2024-06-09T21:57:13.511 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-09T21:57:13.511 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} --archive /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213 -- /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213/orig.config.yaml 2024-06-09T21:57:13.563 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-09T21:57:13.658 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7747213' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 7b7a5d375cf4e4afab96b9d4d229b699ae8304bb owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 178 sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba sleep_before_teardown: 0 subset: 130/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7b7a5d375cf4e4afab96b9d4d229b699ae8304bb syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN6y+NhgWvqMNn46jxYJSwwh1uay2E5HFikiraQ9/xvoyZa2klyf6DR9DRLGMRmqGPyYoiLht8mIwTLzeSoSLyA= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH9KiX+8341IJFTHzUhmLO2XT4W6GXKOhHVTa9U6naLm3Q3gBEQVOJ+qcH64F/Fzjt/cAvoNJZwItgKMr5ULpBY= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJOrlpyGAmpdS0J9617E4IzKUCKBFIAqs28uw4Fw7F/VrJw8qsQJROyAhXyjMfAsNLt8t/CKGzHaulHRTxNWnAM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-06-09_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-09T21:57:13.658 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa; will attempt to use it 2024-06-09T21:57:13.659 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks 2024-06-09T21:57:13.659 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-09T21:57:13.660 INFO:teuthology.task.internal:Checking packages... 2024-06-09T21:57:13.679 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d89accfe3201286f000e065a1faea2a88c7a6ba' 2024-06-09T21:57:13.679 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-09T21:57:13.679 INFO:teuthology.packaging:ref: None 2024-06-09T21:57:13.679 INFO:teuthology.packaging:tag: None 2024-06-09T21:57:13.679 INFO:teuthology.packaging:branch: squid 2024-06-09T21:57:13.679 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T21:57:13.679 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-09T21:57:13.846 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2970.g7b7a5d37 2024-06-09T21:57:13.848 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-09T21:57:13.859 INFO:teuthology.task.internal:no buildpackages task found 2024-06-09T21:57:13.859 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-09T21:57:13.873 INFO:teuthology.task.internal:Saving configuration 2024-06-09T21:57:13.891 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-09T21:57:13.903 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-09T21:57:13.926 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-09 21:52:03.497281', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN6y+NhgWvqMNn46jxYJSwwh1uay2E5HFikiraQ9/xvoyZa2klyf6DR9DRLGMRmqGPyYoiLht8mIwTLzeSoSLyA='} 2024-06-09T21:57:13.942 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-09 21:52:03.499501', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH9KiX+8341IJFTHzUhmLO2XT4W6GXKOhHVTa9U6naLm3Q3gBEQVOJ+qcH64F/Fzjt/cAvoNJZwItgKMr5ULpBY='} 2024-06-09T21:57:13.962 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-09 21:52:03.498505', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJOrlpyGAmpdS0J9617E4IzKUCKBFIAqs28uw4Fw7F/VrJw8qsQJROyAhXyjMfAsNLt8t/CKGzHaulHRTxNWnAM='} 2024-06-09T21:57:13.963 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-09T21:57:13.971 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-06-09T21:57:13.971 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-06-09T21:57:13.971 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-06-09T21:57:13.971 INFO:teuthology.run_tasks:Running task console_log... 2024-06-09T21:57:14.062 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe753e2fbe0>, signals=[15]) 2024-06-09T21:57:14.063 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-09T21:57:14.074 INFO:teuthology.task.internal:Opening connections... 2024-06-09T21:57:14.074 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-06-09T21:57:14.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T21:57:14.149 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-06-09T21:57:14.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T21:57:14.218 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-06-09T21:57:14.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T21:57:14.293 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-09T21:57:14.305 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-06-09T21:57:14.322 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-06-09T21:57:14.323 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-06-09T21:57:14.379 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="9" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="9" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el9" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:LOGO="fedora-logo-icon" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-09T21:57:14.380 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-09T21:57:14.381 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-06-09T21:57:14.403 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-06-09T21:57:14.420 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-06-09T21:57:14.420 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-06-09T21:57:14.478 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-06-09T21:57:14.478 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-06-09T21:57:14.478 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-06-09T21:57:14.478 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-06-09T21:57:14.478 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-06-09T21:57:14.478 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-06-09T21:57:14.478 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-09T21:57:14.478 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-06-09T21:57:14.478 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-06-09T21:57:14.479 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-09T21:57:14.479 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-06-09T21:57:14.479 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-09T21:57:14.479 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-09T21:57:14.479 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-09T21:57:14.479 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-06-09T21:57:14.503 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-06-09T21:57:14.521 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-06-09T21:57:14.521 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-06-09T21:57:14.578 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-06-09T21:57:14.578 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="9" 2024-06-09T21:57:14.578 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-06-09T21:57:14.579 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-06-09T21:57:14.579 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="9" 2024-06-09T21:57:14.579 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el9" 2024-06-09T21:57:14.579 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-09T21:57:14.579 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-06-09T21:57:14.579 INFO:teuthology.orchestra.run.smithi194.stdout:LOGO="fedora-logo-icon" 2024-06-09T21:57:14.579 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-09T21:57:14.579 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-06-09T21:57:14.579 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-09T21:57:14.579 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-09T21:57:14.579 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-09T21:57:14.579 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-06-09T21:57:14.602 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-09T21:57:14.614 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-09T21:57:14.628 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-09T21:57:14.628 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-06-09T21:57:14.631 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-06-09T21:57:14.633 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-06-09T21:57:14.650 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-09T21:57:14.661 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-09T21:57:14.661 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-06-09T21:57:14.688 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-06-09T21:57:14.690 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-06-09T21:57:14.785 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-09T21:57:14.864 INFO:teuthology.run_tasks:Running task kernel... 2024-06-09T21:57:14.885 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-06-09T21:57:14.885 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-06-09T21:57:14.885 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-06-09T21:57:14.886 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-09T21:57:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T21:57:14.902 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-06-09T21:57:14.960 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-437.el9.x86_64 2024-06-09T21:57:14.962 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.14.0-437.el9.x86_64 2024-06-09T21:57:14.962 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum install -y kernel 2024-06-09T21:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.1 MB 00:03 2024-06-09T21:57:21.788 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-06-09T21:57:28.180 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.2 MB 00:02 2024-06-09T21:57:29.854 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 17 kB 00:00 2024-06-09T21:57:31.198 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-06-09T21:57:36.396 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-06-09T21:57:38.343 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-09T21:57:38.344 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-09T21:57:38.441 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-06-09T21:57:38.452 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-06-09T21:57:38.452 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-06-09T21:57:38.452 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-06-09T21:57:38.453 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-06-09T21:57:38.453 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-09T21:57:38.453 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-06-09T21:57:38.453 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-09T21:57:38.453 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-09T21:57:38.453 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-09T21:57:38.453 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T21:57:38.453 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-06-09T21:57:38.453 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-06-09T21:57:38.453 INFO:teuthology.orchestra.run.smithi104.stdout:Install 4 Packages 2024-06-09T21:57:38.453 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T21:57:38.454 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 80 M 2024-06-09T21:57:38.454 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 124 M 2024-06-09T21:57:38.454 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-06-09T21:57:38.804 INFO:teuthology.orchestra.run.smithi104.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:57:38.804 INFO:teuthology.orchestra.run.smithi104.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:57:38.808 INFO:teuthology.orchestra.run.smithi104.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:57:39.022 INFO:teuthology.orchestra.run.smithi104.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 2.9 MB/s | 893 kB 00:00 2024-06-09T21:57:39.572 INFO:teuthology.orchestra.run.smithi104.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-06-09T21:57:40.164 INFO:teuthology.orchestra.run.smithi104.stdout:(3/4): kernel-modules-core-5.14.0-452.el9.x86_6 25 MB/s | 29 MB 00:01 2024-06-09T21:57:41.256 INFO:teuthology.orchestra.run.smithi104.stdout:(4/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 14 MB/s | 35 MB 00:02 2024-06-09T21:57:41.258 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-06-09T21:57:41.259 INFO:teuthology.orchestra.run.smithi104.stdout:Total 29 MB/s | 80 MB 00:02 2024-06-09T21:57:41.955 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-06-09T21:57:42.305 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-06-09T21:57:42.305 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-06-09T21:57:43.270 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-06-09T21:57:43.271 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-06-09T21:57:45.687 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-06-09T21:57:46.370 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-09T21:57:46.463 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-09T21:57:49.106 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-09T21:57:49.254 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-09T21:57:53.949 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-09T21:57:54.075 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:57:58.782 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:58:28.035 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:58:28.074 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:58:28.434 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:58:28.435 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-09T21:58:28.435 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-09T21:58:28.435 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-09T21:58:29.192 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:58:29.192 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T21:58:29.193 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-06-09T21:58:29.193 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-09T21:58:29.193 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-09T21:58:29.193 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-09T21:58:29.193 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-09T21:58:29.193 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T21:58:29.193 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-06-09T21:58:29.585 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-06-09T21:58:29.664 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-09T21:58:29.665 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-09T21:58:29.665 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-09T21:58:29.665 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-09T21:58:29.665 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-09T21:58:29.665 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-06-09T21:58:29.665 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2024-06-09T21:58:29.666 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-06-09T21:58:29.717 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-09T21:58:29.717 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-09T21:58:29.718 DEBUG:teuthology.orchestra.run.smithi104:> sudo rpm -qi grub2-tools 2024-06-09T21:58:29.752 INFO:teuthology.orchestra.run.smithi104.stdout:Name : grub2-tools 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:Epoch : 1 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:Version : 2.06 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:Release : 68.el9 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: x86_64 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:Group : Unspecified 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:Size : 8264254 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:License : GPLv3+ 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-09T21:58:29.753 INFO:teuthology.orchestra.run.smithi104.stdout:Packager : builder@centos.org 2024-06-09T21:58:29.754 INFO:teuthology.orchestra.run.smithi104.stdout:Vendor : CentOS 2024-06-09T21:58:29.754 INFO:teuthology.orchestra.run.smithi104.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-09T21:58:29.754 INFO:teuthology.orchestra.run.smithi104.stdout:Summary : Support tools for GRUB. 2024-06-09T21:58:29.754 INFO:teuthology.orchestra.run.smithi104.stdout:Description : 2024-06-09T21:58:29.754 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T21:58:29.754 INFO:teuthology.orchestra.run.smithi104.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-09T21:58:29.754 INFO:teuthology.orchestra.run.smithi104.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-09T21:58:29.754 INFO:teuthology.orchestra.run.smithi104.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-09T21:58:29.754 INFO:teuthology.orchestra.run.smithi104.stdout:hardware devices. 2024-06-09T21:58:29.754 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T21:58:29.754 INFO:teuthology.orchestra.run.smithi104.stdout:This subpackage provides tools for support of all platforms. 2024-06-09T21:58:29.757 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-09T21:58:29.757 INFO:teuthology.task.kernel:Updating grub on smithi104 to boot 5.14.0-452.el9.x86_64 2024-06-09T21:58:29.757 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-09T21:58:30.349 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-06-09T21:58:31.637 INFO:teuthology.orchestra.run.smithi104.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-09T21:58:31.666 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-06-09T21:58:31.669 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-06-09T21:58:31.684 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.jp9qgz7K7p 2024-06-09T21:58:31.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jp9qgz7K7p 2024-06-09T21:58:31.771 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.jp9qgz7K7p 2024-06-09T21:58:31.863 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.jp9qgz7K7p is 6KB 2024-06-09T21:58:31.885 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.jp9qgz7K7p 2024-06-09T21:58:31.900 DEBUG:teuthology.orchestra.run.smithi104:> sudo /bin/ls /boot/loader/entries || true 2024-06-09T21:58:31.961 INFO:teuthology.orchestra.run.smithi104.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-09T21:58:31.961 INFO:teuthology.orchestra.run.smithi104.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-09T21:58:31.961 INFO:teuthology.orchestra.run.smithi104.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-09T21:58:31.961 INFO:teuthology.orchestra.run.smithi104.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-09T21:58:31.962 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-09T21:58:32.097 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-06-09T21:59:02.100 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-06-09T21:59:02.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-06-09T21:59:02.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T22:00:02.105 DEBUG:teuthology.orchestra.remote:timed out 2024-06-09T22:00:11.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-06-09T22:00:11.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T22:00:44.469 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-06-09T22:00:44.958 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-06-09T22:00:44.958 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-452.el9.x86_64"... 2024-06-09T22:00:44.959 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-06-09T22:00:44.973 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-452.el9.x86_64 2024-06-09T22:00:44.973 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-09T22:00:44.973 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-09T22:00:44.973 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-09T22:00:45.975 DEBUG:teuthology.parallel:result is None 2024-06-09T22:00:45.975 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-09T22:00:45.983 INFO:teuthology.task.internal:Creating test directory... 2024-06-09T22:00:45.984 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-09T22:00:45.986 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-09T22:00:45.989 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-09T22:00:46.007 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-09T22:00:46.016 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-09T22:00:46.031 INFO:teuthology.task.internal:Creating archive directory... 2024-06-09T22:00:46.031 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-09T22:00:46.041 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-09T22:00:46.047 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-09T22:00:46.122 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-09T22:00:46.129 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-09T22:00:46.129 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-09T22:00:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:00:46.146 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-09T22:00:46.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:00:46.162 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-09T22:00:46.165 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-09T22:00:46.188 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-09T22:00:46.221 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-09T22:00:46.236 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-09T22:00:46.239 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-09T22:00:46.255 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-09T22:00:46.255 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-09T22:00:46.275 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-09T22:00:46.277 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-09T22:00:46.286 INFO:teuthology.task.internal:Configuring sudo... 2024-06-09T22:00:46.286 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-09T22:00:46.288 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-09T22:00:46.299 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-09T22:00:46.325 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-09T22:00:46.334 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-09T22:00:46.334 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-09T22:00:46.354 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-09T22:00:46.366 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-09T22:00:46.384 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-09T22:00:46.449 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-09T22:00:46.491 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-09T22:00:46.572 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-09T22:00:46.613 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:00:46.613 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-09T22:00:46.710 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-09T22:00:46.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-09T22:00:46.882 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-09T22:00:46.966 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-09T22:00:47.007 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-09T22:00:47.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-09T22:00:47.072 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-09T22:00:47.118 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-09T22:00:47.173 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-09T22:00:47.257 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-09T22:00:47.299 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-06-09T22:00:47.299 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-09T22:00:47.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-06-09T22:00:47.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-06-09T22:00:47.370 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-06-09T22:00:47.405 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-09T22:00:47.409 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-09T22:00:47.440 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-09T22:00:47.828 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-09T22:00:47.837 INFO:teuthology.task.internal:Starting timer... 2024-06-09T22:00:47.837 INFO:teuthology.run_tasks:Running task pcp... 2024-06-09T22:00:47.850 INFO:teuthology.run_tasks:Running task selinux... 2024-06-09T22:00:47.860 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-09T22:00:47.860 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-06-09T22:00:47.917 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-06-09T22:00:47.919 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-06-09T22:00:47.990 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-06-09T22:00:47.992 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-06-09T22:00:48.059 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-06-09T22:00:48.060 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-09T22:00:48.061 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-06-09T22:00:48.108 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-06-09T22:00:48.108 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-06-09T22:00:48.129 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-06-09T22:00:48.129 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-06-09T22:00:48.153 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-06-09T22:00:48.153 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi104.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-06-09T22:00:48.153 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-09T22:00:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:00:48.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-09T22:00:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:00:48.207 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-09T22:00:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:00:48.233 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-09T22:00:48.233 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-06-09T22:00:48.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-06-09T22:00:48.292 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-06-09T22:00:48.316 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-09T22:00:48.326 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-09T22:00:48.329 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-09T22:00:48.473 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-09T22:00:48.484 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-09T22:00:48.485 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi104.front.sepia.ceph.com,smithi153.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-09T22:05:32.234 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi104.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-06-09T22:05:32.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-06-09T22:05:32.237 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T22:05:32.312 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-06-09T22:05:32.397 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-06-09T22:05:32.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-09T22:05:32.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T22:05:32.487 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-06-09T22:05:32.565 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-09T22:05:32.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-06-09T22:05:32.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T22:05:32.642 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-06-09T22:05:32.726 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-06-09T22:05:32.727 INFO:teuthology.run_tasks:Running task clock... 2024-06-09T22:05:32.737 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-09T22:05:32.738 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-09T22:05:32.738 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-09T22:05:32.740 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-09T22:05:32.741 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-09T22:05:32.743 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-09T22:05:32.743 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-09T22:05:32.768 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-09T22:05:32.770 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-09T22:05:32.782 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-09T22:05:32.783 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-09T22:05:32.796 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-09T22:05:32.805 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-06-09T22:05:32.807 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-06-09T22:05:32.810 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-09T22:05:32.816 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-06-09T22:05:32.817 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-06-09T22:05:32.830 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-09T22:05:32.830 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-09T22:05:32.833 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-06-09T22:05:32.843 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-09T22:05:32.844 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-06-09T22:05:32.844 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-09T22:05:32.857 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-09T22:05:32.870 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-09T22:05:32.899 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-06-09T22:05:32.901 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-06-09T22:05:32.904 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-09T22:05:32.904 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-06-09T22:05:32.904 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:05:32.904 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:05:32.904 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:05:32.904 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:05:32.906 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-09T22:05:32.906 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-06-09T22:05:32.906 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:05:32.906 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:05:32.906 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:05:32.906 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:05:32.930 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-06-09T22:05:32.930 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-09T22:05:32.931 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-06-09T22:05:32.931 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:05:32.931 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:05:32.931 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:05:32.931 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:05:32.932 INFO:teuthology.run_tasks:Running task install... 2024-06-09T22:05:32.940 DEBUG:teuthology.task.install:project ceph 2024-06-09T22:05:32.940 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}} 2024-06-09T22:05:32.941 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-09T22:05:32.941 INFO:teuthology.task.install:Using flavor: default 2024-06-09T22:05:32.946 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-09T22:05:32.946 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-06-09T22:05:32.947 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-09T22:05:32.947 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T22:05:32.949 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-09T22:05:32.949 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T22:05:32.950 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-09T22:05:32.951 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T22:05:33.101 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/ 2024-06-09T22:05:33.101 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2946.g3d89accf 2024-06-09T22:05:33.105 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/ 2024-06-09T22:05:33.105 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2946.g3d89accf 2024-06-09T22:05:33.206 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/ 2024-06-09T22:05:33.207 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2946.g3d89accf 2024-06-09T22:05:33.213 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-09T22:05:33.213 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:05:33.213 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-09T22:05:33.239 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-09T22:05:33.239 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d89accfe3201286f000e065a1faea2a88c7a6ba/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-09T22:05:33.271 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-09T22:05:33.272 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-06-09T22:05:33.272 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-09T22:05:33.296 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-09T22:05:33.296 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d89accfe3201286f000e065a1faea2a88c7a6ba/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-09T22:05:33.305 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-09T22:05:33.317 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-09T22:05:33.317 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-09T22:05:33.317 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-09T22:05:33.342 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-09T22:05:33.342 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d89accfe3201286f000e065a1faea2a88c7a6ba/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-09T22:05:33.361 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-09T22:05:33.379 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-09T22:05:33.405 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-09T22:05:33.433 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-09T22:05:33.443 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2024-06-09T22:05:33.445 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2024-06-09T22:05:33.478 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-09T22:05:33.495 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-06-09T22:05:33.497 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-06-09T22:05:33.541 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-06-09T22:05:33.543 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-06-09T22:05:33.682 INFO:teuthology.orchestra.run.smithi104.stdout:45 files removed 2024-06-09T22:05:33.708 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-09T22:05:33.718 INFO:teuthology.orchestra.run.smithi194.stdout:45 files removed 2024-06-09T22:05:33.745 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-09T22:05:33.770 INFO:teuthology.orchestra.run.smithi153.stdout:45 files removed 2024-06-09T22:05:33.797 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-09T22:05:34.468 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 180 kB/s | 83 kB 00:00 2024-06-09T22:05:34.494 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 162 kB/s | 83 kB 00:00 2024-06-09T22:05:34.582 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 159 kB/s | 83 kB 00:00 2024-06-09T22:05:34.941 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 36 kB/s | 11 kB 00:00 2024-06-09T22:05:34.945 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 27 kB/s | 11 kB 00:00 2024-06-09T22:05:34.962 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-06-09T22:05:35.252 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 6.9 kB/s | 1.9 kB 00:00 2024-06-09T22:05:35.317 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-06-09T22:05:35.362 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-06-09T22:05:36.348 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 8.6 MB/s | 8.1 MB 00:00 2024-06-09T22:05:37.277 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.1 MB 00:01 2024-06-09T22:05:37.455 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.1 MB 00:02 2024-06-09T22:05:38.499 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-06-09T22:05:39.381 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-06-09T22:05:39.661 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-06-09T22:05:43.523 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-06-09T22:05:44.878 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.2 MB 00:01 2024-06-09T22:05:46.123 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 17 kB 00:01 2024-06-09T22:05:47.428 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 17 kB 00:01 2024-06-09T22:05:47.500 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-06-09T22:05:48.456 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-06-09T22:05:49.265 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 881 kB/s | 6.2 MB 00:07 2024-06-09T22:05:50.948 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 17 kB 00:00 2024-06-09T22:05:51.942 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-06-09T22:05:52.864 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 7.1 kB/s | 1.7 kB 00:00 2024-06-09T22:05:53.703 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2024-06-09T22:05:55.132 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-09T22:05:55.132 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-09T22:05:55.143 INFO:teuthology.orchestra.run.smithi194.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-09T22:05:55.144 INFO:teuthology.orchestra.run.smithi194.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-09T22:05:55.145 INFO:teuthology.orchestra.run.smithi194.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-09T22:05:55.145 INFO:teuthology.orchestra.run.smithi194.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-09T22:05:55.147 INFO:teuthology.orchestra.run.smithi194.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-09T22:05:55.149 INFO:teuthology.orchestra.run.smithi194.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-09T22:05:55.150 INFO:teuthology.orchestra.run.smithi194.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-09T22:05:55.211 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-06-09T22:05:55.231 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-06-09T22:05:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-06-09T22:05:55.232 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-06-09T22:05:55.232 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-06-09T22:05:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-09T22:05:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.6 k 2024-06-09T22:05:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-09T22:05:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 890 k 2024-06-09T22:05:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 145 k 2024-06-09T22:05:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-09T22:05:55.232 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 142 k 2024-06-09T22:05:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 3.5 M 2024-06-09T22:05:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 7.4 M 2024-06-09T22:05:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 49 k 2024-06-09T22:05:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 10 M 2024-06-09T22:05:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 48 M 2024-06-09T22:05:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 264 k 2024-06-09T22:05:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 763 k 2024-06-09T22:05:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 221 k 2024-06-09T22:05:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-shell x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 27 k 2024-06-09T22:05:55.233 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-top noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 18 k 2024-06-09T22:05:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-09T22:05:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-09T22:05:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-09T22:05:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 32 k 2024-06-09T22:05:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 747 k 2024-06-09T22:05:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 127 k 2024-06-09T22:05:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-06-09T22:05:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-09T22:05:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-06-09T22:05:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-09T22:05:55.234 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-09T22:05:55.235 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-09T22:05:55.235 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 162 k 2024-06-09T22:05:55.235 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 321 k 2024-06-09T22:05:55.235 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 300 k 2024-06-09T22:05:55.235 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 k 2024-06-09T22:05:55.235 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 85 k 2024-06-09T22:05:55.235 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-09T22:05:55.235 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 171 k 2024-06-09T22:05:55.235 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-06-09T22:05:55.236 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-06-09T22:05:55.236 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager x86_64 1:1.47.91-1.el9 baseos 2.3 M 2024-06-09T22:05:55.236 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-libnm x86_64 1:1.47.91-1.el9 baseos 1.8 M 2024-06-09T22:05:55.236 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-team x86_64 1:1.47.91-1.el9 baseos 41 k 2024-06-09T22:05:55.237 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-tui x86_64 1:1.47.91-1.el9 baseos 248 k 2024-06-09T22:05:55.237 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-09T22:05:55.237 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-09T22:05:55.237 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-09T22:05:55.237 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.4 M 2024-06-09T22:05:55.237 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-09T22:05:55.238 INFO:teuthology.orchestra.run.smithi194.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-06-09T22:05:55.238 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-06-09T22:05:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-06-09T22:05:55.238 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-09T22:05:55.238 INFO:teuthology.orchestra.run.smithi194.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-09T22:05:55.238 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-09T22:05:55.239 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 21 M 2024-06-09T22:05:55.239 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 32 k 2024-06-09T22:05:55.239 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 2.4 M 2024-06-09T22:05:55.239 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 249 k 2024-06-09T22:05:55.239 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 4.7 M 2024-06-09T22:05:55.239 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 17 M 2024-06-09T22:05:55.240 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 16 k 2024-06-09T22:05:55.240 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 25 k 2024-06-09T22:05:55.240 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-09T22:05:55.240 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-09T22:05:55.240 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-09T22:05:55.241 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-09T22:05:55.241 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-09T22:05:55.241 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-09T22:05:55.241 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-09T22:05:55.241 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-09T22:05:55.241 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 163 k 2024-06-09T22:05:55.242 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-09T22:05:55.242 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-09T22:05:55.242 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-09T22:05:55.242 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-09T22:05:55.242 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-09T22:05:55.242 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 501 k 2024-06-09T22:05:55.243 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-09T22:05:55.243 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.3 M 2024-06-09T22:05:55.243 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-09T22:05:55.244 INFO:teuthology.orchestra.run.smithi194.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-09T22:05:55.244 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-09T22:05:55.244 INFO:teuthology.orchestra.run.smithi194.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-09T22:05:55.244 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-09T22:05:55.244 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-09T22:05:55.245 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-09T22:05:55.245 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-09T22:05:55.245 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-09T22:05:55.245 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-09T22:05:55.245 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-09T22:05:55.246 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-09T22:05:55.246 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-09T22:05:55.246 INFO:teuthology.orchestra.run.smithi194.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-09T22:05:55.246 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-09T22:05:55.246 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-09T22:05:55.247 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-09T22:05:55.247 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 k 2024-06-09T22:05:55.247 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 135 k 2024-06-09T22:05:55.247 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-09T22:05:55.247 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-09T22:05:55.247 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-09T22:05:55.248 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-09T22:05:55.248 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-09T22:05:55.248 INFO:teuthology.orchestra.run.smithi194.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-09T22:05:55.248 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-09T22:05:55.248 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-09T22:05:55.249 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-09T22:05:55.249 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-09T22:05:55.249 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-09T22:05:55.249 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-09T22:05:55.250 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-09T22:05:55.250 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-09T22:05:55.250 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-09T22:05:55.250 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-09T22:05:55.250 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-09T22:05:55.250 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-09T22:05:55.251 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-09T22:05:55.251 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-09T22:05:55.251 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-09T22:05:55.251 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-09T22:05:55.251 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-09T22:05:55.252 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-09T22:05:55.252 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-09T22:05:55.252 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-09T22:05:55.252 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-09T22:05:55.252 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-09T22:05:55.253 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-09T22:05:55.253 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-09T22:05:55.253 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-09T22:05:55.253 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-09T22:05:55.253 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-09T22:05:55.253 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-09T22:05:55.254 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-09T22:05:55.254 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-09T22:05:55.254 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-09T22:05:55.254 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-09T22:05:55.254 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-09T22:05:55.255 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-09T22:05:55.255 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-09T22:05:55.255 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-09T22:05:55.255 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-09T22:05:55.255 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-09T22:05:55.255 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-09T22:05:55.256 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-09T22:05:55.256 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-09T22:05:55.256 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-09T22:05:55.256 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-09T22:05:55.256 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-09T22:05:55.257 INFO:teuthology.orchestra.run.smithi194.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-09T22:05:55.257 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-09T22:05:55.257 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-09T22:05:55.257 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-09T22:05:55.257 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-09T22:05:55.257 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-09T22:05:55.258 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-09T22:05:55.258 INFO:teuthology.orchestra.run.smithi194.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-09T22:05:55.258 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-09T22:05:55.258 INFO:teuthology.orchestra.run.smithi194.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-09T22:05:55.258 INFO:teuthology.orchestra.run.smithi194.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-09T22:05:55.259 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-09T22:05:55.259 INFO:teuthology.orchestra.run.smithi194.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-09T22:05:55.259 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-06-09T22:05:55.259 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-09T22:05:55.259 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:05:55.259 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-06-09T22:05:55.260 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-06-09T22:05:55.260 INFO:teuthology.orchestra.run.smithi194.stdout:Install 148 Packages 2024-06-09T22:05:55.260 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 11 Packages 2024-06-09T22:05:55.260 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:05:55.261 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 225 M 2024-06-09T22:05:55.261 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-06-09T22:05:55.686 INFO:teuthology.orchestra.run.smithi194.stdout:(1/159): ceph-19.0.0-2946.g3d89accf.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-06-09T22:05:55.971 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-09T22:05:55.971 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-09T22:05:55.982 INFO:teuthology.orchestra.run.smithi153.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-09T22:05:55.983 INFO:teuthology.orchestra.run.smithi153.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-09T22:05:55.984 INFO:teuthology.orchestra.run.smithi153.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-09T22:05:55.984 INFO:teuthology.orchestra.run.smithi153.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-09T22:05:55.986 INFO:teuthology.orchestra.run.smithi153.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-09T22:05:55.988 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-09T22:05:55.989 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-09T22:05:56.052 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-09T22:05:56.076 INFO:teuthology.orchestra.run.smithi194.stdout:(2/159): ceph-fuse-19.0.0-2946.g3d89accf.el9.x8 2.2 MB/s | 890 kB 00:00 2024-06-09T22:05:56.080 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-06-09T22:05:56.080 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-06-09T22:05:56.080 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-06-09T22:05:56.080 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-09T22:05:56.080 INFO:teuthology.orchestra.run.smithi153.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-09T22:05:56.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.6 k 2024-06-09T22:05:56.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-09T22:05:56.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 890 k 2024-06-09T22:05:56.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 145 k 2024-06-09T22:05:56.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-09T22:05:56.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 142 k 2024-06-09T22:05:56.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 3.5 M 2024-06-09T22:05:56.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 7.4 M 2024-06-09T22:05:56.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 49 k 2024-06-09T22:05:56.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 10 M 2024-06-09T22:05:56.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 48 M 2024-06-09T22:05:56.081 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 264 k 2024-06-09T22:05:56.082 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 763 k 2024-06-09T22:05:56.082 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 221 k 2024-06-09T22:05:56.082 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 27 k 2024-06-09T22:05:56.082 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 18 k 2024-06-09T22:05:56.082 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-09T22:05:56.082 INFO:teuthology.orchestra.run.smithi153.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-09T22:05:56.082 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-09T22:05:56.082 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 32 k 2024-06-09T22:05:56.082 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 747 k 2024-06-09T22:05:56.082 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 127 k 2024-06-09T22:05:56.082 INFO:teuthology.orchestra.run.smithi153.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-06-09T22:05:56.083 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-09T22:05:56.083 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-06-09T22:05:56.083 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-09T22:05:56.083 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-09T22:05:56.083 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-09T22:05:56.083 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 162 k 2024-06-09T22:05:56.083 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 321 k 2024-06-09T22:05:56.083 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 300 k 2024-06-09T22:05:56.083 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 k 2024-06-09T22:05:56.083 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 85 k 2024-06-09T22:05:56.083 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-09T22:05:56.083 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 171 k 2024-06-09T22:05:56.084 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-06-09T22:05:56.084 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-06-09T22:05:56.084 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager x86_64 1:1.47.91-1.el9 baseos 2.3 M 2024-06-09T22:05:56.084 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-libnm x86_64 1:1.47.91-1.el9 baseos 1.8 M 2024-06-09T22:05:56.084 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-team x86_64 1:1.47.91-1.el9 baseos 41 k 2024-06-09T22:05:56.084 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-tui x86_64 1:1.47.91-1.el9 baseos 248 k 2024-06-09T22:05:56.084 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-09T22:05:56.084 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-09T22:05:56.084 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-09T22:05:56.084 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.4 M 2024-06-09T22:05:56.084 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-09T22:05:56.084 INFO:teuthology.orchestra.run.smithi153.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 21 M 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 32 k 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 2.4 M 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 249 k 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 4.7 M 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 17 M 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 16 k 2024-06-09T22:05:56.085 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 25 k 2024-06-09T22:05:56.086 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-09T22:05:56.086 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-09T22:05:56.086 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-09T22:05:56.086 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-09T22:05:56.086 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-09T22:05:56.086 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-09T22:05:56.086 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-09T22:05:56.086 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-09T22:05:56.086 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 163 k 2024-06-09T22:05:56.086 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-09T22:05:56.086 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-09T22:05:56.087 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-09T22:05:56.087 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-09T22:05:56.087 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-09T22:05:56.087 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 501 k 2024-06-09T22:05:56.087 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-09T22:05:56.087 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.3 M 2024-06-09T22:05:56.087 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-09T22:05:56.087 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-09T22:05:56.088 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-09T22:05:56.088 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-09T22:05:56.088 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-09T22:05:56.088 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-09T22:05:56.088 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-09T22:05:56.088 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-09T22:05:56.088 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-09T22:05:56.088 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-09T22:05:56.088 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-09T22:05:56.088 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-09T22:05:56.088 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-09T22:05:56.089 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-09T22:05:56.089 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-09T22:05:56.089 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-09T22:05:56.089 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-09T22:05:56.089 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 k 2024-06-09T22:05:56.089 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 135 k 2024-06-09T22:05:56.089 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-09T22:05:56.089 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-09T22:05:56.089 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-09T22:05:56.089 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-09T22:05:56.089 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-09T22:05:56.089 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-09T22:05:56.090 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-09T22:05:56.090 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-09T22:05:56.090 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-09T22:05:56.090 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-09T22:05:56.090 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-09T22:05:56.090 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-09T22:05:56.090 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-09T22:05:56.090 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-09T22:05:56.090 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-09T22:05:56.090 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-09T22:05:56.090 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-09T22:05:56.091 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-09T22:05:56.094 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-09T22:05:56.094 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-09T22:05:56.094 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-09T22:05:56.094 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-09T22:05:56.094 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-09T22:05:56.094 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-09T22:05:56.094 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-09T22:05:56.095 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-09T22:05:56.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-09T22:05:56.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-09T22:05:56.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-09T22:05:56.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-09T22:05:56.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-09T22:05:56.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-09T22:05:56.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-09T22:05:56.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-09T22:05:56.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-09T22:05:56.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-09T22:05:56.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-09T22:05:56.097 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-09T22:05:56.098 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-09T22:05:56.098 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-06-09T22:05:56.098 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-09T22:05:56.098 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:05:56.098 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-09T22:05:56.098 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-06-09T22:05:56.098 INFO:teuthology.orchestra.run.smithi153.stdout:Install 148 Packages 2024-06-09T22:05:56.098 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 11 Packages 2024-06-09T22:05:56.098 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:05:56.099 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 225 M 2024-06-09T22:05:56.099 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-09T22:05:56.149 INFO:teuthology.orchestra.run.smithi194.stdout:(3/159): ceph-immutable-object-cache-19.0.0-294 1.9 MB/s | 145 kB 00:00 2024-06-09T22:05:56.225 INFO:teuthology.orchestra.run.smithi194.stdout:(4/159): ceph-base-19.0.0-2946.g3d89accf.el9.x8 7.0 MB/s | 5.4 MB 00:00 2024-06-09T22:05:56.408 INFO:teuthology.orchestra.run.smithi194.stdout:(5/159): ceph-mgr-19.0.0-2946.g3d89accf.el9.x86 8.2 MB/s | 1.5 MB 00:00 2024-06-09T22:05:56.708 INFO:teuthology.orchestra.run.smithi194.stdout:(6/159): ceph-mds-19.0.0-2946.g3d89accf.el9.x86 4.2 MB/s | 2.4 MB 00:00 2024-06-09T22:05:56.884 INFO:teuthology.orchestra.run.smithi194.stdout:(7/159): ceph-mon-19.0.0-2946.g3d89accf.el9.x86 9.9 MB/s | 4.7 MB 00:00 2024-06-09T22:05:57.056 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-06-09T22:05:57.430 INFO:teuthology.orchestra.run.smithi153.stdout:(1/159): ceph-19.0.0-2946.g3d89accf.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-06-09T22:05:57.484 INFO:teuthology.orchestra.run.smithi194.stdout:(8/159): ceph-common-19.0.0-2946.g3d89accf.el9. 10 MB/s | 21 MB 00:02 2024-06-09T22:05:57.551 INFO:teuthology.orchestra.run.smithi194.stdout:(9/159): ceph-selinux-19.0.0-2946.g3d89accf.el9 379 kB/s | 25 kB 00:00 2024-06-09T22:05:57.921 INFO:teuthology.orchestra.run.smithi153.stdout:(2/159): ceph-fuse-19.0.0-2946.g3d89accf.el9.x8 1.8 MB/s | 890 kB 00:00 2024-06-09T22:05:57.996 INFO:teuthology.orchestra.run.smithi153.stdout:(3/159): ceph-immutable-object-cache-19.0.0-294 1.9 MB/s | 145 kB 00:00 2024-06-09T22:05:58.001 INFO:teuthology.orchestra.run.smithi194.stdout:(10/159): ceph-radosgw-19.0.0-2946.g3d89accf.el 9.2 MB/s | 10 MB 00:01 2024-06-09T22:05:58.085 INFO:teuthology.orchestra.run.smithi194.stdout:(11/159): cephfs-mirror-19.0.0-2946.g3d89accf.e 2.6 MB/s | 221 kB 00:00 2024-06-09T22:05:58.168 INFO:teuthology.orchestra.run.smithi194.stdout:(12/159): cephfs-shell-19.0.0-2946.g3d89accf.el 325 kB/s | 27 kB 00:00 2024-06-09T22:05:58.260 INFO:teuthology.orchestra.run.smithi194.stdout:(13/159): libcephfs-devel-19.0.0-2946.g3d89accf 346 kB/s | 32 kB 00:00 2024-06-09T22:05:58.660 INFO:teuthology.orchestra.run.smithi194.stdout:(14/159): libcephfs2-19.0.0-2946.g3d89accf.el9. 1.8 MB/s | 747 kB 00:00 2024-06-09T22:05:58.672 INFO:teuthology.orchestra.run.smithi153.stdout:(4/159): ceph-mds-19.0.0-2946.g3d89accf.el9.x86 3.5 MB/s | 2.4 MB 00:00 2024-06-09T22:05:58.794 INFO:teuthology.orchestra.run.smithi194.stdout:(15/159): libcephsqlite-19.0.0-2946.g3d89accf.e 1.2 MB/s | 163 kB 00:00 2024-06-09T22:05:58.936 INFO:teuthology.orchestra.run.smithi194.stdout:(16/159): librados-devel-19.0.0-2946.g3d89accf. 893 kB/s | 127 kB 00:00 2024-06-09T22:05:58.972 INFO:teuthology.orchestra.run.smithi153.stdout:(5/159): ceph-mgr-19.0.0-2946.g3d89accf.el9.x86 5.0 MB/s | 1.5 MB 00:00 2024-06-09T22:05:59.095 INFO:teuthology.orchestra.run.smithi194.stdout:(17/159): ceph-osd-19.0.0-2946.g3d89accf.el9.x8 7.0 MB/s | 17 MB 00:02 2024-06-09T22:05:59.287 INFO:teuthology.orchestra.run.smithi194.stdout:(18/159): libradosstriper1-19.0.0-2946.g3d89acc 1.4 MB/s | 501 kB 00:00 2024-06-09T22:05:59.374 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-09T22:05:59.374 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-09T22:05:59.386 INFO:teuthology.orchestra.run.smithi104.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-09T22:05:59.386 INFO:teuthology.orchestra.run.smithi104.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-09T22:05:59.387 INFO:teuthology.orchestra.run.smithi194.stdout:(19/159): python3-ceph-argparse-19.0.0-2946.g3d 452 kB/s | 45 kB 00:00 2024-06-09T22:05:59.387 INFO:teuthology.orchestra.run.smithi104.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-09T22:05:59.388 INFO:teuthology.orchestra.run.smithi104.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-09T22:05:59.390 INFO:teuthology.orchestra.run.smithi104.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-09T22:05:59.392 INFO:teuthology.orchestra.run.smithi104.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-09T22:05:59.393 INFO:teuthology.orchestra.run.smithi104.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-09T22:05:59.454 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-06-09T22:05:59.465 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-06-09T22:05:59.465 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-06-09T22:05:59.465 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-06-09T22:05:59.465 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-06-09T22:05:59.465 INFO:teuthology.orchestra.run.smithi104.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-09T22:05:59.466 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.6 k 2024-06-09T22:05:59.466 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-09T22:05:59.466 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 890 k 2024-06-09T22:05:59.466 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 145 k 2024-06-09T22:05:59.466 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-09T22:05:59.466 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 142 k 2024-06-09T22:05:59.466 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 3.5 M 2024-06-09T22:05:59.466 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 7.4 M 2024-06-09T22:05:59.466 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 49 k 2024-06-09T22:05:59.466 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 10 M 2024-06-09T22:05:59.467 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 48 M 2024-06-09T22:05:59.467 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 264 k 2024-06-09T22:05:59.467 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 763 k 2024-06-09T22:05:59.467 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 221 k 2024-06-09T22:05:59.467 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-shell x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 27 k 2024-06-09T22:05:59.467 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-top noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 18 k 2024-06-09T22:05:59.467 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-09T22:05:59.467 INFO:teuthology.orchestra.run.smithi104.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-09T22:05:59.468 INFO:teuthology.orchestra.run.smithi104.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-09T22:05:59.468 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 32 k 2024-06-09T22:05:59.468 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 747 k 2024-06-09T22:05:59.468 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 127 k 2024-06-09T22:05:59.468 INFO:teuthology.orchestra.run.smithi104.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-06-09T22:05:59.468 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-09T22:05:59.468 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-06-09T22:05:59.468 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-09T22:05:59.468 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-09T22:05:59.468 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-09T22:05:59.468 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 162 k 2024-06-09T22:05:59.468 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 321 k 2024-06-09T22:05:59.469 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 300 k 2024-06-09T22:05:59.469 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 k 2024-06-09T22:05:59.469 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 85 k 2024-06-09T22:05:59.469 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-09T22:05:59.469 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 171 k 2024-06-09T22:05:59.469 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-06-09T22:05:59.469 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2024-06-09T22:05:59.469 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager x86_64 1:1.47.91-1.el9 baseos 2.3 M 2024-06-09T22:05:59.470 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-libnm x86_64 1:1.47.91-1.el9 baseos 1.8 M 2024-06-09T22:05:59.470 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-team x86_64 1:1.47.91-1.el9 baseos 41 k 2024-06-09T22:05:59.470 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-tui x86_64 1:1.47.91-1.el9 baseos 248 k 2024-06-09T22:05:59.470 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-09T22:05:59.470 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-09T22:05:59.470 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-09T22:05:59.470 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.4 M 2024-06-09T22:05:59.470 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-09T22:05:59.470 INFO:teuthology.orchestra.run.smithi104.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-06-09T22:05:59.470 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-06-09T22:05:59.470 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-06-09T22:05:59.470 INFO:teuthology.orchestra.run.smithi104.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-09T22:05:59.471 INFO:teuthology.orchestra.run.smithi104.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-09T22:05:59.471 INFO:teuthology.orchestra.run.smithi104.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-09T22:05:59.471 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 21 M 2024-06-09T22:05:59.471 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 32 k 2024-06-09T22:05:59.471 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 2.4 M 2024-06-09T22:05:59.471 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 249 k 2024-06-09T22:05:59.471 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 4.7 M 2024-06-09T22:05:59.471 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 17 M 2024-06-09T22:05:59.471 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 16 k 2024-06-09T22:05:59.471 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 25 k 2024-06-09T22:05:59.471 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-09T22:05:59.472 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-09T22:05:59.472 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-09T22:05:59.472 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-09T22:05:59.472 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-09T22:05:59.472 INFO:teuthology.orchestra.run.smithi104.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-09T22:05:59.472 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-09T22:05:59.472 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-09T22:05:59.472 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 163 k 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 501 k 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.3 M 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-09T22:05:59.473 INFO:teuthology.orchestra.run.smithi104.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-09T22:05:59.474 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-09T22:05:59.474 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-09T22:05:59.474 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-09T22:05:59.474 INFO:teuthology.orchestra.run.smithi104.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-09T22:05:59.474 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-09T22:05:59.474 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-09T22:05:59.474 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-09T22:05:59.474 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-09T22:05:59.474 INFO:teuthology.orchestra.run.smithi104.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-09T22:05:59.474 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-09T22:05:59.474 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-09T22:05:59.474 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-09T22:05:59.475 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 k 2024-06-09T22:05:59.475 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 135 k 2024-06-09T22:05:59.475 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-09T22:05:59.475 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-09T22:05:59.475 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-09T22:05:59.475 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-09T22:05:59.475 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-09T22:05:59.475 INFO:teuthology.orchestra.run.smithi104.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-09T22:05:59.475 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-09T22:05:59.475 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-09T22:05:59.475 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-09T22:05:59.475 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-09T22:05:59.476 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-09T22:05:59.477 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-09T22:05:59.477 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-09T22:05:59.477 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-09T22:05:59.477 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-09T22:05:59.477 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-09T22:05:59.477 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-09T22:05:59.477 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-09T22:05:59.477 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-09T22:05:59.477 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-09T22:05:59.477 INFO:teuthology.orchestra.run.smithi104.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-09T22:05:59.477 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-09T22:05:59.478 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-09T22:05:59.479 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-09T22:05:59.479 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-09T22:05:59.479 INFO:teuthology.orchestra.run.smithi104.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-09T22:05:59.479 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-09T22:05:59.479 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-09T22:05:59.479 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-09T22:05:59.479 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-09T22:05:59.479 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-09T22:05:59.479 INFO:teuthology.orchestra.run.smithi104.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-09T22:05:59.479 INFO:teuthology.orchestra.run.smithi104.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-09T22:05:59.480 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-09T22:05:59.480 INFO:teuthology.orchestra.run.smithi104.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-09T22:05:59.480 INFO:teuthology.orchestra.run.smithi104.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-09T22:05:59.480 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-09T22:05:59.480 INFO:teuthology.orchestra.run.smithi104.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-09T22:05:59.480 INFO:teuthology.orchestra.run.smithi104.stdout:Installing weak dependencies: 2024-06-09T22:05:59.480 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-09T22:05:59.480 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:05:59.480 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-06-09T22:05:59.480 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-06-09T22:05:59.480 INFO:teuthology.orchestra.run.smithi104.stdout:Install 148 Packages 2024-06-09T22:05:59.480 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 11 Packages 2024-06-09T22:05:59.481 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:05:59.481 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 225 M 2024-06-09T22:05:59.481 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-06-09T22:05:59.554 INFO:teuthology.orchestra.run.smithi194.stdout:(20/159): python3-ceph-common-19.0.0-2946.g3d89 811 kB/s | 135 kB 00:00 2024-06-09T22:05:59.589 INFO:teuthology.orchestra.run.smithi153.stdout:(6/159): ceph-mon-19.0.0-2946.g3d89accf.el9.x86 7.6 MB/s | 4.7 MB 00:00 2024-06-09T22:05:59.704 INFO:teuthology.orchestra.run.smithi194.stdout:(21/159): python3-cephfs-19.0.0-2946.g3d89accf. 1.1 MB/s | 162 kB 00:00 2024-06-09T22:05:59.764 INFO:teuthology.orchestra.run.smithi153.stdout:(7/159): ceph-base-19.0.0-2946.g3d89accf.el9.x8 1.9 MB/s | 5.4 MB 00:02 2024-06-09T22:05:59.788 INFO:teuthology.orchestra.run.smithi194.stdout:(22/159): librgw2-19.0.0-2946.g3d89accf.el9.x86 7.7 MB/s | 5.3 MB 00:00 2024-06-09T22:05:59.896 INFO:teuthology.orchestra.run.smithi194.stdout:(23/159): python3-rbd-19.0.0-2946.g3d89accf.el9 2.7 MB/s | 300 kB 00:00 2024-06-09T22:05:59.938 INFO:teuthology.orchestra.run.smithi194.stdout:(24/159): python3-rados-19.0.0-2946.g3d89accf.e 1.3 MB/s | 321 kB 00:00 2024-06-09T22:05:59.997 INFO:teuthology.orchestra.run.smithi194.stdout:(25/159): python3-rgw-19.0.0-2946.g3d89accf.el9 988 kB/s | 99 kB 00:00 2024-06-09T22:06:00.030 INFO:teuthology.orchestra.run.smithi194.stdout:(26/159): rbd-fuse-19.0.0-2946.g3d89accf.el9.x8 925 kB/s | 85 kB 00:00 2024-06-09T22:06:00.381 INFO:teuthology.orchestra.run.smithi194.stdout:(27/159): rbd-nbd-19.0.0-2946.g3d89accf.el9.x86 488 kB/s | 171 kB 00:00 2024-06-09T22:06:00.465 INFO:teuthology.orchestra.run.smithi194.stdout:(28/159): ceph-grafana-dashboards-19.0.0-2946.g 381 kB/s | 32 kB 00:00 2024-06-09T22:06:00.523 INFO:teuthology.orchestra.run.smithi104.stdout:(1/159): ceph-19.0.0-2946.g3d89accf.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-06-09T22:06:00.531 INFO:teuthology.orchestra.run.smithi194.stdout:(29/159): rbd-mirror-19.0.0-2946.g3d89accf.el9. 5.8 MB/s | 3.1 MB 00:00 2024-06-09T22:06:00.607 INFO:teuthology.orchestra.run.smithi194.stdout:(30/159): ceph-mgr-cephadm-19.0.0-2946.g3d89acc 1.0 MB/s | 142 kB 00:00 2024-06-09T22:06:01.248 INFO:teuthology.orchestra.run.smithi194.stdout:(31/159): ceph-mgr-dashboard-19.0.0-2946.g3d89a 4.9 MB/s | 3.5 MB 00:00 2024-06-09T22:06:01.349 INFO:teuthology.orchestra.run.smithi194.stdout:(32/159): ceph-mgr-modules-core-19.0.0-2946.g3d 2.4 MB/s | 249 kB 00:00 2024-06-09T22:06:01.424 INFO:teuthology.orchestra.run.smithi194.stdout:(33/159): ceph-mgr-rook-19.0.0-2946.g3d89accf.e 657 kB/s | 49 kB 00:00 2024-06-09T22:06:01.490 INFO:teuthology.orchestra.run.smithi104.stdout:(2/159): ceph-fuse-19.0.0-2946.g3d89accf.el9.x8 920 kB/s | 890 kB 00:00 2024-06-09T22:06:01.499 INFO:teuthology.orchestra.run.smithi194.stdout:(34/159): ceph-prometheus-alerts-19.0.0-2946.g3 220 kB/s | 16 kB 00:00 2024-06-09T22:06:01.564 INFO:teuthology.orchestra.run.smithi153.stdout:(8/159): ceph-osd-19.0.0-2946.g3d89accf.el9.x86 8.5 MB/s | 17 MB 00:01 2024-06-09T22:06:01.583 INFO:teuthology.orchestra.run.smithi194.stdout:(35/159): ceph-volume-19.0.0-2946.g3d89accf.el9 3.1 MB/s | 264 kB 00:00 2024-06-09T22:06:01.624 INFO:teuthology.orchestra.run.smithi104.stdout:(3/159): ceph-immutable-object-cache-19.0.0-294 1.1 MB/s | 145 kB 00:00 2024-06-09T22:06:01.631 INFO:teuthology.orchestra.run.smithi153.stdout:(9/159): ceph-selinux-19.0.0-2946.g3d89accf.el9 379 kB/s | 25 kB 00:00 2024-06-09T22:06:01.775 INFO:teuthology.orchestra.run.smithi194.stdout:(36/159): cephadm-19.0.0-2946.g3d89accf.el9.noa 3.9 MB/s | 763 kB 00:00 2024-06-09T22:06:01.791 INFO:teuthology.orchestra.run.smithi104.stdout:(4/159): ceph-base-19.0.0-2946.g3d89accf.el9.x8 3.2 MB/s | 5.4 MB 00:01 2024-06-09T22:06:01.858 INFO:teuthology.orchestra.run.smithi194.stdout:(37/159): cephfs-top-19.0.0-2946.g3d89accf.el9. 219 kB/s | 18 kB 00:00 2024-06-09T22:06:01.897 INFO:teuthology.orchestra.run.smithi194.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:02.066 INFO:teuthology.orchestra.run.smithi104.stdout:(5/159): ceph-mgr-19.0.0-2946.g3d89accf.el9.x86 5.4 MB/s | 1.5 MB 00:00 2024-06-09T22:06:02.375 INFO:teuthology.orchestra.run.smithi194.stdout:(38/159): fuse-2.9.9-15.el9.x86_64.rpm 155 kB/s | 80 kB 00:00 2024-06-09T22:06:02.377 INFO:teuthology.orchestra.run.smithi194.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:02.751 INFO:teuthology.orchestra.run.smithi194.stdout:(39/159): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-06-09T22:06:02.753 INFO:teuthology.orchestra.run.smithi194.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:02.758 INFO:teuthology.orchestra.run.smithi104.stdout:(6/159): ceph-mon-19.0.0-2946.g3d89accf.el9.x86 6.8 MB/s | 4.7 MB 00:00 2024-06-09T22:06:02.858 INFO:teuthology.orchestra.run.smithi104.stdout:(7/159): ceph-mds-19.0.0-2946.g3d89accf.el9.x86 1.9 MB/s | 2.4 MB 00:01 2024-06-09T22:06:02.867 INFO:teuthology.orchestra.run.smithi194.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-06-09T22:06:02.984 INFO:teuthology.orchestra.run.smithi194.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-06-09T22:06:03.176 INFO:teuthology.orchestra.run.smithi194.stdout:(42/159): python3-cryptography-36.0.1-4.el9.x86 6.4 MB/s | 1.2 MB 00:00 2024-06-09T22:06:03.285 INFO:teuthology.orchestra.run.smithi194.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 984 kB/s | 106 kB 00:00 2024-06-09T22:06:03.394 INFO:teuthology.orchestra.run.smithi194.stdout:(44/159): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-06-09T22:06:03.502 INFO:teuthology.orchestra.run.smithi194.stdout:(45/159): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-06-09T22:06:03.611 INFO:teuthology.orchestra.run.smithi194.stdout:(46/159): python3-urllib3-1.26.5-5.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-06-09T22:06:03.612 INFO:teuthology.orchestra.run.smithi194.stdout:[MIRROR] automake-1.16.2-8.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/automake-1.16.2-8.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-09T22:06:03.807 INFO:teuthology.orchestra.run.smithi194.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 689 kB 00:00 2024-06-09T22:06:03.808 INFO:teuthology.orchestra.run.smithi194.stdout:[MIRROR] bison-3.7.4-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/bison-3.7.4-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:04.086 INFO:teuthology.orchestra.run.smithi194.stdout:(48/159): bison-3.7.4-5.el9.x86_64.rpm 3.3 MB/s | 943 kB 00:00 2024-06-09T22:06:04.088 INFO:teuthology.orchestra.run.smithi194.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:04.145 INFO:teuthology.orchestra.run.smithi194.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-06-09T22:06:04.187 INFO:teuthology.orchestra.run.smithi194.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 522 kB/s | 22 kB 00:00 2024-06-09T22:06:04.296 INFO:teuthology.orchestra.run.smithi194.stdout:(51/159): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-06-09T22:06:04.337 INFO:teuthology.orchestra.run.smithi194.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 787 kB/s | 33 kB 00:00 2024-06-09T22:06:04.897 INFO:teuthology.orchestra.run.smithi194.stdout:(53/159): ceph-mgr-diskprediction-local-19.0.0- 1.7 MB/s | 7.4 MB 00:04 2024-06-09T22:06:04.923 INFO:teuthology.orchestra.run.smithi153.stdout:(10/159): ceph-radosgw-19.0.0-2946.g3d89accf.el 2.0 MB/s | 10 MB 00:05 2024-06-09T22:06:04.925 INFO:teuthology.orchestra.run.smithi104.stdout:(8/159): ceph-osd-19.0.0-2946.g3d89accf.el9.x86 7.7 MB/s | 17 MB 00:02 2024-06-09T22:06:04.972 INFO:teuthology.orchestra.run.smithi194.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 244 kB/s | 18 kB 00:00 2024-06-09T22:06:04.992 INFO:teuthology.orchestra.run.smithi104.stdout:(9/159): ceph-selinux-19.0.0-2946.g3d89accf.el9 380 kB/s | 25 kB 00:00 2024-06-09T22:06:05.013 INFO:teuthology.orchestra.run.smithi194.stdout:(55/159): libaio-devel-0.3.111-13.el9.x86_64.rp 268 kB/s | 11 kB 00:00 2024-06-09T22:06:05.040 INFO:teuthology.orchestra.run.smithi153.stdout:(11/159): cephfs-mirror-19.0.0-2946.g3d89accf.e 1.9 MB/s | 221 kB 00:00 2024-06-09T22:06:05.098 INFO:teuthology.orchestra.run.smithi194.stdout:(56/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-06-09T22:06:05.115 INFO:teuthology.orchestra.run.smithi153.stdout:(12/159): cephfs-shell-19.0.0-2946.g3d89accf.el 361 kB/s | 27 kB 00:00 2024-06-09T22:06:05.156 INFO:teuthology.orchestra.run.smithi194.stdout:(57/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.7 MB/s | 3.0 MB 00:00 2024-06-09T22:06:05.175 INFO:teuthology.orchestra.run.smithi104.stdout:(10/159): ceph-radosgw-19.0.0-2946.g3d89accf.el 4.4 MB/s | 10 MB 00:02 2024-06-09T22:06:05.181 INFO:teuthology.orchestra.run.smithi194.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 543 kB/s | 45 kB 00:00 2024-06-09T22:06:05.190 INFO:teuthology.orchestra.run.smithi153.stdout:(13/159): libcephfs-devel-19.0.0-2946.g3d89accf 425 kB/s | 32 kB 00:00 2024-06-09T22:06:05.257 INFO:teuthology.orchestra.run.smithi194.stdout:(59/159): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 582 kB 00:00 2024-06-09T22:06:05.267 INFO:teuthology.orchestra.run.smithi104.stdout:(11/159): cephfs-mirror-19.0.0-2946.g3d89accf.e 2.4 MB/s | 221 kB 00:00 2024-06-09T22:06:05.299 INFO:teuthology.orchestra.run.smithi194.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 614 kB/s | 25 kB 00:00 2024-06-09T22:06:05.340 INFO:teuthology.orchestra.run.smithi194.stdout:(61/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-06-09T22:06:05.359 INFO:teuthology.orchestra.run.smithi104.stdout:(12/159): cephfs-shell-19.0.0-2946.g3d89accf.el 295 kB/s | 27 kB 00:00 2024-06-09T22:06:05.365 INFO:teuthology.orchestra.run.smithi194.stdout:(62/159): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-06-09T22:06:05.433 INFO:teuthology.orchestra.run.smithi194.stdout:(63/159): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-06-09T22:06:05.451 INFO:teuthology.orchestra.run.smithi104.stdout:(13/159): libcephfs-devel-19.0.0-2946.g3d89accf 346 kB/s | 32 kB 00:00 2024-06-09T22:06:05.458 INFO:teuthology.orchestra.run.smithi194.stdout:(64/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-06-09T22:06:05.483 INFO:teuthology.orchestra.run.smithi194.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 701 kB/s | 35 kB 00:00 2024-06-09T22:06:05.615 INFO:teuthology.orchestra.run.smithi153.stdout:(14/159): libcephfs2-19.0.0-2946.g3d89accf.el9. 1.7 MB/s | 747 kB 00:00 2024-06-09T22:06:05.643 INFO:teuthology.orchestra.run.smithi104.stdout:(14/159): libcephfs2-19.0.0-2946.g3d89accf.el9. 3.8 MB/s | 747 kB 00:00 2024-06-09T22:06:05.743 INFO:teuthology.orchestra.run.smithi104.stdout:(15/159): libcephsqlite-19.0.0-2946.g3d89accf.e 1.6 MB/s | 163 kB 00:00 2024-06-09T22:06:05.757 INFO:teuthology.orchestra.run.smithi153.stdout:(15/159): libcephsqlite-19.0.0-2946.g3d89accf.e 1.1 MB/s | 163 kB 00:00 2024-06-09T22:06:05.836 INFO:teuthology.orchestra.run.smithi104.stdout:(16/159): librados-devel-19.0.0-2946.g3d89accf. 1.3 MB/s | 127 kB 00:00 2024-06-09T22:06:05.899 INFO:teuthology.orchestra.run.smithi153.stdout:(16/159): librados-devel-19.0.0-2946.g3d89accf. 895 kB/s | 127 kB 00:00 2024-06-09T22:06:05.986 INFO:teuthology.orchestra.run.smithi104.stdout:(17/159): libradosstriper1-19.0.0-2946.g3d89acc 3.3 MB/s | 501 kB 00:00 2024-06-09T22:06:06.150 INFO:teuthology.orchestra.run.smithi153.stdout:(17/159): libradosstriper1-19.0.0-2946.g3d89acc 2.0 MB/s | 501 kB 00:00 2024-06-09T22:06:06.267 INFO:teuthology.orchestra.run.smithi194.stdout:(66/159): ceph-test-19.0.0-2946.g3d89accf.el9.x 5.5 MB/s | 48 MB 00:08 2024-06-09T22:06:06.700 INFO:teuthology.orchestra.run.smithi194.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 3.7 MB/s | 1.6 MB 00:00 2024-06-09T22:06:06.834 INFO:teuthology.orchestra.run.smithi194.stdout:(68/159): openblas-openmp-0.3.21-2.el9.x86_64.r 3.4 MB/s | 4.7 MB 00:01 2024-06-09T22:06:06.917 INFO:teuthology.orchestra.run.smithi194.stdout:(69/159): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-06-09T22:06:07.034 INFO:teuthology.orchestra.run.smithi194.stdout:(70/159): postgresql-contrib-13.14-1.el9.x86_64 2.7 MB/s | 911 kB 00:00 2024-06-09T22:06:07.052 INFO:teuthology.orchestra.run.smithi104.stdout:(18/159): librgw2-19.0.0-2946.g3d89accf.el9.x86 5.0 MB/s | 5.3 MB 00:01 2024-06-09T22:06:07.093 INFO:teuthology.orchestra.run.smithi194.stdout:(71/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 2.7 MB/s | 4.4 MB 00:01 2024-06-09T22:06:07.116 INFO:teuthology.orchestra.run.smithi153.stdout:(18/159): ceph-common-19.0.0-2946.g3d89accf.el9 2.1 MB/s | 21 MB 00:10 2024-06-09T22:06:07.136 INFO:teuthology.orchestra.run.smithi104.stdout:(19/159): python3-ceph-argparse-19.0.0-2946.g3d 543 kB/s | 45 kB 00:00 2024-06-09T22:06:07.168 INFO:teuthology.orchestra.run.smithi194.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-06-09T22:06:07.191 INFO:teuthology.orchestra.run.smithi153.stdout:(19/159): python3-ceph-argparse-19.0.0-2946.g3d 603 kB/s | 45 kB 00:00 2024-06-09T22:06:07.228 INFO:teuthology.orchestra.run.smithi104.stdout:(20/159): python3-ceph-common-19.0.0-2946.g3d89 1.4 MB/s | 135 kB 00:00 2024-06-09T22:06:07.267 INFO:teuthology.orchestra.run.smithi153.stdout:(20/159): python3-ceph-common-19.0.0-2946.g3d89 1.8 MB/s | 135 kB 00:00 2024-06-09T22:06:07.328 INFO:teuthology.orchestra.run.smithi104.stdout:(21/159): python3-cephfs-19.0.0-2946.g3d89accf. 1.6 MB/s | 162 kB 00:00 2024-06-09T22:06:07.350 INFO:teuthology.orchestra.run.smithi153.stdout:(21/159): python3-cephfs-19.0.0-2946.g3d89accf. 1.9 MB/s | 162 kB 00:00 2024-06-09T22:06:07.418 INFO:teuthology.orchestra.run.smithi194.stdout:(73/159): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-06-09T22:06:07.428 INFO:teuthology.orchestra.run.smithi104.stdout:(22/159): python3-rados-19.0.0-2946.g3d89accf.e 3.1 MB/s | 321 kB 00:00 2024-06-09T22:06:07.469 INFO:teuthology.orchestra.run.smithi194.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-06-09T22:06:07.492 INFO:teuthology.orchestra.run.smithi153.stdout:(22/159): python3-rados-19.0.0-2946.g3d89accf.e 2.2 MB/s | 321 kB 00:00 2024-06-09T22:06:07.529 INFO:teuthology.orchestra.run.smithi104.stdout:(23/159): python3-rbd-19.0.0-2946.g3d89accf.el9 2.9 MB/s | 300 kB 00:00 2024-06-09T22:06:07.561 INFO:teuthology.orchestra.run.smithi194.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 2.6 MB/s | 244 kB 00:00 2024-06-09T22:06:07.626 INFO:teuthology.orchestra.run.smithi153.stdout:(23/159): python3-rbd-19.0.0-2946.g3d89accf.el9 2.2 MB/s | 300 kB 00:00 2024-06-09T22:06:07.629 INFO:teuthology.orchestra.run.smithi104.stdout:(24/159): python3-rgw-19.0.0-2946.g3d89accf.el9 993 kB/s | 99 kB 00:00 2024-06-09T22:06:07.653 INFO:teuthology.orchestra.run.smithi194.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-06-09T22:06:07.685 INFO:teuthology.orchestra.run.smithi153.stdout:(24/159): python3-rgw-19.0.0-2946.g3d89accf.el9 1.7 MB/s | 99 kB 00:00 2024-06-09T22:06:07.695 INFO:teuthology.orchestra.run.smithi194.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 618 kB/s | 26 kB 00:00 2024-06-09T22:06:07.721 INFO:teuthology.orchestra.run.smithi104.stdout:(25/159): rbd-fuse-19.0.0-2946.g3d89accf.el9.x8 931 kB/s | 85 kB 00:00 2024-06-09T22:06:07.736 INFO:teuthology.orchestra.run.smithi194.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 462 kB/s | 19 kB 00:00 2024-06-09T22:06:07.760 INFO:teuthology.orchestra.run.smithi153.stdout:(25/159): rbd-fuse-19.0.0-2946.g3d89accf.el9.x8 1.1 MB/s | 85 kB 00:00 2024-06-09T22:06:07.812 INFO:teuthology.orchestra.run.smithi194.stdout:(79/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-06-09T22:06:07.854 INFO:teuthology.orchestra.run.smithi194.stdout:(80/159): python3-markupsafe-1.1.1-12.el9.x86_6 832 kB/s | 35 kB 00:00 2024-06-09T22:06:08.296 INFO:teuthology.orchestra.run.smithi104.stdout:(26/159): ceph-common-19.0.0-2946.g3d89accf.el9 2.6 MB/s | 21 MB 00:08 2024-06-09T22:06:08.354 INFO:teuthology.orchestra.run.smithi104.stdout:(27/159): rbd-mirror-19.0.0-2946.g3d89accf.el9. 4.9 MB/s | 3.1 MB 00:00 2024-06-09T22:06:08.413 INFO:teuthology.orchestra.run.smithi104.stdout:(28/159): rbd-nbd-19.0.0-2946.g3d89accf.el9.x86 1.4 MB/s | 171 kB 00:00 2024-06-09T22:06:08.438 INFO:teuthology.orchestra.run.smithi104.stdout:(29/159): ceph-grafana-dashboards-19.0.0-2946.g 380 kB/s | 32 kB 00:00 2024-06-09T22:06:08.472 INFO:teuthology.orchestra.run.smithi104.stdout:(30/159): ceph-mgr-cephadm-19.0.0-2946.g3d89acc 2.4 MB/s | 142 kB 00:00 2024-06-09T22:06:08.719 INFO:teuthology.orchestra.run.smithi153.stdout:(26/159): ceph-test-19.0.0-2946.g3d89accf.el9.x 6.8 MB/s | 48 MB 00:07 2024-06-09T22:06:08.785 INFO:teuthology.orchestra.run.smithi153.stdout:(27/159): librgw2-19.0.0-2946.g3d89accf.el9.x86 2.0 MB/s | 5.3 MB 00:02 2024-06-09T22:06:08.821 INFO:teuthology.orchestra.run.smithi194.stdout:(81/159): postgresql-server-13.14-1.el9.x86_64. 3.0 MB/s | 5.8 MB 00:01 2024-06-09T22:06:08.827 INFO:teuthology.orchestra.run.smithi153.stdout:(28/159): rbd-nbd-19.0.0-2946.g3d89accf.el9.x86 1.5 MB/s | 171 kB 00:00 2024-06-09T22:06:08.844 INFO:teuthology.orchestra.run.smithi153.stdout:(29/159): ceph-grafana-dashboards-19.0.0-2946.g 544 kB/s | 32 kB 00:00 2024-06-09T22:06:08.895 INFO:teuthology.orchestra.run.smithi153.stdout:(30/159): ceph-mgr-cephadm-19.0.0-2946.g3d89acc 2.1 MB/s | 142 kB 00:00 2024-06-09T22:06:08.913 INFO:teuthology.orchestra.run.smithi194.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.7 MB/s | 250 kB 00:00 2024-06-09T22:06:08.922 INFO:teuthology.orchestra.run.smithi104.stdout:(31/159): ceph-mgr-dashboard-19.0.0-2946.g3d89a 7.3 MB/s | 3.5 MB 00:00 2024-06-09T22:06:08.954 INFO:teuthology.orchestra.run.smithi153.stdout:(31/159): rbd-mirror-19.0.0-2946.g3d89accf.el9. 2.6 MB/s | 3.1 MB 00:01 2024-06-09T22:06:09.005 INFO:teuthology.orchestra.run.smithi194.stdout:(83/159): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-06-09T22:06:09.014 INFO:teuthology.orchestra.run.smithi104.stdout:(32/159): ceph-mgr-modules-core-19.0.0-2946.g3d 2.7 MB/s | 249 kB 00:00 2024-06-09T22:06:09.080 INFO:teuthology.orchestra.run.smithi194.stdout:(84/159): python3-babel-2.9.1-2.el9.noarch.rpm 3.1 MB/s | 6.0 MB 00:01 2024-06-09T22:06:09.087 INFO:teuthology.orchestra.run.smithi153.stdout:(32/159): ceph-mgr-modules-core-19.0.0-2946.g3d 1.8 MB/s | 249 kB 00:00 2024-06-09T22:06:09.105 INFO:teuthology.orchestra.run.smithi194.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 771 kB/s | 77 kB 00:00 2024-06-09T22:06:09.122 INFO:teuthology.orchestra.run.smithi194.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-06-09T22:06:09.132 INFO:teuthology.orchestra.run.smithi104.stdout:(33/159): ceph-mgr-rook-19.0.0-2946.g3d89accf.e 422 kB/s | 49 kB 00:00 2024-06-09T22:06:09.145 INFO:teuthology.orchestra.run.smithi153.stdout:(33/159): ceph-mgr-rook-19.0.0-2946.g3d89accf.e 847 kB/s | 49 kB 00:00 2024-06-09T22:06:09.197 INFO:teuthology.orchestra.run.smithi194.stdout:(87/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-06-09T22:06:09.212 INFO:teuthology.orchestra.run.smithi153.stdout:(34/159): ceph-prometheus-alerts-19.0.0-2946.g3 247 kB/s | 16 kB 00:00 2024-06-09T22:06:09.223 INFO:teuthology.orchestra.run.smithi194.stdout:(88/159): python3-protobuf-3.14.0-13.el9.noarch 2.2 MB/s | 269 kB 00:00 2024-06-09T22:06:09.232 INFO:teuthology.orchestra.run.smithi104.stdout:(34/159): ceph-prometheus-alerts-19.0.0-2946.g3 165 kB/s | 16 kB 00:00 2024-06-09T22:06:09.264 INFO:teuthology.orchestra.run.smithi194.stdout:(89/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-06-09T22:06:09.332 INFO:teuthology.orchestra.run.smithi194.stdout:(90/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.4 MB/s | 5.1 MB 00:01 2024-06-09T22:06:09.338 INFO:teuthology.orchestra.run.smithi153.stdout:(35/159): ceph-volume-19.0.0-2946.g3d89accf.el9 2.1 MB/s | 264 kB 00:00 2024-06-09T22:06:09.365 INFO:teuthology.orchestra.run.smithi104.stdout:(35/159): ceph-volume-19.0.0-2946.g3d89accf.el9 1.9 MB/s | 264 kB 00:00 2024-06-09T22:06:09.415 INFO:teuthology.orchestra.run.smithi194.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 356 kB/s | 54 kB 00:00 2024-06-09T22:06:09.457 INFO:teuthology.orchestra.run.smithi194.stdout:(92/159): python3-pyasn1-modules-0.4.8-6.el9.no 1.0 MB/s | 279 kB 00:00 2024-06-09T22:06:09.482 INFO:teuthology.orchestra.run.smithi194.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 622 kB/s | 42 kB 00:00 2024-06-09T22:06:09.516 INFO:teuthology.orchestra.run.smithi194.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 753 kB/s | 44 kB 00:00 2024-06-09T22:06:09.574 INFO:teuthology.orchestra.run.smithi104.stdout:(36/159): cephadm-19.0.0-2946.g3d89accf.el9.noa 3.6 MB/s | 763 kB 00:00 2024-06-09T22:06:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:(95/159): qatlib-service-24.02.0-1.el9.x86_64.r 616 kB/s | 36 kB 00:00 2024-06-09T22:06:09.642 INFO:teuthology.orchestra.run.smithi194.stdout:(96/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 711 kB/s | 47 kB 00:00 2024-06-09T22:06:09.646 INFO:teuthology.orchestra.run.smithi153.stdout:(36/159): cephadm-19.0.0-2946.g3d89accf.el9.noa 2.4 MB/s | 763 kB 00:00 2024-06-09T22:06:09.684 INFO:teuthology.orchestra.run.smithi194.stdout:(97/159): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 221 kB 00:00 2024-06-09T22:06:09.699 INFO:teuthology.orchestra.run.smithi104.stdout:(37/159): cephfs-top-19.0.0-2946.g3d89accf.el9. 146 kB/s | 18 kB 00:00 2024-06-09T22:06:09.714 INFO:teuthology.orchestra.run.smithi153.stdout:(37/159): cephfs-top-19.0.0-2946.g3d89accf.el9. 274 kB/s | 18 kB 00:00 2024-06-09T22:06:09.739 INFO:teuthology.orchestra.run.smithi104.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:09.757 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:09.767 INFO:teuthology.orchestra.run.smithi194.stdout:(98/159): uuid-1.6.2-55.el9.x86_64.rpm 693 kB/s | 58 kB 00:00 2024-06-09T22:06:09.876 INFO:teuthology.orchestra.run.smithi194.stdout:(99/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 867 kB/s | 94 kB 00:00 2024-06-09T22:06:09.880 INFO:teuthology.orchestra.run.smithi153.stdout:(38/159): ceph-mgr-diskprediction-local-19.0.0- 7.5 MB/s | 7.4 MB 00:00 2024-06-09T22:06:09.883 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:09.901 INFO:teuthology.orchestra.run.smithi194.stdout:(100/159): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 305 kB 00:00 2024-06-09T22:06:09.943 INFO:teuthology.orchestra.run.smithi194.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 968 kB/s | 64 kB 00:00 2024-06-09T22:06:09.968 INFO:teuthology.orchestra.run.smithi194.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 670 kB/s | 45 kB 00:00 2024-06-09T22:06:09.989 INFO:teuthology.orchestra.run.smithi153.stdout:(39/159): ceph-mgr-dashboard-19.0.0-2946.g3d89a 3.1 MB/s | 3.5 MB 00:01 2024-06-09T22:06:10.014 INFO:teuthology.orchestra.run.smithi153.stdout:(40/159): fuse-2.9.9-15.el9.x86_64.rpm 267 kB/s | 80 kB 00:00 2024-06-09T22:06:10.015 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:10.017 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:10.027 INFO:teuthology.orchestra.run.smithi194.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:10.037 INFO:teuthology.orchestra.run.smithi194.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:10.081 INFO:teuthology.orchestra.run.smithi153.stdout:(41/159): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-06-09T22:06:10.093 INFO:teuthology.orchestra.run.smithi194.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 148 kB/s | 22 kB 00:00 2024-06-09T22:06:10.106 INFO:teuthology.orchestra.run.smithi153.stdout:(42/159): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-06-09T22:06:10.148 INFO:teuthology.orchestra.run.smithi153.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-06-09T22:06:10.152 INFO:teuthology.orchestra.run.smithi194.stdout:(104/159): abseil-cpp-20211102.0-3.el9.x86_64.r 9.2 MB/s | 548 kB 00:00 2024-06-09T22:06:10.182 INFO:teuthology.orchestra.run.smithi153.stdout:(44/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-06-09T22:06:10.186 INFO:teuthology.orchestra.run.smithi194.stdout:(105/159): gperftools-libs-2.9.1-2.el9.x86_64.r 9.0 MB/s | 309 kB 00:00 2024-06-09T22:06:10.207 INFO:teuthology.orchestra.run.smithi153.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-06-09T22:06:10.211 INFO:teuthology.orchestra.run.smithi194.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 778 kB/s | 19 kB 00:00 2024-06-09T22:06:10.233 INFO:teuthology.orchestra.run.smithi153.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-06-09T22:06:10.233 INFO:teuthology.orchestra.run.smithi104.stdout:(38/159): ceph-test-19.0.0-2946.g3d89accf.el9.x 9.2 MB/s | 48 MB 00:05 2024-06-09T22:06:10.258 INFO:teuthology.orchestra.run.smithi153.stdout:(47/159): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-06-09T22:06:10.258 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] automake-1.16.2-8.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/automake-1.16.2-8.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-09T22:06:10.266 INFO:teuthology.orchestra.run.smithi104.stdout:(39/159): fuse-2.9.9-15.el9.x86_64.rpm 142 kB/s | 80 kB 00:00 2024-06-09T22:06:10.267 INFO:teuthology.orchestra.run.smithi104.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:10.269 INFO:teuthology.orchestra.run.smithi104.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:10.291 INFO:teuthology.orchestra.run.smithi153.stdout:(48/159): python3-cryptography-36.0.1-4.el9.x86 5.9 MB/s | 1.2 MB 00:00 2024-06-09T22:06:10.292 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] bison-3.7.4-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/bison-3.7.4-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:10.294 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:10.359 INFO:teuthology.orchestra.run.smithi104.stdout:(40/159): ceph-mgr-diskprediction-local-19.0.0- 3.9 MB/s | 7.4 MB 00:01 2024-06-09T22:06:10.403 INFO:teuthology.orchestra.run.smithi194.stdout:(107/159): libarrow-9.0.0-11.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2024-06-09T22:06:10.408 INFO:teuthology.orchestra.run.smithi153.stdout:(49/159): automake-1.16.2-8.el9.noarch.rpm 3.8 MB/s | 689 kB 00:00 2024-06-09T22:06:10.425 INFO:teuthology.orchestra.run.smithi104.stdout:(41/159): libgfortran-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 803 kB 00:00 2024-06-09T22:06:10.428 INFO:teuthology.orchestra.run.smithi194.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-06-09T22:06:10.434 INFO:teuthology.orchestra.run.smithi153.stdout:(50/159): boost-program-options-1.75.0-8.el9.x8 753 kB/s | 107 kB 00:00 2024-06-09T22:06:10.451 INFO:teuthology.orchestra.run.smithi153.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 518 kB/s | 22 kB 00:00 2024-06-09T22:06:10.453 INFO:teuthology.orchestra.run.smithi194.stdout:(109/159): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-06-09T22:06:10.481 INFO:teuthology.orchestra.run.smithi194.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 960 kB/s | 49 kB 00:00 2024-06-09T22:06:10.484 INFO:teuthology.orchestra.run.smithi104.stdout:(42/159): libquadmath-11.4.1-3.el9.x86_64.rpm 900 kB/s | 195 kB 00:00 2024-06-09T22:06:10.484 INFO:teuthology.orchestra.run.smithi153.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 986 kB/s | 33 kB 00:00 2024-06-09T22:06:10.504 INFO:teuthology.orchestra.run.smithi194.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-06-09T22:06:10.509 INFO:teuthology.orchestra.run.smithi104.stdout:(43/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-06-09T22:06:10.510 INFO:teuthology.orchestra.run.smithi153.stdout:(53/159): bison-3.7.4-5.el9.x86_64.rpm 3.7 MB/s | 943 kB 00:00 2024-06-09T22:06:10.530 INFO:teuthology.orchestra.run.smithi194.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-06-09T22:06:10.534 INFO:teuthology.orchestra.run.smithi104.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-06-09T22:06:10.543 INFO:teuthology.orchestra.run.smithi153.stdout:(54/159): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-06-09T22:06:10.560 INFO:teuthology.orchestra.run.smithi104.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-06-09T22:06:10.560 INFO:teuthology.orchestra.run.smithi153.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 363 kB/s | 18 kB 00:00 2024-06-09T22:06:10.572 INFO:teuthology.orchestra.run.smithi194.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-06-09T22:06:10.577 INFO:teuthology.orchestra.run.smithi153.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 332 kB/s | 11 kB 00:00 2024-06-09T22:06:10.585 INFO:teuthology.orchestra.run.smithi104.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-06-09T22:06:10.597 INFO:teuthology.orchestra.run.smithi194.stdout:(114/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-06-09T22:06:10.602 INFO:teuthology.orchestra.run.smithi153.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 160 kB 00:00 2024-06-09T22:06:10.619 INFO:teuthology.orchestra.run.smithi104.stdout:(47/159): python3-cryptography-36.0.1-4.el9.x86 6.4 MB/s | 1.2 MB 00:00 2024-06-09T22:06:10.619 INFO:teuthology.orchestra.run.smithi153.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-06-09T22:06:10.620 INFO:teuthology.orchestra.run.smithi104.stdout:[MIRROR] automake-1.16.2-8.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/automake-1.16.2-8.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-09T22:06:10.620 INFO:teuthology.orchestra.run.smithi104.stdout:[MIRROR] bison-3.7.4-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/bison-3.7.4-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:10.630 INFO:teuthology.orchestra.run.smithi194.stdout:(115/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-06-09T22:06:10.652 INFO:teuthology.orchestra.run.smithi104.stdout:(48/159): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-06-09T22:06:10.653 INFO:teuthology.orchestra.run.smithi104.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:10.656 INFO:teuthology.orchestra.run.smithi194.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-06-09T22:06:10.661 INFO:teuthology.orchestra.run.smithi153.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-06-09T22:06:10.681 INFO:teuthology.orchestra.run.smithi194.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-06-09T22:06:10.703 INFO:teuthology.orchestra.run.smithi153.stdout:(60/159): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 582 kB 00:00 2024-06-09T22:06:10.706 INFO:teuthology.orchestra.run.smithi194.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-06-09T22:06:10.728 INFO:teuthology.orchestra.run.smithi153.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 379 kB/s | 25 kB 00:00 2024-06-09T22:06:10.740 INFO:teuthology.orchestra.run.smithi194.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-06-09T22:06:10.754 INFO:teuthology.orchestra.run.smithi153.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-06-09T22:06:10.765 INFO:teuthology.orchestra.run.smithi194.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-06-09T22:06:10.779 INFO:teuthology.orchestra.run.smithi153.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-06-09T22:06:10.799 INFO:teuthology.orchestra.run.smithi194.stdout:(121/159): python3-google-auth-2.29.0-2.el9.noa 3.4 MB/s | 205 kB 00:00 2024-06-09T22:06:10.804 INFO:teuthology.orchestra.run.smithi153.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-06-09T22:06:10.830 INFO:teuthology.orchestra.run.smithi153.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 702 kB/s | 35 kB 00:00 2024-06-09T22:06:10.832 INFO:teuthology.orchestra.run.smithi194.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-06-09T22:06:10.858 INFO:teuthology.orchestra.run.smithi194.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-06-09T22:06:10.883 INFO:teuthology.orchestra.run.smithi194.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 704 kB/s | 18 kB 00:00 2024-06-09T22:06:10.888 INFO:teuthology.orchestra.run.smithi153.stdout:(66/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.4 MB/s | 3.0 MB 00:00 2024-06-09T22:06:10.908 INFO:teuthology.orchestra.run.smithi194.stdout:(125/159): python3-jaraco-collections-3.0.0-8.e 933 kB/s | 23 kB 00:00 2024-06-09T22:06:10.925 INFO:teuthology.orchestra.run.smithi194.stdout:(126/159): python3-jaraco-functools-3.5.0-2.el9 1.1 MB/s | 19 kB 00:00 2024-06-09T22:06:10.967 INFO:teuthology.orchestra.run.smithi194.stdout:(127/159): python3-grpcio-1.46.7-10.el9.x86_64. 10 MB/s | 2.0 MB 00:00 2024-06-09T22:06:10.972 INFO:teuthology.orchestra.run.smithi153.stdout:(67/159): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-06-09T22:06:10.984 INFO:teuthology.orchestra.run.smithi194.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 335 kB/s | 20 kB 00:00 2024-06-09T22:06:11.001 INFO:teuthology.orchestra.run.smithi194.stdout:(129/159): python3-logutils-0.3.5-21.el9.noarch 2.7 MB/s | 46 kB 00:00 2024-06-09T22:06:11.031 INFO:teuthology.orchestra.run.smithi153.stdout:(68/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2024-06-09T22:06:11.035 INFO:teuthology.orchestra.run.smithi194.stdout:(130/159): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-06-09T22:06:11.064 INFO:teuthology.orchestra.run.smithi153.stdout:(69/159): postgresql-13.14-1.el9.x86_64.rpm 9.1 MB/s | 1.6 MB 00:00 2024-06-09T22:06:11.076 INFO:teuthology.orchestra.run.smithi194.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-06-09T22:06:11.081 INFO:teuthology.orchestra.run.smithi153.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-06-09T22:06:11.119 INFO:teuthology.orchestra.run.smithi194.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 689 kB/s | 58 kB 00:00 2024-06-09T22:06:11.123 INFO:teuthology.orchestra.run.smithi153.stdout:(71/159): postgresql-contrib-13.14-1.el9.x86_64 5.9 MB/s | 911 kB 00:00 2024-06-09T22:06:11.144 INFO:teuthology.orchestra.run.smithi194.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 656 kB/s | 16 kB 00:00 2024-06-09T22:06:11.165 INFO:teuthology.orchestra.run.smithi153.stdout:(72/159): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-06-09T22:06:11.169 INFO:teuthology.orchestra.run.smithi194.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-06-09T22:06:11.190 INFO:teuthology.orchestra.run.smithi153.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-06-09T22:06:11.194 INFO:teuthology.orchestra.run.smithi194.stdout:(135/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-06-09T22:06:11.220 INFO:teuthology.orchestra.run.smithi194.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-06-09T22:06:11.233 INFO:teuthology.orchestra.run.smithi153.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-06-09T22:06:11.245 INFO:teuthology.orchestra.run.smithi194.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-06-09T22:06:11.270 INFO:teuthology.orchestra.run.smithi194.stdout:(138/159): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-06-09T22:06:11.296 INFO:teuthology.orchestra.run.smithi194.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-06-09T22:06:11.308 INFO:teuthology.orchestra.run.smithi153.stdout:(75/159): postgresql-server-13.14-1.el9.x86_64. 24 MB/s | 5.8 MB 00:00 2024-06-09T22:06:11.330 INFO:teuthology.orchestra.run.smithi194.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 608 kB/s | 36 kB 00:00 2024-06-09T22:06:11.333 INFO:teuthology.orchestra.run.smithi153.stdout:(76/159): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-06-09T22:06:11.355 INFO:teuthology.orchestra.run.smithi194.stdout:(141/159): python3-typing-extensions-4.4.0-2.el 867 kB/s | 51 kB 00:00 2024-06-09T22:06:11.366 INFO:teuthology.orchestra.run.smithi153.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 776 kB/s | 26 kB 00:00 2024-06-09T22:06:11.388 INFO:teuthology.orchestra.run.smithi194.stdout:(142/159): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-06-09T22:06:11.391 INFO:teuthology.orchestra.run.smithi153.stdout:(78/159): python3-jinja2-2.11.3-6.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-06-09T22:06:11.411 INFO:teuthology.orchestra.run.smithi104.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 141 kB/s | 107 kB 00:00 2024-06-09T22:06:11.414 INFO:teuthology.orchestra.run.smithi194.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-06-09T22:06:11.417 INFO:teuthology.orchestra.run.smithi153.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-06-09T22:06:11.439 INFO:teuthology.orchestra.run.smithi194.stdout:(144/159): python3-zc-lockfile-2.0-10.el9.noarc 808 kB/s | 20 kB 00:00 2024-06-09T22:06:11.442 INFO:teuthology.orchestra.run.smithi153.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-06-09T22:06:11.467 INFO:teuthology.orchestra.run.smithi153.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 694 kB/s | 35 kB 00:00 2024-06-09T22:06:11.472 INFO:teuthology.orchestra.run.smithi194.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-06-09T22:06:11.498 INFO:teuthology.orchestra.run.smithi194.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-06-09T22:06:11.556 INFO:teuthology.orchestra.run.smithi194.stdout:(147/159): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-06-09T22:06:11.559 INFO:teuthology.orchestra.run.smithi153.stdout:(82/159): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-06-09T22:06:11.569 INFO:teuthology.orchestra.run.smithi104.stdout:(50/159): automake-1.16.2-8.el9.noarch.rpm 700 kB/s | 689 kB 00:00 2024-06-09T22:06:11.585 INFO:teuthology.orchestra.run.smithi153.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-06-09T22:06:11.603 INFO:teuthology.orchestra.run.smithi104.stdout:(51/159): bison-3.7.4-5.el9.x86_64.rpm 958 kB/s | 943 kB 00:00 2024-06-09T22:06:11.610 INFO:teuthology.orchestra.run.smithi153.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-06-09T22:06:11.635 INFO:teuthology.orchestra.run.smithi153.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-06-09T22:06:11.660 INFO:teuthology.orchestra.run.smithi153.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-06-09T22:06:11.685 INFO:teuthology.orchestra.run.smithi153.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-06-09T22:06:11.711 INFO:teuthology.orchestra.run.smithi153.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-06-09T22:06:11.737 INFO:teuthology.orchestra.run.smithi153.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-06-09T22:06:11.753 INFO:teuthology.orchestra.run.smithi153.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-06-09T22:06:11.770 INFO:teuthology.orchestra.run.smithi153.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-06-09T22:06:11.804 INFO:teuthology.orchestra.run.smithi153.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-06-09T22:06:11.871 INFO:teuthology.orchestra.run.smithi153.stdout:(93/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-06-09T22:06:11.888 INFO:teuthology.orchestra.run.smithi153.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 529 kB/s | 44 kB 00:00 2024-06-09T22:06:11.913 INFO:teuthology.orchestra.run.smithi153.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-06-09T22:06:11.938 INFO:teuthology.orchestra.run.smithi153.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 716 kB/s | 36 kB 00:00 2024-06-09T22:06:11.955 INFO:teuthology.orchestra.run.smithi153.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-06-09T22:06:11.981 INFO:teuthology.orchestra.run.smithi153.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 305 kB 00:00 2024-06-09T22:06:11.998 INFO:teuthology.orchestra.run.smithi153.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-06-09T22:06:12.003 INFO:teuthology.orchestra.run.smithi104.stdout:(52/159): elfutils-libelf-devel-0.191-4.el9.x86 37 kB/s | 22 kB 00:00 2024-06-09T22:06:12.023 INFO:teuthology.orchestra.run.smithi153.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-06-09T22:06:12.040 INFO:teuthology.orchestra.run.smithi153.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-06-09T22:06:12.057 INFO:teuthology.orchestra.run.smithi153.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-06-09T22:06:12.057 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:12.059 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:12.082 INFO:teuthology.orchestra.run.smithi194.stdout:(148/159): librados2-19.0.0-2946.g3d89accf.el9. 5.8 MB/s | 3.4 MB 00:00 2024-06-09T22:06:12.136 INFO:teuthology.orchestra.run.smithi153.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 232 kB/s | 22 kB 00:00 2024-06-09T22:06:12.199 INFO:teuthology.orchestra.run.smithi153.stdout:(104/159): abseil-cpp-20211102.0-3.el9.x86_64.r 8.6 MB/s | 548 kB 00:00 2024-06-09T22:06:12.220 INFO:teuthology.orchestra.run.smithi104.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 53 kB/s | 33 kB 00:00 2024-06-09T22:06:12.257 INFO:teuthology.orchestra.run.smithi194.stdout:(149/159): NetworkManager-1.47.91-1.el9.x86_64. 13 MB/s | 2.3 MB 00:00 2024-06-09T22:06:12.350 INFO:teuthology.orchestra.run.smithi153.stdout:(105/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-06-09T22:06:12.407 INFO:teuthology.orchestra.run.smithi194.stdout:(150/159): librbd1-19.0.0-2946.g3d89accf.el9.x8 3.7 MB/s | 3.1 MB 00:00 2024-06-09T22:06:12.416 INFO:teuthology.orchestra.run.smithi153.stdout:(106/159): protobuf-compiler-3.14.0-13.el9.x86_ 2.3 MB/s | 863 kB 00:00 2024-06-09T22:06:12.441 INFO:teuthology.orchestra.run.smithi153.stdout:(107/159): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-06-09T22:06:12.449 INFO:teuthology.orchestra.run.smithi194.stdout:(151/159): NetworkManager-libnm-1.47.91-1.el9.x 9.4 MB/s | 1.8 MB 00:00 2024-06-09T22:06:12.454 INFO:teuthology.orchestra.run.smithi104.stdout:(54/159): flex-2.6.4-9.el9.x86_64.rpm 355 kB/s | 313 kB 00:00 2024-06-09T22:06:12.492 INFO:teuthology.orchestra.run.smithi153.stdout:(108/159): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-06-09T22:06:12.508 INFO:teuthology.orchestra.run.smithi194.stdout:(152/159): NetworkManager-team-1.47.91-1.el9.x8 416 kB/s | 41 kB 00:00 2024-06-09T22:06:12.517 INFO:teuthology.orchestra.run.smithi153.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 527 kB/s | 26 kB 00:00 2024-06-09T22:06:12.542 INFO:teuthology.orchestra.run.smithi153.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 1.0 MB/s | 49 kB 00:00 2024-06-09T22:06:12.558 INFO:teuthology.orchestra.run.smithi194.stdout:(153/159): NetworkManager-tui-1.47.91-1.el9.x86 2.2 MB/s | 248 kB 00:00 2024-06-09T22:06:12.568 INFO:teuthology.orchestra.run.smithi153.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-06-09T22:06:12.601 INFO:teuthology.orchestra.run.smithi153.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-06-09T22:06:12.608 INFO:teuthology.orchestra.run.smithi194.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 366 kB/s | 37 kB 00:00 2024-06-09T22:06:12.658 INFO:teuthology.orchestra.run.smithi194.stdout:(155/159): elfutils-libelf-0.191-4.el9.x86_64.r 2.0 MB/s | 207 kB 00:00 2024-06-09T22:06:12.717 INFO:teuthology.orchestra.run.smithi194.stdout:(156/159): elfutils-libs-0.191-4.el9.x86_64.rpm 2.3 MB/s | 260 kB 00:00 2024-06-09T22:06:12.726 INFO:teuthology.orchestra.run.smithi153.stdout:(113/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.2 MB/s | 839 kB 00:00 2024-06-09T22:06:12.752 INFO:teuthology.orchestra.run.smithi153.stdout:(114/159): python3-asyncssh-2.13.2-5.el9.noarch 3.6 MB/s | 548 kB 00:00 2024-06-09T22:06:12.754 INFO:teuthology.orchestra.run.smithi104.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 34 kB/s | 18 kB 00:00 2024-06-09T22:06:12.777 INFO:teuthology.orchestra.run.smithi153.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 864 kB/s | 43 kB 00:00 2024-06-09T22:06:12.801 INFO:teuthology.orchestra.run.smithi194.stdout:(157/159): openssl-3.2.1-1.el9.x86_64.rpm 9.5 MB/s | 1.3 MB 00:00 2024-06-09T22:06:12.802 INFO:teuthology.orchestra.run.smithi153.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 642 kB/s | 32 kB 00:00 2024-06-09T22:06:12.827 INFO:teuthology.orchestra.run.smithi153.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-06-09T22:06:12.861 INFO:teuthology.orchestra.run.smithi153.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-06-09T22:06:12.884 INFO:teuthology.orchestra.run.smithi194.stdout:(158/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-06-09T22:06:12.895 INFO:teuthology.orchestra.run.smithi153.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-06-09T22:06:12.929 INFO:teuthology.orchestra.run.smithi153.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-06-09T22:06:12.954 INFO:teuthology.orchestra.run.smithi153.stdout:(121/159): python3-google-auth-2.29.0-2.el9.noa 3.4 MB/s | 205 kB 00:00 2024-06-09T22:06:12.987 INFO:teuthology.orchestra.run.smithi104.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 22 kB/s | 11 kB 00:00 2024-06-09T22:06:13.021 INFO:teuthology.orchestra.run.smithi153.stdout:(122/159): libarrow-9.0.0-11.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2024-06-09T22:06:13.046 INFO:teuthology.orchestra.run.smithi153.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-06-09T22:06:13.071 INFO:teuthology.orchestra.run.smithi153.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-06-09T22:06:13.097 INFO:teuthology.orchestra.run.smithi153.stdout:(125/159): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-06-09T22:06:13.113 INFO:teuthology.orchestra.run.smithi104.stdout:(57/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.7 MB/s | 3.0 MB 00:01 2024-06-09T22:06:13.122 INFO:teuthology.orchestra.run.smithi153.stdout:(126/159): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-06-09T22:06:13.148 INFO:teuthology.orchestra.run.smithi153.stdout:(127/159): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-06-09T22:06:13.198 INFO:teuthology.orchestra.run.smithi153.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-06-09T22:06:13.223 INFO:teuthology.orchestra.run.smithi153.stdout:(129/159): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-06-09T22:06:13.257 INFO:teuthology.orchestra.run.smithi153.stdout:(130/159): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-06-09T22:06:13.282 INFO:teuthology.orchestra.run.smithi153.stdout:(131/159): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-06-09T22:06:13.315 INFO:teuthology.orchestra.run.smithi153.stdout:(132/159): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-06-09T22:06:13.341 INFO:teuthology.orchestra.run.smithi153.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 657 kB/s | 16 kB 00:00 2024-06-09T22:06:13.366 INFO:teuthology.orchestra.run.smithi153.stdout:(134/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-06-09T22:06:13.391 INFO:teuthology.orchestra.run.smithi153.stdout:(135/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-06-09T22:06:13.417 INFO:teuthology.orchestra.run.smithi153.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-06-09T22:06:13.434 INFO:teuthology.orchestra.run.smithi153.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 736 kB/s | 31 kB 00:00 2024-06-09T22:06:13.475 INFO:teuthology.orchestra.run.smithi153.stdout:(138/159): python3-grpcio-1.46.7-10.el9.x86_64. 3.7 MB/s | 2.0 MB 00:00 2024-06-09T22:06:13.501 INFO:teuthology.orchestra.run.smithi153.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-06-09T22:06:13.518 INFO:teuthology.orchestra.run.smithi153.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 706 kB/s | 59 kB 00:00 2024-06-09T22:06:13.535 INFO:teuthology.orchestra.run.smithi153.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 608 kB/s | 36 kB 00:00 2024-06-09T22:06:13.551 INFO:teuthology.orchestra.run.smithi153.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-06-09T22:06:13.571 INFO:teuthology.orchestra.run.smithi104.stdout:(58/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 196 kB/s | 160 kB 00:00 2024-06-09T22:06:13.577 INFO:teuthology.orchestra.run.smithi153.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-06-09T22:06:13.602 INFO:teuthology.orchestra.run.smithi153.stdout:(144/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-06-09T22:06:13.627 INFO:teuthology.orchestra.run.smithi153.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-06-09T22:06:13.652 INFO:teuthology.orchestra.run.smithi153.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-06-09T22:06:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:(59/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 70 kB/s | 45 kB 00:00 2024-06-09T22:06:13.678 INFO:teuthology.orchestra.run.smithi153.stdout:(147/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-06-09T22:06:13.753 INFO:teuthology.orchestra.run.smithi153.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-06-09T22:06:13.995 INFO:teuthology.orchestra.run.smithi153.stdout:(149/159): librados2-19.0.0-2946.g3d89accf.el9. 9.9 MB/s | 3.4 MB 00:00 2024-06-09T22:06:14.055 INFO:teuthology.orchestra.run.smithi104.stdout:(60/159): librdkafka-1.6.1-102.el9.x86_64.rpm 704 kB/s | 662 kB 00:00 2024-06-09T22:06:14.204 INFO:teuthology.orchestra.run.smithi153.stdout:(150/159): librbd1-19.0.0-2946.g3d89accf.el9.x8 6.0 MB/s | 3.1 MB 00:00 2024-06-09T22:06:14.279 INFO:teuthology.orchestra.run.smithi153.stdout:(151/159): NetworkManager-team-1.47.91-1.el9.x8 555 kB/s | 41 kB 00:00 2024-06-09T22:06:14.314 INFO:teuthology.orchestra.run.smithi104.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 39 kB/s | 25 kB 00:00 2024-06-09T22:06:14.354 INFO:teuthology.orchestra.run.smithi153.stdout:(152/159): NetworkManager-tui-1.47.91-1.el9.x86 3.2 MB/s | 248 kB 00:00 2024-06-09T22:06:14.413 INFO:teuthology.orchestra.run.smithi153.stdout:(153/159): NetworkManager-1.47.91-1.el9.x86_64. 3.5 MB/s | 2.3 MB 00:00 2024-06-09T22:06:14.438 INFO:teuthology.orchestra.run.smithi153.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 439 kB/s | 37 kB 00:00 2024-06-09T22:06:14.480 INFO:teuthology.orchestra.run.smithi153.stdout:(155/159): NetworkManager-libnm-1.47.91-1.el9.x 3.7 MB/s | 1.8 MB 00:00 2024-06-09T22:06:14.506 INFO:teuthology.orchestra.run.smithi153.stdout:(156/159): elfutils-libs-0.191-4.el9.x86_64.rpm 3.8 MB/s | 260 kB 00:00 2024-06-09T22:06:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:(157/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.6 MB/s | 207 kB 00:00 2024-06-09T22:06:14.556 INFO:teuthology.orchestra.run.smithi104.stdout:(62/159): libtool-2.4.6-45.el9.x86_64.rpm 592 kB/s | 582 kB 00:00 2024-06-09T22:06:14.731 INFO:teuthology.orchestra.run.smithi104.stdout:(63/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 70 kB/s | 47 kB 00:00 2024-06-09T22:06:14.834 INFO:teuthology.orchestra.run.smithi194.stdout:(159/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.0 MB/s | 16 MB 00:05 2024-06-09T22:06:14.841 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-06-09T22:06:14.841 INFO:teuthology.orchestra.run.smithi194.stdout:Total 11 MB/s | 225 MB 00:19 2024-06-09T22:06:14.873 INFO:teuthology.orchestra.run.smithi153.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 3.4 MB/s | 1.3 MB 00:00 2024-06-09T22:06:15.098 INFO:teuthology.orchestra.run.smithi153.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 4.0 MB/s | 2.4 MB 00:00 2024-06-09T22:06:15.105 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-09T22:06:15.105 INFO:teuthology.orchestra.run.smithi153.stdout:Total 12 MB/s | 225 MB 00:19 2024-06-09T22:06:15.190 INFO:teuthology.orchestra.run.smithi104.stdout:(64/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 345 kB/s | 292 kB 00:00 2024-06-09T22:06:15.340 INFO:teuthology.orchestra.run.smithi104.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 58 kB/s | 35 kB 00:00 2024-06-09T22:06:15.373 INFO:teuthology.orchestra.run.smithi104.stdout:(66/159): lua-5.4.4-4.el9.x86_64.rpm 231 kB/s | 188 kB 00:00 2024-06-09T22:06:16.028 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-06-09T22:06:16.187 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-06-09T22:06:16.187 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-06-09T22:06:16.330 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-09T22:06:16.407 INFO:teuthology.orchestra.run.smithi104.stdout:(67/159): openblas-openmp-0.3.21-2.el9.x86_64.r 3.9 MB/s | 4.7 MB 00:01 2024-06-09T22:06:16.449 INFO:teuthology.orchestra.run.smithi104.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 1.5 MB/s | 1.6 MB 00:01 2024-06-09T22:06:16.490 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-09T22:06:16.490 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-09T22:06:16.557 INFO:teuthology.orchestra.run.smithi104.stdout:(69/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.6 MB/s | 4.4 MB 00:01 2024-06-09T22:06:17.232 INFO:teuthology.orchestra.run.smithi104.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 170 kB/s | 133 kB 00:00 2024-06-09T22:06:17.416 INFO:teuthology.orchestra.run.smithi104.stdout:(71/159): postgresql-contrib-13.14-1.el9.x86_64 904 kB/s | 911 kB 00:01 2024-06-09T22:06:17.816 INFO:teuthology.orchestra.run.smithi104.stdout:(72/159): postgresql-server-13.14-1.el9.x86_64. 4.6 MB/s | 5.8 MB 00:01 2024-06-09T22:06:18.082 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-06-09T22:06:18.083 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-06-09T22:06:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 123 kB/s | 87 kB 00:00 2024-06-09T22:06:18.233 INFO:teuthology.orchestra.run.smithi104.stdout:(74/159): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:01 2024-06-09T22:06:18.313 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-09T22:06:18.314 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-09T22:06:18.792 INFO:teuthology.orchestra.run.smithi104.stdout:(75/159): python3-colorama-0.4.6-3.el9.noarch.r 92 kB/s | 61 kB 00:00 2024-06-09T22:06:19.042 INFO:teuthology.orchestra.run.smithi104.stdout:(76/159): python3-babel-2.9.1-2.el9.noarch.rpm 4.9 MB/s | 6.0 MB 00:01 2024-06-09T22:06:19.092 INFO:teuthology.orchestra.run.smithi104.stdout:(77/159): python3-devel-3.9.18-3.el9.x86_64.rpm 284 kB/s | 244 kB 00:00 2024-06-09T22:06:19.642 INFO:teuthology.orchestra.run.smithi104.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 35 kB/s | 19 kB 00:00 2024-06-09T22:06:19.668 INFO:teuthology.orchestra.run.smithi104.stdout:(79/159): python3-jsonpatch-1.21-16.el9.noarch. 43 kB/s | 26 kB 00:00 2024-06-09T22:06:19.693 INFO:teuthology.orchestra.run.smithi104.stdout:(80/159): python3-jinja2-2.11.3-6.el9.noarch.rp 276 kB/s | 248 kB 00:00 2024-06-09T22:06:20.285 INFO:teuthology.orchestra.run.smithi104.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 56 kB/s | 35 kB 00:00 2024-06-09T22:06:20.476 INFO:teuthology.orchestra.run.smithi104.stdout:(82/159): python3-mako-1.1.4-6.el9.noarch.rpm 207 kB/s | 172 kB 00:00 2024-06-09T22:06:21.118 INFO:teuthology.orchestra.run.smithi104.stdout:(83/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.6 MB/s | 5.1 MB 00:01 2024-06-09T22:06:21.235 INFO:teuthology.orchestra.run.smithi104.stdout:(84/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 264 kB/s | 250 kB 00:00 2024-06-09T22:06:21.344 INFO:teuthology.orchestra.run.smithi104.stdout:(85/159): python3-oauthlib-3.1.1-5.el9.noarch.r 256 kB/s | 222 kB 00:00 2024-06-09T22:06:21.794 INFO:teuthology.orchestra.run.smithi104.stdout:(86/159): python3-packaging-20.9-5.el9.noarch.r 114 kB/s | 77 kB 00:00 2024-06-09T22:06:21.836 INFO:teuthology.orchestra.run.smithi104.stdout:(87/159): python3-prettytable-0.7.2-27.el9.noar 70 kB/s | 42 kB 00:00 2024-06-09T22:06:22.195 INFO:teuthology.orchestra.run.smithi104.stdout:(88/159): python3-protobuf-3.14.0-13.el9.noarch 327 kB/s | 269 kB 00:00 2024-06-09T22:06:22.570 INFO:teuthology.orchestra.run.smithi104.stdout:(89/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 205 kB/s | 159 kB 00:00 2024-06-09T22:06:22.645 INFO:teuthology.orchestra.run.smithi104.stdout:(90/159): python3-pyasn1-modules-0.4.8-6.el9.no 345 kB/s | 279 kB 00:00 2024-06-09T22:06:22.845 INFO:teuthology.orchestra.run.smithi104.stdout:(91/159): python3-pytz-2021.1-5.el9.noarch.rpm 78 kB/s | 51 kB 00:00 2024-06-09T22:06:23.237 INFO:teuthology.orchestra.run.smithi104.stdout:(92/159): python3-requests-oauthlib-1.3.0-12.el 80 kB/s | 54 kB 00:00 2024-06-09T22:06:23.454 INFO:teuthology.orchestra.run.smithi104.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 68 kB/s | 42 kB 00:00 2024-06-09T22:06:23.854 INFO:teuthology.orchestra.run.smithi104.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 72 kB/s | 44 kB 00:00 2024-06-09T22:06:24.130 INFO:teuthology.orchestra.run.smithi104.stdout:(95/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-06-09T22:06:24.338 INFO:teuthology.orchestra.run.smithi104.stdout:(96/159): qatlib-24.02.0-1.el9.x86_64.rpm 250 kB/s | 221 kB 00:00 2024-06-09T22:06:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-06-09T22:06:24.454 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-06-09T22:06:24.493 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-06-09T22:06:24.554 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-06-09T22:06:24.580 INFO:teuthology.orchestra.run.smithi104.stdout:(97/159): qatlib-service-24.02.0-1.el9.x86_64.r 49 kB/s | 36 kB 00:00 2024-06-09T22:06:24.588 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-06-09T22:06:24.615 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-06-09T22:06:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-09T22:06:24.656 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-06-09T22:06:24.693 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-06-09T22:06:24.731 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-06-09T22:06:24.772 INFO:teuthology.orchestra.run.smithi104.stdout:(98/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 74 kB/s | 47 kB 00:00 2024-06-09T22:06:24.785 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-06-09T22:06:24.818 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-06-09T22:06:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-06-09T22:06:24.887 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-06-09T22:06:24.987 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-06-09T22:06:25.082 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-09T22:06:25.204 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-09T22:06:25.233 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-09T22:06:25.239 INFO:teuthology.orchestra.run.smithi104.stdout:(99/159): socat-1.7.4.1-5.el9.x86_64.rpm 339 kB/s | 305 kB 00:00 2024-06-09T22:06:25.281 INFO:teuthology.orchestra.run.smithi104.stdout:(100/159): uuid-1.6.2-55.el9.x86_64.rpm 86 kB/s | 58 kB 00:00 2024-06-09T22:06:25.284 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-06-09T22:06:25.325 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-09T22:06:25.353 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 10/170 2024-06-09T22:06:25.386 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-06-09T22:06:25.395 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-09T22:06:25.498 INFO:teuthology.orchestra.run.smithi104.stdout:(101/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 129 kB/s | 94 kB 00:00 2024-06-09T22:06:25.542 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-09T22:06:25.581 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-09T22:06:25.585 INFO:teuthology.orchestra.run.smithi104.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:25.586 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-06-09T22:06:25.626 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-06-09T22:06:25.655 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-09T22:06:25.671 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-09T22:06:25.698 INFO:teuthology.orchestra.run.smithi104.stdout:(102/159): lua-devel-5.4.4-4.el9.x86_64.rpm 146 kB/s | 22 kB 00:00 2024-06-09T22:06:25.701 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 10/170 2024-06-09T22:06:25.701 INFO:teuthology.orchestra.run.smithi104.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T22:06:25.733 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-06-09T22:06:25.750 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-09T22:06:25.773 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 15/170 2024-06-09T22:06:25.835 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 15/170 2024-06-09T22:06:25.857 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-09T22:06:25.879 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-09T22:06:25.882 INFO:teuthology.orchestra.run.smithi104.stdout:(103/159): protobuf-compiler-3.14.0-13.el9.x86_ 4.6 MB/s | 863 kB 00:00 2024-06-09T22:06:25.915 INFO:teuthology.orchestra.run.smithi104.stdout:(104/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 71 kB/s | 45 kB 00:00 2024-06-09T22:06:25.924 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-06-09T22:06:25.968 INFO:teuthology.orchestra.run.smithi104.stdout:(105/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 87 kB/s | 64 kB 00:00 2024-06-09T22:06:25.973 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-06-09T22:06:26.004 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-09T22:06:26.057 INFO:teuthology.orchestra.run.smithi104.stdout:(106/159): abseil-cpp-20211102.0-3.el9.x86_64.r 3.1 MB/s | 548 kB 00:00 2024-06-09T22:06:26.065 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-09T22:06:26.091 INFO:teuthology.orchestra.run.smithi104.stdout:(107/159): grpc-data-1.46.7-10.el9.noarch.rpm 197 kB/s | 19 kB 00:00 2024-06-09T22:06:26.106 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-09T22:06:26.116 INFO:teuthology.orchestra.run.smithi104.stdout:(108/159): gperftools-libs-2.9.1-2.el9.x86_64.r 1.5 MB/s | 309 kB 00:00 2024-06-09T22:06:26.130 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 15/170 2024-06-09T22:06:26.141 INFO:teuthology.orchestra.run.smithi104.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 516 kB/s | 26 kB 00:00 2024-06-09T22:06:26.175 INFO:teuthology.orchestra.run.smithi104.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 828 kB/s | 49 kB 00:00 2024-06-09T22:06:26.193 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 15/170 2024-06-09T22:06:26.194 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-09T22:06:26.200 INFO:teuthology.orchestra.run.smithi104.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-06-09T22:06:26.214 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-09T22:06:26.226 INFO:teuthology.orchestra.run.smithi104.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-06-09T22:06:26.232 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-09T22:06:26.276 INFO:teuthology.orchestra.run.smithi104.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-06-09T22:06:26.301 INFO:teuthology.orchestra.run.smithi104.stdout:(114/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-06-09T22:06:26.326 INFO:teuthology.orchestra.run.smithi104.stdout:(115/159): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-06-09T22:06:26.352 INFO:teuthology.orchestra.run.smithi104.stdout:(116/159): python3-certifi-2021.10.8-2.el9.noar 598 kB/s | 15 kB 00:00 2024-06-09T22:06:26.377 INFO:teuthology.orchestra.run.smithi104.stdout:(117/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.6 MB/s | 839 kB 00:00 2024-06-09T22:06:26.389 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-09T22:06:26.402 INFO:teuthology.orchestra.run.smithi104.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-06-09T22:06:26.422 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-09T22:06:26.450 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-09T22:06:26.461 INFO:teuthology.orchestra.run.smithi104.stdout:(119/159): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-06-09T22:06:26.487 INFO:teuthology.orchestra.run.smithi104.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-06-09T22:06:26.512 INFO:teuthology.orchestra.run.smithi104.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-06-09T22:06:26.537 INFO:teuthology.orchestra.run.smithi104.stdout:(122/159): python3-google-auth-2.29.0-2.el9.noa 2.7 MB/s | 205 kB 00:00 2024-06-09T22:06:26.548 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-09T22:06:26.552 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-09T22:06:26.554 INFO:teuthology.orchestra.run.smithi104.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-06-09T22:06:26.571 INFO:teuthology.orchestra.run.smithi104.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 317 kB/s | 11 kB 00:00 2024-06-09T22:06:26.588 INFO:teuthology.orchestra.run.smithi104.stdout:(125/159): python3-jaraco-classes-3.2.1-5.el9.n 526 kB/s | 18 kB 00:00 2024-06-09T22:06:26.605 INFO:teuthology.orchestra.run.smithi104.stdout:(126/159): python3-jaraco-collections-3.0.0-8.e 689 kB/s | 23 kB 00:00 2024-06-09T22:06:26.622 INFO:teuthology.orchestra.run.smithi104.stdout:(127/159): python3-jaraco-functools-3.5.0-2.el9 578 kB/s | 19 kB 00:00 2024-06-09T22:06:26.639 INFO:teuthology.orchestra.run.smithi104.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 585 kB/s | 20 kB 00:00 2024-06-09T22:06:26.643 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-09T22:06:26.656 INFO:teuthology.orchestra.run.smithi104.stdout:(129/159): python3-logutils-0.3.5-21.el9.noarch 2.7 MB/s | 46 kB 00:00 2024-06-09T22:06:26.689 INFO:teuthology.orchestra.run.smithi104.stdout:(130/159): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-06-09T22:06:26.714 INFO:teuthology.orchestra.run.smithi104.stdout:(131/159): python3-natsort-7.1.1-5.el9.noarch.r 2.2 MB/s | 58 kB 00:00 2024-06-09T22:06:26.719 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-09T22:06:26.748 INFO:teuthology.orchestra.run.smithi104.stdout:(132/159): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-06-09T22:06:26.774 INFO:teuthology.orchestra.run.smithi104.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 655 kB/s | 16 kB 00:00 2024-06-09T22:06:26.780 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-09T22:06:26.815 INFO:teuthology.orchestra.run.smithi104.stdout:(134/159): python3-grpcio-1.46.7-10.el9.x86_64. 6.2 MB/s | 2.0 MB 00:00 2024-06-09T22:06:26.841 INFO:teuthology.orchestra.run.smithi104.stdout:(135/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-06-09T22:06:26.848 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-09T22:06:26.866 INFO:teuthology.orchestra.run.smithi104.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-06-09T22:06:26.874 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-09T22:06:26.891 INFO:teuthology.orchestra.run.smithi104.stdout:(137/159): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-06-09T22:06:26.916 INFO:teuthology.orchestra.run.smithi104.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-06-09T22:06:26.942 INFO:teuthology.orchestra.run.smithi104.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-06-09T22:06:26.960 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-06-09T22:06:26.965 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-09T22:06:26.967 INFO:teuthology.orchestra.run.smithi104.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 710 kB/s | 36 kB 00:00 2024-06-09T22:06:26.992 INFO:teuthology.orchestra.run.smithi104.stdout:(141/159): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-06-09T22:06:27.018 INFO:teuthology.orchestra.run.smithi104.stdout:(142/159): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-06-09T22:06:27.043 INFO:teuthology.orchestra.run.smithi104.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-06-09T22:06:27.053 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-06-09T22:06:27.068 INFO:teuthology.orchestra.run.smithi104.stdout:(144/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-06-09T22:06:27.083 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-06-09T22:06:27.093 INFO:teuthology.orchestra.run.smithi104.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-06-09T22:06:27.118 INFO:teuthology.orchestra.run.smithi104.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-06-09T22:06:27.126 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-06-09T22:06:27.144 INFO:teuthology.orchestra.run.smithi104.stdout:(147/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-06-09T22:06:27.153 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-09T22:06:27.192 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-06-09T22:06:27.194 INFO:teuthology.orchestra.run.smithi104.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-06-09T22:06:27.235 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-06-09T22:06:27.263 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-06-09T22:06:27.277 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-06-09T22:06:27.310 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-06-09T22:06:27.338 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-06-09T22:06:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-06-09T22:06:27.369 INFO:teuthology.orchestra.run.smithi104.stdout:(149/159): NetworkManager-1.47.91-1.el9.x86_64. 13 MB/s | 2.3 MB 00:00 2024-06-09T22:06:27.387 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-06-09T22:06:27.406 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-06-09T22:06:27.428 INFO:teuthology.orchestra.run.smithi104.stdout:(150/159): NetworkManager-libnm-1.47.91-1.el9.x 31 MB/s | 1.8 MB 00:00 2024-06-09T22:06:27.430 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-06-09T22:06:27.461 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-06-09T22:06:27.487 INFO:teuthology.orchestra.run.smithi104.stdout:(151/159): librbd1-19.0.0-2946.g3d89accf.el9.x8 9.1 MB/s | 3.1 MB 00:00 2024-06-09T22:06:27.491 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-06-09T22:06:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-06-09T22:06:27.512 INFO:teuthology.orchestra.run.smithi104.stdout:(152/159): NetworkManager-team-1.47.91-1.el9.x8 495 kB/s | 41 kB 00:00 2024-06-09T22:06:27.526 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-06-09T22:06:27.529 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-06-09T22:06:27.554 INFO:teuthology.orchestra.run.smithi104.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 878 kB/s | 37 kB 00:00 2024-06-09T22:06:27.572 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-06-09T22:06:27.596 INFO:teuthology.orchestra.run.smithi104.stdout:(154/159): NetworkManager-tui-1.47.91-1.el9.x86 2.2 MB/s | 248 kB 00:00 2024-06-09T22:06:27.601 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-06-09T22:06:27.605 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-06-09T22:06:27.621 INFO:teuthology.orchestra.run.smithi104.stdout:(155/159): elfutils-libelf-0.191-4.el9.x86_64.r 3.0 MB/s | 207 kB 00:00 2024-06-09T22:06:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-06-09T22:06:27.637 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 34/170 2024-06-09T22:06:27.655 INFO:teuthology.orchestra.run.smithi104.stdout:(156/159): elfutils-libs-0.191-4.el9.x86_64.rpm 4.3 MB/s | 260 kB 00:00 2024-06-09T22:06:27.672 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 35/170 2024-06-09T22:06:27.701 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 36/170 2024-06-09T22:06:27.702 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-06-09T22:06:27.713 INFO:teuthology.orchestra.run.smithi104.stdout:(157/159): librados2-19.0.0-2946.g3d89accf.el9. 5.7 MB/s | 3.4 MB 00:00 2024-06-09T22:06:27.747 INFO:teuthology.orchestra.run.smithi104.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 11 MB/s | 1.3 MB 00:00 2024-06-09T22:06:27.755 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-06-09T22:06:27.770 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-06-09T22:06:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-06-09T22:06:27.824 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-06-09T22:06:27.829 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-06-09T22:06:27.839 INFO:teuthology.orchestra.run.smithi104.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2024-06-09T22:06:27.844 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-06-09T22:06:27.844 INFO:teuthology.orchestra.run.smithi104.stdout:Total 7.9 MB/s | 225 MB 00:28 2024-06-09T22:06:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-06-09T22:06:27.906 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-06-09T22:06:27.940 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 34/170 2024-06-09T22:06:27.983 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 35/170 2024-06-09T22:06:28.012 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 36/170 2024-06-09T22:06:28.031 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-06-09T22:06:28.084 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-06-09T22:06:28.085 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-06-09T22:06:28.127 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-06-09T22:06:28.202 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-06-09T22:06:28.336 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-06-09T22:06:28.397 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-06-09T22:06:28.597 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-06-09T22:06:28.679 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-06-09T22:06:28.709 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-06-09T22:06:28.745 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-06-09T22:06:28.794 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-06-09T22:06:28.833 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 47/170 2024-06-09T22:06:28.874 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-06-09T22:06:28.934 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-06-09T22:06:29.009 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-06-09T22:06:29.022 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-06-09T22:06:29.028 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-06-09T22:06:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-06-09T22:06:29.074 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-06-09T22:06:29.093 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-06-09T22:06:29.123 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-06-09T22:06:29.147 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-06-09T22:06:29.153 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 47/170 2024-06-09T22:06:29.182 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-06-09T22:06:29.182 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-06-09T22:06:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-06-09T22:06:29.213 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-06-09T22:06:29.244 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-06-09T22:06:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-06-09T22:06:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-06-09T22:06:29.349 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-06-09T22:06:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-06-09T22:06:29.397 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-06-09T22:06:29.422 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-06-09T22:06:29.453 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-06-09T22:06:29.465 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-06-09T22:06:29.526 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-06-09T22:06:29.564 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-06-09T22:06:29.599 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-06-09T22:06:29.601 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-06-09T22:06:29.642 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-06-09T22:06:29.648 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-06-09T22:06:29.683 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-06-09T22:06:29.684 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-06-09T22:06:29.727 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-06-09T22:06:29.770 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-06-09T22:06:29.777 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-06-09T22:06:29.808 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 63/170 2024-06-09T22:06:29.907 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-06-09T22:06:29.907 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-06-09T22:06:29.946 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-06-09T22:06:29.987 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-06-09T22:06:30.074 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-06-09T22:06:30.083 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-06-09T22:06:30.099 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-06-09T22:06:30.113 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 63/170 2024-06-09T22:06:30.144 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-06-09T22:06:30.170 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-06-09T22:06:30.187 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-06-09T22:06:30.194 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-06-09T22:06:30.269 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-06-09T22:06:30.319 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-06-09T22:06:30.356 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-06-09T22:06:30.383 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-06-09T22:06:30.397 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-06-09T22:06:30.426 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-06-09T22:06:30.427 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-06-09T22:06:30.455 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-06-09T22:06:30.465 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-06-09T22:06:30.489 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-06-09T22:06:30.496 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-06-09T22:06:30.526 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-06-09T22:06:30.528 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-06-09T22:06:30.560 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-06-09T22:06:30.561 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-06-09T22:06:30.590 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-06-09T22:06:30.602 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-06-09T22:06:30.624 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-06-09T22:06:30.644 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-06-09T22:06:30.658 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-06-09T22:06:30.679 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-06-09T22:06:30.699 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-06-09T22:06:30.734 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-06-09T22:06:30.765 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-06-09T22:06:30.809 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-06-09T22:06:30.822 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-06-09T22:06:30.840 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-06-09T22:06:30.865 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-06-09T22:06:30.891 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-06-09T22:06:30.904 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-06-09T22:06:30.951 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-06-09T22:06:30.993 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-06-09T22:06:31.041 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-06-09T22:06:31.050 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-06-09T22:06:31.105 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-06-09T22:06:31.338 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-06-09T22:06:31.338 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-06-09T22:06:31.765 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-06-09T22:06:31.814 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-06-09T22:06:31.841 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-06-09T22:06:31.847 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-06-09T22:06:31.904 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-06-09T22:06:31.928 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-06-09T22:06:31.948 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-06-09T22:06:32.002 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-06-09T22:06:32.019 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-06-09T22:06:32.047 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-06-09T22:06:32.075 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-06-09T22:06:32.143 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-06-09T22:06:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-06-09T22:06:32.175 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-06-09T22:06:32.242 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-06-09T22:06:32.281 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-06-09T22:06:32.362 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-06-09T22:06:32.398 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-06-09T22:06:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-06-09T22:06:32.476 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-06-09T22:06:32.487 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-06-09T22:06:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-06-09T22:06:33.113 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-06-09T22:06:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-06-09T22:06:33.228 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-06-09T22:06:33.337 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-06-09T22:06:34.440 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-06-09T22:06:34.526 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-06-09T22:06:34.545 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-06-09T22:06:34.577 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-06-09T22:06:34.579 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-06-09T22:06:34.666 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-06-09T22:06:34.694 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-06-09T22:06:34.727 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-06-09T22:06:34.882 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-06-09T22:06:34.925 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 103/170 2024-06-09T22:06:35.016 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 103/170 2024-06-09T22:06:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-06-09T22:06:35.046 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 104/170 2024-06-09T22:06:35.082 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 103/170 2024-06-09T22:06:35.083 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-06-09T22:06:35.172 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 103/170 2024-06-09T22:06:35.204 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 104/170 2024-06-09T22:06:35.239 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-06-09T22:06:35.554 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-06-09T22:06:35.578 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 107/170 2024-06-09T22:06:35.632 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 107/170 2024-06-09T22:06:35.663 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 108/170 2024-06-09T22:06:35.705 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-06-09T22:06:35.735 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 107/170 2024-06-09T22:06:35.789 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 107/170 2024-06-09T22:06:35.820 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 108/170 2024-06-09T22:06:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-06-09T22:06:37.685 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-06-09T22:06:37.722 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-06-09T22:06:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-06-09T22:06:37.834 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-06-09T22:06:37.860 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-06-09T22:06:37.906 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-06-09T22:06:37.914 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-09T22:06:37.951 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-09T22:06:38.057 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-09T22:06:38.108 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-09T22:06:38.289 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-06-09T22:06:38.344 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-09T22:06:38.466 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-09T22:06:38.488 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-09T22:06:38.504 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-09T22:06:38.520 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 110/170 2024-06-09T22:06:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-09T22:06:38.602 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 110/170 2024-06-09T22:06:38.603 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-09T22:06:38.603 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-09T22:06:38.603 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:06:38.608 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 10/170 2024-06-09T22:06:38.630 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 111/170 2024-06-09T22:06:38.647 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-06-09T22:06:38.649 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-09T22:06:38.694 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 110/170 2024-06-09T22:06:38.807 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 110/170 2024-06-09T22:06:38.807 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-09T22:06:38.808 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-09T22:06:38.808 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:06:38.835 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-06-09T22:06:38.837 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 111/170 2024-06-09T22:06:39.006 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-06-09T22:06:39.034 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-09T22:06:39.121 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-09T22:06:39.143 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 15/170 2024-06-09T22:06:39.206 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 15/170 2024-06-09T22:06:39.228 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-09T22:06:39.250 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-09T22:06:39.444 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-09T22:06:39.625 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-09T22:06:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-09T22:06:39.826 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-09T22:06:39.921 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-09T22:06:40.011 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-09T22:06:40.198 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-09T22:06:40.314 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-06-09T22:06:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-06-09T22:06:40.441 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-06-09T22:06:40.475 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-06-09T22:06:40.539 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-06-09T22:06:40.574 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-06-09T22:06:40.618 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-06-09T22:06:40.650 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-06-09T22:06:40.678 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-06-09T22:06:40.746 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-06-09T22:06:40.800 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-06-09T22:06:40.831 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-06-09T22:06:40.866 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-06-09T22:06:40.942 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-06-09T22:06:40.977 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 34/170 2024-06-09T22:06:41.020 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 35/170 2024-06-09T22:06:41.049 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 36/170 2024-06-09T22:06:41.118 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-06-09T22:06:41.164 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-06-09T22:06:41.237 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-06-09T22:06:41.377 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-06-09T22:06:41.433 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-06-09T22:06:41.946 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-06-09T22:06:42.036 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-06-09T22:06:42.074 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-06-09T22:06:42.111 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-06-09T22:06:42.159 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-06-09T22:06:42.197 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 47/170 2024-06-09T22:06:42.238 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-06-09T22:06:42.393 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-06-09T22:06:42.466 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-06-09T22:06:42.520 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-06-09T22:06:42.602 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-06-09T22:06:42.633 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-06-09T22:06:42.670 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-06-09T22:06:42.716 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-06-09T22:06:42.753 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-06-09T22:06:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-06-09T22:06:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-06-09T22:06:42.973 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-06-09T22:06:43.014 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-06-09T22:06:43.055 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-06-09T22:06:43.149 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-06-09T22:06:43.181 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 63/170 2024-06-09T22:06:43.232 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-06-09T22:06:43.263 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-06-09T22:06:43.288 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-06-09T22:06:43.378 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-06-09T22:06:43.505 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-06-09T22:06:43.544 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-06-09T22:06:43.580 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-06-09T22:06:43.639 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-06-09T22:06:43.679 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-06-09T22:06:43.711 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-06-09T22:06:43.740 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-06-09T22:06:43.775 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-06-09T22:06:43.808 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-06-09T22:06:43.850 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-06-09T22:06:43.884 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-06-09T22:06:43.971 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-06-09T22:06:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-06-09T22:06:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-06-09T22:06:44.118 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-06-09T22:06:44.215 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-06-09T22:06:44.263 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-06-09T22:06:44.980 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-06-09T22:06:45.029 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-06-09T22:06:45.063 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-06-09T22:06:45.161 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-06-09T22:06:45.227 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-06-09T22:06:45.271 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-06-09T22:06:45.358 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-06-09T22:06:45.390 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-06-09T22:06:45.568 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-06-09T22:06:45.588 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-06-09T22:06:45.668 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-06-09T22:06:46.310 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-06-09T22:06:46.419 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-06-09T22:06:47.669 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-06-09T22:06:47.750 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-06-09T22:06:47.778 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-06-09T22:06:47.811 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-06-09T22:06:48.122 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-06-09T22:06:48.166 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 103/170 2024-06-09T22:06:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 103/170 2024-06-09T22:06:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 104/170 2024-06-09T22:06:48.332 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-06-09T22:06:48.802 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-06-09T22:06:48.827 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 107/170 2024-06-09T22:06:48.889 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 107/170 2024-06-09T22:06:48.911 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 108/170 2024-06-09T22:06:50.132 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 111/170 2024-06-09T22:06:50.132 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-06-09T22:06:50.132 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-06-09T22:06:50.132 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-06-09T22:06:50.132 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-06-09T22:06:50.132 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-06-09T22:06:50.132 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /root 2024-06-09T22:06:50.132 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-06-09T22:06:50.132 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:06:50.294 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 112/170 2024-06-09T22:06:50.319 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 112/170 2024-06-09T22:06:50.320 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:50.320 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-09T22:06:50.320 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:06:50.501 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 111/170 2024-06-09T22:06:50.501 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-06-09T22:06:50.501 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-06-09T22:06:50.501 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-06-09T22:06:50.501 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-06-09T22:06:50.501 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-06-09T22:06:50.501 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-06-09T22:06:50.501 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-06-09T22:06:50.502 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:06:50.662 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 112/170 2024-06-09T22:06:50.678 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 112/170 2024-06-09T22:06:50.678 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:50.678 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-09T22:06:50.678 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:06:51.192 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-09T22:06:51.225 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-09T22:06:51.772 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 113/170 2024-06-09T22:06:51.776 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-09T22:06:51.798 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 113/170 2024-06-09T22:06:51.798 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:51.798 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-09T22:06:51.798 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-09T22:06:51.798 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-09T22:06:51.799 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:06:51.811 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 110/170 2024-06-09T22:06:51.898 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 110/170 2024-06-09T22:06:51.898 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-09T22:06:51.898 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-09T22:06:51.898 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:06:51.929 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 111/170 2024-06-09T22:06:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 114/170 2024-06-09T22:06:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 114/170 2024-06-09T22:06:52.037 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:52.037 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-09T22:06:52.037 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-09T22:06:52.037 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-09T22:06:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:06:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 113/170 2024-06-09T22:06:52.199 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 113/170 2024-06-09T22:06:52.199 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:52.199 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-09T22:06:52.199 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-09T22:06:52.199 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-09T22:06:52.199 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:06:52.474 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 114/170 2024-06-09T22:06:52.490 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 115/170 2024-06-09T22:06:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 114/170 2024-06-09T22:06:52.497 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:52.497 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-09T22:06:52.497 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-09T22:06:52.497 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-09T22:06:52.497 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:06:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 115/170 2024-06-09T22:06:52.513 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:52.513 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-09T22:06:52.513 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-09T22:06:52.513 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-09T22:06:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:06:52.611 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-06-09T22:06:52.668 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-06-09T22:06:52.723 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-06-09T22:06:52.909 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-06-09T22:06:52.941 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 115/170 2024-06-09T22:06:52.956 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-06-09T22:06:52.964 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 115/170 2024-06-09T22:06:52.964 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:52.964 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-09T22:06:52.964 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-09T22:06:52.964 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-09T22:06:52.964 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:06:53.034 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-06-09T22:06:53.063 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-06-09T22:06:53.120 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-06-09T22:06:53.122 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-06-09T22:06:53.175 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-06-09T22:06:53.262 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-06-09T22:06:53.315 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-06-09T22:06:53.347 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-06-09T22:06:53.361 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-06-09T22:06:53.374 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-06-09T22:06:53.400 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-06-09T22:06:53.487 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-06-09T22:06:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-06-09T22:06:53.773 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-06-09T22:06:53.818 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 127/170 2024-06-09T22:06:53.833 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-06-09T22:06:53.848 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 127/170 2024-06-09T22:06:53.873 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-06-09T22:06:53.879 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 128/170 2024-06-09T22:06:53.901 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-06-09T22:06:53.941 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 129/170 2024-06-09T22:06:53.971 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 130/170 2024-06-09T22:06:54.353 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 127/170 2024-06-09T22:06:54.383 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 127/170 2024-06-09T22:06:54.414 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 128/170 2024-06-09T22:06:54.476 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 129/170 2024-06-09T22:06:54.514 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 130/170 2024-06-09T22:06:55.002 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 130/170 2024-06-09T22:06:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 131/170 2024-06-09T22:06:55.551 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 130/170 2024-06-09T22:06:55.606 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 131/170 2024-06-09T22:06:56.019 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 131/170 2024-06-09T22:06:56.047 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 132/170 2024-06-09T22:06:56.150 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 132/170 2024-06-09T22:06:56.312 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 133/170 2024-06-09T22:06:56.358 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 134/170 2024-06-09T22:06:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 134/170 2024-06-09T22:06:56.381 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:56.381 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-09T22:06:56.381 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-09T22:06:56.382 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-09T22:06:56.382 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:06:56.425 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 135/170 2024-06-09T22:06:56.436 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 135/170 2024-06-09T22:06:56.535 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 136/170 2024-06-09T22:06:56.577 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 131/170 2024-06-09T22:06:56.582 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 137/170 2024-06-09T22:06:56.608 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 132/170 2024-06-09T22:06:56.710 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 132/170 2024-06-09T22:06:56.881 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 133/170 2024-06-09T22:06:56.910 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 134/170 2024-06-09T22:06:56.934 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 134/170 2024-06-09T22:06:56.934 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:56.934 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-09T22:06:56.934 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-09T22:06:56.934 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-09T22:06:56.935 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:06:56.985 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 135/170 2024-06-09T22:06:56.997 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 135/170 2024-06-09T22:06:57.094 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 136/170 2024-06-09T22:06:57.143 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 137/170 2024-06-09T22:06:57.475 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-06-09T22:06:57.503 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 139/170 2024-06-09T22:06:57.525 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 139/170 2024-06-09T22:06:57.525 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:57.525 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-09T22:06:57.525 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-09T22:06:57.525 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-09T22:06:57.525 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:06:57.569 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 140/170 2024-06-09T22:06:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 140/170 2024-06-09T22:06:57.591 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:57.591 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-09T22:06:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:06:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 141/170 2024-06-09T22:06:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 141/170 2024-06-09T22:06:57.659 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:57.659 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-09T22:06:57.659 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-09T22:06:57.659 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-09T22:06:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:06:57.963 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/170 2024-06-09T22:06:57.985 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/170 2024-06-09T22:06:57.985 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:57.985 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-09T22:06:57.985 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-09T22:06:57.985 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-09T22:06:57.985 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:06:58.046 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-06-09T22:06:58.080 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 139/170 2024-06-09T22:06:58.104 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 139/170 2024-06-09T22:06:58.104 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:58.104 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-09T22:06:58.104 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-09T22:06:58.104 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-09T22:06:58.104 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:06:58.147 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 140/170 2024-06-09T22:06:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 140/170 2024-06-09T22:06:58.170 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:58.170 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-09T22:06:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:06:58.214 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 141/170 2024-06-09T22:06:58.237 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 141/170 2024-06-09T22:06:58.237 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:58.237 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-09T22:06:58.237 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-09T22:06:58.238 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-09T22:06:58.238 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:06:58.540 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/170 2024-06-09T22:06:58.564 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/170 2024-06-09T22:06:58.564 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:06:58.564 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-09T22:06:58.564 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-09T22:06:58.564 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-09T22:06:58.564 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:07:02.491 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 143/170 2024-06-09T22:07:02.532 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 144/170 2024-06-09T22:07:02.573 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 145/170 2024-06-09T22:07:02.604 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-06-09T22:07:02.632 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-06-09T22:07:02.815 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 148/170 2024-06-09T22:07:02.861 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-06-09T22:07:02.894 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 150/170 2024-06-09T22:07:02.934 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-06-09T22:07:02.963 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : NetworkManager-team-1:1.47.91-1.el9.x86_64 152/170 2024-06-09T22:07:03.002 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : NetworkManager-tui-1:1.47.91-1.el9.x86_64 153/170 2024-06-09T22:07:03.051 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 143/170 2024-06-09T22:07:03.093 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 144/170 2024-06-09T22:07:03.142 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 145/170 2024-06-09T22:07:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-06-09T22:07:03.210 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-06-09T22:07:03.395 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 148/170 2024-06-09T22:07:03.410 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 111/170 2024-06-09T22:07:03.410 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2024-06-09T22:07:03.410 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2024-06-09T22:07:03.410 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2024-06-09T22:07:03.410 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2024-06-09T22:07:03.411 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2024-06-09T22:07:03.411 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /root 2024-06-09T22:07:03.411 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2024-06-09T22:07:03.411 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:07:03.439 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-06-09T22:07:03.471 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 150/170 2024-06-09T22:07:03.512 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-06-09T22:07:03.542 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-team-1:1.47.91-1.el9.x86_64 152/170 2024-06-09T22:07:03.575 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 112/170 2024-06-09T22:07:03.588 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-tui-1:1.47.91-1.el9.x86_64 153/170 2024-06-09T22:07:03.590 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 112/170 2024-06-09T22:07:03.590 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:07:03.591 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-09T22:07:03.591 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:07:03.599 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-09T22:07:03.640 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-09T22:07:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-09T22:07:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-09T22:07:04.185 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-09T22:07:04.206 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-09T22:07:04.498 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 155/170 2024-06-09T22:07:04.594 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-06-09T22:07:04.636 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-06-09T22:07:04.724 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-06-09T22:07:04.764 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-06-09T22:07:04.789 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-09T22:07:04.814 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-06-09T22:07:04.841 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-06-09T22:07:04.841 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-09T22:07:04.925 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-09T22:07:04.950 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-06-09T22:07:04.950 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-09T22:07:05.018 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-09T22:07:05.042 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-06-09T22:07:05.047 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-09T22:07:05.047 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-09T22:07:05.059 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 155/170 2024-06-09T22:07:05.062 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 113/170 2024-06-09T22:07:05.087 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 113/170 2024-06-09T22:07:05.087 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:07:05.087 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-09T22:07:05.087 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-09T22:07:05.087 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-09T22:07:05.088 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:07:05.148 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-06-09T22:07:05.190 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-06-09T22:07:05.227 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-09T22:07:05.228 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-09T22:07:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-06-09T22:07:05.310 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-06-09T22:07:05.312 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 114/170 2024-06-09T22:07:05.312 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-09T22:07:05.334 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-09T22:07:05.335 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 114/170 2024-06-09T22:07:05.335 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:07:05.335 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-09T22:07:05.335 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-09T22:07:05.335 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-09T22:07:05.335 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:07:05.336 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-06-09T22:07:05.368 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-06-09T22:07:05.395 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-06-09T22:07:05.395 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-09T22:07:05.400 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-09T22:07:05.488 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-09T22:07:05.512 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-06-09T22:07:05.512 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-09T22:07:05.622 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-09T22:07:05.647 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-06-09T22:07:05.653 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-09T22:07:05.653 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-09T22:07:05.779 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 115/170 2024-06-09T22:07:05.802 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 115/170 2024-06-09T22:07:05.802 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:07:05.802 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-09T22:07:05.802 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-09T22:07:05.802 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-09T22:07:05.802 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:07:05.832 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-09T22:07:05.832 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-09T22:07:05.909 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-06-09T22:07:05.925 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-09T22:07:05.949 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-06-09T22:07:05.966 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-06-09T22:07:06.013 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-09T22:07:06.022 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-06-09T22:07:06.208 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-06-09T22:07:06.263 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-06-09T22:07:06.341 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-06-09T22:07:06.429 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-06-09T22:07:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-06-09T22:07:06.605 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-06-09T22:07:06.637 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-06-09T22:07:06.664 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-06-09T22:07:07.127 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 127/170 2024-06-09T22:07:07.159 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 127/170 2024-06-09T22:07:07.194 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 128/170 2024-06-09T22:07:07.257 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 129/170 2024-06-09T22:07:07.286 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 130/170 2024-06-09T22:07:08.317 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 130/170 2024-06-09T22:07:08.354 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 131/170 2024-06-09T22:07:09.320 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 131/170 2024-06-09T22:07:09.355 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 132/170 2024-06-09T22:07:09.456 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 132/170 2024-06-09T22:07:09.627 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 133/170 2024-06-09T22:07:09.664 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 134/170 2024-06-09T22:07:09.687 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 134/170 2024-06-09T22:07:09.688 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:07:09.688 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-09T22:07:09.688 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-09T22:07:09.688 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-09T22:07:09.688 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:07:09.738 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 135/170 2024-06-09T22:07:09.749 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 135/170 2024-06-09T22:07:09.846 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 136/170 2024-06-09T22:07:09.911 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 137/170 2024-06-09T22:07:10.705 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-09T22:07:10.705 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 1/170 2024-06-09T22:07:10.705 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2/170 2024-06-09T22:07:10.705 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 3/170 2024-06-09T22:07:10.705 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 4/170 2024-06-09T22:07:10.705 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 5/170 2024-06-09T22:07:10.705 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 6/170 2024-06-09T22:07:10.706 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 7/170 2024-06-09T22:07:10.706 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-09T22:07:10.706 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-09T22:07:10.706 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 10/170 2024-06-09T22:07:10.706 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 11/170 2024-06-09T22:07:10.706 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 12/170 2024-06-09T22:07:10.706 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 13/170 2024-06-09T22:07:10.706 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-09T22:07:10.706 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 15/170 2024-06-09T22:07:10.706 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 16/170 2024-06-09T22:07:10.706 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 17/170 2024-06-09T22:07:10.707 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 18/170 2024-06-09T22:07:10.707 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 19/170 2024-06-09T22:07:10.707 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 20/170 2024-06-09T22:07:10.707 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 21/170 2024-06-09T22:07:10.707 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 22/170 2024-06-09T22:07:10.707 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 23/170 2024-06-09T22:07:10.707 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 24/170 2024-06-09T22:07:10.707 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 25/170 2024-06-09T22:07:10.707 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 26/170 2024-06-09T22:07:10.707 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 27/170 2024-06-09T22:07:10.707 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 28/170 2024-06-09T22:07:10.707 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 29/170 2024-06-09T22:07:10.708 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 30/170 2024-06-09T22:07:10.719 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 31/170 2024-06-09T22:07:10.719 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 32/170 2024-06-09T22:07:10.719 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 33/170 2024-06-09T22:07:10.719 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 34/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 35/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 36/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 37/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 38/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 39/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-06-09T22:07:10.720 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-06-09T22:07:10.721 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-06-09T22:07:10.722 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-06-09T22:07:10.722 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-06-09T22:07:10.722 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-06-09T22:07:10.722 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-06-09T22:07:10.722 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-06-09T22:07:10.722 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-06-09T22:07:10.722 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-06-09T22:07:10.722 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-06-09T22:07:10.722 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-06-09T22:07:10.722 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-06-09T22:07:10.722 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-06-09T22:07:10.722 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-06-09T22:07:10.723 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-06-09T22:07:10.723 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-06-09T22:07:10.723 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-06-09T22:07:10.723 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-06-09T22:07:10.723 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-06-09T22:07:10.723 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-06-09T22:07:10.723 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-06-09T22:07:10.723 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-06-09T22:07:10.723 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-06-09T22:07:10.723 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-06-09T22:07:10.723 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-06-09T22:07:10.723 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-06-09T22:07:10.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-06-09T22:07:10.725 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-06-09T22:07:10.725 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-06-09T22:07:10.725 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-06-09T22:07:10.725 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-06-09T22:07:10.725 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-06-09T22:07:10.725 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-06-09T22:07:10.725 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-06-09T22:07:10.725 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-06-09T22:07:10.725 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-06-09T22:07:10.725 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-06-09T22:07:10.725 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-06-09T22:07:10.725 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-06-09T22:07:10.726 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 122/170 2024-06-09T22:07:10.727 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-06-09T22:07:10.727 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-06-09T22:07:10.727 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-06-09T22:07:10.727 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-06-09T22:07:10.727 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-06-09T22:07:10.727 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-06-09T22:07:10.727 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-06-09T22:07:10.727 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-06-09T22:07:10.727 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-06-09T22:07:10.727 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-06-09T22:07:10.727 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-06-09T22:07:10.727 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-06-09T22:07:10.728 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 149/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 151/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-1:1.47.91-1.el9.x86_64 153/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 155/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-team-1:1.47.91-1.el9.x86_64 157/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-tui-1:1.47.91-1.el9.x86_64 159/170 2024-06-09T22:07:10.729 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-09T22:07:10.730 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-06-09T22:07:10.730 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-06-09T22:07:10.730 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-06-09T22:07:10.730 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-06-09T22:07:10.730 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-06-09T22:07:10.730 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-06-09T22:07:10.730 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-06-09T22:07:10.730 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-06-09T22:07:10.730 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-06-09T22:07:10.810 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-06-09T22:07:10.840 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 139/170 2024-06-09T22:07:10.862 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 139/170 2024-06-09T22:07:10.863 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:07:10.863 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-09T22:07:10.863 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-09T22:07:10.863 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-09T22:07:10.863 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:07:10.916 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 140/170 2024-06-09T22:07:10.938 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 140/170 2024-06-09T22:07:10.938 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:07:10.938 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-09T22:07:10.938 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:07:10.983 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 141/170 2024-06-09T22:07:11.005 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 141/170 2024-06-09T22:07:11.005 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:07:11.006 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-09T22:07:11.006 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-09T22:07:11.006 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-09T22:07:11.006 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:07:11.345 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/170 2024-06-09T22:07:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/170 2024-06-09T22:07:11.367 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T22:07:11.367 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-09T22:07:11.368 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-09T22:07:11.368 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-09T22:07:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:07:11.579 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-09T22:07:11.579 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 1/170 2024-06-09T22:07:11.579 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2/170 2024-06-09T22:07:11.579 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 3/170 2024-06-09T22:07:11.579 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 4/170 2024-06-09T22:07:11.579 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 5/170 2024-06-09T22:07:11.580 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 6/170 2024-06-09T22:07:11.580 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 7/170 2024-06-09T22:07:11.580 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-09T22:07:11.580 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-09T22:07:11.580 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 10/170 2024-06-09T22:07:11.580 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 11/170 2024-06-09T22:07:11.580 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 12/170 2024-06-09T22:07:11.580 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 13/170 2024-06-09T22:07:11.580 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-09T22:07:11.580 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 15/170 2024-06-09T22:07:11.580 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 16/170 2024-06-09T22:07:11.580 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 17/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 18/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 19/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 20/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 21/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 22/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 23/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 24/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 25/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 26/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 27/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 28/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 29/170 2024-06-09T22:07:11.581 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 30/170 2024-06-09T22:07:11.594 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 31/170 2024-06-09T22:07:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 32/170 2024-06-09T22:07:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 33/170 2024-06-09T22:07:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 34/170 2024-06-09T22:07:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 35/170 2024-06-09T22:07:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 36/170 2024-06-09T22:07:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 37/170 2024-06-09T22:07:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 38/170 2024-06-09T22:07:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 39/170 2024-06-09T22:07:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-06-09T22:07:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-06-09T22:07:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-06-09T22:07:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-06-09T22:07:11.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-06-09T22:07:11.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-06-09T22:07:11.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-06-09T22:07:11.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-06-09T22:07:11.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-06-09T22:07:11.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-06-09T22:07:11.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-06-09T22:07:11.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-06-09T22:07:11.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-06-09T22:07:11.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-06-09T22:07:11.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-06-09T22:07:11.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-06-09T22:07:11.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-06-09T22:07:11.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-06-09T22:07:11.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-06-09T22:07:11.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-06-09T22:07:11.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-06-09T22:07:11.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-06-09T22:07:11.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-06-09T22:07:11.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-06-09T22:07:11.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-06-09T22:07:11.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-06-09T22:07:11.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-06-09T22:07:11.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-06-09T22:07:11.598 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-06-09T22:07:11.599 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-06-09T22:07:11.600 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-06-09T22:07:11.601 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-06-09T22:07:11.601 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-06-09T22:07:11.601 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-06-09T22:07:11.601 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-06-09T22:07:11.601 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-06-09T22:07:11.601 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-06-09T22:07:11.601 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-06-09T22:07:11.601 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-06-09T22:07:11.601 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-06-09T22:07:11.601 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-06-09T22:07:11.601 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-06-09T22:07:11.601 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-06-09T22:07:11.602 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-06-09T22:07:11.602 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-06-09T22:07:11.602 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 122/170 2024-06-09T22:07:11.602 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-06-09T22:07:11.602 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-06-09T22:07:11.602 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-06-09T22:07:11.602 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-06-09T22:07:11.602 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-06-09T22:07:11.602 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-06-09T22:07:11.602 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-06-09T22:07:11.602 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-06-09T22:07:11.602 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-06-09T22:07:11.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 149/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 151/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-1:1.47.91-1.el9.x86_64 153/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 155/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-06-09T22:07:11.604 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-team-1:1.47.91-1.el9.x86_64 157/170 2024-06-09T22:07:11.605 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-06-09T22:07:11.605 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-tui-1:1.47.91-1.el9.x86_64 159/170 2024-06-09T22:07:11.605 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-09T22:07:11.605 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-06-09T22:07:11.605 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-06-09T22:07:11.605 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-06-09T22:07:11.605 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-06-09T22:07:11.605 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-06-09T22:07:11.605 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-06-09T22:07:11.605 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-06-09T22:07:11.605 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-06-09T22:07:11.605 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-06-09T22:07:13.563 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-06-09T22:07:13.563 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:07:13.563 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-06-09T22:07:13.563 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-1:1.47.91-1.el9.x86_64 2024-06-09T22:07:13.563 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-team-1:1.47.91-1.el9.x86_64 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-tui-1:1.47.91-1.el9.x86_64 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: automake-1.16.2-8.el9.noarch 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: bison-3.7.4-5.el9.x86_64 2024-06-09T22:07:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-09T22:07:13.565 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.565 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.565 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.565 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.565 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:13.565 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.565 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.565 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.565 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:13.565 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:13.565 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:13.565 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:13.566 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: flex-2.6.4-9.el9.x86_64 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.567 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.568 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-09T22:07:13.568 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-09T22:07:13.568 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-09T22:07:13.568 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-09T22:07:13.568 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-09T22:07:13.568 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.568 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.568 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-09T22:07:13.568 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.568 INFO:teuthology.orchestra.run.smithi194.stdout: libtool-2.4.6-45.el9.x86_64 2024-06-09T22:07:13.568 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-09T22:07:13.568 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: lua-5.4.4-4.el9.x86_64 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-13.14-1.el9.x86_64 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-09T22:07:13.569 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-09T22:07:13.570 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-09T22:07:13.570 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-09T22:07:13.570 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-09T22:07:13.570 INFO:teuthology.orchestra.run.smithi194.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-09T22:07:13.570 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-09T22:07:13.570 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-09T22:07:13.570 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-09T22:07:13.570 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.570 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.570 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.570 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-09T22:07:13.570 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-09T22:07:13.571 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-09T22:07:13.572 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply-3.11-14.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-09T22:07:13.573 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-09T22:07:13.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: re2-1:20211101-3.el9.x86_64 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-09T22:07:13.575 INFO:teuthology.orchestra.run.smithi194.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-09T22:07:13.576 INFO:teuthology.orchestra.run.smithi194.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-09T22:07:13.576 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-06-09T22:07:13.576 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-09T22:07:13.576 INFO:teuthology.orchestra.run.smithi194.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-09T22:07:13.576 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:07:13.576 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-06-09T22:07:14.016 DEBUG:teuthology.parallel:result is None 2024-06-09T22:07:14.250 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-06-09T22:07:14.250 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:07:14.250 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-1:1.47.91-1.el9.x86_64 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-team-1:1.47.91-1.el9.x86_64 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-tui-1:1.47.91-1.el9.x86_64 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-09T22:07:14.251 INFO:teuthology.orchestra.run.smithi153.stdout: automake-1.16.2-8.el9.noarch 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: bison-3.7.4-5.el9.x86_64 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:14.252 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.253 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: flex-2.6.4-9.el9.x86_64 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.254 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: libtool-2.4.6-45.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-09T22:07:14.255 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-13.14-1.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-09T22:07:14.256 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-09T22:07:14.257 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-09T22:07:14.258 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-09T22:07:14.259 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.260 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-09T22:07:14.261 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-09T22:07:14.262 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.262 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.262 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:14.262 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-06-09T22:07:14.262 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-09T22:07:14.262 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-09T22:07:14.262 INFO:teuthology.orchestra.run.smithi153.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-09T22:07:14.262 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-06-09T22:07:14.262 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-09T22:07:14.262 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-09T22:07:14.263 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-09T22:07:14.263 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-09T22:07:14.769 DEBUG:teuthology.parallel:result is None 2024-06-09T22:07:15.879 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 143/170 2024-06-09T22:07:15.920 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 144/170 2024-06-09T22:07:15.960 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 145/170 2024-06-09T22:07:15.993 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-06-09T22:07:16.020 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-06-09T22:07:16.212 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 148/170 2024-06-09T22:07:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-06-09T22:07:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 150/170 2024-06-09T22:07:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-06-09T22:07:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : NetworkManager-team-1:1.47.91-1.el9.x86_64 152/170 2024-06-09T22:07:16.423 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : NetworkManager-tui-1:1.47.91-1.el9.x86_64 153/170 2024-06-09T22:07:16.998 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-09T22:07:17.037 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-09T22:07:17.057 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-09T22:07:17.887 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 155/170 2024-06-09T22:07:17.974 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-06-09T22:07:18.016 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-06-09T22:07:18.107 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-06-09T22:07:18.144 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-06-09T22:07:18.169 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-09T22:07:18.194 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-06-09T22:07:18.221 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-06-09T22:07:18.221 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-09T22:07:18.305 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-09T22:07:18.339 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-06-09T22:07:18.339 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-09T22:07:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-09T22:07:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-06-09T22:07:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-09T22:07:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-09T22:07:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-09T22:07:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-09T22:07:18.734 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-09T22:07:18.759 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-06-09T22:07:18.831 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-09T22:07:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-09T22:07:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 1/170 2024-06-09T22:07:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2/170 2024-06-09T22:07:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 3/170 2024-06-09T22:07:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 4/170 2024-06-09T22:07:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 5/170 2024-06-09T22:07:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 6/170 2024-06-09T22:07:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 7/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 10/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 11/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 12/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 13/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 15/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 16/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 17/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 18/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 19/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 20/170 2024-06-09T22:07:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 21/170 2024-06-09T22:07:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 22/170 2024-06-09T22:07:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 23/170 2024-06-09T22:07:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 24/170 2024-06-09T22:07:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 25/170 2024-06-09T22:07:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 26/170 2024-06-09T22:07:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 27/170 2024-06-09T22:07:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 28/170 2024-06-09T22:07:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 29/170 2024-06-09T22:07:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 30/170 2024-06-09T22:07:24.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 31/170 2024-06-09T22:07:24.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 32/170 2024-06-09T22:07:24.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 33/170 2024-06-09T22:07:24.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 34/170 2024-06-09T22:07:24.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 35/170 2024-06-09T22:07:24.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 36/170 2024-06-09T22:07:24.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 37/170 2024-06-09T22:07:24.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 38/170 2024-06-09T22:07:24.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 39/170 2024-06-09T22:07:24.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-06-09T22:07:24.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-06-09T22:07:24.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-06-09T22:07:24.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-06-09T22:07:24.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-06-09T22:07:24.233 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-06-09T22:07:24.234 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-06-09T22:07:24.235 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 122/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-06-09T22:07:24.236 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-06-09T22:07:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 149/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-06-09T22:07:24.238 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 151/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-1:1.47.91-1.el9.x86_64 153/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 155/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-team-1:1.47.91-1.el9.x86_64 157/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-tui-1:1.47.91-1.el9.x86_64 159/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-06-09T22:07:24.239 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-06-09T22:07:24.240 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-06-09T22:07:24.240 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-06-09T22:07:24.240 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-06-09T22:07:26.784 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-06-09T22:07:26.784 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:07:26.784 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2024-06-09T22:07:26.784 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-1:1.47.91-1.el9.x86_64 2024-06-09T22:07:26.784 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 2024-06-09T22:07:26.784 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-team-1:1.47.91-1.el9.x86_64 2024-06-09T22:07:26.784 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-tui-1:1.47.91-1.el9.x86_64 2024-06-09T22:07:26.785 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-09T22:07:26.785 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-09T22:07:26.785 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-09T22:07:26.785 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.785 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.785 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-06-09T22:07:26.785 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-06-09T22:07:26.785 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-06-09T22:07:26.785 INFO:teuthology.orchestra.run.smithi104.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-09T22:07:26.785 INFO:teuthology.orchestra.run.smithi104.stdout: automake-1.16.2-8.el9.noarch 2024-06-09T22:07:26.785 INFO:teuthology.orchestra.run.smithi104.stdout: bison-3.7.4-5.el9.x86_64 2024-06-09T22:07:26.785 INFO:teuthology.orchestra.run.smithi104.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-09T22:07:26.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:26.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:26.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:26.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:26.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T22:07:26.787 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: flex-2.6.4-9.el9.x86_64 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.788 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-09T22:07:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-09T22:07:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-09T22:07:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-09T22:07:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-09T22:07:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-09T22:07:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: libtool-2.4.6-45.el9.x86_64 2024-06-09T22:07:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-09T22:07:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-09T22:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-09T22:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-09T22:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout: lua-5.4.4-4.el9.x86_64 2024-06-09T22:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-09T22:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-09T22:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-09T22:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-09T22:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-06-09T22:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-09T22:07:26.790 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-13.14-1.el9.x86_64 2024-06-09T22:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-09T22:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-09T22:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-09T22:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-09T22:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-09T22:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-09T22:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-09T22:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-09T22:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-09T22:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-09T22:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.791 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-09T22:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-09T22:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-09T22:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-09T22:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-09T22:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-09T22:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-09T22:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-09T22:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-09T22:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-09T22:07:26.792 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-09T22:07:26.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-09T22:07:26.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-09T22:07:26.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-09T22:07:26.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-09T22:07:26.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-09T22:07:26.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-09T22:07:26.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply-3.11-14.el9.noarch 2024-06-09T22:07:26.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-09T22:07:26.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-09T22:07:26.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-09T22:07:26.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-09T22:07:26.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-09T22:07:26.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-09T22:07:26.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.796 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T22:07:26.797 INFO:teuthology.orchestra.run.smithi104.stdout: re2-1:20211101-3.el9.x86_64 2024-06-09T22:07:26.797 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-09T22:07:26.797 INFO:teuthology.orchestra.run.smithi104.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-09T22:07:26.797 INFO:teuthology.orchestra.run.smithi104.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-09T22:07:26.797 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-06-09T22:07:26.797 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-09T22:07:26.797 INFO:teuthology.orchestra.run.smithi104.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-09T22:07:26.798 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:07:26.798 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-06-09T22:07:27.217 DEBUG:teuthology.parallel:result is None 2024-06-09T22:07:27.217 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T22:07:27.359 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-09T22:07:27.382 INFO:teuthology.orchestra.run.smithi104.stdout:19.0.0-2946.g3d89accf.el9 2024-06-09T22:07:27.383 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2946.g3d89accf.el9 2024-06-09T22:07:27.383 INFO:teuthology.task.install:The correct ceph version 19.0.0-2946.g3d89accf is installed. 2024-06-09T22:07:27.384 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T22:07:27.601 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-09T22:07:27.626 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-2946.g3d89accf.el9 2024-06-09T22:07:27.627 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2946.g3d89accf.el9 2024-06-09T22:07:27.627 INFO:teuthology.task.install:The correct ceph version 19.0.0-2946.g3d89accf is installed. 2024-06-09T22:07:27.629 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T22:07:27.764 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-09T22:07:27.789 INFO:teuthology.orchestra.run.smithi194.stdout:19.0.0-2946.g3d89accf.el9 2024-06-09T22:07:27.790 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2946.g3d89accf.el9 2024-06-09T22:07:27.790 INFO:teuthology.task.install:The correct ceph version 19.0.0-2946.g3d89accf is installed. 2024-06-09T22:07:27.792 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-09T22:07:27.792 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:07:27.792 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-09T22:07:27.824 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-09T22:07:27.825 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-09T22:07:27.856 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-06-09T22:07:27.856 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-09T22:07:27.886 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-09T22:07:27.887 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:07:27.887 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-06-09T22:07:27.915 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-09T22:07:27.982 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-09T22:07:27.983 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-06-09T22:07:28.010 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-09T22:07:28.078 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-06-09T22:07:28.078 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-06-09T22:07:28.106 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-09T22:07:28.173 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-09T22:07:28.173 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:07:28.173 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-09T22:07:28.200 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-09T22:07:28.267 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-09T22:07:28.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-09T22:07:28.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-09T22:07:28.359 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-06-09T22:07:28.360 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-09T22:07:28.383 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-09T22:07:28.447 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-09T22:07:28.447 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:07:28.448 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-06-09T22:07:28.477 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-09T22:07:28.546 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-09T22:07:28.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-06-09T22:07:28.570 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-09T22:07:28.634 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-06-09T22:07:28.635 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-06-09T22:07:28.659 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-09T22:07:28.722 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-09T22:07:28.811 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-06-09T22:07:28.811 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T22:07:28.812 INFO:tasks.cephadm:Cluster fsid is a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:07:28.812 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-09T22:07:28.812 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.104', 'mon.b': '172.21.15.153', 'mon.c': '172.21.15.194'} 2024-06-09T22:07:28.812 INFO:tasks.cephadm:First mon is mon.a on smithi104 2024-06-09T22:07:28.812 INFO:tasks.cephadm:First mgr is x 2024-06-09T22:07:28.813 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-09T22:07:28.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname $(hostname -s) 2024-06-09T22:07:28.843 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-06-09T22:07:28.867 DEBUG:teuthology.orchestra.run.smithi194:> sudo hostname $(hostname -s) 2024-06-09T22:07:28.890 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-06-09T22:07:28.891 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T22:07:29.112 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55273/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2946-g3d89accf', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2946.g3d89accf'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-30 17:04:17.570799', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-06-09T22:07:29.332 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 3d89accfe3201286f000e065a1faea2a88c7a6ba from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T22:07:29.334 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/x86_64/flavors/default/cephadm 2024-06-09T22:07:29.334 DEBUG:teuthology.orchestra.run.smithi104:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-09T22:07:30.261 INFO:teuthology.orchestra.run.smithi104.stdout:-rw-r--r--. 1 ubuntu ubuntu 781907 Jun 9 22:07 /home/ubuntu/cephtest/cephadm 2024-06-09T22:07:30.261 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-09T22:07:31.199 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 781907 Jun 9 22:07 /home/ubuntu/cephtest/cephadm 2024-06-09T22:07:31.199 DEBUG:teuthology.orchestra.run.smithi194:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-09T22:07:31.816 INFO:teuthology.orchestra.run.smithi194.stdout:-rw-r--r--. 1 ubuntu ubuntu 781907 Jun 9 22:07 /home/ubuntu/cephtest/cephadm 2024-06-09T22:07:31.817 DEBUG:teuthology.orchestra.run.smithi104:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-09T22:07:31.837 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-09T22:07:31.856 DEBUG:teuthology.orchestra.run.smithi194:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-09T22:07:31.889 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba on all hosts... 2024-06-09T22:07:31.889 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba pull 2024-06-09T22:07:31.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba pull 2024-06-09T22:07:31.898 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba pull 2024-06-09T22:07:32.674 INFO:teuthology.orchestra.run.smithi194.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba... 2024-06-09T22:07:32.742 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba... 2024-06-09T22:07:32.791 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba... 2024-06-09T22:08:09.181 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-06-09T22:08:09.181 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)", 2024-06-09T22:08:09.181 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1", 2024-06-09T22:08:09.181 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-06-09T22:08:09.181 INFO:teuthology.orchestra.run.smithi153.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9" 2024-06-09T22:08:09.181 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-06-09T22:08:09.181 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-06-09T22:08:11.781 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-06-09T22:08:11.781 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_version": "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)", 2024-06-09T22:08:11.781 INFO:teuthology.orchestra.run.smithi194.stdout: "image_id": "b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1", 2024-06-09T22:08:11.782 INFO:teuthology.orchestra.run.smithi194.stdout: "repo_digests": [ 2024-06-09T22:08:11.782 INFO:teuthology.orchestra.run.smithi194.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9" 2024-06-09T22:08:11.782 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-06-09T22:08:11.782 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-06-09T22:08:13.109 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:08:13.110 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)", 2024-06-09T22:08:13.110 INFO:teuthology.orchestra.run.smithi104.stdout: "image_id": "b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1", 2024-06-09T22:08:13.110 INFO:teuthology.orchestra.run.smithi104.stdout: "repo_digests": [ 2024-06-09T22:08:13.110 INFO:teuthology.orchestra.run.smithi104.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9" 2024-06-09T22:08:13.110 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-06-09T22:08:13.110 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:08:13.133 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2024-06-09T22:08:13.193 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-06-09T22:08:13.225 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph 2024-06-09T22:08:13.256 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /etc/ceph 2024-06-09T22:08:13.280 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-06-09T22:08:13.307 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /etc/ceph 2024-06-09T22:08:13.334 INFO:tasks.cephadm:Writing seed config... 2024-06-09T22:08:13.335 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-06-09T22:08:13.335 INFO:tasks.cephadm: override: [client] debug client = 20 2024-06-09T22:08:13.335 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-06-09T22:08:13.335 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-06-09T22:08:13.335 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-06-09T22:08:13.335 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-06-09T22:08:13.335 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-06-09T22:08:13.335 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-06-09T22:08:13.335 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-06-09T22:08:13.335 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-06-09T22:08:13.335 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-06-09T22:08:13.336 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-09T22:08:13.337 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-09T22:08:13.338 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-06-09T22:08:13.338 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-06-09T22:08:13.338 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-06-09T22:08:13.338 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-06-09T22:08:13.338 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-06-09T22:08:13.338 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-06-09T22:08:13.338 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-06-09T22:08:13.338 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-06-09T22:08:13.338 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-06-09T22:08:13.339 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:08:13.339 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-09T22:08:13.354 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a948c0ca-26ac-11ef-bc9b-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-06-09T22:08:13.354 DEBUG:teuthology.orchestra.run.smithi104:mon.a> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.a.service 2024-06-09T22:08:13.397 DEBUG:teuthology.orchestra.run.smithi104:mgr.x> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.x.service 2024-06-09T22:08:13.439 INFO:tasks.cephadm:Bootstrapping... 2024-06-09T22:08:13.439 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba -v bootstrap --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.104 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-09T22:08:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-06-09T22:08:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba', '-v', 'bootstrap', '--fsid', 'a948c0ca-26ac-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.104', '--skip-admin-label'] 2024-06-09T22:08:13.629 INFO:teuthology.orchestra.run.smithi104.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-09T22:08:13.630 INFO:teuthology.orchestra.run.smithi104.stdout:Verifying podman|docker is present... 2024-06-09T22:08:13.655 INFO:teuthology.orchestra.run.smithi104.stdout:/bin/podman: stdout 5.0.2 2024-06-09T22:08:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:Verifying lvm2 is present... 2024-06-09T22:08:13.656 INFO:teuthology.orchestra.run.smithi104.stdout:Verifying time synchronization is in place... 2024-06-09T22:08:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-09T22:08:13.663 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-09T22:08:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-09T22:08:13.670 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout inactive 2024-06-09T22:08:13.676 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout enabled 2024-06-09T22:08:13.682 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout active 2024-06-09T22:08:13.682 INFO:teuthology.orchestra.run.smithi104.stdout:Unit chronyd.service is enabled and running 2024-06-09T22:08:13.683 INFO:teuthology.orchestra.run.smithi104.stdout:Repeating the final host check... 2024-06-09T22:08:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:/bin/podman: stdout 5.0.2 2024-06-09T22:08:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:podman (/bin/podman) version 5.0.2 is present 2024-06-09T22:08:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl is present 2024-06-09T22:08:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:lvcreate is present 2024-06-09T22:08:13.712 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-09T22:08:13.712 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-09T22:08:13.718 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-09T22:08:13.718 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout inactive 2024-06-09T22:08:13.725 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout enabled 2024-06-09T22:08:13.731 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout active 2024-06-09T22:08:13.731 INFO:teuthology.orchestra.run.smithi104.stdout:Unit chronyd.service is enabled and running 2024-06-09T22:08:13.731 INFO:teuthology.orchestra.run.smithi104.stdout:Host looks OK 2024-06-09T22:08:13.731 INFO:teuthology.orchestra.run.smithi104.stdout:Cluster fsid: a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:08:13.732 INFO:teuthology.orchestra.run.smithi104.stdout:Acquiring lock 139919950329216 on /run/cephadm/a948c0ca-26ac-11ef-bc9b-c7b262605968.lock 2024-06-09T22:08:13.732 INFO:teuthology.orchestra.run.smithi104.stdout:Lock 139919950329216 acquired on /run/cephadm/a948c0ca-26ac-11ef-bc9b-c7b262605968.lock 2024-06-09T22:08:13.732 INFO:teuthology.orchestra.run.smithi104.stdout:Verifying IP 172.21.15.104 port 3300 ... 2024-06-09T22:08:13.732 INFO:teuthology.orchestra.run.smithi104.stdout:Verifying IP 172.21.15.104 port 6789 ... 2024-06-09T22:08:13.732 INFO:teuthology.orchestra.run.smithi104.stdout:Base mon IP(s) is [172.21.15.104:3300, 172.21.15.104:6789], mon addrv is [v2:172.21.15.104:3300,v1:172.21.15.104:6789] 2024-06-09T22:08:13.735 INFO:teuthology.orchestra.run.smithi104.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.104 metric 100 2024-06-09T22:08:13.735 INFO:teuthology.orchestra.run.smithi104.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.104 metric 100 2024-06-09T22:08:13.738 INFO:teuthology.orchestra.run.smithi104.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-09T22:08:13.738 INFO:teuthology.orchestra.run.smithi104.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-09T22:08:13.738 INFO:teuthology.orchestra.run.smithi104.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1786sec hoplimit 64 pref medium 2024-06-09T22:08:13.741 INFO:teuthology.orchestra.run.smithi104.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-09T22:08:13.741 INFO:teuthology.orchestra.run.smithi104.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-09T22:08:13.741 INFO:teuthology.orchestra.run.smithi104.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-09T22:08:13.741 INFO:teuthology.orchestra.run.smithi104.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-09T22:08:13.741 INFO:teuthology.orchestra.run.smithi104.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce89/64 scope link 2024-06-09T22:08:13.741 INFO:teuthology.orchestra.run.smithi104.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-09T22:08:13.742 INFO:teuthology.orchestra.run.smithi104.stdout:Mon IP `172.21.15.104` is in CIDR network `172.21.0.0/20` 2024-06-09T22:08:13.742 INFO:teuthology.orchestra.run.smithi104.stdout:Mon IP `172.21.15.104` is in CIDR network `172.21.0.0/20` 2024-06-09T22:08:13.742 INFO:teuthology.orchestra.run.smithi104.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-09T22:08:13.742 INFO:teuthology.orchestra.run.smithi104.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-09T22:08:13.743 INFO:teuthology.orchestra.run.smithi104.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba... 2024-06-09T22:08:14.428 INFO:teuthology.orchestra.run.smithi104.stdout:/bin/podman: stdout b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 2024-06-09T22:08:14.429 INFO:teuthology.orchestra.run.smithi104.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba... 2024-06-09T22:08:14.429 INFO:teuthology.orchestra.run.smithi104.stdout:/bin/podman: stderr Getting image source signatures 2024-06-09T22:08:14.429 INFO:teuthology.orchestra.run.smithi104.stdout:/bin/podman: stderr Copying blob sha256:53b2d4191269e9d20972bfa652565296fad6514cf6a3d99f19db644875f67876 2024-06-09T22:08:14.429 INFO:teuthology.orchestra.run.smithi104.stdout:/bin/podman: stderr Copying blob sha256:3ca969c7b6d79946c6f10490fffc00028babe7df67f50ed53c7bc209325e0612 2024-06-09T22:08:14.429 INFO:teuthology.orchestra.run.smithi104.stdout:/bin/podman: stderr Copying config sha256:b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 2024-06-09T22:08:14.429 INFO:teuthology.orchestra.run.smithi104.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-09T22:08:15.433 INFO:teuthology.orchestra.run.smithi104.stdout:ceph: stdout ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc) 2024-06-09T22:08:15.433 INFO:teuthology.orchestra.run.smithi104.stdout:Ceph version: ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc) 2024-06-09T22:08:15.433 INFO:teuthology.orchestra.run.smithi104.stdout:Extracting ceph user uid/gid from container image... 2024-06-09T22:08:16.456 INFO:teuthology.orchestra.run.smithi104.stdout:stat: stdout 167 167 2024-06-09T22:08:16.456 INFO:teuthology.orchestra.run.smithi104.stdout:Creating initial keys... 2024-06-09T22:08:17.335 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph-authtool: stdout AQDQJ2Zmk958MBAAXREjUtffx7vf/PTK73TDpA== 2024-06-09T22:08:18.267 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph-authtool: stdout AQDRJ2ZmtrGDKBAAMqkmPi5ZZNtgfvnNweEtuQ== 2024-06-09T22:08:19.196 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph-authtool: stdout AQDSJ2ZmPB01JRAAFAyo1ns5sACWljJdJsv/UQ== 2024-06-09T22:08:19.197 INFO:teuthology.orchestra.run.smithi104.stdout:Creating initial monmap... 2024-06-09T22:08:20.205 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-09T22:08:20.205 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-06-09T22:08:20.205 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:08:20.205 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-09T22:08:20.205 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool for a [v2:172.21.15.104:3300,v1:172.21.15.104:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-09T22:08:20.206 INFO:teuthology.orchestra.run.smithi104.stdout:setting min_mon_release = quincy 2024-06-09T22:08:20.206 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/monmaptool: set fsid to a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:08:20.206 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-09T22:08:20.206 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:08:20.206 INFO:teuthology.orchestra.run.smithi104.stdout:Creating mon... 2024-06-09T22:08:29.953 INFO:teuthology.orchestra.run.smithi104.stdout:create mon.a on 2024-06-09T22:08:30.290 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-09T22:08:30.426 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-09T22:08:30.597 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968.target. 2024-06-09T22:08:30.597 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968.target. 2024-06-09T22:08:30.821 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.a 2024-06-09T22:08:30.822 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.a.service: Unit ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.a.service not loaded. 2024-06-09T22:08:30.982 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968.target.wants/ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.a.service → /etc/systemd/system/ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@.service. 2024-06-09T22:08:31.242 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:31 smithi104 podman[31998]: 2024-06-09 22:08:31.137862318 +0000 UTC m=+0.017134918 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T22:08:31.485 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-09T22:08:31.485 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout disabled 2024-06-09T22:08:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-09T22:08:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout inactive 2024-06-09T22:08:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:firewalld.service is not enabled 2024-06-09T22:08:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:Not possible to enable service . firewalld.service is not available 2024-06-09T22:08:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:Waiting for mon to start... 2024-06-09T22:08:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:Waiting for mon... 2024-06-09T22:08:31.616 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:31 smithi104 podman[31998]: 2024-06-09 22:08:31.428837153 +0000 UTC m=+0.308109740 container init f4ac17b2af78cef38df9a6c1983a968d1e62e875d1b5e396daf67126e505c8fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-09T22:08:31.617 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:31 smithi104 podman[31998]: 2024-06-09 22:08:31.432047375 +0000 UTC m=+0.311319962 container start f4ac17b2af78cef38df9a6c1983a968d1e62e875d1b5e396daf67126e505c8fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux ) 2024-06-09T22:08:31.617 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:31 smithi104 ceph-mon[32017]: mkfs a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:08:31.617 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:31 smithi104 ceph-mon[32017]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-09T22:08:31.617 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:31 smithi104 bash[31998]: f4ac17b2af78cef38df9a6c1983a968d1e62e875d1b5e396daf67126e505c8fc 2024-06-09T22:08:31.617 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:31 smithi104 systemd[1]: Started Ceph mon.a for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:08:32.619 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:32 smithi104 ceph-mon[32017]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-09T22:08:32.619 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:32 smithi104 ceph-mon[32017]: monmap epoch 1 2024-06-09T22:08:32.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:32 smithi104 ceph-mon[32017]: fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:08:32.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:32 smithi104 ceph-mon[32017]: last_changed 2024-06-09T22:08:19.534976+0000 2024-06-09T22:08:32.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:32 smithi104 ceph-mon[32017]: created 2024-06-09T22:08:19.534976+0000 2024-06-09T22:08:32.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:32 smithi104 ceph-mon[32017]: min_mon_release 19 (squid) 2024-06-09T22:08:32.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:32 smithi104 ceph-mon[32017]: election_strategy: 1 2024-06-09T22:08:32.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:32 smithi104 ceph-mon[32017]: 0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-06-09T22:08:32.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:32 smithi104 ceph-mon[32017]: fsmap 2024-06-09T22:08:32.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:32 smithi104 ceph-mon[32017]: osdmap e1: 0 total, 0 up, 0 in 2024-06-09T22:08:32.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:32 smithi104 ceph-mon[32017]: mgrmap e1: no daemons active 2024-06-09T22:08:32.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:32 smithi104 ceph-mon[32017]: from='client.? 172.21.15.104:0/838318961' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-09T22:08:32.627 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout cluster: 2024-06-09T22:08:32.627 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout id: a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:08:32.627 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-09T22:08:32.627 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:32.627 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout services: 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.568283s) 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout data: 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout pgs: 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.962+0000 7fb70959b700 1 Processor -- start 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.962+0000 7fb70959b700 1 -- start start 2024-06-09T22:08:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.962+0000 7fb70959b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 0x7fb70407cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.962+0000 7fb70959b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb70407d160 con 0x7fb70407e6e0 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.963+0000 7fb702ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 0x7fb70407cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.963+0000 7fb702ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 0x7fb70407cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37360/0 (socket says 172.21.15.104:37360) 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.963+0000 7fb702ffd700 1 -- 172.21.15.104:0/2829810700 learned_addr learned my addr 172.21.15.104:0/2829810700 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.963+0000 7fb702ffd700 1 -- 172.21.15.104:0/2829810700 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70407d2e0 con 0x7fb70407e6e0 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.963+0000 7fb702ffd700 1 --2- 172.21.15.104:0/2829810700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 0x7fb70407cb90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40098a0 tx=0x7fb6f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee5ba243ca93fde server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.963+0000 7fb7027fc700 1 -- 172.21.15.104:0/2829810700 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6f4004160 con 0x7fb70407e6e0 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.963+0000 7fb7027fc700 1 -- 172.21.15.104:0/2829810700 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6f4004300 con 0x7fb70407e6e0 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.963+0000 7fb7027fc700 1 -- 172.21.15.104:0/2829810700 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6f40036a0 con 0x7fb70407e6e0 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.964+0000 7fb70959b700 1 -- 172.21.15.104:0/2829810700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 msgr2=0x7fb70407cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.964+0000 7fb70959b700 1 --2- 172.21.15.104:0/2829810700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 0x7fb70407cb90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40098a0 tx=0x7fb6f4009bb0 comp rx=0 tx=0).stop 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.964+0000 7fb70959b700 1 -- 172.21.15.104:0/2829810700 shutdown_connections 2024-06-09T22:08:32.629 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.964+0000 7fb70959b700 1 --2- 172.21.15.104:0/2829810700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 0x7fb70407cb90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.964+0000 7fb70959b700 1 -- 172.21.15.104:0/2829810700 >> 172.21.15.104:0/2829810700 conn(0x7fb704103540 msgr2=0x7fb704105980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.964+0000 7fb70959b700 1 -- 172.21.15.104:0/2829810700 shutdown_connections 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.964+0000 7fb70959b700 1 -- 172.21.15.104:0/2829810700 wait complete. 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.965+0000 7fb70959b700 1 Processor -- start 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.965+0000 7fb70959b700 1 -- start start 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.965+0000 7fb70959b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 0x7fb70419d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.965+0000 7fb70959b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb70407d7b0 con 0x7fb70407e6e0 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.965+0000 7fb702ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 0x7fb70419d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.965+0000 7fb702ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 0x7fb70419d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37376/0 (socket says 172.21.15.104:37376) 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.965+0000 7fb702ffd700 1 -- 172.21.15.104:0/838318961 learned_addr learned my addr 172.21.15.104:0/838318961 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.965+0000 7fb702ffd700 1 -- 172.21.15.104:0/838318961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70419dd60 con 0x7fb70407e6e0 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.965+0000 7fb702ffd700 1 --2- 172.21.15.104:0/838318961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 0x7fb70419d820 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4000c00 tx=0x7fb6f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:32.630 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.965+0000 7fb700ff9700 1 -- 172.21.15.104:0/838318961 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6f4003db0 con 0x7fb70407e6e0 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.966+0000 7fb700ff9700 1 -- 172.21.15.104:0/838318961 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6f4011630 con 0x7fb70407e6e0 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.966+0000 7fb70959b700 1 -- 172.21.15.104:0/838318961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb70419dff0 con 0x7fb70407e6e0 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.966+0000 7fb700ff9700 1 -- 172.21.15.104:0/838318961 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6f4011950 con 0x7fb70407e6e0 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.966+0000 7fb70959b700 1 -- 172.21.15.104:0/838318961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7041a0ce0 con 0x7fb70407e6e0 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.966+0000 7fb700ff9700 1 -- 172.21.15.104:0/838318961 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb6f4011af0 con 0x7fb70407e6e0 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.966+0000 7fb700ff9700 1 -- 172.21.15.104:0/838318961 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb6f401c790 con 0x7fb70407e6e0 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.967+0000 7fb70959b700 1 -- 172.21.15.104:0/838318961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb704063af0 con 0x7fb70407e6e0 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:31.969+0000 7fb700ff9700 1 -- 172.21.15.104:0/838318961 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb6f4032020 con 0x7fb70407e6e0 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:32.037+0000 7fb70959b700 1 -- 172.21.15.104:0/838318961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fb70402de20 con 0x7fb70407e6e0 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:32.038+0000 7fb700ff9700 1 -- 172.21.15.104:0/838318961 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fb6f4004260 con 0x7fb70407e6e0 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:32.038+0000 7fb70959b700 1 -- 172.21.15.104:0/838318961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 msgr2=0x7fb70419d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:32.038+0000 7fb70959b700 1 --2- 172.21.15.104:0/838318961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 0x7fb70419d820 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4000c00 tx=0x7fb6f4004750 comp rx=0 tx=0).stop 2024-06-09T22:08:32.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:32.038+0000 7fb70959b700 1 -- 172.21.15.104:0/838318961 shutdown_connections 2024-06-09T22:08:32.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:32.038+0000 7fb70959b700 1 --2- 172.21.15.104:0/838318961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70407e6e0 0x7fb70419d820 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:32.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:32.038+0000 7fb70959b700 1 -- 172.21.15.104:0/838318961 >> 172.21.15.104:0/838318961 conn(0x7fb704103540 msgr2=0x7fb7041043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:32.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:32.039+0000 7fb70959b700 1 -- 172.21.15.104:0/838318961 shutdown_connections 2024-06-09T22:08:32.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:32.039+0000 7fb70959b700 1 -- 172.21.15.104:0/838318961 wait complete. 2024-06-09T22:08:32.632 INFO:teuthology.orchestra.run.smithi104.stdout:mon is available 2024-06-09T22:08:32.632 INFO:teuthology.orchestra.run.smithi104.stdout:Assimilating anything we can from ceph.conf... 2024-06-09T22:08:33.701 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:33 smithi104 ceph-mon[32017]: from='client.? 172.21.15.104:0/1472439018' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-09T22:08:33.701 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:33 smithi104 ceph-mon[32017]: from='client.? 172.21.15.104:0/1472439018' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-09T22:08:33.709 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:33.709 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout [global] 2024-06-09T22:08:33.709 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout fsid = a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:08:33.709 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.104:3300,v1:172.21.15.104:6789] 2024-06-09T22:08:33.709 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-09T22:08:33.709 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-09T22:08:33.709 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-09T22:08:33.709 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-09T22:08:33.709 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:33.709 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-09T22:08:33.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-09T22:08:33.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:33.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout [osd] 2024-06-09T22:08:33.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-09T22:08:33.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-09T22:08:33.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-09T22:08:33.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-09T22:08:33.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-09T22:08:33.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.093+0000 7f58fde71700 1 Processor -- start 2024-06-09T22:08:33.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.093+0000 7f58fde71700 1 -- start start 2024-06-09T22:08:33.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.093+0000 7f58fde71700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 0x7f58f8108360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:33.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.093+0000 7f58fde71700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f58f81088a0 con 0x7f58f8107f40 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.093+0000 7f58f77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 0x7f58f8108360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.093+0000 7f58f77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 0x7f58f8108360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37388/0 (socket says 172.21.15.104:37388) 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.093+0000 7f58f77fe700 1 -- 172.21.15.104:0/2840430157 learned_addr learned my addr 172.21.15.104:0/2840430157 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.093+0000 7f58f77fe700 1 -- 172.21.15.104:0/2840430157 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f8109090 con 0x7f58f8107f40 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.093+0000 7f58f77fe700 1 --2- 172.21.15.104:0/2840430157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 0x7f58f8108360 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f58e80098a0 tx=0x7f58e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a560803a320796 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.094+0000 7f58f6ffd700 1 -- 172.21.15.104:0/2840430157 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58e8004160 con 0x7f58f8107f40 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.094+0000 7f58f6ffd700 1 -- 172.21.15.104:0/2840430157 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58e8004300 con 0x7f58f8107f40 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.094+0000 7f58f6ffd700 1 -- 172.21.15.104:0/2840430157 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58e80036a0 con 0x7f58f8107f40 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.094+0000 7f58fde71700 1 -- 172.21.15.104:0/2840430157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 msgr2=0x7f58f8108360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.094+0000 7f58fde71700 1 --2- 172.21.15.104:0/2840430157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 0x7f58f8108360 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f58e80098a0 tx=0x7f58e8009bb0 comp rx=0 tx=0).stop 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.094+0000 7f58fde71700 1 -- 172.21.15.104:0/2840430157 shutdown_connections 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.094+0000 7f58fde71700 1 --2- 172.21.15.104:0/2840430157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 0x7f58f8108360 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.094+0000 7f58fde71700 1 -- 172.21.15.104:0/2840430157 >> 172.21.15.104:0/2840430157 conn(0x7f58f81035b0 msgr2=0x7f58f81059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.094+0000 7f58fde71700 1 -- 172.21.15.104:0/2840430157 shutdown_connections 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.094+0000 7f58fde71700 1 -- 172.21.15.104:0/2840430157 wait complete. 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.095+0000 7f58fde71700 1 Processor -- start 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.095+0000 7f58fde71700 1 -- start start 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.095+0000 7f58fde71700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 0x7f58f819b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.095+0000 7f58fde71700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f58f8109560 con 0x7f58f8107f40 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.095+0000 7f58f77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 0x7f58f819b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.095+0000 7f58f77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 0x7f58f819b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37400/0 (socket says 172.21.15.104:37400) 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.095+0000 7f58f77fe700 1 -- 172.21.15.104:0/1472439018 learned_addr learned my addr 172.21.15.104:0/1472439018 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.095+0000 7f58f77fe700 1 -- 172.21.15.104:0/1472439018 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f819bb80 con 0x7f58f8107f40 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.096+0000 7f58f77fe700 1 --2- 172.21.15.104:0/1472439018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 0x7f58f819b640 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f58e8000c00 tx=0x7f58e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.096+0000 7f58f57fa700 1 -- 172.21.15.104:0/1472439018 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58e8003db0 con 0x7f58f8107f40 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.096+0000 7f58f57fa700 1 -- 172.21.15.104:0/1472439018 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58e8011630 con 0x7f58f8107f40 2024-06-09T22:08:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.096+0000 7f58fde71700 1 -- 172.21.15.104:0/1472439018 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f819be10 con 0x7f58f8107f40 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.096+0000 7f58f57fa700 1 -- 172.21.15.104:0/1472439018 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58e8011950 con 0x7f58f8107f40 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.096+0000 7f58fde71700 1 -- 172.21.15.104:0/1472439018 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f819eb00 con 0x7f58f8107f40 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.096+0000 7f58f57fa700 1 -- 172.21.15.104:0/1472439018 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f58e8011af0 con 0x7f58f8107f40 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.096+0000 7f58f57fa700 1 -- 172.21.15.104:0/1472439018 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f58e801c450 con 0x7f58f8107f40 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.097+0000 7f58fde71700 1 -- 172.21.15.104:0/1472439018 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f8063af0 con 0x7f58f8107f40 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.099+0000 7f58f57fa700 1 -- 172.21.15.104:0/1472439018 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f58e8032020 con 0x7f58f8107f40 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.166+0000 7f58fde71700 1 -- 172.21.15.104:0/1472439018 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f58f802de20 con 0x7f58f8107f40 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.169+0000 7f58f57fa700 1 -- 172.21.15.104:0/1472439018 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+431 (secure 0 0 0) 0x7f58e8004260 con 0x7f58f8107f40 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.170+0000 7f58f57fa700 1 -- 172.21.15.104:0/1472439018 <== mon.0 v2:172.21.15.104:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f58e801b450 con 0x7f58f8107f40 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.170+0000 7f58fde71700 1 -- 172.21.15.104:0/1472439018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 msgr2=0x7f58f819b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.170+0000 7f58fde71700 1 --2- 172.21.15.104:0/1472439018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 0x7f58f819b640 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f58e8000c00 tx=0x7f58e8004750 comp rx=0 tx=0).stop 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.170+0000 7f58fde71700 1 -- 172.21.15.104:0/1472439018 shutdown_connections 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.170+0000 7f58fde71700 1 --2- 172.21.15.104:0/1472439018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58f8107f40 0x7f58f819b640 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:33.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.170+0000 7f58fde71700 1 -- 172.21.15.104:0/1472439018 >> 172.21.15.104:0/1472439018 conn(0x7f58f81035b0 msgr2=0x7f58f81921e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:33.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.170+0000 7f58fde71700 1 -- 172.21.15.104:0/1472439018 shutdown_connections 2024-06-09T22:08:33.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:33.170+0000 7f58fde71700 1 -- 172.21.15.104:0/1472439018 wait complete. 2024-06-09T22:08:33.714 INFO:teuthology.orchestra.run.smithi104.stdout:Generating new minimal ceph.conf... 2024-06-09T22:08:34.742 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:34 smithi104 ceph-mon[32017]: from='client.? 172.21.15.104:0/2398597930' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:08:34.823 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.133+0000 7f431f1dc700 1 Processor -- start 2024-06-09T22:08:34.823 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.133+0000 7f431f1dc700 1 -- start start 2024-06-09T22:08:34.823 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.133+0000 7f431f1dc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 0x7f43181072b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:34.823 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.133+0000 7f431f1dc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f43181077f0 con 0x7f4318106e90 2024-06-09T22:08:34.823 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.133+0000 7f431cf78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 0x7f43181072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:34.823 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.133+0000 7f431cf78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 0x7f43181072b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37412/0 (socket says 172.21.15.104:37412) 2024-06-09T22:08:34.823 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.133+0000 7f431cf78700 1 -- 172.21.15.104:0/4098733335 learned_addr learned my addr 172.21.15.104:0/4098733335 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:34.823 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.133+0000 7f431cf78700 1 -- 172.21.15.104:0/4098733335 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4318107fa0 con 0x7f4318106e90 2024-06-09T22:08:34.823 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.133+0000 7f431cf78700 1 --2- 172.21.15.104:0/4098733335 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 0x7f43181072b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f430c0098a0 tx=0x7f430c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6292528dde8bf2c1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:34.823 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.134+0000 7f43177fe700 1 -- 172.21.15.104:0/4098733335 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f430c004160 con 0x7f4318106e90 2024-06-09T22:08:34.823 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.134+0000 7f43177fe700 1 -- 172.21.15.104:0/4098733335 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f430c004300 con 0x7f4318106e90 2024-06-09T22:08:34.824 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.134+0000 7f43177fe700 1 -- 172.21.15.104:0/4098733335 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f430c011b50 con 0x7f4318106e90 2024-06-09T22:08:34.824 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.134+0000 7f431f1dc700 1 -- 172.21.15.104:0/4098733335 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 msgr2=0x7f43181072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:34.824 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.134+0000 7f431f1dc700 1 --2- 172.21.15.104:0/4098733335 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 0x7f43181072b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f430c0098a0 tx=0x7f430c009bb0 comp rx=0 tx=0).stop 2024-06-09T22:08:34.824 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.134+0000 7f431f1dc700 1 -- 172.21.15.104:0/4098733335 shutdown_connections 2024-06-09T22:08:34.824 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.134+0000 7f431f1dc700 1 --2- 172.21.15.104:0/4098733335 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 0x7f43181072b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:34.824 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.134+0000 7f431f1dc700 1 -- 172.21.15.104:0/4098733335 >> 172.21.15.104:0/4098733335 conn(0x7f4318102470 msgr2=0x7f4318104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:34.824 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.134+0000 7f431f1dc700 1 -- 172.21.15.104:0/4098733335 shutdown_connections 2024-06-09T22:08:34.824 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.134+0000 7f431f1dc700 1 -- 172.21.15.104:0/4098733335 wait complete. 2024-06-09T22:08:34.824 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.135+0000 7f431f1dc700 1 Processor -- start 2024-06-09T22:08:34.824 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.135+0000 7f431f1dc700 1 -- start start 2024-06-09T22:08:34.824 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.135+0000 7f431f1dc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 0x7f431819bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:34.824 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.135+0000 7f431f1dc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f431810b3e0 con 0x7f4318106e90 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.135+0000 7f431cf78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 0x7f431819bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.135+0000 7f431cf78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 0x7f431819bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37426/0 (socket says 172.21.15.104:37426) 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.135+0000 7f431cf78700 1 -- 172.21.15.104:0/2398597930 learned_addr learned my addr 172.21.15.104:0/2398597930 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.136+0000 7f431cf78700 1 -- 172.21.15.104:0/2398597930 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431819c490 con 0x7f4318106e90 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.136+0000 7f431cf78700 1 --2- 172.21.15.104:0/2398597930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 0x7f431819bf50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f430c011b60 tx=0x7f430c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.136+0000 7f4315ffb700 1 -- 172.21.15.104:0/2398597930 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f430c003b20 con 0x7f4318106e90 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.136+0000 7f431f1dc700 1 -- 172.21.15.104:0/2398597930 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f431819c720 con 0x7f4318106e90 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.136+0000 7f4315ffb700 1 -- 172.21.15.104:0/2398597930 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f430c003cc0 con 0x7f4318106e90 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.136+0000 7f4315ffb700 1 -- 172.21.15.104:0/2398597930 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f430c024840 con 0x7f4318106e90 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.136+0000 7f431f1dc700 1 -- 172.21.15.104:0/2398597930 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f431819f410 con 0x7f4318106e90 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.136+0000 7f4315ffb700 1 -- 172.21.15.104:0/2398597930 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f430c022020 con 0x7f4318106e90 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.136+0000 7f4315ffb700 1 -- 172.21.15.104:0/2398597930 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f430c011e00 con 0x7f4318106e90 2024-06-09T22:08:34.825 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.137+0000 7f431f1dc700 1 -- 172.21.15.104:0/2398597930 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4318063af0 con 0x7f4318106e90 2024-06-09T22:08:34.826 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.139+0000 7f4315ffb700 1 -- 172.21.15.104:0/2398597930 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f430c043dd0 con 0x7f4318106e90 2024-06-09T22:08:34.826 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.208+0000 7f431f1dc700 1 -- 172.21.15.104:0/2398597930 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f431802df40 con 0x7f4318106e90 2024-06-09T22:08:34.826 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.208+0000 7f4315ffb700 1 -- 172.21.15.104:0/2398597930 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f430c032030 con 0x7f4318106e90 2024-06-09T22:08:34.826 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.209+0000 7f431f1dc700 1 -- 172.21.15.104:0/2398597930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 msgr2=0x7f431819bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:34.826 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.209+0000 7f431f1dc700 1 --2- 172.21.15.104:0/2398597930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 0x7f431819bf50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f430c011b60 tx=0x7f430c011c40 comp rx=0 tx=0).stop 2024-06-09T22:08:34.826 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.209+0000 7f431f1dc700 1 -- 172.21.15.104:0/2398597930 shutdown_connections 2024-06-09T22:08:34.826 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.209+0000 7f431f1dc700 1 --2- 172.21.15.104:0/2398597930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4318106e90 0x7f431819bf50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:34.826 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.209+0000 7f431f1dc700 1 -- 172.21.15.104:0/2398597930 >> 172.21.15.104:0/2398597930 conn(0x7f4318102470 msgr2=0x7f4318103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:34.826 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.210+0000 7f431f1dc700 1 -- 172.21.15.104:0/2398597930 shutdown_connections 2024-06-09T22:08:34.826 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:34.210+0000 7f431f1dc700 1 -- 172.21.15.104:0/2398597930 wait complete. 2024-06-09T22:08:34.826 INFO:teuthology.orchestra.run.smithi104.stdout:Restarting the monitor... 2024-06-09T22:08:35.242 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:34 smithi104 systemd[1]: Stopping Ceph mon.a for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:08:35.992 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:35 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a[32013]: 2024-06-09T22:08:35.733+0000 7fdccf38a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:08:35.992 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:35 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a[32013]: 2024-06-09T22:08:35.733+0000 7fdccf38a700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-06-09T22:08:35.992 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:35 smithi104 podman[32265]: 2024-06-09 22:08:35.93854725 +0000 UTC m=+0.934115246 container died f4ac17b2af78cef38df9a6c1983a968d1e62e875d1b5e396daf67126e505c8fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:08:36.492 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:36 smithi104 podman[32265]: 2024-06-09 22:08:36.298755133 +0000 UTC m=+1.294323123 container cleanup f4ac17b2af78cef38df9a6c1983a968d1e62e875d1b5e396daf67126e505c8fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-06-09T22:08:36.492 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:36 smithi104 bash[32265]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a 2024-06-09T22:08:36.983 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:36 smithi104 podman[32279]: 2024-06-09 22:08:36.616672098 +0000 UTC m=+0.669618019 container remove f4ac17b2af78cef38df9a6c1983a968d1e62e875d1b5e396daf67126e505c8fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0) 2024-06-09T22:08:36.983 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:36 smithi104 systemd[1]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-09T22:08:36.983 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:36 smithi104 systemd[1]: Stopped Ceph mon.a for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:08:36.984 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:36 smithi104 systemd[1]: Starting Ceph mon.a for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:08:37.242 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:36 smithi104 podman[32370]: 2024-06-09 22:08:36.901308394 +0000 UTC m=+0.017543909 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T22:08:37.242 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 podman[32370]: 2024-06-09 22:08:37.000619277 +0000 UTC m=+0.116854776 container create 2cb3d4ded8f69f6e28cefd01ca38469df63e8442bf5ea5c405eed635a0b1e429 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-09T22:08:37.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 podman[32370]: 2024-06-09 22:08:37.236580482 +0000 UTC m=+0.352815989 container init 2cb3d4ded8f69f6e28cefd01ca38469df63e8442bf5ea5c405eed635a0b1e429 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-06-09T22:08:37.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 podman[32370]: 2024-06-09 22:08:37.239812593 +0000 UTC m=+0.356048083 container start 2cb3d4ded8f69f6e28cefd01ca38469df63e8442bf5ea5c405eed635a0b1e429 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-09T22:08:37.333 INFO:teuthology.orchestra.run.smithi104.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-09T22:08:37.525 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: set uid:gid to 167:167 (ceph:ceph) 2024-06-09T22:08:37.525 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc), process ceph-mon, pid 2 2024-06-09T22:08:37.525 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: pidfile_write: ignore empty --pid-file 2024-06-09T22:08:37.525 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: load: jerasure load: lrc 2024-06-09T22:08:37.525 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: RocksDB version: 7.9.2 2024-06-09T22:08:37.525 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Git sha 0 2024-06-09T22:08:37.525 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Compile date 2024-05-30 15:42:47 2024-06-09T22:08:37.525 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: DB SUMMARY 2024-06-09T22:08:37.525 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: DB Session ID: V2YFA5MY3NURTV1J6RNS 2024-06-09T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: CURRENT file: CURRENT 2024-06-09T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: IDENTITY file: IDENTITY 2024-06-09T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-06-09T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-06-09T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105364 ; 2024-06-09T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.error_if_exists: 0 2024-06-09T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.create_if_missing: 0 2024-06-09T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.paranoid_checks: 1 2024-06-09T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-09T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-09T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.env: 0x558709b22c60 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.fs: PosixFileSystem 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.info_log: 0x55870b915b20 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_file_opening_threads: 16 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.statistics: (nil) 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.use_fsync: 0 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_log_file_size: 0 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.keep_log_file_num: 1000 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.recycle_log_file_num: 0 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.allow_fallocate: 1 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.allow_mmap_reads: 0 2024-06-09T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.allow_mmap_writes: 0 2024-06-09T22:08:37.528 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.use_direct_reads: 0 2024-06-09T22:08:37.528 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-09T22:08:37.528 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.create_missing_column_families: 0 2024-06-09T22:08:37.528 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.db_log_dir: 2024-06-09T22:08:37.528 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.wal_dir: 2024-06-09T22:08:37.528 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-09T22:08:37.529 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-09T22:08:37.529 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-09T22:08:37.529 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-09T22:08:37.529 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-09T22:08:37.529 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.advise_random_on_open: 1 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.db_write_buffer_size: 0 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.write_buffer_manager: 0x55870b8e61e0 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.rate_limiter: (nil) 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.wal_recovery_mode: 2 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.enable_thread_tracking: 0 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.enable_pipelined_write: 0 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.unordered_write: 0 2024-06-09T22:08:37.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-09T22:08:37.531 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-09T22:08:37.531 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-09T22:08:37.531 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-09T22:08:37.531 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.row_cache: None 2024-06-09T22:08:37.531 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.wal_filter: None 2024-06-09T22:08:37.531 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-09T22:08:37.531 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.allow_ingest_behind: 0 2024-06-09T22:08:37.531 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.two_write_queues: 0 2024-06-09T22:08:37.531 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.manual_wal_flush: 0 2024-06-09T22:08:37.531 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.wal_compression: 0 2024-06-09T22:08:37.531 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.atomic_flush: 0 2024-06-09T22:08:37.531 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.log_readahead_size: 0 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.best_efforts_recovery: 0 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.allow_data_in_errors: 0 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.db_host_id: __hostname__ 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_background_jobs: 2 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_background_compactions: -1 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_subcompactions: 1 2024-06-09T22:08:37.532 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_total_wal_size: 0 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_open_files: -1 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bytes_per_sync: 0 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_readahead_size: 0 2024-06-09T22:08:37.533 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_background_flushes: -1 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Compression algorithms supported: 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: kZSTD supported: 0 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: kXpressCompression supported: 0 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: kLZ4HCCompression supported: 1 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: kZlibCompression supported: 1 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: kSnappyCompression supported: 1 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: kLZ4Compression supported: 1 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: kBZip2Compression supported: 0 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-09T22:08:37.534 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.merge_operator: 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_filter: None 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_filter_factory: None 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.sst_partitioner_factory: None 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55870b915c40) 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout: cache_index_and_filter_blocks: 1 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout: pin_top_level_index_and_filter: 1 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout: index_type: 0 2024-06-09T22:08:37.535 INFO:journalctl@ceph.mon.a.smithi104.stdout: data_block_index_type: 0 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: index_shortening: 1 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: checksum: 4 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: no_block_cache: 0 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: block_cache: 0x55870b917090 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: block_cache_name: BinnedLRUCache 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: block_cache_options: 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: capacity : 536870912 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: num_shard_bits : 4 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: strict_capacity_limit : 0 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: high_pri_pool_ratio: 0.000 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: block_cache_compressed: (nil) 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: persistent_cache: (nil) 2024-06-09T22:08:37.536 INFO:journalctl@ceph.mon.a.smithi104.stdout: block_size: 4096 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: block_size_deviation: 10 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: block_restart_interval: 16 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: index_block_restart_interval: 1 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: metadata_block_size: 4096 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: partition_filters: 0 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: use_delta_encoding: 1 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: filter_policy: bloomfilter 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: whole_key_filtering: 1 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: verify_compression: 0 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: read_amp_bytes_per_bit: 0 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: format_version: 5 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: enable_index_compression: 1 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: block_align: 0 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: max_auto_readahead_size: 262144 2024-06-09T22:08:37.537 INFO:journalctl@ceph.mon.a.smithi104.stdout: prepopulate_block_cache: 0 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout: initial_auto_readahead_size: 8192 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout: num_file_reads_for_auto_readahead: 2 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.write_buffer_size: 33554432 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_write_buffer_number: 2 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compression: NoCompression 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bottommost_compression: Disabled 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.prefix_extractor: nullptr 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.num_levels: 7 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-09T22:08:37.538 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compression_opts.level: 32767 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compression_opts.strategy: 0 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-09T22:08:37.539 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compression_opts.enabled: false 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.target_file_size_base: 67108864 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-09T22:08:37.540 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.arena_block_size: 1048576 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.disable_auto_compactions: 0 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-09T22:08:37.541 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.table_properties_collectors: 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.inplace_update_support: 0 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-09T22:08:37.542 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.bloom_locality: 0 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.max_successive_merges: 0 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.paranoid_file_checks: 0 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.force_consistency_checks: 1 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.report_bg_io_stats: 0 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.ttl: 2592000 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.enable_blob_files: false 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.min_blob_size: 0 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.blob_file_size: 268435456 2024-06-09T22:08:37.543 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.blob_file_starting_level: 0 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3ad2663b-5bad-467e-8105-28b62fa93257 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717970917270139, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717970917271145, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100407, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98327, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14306, "raw_average_key_size": 49, "raw_value_size": 90573, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1717970917, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3ad2663b-5bad-467e-8105-28b62fa93257", "db_session_id": "V2YFA5MY3NURTV1J6RNS", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717970917271226, "job": 1, "event": "recovery_finished"} 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-06-09T22:08:37.544 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55870ba06000 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: DB pointer 0x55870b9f0000 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout: ** DB Stats ** 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout: 2024-06-09T22:08:37.545 INFO:journalctl@ceph.mon.a.smithi104.stdout: ** Compaction Stats [default] ** 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: L0 2/0 99.89 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: Sum 2/0 99.89 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: ** Compaction Stats [default] ** 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: 2024-06-09T22:08:37.546 INFO:journalctl@ceph.mon.a.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout: Cumulative compaction: 0.00 GB write, 31.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout: Interval compaction: 0.00 GB write, 31.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout: Block cache BinnedLRUCache@0x55870b917090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout: 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: starting mon.a rank 0 at public addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] at bind addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:08:37.547 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: mon.a@-1(???) e1 preinit fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: mon.a@-1(???).mds e1 new map 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: mon.a@-1(???).mds e1 print_map 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout: e1 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout: legacy client fscid: -1 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout: 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout: No filesystems configured 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-09T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-09T22:08:37.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: monmap epoch 1 2024-06-09T22:08:37.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:08:37.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: last_changed 2024-06-09T22:08:19.534976+0000 2024-06-09T22:08:37.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: created 2024-06-09T22:08:19.534976+0000 2024-06-09T22:08:37.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: min_mon_release 19 (squid) 2024-06-09T22:08:37.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: election_strategy: 1 2024-06-09T22:08:37.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: 0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-06-09T22:08:37.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: fsmap 2024-06-09T22:08:37.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: osdmap e1: 0 total, 0 up, 0 in 2024-06-09T22:08:37.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 ceph-mon[32389]: mgrmap e1: no daemons active 2024-06-09T22:08:37.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 bash[32370]: 2cb3d4ded8f69f6e28cefd01ca38469df63e8442bf5ea5c405eed635a0b1e429 2024-06-09T22:08:37.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:37 smithi104 systemd[1]: Started Ceph mon.a for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:08:38.346 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.740+0000 7fcc7b9a3700 1 Processor -- start 2024-06-09T22:08:38.350 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.740+0000 7fcc7b9a3700 1 -- start start 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.740+0000 7fcc7b9a3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 0x7fcc7410add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.740+0000 7fcc7b9a3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7407be10 con 0x7fcc741089e0 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.741+0000 7fcc7973f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 0x7fcc7410add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.741+0000 7fcc7973f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 0x7fcc7410add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47052/0 (socket says 172.21.15.104:47052) 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.741+0000 7fcc7973f700 1 -- 172.21.15.104:0/1744293373 learned_addr learned my addr 172.21.15.104:0/1744293373 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.741+0000 7fcc7973f700 1 -- 172.21.15.104:0/1744293373 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7410b310 con 0x7fcc741089e0 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.741+0000 7fcc7973f700 1 --2- 172.21.15.104:0/1744293373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 0x7fcc7410add0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcc700098a0 tx=0x7fcc70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fdc7f474ae02c91 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.741+0000 7fcc6bfff700 1 -- 172.21.15.104:0/1744293373 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc70004160 con 0x7fcc741089e0 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.741+0000 7fcc6bfff700 1 -- 172.21.15.104:0/1744293373 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcc70004300 con 0x7fcc741089e0 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.742+0000 7fcc6bfff700 1 -- 172.21.15.104:0/1744293373 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc70011b50 con 0x7fcc741089e0 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.742+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/1744293373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 msgr2=0x7fcc7410add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.742+0000 7fcc7b9a3700 1 --2- 172.21.15.104:0/1744293373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 0x7fcc7410add0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcc700098a0 tx=0x7fcc70009bb0 comp rx=0 tx=0).stop 2024-06-09T22:08:38.351 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.742+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/1744293373 shutdown_connections 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.742+0000 7fcc7b9a3700 1 --2- 172.21.15.104:0/1744293373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 0x7fcc7410add0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.742+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/1744293373 >> 172.21.15.104:0/1744293373 conn(0x7fcc74102370 msgr2=0x7fcc741047d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.742+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/1744293373 shutdown_connections 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.742+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/1744293373 wait complete. 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.743+0000 7fcc7b9a3700 1 Processor -- start 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.743+0000 7fcc7b9a3700 1 -- start start 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.743+0000 7fcc7b9a3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 0x7fcc74108260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.743+0000 7fcc7b9a3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7410c2d0 con 0x7fcc741089e0 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.743+0000 7fcc7973f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 0x7fcc74108260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.743+0000 7fcc7973f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 0x7fcc74108260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47060/0 (socket says 172.21.15.104:47060) 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.743+0000 7fcc7973f700 1 -- 172.21.15.104:0/2085770842 learned_addr learned my addr 172.21.15.104:0/2085770842 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.743+0000 7fcc7973f700 1 -- 172.21.15.104:0/2085770842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc741087a0 con 0x7fcc741089e0 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.743+0000 7fcc7973f700 1 --2- 172.21.15.104:0/2085770842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 0x7fcc74108260 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fcc70011670 tx=0x7fcc70011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:38.352 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.744+0000 7fcc6a7fc700 1 -- 172.21.15.104:0/2085770842 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc700039c0 con 0x7fcc741089e0 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.744+0000 7fcc6a7fc700 1 -- 172.21.15.104:0/2085770842 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcc70011e50 con 0x7fcc741089e0 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.744+0000 7fcc6a7fc700 1 -- 172.21.15.104:0/2085770842 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc7001a5c0 con 0x7fcc741089e0 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.744+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/2085770842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc74106990 con 0x7fcc741089e0 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.744+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/2085770842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc74106ed0 con 0x7fcc741089e0 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.744+0000 7fcc6a7fc700 1 -- 172.21.15.104:0/2085770842 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcc7001ac50 con 0x7fcc741089e0 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.744+0000 7fcc6a7fc700 1 -- 172.21.15.104:0/2085770842 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcc70019700 con 0x7fcc741089e0 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.745+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/2085770842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc74063af0 con 0x7fcc741089e0 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.747+0000 7fcc6a7fc700 1 -- 172.21.15.104:0/2085770842 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fcc70003b60 con 0x7fcc741089e0 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.814+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/2085770842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fcc74107400 con 0x7fcc741089e0 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.819+0000 7fcc6a7fc700 1 -- 172.21.15.104:0/2085770842 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fcc70019900 con 0x7fcc741089e0 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.819+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/2085770842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 msgr2=0x7fcc74108260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.819+0000 7fcc7b9a3700 1 --2- 172.21.15.104:0/2085770842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 0x7fcc74108260 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fcc70011670 tx=0x7fcc70011b60 comp rx=0 tx=0).stop 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.819+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/2085770842 shutdown_connections 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.819+0000 7fcc7b9a3700 1 --2- 172.21.15.104:0/2085770842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc741089e0 0x7fcc74108260 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:38.353 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.819+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/2085770842 >> 172.21.15.104:0/2085770842 conn(0x7fcc74102370 msgr2=0x7fcc74102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:38.354 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.820+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/2085770842 shutdown_connections 2024-06-09T22:08:38.354 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:37.820+0000 7fcc7b9a3700 1 -- 172.21.15.104:0/2085770842 wait complete. 2024-06-09T22:08:38.354 INFO:teuthology.orchestra.run.smithi104.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-09T22:08:38.376 INFO:teuthology.orchestra.run.smithi104.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-09T22:08:38.376 INFO:teuthology.orchestra.run.smithi104.stdout:Creating mgr... 2024-06-09T22:08:38.376 INFO:teuthology.orchestra.run.smithi104.stdout:Verifying port 0.0.0.0:9283 ... 2024-06-09T22:08:38.377 INFO:teuthology.orchestra.run.smithi104.stdout:Verifying port 0.0.0.0:8765 ... 2024-06-09T22:08:38.627 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.x 2024-06-09T22:08:38.627 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.x.service: Unit ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.x.service not loaded. 2024-06-09T22:08:38.770 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968.target.wants/ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@.service. 2024-06-09T22:08:38.992 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:38 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2085770842' entity='client.admin' 2024-06-09T22:08:39.297 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:39 smithi104 podman[32601]: 2024-06-09 22:08:39.240622038 +0000 UTC m=+0.332842643 container start b6c0d3ce81dbb43d65a11537b077ccb874d44886be4274548f615e6faaf21166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS) 2024-06-09T22:08:39.311 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-09T22:08:39.311 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout disabled 2024-06-09T22:08:39.318 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-09T22:08:39.318 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout inactive 2024-06-09T22:08:39.319 INFO:teuthology.orchestra.run.smithi104.stdout:firewalld.service is not enabled 2024-06-09T22:08:39.319 INFO:teuthology.orchestra.run.smithi104.stdout:Not possible to enable service . firewalld.service is not available 2024-06-09T22:08:39.327 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-09T22:08:39.327 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout disabled 2024-06-09T22:08:39.334 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-09T22:08:39.334 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout inactive 2024-06-09T22:08:39.335 INFO:teuthology.orchestra.run.smithi104.stdout:firewalld.service is not enabled 2024-06-09T22:08:39.335 INFO:teuthology.orchestra.run.smithi104.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-06-09T22:08:39.335 INFO:teuthology.orchestra.run.smithi104.stdout:Waiting for mgr to start... 2024-06-09T22:08:39.335 INFO:teuthology.orchestra.run.smithi104.stdout:Waiting for mgr... 2024-06-09T22:08:39.548 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:39 smithi104 bash[32601]: b6c0d3ce81dbb43d65a11537b077ccb874d44886be4274548f615e6faaf21166 2024-06-09T22:08:39.548 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:39 smithi104 systemd[1]: Started Ceph mgr.x for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:08:39.548 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:39 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:39.394+0000 7f284080f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-09T22:08:39.840 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:39 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:39.582+0000 7f284080f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:08:39.840 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:39 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:39.686+0000 7f284080f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-09T22:08:40.091 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:39 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3921184397' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-09T22:08:40.091 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:40 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:40.016+0000 7f284080f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-09T22:08:40.091 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:40 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:40.105+0000 7f284080f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-09T22:08:40.400 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:40.401 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout { 2024-06-09T22:08:40.401 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:08:40.401 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "health": { 2024-06-09T22:08:40.401 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-09T22:08:40.401 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-09T22:08:40.401 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-09T22:08:40.401 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:40.401 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-09T22:08:40.401 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-09T22:08:40.401 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 0 2024-06-09T22:08:40.401 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ], 2024-06-09T22:08:40.401 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "a" 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ], 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-09T22:08:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-09T22:08:40.410 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-09T22:08:40.410 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-09T22:08:40.410 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-09T22:08:40.410 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:40.410 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-09T22:08:40.410 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-09T22:08:40.410 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-09T22:08:40.410 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-09T22:08:40.410 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-09T22:08:40.410 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-09T22:08:40.411 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "restful" 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ], 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "modified": "2024-06-09T22:08:31.471888+0000", 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout } 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.774+0000 7fc864831700 1 Processor -- start 2024-06-09T22:08:40.412 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.774+0000 7fc864831700 1 -- start start 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.774+0000 7fc864831700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 0x7fc860108380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.774+0000 7fc864831700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc8601088c0 con 0x7fc860107f40 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.775+0000 7fc85e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 0x7fc860108380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.775+0000 7fc85e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 0x7fc860108380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47084/0 (socket says 172.21.15.104:47084) 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.775+0000 7fc85e59c700 1 -- 172.21.15.104:0/3776037867 learned_addr learned my addr 172.21.15.104:0/3776037867 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.775+0000 7fc85e59c700 1 -- 172.21.15.104:0/3776037867 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8601090d0 con 0x7fc860107f40 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.775+0000 7fc85e59c700 1 --2- 172.21.15.104:0/3776037867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 0x7fc860108380 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc8540098a0 tx=0x7fc854009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44d9cf295c2dfdb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.775+0000 7fc85d59a700 1 -- 172.21.15.104:0/3776037867 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc854004160 con 0x7fc860107f40 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.776+0000 7fc85d59a700 1 -- 172.21.15.104:0/3776037867 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc854004300 con 0x7fc860107f40 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.776+0000 7fc85d59a700 1 -- 172.21.15.104:0/3776037867 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc854011b50 con 0x7fc860107f40 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.776+0000 7fc864831700 1 -- 172.21.15.104:0/3776037867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 msgr2=0x7fc860108380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.776+0000 7fc864831700 1 --2- 172.21.15.104:0/3776037867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 0x7fc860108380 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc8540098a0 tx=0x7fc854009bb0 comp rx=0 tx=0).stop 2024-06-09T22:08:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.776+0000 7fc864831700 1 -- 172.21.15.104:0/3776037867 shutdown_connections 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.776+0000 7fc864831700 1 --2- 172.21.15.104:0/3776037867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 0x7fc860108380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.776+0000 7fc864831700 1 -- 172.21.15.104:0/3776037867 >> 172.21.15.104:0/3776037867 conn(0x7fc8601035b0 msgr2=0x7fc8601059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.776+0000 7fc864831700 1 -- 172.21.15.104:0/3776037867 shutdown_connections 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.776+0000 7fc864831700 1 -- 172.21.15.104:0/3776037867 wait complete. 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.777+0000 7fc864831700 1 Processor -- start 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.777+0000 7fc864831700 1 -- start start 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.777+0000 7fc864831700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 0x7fc86019bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.777+0000 7fc864831700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc86010c390 con 0x7fc860107f40 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.777+0000 7fc85e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 0x7fc86019bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.777+0000 7fc85e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 0x7fc86019bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47092/0 (socket says 172.21.15.104:47092) 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.777+0000 7fc85e59c700 1 -- 172.21.15.104:0/3921184397 learned_addr learned my addr 172.21.15.104:0/3921184397 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.778+0000 7fc85e59c700 1 -- 172.21.15.104:0/3921184397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86019c4c0 con 0x7fc860107f40 2024-06-09T22:08:40.414 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.778+0000 7fc85e59c700 1 --2- 172.21.15.104:0/3921184397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 0x7fc86019bf80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc854009fb0 tx=0x7fc854011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.778+0000 7fc84f7fe700 1 -- 172.21.15.104:0/3921184397 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8540038b0 con 0x7fc860107f40 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.778+0000 7fc84f7fe700 1 -- 172.21.15.104:0/3921184397 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc854011e50 con 0x7fc860107f40 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.778+0000 7fc84f7fe700 1 -- 172.21.15.104:0/3921184397 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc854019550 con 0x7fc860107f40 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.778+0000 7fc864831700 1 -- 172.21.15.104:0/3921184397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc86019c750 con 0x7fc860107f40 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.778+0000 7fc864831700 1 -- 172.21.15.104:0/3921184397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc86019f440 con 0x7fc860107f40 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.778+0000 7fc84f7fe700 1 -- 172.21.15.104:0/3921184397 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc854019b90 con 0x7fc860107f40 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.778+0000 7fc84f7fe700 1 -- 172.21.15.104:0/3921184397 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc854024730 con 0x7fc860107f40 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.779+0000 7fc864831700 1 -- 172.21.15.104:0/3921184397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc86010c510 con 0x7fc860107f40 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.783+0000 7fc84f7fe700 1 -- 172.21.15.104:0/3921184397 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc854003a50 con 0x7fc860107f40 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.854+0000 7fc864831700 1 -- 172.21.15.104:0/3921184397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc860063af0 con 0x7fc860107f40 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.855+0000 7fc84f7fe700 1 -- 172.21.15.104:0/3921184397 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc8540249d0 con 0x7fc860107f40 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.856+0000 7fc864831700 1 -- 172.21.15.104:0/3921184397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 msgr2=0x7fc86019bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.856+0000 7fc864831700 1 --2- 172.21.15.104:0/3921184397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 0x7fc86019bf80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc854009fb0 tx=0x7fc854011bf0 comp rx=0 tx=0).stop 2024-06-09T22:08:40.415 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.856+0000 7fc864831700 1 -- 172.21.15.104:0/3921184397 shutdown_connections 2024-06-09T22:08:40.416 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.856+0000 7fc864831700 1 --2- 172.21.15.104:0/3921184397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc860107f40 0x7fc86019bf80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:40.416 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.856+0000 7fc864831700 1 -- 172.21.15.104:0/3921184397 >> 172.21.15.104:0/3921184397 conn(0x7fc8601035b0 msgr2=0x7fc860104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:40.416 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.856+0000 7fc864831700 1 -- 172.21.15.104:0/3921184397 shutdown_connections 2024-06-09T22:08:40.416 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:39.856+0000 7fc864831700 1 -- 172.21.15.104:0/3921184397 wait complete. 2024-06-09T22:08:40.416 INFO:teuthology.orchestra.run.smithi104.stdout:mgr not available, waiting (1/15)... 2024-06-09T22:08:41.612 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:41 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:41.293+0000 7f284080f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-09T22:08:41.612 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:41 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:41.460+0000 7f284080f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-09T22:08:41.612 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:41 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:41.626+0000 7f284080f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-09T22:08:42.186 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:41 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:41.910+0000 7f284080f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-09T22:08:42.189 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:42 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:42.001+0000 7f284080f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-09T22:08:42.189 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:42 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:42.201+0000 7f284080f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-09T22:08:42.464 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:42 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:42.400+0000 7f284080f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:08:43.242 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:43 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2749279128' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-09T22:08:43.242 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:43 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:43.041+0000 7f284080f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-09T22:08:43.552 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:43 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:43.478+0000 7f284080f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-09T22:08:43.555 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:43.564 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout { 2024-06-09T22:08:43.564 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:08:43.564 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "health": { 2024-06-09T22:08:43.564 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-09T22:08:43.564 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 0 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ], 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "a" 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ], 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-09T22:08:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-09T22:08:43.566 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-09T22:08:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "restful" 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ], 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "modified": "2024-06-09T22:08:31.471888+0000", 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-09T22:08:43.568 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout } 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.912+0000 7fd10677a700 1 Processor -- start 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.912+0000 7fd10677a700 1 -- start start 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.913+0000 7fd10677a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 0x7fd1001083f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.913+0000 7fd10677a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd100108930 con 0x7fd100107fb0 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.913+0000 7fd0fffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 0x7fd1001083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.913+0000 7fd0fffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 0x7fd1001083f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47100/0 (socket says 172.21.15.104:47100) 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.913+0000 7fd0fffff700 1 -- 172.21.15.104:0/1139234209 learned_addr learned my addr 172.21.15.104:0/1139234209 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.913+0000 7fd0fffff700 1 -- 172.21.15.104:0/1139234209 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1001090f0 con 0x7fd100107fb0 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.913+0000 7fd0fffff700 1 --2- 172.21.15.104:0/1139234209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 0x7fd1001083f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00098a0 tx=0x7fd0f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5023b3216f30d06 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.914+0000 7fd0feffd700 1 -- 172.21.15.104:0/1139234209 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0f0004160 con 0x7fd100107fb0 2024-06-09T22:08:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.914+0000 7fd0feffd700 1 -- 172.21.15.104:0/1139234209 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd0f0004300 con 0x7fd100107fb0 2024-06-09T22:08:43.570 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.914+0000 7fd0feffd700 1 -- 172.21.15.104:0/1139234209 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0f0011b50 con 0x7fd100107fb0 2024-06-09T22:08:43.570 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.914+0000 7fd10677a700 1 -- 172.21.15.104:0/1139234209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 msgr2=0x7fd1001083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:43.570 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.914+0000 7fd10677a700 1 --2- 172.21.15.104:0/1139234209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 0x7fd1001083f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00098a0 tx=0x7fd0f0009bb0 comp rx=0 tx=0).stop 2024-06-09T22:08:43.570 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.914+0000 7fd10677a700 1 -- 172.21.15.104:0/1139234209 shutdown_connections 2024-06-09T22:08:43.570 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.914+0000 7fd10677a700 1 --2- 172.21.15.104:0/1139234209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 0x7fd1001083f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:43.570 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.914+0000 7fd10677a700 1 -- 172.21.15.104:0/1139234209 >> 172.21.15.104:0/1139234209 conn(0x7fd100103560 msgr2=0x7fd1001059e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:43.570 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.914+0000 7fd10677a700 1 -- 172.21.15.104:0/1139234209 shutdown_connections 2024-06-09T22:08:43.570 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.914+0000 7fd10677a700 1 -- 172.21.15.104:0/1139234209 wait complete. 2024-06-09T22:08:43.570 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.915+0000 7fd10677a700 1 Processor -- start 2024-06-09T22:08:43.570 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.915+0000 7fd10677a700 1 -- start start 2024-06-09T22:08:43.570 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.915+0000 7fd10677a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 0x7fd10007e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.915+0000 7fd10677a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd10010c3b0 con 0x7fd100107fb0 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.916+0000 7fd0fffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 0x7fd10007e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.916+0000 7fd0fffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 0x7fd10007e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47116/0 (socket says 172.21.15.104:47116) 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.916+0000 7fd0fffff700 1 -- 172.21.15.104:0/2749279128 learned_addr learned my addr 172.21.15.104:0/2749279128 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.916+0000 7fd0fffff700 1 -- 172.21.15.104:0/2749279128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10007e920 con 0x7fd100107fb0 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.916+0000 7fd0fffff700 1 --2- 172.21.15.104:0/2749279128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 0x7fd10007e3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0000c00 tx=0x7fd0f0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.916+0000 7fd0fd7fa700 1 -- 172.21.15.104:0/2749279128 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0f0003b20 con 0x7fd100107fb0 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.916+0000 7fd0fd7fa700 1 -- 172.21.15.104:0/2749279128 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd0f0003cc0 con 0x7fd100107fb0 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.916+0000 7fd10677a700 1 -- 172.21.15.104:0/2749279128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd10007cb10 con 0x7fd100107fb0 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.916+0000 7fd0fd7fa700 1 -- 172.21.15.104:0/2749279128 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0f0024910 con 0x7fd100107fb0 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.916+0000 7fd10677a700 1 -- 172.21.15.104:0/2749279128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd10007d050 con 0x7fd100107fb0 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.917+0000 7fd0fd7fa700 1 -- 172.21.15.104:0/2749279128 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd0f0022020 con 0x7fd100107fb0 2024-06-09T22:08:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.917+0000 7fd0fd7fa700 1 -- 172.21.15.104:0/2749279128 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd0f0011e00 con 0x7fd100107fb0 2024-06-09T22:08:43.572 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.918+0000 7fd10677a700 1 -- 172.21.15.104:0/2749279128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd10010c530 con 0x7fd100107fb0 2024-06-09T22:08:43.572 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.920+0000 7fd0fd7fa700 1 -- 172.21.15.104:0/2749279128 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd0f001acd0 con 0x7fd100107fb0 2024-06-09T22:08:43.572 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.992+0000 7fd10677a700 1 -- 172.21.15.104:0/2749279128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd10007d6e0 con 0x7fd100107fb0 2024-06-09T22:08:43.572 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.993+0000 7fd0fd7fa700 1 -- 172.21.15.104:0/2749279128 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd0f0032030 con 0x7fd100107fb0 2024-06-09T22:08:43.572 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.994+0000 7fd10677a700 1 -- 172.21.15.104:0/2749279128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 msgr2=0x7fd10007e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:43.572 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.994+0000 7fd10677a700 1 --2- 172.21.15.104:0/2749279128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 0x7fd10007e3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0000c00 tx=0x7fd0f0011c10 comp rx=0 tx=0).stop 2024-06-09T22:08:43.572 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.994+0000 7fd10677a700 1 -- 172.21.15.104:0/2749279128 shutdown_connections 2024-06-09T22:08:43.572 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.994+0000 7fd10677a700 1 --2- 172.21.15.104:0/2749279128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd100107fb0 0x7fd10007e3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:43.572 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.994+0000 7fd10677a700 1 -- 172.21.15.104:0/2749279128 >> 172.21.15.104:0/2749279128 conn(0x7fd100103560 msgr2=0x7fd100104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:43.572 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.994+0000 7fd10677a700 1 -- 172.21.15.104:0/2749279128 shutdown_connections 2024-06-09T22:08:43.572 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:42.994+0000 7fd10677a700 1 -- 172.21.15.104:0/2749279128 wait complete. 2024-06-09T22:08:43.572 INFO:teuthology.orchestra.run.smithi104.stdout:mgr not available, waiting (2/15)... 2024-06-09T22:08:43.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:43 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:43.579+0000 7f284080f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-09T22:08:43.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:43 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:43.780+0000 7f284080f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-09T22:08:44.628 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:44 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:44.312+0000 7f284080f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-09T22:08:44.628 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:44 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:44.404+0000 7f284080f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-09T22:08:44.971 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:44 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:44.642+0000 7f284080f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-09T22:08:44.971 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:44 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:44.774+0000 7f284080f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-09T22:08:44.971 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:44 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:44.870+0000 7f284080f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-09T22:08:44.971 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:44 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:44.985+0000 7f284080f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-09T22:08:45.669 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:45 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:45.279+0000 7f284080f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-09T22:08:45.979 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:45 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:45.889+0000 7f284080f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-09T22:08:46.242 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:46 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:46.119+0000 7f284080f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-09T22:08:46.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:46 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/422867098' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-09T22:08:46.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:46 smithi104 ceph-mon[32389]: Activating manager daemon x 2024-06-09T22:08:46.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:46 smithi104 ceph-mon[32389]: mgrmap e2: x(active, starting, since 0.00304754s) 2024-06-09T22:08:46.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:46 smithi104 ceph-mon[32389]: from='mgr.14100 172.21.15.104:0/2937789543' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-09T22:08:46.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:46 smithi104 ceph-mon[32389]: from='mgr.14100 172.21.15.104:0/2937789543' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-09T22:08:46.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:46 smithi104 ceph-mon[32389]: from='mgr.14100 172.21.15.104:0/2937789543' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-09T22:08:46.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:46 smithi104 ceph-mon[32389]: from='mgr.14100 172.21.15.104:0/2937789543' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:08:46.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:46 smithi104 ceph-mon[32389]: from='mgr.14100 172.21.15.104:0/2937789543' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-09T22:08:46.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:46 smithi104 ceph-mon[32389]: Manager daemon x is now available 2024-06-09T22:08:46.620 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:46.621 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout { 2024-06-09T22:08:46.621 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:08:46.621 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "health": { 2024-06-09T22:08:46.621 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-09T22:08:46.621 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-09T22:08:46.621 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-09T22:08:46.621 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:46.621 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-09T22:08:46.621 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-09T22:08:46.621 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 0 2024-06-09T22:08:46.621 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ], 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "a" 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ], 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-09T22:08:46.622 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-09T22:08:46.623 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-09T22:08:46.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-09T22:08:46.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-09T22:08:46.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-09T22:08:46.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:46.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-09T22:08:46.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-09T22:08:46.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-09T22:08:46.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-09T22:08:46.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-09T22:08:46.631 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-09T22:08:46.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-09T22:08:46.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-09T22:08:46.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-09T22:08:46.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:46.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-09T22:08:46.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:46.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-09T22:08:46.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-09T22:08:46.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:46.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-09T22:08:46.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-09T22:08:46.632 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-09T22:08:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-09T22:08:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-09T22:08:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-09T22:08:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "restful" 2024-06-09T22:08:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ], 2024-06-09T22:08:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-09T22:08:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-09T22:08:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "modified": "2024-06-09T22:08:31.471888+0000", 2024-06-09T22:08:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-09T22:08:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:46.634 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-09T22:08:46.634 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout } 2024-06-09T22:08:46.634 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.002+0000 7f91688ad700 1 Processor -- start 2024-06-09T22:08:46.634 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.002+0000 7f91688ad700 1 -- start start 2024-06-09T22:08:46.634 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.002+0000 7f91688ad700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 0x7f916407cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:46.634 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.002+0000 7f91688ad700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f916407d140 con 0x7f916407e6c0 2024-06-09T22:08:46.634 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.003+0000 7f916259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 0x7f916407cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:46.634 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.003+0000 7f916259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 0x7f916407cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53246/0 (socket says 172.21.15.104:53246) 2024-06-09T22:08:46.634 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.003+0000 7f916259c700 1 -- 172.21.15.104:0/3511278046 learned_addr learned my addr 172.21.15.104:0/3511278046 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:46.634 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.003+0000 7f916259c700 1 -- 172.21.15.104:0/3511278046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916407d2c0 con 0x7f916407e6c0 2024-06-09T22:08:46.634 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.003+0000 7f916259c700 1 --2- 172.21.15.104:0/3511278046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 0x7f916407cb70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f91580098a0 tx=0x7f9158009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28219b3ccc5a064 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:46.634 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.003+0000 7f916159a700 1 -- 172.21.15.104:0/3511278046 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9158004160 con 0x7f916407e6c0 2024-06-09T22:08:46.635 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.003+0000 7f916159a700 1 -- 172.21.15.104:0/3511278046 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9158004300 con 0x7f916407e6c0 2024-06-09T22:08:46.635 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.003+0000 7f916159a700 1 -- 172.21.15.104:0/3511278046 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9158011b50 con 0x7f916407e6c0 2024-06-09T22:08:46.635 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.004+0000 7f91688ad700 1 -- 172.21.15.104:0/3511278046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 msgr2=0x7f916407cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:46.635 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.004+0000 7f91688ad700 1 --2- 172.21.15.104:0/3511278046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 0x7f916407cb70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f91580098a0 tx=0x7f9158009bb0 comp rx=0 tx=0).stop 2024-06-09T22:08:46.635 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.004+0000 7f91688ad700 1 -- 172.21.15.104:0/3511278046 shutdown_connections 2024-06-09T22:08:46.635 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.004+0000 7f91688ad700 1 --2- 172.21.15.104:0/3511278046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 0x7f916407cb70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:46.635 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.004+0000 7f91688ad700 1 -- 172.21.15.104:0/3511278046 >> 172.21.15.104:0/3511278046 conn(0x7f9164103540 msgr2=0x7f9164105980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:46.635 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.004+0000 7f91688ad700 1 -- 172.21.15.104:0/3511278046 shutdown_connections 2024-06-09T22:08:46.635 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.004+0000 7f91688ad700 1 -- 172.21.15.104:0/3511278046 wait complete. 2024-06-09T22:08:46.635 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.005+0000 7f91688ad700 1 Processor -- start 2024-06-09T22:08:46.635 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.005+0000 7f91688ad700 1 -- start start 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.005+0000 7f91688ad700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 0x7f916419e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.005+0000 7f91688ad700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f916410e550 con 0x7f916407e6c0 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.005+0000 7f916259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 0x7f916419e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.005+0000 7f916259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 0x7f916419e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53248/0 (socket says 172.21.15.104:53248) 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.005+0000 7f916259c700 1 -- 172.21.15.104:0/422867098 learned_addr learned my addr 172.21.15.104:0/422867098 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.005+0000 7f916259c700 1 -- 172.21.15.104:0/422867098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916419e610 con 0x7f916407e6c0 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.005+0000 7f916259c700 1 --2- 172.21.15.104:0/422867098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 0x7f916419e0d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9158011670 tx=0x7f9158011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.006+0000 7f91537fe700 1 -- 172.21.15.104:0/422867098 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91580039c0 con 0x7f916407e6c0 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.006+0000 7f91688ad700 1 -- 172.21.15.104:0/422867098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f916419e8a0 con 0x7f916407e6c0 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.006+0000 7f91537fe700 1 -- 172.21.15.104:0/422867098 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9158011e50 con 0x7f916407e6c0 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.006+0000 7f91537fe700 1 -- 172.21.15.104:0/422867098 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f915801a5c0 con 0x7f916407e6c0 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.006+0000 7f91688ad700 1 -- 172.21.15.104:0/422867098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91641a1590 con 0x7f916407e6c0 2024-06-09T22:08:46.636 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.006+0000 7f91537fe700 1 -- 172.21.15.104:0/422867098 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f915801ac50 con 0x7f916407e6c0 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.006+0000 7f91537fe700 1 -- 172.21.15.104:0/422867098 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9158019700 con 0x7f916407e6c0 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.007+0000 7f91688ad700 1 -- 172.21.15.104:0/422867098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9164063af0 con 0x7f916407e6c0 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.009+0000 7f91537fe700 1 -- 172.21.15.104:0/422867098 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9158003b60 con 0x7f916407e6c0 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.080+0000 7f91688ad700 1 -- 172.21.15.104:0/422867098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f91641a1760 con 0x7f916407e6c0 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.081+0000 7f91537fe700 1 -- 172.21.15.104:0/422867098 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9158019900 con 0x7f916407e6c0 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.082+0000 7f91688ad700 1 -- 172.21.15.104:0/422867098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 msgr2=0x7f916419e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.082+0000 7f91688ad700 1 --2- 172.21.15.104:0/422867098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 0x7f916419e0d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9158011670 tx=0x7f9158011b60 comp rx=0 tx=0).stop 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.082+0000 7f91688ad700 1 -- 172.21.15.104:0/422867098 shutdown_connections 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.082+0000 7f91688ad700 1 --2- 172.21.15.104:0/422867098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f916407e6c0 0x7f916419e0d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.082+0000 7f91688ad700 1 -- 172.21.15.104:0/422867098 >> 172.21.15.104:0/422867098 conn(0x7f9164103540 msgr2=0x7f9164104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.082+0000 7f91688ad700 1 -- 172.21.15.104:0/422867098 shutdown_connections 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:46.082+0000 7f91688ad700 1 -- 172.21.15.104:0/422867098 wait complete. 2024-06-09T22:08:46.637 INFO:teuthology.orchestra.run.smithi104.stdout:mgr not available, waiting (3/15)... 2024-06-09T22:08:47.492 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:47 smithi104 ceph-mon[32389]: from='mgr.14100 172.21.15.104:0/2937789543' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-09T22:08:47.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:47 smithi104 ceph-mon[32389]: from='mgr.14100 172.21.15.104:0/2937789543' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-09T22:08:47.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:47 smithi104 ceph-mon[32389]: from='mgr.14100 172.21.15.104:0/2937789543' entity='mgr.x' 2024-06-09T22:08:47.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:47 smithi104 ceph-mon[32389]: from='mgr.14100 172.21.15.104:0/2937789543' entity='mgr.x' 2024-06-09T22:08:47.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:47 smithi104 ceph-mon[32389]: from='mgr.14100 172.21.15.104:0/2937789543' entity='mgr.x' 2024-06-09T22:08:47.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:47 smithi104 ceph-mon[32389]: mgrmap e3: x(active, since 1.0077s) 2024-06-09T22:08:49.334 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:49 smithi104 ceph-mon[32389]: mgrmap e4: x(active, since 2s) 2024-06-09T22:08:50.019 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:50.019 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout { 2024-06-09T22:08:50.019 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:08:50.019 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "health": { 2024-06-09T22:08:50.019 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-09T22:08:50.019 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-09T22:08:50.019 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-09T22:08:50.019 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:50.019 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-09T22:08:50.019 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-09T22:08:50.019 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 0 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ], 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "a" 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ], 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-09T22:08:50.020 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-09T22:08:50.028 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-09T22:08:50.028 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-09T22:08:50.028 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-09T22:08:50.028 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:50.028 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-09T22:08:50.028 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-09T22:08:50.028 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-09T22:08:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "restful" 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ], 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "modified": "2024-06-09T22:08:31.471888+0000", 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout }, 2024-06-09T22:08:50.030 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout } 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.062+0000 7f4614fb9700 1 Processor -- start 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.062+0000 7f4614fb9700 1 -- start start 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.062+0000 7f4614fb9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 0x7f4610108390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.062+0000 7f4614fb9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f46101088d0 con 0x7f4610107f50 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.062+0000 7f460e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 0x7f4610108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.062+0000 7f460e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 0x7f4610108390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53340/0 (socket says 172.21.15.104:53340) 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.062+0000 7f460e59c700 1 -- 172.21.15.104:0/713090560 learned_addr learned my addr 172.21.15.104:0/713090560 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.063+0000 7f460e59c700 1 -- 172.21.15.104:0/713090560 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46101090e0 con 0x7f4610107f50 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.063+0000 7f460e59c700 1 --2- 172.21.15.104:0/713090560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 0x7f4610108390 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f46000098a0 tx=0x7f4600009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e020bbe989b372e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.063+0000 7f460d59a700 1 -- 172.21.15.104:0/713090560 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4600004160 con 0x7f4610107f50 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.063+0000 7f460d59a700 1 -- 172.21.15.104:0/713090560 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4600004300 con 0x7f4610107f50 2024-06-09T22:08:50.031 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.063+0000 7f460d59a700 1 -- 172.21.15.104:0/713090560 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4600011b50 con 0x7f4610107f50 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.064+0000 7f4614fb9700 1 -- 172.21.15.104:0/713090560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 msgr2=0x7f4610108390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.064+0000 7f4614fb9700 1 --2- 172.21.15.104:0/713090560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 0x7f4610108390 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f46000098a0 tx=0x7f4600009bb0 comp rx=0 tx=0).stop 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.064+0000 7f4614fb9700 1 -- 172.21.15.104:0/713090560 shutdown_connections 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.064+0000 7f4614fb9700 1 --2- 172.21.15.104:0/713090560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 0x7f4610108390 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.064+0000 7f4614fb9700 1 -- 172.21.15.104:0/713090560 >> 172.21.15.104:0/713090560 conn(0x7f4610103540 msgr2=0x7f4610105980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.064+0000 7f4614fb9700 1 -- 172.21.15.104:0/713090560 shutdown_connections 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.064+0000 7f4614fb9700 1 -- 172.21.15.104:0/713090560 wait complete. 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f4614fb9700 1 Processor -- start 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f4614fb9700 1 -- start start 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f4614fb9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 0x7f461019bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f4614fb9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f461010c3a0 con 0x7f4610107f50 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f460e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 0x7f461019bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:50.032 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f460e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 0x7f461019bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53346/0 (socket says 172.21.15.104:53346) 2024-06-09T22:08:50.033 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f460e59c700 1 -- 172.21.15.104:0/1690034098 learned_addr learned my addr 172.21.15.104:0/1690034098 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:50.033 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f460e59c700 1 -- 172.21.15.104:0/1690034098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461019c480 con 0x7f4610107f50 2024-06-09T22:08:50.033 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f460e59c700 1 --2- 172.21.15.104:0/1690034098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 0x7f461019bf40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4600011670 tx=0x7f4600011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:50.033 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f45ff7fe700 1 -- 172.21.15.104:0/1690034098 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46000039c0 con 0x7f4610107f50 2024-06-09T22:08:50.033 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f4614fb9700 1 -- 172.21.15.104:0/1690034098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461019c710 con 0x7f4610107f50 2024-06-09T22:08:50.033 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f45ff7fe700 1 -- 172.21.15.104:0/1690034098 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4600011e50 con 0x7f4610107f50 2024-06-09T22:08:50.033 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.065+0000 7f45ff7fe700 1 -- 172.21.15.104:0/1690034098 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f460001a5c0 con 0x7f4610107f50 2024-06-09T22:08:50.033 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.066+0000 7f4614fb9700 1 -- 172.21.15.104:0/1690034098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f461019f400 con 0x7f4610107f50 2024-06-09T22:08:50.033 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.066+0000 7f45ff7fe700 1 -- 172.21.15.104:0/1690034098 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f460001a8c0 con 0x7f4610107f50 2024-06-09T22:08:50.033 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.067+0000 7f45ff7fe700 1 --2- 172.21.15.104:0/1690034098 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f45f8038fb0 0x7f45f803b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:50.033 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.067+0000 7f45ff7fe700 1 -- 172.21.15.104:0/1690034098 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f460004d420 con 0x7f4610107f50 2024-06-09T22:08:50.033 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.067+0000 7f4614fb9700 1 -- 172.21.15.104:0/1690034098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f461010c520 con 0x7f4610107f50 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.067+0000 7f460dd9b700 1 --2- 172.21.15.104:0/1690034098 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f45f8038fb0 0x7f45f803b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.067+0000 7f460dd9b700 1 --2- 172.21.15.104:0/1690034098 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f45f8038fb0 0x7f45f803b470 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4604006fd0 tx=0x7f4604006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.072+0000 7f45ff7fe700 1 -- 172.21.15.104:0/1690034098 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4600019c70 con 0x7f4610107f50 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.348+0000 7f4614fb9700 1 -- 172.21.15.104:0/1690034098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f461019f5d0 con 0x7f4610107f50 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.349+0000 7f45ff7fe700 1 -- 172.21.15.104:0/1690034098 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f460002cc00 con 0x7f4610107f50 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.351+0000 7f4614fb9700 1 -- 172.21.15.104:0/1690034098 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f45f8038fb0 msgr2=0x7f45f803b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.351+0000 7f4614fb9700 1 --2- 172.21.15.104:0/1690034098 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f45f8038fb0 0x7f45f803b470 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4604006fd0 tx=0x7f4604006e40 comp rx=0 tx=0).stop 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.351+0000 7f4614fb9700 1 -- 172.21.15.104:0/1690034098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 msgr2=0x7f461019bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.351+0000 7f4614fb9700 1 --2- 172.21.15.104:0/1690034098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 0x7f461019bf40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4600011670 tx=0x7f4600011b60 comp rx=0 tx=0).stop 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.351+0000 7f4614fb9700 1 -- 172.21.15.104:0/1690034098 shutdown_connections 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.351+0000 7f4614fb9700 1 --2- 172.21.15.104:0/1690034098 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f45f8038fb0 0x7f45f803b470 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.351+0000 7f4614fb9700 1 --2- 172.21.15.104:0/1690034098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4610107f50 0x7f461019bf40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.351+0000 7f4614fb9700 1 -- 172.21.15.104:0/1690034098 >> 172.21.15.104:0/1690034098 conn(0x7f4610103540 msgr2=0x7f46101040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:50.035 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.351+0000 7f4614fb9700 1 -- 172.21.15.104:0/1690034098 shutdown_connections 2024-06-09T22:08:50.035 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:49.351+0000 7f4614fb9700 1 -- 172.21.15.104:0/1690034098 wait complete. 2024-06-09T22:08:50.035 INFO:teuthology.orchestra.run.smithi104.stdout:mgr is available 2024-06-09T22:08:50.433 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:50 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1690034098' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-09T22:08:51.213 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:51.213 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout [global] 2024-06-09T22:08:51.213 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout fsid = a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:08:51.213 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-09T22:08:51.213 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout [osd] 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-09T22:08:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.457+0000 7fda94d92700 1 Processor -- start 2024-06-09T22:08:51.215 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.457+0000 7fda94d92700 1 -- start start 2024-06-09T22:08:51.215 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.457+0000 7fda94d92700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 0x7fda901083e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:51.215 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.457+0000 7fda94d92700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fda90108920 con 0x7fda90107fc0 2024-06-09T22:08:51.215 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.457+0000 7fda8e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 0x7fda901083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:51.215 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.457+0000 7fda8e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 0x7fda901083e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53362/0 (socket says 172.21.15.104:53362) 2024-06-09T22:08:51.215 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.457+0000 7fda8e59c700 1 -- 172.21.15.104:0/1645505968 learned_addr learned my addr 172.21.15.104:0/1645505968 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:51.215 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.458+0000 7fda8e59c700 1 -- 172.21.15.104:0/1645505968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda90108aa0 con 0x7fda90107fc0 2024-06-09T22:08:51.215 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.458+0000 7fda8e59c700 1 --2- 172.21.15.104:0/1645505968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 0x7fda901083e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fda800098a0 tx=0x7fda80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73515bce7780f3b9 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:51.215 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.458+0000 7fda8dd9b700 1 -- 172.21.15.104:0/1645505968 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda80004160 con 0x7fda90107fc0 2024-06-09T22:08:51.215 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.458+0000 7fda8dd9b700 1 -- 172.21.15.104:0/1645505968 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fda80004300 con 0x7fda90107fc0 2024-06-09T22:08:51.215 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.458+0000 7fda8dd9b700 1 -- 172.21.15.104:0/1645505968 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda80011b50 con 0x7fda90107fc0 2024-06-09T22:08:51.216 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.458+0000 7fda94d92700 1 -- 172.21.15.104:0/1645505968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 msgr2=0x7fda901083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:51.216 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.458+0000 7fda94d92700 1 --2- 172.21.15.104:0/1645505968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 0x7fda901083e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fda800098a0 tx=0x7fda80009bb0 comp rx=0 tx=0).stop 2024-06-09T22:08:51.216 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.459+0000 7fda94d92700 1 -- 172.21.15.104:0/1645505968 shutdown_connections 2024-06-09T22:08:51.216 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.459+0000 7fda94d92700 1 --2- 172.21.15.104:0/1645505968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 0x7fda901083e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:51.216 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.459+0000 7fda94d92700 1 -- 172.21.15.104:0/1645505968 >> 172.21.15.104:0/1645505968 conn(0x7fda901035e0 msgr2=0x7fda90105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:51.216 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.459+0000 7fda94d92700 1 -- 172.21.15.104:0/1645505968 shutdown_connections 2024-06-09T22:08:51.216 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.459+0000 7fda94d92700 1 -- 172.21.15.104:0/1645505968 wait complete. 2024-06-09T22:08:51.216 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.459+0000 7fda94d92700 1 Processor -- start 2024-06-09T22:08:51.216 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.459+0000 7fda94d92700 1 -- start start 2024-06-09T22:08:51.216 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.460+0000 7fda94d92700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 0x7fda9019bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.460+0000 7fda94d92700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fda9010c430 con 0x7fda90107fc0 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.460+0000 7fda8e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 0x7fda9019bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.460+0000 7fda8e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 0x7fda9019bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53374/0 (socket says 172.21.15.104:53374) 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.460+0000 7fda8e59c700 1 -- 172.21.15.104:0/2891508187 learned_addr learned my addr 172.21.15.104:0/2891508187 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.460+0000 7fda8e59c700 1 -- 172.21.15.104:0/2891508187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9019c430 con 0x7fda90107fc0 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.460+0000 7fda8e59c700 1 --2- 172.21.15.104:0/2891508187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 0x7fda9019bef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fda80011670 tx=0x7fda80011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.460+0000 7fda877fe700 1 -- 172.21.15.104:0/2891508187 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda800039c0 con 0x7fda90107fc0 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.460+0000 7fda877fe700 1 -- 172.21.15.104:0/2891508187 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fda80011e50 con 0x7fda90107fc0 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.460+0000 7fda94d92700 1 -- 172.21.15.104:0/2891508187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9019c6c0 con 0x7fda90107fc0 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.460+0000 7fda877fe700 1 -- 172.21.15.104:0/2891508187 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda8001a5c0 con 0x7fda90107fc0 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.460+0000 7fda94d92700 1 -- 172.21.15.104:0/2891508187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9019f3b0 con 0x7fda90107fc0 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.461+0000 7fda877fe700 1 -- 172.21.15.104:0/2891508187 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fda8001a8c0 con 0x7fda90107fc0 2024-06-09T22:08:51.217 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.461+0000 7fda94d92700 1 -- 172.21.15.104:0/2891508187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda90063af0 con 0x7fda90107fc0 2024-06-09T22:08:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.462+0000 7fda877fe700 1 --2- 172.21.15.104:0/2891508187 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fda7c038fb0 0x7fda7c03b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.462+0000 7fda877fe700 1 -- 172.21.15.104:0/2891508187 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fda8004d410 con 0x7fda90107fc0 2024-06-09T22:08:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.462+0000 7fda87fff700 1 --2- 172.21.15.104:0/2891508187 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fda7c038fb0 0x7fda7c03b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.462+0000 7fda87fff700 1 --2- 172.21.15.104:0/2891508187 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fda7c038fb0 0x7fda7c03b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fda78006fd0 tx=0x7fda78006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.467+0000 7fda877fe700 1 -- 172.21.15.104:0/2891508187 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fda8000f2e0 con 0x7fda90107fc0 2024-06-09T22:08:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.659+0000 7fda94d92700 1 -- 172.21.15.104:0/2891508187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fda9019f580 con 0x7fda90107fc0 2024-06-09T22:08:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.660+0000 7fda877fe700 1 -- 172.21.15.104:0/2891508187 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fda9019f580 con 0x7fda90107fc0 2024-06-09T22:08:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.662+0000 7fda94d92700 1 -- 172.21.15.104:0/2891508187 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fda7c038fb0 msgr2=0x7fda7c03b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.662+0000 7fda94d92700 1 --2- 172.21.15.104:0/2891508187 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fda7c038fb0 0x7fda7c03b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fda78006fd0 tx=0x7fda78006e40 comp rx=0 tx=0).stop 2024-06-09T22:08:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.662+0000 7fda94d92700 1 -- 172.21.15.104:0/2891508187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 msgr2=0x7fda9019bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.662+0000 7fda94d92700 1 --2- 172.21.15.104:0/2891508187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 0x7fda9019bef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fda80011670 tx=0x7fda80011b60 comp rx=0 tx=0).stop 2024-06-09T22:08:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.662+0000 7fda94d92700 1 -- 172.21.15.104:0/2891508187 shutdown_connections 2024-06-09T22:08:51.219 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.662+0000 7fda94d92700 1 --2- 172.21.15.104:0/2891508187 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fda7c038fb0 0x7fda7c03b470 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:51.219 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.662+0000 7fda94d92700 1 --2- 172.21.15.104:0/2891508187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda90107fc0 0x7fda9019bef0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:51.219 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.662+0000 7fda94d92700 1 -- 172.21.15.104:0/2891508187 >> 172.21.15.104:0/2891508187 conn(0x7fda901035e0 msgr2=0x7fda90104050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:51.219 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.662+0000 7fda94d92700 1 -- 172.21.15.104:0/2891508187 shutdown_connections 2024-06-09T22:08:51.219 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:50.662+0000 7fda94d92700 1 -- 172.21.15.104:0/2891508187 wait complete. 2024-06-09T22:08:51.219 INFO:teuthology.orchestra.run.smithi104.stdout:Enabling cephadm module... 2024-06-09T22:08:51.242 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:51 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2891508187' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-09T22:08:52.124 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:52 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2538035083' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-09T22:08:52.444 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: ignoring --setuser ceph since I am not root 2024-06-09T22:08:52.444 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: ignoring --setgroup ceph since I am not root 2024-06-09T22:08:52.444 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:52.305+0000 7f4d3fb5f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-09T22:08:52.703 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:52.491+0000 7f4d3fb5f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:08:52.703 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:52.593+0000 7f4d3fb5f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-09T22:08:52.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.631+0000 7f7f92b5d700 1 Processor -- start 2024-06-09T22:08:52.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.631+0000 7f7f92b5d700 1 -- start start 2024-06-09T22:08:52.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.631+0000 7f7f92b5d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 0x7f7f8c107290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:52.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.631+0000 7f7f92b5d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c1077d0 con 0x7f7f8c106e70 2024-06-09T22:08:52.710 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.632+0000 7f7f908f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 0x7f7f8c107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.632+0000 7f7f908f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 0x7f7f8c107290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53380/0 (socket says 172.21.15.104:53380) 2024-06-09T22:08:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.632+0000 7f7f908f9700 1 -- 172.21.15.104:0/728864966 learned_addr learned my addr 172.21.15.104:0/728864966 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.632+0000 7f7f908f9700 1 -- 172.21.15.104:0/728864966 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f8c107f80 con 0x7f7f8c106e70 2024-06-09T22:08:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.632+0000 7f7f908f9700 1 --2- 172.21.15.104:0/728864966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 0x7f7f8c107290 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009e80 tx=0x7f7f8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1579cda68971fd4f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.632+0000 7f7f8b7fe700 1 -- 172.21.15.104:0/728864966 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f80017070 con 0x7f7f8c106e70 2024-06-09T22:08:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.632+0000 7f7f8b7fe700 1 -- 172.21.15.104:0/728864966 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7f8000b8c0 con 0x7f7f8c106e70 2024-06-09T22:08:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.632+0000 7f7f8b7fe700 1 -- 172.21.15.104:0/728864966 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f800045c0 con 0x7f7f8c106e70 2024-06-09T22:08:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.633+0000 7f7f92b5d700 1 -- 172.21.15.104:0/728864966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 msgr2=0x7f7f8c107290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.633+0000 7f7f92b5d700 1 --2- 172.21.15.104:0/728864966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 0x7f7f8c107290 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009e80 tx=0x7f7f8000b330 comp rx=0 tx=0).stop 2024-06-09T22:08:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.633+0000 7f7f92b5d700 1 -- 172.21.15.104:0/728864966 shutdown_connections 2024-06-09T22:08:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.633+0000 7f7f92b5d700 1 --2- 172.21.15.104:0/728864966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 0x7f7f8c107290 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.633+0000 7f7f92b5d700 1 -- 172.21.15.104:0/728864966 >> 172.21.15.104:0/728864966 conn(0x7f7f8c102410 msgr2=0x7f7f8c104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:52.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.633+0000 7f7f92b5d700 1 -- 172.21.15.104:0/728864966 shutdown_connections 2024-06-09T22:08:52.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.633+0000 7f7f92b5d700 1 -- 172.21.15.104:0/728864966 wait complete. 2024-06-09T22:08:52.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.633+0000 7f7f92b5d700 1 Processor -- start 2024-06-09T22:08:52.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.634+0000 7f7f92b5d700 1 -- start start 2024-06-09T22:08:52.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.634+0000 7f7f92b5d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 0x7f7f8c104770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:52.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.634+0000 7f7f92b5d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c10b3c0 con 0x7f7f8c106e70 2024-06-09T22:08:52.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.634+0000 7f7f908f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 0x7f7f8c104770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:52.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.634+0000 7f7f908f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 0x7f7f8c104770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53392/0 (socket says 172.21.15.104:53392) 2024-06-09T22:08:52.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.634+0000 7f7f908f9700 1 -- 172.21.15.104:0/2538035083 learned_addr learned my addr 172.21.15.104:0/2538035083 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:52.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.634+0000 7f7f908f9700 1 -- 172.21.15.104:0/2538035083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f8c104cb0 con 0x7f7f8c106e70 2024-06-09T22:08:52.712 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.634+0000 7f7f908f9700 1 --2- 172.21.15.104:0/2538035083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 0x7f7f8c104770 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7f800072a0 tx=0x7f7f8000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.634+0000 7f7f89ffb700 1 -- 172.21.15.104:0/2538035083 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f80020020 con 0x7f7f8c106e70 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.634+0000 7f7f92b5d700 1 -- 172.21.15.104:0/2538035083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f8c102ea0 con 0x7f7f8c106e70 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.634+0000 7f7f89ffb700 1 -- 172.21.15.104:0/2538035083 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7f80003b50 con 0x7f7f8c106e70 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.634+0000 7f7f89ffb700 1 -- 172.21.15.104:0/2538035083 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f80017040 con 0x7f7f8c106e70 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.635+0000 7f7f92b5d700 1 -- 172.21.15.104:0/2538035083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f8c1033e0 con 0x7f7f8c106e70 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.635+0000 7f7f89ffb700 1 -- 172.21.15.104:0/2538035083 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f7f80023020 con 0x7f7f8c106e70 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.635+0000 7f7f92b5d700 1 -- 172.21.15.104:0/2538035083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f8c063af0 con 0x7f7f8c106e70 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.635+0000 7f7f89ffb700 1 --2- 172.21.15.104:0/2538035083 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f7f7c038fb0 0x7f7f7c03b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.636+0000 7f7f89ffb700 1 -- 172.21.15.104:0/2538035083 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7f8000f050 con 0x7f7f8c106e70 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.636+0000 7f7f8bfff700 1 --2- 172.21.15.104:0/2538035083 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f7f7c038fb0 0x7f7f7c03b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.636+0000 7f7f8bfff700 1 --2- 172.21.15.104:0/2538035083 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f7f7c038fb0 0x7f7f7c03b470 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7f78006fd0 tx=0x7f7f78006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.640+0000 7f7f89ffb700 1 -- 172.21.15.104:0/2538035083 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7f8002a020 con 0x7f7f8c106e70 2024-06-09T22:08:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:51.888+0000 7f7f92b5d700 1 -- 172.21.15.104:0/2538035083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f7f8c1952c0 con 0x7f7f8c106e70 2024-06-09T22:08:52.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:52.138+0000 7f7f89ffb700 1 -- 172.21.15.104:0/2538035083 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f7f8c1952c0 con 0x7f7f8c106e70 2024-06-09T22:08:52.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:52.140+0000 7f7f89ffb700 1 -- 172.21.15.104:0/2538035083 <== mon.0 v2:172.21.15.104:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f7f80025400 con 0x7f7f8c106e70 2024-06-09T22:08:52.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:52.141+0000 7f7f92b5d700 1 -- 172.21.15.104:0/2538035083 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f7f7c038fb0 msgr2=0x7f7f7c03b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:52.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:52.141+0000 7f7f92b5d700 1 --2- 172.21.15.104:0/2538035083 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f7f7c038fb0 0x7f7f7c03b470 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7f78006fd0 tx=0x7f7f78006e40 comp rx=0 tx=0).stop 2024-06-09T22:08:52.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:52.141+0000 7f7f92b5d700 1 -- 172.21.15.104:0/2538035083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 msgr2=0x7f7f8c104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:52.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:52.141+0000 7f7f92b5d700 1 --2- 172.21.15.104:0/2538035083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 0x7f7f8c104770 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7f800072a0 tx=0x7f7f8000bd50 comp rx=0 tx=0).stop 2024-06-09T22:08:52.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:52.141+0000 7f7f92b5d700 1 -- 172.21.15.104:0/2538035083 shutdown_connections 2024-06-09T22:08:52.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:52.141+0000 7f7f92b5d700 1 --2- 172.21.15.104:0/2538035083 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f7f7c038fb0 0x7f7f7c03b470 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:52.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:52.141+0000 7f7f92b5d700 1 --2- 172.21.15.104:0/2538035083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f8c106e70 0x7f7f8c104770 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:52.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:52.141+0000 7f7f92b5d700 1 -- 172.21.15.104:0/2538035083 >> 172.21.15.104:0/2538035083 conn(0x7f7f8c102410 msgr2=0x7f7f8c192a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:52.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:52.142+0000 7f7f92b5d700 1 -- 172.21.15.104:0/2538035083 shutdown_connections 2024-06-09T22:08:52.714 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:52.142+0000 7f7f92b5d700 1 -- 172.21.15.104:0/2538035083 wait complete. 2024-06-09T22:08:52.988 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:52.914+0000 7f4d3fb5f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-09T22:08:53.242 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:53 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2538035083' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-09T22:08:53.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:53 smithi104 ceph-mon[32389]: mgrmap e5: x(active, since 6s) 2024-06-09T22:08:53.243 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:53 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:53.004+0000 7f4d3fb5f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-09T22:08:53.951 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout { 2024-06-09T22:08:53.951 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout } 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.103+0000 7f557410e700 1 Processor -- start 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.104+0000 7f557410e700 1 -- start start 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.104+0000 7f557410e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 0x7f556c07cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.104+0000 7f557410e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f556c07d140 con 0x7f556c07e6c0 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.104+0000 7f5571eaa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 0x7f556c07cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.104+0000 7f5571eaa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 0x7f556c07cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53412/0 (socket says 172.21.15.104:53412) 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.104+0000 7f5571eaa700 1 -- 172.21.15.104:0/1854087260 learned_addr learned my addr 172.21.15.104:0/1854087260 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.104+0000 7f5571eaa700 1 -- 172.21.15.104:0/1854087260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556c07d2c0 con 0x7f556c07e6c0 2024-06-09T22:08:53.952 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.104+0000 7f5571eaa700 1 --2- 172.21.15.104:0/1854087260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 0x7f556c07cb70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f55680098a0 tx=0x7f5568009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0cfdf597b641eed server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.105+0000 7f5570ea8700 1 -- 172.21.15.104:0/1854087260 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5568004160 con 0x7f556c07e6c0 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.105+0000 7f5570ea8700 1 -- 172.21.15.104:0/1854087260 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5568004300 con 0x7f556c07e6c0 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.105+0000 7f5570ea8700 1 -- 172.21.15.104:0/1854087260 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5568011b50 con 0x7f556c07e6c0 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.105+0000 7f557410e700 1 -- 172.21.15.104:0/1854087260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 msgr2=0x7f556c07cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.105+0000 7f557410e700 1 --2- 172.21.15.104:0/1854087260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 0x7f556c07cb70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f55680098a0 tx=0x7f5568009bb0 comp rx=0 tx=0).stop 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.105+0000 7f557410e700 1 -- 172.21.15.104:0/1854087260 shutdown_connections 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.105+0000 7f557410e700 1 --2- 172.21.15.104:0/1854087260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 0x7f556c07cb70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.105+0000 7f557410e700 1 -- 172.21.15.104:0/1854087260 >> 172.21.15.104:0/1854087260 conn(0x7f556c103580 msgr2=0x7f556c1059a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.105+0000 7f557410e700 1 -- 172.21.15.104:0/1854087260 shutdown_connections 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.105+0000 7f557410e700 1 -- 172.21.15.104:0/1854087260 wait complete. 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.106+0000 7f557410e700 1 Processor -- start 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.106+0000 7f557410e700 1 -- start start 2024-06-09T22:08:53.953 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.106+0000 7f557410e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 0x7f556c19e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:53.954 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.106+0000 7f557410e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f556c10e550 con 0x7f556c07e6c0 2024-06-09T22:08:53.954 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.106+0000 7f5571eaa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 0x7f556c19e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:08:53.954 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.106+0000 7f5571eaa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 0x7f556c19e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53422/0 (socket says 172.21.15.104:53422) 2024-06-09T22:08:53.954 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.106+0000 7f5571eaa700 1 -- 172.21.15.104:0/4158332381 learned_addr learned my addr 172.21.15.104:0/4158332381 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:08:53.954 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.106+0000 7f5571eaa700 1 -- 172.21.15.104:0/4158332381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556c19e680 con 0x7f556c07e6c0 2024-06-09T22:08:53.954 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.107+0000 7f5571eaa700 1 --2- 172.21.15.104:0/4158332381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 0x7f556c19e140 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5568011670 tx=0x7f5568011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:08:53.954 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.107+0000 7f5562ffd700 1 -- 172.21.15.104:0/4158332381 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55680039c0 con 0x7f556c07e6c0 2024-06-09T22:08:53.954 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.107+0000 7f5562ffd700 1 -- 172.21.15.104:0/4158332381 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5568011e50 con 0x7f556c07e6c0 2024-06-09T22:08:53.954 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.107+0000 7f557410e700 1 -- 172.21.15.104:0/4158332381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f556c19e910 con 0x7f556c07e6c0 2024-06-09T22:08:53.954 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.107+0000 7f5562ffd700 1 -- 172.21.15.104:0/4158332381 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f556801a5c0 con 0x7f556c07e6c0 2024-06-09T22:08:53.954 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.107+0000 7f557410e700 1 -- 172.21.15.104:0/4158332381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f556c1a1600 con 0x7f556c07e6c0 2024-06-09T22:08:53.954 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.108+0000 7f5562ffd700 1 -- 172.21.15.104:0/4158332381 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f5568024430 con 0x7f556c07e6c0 2024-06-09T22:08:53.955 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.108+0000 7f557410e700 1 -- 172.21.15.104:0/4158332381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f556c063af0 con 0x7f556c07e6c0 2024-06-09T22:08:53.955 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.108+0000 7f5562ffd700 1 --2- 172.21.15.104:0/4158332381 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f5558039050 0x7f555803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:08:53.955 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.108+0000 7f5562ffd700 1 -- 172.21.15.104:0/4158332381 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f556804d3f0 con 0x7f556c07e6c0 2024-06-09T22:08:53.955 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.108+0000 7f55716a9700 1 -- 172.21.15.104:0/4158332381 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f5558039050 msgr2=0x7f555803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1879478526 2024-06-09T22:08:53.955 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.108+0000 7f55716a9700 1 --2- 172.21.15.104:0/4158332381 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f5558039050 0x7f555803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:08:53.955 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.113+0000 7f5562ffd700 1 -- 172.21.15.104:0/4158332381 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f556804ce10 con 0x7f556c07e6c0 2024-06-09T22:08:53.955 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.309+0000 7f55716a9700 1 -- 172.21.15.104:0/4158332381 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f5558039050 msgr2=0x7f555803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1879478526 2024-06-09T22:08:53.955 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.309+0000 7f55716a9700 1 --2- 172.21.15.104:0/4158332381 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f5558039050 0x7f555803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:08:53.955 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.371+0000 7f557410e700 1 -- 172.21.15.104:0/4158332381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f556c1a17d0 con 0x7f556c07e6c0 2024-06-09T22:08:53.955 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.372+0000 7f5562ffd700 1 -- 172.21.15.104:0/4158332381 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f556c1a17d0 con 0x7f556c07e6c0 2024-06-09T22:08:53.955 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.374+0000 7f557410e700 1 -- 172.21.15.104:0/4158332381 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f5558039050 msgr2=0x7f555803b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-09T22:08:53.955 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.374+0000 7f557410e700 1 --2- 172.21.15.104:0/4158332381 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f5558039050 0x7f555803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:53.956 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.374+0000 7f557410e700 1 -- 172.21.15.104:0/4158332381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 msgr2=0x7f556c19e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:08:53.956 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.374+0000 7f557410e700 1 --2- 172.21.15.104:0/4158332381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 0x7f556c19e140 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5568011670 tx=0x7f5568011b60 comp rx=0 tx=0).stop 2024-06-09T22:08:53.956 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.374+0000 7f557410e700 1 -- 172.21.15.104:0/4158332381 shutdown_connections 2024-06-09T22:08:53.956 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.374+0000 7f557410e700 1 --2- 172.21.15.104:0/4158332381 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7f5558039050 0x7f555803b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:53.956 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.374+0000 7f557410e700 1 --2- 172.21.15.104:0/4158332381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f556c07e6c0 0x7f556c19e140 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:08:53.956 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.374+0000 7f557410e700 1 -- 172.21.15.104:0/4158332381 >> 172.21.15.104:0/4158332381 conn(0x7f556c103580 msgr2=0x7f556c1040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:08:53.956 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.374+0000 7f557410e700 1 -- 172.21.15.104:0/4158332381 shutdown_connections 2024-06-09T22:08:53.956 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:53.374+0000 7f557410e700 1 -- 172.21.15.104:0/4158332381 wait complete. 2024-06-09T22:08:53.956 INFO:teuthology.orchestra.run.smithi104.stdout:Waiting for the mgr to restart... 2024-06-09T22:08:53.956 INFO:teuthology.orchestra.run.smithi104.stdout:Waiting for mgr epoch 5... 2024-06-09T22:08:54.211 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:54 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/4158332381' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-09T22:08:54.493 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:54.224+0000 7f4d3fb5f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-09T22:08:54.493 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:54.398+0000 7f4d3fb5f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-09T22:08:54.841 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:54.568+0000 7f4d3fb5f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-09T22:08:54.841 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:54.854+0000 7f4d3fb5f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-09T22:08:55.134 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:54.946+0000 7f4d3fb5f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-09T22:08:55.134 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:55 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:55.147+0000 7f4d3fb5f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-09T22:08:55.492 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:55 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:55.340+0000 7f4d3fb5f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:08:56.242 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:55 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:55.934+0000 7f4d3fb5f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-09T22:08:56.637 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:56.354+0000 7f4d3fb5f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-09T22:08:56.637 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:56.450+0000 7f4d3fb5f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-09T22:08:56.637 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:56.651+0000 7f4d3fb5f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-09T22:08:57.492 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:57.184+0000 7f4d3fb5f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-09T22:08:57.493 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:57.275+0000 7f4d3fb5f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-09T22:08:57.846 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:57.518+0000 7f4d3fb5f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-09T22:08:57.846 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:57.651+0000 7f4d3fb5f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-09T22:08:57.846 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:57.745+0000 7f4d3fb5f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-09T22:08:57.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:57.860+0000 7f4d3fb5f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-09T22:08:58.492 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:58.154+0000 7f4d3fb5f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-09T22:08:58.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:58.751+0000 7f4d3fb5f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-09T22:08:58.993 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:08:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:08:58.966+0000 7f4d3fb5f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-09T22:08:59.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: Active manager daemon x restarted 2024-06-09T22:08:59.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: Activating manager daemon x 2024-06-09T22:08:59.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: osdmap e2: 0 total, 0 up, 0 in 2024-06-09T22:08:59.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: mgrmap e6: x(active, starting, since 0.00344692s) 2024-06-09T22:08:59.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:08:59.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-09T22:08:59.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-09T22:08:59.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-09T22:08:59.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-09T22:08:59.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: Manager daemon x is now available 2024-06-09T22:08:59.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' 2024-06-09T22:08:59.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' 2024-06-09T22:08:59.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:08:59.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:08:59.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-09T22:08:59.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:08:59 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-09T22:09:00.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:00 smithi104 ceph-mon[32389]: Found migration_current of "None". Setting to last migration. 2024-06-09T22:09:00.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:00 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' 2024-06-09T22:09:00.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:00 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' 2024-06-09T22:09:00.243 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:00 smithi104 ceph-mon[32389]: mgrmap e7: x(active, since 1.00529s) 2024-06-09T22:09:00.558 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout { 2024-06-09T22:09:00.558 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-06-09T22:09:00.558 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-09T22:09:00.558 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout } 2024-06-09T22:09:00.558 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.402+0000 7fd08c1e8700 1 Processor -- start 2024-06-09T22:09:00.558 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.402+0000 7fd08c1e8700 1 -- start start 2024-06-09T22:09:00.558 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.403+0000 7fd08c1e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 0x7fd084107260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:00.558 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.403+0000 7fd08c1e8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd0841077a0 con 0x7fd084106e40 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.403+0000 7fd089f84700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 0x7fd084107260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.403+0000 7fd089f84700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 0x7fd084107260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53424/0 (socket says 172.21.15.104:53424) 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.403+0000 7fd089f84700 1 -- 172.21.15.104:0/153875204 learned_addr learned my addr 172.21.15.104:0/153875204 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.403+0000 7fd089f84700 1 -- 172.21.15.104:0/153875204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd084107fa0 con 0x7fd084106e40 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.403+0000 7fd089f84700 1 --2- 172.21.15.104:0/153875204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 0x7fd084107260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd0800098a0 tx=0x7fd080009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9945945fe90c6c83 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.403+0000 7fd088f82700 1 -- 172.21.15.104:0/153875204 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd080004160 con 0x7fd084106e40 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.403+0000 7fd088f82700 1 -- 172.21.15.104:0/153875204 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd080004300 con 0x7fd084106e40 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.403+0000 7fd088f82700 1 -- 172.21.15.104:0/153875204 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd080011b50 con 0x7fd084106e40 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.404+0000 7fd08c1e8700 1 -- 172.21.15.104:0/153875204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 msgr2=0x7fd084107260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.404+0000 7fd08c1e8700 1 --2- 172.21.15.104:0/153875204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 0x7fd084107260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd0800098a0 tx=0x7fd080009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.404+0000 7fd08c1e8700 1 -- 172.21.15.104:0/153875204 shutdown_connections 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.404+0000 7fd08c1e8700 1 --2- 172.21.15.104:0/153875204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 0x7fd084107260 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:00.559 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.404+0000 7fd08c1e8700 1 -- 172.21.15.104:0/153875204 >> 172.21.15.104:0/153875204 conn(0x7fd084102400 msgr2=0x7fd084104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:00.560 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.404+0000 7fd08c1e8700 1 -- 172.21.15.104:0/153875204 shutdown_connections 2024-06-09T22:09:00.560 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.404+0000 7fd08c1e8700 1 -- 172.21.15.104:0/153875204 wait complete. 2024-06-09T22:09:00.560 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.405+0000 7fd08c1e8700 1 Processor -- start 2024-06-09T22:09:00.560 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.405+0000 7fd08c1e8700 1 -- start start 2024-06-09T22:09:00.560 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.405+0000 7fd08c1e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 0x7fd08419bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:00.560 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.405+0000 7fd08c1e8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd08410b3e0 con 0x7fd084106e40 2024-06-09T22:09:00.560 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.405+0000 7fd089f84700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 0x7fd08419bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:00.560 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.405+0000 7fd089f84700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 0x7fd08419bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53430/0 (socket says 172.21.15.104:53430) 2024-06-09T22:09:00.560 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.405+0000 7fd089f84700 1 -- 172.21.15.104:0/731521286 learned_addr learned my addr 172.21.15.104:0/731521286 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:00.560 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.405+0000 7fd089f84700 1 -- 172.21.15.104:0/731521286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd08419c480 con 0x7fd084106e40 2024-06-09T22:09:00.560 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.405+0000 7fd089f84700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 0x7fd08419bf40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd080011b60 tx=0x7fd080011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:00.560 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.405+0000 7fd07affd700 1 -- 172.21.15.104:0/731521286 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd080003970 con 0x7fd084106e40 2024-06-09T22:09:00.561 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.406+0000 7fd07affd700 1 -- 172.21.15.104:0/731521286 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd080003b10 con 0x7fd084106e40 2024-06-09T22:09:00.561 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.406+0000 7fd08c1e8700 1 -- 172.21.15.104:0/731521286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd08419c710 con 0x7fd084106e40 2024-06-09T22:09:00.561 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.406+0000 7fd07affd700 1 -- 172.21.15.104:0/731521286 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd080024610 con 0x7fd084106e40 2024-06-09T22:09:00.561 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.406+0000 7fd08c1e8700 1 -- 172.21.15.104:0/731521286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd08419f400 con 0x7fd084106e40 2024-06-09T22:09:00.561 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.406+0000 7fd07affd700 1 -- 172.21.15.104:0/731521286 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fd080024910 con 0x7fd084106e40 2024-06-09T22:09:00.561 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.407+0000 7fd07affd700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 0x7fd07003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:00.561 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.407+0000 7fd07affd700 1 -- 172.21.15.104:0/731521286 --> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd07003bc20 con 0x7fd070039050 2024-06-09T22:09:00.561 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.407+0000 7fd089783700 1 -- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 msgr2=0x7fd07003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1879478526 2024-06-09T22:09:00.561 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.407+0000 7fd07affd700 1 -- 172.21.15.104:0/731521286 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd080011e00 con 0x7fd084106e40 2024-06-09T22:09:00.561 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.407+0000 7fd089783700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 0x7fd07003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:09:00.561 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.608+0000 7fd089783700 1 -- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 msgr2=0x7fd07003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1879478526 2024-06-09T22:09:00.562 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:54.608+0000 7fd089783700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 0x7fd07003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:09:00.562 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:55.009+0000 7fd089783700 1 -- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 msgr2=0x7fd07003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1879478526 2024-06-09T22:09:00.562 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:55.009+0000 7fd089783700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 0x7fd07003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-09T22:09:00.562 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:55.810+0000 7fd089783700 1 -- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 msgr2=0x7fd07003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1879478526 2024-06-09T22:09:00.562 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:55.810+0000 7fd089783700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 0x7fd07003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-09T22:09:00.562 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:57.411+0000 7fd089783700 1 -- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 msgr2=0x7fd07003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1879478526 2024-06-09T22:09:00.562 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:57.411+0000 7fd089783700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 0x7fd07003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-06-09T22:09:00.562 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:58.970+0000 7fd07affd700 1 -- 172.21.15.104:0/731521286 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7fd080024c00 con 0x7fd084106e40 2024-06-09T22:09:00.562 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:58.970+0000 7fd07affd700 1 -- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 msgr2=0x7fd07003b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-09T22:09:00.562 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:58.970+0000 7fd07affd700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 0x7fd07003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:00.562 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.971+0000 7fd07affd700 1 -- 172.21.15.104:0/731521286 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fd08004eeb0 con 0x7fd084106e40 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.971+0000 7fd07affd700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fd07003c980 0x7fd07003ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.971+0000 7fd07affd700 1 -- 172.21.15.104:0/731521286 --> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd08000af30 con 0x7fd07003c980 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.972+0000 7fd089783700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fd07003c980 0x7fd07003ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.972+0000 7fd089783700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fd07003c980 0x7fd07003ed70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd074003a10 tx=0x7fd074009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.973+0000 7fd07affd700 1 -- 172.21.15.104:0/731521286 <== mgr.14120 v2:172.21.15.104:6800/85752866 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fd08000af30 con 0x7fd07003c980 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.979+0000 7fd08c1e8700 1 -- 172.21.15.104:0/731521286 --> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd0840695d0 con 0x7fd07003c980 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.979+0000 7fd07affd700 1 -- 172.21.15.104:0/731521286 <== mgr.14120 v2:172.21.15.104:6800/85752866 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fd0840695d0 con 0x7fd07003c980 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.980+0000 7fd08c1e8700 1 -- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fd07003c980 msgr2=0x7fd07003ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.980+0000 7fd08c1e8700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fd07003c980 0x7fd07003ed70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd074003a10 tx=0x7fd074009290 comp rx=0 tx=0).stop 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.980+0000 7fd08c1e8700 1 -- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 msgr2=0x7fd08419bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.980+0000 7fd08c1e8700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 0x7fd08419bf40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd080011b60 tx=0x7fd080011c40 comp rx=0 tx=0).stop 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.980+0000 7fd08c1e8700 1 -- 172.21.15.104:0/731521286 shutdown_connections 2024-06-09T22:09:00.563 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.980+0000 7fd08c1e8700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fd07003c980 0x7fd07003ed70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:00.564 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.980+0000 7fd08c1e8700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd084106e40 0x7fd08419bf40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:00.564 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.980+0000 7fd08c1e8700 1 --2- 172.21.15.104:0/731521286 >> [v2:172.21.15.104:6800/1879478526,v1:172.21.15.104:6801/1879478526] conn(0x7fd070039050 0x7fd07003b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:00.564 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.980+0000 7fd08c1e8700 1 -- 172.21.15.104:0/731521286 >> 172.21.15.104:0/731521286 conn(0x7fd084102400 msgr2=0x7fd0841032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:00.564 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.980+0000 7fd08c1e8700 1 -- 172.21.15.104:0/731521286 shutdown_connections 2024-06-09T22:09:00.564 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:08:59.980+0000 7fd08c1e8700 1 -- 172.21.15.104:0/731521286 wait complete. 2024-06-09T22:09:00.564 INFO:teuthology.orchestra.run.smithi104.stdout:mgr epoch 5 is available 2024-06-09T22:09:00.564 INFO:teuthology.orchestra.run.smithi104.stdout:Setting orchestrator backend to cephadm... 2024-06-09T22:09:01.169 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:01 smithi104 ceph-mon[32389]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-09T22:09:01.169 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:01 smithi104 ceph-mon[32389]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-09T22:09:01.169 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:01 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:09:01.686 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.957+0000 7fb757db9700 1 Processor -- start 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.957+0000 7fb757db9700 1 -- start start 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.958+0000 7fb757db9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 0x7fb750109940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.958+0000 7fb757db9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb750109f10 con 0x7fb750109520 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.958+0000 7fb755b55700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 0x7fb750109940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.958+0000 7fb755b55700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 0x7fb750109940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41880/0 (socket says 172.21.15.104:41880) 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.958+0000 7fb755b55700 1 -- 172.21.15.104:0/1132148575 learned_addr learned my addr 172.21.15.104:0/1132148575 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.958+0000 7fb755b55700 1 -- 172.21.15.104:0/1132148575 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb75010a090 con 0x7fb750109520 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.958+0000 7fb755b55700 1 --2- 172.21.15.104:0/1132148575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 0x7fb750109940 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb7440098a0 tx=0x7fb744009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7927e3ce6f5744e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.958+0000 7fb754b53700 1 -- 172.21.15.104:0/1132148575 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb744004160 con 0x7fb750109520 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.958+0000 7fb754b53700 1 -- 172.21.15.104:0/1132148575 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb744004300 con 0x7fb750109520 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.958+0000 7fb754b53700 1 -- 172.21.15.104:0/1132148575 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb744011b50 con 0x7fb750109520 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.959+0000 7fb757db9700 1 -- 172.21.15.104:0/1132148575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 msgr2=0x7fb750109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:01.687 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.959+0000 7fb757db9700 1 --2- 172.21.15.104:0/1132148575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 0x7fb750109940 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb7440098a0 tx=0x7fb744009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.959+0000 7fb757db9700 1 -- 172.21.15.104:0/1132148575 shutdown_connections 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.959+0000 7fb757db9700 1 --2- 172.21.15.104:0/1132148575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 0x7fb750109940 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.959+0000 7fb757db9700 1 -- 172.21.15.104:0/1132148575 >> 172.21.15.104:0/1132148575 conn(0x7fb750101b40 msgr2=0x7fb750103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.959+0000 7fb757db9700 1 -- 172.21.15.104:0/1132148575 shutdown_connections 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.959+0000 7fb757db9700 1 -- 172.21.15.104:0/1132148575 wait complete. 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.960+0000 7fb757db9700 1 Processor -- start 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.960+0000 7fb757db9700 1 -- start start 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.960+0000 7fb757db9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 0x7fb75019bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.960+0000 7fb757db9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb750106d80 con 0x7fb750109520 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.960+0000 7fb755b55700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 0x7fb75019bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.960+0000 7fb755b55700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 0x7fb75019bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41886/0 (socket says 172.21.15.104:41886) 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.960+0000 7fb755b55700 1 -- 172.21.15.104:0/2102467852 learned_addr learned my addr 172.21.15.104:0/2102467852 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.960+0000 7fb755b55700 1 -- 172.21.15.104:0/2102467852 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb75019c420 con 0x7fb750109520 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.960+0000 7fb755b55700 1 --2- 172.21.15.104:0/2102467852 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 0x7fb75019bee0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb744011670 tx=0x7fb744011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.961+0000 7fb742ffd700 1 -- 172.21.15.104:0/2102467852 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7440039c0 con 0x7fb750109520 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.961+0000 7fb742ffd700 1 -- 172.21.15.104:0/2102467852 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb744011e50 con 0x7fb750109520 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.961+0000 7fb757db9700 1 -- 172.21.15.104:0/2102467852 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb75019c710 con 0x7fb750109520 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.961+0000 7fb742ffd700 1 -- 172.21.15.104:0/2102467852 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb74401a5c0 con 0x7fb750109520 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.961+0000 7fb757db9700 1 -- 172.21.15.104:0/2102467852 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb75019f3a0 con 0x7fb750109520 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.961+0000 7fb742ffd700 1 -- 172.21.15.104:0/2102467852 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fb74401a8c0 con 0x7fb750109520 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.961+0000 7fb757db9700 1 -- 172.21.15.104:0/2102467852 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb750063af0 con 0x7fb750109520 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.962+0000 7fb742ffd700 1 --2- 172.21.15.104:0/2102467852 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb73c038ef0 0x7fb73c03b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.962+0000 7fb742ffd700 1 -- 172.21.15.104:0/2102467852 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb74404d3e0 con 0x7fb750109520 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.962+0000 7fb755354700 1 --2- 172.21.15.104:0/2102467852 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb73c038ef0 0x7fb73c03b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.962+0000 7fb755354700 1 --2- 172.21.15.104:0/2102467852 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb73c038ef0 0x7fb73c03b3b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb74c006fd0 tx=0x7fb74c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:00.967+0000 7fb742ffd700 1 -- 172.21.15.104:0/2102467852 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb744019c70 con 0x7fb750109520 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.022+0000 7fb742ffd700 1 -- 172.21.15.104:0/2102467852 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fb74402d910 con 0x7fb750109520 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.183+0000 7fb757db9700 1 -- 172.21.15.104:0/2102467852 --> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fb750102e30 con 0x7fb73c038ef0 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.187+0000 7fb742ffd700 1 -- 172.21.15.104:0/2102467852 <== mgr.14120 v2:172.21.15.104:6800/85752866 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb750102e30 con 0x7fb73c038ef0 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.189+0000 7fb757db9700 1 -- 172.21.15.104:0/2102467852 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb73c038ef0 msgr2=0x7fb73c03b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.189+0000 7fb757db9700 1 --2- 172.21.15.104:0/2102467852 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb73c038ef0 0x7fb73c03b3b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb74c006fd0 tx=0x7fb74c006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.189+0000 7fb757db9700 1 -- 172.21.15.104:0/2102467852 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 msgr2=0x7fb75019bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.189+0000 7fb757db9700 1 --2- 172.21.15.104:0/2102467852 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 0x7fb75019bee0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb744011670 tx=0x7fb744011b60 comp rx=0 tx=0).stop 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.190+0000 7fb757db9700 1 -- 172.21.15.104:0/2102467852 shutdown_connections 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.190+0000 7fb757db9700 1 --2- 172.21.15.104:0/2102467852 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb73c038ef0 0x7fb73c03b3b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.190+0000 7fb757db9700 1 --2- 172.21.15.104:0/2102467852 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb750109520 0x7fb75019bee0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.190+0000 7fb757db9700 1 -- 172.21.15.104:0/2102467852 >> 172.21.15.104:0/2102467852 conn(0x7fb750101b40 msgr2=0x7fb750102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.190+0000 7fb757db9700 1 -- 172.21.15.104:0/2102467852 shutdown_connections 2024-06-09T22:09:01.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:01.190+0000 7fb757db9700 1 -- 172.21.15.104:0/2102467852 wait complete. 2024-06-09T22:09:02.057 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:02 smithi104 ceph-mon[32389]: [09/Jun/2024:22:09:00] ENGINE Bus STARTING 2024-06-09T22:09:02.057 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:02 smithi104 ceph-mon[32389]: [09/Jun/2024:22:09:00] ENGINE Serving on https://172.21.15.104:7150 2024-06-09T22:09:02.057 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:02 smithi104 ceph-mon[32389]: [09/Jun/2024:22:09:00] ENGINE Serving on http://172.21.15.104:8765 2024-06-09T22:09:02.057 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:02 smithi104 ceph-mon[32389]: [09/Jun/2024:22:09:00] ENGINE Bus STARTED 2024-06-09T22:09:02.058 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:02 smithi104 ceph-mon[32389]: mgrmap e8: x(active, since 2s) 2024-06-09T22:09:02.058 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:02 smithi104 ceph-mon[32389]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:09:02.058 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:02 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' 2024-06-09T22:09:02.058 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:02 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:09:02.818 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-09T22:09:02.818 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.096+0000 7fa636842700 1 Processor -- start 2024-06-09T22:09:02.818 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.096+0000 7fa636842700 1 -- start start 2024-06-09T22:09:02.818 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.096+0000 7fa636842700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 0x7fa63010ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:02.818 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.096+0000 7fa636842700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa63007c720 con 0x7fa6301089b0 2024-06-09T22:09:02.818 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.096+0000 7fa62ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 0x7fa63010ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:02.818 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.097+0000 7fa62ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 0x7fa63010ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41894/0 (socket says 172.21.15.104:41894) 2024-06-09T22:09:02.818 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.097+0000 7fa62ffff700 1 -- 172.21.15.104:0/3411452143 learned_addr learned my addr 172.21.15.104:0/3411452143 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:02.818 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.097+0000 7fa62ffff700 1 -- 172.21.15.104:0/3411452143 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63010b2e0 con 0x7fa6301089b0 2024-06-09T22:09:02.818 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.097+0000 7fa62ffff700 1 --2- 172.21.15.104:0/3411452143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 0x7fa63010ada0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa6200098a0 tx=0x7fa620009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110e3212b1ba43fe server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:02.818 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.097+0000 7fa62effd700 1 -- 172.21.15.104:0/3411452143 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa620004160 con 0x7fa6301089b0 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.097+0000 7fa62effd700 1 -- 172.21.15.104:0/3411452143 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa620004300 con 0x7fa6301089b0 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.097+0000 7fa62effd700 1 -- 172.21.15.104:0/3411452143 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa620011b50 con 0x7fa6301089b0 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.098+0000 7fa636842700 1 -- 172.21.15.104:0/3411452143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 msgr2=0x7fa63010ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.098+0000 7fa636842700 1 --2- 172.21.15.104:0/3411452143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 0x7fa63010ada0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa6200098a0 tx=0x7fa620009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.098+0000 7fa636842700 1 -- 172.21.15.104:0/3411452143 shutdown_connections 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.098+0000 7fa636842700 1 --2- 172.21.15.104:0/3411452143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 0x7fa63010ada0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.098+0000 7fa636842700 1 -- 172.21.15.104:0/3411452143 >> 172.21.15.104:0/3411452143 conn(0x7fa630102400 msgr2=0x7fa630104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.098+0000 7fa636842700 1 -- 172.21.15.104:0/3411452143 shutdown_connections 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.098+0000 7fa636842700 1 -- 172.21.15.104:0/3411452143 wait complete. 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.098+0000 7fa636842700 1 Processor -- start 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.098+0000 7fa636842700 1 -- start start 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.099+0000 7fa636842700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 0x7fa63019bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.099+0000 7fa636842700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa63010c310 con 0x7fa6301089b0 2024-06-09T22:09:02.819 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.099+0000 7fa62ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 0x7fa63019bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.099+0000 7fa62ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 0x7fa63019bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41898/0 (socket says 172.21.15.104:41898) 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.099+0000 7fa62ffff700 1 -- 172.21.15.104:0/709467903 learned_addr learned my addr 172.21.15.104:0/709467903 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.099+0000 7fa62ffff700 1 -- 172.21.15.104:0/709467903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63019c480 con 0x7fa6301089b0 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.099+0000 7fa62ffff700 1 --2- 172.21.15.104:0/709467903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 0x7fa63019bf40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa620009fb0 tx=0x7fa620011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.099+0000 7fa62d7fa700 1 -- 172.21.15.104:0/709467903 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6200038b0 con 0x7fa6301089b0 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.099+0000 7fa62d7fa700 1 -- 172.21.15.104:0/709467903 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa620011e50 con 0x7fa6301089b0 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.099+0000 7fa636842700 1 -- 172.21.15.104:0/709467903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63019c710 con 0x7fa6301089b0 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.099+0000 7fa62d7fa700 1 -- 172.21.15.104:0/709467903 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa620019420 con 0x7fa6301089b0 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.099+0000 7fa636842700 1 -- 172.21.15.104:0/709467903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa63019f400 con 0x7fa6301089b0 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.100+0000 7fa62d7fa700 1 -- 172.21.15.104:0/709467903 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fa620019720 con 0x7fa6301089b0 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.100+0000 7fa636842700 1 -- 172.21.15.104:0/709467903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6300695d0 con 0x7fa6301089b0 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.100+0000 7fa62d7fa700 1 --2- 172.21.15.104:0/709467903 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa618039000 0x7fa61803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.100+0000 7fa62d7fa700 1 -- 172.21.15.104:0/709467903 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa62004d330 con 0x7fa6301089b0 2024-06-09T22:09:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.101+0000 7fa62f7fe700 1 --2- 172.21.15.104:0/709467903 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa618039000 0x7fa61803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.101+0000 7fa62f7fe700 1 --2- 172.21.15.104:0/709467903 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa618039000 0x7fa61803b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa624006fd0 tx=0x7fa624006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.106+0000 7fa62d7fa700 1 -- 172.21.15.104:0/709467903 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa620020240 con 0x7fa6301089b0 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.303+0000 7fa636842700 1 -- 172.21.15.104:0/709467903 --> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fa630103670 con 0x7fa618039000 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.304+0000 7fa62d7fa700 1 -- 172.21.15.104:0/709467903 <== mgr.14120 v2:172.21.15.104:6800/85752866 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fa630103670 con 0x7fa618039000 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.306+0000 7fa636842700 1 -- 172.21.15.104:0/709467903 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa618039000 msgr2=0x7fa61803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.306+0000 7fa636842700 1 --2- 172.21.15.104:0/709467903 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa618039000 0x7fa61803b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa624006fd0 tx=0x7fa624006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.306+0000 7fa636842700 1 -- 172.21.15.104:0/709467903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 msgr2=0x7fa63019bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.306+0000 7fa636842700 1 --2- 172.21.15.104:0/709467903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 0x7fa63019bf40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa620009fb0 tx=0x7fa620011bf0 comp rx=0 tx=0).stop 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.306+0000 7fa636842700 1 -- 172.21.15.104:0/709467903 shutdown_connections 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.306+0000 7fa636842700 1 --2- 172.21.15.104:0/709467903 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa618039000 0x7fa61803b4c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.306+0000 7fa636842700 1 --2- 172.21.15.104:0/709467903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6301089b0 0x7fa63019bf40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.306+0000 7fa636842700 1 -- 172.21.15.104:0/709467903 >> 172.21.15.104:0/709467903 conn(0x7fa630102400 msgr2=0x7fa630102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.306+0000 7fa636842700 1 -- 172.21.15.104:0/709467903 shutdown_connections 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:02.307+0000 7fa636842700 1 -- 172.21.15.104:0/709467903 wait complete. 2024-06-09T22:09:02.821 INFO:teuthology.orchestra.run.smithi104.stdout:Generating ssh key... 2024-06-09T22:09:03.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-mon[32389]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:09:03.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: Generating public/private rsa key pair. 2024-06-09T22:09:03.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: Your identification has been saved in /tmp/tmpakhhxhbl/key. 2024-06-09T22:09:03.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: Your public key has been saved in /tmp/tmpakhhxhbl/key.pub. 2024-06-09T22:09:03.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: The key fingerprint is: 2024-06-09T22:09:03.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: SHA256:Bd98r4GB83xtyFnlUFpzEHAieeUH+4MUDblq2Z6W/UY ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:09:03.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: The key's randomart image is: 2024-06-09T22:09:03.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: +---[RSA 3072]----+ 2024-06-09T22:09:03.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: | . ..o=@==| 2024-06-09T22:09:03.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: | o.=.=.O+| 2024-06-09T22:09:03.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: | =.+.* +| 2024-06-09T22:09:03.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: | . +.*.O | 2024-06-09T22:09:03.992 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: | S B.*.=| 2024-06-09T22:09:03.993 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: | + o +E| 2024-06-09T22:09:03.993 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: | . . =. | 2024-06-09T22:09:03.993 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: | = ..| 2024-06-09T22:09:03.993 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: | . .o| 2024-06-09T22:09:03.993 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: +----[SHA256]-----+ 2024-06-09T22:09:04.315 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.210+0000 7fc5880d6700 1 Processor -- start 2024-06-09T22:09:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.210+0000 7fc5880d6700 1 -- start start 2024-06-09T22:09:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.210+0000 7fc5880d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 0x7fc580107290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.210+0000 7fc5880d6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801077d0 con 0x7fc580106e70 2024-06-09T22:09:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.210+0000 7fc585e72700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 0x7fc580107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.210+0000 7fc585e72700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 0x7fc580107290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41912/0 (socket says 172.21.15.104:41912) 2024-06-09T22:09:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.210+0000 7fc585e72700 1 -- 172.21.15.104:0/1862192557 learned_addr learned my addr 172.21.15.104:0/1862192557 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.211+0000 7fc585e72700 1 -- 172.21.15.104:0/1862192557 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc580107f80 con 0x7fc580106e70 2024-06-09T22:09:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.211+0000 7fc585e72700 1 --2- 172.21.15.104:0/1862192557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 0x7fc580107290 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc57c0098a0 tx=0x7fc57c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78c7f8893e31422 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.211+0000 7fc584e70700 1 -- 172.21.15.104:0/1862192557 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc57c004160 con 0x7fc580106e70 2024-06-09T22:09:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.211+0000 7fc584e70700 1 -- 172.21.15.104:0/1862192557 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc57c004300 con 0x7fc580106e70 2024-06-09T22:09:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.211+0000 7fc584e70700 1 -- 172.21.15.104:0/1862192557 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc57c011b50 con 0x7fc580106e70 2024-06-09T22:09:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.212+0000 7fc5880d6700 1 -- 172.21.15.104:0/1862192557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 msgr2=0x7fc580107290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.212+0000 7fc5880d6700 1 --2- 172.21.15.104:0/1862192557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 0x7fc580107290 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc57c0098a0 tx=0x7fc57c009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.212+0000 7fc5880d6700 1 -- 172.21.15.104:0/1862192557 shutdown_connections 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.212+0000 7fc5880d6700 1 --2- 172.21.15.104:0/1862192557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 0x7fc580107290 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.212+0000 7fc5880d6700 1 -- 172.21.15.104:0/1862192557 >> 172.21.15.104:0/1862192557 conn(0x7fc580102410 msgr2=0x7fc580104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.212+0000 7fc5880d6700 1 -- 172.21.15.104:0/1862192557 shutdown_connections 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.212+0000 7fc5880d6700 1 -- 172.21.15.104:0/1862192557 wait complete. 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.212+0000 7fc5880d6700 1 Processor -- start 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.212+0000 7fc5880d6700 1 -- start start 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.212+0000 7fc5880d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 0x7fc58019ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.212+0000 7fc5880d6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc58010b3c0 con 0x7fc580106e70 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.213+0000 7fc585e72700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 0x7fc58019ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.213+0000 7fc585e72700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 0x7fc58019ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41914/0 (socket says 172.21.15.104:41914) 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.213+0000 7fc585e72700 1 -- 172.21.15.104:0/1095146094 learned_addr learned my addr 172.21.15.104:0/1095146094 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:04.317 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.213+0000 7fc585e72700 1 -- 172.21.15.104:0/1095146094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58019b390 con 0x7fc580106e70 2024-06-09T22:09:04.318 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.213+0000 7fc585e72700 1 --2- 172.21.15.104:0/1095146094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 0x7fc58019ae50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc57c011b60 tx=0x7fc57c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:04.318 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.213+0000 7fc576ffd700 1 -- 172.21.15.104:0/1095146094 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc57c003b20 con 0x7fc580106e70 2024-06-09T22:09:04.318 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.213+0000 7fc576ffd700 1 -- 172.21.15.104:0/1095146094 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc57c003cc0 con 0x7fc580106e70 2024-06-09T22:09:04.318 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.213+0000 7fc5880d6700 1 -- 172.21.15.104:0/1095146094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc58019b620 con 0x7fc580106e70 2024-06-09T22:09:04.318 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.213+0000 7fc576ffd700 1 -- 172.21.15.104:0/1095146094 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc57c024840 con 0x7fc580106e70 2024-06-09T22:09:04.318 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.213+0000 7fc5880d6700 1 -- 172.21.15.104:0/1095146094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc58019e310 con 0x7fc580106e70 2024-06-09T22:09:04.318 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.214+0000 7fc576ffd700 1 -- 172.21.15.104:0/1095146094 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fc57c022020 con 0x7fc580106e70 2024-06-09T22:09:04.318 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.214+0000 7fc5880d6700 1 -- 172.21.15.104:0/1095146094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc580063af0 con 0x7fc580106e70 2024-06-09T22:09:04.318 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.214+0000 7fc576ffd700 1 --2- 172.21.15.104:0/1095146094 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fc56c0390a0 0x7fc56c03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:04.318 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.214+0000 7fc576ffd700 1 -- 172.21.15.104:0/1095146094 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc57c011e00 con 0x7fc580106e70 2024-06-09T22:09:04.318 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.214+0000 7fc585671700 1 --2- 172.21.15.104:0/1095146094 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fc56c0390a0 0x7fc56c03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:04.318 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.215+0000 7fc585671700 1 --2- 172.21.15.104:0/1095146094 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fc56c0390a0 0x7fc56c03b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc570006fd0 tx=0x7fc570006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.219+0000 7fc576ffd700 1 -- 172.21.15.104:0/1095146094 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc57c01a7b0 con 0x7fc580106e70 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.415+0000 7fc5880d6700 1 -- 172.21.15.104:0/1095146094 --> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fc580103680 con 0x7fc56c0390a0 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.696+0000 7fc576ffd700 1 -- 172.21.15.104:0/1095146094 <== mgr.14120 v2:172.21.15.104:6800/85752866 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc580103680 con 0x7fc56c0390a0 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.699+0000 7fc5880d6700 1 -- 172.21.15.104:0/1095146094 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fc56c0390a0 msgr2=0x7fc56c03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.699+0000 7fc5880d6700 1 --2- 172.21.15.104:0/1095146094 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fc56c0390a0 0x7fc56c03b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc570006fd0 tx=0x7fc570006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.699+0000 7fc5880d6700 1 -- 172.21.15.104:0/1095146094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 msgr2=0x7fc58019ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.699+0000 7fc5880d6700 1 --2- 172.21.15.104:0/1095146094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 0x7fc58019ae50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc57c011b60 tx=0x7fc57c011c40 comp rx=0 tx=0).stop 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.699+0000 7fc5880d6700 1 -- 172.21.15.104:0/1095146094 shutdown_connections 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.699+0000 7fc5880d6700 1 --2- 172.21.15.104:0/1095146094 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fc56c0390a0 0x7fc56c03b560 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.699+0000 7fc5880d6700 1 --2- 172.21.15.104:0/1095146094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc580106e70 0x7fc58019ae50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.699+0000 7fc5880d6700 1 -- 172.21.15.104:0/1095146094 >> 172.21.15.104:0/1095146094 conn(0x7fc580102410 msgr2=0x7fc580102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.699+0000 7fc5880d6700 1 -- 172.21.15.104:0/1095146094 shutdown_connections 2024-06-09T22:09:04.319 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:03.699+0000 7fc5880d6700 1 -- 172.21.15.104:0/1095146094 wait complete. 2024-06-09T22:09:04.917 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:04 smithi104 ceph-mon[32389]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:09:04.917 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:04 smithi104 ceph-mon[32389]: Generating ssh key... 2024-06-09T22:09:04.917 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:04 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' 2024-06-09T22:09:04.917 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:04 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' 2024-06-09T22:09:05.505 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/VUyIH7cF72dyZ84ZWviMNeJxtiUfX9MOxaJkPOk5H7yzbOIkVWycAP09tklpSNBlPcxQOVKN0cdOLjPsyGqBF9NvY+A2B/snv37M7AMm9E4uLoylwO3EXa1acTx161qDgLqD2IBKZ2fGowA/tPFZv1A/ncIpqZnFIWtyaASgssGRaJKegxdTVZSwRZwm8VhR1VNVJiD6bxQ1gFddNy1wxrmmkoGScgleMOOvp2Jk32+a2hYKy8gvRtIZ1CVsmmeGMyFGG+2dlY0m1LEIIM2mhf/cmzUfHK3n1dFXxC+wGi/5yhYF+O3zz3vHNfC+Axz8m6+9im5g624avZraIG2PvhYoK1gX5BMkgFdNVYEX9OQclzCqqyIzSAJnHmOYwLC//oPiNFYMONLZt1USy1asim+BeECtUvJvybHo3b4Mfz92W4a9jRK7Y/pyu8zNJzSuk+Ld5oy0mJtDPCWc+d3SAJPcHIQvVHOaJ10TEj/DysNBbI1KXOUvG6sFxcZGqfk= ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:09:05.505 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.726+0000 7fb05fa43700 1 Processor -- start 2024-06-09T22:09:05.505 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.726+0000 7fb05fa43700 1 -- start start 2024-06-09T22:09:05.505 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.726+0000 7fb05fa43700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 0x7fb05810ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:05.505 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.726+0000 7fb05fa43700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb05807c690 con 0x7fb058108a30 2024-06-09T22:09:05.505 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.726+0000 7fb05d7df700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 0x7fb05810ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:05.505 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.726+0000 7fb05d7df700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 0x7fb05810ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41920/0 (socket says 172.21.15.104:41920) 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.726+0000 7fb05d7df700 1 -- 172.21.15.104:0/4050304411 learned_addr learned my addr 172.21.15.104:0/4050304411 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.726+0000 7fb05d7df700 1 -- 172.21.15.104:0/4050304411 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05810b360 con 0x7fb058108a30 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.727+0000 7fb05d7df700 1 --2- 172.21.15.104:0/4050304411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 0x7fb05810ae20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb0480098a0 tx=0x7fb048009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44545552e80b2e08 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.727+0000 7fb04ffff700 1 -- 172.21.15.104:0/4050304411 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb048004160 con 0x7fb058108a30 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.727+0000 7fb04ffff700 1 -- 172.21.15.104:0/4050304411 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb048004300 con 0x7fb058108a30 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.727+0000 7fb04ffff700 1 -- 172.21.15.104:0/4050304411 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb048011b50 con 0x7fb058108a30 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.727+0000 7fb05fa43700 1 -- 172.21.15.104:0/4050304411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 msgr2=0x7fb05810ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.727+0000 7fb05fa43700 1 --2- 172.21.15.104:0/4050304411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 0x7fb05810ae20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb0480098a0 tx=0x7fb048009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.727+0000 7fb05fa43700 1 -- 172.21.15.104:0/4050304411 shutdown_connections 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.727+0000 7fb05fa43700 1 --2- 172.21.15.104:0/4050304411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 0x7fb05810ae20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.727+0000 7fb05fa43700 1 -- 172.21.15.104:0/4050304411 >> 172.21.15.104:0/4050304411 conn(0x7fb058102440 msgr2=0x7fb058104860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.727+0000 7fb05fa43700 1 -- 172.21.15.104:0/4050304411 shutdown_connections 2024-06-09T22:09:05.506 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.727+0000 7fb05fa43700 1 -- 172.21.15.104:0/4050304411 wait complete. 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.728+0000 7fb05fa43700 1 Processor -- start 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.728+0000 7fb05fa43700 1 -- start start 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.728+0000 7fb05fa43700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 0x7fb05819beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.728+0000 7fb05fa43700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb05810c320 con 0x7fb058108a30 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.728+0000 7fb05d7df700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 0x7fb05819beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.728+0000 7fb05d7df700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 0x7fb05819beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41926/0 (socket says 172.21.15.104:41926) 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.728+0000 7fb05d7df700 1 -- 172.21.15.104:0/418336942 learned_addr learned my addr 172.21.15.104:0/418336942 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.729+0000 7fb05d7df700 1 -- 172.21.15.104:0/418336942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05819c3f0 con 0x7fb058108a30 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.729+0000 7fb05d7df700 1 --2- 172.21.15.104:0/418336942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 0x7fb05819beb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb048011670 tx=0x7fb048011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.729+0000 7fb04e7fc700 1 -- 172.21.15.104:0/418336942 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0480039c0 con 0x7fb058108a30 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.729+0000 7fb05fa43700 1 -- 172.21.15.104:0/418336942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05819c680 con 0x7fb058108a30 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.729+0000 7fb04e7fc700 1 -- 172.21.15.104:0/418336942 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb048011e50 con 0x7fb058108a30 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.729+0000 7fb04e7fc700 1 -- 172.21.15.104:0/418336942 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb04801a5c0 con 0x7fb058108a30 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.729+0000 7fb05fa43700 1 -- 172.21.15.104:0/418336942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05819f370 con 0x7fb058108a30 2024-06-09T22:09:05.507 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.730+0000 7fb04e7fc700 1 -- 172.21.15.104:0/418336942 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fb048024430 con 0x7fb058108a30 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.730+0000 7fb04e7fc700 1 --2- 172.21.15.104:0/418336942 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb044039000 0x7fb04403b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.730+0000 7fb04e7fc700 1 -- 172.21.15.104:0/418336942 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb04804d520 con 0x7fb058108a30 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.730+0000 7fb05fa43700 1 -- 172.21.15.104:0/418336942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb058063af0 con 0x7fb058108a30 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.730+0000 7fb05cfde700 1 --2- 172.21.15.104:0/418336942 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb044039000 0x7fb04403b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.730+0000 7fb05cfde700 1 --2- 172.21.15.104:0/418336942 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb044039000 0x7fb04403b4c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb054006fd0 tx=0x7fb054006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.735+0000 7fb04e7fc700 1 -- 172.21.15.104:0/418336942 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb04802d910 con 0x7fb058108a30 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.931+0000 7fb05fa43700 1 -- 172.21.15.104:0/418336942 --> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fb0581036b0 con 0x7fb044039000 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.931+0000 7fb04e7fc700 1 -- 172.21.15.104:0/418336942 <== mgr.14120 v2:172.21.15.104:6800/85752866 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fb0581036b0 con 0x7fb044039000 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.933+0000 7fb05fa43700 1 -- 172.21.15.104:0/418336942 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb044039000 msgr2=0x7fb04403b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.933+0000 7fb05fa43700 1 --2- 172.21.15.104:0/418336942 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb044039000 0x7fb04403b4c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb054006fd0 tx=0x7fb054006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.933+0000 7fb05fa43700 1 -- 172.21.15.104:0/418336942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 msgr2=0x7fb05819beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.933+0000 7fb05fa43700 1 --2- 172.21.15.104:0/418336942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 0x7fb05819beb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb048011670 tx=0x7fb048011b60 comp rx=0 tx=0).stop 2024-06-09T22:09:05.508 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.933+0000 7fb05fa43700 1 -- 172.21.15.104:0/418336942 shutdown_connections 2024-06-09T22:09:05.509 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.933+0000 7fb05fa43700 1 --2- 172.21.15.104:0/418336942 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb044039000 0x7fb04403b4c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:05.509 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.933+0000 7fb05fa43700 1 --2- 172.21.15.104:0/418336942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb058108a30 0x7fb05819beb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:05.509 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.934+0000 7fb05fa43700 1 -- 172.21.15.104:0/418336942 >> 172.21.15.104:0/418336942 conn(0x7fb058102440 msgr2=0x7fb058102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:05.509 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.934+0000 7fb05fa43700 1 -- 172.21.15.104:0/418336942 shutdown_connections 2024-06-09T22:09:05.509 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:04.934+0000 7fb05fa43700 1 -- 172.21.15.104:0/418336942 wait complete. 2024-06-09T22:09:05.509 INFO:teuthology.orchestra.run.smithi104.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-09T22:09:05.509 INFO:teuthology.orchestra.run.smithi104.stdout:Adding key to root@localhost authorized_keys... 2024-06-09T22:09:05.509 INFO:teuthology.orchestra.run.smithi104.stdout:Adding host smithi104... 2024-06-09T22:09:05.993 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:05 smithi104 ceph-mon[32389]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:09:06.726 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:06 smithi104 ceph-mon[32389]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "addr": "172.21.15.104", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:09:07.993 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:07 smithi104 ceph-mon[32389]: Deploying cephadm binary to smithi104 2024-06-09T22:09:10.697 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout Added host 'smithi104' with addr '172.21.15.104' 2024-06-09T22:09:10.697 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.926+0000 7fa6750a4700 1 Processor -- start 2024-06-09T22:09:10.697 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.927+0000 7fa6750a4700 1 -- start start 2024-06-09T22:09:10.697 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.927+0000 7fa6750a4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 0x7fa67010ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:10.697 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.927+0000 7fa6750a4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa67007c670 con 0x7fa670108a40 2024-06-09T22:09:10.697 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.927+0000 7fa66ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 0x7fa67010ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:10.697 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.927+0000 7fa66ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 0x7fa67010ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41934/0 (socket says 172.21.15.104:41934) 2024-06-09T22:09:10.697 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.927+0000 7fa66ed9d700 1 -- 172.21.15.104:0/1191925648 learned_addr learned my addr 172.21.15.104:0/1191925648 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:10.697 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.927+0000 7fa66ed9d700 1 -- 172.21.15.104:0/1191925648 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67010b370 con 0x7fa670108a40 2024-06-09T22:09:10.697 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.927+0000 7fa66ed9d700 1 --2- 172.21.15.104:0/1191925648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 0x7fa67010ae30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa6580098a0 tx=0x7fa658009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62d0b9f35fbc942 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:10.698 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.927+0000 7fa66dd9b700 1 -- 172.21.15.104:0/1191925648 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa658004160 con 0x7fa670108a40 2024-06-09T22:09:10.698 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.927+0000 7fa66dd9b700 1 -- 172.21.15.104:0/1191925648 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa658004300 con 0x7fa670108a40 2024-06-09T22:09:10.698 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.927+0000 7fa66dd9b700 1 -- 172.21.15.104:0/1191925648 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa658011b50 con 0x7fa670108a40 2024-06-09T22:09:10.698 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.928+0000 7fa6750a4700 1 -- 172.21.15.104:0/1191925648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 msgr2=0x7fa67010ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:10.698 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.928+0000 7fa6750a4700 1 --2- 172.21.15.104:0/1191925648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 0x7fa67010ae30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa6580098a0 tx=0x7fa658009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:10.698 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.928+0000 7fa6750a4700 1 -- 172.21.15.104:0/1191925648 shutdown_connections 2024-06-09T22:09:10.698 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.928+0000 7fa6750a4700 1 --2- 172.21.15.104:0/1191925648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 0x7fa67010ae30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:10.698 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.928+0000 7fa6750a4700 1 -- 172.21.15.104:0/1191925648 >> 172.21.15.104:0/1191925648 conn(0x7fa670102410 msgr2=0x7fa670104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:10.698 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.928+0000 7fa6750a4700 1 -- 172.21.15.104:0/1191925648 shutdown_connections 2024-06-09T22:09:10.698 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.928+0000 7fa6750a4700 1 -- 172.21.15.104:0/1191925648 wait complete. 2024-06-09T22:09:10.698 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.929+0000 7fa6750a4700 1 Processor -- start 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.929+0000 7fa6750a4700 1 -- start start 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.929+0000 7fa6750a4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 0x7fa6701082c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.929+0000 7fa6750a4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa67010c330 con 0x7fa670108a40 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.929+0000 7fa66ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 0x7fa6701082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.929+0000 7fa66ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 0x7fa6701082c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41940/0 (socket says 172.21.15.104:41940) 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.929+0000 7fa66ed9d700 1 -- 172.21.15.104:0/2871293814 learned_addr learned my addr 172.21.15.104:0/2871293814 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.929+0000 7fa66ed9d700 1 -- 172.21.15.104:0/2871293814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa670108800 con 0x7fa670108a40 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.930+0000 7fa66ed9d700 1 --2- 172.21.15.104:0/2871293814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 0x7fa6701082c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa658011670 tx=0x7fa658011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.930+0000 7fa667fff700 1 -- 172.21.15.104:0/2871293814 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6580039c0 con 0x7fa670108a40 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.930+0000 7fa667fff700 1 -- 172.21.15.104:0/2871293814 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa658011e50 con 0x7fa670108a40 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.930+0000 7fa6750a4700 1 -- 172.21.15.104:0/2871293814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6701069f0 con 0x7fa670108a40 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.930+0000 7fa667fff700 1 -- 172.21.15.104:0/2871293814 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa65801a5c0 con 0x7fa670108a40 2024-06-09T22:09:10.699 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.930+0000 7fa6750a4700 1 -- 172.21.15.104:0/2871293814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa670106f30 con 0x7fa670108a40 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.931+0000 7fa667fff700 1 -- 172.21.15.104:0/2871293814 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fa65801a8c0 con 0x7fa670108a40 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.931+0000 7fa6750a4700 1 -- 172.21.15.104:0/2871293814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa670063af0 con 0x7fa670108a40 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.931+0000 7fa667fff700 1 --2- 172.21.15.104:0/2871293814 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa65c039000 0x7fa65c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.931+0000 7fa667fff700 1 -- 172.21.15.104:0/2871293814 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa65804d460 con 0x7fa670108a40 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.931+0000 7fa66e59c700 1 --2- 172.21.15.104:0/2871293814 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa65c039000 0x7fa65c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.931+0000 7fa66e59c700 1 --2- 172.21.15.104:0/2871293814 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa65c039000 0x7fa65c03b4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa660006fd0 tx=0x7fa660006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:05.936+0000 7fa667fff700 1 -- 172.21.15.104:0/2871293814 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa658019c70 con 0x7fa670108a40 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:06.137+0000 7fa6750a4700 1 -- 172.21.15.104:0/2871293814 --> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi104", "addr": "172.21.15.104", "target": ["mon-mgr", ""]}) -- 0x7fa6701036d0 con 0x7fa65c039000 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:10.206+0000 7fa667fff700 1 -- 172.21.15.104:0/2871293814 <== mgr.14120 v2:172.21.15.104:6800/85752866 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fa6701036d0 con 0x7fa65c039000 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:10.208+0000 7fa6750a4700 1 -- 172.21.15.104:0/2871293814 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa65c039000 msgr2=0x7fa65c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:10.208+0000 7fa6750a4700 1 --2- 172.21.15.104:0/2871293814 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa65c039000 0x7fa65c03b4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa660006fd0 tx=0x7fa660006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:10.208+0000 7fa6750a4700 1 -- 172.21.15.104:0/2871293814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 msgr2=0x7fa6701082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:10.208+0000 7fa6750a4700 1 --2- 172.21.15.104:0/2871293814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 0x7fa6701082c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa658011670 tx=0x7fa658011b60 comp rx=0 tx=0).stop 2024-06-09T22:09:10.700 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:10.209+0000 7fa6750a4700 1 -- 172.21.15.104:0/2871293814 shutdown_connections 2024-06-09T22:09:10.701 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:10.209+0000 7fa6750a4700 1 --2- 172.21.15.104:0/2871293814 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa65c039000 0x7fa65c03b4c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:10.701 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:10.209+0000 7fa6750a4700 1 --2- 172.21.15.104:0/2871293814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa670108a40 0x7fa6701082c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:10.701 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:10.209+0000 7fa6750a4700 1 -- 172.21.15.104:0/2871293814 >> 172.21.15.104:0/2871293814 conn(0x7fa670102410 msgr2=0x7fa670102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:10.701 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:10.209+0000 7fa6750a4700 1 -- 172.21.15.104:0/2871293814 shutdown_connections 2024-06-09T22:09:10.701 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:10.209+0000 7fa6750a4700 1 -- 172.21.15.104:0/2871293814 wait complete. 2024-06-09T22:09:10.701 INFO:teuthology.orchestra.run.smithi104.stdout:Deploying unmanaged mon service... 2024-06-09T22:09:11.374 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:11 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' 2024-06-09T22:09:11.374 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:11 smithi104 ceph-mon[32389]: Added host smithi104 2024-06-09T22:09:11.375 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:11 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:09:11.937 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-09T22:09:11.937 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.157+0000 7f4977505700 1 Processor -- start 2024-06-09T22:09:11.937 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.157+0000 7f4977505700 1 -- start start 2024-06-09T22:09:11.937 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.157+0000 7f4977505700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 0x7f497007cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:11.937 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.157+0000 7f4977505700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f497007d0b0 con 0x7f497007e600 2024-06-09T22:09:11.937 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.157+0000 7f49752a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 0x7f497007cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:11.937 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.157+0000 7f49752a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 0x7f497007cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34292/0 (socket says 172.21.15.104:34292) 2024-06-09T22:09:11.937 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.157+0000 7f49752a1700 1 -- 172.21.15.104:0/578298487 learned_addr learned my addr 172.21.15.104:0/578298487 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:11.937 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.158+0000 7f49752a1700 1 -- 172.21.15.104:0/578298487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497007d230 con 0x7f497007e600 2024-06-09T22:09:11.938 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.158+0000 7f49752a1700 1 --2- 172.21.15.104:0/578298487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 0x7f497007cb70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f496c0098a0 tx=0x7f496c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510f4b3d41ed9206 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:11.938 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.158+0000 7f4967fff700 1 -- 172.21.15.104:0/578298487 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f496c004160 con 0x7f497007e600 2024-06-09T22:09:11.938 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.158+0000 7f4967fff700 1 -- 172.21.15.104:0/578298487 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f496c004300 con 0x7f497007e600 2024-06-09T22:09:11.938 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.158+0000 7f4967fff700 1 -- 172.21.15.104:0/578298487 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f496c011b50 con 0x7f497007e600 2024-06-09T22:09:11.938 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.159+0000 7f4977505700 1 -- 172.21.15.104:0/578298487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 msgr2=0x7f497007cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:11.938 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.159+0000 7f4977505700 1 --2- 172.21.15.104:0/578298487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 0x7f497007cb70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f496c0098a0 tx=0x7f496c009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:11.938 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.159+0000 7f4977505700 1 -- 172.21.15.104:0/578298487 shutdown_connections 2024-06-09T22:09:11.938 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.159+0000 7f4977505700 1 --2- 172.21.15.104:0/578298487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 0x7f497007cb70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:11.938 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.159+0000 7f4977505700 1 -- 172.21.15.104:0/578298487 >> 172.21.15.104:0/578298487 conn(0x7f49701035e0 msgr2=0x7f4970105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:11.938 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.159+0000 7f4977505700 1 -- 172.21.15.104:0/578298487 shutdown_connections 2024-06-09T22:09:11.938 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.159+0000 7f4977505700 1 -- 172.21.15.104:0/578298487 wait complete. 2024-06-09T22:09:11.938 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.159+0000 7f4977505700 1 Processor -- start 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.159+0000 7f4977505700 1 -- start start 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.160+0000 7f4977505700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 0x7f497019e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.160+0000 7f4977505700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f497010e490 con 0x7f497007e600 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.160+0000 7f49752a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 0x7f497019e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.160+0000 7f49752a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 0x7f497019e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34296/0 (socket says 172.21.15.104:34296) 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.160+0000 7f49752a1700 1 -- 172.21.15.104:0/2278272492 learned_addr learned my addr 172.21.15.104:0/2278272492 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.160+0000 7f49752a1700 1 -- 172.21.15.104:0/2278272492 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497019e660 con 0x7f497007e600 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.160+0000 7f49752a1700 1 --2- 172.21.15.104:0/2278272492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 0x7f497019e120 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f496c011670 tx=0x7f496c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.160+0000 7f49667fc700 1 -- 172.21.15.104:0/2278272492 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f496c0039c0 con 0x7f497007e600 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.160+0000 7f49667fc700 1 -- 172.21.15.104:0/2278272492 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f496c011e50 con 0x7f497007e600 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.160+0000 7f4977505700 1 -- 172.21.15.104:0/2278272492 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497019e8f0 con 0x7f497007e600 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.160+0000 7f49667fc700 1 -- 172.21.15.104:0/2278272492 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f496c01a5c0 con 0x7f497007e600 2024-06-09T22:09:11.939 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.160+0000 7f4977505700 1 -- 172.21.15.104:0/2278272492 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49701a15e0 con 0x7f497007e600 2024-06-09T22:09:11.940 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.161+0000 7f49667fc700 1 -- 172.21.15.104:0/2278272492 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f496c01a870 con 0x7f497007e600 2024-06-09T22:09:11.940 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.161+0000 7f49667fc700 1 --2- 172.21.15.104:0/2278272492 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f495c039050 0x7f495c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:11.940 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.161+0000 7f4977505700 1 -- 172.21.15.104:0/2278272492 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4970063af0 con 0x7f497007e600 2024-06-09T22:09:11.940 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.161+0000 7f49667fc700 1 -- 172.21.15.104:0/2278272492 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f496c04d340 con 0x7f497007e600 2024-06-09T22:09:11.940 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.161+0000 7f4974aa0700 1 --2- 172.21.15.104:0/2278272492 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f495c039050 0x7f495c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:11.940 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.162+0000 7f4974aa0700 1 --2- 172.21.15.104:0/2278272492 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f495c039050 0x7f495c03b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4960006fd0 tx=0x7f4960006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:11.940 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.166+0000 7f49667fc700 1 -- 172.21.15.104:0/2278272492 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f496c019c70 con 0x7f497007e600 2024-06-09T22:09:11.940 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.388+0000 7f4977505700 1 -- 172.21.15.104:0/2278272492 --> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4970104830 con 0x7f495c039050 2024-06-09T22:09:11.940 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.395+0000 7f49667fc700 1 -- 172.21.15.104:0/2278272492 <== mgr.14120 v2:172.21.15.104:6800/85752866 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4970104830 con 0x7f495c039050 2024-06-09T22:09:11.940 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.398+0000 7f4977505700 1 -- 172.21.15.104:0/2278272492 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f495c039050 msgr2=0x7f495c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:11.940 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.398+0000 7f4977505700 1 --2- 172.21.15.104:0/2278272492 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f495c039050 0x7f495c03b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4960006fd0 tx=0x7f4960006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:11.940 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.398+0000 7f4977505700 1 -- 172.21.15.104:0/2278272492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 msgr2=0x7f497019e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:11.941 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.398+0000 7f4977505700 1 --2- 172.21.15.104:0/2278272492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 0x7f497019e120 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f496c011670 tx=0x7f496c011b60 comp rx=0 tx=0).stop 2024-06-09T22:09:11.941 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.398+0000 7f4977505700 1 -- 172.21.15.104:0/2278272492 shutdown_connections 2024-06-09T22:09:11.941 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.398+0000 7f4977505700 1 --2- 172.21.15.104:0/2278272492 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f495c039050 0x7f495c03b510 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:11.941 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.398+0000 7f4977505700 1 --2- 172.21.15.104:0/2278272492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f497007e600 0x7f497019e120 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:11.941 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.398+0000 7f4977505700 1 -- 172.21.15.104:0/2278272492 >> 172.21.15.104:0/2278272492 conn(0x7f49701035e0 msgr2=0x7f4970104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:11.941 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.398+0000 7f4977505700 1 -- 172.21.15.104:0/2278272492 shutdown_connections 2024-06-09T22:09:11.941 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:11.398+0000 7f4977505700 1 -- 172.21.15.104:0/2278272492 wait complete. 2024-06-09T22:09:11.941 INFO:teuthology.orchestra.run.smithi104.stdout:Deploying unmanaged mgr service... 2024-06-09T22:09:12.589 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:12 smithi104 ceph-mon[32389]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:09:12.589 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:12 smithi104 ceph-mon[32389]: Saving service mon spec with placement count:5 2024-06-09T22:09:12.589 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:12 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' 2024-06-09T22:09:13.151 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-09T22:09:13.151 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.381+0000 7f5d1d7ee700 1 Processor -- start 2024-06-09T22:09:13.151 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.381+0000 7f5d1d7ee700 1 -- start start 2024-06-09T22:09:13.151 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.381+0000 7f5d1d7ee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 0x7f5d1810adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:13.151 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.381+0000 7f5d1d7ee700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1807be50 con 0x7f5d181089d0 2024-06-09T22:09:13.151 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.381+0000 7f5d16ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 0x7f5d1810adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:13.151 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.381+0000 7f5d16ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 0x7f5d1810adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34308/0 (socket says 172.21.15.104:34308) 2024-06-09T22:09:13.151 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.381+0000 7f5d16ffd700 1 -- 172.21.15.104:0/1570171041 learned_addr learned my addr 172.21.15.104:0/1570171041 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.382+0000 7f5d16ffd700 1 -- 172.21.15.104:0/1570171041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1810b300 con 0x7f5d181089d0 2024-06-09T22:09:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.382+0000 7f5d16ffd700 1 --2- 172.21.15.104:0/1570171041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 0x7f5d1810adc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5d080098a0 tx=0x7f5d08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420a9a5e84104528 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.382+0000 7f5d167fc700 1 -- 172.21.15.104:0/1570171041 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d08004160 con 0x7f5d181089d0 2024-06-09T22:09:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.382+0000 7f5d167fc700 1 -- 172.21.15.104:0/1570171041 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5d08004300 con 0x7f5d181089d0 2024-06-09T22:09:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.382+0000 7f5d167fc700 1 -- 172.21.15.104:0/1570171041 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d08011b50 con 0x7f5d181089d0 2024-06-09T22:09:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.383+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/1570171041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 msgr2=0x7f5d1810adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.383+0000 7f5d1d7ee700 1 --2- 172.21.15.104:0/1570171041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 0x7f5d1810adc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5d080098a0 tx=0x7f5d08009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.383+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/1570171041 shutdown_connections 2024-06-09T22:09:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.383+0000 7f5d1d7ee700 1 --2- 172.21.15.104:0/1570171041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 0x7f5d1810adc0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.383+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/1570171041 >> 172.21.15.104:0/1570171041 conn(0x7f5d181023c0 msgr2=0x7f5d18104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.383+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/1570171041 shutdown_connections 2024-06-09T22:09:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.383+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/1570171041 wait complete. 2024-06-09T22:09:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.383+0000 7f5d1d7ee700 1 Processor -- start 2024-06-09T22:09:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d1d7ee700 1 -- start start 2024-06-09T22:09:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d1d7ee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 0x7f5d18108250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d1d7ee700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1810c320 con 0x7f5d181089d0 2024-06-09T22:09:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d16ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 0x7f5d18108250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d16ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 0x7f5d18108250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34316/0 (socket says 172.21.15.104:34316) 2024-06-09T22:09:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d16ffd700 1 -- 172.21.15.104:0/3870804487 learned_addr learned my addr 172.21.15.104:0/3870804487 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d16ffd700 1 -- 172.21.15.104:0/3870804487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d18108790 con 0x7f5d181089d0 2024-06-09T22:09:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d16ffd700 1 --2- 172.21.15.104:0/3870804487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 0x7f5d18108250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5d08011b60 tx=0x7f5d08011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d14ff9700 1 -- 172.21.15.104:0/3870804487 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d08003b20 con 0x7f5d181089d0 2024-06-09T22:09:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d14ff9700 1 -- 172.21.15.104:0/3870804487 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5d08003cc0 con 0x7f5d181089d0 2024-06-09T22:09:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/3870804487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d18106980 con 0x7f5d181089d0 2024-06-09T22:09:13.154 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d14ff9700 1 -- 172.21.15.104:0/3870804487 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d08024840 con 0x7f5d181089d0 2024-06-09T22:09:13.154 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.384+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/3870804487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d18106ec0 con 0x7f5d181089d0 2024-06-09T22:09:13.154 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.385+0000 7f5d14ff9700 1 -- 172.21.15.104:0/3870804487 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f5d08022020 con 0x7f5d181089d0 2024-06-09T22:09:13.154 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.385+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/3870804487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d18063af0 con 0x7f5d181089d0 2024-06-09T22:09:13.154 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.386+0000 7f5d14ff9700 1 --2- 172.21.15.104:0/3870804487 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f5d04039050 0x7f5d0403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:13.154 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.386+0000 7f5d14ff9700 1 -- 172.21.15.104:0/3870804487 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5d08011e00 con 0x7f5d181089d0 2024-06-09T22:09:13.154 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.386+0000 7f5d0e7fc700 1 --2- 172.21.15.104:0/3870804487 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f5d04039050 0x7f5d0403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:13.154 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.386+0000 7f5d0e7fc700 1 --2- 172.21.15.104:0/3870804487 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f5d04039050 0x7f5d0403b510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5d00006fd0 tx=0x7f5d00006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:13.154 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.391+0000 7f5d14ff9700 1 -- 172.21.15.104:0/3870804487 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5d0801a950 con 0x7f5d181089d0 2024-06-09T22:09:13.154 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.602+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/3870804487 --> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f5d181035e0 con 0x7f5d04039050 2024-06-09T22:09:13.154 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.610+0000 7f5d14ff9700 1 -- 172.21.15.104:0/3870804487 <== mgr.14120 v2:172.21.15.104:6800/85752866 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5d181035e0 con 0x7f5d04039050 2024-06-09T22:09:13.154 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.612+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/3870804487 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f5d04039050 msgr2=0x7f5d0403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:13.155 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.612+0000 7f5d1d7ee700 1 --2- 172.21.15.104:0/3870804487 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f5d04039050 0x7f5d0403b510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5d00006fd0 tx=0x7f5d00006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:13.155 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.612+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/3870804487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 msgr2=0x7f5d18108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:13.155 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.612+0000 7f5d1d7ee700 1 --2- 172.21.15.104:0/3870804487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 0x7f5d18108250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5d08011b60 tx=0x7f5d08011c40 comp rx=0 tx=0).stop 2024-06-09T22:09:13.155 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.612+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/3870804487 shutdown_connections 2024-06-09T22:09:13.155 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.612+0000 7f5d1d7ee700 1 --2- 172.21.15.104:0/3870804487 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f5d04039050 0x7f5d0403b510 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:13.155 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.612+0000 7f5d1d7ee700 1 --2- 172.21.15.104:0/3870804487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d181089d0 0x7f5d18108250 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:13.155 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.612+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/3870804487 >> 172.21.15.104:0/3870804487 conn(0x7f5d181023c0 msgr2=0x7f5d18102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:13.155 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.612+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/3870804487 shutdown_connections 2024-06-09T22:09:13.155 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:12.612+0000 7f5d1d7ee700 1 -- 172.21.15.104:0/3870804487 wait complete. 2024-06-09T22:09:13.801 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:13 smithi104 ceph-mon[32389]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:09:13.801 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:13 smithi104 ceph-mon[32389]: Saving service mgr spec with placement count:2 2024-06-09T22:09:13.801 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:13 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' 2024-06-09T22:09:14.448 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.582+0000 7fb0e9da1700 1 Processor -- start 2024-06-09T22:09:14.448 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.582+0000 7fb0e9da1700 1 -- start start 2024-06-09T22:09:14.448 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.582+0000 7fb0e9da1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 0x7fb0e410ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.582+0000 7fb0e9da1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e407be50 con 0x7fb0e4108870 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.583+0000 7fb0e8d9f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 0x7fb0e410ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.583+0000 7fb0e8d9f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 0x7fb0e410ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34326/0 (socket says 172.21.15.104:34326) 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.583+0000 7fb0e8d9f700 1 -- 172.21.15.104:0/3949319974 learned_addr learned my addr 172.21.15.104:0/3949319974 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.583+0000 7fb0e8d9f700 1 -- 172.21.15.104:0/3949319974 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e410b1a0 con 0x7fb0e4108870 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.583+0000 7fb0e8d9f700 1 --2- 172.21.15.104:0/3949319974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 0x7fb0e410ac60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80098a0 tx=0x7fb0d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49917220b19dc44 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.583+0000 7fb0e37fe700 1 -- 172.21.15.104:0/3949319974 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d8004160 con 0x7fb0e4108870 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.583+0000 7fb0e37fe700 1 -- 172.21.15.104:0/3949319974 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb0d8004300 con 0x7fb0e4108870 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.583+0000 7fb0e37fe700 1 -- 172.21.15.104:0/3949319974 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d8011b50 con 0x7fb0e4108870 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.584+0000 7fb0e9da1700 1 -- 172.21.15.104:0/3949319974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 msgr2=0x7fb0e410ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.584+0000 7fb0e9da1700 1 --2- 172.21.15.104:0/3949319974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 0x7fb0e410ac60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80098a0 tx=0x7fb0d8009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.584+0000 7fb0e9da1700 1 -- 172.21.15.104:0/3949319974 shutdown_connections 2024-06-09T22:09:14.449 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.584+0000 7fb0e9da1700 1 --2- 172.21.15.104:0/3949319974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 0x7fb0e410ac60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.584+0000 7fb0e9da1700 1 -- 172.21.15.104:0/3949319974 >> 172.21.15.104:0/3949319974 conn(0x7fb0e41022e0 msgr2=0x7fb0e4104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.584+0000 7fb0e9da1700 1 -- 172.21.15.104:0/3949319974 shutdown_connections 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.584+0000 7fb0e9da1700 1 -- 172.21.15.104:0/3949319974 wait complete. 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.585+0000 7fb0e9da1700 1 Processor -- start 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.585+0000 7fb0e9da1700 1 -- start start 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.585+0000 7fb0e9da1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 0x7fb0e419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.585+0000 7fb0e9da1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e410c1c0 con 0x7fb0e4108870 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.585+0000 7fb0e8d9f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 0x7fb0e419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.585+0000 7fb0e8d9f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 0x7fb0e419dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34340/0 (socket says 172.21.15.104:34340) 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.585+0000 7fb0e8d9f700 1 -- 172.21.15.104:0/2970246827 learned_addr learned my addr 172.21.15.104:0/2970246827 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.585+0000 7fb0e8d9f700 1 -- 172.21.15.104:0/2970246827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e419e520 con 0x7fb0e4108870 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.585+0000 7fb0e8d9f700 1 --2- 172.21.15.104:0/2970246827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 0x7fb0e419dfe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009fb0 tx=0x7fb0d8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:14.450 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.585+0000 7fb0e1ffb700 1 -- 172.21.15.104:0/2970246827 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d80038b0 con 0x7fb0e4108870 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.585+0000 7fb0e1ffb700 1 -- 172.21.15.104:0/2970246827 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb0d8011e50 con 0x7fb0e4108870 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.586+0000 7fb0e9da1700 1 -- 172.21.15.104:0/2970246827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e419e7b0 con 0x7fb0e4108870 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.586+0000 7fb0e1ffb700 1 -- 172.21.15.104:0/2970246827 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d8019420 con 0x7fb0e4108870 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.586+0000 7fb0e9da1700 1 -- 172.21.15.104:0/2970246827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e41a14a0 con 0x7fb0e4108870 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.587+0000 7fb0e1ffb700 1 -- 172.21.15.104:0/2970246827 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fb0d80196f0 con 0x7fb0e4108870 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.587+0000 7fb0e9da1700 1 -- 172.21.15.104:0/2970246827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e40695d0 con 0x7fb0e4108870 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.587+0000 7fb0e1ffb700 1 --2- 172.21.15.104:0/2970246827 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb0d4039050 0x7fb0d403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.587+0000 7fb0e1ffb700 1 -- 172.21.15.104:0/2970246827 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb0d804d700 con 0x7fb0e4108870 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.587+0000 7fb0e3fff700 1 --2- 172.21.15.104:0/2970246827 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb0d4039050 0x7fb0d403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.588+0000 7fb0e3fff700 1 --2- 172.21.15.104:0/2970246827 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb0d4039050 0x7fb0d403b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0006fd0 tx=0x7fb0d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.592+0000 7fb0e1ffb700 1 -- 172.21.15.104:0/2970246827 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb0d800ba30 con 0x7fb0e4108870 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.795+0000 7fb0e9da1700 1 -- 172.21.15.104:0/2970246827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fb0e41a1af0 con 0x7fb0e4108870 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.800+0000 7fb0e1ffb700 1 -- 172.21.15.104:0/2970246827 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7fb0d800bc90 con 0x7fb0e4108870 2024-06-09T22:09:14.451 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.803+0000 7fb0e9da1700 1 -- 172.21.15.104:0/2970246827 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb0d4039050 msgr2=0x7fb0d403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:14.452 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.803+0000 7fb0e9da1700 1 --2- 172.21.15.104:0/2970246827 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb0d4039050 0x7fb0d403b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0006fd0 tx=0x7fb0d0006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:14.452 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.803+0000 7fb0e9da1700 1 -- 172.21.15.104:0/2970246827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 msgr2=0x7fb0e419dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:14.452 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.803+0000 7fb0e9da1700 1 --2- 172.21.15.104:0/2970246827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 0x7fb0e419dfe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009fb0 tx=0x7fb0d8011bf0 comp rx=0 tx=0).stop 2024-06-09T22:09:14.452 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.803+0000 7fb0e9da1700 1 -- 172.21.15.104:0/2970246827 shutdown_connections 2024-06-09T22:09:14.452 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.803+0000 7fb0e9da1700 1 --2- 172.21.15.104:0/2970246827 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fb0d4039050 0x7fb0d403b510 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:14.452 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.803+0000 7fb0e9da1700 1 --2- 172.21.15.104:0/2970246827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e4108870 0x7fb0e419dfe0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:14.452 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.803+0000 7fb0e9da1700 1 -- 172.21.15.104:0/2970246827 >> 172.21.15.104:0/2970246827 conn(0x7fb0e41022e0 msgr2=0x7fb0e4102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:14.452 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.803+0000 7fb0e9da1700 1 -- 172.21.15.104:0/2970246827 shutdown_connections 2024-06-09T22:09:14.452 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:13.803+0000 7fb0e9da1700 1 -- 172.21.15.104:0/2970246827 wait complete. 2024-06-09T22:09:14.993 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:14 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2970246827' entity='client.admin' 2024-06-09T22:09:16.287 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:16 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1959843645' entity='client.admin' 2024-06-09T22:09:16.294 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.894+0000 7f7b77c92700 1 Processor -- start 2024-06-09T22:09:16.294 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.894+0000 7f7b77c92700 1 -- start start 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.894+0000 7f7b77c92700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 0x7f7b7010ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.894+0000 7f7b77c92700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7007be70 con 0x7f7b701089b0 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.894+0000 7f7b75a2e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 0x7f7b7010ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.894+0000 7f7b75a2e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 0x7f7b7010ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34344/0 (socket says 172.21.15.104:34344) 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.894+0000 7f7b75a2e700 1 -- 172.21.15.104:0/3049881142 learned_addr learned my addr 172.21.15.104:0/3049881142 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.895+0000 7f7b75a2e700 1 -- 172.21.15.104:0/3049881142 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7010b2e0 con 0x7f7b701089b0 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.895+0000 7f7b75a2e700 1 --2- 172.21.15.104:0/3049881142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 0x7f7b7010ada0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0098a0 tx=0x7f7b6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8450a2b383797e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.895+0000 7f7b74a2c700 1 -- 172.21.15.104:0/3049881142 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b6c004160 con 0x7f7b701089b0 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.895+0000 7f7b74a2c700 1 -- 172.21.15.104:0/3049881142 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7b6c004300 con 0x7f7b701089b0 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.895+0000 7f7b74a2c700 1 -- 172.21.15.104:0/3049881142 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b6c011b50 con 0x7f7b701089b0 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.896+0000 7f7b77c92700 1 -- 172.21.15.104:0/3049881142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 msgr2=0x7f7b7010ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.896+0000 7f7b77c92700 1 --2- 172.21.15.104:0/3049881142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 0x7f7b7010ada0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0098a0 tx=0x7f7b6c009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:16.295 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.896+0000 7f7b77c92700 1 -- 172.21.15.104:0/3049881142 shutdown_connections 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.896+0000 7f7b77c92700 1 --2- 172.21.15.104:0/3049881142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 0x7f7b7010ada0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.896+0000 7f7b77c92700 1 -- 172.21.15.104:0/3049881142 >> 172.21.15.104:0/3049881142 conn(0x7f7b701023c0 msgr2=0x7f7b70104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.896+0000 7f7b77c92700 1 -- 172.21.15.104:0/3049881142 shutdown_connections 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.896+0000 7f7b77c92700 1 -- 172.21.15.104:0/3049881142 wait complete. 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.896+0000 7f7b77c92700 1 Processor -- start 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.896+0000 7f7b77c92700 1 -- start start 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.897+0000 7f7b77c92700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 0x7f7b7019be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.897+0000 7f7b77c92700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7010c290 con 0x7f7b701089b0 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.897+0000 7f7b75a2e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 0x7f7b7019be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.897+0000 7f7b75a2e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 0x7f7b7019be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34354/0 (socket says 172.21.15.104:34354) 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.897+0000 7f7b75a2e700 1 -- 172.21.15.104:0/1959843645 learned_addr learned my addr 172.21.15.104:0/1959843645 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.897+0000 7f7b75a2e700 1 -- 172.21.15.104:0/1959843645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7019c3d0 con 0x7f7b701089b0 2024-06-09T22:09:16.296 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.897+0000 7f7b75a2e700 1 --2- 172.21.15.104:0/1959843645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 0x7f7b7019be90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c011670 tx=0x7f7b6c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:16.297 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.897+0000 7f7b66ffd700 1 -- 172.21.15.104:0/1959843645 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b6c0039c0 con 0x7f7b701089b0 2024-06-09T22:09:16.297 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.897+0000 7f7b66ffd700 1 -- 172.21.15.104:0/1959843645 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7b6c011e50 con 0x7f7b701089b0 2024-06-09T22:09:16.297 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.897+0000 7f7b77c92700 1 -- 172.21.15.104:0/1959843645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7019c6c0 con 0x7f7b701089b0 2024-06-09T22:09:16.297 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.897+0000 7f7b66ffd700 1 -- 172.21.15.104:0/1959843645 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b6c01a5c0 con 0x7f7b701089b0 2024-06-09T22:09:16.297 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.897+0000 7f7b77c92700 1 -- 172.21.15.104:0/1959843645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b7019f350 con 0x7f7b701089b0 2024-06-09T22:09:16.297 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.898+0000 7f7b66ffd700 1 -- 172.21.15.104:0/1959843645 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f7b6c01a870 con 0x7f7b701089b0 2024-06-09T22:09:16.297 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.898+0000 7f7b77c92700 1 -- 172.21.15.104:0/1959843645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b70063af0 con 0x7f7b701089b0 2024-06-09T22:09:16.297 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.898+0000 7f7b66ffd700 1 --2- 172.21.15.104:0/1959843645 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f7b5c039050 0x7f7b5c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:16.297 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.898+0000 7f7b66ffd700 1 -- 172.21.15.104:0/1959843645 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7b6c04d500 con 0x7f7b701089b0 2024-06-09T22:09:16.297 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.898+0000 7f7b7522d700 1 --2- 172.21.15.104:0/1959843645 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f7b5c039050 0x7f7b5c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:16.297 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.899+0000 7f7b7522d700 1 --2- 172.21.15.104:0/1959843645 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f7b5c039050 0x7f7b5c03b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7b60006fd0 tx=0x7f7b60006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:16.297 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:14.903+0000 7f7b66ffd700 1 -- 172.21.15.104:0/1959843645 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7b6c019c70 con 0x7f7b701089b0 2024-06-09T22:09:16.298 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:15.098+0000 7f7b77c92700 1 -- 172.21.15.104:0/1959843645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f7b7019f520 con 0x7f7b701089b0 2024-06-09T22:09:16.298 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:15.100+0000 7f7b66ffd700 1 -- 172.21.15.104:0/1959843645 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v7)=0 v7) ==== 130+0+0 (secure 0 0 0) 0x7f7b6c003b60 con 0x7f7b701089b0 2024-06-09T22:09:16.298 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:15.103+0000 7f7b77c92700 1 -- 172.21.15.104:0/1959843645 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f7b5c039050 msgr2=0x7f7b5c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:16.298 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:15.103+0000 7f7b77c92700 1 --2- 172.21.15.104:0/1959843645 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f7b5c039050 0x7f7b5c03b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7b60006fd0 tx=0x7f7b60006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:16.298 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:15.103+0000 7f7b77c92700 1 -- 172.21.15.104:0/1959843645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 msgr2=0x7f7b7019be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:16.298 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:15.103+0000 7f7b77c92700 1 --2- 172.21.15.104:0/1959843645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 0x7f7b7019be90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c011670 tx=0x7f7b6c011b60 comp rx=0 tx=0).stop 2024-06-09T22:09:16.298 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:15.103+0000 7f7b77c92700 1 -- 172.21.15.104:0/1959843645 shutdown_connections 2024-06-09T22:09:16.298 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:15.103+0000 7f7b77c92700 1 --2- 172.21.15.104:0/1959843645 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f7b5c039050 0x7f7b5c03b510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:16.298 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:15.103+0000 7f7b77c92700 1 --2- 172.21.15.104:0/1959843645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b701089b0 0x7f7b7019be90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:16.298 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:15.103+0000 7f7b77c92700 1 -- 172.21.15.104:0/1959843645 >> 172.21.15.104:0/1959843645 conn(0x7f7b701023c0 msgr2=0x7f7b70102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:16.298 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:15.103+0000 7f7b77c92700 1 -- 172.21.15.104:0/1959843645 shutdown_connections 2024-06-09T22:09:16.299 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:15.103+0000 7f7b77c92700 1 -- 172.21.15.104:0/1959843645 wait complete. 2024-06-09T22:09:16.299 INFO:teuthology.orchestra.run.smithi104.stdout:Enabling the dashboard module... 2024-06-09T22:09:18.104 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:17 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' 2024-06-09T22:09:18.104 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:17 smithi104 ceph-mon[32389]: from='mgr.14120 172.21.15.104:0/3713486194' entity='mgr.x' 2024-06-09T22:09:18.104 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:17 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3379774491' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-09T22:09:18.731 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:18 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: ignoring --setuser ceph since I am not root 2024-06-09T22:09:18.731 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:18 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: ignoring --setgroup ceph since I am not root 2024-06-09T22:09:18.731 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:18 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:18.543+0000 7f1647238200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-09T22:09:18.731 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:18 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:18.744+0000 7f1647238200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:09:18.993 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:18 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:18.848+0000 7f1647238200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-09T22:09:19.131 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.304+0000 7f9115a31700 1 Processor -- start 2024-06-09T22:09:19.131 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.304+0000 7f9115a31700 1 -- start start 2024-06-09T22:09:19.132 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.304+0000 7f9115a31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 0x7f9110109940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:19.132 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.304+0000 7f9115a31700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9110109f10 con 0x7f9110109520 2024-06-09T22:09:19.132 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.304+0000 7f910effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 0x7f9110109940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:19.132 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.304+0000 7f910effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 0x7f9110109940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43082/0 (socket says 172.21.15.104:43082) 2024-06-09T22:09:19.132 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.304+0000 7f910effd700 1 -- 172.21.15.104:0/3734383308 learned_addr learned my addr 172.21.15.104:0/3734383308 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:19.132 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.304+0000 7f910effd700 1 -- 172.21.15.104:0/3734383308 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911010a090 con 0x7f9110109520 2024-06-09T22:09:19.132 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.305+0000 7f910effd700 1 --2- 172.21.15.104:0/3734383308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 0x7f9110109940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f91000098a0 tx=0x7f9100009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cde76a9ef1b489 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:19.132 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.305+0000 7f910dffb700 1 -- 172.21.15.104:0/3734383308 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9100004160 con 0x7f9110109520 2024-06-09T22:09:19.132 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.305+0000 7f910dffb700 1 -- 172.21.15.104:0/3734383308 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9100004300 con 0x7f9110109520 2024-06-09T22:09:19.132 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.305+0000 7f910dffb700 1 -- 172.21.15.104:0/3734383308 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9100011b20 con 0x7f9110109520 2024-06-09T22:09:19.132 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.306+0000 7f9115a31700 1 -- 172.21.15.104:0/3734383308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 msgr2=0x7f9110109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:19.132 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.306+0000 7f9115a31700 1 --2- 172.21.15.104:0/3734383308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 0x7f9110109940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f91000098a0 tx=0x7f9100009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.306+0000 7f9115a31700 1 -- 172.21.15.104:0/3734383308 shutdown_connections 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.306+0000 7f9115a31700 1 --2- 172.21.15.104:0/3734383308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 0x7f9110109940 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.306+0000 7f9115a31700 1 -- 172.21.15.104:0/3734383308 >> 172.21.15.104:0/3734383308 conn(0x7f9110101b40 msgr2=0x7f9110103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.306+0000 7f9115a31700 1 -- 172.21.15.104:0/3734383308 shutdown_connections 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.306+0000 7f9115a31700 1 -- 172.21.15.104:0/3734383308 wait complete. 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.306+0000 7f9115a31700 1 Processor -- start 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.306+0000 7f9115a31700 1 -- start start 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.307+0000 7f9115a31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 0x7f911019be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.307+0000 7f9115a31700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9110106e30 con 0x7f9110109520 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.307+0000 7f910effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 0x7f911019be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.307+0000 7f910effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 0x7f911019be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43084/0 (socket says 172.21.15.104:43084) 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.307+0000 7f910effd700 1 -- 172.21.15.104:0/3379774491 learned_addr learned my addr 172.21.15.104:0/3379774491 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.307+0000 7f910effd700 1 -- 172.21.15.104:0/3379774491 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911019c3b0 con 0x7f9110109520 2024-06-09T22:09:19.133 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.307+0000 7f910effd700 1 --2- 172.21.15.104:0/3379774491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 0x7f911019be70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f91000117e0 tx=0x7f9100011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.307+0000 7f9114a2f700 1 -- 172.21.15.104:0/3379774491 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9100003bf0 con 0x7f9110109520 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.307+0000 7f9115a31700 1 -- 172.21.15.104:0/3379774491 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f911019c6a0 con 0x7f9110109520 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.307+0000 7f9114a2f700 1 -- 172.21.15.104:0/3379774491 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f910001a910 con 0x7f9110109520 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.307+0000 7f9114a2f700 1 -- 172.21.15.104:0/3379774491 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91000249a0 con 0x7f9110109520 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.307+0000 7f9115a31700 1 -- 172.21.15.104:0/3379774491 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f911019f330 con 0x7f9110109520 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.308+0000 7f9114a2f700 1 -- 172.21.15.104:0/3379774491 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f9100022020 con 0x7f9110109520 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.308+0000 7f9115a31700 1 -- 172.21.15.104:0/3379774491 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90fc005360 con 0x7f9110109520 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.309+0000 7f9114a2f700 1 --2- 172.21.15.104:0/3379774491 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f90f8039000 0x7f90f803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.309+0000 7f9114a2f700 1 -- 172.21.15.104:0/3379774491 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f910004d320 con 0x7f9110109520 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.309+0000 7f910e7fc700 1 --2- 172.21.15.104:0/3379774491 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f90f8039000 0x7f90f803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.309+0000 7f910e7fc700 1 --2- 172.21.15.104:0/3379774491 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f90f8039000 0x7f90f803b4c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9104006fd0 tx=0x7f9104006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.314+0000 7f9114a2f700 1 -- 172.21.15.104:0/3379774491 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9100011e20 con 0x7f9110109520 2024-06-09T22:09:19.134 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:17.592+0000 7f9115a31700 1 -- 172.21.15.104:0/3379774491 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f90fc005650 con 0x7f9110109520 2024-06-09T22:09:19.135 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:18.367+0000 7f9114a2f700 1 -- 172.21.15.104:0/3379774491 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f91000172b0 con 0x7f9110109520 2024-06-09T22:09:19.135 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:18.368+0000 7f9114a2f700 1 -- 172.21.15.104:0/3379774491 <== mon.0 v2:172.21.15.104:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f910002d910 con 0x7f9110109520 2024-06-09T22:09:19.135 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:18.370+0000 7f9115a31700 1 -- 172.21.15.104:0/3379774491 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f90f8039000 msgr2=0x7f90f803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:19.135 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:18.370+0000 7f9115a31700 1 --2- 172.21.15.104:0/3379774491 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f90f8039000 0x7f90f803b4c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9104006fd0 tx=0x7f9104006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:19.135 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:18.370+0000 7f9115a31700 1 -- 172.21.15.104:0/3379774491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 msgr2=0x7f911019be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:19.135 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:18.370+0000 7f9115a31700 1 --2- 172.21.15.104:0/3379774491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 0x7f911019be70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f91000117e0 tx=0x7f9100011c10 comp rx=0 tx=0).stop 2024-06-09T22:09:19.135 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:18.370+0000 7f9115a31700 1 -- 172.21.15.104:0/3379774491 shutdown_connections 2024-06-09T22:09:19.135 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:18.370+0000 7f9115a31700 1 --2- 172.21.15.104:0/3379774491 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7f90f8039000 0x7f90f803b4c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:19.135 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:18.370+0000 7f9115a31700 1 --2- 172.21.15.104:0/3379774491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9110109520 0x7f911019be70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:19.135 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:18.370+0000 7f9115a31700 1 -- 172.21.15.104:0/3379774491 >> 172.21.15.104:0/3379774491 conn(0x7f9110101b40 msgr2=0x7f9110102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:19.135 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:18.370+0000 7f9115a31700 1 -- 172.21.15.104:0/3379774491 shutdown_connections 2024-06-09T22:09:19.135 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:18.370+0000 7f9115a31700 1 -- 172.21.15.104:0/3379774491 wait complete. 2024-06-09T22:09:19.444 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:19 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3379774491' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-09T22:09:19.444 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:19 smithi104 ceph-mon[32389]: mgrmap e9: x(active, since 19s) 2024-06-09T22:09:19.444 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:19 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:19.157+0000 7f1647238200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-09T22:09:19.445 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:19 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:19.245+0000 7f1647238200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-09T22:09:20.456 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout { 2024-06-09T22:09:20.456 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-09T22:09:20.456 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-09T22:09:20.456 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-06-09T22:09:20.456 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-09T22:09:20.456 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout } 2024-06-09T22:09:20.456 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.559+0000 7fa747dd3700 1 Processor -- start 2024-06-09T22:09:20.456 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.559+0000 7fa747dd3700 1 -- start start 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.559+0000 7fa747dd3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 0x7fa74010ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.559+0000 7fa747dd3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa74007bdf0 con 0x7fa740108a30 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.559+0000 7fa745b6f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 0x7fa74010ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.559+0000 7fa745b6f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 0x7fa74010ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43112/0 (socket says 172.21.15.104:43112) 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.559+0000 7fa745b6f700 1 -- 172.21.15.104:0/1958884987 learned_addr learned my addr 172.21.15.104:0/1958884987 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.560+0000 7fa745b6f700 1 -- 172.21.15.104:0/1958884987 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74010b360 con 0x7fa740108a30 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.560+0000 7fa745b6f700 1 --2- 172.21.15.104:0/1958884987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 0x7fa74010ae20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0098a0 tx=0x7fa73c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b07bde72680b63 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.560+0000 7fa744b6d700 1 -- 172.21.15.104:0/1958884987 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa73c004160 con 0x7fa740108a30 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.560+0000 7fa744b6d700 1 -- 172.21.15.104:0/1958884987 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa73c004300 con 0x7fa740108a30 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.560+0000 7fa744b6d700 1 -- 172.21.15.104:0/1958884987 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa73c011b20 con 0x7fa740108a30 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.561+0000 7fa747dd3700 1 -- 172.21.15.104:0/1958884987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 msgr2=0x7fa74010ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.561+0000 7fa747dd3700 1 --2- 172.21.15.104:0/1958884987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 0x7fa74010ae20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0098a0 tx=0x7fa73c009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.561+0000 7fa747dd3700 1 -- 172.21.15.104:0/1958884987 shutdown_connections 2024-06-09T22:09:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.561+0000 7fa747dd3700 1 --2- 172.21.15.104:0/1958884987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 0x7fa74010ae20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:20.458 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.561+0000 7fa747dd3700 1 -- 172.21.15.104:0/1958884987 >> 172.21.15.104:0/1958884987 conn(0x7fa7401023c0 msgr2=0x7fa740104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:20.458 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.561+0000 7fa747dd3700 1 -- 172.21.15.104:0/1958884987 shutdown_connections 2024-06-09T22:09:20.458 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.561+0000 7fa747dd3700 1 -- 172.21.15.104:0/1958884987 wait complete. 2024-06-09T22:09:20.458 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.561+0000 7fa747dd3700 1 Processor -- start 2024-06-09T22:09:20.458 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa747dd3700 1 -- start start 2024-06-09T22:09:20.458 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa747dd3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 0x7fa74019be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:20.458 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa747dd3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa74010c360 con 0x7fa740108a30 2024-06-09T22:09:20.458 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa745b6f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 0x7fa74019be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:20.458 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa745b6f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 0x7fa74019be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43126/0 (socket says 172.21.15.104:43126) 2024-06-09T22:09:20.458 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa745b6f700 1 -- 172.21.15.104:0/454228319 learned_addr learned my addr 172.21.15.104:0/454228319 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:20.458 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa745b6f700 1 -- 172.21.15.104:0/454228319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74019c3c0 con 0x7fa740108a30 2024-06-09T22:09:20.458 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa745b6f700 1 --2- 172.21.15.104:0/454228319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 0x7fa74019be80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0117e0 tx=0x7fa73c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa736ffd700 1 -- 172.21.15.104:0/454228319 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa73c003ba0 con 0x7fa740108a30 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa736ffd700 1 -- 172.21.15.104:0/454228319 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa73c01a910 con 0x7fa740108a30 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa736ffd700 1 -- 172.21.15.104:0/454228319 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa73c0248f0 con 0x7fa740108a30 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa747dd3700 1 -- 172.21.15.104:0/454228319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa74019c650 con 0x7fa740108a30 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.562+0000 7fa747dd3700 1 -- 172.21.15.104:0/454228319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa74019cab0 con 0x7fa740108a30 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.563+0000 7fa747dd3700 1 -- 172.21.15.104:0/454228319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa740063af0 con 0x7fa740108a30 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.563+0000 7fa736ffd700 1 -- 172.21.15.104:0/454228319 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fa73c022020 con 0x7fa740108a30 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.563+0000 7fa736ffd700 1 --2- 172.21.15.104:0/454228319 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa72c039100 0x7fa72c03b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.564+0000 7fa736ffd700 1 -- 172.21.15.104:0/454228319 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa73c04d670 con 0x7fa740108a30 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.564+0000 7fa74536e700 1 -- 172.21.15.104:0/454228319 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa72c039100 msgr2=0x7fa72c03b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/85752866 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.564+0000 7fa74536e700 1 --2- 172.21.15.104:0/454228319 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa72c039100 0x7fa72c03b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.568+0000 7fa736ffd700 1 -- 172.21.15.104:0/454228319 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa73c02dc90 con 0x7fa740108a30 2024-06-09T22:09:20.459 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.765+0000 7fa74536e700 1 -- 172.21.15.104:0/454228319 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa72c039100 msgr2=0x7fa72c03b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/85752866 2024-06-09T22:09:20.460 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.765+0000 7fa74536e700 1 --2- 172.21.15.104:0/454228319 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa72c039100 0x7fa72c03b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:09:20.460 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.863+0000 7fa747dd3700 1 -- 172.21.15.104:0/454228319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa74019f520 con 0x7fa740108a30 2024-06-09T22:09:20.460 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.863+0000 7fa736ffd700 1 -- 172.21.15.104:0/454228319 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fa73c018aa0 con 0x7fa740108a30 2024-06-09T22:09:20.460 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.865+0000 7fa747dd3700 1 -- 172.21.15.104:0/454228319 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa72c039100 msgr2=0x7fa72c03b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-09T22:09:20.460 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.865+0000 7fa747dd3700 1 --2- 172.21.15.104:0/454228319 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa72c039100 0x7fa72c03b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:20.460 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.865+0000 7fa747dd3700 1 -- 172.21.15.104:0/454228319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 msgr2=0x7fa74019be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:20.460 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.865+0000 7fa747dd3700 1 --2- 172.21.15.104:0/454228319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 0x7fa74019be80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0117e0 tx=0x7fa73c011ba0 comp rx=0 tx=0).stop 2024-06-09T22:09:20.460 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.865+0000 7fa747dd3700 1 -- 172.21.15.104:0/454228319 shutdown_connections 2024-06-09T22:09:20.460 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.865+0000 7fa747dd3700 1 --2- 172.21.15.104:0/454228319 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fa72c039100 0x7fa72c03b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:20.460 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.865+0000 7fa747dd3700 1 --2- 172.21.15.104:0/454228319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa740108a30 0x7fa74019be80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:20.460 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.865+0000 7fa747dd3700 1 -- 172.21.15.104:0/454228319 >> 172.21.15.104:0/454228319 conn(0x7fa7401023c0 msgr2=0x7fa740102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:20.460 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.865+0000 7fa747dd3700 1 -- 172.21.15.104:0/454228319 shutdown_connections 2024-06-09T22:09:20.461 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:19.865+0000 7fa747dd3700 1 -- 172.21.15.104:0/454228319 wait complete. 2024-06-09T22:09:20.461 INFO:teuthology.orchestra.run.smithi104.stdout:Waiting for the mgr to restart... 2024-06-09T22:09:20.461 INFO:teuthology.orchestra.run.smithi104.stdout:Waiting for mgr epoch 9... 2024-06-09T22:09:20.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:20 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/454228319' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-09T22:09:20.493 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:20 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:20.460+0000 7f1647238200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-09T22:09:20.850 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:20 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:20.637+0000 7f1647238200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-09T22:09:20.850 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:20 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:20.810+0000 7f1647238200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-09T22:09:21.182 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:21 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:21.104+0000 7f1647238200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-09T22:09:21.182 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:21 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:21.195+0000 7f1647238200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-09T22:09:21.476 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:21 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:21.397+0000 7f1647238200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-09T22:09:21.743 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:21 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:21.590+0000 7f1647238200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:09:22.493 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:22 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:22.188+0000 7f1647238200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-09T22:09:22.894 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:22 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:22.609+0000 7f1647238200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-09T22:09:22.894 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:22 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:22.707+0000 7f1647238200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-09T22:09:22.894 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:22 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:22.907+0000 7f1647238200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-09T22:09:23.742 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:23 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:23.422+0000 7f1647238200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-09T22:09:23.742 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:23 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:23.514+0000 7f1647238200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-09T22:09:23.993 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:23 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:23.755+0000 7f1647238200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-09T22:09:23.993 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:23 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:23.888+0000 7f1647238200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-09T22:09:23.994 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:23 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:23.984+0000 7f1647238200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-09T22:09:24.388 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:24 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:24.104+0000 7f1647238200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-09T22:09:24.658 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:24 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:24.401+0000 7f1647238200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-09T22:09:25.243 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:25.002+0000 7f1647238200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-09T22:09:25.243 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:09:25.219+0000 7f1647238200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-09T22:09:25.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-mon[32389]: Active manager daemon x restarted 2024-06-09T22:09:25.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-mon[32389]: Activating manager daemon x 2024-06-09T22:09:25.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-mon[32389]: osdmap e3: 0 total, 0 up, 0 in 2024-06-09T22:09:25.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-mon[32389]: mgrmap e10: x(active, starting, since 0.00332034s) 2024-06-09T22:09:25.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:09:25.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-09T22:09:25.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-09T22:09:25.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-09T22:09:25.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-09T22:09:25.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-mon[32389]: Manager daemon x is now available 2024-06-09T22:09:25.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:09:25.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-09T22:09:26.436 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-09T22:09:26.436 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:26 smithi104 ceph-mon[32389]: mgrmap e11: x(active, since 1.00672s) 2024-06-09T22:09:27.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout { 2024-06-09T22:09:27.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-09T22:09:27.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-09T22:09:27.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout } 2024-06-09T22:09:27.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.882+0000 7fe99759e700 1 Processor -- start 2024-06-09T22:09:27.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.882+0000 7fe99759e700 1 -- start start 2024-06-09T22:09:27.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.882+0000 7fe99759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 0x7fe99810aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:27.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.882+0000 7fe99759e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe99807be10 con 0x7fe9981088b0 2024-06-09T22:09:27.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.883+0000 7fe99659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 0x7fe99810aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:27.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.883+0000 7fe99659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 0x7fe99810aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43132/0 (socket says 172.21.15.104:43132) 2024-06-09T22:09:27.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.883+0000 7fe99659c700 1 -- 172.21.15.104:0/496769901 learned_addr learned my addr 172.21.15.104:0/496769901 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:27.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.883+0000 7fe99659c700 1 -- 172.21.15.104:0/496769901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99810b1e0 con 0x7fe9981088b0 2024-06-09T22:09:27.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.883+0000 7fe99659c700 1 --2- 172.21.15.104:0/496769901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 0x7fe99810aca0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0098a0 tx=0x7fe98c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c580bf57f92d6a9 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:27.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.883+0000 7fe99559a700 1 -- 172.21.15.104:0/496769901 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe98c004160 con 0x7fe9981088b0 2024-06-09T22:09:27.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.883+0000 7fe99559a700 1 -- 172.21.15.104:0/496769901 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe98c004300 con 0x7fe9981088b0 2024-06-09T22:09:27.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.883+0000 7fe99559a700 1 -- 172.21.15.104:0/496769901 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe98c011b20 con 0x7fe9981088b0 2024-06-09T22:09:27.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.884+0000 7fe99759e700 1 -- 172.21.15.104:0/496769901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 msgr2=0x7fe99810aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:27.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.884+0000 7fe99759e700 1 --2- 172.21.15.104:0/496769901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 0x7fe99810aca0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0098a0 tx=0x7fe98c009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:27.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.884+0000 7fe99759e700 1 -- 172.21.15.104:0/496769901 shutdown_connections 2024-06-09T22:09:27.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.884+0000 7fe99759e700 1 --2- 172.21.15.104:0/496769901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 0x7fe99810aca0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:27.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.884+0000 7fe99759e700 1 -- 172.21.15.104:0/496769901 >> 172.21.15.104:0/496769901 conn(0x7fe9981022e0 msgr2=0x7fe998104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:27.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.884+0000 7fe99759e700 1 -- 172.21.15.104:0/496769901 shutdown_connections 2024-06-09T22:09:27.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.884+0000 7fe99759e700 1 -- 172.21.15.104:0/496769901 wait complete. 2024-06-09T22:09:27.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.885+0000 7fe99759e700 1 Processor -- start 2024-06-09T22:09:27.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.885+0000 7fe99759e700 1 -- start start 2024-06-09T22:09:27.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.885+0000 7fe99759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 0x7fe99819bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:27.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.885+0000 7fe99759e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe99810c1e0 con 0x7fe9981088b0 2024-06-09T22:09:27.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.885+0000 7fe99659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 0x7fe99819bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:27.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.885+0000 7fe99659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 0x7fe99819bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43142/0 (socket says 172.21.15.104:43142) 2024-06-09T22:09:27.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.885+0000 7fe99659c700 1 -- 172.21.15.104:0/2842995952 learned_addr learned my addr 172.21.15.104:0/2842995952 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:27.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.886+0000 7fe99659c700 1 -- 172.21.15.104:0/2842995952 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99819c2e0 con 0x7fe9981088b0 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.886+0000 7fe99659c700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 0x7fe99819bda0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe98c011b60 tx=0x7fe98c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.886+0000 7fe9877fe700 1 -- 172.21.15.104:0/2842995952 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe98c003b60 con 0x7fe9981088b0 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.886+0000 7fe9877fe700 1 -- 172.21.15.104:0/2842995952 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe98c01a910 con 0x7fe9981088b0 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.886+0000 7fe9877fe700 1 -- 172.21.15.104:0/2842995952 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe98c024930 con 0x7fe9981088b0 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.886+0000 7fe99759e700 1 -- 172.21.15.104:0/2842995952 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe99819c570 con 0x7fe9981088b0 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.886+0000 7fe99759e700 1 -- 172.21.15.104:0/2842995952 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe99819c9d0 con 0x7fe9981088b0 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.887+0000 7fe9877fe700 1 -- 172.21.15.104:0/2842995952 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fe98c022020 con 0x7fe9981088b0 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.887+0000 7fe9877fe700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 0x7fe98003b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.887+0000 7fe9877fe700 1 -- 172.21.15.104:0/2842995952 --> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe98003bcd0 con 0x7fe980039100 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.887+0000 7fe9877fe700 1 -- 172.21.15.104:0/2842995952 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe98c04d420 con 0x7fe9981088b0 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.887+0000 7fe995d9b700 1 -- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 msgr2=0x7fe98003b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/85752866 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:20.887+0000 7fe995d9b700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 0x7fe98003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:09:27.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:21.088+0000 7fe995d9b700 1 -- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 msgr2=0x7fe98003b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/85752866 2024-06-09T22:09:27.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:21.088+0000 7fe995d9b700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 0x7fe98003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:09:27.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:21.489+0000 7fe995d9b700 1 -- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 msgr2=0x7fe98003b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/85752866 2024-06-09T22:09:27.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:21.489+0000 7fe995d9b700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 0x7fe98003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-09T22:09:27.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:22.290+0000 7fe995d9b700 1 -- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 msgr2=0x7fe98003b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/85752866 2024-06-09T22:09:27.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:22.290+0000 7fe995d9b700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 0x7fe98003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-09T22:09:27.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:23.891+0000 7fe995d9b700 1 -- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 msgr2=0x7fe98003b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/85752866 2024-06-09T22:09:27.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:23.891+0000 7fe995d9b700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 0x7fe98003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-06-09T22:09:27.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:25.222+0000 7fe9877fe700 1 -- 172.21.15.104:0/2842995952 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7fe98c01ab30 con 0x7fe9981088b0 2024-06-09T22:09:27.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:25.222+0000 7fe9877fe700 1 -- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 msgr2=0x7fe98003b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-09T22:09:27.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:25.222+0000 7fe9877fe700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 0x7fe98003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:27.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.224+0000 7fe9877fe700 1 -- 172.21.15.104:0/2842995952 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7fe98c04f900 con 0x7fe9981088b0 2024-06-09T22:09:27.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.225+0000 7fe9877fe700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe98003cbc0 0x7fe98003efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.225+0000 7fe9877fe700 1 -- 172.21.15.104:0/2842995952 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe98003bcd0 con 0x7fe98003cbc0 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.225+0000 7fe995d9b700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe98003cbc0 0x7fe98003efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.226+0000 7fe995d9b700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe98003cbc0 0x7fe98003efb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe988003a10 tx=0x7fe988009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.227+0000 7fe9877fe700 1 -- 172.21.15.104:0/2842995952 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fe98003bcd0 con 0x7fe98003cbc0 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.238+0000 7fe99759e700 1 -- 172.21.15.104:0/2842995952 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe9980695d0 con 0x7fe98003cbc0 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.238+0000 7fe9877fe700 1 -- 172.21.15.104:0/2842995952 <== mgr.14152 v2:172.21.15.104:6800/1906048760 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fe9980695d0 con 0x7fe98003cbc0 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.238+0000 7fe99759e700 1 -- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe98003cbc0 msgr2=0x7fe98003efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.238+0000 7fe99759e700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe98003cbc0 0x7fe98003efb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe988003a10 tx=0x7fe988009270 comp rx=0 tx=0).stop 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.238+0000 7fe99759e700 1 -- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 msgr2=0x7fe99819bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.238+0000 7fe99759e700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 0x7fe99819bda0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe98c011b60 tx=0x7fe98c011c40 comp rx=0 tx=0).stop 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.238+0000 7fe99759e700 1 -- 172.21.15.104:0/2842995952 shutdown_connections 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.238+0000 7fe99759e700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe98003cbc0 0x7fe98003efb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:27.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.238+0000 7fe99759e700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9981088b0 0x7fe99819bda0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:27.122 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.238+0000 7fe99759e700 1 --2- 172.21.15.104:0/2842995952 >> [v2:172.21.15.104:6800/85752866,v1:172.21.15.104:6801/85752866] conn(0x7fe980039100 0x7fe98003b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:27.122 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.238+0000 7fe99759e700 1 -- 172.21.15.104:0/2842995952 >> 172.21.15.104:0/2842995952 conn(0x7fe9981022e0 msgr2=0x7fe998102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:27.122 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.238+0000 7fe99759e700 1 -- 172.21.15.104:0/2842995952 shutdown_connections 2024-06-09T22:09:27.122 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:26.239+0000 7fe99759e700 1 -- 172.21.15.104:0/2842995952 wait complete. 2024-06-09T22:09:27.122 INFO:teuthology.orchestra.run.smithi104.stdout:mgr epoch 9 is available 2024-06-09T22:09:27.122 INFO:teuthology.orchestra.run.smithi104.stdout:Generating a dashboard self-signed certificate... 2024-06-09T22:09:27.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:27 smithi104 ceph-mon[32389]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-09T22:09:27.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:27 smithi104 ceph-mon[32389]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-09T22:09:27.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:27 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:28.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:28 smithi104 ceph-mon[32389]: [09/Jun/2024:22:09:26] ENGINE Bus STARTING 2024-06-09T22:09:28.493 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:28 smithi104 ceph-mon[32389]: [09/Jun/2024:22:09:26] ENGINE Serving on http://172.21.15.104:8765 2024-06-09T22:09:28.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:28 smithi104 ceph-mon[32389]: [09/Jun/2024:22:09:26] ENGINE Serving on https://172.21.15.104:7150 2024-06-09T22:09:28.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:28 smithi104 ceph-mon[32389]: [09/Jun/2024:22:09:26] ENGINE Bus STARTED 2024-06-09T22:09:28.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:28 smithi104 ceph-mon[32389]: mgrmap e12: x(active, since 2s) 2024-06-09T22:09:28.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:28 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:28.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:28 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:28.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:28 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:28.521 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-09T22:09:28.521 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.574+0000 7f9808b43700 1 Processor -- start 2024-06-09T22:09:28.521 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.574+0000 7f9808b43700 1 -- start start 2024-06-09T22:09:28.521 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.575+0000 7f9808b43700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 0x7f9804107240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:28.521 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.575+0000 7f9808b43700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9804107780 con 0x7f9804106e20 2024-06-09T22:09:28.521 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.575+0000 7f980259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 0x7f9804107240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:28.521 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.575+0000 7f980259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 0x7f9804107240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40672/0 (socket says 172.21.15.104:40672) 2024-06-09T22:09:28.521 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.575+0000 7f980259c700 1 -- 172.21.15.104:0/2615232415 learned_addr learned my addr 172.21.15.104:0/2615232415 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:28.521 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.575+0000 7f980259c700 1 -- 172.21.15.104:0/2615232415 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9804107f80 con 0x7f9804106e20 2024-06-09T22:09:28.521 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.575+0000 7f980259c700 1 --2- 172.21.15.104:0/2615232415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 0x7f9804107240 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f97f4010b60 tx=0x7f97f4010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cbb9bd78f20c6c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:28.522 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.576+0000 7f980159a700 1 -- 172.21.15.104:0/2615232415 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97f400bdd0 con 0x7f9804106e20 2024-06-09T22:09:28.522 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.576+0000 7f980159a700 1 -- 172.21.15.104:0/2615232415 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f4004510 con 0x7f9804106e20 2024-06-09T22:09:28.522 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.576+0000 7f980159a700 1 -- 172.21.15.104:0/2615232415 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97f40185b0 con 0x7f9804106e20 2024-06-09T22:09:28.522 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.576+0000 7f9808b43700 1 -- 172.21.15.104:0/2615232415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 msgr2=0x7f9804107240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:28.522 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.576+0000 7f9808b43700 1 --2- 172.21.15.104:0/2615232415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 0x7f9804107240 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f97f4010b60 tx=0x7f97f4010e70 comp rx=0 tx=0).stop 2024-06-09T22:09:28.522 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.576+0000 7f9808b43700 1 -- 172.21.15.104:0/2615232415 shutdown_connections 2024-06-09T22:09:28.522 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.576+0000 7f9808b43700 1 --2- 172.21.15.104:0/2615232415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 0x7f9804107240 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:28.522 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.576+0000 7f9808b43700 1 -- 172.21.15.104:0/2615232415 >> 172.21.15.104:0/2615232415 conn(0x7f98041023c0 msgr2=0x7f9804104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:28.522 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.576+0000 7f9808b43700 1 -- 172.21.15.104:0/2615232415 shutdown_connections 2024-06-09T22:09:28.522 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.577+0000 7f9808b43700 1 -- 172.21.15.104:0/2615232415 wait complete. 2024-06-09T22:09:28.522 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.577+0000 7f9808b43700 1 Processor -- start 2024-06-09T22:09:28.522 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.577+0000 7f9808b43700 1 -- start start 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.578+0000 7f9808b43700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 0x7f9804199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.578+0000 7f9808b43700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f980407afb0 con 0x7f9804106e20 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.578+0000 7f980259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 0x7f9804199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.578+0000 7f980259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 0x7f9804199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40682/0 (socket says 172.21.15.104:40682) 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.578+0000 7f980259c700 1 -- 172.21.15.104:0/2860456709 learned_addr learned my addr 172.21.15.104:0/2860456709 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.578+0000 7f980259c700 1 -- 172.21.15.104:0/2860456709 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f980419a2b0 con 0x7f9804106e20 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.578+0000 7f980259c700 1 --2- 172.21.15.104:0/2860456709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 0x7f9804199d70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f97f4000c00 tx=0x7f97f40039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.578+0000 7f97f37fe700 1 -- 172.21.15.104:0/2860456709 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97f40188c0 con 0x7f9804106e20 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.578+0000 7f9808b43700 1 -- 172.21.15.104:0/2860456709 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f980419a540 con 0x7f9804106e20 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.578+0000 7f97f37fe700 1 -- 172.21.15.104:0/2860456709 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f4026070 con 0x7f9804106e20 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.578+0000 7f97f37fe700 1 -- 172.21.15.104:0/2860456709 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97f40173d0 con 0x7f9804106e20 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.578+0000 7f9808b43700 1 -- 172.21.15.104:0/2860456709 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f980419d1d0 con 0x7f9804106e20 2024-06-09T22:09:28.523 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.579+0000 7f97f37fe700 1 -- 172.21.15.104:0/2860456709 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f97f4017680 con 0x7f9804106e20 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.579+0000 7f9808b43700 1 -- 172.21.15.104:0/2860456709 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9804063af0 con 0x7f9804106e20 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.580+0000 7f97f37fe700 1 --2- 172.21.15.104:0/2860456709 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f97ec038f80 0x7f97ec03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.580+0000 7f97f37fe700 1 -- 172.21.15.104:0/2860456709 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f97f4051990 con 0x7f9804106e20 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.580+0000 7f9801d9b700 1 --2- 172.21.15.104:0/2860456709 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f97ec038f80 0x7f97ec03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.580+0000 7f9801d9b700 1 --2- 172.21.15.104:0/2860456709 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f97ec038f80 0x7f97ec03b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f97f8006fd0 tx=0x7f97f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.585+0000 7f97f37fe700 1 -- 172.21.15.104:0/2860456709 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f97f4021480 con 0x7f9804106e20 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.793+0000 7f9808b43700 1 -- 172.21.15.104:0/2860456709 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f98040628d0 con 0x7f97ec038f80 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.876+0000 7f97f37fe700 1 -- 172.21.15.104:0/2860456709 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f97f402b7e0 con 0x7f9804106e20 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.909+0000 7f97f37fe700 1 -- 172.21.15.104:0/2860456709 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f98040628d0 con 0x7f97ec038f80 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.911+0000 7f9808b43700 1 -- 172.21.15.104:0/2860456709 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f97ec038f80 msgr2=0x7f97ec03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.911+0000 7f9808b43700 1 --2- 172.21.15.104:0/2860456709 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f97ec038f80 0x7f97ec03b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f97f8006fd0 tx=0x7f97f8006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.911+0000 7f9808b43700 1 -- 172.21.15.104:0/2860456709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 msgr2=0x7f9804199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.911+0000 7f9808b43700 1 --2- 172.21.15.104:0/2860456709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 0x7f9804199d70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f97f4000c00 tx=0x7f97f40039f0 comp rx=0 tx=0).stop 2024-06-09T22:09:28.524 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.911+0000 7f9808b43700 1 -- 172.21.15.104:0/2860456709 shutdown_connections 2024-06-09T22:09:28.525 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.911+0000 7f9808b43700 1 --2- 172.21.15.104:0/2860456709 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f97ec038f80 0x7f97ec03b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:28.525 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.911+0000 7f9808b43700 1 --2- 172.21.15.104:0/2860456709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9804106e20 0x7f9804199d70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:28.525 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.911+0000 7f9808b43700 1 -- 172.21.15.104:0/2860456709 >> 172.21.15.104:0/2860456709 conn(0x7f98041023c0 msgr2=0x7f98041902f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:28.525 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.911+0000 7f9808b43700 1 -- 172.21.15.104:0/2860456709 shutdown_connections 2024-06-09T22:09:28.525 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:27.911+0000 7f9808b43700 1 -- 172.21.15.104:0/2860456709 wait complete. 2024-06-09T22:09:28.525 INFO:teuthology.orchestra.run.smithi104.stdout:Creating initial admin user... 2024-06-09T22:09:29.408 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:29 smithi104 ceph-mon[32389]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:09:30.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Q3ETmgFS1h18nZtJuvSrQOck04J/gvSUBq0vzWdt9CcPY/A7K4rRO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1717970969, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-09T22:09:30.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.958+0000 7f2a03c0c700 1 Processor -- start 2024-06-09T22:09:30.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.958+0000 7f2a03c0c700 1 -- start start 2024-06-09T22:09:30.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.958+0000 7f2a03c0c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 0x7f29fc107290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:30.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.958+0000 7f2a03c0c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f29fc1077d0 con 0x7f29fc106e70 2024-06-09T22:09:30.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.958+0000 7f2a019a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 0x7f29fc107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:30.116 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.959+0000 7f2a019a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 0x7f29fc107290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40690/0 (socket says 172.21.15.104:40690) 2024-06-09T22:09:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.959+0000 7f2a019a8700 1 -- 172.21.15.104:0/2066917824 learned_addr learned my addr 172.21.15.104:0/2066917824 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.959+0000 7f2a019a8700 1 -- 172.21.15.104:0/2066917824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29fc107f80 con 0x7f29fc106e70 2024-06-09T22:09:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.959+0000 7f2a019a8700 1 --2- 172.21.15.104:0/2066917824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 0x7f29fc107290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29f80098a0 tx=0x7f29f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12476965d3c0de6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.959+0000 7f2a009a6700 1 -- 172.21.15.104:0/2066917824 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29f8004160 con 0x7f29fc106e70 2024-06-09T22:09:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.959+0000 7f2a009a6700 1 -- 172.21.15.104:0/2066917824 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29f8004300 con 0x7f29fc106e70 2024-06-09T22:09:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.959+0000 7f2a009a6700 1 -- 172.21.15.104:0/2066917824 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29f8011b20 con 0x7f29fc106e70 2024-06-09T22:09:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.960+0000 7f2a03c0c700 1 -- 172.21.15.104:0/2066917824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 msgr2=0x7f29fc107290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.960+0000 7f2a03c0c700 1 --2- 172.21.15.104:0/2066917824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 0x7f29fc107290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29f80098a0 tx=0x7f29f8009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.960+0000 7f2a03c0c700 1 -- 172.21.15.104:0/2066917824 shutdown_connections 2024-06-09T22:09:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.960+0000 7f2a03c0c700 1 --2- 172.21.15.104:0/2066917824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 0x7f29fc107290 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.960+0000 7f2a03c0c700 1 -- 172.21.15.104:0/2066917824 >> 172.21.15.104:0/2066917824 conn(0x7f29fc102410 msgr2=0x7f29fc104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.960+0000 7f2a03c0c700 1 -- 172.21.15.104:0/2066917824 shutdown_connections 2024-06-09T22:09:30.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.960+0000 7f2a03c0c700 1 -- 172.21.15.104:0/2066917824 wait complete. 2024-06-09T22:09:30.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.960+0000 7f2a03c0c700 1 Processor -- start 2024-06-09T22:09:30.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.960+0000 7f2a03c0c700 1 -- start start 2024-06-09T22:09:30.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.961+0000 7f2a03c0c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 0x7f29fc07c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:30.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.961+0000 7f2a03c0c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f29fc10b3c0 con 0x7f29fc106e70 2024-06-09T22:09:30.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.961+0000 7f2a019a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 0x7f29fc07c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:30.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.961+0000 7f2a019a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 0x7f29fc07c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40702/0 (socket says 172.21.15.104:40702) 2024-06-09T22:09:30.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.961+0000 7f2a019a8700 1 -- 172.21.15.104:0/1990830956 learned_addr learned my addr 172.21.15.104:0/1990830956 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:30.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.961+0000 7f2a019a8700 1 -- 172.21.15.104:0/1990830956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29fc19ebf0 con 0x7f29fc106e70 2024-06-09T22:09:30.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.961+0000 7f2a019a8700 1 --2- 172.21.15.104:0/1990830956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 0x7f29fc07c2c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f29f80117e0 tx=0x7f29f8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:30.118 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.961+0000 7f29f2ffd700 1 -- 172.21.15.104:0/1990830956 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29f8003bf0 con 0x7f29fc106e70 2024-06-09T22:09:30.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.961+0000 7f29f2ffd700 1 -- 172.21.15.104:0/1990830956 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29f801a910 con 0x7f29fc106e70 2024-06-09T22:09:30.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.961+0000 7f2a03c0c700 1 -- 172.21.15.104:0/1990830956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29fc19edc0 con 0x7f29fc106e70 2024-06-09T22:09:30.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.961+0000 7f29f2ffd700 1 -- 172.21.15.104:0/1990830956 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29f8024970 con 0x7f29fc106e70 2024-06-09T22:09:30.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.961+0000 7f2a03c0c700 1 -- 172.21.15.104:0/1990830956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29fc19f160 con 0x7f29fc106e70 2024-06-09T22:09:30.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.962+0000 7f29f2ffd700 1 -- 172.21.15.104:0/1990830956 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f29f8022020 con 0x7f29fc106e70 2024-06-09T22:09:30.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.962+0000 7f2a03c0c700 1 -- 172.21.15.104:0/1990830956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29fc063af0 con 0x7f29fc106e70 2024-06-09T22:09:30.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.962+0000 7f29f2ffd700 1 --2- 172.21.15.104:0/1990830956 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f29e80390b0 0x7f29e803b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:30.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.962+0000 7f29f2ffd700 1 -- 172.21.15.104:0/1990830956 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f29f804d600 con 0x7f29fc106e70 2024-06-09T22:09:30.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.962+0000 7f2a011a7700 1 --2- 172.21.15.104:0/1990830956 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f29e80390b0 0x7f29e803b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:30.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.963+0000 7f2a011a7700 1 --2- 172.21.15.104:0/1990830956 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f29e80390b0 0x7f29e803b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29ec006fd0 tx=0x7f29ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:30.119 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:28.967+0000 7f29f2ffd700 1 -- 172.21.15.104:0/1990830956 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f29f8011e20 con 0x7f29fc106e70 2024-06-09T22:09:30.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:29.196+0000 7f2a03c0c700 1 -- 172.21.15.104:0/1990830956 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f29fc19f450 con 0x7f29e80390b0 2024-06-09T22:09:30.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:29.421+0000 7f29f2ffd700 1 -- 172.21.15.104:0/1990830956 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f29fc19f450 con 0x7f29e80390b0 2024-06-09T22:09:30.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:29.424+0000 7f2a03c0c700 1 -- 172.21.15.104:0/1990830956 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f29e80390b0 msgr2=0x7f29e803b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:30.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:29.424+0000 7f2a03c0c700 1 --2- 172.21.15.104:0/1990830956 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f29e80390b0 0x7f29e803b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29ec006fd0 tx=0x7f29ec006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:30.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:29.424+0000 7f2a03c0c700 1 -- 172.21.15.104:0/1990830956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 msgr2=0x7f29fc07c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:30.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:29.424+0000 7f2a03c0c700 1 --2- 172.21.15.104:0/1990830956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 0x7f29fc07c2c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f29f80117e0 tx=0x7f29f8011c10 comp rx=0 tx=0).stop 2024-06-09T22:09:30.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:29.424+0000 7f2a03c0c700 1 -- 172.21.15.104:0/1990830956 shutdown_connections 2024-06-09T22:09:30.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:29.424+0000 7f2a03c0c700 1 --2- 172.21.15.104:0/1990830956 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f29e80390b0 0x7f29e803b570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:30.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:29.424+0000 7f2a03c0c700 1 --2- 172.21.15.104:0/1990830956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29fc106e70 0x7f29fc07c2c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:30.120 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:29.424+0000 7f2a03c0c700 1 -- 172.21.15.104:0/1990830956 >> 172.21.15.104:0/1990830956 conn(0x7f29fc102410 msgr2=0x7f29fc102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:30.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:29.424+0000 7f2a03c0c700 1 -- 172.21.15.104:0/1990830956 shutdown_connections 2024-06-09T22:09:30.121 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:29.424+0000 7f2a03c0c700 1 -- 172.21.15.104:0/1990830956 wait complete. 2024-06-09T22:09:30.121 INFO:teuthology.orchestra.run.smithi104.stdout:Fetching dashboard port number... 2024-06-09T22:09:30.743 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:30 smithi104 ceph-mon[32389]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:09:30.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:31.491 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stdout 8443 2024-06-09T22:09:31.491 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.598+0000 7f60c8fd3700 1 Processor -- start 2024-06-09T22:09:31.491 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.598+0000 7f60c8fd3700 1 -- start start 2024-06-09T22:09:31.491 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.599+0000 7f60c8fd3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 0x7f60c410ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:31.491 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.599+0000 7f60c8fd3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f60c407be70 con 0x7f60c41089b0 2024-06-09T22:09:31.491 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.599+0000 7f60c259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 0x7f60c410ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:31.491 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.599+0000 7f60c259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 0x7f60c410ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40704/0 (socket says 172.21.15.104:40704) 2024-06-09T22:09:31.491 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.599+0000 7f60c259c700 1 -- 172.21.15.104:0/1828974807 learned_addr learned my addr 172.21.15.104:0/1828974807 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:31.491 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.599+0000 7f60c259c700 1 -- 172.21.15.104:0/1828974807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c410b2e0 con 0x7f60c41089b0 2024-06-09T22:09:31.491 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.599+0000 7f60c259c700 1 --2- 172.21.15.104:0/1828974807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 0x7f60c410ada0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f60b40098a0 tx=0x7f60b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb3558fa6a184d2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:31.491 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.599+0000 7f60c159a700 1 -- 172.21.15.104:0/1828974807 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60b4004160 con 0x7f60c41089b0 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.599+0000 7f60c159a700 1 -- 172.21.15.104:0/1828974807 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60b4004300 con 0x7f60c41089b0 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.599+0000 7f60c159a700 1 -- 172.21.15.104:0/1828974807 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60b4011b20 con 0x7f60c41089b0 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.600+0000 7f60c8fd3700 1 -- 172.21.15.104:0/1828974807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 msgr2=0x7f60c410ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.600+0000 7f60c8fd3700 1 --2- 172.21.15.104:0/1828974807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 0x7f60c410ada0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f60b40098a0 tx=0x7f60b4009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.600+0000 7f60c8fd3700 1 -- 172.21.15.104:0/1828974807 shutdown_connections 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.600+0000 7f60c8fd3700 1 --2- 172.21.15.104:0/1828974807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 0x7f60c410ada0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.600+0000 7f60c8fd3700 1 -- 172.21.15.104:0/1828974807 >> 172.21.15.104:0/1828974807 conn(0x7f60c41023c0 msgr2=0x7f60c4104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.600+0000 7f60c8fd3700 1 -- 172.21.15.104:0/1828974807 shutdown_connections 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.600+0000 7f60c8fd3700 1 -- 172.21.15.104:0/1828974807 wait complete. 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.601+0000 7f60c8fd3700 1 Processor -- start 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.601+0000 7f60c8fd3700 1 -- start start 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.601+0000 7f60c8fd3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 0x7f60c419be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.601+0000 7f60c8fd3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f60c410c340 con 0x7f60c41089b0 2024-06-09T22:09:31.492 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.601+0000 7f60c259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 0x7f60c419be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.601+0000 7f60c259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 0x7f60c419be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40710/0 (socket says 172.21.15.104:40710) 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.601+0000 7f60c259c700 1 -- 172.21.15.104:0/3857313956 learned_addr learned my addr 172.21.15.104:0/3857313956 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.602+0000 7f60c259c700 1 -- 172.21.15.104:0/3857313956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c419c370 con 0x7f60c41089b0 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.602+0000 7f60c259c700 1 --2- 172.21.15.104:0/3857313956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 0x7f60c419be30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f60b40117e0 tx=0x7f60b4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.602+0000 7f60b37fe700 1 -- 172.21.15.104:0/3857313956 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60b4003bf0 con 0x7f60c41089b0 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.602+0000 7f60b37fe700 1 -- 172.21.15.104:0/3857313956 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60b401a910 con 0x7f60c41089b0 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.602+0000 7f60c8fd3700 1 -- 172.21.15.104:0/3857313956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c419c660 con 0x7f60c41089b0 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.602+0000 7f60b37fe700 1 -- 172.21.15.104:0/3857313956 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60b4024970 con 0x7f60c41089b0 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.602+0000 7f60c8fd3700 1 -- 172.21.15.104:0/3857313956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c419f2f0 con 0x7f60c41089b0 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.603+0000 7f60b37fe700 1 -- 172.21.15.104:0/3857313956 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f60b4022020 con 0x7f60c41089b0 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.603+0000 7f60c8fd3700 1 -- 172.21.15.104:0/3857313956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60c4063af0 con 0x7f60c41089b0 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.603+0000 7f60b37fe700 1 --2- 172.21.15.104:0/3857313956 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f60ac039050 0x7f60ac03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:31.493 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.603+0000 7f60b37fe700 1 -- 172.21.15.104:0/3857313956 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f60b404d480 con 0x7f60c41089b0 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.603+0000 7f60c1d9b700 1 --2- 172.21.15.104:0/3857313956 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f60ac039050 0x7f60ac03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.604+0000 7f60c1d9b700 1 --2- 172.21.15.104:0/3857313956 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f60ac039050 0x7f60ac03b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f60b8006fd0 tx=0x7f60b8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.608+0000 7f60b37fe700 1 -- 172.21.15.104:0/3857313956 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f60b4011e20 con 0x7f60c41089b0 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.806+0000 7f60c8fd3700 1 -- 172.21.15.104:0/3857313956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f60c419f4c0 con 0x7f60c41089b0 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.806+0000 7f60b37fe700 1 -- 172.21.15.104:0/3857313956 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f60b40172b0 con 0x7f60c41089b0 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.808+0000 7f60c8fd3700 1 -- 172.21.15.104:0/3857313956 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f60ac039050 msgr2=0x7f60ac03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.808+0000 7f60c8fd3700 1 --2- 172.21.15.104:0/3857313956 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f60ac039050 0x7f60ac03b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f60b8006fd0 tx=0x7f60b8006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.808+0000 7f60c8fd3700 1 -- 172.21.15.104:0/3857313956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 msgr2=0x7f60c419be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.808+0000 7f60c8fd3700 1 --2- 172.21.15.104:0/3857313956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 0x7f60c419be30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f60b40117e0 tx=0x7f60b4011c10 comp rx=0 tx=0).stop 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.808+0000 7f60c8fd3700 1 -- 172.21.15.104:0/3857313956 shutdown_connections 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.808+0000 7f60c8fd3700 1 --2- 172.21.15.104:0/3857313956 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f60ac039050 0x7f60ac03b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.808+0000 7f60c8fd3700 1 --2- 172.21.15.104:0/3857313956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c41089b0 0x7f60c419be30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:31.494 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.808+0000 7f60c8fd3700 1 -- 172.21.15.104:0/3857313956 >> 172.21.15.104:0/3857313956 conn(0x7f60c41023c0 msgr2=0x7f60c4102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:31.495 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.808+0000 7f60c8fd3700 1 -- 172.21.15.104:0/3857313956 shutdown_connections 2024-06-09T22:09:31.495 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:30.808+0000 7f60c8fd3700 1 -- 172.21.15.104:0/3857313956 wait complete. 2024-06-09T22:09:31.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:31 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3857313956' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-09T22:09:31.498 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-09T22:09:31.498 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout disabled 2024-06-09T22:09:31.506 INFO:teuthology.orchestra.run.smithi104.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-09T22:09:31.506 INFO:teuthology.orchestra.run.smithi104.stdout:systemctl: stdout inactive 2024-06-09T22:09:31.506 INFO:teuthology.orchestra.run.smithi104.stdout:firewalld.service is not enabled 2024-06-09T22:09:31.506 INFO:teuthology.orchestra.run.smithi104.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-09T22:09:31.508 INFO:teuthology.orchestra.run.smithi104.stdout:Ceph Dashboard is now available at: 2024-06-09T22:09:31.508 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:09:31.508 INFO:teuthology.orchestra.run.smithi104.stdout: URL: https://smithi104.front.sepia.ceph.com:8443/ 2024-06-09T22:09:31.508 INFO:teuthology.orchestra.run.smithi104.stdout: User: admin 2024-06-09T22:09:31.508 INFO:teuthology.orchestra.run.smithi104.stdout: Password: vmabypg48h 2024-06-09T22:09:31.508 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:09:31.509 INFO:teuthology.orchestra.run.smithi104.stdout:Saving cluster configuration to /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config directory 2024-06-09T22:09:31.509 INFO:teuthology.orchestra.run.smithi104.stdout:Enabling autotune for osd_memory_target 2024-06-09T22:09:32.681 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:32 smithi104 ceph-mon[32389]: mgrmap e13: x(active, since 6s) 2024-06-09T22:09:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.928+0000 7f82c4762700 1 Processor -- start 2024-06-09T22:09:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.928+0000 7f82c4762700 1 -- start start 2024-06-09T22:09:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.928+0000 7f82c4762700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 0x7f82bc108390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.928+0000 7f82c4762700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc1088d0 con 0x7f82bc107f50 2024-06-09T22:09:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.928+0000 7f82c24fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 0x7f82bc108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.928+0000 7f82c24fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 0x7f82bc108390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40718/0 (socket says 172.21.15.104:40718) 2024-06-09T22:09:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.928+0000 7f82c24fe700 1 -- 172.21.15.104:0/1558094163 learned_addr learned my addr 172.21.15.104:0/1558094163 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.929+0000 7f82c24fe700 1 -- 172.21.15.104:0/1558094163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82bc1090e0 con 0x7f82bc107f50 2024-06-09T22:09:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.929+0000 7f82c24fe700 1 --2- 172.21.15.104:0/1558094163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 0x7f82bc108390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f82b80098a0 tx=0x7f82b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33e72d764d48fe8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.929+0000 7f82c14fc700 1 -- 172.21.15.104:0/1558094163 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82b8004160 con 0x7f82bc107f50 2024-06-09T22:09:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.929+0000 7f82c14fc700 1 -- 172.21.15.104:0/1558094163 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82b8004300 con 0x7f82bc107f50 2024-06-09T22:09:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.929+0000 7f82c14fc700 1 -- 172.21.15.104:0/1558094163 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82b8011b20 con 0x7f82bc107f50 2024-06-09T22:09:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.929+0000 7f82c4762700 1 -- 172.21.15.104:0/1558094163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 msgr2=0x7f82bc108390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.929+0000 7f82c4762700 1 --2- 172.21.15.104:0/1558094163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 0x7f82bc108390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f82b80098a0 tx=0x7f82b8009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.930+0000 7f82c4762700 1 -- 172.21.15.104:0/1558094163 shutdown_connections 2024-06-09T22:09:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.930+0000 7f82c4762700 1 --2- 172.21.15.104:0/1558094163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 0x7f82bc108390 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.930+0000 7f82c4762700 1 -- 172.21.15.104:0/1558094163 >> 172.21.15.104:0/1558094163 conn(0x7f82bc103580 msgr2=0x7f82bc1059a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.930+0000 7f82c4762700 1 -- 172.21.15.104:0/1558094163 shutdown_connections 2024-06-09T22:09:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.930+0000 7f82c4762700 1 -- 172.21.15.104:0/1558094163 wait complete. 2024-06-09T22:09:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.930+0000 7f82c4762700 1 Processor -- start 2024-06-09T22:09:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.930+0000 7f82c4762700 1 -- start start 2024-06-09T22:09:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.930+0000 7f82c4762700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 0x7f82bc07e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.930+0000 7f82c4762700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc10c3a0 con 0x7f82bc107f50 2024-06-09T22:09:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.931+0000 7f82c24fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 0x7f82bc07e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.931+0000 7f82c24fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 0x7f82bc07e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40724/0 (socket says 172.21.15.104:40724) 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.931+0000 7f82c24fe700 1 -- 172.21.15.104:0/3194945421 learned_addr learned my addr 172.21.15.104:0/3194945421 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.931+0000 7f82c24fe700 1 -- 172.21.15.104:0/3194945421 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82bc07e920 con 0x7f82bc107f50 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.931+0000 7f82c24fe700 1 --2- 172.21.15.104:0/3194945421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 0x7f82bc07e3e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f82b80117e0 tx=0x7f82b8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.931+0000 7f82b37fe700 1 -- 172.21.15.104:0/3194945421 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82b8003bf0 con 0x7f82bc107f50 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.931+0000 7f82b37fe700 1 -- 172.21.15.104:0/3194945421 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82b801a910 con 0x7f82bc107f50 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.931+0000 7f82c4762700 1 -- 172.21.15.104:0/3194945421 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82bc07cb10 con 0x7f82bc107f50 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.931+0000 7f82b37fe700 1 -- 172.21.15.104:0/3194945421 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82b8024970 con 0x7f82bc107f50 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.931+0000 7f82c4762700 1 -- 172.21.15.104:0/3194945421 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82bc07cfd0 con 0x7f82bc107f50 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.932+0000 7f82b37fe700 1 -- 172.21.15.104:0/3194945421 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f82b8022020 con 0x7f82bc107f50 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.932+0000 7f82c4762700 1 -- 172.21.15.104:0/3194945421 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82bc10c520 con 0x7f82bc107f50 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.932+0000 7f82b37fe700 1 --2- 172.21.15.104:0/3194945421 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f82a80391a0 0x7f82a803b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.932+0000 7f82b37fe700 1 -- 172.21.15.104:0/3194945421 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f82b804d6b0 con 0x7f82bc107f50 2024-06-09T22:09:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.933+0000 7f82c1cfd700 1 --2- 172.21.15.104:0/3194945421 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f82a80391a0 0x7f82a803b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.933+0000 7f82c1cfd700 1 --2- 172.21.15.104:0/3194945421 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f82a80391a0 0x7f82a803b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f82ac006fd0 tx=0x7f82ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:31.937+0000 7f82b37fe700 1 -- 172.21.15.104:0/3194945421 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f82b800b3e0 con 0x7f82bc107f50 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:32.129+0000 7f82c4762700 1 -- 172.21.15.104:0/3194945421 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f82bc063af0 con 0x7f82bc107f50 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:32.130+0000 7f82b37fe700 1 -- 172.21.15.104:0/3194945421 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f82b8018e10 con 0x7f82bc107f50 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:32.132+0000 7f82c4762700 1 -- 172.21.15.104:0/3194945421 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f82a80391a0 msgr2=0x7f82a803b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:32.132+0000 7f82c4762700 1 --2- 172.21.15.104:0/3194945421 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f82a80391a0 0x7f82a803b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f82ac006fd0 tx=0x7f82ac006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:32.132+0000 7f82c4762700 1 -- 172.21.15.104:0/3194945421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 msgr2=0x7f82bc07e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:32.132+0000 7f82c4762700 1 --2- 172.21.15.104:0/3194945421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 0x7f82bc07e3e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f82b80117e0 tx=0x7f82b8011ba0 comp rx=0 tx=0).stop 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:32.132+0000 7f82c4762700 1 -- 172.21.15.104:0/3194945421 shutdown_connections 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:32.132+0000 7f82c4762700 1 --2- 172.21.15.104:0/3194945421 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f82a80391a0 0x7f82a803b660 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:32.132+0000 7f82c4762700 1 --2- 172.21.15.104:0/3194945421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82bc107f50 0x7f82bc07e3e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:32.132+0000 7f82c4762700 1 -- 172.21.15.104:0/3194945421 >> 172.21.15.104:0/3194945421 conn(0x7f82bc103580 msgr2=0x7f82bc104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:32.132+0000 7f82c4762700 1 -- 172.21.15.104:0/3194945421 shutdown_connections 2024-06-09T22:09:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:32.132+0000 7f82c4762700 1 -- 172.21.15.104:0/3194945421 wait complete. 2024-06-09T22:09:33.962 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.090+0000 7f1f9f225700 1 Processor -- start 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.090+0000 7f1f9f225700 1 -- start start 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.090+0000 7f1f9f225700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 0x7f1f981072b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.090+0000 7f1f9f225700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1f981077f0 con 0x7f1f98106e90 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.091+0000 7f1f9cfc1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 0x7f1f981072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.091+0000 7f1f9cfc1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 0x7f1f981072b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40732/0 (socket says 172.21.15.104:40732) 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.091+0000 7f1f9cfc1700 1 -- 172.21.15.104:0/1792112885 learned_addr learned my addr 172.21.15.104:0/1792112885 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.091+0000 7f1f9cfc1700 1 -- 172.21.15.104:0/1792112885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f98107fa0 con 0x7f1f98106e90 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.091+0000 7f1f9cfc1700 1 --2- 172.21.15.104:0/1792112885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 0x7f1f981072b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0098a0 tx=0x7f1f8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=811befa29db5bc73 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.091+0000 7f1f977fe700 1 -- 172.21.15.104:0/1792112885 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f8c004160 con 0x7f1f98106e90 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.091+0000 7f1f977fe700 1 -- 172.21.15.104:0/1792112885 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f8c004300 con 0x7f1f98106e90 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.091+0000 7f1f977fe700 1 -- 172.21.15.104:0/1792112885 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f8c011b20 con 0x7f1f98106e90 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.092+0000 7f1f9f225700 1 -- 172.21.15.104:0/1792112885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 msgr2=0x7f1f981072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:33.963 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.092+0000 7f1f9f225700 1 --2- 172.21.15.104:0/1792112885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 0x7f1f981072b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0098a0 tx=0x7f1f8c009bb0 comp rx=0 tx=0).stop 2024-06-09T22:09:33.964 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.092+0000 7f1f9f225700 1 -- 172.21.15.104:0/1792112885 shutdown_connections 2024-06-09T22:09:33.964 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.092+0000 7f1f9f225700 1 --2- 172.21.15.104:0/1792112885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 0x7f1f981072b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:33.964 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.092+0000 7f1f9f225700 1 -- 172.21.15.104:0/1792112885 >> 172.21.15.104:0/1792112885 conn(0x7f1f98102470 msgr2=0x7f1f98104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:33.964 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.092+0000 7f1f9f225700 1 -- 172.21.15.104:0/1792112885 shutdown_connections 2024-06-09T22:09:33.964 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.092+0000 7f1f9f225700 1 -- 172.21.15.104:0/1792112885 wait complete. 2024-06-09T22:09:33.964 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.092+0000 7f1f9f225700 1 Processor -- start 2024-06-09T22:09:33.964 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.092+0000 7f1f9f225700 1 -- start start 2024-06-09T22:09:33.964 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.093+0000 7f1f9f225700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 0x7f1f981047b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:33.964 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.093+0000 7f1f9f225700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9810b3e0 con 0x7f1f98106e90 2024-06-09T22:09:33.964 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.093+0000 7f1f9cfc1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 0x7f1f981047b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:33.964 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.093+0000 7f1f9cfc1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 0x7f1f981047b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40746/0 (socket says 172.21.15.104:40746) 2024-06-09T22:09:33.964 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.093+0000 7f1f9cfc1700 1 -- 172.21.15.104:0/4038759517 learned_addr learned my addr 172.21.15.104:0/4038759517 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:33.965 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.093+0000 7f1f9cfc1700 1 -- 172.21.15.104:0/4038759517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f98104cf0 con 0x7f1f98106e90 2024-06-09T22:09:33.965 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.093+0000 7f1f9cfc1700 1 --2- 172.21.15.104:0/4038759517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 0x7f1f981047b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0117e0 tx=0x7f1f8c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:33.965 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.093+0000 7f1f95ffb700 1 -- 172.21.15.104:0/4038759517 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f8c003bf0 con 0x7f1f98106e90 2024-06-09T22:09:33.965 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.093+0000 7f1f95ffb700 1 -- 172.21.15.104:0/4038759517 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f8c01a910 con 0x7f1f98106e90 2024-06-09T22:09:33.965 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.093+0000 7f1f9f225700 1 -- 172.21.15.104:0/4038759517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f98102ee0 con 0x7f1f98106e90 2024-06-09T22:09:33.965 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.093+0000 7f1f95ffb700 1 -- 172.21.15.104:0/4038759517 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f8c024970 con 0x7f1f98106e90 2024-06-09T22:09:33.965 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.093+0000 7f1f9f225700 1 -- 172.21.15.104:0/4038759517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f981033a0 con 0x7f1f98106e90 2024-06-09T22:09:33.965 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.094+0000 7f1f95ffb700 1 -- 172.21.15.104:0/4038759517 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f1f8c022020 con 0x7f1f98106e90 2024-06-09T22:09:33.965 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.094+0000 7f1f9f225700 1 -- 172.21.15.104:0/4038759517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f98063af0 con 0x7f1f98106e90 2024-06-09T22:09:33.965 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.094+0000 7f1f95ffb700 1 --2- 172.21.15.104:0/4038759517 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1f880390f0 0x7f1f8803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:33.965 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.095+0000 7f1f95ffb700 1 -- 172.21.15.104:0/4038759517 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1f8c04d5f0 con 0x7f1f98106e90 2024-06-09T22:09:33.965 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.095+0000 7f1f97fff700 1 --2- 172.21.15.104:0/4038759517 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1f880390f0 0x7f1f8803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.095+0000 7f1f97fff700 1 --2- 172.21.15.104:0/4038759517 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1f880390f0 0x7f1f8803b5b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1f84006fd0 tx=0x7f1f84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.100+0000 7f1f95ffb700 1 -- 172.21.15.104:0/4038759517 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1f8c011e20 con 0x7f1f98106e90 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.404+0000 7f1f9f225700 1 -- 172.21.15.104:0/4038759517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f1f981956a0 con 0x7f1f98106e90 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.407+0000 7f1f95ffb700 1 -- 172.21.15.104:0/4038759517 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) ==== 153+0+0 (secure 0 0 0) 0x7f1f8c00ba30 con 0x7f1f98106e90 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.409+0000 7f1f9f225700 1 -- 172.21.15.104:0/4038759517 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1f880390f0 msgr2=0x7f1f8803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.409+0000 7f1f9f225700 1 --2- 172.21.15.104:0/4038759517 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1f880390f0 0x7f1f8803b5b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1f84006fd0 tx=0x7f1f84006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.409+0000 7f1f9f225700 1 -- 172.21.15.104:0/4038759517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 msgr2=0x7f1f981047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.409+0000 7f1f9f225700 1 --2- 172.21.15.104:0/4038759517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 0x7f1f981047b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0117e0 tx=0x7f1f8c011c10 comp rx=0 tx=0).stop 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.409+0000 7f1f9f225700 1 -- 172.21.15.104:0/4038759517 shutdown_connections 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.409+0000 7f1f9f225700 1 --2- 172.21.15.104:0/4038759517 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1f880390f0 0x7f1f8803b5b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.409+0000 7f1f9f225700 1 --2- 172.21.15.104:0/4038759517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f98106e90 0x7f1f981047b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.409+0000 7f1f9f225700 1 -- 172.21.15.104:0/4038759517 >> 172.21.15.104:0/4038759517 conn(0x7f1f98102470 msgr2=0x7f1f981929c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.409+0000 7f1f9f225700 1 -- 172.21.15.104:0/4038759517 shutdown_connections 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr 2024-06-09T22:09:33.409+0000 7f1f9f225700 1 -- 172.21.15.104:0/4038759517 wait complete. 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout:Or, if you are only running a single cluster on this host: 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:09:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: ceph telemetry on 2024-06-09T22:09:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:09:33.968 INFO:teuthology.orchestra.run.smithi104.stdout:For more information see: 2024-06-09T22:09:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:09:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-06-09T22:09:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:09:33.968 INFO:teuthology.orchestra.run.smithi104.stdout:Bootstrap complete. 2024-06-09T22:09:33.997 INFO:tasks.cephadm:Fetching config... 2024-06-09T22:09:33.997 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:09:33.997 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-09T22:09:34.013 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-09T22:09:34.013 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:09:34.013 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-09T22:09:34.070 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-09T22:09:34.070 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:09:34.070 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/keyring of=/dev/stdout 2024-06-09T22:09:34.135 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-09T22:09:34.135 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:09:34.135 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-09T22:09:34.191 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-09T22:09:34.191 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/VUyIH7cF72dyZ84ZWviMNeJxtiUfX9MOxaJkPOk5H7yzbOIkVWycAP09tklpSNBlPcxQOVKN0cdOLjPsyGqBF9NvY+A2B/snv37M7AMm9E4uLoylwO3EXa1acTx161qDgLqD2IBKZ2fGowA/tPFZv1A/ncIpqZnFIWtyaASgssGRaJKegxdTVZSwRZwm8VhR1VNVJiD6bxQ1gFddNy1wxrmmkoGScgleMOOvp2Jk32+a2hYKy8gvRtIZ1CVsmmeGMyFGG+2dlY0m1LEIIM2mhf/cmzUfHK3n1dFXxC+wGi/5yhYF+O3zz3vHNfC+Axz8m6+9im5g624avZraIG2PvhYoK1gX5BMkgFdNVYEX9OQclzCqqyIzSAJnHmOYwLC//oPiNFYMONLZt1USy1asim+BeECtUvJvybHo3b4Mfz92W4a9jRK7Y/pyu8zNJzSuk+Ld5oy0mJtDPCWc+d3SAJPcHIQvVHOaJ10TEj/DysNBbI1KXOUvG6sFxcZGqfk= ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-09T22:09:34.264 INFO:teuthology.orchestra.run.smithi104.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/VUyIH7cF72dyZ84ZWviMNeJxtiUfX9MOxaJkPOk5H7yzbOIkVWycAP09tklpSNBlPcxQOVKN0cdOLjPsyGqBF9NvY+A2B/snv37M7AMm9E4uLoylwO3EXa1acTx161qDgLqD2IBKZ2fGowA/tPFZv1A/ncIpqZnFIWtyaASgssGRaJKegxdTVZSwRZwm8VhR1VNVJiD6bxQ1gFddNy1wxrmmkoGScgleMOOvp2Jk32+a2hYKy8gvRtIZ1CVsmmeGMyFGG+2dlY0m1LEIIM2mhf/cmzUfHK3n1dFXxC+wGi/5yhYF+O3zz3vHNfC+Axz8m6+9im5g624avZraIG2PvhYoK1gX5BMkgFdNVYEX9OQclzCqqyIzSAJnHmOYwLC//oPiNFYMONLZt1USy1asim+BeECtUvJvybHo3b4Mfz92W4a9jRK7Y/pyu8zNJzSuk+Ld5oy0mJtDPCWc+d3SAJPcHIQvVHOaJ10TEj/DysNBbI1KXOUvG6sFxcZGqfk= ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:09:34.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/VUyIH7cF72dyZ84ZWviMNeJxtiUfX9MOxaJkPOk5H7yzbOIkVWycAP09tklpSNBlPcxQOVKN0cdOLjPsyGqBF9NvY+A2B/snv37M7AMm9E4uLoylwO3EXa1acTx161qDgLqD2IBKZ2fGowA/tPFZv1A/ncIpqZnFIWtyaASgssGRaJKegxdTVZSwRZwm8VhR1VNVJiD6bxQ1gFddNy1wxrmmkoGScgleMOOvp2Jk32+a2hYKy8gvRtIZ1CVsmmeGMyFGG+2dlY0m1LEIIM2mhf/cmzUfHK3n1dFXxC+wGi/5yhYF+O3zz3vHNfC+Axz8m6+9im5g624avZraIG2PvhYoK1gX5BMkgFdNVYEX9OQclzCqqyIzSAJnHmOYwLC//oPiNFYMONLZt1USy1asim+BeECtUvJvybHo3b4Mfz92W4a9jRK7Y/pyu8zNJzSuk+Ld5oy0mJtDPCWc+d3SAJPcHIQvVHOaJ10TEj/DysNBbI1KXOUvG6sFxcZGqfk= ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-09T22:09:34.316 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/VUyIH7cF72dyZ84ZWviMNeJxtiUfX9MOxaJkPOk5H7yzbOIkVWycAP09tklpSNBlPcxQOVKN0cdOLjPsyGqBF9NvY+A2B/snv37M7AMm9E4uLoylwO3EXa1acTx161qDgLqD2IBKZ2fGowA/tPFZv1A/ncIpqZnFIWtyaASgssGRaJKegxdTVZSwRZwm8VhR1VNVJiD6bxQ1gFddNy1wxrmmkoGScgleMOOvp2Jk32+a2hYKy8gvRtIZ1CVsmmeGMyFGG+2dlY0m1LEIIM2mhf/cmzUfHK3n1dFXxC+wGi/5yhYF+O3zz3vHNfC+Axz8m6+9im5g624avZraIG2PvhYoK1gX5BMkgFdNVYEX9OQclzCqqyIzSAJnHmOYwLC//oPiNFYMONLZt1USy1asim+BeECtUvJvybHo3b4Mfz92W4a9jRK7Y/pyu8zNJzSuk+Ld5oy0mJtDPCWc+d3SAJPcHIQvVHOaJ10TEj/DysNBbI1KXOUvG6sFxcZGqfk= ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:09:34.328 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/VUyIH7cF72dyZ84ZWviMNeJxtiUfX9MOxaJkPOk5H7yzbOIkVWycAP09tklpSNBlPcxQOVKN0cdOLjPsyGqBF9NvY+A2B/snv37M7AMm9E4uLoylwO3EXa1acTx161qDgLqD2IBKZ2fGowA/tPFZv1A/ncIpqZnFIWtyaASgssGRaJKegxdTVZSwRZwm8VhR1VNVJiD6bxQ1gFddNy1wxrmmkoGScgleMOOvp2Jk32+a2hYKy8gvRtIZ1CVsmmeGMyFGG+2dlY0m1LEIIM2mhf/cmzUfHK3n1dFXxC+wGi/5yhYF+O3zz3vHNfC+Axz8m6+9im5g624avZraIG2PvhYoK1gX5BMkgFdNVYEX9OQclzCqqyIzSAJnHmOYwLC//oPiNFYMONLZt1USy1asim+BeECtUvJvybHo3b4Mfz92W4a9jRK7Y/pyu8zNJzSuk+Ld5oy0mJtDPCWc+d3SAJPcHIQvVHOaJ10TEj/DysNBbI1KXOUvG6sFxcZGqfk= ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-09T22:09:34.366 INFO:teuthology.orchestra.run.smithi194.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/VUyIH7cF72dyZ84ZWviMNeJxtiUfX9MOxaJkPOk5H7yzbOIkVWycAP09tklpSNBlPcxQOVKN0cdOLjPsyGqBF9NvY+A2B/snv37M7AMm9E4uLoylwO3EXa1acTx161qDgLqD2IBKZ2fGowA/tPFZv1A/ncIpqZnFIWtyaASgssGRaJKegxdTVZSwRZwm8VhR1VNVJiD6bxQ1gFddNy1wxrmmkoGScgleMOOvp2Jk32+a2hYKy8gvRtIZ1CVsmmeGMyFGG+2dlY0m1LEIIM2mhf/cmzUfHK3n1dFXxC+wGi/5yhYF+O3zz3vHNfC+Axz8m6+9im5g624avZraIG2PvhYoK1gX5BMkgFdNVYEX9OQclzCqqyIzSAJnHmOYwLC//oPiNFYMONLZt1USy1asim+BeECtUvJvybHo3b4Mfz92W4a9jRK7Y/pyu8zNJzSuk+Ld5oy0mJtDPCWc+d3SAJPcHIQvVHOaJ10TEj/DysNBbI1KXOUvG6sFxcZGqfk= ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:09:34.378 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-09T22:09:34.617 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:09:34.740 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:34 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/4038759517' entity='client.admin' 2024-06-09T22:09:36.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.756+0000 7fa7d2067700 1 -- 172.21.15.104:0/3924450453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7cc102a00 msgr2=0x7fa7cc104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:36.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.756+0000 7fa7d2067700 1 --2- 172.21.15.104:0/3924450453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7cc102a00 0x7fa7cc104df0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00b0a0 tx=0x7fa7bc00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:09:36.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.756+0000 7fa7d2067700 1 -- 172.21.15.104:0/3924450453 shutdown_connections 2024-06-09T22:09:36.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.756+0000 7fa7d2067700 1 --2- 172.21.15.104:0/3924450453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7cc102a00 0x7fa7cc104df0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:36.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.756+0000 7fa7d2067700 1 -- 172.21.15.104:0/3924450453 >> 172.21.15.104:0/3924450453 conn(0x7fa7cc0fc390 msgr2=0x7fa7cc0fe7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:36.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.756+0000 7fa7d2067700 1 -- 172.21.15.104:0/3924450453 shutdown_connections 2024-06-09T22:09:36.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.756+0000 7fa7d2067700 1 -- 172.21.15.104:0/3924450453 wait complete. 2024-06-09T22:09:36.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.757+0000 7fa7d2067700 1 Processor -- start 2024-06-09T22:09:36.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.757+0000 7fa7d2067700 1 -- start start 2024-06-09T22:09:36.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.757+0000 7fa7d2067700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7cc102a00 0x7fa7cc1975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:36.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.757+0000 7fa7d2067700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc107fb0 con 0x7fa7cc102a00 2024-06-09T22:09:36.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.757+0000 7fa7cb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7cc102a00 0x7fa7cc1975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:36.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.757+0000 7fa7cb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7cc102a00 0x7fa7cc1975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52186/0 (socket says 172.21.15.104:52186) 2024-06-09T22:09:36.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.757+0000 7fa7cb7fe700 1 -- 172.21.15.104:0/3673685537 learned_addr learned my addr 172.21.15.104:0/3673685537 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:36.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.757+0000 7fa7cb7fe700 1 -- 172.21.15.104:0/3673685537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7cc197b00 con 0x7fa7cc102a00 2024-06-09T22:09:36.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.757+0000 7fa7cb7fe700 1 --2- 172.21.15.104:0/3673685537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7cc102a00 0x7fa7cc1975c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc015040 tx=0x7fa7bc0075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:36.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.758+0000 7fa7c8ff9700 1 -- 172.21.15.104:0/3673685537 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7bc017040 con 0x7fa7cc102a00 2024-06-09T22:09:36.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.758+0000 7fa7c8ff9700 1 -- 172.21.15.104:0/3673685537 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7bc022070 con 0x7fa7cc102a00 2024-06-09T22:09:36.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.758+0000 7fa7c8ff9700 1 -- 172.21.15.104:0/3673685537 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7bc01d5d0 con 0x7fa7cc102a00 2024-06-09T22:09:36.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.758+0000 7fa7d2067700 1 -- 172.21.15.104:0/3673685537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7cc197d90 con 0x7fa7cc102a00 2024-06-09T22:09:36.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.758+0000 7fa7d2067700 1 -- 172.21.15.104:0/3673685537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7cc19aa80 con 0x7fa7cc102a00 2024-06-09T22:09:36.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.759+0000 7fa7c8ff9700 1 -- 172.21.15.104:0/3673685537 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fa7bc01d880 con 0x7fa7cc102a00 2024-06-09T22:09:36.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.759+0000 7fa7d2067700 1 -- 172.21.15.104:0/3673685537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7cc108730 con 0x7fa7cc102a00 2024-06-09T22:09:36.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.759+0000 7fa7c8ff9700 1 --2- 172.21.15.104:0/3673685537 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7b4039150 0x7fa7b403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:36.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.759+0000 7fa7c8ff9700 1 -- 172.21.15.104:0/3673685537 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa7bc00f050 con 0x7fa7cc102a00 2024-06-09T22:09:36.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.759+0000 7fa7caffd700 1 --2- 172.21.15.104:0/3673685537 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7b4039150 0x7fa7b403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:36.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.759+0000 7fa7caffd700 1 --2- 172.21.15.104:0/3673685537 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7b4039150 0x7fa7b403b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0006fd0 tx=0x7fa7c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:36.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.764+0000 7fa7c8ff9700 1 -- 172.21.15.104:0/3673685537 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa7bc007e20 con 0x7fa7cc102a00 2024-06-09T22:09:36.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.958+0000 7fa7d2067700 1 -- 172.21.15.104:0/3673685537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fa7cc02ddb0 con 0x7fa7cc102a00 2024-06-09T22:09:36.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.960+0000 7fa7c8ff9700 1 -- 172.21.15.104:0/3673685537 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fa7bc025be0 con 0x7fa7cc102a00 2024-06-09T22:09:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.962+0000 7fa7d2067700 1 -- 172.21.15.104:0/3673685537 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7b4039150 msgr2=0x7fa7b403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.962+0000 7fa7d2067700 1 --2- 172.21.15.104:0/3673685537 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7b4039150 0x7fa7b403b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0006fd0 tx=0x7fa7c0006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.962+0000 7fa7d2067700 1 -- 172.21.15.104:0/3673685537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7cc102a00 msgr2=0x7fa7cc1975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.962+0000 7fa7d2067700 1 --2- 172.21.15.104:0/3673685537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7cc102a00 0x7fa7cc1975c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc015040 tx=0x7fa7bc0075c0 comp rx=0 tx=0).stop 2024-06-09T22:09:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.962+0000 7fa7d2067700 1 -- 172.21.15.104:0/3673685537 shutdown_connections 2024-06-09T22:09:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.962+0000 7fa7d2067700 1 --2- 172.21.15.104:0/3673685537 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7b4039150 0x7fa7b403b610 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.962+0000 7fa7d2067700 1 --2- 172.21.15.104:0/3673685537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7cc102a00 0x7fa7cc1975c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.962+0000 7fa7d2067700 1 -- 172.21.15.104:0/3673685537 >> 172.21.15.104:0/3673685537 conn(0x7fa7cc0fc390 msgr2=0x7fa7cc0fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.963+0000 7fa7d2067700 1 -- 172.21.15.104:0/3673685537 shutdown_connections 2024-06-09T22:09:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:36.963+0000 7fa7d2067700 1 -- 172.21.15.104:0/3673685537 wait complete. 2024-06-09T22:09:37.241 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:37.242 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:37.242 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:09:37.242 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:37.242 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:09:37.242 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:37.552 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-09T22:09:37.553 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-09T22:09:37.787 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:09:38.194 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:37 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3673685537' entity='client.admin' 2024-06-09T22:09:38.194 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:37 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:09:38.195 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:37 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:39.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.320+0000 7f162e277700 1 -- 172.21.15.104:0/2890226187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16281029c0 msgr2=0x7f1628104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:39.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.320+0000 7f162e277700 1 --2- 172.21.15.104:0/2890226187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16281029c0 0x7f1628104db0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1618009910 tx=0x7f1618009c20 comp rx=0 tx=0).stop 2024-06-09T22:09:39.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.320+0000 7f162e277700 1 -- 172.21.15.104:0/2890226187 shutdown_connections 2024-06-09T22:09:39.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.320+0000 7f162e277700 1 --2- 172.21.15.104:0/2890226187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16281029c0 0x7f1628104db0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:39.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.320+0000 7f162e277700 1 -- 172.21.15.104:0/2890226187 >> 172.21.15.104:0/2890226187 conn(0x7f16280fc3f0 msgr2=0x7f16280fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:39.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.320+0000 7f162e277700 1 -- 172.21.15.104:0/2890226187 shutdown_connections 2024-06-09T22:09:39.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.320+0000 7f162e277700 1 -- 172.21.15.104:0/2890226187 wait complete. 2024-06-09T22:09:39.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.321+0000 7f162e277700 1 Processor -- start 2024-06-09T22:09:39.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.321+0000 7f162e277700 1 -- start start 2024-06-09T22:09:39.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.321+0000 7f162e277700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16281029c0 0x7f1628102240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:39.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.321+0000 7f162e277700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1628107f60 con 0x7f16281029c0 2024-06-09T22:09:39.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.321+0000 7f1627fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16281029c0 0x7f1628102240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:39.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.321+0000 7f1627fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16281029c0 0x7f1628102240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52204/0 (socket says 172.21.15.104:52204) 2024-06-09T22:09:39.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.321+0000 7f1627fff700 1 -- 172.21.15.104:0/3722548474 learned_addr learned my addr 172.21.15.104:0/3722548474 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:39.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.321+0000 7f1627fff700 1 -- 172.21.15.104:0/3722548474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1628102780 con 0x7f16281029c0 2024-06-09T22:09:39.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.321+0000 7f1627fff700 1 --2- 172.21.15.104:0/3722548474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16281029c0 0x7f1628102240 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1618005b40 tx=0x7f1618005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:39.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.322+0000 7f16257fa700 1 -- 172.21.15.104:0/3722548474 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1618027070 con 0x7f16281029c0 2024-06-09T22:09:39.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.322+0000 7f16257fa700 1 -- 172.21.15.104:0/3722548474 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f161800be40 con 0x7f16281029c0 2024-06-09T22:09:39.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.322+0000 7f162e277700 1 -- 172.21.15.104:0/3722548474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1628100970 con 0x7f16281029c0 2024-06-09T22:09:39.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.322+0000 7f16257fa700 1 -- 172.21.15.104:0/3722548474 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f161800f460 con 0x7f16281029c0 2024-06-09T22:09:39.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.322+0000 7f162e277700 1 -- 172.21.15.104:0/3722548474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1628100eb0 con 0x7f16281029c0 2024-06-09T22:09:39.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.322+0000 7f16257fa700 1 -- 172.21.15.104:0/3722548474 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f161802a070 con 0x7f16281029c0 2024-06-09T22:09:39.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.322+0000 7f162e277700 1 -- 172.21.15.104:0/3722548474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16281086e0 con 0x7f16281029c0 2024-06-09T22:09:39.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.323+0000 7f16257fa700 1 --2- 172.21.15.104:0/3722548474 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f16100390f0 0x7f161003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:39.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.323+0000 7f16257fa700 1 -- 172.21.15.104:0/3722548474 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f161801e030 con 0x7f16281029c0 2024-06-09T22:09:39.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.323+0000 7f16277fe700 1 --2- 172.21.15.104:0/3722548474 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f16100390f0 0x7f161003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:39.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.323+0000 7f16277fe700 1 --2- 172.21.15.104:0/3722548474 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f16100390f0 0x7f161003b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f161c006fd0 tx=0x7f161c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:39.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.328+0000 7f16257fa700 1 -- 172.21.15.104:0/3722548474 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1618013070 con 0x7f16281029c0 2024-06-09T22:09:39.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.528+0000 7f162e277700 1 -- 172.21.15.104:0/3722548474 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f1628190b00 con 0x7f16100390f0 2024-06-09T22:09:39.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.532+0000 7f16257fa700 1 -- 172.21.15.104:0/3722548474 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1628190b00 con 0x7f16100390f0 2024-06-09T22:09:39.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.534+0000 7f162e277700 1 -- 172.21.15.104:0/3722548474 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f16100390f0 msgr2=0x7f161003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:39.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.534+0000 7f162e277700 1 --2- 172.21.15.104:0/3722548474 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f16100390f0 0x7f161003b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f161c006fd0 tx=0x7f161c006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:39.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.534+0000 7f162e277700 1 -- 172.21.15.104:0/3722548474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16281029c0 msgr2=0x7f1628102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:39.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.534+0000 7f162e277700 1 --2- 172.21.15.104:0/3722548474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16281029c0 0x7f1628102240 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1618005b40 tx=0x7f1618005030 comp rx=0 tx=0).stop 2024-06-09T22:09:39.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.534+0000 7f162e277700 1 -- 172.21.15.104:0/3722548474 shutdown_connections 2024-06-09T22:09:39.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.534+0000 7f162e277700 1 --2- 172.21.15.104:0/3722548474 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f16100390f0 0x7f161003b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:39.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.534+0000 7f162e277700 1 --2- 172.21.15.104:0/3722548474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16281029c0 0x7f1628102240 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:39.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.534+0000 7f162e277700 1 -- 172.21.15.104:0/3722548474 >> 172.21.15.104:0/3722548474 conn(0x7f16280fc3f0 msgr2=0x7f16280fce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:39.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.534+0000 7f162e277700 1 -- 172.21.15.104:0/3722548474 shutdown_connections 2024-06-09T22:09:39.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:39.534+0000 7f162e277700 1 -- 172.21.15.104:0/3722548474 wait complete. 2024-06-09T22:09:40.218 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-06-09T22:09:40.218 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-09T22:09:40.218 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-06-09T22:09:40.237 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-09T22:09:40.237 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:09:40.295 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-06-09T22:09:40.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch host add smithi153 2024-06-09T22:09:40.533 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:09:40.771 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:40 smithi104 ceph-mon[32389]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:09:40.771 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:40.771 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:09:40.771 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:09:40.771 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:09:40.771 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:40 smithi104 ceph-mon[32389]: Updating smithi104:/etc/ceph/ceph.conf 2024-06-09T22:09:40.771 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:40 smithi104 ceph-mon[32389]: Updating smithi104:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:09:40.771 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:40 smithi104 ceph-mon[32389]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:09:40.772 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:40 smithi104 ceph-mon[32389]: Updating smithi104:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-09T22:09:40.772 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:40.772 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:40.772 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:42.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.126+0000 7fee4928e700 1 -- 172.21.15.104:0/3235504512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee44100fd0 msgr2=0x7fee441013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:42.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.126+0000 7fee4928e700 1 --2- 172.21.15.104:0/3235504512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee44100fd0 0x7fee441013b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fee34009910 tx=0x7fee34009c20 comp rx=0 tx=0).stop 2024-06-09T22:09:42.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.126+0000 7fee4928e700 1 -- 172.21.15.104:0/3235504512 shutdown_connections 2024-06-09T22:09:42.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.126+0000 7fee4928e700 1 --2- 172.21.15.104:0/3235504512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee44100fd0 0x7fee441013b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:42.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.126+0000 7fee4928e700 1 -- 172.21.15.104:0/3235504512 >> 172.21.15.104:0/3235504512 conn(0x7fee44077a60 msgr2=0x7fee440fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:42.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.126+0000 7fee4928e700 1 -- 172.21.15.104:0/3235504512 shutdown_connections 2024-06-09T22:09:42.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.126+0000 7fee4928e700 1 -- 172.21.15.104:0/3235504512 wait complete. 2024-06-09T22:09:42.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.127+0000 7fee4928e700 1 Processor -- start 2024-06-09T22:09:42.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.127+0000 7fee4928e700 1 -- start start 2024-06-09T22:09:42.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.127+0000 7fee4928e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee44100fd0 0x7fee44198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:42.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.127+0000 7fee4928e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fee4410b3a0 con 0x7fee44100fd0 2024-06-09T22:09:42.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.127+0000 7fee42ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee44100fd0 0x7fee44198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:42.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.127+0000 7fee42ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee44100fd0 0x7fee44198820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52226/0 (socket says 172.21.15.104:52226) 2024-06-09T22:09:42.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.128+0000 7fee42ffd700 1 -- 172.21.15.104:0/2118337545 learned_addr learned my addr 172.21.15.104:0/2118337545 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:42.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.128+0000 7fee42ffd700 1 -- 172.21.15.104:0/2118337545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee44198d60 con 0x7fee44100fd0 2024-06-09T22:09:42.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.128+0000 7fee42ffd700 1 --2- 172.21.15.104:0/2118337545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee44100fd0 0x7fee44198820 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fee34005850 tx=0x7fee34005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:42.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.128+0000 7fee40ff9700 1 -- 172.21.15.104:0/2118337545 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee34026070 con 0x7fee44100fd0 2024-06-09T22:09:42.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.128+0000 7fee4928e700 1 -- 172.21.15.104:0/2118337545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee44199050 con 0x7fee44100fd0 2024-06-09T22:09:42.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.128+0000 7fee40ff9700 1 -- 172.21.15.104:0/2118337545 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee3400bbd0 con 0x7fee44100fd0 2024-06-09T22:09:42.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.128+0000 7fee4928e700 1 -- 172.21.15.104:0/2118337545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee44199510 con 0x7fee44100fd0 2024-06-09T22:09:42.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.128+0000 7fee40ff9700 1 -- 172.21.15.104:0/2118337545 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee34003970 con 0x7fee44100fd0 2024-06-09T22:09:42.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.129+0000 7fee40ff9700 1 -- 172.21.15.104:0/2118337545 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fee34029070 con 0x7fee44100fd0 2024-06-09T22:09:42.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.129+0000 7fee4928e700 1 -- 172.21.15.104:0/2118337545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee24005330 con 0x7fee44100fd0 2024-06-09T22:09:42.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.129+0000 7fee40ff9700 1 --2- 172.21.15.104:0/2118337545 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fee300390f0 0x7fee3003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:42.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.129+0000 7fee40ff9700 1 -- 172.21.15.104:0/2118337545 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fee3401e070 con 0x7fee44100fd0 2024-06-09T22:09:42.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.130+0000 7fee3bfff700 1 --2- 172.21.15.104:0/2118337545 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fee300390f0 0x7fee3003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:42.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.130+0000 7fee3bfff700 1 --2- 172.21.15.104:0/2118337545 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fee300390f0 0x7fee3003b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee2c006fd0 tx=0x7fee2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:42.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.135+0000 7fee40ff9700 1 -- 172.21.15.104:0/2118337545 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fee34052070 con 0x7fee44100fd0 2024-06-09T22:09:42.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:42.334+0000 7fee4928e700 1 -- 172.21.15.104:0/2118337545 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}) -- 0x7fee24000bc0 con 0x7fee300390f0 2024-06-09T22:09:42.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:42 smithi104 ceph-mon[32389]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:09:43.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:43 smithi104 ceph-mon[32389]: Deploying cephadm binary to smithi153 2024-06-09T22:09:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:46.493+0000 7fee40ff9700 1 -- 172.21.15.104:0/2118337545 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fee24000bc0 con 0x7fee300390f0 2024-06-09T22:09:46.480 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:46 smithi104 ceph-mon[32389]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:09:46.480 INFO:teuthology.orchestra.run.smithi104.stdout:Added host 'smithi153' with addr '172.21.15.153' 2024-06-09T22:09:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:46.496+0000 7fee4928e700 1 -- 172.21.15.104:0/2118337545 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fee300390f0 msgr2=0x7fee3003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:46.496+0000 7fee4928e700 1 --2- 172.21.15.104:0/2118337545 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fee300390f0 0x7fee3003b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee2c006fd0 tx=0x7fee2c006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:46.496+0000 7fee4928e700 1 -- 172.21.15.104:0/2118337545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee44100fd0 msgr2=0x7fee44198820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:46.496+0000 7fee4928e700 1 --2- 172.21.15.104:0/2118337545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee44100fd0 0x7fee44198820 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fee34005850 tx=0x7fee34005030 comp rx=0 tx=0).stop 2024-06-09T22:09:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:46.496+0000 7fee4928e700 1 -- 172.21.15.104:0/2118337545 shutdown_connections 2024-06-09T22:09:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:46.496+0000 7fee4928e700 1 --2- 172.21.15.104:0/2118337545 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fee300390f0 0x7fee3003b5b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:46.496+0000 7fee4928e700 1 --2- 172.21.15.104:0/2118337545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee44100fd0 0x7fee44198820 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:46.496+0000 7fee4928e700 1 -- 172.21.15.104:0/2118337545 >> 172.21.15.104:0/2118337545 conn(0x7fee44077a60 msgr2=0x7fee441029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:46.496+0000 7fee4928e700 1 -- 172.21.15.104:0/2118337545 shutdown_connections 2024-06-09T22:09:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:46.496+0000 7fee4928e700 1 -- 172.21.15.104:0/2118337545 wait complete. 2024-06-09T22:09:47.120 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-06-09T22:09:47.361 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:09:47.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:47.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:47 smithi104 ceph-mon[32389]: Added host smithi153 2024-06-09T22:09:47.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:09:47.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:47.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:47 smithi104 ceph-mon[32389]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:09:48.627 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:48 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:48.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.861+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/4055517053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5d01020e0 msgr2=0x7fc5d01024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:48.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.861+0000 7fc5d7ffc700 1 --2- 172.21.15.104:0/4055517053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5d01020e0 0x7fc5d01024c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc009910 tx=0x7fc5cc009c20 comp rx=0 tx=0).stop 2024-06-09T22:09:48.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.861+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/4055517053 shutdown_connections 2024-06-09T22:09:48.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.861+0000 7fc5d7ffc700 1 --2- 172.21.15.104:0/4055517053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5d01020e0 0x7fc5d01024c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:48.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.861+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/4055517053 >> 172.21.15.104:0/4055517053 conn(0x7fc5d00fd9b0 msgr2=0x7fc5d00ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:48.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.861+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/4055517053 shutdown_connections 2024-06-09T22:09:48.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.861+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/4055517053 wait complete. 2024-06-09T22:09:48.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.862+0000 7fc5d7ffc700 1 Processor -- start 2024-06-09T22:09:48.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.862+0000 7fc5d7ffc700 1 -- start start 2024-06-09T22:09:48.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.862+0000 7fc5d7ffc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5d01020e0 0x7fc5d019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:48.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.862+0000 7fc5d7ffc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d010c4b0 con 0x7fc5d01020e0 2024-06-09T22:09:48.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.862+0000 7fc5d5d98700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5d01020e0 0x7fc5d019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:48.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.862+0000 7fc5d5d98700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5d01020e0 0x7fc5d019d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60432/0 (socket says 172.21.15.104:60432) 2024-06-09T22:09:48.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.862+0000 7fc5d5d98700 1 -- 172.21.15.104:0/403422206 learned_addr learned my addr 172.21.15.104:0/403422206 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:48.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.863+0000 7fc5d5d98700 1 -- 172.21.15.104:0/403422206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d019dbb0 con 0x7fc5d01020e0 2024-06-09T22:09:48.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.863+0000 7fc5d5d98700 1 --2- 172.21.15.104:0/403422206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5d01020e0 0x7fc5d019d670 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc005850 tx=0x7fc5cc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:48.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.863+0000 7fc5c6ffd700 1 -- 172.21.15.104:0/403422206 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5cc026070 con 0x7fc5d01020e0 2024-06-09T22:09:48.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.863+0000 7fc5c6ffd700 1 -- 172.21.15.104:0/403422206 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5cc00bbd0 con 0x7fc5d01020e0 2024-06-09T22:09:48.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.863+0000 7fc5c6ffd700 1 -- 172.21.15.104:0/403422206 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5cc003970 con 0x7fc5d01020e0 2024-06-09T22:09:48.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.863+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/403422206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d019dea0 con 0x7fc5d01020e0 2024-06-09T22:09:48.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.863+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/403422206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d01977f0 con 0x7fc5d01020e0 2024-06-09T22:09:48.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.864+0000 7fc5c6ffd700 1 -- 172.21.15.104:0/403422206 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fc5cc029070 con 0x7fc5d01020e0 2024-06-09T22:09:48.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.864+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/403422206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5d01035c0 con 0x7fc5d01020e0 2024-06-09T22:09:48.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.864+0000 7fc5c6ffd700 1 --2- 172.21.15.104:0/403422206 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc5bc039150 0x7fc5bc03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:48.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.864+0000 7fc5c6ffd700 1 -- 172.21.15.104:0/403422206 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc5cc01e070 con 0x7fc5d01020e0 2024-06-09T22:09:48.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.864+0000 7fc5d5597700 1 --2- 172.21.15.104:0/403422206 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc5bc039150 0x7fc5bc03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:48.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.864+0000 7fc5d5597700 1 --2- 172.21.15.104:0/403422206 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc5bc039150 0x7fc5bc03b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0006fd0 tx=0x7fc5c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:48.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:48.869+0000 7fc5c6ffd700 1 -- 172.21.15.104:0/403422206 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc5cc050030 con 0x7fc5d01020e0 2024-06-09T22:09:49.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:49.071+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/403422206 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc5d0198d20 con 0x7fc5bc039150 2024-06-09T22:09:49.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:49.072+0000 7fc5c6ffd700 1 -- 172.21.15.104:0/403422206 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7fc5d0198d20 con 0x7fc5bc039150 2024-06-09T22:09:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:09:49.059 INFO:teuthology.orchestra.run.smithi104.stdout:[{"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-06-09T22:09:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:49.074+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/403422206 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc5bc039150 msgr2=0x7fc5bc03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:49.074+0000 7fc5d7ffc700 1 --2- 172.21.15.104:0/403422206 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc5bc039150 0x7fc5bc03b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0006fd0 tx=0x7fc5c0006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:49.074+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/403422206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5d01020e0 msgr2=0x7fc5d019d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:49.074+0000 7fc5d7ffc700 1 --2- 172.21.15.104:0/403422206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5d01020e0 0x7fc5d019d670 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc005850 tx=0x7fc5cc005030 comp rx=0 tx=0).stop 2024-06-09T22:09:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:49.074+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/403422206 shutdown_connections 2024-06-09T22:09:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:49.074+0000 7fc5d7ffc700 1 --2- 172.21.15.104:0/403422206 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc5bc039150 0x7fc5bc03b610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:49.074+0000 7fc5d7ffc700 1 --2- 172.21.15.104:0/403422206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5d01020e0 0x7fc5d019d670 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:49.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:49.074+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/403422206 >> 172.21.15.104:0/403422206 conn(0x7fc5d00fd9b0 msgr2=0x7fc5d0103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:49.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:49.074+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/403422206 shutdown_connections 2024-06-09T22:09:49.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:49.074+0000 7fc5d7ffc700 1 -- 172.21.15.104:0/403422206 wait complete. 2024-06-09T22:09:49.583 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi194 2024-06-09T22:09:49.583 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-06-09T22:09:49.584 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.conf 2024-06-09T22:09:49.603 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-06-09T22:09:49.603 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:09:49.660 INFO:tasks.cephadm:Adding host smithi194 to orchestrator... 2024-06-09T22:09:49.660 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch host add smithi194 2024-06-09T22:09:49.893 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:09:50.301 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:50 smithi104 ceph-mon[32389]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:09:50.301 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:50 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:50.301 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:50 smithi104 ceph-mon[32389]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:09:51.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.358+0000 7f9f72f81700 1 -- 172.21.15.104:0/1743722969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f6c101000 msgr2=0x7f9f6c1013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:51.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.358+0000 7f9f72f81700 1 --2- 172.21.15.104:0/1743722969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f6c101000 0x7f9f6c1013e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009910 tx=0x7f9f60009c20 comp rx=0 tx=0).stop 2024-06-09T22:09:51.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.358+0000 7f9f72f81700 1 -- 172.21.15.104:0/1743722969 shutdown_connections 2024-06-09T22:09:51.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.358+0000 7f9f72f81700 1 --2- 172.21.15.104:0/1743722969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f6c101000 0x7f9f6c1013e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:51.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.358+0000 7f9f72f81700 1 -- 172.21.15.104:0/1743722969 >> 172.21.15.104:0/1743722969 conn(0x7f9f6c0fc8b0 msgr2=0x7f9f6c0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:51.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.358+0000 7f9f72f81700 1 -- 172.21.15.104:0/1743722969 shutdown_connections 2024-06-09T22:09:51.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.358+0000 7f9f72f81700 1 -- 172.21.15.104:0/1743722969 wait complete. 2024-06-09T22:09:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.359+0000 7f9f72f81700 1 Processor -- start 2024-06-09T22:09:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.359+0000 7f9f72f81700 1 -- start start 2024-06-09T22:09:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.359+0000 7f9f72f81700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f6c101000 0x7f9f6c1988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.359+0000 7f9f72f81700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6c10b3d0 con 0x7f9f6c101000 2024-06-09T22:09:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.359+0000 7f9f70d1d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f6c101000 0x7f9f6c1988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.359+0000 7f9f70d1d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f6c101000 0x7f9f6c1988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60442/0 (socket says 172.21.15.104:60442) 2024-06-09T22:09:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.359+0000 7f9f70d1d700 1 -- 172.21.15.104:0/593764207 learned_addr learned my addr 172.21.15.104:0/593764207 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.359+0000 7f9f70d1d700 1 -- 172.21.15.104:0/593764207 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f6c198e10 con 0x7f9f6c101000 2024-06-09T22:09:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.360+0000 7f9f70d1d700 1 --2- 172.21.15.104:0/593764207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f6c101000 0x7f9f6c1988d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9f60005850 tx=0x7f9f60005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.360+0000 7f9f69ffb700 1 -- 172.21.15.104:0/593764207 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f60026070 con 0x7f9f6c101000 2024-06-09T22:09:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.360+0000 7f9f69ffb700 1 -- 172.21.15.104:0/593764207 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f6000bbd0 con 0x7f9f6c101000 2024-06-09T22:09:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.360+0000 7f9f72f81700 1 -- 172.21.15.104:0/593764207 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f6c199100 con 0x7f9f6c101000 2024-06-09T22:09:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.360+0000 7f9f69ffb700 1 -- 172.21.15.104:0/593764207 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f60003970 con 0x7f9f6c101000 2024-06-09T22:09:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.360+0000 7f9f72f81700 1 -- 172.21.15.104:0/593764207 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f6c1995c0 con 0x7f9f6c101000 2024-06-09T22:09:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.361+0000 7f9f69ffb700 1 -- 172.21.15.104:0/593764207 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f9f60029070 con 0x7f9f6c101000 2024-06-09T22:09:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.361+0000 7f9f69ffb700 1 --2- 172.21.15.104:0/593764207 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9f5c0390f0 0x7f9f5c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.361+0000 7f9f72f81700 1 -- 172.21.15.104:0/593764207 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f6c1024e0 con 0x7f9f6c101000 2024-06-09T22:09:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.361+0000 7f9f69ffb700 1 -- 172.21.15.104:0/593764207 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9f6001e070 con 0x7f9f6c101000 2024-06-09T22:09:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.361+0000 7f9f6bfff700 1 --2- 172.21.15.104:0/593764207 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9f5c0390f0 0x7f9f5c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.361+0000 7f9f6bfff700 1 --2- 172.21.15.104:0/593764207 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9f5c0390f0 0x7f9f5c03b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9f58006fd0 tx=0x7f9f58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:51.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.366+0000 7f9f69ffb700 1 -- 172.21.15.104:0/593764207 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9f60013070 con 0x7f9f6c101000 2024-06-09T22:09:51.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:51.569+0000 7f9f72f81700 1 -- 172.21.15.104:0/593764207 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}) -- 0x7f9f6c06a840 con 0x7f9f5c0390f0 2024-06-09T22:09:52.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:52 smithi104 ceph-mon[32389]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:09:52.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:52 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:52.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:52 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:52.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:52 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:52.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:52 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:52.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:52 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:09:52.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:52 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:09:52.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:52 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:09:53.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:53 smithi104 ceph-mon[32389]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:09:53.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:53 smithi104 ceph-mon[32389]: Deploying cephadm binary to smithi194 2024-06-09T22:09:53.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:53 smithi104 ceph-mon[32389]: Updating smithi153:/etc/ceph/ceph.conf 2024-06-09T22:09:54.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:54 smithi104 ceph-mon[32389]: Updating smithi153:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:09:54.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:54 smithi104 ceph-mon[32389]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:09:54.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:54 smithi104 ceph-mon[32389]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:09:55.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:55.737+0000 7f9f69ffb700 1 -- 172.21.15.104:0/593764207 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f9f6c06a840 con 0x7f9f5c0390f0 2024-06-09T22:09:55.725 INFO:teuthology.orchestra.run.smithi104.stdout:Added host 'smithi194' with addr '172.21.15.194' 2024-06-09T22:09:55.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:55.739+0000 7f9f72f81700 1 -- 172.21.15.104:0/593764207 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9f5c0390f0 msgr2=0x7f9f5c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:55.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:55.739+0000 7f9f72f81700 1 --2- 172.21.15.104:0/593764207 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9f5c0390f0 0x7f9f5c03b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9f58006fd0 tx=0x7f9f58006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:55.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:55.739+0000 7f9f72f81700 1 -- 172.21.15.104:0/593764207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f6c101000 msgr2=0x7f9f6c1988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:55.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:55.739+0000 7f9f72f81700 1 --2- 172.21.15.104:0/593764207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f6c101000 0x7f9f6c1988d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9f60005850 tx=0x7f9f60005030 comp rx=0 tx=0).stop 2024-06-09T22:09:55.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:55.740+0000 7f9f72f81700 1 -- 172.21.15.104:0/593764207 shutdown_connections 2024-06-09T22:09:55.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:55.740+0000 7f9f72f81700 1 --2- 172.21.15.104:0/593764207 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9f5c0390f0 0x7f9f5c03b5b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:55.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:55.740+0000 7f9f72f81700 1 --2- 172.21.15.104:0/593764207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f6c101000 0x7f9f6c1988d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:55.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:55.740+0000 7f9f72f81700 1 -- 172.21.15.104:0/593764207 >> 172.21.15.104:0/593764207 conn(0x7f9f6c0fc8b0 msgr2=0x7f9f6c103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:55.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:55.740+0000 7f9f72f81700 1 -- 172.21.15.104:0/593764207 shutdown_connections 2024-06-09T22:09:55.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:55.740+0000 7f9f72f81700 1 -- 172.21.15.104:0/593764207 wait complete. 2024-06-09T22:09:56.286 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:56 smithi104 ceph-mon[32389]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:09:56.286 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:56 smithi104 ceph-mon[32389]: Updating smithi153:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-09T22:09:56.286 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:56.286 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:56.286 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:56.286 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:56.286 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:09:56.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-06-09T22:09:56.544 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:09:57.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:57 smithi104 ceph-mon[32389]: Added host smithi194 2024-06-09T22:09:57.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:57.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:09:58.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.050+0000 7f82d77b5700 1 -- 172.21.15.104:0/1401517911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d01020d0 msgr2=0x7f82d01024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:58.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.050+0000 7f82d77b5700 1 --2- 172.21.15.104:0/1401517911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d01020d0 0x7f82d01024b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009910 tx=0x7f82cc009c20 comp rx=0 tx=0).stop 2024-06-09T22:09:58.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.051+0000 7f82d77b5700 1 -- 172.21.15.104:0/1401517911 shutdown_connections 2024-06-09T22:09:58.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.051+0000 7f82d77b5700 1 --2- 172.21.15.104:0/1401517911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d01020d0 0x7f82d01024b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:58.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.051+0000 7f82d77b5700 1 -- 172.21.15.104:0/1401517911 >> 172.21.15.104:0/1401517911 conn(0x7f82d00fd980 msgr2=0x7f82d00ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:58.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.051+0000 7f82d77b5700 1 -- 172.21.15.104:0/1401517911 shutdown_connections 2024-06-09T22:09:58.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.051+0000 7f82d77b5700 1 -- 172.21.15.104:0/1401517911 wait complete. 2024-06-09T22:09:58.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.051+0000 7f82d77b5700 1 Processor -- start 2024-06-09T22:09:58.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.051+0000 7f82d77b5700 1 -- start start 2024-06-09T22:09:58.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.052+0000 7f82d77b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d01020d0 0x7f82d019d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:58.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.052+0000 7f82d77b5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f82d010c4a0 con 0x7f82d01020d0 2024-06-09T22:09:58.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.052+0000 7f82d5551700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d01020d0 0x7f82d019d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:58.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.052+0000 7f82d5551700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d01020d0 0x7f82d019d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38918/0 (socket says 172.21.15.104:38918) 2024-06-09T22:09:58.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.052+0000 7f82d5551700 1 -- 172.21.15.104:0/2559917041 learned_addr learned my addr 172.21.15.104:0/2559917041 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:09:58.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.052+0000 7f82d5551700 1 -- 172.21.15.104:0/2559917041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d019dbf0 con 0x7f82d01020d0 2024-06-09T22:09:58.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.052+0000 7f82d5551700 1 --2- 172.21.15.104:0/2559917041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d01020d0 0x7f82d019d6b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f82cc005850 tx=0x7f82cc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:58.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.052+0000 7f82c67fc700 1 -- 172.21.15.104:0/2559917041 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82cc026070 con 0x7f82d01020d0 2024-06-09T22:09:58.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.052+0000 7f82c67fc700 1 -- 172.21.15.104:0/2559917041 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82cc00bbd0 con 0x7f82d01020d0 2024-06-09T22:09:58.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.052+0000 7f82d77b5700 1 -- 172.21.15.104:0/2559917041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d019dee0 con 0x7f82d01020d0 2024-06-09T22:09:58.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.052+0000 7f82c67fc700 1 -- 172.21.15.104:0/2559917041 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82cc003970 con 0x7f82d01020d0 2024-06-09T22:09:58.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.052+0000 7f82d77b5700 1 -- 172.21.15.104:0/2559917041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d0197870 con 0x7f82d01020d0 2024-06-09T22:09:58.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.053+0000 7f82c67fc700 1 -- 172.21.15.104:0/2559917041 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f82cc029070 con 0x7f82d01020d0 2024-06-09T22:09:58.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.053+0000 7f82d77b5700 1 -- 172.21.15.104:0/2559917041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d01035b0 con 0x7f82d01020d0 2024-06-09T22:09:58.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.053+0000 7f82c67fc700 1 --2- 172.21.15.104:0/2559917041 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f82bc039150 0x7f82bc03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:09:58.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.053+0000 7f82c67fc700 1 -- 172.21.15.104:0/2559917041 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f82cc01e070 con 0x7f82d01020d0 2024-06-09T22:09:58.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.053+0000 7f82d4d50700 1 --2- 172.21.15.104:0/2559917041 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f82bc039150 0x7f82bc03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:09:58.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.054+0000 7f82d4d50700 1 --2- 172.21.15.104:0/2559917041 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f82bc039150 0x7f82bc03b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f82c0006fd0 tx=0x7f82c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:09:58.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.058+0000 7f82c67fc700 1 -- 172.21.15.104:0/2559917041 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f82cc013070 con 0x7f82d01020d0 2024-06-09T22:09:58.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.261+0000 7f82d77b5700 1 -- 172.21.15.104:0/2559917041 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f82d0198da0 con 0x7f82bc039150 2024-06-09T22:09:58.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.262+0000 7f82c67fc700 1 -- 172.21.15.104:0/2559917041 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f82d0198da0 con 0x7f82bc039150 2024-06-09T22:09:58.251 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:09:58.251 INFO:teuthology.orchestra.run.smithi104.stdout:[{"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}, {"addr": "172.21.15.194", "hostname": "smithi194", "labels": [], "status": ""}] 2024-06-09T22:09:58.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.265+0000 7f82d77b5700 1 -- 172.21.15.104:0/2559917041 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f82bc039150 msgr2=0x7f82bc03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.265+0000 7f82d77b5700 1 --2- 172.21.15.104:0/2559917041 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f82bc039150 0x7f82bc03b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f82c0006fd0 tx=0x7f82c0006e40 comp rx=0 tx=0).stop 2024-06-09T22:09:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.265+0000 7f82d77b5700 1 -- 172.21.15.104:0/2559917041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d01020d0 msgr2=0x7f82d019d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:09:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.265+0000 7f82d77b5700 1 --2- 172.21.15.104:0/2559917041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d01020d0 0x7f82d019d6b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f82cc005850 tx=0x7f82cc005030 comp rx=0 tx=0).stop 2024-06-09T22:09:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.265+0000 7f82d77b5700 1 -- 172.21.15.104:0/2559917041 shutdown_connections 2024-06-09T22:09:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.265+0000 7f82d77b5700 1 --2- 172.21.15.104:0/2559917041 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f82bc039150 0x7f82bc03b610 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.265+0000 7f82d77b5700 1 --2- 172.21.15.104:0/2559917041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d01020d0 0x7f82d019d6b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:09:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.265+0000 7f82d77b5700 1 -- 172.21.15.104:0/2559917041 >> 172.21.15.104:0/2559917041 conn(0x7f82d00fd980 msgr2=0x7f82d0104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:09:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.265+0000 7f82d77b5700 1 -- 172.21.15.104:0/2559917041 shutdown_connections 2024-06-09T22:09:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:09:58.265+0000 7f82d77b5700 1 -- 172.21.15.104:0/2559917041 wait complete. 2024-06-09T22:09:58.548 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:58 smithi104 ceph-mon[32389]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:09:58.800 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-09T22:09:58.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-06-09T22:09:59.032 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:09:59.459 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:59 smithi104 ceph-mon[32389]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:09:59.459 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:09:59 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:00.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.321+0000 7f74d5a19700 1 -- 172.21.15.104:0/1170089046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74d0106e00 msgr2=0x7f74d01071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:00.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.321+0000 7f74d5a19700 1 --2- 172.21.15.104:0/1170089046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74d0106e00 0x7f74d01071e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f74c0009910 tx=0x7f74c0009c20 comp rx=0 tx=0).stop 2024-06-09T22:10:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.321+0000 7f74d5a19700 1 -- 172.21.15.104:0/1170089046 shutdown_connections 2024-06-09T22:10:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.321+0000 7f74d5a19700 1 --2- 172.21.15.104:0/1170089046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74d0106e00 0x7f74d01071e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.321+0000 7f74d5a19700 1 -- 172.21.15.104:0/1170089046 >> 172.21.15.104:0/1170089046 conn(0x7f74d0077480 msgr2=0x7f74d0077890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.321+0000 7f74d5a19700 1 -- 172.21.15.104:0/1170089046 shutdown_connections 2024-06-09T22:10:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.321+0000 7f74d5a19700 1 -- 172.21.15.104:0/1170089046 wait complete. 2024-06-09T22:10:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.321+0000 7f74d5a19700 1 Processor -- start 2024-06-09T22:10:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.321+0000 7f74d5a19700 1 -- start start 2024-06-09T22:10:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.322+0000 7f74d5a19700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74d0106e00 0x7f74d0198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.322+0000 7f74d5a19700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f74d010b3b0 con 0x7f74d0106e00 2024-06-09T22:10:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.322+0000 7f74ceffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74d0106e00 0x7f74d0198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.322+0000 7f74ceffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74d0106e00 0x7f74d0198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38940/0 (socket says 172.21.15.104:38940) 2024-06-09T22:10:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.322+0000 7f74ceffd700 1 -- 172.21.15.104:0/45600837 learned_addr learned my addr 172.21.15.104:0/45600837 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:10:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.322+0000 7f74ceffd700 1 -- 172.21.15.104:0/45600837 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d0198d70 con 0x7f74d0106e00 2024-06-09T22:10:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.322+0000 7f74ceffd700 1 --2- 172.21.15.104:0/45600837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74d0106e00 0x7f74d0198830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f74c0005850 tx=0x7f74c0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.322+0000 7f74d4a17700 1 -- 172.21.15.104:0/45600837 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74c0026070 con 0x7f74d0106e00 2024-06-09T22:10:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.322+0000 7f74d4a17700 1 -- 172.21.15.104:0/45600837 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74c000bbd0 con 0x7f74d0106e00 2024-06-09T22:10:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.322+0000 7f74d5a19700 1 -- 172.21.15.104:0/45600837 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d0199060 con 0x7f74d0106e00 2024-06-09T22:10:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.322+0000 7f74d4a17700 1 -- 172.21.15.104:0/45600837 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74c0003970 con 0x7f74d0106e00 2024-06-09T22:10:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.322+0000 7f74d5a19700 1 -- 172.21.15.104:0/45600837 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d0199520 con 0x7f74d0106e00 2024-06-09T22:10:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.323+0000 7f74d5a19700 1 -- 172.21.15.104:0/45600837 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74d01025a0 con 0x7f74d0106e00 2024-06-09T22:10:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.323+0000 7f74d4a17700 1 -- 172.21.15.104:0/45600837 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f74c0029070 con 0x7f74d0106e00 2024-06-09T22:10:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.324+0000 7f74d4a17700 1 --2- 172.21.15.104:0/45600837 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f74b80390f0 0x7f74b803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.324+0000 7f74d4a17700 1 -- 172.21.15.104:0/45600837 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f74c001e070 con 0x7f74d0106e00 2024-06-09T22:10:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.324+0000 7f74ce7fc700 1 --2- 172.21.15.104:0/45600837 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f74b80390f0 0x7f74b803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:00.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.324+0000 7f74ce7fc700 1 --2- 172.21.15.104:0/45600837 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f74b80390f0 0x7f74b803b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006fd0 tx=0x7f74c4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:00.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.328+0000 7f74d4a17700 1 -- 172.21.15.104:0/45600837 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f74c0013070 con 0x7f74d0106e00 2024-06-09T22:10:00.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:00 smithi104 ceph-mon[32389]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:00.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:00.533+0000 7f74d5a19700 1 -- 172.21.15.104:0/45600837 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f74d0100a90 con 0x7f74d0106e00 2024-06-09T22:10:01.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:01.351+0000 7f74d4a17700 1 -- 172.21.15.104:0/45600837 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f74c0030090 con 0x7f74d0106e00 2024-06-09T22:10:01.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:01.355+0000 7f74d5a19700 1 -- 172.21.15.104:0/45600837 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f74b80390f0 msgr2=0x7f74b803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:01.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:01.355+0000 7f74d5a19700 1 --2- 172.21.15.104:0/45600837 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f74b80390f0 0x7f74b803b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006fd0 tx=0x7f74c4006e40 comp rx=0 tx=0).stop 2024-06-09T22:10:01.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:01.355+0000 7f74d5a19700 1 -- 172.21.15.104:0/45600837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74d0106e00 msgr2=0x7f74d0198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:01.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:01.355+0000 7f74d5a19700 1 --2- 172.21.15.104:0/45600837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74d0106e00 0x7f74d0198830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f74c0005850 tx=0x7f74c0005030 comp rx=0 tx=0).stop 2024-06-09T22:10:01.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:01.355+0000 7f74d5a19700 1 -- 172.21.15.104:0/45600837 shutdown_connections 2024-06-09T22:10:01.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:01.355+0000 7f74d5a19700 1 --2- 172.21.15.104:0/45600837 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f74b80390f0 0x7f74b803b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:01.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:01.355+0000 7f74d5a19700 1 --2- 172.21.15.104:0/45600837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74d0106e00 0x7f74d0198830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:01.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:01.355+0000 7f74d5a19700 1 -- 172.21.15.104:0/45600837 >> 172.21.15.104:0/45600837 conn(0x7f74d0077480 msgr2=0x7f74d00fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:01.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:01.355+0000 7f74d5a19700 1 -- 172.21.15.104:0/45600837 shutdown_connections 2024-06-09T22:10:01.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:01.355+0000 7f74d5a19700 1 -- 172.21.15.104:0/45600837 wait complete. 2024-06-09T22:10:01.344 INFO:teuthology.orchestra.run.smithi104.stderr:adjusted tunables profile to default 2024-06-09T22:10:01.668 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:01 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/45600837' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-09T22:10:01.981 INFO:tasks.cephadm:Adding mon.a on smithi104 2024-06-09T22:10:01.981 INFO:tasks.cephadm:Adding mon.b on smithi153 2024-06-09T22:10:01.981 INFO:tasks.cephadm:Adding mon.c on smithi194 2024-06-09T22:10:01.981 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch apply mon '3;smithi104:172.21.15.104=a;smithi153:172.21.15.153=b;smithi194:172.21.15.194=c' 2024-06-09T22:10:02.220 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:10:02.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:02.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/45600837' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-09T22:10:02.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: osdmap e4: 0 total, 0 up, 0 in 2024-06-09T22:10:02.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:02.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:02.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:02.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:02.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:10:02.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:02.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:02.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:02.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:02.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:03.736 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.748+0000 7ff7e2951700 1 -- 172.21.15.194:0/4061487079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7dc1065e0 msgr2=0x7ff7dc1069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:03.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.748+0000 7ff7e2951700 1 --2- 172.21.15.194:0/4061487079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7dc1065e0 0x7ff7dc1069c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009920 tx=0x7ff7cc009c30 comp rx=0 tx=0).stop 2024-06-09T22:10:03.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.748+0000 7ff7e2951700 1 -- 172.21.15.194:0/4061487079 shutdown_connections 2024-06-09T22:10:03.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.748+0000 7ff7e2951700 1 --2- 172.21.15.194:0/4061487079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7dc1065e0 0x7ff7dc1069c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:03.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.748+0000 7ff7e2951700 1 -- 172.21.15.194:0/4061487079 >> 172.21.15.194:0/4061487079 conn(0x7ff7dc077440 msgr2=0x7ff7dc077850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:03.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.748+0000 7ff7e2951700 1 -- 172.21.15.194:0/4061487079 shutdown_connections 2024-06-09T22:10:03.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.748+0000 7ff7e2951700 1 -- 172.21.15.194:0/4061487079 wait complete. 2024-06-09T22:10:03.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.749+0000 7ff7e2951700 1 Processor -- start 2024-06-09T22:10:03.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.749+0000 7ff7e2951700 1 -- start start 2024-06-09T22:10:03.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.749+0000 7ff7e2951700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7dc1065e0 0x7ff7dc196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:03.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.749+0000 7ff7e2951700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc10aba0 con 0x7ff7dc1065e0 2024-06-09T22:10:03.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.749+0000 7ff7dbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7dc1065e0 0x7ff7dc196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:03.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.749+0000 7ff7dbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7dc1065e0 0x7ff7dc196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.194:58206/0 (socket says 172.21.15.194:58206) 2024-06-09T22:10:03.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.749+0000 7ff7dbfff700 1 -- 172.21.15.194:0/4072355025 learned_addr learned my addr 172.21.15.194:0/4072355025 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-06-09T22:10:03.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.750+0000 7ff7dbfff700 1 -- 172.21.15.194:0/4072355025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7dc0fe580 con 0x7ff7dc1065e0 2024-06-09T22:10:03.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.750+0000 7ff7dbfff700 1 --2- 172.21.15.194:0/4072355025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7dc1065e0 0x7ff7dc196530 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc00b5c0 tx=0x7ff7cc0055d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:03.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.750+0000 7ff7d97fa700 1 -- 172.21.15.194:0/4072355025 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7cc026070 con 0x7ff7dc1065e0 2024-06-09T22:10:03.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.750+0000 7ff7e2951700 1 -- 172.21.15.194:0/4072355025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7dc196a70 con 0x7ff7dc1065e0 2024-06-09T22:10:03.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.750+0000 7ff7d97fa700 1 -- 172.21.15.194:0/4072355025 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7cc00bc30 con 0x7ff7dc1065e0 2024-06-09T22:10:03.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.750+0000 7ff7d97fa700 1 -- 172.21.15.194:0/4072355025 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7cc003a40 con 0x7ff7dc1065e0 2024-06-09T22:10:03.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.750+0000 7ff7e2951700 1 -- 172.21.15.194:0/4072355025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7dc196ed0 con 0x7ff7dc1065e0 2024-06-09T22:10:03.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.751+0000 7ff7d97fa700 1 -- 172.21.15.194:0/4072355025 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7ff7cc029070 con 0x7ff7dc1065e0 2024-06-09T22:10:03.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.751+0000 7ff7e2951700 1 -- 172.21.15.194:0/4072355025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7dc101c50 con 0x7ff7dc1065e0 2024-06-09T22:10:03.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.751+0000 7ff7d97fa700 1 --2- 172.21.15.194:0/4072355025 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff7c40390f0 0x7ff7c403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:03.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.751+0000 7ff7d97fa700 1 -- 172.21.15.194:0/4072355025 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff7cc01e070 con 0x7ff7dc1065e0 2024-06-09T22:10:03.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.751+0000 7ff7db7fe700 1 --2- 172.21.15.194:0/4072355025 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff7c40390f0 0x7ff7c403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:03.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.752+0000 7ff7db7fe700 1 --2- 172.21.15.194:0/4072355025 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff7c40390f0 0x7ff7c403b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0006fd0 tx=0x7ff7d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:03.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:03 smithi104 ceph-mon[32389]: Updating smithi194:/etc/ceph/ceph.conf 2024-06-09T22:10:03.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:03 smithi104 ceph-mon[32389]: Updating smithi194:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:10:03.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:03 smithi104 ceph-mon[32389]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:10:03.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:03 smithi104 ceph-mon[32389]: Updating smithi194:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-09T22:10:03.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.757+0000 7ff7d97fa700 1 -- 172.21.15.194:0/4072355025 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff7cc013020 con 0x7ff7dc1065e0 2024-06-09T22:10:03.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.968+0000 7ff7e2951700 1 -- 172.21.15.194:0/4072355025 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi104:172.21.15.104=a;smithi153:172.21.15.153=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}) -- 0x7ff7dc0fdde0 con 0x7ff7c40390f0 2024-06-09T22:10:03.962 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.975+0000 7ff7d97fa700 1 -- 172.21.15.194:0/4072355025 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff7dc0fdde0 con 0x7ff7c40390f0 2024-06-09T22:10:03.962 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mon update... 2024-06-09T22:10:03.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.977+0000 7ff7e2951700 1 -- 172.21.15.194:0/4072355025 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff7c40390f0 msgr2=0x7ff7c403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:03.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.977+0000 7ff7e2951700 1 --2- 172.21.15.194:0/4072355025 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff7c40390f0 0x7ff7c403b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0006fd0 tx=0x7ff7d0006e40 comp rx=0 tx=0).stop 2024-06-09T22:10:03.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.977+0000 7ff7e2951700 1 -- 172.21.15.194:0/4072355025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7dc1065e0 msgr2=0x7ff7dc196530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:03.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.977+0000 7ff7e2951700 1 --2- 172.21.15.194:0/4072355025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7dc1065e0 0x7ff7dc196530 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc00b5c0 tx=0x7ff7cc0055d0 comp rx=0 tx=0).stop 2024-06-09T22:10:03.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.977+0000 7ff7e2951700 1 -- 172.21.15.194:0/4072355025 shutdown_connections 2024-06-09T22:10:03.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.977+0000 7ff7e2951700 1 --2- 172.21.15.194:0/4072355025 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff7c40390f0 0x7ff7c403b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:03.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.977+0000 7ff7e2951700 1 --2- 172.21.15.194:0/4072355025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7dc1065e0 0x7ff7dc196530 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:03.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.977+0000 7ff7e2951700 1 -- 172.21.15.194:0/4072355025 >> 172.21.15.194:0/4072355025 conn(0x7ff7dc077440 msgr2=0x7ff7dc0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:03.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.977+0000 7ff7e2951700 1 -- 172.21.15.194:0/4072355025 shutdown_connections 2024-06-09T22:10:03.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:03.977+0000 7ff7e2951700 1 -- 172.21.15.194:0/4072355025 wait complete. 2024-06-09T22:10:04.570 DEBUG:teuthology.orchestra.run.smithi153:mon.b> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.b.service 2024-06-09T22:10:04.573 DEBUG:teuthology.orchestra.run.smithi194:mon.c> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.c.service 2024-06-09T22:10:04.575 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-09T22:10:04.576 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-09T22:10:04.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:04 smithi104 ceph-mon[32389]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:04.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:04.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:04.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:04.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:04.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:04.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-09T22:10:04.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:05.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:05 smithi104 ceph-mon[32389]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi104:172.21.15.104=a;smithi153:172.21.15.153=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:10:05.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:05 smithi104 ceph-mon[32389]: Saving service mon spec with placement smithi104:172.21.15.104=a;smithi153:172.21.15.153=b;smithi194:172.21.15.194=c;count:3 2024-06-09T22:10:05.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:05 smithi104 ceph-mon[32389]: Deploying daemon mon.c on smithi194 2024-06-09T22:10:06.240 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:10:06.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:06 smithi104 ceph-mon[32389]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:08.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.654+0000 7f35c7fa7700 1 -- 172.21.15.194:0/135895116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c01006e0 msgr2=0x7f35c0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:08.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.654+0000 7f35c7fa7700 1 --2- 172.21.15.194:0/135895116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c01006e0 0x7f35c0100ac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f35b800b0a0 tx=0x7f35b800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:10:08.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.654+0000 7f35c7fa7700 1 -- 172.21.15.194:0/135895116 shutdown_connections 2024-06-09T22:10:08.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.654+0000 7f35c7fa7700 1 --2- 172.21.15.194:0/135895116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c01006e0 0x7f35c0100ac0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:08.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.654+0000 7f35c7fa7700 1 -- 172.21.15.194:0/135895116 >> 172.21.15.194:0/135895116 conn(0x7f35c0077480 msgr2=0x7f35c0077890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:08.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.654+0000 7f35c7fa7700 1 -- 172.21.15.194:0/135895116 shutdown_connections 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.654+0000 7f35c7fa7700 1 -- 172.21.15.194:0/135895116 wait complete. 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.655+0000 7f35c7fa7700 1 Processor -- start 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.655+0000 7f35c7fa7700 1 -- start start 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.655+0000 7f35c7fa7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c01006e0 0x7f35c0101660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.655+0000 7f35c7fa7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f35c010c490 con 0x7f35c01006e0 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.655+0000 7f35c5d43700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c01006e0 0x7f35c0101660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.656+0000 7f35c5d43700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c01006e0 0x7f35c0101660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.194:58236/0 (socket says 172.21.15.194:58236) 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.656+0000 7f35c5d43700 1 -- 172.21.15.194:0/3143039474 learned_addr learned my addr 172.21.15.194:0/3143039474 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.656+0000 7f35c5d43700 1 -- 172.21.15.194:0/3143039474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c0103550 con 0x7f35c01006e0 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.656+0000 7f35c5d43700 1 --2- 172.21.15.194:0/3143039474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c01006e0 0x7f35c0101660 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f35b8015040 tx=0x7f35b8003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.656+0000 7f35b6ffd700 1 -- 172.21.15.194:0/3143039474 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35b8017040 con 0x7f35c01006e0 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.656+0000 7f35c7fa7700 1 -- 172.21.15.194:0/3143039474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c0101c00 con 0x7f35c01006e0 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.656+0000 7f35b6ffd700 1 -- 172.21.15.194:0/3143039474 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35b8022070 con 0x7f35c01006e0 2024-06-09T22:10:08.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.656+0000 7f35b6ffd700 1 -- 172.21.15.194:0/3143039474 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35b801d5d0 con 0x7f35c01006e0 2024-06-09T22:10:08.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.656+0000 7f35c7fa7700 1 -- 172.21.15.194:0/3143039474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c01020c0 con 0x7f35c01006e0 2024-06-09T22:10:08.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.657+0000 7f35b6ffd700 1 -- 172.21.15.194:0/3143039474 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f35b8004000 con 0x7f35c01006e0 2024-06-09T22:10:08.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.657+0000 7f35c7fa7700 1 -- 172.21.15.194:0/3143039474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35c0063af0 con 0x7f35c01006e0 2024-06-09T22:10:08.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.657+0000 7f35b6ffd700 1 --2- 172.21.15.194:0/3143039474 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f35ac0390f0 0x7f35ac03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:08.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.657+0000 7f35b6ffd700 1 -- 172.21.15.194:0/3143039474 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f35b800f050 con 0x7f35c01006e0 2024-06-09T22:10:08.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.657+0000 7f35c5542700 1 --2- 172.21.15.194:0/3143039474 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f35ac0390f0 0x7f35ac03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:08.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.658+0000 7f35c5542700 1 --2- 172.21.15.194:0/3143039474 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f35ac0390f0 0x7f35ac03b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f35b0006fd0 tx=0x7f35b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:08.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.663+0000 7f35b6ffd700 1 -- 172.21.15.194:0/3143039474 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f35b8007e20 con 0x7f35c01006e0 2024-06-09T22:10:08.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:08 smithi104 ceph-mon[32389]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:08.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.954+0000 7f35c7fa7700 1 -- 172.21.15.194:0/3143039474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f35c0103780 con 0x7f35c01006e0 2024-06-09T22:10:08.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.955+0000 7f35b6ffd700 1 -- 172.21.15.194:0/3143039474 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f35b801b3f0 con 0x7f35c01006e0 2024-06-09T22:10:08.943 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:10:08.943 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","modified":"2024-06-09T22:08:19.534976Z","created":"2024-06-09T22:08:19.534976Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-09T22:10:08.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.957+0000 7f35c7fa7700 1 -- 172.21.15.194:0/3143039474 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f35ac0390f0 msgr2=0x7f35ac03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:08.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.957+0000 7f35c7fa7700 1 --2- 172.21.15.194:0/3143039474 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f35ac0390f0 0x7f35ac03b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f35b0006fd0 tx=0x7f35b0006e40 comp rx=0 tx=0).stop 2024-06-09T22:10:08.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.957+0000 7f35c7fa7700 1 -- 172.21.15.194:0/3143039474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c01006e0 msgr2=0x7f35c0101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:08.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.957+0000 7f35c7fa7700 1 --2- 172.21.15.194:0/3143039474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c01006e0 0x7f35c0101660 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f35b8015040 tx=0x7f35b8003d50 comp rx=0 tx=0).stop 2024-06-09T22:10:08.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.957+0000 7f35c7fa7700 1 -- 172.21.15.194:0/3143039474 shutdown_connections 2024-06-09T22:10:08.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.957+0000 7f35c7fa7700 1 --2- 172.21.15.194:0/3143039474 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f35ac0390f0 0x7f35ac03b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:08.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.957+0000 7f35c7fa7700 1 --2- 172.21.15.194:0/3143039474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c01006e0 0x7f35c0101660 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:08.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.957+0000 7f35c7fa7700 1 -- 172.21.15.194:0/3143039474 >> 172.21.15.194:0/3143039474 conn(0x7f35c0077480 msgr2=0x7f35c0103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:08.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.957+0000 7f35c7fa7700 1 -- 172.21.15.194:0/3143039474 shutdown_connections 2024-06-09T22:10:08.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:08.957+0000 7f35c7fa7700 1 -- 172.21.15.194:0/3143039474 wait complete. 2024-06-09T22:10:08.946 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-06-09T22:10:09.297 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:09 smithi194 ceph-mon[36856]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-06-09T22:10:10.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:09 smithi194 bash[36806]: 1e622cac1cc78614d23a20f71f7b142bbccb0a63fc633bbe57e46f2d5415c573 2024-06-09T22:10:10.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:09 smithi194 systemd[1]: Started Ceph mon.c for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:10:10.594 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-09T22:10:10.594 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-09T22:10:10.836 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:10:14.414 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:10:14.414 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:14.414 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: mon.a calling monitor election 2024-06-09T22:10:14.414 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:14.414 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:14.414 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:14.414 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: mon.c calling monitor election 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: monmap epoch 2 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: last_changed 2024-06-09T22:10:09.043177+0000 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: created 2024-06-09T22:08:19.534976+0000 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: min_mon_release 19 (squid) 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: election_strategy: 1 2024-06-09T22:10:14.415 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: 0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-06-09T22:10:14.416 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-06-09T22:10:14.416 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: fsmap 2024-06-09T22:10:14.416 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: osdmap e4: 0 total, 0 up, 0 in 2024-06-09T22:10:14.416 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: mgrmap e13: x(active, since 48s) 2024-06-09T22:10:14.416 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: overall HEALTH_OK 2024-06-09T22:10:14.416 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:14.416 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:14.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:10:14.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:14.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: mon.a calling monitor election 2024-06-09T22:10:14.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:14.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:14.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:14.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: mon.c calling monitor election 2024-06-09T22:10:14.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:14.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:14.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:14.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:14.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: monmap epoch 2 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: last_changed 2024-06-09T22:10:09.043177+0000 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: created 2024-06-09T22:08:19.534976+0000 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: min_mon_release 19 (squid) 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: election_strategy: 1 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: 0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: fsmap 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: osdmap e4: 0 total, 0 up, 0 in 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: mgrmap e13: x(active, since 48s) 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: overall HEALTH_OK 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:14.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:15.175 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.186+0000 7efc7873d700 1 -- 172.21.15.194:0/2254198770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc580035c0 msgr2=0x7efc58005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:15.175 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.186+0000 7efc7873d700 1 --2- 172.21.15.194:0/2254198770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc580035c0 0x7efc58005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efc70101ed0 tx=0x7efc6000feb0 comp rx=0 tx=0).stop 2024-06-09T22:10:15.175 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.186+0000 7efc7873d700 1 -- 172.21.15.194:0/2254198770 shutdown_connections 2024-06-09T22:10:15.175 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.186+0000 7efc7873d700 1 --2- 172.21.15.194:0/2254198770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc580035c0 0x7efc58005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:15.175 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.186+0000 7efc7873d700 1 -- 172.21.15.194:0/2254198770 >> 172.21.15.194:0/2254198770 conn(0x7efc700fda90 msgr2=0x7efc700ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.186+0000 7efc7873d700 1 -- 172.21.15.194:0/2254198770 shutdown_connections 2024-06-09T22:10:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.186+0000 7efc7873d700 1 -- 172.21.15.194:0/2254198770 wait complete. 2024-06-09T22:10:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.187+0000 7efc7873d700 1 Processor -- start 2024-06-09T22:10:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.187+0000 7efc7873d700 1 -- start start 2024-06-09T22:10:15.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc7873d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc580035c0 0x7efc7019d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:15.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc7873d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc7019dcf0 0x7efc70197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:15.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc7873d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efc7010a030 con 0x7efc580035c0 2024-06-09T22:10:15.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc7873d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efc70109eb0 con 0x7efc7019dcf0 2024-06-09T22:10:15.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc75cd8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc7019dcf0 0x7efc70197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:15.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc75cd8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc7019dcf0 0x7efc70197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:50308/0 (socket says 172.21.15.194:50308) 2024-06-09T22:10:15.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc75cd8700 1 -- 172.21.15.194:0/3405563443 learned_addr learned my addr 172.21.15.194:0/3405563443 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc764d9700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc580035c0 0x7efc7019d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc75cd8700 1 -- 172.21.15.194:0/3405563443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc7019dcf0 msgr2=0x7efc70197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc75cd8700 1 -- 172.21.15.194:0/3405563443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc7019dcf0 msgr2=0x7efc70197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc75cd8700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc7019dcf0 0x7efc70197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc75cd8700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc7019dcf0 0x7efc70197970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc764d9700 1 -- 172.21.15.194:0/3405563443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc7019dcf0 msgr2=0x7efc70197970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc764d9700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc7019dcf0 0x7efc70197970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc764d9700 1 -- 172.21.15.194:0/3405563443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7019e330 con 0x7efc580035c0 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.188+0000 7efc764d9700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc580035c0 0x7efc7019d7b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efc70101ed0 tx=0x7efc600055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.189+0000 7efc6f7fe700 1 -- 172.21.15.194:0/3405563443 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7efc60004b60 con 0x7efc580035c0 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.189+0000 7efc6f7fe700 1 -- 172.21.15.194:0/3405563443 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc60013e40 con 0x7efc580035c0 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.189+0000 7efc7873d700 1 -- 172.21.15.194:0/3405563443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc70197eb0 con 0x7efc580035c0 2024-06-09T22:10:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.189+0000 7efc6f7fe700 1 -- 172.21.15.194:0/3405563443 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7efc60011b20 con 0x7efc580035c0 2024-06-09T22:10:15.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.189+0000 7efc7873d700 1 -- 172.21.15.194:0/3405563443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc70198440 con 0x7efc580035c0 2024-06-09T22:10:15.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.190+0000 7efc6f7fe700 1 -- 172.21.15.194:0/3405563443 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7efc60023070 con 0x7efc580035c0 2024-06-09T22:10:15.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.190+0000 7efc6f7fe700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7efc5c039080 0x7efc5c03b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:15.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.190+0000 7efc6f7fe700 1 -- 172.21.15.194:0/3405563443 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efc60020050 con 0x7efc580035c0 2024-06-09T22:10:15.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.190+0000 7efc7873d700 1 -- 172.21.15.194:0/3405563443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc701036e0 con 0x7efc580035c0 2024-06-09T22:10:15.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.190+0000 7efc75cd8700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7efc5c039080 0x7efc5c03b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:15.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.190+0000 7efc75cd8700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7efc5c039080 0x7efc5c03b540 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc64009400 tx=0x7efc64007040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:15.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.195+0000 7efc6f7fe700 1 -- 172.21.15.194:0/3405563443 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7efc60028020 con 0x7efc580035c0 2024-06-09T22:10:15.357 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:15.357 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-09T22:10:15.357 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:15.357 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:15 smithi104 ceph-mon[32389]: Deploying daemon mon.b on smithi153 2024-06-09T22:10:15.358 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:15.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.476+0000 7efc7873d700 1 -- 172.21.15.194:0/3405563443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7efc70198ea0 con 0x7efc580035c0 2024-06-09T22:10:15.466 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:15.466 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-09T22:10:15.466 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:15.466 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:15 smithi194 ceph-mon[36856]: Deploying daemon mon.b on smithi153 2024-06-09T22:10:15.466 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:15.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.477+0000 7efc6f7fe700 1 -- 172.21.15.194:0/3405563443 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7efc6001a390 con 0x7efc580035c0 2024-06-09T22:10:15.467 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:10:15.467 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":2,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","modified":"2024-06-09T22:10:09.043177Z","created":"2024-06-09T22:08:19.534976Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-09T22:10:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.479+0000 7efc7873d700 1 -- 172.21.15.194:0/3405563443 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7efc5c039080 msgr2=0x7efc5c03b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.479+0000 7efc7873d700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7efc5c039080 0x7efc5c03b540 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc64009400 tx=0x7efc64007040 comp rx=0 tx=0).stop 2024-06-09T22:10:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.479+0000 7efc7873d700 1 -- 172.21.15.194:0/3405563443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc580035c0 msgr2=0x7efc7019d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.479+0000 7efc7873d700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc580035c0 0x7efc7019d7b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efc70101ed0 tx=0x7efc600055a0 comp rx=0 tx=0).stop 2024-06-09T22:10:15.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.479+0000 7efc7873d700 1 -- 172.21.15.194:0/3405563443 shutdown_connections 2024-06-09T22:10:15.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.479+0000 7efc7873d700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7efc5c039080 0x7efc5c03b540 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:15.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.479+0000 7efc7873d700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc580035c0 0x7efc7019d7b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:15.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.479+0000 7efc7873d700 1 --2- 172.21.15.194:0/3405563443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc7019dcf0 0x7efc70197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:15.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.479+0000 7efc7873d700 1 -- 172.21.15.194:0/3405563443 >> 172.21.15.194:0/3405563443 conn(0x7efc700fda90 msgr2=0x7efc700feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:15.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.479+0000 7efc7873d700 1 -- 172.21.15.194:0/3405563443 shutdown_connections 2024-06-09T22:10:15.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:15.480+0000 7efc7873d700 1 -- 172.21.15.194:0/3405563443 wait complete. 2024-06-09T22:10:15.469 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 2 2024-06-09T22:10:16.297 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:16 smithi194 ceph-mon[36856]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:16.297 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:16 smithi194 ceph-mon[36856]: from='client.? 172.21.15.194:0/3405563443' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-09T22:10:16.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:16 smithi104 ceph-mon[32389]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:16.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:16 smithi104 ceph-mon[32389]: from='client.? 172.21.15.194:0/3405563443' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-09T22:10:17.055 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-09T22:10:17.056 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-09T22:10:17.300 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:10:23.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:23 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:23.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:10:23.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:23.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: mon.a calling monitor election 2024-06-09T22:10:23.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:23.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: mon.c calling monitor election 2024-06-09T22:10:23.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:23.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:23.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:23.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: mon.b calling monitor election 2024-06-09T22:10:23.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: monmap epoch 3 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: last_changed 2024-06-09T22:10:18.113935+0000 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: created 2024-06-09T22:08:19.534976+0000 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: min_mon_release 19 (squid) 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: election_strategy: 1 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: 0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-06-09T22:10:23.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: fsmap 2024-06-09T22:10:23.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: osdmap e4: 0 total, 0 up, 0 in 2024-06-09T22:10:23.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: mgrmap e13: x(active, since 57s) 2024-06-09T22:10:23.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: overall HEALTH_OK 2024-06-09T22:10:23.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:23.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:23.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:23.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:23.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:10:23.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:23.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: mon.a calling monitor election 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: mon.c calling monitor election 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: mon.b calling monitor election 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: monmap epoch 3 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: last_changed 2024-06-09T22:10:18.113935+0000 2024-06-09T22:10:23.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: created 2024-06-09T22:08:19.534976+0000 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: min_mon_release 19 (squid) 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: election_strategy: 1 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: 0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: fsmap 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: osdmap e4: 0 total, 0 up, 0 in 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: mgrmap e13: x(active, since 57s) 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: overall HEALTH_OK 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:23.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:24.481 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:24 smithi153 ceph-mon[36599]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:24.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:24 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:24.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:24 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:24.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:24 smithi104 ceph-mon[32389]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:24.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:24 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:24.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:24 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:24.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:24 smithi194 ceph-mon[36856]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:24.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:24 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:24.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:24 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:10:24.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.655+0000 7f96d4f18700 1 -- 172.21.15.194:0/2596374147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96bc0062a0 msgr2=0x7f96bc008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:24.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.655+0000 7f96d4f18700 1 --2- 172.21.15.194:0/2596374147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96bc0062a0 0x7f96bc008710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f96d0103070 tx=0x7f96c000fe90 comp rx=0 tx=0).stop 2024-06-09T22:10:24.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.656+0000 7f96d4f18700 1 -- 172.21.15.194:0/2596374147 shutdown_connections 2024-06-09T22:10:24.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.656+0000 7f96d4f18700 1 --2- 172.21.15.194:0/2596374147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96bc0062a0 0x7f96bc008710 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:24.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.656+0000 7f96d4f18700 1 -- 172.21.15.194:0/2596374147 >> 172.21.15.194:0/2596374147 conn(0x7f96d00fdaa0 msgr2=0x7f96d00ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:24.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.656+0000 7f96d4f18700 1 -- 172.21.15.194:0/2596374147 shutdown_connections 2024-06-09T22:10:24.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.656+0000 7f96d4f18700 1 -- 172.21.15.194:0/2596374147 wait complete. 2024-06-09T22:10:24.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.656+0000 7f96d4f18700 1 Processor -- start 2024-06-09T22:10:24.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96d4f18700 1 -- start start 2024-06-09T22:10:24.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96d4f18700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96bc0062a0 0x7f96d0074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:24.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96d4f18700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96d0074cd0 0x7f96d0075150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:24.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96d4f18700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96d01a3ee0 0x7f96d01a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:24.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96d4f18700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f96d0109fd0 con 0x7f96d01a3ee0 2024-06-09T22:10:24.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96d4f18700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96d0109e50 con 0x7f96d0074cd0 2024-06-09T22:10:24.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96d4f18700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f96d010a150 con 0x7f96bc0062a0 2024-06-09T22:10:24.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96ce59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96bc0062a0 0x7f96d0074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96ced9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96d01a3ee0 0x7f96d01a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96ce59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96bc0062a0 0x7f96d0074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:43372/0 (socket says 172.21.15.194:43372) 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96ce59c700 1 -- 172.21.15.194:0/2849144611 learned_addr learned my addr 172.21.15.194:0/2849144611 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96c5d9b700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96d0074cd0 0x7f96d0075150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96ce59c700 1 -- 172.21.15.194:0/2849144611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96d0074cd0 msgr2=0x7f96d0075150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96ce59c700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96d0074cd0 0x7f96d0075150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96ce59c700 1 -- 172.21.15.194:0/2849144611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96d01a3ee0 msgr2=0x7f96d01a62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96ce59c700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96d01a3ee0 0x7f96d01a62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96ce59c700 1 -- 172.21.15.194:0/2849144611 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d01a6a00 con 0x7f96bc0062a0 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.657+0000 7f96ced9d700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96d01a3ee0 0x7f96d01a62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.658+0000 7f96ce59c700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96bc0062a0 0x7f96d0074790 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f96c0015040 tx=0x7f96c0012c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.658+0000 7f96c7fff700 1 -- 172.21.15.194:0/2849144611 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c0011d20 con 0x7f96bc0062a0 2024-06-09T22:10:24.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.658+0000 7f96c7fff700 1 -- 172.21.15.194:0/2849144611 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96c001d410 con 0x7f96bc0062a0 2024-06-09T22:10:24.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.658+0000 7f96d4f18700 1 -- 172.21.15.194:0/2849144611 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d01a6c90 con 0x7f96bc0062a0 2024-06-09T22:10:24.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.658+0000 7f96c7fff700 1 -- 172.21.15.194:0/2849144611 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c00038e0 con 0x7f96bc0062a0 2024-06-09T22:10:24.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.658+0000 7f96d4f18700 1 -- 172.21.15.194:0/2849144611 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d01a71c0 con 0x7f96bc0062a0 2024-06-09T22:10:24.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.659+0000 7f96c7fff700 1 -- 172.21.15.194:0/2849144611 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f96c0003a80 con 0x7f96bc0062a0 2024-06-09T22:10:24.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.659+0000 7f96d4f18700 1 -- 172.21.15.194:0/2849144611 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d0063af0 con 0x7f96bc0062a0 2024-06-09T22:10:24.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.659+0000 7f96c7fff700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f96b0039580 0x7f96b003ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:24.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.659+0000 7f96c7fff700 1 -- 172.21.15.194:0/2849144611 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f96c0050120 con 0x7f96bc0062a0 2024-06-09T22:10:24.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.659+0000 7f96c5d9b700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f96b0039580 0x7f96b003ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:24.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.660+0000 7f96c5d9b700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f96b0039580 0x7f96b003ba40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009ad0 tx=0x7f96b8009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:24.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.664+0000 7f96c7fff700 1 -- 172.21.15.194:0/2849144611 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f96c0003e20 con 0x7f96bc0062a0 2024-06-09T22:10:24.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.966+0000 7f96d4f18700 1 -- 172.21.15.194:0/2849144611 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f96d002de60 con 0x7f96bc0062a0 2024-06-09T22:10:24.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.967+0000 7f96c7fff700 1 -- 172.21.15.194:0/2849144611 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1289 (secure 0 0 0) 0x7f96c000de00 con 0x7f96bc0062a0 2024-06-09T22:10:24.956 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-06-09T22:10:24.956 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":3,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","modified":"2024-06-09T22:10:18.113935Z","created":"2024-06-09T22:08:19.534976Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-06-09T22:10:24.959 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.970+0000 7f96d4f18700 1 -- 172.21.15.194:0/2849144611 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f96b0039580 msgr2=0x7f96b003ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:24.959 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.970+0000 7f96d4f18700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f96b0039580 0x7f96b003ba40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009ad0 tx=0x7f96b8009210 comp rx=0 tx=0).stop 2024-06-09T22:10:24.959 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.970+0000 7f96d4f18700 1 -- 172.21.15.194:0/2849144611 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96bc0062a0 msgr2=0x7f96d0074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:24.959 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.970+0000 7f96d4f18700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96bc0062a0 0x7f96d0074790 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f96c0015040 tx=0x7f96c0012c00 comp rx=0 tx=0).stop 2024-06-09T22:10:24.959 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.970+0000 7f96d4f18700 1 -- 172.21.15.194:0/2849144611 shutdown_connections 2024-06-09T22:10:24.959 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.970+0000 7f96d4f18700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f96b0039580 0x7f96b003ba40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:24.959 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.970+0000 7f96d4f18700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96d01a3ee0 0x7f96d01a62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:24.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.970+0000 7f96d4f18700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96bc0062a0 0x7f96d0074790 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:24.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.970+0000 7f96d4f18700 1 --2- 172.21.15.194:0/2849144611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96d0074cd0 0x7f96d0075150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:24.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.970+0000 7f96d4f18700 1 -- 172.21.15.194:0/2849144611 >> 172.21.15.194:0/2849144611 conn(0x7f96d00fdaa0 msgr2=0x7f96d00fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:24.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.970+0000 7f96d4f18700 1 -- 172.21.15.194:0/2849144611 shutdown_connections 2024-06-09T22:10:24.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:24.970+0000 7f96d4f18700 1 -- 172.21.15.194:0/2849144611 wait complete. 2024-06-09T22:10:24.960 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 3 2024-06-09T22:10:25.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:25.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: Updating smithi104:/etc/ceph/ceph.conf 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: Updating smithi153:/etc/ceph/ceph.conf 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: Updating smithi194:/etc/ceph/ceph.conf 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: Updating smithi153:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: Updating smithi194:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: Updating smithi104:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='client.? 172.21.15.194:0/2849144611' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:25.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: Reconfiguring mon.a (unknown last config time)... 2024-06-09T22:10:25.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-09T22:10:25.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-09T22:10:25.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:25.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: Reconfiguring daemon mon.a on smithi104 2024-06-09T22:10:25.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.604 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-09T22:10:25.604 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-06-09T22:10:25.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: Updating smithi104:/etc/ceph/ceph.conf 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: Updating smithi153:/etc/ceph/ceph.conf 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: Updating smithi194:/etc/ceph/ceph.conf 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: Updating smithi153:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: Updating smithi194:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: Updating smithi104:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='client.? 172.21.15.194:0/2849144611' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:25.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: Reconfiguring mon.a (unknown last config time)... 2024-06-09T22:10:25.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-09T22:10:25.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-09T22:10:25.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:25.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: Reconfiguring daemon mon.a on smithi104 2024-06-09T22:10:25.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: Updating smithi104:/etc/ceph/ceph.conf 2024-06-09T22:10:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: Updating smithi153:/etc/ceph/ceph.conf 2024-06-09T22:10:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: Updating smithi194:/etc/ceph/ceph.conf 2024-06-09T22:10:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: Updating smithi153:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:10:25.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: Updating smithi194:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: Updating smithi104:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='client.? 172.21.15.194:0/2849144611' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: Reconfiguring mon.a (unknown last config time)... 2024-06-09T22:10:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-09T22:10:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-09T22:10:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: Reconfiguring daemon mon.a on smithi104 2024-06-09T22:10:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:26.765 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:10:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:27 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:27 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:27 smithi153 ceph-mon[36599]: Reconfiguring mon.b (monmap changed)... 2024-06-09T22:10:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:27 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-09T22:10:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:27 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-09T22:10:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:27 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:27 smithi153 ceph-mon[36599]: Reconfiguring daemon mon.b on smithi153 2024-06-09T22:10:27.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:27 smithi153 ceph-mon[36599]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:28.028 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:27 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:28.028 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:27 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:28.028 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:27 smithi104 ceph-mon[32389]: Reconfiguring mon.b (monmap changed)... 2024-06-09T22:10:28.028 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:27 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-09T22:10:28.028 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:27 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-09T22:10:28.028 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:27 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:28.028 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:27 smithi104 ceph-mon[32389]: Reconfiguring daemon mon.b on smithi153 2024-06-09T22:10:28.028 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:27 smithi104 ceph-mon[32389]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:28.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:27 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:28.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:27 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:28.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:27 smithi194 ceph-mon[36856]: Reconfiguring mon.b (monmap changed)... 2024-06-09T22:10:28.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:27 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-09T22:10:28.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:27 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-09T22:10:28.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:27 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:28.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:27 smithi194 ceph-mon[36856]: Reconfiguring daemon mon.b on smithi153 2024-06-09T22:10:28.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:27 smithi194 ceph-mon[36856]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:28.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.267+0000 7f551a33a700 1 -- 172.21.15.104:0/2575363468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551406b810 msgr2=0x7f55141070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:28.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.267+0000 7f551a33a700 1 --2- 172.21.15.104:0/2575363468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551406b810 0x7f55141070a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f55080099c0 tx=0x7f5508009cd0 comp rx=0 tx=0).stop 2024-06-09T22:10:28.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.267+0000 7f551a33a700 1 -- 172.21.15.104:0/2575363468 shutdown_connections 2024-06-09T22:10:28.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.267+0000 7f551a33a700 1 --2- 172.21.15.104:0/2575363468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5514107780 0x7f551410f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:28.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.267+0000 7f551a33a700 1 --2- 172.21.15.104:0/2575363468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f551406ae60 0x7f551406b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:28.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.267+0000 7f551a33a700 1 --2- 172.21.15.104:0/2575363468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551406b810 0x7f55141070a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:28.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.267+0000 7f551a33a700 1 -- 172.21.15.104:0/2575363468 >> 172.21.15.104:0/2575363468 conn(0x7f55140773e0 msgr2=0x7f55140777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:28.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.267+0000 7f551a33a700 1 -- 172.21.15.104:0/2575363468 shutdown_connections 2024-06-09T22:10:28.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.267+0000 7f551a33a700 1 -- 172.21.15.104:0/2575363468 wait complete. 2024-06-09T22:10:28.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.267+0000 7f551a33a700 1 Processor -- start 2024-06-09T22:10:28.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.267+0000 7f551a33a700 1 -- start start 2024-06-09T22:10:28.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f551a33a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f551406ae60 0x7f551419e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:28.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f551a33a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551406b810 0x7f551419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:28.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f551a33a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5514107780 0x7f5514198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:28.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f551a33a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5514112780 con 0x7f551406b810 2024-06-09T22:10:28.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f551a33a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5514112600 con 0x7f5514107780 2024-06-09T22:10:28.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f551a33a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5514112900 con 0x7f551406ae60 2024-06-09T22:10:28.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f55137fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551406b810 0x7f551419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:28.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f55137fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551406b810 0x7f551419ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35540/0 (socket says 172.21.15.104:35540) 2024-06-09T22:10:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f55137fe700 1 -- 172.21.15.104:0/4214684851 learned_addr learned my addr 172.21.15.104:0/4214684851 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:10:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f55188d7700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5514107780 0x7f5514198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f5513fff700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f551406ae60 0x7f551419e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f55137fe700 1 -- 172.21.15.104:0/4214684851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f551406ae60 msgr2=0x7f551419e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f55137fe700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f551406ae60 0x7f551419e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f55137fe700 1 -- 172.21.15.104:0/4214684851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5514107780 msgr2=0x7f5514198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f55137fe700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5514107780 0x7f5514198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f55137fe700 1 -- 172.21.15.104:0/4214684851 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5514199180 con 0x7f551406b810 2024-06-09T22:10:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f55188d7700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5514107780 0x7f5514198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:10:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.268+0000 7f55137fe700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551406b810 0x7f551419ed50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5508006010 tx=0x7f5508003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.269+0000 7f55117fa700 1 -- 172.21.15.104:0/4214684851 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5508003680 con 0x7f551406b810 2024-06-09T22:10:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.269+0000 7f55117fa700 1 -- 172.21.15.104:0/4214684851 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f550800f460 con 0x7f551406b810 2024-06-09T22:10:28.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.269+0000 7f551a33a700 1 -- 172.21.15.104:0/4214684851 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5514199410 con 0x7f551406b810 2024-06-09T22:10:28.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.269+0000 7f55117fa700 1 -- 172.21.15.104:0/4214684851 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5508017670 con 0x7f551406b810 2024-06-09T22:10:28.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.269+0000 7f551a33a700 1 -- 172.21.15.104:0/4214684851 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55141a56e0 con 0x7f551406b810 2024-06-09T22:10:28.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.269+0000 7f55117fa700 1 -- 172.21.15.104:0/4214684851 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f5508017810 con 0x7f551406b810 2024-06-09T22:10:28.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.270+0000 7f551a33a700 1 -- 172.21.15.104:0/4214684851 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5514069180 con 0x7f551406b810 2024-06-09T22:10:28.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.270+0000 7f55117fa700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f54fc0394d0 0x7f54fc03b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:28.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.270+0000 7f55117fa700 1 -- 172.21.15.104:0/4214684851 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f550804d1b0 con 0x7f551406b810 2024-06-09T22:10:28.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.270+0000 7f5513fff700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f54fc0394d0 0x7f54fc03b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:28.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.270+0000 7f5513fff700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f54fc0394d0 0x7f54fc03b990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5514106b90 tx=0x7f5504009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:28.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.275+0000 7f55117fa700 1 -- 172.21.15.104:0/4214684851 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5508013070 con 0x7f551406b810 2024-06-09T22:10:28.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.475+0000 7f551a33a700 1 -- 172.21.15.104:0/4214684851 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f551402dca0 con 0x7f551406b810 2024-06-09T22:10:28.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.475+0000 7f55117fa700 1 -- 172.21.15.104:0/4214684851 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7f5508020bf0 con 0x7f551406b810 2024-06-09T22:10:28.464 INFO:teuthology.orchestra.run.smithi104.stdout:# minimal ceph.conf for a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:10:28.464 INFO:teuthology.orchestra.run.smithi104.stdout:[global] 2024-06-09T22:10:28.464 INFO:teuthology.orchestra.run.smithi104.stdout: fsid = a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:10:28.465 INFO:teuthology.orchestra.run.smithi104.stdout: mon_host = [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-06-09T22:10:28.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.478+0000 7f551a33a700 1 -- 172.21.15.104:0/4214684851 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f54fc0394d0 msgr2=0x7f54fc03b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:28.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.478+0000 7f551a33a700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f54fc0394d0 0x7f54fc03b990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5514106b90 tx=0x7f5504009260 comp rx=0 tx=0).stop 2024-06-09T22:10:28.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.478+0000 7f551a33a700 1 -- 172.21.15.104:0/4214684851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551406b810 msgr2=0x7f551419ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:28.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.478+0000 7f551a33a700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551406b810 0x7f551419ed50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5508006010 tx=0x7f5508003eb0 comp rx=0 tx=0).stop 2024-06-09T22:10:28.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.478+0000 7f551a33a700 1 -- 172.21.15.104:0/4214684851 shutdown_connections 2024-06-09T22:10:28.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.478+0000 7f551a33a700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f54fc0394d0 0x7f54fc03b990 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:28.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.478+0000 7f551a33a700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5514107780 0x7f5514198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:28.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.478+0000 7f551a33a700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f551406ae60 0x7f551419e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:28.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.478+0000 7f551a33a700 1 --2- 172.21.15.104:0/4214684851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551406b810 0x7f551419ed50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:28.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.478+0000 7f551a33a700 1 -- 172.21.15.104:0/4214684851 >> 172.21.15.104:0/4214684851 conn(0x7f55140773e0 msgr2=0x7f55141005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:28.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.478+0000 7f551a33a700 1 -- 172.21.15.104:0/4214684851 shutdown_connections 2024-06-09T22:10:28.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:28.478+0000 7f551a33a700 1 -- 172.21.15.104:0/4214684851 wait complete. 2024-06-09T22:10:29.021 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-09T22:10:29.022 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:10:29.022 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.conf 2024-06-09T22:10:29.053 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:10:29.054 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:10:29.119 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-09T22:10:29.120 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-06-09T22:10:29.149 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-09T22:10:29.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:10:29.215 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-06-09T22:10:29.215 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.conf 2024-06-09T22:10:29.242 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-06-09T22:10:29.242 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:10:29.306 INFO:tasks.cephadm:Adding mgr.x on smithi104 2024-06-09T22:10:29.306 INFO:tasks.cephadm:Adding mgr.y on smithi153 2024-06-09T22:10:29.306 INFO:tasks.cephadm:Adding mgr.z on smithi194 2024-06-09T22:10:29.306 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch apply mgr '3;smithi104=x;smithi153=y;smithi194=z' 2024-06-09T22:10:29.481 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:29 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:29.481 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:29 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:29.481 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:29 smithi153 ceph-mon[36599]: Reconfiguring mon.c (monmap changed)... 2024-06-09T22:10:29.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:29 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-09T22:10:29.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:29 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-09T22:10:29.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:29 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:29.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:29 smithi153 ceph-mon[36599]: Reconfiguring daemon mon.c on smithi194 2024-06-09T22:10:29.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:29 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/4214684851' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:29.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:29 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:29.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:29 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:29.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:29 smithi194 ceph-mon[36856]: Reconfiguring mon.c (monmap changed)... 2024-06-09T22:10:29.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:29 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-09T22:10:29.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:29 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-09T22:10:29.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:29 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:29.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:29 smithi194 ceph-mon[36856]: Reconfiguring daemon mon.c on smithi194 2024-06-09T22:10:29.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:29 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/4214684851' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:29.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:29 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:29.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:29 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:29.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:29 smithi104 ceph-mon[32389]: Reconfiguring mon.c (monmap changed)... 2024-06-09T22:10:29.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:29 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-09T22:10:29.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:29 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-09T22:10:29.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:29 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:29.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:29 smithi104 ceph-mon[32389]: Reconfiguring daemon mon.c on smithi194 2024-06-09T22:10:29.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:29 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/4214684851' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:29.906 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:10:30.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:30 smithi104 ceph-mon[32389]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:30.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:30.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:30.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:30.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:30 smithi194 ceph-mon[36856]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:30.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:30.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:30.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:30.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:30 smithi153 ceph-mon[36599]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:30.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:30.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:30.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:31.288 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.297+0000 7f0676cbb700 1 -- 172.21.15.194:0/2400003630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0670101d60 msgr2=0x7f06701021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:31.288 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.297+0000 7f0676cbb700 1 --2- 172.21.15.194:0/2400003630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0670101d60 0x7f06701021c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f065c009990 tx=0x7f065c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:10:31.288 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.297+0000 7f0676cbb700 1 -- 172.21.15.194:0/2400003630 shutdown_connections 2024-06-09T22:10:31.288 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.297+0000 7f0676cbb700 1 --2- 172.21.15.194:0/2400003630 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0670102700 0x7f0670108fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:31.288 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.297+0000 7f0676cbb700 1 --2- 172.21.15.194:0/2400003630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0670100b60 0x7f0670100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:31.288 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.297+0000 7f0676cbb700 1 --2- 172.21.15.194:0/2400003630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0670101d60 0x7f06701021c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.297+0000 7f0676cbb700 1 -- 172.21.15.194:0/2400003630 >> 172.21.15.194:0/2400003630 conn(0x7f06700fc120 msgr2=0x7f06700fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.297+0000 7f0676cbb700 1 -- 172.21.15.194:0/2400003630 shutdown_connections 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.297+0000 7f0676cbb700 1 -- 172.21.15.194:0/2400003630 wait complete. 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0676cbb700 1 Processor -- start 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0676cbb700 1 -- start start 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0676cbb700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0670100b60 0x7f0670198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0676cbb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0670101d60 0x7f0670198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0676cbb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0670102700 0x7f067019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0676cbb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f067010bf00 con 0x7f0670102700 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0676cbb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f067010bd80 con 0x7f0670101d60 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0676cbb700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f067010c080 con 0x7f0670100b60 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0674a57700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0670100b60 0x7f0670198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:31.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0674a57700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0670100b60 0x7f0670198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:39192/0 (socket says 172.21.15.194:39192) 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0674a57700 1 -- 172.21.15.194:0/1774169523 learned_addr learned my addr 172.21.15.194:0/1774169523 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f066ffff700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0670101d60 0x7f0670198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0674a57700 1 -- 172.21.15.194:0/1774169523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0670101d60 msgr2=0x7f0670198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0674a57700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0670101d60 0x7f0670198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0674a57700 1 -- 172.21.15.194:0/1774169523 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0670102700 msgr2=0x7f067019fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0674a57700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0670102700 0x7f067019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f0674a57700 1 -- 172.21.15.194:0/1774169523 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06701a0390 con 0x7f0670100b60 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.298+0000 7f066ffff700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0670101d60 0x7f0670198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.299+0000 7f0674a57700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0670100b60 0x7f0670198730 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f066400dbf0 tx=0x7f06640072a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.299+0000 7f066dffb700 1 -- 172.21.15.194:0/1774169523 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664007de0 con 0x7f0670100b60 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.299+0000 7f066dffb700 1 -- 172.21.15.194:0/1774169523 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f066400e930 con 0x7f0670100b60 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.299+0000 7f0676cbb700 1 -- 172.21.15.194:0/1774169523 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06701a0680 con 0x7f0670100b60 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.299+0000 7f066dffb700 1 -- 172.21.15.194:0/1774169523 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06640169b0 con 0x7f0670100b60 2024-06-09T22:10:31.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.299+0000 7f0676cbb700 1 -- 172.21.15.194:0/1774169523 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06701a0bc0 con 0x7f0670100b60 2024-06-09T22:10:31.291 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.300+0000 7f066dffb700 1 -- 172.21.15.194:0/1774169523 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f066401d020 con 0x7f0670100b60 2024-06-09T22:10:31.291 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.300+0000 7f066dffb700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0660039330 0x7f066003b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:31.291 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.300+0000 7f066dffb700 1 -- 172.21.15.194:0/1774169523 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f066404c370 con 0x7f0670100b60 2024-06-09T22:10:31.291 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.300+0000 7f0676cbb700 1 -- 172.21.15.194:0/1774169523 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0670069180 con 0x7f0670100b60 2024-06-09T22:10:31.291 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.300+0000 7f066ffff700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0660039330 0x7f066003b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:31.291 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.301+0000 7f066ffff700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0660039330 0x7f066003b7f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f065c00b5c0 tx=0x7f065c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:31.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.305+0000 7f066dffb700 1 -- 172.21.15.194:0/1774169523 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f066400a330 con 0x7f0670100b60 2024-06-09T22:10:31.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.510+0000 7f0676cbb700 1 -- 172.21.15.194:0/1774169523 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi104=x;smithi153=y;smithi194=z", "target": ["mon-mgr", ""]}) -- 0x7f0670108930 con 0x7f0660039330 2024-06-09T22:10:31.499 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:31.499 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:31.499 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:31.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.517+0000 7f066dffb700 1 -- 172.21.15.194:0/1774169523 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0670108930 con 0x7f0660039330 2024-06-09T22:10:31.505 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mgr update... 2024-06-09T22:10:31.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.519+0000 7f0676cbb700 1 -- 172.21.15.194:0/1774169523 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0660039330 msgr2=0x7f066003b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:31.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.519+0000 7f0676cbb700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0660039330 0x7f066003b7f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f065c00b5c0 tx=0x7f065c005fb0 comp rx=0 tx=0).stop 2024-06-09T22:10:31.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.519+0000 7f0676cbb700 1 -- 172.21.15.194:0/1774169523 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0670100b60 msgr2=0x7f0670198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:31.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.519+0000 7f0676cbb700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0670100b60 0x7f0670198730 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f066400dbf0 tx=0x7f06640072a0 comp rx=0 tx=0).stop 2024-06-09T22:10:31.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.519+0000 7f0676cbb700 1 -- 172.21.15.194:0/1774169523 shutdown_connections 2024-06-09T22:10:31.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.519+0000 7f0676cbb700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0660039330 0x7f066003b7f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:31.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.519+0000 7f0676cbb700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0670102700 0x7f067019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:31.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.519+0000 7f0676cbb700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0670100b60 0x7f0670198730 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:31.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.519+0000 7f0676cbb700 1 --2- 172.21.15.194:0/1774169523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0670101d60 0x7f0670198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:31.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.519+0000 7f0676cbb700 1 -- 172.21.15.194:0/1774169523 >> 172.21.15.194:0/1774169523 conn(0x7f06700fc120 msgr2=0x7f0670107210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:31.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.519+0000 7f0676cbb700 1 -- 172.21.15.194:0/1774169523 shutdown_connections 2024-06-09T22:10:31.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:10:31.519+0000 7f0676cbb700 1 -- 172.21.15.194:0/1774169523 wait complete. 2024-06-09T22:10:31.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:31.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:31.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:31.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:31.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:31.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:32.059 DEBUG:teuthology.orchestra.run.smithi153:mgr.y> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.y.service 2024-06-09T22:10:32.062 DEBUG:teuthology.orchestra.run.smithi194:mgr.z> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.z.service 2024-06-09T22:10:32.064 INFO:tasks.cephadm:Deploying OSDs... 2024-06-09T22:10:32.064 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:10:32.064 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-06-09T22:10:32.084 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-09T22:10:32.084 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-06-09T22:10:32.142 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-09T22:10:32.143 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T22:10:32.143 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-06-09T22:10:32.143 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:10:32.143 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T22:10:32.143 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-06-09 22:09:34.949188129 +0000 2024-06-09T22:10:32.143 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-06-09 22:04:24.846158334 +0000 2024-06-09T22:10:32.143 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-06-09 22:04:24.846158334 +0000 2024-06-09T22:10:32.143 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-06-09 22:04:24.846158334 +0000 2024-06-09T22:10:32.143 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-09T22:10:32.213 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-06-09T22:10:32.213 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-06-09T22:10:32.213 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000204292 s, 2.5 MB/s 2024-06-09T22:10:32.214 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-09T22:10:32.272 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-06-09T22:10:32.330 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-09T22:10:32.330 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T22:10:32.330 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-06-09T22:10:32.330 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:10:32.330 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T22:10:32.330 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-06-09 22:09:34.980187785 +0000 2024-06-09T22:10:32.330 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-06-09 22:04:25.194154431 +0000 2024-06-09T22:10:32.330 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-06-09 22:04:25.194154431 +0000 2024-06-09T22:10:32.330 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-06-09 22:04:25.194154431 +0000 2024-06-09T22:10:32.331 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-09T22:10:32.396 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-06-09T22:10:32.396 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-06-09T22:10:32.396 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.0001823 s, 2.8 MB/s 2024-06-09T22:10:32.397 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-09T22:10:32.454 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-06-09T22:10:32.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:32 smithi104 ceph-mon[32389]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:32.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:32.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:32.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:32.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:32.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:32.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-09T22:10:32.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-09T22:10:32.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-09T22:10:32.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:32.510 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-09T22:10:32.510 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T22:10:32.510 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-06-09T22:10:32.510 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:10:32.510 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T22:10:32.511 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-06-09 22:09:35.009187463 +0000 2024-06-09T22:10:32.511 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-06-09 22:04:25.528150686 +0000 2024-06-09T22:10:32.511 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-06-09 22:04:25.528150686 +0000 2024-06-09T22:10:32.511 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-06-09 22:04:25.528150686 +0000 2024-06-09T22:10:32.511 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-09T22:10:32.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:32 smithi194 ceph-mon[36856]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:32.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:32.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:32.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:32.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:32.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:32.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-09T22:10:32.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-09T22:10:32.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-09T22:10:32.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:32.575 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-06-09T22:10:32.575 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-06-09T22:10:32.575 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000206799 s, 2.5 MB/s 2024-06-09T22:10:32.576 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-09T22:10:32.632 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-06-09T22:10:32.687 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-09T22:10:32.687 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T22:10:32.688 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-06-09T22:10:32.688 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:10:32.688 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T22:10:32.688 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-06-09 22:09:35.038187140 +0000 2024-06-09T22:10:32.688 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-06-09 22:04:25.856147008 +0000 2024-06-09T22:10:32.688 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-06-09 22:04:25.856147008 +0000 2024-06-09T22:10:32.688 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-06-09 22:04:25.856147008 +0000 2024-06-09T22:10:32.688 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-09T22:10:32.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:32 smithi153 ceph-mon[36599]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:32.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:32.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:32.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:32.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:32.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:32.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-09T22:10:32.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-09T22:10:32.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-09T22:10:32.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:32.750 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-06-09T22:10:32.751 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-06-09T22:10:32.751 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000229828 s, 2.2 MB/s 2024-06-09T22:10:32.752 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-09T22:10:32.808 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-09T22:10:32.808 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-06-09T22:10:32.827 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-09T22:10:32.827 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-06-09T22:10:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-09T22:10:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T22:10:32.885 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-06-09T22:10:32.885 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:10:32.885 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T22:10:32.885 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-06-09 22:10:17.510457914 +0000 2024-06-09T22:10:32.885 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-06-09 22:04:50.824674608 +0000 2024-06-09T22:10:32.885 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-06-09 22:04:50.824674608 +0000 2024-06-09T22:10:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-06-09 22:04:50.824674608 +0000 2024-06-09T22:10:32.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-09T22:10:32.951 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-06-09T22:10:32.951 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-06-09T22:10:32.951 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000289813 s, 1.8 MB/s 2024-06-09T22:10:32.952 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-09T22:10:33.011 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-06-09T22:10:33.067 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-09T22:10:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T22:10:33.068 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-06-09T22:10:33.068 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:10:33.068 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T22:10:33.068 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-06-09 22:10:17.510457914 +0000 2024-06-09T22:10:33.068 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-06-09 22:04:51.150668668 +0000 2024-06-09T22:10:33.068 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-06-09 22:04:51.150668668 +0000 2024-06-09T22:10:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-06-09 22:04:51.150668668 +0000 2024-06-09T22:10:33.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-09T22:10:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-06-09T22:10:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-06-09T22:10:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000209789 s, 2.4 MB/s 2024-06-09T22:10:33.134 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-09T22:10:33.192 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-06-09T22:10:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-09T22:10:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T22:10:33.247 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-06-09T22:10:33.247 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:10:33.247 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T22:10:33.247 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-06-09 22:10:17.511457896 +0000 2024-06-09T22:10:33.247 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-06-09 22:04:51.491662455 +0000 2024-06-09T22:10:33.248 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-06-09 22:04:51.491662455 +0000 2024-06-09T22:10:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-06-09 22:04:51.491662455 +0000 2024-06-09T22:10:33.248 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-09T22:10:33.311 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-06-09T22:10:33.312 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-06-09T22:10:33.312 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000298376 s, 1.7 MB/s 2024-06-09T22:10:33.313 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-09T22:10:33.370 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-06-09T22:10:33.427 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-09T22:10:33.427 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T22:10:33.427 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-06-09T22:10:33.427 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:10:33.427 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T22:10:33.427 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-06-09 22:10:17.511457896 +0000 2024-06-09T22:10:33.427 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-06-09 22:04:51.810656642 +0000 2024-06-09T22:10:33.427 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-06-09 22:04:51.810656642 +0000 2024-06-09T22:10:33.427 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-06-09 22:04:51.810656642 +0000 2024-06-09T22:10:33.427 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-09T22:10:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-06-09T22:10:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-06-09T22:10:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000228691 s, 2.2 MB/s 2024-06-09T22:10:33.492 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-09T22:10:33.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:33 smithi194 ceph-mon[36856]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi104=x;smithi153=y;smithi194=z", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:10:33.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:33 smithi194 ceph-mon[36856]: Saving service mgr spec with placement smithi104=x;smithi153=y;smithi194=z;count:3 2024-06-09T22:10:33.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:33 smithi194 ceph-mon[36856]: Deploying daemon mgr.z on smithi194 2024-06-09T22:10:33.550 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-06-09T22:10:33.550 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-06-09T22:10:33.567 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-09T22:10:33.567 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-06-09T22:10:33.622 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-09T22:10:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T22:10:33.623 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-06-09T22:10:33.623 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:10:33.623 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T22:10:33.623 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-06-09 22:10:33.489636281 +0000 2024-06-09T22:10:33.623 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-06-09 22:04:43.974410815 +0000 2024-06-09T22:10:33.623 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-06-09 22:04:43.974410815 +0000 2024-06-09T22:10:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-06-09 22:04:43.974410815 +0000 2024-06-09T22:10:33.623 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-09T22:10:33.689 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-06-09T22:10:33.689 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-06-09T22:10:33.689 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.00027162 s, 1.9 MB/s 2024-06-09T22:10:33.690 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-09T22:10:33.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:33 smithi153 ceph-mon[36599]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi104=x;smithi153=y;smithi194=z", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:10:33.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:33 smithi153 ceph-mon[36599]: Saving service mgr spec with placement smithi104=x;smithi153=y;smithi194=z;count:3 2024-06-09T22:10:33.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:33 smithi153 ceph-mon[36599]: Deploying daemon mgr.z on smithi194 2024-06-09T22:10:33.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:33 smithi104 ceph-mon[32389]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi104=x;smithi153=y;smithi194=z", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:10:33.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:33 smithi104 ceph-mon[32389]: Saving service mgr spec with placement smithi104=x;smithi153=y;smithi194=z;count:3 2024-06-09T22:10:33.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:33 smithi104 ceph-mon[32389]: Deploying daemon mgr.z on smithi194 2024-06-09T22:10:33.746 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-06-09T22:10:33.797 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:33 smithi194 podman[38513]: 2024-06-09 22:10:33.790311243 +0000 UTC m=+0.142071740 container create 0b98115dc341f2deb77d99544c2f0de42feac370f4740c3fb39bfeffacfd119e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-09T22:10:33.800 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-09T22:10:33.800 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T22:10:33.800 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-06-09T22:10:33.800 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:10:33.801 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T22:10:33.801 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-06-09 22:10:33.489636281 +0000 2024-06-09T22:10:33.801 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-06-09 22:04:44.301404157 +0000 2024-06-09T22:10:33.801 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-06-09 22:04:44.301404157 +0000 2024-06-09T22:10:33.801 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-06-09 22:04:44.301404157 +0000 2024-06-09T22:10:33.801 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-09T22:10:33.863 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-06-09T22:10:33.863 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-06-09T22:10:33.863 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000172981 s, 3.0 MB/s 2024-06-09T22:10:33.864 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-09T22:10:33.936 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-06-09T22:10:33.990 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-09T22:10:33.990 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T22:10:33.990 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-06-09T22:10:33.991 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:10:33.991 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T22:10:33.991 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-06-09 22:10:33.490636261 +0000 2024-06-09T22:10:33.991 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-06-09 22:04:44.623397601 +0000 2024-06-09T22:10:33.991 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-06-09 22:04:44.623397601 +0000 2024-06-09T22:10:33.991 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-06-09 22:04:44.623397601 +0000 2024-06-09T22:10:33.991 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-09T22:10:34.054 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-06-09T22:10:34.054 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-06-09T22:10:34.054 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000198901 s, 2.6 MB/s 2024-06-09T22:10:34.055 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-09T22:10:34.111 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-06-09T22:10:34.122 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:34 smithi194 podman[38513]: 2024-06-09 22:10:34.051191994 +0000 UTC m=+0.402952500 container init 0b98115dc341f2deb77d99544c2f0de42feac370f4740c3fb39bfeffacfd119e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:10:34.122 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:34 smithi194 podman[38513]: 2024-06-09 22:10:34.053997544 +0000 UTC m=+0.405758048 container start 0b98115dc341f2deb77d99544c2f0de42feac370f4740c3fb39bfeffacfd119e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528) 2024-06-09T22:10:34.123 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-mgr[38651]: -- 172.21.15.194:0/3070381182 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e02622e5a0 con 0x55e026202800 2024-06-09T22:10:34.123 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:34 smithi194 bash[38513]: 0b98115dc341f2deb77d99544c2f0de42feac370f4740c3fb39bfeffacfd119e 2024-06-09T22:10:34.139 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-09T22:10:34.139 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T22:10:34.139 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-06-09T22:10:34.139 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:10:34.139 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T22:10:34.139 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-06-09 22:10:33.490636261 +0000 2024-06-09T22:10:34.139 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-06-09 22:04:44.948390984 +0000 2024-06-09T22:10:34.139 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-06-09 22:04:44.948390984 +0000 2024-06-09T22:10:34.139 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-06-09 22:04:44.948390984 +0000 2024-06-09T22:10:34.140 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-09T22:10:34.205 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-06-09T22:10:34.205 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-06-09T22:10:34.205 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000212217 s, 2.4 MB/s 2024-06-09T22:10:34.206 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-09T22:10:34.262 INFO:tasks.cephadm:Deploying osd.0 on smithi104 with /dev/vg_nvme/lv_4... 2024-06-09T22:10:34.263 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-09T22:10:34.388 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:34 smithi194 systemd[1]: Started Ceph mgr.z for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:10:34.388 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:34.210+0000 7fe68068e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-09T22:10:34.388 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:34.399+0000 7fe68068e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:10:34.388 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-mon[36856]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:34.388 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:34.388 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:34.388 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:34.388 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-09T22:10:34.388 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-09T22:10:34.388 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-09T22:10:34.389 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:34.505 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:10:34.626 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:34 smithi104 ceph-mon[32389]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:34.626 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:34 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:34.627 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:34 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:34.627 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:34 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:34.627 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:34 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-09T22:10:34.627 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:34 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-09T22:10:34.627 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:34 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-09T22:10:34.627 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:34 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:34.633 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:34 smithi153 ceph-mon[36599]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:34.633 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:34 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:34.633 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:34 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:34.633 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:34 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:34.633 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:34 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-09T22:10:34.633 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:34 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-09T22:10:34.634 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:34 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-09T22:10:34.634 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:34 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:34.795 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:34.501+0000 7fe68068e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-09T22:10:34.795 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:34.806+0000 7fe68068e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-09T22:10:35.047 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:34 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:34.891+0000 7fe68068e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-09T22:10:35.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:35 smithi153 ceph-mon[36599]: Deploying daemon mgr.y on smithi153 2024-06-09T22:10:35.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:35 smithi194 ceph-mon[36856]: Deploying daemon mgr.y on smithi153 2024-06-09T22:10:35.631 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:35 smithi104 ceph-mon[32389]: Deploying daemon mgr.y on smithi153 2024-06-09T22:10:36.253 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-09T22:10:36.253 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-09T22:10:36.253 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-06-09T22:10:36.253 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-06-09T22:10:36.253 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012875 s, 814 MB/s 2024-06-09T22:10:36.253 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-06-09T22:10:36.365 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:36 smithi153 systemd[1]: Starting Ceph mgr.y for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:10:36.365 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:36 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:36.121+0000 7fe68068e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-09T22:10:36.365 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:36 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:36.294+0000 7fe68068e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-09T22:10:36.629 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:36 smithi104 ceph-mon[32389]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:36.641 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:36 smithi153 ceph-mon[36599]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:36.642 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:36 smithi153 podman[38005]: 2024-06-09 22:10:36.348363461 +0000 UTC m=+0.017218405 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:10:36.642 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:36 smithi153 podman[38005]: 2024-06-09 22:10:36.451663714 +0000 UTC m=+0.120518651 container create 0827a1ba0af3fc1257e8739c8c2ce0d821daeb55a4e097f708754f23cbe478f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:10:36.642 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:36 smithi153 podman[38005]: 2024-06-09 22:10:36.654058527 +0000 UTC m=+0.322913468 container init 0827a1ba0af3fc1257e8739c8c2ce0d821daeb55a4e097f708754f23cbe478f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y, RELEASE=squid-3d89acc, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS) 2024-06-09T22:10:36.751 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:36 smithi194 ceph-mon[36856]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:36.752 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:36 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:36.471+0000 7fe68068e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-09T22:10:36.752 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:36 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:36.763+0000 7fe68068e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-09T22:10:36.802 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_4 2024-06-09T22:10:36.981 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:36 smithi153 podman[38005]: 2024-06-09 22:10:36.656791727 +0000 UTC m=+0.325646671 container start 0827a1ba0af3fc1257e8739c8c2ce0d821daeb55a4e097f708754f23cbe478f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-06-09T22:10:36.982 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:36 smithi153 ceph-mgr[38024]: -- 172.21.15.153:0/820738231 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55839d13c5a0 con 0x55839d110800 2024-06-09T22:10:36.982 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:36 smithi153 bash[38005]: 0827a1ba0af3fc1257e8739c8c2ce0d821daeb55a4e097f708754f23cbe478f2 2024-06-09T22:10:36.982 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:36 smithi153 systemd[1]: Started Ceph mgr.y for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:10:36.982 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:36 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:36.816+0000 7f61cd09b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-09T22:10:37.047 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:36 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:36.859+0000 7fe68068e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-09T22:10:37.051 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:10:37.430 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:37 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:37.004+0000 7f61cd09b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:10:37.430 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:37 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:37.113+0000 7f61cd09b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-09T22:10:37.430 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:37 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:37.441+0000 7f61cd09b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-09T22:10:37.547 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:37 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:37.069+0000 7fe68068e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-09T22:10:37.548 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:37 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:37.275+0000 7fe68068e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:10:37.731 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:37 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:37.538+0000 7f61cd09b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-09T22:10:38.047 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:37 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:37.910+0000 7fe68068e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-09T22:10:38.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:37 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:38.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:37 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:38.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:37 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:38.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:37 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:38.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:37 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:38.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:37 smithi194 ceph-mon[36856]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:38.060 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:37 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:38.060 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:37 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:38.060 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:37 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:38.060 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:37 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:38.060 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:37 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:38.060 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:37 smithi104 ceph-mon[32389]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:38.231 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:37 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:38.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:37 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:38.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:37 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:38.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:37 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:38.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:37 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:38.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:37 smithi153 ceph-mon[36599]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.555+0000 7f73951bc700 1 -- 172.21.15.104:0/1460746561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7390104090 msgr2=0x7f739010a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.555+0000 7f73951bc700 1 --2- 172.21.15.104:0/1460746561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7390104090 0x7f739010a940 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f738800b0d0 tx=0x7f738800b3e0 comp rx=0 tx=0).stop 2024-06-09T22:10:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.555+0000 7f73951bc700 1 -- 172.21.15.104:0/1460746561 shutdown_connections 2024-06-09T22:10:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.555+0000 7f73951bc700 1 --2- 172.21.15.104:0/1460746561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7390104090 0x7f739010a940 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.555+0000 7f73951bc700 1 --2- 172.21.15.104:0/1460746561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73901024f0 0x7f7390102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.555+0000 7f73951bc700 1 --2- 172.21.15.104:0/1460746561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73901036f0 0x7f7390103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.555+0000 7f73951bc700 1 -- 172.21.15.104:0/1460746561 >> 172.21.15.104:0/1460746561 conn(0x7f73900fda90 msgr2=0x7f73900ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.555+0000 7f73951bc700 1 -- 172.21.15.104:0/1460746561 shutdown_connections 2024-06-09T22:10:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.555+0000 7f73951bc700 1 -- 172.21.15.104:0/1460746561 wait complete. 2024-06-09T22:10:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.556+0000 7f73951bc700 1 Processor -- start 2024-06-09T22:10:38.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.556+0000 7f73951bc700 1 -- start start 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.556+0000 7f73951bc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73901024f0 0x7f7390198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.556+0000 7f73951bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73901036f0 0x7f7390198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.556+0000 7f73951bc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7390104090 0x7f739019fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.556+0000 7f73951bc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f739010e0e0 con 0x7f73901036f0 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.556+0000 7f73951bc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f739010df60 con 0x7f7390104090 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.556+0000 7f73951bc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f739010e260 con 0x7f73901024f0 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f738e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73901036f0 0x7f7390198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f738ed9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73901024f0 0x7f7390198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f738ed9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73901024f0 0x7f7390198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:45858/0 (socket says 172.21.15.104:45858) 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f738e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73901036f0 0x7f7390198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35430/0 (socket says 172.21.15.104:35430) 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f738ed9d700 1 -- 172.21.15.104:0/1198367210 learned_addr learned my addr 172.21.15.104:0/1198367210 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f738f59e700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7390104090 0x7f739019fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f738e59c700 1 -- 172.21.15.104:0/1198367210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73901024f0 msgr2=0x7f7390198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f738e59c700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73901024f0 0x7f7390198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f738e59c700 1 -- 172.21.15.104:0/1198367210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7390104090 msgr2=0x7f739019fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f738e59c700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7390104090 0x7f739019fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f738e59c700 1 -- 172.21.15.104:0/1198367210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73901a02f0 con 0x7f73901036f0 2024-06-09T22:10:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f738e59c700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73901036f0 0x7f7390198bd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f738000d980 tx=0x7f738000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f7387fff700 1 -- 172.21.15.104:0/1198367210 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7380017070 con 0x7f73901036f0 2024-06-09T22:10:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f7387fff700 1 -- 172.21.15.104:0/1198367210 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f738000fd70 con 0x7f73901036f0 2024-06-09T22:10:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f73951bc700 1 -- 172.21.15.104:0/1198367210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73901a05e0 con 0x7f73901036f0 2024-06-09T22:10:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f7387fff700 1 -- 172.21.15.104:0/1198367210 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738000e3b0 con 0x7f73901036f0 2024-06-09T22:10:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.557+0000 7f73951bc700 1 -- 172.21.15.104:0/1198367210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73901a0b20 con 0x7f73901036f0 2024-06-09T22:10:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.558+0000 7f7387fff700 1 -- 172.21.15.104:0/1198367210 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f738001a070 con 0x7f73901036f0 2024-06-09T22:10:38.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.558+0000 7f7387fff700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f737c039330 0x7f737c03b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:10:38.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.558+0000 7f73951bc700 1 -- 172.21.15.104:0/1198367210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7390069180 con 0x7f73901036f0 2024-06-09T22:10:38.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.558+0000 7f7387fff700 1 -- 172.21.15.104:0/1198367210 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f738004bb20 con 0x7f73901036f0 2024-06-09T22:10:38.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.558+0000 7f738ed9d700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f737c039330 0x7f737c03b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:10:38.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.559+0000 7f738ed9d700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f737c039330 0x7f737c03b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f737800bd20 tx=0x7f737800b450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:10:38.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.563+0000 7f7387fff700 1 -- 172.21.15.104:0/1198367210 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7380010460 con 0x7f73901036f0 2024-06-09T22:10:38.650 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:38 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:38.356+0000 7fe68068e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-09T22:10:38.650 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:38 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:38.459+0000 7fe68068e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-09T22:10:38.650 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:38 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:38.661+0000 7fe68068e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-09T22:10:38.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:38.784+0000 7f73951bc700 1 -- 172.21.15.104:0/1198367210 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f73900628d0 con 0x7f737c039330 2024-06-09T22:10:39.104 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:38 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:38.779+0000 7f61cd09b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-09T22:10:39.105 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:38 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:38.946+0000 7f61cd09b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-09T22:10:39.105 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:39.116+0000 7f61cd09b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-09T22:10:39.390 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.390 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.390 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:39.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:39.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-09T22:10:39.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-09T22:10:39.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:39.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:10:39.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:10:39.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:39.449 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.449 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.450 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.450 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.450 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:39.450 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:39.450 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.450 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-09T22:10:39.450 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-09T22:10:39.450 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:39.450 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:10:39.450 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:10:39.450 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:39.509 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:39.182+0000 7fe68068e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-09T22:10:39.509 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:39.275+0000 7fe68068e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-09T22:10:39.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:39.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:39.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:39.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-09T22:10:39.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-09T22:10:39.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:39.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:10:39.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:10:39.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:39.688 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:39.401+0000 7f61cd09b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-09T22:10:39.688 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:39.494+0000 7f61cd09b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-09T22:10:39.688 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:39.700+0000 7f61cd09b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-09T22:10:39.797 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:39.520+0000 7fe68068e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-09T22:10:39.798 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:39.656+0000 7fe68068e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-09T22:10:39.798 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:39.754+0000 7fe68068e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-09T22:10:39.981 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:39 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:39.890+0000 7f61cd09b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:10:40.176 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:39 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:39.870+0000 7fe68068e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-09T22:10:40.245 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:40 smithi104 ceph-mon[32389]: Reconfiguring mgr.x (unknown last config time)... 2024-06-09T22:10:40.245 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:40 smithi104 ceph-mon[32389]: Reconfiguring daemon mgr.x on smithi104 2024-06-09T22:10:40.245 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:40 smithi104 ceph-mon[32389]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:10:40.245 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:40 smithi104 ceph-mon[32389]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:40.479 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:40 smithi153 ceph-mon[36599]: Reconfiguring mgr.x (unknown last config time)... 2024-06-09T22:10:40.479 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:40 smithi153 ceph-mon[36599]: Reconfiguring daemon mgr.x on smithi104 2024-06-09T22:10:40.479 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:40 smithi153 ceph-mon[36599]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:10:40.479 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:40 smithi153 ceph-mon[36599]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:40.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:40 smithi194 ceph-mon[36856]: Reconfiguring mgr.x (unknown last config time)... 2024-06-09T22:10:40.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:40 smithi194 ceph-mon[36856]: Reconfiguring daemon mgr.x on smithi104 2024-06-09T22:10:40.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:40 smithi194 ceph-mon[36856]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:10:40.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:40 smithi194 ceph-mon[36856]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:40.548 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:40 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:40.196+0000 7fe68068e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-09T22:10:40.731 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:40 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:40.490+0000 7f61cd09b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-09T22:10:41.047 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:40 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:40.795+0000 7fe68068e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-09T22:10:41.048 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:10:41.011+0000 7fe68068e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-09T22:10:41.195 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:40 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:40.910+0000 7f61cd09b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-09T22:10:41.195 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:41.007+0000 7f61cd09b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-09T22:10:41.195 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:41.206+0000 7f61cd09b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-09T22:10:41.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:41.284+0000 7f7387fff700 1 -- 172.21.15.104:0/1198367210 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 14) ==== 92061+0+0 (secure 0 0 0) 0x7f7380005170 con 0x7f73901036f0 2024-06-09T22:10:41.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:41.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-mon[36599]: Standby manager daemon z started 2024-06-09T22:10:41.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.194:0/3716486173' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-09T22:10:41.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.194:0/3716486173' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-09T22:10:41.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.194:0/3716486173' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-09T22:10:41.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.194:0/3716486173' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-09T22:10:41.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:41.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:41.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:41.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:41.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:41.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:41.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:41.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-mon[36856]: Standby manager daemon z started 2024-06-09T22:10:41.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.194:0/3716486173' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-09T22:10:41.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.194:0/3716486173' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-09T22:10:41.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.194:0/3716486173' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-09T22:10:41.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.194:0/3716486173' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-09T22:10:41.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:41.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:41.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:41.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:41.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:41.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:41.562 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:41.562 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:41 smithi104 ceph-mon[32389]: Standby manager daemon z started 2024-06-09T22:10:41.562 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:41 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.194:0/3716486173' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-09T22:10:41.562 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:41 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.194:0/3716486173' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-09T22:10:41.562 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:41 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.194:0/3716486173' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-09T22:10:41.562 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:41 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.194:0/3716486173' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-09T22:10:41.562 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:41.562 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:41.562 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:41.562 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:41.563 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:41.563 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:41.981 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:41.732+0000 7f61cd09b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-09T22:10:41.982 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:41 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:41.824+0000 7f61cd09b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-09T22:10:42.396 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:42 smithi153 ceph-mon[36599]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:42.396 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:42 smithi153 ceph-mon[36599]: mgrmap e14: x(active, since 76s), standbys: z 2024-06-09T22:10:42.396 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:42 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-09T22:10:42.396 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:42.064+0000 7f61cd09b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-09T22:10:42.396 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:42.197+0000 7f61cd09b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-09T22:10:42.396 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:42.291+0000 7f61cd09b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-09T22:10:42.396 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:42.407+0000 7f61cd09b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-09T22:10:42.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:42 smithi194 ceph-mon[36856]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:42.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:42 smithi194 ceph-mon[36856]: mgrmap e14: x(active, since 76s), standbys: z 2024-06-09T22:10:42.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:42 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-09T22:10:42.619 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:42 smithi104 ceph-mon[32389]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:42.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:42 smithi104 ceph-mon[32389]: mgrmap e14: x(active, since 76s), standbys: z 2024-06-09T22:10:42.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:42 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-09T22:10:42.981 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:42.710+0000 7f61cd09b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-09T22:10:43.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:43 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3678108053' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6"}]: dispatch 2024-06-09T22:10:43.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:43 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3678108053' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6"}]': finished 2024-06-09T22:10:43.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:43 smithi194 ceph-mon[36856]: osdmap e5: 1 total, 0 up, 1 in 2024-06-09T22:10:43.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:43 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:10:43.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:43 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2012852632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:10:43.732 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:43 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:43.305+0000 7f61cd09b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-09T22:10:43.732 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:10:43 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:10:43.519+0000 7f61cd09b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-09T22:10:43.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:43 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3678108053' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6"}]: dispatch 2024-06-09T22:10:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:43 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3678108053' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6"}]': finished 2024-06-09T22:10:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:43 smithi153 ceph-mon[36599]: osdmap e5: 1 total, 0 up, 1 in 2024-06-09T22:10:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:43 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:10:43.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:43 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2012852632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:10:43.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:43 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3678108053' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6"}]: dispatch 2024-06-09T22:10:43.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:43 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3678108053' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6"}]': finished 2024-06-09T22:10:43.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:43 smithi104 ceph-mon[32389]: osdmap e5: 1 total, 0 up, 1 in 2024-06-09T22:10:43.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:43 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:10:43.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:43 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2012852632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:10:44.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:44.302+0000 7f7387fff700 1 -- 172.21.15.104:0/1198367210 <== mon.0 v2:172.21.15.104:3300/0 8 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f738001b3b0 con 0x7f73901036f0 2024-06-09T22:10:44.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:44 smithi194 ceph-mon[36856]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:44.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:44 smithi194 ceph-mon[36856]: Standby manager daemon y started 2024-06-09T22:10:44.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:44 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.153:0/1491448862' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-09T22:10:44.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:44 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.153:0/1491448862' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-09T22:10:44.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:44 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.153:0/1491448862' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-09T22:10:44.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:44 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.153:0/1491448862' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-09T22:10:44.647 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:44 smithi104 ceph-mon[32389]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:44.647 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:44 smithi104 ceph-mon[32389]: Standby manager daemon y started 2024-06-09T22:10:44.647 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:44 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.153:0/1491448862' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-09T22:10:44.647 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:44 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.153:0/1491448862' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-09T22:10:44.647 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:44 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.153:0/1491448862' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-09T22:10:44.648 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:44 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.153:0/1491448862' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-09T22:10:44.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:44 smithi153 ceph-mon[36599]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:44.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:44 smithi153 ceph-mon[36599]: Standby manager daemon y started 2024-06-09T22:10:44.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:44 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.153:0/1491448862' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-09T22:10:44.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:44 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.153:0/1491448862' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-09T22:10:44.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:44 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.153:0/1491448862' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-09T22:10:44.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:44 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.153:0/1491448862' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-09T22:10:45.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:45 smithi194 ceph-mon[36856]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-06-09T22:10:45.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:45 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-09T22:10:45.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:45 smithi153 ceph-mon[36599]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-06-09T22:10:45.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:45 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-09T22:10:45.745 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:45 smithi104 ceph-mon[32389]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-06-09T22:10:45.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:45 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-09T22:10:46.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:46 smithi104 ceph-mon[32389]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:46.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:46 smithi194 ceph-mon[36856]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:46.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:46 smithi153 ceph-mon[36599]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:48.392 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:48 smithi104 ceph-mon[32389]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:48.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:48 smithi194 ceph-mon[36856]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:48.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:48 smithi153 ceph-mon[36599]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:49.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-09T22:10:49.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:49.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-09T22:10:49.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:49.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-09T22:10:49.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:50.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:50 smithi104 ceph-mon[32389]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:50.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:50 smithi104 ceph-mon[32389]: Deploying daemon osd.0 on smithi104 2024-06-09T22:10:50.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:50 smithi153 ceph-mon[36599]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:50.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:50 smithi153 ceph-mon[36599]: Deploying daemon osd.0 on smithi104 2024-06-09T22:10:50.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:50 smithi194 ceph-mon[36856]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:50.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:50 smithi194 ceph-mon[36856]: Deploying daemon osd.0 on smithi104 2024-06-09T22:10:52.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:52 smithi104 ceph-mon[32389]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:52.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:52 smithi153 ceph-mon[36599]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:52.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:52 smithi194 ceph-mon[36856]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:54.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:54 smithi153 ceph-mon[36599]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:54.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:54 smithi104 ceph-mon[32389]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:54.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:54 smithi194 ceph-mon[36856]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:55.914 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:55 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:55.914 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:55 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:55.914 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:55 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:55.914 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:55 smithi104 ceph-mon[32389]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:55 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:55 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:55 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:55.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:55 smithi153 ceph-mon[36599]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:56.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:55 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:56.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:55 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:56.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:55 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:56.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:55 smithi194 ceph-mon[36856]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:57.611 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:57.611 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:57.611 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:57.611 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:57.611 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:57.611 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:57 smithi104 ceph-mon[32389]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:57.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:57.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:57.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:57.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:57.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:57.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:57 smithi194 ceph-mon[36856]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:57.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:57.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:57.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:10:57.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:10:57.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:57.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:57 smithi153 ceph-mon[36599]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:10:58.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.292+0000 7f7387fff700 1 -- 172.21.15.104:0/1198367210 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f73900628d0 con 0x7f737c039330 2024-06-09T22:10:58.282 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 0 on host 'smithi104' 2024-06-09T22:10:58.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.294+0000 7f73951bc700 1 -- 172.21.15.104:0/1198367210 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f737c039330 msgr2=0x7f737c03b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:58.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.294+0000 7f73951bc700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f737c039330 0x7f737c03b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f737800bd20 tx=0x7f737800b450 comp rx=0 tx=0).stop 2024-06-09T22:10:58.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.294+0000 7f73951bc700 1 -- 172.21.15.104:0/1198367210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73901036f0 msgr2=0x7f7390198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:10:58.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.294+0000 7f73951bc700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73901036f0 0x7f7390198bd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f738000d980 tx=0x7f738000dc90 comp rx=0 tx=0).stop 2024-06-09T22:10:58.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.294+0000 7f73951bc700 1 -- 172.21.15.104:0/1198367210 shutdown_connections 2024-06-09T22:10:58.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.294+0000 7f73951bc700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f737c039330 0x7f737c03b7f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:58.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.294+0000 7f73951bc700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7390104090 0x7f739019fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:58.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.294+0000 7f73951bc700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73901024f0 0x7f7390198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:58.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.294+0000 7f73951bc700 1 --2- 172.21.15.104:0/1198367210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73901036f0 0x7f7390198bd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:10:58.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.294+0000 7f73951bc700 1 -- 172.21.15.104:0/1198367210 >> 172.21.15.104:0/1198367210 conn(0x7f73900fda90 msgr2=0x7f73901069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:10:58.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.294+0000 7f73951bc700 1 -- 172.21.15.104:0/1198367210 shutdown_connections 2024-06-09T22:10:58.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:10:58.294+0000 7f73951bc700 1 -- 172.21.15.104:0/1198367210 wait complete. 2024-06-09T22:10:58.842 DEBUG:teuthology.orchestra.run.smithi104:osd.0> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.0.service 2024-06-09T22:10:58.844 INFO:tasks.cephadm:Deploying osd.1 on smithi153 with /dev/vg_nvme/lv_4... 2024-06-09T22:10:58.844 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-09T22:10:59.089 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.b/config 2024-06-09T22:10:59.471 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:59 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:59.472 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:59 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:59.472 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:59 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:59.472 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:10:59 smithi153 ceph-mon[36599]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-09T22:10:59.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:59 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:59.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:59 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:59.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:59 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:59.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:10:59 smithi104 ceph-mon[32389]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-09T22:10:59.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:59 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:59.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:59 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:10:59.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:59 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:10:59.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:10:59 smithi194 ceph-mon[36856]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-09T22:11:00.496 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:00 smithi104 ceph-mon[32389]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:11:00.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:00 smithi104 ceph-mon[32389]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-09T22:11:00.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:00 smithi104 ceph-mon[32389]: osdmap e6: 1 total, 0 up, 1 in 2024-06-09T22:11:00.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:00 smithi104 ceph-mon[32389]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-06-09T22:11:00.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:00 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:00.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:00 smithi194 ceph-mon[36856]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:11:00.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:00 smithi194 ceph-mon[36856]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-09T22:11:00.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:00 smithi194 ceph-mon[36856]: osdmap e6: 1 total, 0 up, 1 in 2024-06-09T22:11:00.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:00 smithi194 ceph-mon[36856]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-06-09T22:11:00.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:00 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:00.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:00 smithi153 ceph-mon[36599]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:11:00.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:00 smithi153 ceph-mon[36599]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-09T22:11:00.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:00 smithi153 ceph-mon[36599]: osdmap e6: 1 total, 0 up, 1 in 2024-06-09T22:11:00.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:00 smithi153 ceph-mon[36599]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-06-09T22:11:00.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:00 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:00.820 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-09T22:11:00.820 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-09T22:11:00.820 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-06-09T22:11:00.820 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-06-09T22:11:00.820 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120925 s, 867 MB/s 2024-06-09T22:11:00.820 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-06-09T22:11:01.096 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:11:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0[44576]: 2024-06-09T22:11:00.832+0000 7f29bd853700 -1 osd.0 0 waiting for initial osdmap 2024-06-09T22:11:01.096 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:11:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0[44576]: 2024-06-09T22:11:00.836+0000 7f29b563e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:11:01.349 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:01 smithi104 ceph-mon[32389]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-06-09T22:11:01.349 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:01 smithi104 ceph-mon[32389]: osdmap e7: 1 total, 0 up, 1 in 2024-06-09T22:11:01.349 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:01 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:01.349 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:01 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:01.349 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:01 smithi104 ceph-mon[32389]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' 2024-06-09T22:11:01.349 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:01 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:01.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-06-09T22:11:01.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:01 smithi153 ceph-mon[36599]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-06-09T22:11:01.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:01 smithi153 ceph-mon[36599]: osdmap e7: 1 total, 0 up, 1 in 2024-06-09T22:11:01.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:01 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:01.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:01 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:01.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:01 smithi153 ceph-mon[36599]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' 2024-06-09T22:11:01.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:01 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:01.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:01 smithi194 ceph-mon[36856]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-06-09T22:11:01.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:01 smithi194 ceph-mon[36856]: osdmap e7: 1 total, 0 up, 1 in 2024-06-09T22:11:01.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:01 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:01.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:01 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:01.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:01 smithi194 ceph-mon[36856]: from='osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505]' entity='osd.0' 2024-06-09T22:11:01.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:01 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:01.648 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.b/config 2024-06-09T22:11:02.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: purged_snaps scrub starts 2024-06-09T22:11:02.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: purged_snaps scrub ok 2024-06-09T22:11:02.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:11:02.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505] boot 2024-06-09T22:11:02.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: osdmap e8: 1 total, 1 up, 1 in 2024-06-09T22:11:02.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:02.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:02.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:02.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:11:02.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:02.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:02.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:02.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:02.604 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: purged_snaps scrub starts 2024-06-09T22:11:02.604 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: purged_snaps scrub ok 2024-06-09T22:11:02.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:11:02.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505] boot 2024-06-09T22:11:02.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: osdmap e8: 1 total, 1 up, 1 in 2024-06-09T22:11:02.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:02.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:02.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:02.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:11:02.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:02.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:02.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:02.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:02.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: purged_snaps scrub starts 2024-06-09T22:11:02.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: purged_snaps scrub ok 2024-06-09T22:11:02.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-09T22:11:02.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: osd.0 [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505] boot 2024-06-09T22:11:02.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: osdmap e8: 1 total, 1 up, 1 in 2024-06-09T22:11:02.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:11:02.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:02.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:02.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:11:02.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:02.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:02.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:02.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:03.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.063+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/3166133366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d941013c0 msgr2=0x7f3d941017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.063+0000 7f3d9bbc9700 1 --2- 172.21.15.153:0/3166133366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d941013c0 0x7f3d941017e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d900099c0 tx=0x7f3d90009cd0 comp rx=0 tx=0).stop 2024-06-09T22:11:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.063+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/3166133366 shutdown_connections 2024-06-09T22:11:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.063+0000 7f3d9bbc9700 1 --2- 172.21.15.153:0/3166133366 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d94102f60 0x7f3d94109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.063+0000 7f3d9bbc9700 1 --2- 172.21.15.153:0/3166133366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d941013c0 0x7f3d941017e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.063+0000 7f3d9bbc9700 1 --2- 172.21.15.153:0/3166133366 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d941025c0 0x7f3d94102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.063+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/3166133366 >> 172.21.15.153:0/3166133366 conn(0x7f3d940fc960 msgr2=0x7f3d940feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:11:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.063+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/3166133366 shutdown_connections 2024-06-09T22:11:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.063+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/3166133366 wait complete. 2024-06-09T22:11:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9bbc9700 1 Processor -- start 2024-06-09T22:11:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9bbc9700 1 -- start start 2024-06-09T22:11:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9bbc9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d941013c0 0x7f3d94197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:11:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9bbc9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d941025c0 0x7f3d94197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:11:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9bbc9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d94102f60 0x7f3d9419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:11:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9bbc9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9410cf80 con 0x7f3d941013c0 2024-06-09T22:11:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9bbc9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9410ce00 con 0x7f3d94102f60 2024-06-09T22:11:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9bbc9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9410d100 con 0x7f3d941025c0 2024-06-09T22:11:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9a166700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d94102f60 0x7f3d9419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:11:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9a166700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d94102f60 0x7f3d9419ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57834/0 (socket says 172.21.15.153:57834) 2024-06-09T22:11:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9a166700 1 -- 172.21.15.153:0/1706809767 learned_addr learned my addr 172.21.15.153:0/1706809767 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-09T22:11:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d99164700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d941025c0 0x7f3d94197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:11:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d99965700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d941013c0 0x7f3d94197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:11:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9a166700 1 -- 172.21.15.153:0/1706809767 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d941025c0 msgr2=0x7f3d94197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9a166700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d941025c0 0x7f3d94197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9a166700 1 -- 172.21.15.153:0/1706809767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d941013c0 msgr2=0x7f3d94197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9a166700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d941013c0 0x7f3d94197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.064+0000 7f3d9a166700 1 -- 172.21.15.153:0/1706809767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9419f2b0 con 0x7f3d94102f60 2024-06-09T22:11:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.065+0000 7f3d9a166700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d94102f60 0x7f3d9419ec10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c007ee0 tx=0x7f3d8c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:11:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.065+0000 7f3d8affd700 1 -- 172.21.15.153:0/1706809767 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c00dbd0 con 0x7f3d94102f60 2024-06-09T22:11:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.065+0000 7f3d8affd700 1 -- 172.21.15.153:0/1706809767 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d8c00dd70 con 0x7f3d94102f60 2024-06-09T22:11:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.065+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/1706809767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9419f5a0 con 0x7f3d94102f60 2024-06-09T22:11:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.065+0000 7f3d8affd700 1 -- 172.21.15.153:0/1706809767 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c005a40 con 0x7f3d94102f60 2024-06-09T22:11:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.065+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/1706809767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d9419fa90 con 0x7f3d94102f60 2024-06-09T22:11:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.066+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/1706809767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d94069180 con 0x7f3d94102f60 2024-06-09T22:11:03.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.067+0000 7f3d8affd700 1 -- 172.21.15.153:0/1706809767 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f3d8c01a020 con 0x7f3d94102f60 2024-06-09T22:11:03.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.068+0000 7f3d8affd700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3d800a2b90 0x7f3d800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:11:03.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.068+0000 7f3d8affd700 1 -- 172.21.15.153:0/1706809767 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7f3d8c0ccca0 con 0x7f3d94102f60 2024-06-09T22:11:03.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.068+0000 7f3d99965700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3d800a2b90 0x7f3d800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:11:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.068+0000 7f3d99965700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3d800a2b90 0x7f3d800a5050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3d90000c00 tx=0x7f3d90005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:11:03.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.072+0000 7f3d8affd700 1 -- 172.21.15.153:0/1706809767 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3d8c010070 con 0x7f3d94102f60 2024-06-09T22:11:03.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:03.276+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/1706809767 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f3d940628d0 con 0x7f3d800a2b90 2024-06-09T22:11:03.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:03 smithi153 ceph-mon[36599]: Detected new or changed devices on smithi104 2024-06-09T22:11:03.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:03 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi104 to 17088M 2024-06-09T22:11:03.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:03 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:11:03.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:03 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:11:03.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:03 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:03.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:03 smithi104 ceph-mon[32389]: Detected new or changed devices on smithi104 2024-06-09T22:11:03.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:03 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi104 to 17088M 2024-06-09T22:11:03.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:03 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:11:03.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:03 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:11:03.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:03 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:03.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:03 smithi194 ceph-mon[36856]: Detected new or changed devices on smithi104 2024-06-09T22:11:03.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:03 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi104 to 17088M 2024-06-09T22:11:03.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:03 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:11:03.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:03 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:11:03.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:03 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:04.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:04 smithi153 ceph-mon[36599]: pgmap v47: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:04.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:04 smithi153 ceph-mon[36599]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:11:04.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:04 smithi153 ceph-mon[36599]: osdmap e9: 1 total, 1 up, 1 in 2024-06-09T22:11:04.700 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:04 smithi194 ceph-mon[36856]: pgmap v47: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:04.700 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:04 smithi194 ceph-mon[36856]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:11:04.700 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:04 smithi194 ceph-mon[36856]: osdmap e9: 1 total, 1 up, 1 in 2024-06-09T22:11:04.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:04 smithi104 ceph-mon[32389]: pgmap v47: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:04.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:04 smithi104 ceph-mon[32389]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:11:04.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:04 smithi104 ceph-mon[32389]: osdmap e9: 1 total, 1 up, 1 in 2024-06-09T22:11:06.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:06 smithi153 ceph-mon[36599]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:06.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:06 smithi153 ceph-mon[36599]: from='client.? 172.21.15.153:0/1604493446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa718c73-5826-43d9-9b5b-c99ddec256da"}]: dispatch 2024-06-09T22:11:06.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:06 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa718c73-5826-43d9-9b5b-c99ddec256da"}]: dispatch 2024-06-09T22:11:06.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:06 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa718c73-5826-43d9-9b5b-c99ddec256da"}]': finished 2024-06-09T22:11:06.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:06 smithi153 ceph-mon[36599]: osdmap e10: 2 total, 1 up, 2 in 2024-06-09T22:11:06.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:06.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:06 smithi104 ceph-mon[32389]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:06.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:06 smithi104 ceph-mon[32389]: from='client.? 172.21.15.153:0/1604493446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa718c73-5826-43d9-9b5b-c99ddec256da"}]: dispatch 2024-06-09T22:11:06.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:06 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa718c73-5826-43d9-9b5b-c99ddec256da"}]: dispatch 2024-06-09T22:11:06.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:06 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa718c73-5826-43d9-9b5b-c99ddec256da"}]': finished 2024-06-09T22:11:06.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:06 smithi104 ceph-mon[32389]: osdmap e10: 2 total, 1 up, 2 in 2024-06-09T22:11:06.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:06.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:06 smithi194 ceph-mon[36856]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:06.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:06 smithi194 ceph-mon[36856]: from='client.? 172.21.15.153:0/1604493446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa718c73-5826-43d9-9b5b-c99ddec256da"}]: dispatch 2024-06-09T22:11:06.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:06 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa718c73-5826-43d9-9b5b-c99ddec256da"}]: dispatch 2024-06-09T22:11:06.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:06 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa718c73-5826-43d9-9b5b-c99ddec256da"}]': finished 2024-06-09T22:11:06.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:06 smithi194 ceph-mon[36856]: osdmap e10: 2 total, 1 up, 2 in 2024-06-09T22:11:06.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:07.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:07 smithi153 ceph-mon[36599]: from='client.? 172.21.15.153:0/4259572124' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:11:07.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:07 smithi104 ceph-mon[32389]: from='client.? 172.21.15.153:0/4259572124' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:11:07.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:07 smithi194 ceph-mon[36856]: from='client.? 172.21.15.153:0/4259572124' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:11:08.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:08 smithi153 ceph-mon[36599]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:08.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:08 smithi104 ceph-mon[32389]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:08.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:08 smithi194 ceph-mon[36856]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:10.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:10 smithi153 ceph-mon[36599]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:10.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:10 smithi104 ceph-mon[32389]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:10.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:10 smithi194 ceph-mon[36856]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:12.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:12 smithi153 ceph-mon[36599]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:12.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:12 smithi104 ceph-mon[32389]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:12.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:12 smithi194 ceph-mon[36856]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:13.595 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-09T22:11:13.595 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:13.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-09T22:11:13.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:13.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-09T22:11:13.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:14.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:14 smithi153 ceph-mon[36599]: Deploying daemon osd.1 on smithi153 2024-06-09T22:11:14.482 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:14 smithi153 ceph-mon[36599]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:14.647 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:14 smithi104 ceph-mon[32389]: Deploying daemon osd.1 on smithi153 2024-06-09T22:11:14.648 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:14 smithi104 ceph-mon[32389]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:14.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:14 smithi194 ceph-mon[36856]: Deploying daemon osd.1 on smithi153 2024-06-09T22:11:14.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:14 smithi194 ceph-mon[36856]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:16.427 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:16 smithi153 ceph-mon[36599]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:16.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:16 smithi104 ceph-mon[32389]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:16.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:16 smithi194 ceph-mon[36856]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:18.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:18 smithi153 ceph-mon[36599]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:18.677 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:18 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:18.677 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:18 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:18.677 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:18 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:11:18.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:18 smithi104 ceph-mon[32389]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:18.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:18 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:18.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:18 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:18.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:18 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:11:18.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:18 smithi194 ceph-mon[36856]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:18.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:18 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:18.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:18 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:18.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:18 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:11:20.479 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:20 smithi153 ceph-mon[36599]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:20.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:20.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:20.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:20.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:20.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:20.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:20 smithi104 ceph-mon[32389]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:20.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:20.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:20.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:20.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:20.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:20.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:20 smithi194 ceph-mon[36856]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:20.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:20.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:20.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:20.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:20.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.638+0000 7f3d8affd700 1 -- 172.21.15.153:0/1706809767 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3d940628d0 con 0x7f3d800a2b90 2024-06-09T22:11:21.628 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 1 on host 'smithi153' 2024-06-09T22:11:21.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.640+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/1706809767 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3d800a2b90 msgr2=0x7f3d800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:21.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.640+0000 7f3d9bbc9700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3d800a2b90 0x7f3d800a5050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3d90000c00 tx=0x7f3d90005fd0 comp rx=0 tx=0).stop 2024-06-09T22:11:21.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.640+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/1706809767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d94102f60 msgr2=0x7f3d9419ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:21.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.640+0000 7f3d9bbc9700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d94102f60 0x7f3d9419ec10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c007ee0 tx=0x7f3d8c00d3b0 comp rx=0 tx=0).stop 2024-06-09T22:11:21.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.641+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/1706809767 shutdown_connections 2024-06-09T22:11:21.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.641+0000 7f3d9bbc9700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3d800a2b90 0x7f3d800a5050 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:21.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.641+0000 7f3d9bbc9700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d941013c0 0x7f3d94197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:21.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.641+0000 7f3d9bbc9700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d94102f60 0x7f3d9419ec10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:21.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.641+0000 7f3d9bbc9700 1 --2- 172.21.15.153:0/1706809767 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d941025c0 0x7f3d94197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:21.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.641+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/1706809767 >> 172.21.15.153:0/1706809767 conn(0x7f3d940fc960 msgr2=0x7f3d94105980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:11:21.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.641+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/1706809767 shutdown_connections 2024-06-09T22:11:21.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:11:21.641+0000 7f3d9bbc9700 1 -- 172.21.15.153:0/1706809767 wait complete. 2024-06-09T22:11:22.314 DEBUG:teuthology.orchestra.run.smithi153:osd.1> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.1.service 2024-06-09T22:11:22.316 INFO:tasks.cephadm:Deploying osd.2 on smithi194 with /dev/vg_nvme/lv_4... 2024-06-09T22:11:22.316 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-09T22:11:22.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:22 smithi153 ceph-mon[36599]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:22.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:22 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:11:22.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:22 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:22.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:22 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:22.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:22 smithi153 ceph-mon[36599]: from='osd.1 [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-09T22:11:22.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:22 smithi153 ceph-mon[36599]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-09T22:11:22.561 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:11:22.683 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:22 smithi194 ceph-mon[36856]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:22.683 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:22 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:11:22.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:22 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:22.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:22 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:22.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:22 smithi194 ceph-mon[36856]: from='osd.1 [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-09T22:11:22.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:22 smithi194 ceph-mon[36856]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-09T22:11:22.746 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:22 smithi104 ceph-mon[32389]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:22.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:22 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:11:22.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:22 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:22.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:22 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:22.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:22 smithi104 ceph-mon[32389]: from='osd.1 [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-09T22:11:22.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:22 smithi104 ceph-mon[32389]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-09T22:11:23.857 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:23 smithi153 ceph-mon[36599]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-09T22:11:23.857 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:23 smithi153 ceph-mon[36599]: from='osd.1 [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:11:23.857 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:23 smithi153 ceph-mon[36599]: osdmap e11: 2 total, 1 up, 2 in 2024-06-09T22:11:23.858 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:23 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:23.858 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:23 smithi153 ceph-mon[36599]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:11:23.858 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:23 smithi153 ceph-mon[36599]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:23.898 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:23 smithi194 ceph-mon[36856]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-09T22:11:23.898 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:23 smithi194 ceph-mon[36856]: from='osd.1 [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:11:23.898 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:23 smithi194 ceph-mon[36856]: osdmap e11: 2 total, 1 up, 2 in 2024-06-09T22:11:23.898 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:23.898 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:23 smithi194 ceph-mon[36856]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:11:23.898 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:23 smithi194 ceph-mon[36856]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:23.996 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:23 smithi104 ceph-mon[32389]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-09T22:11:23.996 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:23 smithi104 ceph-mon[32389]: from='osd.1 [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:11:23.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:23 smithi104 ceph-mon[32389]: osdmap e11: 2 total, 1 up, 2 in 2024-06-09T22:11:23.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:23.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:23 smithi104 ceph-mon[32389]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:11:23.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:23 smithi104 ceph-mon[32389]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:24.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:11:24 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1[41180]: 2024-06-09T22:11:24.076+0000 7fd797a83700 -1 osd.1 0 waiting for initial osdmap 2024-06-09T22:11:24.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:11:24 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1[41180]: 2024-06-09T22:11:24.079+0000 7fd78f86e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:11:24.483 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-09T22:11:24.484 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-09T22:11:24.484 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-06-09T22:11:24.484 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-06-09T22:11:24.484 INFO:teuthology.orchestra.run.smithi194.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111956 s, 937 MB/s 2024-06-09T22:11:24.484 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-06-09T22:11:24.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:24 smithi153 ceph-mon[36599]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-09T22:11:24.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:24 smithi153 ceph-mon[36599]: osdmap e12: 2 total, 1 up, 2 in 2024-06-09T22:11:24.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:24 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:24.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:24 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:24.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:24 smithi153 ceph-mon[36599]: from='osd.1 ' entity='osd.1' 2024-06-09T22:11:24.996 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:24 smithi104 ceph-mon[32389]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-09T22:11:24.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:24 smithi104 ceph-mon[32389]: osdmap e12: 2 total, 1 up, 2 in 2024-06-09T22:11:24.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:24 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:24.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:24 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:24.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:24 smithi104 ceph-mon[32389]: from='osd.1 ' entity='osd.1' 2024-06-09T22:11:25.035 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:24 smithi194 ceph-mon[36856]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-09T22:11:25.035 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:24 smithi194 ceph-mon[36856]: osdmap e12: 2 total, 1 up, 2 in 2024-06-09T22:11:25.035 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:24 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:25.035 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:24 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:25.035 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:24 smithi194 ceph-mon[36856]: from='osd.1 ' entity='osd.1' 2024-06-09T22:11:25.063 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_4 2024-06-09T22:11:25.302 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:11:25.675 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: purged_snaps scrub starts 2024-06-09T22:11:25.675 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: purged_snaps scrub ok 2024-06-09T22:11:25.675 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:25.675 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: osd.1 [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097] boot 2024-06-09T22:11:25.675 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: osdmap e13: 2 total, 2 up, 2 in 2024-06-09T22:11:25.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:25.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:25.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: Detected new or changed devices on smithi153 2024-06-09T22:11:25.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:25.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:25.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:11:25.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi153 to 17088M 2024-06-09T22:11:25.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:25.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:25.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:25.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: purged_snaps scrub starts 2024-06-09T22:11:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: purged_snaps scrub ok 2024-06-09T22:11:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: osd.1 [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097] boot 2024-06-09T22:11:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: osdmap e13: 2 total, 2 up, 2 in 2024-06-09T22:11:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:25.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: Detected new or changed devices on smithi153 2024-06-09T22:11:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:11:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi153 to 17088M 2024-06-09T22:11:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:25.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:25.996 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: purged_snaps scrub starts 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: purged_snaps scrub ok 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: osd.1 [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097] boot 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: osdmap e13: 2 total, 2 up, 2 in 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: Detected new or changed devices on smithi153 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi153 to 17088M 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:25.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:25.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:25.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:26.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.884+0000 7f4abf0c3700 1 -- 172.21.15.194:0/1010963835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab8102f90 msgr2=0x7f4ab8109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:26.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.884+0000 7f4abf0c3700 1 --2- 172.21.15.194:0/1010963835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab8102f90 0x7f4ab8109840 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4ab000b0a0 tx=0x7f4ab000b3b0 comp rx=0 tx=0).stop 2024-06-09T22:11:26.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.884+0000 7f4abf0c3700 1 -- 172.21.15.194:0/1010963835 shutdown_connections 2024-06-09T22:11:26.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.884+0000 7f4abf0c3700 1 --2- 172.21.15.194:0/1010963835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab8102f90 0x7f4ab8109840 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:26.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.884+0000 7f4abf0c3700 1 --2- 172.21.15.194:0/1010963835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab81013f0 0x7f4ab8101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.884+0000 7f4abf0c3700 1 --2- 172.21.15.194:0/1010963835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab81025f0 0x7f4ab8102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.884+0000 7f4abf0c3700 1 -- 172.21.15.194:0/1010963835 >> 172.21.15.194:0/1010963835 conn(0x7f4ab80fc990 msgr2=0x7f4ab80fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.884+0000 7f4abf0c3700 1 -- 172.21.15.194:0/1010963835 shutdown_connections 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.884+0000 7f4abf0c3700 1 -- 172.21.15.194:0/1010963835 wait complete. 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abf0c3700 1 Processor -- start 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abf0c3700 1 -- start start 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abf0c3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab81013f0 0x7f4ab81976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abf0c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab81025f0 0x7f4ab8197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abf0c3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab8102f90 0x7f4ab819ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abf0c3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab810d0b0 con 0x7f4ab81025f0 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abf0c3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab810cf30 con 0x7f4ab8102f90 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abf0c3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab810d230 con 0x7f4ab81013f0 2024-06-09T22:11:26.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abce5f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab81013f0 0x7f4ab81976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:11:26.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abce5f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab81013f0 0x7f4ab81976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:54594/0 (socket says 172.21.15.194:54594) 2024-06-09T22:11:26.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abce5f700 1 -- 172.21.15.194:0/12638901 learned_addr learned my addr 172.21.15.194:0/12638901 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-06-09T22:11:26.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4ab7fff700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab81025f0 0x7f4ab8197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:11:26.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abd660700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab8102f90 0x7f4ab819ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:11:26.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abce5f700 1 -- 172.21.15.194:0/12638901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab8102f90 msgr2=0x7f4ab819ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:26.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abce5f700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab8102f90 0x7f4ab819ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:26.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abce5f700 1 -- 172.21.15.194:0/12638901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab81025f0 msgr2=0x7f4ab8197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:26.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abce5f700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab81025f0 0x7f4ab8197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:26.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abce5f700 1 -- 172.21.15.194:0/12638901 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab806a270 con 0x7f4ab81013f0 2024-06-09T22:11:26.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.885+0000 7f4abd660700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab8102f90 0x7f4ab819ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:11:26.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.886+0000 7f4abce5f700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab81013f0 0x7f4ab81976c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800b500 tx=0x7f4aa800b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:11:26.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.886+0000 7f4ab5ffb700 1 -- 172.21.15.194:0/12638901 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8015020 con 0x7f4ab81013f0 2024-06-09T22:11:26.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.886+0000 7f4ab5ffb700 1 -- 172.21.15.194:0/12638901 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4aa8011d10 con 0x7f4ab81013f0 2024-06-09T22:11:26.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.886+0000 7f4abf0c3700 1 -- 172.21.15.194:0/12638901 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab806a560 con 0x7f4ab81013f0 2024-06-09T22:11:26.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.886+0000 7f4ab5ffb700 1 -- 172.21.15.194:0/12638901 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8010e50 con 0x7f4ab81013f0 2024-06-09T22:11:26.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.886+0000 7f4abf0c3700 1 -- 172.21.15.194:0/12638901 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab806ab20 con 0x7f4ab81013f0 2024-06-09T22:11:26.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.887+0000 7f4abf0c3700 1 -- 172.21.15.194:0/12638901 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ab8069180 con 0x7f4ab81013f0 2024-06-09T22:11:26.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.888+0000 7f4ab5ffb700 1 -- 172.21.15.194:0/12638901 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f4aa8018020 con 0x7f4ab81013f0 2024-06-09T22:11:26.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.888+0000 7f4ab5ffb700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4aa00a2d30 0x7f4aa00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:11:26.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.888+0000 7f4ab5ffb700 1 -- 172.21.15.194:0/12638901 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f4aa80ccf50 con 0x7f4ab81013f0 2024-06-09T22:11:26.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.888+0000 7f4ab7fff700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4aa00a2d30 0x7f4aa00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:11:26.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.889+0000 7f4ab7fff700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4aa00a2d30 0x7f4aa00a51f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4aac005950 tx=0x7f4aac00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:11:26.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:26.893+0000 7f4ab5ffb700 1 -- 172.21.15.194:0/12638901 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4aa801f020 con 0x7f4ab81013f0 2024-06-09T22:11:27.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:27.100+0000 7f4abf0c3700 1 -- 172.21.15.194:0/12638901 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4ab802dd10 con 0x7f4aa00a2d30 2024-06-09T22:11:27.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:27 smithi194 ceph-mon[36856]: osdmap e14: 2 total, 2 up, 2 in 2024-06-09T22:11:27.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:27 smithi194 ceph-mon[36856]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:11:27.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:27 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:11:27.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:27 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:11:27.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:27 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:27.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:27 smithi194 ceph-mon[36856]: pgmap v65: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:27.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:27 smithi153 ceph-mon[36599]: osdmap e14: 2 total, 2 up, 2 in 2024-06-09T22:11:27.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:27 smithi153 ceph-mon[36599]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:11:27.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:27 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:11:27.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:27 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:11:27.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:27 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:27.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:27 smithi153 ceph-mon[36599]: pgmap v65: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:27.996 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:27 smithi104 ceph-mon[32389]: osdmap e14: 2 total, 2 up, 2 in 2024-06-09T22:11:27.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:27 smithi104 ceph-mon[32389]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:11:27.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:27 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:11:27.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:27 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:11:27.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:27 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:27.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:27 smithi104 ceph-mon[32389]: pgmap v65: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:30.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:30 smithi194 ceph-mon[36856]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:30.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:30 smithi194 ceph-mon[36856]: from='client.? 172.21.15.194:0/2406870717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35204639-9a6e-485d-a3b7-2d2330406b3e"}]: dispatch 2024-06-09T22:11:30.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:30 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35204639-9a6e-485d-a3b7-2d2330406b3e"}]: dispatch 2024-06-09T22:11:30.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:30 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35204639-9a6e-485d-a3b7-2d2330406b3e"}]': finished 2024-06-09T22:11:30.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:30 smithi194 ceph-mon[36856]: osdmap e15: 3 total, 2 up, 3 in 2024-06-09T22:11:30.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:30.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:30 smithi194 ceph-mon[36856]: from='client.? 172.21.15.194:0/3371317692' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:11:30.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:30 smithi153 ceph-mon[36599]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:30.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:30 smithi153 ceph-mon[36599]: from='client.? 172.21.15.194:0/2406870717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35204639-9a6e-485d-a3b7-2d2330406b3e"}]: dispatch 2024-06-09T22:11:30.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:30 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35204639-9a6e-485d-a3b7-2d2330406b3e"}]: dispatch 2024-06-09T22:11:30.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:30 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35204639-9a6e-485d-a3b7-2d2330406b3e"}]': finished 2024-06-09T22:11:30.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:30 smithi153 ceph-mon[36599]: osdmap e15: 3 total, 2 up, 3 in 2024-06-09T22:11:30.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:30.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:30 smithi153 ceph-mon[36599]: from='client.? 172.21.15.194:0/3371317692' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:11:30.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:30 smithi104 ceph-mon[32389]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:30.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:30 smithi104 ceph-mon[32389]: from='client.? 172.21.15.194:0/2406870717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35204639-9a6e-485d-a3b7-2d2330406b3e"}]: dispatch 2024-06-09T22:11:30.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:30 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35204639-9a6e-485d-a3b7-2d2330406b3e"}]: dispatch 2024-06-09T22:11:30.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:30 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35204639-9a6e-485d-a3b7-2d2330406b3e"}]': finished 2024-06-09T22:11:30.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:30 smithi104 ceph-mon[32389]: osdmap e15: 3 total, 2 up, 3 in 2024-06-09T22:11:30.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:30.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:30 smithi104 ceph-mon[32389]: from='client.? 172.21.15.194:0/3371317692' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:11:32.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:32 smithi194 ceph-mon[36856]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:32.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:32 smithi153 ceph-mon[36599]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:32.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:32 smithi104 ceph-mon[32389]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:34.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:34 smithi194 ceph-mon[36856]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:34.551 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:34 smithi153 ceph-mon[36599]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:34.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:34 smithi104 ceph-mon[32389]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:36.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:36 smithi194 ceph-mon[36856]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:36.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:36 smithi153 ceph-mon[36599]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:36.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:36 smithi104 ceph-mon[32389]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:37.613 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:37 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-09T22:11:37.613 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:37 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:37.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:37 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-09T22:11:37.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:37 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:37.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:37 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-09T22:11:37.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:37 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:38.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:38 smithi194 ceph-mon[36856]: Deploying daemon osd.2 on smithi194 2024-06-09T22:11:38.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:38 smithi194 ceph-mon[36856]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:38.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:38 smithi153 ceph-mon[36599]: Deploying daemon osd.2 on smithi194 2024-06-09T22:11:38.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:38 smithi153 ceph-mon[36599]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:38.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:38 smithi104 ceph-mon[32389]: Deploying daemon osd.2 on smithi194 2024-06-09T22:11:38.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:38 smithi104 ceph-mon[32389]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:40.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:40 smithi194 ceph-mon[36856]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:40.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:40 smithi153 ceph-mon[36599]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:40.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:40 smithi104 ceph-mon[32389]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:42 smithi153 ceph-mon[36599]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:42.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:42 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:42.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:42 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:11:42.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:42 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:42.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:42 smithi104 ceph-mon[32389]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:42.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:42 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:42.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:42 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:11:42.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:42 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:42.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:42 smithi194 ceph-mon[36856]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:42.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:42 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:42.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:42 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:11:42.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:42 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:44.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:44 smithi153 ceph-mon[36599]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:44.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:44 smithi104 ceph-mon[32389]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:44.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:44 smithi194 ceph-mon[36856]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:45.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.461+0000 7f4ab5ffb700 1 -- 172.21.15.194:0/12638901 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4ab802dd10 con 0x7f4aa00a2d30 2024-06-09T22:11:45.452 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 2 on host 'smithi194' 2024-06-09T22:11:45.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.464+0000 7f4abf0c3700 1 -- 172.21.15.194:0/12638901 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4aa00a2d30 msgr2=0x7f4aa00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:45.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.464+0000 7f4abf0c3700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4aa00a2d30 0x7f4aa00a51f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4aac005950 tx=0x7f4aac00b410 comp rx=0 tx=0).stop 2024-06-09T22:11:45.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.464+0000 7f4abf0c3700 1 -- 172.21.15.194:0/12638901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab81013f0 msgr2=0x7f4ab81976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:45.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.464+0000 7f4abf0c3700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab81013f0 0x7f4ab81976c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800b500 tx=0x7f4aa800b810 comp rx=0 tx=0).stop 2024-06-09T22:11:45.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.464+0000 7f4abf0c3700 1 -- 172.21.15.194:0/12638901 shutdown_connections 2024-06-09T22:11:45.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.464+0000 7f4abf0c3700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4aa00a2d30 0x7f4aa00a51f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:45.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.464+0000 7f4abf0c3700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab8102f90 0x7f4ab819ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:45.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.464+0000 7f4abf0c3700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab81013f0 0x7f4ab81976c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:45.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.464+0000 7f4abf0c3700 1 --2- 172.21.15.194:0/12638901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab81025f0 0x7f4ab8197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:45.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.464+0000 7f4abf0c3700 1 -- 172.21.15.194:0/12638901 >> 172.21.15.194:0/12638901 conn(0x7f4ab80fc990 msgr2=0x7f4ab81058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:11:45.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.464+0000 7f4abf0c3700 1 -- 172.21.15.194:0/12638901 shutdown_connections 2024-06-09T22:11:45.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:11:45.464+0000 7f4abf0c3700 1 -- 172.21.15.194:0/12638901 wait complete. 2024-06-09T22:11:46.049 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:45 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.050 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:45 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.050 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:45 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:46.050 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:45 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:46.050 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:45 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.050 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:45 smithi194 ceph-mon[36856]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:46.050 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:45 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.050 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:45 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.050 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:45 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:11:46.096 DEBUG:teuthology.orchestra.run.smithi194:osd.2> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.2.service 2024-06-09T22:11:46.099 INFO:tasks.cephadm:Deploying osd.3 on smithi104 with /dev/vg_nvme/lv_3... 2024-06-09T22:11:46.099 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-09T22:11:46.127 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:45 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.127 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:45 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.127 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:45 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:46.127 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:45 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:46.127 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:45 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.127 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:45 smithi104 ceph-mon[32389]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:46.127 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:45 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.128 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:45 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.128 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:45 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:11:46.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:45 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:45 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:45 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:46.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:45 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:46.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:45 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:45 smithi153 ceph-mon[36599]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:46.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:45 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:45 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:46.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:45 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:11:46.344 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:11:47.107 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:46 smithi104 ceph-mon[32389]: from='osd.2 [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-09T22:11:47.107 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:46 smithi104 ceph-mon[32389]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-09T22:11:47.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:46 smithi153 ceph-mon[36599]: from='osd.2 [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-09T22:11:47.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:46 smithi153 ceph-mon[36599]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-09T22:11:47.241 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:46 smithi194 ceph-mon[36856]: from='osd.2 [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-09T22:11:47.241 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:46 smithi194 ceph-mon[36856]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-09T22:11:47.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:47 smithi104 ceph-mon[32389]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-09T22:11:47.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:47 smithi104 ceph-mon[32389]: osdmap e16: 3 total, 2 up, 3 in 2024-06-09T22:11:47.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:47.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:47 smithi104 ceph-mon[32389]: from='osd.2 [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:11:47.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:47 smithi104 ceph-mon[32389]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:11:47.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:47 smithi104 ceph-mon[32389]: pgmap v77: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:48.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:47 smithi153 ceph-mon[36599]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-09T22:11:48.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:47 smithi153 ceph-mon[36599]: osdmap e16: 3 total, 2 up, 3 in 2024-06-09T22:11:48.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:47 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:48.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:47 smithi153 ceph-mon[36599]: from='osd.2 [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:11:48.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:47 smithi153 ceph-mon[36599]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:11:48.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:47 smithi153 ceph-mon[36599]: pgmap v77: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:48.245 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-09T22:11:48.245 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-09T22:11:48.245 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-06-09T22:11:48.245 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-06-09T22:11:48.245 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115168 s, 910 MB/s 2024-06-09T22:11:48.245 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-06-09T22:11:48.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:47 smithi194 ceph-mon[36856]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-09T22:11:48.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:47 smithi194 ceph-mon[36856]: osdmap e16: 3 total, 2 up, 3 in 2024-06-09T22:11:48.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:47 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:48.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:47 smithi194 ceph-mon[36856]: from='osd.2 [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:11:48.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:47 smithi194 ceph-mon[36856]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:11:48.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:47 smithi194 ceph-mon[36856]: pgmap v77: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-06-09T22:11:48.300 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:11:47 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2[41983]: 2024-06-09T22:11:47.883+0000 7fbdbf43e700 -1 osd.2 0 waiting for initial osdmap 2024-06-09T22:11:48.301 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:11:47 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2[41983]: 2024-06-09T22:11:47.888+0000 7fbdbb231700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:11:48.945 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_3 2024-06-09T22:11:49.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:48 smithi194 ceph-mon[36856]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-06-09T22:11:49.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:48 smithi194 ceph-mon[36856]: osdmap e17: 3 total, 2 up, 3 in 2024-06-09T22:11:49.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:48 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:49.048 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:48 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:49.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:48 smithi153 ceph-mon[36599]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-06-09T22:11:49.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:48 smithi153 ceph-mon[36599]: osdmap e17: 3 total, 2 up, 3 in 2024-06-09T22:11:49.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:48 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:49.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:48 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:49.234 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:11:49.247 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:48 smithi104 ceph-mon[32389]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-06-09T22:11:49.247 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:48 smithi104 ceph-mon[32389]: osdmap e17: 3 total, 2 up, 3 in 2024-06-09T22:11:49.247 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:48 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:49.247 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:48 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:49.997 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: purged_snaps scrub starts 2024-06-09T22:11:49.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: purged_snaps scrub ok 2024-06-09T22:11:49.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: osd.2 [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084] boot 2024-06-09T22:11:49.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: osdmap e18: 3 total, 3 up, 3 in 2024-06-09T22:11:49.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:49.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: Detected new or changed devices on smithi194 2024-06-09T22:11:49.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:49.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:49.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:11:49.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi194 to 17088M 2024-06-09T22:11:49.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:49.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:49.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:49.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:49.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: pgmap v80: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:49.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-09T22:11:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: purged_snaps scrub starts 2024-06-09T22:11:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: purged_snaps scrub ok 2024-06-09T22:11:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: osd.2 [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084] boot 2024-06-09T22:11:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: osdmap e18: 3 total, 3 up, 3 in 2024-06-09T22:11:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: Detected new or changed devices on smithi194 2024-06-09T22:11:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:11:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi194 to 17088M 2024-06-09T22:11:50.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:50.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:50.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:50.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:50.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: pgmap v80: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:50.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-09T22:11:50.299 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: purged_snaps scrub starts 2024-06-09T22:11:50.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: purged_snaps scrub ok 2024-06-09T22:11:50.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: osd.2 [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084] boot 2024-06-09T22:11:50.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: osdmap e18: 3 total, 3 up, 3 in 2024-06-09T22:11:50.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:11:50.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: Detected new or changed devices on smithi194 2024-06-09T22:11:50.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:50.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:50.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:11:50.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi194 to 17088M 2024-06-09T22:11:50.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:50.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:50.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:11:50.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:50.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: pgmap v80: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:50.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-09T22:11:50.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.808+0000 7f492656c700 1 -- 172.21.15.104:0/4087522555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4920104220 msgr2=0x7f492010aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:50.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.808+0000 7f492656c700 1 --2- 172.21.15.104:0/4087522555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4920104220 0x7f492010aad0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f491800b0d0 tx=0x7f491800b3e0 comp rx=0 tx=0).stop 2024-06-09T22:11:50.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.808+0000 7f492656c700 1 -- 172.21.15.104:0/4087522555 shutdown_connections 2024-06-09T22:11:50.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.808+0000 7f492656c700 1 --2- 172.21.15.104:0/4087522555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4920104220 0x7f492010aad0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:50.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.808+0000 7f492656c700 1 --2- 172.21.15.104:0/4087522555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4920100800 0x7f4920100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:50.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.808+0000 7f492656c700 1 --2- 172.21.15.104:0/4087522555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4920103880 0x7f4920103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:50.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.808+0000 7f492656c700 1 -- 172.21.15.104:0/4087522555 >> 172.21.15.104:0/4087522555 conn(0x7f49200fc140 msgr2=0x7f49200fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:11:50.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.808+0000 7f492656c700 1 -- 172.21.15.104:0/4087522555 shutdown_connections 2024-06-09T22:11:50.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.808+0000 7f492656c700 1 -- 172.21.15.104:0/4087522555 wait complete. 2024-06-09T22:11:50.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f492656c700 1 Processor -- start 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f492656c700 1 -- start start 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f492656c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4920100800 0x7f49201985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f492656c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4920103880 0x7f4920198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f492656c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4920104220 0x7f492019fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f492656c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f492010e110 con 0x7f4920100800 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f492656c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f492010df90 con 0x7f4920104220 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f492656c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f492010e290 con 0x7f4920103880 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f491ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4920100800 0x7f49201985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f491ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4920100800 0x7f49201985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57926/0 (socket says 172.21.15.104:57926) 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f491ffff700 1 -- 172.21.15.104:0/1800840881 learned_addr learned my addr 172.21.15.104:0/1800840881 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f4924b09700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4920104220 0x7f492019fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f4917fff700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4920103880 0x7f4920198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:11:50.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f491ffff700 1 -- 172.21.15.104:0/1800840881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4920103880 msgr2=0x7f4920198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:50.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f491ffff700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4920103880 0x7f4920198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:50.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f491ffff700 1 -- 172.21.15.104:0/1800840881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4920104220 msgr2=0x7f492019fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:11:50.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f491ffff700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4920104220 0x7f492019fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:11:50.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.809+0000 7f491ffff700 1 -- 172.21.15.104:0/1800840881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49201a0210 con 0x7f4920100800 2024-06-09T22:11:50.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.810+0000 7f491ffff700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4920100800 0x7f49201985b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f491000c9c0 tx=0x7f491000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:11:50.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.810+0000 7f491dffb700 1 -- 172.21.15.104:0/1800840881 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4910007ac0 con 0x7f4920100800 2024-06-09T22:11:50.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.810+0000 7f491dffb700 1 -- 172.21.15.104:0/1800840881 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4910007c60 con 0x7f4920100800 2024-06-09T22:11:50.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.810+0000 7f492656c700 1 -- 172.21.15.104:0/1800840881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49201a0500 con 0x7f4920100800 2024-06-09T22:11:50.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.810+0000 7f491dffb700 1 -- 172.21.15.104:0/1800840881 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4910018660 con 0x7f4920100800 2024-06-09T22:11:50.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.810+0000 7f492656c700 1 -- 172.21.15.104:0/1800840881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49201a0a40 con 0x7f4920100800 2024-06-09T22:11:50.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.811+0000 7f492656c700 1 -- 172.21.15.104:0/1800840881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4920069180 con 0x7f4920100800 2024-06-09T22:11:50.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.812+0000 7f491dffb700 1 -- 172.21.15.104:0/1800840881 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f4910004020 con 0x7f4920100800 2024-06-09T22:11:50.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.813+0000 7f491dffb700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f490c0a2b90 0x7f490c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:11:50.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.813+0000 7f4917fff700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f490c0a2b90 0x7f490c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:11:50.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.813+0000 7f491dffb700 1 -- 172.21.15.104:0/1800840881 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f49100cdd00 con 0x7f4920100800 2024-06-09T22:11:50.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.814+0000 7f4917fff700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f490c0a2b90 0x7f490c0a5050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4908005950 tx=0x7f49080058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:11:50.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:50.817+0000 7f491dffb700 1 -- 172.21.15.104:0/1800840881 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4910014030 con 0x7f4920100800 2024-06-09T22:11:51.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:51.037+0000 7f492656c700 1 -- 172.21.15.104:0/1800840881 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f49200628d0 con 0x7f490c0a2b90 2024-06-09T22:11:51.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:50 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-09T22:11:51.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:50 smithi153 ceph-mon[36599]: osdmap e19: 3 total, 3 up, 3 in 2024-06-09T22:11:51.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:50 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-09T22:11:51.245 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:50 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-09T22:11:51.245 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:50 smithi104 ceph-mon[32389]: osdmap e19: 3 total, 3 up, 3 in 2024-06-09T22:11:51.245 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:50 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-09T22:11:51.299 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:50 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-09T22:11:51.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:50 smithi194 ceph-mon[36856]: osdmap e19: 3 total, 3 up, 3 in 2024-06-09T22:11:51.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:50 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-09T22:11:51.497 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:11:51 smithi104 sudo[49317]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-09T22:11:51.497 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:11:51 smithi104 sudo[49317]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-09T22:11:51.497 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:11:51 smithi104 sudo[49317]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-09T22:11:51.497 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:11:51 smithi104 sudo[49317]: pam_unix(sudo:session): session closed for user root 2024-06-09T22:11:51.829 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:11:51 smithi104 sudo[49352]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-09T22:11:51.829 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:11:51 smithi104 sudo[49352]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-09T22:11:51.829 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:11:51 smithi104 sudo[49352]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-09T22:11:51.829 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:11:51 smithi104 sudo[49352]: pam_unix(sudo:session): session closed for user root 2024-06-09T22:11:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:11:51.915+0000 7f491dffb700 1 -- 172.21.15.104:0/1800840881 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4910098120 con 0x7f4920100800 2024-06-09T22:11:52.035 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:11:51 smithi153 sudo[45319]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-09T22:11:52.035 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:11:51 smithi153 sudo[45319]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-09T22:11:52.035 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:11:51 smithi153 sudo[45319]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-09T22:11:52.035 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:11:51 smithi153 sudo[45319]: pam_unix(sudo:session): session closed for user root 2024-06-09T22:11:52.035 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-09T22:11:52.036 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:51 smithi153 ceph-mon[36599]: osdmap e20: 3 total, 3 up, 3 in 2024-06-09T22:11:52.036 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:51 smithi153 ceph-mon[36599]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:11:52.036 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:11:52.036 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:11:52.036 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:52.036 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:51 smithi153 ceph-mon[36599]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:52.036 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:52.247 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-09T22:11:52.247 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:51 smithi104 ceph-mon[32389]: osdmap e20: 3 total, 3 up, 3 in 2024-06-09T22:11:52.247 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:51 smithi104 ceph-mon[32389]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:11:52.248 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:11:52.248 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:11:52.248 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:52.248 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:51 smithi104 ceph-mon[32389]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:52.248 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:52.299 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-09T22:11:52.299 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:51 smithi194 ceph-mon[36856]: osdmap e20: 3 total, 3 up, 3 in 2024-06-09T22:11:52.299 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:51 smithi194 ceph-mon[36856]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:11:52.299 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:11:52.299 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:11:52.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:11:52.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:51 smithi194 ceph-mon[36856]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:52.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:52.483 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:11:52 smithi153 sudo[45322]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-09T22:11:52.483 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:11:52 smithi153 sudo[45322]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-09T22:11:52.483 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:11:52 smithi153 sudo[45322]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-09T22:11:52.483 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:11:52 smithi153 sudo[45322]: pam_unix(sudo:session): session closed for user root 2024-06-09T22:11:52.558 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:11:52 smithi194 sudo[45584]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-09T22:11:52.558 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:11:52 smithi194 sudo[45584]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-09T22:11:52.558 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:11:52 smithi194 sudo[45584]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-09T22:11:52.558 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:11:52 smithi194 sudo[45584]: pam_unix(sudo:session): session closed for user root 2024-06-09T22:11:53.049 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:11:52 smithi194 sudo[45587]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-09T22:11:53.050 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:11:52 smithi194 sudo[45587]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-09T22:11:53.050 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:11:52 smithi194 sudo[45587]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-09T22:11:53.050 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:11:52 smithi194 sudo[45587]: pam_unix(sudo:session): session closed for user root 2024-06-09T22:11:53.094 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:52 smithi104 sudo[49420]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-09T22:11:53.094 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:52 smithi104 sudo[49420]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-09T22:11:53.094 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:52 smithi104 sudo[49420]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-09T22:11:53.094 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:52 smithi104 sudo[49420]: pam_unix(sudo:session): session closed for user root 2024-06-09T22:11:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 ceph-mon[36856]: osdmap e21: 3 total, 3 up, 3 in 2024-06-09T22:11:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 ceph-mon[36856]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-06-09T22:11:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 ceph-mon[36856]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-09T22:11:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 sudo[45590]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-09T22:11:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 sudo[45590]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-09T22:11:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 sudo[45590]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-09T22:11:53.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 sudo[45590]: pam_unix(sudo:session): session closed for user root 2024-06-09T22:11:53.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:53 smithi153 ceph-mon[36599]: osdmap e21: 3 total, 3 up, 3 in 2024-06-09T22:11:53.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:53 smithi153 ceph-mon[36599]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-06-09T22:11:53.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:53 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:53.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:53 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:53.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:53 smithi153 ceph-mon[36599]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-09T22:11:53.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:53 smithi104 sudo[49430]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-09T22:11:53.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:53 smithi104 sudo[49430]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-09T22:11:53.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:53 smithi104 sudo[49430]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-09T22:11:53.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:53 smithi104 sudo[49430]: pam_unix(sudo:session): session closed for user root 2024-06-09T22:11:53.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:53 smithi104 ceph-mon[32389]: osdmap e21: 3 total, 3 up, 3 in 2024-06-09T22:11:53.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:53 smithi104 ceph-mon[32389]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-06-09T22:11:53.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:53 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:53.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:53 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:11:53.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:53 smithi104 ceph-mon[32389]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-09T22:11:53.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 sudo[45593]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-09T22:11:53.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 sudo[45593]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-09T22:11:53.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 sudo[45593]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-09T22:11:53.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:53 smithi194 sudo[45593]: pam_unix(sudo:session): session closed for user root 2024-06-09T22:11:54.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:53 smithi153 sudo[45325]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-09T22:11:54.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:53 smithi153 sudo[45325]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-09T22:11:54.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:53 smithi153 sudo[45325]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-09T22:11:54.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:53 smithi153 sudo[45325]: pam_unix(sudo:session): session closed for user root 2024-06-09T22:11:54.476 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-09T22:11:54.477 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:11:54.477 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:11:54.477 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:11:54.477 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-09T22:11:54.477 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:11:54.478 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:11:54.478 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:11:54.478 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:54.478 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-09T22:11:54.478 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-09T22:11:54.479 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:11:54.479 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:11:54.479 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:11:54.479 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/499078127' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3acd0acc-fcf7-4122-a939-d961bfc03c0b"}]: dispatch 2024-06-09T22:11:54.479 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-09T22:11:54.479 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/499078127' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3acd0acc-fcf7-4122-a939-d961bfc03c0b"}]': finished 2024-06-09T22:11:54.480 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: osdmap e22: 4 total, 3 up, 4 in 2024-06-09T22:11:54.480 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:11:54.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 sudo[45328]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-09T22:11:54.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 sudo[45328]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-09T22:11:54.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 sudo[45328]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-09T22:11:54.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 sudo[45328]: pam_unix(sudo:session): session closed for user root 2024-06-09T22:11:54.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-09T22:11:54.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:11:54.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:11:54.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:11:54.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-09T22:11:54.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:11:54.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:11:54.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:11:54.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:54.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-09T22:11:54.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-09T22:11:54.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:11:54.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:11:54.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:11:54.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/499078127' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3acd0acc-fcf7-4122-a939-d961bfc03c0b"}]: dispatch 2024-06-09T22:11:54.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-09T22:11:54.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/499078127' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3acd0acc-fcf7-4122-a939-d961bfc03c0b"}]': finished 2024-06-09T22:11:54.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: osdmap e22: 4 total, 3 up, 4 in 2024-06-09T22:11:54.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:11:54.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-09T22:11:54.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:11:54.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:11:54.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:11:54.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-09T22:11:54.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:11:54.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:11:54.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:11:54.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:54.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-09T22:11:54.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-09T22:11:54.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:11:54.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:11:54.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:11:54.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/499078127' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3acd0acc-fcf7-4122-a939-d961bfc03c0b"}]: dispatch 2024-06-09T22:11:54.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-09T22:11:54.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/499078127' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3acd0acc-fcf7-4122-a939-d961bfc03c0b"}]': finished 2024-06-09T22:11:54.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: osdmap e22: 4 total, 3 up, 4 in 2024-06-09T22:11:54.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:11:55.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:55 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2638418802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:11:55.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:55 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2638418802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:11:55.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:55 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2638418802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:11:56.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:56 smithi153 ceph-mon[36599]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:56.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:56 smithi104 ceph-mon[32389]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:56.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:56 smithi194 ceph-mon[36856]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:58.478 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:11:58 smithi104 ceph-mon[32389]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:58.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:11:58 smithi194 ceph-mon[36856]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:11:58.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:11:58 smithi153 ceph-mon[36599]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:00.414 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:00 smithi104 ceph-mon[32389]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:00.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:00 smithi194 ceph-mon[36856]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:00.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:00 smithi153 ceph-mon[36599]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:02.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:02 smithi104 ceph-mon[32389]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:02.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-09T22:12:02.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:02.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:02 smithi194 ceph-mon[36856]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:02.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-09T22:12:02.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:02.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:02 smithi153 ceph-mon[36599]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:02.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-09T22:12:02.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:03.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:03 smithi104 ceph-mon[32389]: Deploying daemon osd.3 on smithi104 2024-06-09T22:12:03.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:03 smithi153 ceph-mon[36599]: Deploying daemon osd.3 on smithi104 2024-06-09T22:12:03.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:03 smithi194 ceph-mon[36856]: Deploying daemon osd.3 on smithi104 2024-06-09T22:12:04.635 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:04 smithi153 ceph-mon[36599]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:04.684 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:04 smithi104 ceph-mon[32389]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:04.701 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:04 smithi194 ceph-mon[36856]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:06.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:06 smithi104 ceph-mon[32389]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:06.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:06 smithi153 ceph-mon[36599]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:06.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:06 smithi194 ceph-mon[36856]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:08.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:08 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:08.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:08 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:08.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:08 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:08.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:08 smithi153 ceph-mon[36599]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:08.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:08 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:08.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:08 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:08.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:08 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:08.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:08 smithi104 ceph-mon[32389]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:08.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:08 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:08.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:08 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:08.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:08 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:08.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:08 smithi194 ceph-mon[36856]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:10.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:10 smithi194 ceph-mon[36856]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:10.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:10 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:10.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:10 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:10.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:10 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:10.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:10 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:12:10.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:10 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:10.713 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:10 smithi104 ceph-mon[32389]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:10.714 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:10 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:10.714 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:10 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:10.714 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:10 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:10.714 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:10 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:12:10.714 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:10 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:10.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:10 smithi153 ceph-mon[36599]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:10.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:10 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:10.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:10 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:10.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:10 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:10.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:10 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:12:10.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:10 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:11.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.065+0000 7f491dffb700 1 -- 172.21.15.104:0/1800840881 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f49200628d0 con 0x7f490c0a2b90 2024-06-09T22:12:11.056 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 3 on host 'smithi104' 2024-06-09T22:12:11.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.067+0000 7f492656c700 1 -- 172.21.15.104:0/1800840881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f490c0a2b90 msgr2=0x7f490c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:12:11.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.067+0000 7f492656c700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f490c0a2b90 0x7f490c0a5050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4908005950 tx=0x7f49080058e0 comp rx=0 tx=0).stop 2024-06-09T22:12:11.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.067+0000 7f492656c700 1 -- 172.21.15.104:0/1800840881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4920100800 msgr2=0x7f49201985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:12:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.067+0000 7f492656c700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4920100800 0x7f49201985b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f491000c9c0 tx=0x7f491000ccd0 comp rx=0 tx=0).stop 2024-06-09T22:12:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.067+0000 7f492656c700 1 -- 172.21.15.104:0/1800840881 shutdown_connections 2024-06-09T22:12:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.067+0000 7f492656c700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f490c0a2b90 0x7f490c0a5050 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.067+0000 7f492656c700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4920100800 0x7f49201985b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.067+0000 7f492656c700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4920104220 0x7f492019fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.067+0000 7f492656c700 1 --2- 172.21.15.104:0/1800840881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4920103880 0x7f4920198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.067+0000 7f492656c700 1 -- 172.21.15.104:0/1800840881 >> 172.21.15.104:0/1800840881 conn(0x7f49200fc140 msgr2=0x7f4920106ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:12:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.067+0000 7f492656c700 1 -- 172.21.15.104:0/1800840881 shutdown_connections 2024-06-09T22:12:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:12:11.067+0000 7f492656c700 1 -- 172.21.15.104:0/1800840881 wait complete. 2024-06-09T22:12:11.726 DEBUG:teuthology.orchestra.run.smithi104:osd.3> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.3.service 2024-06-09T22:12:11.728 INFO:tasks.cephadm:Deploying osd.4 on smithi153 with /dev/vg_nvme/lv_3... 2024-06-09T22:12:11.729 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-09T22:12:11.977 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.b/config 2024-06-09T22:12:12.248 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:12 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:12.248 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:12 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:12.248 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:12 smithi104 ceph-mon[32389]: from='osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-09T22:12:12.248 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:12 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:12.248 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:12 smithi104 ceph-mon[32389]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:12.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:12 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:12.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:12 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:12.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:12 smithi153 ceph-mon[36599]: from='osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-09T22:12:12.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:12 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:12.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:12 smithi153 ceph-mon[36599]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:12.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:12 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:12.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:12 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:12.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:12 smithi194 ceph-mon[36856]: from='osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-09T22:12:12.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:12 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:12.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:12 smithi194 ceph-mon[36856]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:13.225 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:13 smithi153 ceph-mon[36599]: from='osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-09T22:12:13.225 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:13 smithi153 ceph-mon[36599]: osdmap e23: 4 total, 3 up, 4 in 2024-06-09T22:12:13.226 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:13 smithi153 ceph-mon[36599]: from='osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-06-09T22:12:13.226 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:12:13.293 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:13 smithi104 ceph-mon[32389]: from='osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-09T22:12:13.293 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:13 smithi104 ceph-mon[32389]: osdmap e23: 4 total, 3 up, 4 in 2024-06-09T22:12:13.293 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:13 smithi104 ceph-mon[32389]: from='osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-06-09T22:12:13.293 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:12:13.294 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:12:13 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3[51484]: 2024-06-09T22:12:13.046+0000 7f1f621f3700 -1 osd.3 0 waiting for initial osdmap 2024-06-09T22:12:13.294 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:12:13 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3[51484]: 2024-06-09T22:12:13.053+0000 7f1f5dfe6700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:12:13.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:13 smithi194 ceph-mon[36856]: from='osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-09T22:12:13.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:13 smithi194 ceph-mon[36856]: osdmap e23: 4 total, 3 up, 4 in 2024-06-09T22:12:13.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:13 smithi194 ceph-mon[36856]: from='osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-06-09T22:12:13.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:12:13.783 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-09T22:12:13.783 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-09T22:12:13.783 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-06-09T22:12:13.783 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-06-09T22:12:13.783 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129453 s, 810 MB/s 2024-06-09T22:12:13.783 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-06-09T22:12:14.254 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:14 smithi104 ceph-mon[32389]: purged_snaps scrub starts 2024-06-09T22:12:14.255 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:14 smithi104 ceph-mon[32389]: purged_snaps scrub ok 2024-06-09T22:12:14.255 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:14 smithi104 ceph-mon[32389]: from='osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-06-09T22:12:14.255 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:14 smithi104 ceph-mon[32389]: osdmap e24: 4 total, 3 up, 4 in 2024-06-09T22:12:14.255 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:12:14.255 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:12:14.255 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:14 smithi104 ceph-mon[32389]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:14.255 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:14 smithi104 ceph-mon[32389]: osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340] boot 2024-06-09T22:12:14.255 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:14 smithi104 ceph-mon[32389]: osdmap e25: 4 total, 4 up, 4 in 2024-06-09T22:12:14.255 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:12:14.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:14 smithi194 ceph-mon[36856]: purged_snaps scrub starts 2024-06-09T22:12:14.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:14 smithi194 ceph-mon[36856]: purged_snaps scrub ok 2024-06-09T22:12:14.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:14 smithi194 ceph-mon[36856]: from='osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-06-09T22:12:14.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:14 smithi194 ceph-mon[36856]: osdmap e24: 4 total, 3 up, 4 in 2024-06-09T22:12:14.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:12:14.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:12:14.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:14 smithi194 ceph-mon[36856]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:14.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:14 smithi194 ceph-mon[36856]: osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340] boot 2024-06-09T22:12:14.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:14 smithi194 ceph-mon[36856]: osdmap e25: 4 total, 4 up, 4 in 2024-06-09T22:12:14.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:12:14.405 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:14 smithi153 ceph-mon[36599]: purged_snaps scrub starts 2024-06-09T22:12:14.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:14 smithi153 ceph-mon[36599]: purged_snaps scrub ok 2024-06-09T22:12:14.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:14 smithi153 ceph-mon[36599]: from='osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-06-09T22:12:14.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:14 smithi153 ceph-mon[36599]: osdmap e24: 4 total, 3 up, 4 in 2024-06-09T22:12:14.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:14 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:12:14.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:14 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:12:14.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:14 smithi153 ceph-mon[36599]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-09T22:12:14.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:14 smithi153 ceph-mon[36599]: osd.3 [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340] boot 2024-06-09T22:12:14.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:14 smithi153 ceph-mon[36599]: osdmap e25: 4 total, 4 up, 4 in 2024-06-09T22:12:14.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:14 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:12:14.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-06-09T22:12:14.663 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.b/config 2024-06-09T22:12:16.091 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:15 smithi153 ceph-mon[36599]: Detected new or changed devices on smithi104 2024-06-09T22:12:16.091 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:15 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:16.091 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:15 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:16.092 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:15 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:12:16.092 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:15 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:12:16.092 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:15 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi104 to 8544M 2024-06-09T22:12:16.092 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:15 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:16.092 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:15 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:16.092 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:15 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:12:16.092 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:15 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:16.092 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:15 smithi153 ceph-mon[36599]: osdmap e26: 4 total, 4 up, 4 in 2024-06-09T22:12:16.093 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:15 smithi153 ceph-mon[36599]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:16.248 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:15 smithi104 ceph-mon[32389]: Detected new or changed devices on smithi104 2024-06-09T22:12:16.248 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:16.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:16.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:12:16.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:12:16.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:15 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi104 to 8544M 2024-06-09T22:12:16.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:16.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:16.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:12:16.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:16.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:15 smithi104 ceph-mon[32389]: osdmap e26: 4 total, 4 up, 4 in 2024-06-09T22:12:16.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:15 smithi104 ceph-mon[32389]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:16.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:15 smithi194 ceph-mon[36856]: Detected new or changed devices on smithi104 2024-06-09T22:12:16.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:16.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:16.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:12:16.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:12:16.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:15 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi104 to 8544M 2024-06-09T22:12:16.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:16.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:16.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:12:16.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:16.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:15 smithi194 ceph-mon[36856]: osdmap e26: 4 total, 4 up, 4 in 2024-06-09T22:12:16.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:15 smithi194 ceph-mon[36856]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.480+0000 7f746d704700 1 -- 172.21.15.153:0/2510703978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74681058b0 msgr2=0x7f7468107ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:12:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.480+0000 7f746d704700 1 --2- 172.21.15.153:0/2510703978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74681058b0 0x7f7468107ca0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7450009990 tx=0x7f7450009ca0 comp rx=0 tx=0).stop 2024-06-09T22:12:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.480+0000 7f746d704700 1 -- 172.21.15.153:0/2510703978 shutdown_connections 2024-06-09T22:12:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.480+0000 7f746d704700 1 --2- 172.21.15.153:0/2510703978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74681081e0 0x7f746810a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.480+0000 7f746d704700 1 --2- 172.21.15.153:0/2510703978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7468102f80 0x7f7468105370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.480+0000 7f746d704700 1 --2- 172.21.15.153:0/2510703978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74681058b0 0x7f7468107ca0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.480+0000 7f746d704700 1 -- 172.21.15.153:0/2510703978 >> 172.21.15.153:0/2510703978 conn(0x7f74680fc990 msgr2=0x7f74680fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:12:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.480+0000 7f746d704700 1 -- 172.21.15.153:0/2510703978 shutdown_connections 2024-06-09T22:12:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.480+0000 7f746d704700 1 -- 172.21.15.153:0/2510703978 wait complete. 2024-06-09T22:12:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.481+0000 7f746d704700 1 Processor -- start 2024-06-09T22:12:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.481+0000 7f746d704700 1 -- start start 2024-06-09T22:12:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.481+0000 7f746d704700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7468102f80 0x7f74681028e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:12:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.481+0000 7f746d704700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74681058b0 0x7f7468100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:12:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.481+0000 7f746d704700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f74681081e0 0x7f7468101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:12:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.481+0000 7f746d704700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f746810e140 con 0x7f74681058b0 2024-06-09T22:12:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.481+0000 7f746d704700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f746810dfc0 con 0x7f7468102f80 2024-06-09T22:12:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.481+0000 7f746d704700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f746810e2c0 con 0x7f74681081e0 2024-06-09T22:12:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.481+0000 7f7466ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7468102f80 0x7f74681028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:12:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.481+0000 7f7466ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7468102f80 0x7f74681028e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52482/0 (socket says 172.21.15.153:52482) 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.481+0000 7f7466ffd700 1 -- 172.21.15.153:0/543891503 learned_addr learned my addr 172.21.15.153:0/543891503 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f74667fc700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74681058b0 0x7f7468100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f74677fe700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f74681081e0 0x7f7468101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f7466ffd700 1 -- 172.21.15.153:0/543891503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f74681081e0 msgr2=0x7f7468101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f7466ffd700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f74681081e0 0x7f7468101470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f7466ffd700 1 -- 172.21.15.153:0/543891503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74681058b0 msgr2=0x7f7468100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f7466ffd700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74681058b0 0x7f7468100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f7466ffd700 1 -- 172.21.15.153:0/543891503 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7468101d20 con 0x7f7468102f80 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f7466ffd700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7468102f80 0x7f74681028e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f745800ba50 tx=0x7f745800be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f745ffff700 1 -- 172.21.15.153:0/543891503 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458019070 con 0x7f7468102f80 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f745ffff700 1 -- 172.21.15.153:0/543891503 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7458014440 con 0x7f7468102f80 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f746d704700 1 -- 172.21.15.153:0/543891503 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74681a28e0 con 0x7f7468102f80 2024-06-09T22:12:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f745ffff700 1 -- 172.21.15.153:0/543891503 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74580075d0 con 0x7f7468102f80 2024-06-09T22:12:16.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.482+0000 7f746d704700 1 -- 172.21.15.153:0/543891503 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74681a2da0 con 0x7f7468102f80 2024-06-09T22:12:16.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.483+0000 7f746d704700 1 -- 172.21.15.153:0/543891503 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7468069180 con 0x7f7468102f80 2024-06-09T22:12:16.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.484+0000 7f745ffff700 1 -- 172.21.15.153:0/543891503 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f745801c070 con 0x7f7468102f80 2024-06-09T22:12:16.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.485+0000 7f745ffff700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f74540a2c20 0x7f74540a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:12:16.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.485+0000 7f745ffff700 1 -- 172.21.15.153:0/543891503 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f74580cd360 con 0x7f7468102f80 2024-06-09T22:12:16.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.485+0000 7f74667fc700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f74540a2c20 0x7f74540a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:12:16.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.485+0000 7f74667fc700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f74540a2c20 0x7f74540a50e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f74500039a0 tx=0x7f745000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:12:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.489+0000 7f745ffff700 1 -- 172.21.15.153:0/543891503 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f745809b4c0 con 0x7f7468102f80 2024-06-09T22:12:16.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:16.691+0000 7f746d704700 1 -- 172.21.15.153:0/543891503 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f74680628d0 con 0x7f74540a2c20 2024-06-09T22:12:17.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:17 smithi153 ceph-mon[36599]: osdmap e27: 4 total, 4 up, 4 in 2024-06-09T22:12:17.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:12:17.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:12:17.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:17.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:17 smithi194 ceph-mon[36856]: osdmap e27: 4 total, 4 up, 4 in 2024-06-09T22:12:17.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:12:17.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:12:17.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:17.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:17 smithi104 ceph-mon[32389]: osdmap e27: 4 total, 4 up, 4 in 2024-06-09T22:12:17.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:12:17.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:12:17.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:18.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:18 smithi194 ceph-mon[36856]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:12:18.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:18 smithi194 ceph-mon[36856]: osdmap e28: 4 total, 4 up, 4 in 2024-06-09T22:12:18.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:18 smithi194 ceph-mon[36856]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:18.381 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:18 smithi153 ceph-mon[36599]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:12:18.382 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:18 smithi153 ceph-mon[36599]: osdmap e28: 4 total, 4 up, 4 in 2024-06-09T22:12:18.382 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:18 smithi153 ceph-mon[36599]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:18.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:18 smithi104 ceph-mon[32389]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:12:18.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:18 smithi104 ceph-mon[32389]: osdmap e28: 4 total, 4 up, 4 in 2024-06-09T22:12:18.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:18 smithi104 ceph-mon[32389]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:19.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:19 smithi194 ceph-mon[36856]: osdmap e29: 4 total, 4 up, 4 in 2024-06-09T22:12:19.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:19 smithi153 ceph-mon[36599]: osdmap e29: 4 total, 4 up, 4 in 2024-06-09T22:12:19.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:19 smithi104 ceph-mon[32389]: osdmap e29: 4 total, 4 up, 4 in 2024-06-09T22:12:20.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:20 smithi194 ceph-mon[36856]: from='client.? 172.21.15.153:0/753615909' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0d2ceee-535e-48d8-90e7-5c5edb3eebe4"}]: dispatch 2024-06-09T22:12:20.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:20 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0d2ceee-535e-48d8-90e7-5c5edb3eebe4"}]: dispatch 2024-06-09T22:12:20.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:20 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0d2ceee-535e-48d8-90e7-5c5edb3eebe4"}]': finished 2024-06-09T22:12:20.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:20 smithi194 ceph-mon[36856]: osdmap e30: 5 total, 4 up, 5 in 2024-06-09T22:12:20.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:20.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:20 smithi194 ceph-mon[36856]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:20.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:20 smithi194 ceph-mon[36856]: from='client.? 172.21.15.153:0/146458846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:12:20.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:20 smithi153 ceph-mon[36599]: from='client.? 172.21.15.153:0/753615909' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0d2ceee-535e-48d8-90e7-5c5edb3eebe4"}]: dispatch 2024-06-09T22:12:20.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:20 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0d2ceee-535e-48d8-90e7-5c5edb3eebe4"}]: dispatch 2024-06-09T22:12:20.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:20 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0d2ceee-535e-48d8-90e7-5c5edb3eebe4"}]': finished 2024-06-09T22:12:20.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:20 smithi153 ceph-mon[36599]: osdmap e30: 5 total, 4 up, 5 in 2024-06-09T22:12:20.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:20.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:20 smithi153 ceph-mon[36599]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:20.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:20 smithi153 ceph-mon[36599]: from='client.? 172.21.15.153:0/146458846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:12:20.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:20 smithi104 ceph-mon[32389]: from='client.? 172.21.15.153:0/753615909' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0d2ceee-535e-48d8-90e7-5c5edb3eebe4"}]: dispatch 2024-06-09T22:12:20.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:20 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0d2ceee-535e-48d8-90e7-5c5edb3eebe4"}]: dispatch 2024-06-09T22:12:20.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:20 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0d2ceee-535e-48d8-90e7-5c5edb3eebe4"}]': finished 2024-06-09T22:12:20.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:20 smithi104 ceph-mon[32389]: osdmap e30: 5 total, 4 up, 5 in 2024-06-09T22:12:20.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:20.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:20 smithi104 ceph-mon[32389]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:20.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:20 smithi104 ceph-mon[32389]: from='client.? 172.21.15.153:0/146458846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:12:22.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:22 smithi194 ceph-mon[36856]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-06-09T22:12:22.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:22 smithi153 ceph-mon[36599]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-06-09T22:12:22.747 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:22 smithi104 ceph-mon[32389]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-06-09T22:12:24.417 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:24 smithi153 ceph-mon[36599]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-06-09T22:12:24.715 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:24 smithi104 ceph-mon[32389]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-06-09T22:12:24.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:24 smithi194 ceph-mon[36856]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-06-09T22:12:26.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:26 smithi153 ceph-mon[36599]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-09T22:12:26.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-09T22:12:26.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:26.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:26 smithi104 ceph-mon[32389]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-09T22:12:26.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-09T22:12:26.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:26.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:26 smithi194 ceph-mon[36856]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-09T22:12:26.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-09T22:12:26.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:27.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:27 smithi153 ceph-mon[36599]: Deploying daemon osd.4 on smithi153 2024-06-09T22:12:27.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:27 smithi104 ceph-mon[32389]: Deploying daemon osd.4 on smithi153 2024-06-09T22:12:27.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:27 smithi194 ceph-mon[36856]: Deploying daemon osd.4 on smithi153 2024-06-09T22:12:28.464 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:28 smithi153 ceph-mon[36599]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-06-09T22:12:28.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:28 smithi104 ceph-mon[32389]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-06-09T22:12:28.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:28 smithi194 ceph-mon[36856]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-06-09T22:12:30.589 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:30 smithi153 ceph-mon[36599]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-06-09T22:12:30.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:30 smithi104 ceph-mon[32389]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-06-09T22:12:30.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:30 smithi194 ceph-mon[36856]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-06-09T22:12:32.428 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:32 smithi153 ceph-mon[36599]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-06-09T22:12:32.429 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:32.429 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:32.429 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:32.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:32 smithi104 ceph-mon[32389]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-06-09T22:12:32.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:32.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:32.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:32.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:32 smithi194 ceph-mon[36856]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-06-09T22:12:32.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:32.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:32.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:34.602 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:34 smithi194 ceph-mon[36856]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:34.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:34 smithi153 ceph-mon[36599]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:34.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:34 smithi104 ceph-mon[32389]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:35.497 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 4 on host 'smithi153' 2024-06-09T22:12:35.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.506+0000 7f745ffff700 1 -- 172.21.15.153:0/543891503 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f74680628d0 con 0x7f74540a2c20 2024-06-09T22:12:35.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.509+0000 7f746d704700 1 -- 172.21.15.153:0/543891503 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f74540a2c20 msgr2=0x7f74540a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:12:35.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.509+0000 7f746d704700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f74540a2c20 0x7f74540a50e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f74500039a0 tx=0x7f745000b540 comp rx=0 tx=0).stop 2024-06-09T22:12:35.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.509+0000 7f746d704700 1 -- 172.21.15.153:0/543891503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7468102f80 msgr2=0x7f74681028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:12:35.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.509+0000 7f746d704700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7468102f80 0x7f74681028e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f745800ba50 tx=0x7f745800be10 comp rx=0 tx=0).stop 2024-06-09T22:12:35.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.509+0000 7f746d704700 1 -- 172.21.15.153:0/543891503 shutdown_connections 2024-06-09T22:12:35.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.509+0000 7f746d704700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f74540a2c20 0x7f74540a50e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:35.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.509+0000 7f746d704700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f74681081e0 0x7f7468101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:35.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.509+0000 7f746d704700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7468102f80 0x7f74681028e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:35.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.509+0000 7f746d704700 1 --2- 172.21.15.153:0/543891503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74681058b0 0x7f7468100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:35.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.509+0000 7f746d704700 1 -- 172.21.15.153:0/543891503 >> 172.21.15.153:0/543891503 conn(0x7f74680fc990 msgr2=0x7f74681039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:12:35.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.509+0000 7f746d704700 1 -- 172.21.15.153:0/543891503 shutdown_connections 2024-06-09T22:12:35.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:12:35.509+0000 7f746d704700 1 -- 172.21.15.153:0/543891503 wait complete. 2024-06-09T22:12:35.675 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:35 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:35.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:35 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:35.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:35 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:35.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:35 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:12:35.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:35 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:35.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:35 smithi153 ceph-mon[36599]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:35.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:35 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:35.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:35 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:35.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:35 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:35.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:35 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:35.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:35 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:35.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:35 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:35.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:35 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:12:35.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:35 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:35.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:35 smithi104 ceph-mon[32389]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:35.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:35 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:35.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:35 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:35.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:35 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:36.050 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:35 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:36.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:35 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:36.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:35 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:36.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:35 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:12:36.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:35 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:36.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:35 smithi194 ceph-mon[36856]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:36.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:35 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:36.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:35 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:36.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:35 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:36.126 DEBUG:teuthology.orchestra.run.smithi153:osd.4> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.4.service 2024-06-09T22:12:36.128 INFO:tasks.cephadm:Deploying osd.5 on smithi194 with /dev/vg_nvme/lv_3... 2024-06-09T22:12:36.128 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-09T22:12:36.371 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:12:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:36 smithi153 ceph-mon[36599]: from='osd.4 [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-09T22:12:36.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:36 smithi153 ceph-mon[36599]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-09T22:12:36.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:36 smithi194 ceph-mon[36856]: from='osd.4 [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-09T22:12:36.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:36 smithi194 ceph-mon[36856]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-09T22:12:36.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:36 smithi104 ceph-mon[32389]: from='osd.4 [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-09T22:12:36.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:36 smithi104 ceph-mon[32389]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-09T22:12:37.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:37 smithi153 ceph-mon[36599]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-09T22:12:37.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:37 smithi153 ceph-mon[36599]: from='osd.4 [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:12:37.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:37 smithi153 ceph-mon[36599]: osdmap e31: 5 total, 4 up, 5 in 2024-06-09T22:12:37.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:37 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:37.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:37 smithi153 ceph-mon[36599]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:12:37.626 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:37 smithi153 ceph-mon[36599]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:37.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:37 smithi104 ceph-mon[32389]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-09T22:12:37.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:37 smithi104 ceph-mon[32389]: from='osd.4 [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:12:37.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:37 smithi104 ceph-mon[32389]: osdmap e31: 5 total, 4 up, 5 in 2024-06-09T22:12:37.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:37 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:37.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:37 smithi104 ceph-mon[32389]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:12:37.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:37 smithi104 ceph-mon[32389]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:38.050 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:37 smithi194 ceph-mon[36856]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-09T22:12:38.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:37 smithi194 ceph-mon[36856]: from='osd.4 [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:12:38.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:37 smithi194 ceph-mon[36856]: osdmap e31: 5 total, 4 up, 5 in 2024-06-09T22:12:38.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:37 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:38.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:37 smithi194 ceph-mon[36856]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:12:38.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:37 smithi194 ceph-mon[36856]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:38.259 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-09T22:12:38.259 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-09T22:12:38.259 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-06-09T22:12:38.259 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-06-09T22:12:38.259 INFO:teuthology.orchestra.run.smithi194.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0118596 s, 884 MB/s 2024-06-09T22:12:38.260 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 2024-06-09T22:12:38.260 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-06-09T22:12:38.365 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:12:38 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4[48121]: 2024-06-09T22:12:38.101+0000 7f94a3ca4700 -1 osd.4 0 waiting for initial osdmap 2024-06-09T22:12:38.365 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:12:38 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4[48121]: 2024-06-09T22:12:38.110+0000 7f949ba8f700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:12:38.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:38 smithi153 ceph-mon[36599]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-09T22:12:38.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:38 smithi153 ceph-mon[36599]: osdmap e32: 5 total, 4 up, 5 in 2024-06-09T22:12:38.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:38 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:38.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:38 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:38.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:38 smithi153 ceph-mon[36599]: from='osd.4 ' entity='osd.4' 2024-06-09T22:12:38.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:38 smithi194 ceph-mon[36856]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-09T22:12:38.951 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:38 smithi194 ceph-mon[36856]: osdmap e32: 5 total, 4 up, 5 in 2024-06-09T22:12:38.951 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:38 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:38.951 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:38 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:38.951 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:38 smithi194 ceph-mon[36856]: from='osd.4 ' entity='osd.4' 2024-06-09T22:12:38.979 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_3 2024-06-09T22:12:38.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:38 smithi104 ceph-mon[32389]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-09T22:12:38.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:38 smithi104 ceph-mon[32389]: osdmap e32: 5 total, 4 up, 5 in 2024-06-09T22:12:38.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:38 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:38.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:38 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:38.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:38 smithi104 ceph-mon[32389]: from='osd.4 ' entity='osd.4' 2024-06-09T22:12:39.225 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:12:39.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: purged_snaps scrub starts 2024-06-09T22:12:39.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: purged_snaps scrub ok 2024-06-09T22:12:39.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:39.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: Detected new or changed devices on smithi153 2024-06-09T22:12:39.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:39.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:39.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:12:39.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:12:39.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi153 to 8544M 2024-06-09T22:12:39.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:39.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:39.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:12:39.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:39.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: osd.4 [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981] boot 2024-06-09T22:12:39.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: osdmap e33: 5 total, 5 up, 5 in 2024-06-09T22:12:39.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:39.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:39 smithi153 ceph-mon[36599]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:39.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: purged_snaps scrub starts 2024-06-09T22:12:39.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: purged_snaps scrub ok 2024-06-09T22:12:39.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:39.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: Detected new or changed devices on smithi153 2024-06-09T22:12:39.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:39.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:39.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:12:39.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:12:39.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi153 to 8544M 2024-06-09T22:12:39.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:39.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:39.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:12:39.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:39.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: osd.4 [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981] boot 2024-06-09T22:12:39.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: osdmap e33: 5 total, 5 up, 5 in 2024-06-09T22:12:39.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:39.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:39 smithi104 ceph-mon[32389]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:40.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: purged_snaps scrub starts 2024-06-09T22:12:40.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: purged_snaps scrub ok 2024-06-09T22:12:40.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:40.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: Detected new or changed devices on smithi153 2024-06-09T22:12:40.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:40.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:40.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:12:40.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:12:40.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi153 to 8544M 2024-06-09T22:12:40.016 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:40.017 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:40.017 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:12:40.017 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:40.017 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: osd.4 [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981] boot 2024-06-09T22:12:40.017 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: osdmap e33: 5 total, 5 up, 5 in 2024-06-09T22:12:40.017 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:12:40.017 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:39 smithi194 ceph-mon[36856]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-06-09T22:12:40.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.876+0000 7fac00e3f700 1 -- 172.21.15.194:0/118270107 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabfc102fb0 msgr2=0x7fabfc109860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:12:40.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.876+0000 7fac00e3f700 1 --2- 172.21.15.194:0/118270107 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabfc102fb0 0x7fabfc109860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fabf400b0a0 tx=0x7fabf400b3b0 comp rx=0 tx=0).stop 2024-06-09T22:12:40.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 -- 172.21.15.194:0/118270107 shutdown_connections 2024-06-09T22:12:40.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 --2- 172.21.15.194:0/118270107 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabfc102fb0 0x7fabfc109860 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:40.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 --2- 172.21.15.194:0/118270107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabfc101410 0x7fabfc101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:40.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 --2- 172.21.15.194:0/118270107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabfc102610 0x7fabfc102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:40.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 -- 172.21.15.194:0/118270107 >> 172.21.15.194:0/118270107 conn(0x7fabfc0fc9b0 msgr2=0x7fabfc0fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:12:40.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 -- 172.21.15.194:0/118270107 shutdown_connections 2024-06-09T22:12:40.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 -- 172.21.15.194:0/118270107 wait complete. 2024-06-09T22:12:40.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 Processor -- start 2024-06-09T22:12:40.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 -- start start 2024-06-09T22:12:40.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabfc101410 0x7fabfc198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:12:40.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabfc102610 0x7fabfc198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:12:40.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabfc102fb0 0x7fabfc19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:12:40.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fabfc10d070 con 0x7fabfc101410 2024-06-09T22:12:40.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fabfc10cef0 con 0x7fabfc102610 2024-06-09T22:12:40.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.877+0000 7fac00e3f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fabfc10d1f0 con 0x7fabfc102fb0 2024-06-09T22:12:40.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabfad9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabfc102fb0 0x7fabfc19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:12:40.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabfad9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabfc102fb0 0x7fabfc19fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46580/0 (socket says 172.21.15.194:46580) 2024-06-09T22:12:40.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabfad9d700 1 -- 172.21.15.194:0/3232667599 learned_addr learned my addr 172.21.15.194:0/3232667599 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-06-09T22:12:40.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabfa59c700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabfc101410 0x7fabfc198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:12:40.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabf9d9b700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabfc102610 0x7fabfc198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:12:40.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabfad9d700 1 -- 172.21.15.194:0/3232667599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabfc102610 msgr2=0x7fabfc198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:12:40.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabfad9d700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabfc102610 0x7fabfc198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:40.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabfad9d700 1 -- 172.21.15.194:0/3232667599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabfc101410 msgr2=0x7fabfc198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:12:40.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabfad9d700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabfc101410 0x7fabfc198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:12:40.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabfad9d700 1 -- 172.21.15.194:0/3232667599 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabfc1a0360 con 0x7fabfc102fb0 2024-06-09T22:12:40.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabfa59c700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabfc101410 0x7fabfc198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:12:40.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabfad9d700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabfc102fb0 0x7fabfc19fcc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fabf4009800 tx=0x7fabf400be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:12:40.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabeb7fe700 1 -- 172.21.15.194:0/3232667599 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf4021070 con 0x7fabfc102fb0 2024-06-09T22:12:40.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabeb7fe700 1 -- 172.21.15.194:0/3232667599 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabf4005520 con 0x7fabfc102fb0 2024-06-09T22:12:40.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fac00e3f700 1 -- 172.21.15.194:0/3232667599 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabfc1a05f0 con 0x7fabfc102fb0 2024-06-09T22:12:40.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fabeb7fe700 1 -- 172.21.15.194:0/3232667599 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf401b930 con 0x7fabfc102fb0 2024-06-09T22:12:40.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.878+0000 7fac00e3f700 1 -- 172.21.15.194:0/3232667599 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabfc1a09f0 con 0x7fabfc102fb0 2024-06-09T22:12:40.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.879+0000 7fac00e3f700 1 -- 172.21.15.194:0/3232667599 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabfc069180 con 0x7fabfc102fb0 2024-06-09T22:12:40.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.880+0000 7fabeb7fe700 1 -- 172.21.15.194:0/3232667599 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fabf4024070 con 0x7fabfc102fb0 2024-06-09T22:12:40.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.881+0000 7fabeb7fe700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fabe40a2c20 0x7fabe40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:12:40.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.881+0000 7fabeb7fe700 1 -- 172.21.15.194:0/3232667599 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fabf40cf430 con 0x7fabfc102fb0 2024-06-09T22:12:40.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.881+0000 7fabfa59c700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fabe40a2c20 0x7fabe40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:12:40.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.882+0000 7fabfa59c700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fabe40a2c20 0x7fabe40a50e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabfc102470 tx=0x7fabec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:12:40.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:40.885+0000 7fabeb7fe700 1 -- 172.21.15.194:0/3232667599 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fabf409d490 con 0x7fabfc102fb0 2024-06-09T22:12:41.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:12:41.092+0000 7fac00e3f700 1 -- 172.21.15.194:0/3232667599 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fabfc0fea90 con 0x7fabe40a2c20 2024-06-09T22:12:41.972 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:41 smithi194 ceph-mon[36856]: osdmap e34: 5 total, 5 up, 5 in 2024-06-09T22:12:41.972 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:41 smithi194 ceph-mon[36856]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:12:41.972 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:12:41.973 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:12:41.973 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:41.973 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:41 smithi194 ceph-mon[36856]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:41.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:41 smithi153 ceph-mon[36599]: osdmap e34: 5 total, 5 up, 5 in 2024-06-09T22:12:41.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:41 smithi153 ceph-mon[36599]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:12:41.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:12:41.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:12:41.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:41.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:41 smithi153 ceph-mon[36599]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:41.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:41 smithi104 ceph-mon[32389]: osdmap e34: 5 total, 5 up, 5 in 2024-06-09T22:12:41.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:41 smithi104 ceph-mon[32389]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:12:41.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:12:41.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:12:41.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:41.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:41 smithi104 ceph-mon[32389]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:44.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:44 smithi194 ceph-mon[36856]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:44.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:44 smithi194 ceph-mon[36856]: from='client.? 172.21.15.194:0/3462004431' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49a8983d-15aa-47ef-83b9-72ea75fe4505"}]: dispatch 2024-06-09T22:12:44.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:44 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49a8983d-15aa-47ef-83b9-72ea75fe4505"}]: dispatch 2024-06-09T22:12:44.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:44 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49a8983d-15aa-47ef-83b9-72ea75fe4505"}]': finished 2024-06-09T22:12:44.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:44 smithi194 ceph-mon[36856]: osdmap e35: 6 total, 5 up, 6 in 2024-06-09T22:12:44.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:44 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:12:44.701 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:44 smithi104 ceph-mon[32389]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:44.702 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:44 smithi104 ceph-mon[32389]: from='client.? 172.21.15.194:0/3462004431' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49a8983d-15aa-47ef-83b9-72ea75fe4505"}]: dispatch 2024-06-09T22:12:44.702 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:44 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49a8983d-15aa-47ef-83b9-72ea75fe4505"}]: dispatch 2024-06-09T22:12:44.702 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:44 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49a8983d-15aa-47ef-83b9-72ea75fe4505"}]': finished 2024-06-09T22:12:44.702 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:44 smithi104 ceph-mon[32389]: osdmap e35: 6 total, 5 up, 6 in 2024-06-09T22:12:44.702 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:44 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:12:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:44 smithi153 ceph-mon[36599]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:44 smithi153 ceph-mon[36599]: from='client.? 172.21.15.194:0/3462004431' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49a8983d-15aa-47ef-83b9-72ea75fe4505"}]: dispatch 2024-06-09T22:12:44.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:44 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49a8983d-15aa-47ef-83b9-72ea75fe4505"}]: dispatch 2024-06-09T22:12:44.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:44 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49a8983d-15aa-47ef-83b9-72ea75fe4505"}]': finished 2024-06-09T22:12:44.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:44 smithi153 ceph-mon[36599]: osdmap e35: 6 total, 5 up, 6 in 2024-06-09T22:12:44.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:44 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:12:45.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:45 smithi153 ceph-mon[36599]: from='client.? 172.21.15.194:0/1732846811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:12:45.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:45 smithi104 ceph-mon[32389]: from='client.? 172.21.15.194:0/1732846811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:12:45.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:45 smithi194 ceph-mon[36856]: from='client.? 172.21.15.194:0/1732846811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:12:46.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:46 smithi153 ceph-mon[36599]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:46.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:46 smithi104 ceph-mon[32389]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:46.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:46 smithi194 ceph-mon[36856]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:48.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:48 smithi194 ceph-mon[36856]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:48.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:48 smithi153 ceph-mon[36599]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:48.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:48 smithi104 ceph-mon[32389]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:50.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:50 smithi153 ceph-mon[36599]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:50.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:50 smithi104 ceph-mon[32389]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:50.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:50 smithi194 ceph-mon[36856]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:51.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-09T22:12:51.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:51.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-09T22:12:51.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:51.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-09T22:12:51.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:12:52.687 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:52 smithi194 ceph-mon[36856]: Deploying daemon osd.5 on smithi194 2024-06-09T22:12:52.687 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:52 smithi194 ceph-mon[36856]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:52.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:52 smithi153 ceph-mon[36599]: Deploying daemon osd.5 on smithi194 2024-06-09T22:12:52.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:52 smithi153 ceph-mon[36599]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:52.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:52 smithi104 ceph-mon[32389]: Deploying daemon osd.5 on smithi194 2024-06-09T22:12:52.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:52 smithi104 ceph-mon[32389]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:54.449 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:54 smithi194 ceph-mon[36856]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:54.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:54 smithi153 ceph-mon[36599]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:54.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:54 smithi104 ceph-mon[32389]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:56.499 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:56 smithi194 ceph-mon[36856]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:56.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:56 smithi153 ceph-mon[36599]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:56.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:56 smithi104 ceph-mon[32389]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:57.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:57.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:57.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:57.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:12:57 smithi104 ceph-mon[32389]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:58.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:58.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:58.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:58.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:12:57 smithi194 ceph-mon[36856]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:12:58.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:58.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:12:58.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:12:58.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:12:57 smithi153 ceph-mon[36599]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:13:00.523 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:00 smithi194 ceph-mon[36856]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:13:00.523 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:00 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:00.523 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:00 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:00.523 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:00 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:00.523 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:00 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:00.523 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:00 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:00.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.609+0000 7fabeb7fe700 1 -- 172.21.15.194:0/3232667599 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fabfc0fea90 con 0x7fabe40a2c20 2024-06-09T22:13:00.602 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 5 on host 'smithi194' 2024-06-09T22:13:00.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.612+0000 7fac00e3f700 1 -- 172.21.15.194:0/3232667599 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fabe40a2c20 msgr2=0x7fabe40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:00.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.612+0000 7fac00e3f700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fabe40a2c20 0x7fabe40a50e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabfc102470 tx=0x7fabec008040 comp rx=0 tx=0).stop 2024-06-09T22:13:00.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.612+0000 7fac00e3f700 1 -- 172.21.15.194:0/3232667599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabfc102fb0 msgr2=0x7fabfc19fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:00.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.612+0000 7fac00e3f700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabfc102fb0 0x7fabfc19fcc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fabf4009800 tx=0x7fabf400be20 comp rx=0 tx=0).stop 2024-06-09T22:13:00.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.612+0000 7fac00e3f700 1 -- 172.21.15.194:0/3232667599 shutdown_connections 2024-06-09T22:13:00.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.612+0000 7fac00e3f700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fabe40a2c20 0x7fabe40a50e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:00.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.612+0000 7fac00e3f700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabfc101410 0x7fabfc198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:00.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.612+0000 7fac00e3f700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabfc102fb0 0x7fabfc19fcc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:00.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.612+0000 7fac00e3f700 1 --2- 172.21.15.194:0/3232667599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabfc102610 0x7fabfc198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:00.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.612+0000 7fac00e3f700 1 -- 172.21.15.194:0/3232667599 >> 172.21.15.194:0/3232667599 conn(0x7fabfc0fc9b0 msgr2=0x7fabfc0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:13:00.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.612+0000 7fac00e3f700 1 -- 172.21.15.194:0/3232667599 shutdown_connections 2024-06-09T22:13:00.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:00.612+0000 7fac00e3f700 1 -- 172.21.15.194:0/3232667599 wait complete. 2024-06-09T22:13:00.734 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:00 smithi153 ceph-mon[36599]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:13:00.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:00 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:00.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:00 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:00.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:00 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:00.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:00 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:00.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:00 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:00.748 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:00 smithi104 ceph-mon[32389]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:13:00.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:00 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:00.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:00 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:00.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:00 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:00.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:00 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:00.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:00 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:01.271 DEBUG:teuthology.orchestra.run.smithi194:osd.5> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.5.service 2024-06-09T22:13:01.273 INFO:tasks.cephadm:Deploying osd.6 on smithi104 with /dev/vg_nvme/lv_2... 2024-06-09T22:13:01.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-09T22:13:01.522 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:13:01.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:01 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:01.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:01 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:01.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:01 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:01.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:01 smithi194 ceph-mon[36856]: from='osd.5 [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-09T22:13:01.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:01 smithi194 ceph-mon[36856]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-09T22:13:01.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:01 smithi194 ceph-mon[36856]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:13:01.839 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:01 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:01.840 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:01 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:01.840 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:01 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:01.840 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:01 smithi104 ceph-mon[32389]: from='osd.5 [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-09T22:13:01.840 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:01 smithi104 ceph-mon[32389]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-09T22:13:01.840 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:01 smithi104 ceph-mon[32389]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:13:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:01 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:01 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:01 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:01 smithi153 ceph-mon[36599]: from='osd.5 [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-09T22:13:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:01 smithi153 ceph-mon[36599]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-09T22:13:01.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:01 smithi153 ceph-mon[36599]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:13:02.776 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:02 smithi194 ceph-mon[36856]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-09T22:13:02.776 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:02 smithi194 ceph-mon[36856]: osdmap e36: 6 total, 5 up, 6 in 2024-06-09T22:13:02.776 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:02.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:02 smithi194 ceph-mon[36856]: from='osd.5 [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:13:02.777 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:02 smithi194 ceph-mon[36856]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:13:02.857 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:02 smithi104 ceph-mon[32389]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-09T22:13:02.858 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:02 smithi104 ceph-mon[32389]: osdmap e36: 6 total, 5 up, 6 in 2024-06-09T22:13:02.858 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:02.858 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:02 smithi104 ceph-mon[32389]: from='osd.5 [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:13:02.858 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:02 smithi104 ceph-mon[32389]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:13:02.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:02 smithi153 ceph-mon[36599]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-09T22:13:02.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:02 smithi153 ceph-mon[36599]: osdmap e36: 6 total, 5 up, 6 in 2024-06-09T22:13:02.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:02.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:02 smithi153 ceph-mon[36599]: from='osd.5 [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:13:02.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:02 smithi153 ceph-mon[36599]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:13:03.051 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:13:03 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:13:03.024+0000 7f4a9cb04700 -1 osd.5 0 waiting for initial osdmap 2024-06-09T22:13:03.051 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:13:03 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:13:03.032+0000 7f4a948ef700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:13:03.451 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-09T22:13:03.451 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-09T22:13:03.451 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-06-09T22:13:03.451 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-06-09T22:13:03.451 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132825 s, 789 MB/s 2024-06-09T22:13:03.452 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-06-09T22:13:03.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:03 smithi104 ceph-mon[32389]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-06-09T22:13:03.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:03 smithi104 ceph-mon[32389]: osdmap e37: 6 total, 5 up, 6 in 2024-06-09T22:13:03.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:03 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:03.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:03 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:03.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:03 smithi104 ceph-mon[32389]: from='osd.5 ' entity='osd.5' 2024-06-09T22:13:03.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:03 smithi104 ceph-mon[32389]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:13:03.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:03 smithi153 ceph-mon[36599]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-06-09T22:13:03.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:03 smithi153 ceph-mon[36599]: osdmap e37: 6 total, 5 up, 6 in 2024-06-09T22:13:03.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:03 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:03.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:03 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:03.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:03 smithi153 ceph-mon[36599]: from='osd.5 ' entity='osd.5' 2024-06-09T22:13:03.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:03 smithi153 ceph-mon[36599]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:13:04.012 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:03 smithi194 ceph-mon[36856]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-06-09T22:13:04.012 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:03 smithi194 ceph-mon[36856]: osdmap e37: 6 total, 5 up, 6 in 2024-06-09T22:13:04.012 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:03 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:04.012 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:03 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:04.012 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:03 smithi194 ceph-mon[36856]: from='osd.5 ' entity='osd.5' 2024-06-09T22:13:04.012 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:03 smithi194 ceph-mon[36856]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-06-09T22:13:04.146 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_2 2024-06-09T22:13:04.386 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:13:04.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: purged_snaps scrub starts 2024-06-09T22:13:04.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: purged_snaps scrub ok 2024-06-09T22:13:04.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:04.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: osd.5 [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494] boot 2024-06-09T22:13:04.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: osdmap e38: 6 total, 6 up, 6 in 2024-06-09T22:13:04.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:04.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: Detected new or changed devices on smithi194 2024-06-09T22:13:04.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:04.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:04.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:04.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:04.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi194 to 8544M 2024-06-09T22:13:04.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:04.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:04.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:04.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:04 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:04.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: purged_snaps scrub starts 2024-06-09T22:13:04.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: purged_snaps scrub ok 2024-06-09T22:13:04.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:04.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: osd.5 [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494] boot 2024-06-09T22:13:04.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: osdmap e38: 6 total, 6 up, 6 in 2024-06-09T22:13:04.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:04.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: Detected new or changed devices on smithi194 2024-06-09T22:13:04.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:04.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:04.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:04.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:04.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi194 to 8544M 2024-06-09T22:13:04.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:04.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:04.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:04.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:04 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:04.886 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: purged_snaps scrub starts 2024-06-09T22:13:04.886 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: purged_snaps scrub ok 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: osd.5 [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494] boot 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: osdmap e38: 6 total, 6 up, 6 in 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: Detected new or changed devices on smithi194 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi194 to 8544M 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:04.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:04 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:05.804 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:05 smithi104 ceph-mon[32389]: osdmap e39: 6 total, 6 up, 6 in 2024-06-09T22:13:05.804 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:05 smithi104 ceph-mon[32389]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:05.886 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:05 smithi194 ceph-mon[36856]: osdmap e39: 6 total, 6 up, 6 in 2024-06-09T22:13:05.886 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:05 smithi194 ceph-mon[36856]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:05.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:05 smithi153 ceph-mon[36599]: osdmap e39: 6 total, 6 up, 6 in 2024-06-09T22:13:05.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:05 smithi153 ceph-mon[36599]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:06.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.054+0000 7f1cea448700 1 -- 172.21.15.104:0/236682679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce4101410 msgr2=0x7f1ce4101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.054+0000 7f1cea448700 1 --2- 172.21.15.104:0/236682679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce4101410 0x7f1ce4101830 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4009990 tx=0x7f1cd4009ca0 comp rx=0 tx=0).stop 2024-06-09T22:13:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.054+0000 7f1cea448700 1 -- 172.21.15.104:0/236682679 shutdown_connections 2024-06-09T22:13:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.054+0000 7f1cea448700 1 --2- 172.21.15.104:0/236682679 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce4102fb0 0x7f1ce4109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.054+0000 7f1cea448700 1 --2- 172.21.15.104:0/236682679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce4101410 0x7f1ce4101830 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.054+0000 7f1cea448700 1 --2- 172.21.15.104:0/236682679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4102610 0x7f1ce4102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.054+0000 7f1cea448700 1 -- 172.21.15.104:0/236682679 >> 172.21.15.104:0/236682679 conn(0x7f1ce40fc9b0 msgr2=0x7f1ce40fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:13:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.054+0000 7f1cea448700 1 -- 172.21.15.104:0/236682679 shutdown_connections 2024-06-09T22:13:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.054+0000 7f1cea448700 1 -- 172.21.15.104:0/236682679 wait complete. 2024-06-09T22:13:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.054+0000 7f1cea448700 1 Processor -- start 2024-06-09T22:13:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1cea448700 1 -- start start 2024-06-09T22:13:06.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1cea448700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4101410 0x7f1ce41964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:13:06.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1cea448700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce4102610 0x7f1ce4196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:13:06.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1cea448700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce4102fb0 0x7f1ce419dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:13:06.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1cea448700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce410bf40 con 0x7f1ce4102fb0 2024-06-09T22:13:06.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1cea448700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce410bdc0 con 0x7f1ce4101410 2024-06-09T22:13:06.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1cea448700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce410c0c0 con 0x7f1ce4102610 2024-06-09T22:13:06.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce89e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce4102fb0 0x7f1ce419dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:13:06.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4101410 0x7f1ce41964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:13:06.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4101410 0x7f1ce41964f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:55862/0 (socket says 172.21.15.104:55862) 2024-06-09T22:13:06.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce89e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce4102fb0 0x7f1ce419dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36732/0 (socket says 172.21.15.104:36732) 2024-06-09T22:13:06.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce3fff700 1 -- 172.21.15.104:0/2412334376 learned_addr learned my addr 172.21.15.104:0/2412334376 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:13:06.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce37fe700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce4102610 0x7f1ce4196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:13:06.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce3fff700 1 -- 172.21.15.104:0/2412334376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce4102610 msgr2=0x7f1ce4196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:06.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce3fff700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce4102610 0x7f1ce4196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce3fff700 1 -- 172.21.15.104:0/2412334376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce4102fb0 msgr2=0x7f1ce419dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce3fff700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce4102fb0 0x7f1ce419dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce3fff700 1 -- 172.21.15.104:0/2412334376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce419e150 con 0x7f1ce4101410 2024-06-09T22:13:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce37fe700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce4102610 0x7f1ce4196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:13:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce89e5700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce4102fb0 0x7f1ce419dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:13:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.055+0000 7f1ce3fff700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4101410 0x7f1ce41964f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4000c00 tx=0x7f1cd4005710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:13:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.056+0000 7f1ce17fa700 1 -- 172.21.15.104:0/2412334376 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd400b610 con 0x7f1ce4101410 2024-06-09T22:13:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.056+0000 7f1cea448700 1 -- 172.21.15.104:0/2412334376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce419e3e0 con 0x7f1ce4101410 2024-06-09T22:13:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.056+0000 7f1ce17fa700 1 -- 172.21.15.104:0/2412334376 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd4025070 con 0x7f1ce4101410 2024-06-09T22:13:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.056+0000 7f1ce17fa700 1 -- 172.21.15.104:0/2412334376 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd40206c0 con 0x7f1ce4101410 2024-06-09T22:13:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.056+0000 7f1cea448700 1 -- 172.21.15.104:0/2412334376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce419e890 con 0x7f1ce4101410 2024-06-09T22:13:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.057+0000 7f1cea448700 1 -- 172.21.15.104:0/2412334376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ce4069180 con 0x7f1ce4101410 2024-06-09T22:13:06.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.058+0000 7f1ce17fa700 1 -- 172.21.15.104:0/2412334376 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f1cd4009d10 con 0x7f1ce4101410 2024-06-09T22:13:06.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.058+0000 7f1ce17fa700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1ccc0a2c20 0x7f1ccc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:13:06.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.059+0000 7f1ce37fe700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1ccc0a2c20 0x7f1ccc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:13:06.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.059+0000 7f1ce17fa700 1 -- 172.21.15.104:0/2412334376 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f1cd40d1d80 con 0x7f1ce4101410 2024-06-09T22:13:06.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.059+0000 7f1ce37fe700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1ccc0a2c20 0x7f1ccc0a50e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009be0 tx=0x7f1cd8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:13:06.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.062+0000 7f1ce17fa700 1 -- 172.21.15.104:0/2412334376 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1cd409fb70 con 0x7f1ce4101410 2024-06-09T22:13:06.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:06.270+0000 7f1cea448700 1 -- 172.21.15.104:0/2412334376 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f1ce40628d0 con 0x7f1ccc0a2c20 2024-06-09T22:13:06.872 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:06 smithi104 ceph-mon[32389]: osdmap e40: 6 total, 6 up, 6 in 2024-06-09T22:13:06.873 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:06 smithi104 ceph-mon[32389]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:13:06.873 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:13:06.873 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:13:06.873 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:06.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:06 smithi153 ceph-mon[36599]: osdmap e40: 6 total, 6 up, 6 in 2024-06-09T22:13:06.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:06 smithi153 ceph-mon[36599]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:13:06.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:13:06.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:13:06.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:07.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:06 smithi194 ceph-mon[36856]: osdmap e40: 6 total, 6 up, 6 in 2024-06-09T22:13:07.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:06 smithi194 ceph-mon[36856]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:13:07.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:13:07.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:13:07.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:07.884 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:07 smithi104 ceph-mon[32389]: osdmap e41: 6 total, 6 up, 6 in 2024-06-09T22:13:07.884 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:07 smithi104 ceph-mon[32389]: pgmap v142: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-06-09T22:13:07.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:07 smithi153 ceph-mon[36599]: osdmap e41: 6 total, 6 up, 6 in 2024-06-09T22:13:07.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:07 smithi153 ceph-mon[36599]: pgmap v142: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-06-09T22:13:08.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:07 smithi194 ceph-mon[36856]: osdmap e41: 6 total, 6 up, 6 in 2024-06-09T22:13:08.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:07 smithi194 ceph-mon[36856]: pgmap v142: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-06-09T22:13:08.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:08 smithi153 ceph-mon[36599]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-06-09T22:13:08.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:08 smithi153 ceph-mon[36599]: osdmap e42: 6 total, 6 up, 6 in 2024-06-09T22:13:08.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:08 smithi104 ceph-mon[32389]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-06-09T22:13:08.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:08 smithi104 ceph-mon[32389]: osdmap e42: 6 total, 6 up, 6 in 2024-06-09T22:13:09.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:08 smithi194 ceph-mon[36856]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-06-09T22:13:09.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:08 smithi194 ceph-mon[36856]: osdmap e42: 6 total, 6 up, 6 in 2024-06-09T22:13:09.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:09 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3361594568' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08563350-3ded-4ff6-a214-12d9242f984a"}]: dispatch 2024-06-09T22:13:09.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:09 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3361594568' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08563350-3ded-4ff6-a214-12d9242f984a"}]': finished 2024-06-09T22:13:09.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:09 smithi104 ceph-mon[32389]: osdmap e43: 7 total, 6 up, 7 in 2024-06-09T22:13:09.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:09 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:09.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:09 smithi104 ceph-mon[32389]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-06-09T22:13:09.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:09 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3361594568' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08563350-3ded-4ff6-a214-12d9242f984a"}]: dispatch 2024-06-09T22:13:09.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:09 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3361594568' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08563350-3ded-4ff6-a214-12d9242f984a"}]': finished 2024-06-09T22:13:09.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:09 smithi153 ceph-mon[36599]: osdmap e43: 7 total, 6 up, 7 in 2024-06-09T22:13:09.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:09 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:09.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:09 smithi153 ceph-mon[36599]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-06-09T22:13:10.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:09 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3361594568' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08563350-3ded-4ff6-a214-12d9242f984a"}]: dispatch 2024-06-09T22:13:10.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:09 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3361594568' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08563350-3ded-4ff6-a214-12d9242f984a"}]': finished 2024-06-09T22:13:10.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:09 smithi194 ceph-mon[36856]: osdmap e43: 7 total, 6 up, 7 in 2024-06-09T22:13:10.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:09 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:10.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:09 smithi194 ceph-mon[36856]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-06-09T22:13:10.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:10 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3229982864' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:13:10.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:10 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3229982864' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:13:11.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:10 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3229982864' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:13:11.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:11 smithi153 ceph-mon[36599]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-06-09T22:13:11.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:11 smithi104 ceph-mon[32389]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-06-09T22:13:12.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:11 smithi194 ceph-mon[36856]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-06-09T22:13:12.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:12 smithi153 ceph-mon[36599]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-06-09T22:13:12.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:12 smithi153 ceph-mon[36599]: Cluster is now healthy 2024-06-09T22:13:12.998 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:12 smithi104 ceph-mon[32389]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-06-09T22:13:12.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:12 smithi104 ceph-mon[32389]: Cluster is now healthy 2024-06-09T22:13:13.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:12 smithi194 ceph-mon[36856]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-06-09T22:13:13.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:12 smithi194 ceph-mon[36856]: Cluster is now healthy 2024-06-09T22:13:13.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:13 smithi104 ceph-mon[32389]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-06-09T22:13:13.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:13 smithi153 ceph-mon[36599]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-06-09T22:13:14.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:13 smithi194 ceph-mon[36856]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-06-09T22:13:16.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:16 smithi153 ceph-mon[36599]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-09T22:13:16.499 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:16 smithi104 ceph-mon[32389]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-09T22:13:16.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:16 smithi194 ceph-mon[36856]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-09T22:13:17.637 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-09T22:13:17.637 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:17.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-09T22:13:17.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:17.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-09T22:13:17.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:18.722 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:18 smithi104 ceph-mon[32389]: Deploying daemon osd.6 on smithi104 2024-06-09T22:13:18.722 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:18 smithi104 ceph-mon[32389]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-06-09T22:13:18.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:18 smithi153 ceph-mon[36599]: Deploying daemon osd.6 on smithi104 2024-06-09T22:13:18.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:18 smithi153 ceph-mon[36599]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-06-09T22:13:18.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:18 smithi194 ceph-mon[36856]: Deploying daemon osd.6 on smithi104 2024-06-09T22:13:18.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:18 smithi194 ceph-mon[36856]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-06-09T22:13:20.731 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:20 smithi104 ceph-mon[32389]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-06-09T22:13:20.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:20 smithi153 ceph-mon[36599]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-06-09T22:13:20.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:20 smithi194 ceph-mon[36856]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-06-09T22:13:22.605 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:22 smithi104 ceph-mon[32389]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-06-09T22:13:22.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:22 smithi153 ceph-mon[36599]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-06-09T22:13:22.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:22 smithi194 ceph-mon[36856]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-06-09T22:13:23.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:23 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:23.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:23 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:23.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:23 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:23.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:23 smithi153 ceph-mon[36599]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:23.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:23.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:23.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:23 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:23.999 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:23 smithi104 ceph-mon[32389]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:24.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:24.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:24.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:23 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:24.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:23 smithi194 ceph-mon[36856]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:26.641 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:26 smithi104 ceph-mon[32389]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:26.641 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:26.641 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:26.641 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:26.641 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:26.642 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:26.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.732+0000 7f1ce17fa700 1 -- 172.21.15.104:0/2412334376 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1ce40628d0 con 0x7f1ccc0a2c20 2024-06-09T22:13:26.725 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 6 on host 'smithi104' 2024-06-09T22:13:26.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.734+0000 7f1cea448700 1 -- 172.21.15.104:0/2412334376 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1ccc0a2c20 msgr2=0x7f1ccc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:26.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.734+0000 7f1cea448700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1ccc0a2c20 0x7f1ccc0a50e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009be0 tx=0x7f1cd8009260 comp rx=0 tx=0).stop 2024-06-09T22:13:26.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.734+0000 7f1cea448700 1 -- 172.21.15.104:0/2412334376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4101410 msgr2=0x7f1ce41964f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:26.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.734+0000 7f1cea448700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4101410 0x7f1ce41964f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4000c00 tx=0x7f1cd4005710 comp rx=0 tx=0).stop 2024-06-09T22:13:26.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.735+0000 7f1cea448700 1 -- 172.21.15.104:0/2412334376 shutdown_connections 2024-06-09T22:13:26.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.735+0000 7f1cea448700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1ccc0a2c20 0x7f1ccc0a50e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:26.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.735+0000 7f1cea448700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce4102fb0 0x7f1ce419dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:26.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.735+0000 7f1cea448700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4101410 0x7f1ce41964f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:26.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.735+0000 7f1cea448700 1 --2- 172.21.15.104:0/2412334376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce4102610 0x7f1ce4196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:26.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.735+0000 7f1cea448700 1 -- 172.21.15.104:0/2412334376 >> 172.21.15.104:0/2412334376 conn(0x7f1ce40fc9b0 msgr2=0x7f1ce4105910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:13:26.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.735+0000 7f1cea448700 1 -- 172.21.15.104:0/2412334376 shutdown_connections 2024-06-09T22:13:26.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:13:26.735+0000 7f1cea448700 1 -- 172.21.15.104:0/2412334376 wait complete. 2024-06-09T22:13:26.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:26 smithi153 ceph-mon[36599]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:26.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:26.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:26.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:26.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:26.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:26.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:26 smithi194 ceph-mon[36856]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:26.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:26.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:26.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:26.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:26.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:27.405 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:27 smithi104 ceph-mon[32389]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-09T22:13:27.405 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:27 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:27.405 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:27 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:27.405 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:27 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:27.431 DEBUG:teuthology.orchestra.run.smithi104:osd.6> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.6.service 2024-06-09T22:13:27.433 INFO:tasks.cephadm:Deploying osd.7 on smithi153 with /dev/vg_nvme/lv_2... 2024-06-09T22:13:27.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-09T22:13:27.678 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.b/config 2024-06-09T22:13:27.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:27 smithi153 ceph-mon[36599]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-09T22:13:27.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:27 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:27.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:27 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:27.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:27 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:27.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:27 smithi194 ceph-mon[36856]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-09T22:13:27.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:27 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:27.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:27 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:27.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:27 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:28.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:28 smithi153 ceph-mon[36599]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:28.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:28 smithi153 ceph-mon[36599]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-09T22:13:28.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:28 smithi153 ceph-mon[36599]: osdmap e44: 7 total, 6 up, 7 in 2024-06-09T22:13:28.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:28 smithi153 ceph-mon[36599]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-06-09T22:13:28.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:28 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:28.596 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:28 smithi104 ceph-mon[32389]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:28.596 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:28 smithi104 ceph-mon[32389]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-09T22:13:28.596 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:28 smithi104 ceph-mon[32389]: osdmap e44: 7 total, 6 up, 7 in 2024-06-09T22:13:28.596 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:28 smithi104 ceph-mon[32389]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-06-09T22:13:28.596 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:28 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:28.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:28 smithi194 ceph-mon[36856]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:28.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:28 smithi194 ceph-mon[36856]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-09T22:13:28.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:28 smithi194 ceph-mon[36856]: osdmap e44: 7 total, 6 up, 7 in 2024-06-09T22:13:28.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:28 smithi194 ceph-mon[36856]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-06-09T22:13:28.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:28 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:28.999 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:13:28 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:13:28.781+0000 7f81079d4700 -1 osd.6 0 waiting for initial osdmap 2024-06-09T22:13:28.999 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:13:28 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:13:28.790+0000 7f80ff7bf700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:13:29.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:29 smithi153 ceph-mon[36599]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-06-09T22:13:29.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:29 smithi153 ceph-mon[36599]: osdmap e45: 7 total, 6 up, 7 in 2024-06-09T22:13:29.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:29 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:29.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:29 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:29.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:29 smithi153 ceph-mon[36599]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' 2024-06-09T22:13:29.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:29 smithi104 ceph-mon[32389]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-06-09T22:13:29.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:29 smithi104 ceph-mon[32389]: osdmap e45: 7 total, 6 up, 7 in 2024-06-09T22:13:29.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:29 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:29.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:29 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:29.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:29 smithi104 ceph-mon[32389]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' 2024-06-09T22:13:29.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:29 smithi194 ceph-mon[36856]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-06-09T22:13:29.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:29 smithi194 ceph-mon[36856]: osdmap e45: 7 total, 6 up, 7 in 2024-06-09T22:13:29.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:29 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:29.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:29 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:29.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:29 smithi194 ceph-mon[36856]: from='osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419]' entity='osd.6' 2024-06-09T22:13:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-09T22:13:29.864 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-09T22:13:29.864 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-06-09T22:13:29.864 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-06-09T22:13:29.864 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0129998 s, 807 MB/s 2024-06-09T22:13:29.864 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 2024-06-09T22:13:29.864 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-06-09T22:13:30.454 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:30 smithi104 ceph-mon[32389]: purged_snaps scrub starts 2024-06-09T22:13:30.455 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:30 smithi104 ceph-mon[32389]: purged_snaps scrub ok 2024-06-09T22:13:30.455 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:30 smithi104 ceph-mon[32389]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:30.455 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:30.455 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:30 smithi104 ceph-mon[32389]: osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419] boot 2024-06-09T22:13:30.455 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:30 smithi104 ceph-mon[32389]: osdmap e46: 7 total, 7 up, 7 in 2024-06-09T22:13:30.455 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:30.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:30 smithi153 ceph-mon[36599]: purged_snaps scrub starts 2024-06-09T22:13:30.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:30 smithi153 ceph-mon[36599]: purged_snaps scrub ok 2024-06-09T22:13:30.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:30 smithi153 ceph-mon[36599]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:30.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:30.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:30 smithi153 ceph-mon[36599]: osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419] boot 2024-06-09T22:13:30.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:30 smithi153 ceph-mon[36599]: osdmap e46: 7 total, 7 up, 7 in 2024-06-09T22:13:30.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:30.557 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-06-09T22:13:30.800 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.b/config 2024-06-09T22:13:30.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:30 smithi194 ceph-mon[36856]: purged_snaps scrub starts 2024-06-09T22:13:30.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:30 smithi194 ceph-mon[36856]: purged_snaps scrub ok 2024-06-09T22:13:30.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:30 smithi194 ceph-mon[36856]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-06-09T22:13:30.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:30.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:30 smithi194 ceph-mon[36856]: osd.6 [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419] boot 2024-06-09T22:13:30.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:30 smithi194 ceph-mon[36856]: osdmap e46: 7 total, 7 up, 7 in 2024-06-09T22:13:30.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:13:32.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: Detected new or changed devices on smithi104 2024-06-09T22:13:32.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:32.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:32.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:32.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:32.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:32.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi104 to 5696M 2024-06-09T22:13:32.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:32.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:32.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:32.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:32.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:32.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:31 smithi153 ceph-mon[36599]: osdmap e47: 7 total, 7 up, 7 in 2024-06-09T22:13:32.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: Detected new or changed devices on smithi104 2024-06-09T22:13:32.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:32.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:32.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:32.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:32.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:32.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi104 to 5696M 2024-06-09T22:13:32.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:32.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:32.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:32.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:32.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:32.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:31 smithi104 ceph-mon[32389]: osdmap e47: 7 total, 7 up, 7 in 2024-06-09T22:13:32.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: Detected new or changed devices on smithi104 2024-06-09T22:13:32.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:32.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:32.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:32.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:32.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:32.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi104 to 5696M 2024-06-09T22:13:32.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:32.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:32.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:32.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:32.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:32.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:31 smithi194 ceph-mon[36856]: osdmap e47: 7 total, 7 up, 7 in 2024-06-09T22:13:32.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.453+0000 7f27c9890700 1 -- 172.21.15.153:0/967662572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c4104090 msgr2=0x7f27c410a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:32.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.453+0000 7f27c9890700 1 --2- 172.21.15.153:0/967662572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c4104090 0x7f27c410a940 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f27b40099c0 tx=0x7f27b4009cd0 comp rx=0 tx=0).stop 2024-06-09T22:13:32.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.453+0000 7f27c9890700 1 -- 172.21.15.153:0/967662572 shutdown_connections 2024-06-09T22:13:32.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.453+0000 7f27c9890700 1 --2- 172.21.15.153:0/967662572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c41024f0 0x7f27c4102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:32.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.453+0000 7f27c9890700 1 --2- 172.21.15.153:0/967662572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c4104090 0x7f27c410a940 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:32.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.453+0000 7f27c9890700 1 --2- 172.21.15.153:0/967662572 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27c41036f0 0x7f27c4103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:32.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.453+0000 7f27c9890700 1 -- 172.21.15.153:0/967662572 >> 172.21.15.153:0/967662572 conn(0x7f27c40fda90 msgr2=0x7f27c40ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:13:32.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.453+0000 7f27c9890700 1 -- 172.21.15.153:0/967662572 shutdown_connections 2024-06-09T22:13:32.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.453+0000 7f27c9890700 1 -- 172.21.15.153:0/967662572 wait complete. 2024-06-09T22:13:32.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.454+0000 7f27c9890700 1 Processor -- start 2024-06-09T22:13:32.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.454+0000 7f27c9890700 1 -- start start 2024-06-09T22:13:32.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.454+0000 7f27c9890700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27c41024f0 0x7f27c4076230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:13:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.454+0000 7f27c9890700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c41036f0 0x7f27c4074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:13:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.454+0000 7f27c9890700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c4104090 0x7f27c4074dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:13:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.454+0000 7f27c9890700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f27c410e0c0 con 0x7f27c41036f0 2024-06-09T22:13:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.454+0000 7f27c9890700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27c410df40 con 0x7f27c4104090 2024-06-09T22:13:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.454+0000 7f27c9890700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f27c410e240 con 0x7f27c41024f0 2024-06-09T22:13:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.454+0000 7f27c37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c4104090 0x7f27c4074dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:13:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.454+0000 7f27c37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c4104090 0x7f27c4074dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48250/0 (socket says 172.21.15.153:48250) 2024-06-09T22:13:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.454+0000 7f27c37fe700 1 -- 172.21.15.153:0/435718599 learned_addr learned my addr 172.21.15.153:0/435718599 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-09T22:13:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.454+0000 7f27c27fc700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c41036f0 0x7f27c4074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:13:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.455+0000 7f27c2ffd700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27c41024f0 0x7f27c4076230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:13:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.455+0000 7f27c37fe700 1 -- 172.21.15.153:0/435718599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27c41024f0 msgr2=0x7f27c4076230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.455+0000 7f27c37fe700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27c41024f0 0x7f27c4076230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:32.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.455+0000 7f27c37fe700 1 -- 172.21.15.153:0/435718599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c41036f0 msgr2=0x7f27c4074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:32.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.455+0000 7f27c37fe700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c41036f0 0x7f27c4074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:32.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.455+0000 7f27c37fe700 1 -- 172.21.15.153:0/435718599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c4075300 con 0x7f27c4104090 2024-06-09T22:13:32.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.455+0000 7f27c37fe700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c4104090 0x7f27c4074dc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009990 tx=0x7f27b40047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:13:32.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.455+0000 7f27c888e700 1 -- 172.21.15.153:0/435718599 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b4003d30 con 0x7f27c4104090 2024-06-09T22:13:32.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.455+0000 7f27c9890700 1 -- 172.21.15.153:0/435718599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c4075590 con 0x7f27c4104090 2024-06-09T22:13:32.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.455+0000 7f27c888e700 1 -- 172.21.15.153:0/435718599 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27b4004830 con 0x7f27c4104090 2024-06-09T22:13:32.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.455+0000 7f27c888e700 1 -- 172.21.15.153:0/435718599 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b4005590 con 0x7f27c4104090 2024-06-09T22:13:32.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.455+0000 7f27c9890700 1 -- 172.21.15.153:0/435718599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c41a6f80 con 0x7f27c4104090 2024-06-09T22:13:32.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.456+0000 7f27c9890700 1 -- 172.21.15.153:0/435718599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27c4069180 con 0x7f27c4104090 2024-06-09T22:13:32.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.457+0000 7f27c888e700 1 -- 172.21.15.153:0/435718599 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f27b40049e0 con 0x7f27c4104090 2024-06-09T22:13:32.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.458+0000 7f27c888e700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f27ac0a2c20 0x7f27ac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:13:32.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.458+0000 7f27c888e700 1 -- 172.21.15.153:0/435718599 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f27b40cf4a0 con 0x7f27c4104090 2024-06-09T22:13:32.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.458+0000 7f27c2ffd700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f27ac0a2c20 0x7f27ac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:13:32.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.459+0000 7f27c2ffd700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f27ac0a2c20 0x7f27ac0a50e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f27c4103550 tx=0x7f27bc009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:13:32.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.462+0000 7f27c888e700 1 -- 172.21.15.153:0/435718599 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f27b4013070 con 0x7f27c4104090 2024-06-09T22:13:32.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:32.665+0000 7f27c9890700 1 -- 172.21.15.153:0/435718599 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f27c40628d0 con 0x7f27ac0a2c20 2024-06-09T22:13:33.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:13:33.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:13:33.235 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:33.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:13:33.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:13:33.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:33.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:13:33.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:13:33.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:34.203 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:33 smithi194 ceph-mon[36856]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:13:34.203 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:33 smithi194 ceph-mon[36856]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:34.216 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:33 smithi153 ceph-mon[36599]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:13:34.216 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:33 smithi153 ceph-mon[36599]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:34.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:33 smithi104 ceph-mon[32389]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:13:34.249 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:33 smithi104 ceph-mon[32389]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:36 smithi153 ceph-mon[36599]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:36 smithi153 ceph-mon[36599]: from='client.? 172.21.15.153:0/206871967' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b23baa2c-c27d-43a3-9820-8626c26d3f5f"}]: dispatch 2024-06-09T22:13:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:36 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b23baa2c-c27d-43a3-9820-8626c26d3f5f"}]: dispatch 2024-06-09T22:13:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:36 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b23baa2c-c27d-43a3-9820-8626c26d3f5f"}]': finished 2024-06-09T22:13:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:36 smithi153 ceph-mon[36599]: osdmap e48: 8 total, 7 up, 8 in 2024-06-09T22:13:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:36 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:36 smithi153 ceph-mon[36599]: from='client.? 172.21.15.153:0/2749820946' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:13:36.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:36 smithi104 ceph-mon[32389]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:36.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:36 smithi104 ceph-mon[32389]: from='client.? 172.21.15.153:0/206871967' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b23baa2c-c27d-43a3-9820-8626c26d3f5f"}]: dispatch 2024-06-09T22:13:36.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:36 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b23baa2c-c27d-43a3-9820-8626c26d3f5f"}]: dispatch 2024-06-09T22:13:36.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:36 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b23baa2c-c27d-43a3-9820-8626c26d3f5f"}]': finished 2024-06-09T22:13:36.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:36 smithi104 ceph-mon[32389]: osdmap e48: 8 total, 7 up, 8 in 2024-06-09T22:13:36.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:36.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:36 smithi104 ceph-mon[32389]: from='client.? 172.21.15.153:0/2749820946' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:13:36.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:36 smithi194 ceph-mon[36856]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:36.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:36 smithi194 ceph-mon[36856]: from='client.? 172.21.15.153:0/206871967' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b23baa2c-c27d-43a3-9820-8626c26d3f5f"}]: dispatch 2024-06-09T22:13:36.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:36 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b23baa2c-c27d-43a3-9820-8626c26d3f5f"}]: dispatch 2024-06-09T22:13:36.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:36 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b23baa2c-c27d-43a3-9820-8626c26d3f5f"}]': finished 2024-06-09T22:13:36.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:36 smithi194 ceph-mon[36856]: osdmap e48: 8 total, 7 up, 8 in 2024-06-09T22:13:36.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:36 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:36.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:36 smithi194 ceph-mon[36856]: from='client.? 172.21.15.153:0/2749820946' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:13:38.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:38 smithi153 ceph-mon[36599]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:38.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:38 smithi104 ceph-mon[32389]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:38.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:38 smithi194 ceph-mon[36856]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:40.578 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:40 smithi153 ceph-mon[36599]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:40.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:40 smithi104 ceph-mon[32389]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:40.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:40 smithi194 ceph-mon[36856]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:42.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:42 smithi153 ceph-mon[36599]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:42.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:42 smithi104 ceph-mon[32389]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:42.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:42 smithi194 ceph-mon[36856]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:43.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:43 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-09T22:13:43.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:43 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:43.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:43 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-09T22:13:43.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:43 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:43.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:43 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-09T22:13:43.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:43 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:44.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:44 smithi153 ceph-mon[36599]: Deploying daemon osd.7 on smithi153 2024-06-09T22:13:44.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:44 smithi153 ceph-mon[36599]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:44.651 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:44 smithi104 ceph-mon[32389]: Deploying daemon osd.7 on smithi153 2024-06-09T22:13:44.651 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:44 smithi104 ceph-mon[32389]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:44.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:44 smithi194 ceph-mon[36856]: Deploying daemon osd.7 on smithi153 2024-06-09T22:13:44.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:44 smithi194 ceph-mon[36856]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:46.644 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:46 smithi153 ceph-mon[36599]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:46.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:46 smithi104 ceph-mon[32389]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:46.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:46 smithi194 ceph-mon[36856]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:48.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:48 smithi153 ceph-mon[36599]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:48.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:48 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:48.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:48 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:48.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:48 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:48.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:48 smithi104 ceph-mon[32389]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:48.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:48 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:48.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:48 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:48.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:48 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:48.801 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:48 smithi194 ceph-mon[36856]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:48.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:48 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:48.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:48 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:48.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:48 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:50.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:50 smithi153 ceph-mon[36599]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:50.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:50 smithi104 ceph-mon[32389]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:50.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:50 smithi194 ceph-mon[36856]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:51.470 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 7 on host 'smithi153' 2024-06-09T22:13:51.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.476+0000 7f27c888e700 1 -- 172.21.15.153:0/435718599 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f27c40628d0 con 0x7f27ac0a2c20 2024-06-09T22:13:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.479+0000 7f27c9890700 1 -- 172.21.15.153:0/435718599 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f27ac0a2c20 msgr2=0x7f27ac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.479+0000 7f27c9890700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f27ac0a2c20 0x7f27ac0a50e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f27c4103550 tx=0x7f27bc009040 comp rx=0 tx=0).stop 2024-06-09T22:13:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.479+0000 7f27c9890700 1 -- 172.21.15.153:0/435718599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c4104090 msgr2=0x7f27c4074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.479+0000 7f27c9890700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c4104090 0x7f27c4074dc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009990 tx=0x7f27b40047b0 comp rx=0 tx=0).stop 2024-06-09T22:13:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.479+0000 7f27c9890700 1 -- 172.21.15.153:0/435718599 shutdown_connections 2024-06-09T22:13:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.479+0000 7f27c9890700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f27ac0a2c20 0x7f27ac0a50e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:51.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.479+0000 7f27c9890700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c4104090 0x7f27c4074dc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:51.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.479+0000 7f27c9890700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27c41024f0 0x7f27c4076230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:51.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.479+0000 7f27c9890700 1 --2- 172.21.15.153:0/435718599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c41036f0 0x7f27c4074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:51.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.479+0000 7f27c9890700 1 -- 172.21.15.153:0/435718599 >> 172.21.15.153:0/435718599 conn(0x7f27c40fda90 msgr2=0x7f27c4106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:13:51.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.480+0000 7f27c9890700 1 -- 172.21.15.153:0/435718599 shutdown_connections 2024-06-09T22:13:51.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:13:51.480+0000 7f27c9890700 1 -- 172.21.15.153:0/435718599 wait complete. 2024-06-09T22:13:52.125 DEBUG:teuthology.orchestra.run.smithi153:osd.7> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.7.service 2024-06-09T22:13:52.128 INFO:tasks.cephadm:Deploying osd.8 on smithi194 with /dev/vg_nvme/lv_2... 2024-06-09T22:13:52.128 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-09T22:13:52.375 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:13:52.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:52 smithi153 ceph-mon[36599]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:52.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:52 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:52.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:52 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:52.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:52 smithi153 ceph-mon[36599]: from='osd.7 [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-09T22:13:52.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:52 smithi153 ceph-mon[36599]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-09T22:13:52.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:52 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:52.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:52 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:52.749 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:52 smithi194 ceph-mon[36856]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:52.749 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:52 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:52.749 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:52 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:52.750 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:52 smithi194 ceph-mon[36856]: from='osd.7 [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-09T22:13:52.750 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:52 smithi194 ceph-mon[36856]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-09T22:13:52.750 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:52 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:52.750 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:52 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:52.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:52 smithi104 ceph-mon[32389]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:52.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:52 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:52.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:52 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:52.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:52 smithi104 ceph-mon[32389]: from='osd.7 [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-09T22:13:52.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:52 smithi104 ceph-mon[32389]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-09T22:13:52.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:52 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:52.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:52 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:53.552 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:53 smithi194 ceph-mon[36856]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-09T22:13:53.552 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:53 smithi194 ceph-mon[36856]: osdmap e49: 8 total, 7 up, 8 in 2024-06-09T22:13:53.552 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:53 smithi194 ceph-mon[36856]: from='osd.7 [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:13:53.552 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:53 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:53.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:53 smithi194 ceph-mon[36856]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:13:53.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:53 smithi153 ceph-mon[36599]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-09T22:13:53.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:53 smithi153 ceph-mon[36599]: osdmap e49: 8 total, 7 up, 8 in 2024-06-09T22:13:53.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:53 smithi153 ceph-mon[36599]: from='osd.7 [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:13:53.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:53 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:53.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:53 smithi153 ceph-mon[36599]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:13:53.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:53 smithi104 ceph-mon[32389]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-09T22:13:53.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:53 smithi104 ceph-mon[32389]: osdmap e49: 8 total, 7 up, 8 in 2024-06-09T22:13:53.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:53 smithi104 ceph-mon[32389]: from='osd.7 [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:13:53.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:53 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:53.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:53 smithi104 ceph-mon[32389]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:13:54.135 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:13:53 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:13:53.916+0000 7fced580f700 -1 osd.7 0 waiting for initial osdmap 2024-06-09T22:13:54.135 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:13:53 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:13:53.928+0000 7fcecd5fa700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:13:54.445 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-09T22:13:54.445 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-09T22:13:54.446 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-06-09T22:13:54.446 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-06-09T22:13:54.446 INFO:teuthology.orchestra.run.smithi194.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114047 s, 919 MB/s 2024-06-09T22:13:54.446 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-06-09T22:13:54.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:54 smithi153 ceph-mon[36599]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:54.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:54 smithi153 ceph-mon[36599]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-09T22:13:54.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:54 smithi153 ceph-mon[36599]: osdmap e50: 8 total, 7 up, 8 in 2024-06-09T22:13:54.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:54.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:54.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:54 smithi153 ceph-mon[36599]: from='osd.7 ' entity='osd.7' 2024-06-09T22:13:54.742 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:54 smithi194 ceph-mon[36856]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:54.742 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:54 smithi194 ceph-mon[36856]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-09T22:13:54.742 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:54 smithi194 ceph-mon[36856]: osdmap e50: 8 total, 7 up, 8 in 2024-06-09T22:13:54.742 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:54.742 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:54.742 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:54 smithi194 ceph-mon[36856]: from='osd.7 ' entity='osd.7' 2024-06-09T22:13:54.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:54 smithi104 ceph-mon[32389]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-06-09T22:13:54.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:54 smithi104 ceph-mon[32389]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-09T22:13:54.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:54 smithi104 ceph-mon[32389]: osdmap e50: 8 total, 7 up, 8 in 2024-06-09T22:13:54.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:54.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:54.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:54 smithi104 ceph-mon[32389]: from='osd.7 ' entity='osd.7' 2024-06-09T22:13:55.090 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_2 2024-06-09T22:13:55.328 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:13:55.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:55 smithi194 ceph-mon[36856]: purged_snaps scrub starts 2024-06-09T22:13:55.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:55 smithi194 ceph-mon[36856]: purged_snaps scrub ok 2024-06-09T22:13:55.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:55 smithi194 ceph-mon[36856]: osd.7 [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024] boot 2024-06-09T22:13:55.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:55 smithi194 ceph-mon[36856]: osdmap e51: 8 total, 8 up, 8 in 2024-06-09T22:13:55.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:55 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:55.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:55 smithi194 ceph-mon[36856]: pgmap v176: 1 pgs: 1 remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:13:55.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:55 smithi153 ceph-mon[36599]: purged_snaps scrub starts 2024-06-09T22:13:55.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:55 smithi153 ceph-mon[36599]: purged_snaps scrub ok 2024-06-09T22:13:55.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:55 smithi153 ceph-mon[36599]: osd.7 [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024] boot 2024-06-09T22:13:55.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:55 smithi153 ceph-mon[36599]: osdmap e51: 8 total, 8 up, 8 in 2024-06-09T22:13:55.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:55 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:55.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:55 smithi153 ceph-mon[36599]: pgmap v176: 1 pgs: 1 remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:13:55.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:55 smithi104 ceph-mon[32389]: purged_snaps scrub starts 2024-06-09T22:13:55.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:55 smithi104 ceph-mon[32389]: purged_snaps scrub ok 2024-06-09T22:13:55.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:55 smithi104 ceph-mon[32389]: osd.7 [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024] boot 2024-06-09T22:13:55.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:55 smithi104 ceph-mon[32389]: osdmap e51: 8 total, 8 up, 8 in 2024-06-09T22:13:55.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:55 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:13:55.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:55 smithi104 ceph-mon[32389]: pgmap v176: 1 pgs: 1 remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:13:56.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: osdmap e52: 8 total, 8 up, 8 in 2024-06-09T22:13:56.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: Detected new or changed devices on smithi153 2024-06-09T22:13:56.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:56.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:56.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:56.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi153 to 5696M 2024-06-09T22:13:56.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:56.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:56.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:56.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:56.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:56.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:56 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: osdmap e52: 8 total, 8 up, 8 in 2024-06-09T22:13:56.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: Detected new or changed devices on smithi153 2024-06-09T22:13:56.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:56.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:56.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:56.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi153 to 5696M 2024-06-09T22:13:56.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:56.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:56.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:56.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:56.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:56.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:56 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: osdmap e52: 8 total, 8 up, 8 in 2024-06-09T22:13:56.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: Detected new or changed devices on smithi153 2024-06-09T22:13:56.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:56.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:56.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:13:56.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi153 to 5696M 2024-06-09T22:13:56.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:56.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:56.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:56.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:13:56.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:56.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:13:56.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:56 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:13:57.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.031+0000 7fcc2448b700 1 -- 172.21.15.194:0/3677135078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc1c103780 msgr2=0x7fcc1c103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:57.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.031+0000 7fcc2448b700 1 --2- 172.21.15.194:0/3677135078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc1c103780 0x7fcc1c103be0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009990 tx=0x7fcc0c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:13:57.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.031+0000 7fcc2448b700 1 -- 172.21.15.194:0/3677135078 shutdown_connections 2024-06-09T22:13:57.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.031+0000 7fcc2448b700 1 --2- 172.21.15.194:0/3677135078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c1007a0 0x7fcc1c100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:57.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.031+0000 7fcc2448b700 1 --2- 172.21.15.194:0/3677135078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc1c104120 0x7fcc1c10a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.031+0000 7fcc2448b700 1 --2- 172.21.15.194:0/3677135078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc1c103780 0x7fcc1c103be0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.031+0000 7fcc2448b700 1 -- 172.21.15.194:0/3677135078 >> 172.21.15.194:0/3677135078 conn(0x7fcc1c0fc120 msgr2=0x7fcc1c0fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.031+0000 7fcc2448b700 1 -- 172.21.15.194:0/3677135078 shutdown_connections 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.031+0000 7fcc2448b700 1 -- 172.21.15.194:0/3677135078 wait complete. 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc2448b700 1 Processor -- start 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc2448b700 1 -- start start 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc2448b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc1c1007a0 0x7fcc1c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc2448b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc1c103780 0x7fcc1c198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc2448b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c104120 0x7fcc1c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc2448b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c10e0c0 con 0x7fcc1c104120 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc2448b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c10df40 con 0x7fcc1c1007a0 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc2448b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c10e240 con 0x7fcc1c103780 2024-06-09T22:13:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc21a26700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc1c103780 0x7fcc1c198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc21a26700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc1c103780 0x7fcc1c198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44798/0 (socket says 172.21.15.194:44798) 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc21a26700 1 -- 172.21.15.194:0/2790837046 learned_addr learned my addr 172.21.15.194:0/2790837046 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc22227700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc1c1007a0 0x7fcc1c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc22a28700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c104120 0x7fcc1c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc21a26700 1 -- 172.21.15.194:0/2790837046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc1c1007a0 msgr2=0x7fcc1c1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc21a26700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc1c1007a0 0x7fcc1c1986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc21a26700 1 -- 172.21.15.194:0/2790837046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c104120 msgr2=0x7fcc1c19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc21a26700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c104120 0x7fcc1c19fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.032+0000 7fcc21a26700 1 -- 172.21.15.194:0/2790837046 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1c1a0300 con 0x7fcc1c103780 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.033+0000 7fcc21a26700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc1c103780 0x7fcc1c198be0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c004790 tx=0x7fcc0c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.033+0000 7fcc137fe700 1 -- 172.21.15.194:0/2790837046 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c027070 con 0x7fcc1c103780 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.033+0000 7fcc2448b700 1 -- 172.21.15.194:0/2790837046 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1c1a0590 con 0x7fcc1c103780 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.033+0000 7fcc137fe700 1 -- 172.21.15.194:0/2790837046 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc0c004150 con 0x7fcc1c103780 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.033+0000 7fcc137fe700 1 -- 172.21.15.194:0/2790837046 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00f460 con 0x7fcc1c103780 2024-06-09T22:13:57.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.033+0000 7fcc2448b700 1 -- 172.21.15.194:0/2790837046 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc1c1a0a40 con 0x7fcc1c103780 2024-06-09T22:13:57.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.034+0000 7fcc2448b700 1 -- 172.21.15.194:0/2790837046 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc1c069180 con 0x7fcc1c103780 2024-06-09T22:13:57.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.035+0000 7fcc137fe700 1 -- 172.21.15.194:0/2790837046 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fcc0c02a070 con 0x7fcc1c103780 2024-06-09T22:13:57.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.035+0000 7fcc137fe700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fcc080a2c20 0x7fcc080a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:13:57.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.036+0000 7fcc137fe700 1 -- 172.21.15.194:0/2790837046 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7fcc0c0cfb70 con 0x7fcc1c103780 2024-06-09T22:13:57.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.036+0000 7fcc22227700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fcc080a2c20 0x7fcc080a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:13:57.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.036+0000 7fcc22227700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fcc080a2c20 0x7fcc080a50e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c069f70 tx=0x7fcc1800c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:13:57.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.039+0000 7fcc137fe700 1 -- 172.21.15.194:0/2790837046 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fcc0c031070 con 0x7fcc1c103780 2024-06-09T22:13:57.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:13:57.239+0000 7fcc2448b700 1 -- 172.21.15.194:0/2790837046 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fcc1c0628d0 con 0x7fcc080a2c20 2024-06-09T22:13:57.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:57 smithi153 ceph-mon[36599]: osdmap e53: 8 total, 8 up, 8 in 2024-06-09T22:13:57.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:57 smithi153 ceph-mon[36599]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:13:57.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:13:57.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:13:57.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:57.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:57 smithi153 ceph-mon[36599]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:13:57.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:57 smithi153 ceph-mon[36599]: osdmap e54: 8 total, 8 up, 8 in 2024-06-09T22:13:57.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:57 smithi104 ceph-mon[32389]: osdmap e53: 8 total, 8 up, 8 in 2024-06-09T22:13:57.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:57 smithi104 ceph-mon[32389]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:13:57.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:13:57.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:13:57.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:57.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:57 smithi104 ceph-mon[32389]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:13:57.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:57 smithi104 ceph-mon[32389]: osdmap e54: 8 total, 8 up, 8 in 2024-06-09T22:13:57.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:57 smithi194 ceph-mon[36856]: osdmap e53: 8 total, 8 up, 8 in 2024-06-09T22:13:57.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:57 smithi194 ceph-mon[36856]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:13:57.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:13:57.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:13:57.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:13:57.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:57 smithi194 ceph-mon[36856]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:13:57.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:57 smithi194 ceph-mon[36856]: osdmap e54: 8 total, 8 up, 8 in 2024-06-09T22:13:59.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:13:59 smithi194 ceph-mon[36856]: osdmap e55: 8 total, 8 up, 8 in 2024-06-09T22:13:59.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:13:59 smithi153 ceph-mon[36599]: osdmap e55: 8 total, 8 up, 8 in 2024-06-09T22:13:59.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:13:59 smithi104 ceph-mon[32389]: osdmap e55: 8 total, 8 up, 8 in 2024-06-09T22:14:00.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:00 smithi153 ceph-mon[36599]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:00.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:00 smithi153 ceph-mon[36599]: from='client.? 172.21.15.194:0/339150246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8beddca3-bc69-4ef9-955b-f0296e0f28bc"}]: dispatch 2024-06-09T22:14:00.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:00 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8beddca3-bc69-4ef9-955b-f0296e0f28bc"}]: dispatch 2024-06-09T22:14:00.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:00 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8beddca3-bc69-4ef9-955b-f0296e0f28bc"}]': finished 2024-06-09T22:14:00.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:00 smithi153 ceph-mon[36599]: osdmap e56: 9 total, 8 up, 9 in 2024-06-09T22:14:00.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:00 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:00.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:00 smithi104 ceph-mon[32389]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:00.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:00 smithi104 ceph-mon[32389]: from='client.? 172.21.15.194:0/339150246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8beddca3-bc69-4ef9-955b-f0296e0f28bc"}]: dispatch 2024-06-09T22:14:00.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:00 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8beddca3-bc69-4ef9-955b-f0296e0f28bc"}]: dispatch 2024-06-09T22:14:00.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:00 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8beddca3-bc69-4ef9-955b-f0296e0f28bc"}]': finished 2024-06-09T22:14:00.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:00 smithi104 ceph-mon[32389]: osdmap e56: 9 total, 8 up, 9 in 2024-06-09T22:14:00.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:00 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:00.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:00 smithi194 ceph-mon[36856]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:00.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:00 smithi194 ceph-mon[36856]: from='client.? 172.21.15.194:0/339150246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8beddca3-bc69-4ef9-955b-f0296e0f28bc"}]: dispatch 2024-06-09T22:14:00.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:00 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8beddca3-bc69-4ef9-955b-f0296e0f28bc"}]: dispatch 2024-06-09T22:14:00.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:00 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8beddca3-bc69-4ef9-955b-f0296e0f28bc"}]': finished 2024-06-09T22:14:00.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:00 smithi194 ceph-mon[36856]: osdmap e56: 9 total, 8 up, 9 in 2024-06-09T22:14:00.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:00 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:01.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:01 smithi153 ceph-mon[36599]: from='client.? 172.21.15.194:0/1661730210' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:14:01.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:01 smithi104 ceph-mon[32389]: from='client.? 172.21.15.194:0/1661730210' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:14:01.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:01 smithi194 ceph-mon[36856]: from='client.? 172.21.15.194:0/1661730210' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:14:02.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:02 smithi153 ceph-mon[36599]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:02.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:02 smithi104 ceph-mon[32389]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:02.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:02 smithi194 ceph-mon[36856]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:04.552 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:04 smithi194 ceph-mon[36856]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:04.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:04 smithi153 ceph-mon[36599]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:04.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:04 smithi104 ceph-mon[32389]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:06.552 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:06 smithi194 ceph-mon[36856]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:06.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:06 smithi153 ceph-mon[36599]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:06.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:06 smithi104 ceph-mon[32389]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:07.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:07 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-09T22:14:07.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:07 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:07.749 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:07 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-09T22:14:07.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:07 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:07.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:07 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-09T22:14:07.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:07 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:08.552 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:08 smithi194 ceph-mon[36856]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:08.552 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:08 smithi194 ceph-mon[36856]: Deploying daemon osd.8 on smithi194 2024-06-09T22:14:08.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:08 smithi153 ceph-mon[36599]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:08.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:08 smithi153 ceph-mon[36599]: Deploying daemon osd.8 on smithi194 2024-06-09T22:14:08.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:08 smithi104 ceph-mon[32389]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:08.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:08 smithi104 ceph-mon[32389]: Deploying daemon osd.8 on smithi194 2024-06-09T22:14:10.552 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:10 smithi194 ceph-mon[36856]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:10.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:10 smithi153 ceph-mon[36599]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:10.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:10 smithi104 ceph-mon[32389]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:12.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:12 smithi194 ceph-mon[36856]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:12.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:12 smithi153 ceph-mon[36599]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:12.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:12 smithi104 ceph-mon[32389]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:14.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:14.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:14.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:14:14.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:14 smithi104 ceph-mon[32389]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:14.302 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:14.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:14.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:14:14.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:14 smithi194 ceph-mon[36856]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:14.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:14 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:14.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:14 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:14.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:14 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:14:14.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:14 smithi153 ceph-mon[36599]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:16.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:16 smithi194 ceph-mon[36856]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:16.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.565+0000 7fcc137fe700 1 -- 172.21.15.194:0/2790837046 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcc1c0628d0 con 0x7fcc080a2c20 2024-06-09T22:14:16.559 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 8 on host 'smithi194' 2024-06-09T22:14:16.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.568+0000 7fcc2448b700 1 -- 172.21.15.194:0/2790837046 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fcc080a2c20 msgr2=0x7fcc080a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:14:16.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.568+0000 7fcc2448b700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fcc080a2c20 0x7fcc080a50e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c069f70 tx=0x7fcc1800c040 comp rx=0 tx=0).stop 2024-06-09T22:14:16.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.568+0000 7fcc2448b700 1 -- 172.21.15.194:0/2790837046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc1c103780 msgr2=0x7fcc1c198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:14:16.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.568+0000 7fcc2448b700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc1c103780 0x7fcc1c198be0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c004790 tx=0x7fcc0c009580 comp rx=0 tx=0).stop 2024-06-09T22:14:16.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.568+0000 7fcc2448b700 1 -- 172.21.15.194:0/2790837046 shutdown_connections 2024-06-09T22:14:16.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.568+0000 7fcc2448b700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fcc080a2c20 0x7fcc080a50e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:16.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.568+0000 7fcc2448b700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c104120 0x7fcc1c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:16.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.568+0000 7fcc2448b700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc1c1007a0 0x7fcc1c1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:16.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.568+0000 7fcc2448b700 1 --2- 172.21.15.194:0/2790837046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc1c103780 0x7fcc1c198be0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:16.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.568+0000 7fcc2448b700 1 -- 172.21.15.194:0/2790837046 >> 172.21.15.194:0/2790837046 conn(0x7fcc1c0fc120 msgr2=0x7fcc1c106a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:14:16.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.568+0000 7fcc2448b700 1 -- 172.21.15.194:0/2790837046 shutdown_connections 2024-06-09T22:14:16.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:14:16.568+0000 7fcc2448b700 1 -- 172.21.15.194:0/2790837046 wait complete. 2024-06-09T22:14:16.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:16 smithi153 ceph-mon[36599]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:16.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:16 smithi104 ceph-mon[32389]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:17.237 DEBUG:teuthology.orchestra.run.smithi194:osd.8> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.8.service 2024-06-09T22:14:17.239 INFO:tasks.cephadm:Deploying osd.9 on smithi104 with /dev/vg_nvme/lv_1... 2024-06-09T22:14:17.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-09T22:14:17.487 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:14:17.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:17.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:17.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:17.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:17.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:17 smithi194 ceph-mon[36856]: from='osd.8 [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-09T22:14:17.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:17 smithi194 ceph-mon[36856]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-09T22:14:17.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:17 smithi194 ceph-mon[36856]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:17.888 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:17.888 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:17.888 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:17.889 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:17.889 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:17 smithi104 ceph-mon[32389]: from='osd.8 [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-09T22:14:17.889 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:17 smithi104 ceph-mon[32389]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-09T22:14:17.889 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:17 smithi104 ceph-mon[32389]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:17.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:17.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:17.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:17.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:17.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:17 smithi153 ceph-mon[36599]: from='osd.8 [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-09T22:14:17.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:17 smithi153 ceph-mon[36599]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-09T22:14:17.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:17 smithi153 ceph-mon[36599]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:18.747 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:18 smithi194 ceph-mon[36856]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-09T22:14:18.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:18 smithi194 ceph-mon[36856]: from='osd.8 [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:14:18.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:18 smithi194 ceph-mon[36856]: osdmap e57: 9 total, 8 up, 9 in 2024-06-09T22:14:18.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:18 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:18.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:18 smithi194 ceph-mon[36856]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:14:18.889 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:18 smithi104 ceph-mon[32389]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-09T22:14:18.889 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:18 smithi104 ceph-mon[32389]: from='osd.8 [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:14:18.889 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:18 smithi104 ceph-mon[32389]: osdmap e57: 9 total, 8 up, 9 in 2024-06-09T22:14:18.889 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:18 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:18.889 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:18 smithi104 ceph-mon[32389]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:14:18.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:18 smithi153 ceph-mon[36599]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-09T22:14:18.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:18 smithi153 ceph-mon[36599]: from='osd.8 [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:14:18.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:18 smithi153 ceph-mon[36599]: osdmap e57: 9 total, 8 up, 9 in 2024-06-09T22:14:18.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:18 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:18.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:18 smithi153 ceph-mon[36599]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:14:19.553 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:14:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:14:19.071+0000 7fa9de8ea700 -1 osd.8 0 waiting for initial osdmap 2024-06-09T22:14:19.553 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:14:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:14:19.081+0000 7fa9d66d5700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:14:19.626 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-09T22:14:19.626 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-09T22:14:19.626 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-06-09T22:14:19.626 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-06-09T22:14:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.01142 s, 918 MB/s 2024-06-09T22:14:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-06-09T22:14:19.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:19 smithi104 ceph-mon[32389]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-06-09T22:14:19.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:19 smithi104 ceph-mon[32389]: osdmap e58: 9 total, 8 up, 9 in 2024-06-09T22:14:19.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:19 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:19.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:19 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:19.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:19 smithi104 ceph-mon[32389]: from='osd.8 ' entity='osd.8' 2024-06-09T22:14:19.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:19 smithi104 ceph-mon[32389]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:19.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:19 smithi194 ceph-mon[36856]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-06-09T22:14:19.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:19 smithi194 ceph-mon[36856]: osdmap e58: 9 total, 8 up, 9 in 2024-06-09T22:14:19.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:19 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:19.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:19 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:19.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:19 smithi194 ceph-mon[36856]: from='osd.8 ' entity='osd.8' 2024-06-09T22:14:19.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:19 smithi194 ceph-mon[36856]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:19.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:19 smithi153 ceph-mon[36599]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-06-09T22:14:19.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:19 smithi153 ceph-mon[36599]: osdmap e58: 9 total, 8 up, 9 in 2024-06-09T22:14:19.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:19 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:19.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:19 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:19.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:19 smithi153 ceph-mon[36599]: from='osd.8 ' entity='osd.8' 2024-06-09T22:14:19.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:19 smithi153 ceph-mon[36599]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-06-09T22:14:20.473 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_1 2024-06-09T22:14:20.715 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:14:20.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: purged_snaps scrub starts 2024-06-09T22:14:20.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: purged_snaps scrub ok 2024-06-09T22:14:20.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:20.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: osd.8 [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300] boot 2024-06-09T22:14:20.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: osdmap e59: 9 total, 9 up, 9 in 2024-06-09T22:14:20.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:20.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: Detected new or changed devices on smithi194 2024-06-09T22:14:20.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:20.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:20.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:20.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:20.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:20.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi194 to 5696M 2024-06-09T22:14:20.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:20.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:20.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:14:20.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:20.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:14:20.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:20.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:14:20.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:20.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: purged_snaps scrub starts 2024-06-09T22:14:20.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: purged_snaps scrub ok 2024-06-09T22:14:20.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:20.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: osd.8 [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300] boot 2024-06-09T22:14:20.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: osdmap e59: 9 total, 9 up, 9 in 2024-06-09T22:14:20.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:20.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: Detected new or changed devices on smithi194 2024-06-09T22:14:20.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:20.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:20.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:20.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:20.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:20.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi194 to 5696M 2024-06-09T22:14:20.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:20.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:20.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:14:20.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:20.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:14:20.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:20.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:14:20.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:21.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: purged_snaps scrub starts 2024-06-09T22:14:21.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: purged_snaps scrub ok 2024-06-09T22:14:21.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:21.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: osd.8 [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300] boot 2024-06-09T22:14:21.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: osdmap e59: 9 total, 9 up, 9 in 2024-06-09T22:14:21.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:14:21.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: Detected new or changed devices on smithi194 2024-06-09T22:14:21.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:21.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:21.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:21.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi194 to 5696M 2024-06-09T22:14:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:14:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:14:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:14:21.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:21.897 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:21 smithi104 ceph-mon[32389]: osdmap e60: 9 total, 9 up, 9 in 2024-06-09T22:14:21.897 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:21 smithi104 ceph-mon[32389]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:21.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:21 smithi153 ceph-mon[36599]: osdmap e60: 9 total, 9 up, 9 in 2024-06-09T22:14:21.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:21 smithi153 ceph-mon[36599]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:22.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:21 smithi194 ceph-mon[36856]: osdmap e60: 9 total, 9 up, 9 in 2024-06-09T22:14:22.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:21 smithi194 ceph-mon[36856]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:22.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.439+0000 7f60c1c6e700 1 -- 172.21.15.104:0/4046882122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc104130 msgr2=0x7f60bc10a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:14:22.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.439+0000 7f60c1c6e700 1 --2- 172.21.15.104:0/4046882122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc104130 0x7f60bc10a9e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009990 tx=0x7f60b4009ca0 comp rx=0 tx=0).stop 2024-06-09T22:14:22.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.439+0000 7f60c1c6e700 1 -- 172.21.15.104:0/4046882122 shutdown_connections 2024-06-09T22:14:22.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.439+0000 7f60c1c6e700 1 --2- 172.21.15.104:0/4046882122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc104130 0x7f60bc10a9e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:22.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.439+0000 7f60c1c6e700 1 --2- 172.21.15.104:0/4046882122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60bc1007b0 0x7f60bc100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:22.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.439+0000 7f60c1c6e700 1 --2- 172.21.15.104:0/4046882122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60bc103790 0x7f60bc103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:22.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.439+0000 7f60c1c6e700 1 -- 172.21.15.104:0/4046882122 >> 172.21.15.104:0/4046882122 conn(0x7f60bc0fc0f0 msgr2=0x7f60bc0fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:14:22.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.439+0000 7f60c1c6e700 1 -- 172.21.15.104:0/4046882122 shutdown_connections 2024-06-09T22:14:22.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.439+0000 7f60c1c6e700 1 -- 172.21.15.104:0/4046882122 wait complete. 2024-06-09T22:14:22.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60c1c6e700 1 Processor -- start 2024-06-09T22:14:22.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60c1c6e700 1 -- start start 2024-06-09T22:14:22.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60c1c6e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60bc1007b0 0x7f60bc1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:14:22.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60c1c6e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60bc103790 0x7f60bc198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:14:22.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60c1c6e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc104130 0x7f60bc19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:14:22.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60c1c6e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc10e0b0 con 0x7f60bc104130 2024-06-09T22:14:22.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60c1c6e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc10df30 con 0x7f60bc1007b0 2024-06-09T22:14:22.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60c1c6e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc10e230 con 0x7f60bc103790 2024-06-09T22:14:22.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60bbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc104130 0x7f60bc19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:14:22.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60bbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc104130 0x7f60bc19fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34398/0 (socket says 172.21.15.104:34398) 2024-06-09T22:14:22.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60bbfff700 1 -- 172.21.15.104:0/2746907772 learned_addr learned my addr 172.21.15.104:0/2746907772 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:14:22.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60baffd700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60bc103790 0x7f60bc198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:14:22.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60bb7fe700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60bc1007b0 0x7f60bc1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:14:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60bbfff700 1 -- 172.21.15.104:0/2746907772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60bc103790 msgr2=0x7f60bc198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:14:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60bbfff700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60bc103790 0x7f60bc198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60bbfff700 1 -- 172.21.15.104:0/2746907772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60bc1007b0 msgr2=0x7f60bc1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:14:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60bbfff700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60bc1007b0 0x7f60bc1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.440+0000 7f60bbfff700 1 -- 172.21.15.104:0/2746907772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60bc1a0230 con 0x7f60bc104130 2024-06-09T22:14:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.441+0000 7f60bbfff700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc104130 0x7f60bc19fb90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f60b40059d0 tx=0x7f60b4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:14:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.441+0000 7f60b8ff9700 1 -- 172.21.15.104:0/2746907772 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b4003680 con 0x7f60bc104130 2024-06-09T22:14:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.441+0000 7f60b8ff9700 1 -- 172.21.15.104:0/2746907772 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60b4004150 con 0x7f60bc104130 2024-06-09T22:14:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.441+0000 7f60b8ff9700 1 -- 172.21.15.104:0/2746907772 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b4005440 con 0x7f60bc104130 2024-06-09T22:14:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.441+0000 7f60c1c6e700 1 -- 172.21.15.104:0/2746907772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60bc1a04c0 con 0x7f60bc104130 2024-06-09T22:14:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.441+0000 7f60c1c6e700 1 -- 172.21.15.104:0/2746907772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60bc1a08b0 con 0x7f60bc104130 2024-06-09T22:14:22.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.442+0000 7f60c1c6e700 1 -- 172.21.15.104:0/2746907772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60bc069180 con 0x7f60bc104130 2024-06-09T22:14:22.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.443+0000 7f60b8ff9700 1 -- 172.21.15.104:0/2746907772 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f60b4029060 con 0x7f60bc104130 2024-06-09T22:14:22.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.444+0000 7f60b8ff9700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f60a40a2c20 0x7f60a40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:14:22.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.444+0000 7f60bb7fe700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f60a40a2c20 0x7f60a40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:14:22.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.444+0000 7f60b8ff9700 1 -- 172.21.15.104:0/2746907772 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7f60b40cf3e0 con 0x7f60bc104130 2024-06-09T22:14:22.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.444+0000 7f60bb7fe700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f60a40a2c20 0x7f60a40a50e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f60bc06aa60 tx=0x7f60ac009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:14:22.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.447+0000 7f60b8ff9700 1 -- 172.21.15.104:0/2746907772 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f60b405d850 con 0x7f60bc104130 2024-06-09T22:14:22.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:22.652+0000 7f60c1c6e700 1 -- 172.21.15.104:0/2746907772 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f60bc0628d0 con 0x7f60a40a2c20 2024-06-09T22:14:22.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:22 smithi153 ceph-mon[36599]: osdmap e61: 9 total, 9 up, 9 in 2024-06-09T22:14:22.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:22 smithi153 ceph-mon[36599]: osdmap e62: 9 total, 9 up, 9 in 2024-06-09T22:14:22.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:22 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:14:22.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:22 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:14:22.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:22 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:23.000 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:22 smithi104 ceph-mon[32389]: osdmap e61: 9 total, 9 up, 9 in 2024-06-09T22:14:23.000 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:22 smithi104 ceph-mon[32389]: osdmap e62: 9 total, 9 up, 9 in 2024-06-09T22:14:23.000 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:22 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:14:23.000 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:22 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:14:23.000 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:22 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:23.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:22 smithi194 ceph-mon[36856]: osdmap e61: 9 total, 9 up, 9 in 2024-06-09T22:14:23.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:22 smithi194 ceph-mon[36856]: osdmap e62: 9 total, 9 up, 9 in 2024-06-09T22:14:23.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:22 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:14:23.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:22 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:14:23.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:22 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:24.515 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:24 smithi104 ceph-mon[32389]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:14:24.516 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:24 smithi104 ceph-mon[32389]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:24.516 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:24 smithi104 ceph-mon[32389]: osdmap e63: 9 total, 9 up, 9 in 2024-06-09T22:14:24.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:24 smithi153 ceph-mon[36599]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:14:24.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:24 smithi153 ceph-mon[36599]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:24.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:24 smithi153 ceph-mon[36599]: osdmap e63: 9 total, 9 up, 9 in 2024-06-09T22:14:24.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:24 smithi194 ceph-mon[36856]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:14:24.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:24 smithi194 ceph-mon[36856]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:24.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:24 smithi194 ceph-mon[36856]: osdmap e63: 9 total, 9 up, 9 in 2024-06-09T22:14:26.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:26 smithi153 ceph-mon[36599]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-06-09T22:14:26.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:26 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2321169448' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d82b0c48-b957-4209-afc9-020ded569c2c"}]: dispatch 2024-06-09T22:14:26.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:26 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2321169448' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d82b0c48-b957-4209-afc9-020ded569c2c"}]': finished 2024-06-09T22:14:26.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:26 smithi153 ceph-mon[36599]: osdmap e64: 10 total, 9 up, 10 in 2024-06-09T22:14:26.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:26.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:26 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2028583032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:14:26.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:26 smithi104 ceph-mon[32389]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-06-09T22:14:26.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:26 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2321169448' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d82b0c48-b957-4209-afc9-020ded569c2c"}]: dispatch 2024-06-09T22:14:26.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:26 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2321169448' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d82b0c48-b957-4209-afc9-020ded569c2c"}]': finished 2024-06-09T22:14:26.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:26 smithi104 ceph-mon[32389]: osdmap e64: 10 total, 9 up, 10 in 2024-06-09T22:14:26.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:26.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:26 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2028583032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:14:26.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:26 smithi194 ceph-mon[36856]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-06-09T22:14:26.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:26 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2321169448' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d82b0c48-b957-4209-afc9-020ded569c2c"}]: dispatch 2024-06-09T22:14:26.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:26 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2321169448' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d82b0c48-b957-4209-afc9-020ded569c2c"}]': finished 2024-06-09T22:14:26.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:26 smithi194 ceph-mon[36856]: osdmap e64: 10 total, 9 up, 10 in 2024-06-09T22:14:26.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:26.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:26 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2028583032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:14:28.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:28 smithi153 ceph-mon[36599]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-06-09T22:14:28.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:28 smithi104 ceph-mon[32389]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-06-09T22:14:28.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:28 smithi194 ceph-mon[36856]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-06-09T22:14:30.640 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:30 smithi104 ceph-mon[32389]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-06-09T22:14:30.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:30 smithi153 ceph-mon[36599]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-06-09T22:14:30.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:30 smithi194 ceph-mon[36856]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-06-09T22:14:32.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:32 smithi153 ceph-mon[36599]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-09T22:14:32.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:32 smithi104 ceph-mon[32389]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-09T22:14:32.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:32 smithi194 ceph-mon[36856]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-09T22:14:33.624 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:33 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-09T22:14:33.624 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:33 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:33.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:33 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-09T22:14:33.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:33 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:33.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:33 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-09T22:14:33.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:33 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:34.638 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:34 smithi153 ceph-mon[36599]: Deploying daemon osd.9 on smithi104 2024-06-09T22:14:34.638 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:34 smithi153 ceph-mon[36599]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-09T22:14:34.704 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:34 smithi194 ceph-mon[36856]: Deploying daemon osd.9 on smithi104 2024-06-09T22:14:34.704 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:34 smithi194 ceph-mon[36856]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-09T22:14:34.719 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:34 smithi104 ceph-mon[32389]: Deploying daemon osd.9 on smithi104 2024-06-09T22:14:34.719 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:34 smithi104 ceph-mon[32389]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-09T22:14:36.687 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:36 smithi104 ceph-mon[32389]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:36.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:36 smithi153 ceph-mon[36599]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:36.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:36 smithi194 ceph-mon[36856]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:38.710 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:38 smithi104 ceph-mon[32389]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:38.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:38 smithi153 ceph-mon[36599]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:38.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:38 smithi194 ceph-mon[36856]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:40.236 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:40.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:40.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:14:40.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:39 smithi153 ceph-mon[36599]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:40.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:40.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:40.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:14:40.250 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:39 smithi104 ceph-mon[32389]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:40.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:40.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:40.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:14:40.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:39 smithi194 ceph-mon[36856]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:42.602 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:42 smithi104 ceph-mon[32389]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:42.602 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:42 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:42.602 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:42 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:42.603 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:42 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:42.603 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:42 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:14:42.603 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:42 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:42.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:42 smithi153 ceph-mon[36599]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:42.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:42 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:42.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:42 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:42.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:42 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:42.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:42 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:14:42.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:42 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:42.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:42 smithi194 ceph-mon[36856]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:42.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:42 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:42.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:42 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:42.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:42 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:42.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:42 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:14:42.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:42 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:42.975 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 9 on host 'smithi104' 2024-06-09T22:14:42.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.981+0000 7f60b8ff9700 1 -- 172.21.15.104:0/2746907772 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f60bc0628d0 con 0x7f60a40a2c20 2024-06-09T22:14:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.984+0000 7f60c1c6e700 1 -- 172.21.15.104:0/2746907772 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f60a40a2c20 msgr2=0x7f60a40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:14:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.984+0000 7f60c1c6e700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f60a40a2c20 0x7f60a40a50e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f60bc06aa60 tx=0x7f60ac009040 comp rx=0 tx=0).stop 2024-06-09T22:14:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.984+0000 7f60c1c6e700 1 -- 172.21.15.104:0/2746907772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc104130 msgr2=0x7f60bc19fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:14:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.984+0000 7f60c1c6e700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc104130 0x7f60bc19fb90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f60b40059d0 tx=0x7f60b4009580 comp rx=0 tx=0).stop 2024-06-09T22:14:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.984+0000 7f60c1c6e700 1 -- 172.21.15.104:0/2746907772 shutdown_connections 2024-06-09T22:14:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.984+0000 7f60c1c6e700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f60a40a2c20 0x7f60a40a50e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.984+0000 7f60c1c6e700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc104130 0x7f60bc19fb90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.984+0000 7f60c1c6e700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60bc1007b0 0x7f60bc1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.984+0000 7f60c1c6e700 1 --2- 172.21.15.104:0/2746907772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60bc103790 0x7f60bc198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.984+0000 7f60c1c6e700 1 -- 172.21.15.104:0/2746907772 >> 172.21.15.104:0/2746907772 conn(0x7f60bc0fc0f0 msgr2=0x7f60bc106a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:14:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.984+0000 7f60c1c6e700 1 -- 172.21.15.104:0/2746907772 shutdown_connections 2024-06-09T22:14:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:14:42.984+0000 7f60c1c6e700 1 -- 172.21.15.104:0/2746907772 wait complete. 2024-06-09T22:14:43.631 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:43 smithi104 ceph-mon[32389]: from='osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-09T22:14:43.632 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:43 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:43.632 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:43 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:43.632 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:43 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:14:43.660 DEBUG:teuthology.orchestra.run.smithi104:osd.9> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.9.service 2024-06-09T22:14:43.662 INFO:tasks.cephadm:Deploying osd.10 on smithi153 with /dev/vg_nvme/lv_1... 2024-06-09T22:14:43.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-09T22:14:43.689 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:43 smithi153 ceph-mon[36599]: from='osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-09T22:14:43.689 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:43 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:43.689 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:43 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:43.689 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:43 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:14:43.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:43 smithi194 ceph-mon[36856]: from='osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-09T22:14:43.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:43 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:43.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:43 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:43.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:43 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:14:43.909 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.b/config 2024-06-09T22:14:44.514 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:44 smithi104 ceph-mon[32389]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:44.514 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:44 smithi104 ceph-mon[32389]: from='osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-09T22:14:44.514 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:44 smithi104 ceph-mon[32389]: osdmap e65: 10 total, 9 up, 10 in 2024-06-09T22:14:44.514 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:44 smithi104 ceph-mon[32389]: from='osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-06-09T22:14:44.514 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:44 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:44.514 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:14:44 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:14:44.347+0000 7f1977d20700 -1 osd.9 0 waiting for initial osdmap 2024-06-09T22:14:44.514 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:14:44 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:14:44.363+0000 7f1973b13700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:14:44.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:44 smithi153 ceph-mon[36599]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:44.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:44 smithi153 ceph-mon[36599]: from='osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-09T22:14:44.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:44 smithi153 ceph-mon[36599]: osdmap e65: 10 total, 9 up, 10 in 2024-06-09T22:14:44.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:44 smithi153 ceph-mon[36599]: from='osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-06-09T22:14:44.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:44 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:44.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:44 smithi194 ceph-mon[36856]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:44.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:44 smithi194 ceph-mon[36856]: from='osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-09T22:14:44.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:44 smithi194 ceph-mon[36856]: osdmap e65: 10 total, 9 up, 10 in 2024-06-09T22:14:44.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:44 smithi194 ceph-mon[36856]: from='osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-06-09T22:14:44.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:44 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:45.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:45 smithi153 ceph-mon[36599]: from='osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-06-09T22:14:45.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:45 smithi153 ceph-mon[36599]: osdmap e66: 10 total, 9 up, 10 in 2024-06-09T22:14:45.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:45 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:45.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:45 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:45.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:45 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:45.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:45 smithi104 ceph-mon[32389]: from='osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-06-09T22:14:45.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:45 smithi104 ceph-mon[32389]: osdmap e66: 10 total, 9 up, 10 in 2024-06-09T22:14:45.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:45 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:45.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:45 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:45.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:45 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:45.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:45 smithi194 ceph-mon[36856]: from='osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-06-09T22:14:45.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:45 smithi194 ceph-mon[36856]: osdmap e66: 10 total, 9 up, 10 in 2024-06-09T22:14:45.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:45 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:45.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:45 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:45.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:45 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-09T22:14:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-09T22:14:46.087 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-06-09T22:14:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-06-09T22:14:46.087 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134795 s, 778 MB/s 2024-06-09T22:14:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-06-09T22:14:46.690 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:46 smithi104 ceph-mon[32389]: purged_snaps scrub starts 2024-06-09T22:14:46.691 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:46 smithi104 ceph-mon[32389]: purged_snaps scrub ok 2024-06-09T22:14:46.691 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:46 smithi104 ceph-mon[32389]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:46.691 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:46 smithi104 ceph-mon[32389]: osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298] boot 2024-06-09T22:14:46.691 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:46 smithi104 ceph-mon[32389]: osdmap e67: 10 total, 10 up, 10 in 2024-06-09T22:14:46.691 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:46 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:46.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:46 smithi153 ceph-mon[36599]: purged_snaps scrub starts 2024-06-09T22:14:46.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:46 smithi153 ceph-mon[36599]: purged_snaps scrub ok 2024-06-09T22:14:46.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:46 smithi153 ceph-mon[36599]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:46.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:46 smithi153 ceph-mon[36599]: osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298] boot 2024-06-09T22:14:46.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:46 smithi153 ceph-mon[36599]: osdmap e67: 10 total, 10 up, 10 in 2024-06-09T22:14:46.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:46 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:46.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:46 smithi194 ceph-mon[36856]: purged_snaps scrub starts 2024-06-09T22:14:46.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:46 smithi194 ceph-mon[36856]: purged_snaps scrub ok 2024-06-09T22:14:46.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:46 smithi194 ceph-mon[36856]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-06-09T22:14:46.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:46 smithi194 ceph-mon[36856]: osd.9 [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298] boot 2024-06-09T22:14:46.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:46 smithi194 ceph-mon[36856]: osdmap e67: 10 total, 10 up, 10 in 2024-06-09T22:14:46.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:46 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:14:46.831 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_1 2024-06-09T22:14:47.110 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.b/config 2024-06-09T22:14:47.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:47 smithi153 ceph-mon[36599]: osdmap e68: 10 total, 10 up, 10 in 2024-06-09T22:14:47.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:47 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:47.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:47 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:47.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:47 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:47.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:47 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:47.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:47 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:47.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:47 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:47.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:47 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:47.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:47 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:47.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:47 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:14:47.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:47 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:47.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:47 smithi104 ceph-mon[32389]: osdmap e68: 10 total, 10 up, 10 in 2024-06-09T22:14:47.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:47.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:47.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:47.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:47.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:47.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:47.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:47.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:47.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:14:47.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:47 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:47.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:47 smithi194 ceph-mon[36856]: osdmap e68: 10 total, 10 up, 10 in 2024-06-09T22:14:47.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:47 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:47.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:47 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:47.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:47 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:47.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:47 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:47.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:47 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:47.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:47 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:14:47.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:47 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:47.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:47 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:47.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:47 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:14:47.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:47 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:14:48.486 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:48 smithi153 ceph-mon[36599]: Detected new or changed devices on smithi104 2024-06-09T22:14:48.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:48 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi104 to 4272M 2024-06-09T22:14:48.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:48 smithi153 ceph-mon[36599]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:48.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:48 smithi104 ceph-mon[32389]: Detected new or changed devices on smithi104 2024-06-09T22:14:48.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:48 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi104 to 4272M 2024-06-09T22:14:48.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:48 smithi104 ceph-mon[32389]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.773+0000 7fc9c270f700 1 -- 172.21.15.153:0/2544119153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9bc1082a0 msgr2=0x7fc9bc10a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:14:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.773+0000 7fc9c270f700 1 --2- 172.21.15.153:0/2544119153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9bc1082a0 0x7fc9bc10a690 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400b0a0 tx=0x7fc9b400b3b0 comp rx=0 tx=0).stop 2024-06-09T22:14:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.773+0000 7fc9c270f700 1 -- 172.21.15.153:0/2544119153 shutdown_connections 2024-06-09T22:14:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.773+0000 7fc9c270f700 1 --2- 172.21.15.153:0/2544119153 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9bc103040 0x7fc9bc105430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.773+0000 7fc9c270f700 1 --2- 172.21.15.153:0/2544119153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9bc1082a0 0x7fc9bc10a690 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.773+0000 7fc9c270f700 1 --2- 172.21.15.153:0/2544119153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9bc105970 0x7fc9bc107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.773+0000 7fc9c270f700 1 -- 172.21.15.153:0/2544119153 >> 172.21.15.153:0/2544119153 conn(0x7fc9bc0fc9f0 msgr2=0x7fc9bc0fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:14:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.774+0000 7fc9c270f700 1 -- 172.21.15.153:0/2544119153 shutdown_connections 2024-06-09T22:14:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.774+0000 7fc9c270f700 1 -- 172.21.15.153:0/2544119153 wait complete. 2024-06-09T22:14:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.774+0000 7fc9c270f700 1 Processor -- start 2024-06-09T22:14:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.774+0000 7fc9c270f700 1 -- start start 2024-06-09T22:14:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c270f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9bc103040 0x7fc9bc1964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:14:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c270f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9bc105970 0x7fc9bc196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:14:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c270f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9bc1082a0 0x7fc9bc1a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:14:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c270f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc10e100 con 0x7fc9bc103040 2024-06-09T22:14:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c270f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc10df80 con 0x7fc9bc1082a0 2024-06-09T22:14:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c270f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc10e280 con 0x7fc9bc105970 2024-06-09T22:14:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c0cac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9bc1082a0 0x7fc9bc1a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:14:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c0cac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9bc1082a0 0x7fc9bc1a6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59832/0 (socket says 172.21.15.153:59832) 2024-06-09T22:14:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c0cac700 1 -- 172.21.15.153:0/2280256570 learned_addr learned my addr 172.21.15.153:0/2280256570 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-09T22:14:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9bbfff700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9bc103040 0x7fc9bc1964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:14:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9bb7fe700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9bc105970 0x7fc9bc196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:14:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c0cac700 1 -- 172.21.15.153:0/2280256570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9bc105970 msgr2=0x7fc9bc196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:14:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c0cac700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9bc105970 0x7fc9bc196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c0cac700 1 -- 172.21.15.153:0/2280256570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9bc103040 msgr2=0x7fc9bc1964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c0cac700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9bc103040 0x7fc9bc1964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c0cac700 1 -- 172.21.15.153:0/2280256570 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9bc1a6980 con 0x7fc9bc1082a0 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9c0cac700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9bc1082a0 0x7fc9bc1a6410 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009bc0 tx=0x7fc9b4004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9b97fa700 1 -- 172.21.15.153:0/2280256570 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4021070 con 0x7fc9bc1082a0 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.775+0000 7fc9b97fa700 1 -- 172.21.15.153:0/2280256570 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9b4007b80 con 0x7fc9bc1082a0 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.776+0000 7fc9c270f700 1 -- 172.21.15.153:0/2280256570 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9bc1a6c10 con 0x7fc9bc1082a0 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.776+0000 7fc9b97fa700 1 -- 172.21.15.153:0/2280256570 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b401d450 con 0x7fc9bc1082a0 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.776+0000 7fc9c270f700 1 -- 172.21.15.153:0/2280256570 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9bc1a70c0 con 0x7fc9bc1082a0 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.776+0000 7fc9c270f700 1 -- 172.21.15.153:0/2280256570 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9bc069180 con 0x7fc9bc1082a0 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.778+0000 7fc9b97fa700 1 -- 172.21.15.153:0/2280256570 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc9b4003a00 con 0x7fc9bc1082a0 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.778+0000 7fc9b97fa700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc9a40a2c20 0x7fc9a40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.778+0000 7fc9b97fa700 1 -- 172.21.15.153:0/2280256570 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fc9b400f020 con 0x7fc9bc1082a0 2024-06-09T22:14:48.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.779+0000 7fc9bbfff700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc9a40a2c20 0x7fc9a40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:14:48.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.779+0000 7fc9bbfff700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc9a40a2c20 0x7fc9a40a50e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac009c60 tx=0x7fc9ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:14:48.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.782+0000 7fc9b97fa700 1 -- 172.21.15.153:0/2280256570 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc9b402a020 con 0x7fc9bc1082a0 2024-06-09T22:14:48.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:48 smithi194 ceph-mon[36856]: Detected new or changed devices on smithi104 2024-06-09T22:14:48.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:48 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi104 to 4272M 2024-06-09T22:14:48.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:48 smithi194 ceph-mon[36856]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:48.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:14:48.986+0000 7fc9c270f700 1 -- 172.21.15.153:0/2280256570 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc9bc0628d0 con 0x7fc9a40a2c20 2024-06-09T22:14:49.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:14:49.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:14:49.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:49 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:49.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:14:49.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:14:49.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:49 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:49.802 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:14:49.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:14:49.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:49 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:50.681 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:50 smithi153 ceph-mon[36599]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:14:50.681 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:50 smithi153 ceph-mon[36599]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:50.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:50 smithi104 ceph-mon[32389]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:14:50.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:50 smithi104 ceph-mon[32389]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:50.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:50 smithi194 ceph-mon[36856]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:14:50.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:50 smithi194 ceph-mon[36856]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:52.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:52 smithi153 ceph-mon[36599]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:52.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:52 smithi153 ceph-mon[36599]: from='client.? 172.21.15.153:0/1259264947' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98aa6a7b-baac-4816-bf0c-fa0b01931b86"}]: dispatch 2024-06-09T22:14:52.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:52 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98aa6a7b-baac-4816-bf0c-fa0b01931b86"}]: dispatch 2024-06-09T22:14:52.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:52 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98aa6a7b-baac-4816-bf0c-fa0b01931b86"}]': finished 2024-06-09T22:14:52.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:52 smithi153 ceph-mon[36599]: osdmap e69: 11 total, 10 up, 11 in 2024-06-09T22:14:52.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:52 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:14:52.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:52 smithi104 ceph-mon[32389]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:52.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:52 smithi104 ceph-mon[32389]: from='client.? 172.21.15.153:0/1259264947' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98aa6a7b-baac-4816-bf0c-fa0b01931b86"}]: dispatch 2024-06-09T22:14:52.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:52 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98aa6a7b-baac-4816-bf0c-fa0b01931b86"}]: dispatch 2024-06-09T22:14:52.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:52 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98aa6a7b-baac-4816-bf0c-fa0b01931b86"}]': finished 2024-06-09T22:14:52.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:52 smithi104 ceph-mon[32389]: osdmap e69: 11 total, 10 up, 11 in 2024-06-09T22:14:52.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:52 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:14:52.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:52 smithi194 ceph-mon[36856]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:52.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:52 smithi194 ceph-mon[36856]: from='client.? 172.21.15.153:0/1259264947' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98aa6a7b-baac-4816-bf0c-fa0b01931b86"}]: dispatch 2024-06-09T22:14:52.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:52 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98aa6a7b-baac-4816-bf0c-fa0b01931b86"}]: dispatch 2024-06-09T22:14:52.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:52 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98aa6a7b-baac-4816-bf0c-fa0b01931b86"}]': finished 2024-06-09T22:14:52.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:52 smithi194 ceph-mon[36856]: osdmap e69: 11 total, 10 up, 11 in 2024-06-09T22:14:52.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:52 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:14:53.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:53 smithi153 ceph-mon[36599]: from='client.? 172.21.15.153:0/3183503642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:14:53.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:53 smithi104 ceph-mon[32389]: from='client.? 172.21.15.153:0/3183503642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:14:53.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:53 smithi194 ceph-mon[36856]: from='client.? 172.21.15.153:0/3183503642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:14:54.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:54 smithi153 ceph-mon[36599]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:54.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:54 smithi104 ceph-mon[32389]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:54.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:54 smithi194 ceph-mon[36856]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:56.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:56 smithi153 ceph-mon[36599]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:56.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:56 smithi104 ceph-mon[32389]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:56.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:56 smithi194 ceph-mon[36856]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:58.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:58 smithi153 ceph-mon[36599]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:58.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:58 smithi104 ceph-mon[32389]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:58.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:58 smithi194 ceph-mon[36856]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:14:59.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:59 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-09T22:14:59.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:14:59 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:59.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:59 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-09T22:14:59.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:14:59 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:14:59.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:59 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-09T22:14:59.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:14:59 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:00.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:00 smithi153 ceph-mon[36599]: Deploying daemon osd.10 on smithi153 2024-06-09T22:15:00.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:00 smithi153 ceph-mon[36599]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:00.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:00 smithi104 ceph-mon[32389]: Deploying daemon osd.10 on smithi153 2024-06-09T22:15:00.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:00 smithi104 ceph-mon[32389]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:00.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:00 smithi194 ceph-mon[36856]: Deploying daemon osd.10 on smithi153 2024-06-09T22:15:00.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:00 smithi194 ceph-mon[36856]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:02.721 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:02 smithi153 ceph-mon[36599]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:02.750 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:02 smithi104 ceph-mon[32389]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:02.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:02 smithi194 ceph-mon[36856]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:04.667 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:04 smithi153 ceph-mon[36599]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:04.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:04 smithi104 ceph-mon[32389]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:04.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:04 smithi194 ceph-mon[36856]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:06.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:06.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:06.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:06.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:06 smithi194 ceph-mon[36856]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:06.347 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:06.347 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:06.347 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:06.347 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:06 smithi153 ceph-mon[36599]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:06.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:06.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:06.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:06.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:06 smithi104 ceph-mon[32389]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:08.639 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:08 smithi153 ceph-mon[36599]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:08.639 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:08 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:08.639 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:08 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:08.639 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:08 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:08.639 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:08 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:08.639 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:08 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:08.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:08 smithi104 ceph-mon[32389]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:08.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:08 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:08.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:08 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:08.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:08 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:08.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:08 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:08.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:08 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:08.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:08 smithi194 ceph-mon[36856]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:08.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:08 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:08.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:08 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:08.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:08 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:08.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:08 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:08.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:08 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:08.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.005+0000 7fc9b97fa700 1 -- 172.21.15.153:0/2280256570 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fc9bc0628d0 con 0x7fc9a40a2c20 2024-06-09T22:15:08.999 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 10 on host 'smithi153' 2024-06-09T22:15:09.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.007+0000 7fc9c270f700 1 -- 172.21.15.153:0/2280256570 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc9a40a2c20 msgr2=0x7fc9a40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:09.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.007+0000 7fc9c270f700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc9a40a2c20 0x7fc9a40a50e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac009c60 tx=0x7fc9ac009260 comp rx=0 tx=0).stop 2024-06-09T22:15:09.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.007+0000 7fc9c270f700 1 -- 172.21.15.153:0/2280256570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9bc1082a0 msgr2=0x7fc9bc1a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:09.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.007+0000 7fc9c270f700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9bc1082a0 0x7fc9bc1a6410 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009bc0 tx=0x7fc9b4004d00 comp rx=0 tx=0).stop 2024-06-09T22:15:09.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.007+0000 7fc9c270f700 1 -- 172.21.15.153:0/2280256570 shutdown_connections 2024-06-09T22:15:09.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.007+0000 7fc9c270f700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc9a40a2c20 0x7fc9a40a50e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:09.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.007+0000 7fc9c270f700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9bc103040 0x7fc9bc1964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:09.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.007+0000 7fc9c270f700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9bc1082a0 0x7fc9bc1a6410 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:09.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.007+0000 7fc9c270f700 1 --2- 172.21.15.153:0/2280256570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9bc105970 0x7fc9bc196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:09.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.007+0000 7fc9c270f700 1 -- 172.21.15.153:0/2280256570 >> 172.21.15.153:0/2280256570 conn(0x7fc9bc0fc9f0 msgr2=0x7fc9bc1027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:09.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.007+0000 7fc9c270f700 1 -- 172.21.15.153:0/2280256570 shutdown_connections 2024-06-09T22:15:09.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-09T22:15:09.007+0000 7fc9c270f700 1 -- 172.21.15.153:0/2280256570 wait complete. 2024-06-09T22:15:09.623 DEBUG:teuthology.orchestra.run.smithi153:osd.10> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.10.service 2024-06-09T22:15:09.625 INFO:tasks.cephadm:Deploying osd.11 on smithi194 with /dev/vg_nvme/lv_1... 2024-06-09T22:15:09.625 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-09T22:15:09.870 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:15:10.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:09 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:10.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:09 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:10.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:09 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:10.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:09 smithi153 ceph-mon[36599]: from='osd.10 [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-09T22:15:10.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:09 smithi153 ceph-mon[36599]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-09T22:15:10.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:09 smithi153 ceph-mon[36599]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:10.251 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:09 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:10.251 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:09 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:10.251 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:09 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:10.251 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:09 smithi104 ceph-mon[32389]: from='osd.10 [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-09T22:15:10.251 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:09 smithi104 ceph-mon[32389]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-09T22:15:10.251 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:09 smithi104 ceph-mon[32389]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:10.252 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:09 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:10.252 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:09 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:10.252 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:09 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:10.252 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:09 smithi194 ceph-mon[36856]: from='osd.10 [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-09T22:15:10.252 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:09 smithi194 ceph-mon[36856]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-09T22:15:10.252 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:09 smithi194 ceph-mon[36856]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:11.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:11 smithi153 ceph-mon[36599]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-09T22:15:11.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:11 smithi153 ceph-mon[36599]: from='osd.10 [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:15:11.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:11 smithi153 ceph-mon[36599]: osdmap e70: 11 total, 10 up, 11 in 2024-06-09T22:15:11.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:11 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:11.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:11 smithi153 ceph-mon[36599]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:15:11.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:11 smithi194 ceph-mon[36856]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-09T22:15:11.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:11 smithi194 ceph-mon[36856]: from='osd.10 [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:15:11.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:11 smithi194 ceph-mon[36856]: osdmap e70: 11 total, 10 up, 11 in 2024-06-09T22:15:11.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:11 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:11.304 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:11 smithi194 ceph-mon[36856]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:15:11.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:11 smithi104 ceph-mon[32389]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-09T22:15:11.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:11 smithi104 ceph-mon[32389]: from='osd.10 [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:15:11.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:11 smithi104 ceph-mon[32389]: osdmap e70: 11 total, 10 up, 11 in 2024-06-09T22:15:11.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:11 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:11.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:11 smithi104 ceph-mon[32389]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-09T22:15:11.737 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:15:11 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:15:11.418+0000 7ffa8c950700 -1 osd.10 0 waiting for initial osdmap 2024-06-09T22:15:11.737 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:15:11 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:15:11.430+0000 7ffa8473b700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:15:12.011 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:12 smithi194 ceph-mon[36856]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-09T22:15:12.011 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:12 smithi194 ceph-mon[36856]: osdmap e71: 11 total, 10 up, 11 in 2024-06-09T22:15:12.011 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:12 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:12.011 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:12 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:12.011 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:12 smithi194 ceph-mon[36856]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:12.011 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:12 smithi194 ceph-mon[36856]: from='osd.10 ' entity='osd.10' 2024-06-09T22:15:12.022 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-09T22:15:12.022 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-09T22:15:12.022 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-06-09T22:15:12.022 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-06-09T22:15:12.022 INFO:teuthology.orchestra.run.smithi194.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128766 s, 814 MB/s 2024-06-09T22:15:12.022 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-06-09T22:15:12.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:12 smithi153 ceph-mon[36599]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-09T22:15:12.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:12 smithi153 ceph-mon[36599]: osdmap e71: 11 total, 10 up, 11 in 2024-06-09T22:15:12.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:12 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:12.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:12 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:12.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:12 smithi153 ceph-mon[36599]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:12.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:12 smithi153 ceph-mon[36599]: from='osd.10 ' entity='osd.10' 2024-06-09T22:15:12.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:12 smithi104 ceph-mon[32389]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-09T22:15:12.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:12 smithi104 ceph-mon[32389]: osdmap e71: 11 total, 10 up, 11 in 2024-06-09T22:15:12.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:12 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:12.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:12 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:12.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:12 smithi104 ceph-mon[32389]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-06-09T22:15:12.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:12 smithi104 ceph-mon[32389]: from='osd.10 ' entity='osd.10' 2024-06-09T22:15:12.836 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_1 2024-06-09T22:15:13.078 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:15:13.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: purged_snaps scrub starts 2024-06-09T22:15:13.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: purged_snaps scrub ok 2024-06-09T22:15:13.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:13.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: osd.10 [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655] boot 2024-06-09T22:15:13.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: osdmap e72: 11 total, 11 up, 11 in 2024-06-09T22:15:13.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:13.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: osdmap e73: 11 total, 11 up, 11 in 2024-06-09T22:15:13.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:13.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:13.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:13.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:13.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:13.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:13.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:13.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:13.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:13.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:13 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:13.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: purged_snaps scrub starts 2024-06-09T22:15:13.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: purged_snaps scrub ok 2024-06-09T22:15:13.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:13.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: osd.10 [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655] boot 2024-06-09T22:15:13.237 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: osdmap e72: 11 total, 11 up, 11 in 2024-06-09T22:15:13.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:13.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: osdmap e73: 11 total, 11 up, 11 in 2024-06-09T22:15:13.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:13.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:13.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:13.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:13.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:13.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:13.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:13.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:13.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:13.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:13 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:13.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: purged_snaps scrub starts 2024-06-09T22:15:13.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: purged_snaps scrub ok 2024-06-09T22:15:13.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:13.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: osd.10 [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655] boot 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: osdmap e72: 11 total, 11 up, 11 in 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: osdmap e73: 11 total, 11 up, 11 in 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:13.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:13 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:14.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:14 smithi194 ceph-mon[36856]: Detected new or changed devices on smithi153 2024-06-09T22:15:14.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:14 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi153 to 4272M 2024-06-09T22:15:14.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:14 smithi194 ceph-mon[36856]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:14.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:14 smithi194 ceph-mon[36856]: osdmap e74: 11 total, 11 up, 11 in 2024-06-09T22:15:14.682 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:14 smithi104 ceph-mon[32389]: Detected new or changed devices on smithi153 2024-06-09T22:15:14.683 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:14 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi153 to 4272M 2024-06-09T22:15:14.683 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:14 smithi104 ceph-mon[32389]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:14.683 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:14 smithi104 ceph-mon[32389]: osdmap e74: 11 total, 11 up, 11 in 2024-06-09T22:15:14.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:14 smithi153 ceph-mon[36599]: Detected new or changed devices on smithi153 2024-06-09T22:15:14.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:14 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi153 to 4272M 2024-06-09T22:15:14.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:14 smithi153 ceph-mon[36599]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:14.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:14 smithi153 ceph-mon[36599]: osdmap e74: 11 total, 11 up, 11 in 2024-06-09T22:15:14.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.824+0000 7f2e37bf9700 1 -- 172.21.15.194:0/1794381569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e30108260 msgr2=0x7f2e3010a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:14.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.824+0000 7f2e37bf9700 1 --2- 172.21.15.194:0/1794381569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e30108260 0x7f2e3010a670 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800b0a0 tx=0x7f2e2800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:15:14.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.824+0000 7f2e37bf9700 1 -- 172.21.15.194:0/1794381569 shutdown_connections 2024-06-09T22:15:14.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.824+0000 7f2e37bf9700 1 --2- 172.21.15.194:0/1794381569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e30108260 0x7f2e3010a670 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:14.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.824+0000 7f2e37bf9700 1 --2- 172.21.15.194:0/1794381569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e30103000 0x7f2e301053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:14.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.824+0000 7f2e37bf9700 1 --2- 172.21.15.194:0/1794381569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e30105930 0x7f2e30107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:14.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.824+0000 7f2e37bf9700 1 -- 172.21.15.194:0/1794381569 >> 172.21.15.194:0/1794381569 conn(0x7f2e300fc9f0 msgr2=0x7f2e300fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:14.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.824+0000 7f2e37bf9700 1 -- 172.21.15.194:0/1794381569 shutdown_connections 2024-06-09T22:15:14.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.824+0000 7f2e37bf9700 1 -- 172.21.15.194:0/1794381569 wait complete. 2024-06-09T22:15:14.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.825+0000 7f2e37bf9700 1 Processor -- start 2024-06-09T22:15:14.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.825+0000 7f2e37bf9700 1 -- start start 2024-06-09T22:15:14.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.825+0000 7f2e37bf9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e30103000 0x7f2e30198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:14.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.825+0000 7f2e37bf9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e30105930 0x7f2e30198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:14.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.825+0000 7f2e37bf9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e30108260 0x7f2e3019fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:14.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.825+0000 7f2e37bf9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3010e1c0 con 0x7f2e30108260 2024-06-09T22:15:14.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.825+0000 7f2e37bf9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3010e040 con 0x7f2e30103000 2024-06-09T22:15:14.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.825+0000 7f2e37bf9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3010e340 con 0x7f2e30105930 2024-06-09T22:15:14.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.825+0000 7f2e35194700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e30105930 0x7f2e30198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:14.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.825+0000 7f2e35194700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e30105930 0x7f2e30198cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:37812/0 (socket says 172.21.15.194:37812) 2024-06-09T22:15:14.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.825+0000 7f2e35194700 1 -- 172.21.15.194:0/833084381 learned_addr learned my addr 172.21.15.194:0/833084381 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-06-09T22:15:14.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e36196700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e30108260 0x7f2e3019fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:14.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e35995700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e30103000 0x7f2e30198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:14.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e35194700 1 -- 172.21.15.194:0/833084381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e30103000 msgr2=0x7f2e30198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:14.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e35194700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e30103000 0x7f2e30198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:14.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e35194700 1 -- 172.21.15.194:0/833084381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e30108260 msgr2=0x7f2e3019fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:14.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e35194700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e30108260 0x7f2e3019fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:14.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e35194700 1 -- 172.21.15.194:0/833084381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e301a03d0 con 0x7f2e30105930 2024-06-09T22:15:14.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e35995700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e30103000 0x7f2e30198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:15:14.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e35194700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e30105930 0x7f2e30198cb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00b560 tx=0x7f2e2c00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:14.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e26ffd700 1 -- 172.21.15.194:0/833084381 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c015020 con 0x7f2e30105930 2024-06-09T22:15:14.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e26ffd700 1 -- 172.21.15.194:0/833084381 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e2c011d10 con 0x7f2e30105930 2024-06-09T22:15:14.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e37bf9700 1 -- 172.21.15.194:0/833084381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e301a06c0 con 0x7f2e30105930 2024-06-09T22:15:14.825 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e26ffd700 1 -- 172.21.15.194:0/833084381 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c010e50 con 0x7f2e30105930 2024-06-09T22:15:14.825 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.826+0000 7f2e37bf9700 1 -- 172.21.15.194:0/833084381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e301a0c80 con 0x7f2e30105930 2024-06-09T22:15:14.825 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.827+0000 7f2e37bf9700 1 -- 172.21.15.194:0/833084381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e30069180 con 0x7f2e30105930 2024-06-09T22:15:14.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.828+0000 7f2e26ffd700 1 -- 172.21.15.194:0/833084381 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f2e2c018020 con 0x7f2e30105930 2024-06-09T22:15:14.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.829+0000 7f2e26ffd700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2e1c0a2dc0 0x7f2e1c0a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:14.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.829+0000 7f2e26ffd700 1 -- 172.21.15.194:0/833084381 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f2e2c0ce380 con 0x7f2e30105930 2024-06-09T22:15:14.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.829+0000 7f2e35995700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2e1c0a2dc0 0x7f2e1c0a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:14.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.829+0000 7f2e35995700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2e1c0a2dc0 0x7f2e1c0a5280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2e200099d0 tx=0x7f2e20008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:14.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:14.833+0000 7f2e26ffd700 1 -- 172.21.15.194:0/833084381 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2e2c01f020 con 0x7f2e30105930 2024-06-09T22:15:15.033 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:15.039+0000 7f2e37bf9700 1 -- 172.21.15.194:0/833084381 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2e300628d0 con 0x7f2e1c0a2dc0 2024-06-09T22:15:15.627 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:15 smithi104 ceph-mon[32389]: osdmap e75: 11 total, 11 up, 11 in 2024-06-09T22:15:15.627 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:15:15.627 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:15:15.627 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:15.642 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:15 smithi194 ceph-mon[36856]: osdmap e75: 11 total, 11 up, 11 in 2024-06-09T22:15:15.642 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:15:15.642 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:15:15.642 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:15.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:15 smithi153 ceph-mon[36599]: osdmap e75: 11 total, 11 up, 11 in 2024-06-09T22:15:15.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:15 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-09T22:15:15.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:15 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-09T22:15:15.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:15 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:16.656 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:16 smithi194 ceph-mon[36856]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:15:16.656 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:16 smithi194 ceph-mon[36856]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-09T22:15:16.656 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:16 smithi194 ceph-mon[36856]: osdmap e76: 11 total, 11 up, 11 in 2024-06-09T22:15:16.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:16 smithi153 ceph-mon[36599]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:15:16.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:16 smithi153 ceph-mon[36599]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-09T22:15:16.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:16 smithi153 ceph-mon[36599]: osdmap e76: 11 total, 11 up, 11 in 2024-06-09T22:15:16.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:16 smithi104 ceph-mon[32389]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:15:16.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:16 smithi104 ceph-mon[32389]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-09T22:15:16.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:16 smithi104 ceph-mon[32389]: osdmap e76: 11 total, 11 up, 11 in 2024-06-09T22:15:18.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:18 smithi153 ceph-mon[36599]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-06-09T22:15:18.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:18 smithi153 ceph-mon[36599]: from='client.? 172.21.15.194:0/1534223750' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f107b3c6-8206-4cd0-bc41-5cc8eb8f179d"}]: dispatch 2024-06-09T22:15:18.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:18 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f107b3c6-8206-4cd0-bc41-5cc8eb8f179d"}]: dispatch 2024-06-09T22:15:18.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:18 smithi153 ceph-mon[36599]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f107b3c6-8206-4cd0-bc41-5cc8eb8f179d"}]': finished 2024-06-09T22:15:18.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:18 smithi153 ceph-mon[36599]: osdmap e77: 12 total, 11 up, 12 in 2024-06-09T22:15:18.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:18 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:18.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:18 smithi104 ceph-mon[32389]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-06-09T22:15:18.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:18 smithi104 ceph-mon[32389]: from='client.? 172.21.15.194:0/1534223750' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f107b3c6-8206-4cd0-bc41-5cc8eb8f179d"}]: dispatch 2024-06-09T22:15:18.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:18 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f107b3c6-8206-4cd0-bc41-5cc8eb8f179d"}]: dispatch 2024-06-09T22:15:18.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:18 smithi104 ceph-mon[32389]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f107b3c6-8206-4cd0-bc41-5cc8eb8f179d"}]': finished 2024-06-09T22:15:18.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:18 smithi104 ceph-mon[32389]: osdmap e77: 12 total, 11 up, 12 in 2024-06-09T22:15:18.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:18 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:18.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:18 smithi194 ceph-mon[36856]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-06-09T22:15:18.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:18 smithi194 ceph-mon[36856]: from='client.? 172.21.15.194:0/1534223750' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f107b3c6-8206-4cd0-bc41-5cc8eb8f179d"}]: dispatch 2024-06-09T22:15:18.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:18 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f107b3c6-8206-4cd0-bc41-5cc8eb8f179d"}]: dispatch 2024-06-09T22:15:18.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:18 smithi194 ceph-mon[36856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f107b3c6-8206-4cd0-bc41-5cc8eb8f179d"}]': finished 2024-06-09T22:15:18.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:18 smithi194 ceph-mon[36856]: osdmap e77: 12 total, 11 up, 12 in 2024-06-09T22:15:18.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:18 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:19.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:19 smithi153 ceph-mon[36599]: from='client.? 172.21.15.194:0/3293128256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:15:19.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:19 smithi104 ceph-mon[32389]: from='client.? 172.21.15.194:0/3293128256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:15:19.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:19 smithi194 ceph-mon[36856]: from='client.? 172.21.15.194:0/3293128256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-09T22:15:20.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:20 smithi153 ceph-mon[36599]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-06-09T22:15:20.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:20 smithi104 ceph-mon[32389]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-06-09T22:15:20.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:20 smithi194 ceph-mon[36856]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-06-09T22:15:22.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:22 smithi153 ceph-mon[36599]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-06-09T22:15:22.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:22 smithi104 ceph-mon[32389]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-06-09T22:15:22.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:22 smithi194 ceph-mon[36856]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-06-09T22:15:24.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:24 smithi194 ceph-mon[36856]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:24.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:24 smithi153 ceph-mon[36599]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:24.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:24 smithi104 ceph-mon[32389]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:26.639 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:26 smithi194 ceph-mon[36856]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:26.639 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-09T22:15:26.639 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:26.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:26 smithi153 ceph-mon[36599]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:26.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-09T22:15:26.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:26.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:26 smithi104 ceph-mon[32389]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:26.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-09T22:15:26.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:27.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:27 smithi153 ceph-mon[36599]: Deploying daemon osd.11 on smithi194 2024-06-09T22:15:27.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:27 smithi194 ceph-mon[36856]: Deploying daemon osd.11 on smithi194 2024-06-09T22:15:27.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:27 smithi104 ceph-mon[32389]: Deploying daemon osd.11 on smithi194 2024-06-09T22:15:28.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:28 smithi153 ceph-mon[36599]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:28.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:28 smithi104 ceph-mon[32389]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:28.800 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:28 smithi194 ceph-mon[36856]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:30.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:30 smithi194 ceph-mon[36856]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:30.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:30 smithi153 ceph-mon[36599]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:30.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:30 smithi104 ceph-mon[32389]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:32.634 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:32 smithi194 ceph-mon[36856]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:32.634 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:32.634 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:32.634 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:32.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:32 smithi153 ceph-mon[36599]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:32.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:32.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:32.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:32.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:32 smithi104 ceph-mon[32389]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:32.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:32.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:32.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:34.472 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:34 smithi194 ceph-mon[36856]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:34.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:34 smithi153 ceph-mon[36599]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:34.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:34 smithi104 ceph-mon[32389]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:35.450 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 11 on host 'smithi194' 2024-06-09T22:15:35.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.455+0000 7f2e26ffd700 1 -- 172.21.15.194:0/833084381 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2e300628d0 con 0x7f2e1c0a2dc0 2024-06-09T22:15:35.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.458+0000 7f2e37bf9700 1 -- 172.21.15.194:0/833084381 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2e1c0a2dc0 msgr2=0x7f2e1c0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:35.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.458+0000 7f2e37bf9700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2e1c0a2dc0 0x7f2e1c0a5280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2e200099d0 tx=0x7f2e20008040 comp rx=0 tx=0).stop 2024-06-09T22:15:35.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.458+0000 7f2e37bf9700 1 -- 172.21.15.194:0/833084381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e30105930 msgr2=0x7f2e30198cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:35.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.458+0000 7f2e37bf9700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e30105930 0x7f2e30198cb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00b560 tx=0x7f2e2c00b870 comp rx=0 tx=0).stop 2024-06-09T22:15:35.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.458+0000 7f2e37bf9700 1 -- 172.21.15.194:0/833084381 shutdown_connections 2024-06-09T22:15:35.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.458+0000 7f2e37bf9700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2e1c0a2dc0 0x7f2e1c0a5280 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:35.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.458+0000 7f2e37bf9700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e30108260 0x7f2e3019fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:35.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.458+0000 7f2e37bf9700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e30103000 0x7f2e30198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:35.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.458+0000 7f2e37bf9700 1 --2- 172.21.15.194:0/833084381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e30105930 0x7f2e30198cb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:35.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.458+0000 7f2e37bf9700 1 -- 172.21.15.194:0/833084381 >> 172.21.15.194:0/833084381 conn(0x7f2e300fc9f0 msgr2=0x7f2e30103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:35.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.458+0000 7f2e37bf9700 1 -- 172.21.15.194:0/833084381 shutdown_connections 2024-06-09T22:15:35.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:35.458+0000 7f2e37bf9700 1 -- 172.21.15.194:0/833084381 wait complete. 2024-06-09T22:15:36.149 DEBUG:teuthology.orchestra.run.smithi194:osd.11> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.11.service 2024-06-09T22:15:36.152 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-06-09T22:15:36.152 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-09T22:15:36.410 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:15:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:36 smithi153 ceph-mon[36599]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:36 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:36 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:36 smithi153 ceph-mon[36599]: from='osd.11 [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-09T22:15:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:36 smithi153 ceph-mon[36599]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-09T22:15:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:36 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:36 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:36.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:36 smithi104 ceph-mon[32389]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:36.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:36.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:36.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:36 smithi104 ceph-mon[32389]: from='osd.11 [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-09T22:15:36.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:36 smithi104 ceph-mon[32389]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-09T22:15:36.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:36.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:36.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:36 smithi194 ceph-mon[36856]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:36.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:36 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:36.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:36 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:36.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:36 smithi194 ceph-mon[36856]: from='osd.11 [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-09T22:15:36.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:36 smithi194 ceph-mon[36856]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-09T22:15:36.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:36 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:36.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:36 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:37.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:37 smithi194 ceph-mon[36856]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-09T22:15:37.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:37 smithi194 ceph-mon[36856]: osdmap e78: 12 total, 11 up, 12 in 2024-06-09T22:15:37.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:37 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:37.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:37 smithi194 ceph-mon[36856]: from='osd.11 [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:15:37.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:37 smithi194 ceph-mon[36856]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:15:37.554 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:15:37 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:15:37.463+0000 7f1a90687700 -1 osd.11 0 waiting for initial osdmap 2024-06-09T22:15:37.554 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:15:37 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:15:37.474+0000 7f1a8c47a700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:15:37.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:37 smithi153 ceph-mon[36599]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-09T22:15:37.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:37 smithi153 ceph-mon[36599]: osdmap e78: 12 total, 11 up, 12 in 2024-06-09T22:15:37.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:37 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:37.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:37 smithi153 ceph-mon[36599]: from='osd.11 [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:15:37.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:37 smithi153 ceph-mon[36599]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:15:37.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:37 smithi104 ceph-mon[32389]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-09T22:15:37.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:37 smithi104 ceph-mon[32389]: osdmap e78: 12 total, 11 up, 12 in 2024-06-09T22:15:37.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:37 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:37.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:37 smithi104 ceph-mon[32389]: from='osd.11 [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:15:37.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:37 smithi104 ceph-mon[32389]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-06-09T22:15:38.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.144+0000 7f5c8c96e700 1 -- 172.21.15.104:0/3249430215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c88076480 msgr2=0x7f5c880748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:38.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.144+0000 7f5c8c96e700 1 --2- 172.21.15.104:0/3249430215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c88076480 0x7f5c880748e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009990 tx=0x7f5c78009ca0 comp rx=0 tx=0).stop 2024-06-09T22:15:38.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.144+0000 7f5c8c96e700 1 -- 172.21.15.104:0/3249430215 shutdown_connections 2024-06-09T22:15:38.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.144+0000 7f5c8c96e700 1 --2- 172.21.15.104:0/3249430215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c88076480 0x7f5c880748e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:38.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.144+0000 7f5c8c96e700 1 --2- 172.21.15.104:0/3249430215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c8810a3e0 0x7f5c8810c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:38.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.144+0000 7f5c8c96e700 1 --2- 172.21.15.104:0/3249430215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c88074eb0 0x7f5c88075310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:38.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.144+0000 7f5c8c96e700 1 -- 172.21.15.104:0/3249430215 >> 172.21.15.104:0/3249430215 conn(0x7f5c880fdae0 msgr2=0x7f5c880fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:38.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.144+0000 7f5c8c96e700 1 -- 172.21.15.104:0/3249430215 shutdown_connections 2024-06-09T22:15:38.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.144+0000 7f5c8c96e700 1 -- 172.21.15.104:0/3249430215 wait complete. 2024-06-09T22:15:38.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.145+0000 7f5c8c96e700 1 Processor -- start 2024-06-09T22:15:38.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.145+0000 7f5c8c96e700 1 -- start start 2024-06-09T22:15:38.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.145+0000 7f5c8c96e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c88074eb0 0x7f5c8819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:38.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.145+0000 7f5c8c96e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c88076480 0x7f5c8819ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:38.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.145+0000 7f5c8c96e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c8810a3e0 0x7f5c881a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:38.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.145+0000 7f5c8c96e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5c88110370 con 0x7f5c8810a3e0 2024-06-09T22:15:38.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.145+0000 7f5c8c96e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5c881101f0 con 0x7f5c88074eb0 2024-06-09T22:15:38.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.145+0000 7f5c8c96e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c881104f0 con 0x7f5c88076480 2024-06-09T22:15:38.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c86d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c8810a3e0 0x7f5c881a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:38.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c86d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c8810a3e0 0x7f5c881a1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42152/0 (socket says 172.21.15.104:42152) 2024-06-09T22:15:38.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c86d9d700 1 -- 172.21.15.104:0/1096308692 learned_addr learned my addr 172.21.15.104:0/1096308692 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:15:38.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c8659c700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c88074eb0 0x7f5c8819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:38.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c85d9b700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c88076480 0x7f5c8819ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:38.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c86d9d700 1 -- 172.21.15.104:0/1096308692 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c88076480 msgr2=0x7f5c8819ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:38.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c86d9d700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c88076480 0x7f5c8819ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:38.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c86d9d700 1 -- 172.21.15.104:0/1096308692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c88074eb0 msgr2=0x7f5c8819a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:38.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c86d9d700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c88074eb0 0x7f5c8819a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:38.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c86d9d700 1 -- 172.21.15.104:0/1096308692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c881a25b0 con 0x7f5c8810a3e0 2024-06-09T22:15:38.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c86d9d700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c8810a3e0 0x7f5c881a1f10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5c8000b9a0 tx=0x7f5c8000bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:38.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c777fe700 1 -- 172.21.15.104:0/1096308692 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8000d850 con 0x7f5c8810a3e0 2024-06-09T22:15:38.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c777fe700 1 -- 172.21.15.104:0/1096308692 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c8000e420 con 0x7f5c8810a3e0 2024-06-09T22:15:38.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c8c96e700 1 -- 172.21.15.104:0/1096308692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c881a28a0 con 0x7f5c8810a3e0 2024-06-09T22:15:38.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.146+0000 7f5c777fe700 1 -- 172.21.15.104:0/1096308692 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c80017660 con 0x7f5c8810a3e0 2024-06-09T22:15:38.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.147+0000 7f5c8c96e700 1 -- 172.21.15.104:0/1096308692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c881a2e60 con 0x7f5c8810a3e0 2024-06-09T22:15:38.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.148+0000 7f5c8c96e700 1 -- 172.21.15.104:0/1096308692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c88069180 con 0x7f5c8810a3e0 2024-06-09T22:15:38.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.148+0000 7f5c777fe700 1 -- 172.21.15.104:0/1096308692 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f5c8000d9f0 con 0x7f5c8810a3e0 2024-06-09T22:15:38.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.149+0000 7f5c777fe700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f5c700a2ca0 0x7f5c700a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:38.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.149+0000 7f5c8659c700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f5c700a2ca0 0x7f5c700a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:38.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.149+0000 7f5c777fe700 1 -- 172.21.15.104:0/1096308692 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f5c800ce590 con 0x7f5c8810a3e0 2024-06-09T22:15:38.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.149+0000 7f5c8659c700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f5c700a2ca0 0x7f5c700a5160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5c78003890 tx=0x7f5c78005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:38.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.154+0000 7f5c777fe700 1 -- 172.21.15.104:0/1096308692 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5c80013030 con 0x7f5c8810a3e0 2024-06-09T22:15:38.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.358+0000 7f5c8c96e700 1 -- 172.21.15.104:0/1096308692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f5c8802e0e0 con 0x7f5c8810a3e0 2024-06-09T22:15:38.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.358+0000 7f5c777fe700 1 -- 172.21.15.104:0/1096308692 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f5c8802e0e0 con 0x7f5c8810a3e0 2024-06-09T22:15:38.354 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:15:38.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.361+0000 7f5c8c96e700 1 -- 172.21.15.104:0/1096308692 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f5c700a2ca0 msgr2=0x7f5c700a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:38.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.361+0000 7f5c8c96e700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f5c700a2ca0 0x7f5c700a5160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5c78003890 tx=0x7f5c78005fd0 comp rx=0 tx=0).stop 2024-06-09T22:15:38.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.361+0000 7f5c8c96e700 1 -- 172.21.15.104:0/1096308692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c8810a3e0 msgr2=0x7f5c881a1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:38.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.361+0000 7f5c8c96e700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c8810a3e0 0x7f5c881a1f10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5c8000b9a0 tx=0x7f5c8000bcb0 comp rx=0 tx=0).stop 2024-06-09T22:15:38.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.361+0000 7f5c8c96e700 1 -- 172.21.15.104:0/1096308692 shutdown_connections 2024-06-09T22:15:38.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.361+0000 7f5c8c96e700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f5c700a2ca0 0x7f5c700a5160 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:38.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.361+0000 7f5c8c96e700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c8810a3e0 0x7f5c881a1f10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:38.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.361+0000 7f5c8c96e700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c88074eb0 0x7f5c8819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:38.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.361+0000 7f5c8c96e700 1 --2- 172.21.15.104:0/1096308692 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c88076480 0x7f5c8819ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:38.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.361+0000 7f5c8c96e700 1 -- 172.21.15.104:0/1096308692 >> 172.21.15.104:0/1096308692 conn(0x7f5c880fdae0 msgr2=0x7f5c88106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:38.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.361+0000 7f5c8c96e700 1 -- 172.21.15.104:0/1096308692 shutdown_connections 2024-06-09T22:15:38.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:38.361+0000 7f5c8c96e700 1 -- 172.21.15.104:0/1096308692 wait complete. 2024-06-09T22:15:38.678 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:38 smithi104 ceph-mon[32389]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:38.678 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:38 smithi104 ceph-mon[32389]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-06-09T22:15:38.678 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:38 smithi104 ceph-mon[32389]: osdmap e79: 12 total, 11 up, 12 in 2024-06-09T22:15:38.678 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:38 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:38.678 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:38 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:38.678 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:38 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1096308692' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-09T22:15:38.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:38 smithi153 ceph-mon[36599]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:38.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:38 smithi153 ceph-mon[36599]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-06-09T22:15:38.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:38 smithi153 ceph-mon[36599]: osdmap e79: 12 total, 11 up, 12 in 2024-06-09T22:15:38.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:38 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:38.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:38 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:38.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:38 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1096308692' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-09T22:15:38.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:38 smithi194 ceph-mon[36856]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:38.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:38 smithi194 ceph-mon[36856]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-06-09T22:15:38.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:38 smithi194 ceph-mon[36856]: osdmap e79: 12 total, 11 up, 12 in 2024-06-09T22:15:38.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:38 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:38.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:38 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:38.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:38 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1096308692' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-09T22:15:39.046 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1717971312,"num_in_osds":12,"osd_in_since":1717971318,"num_remapped_pgs":0} 2024-06-09T22:15:39.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:39 smithi153 ceph-mon[36599]: purged_snaps scrub starts 2024-06-09T22:15:39.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:39 smithi153 ceph-mon[36599]: purged_snaps scrub ok 2024-06-09T22:15:39.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:39.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:39 smithi153 ceph-mon[36599]: osd.11 [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515] boot 2024-06-09T22:15:39.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:39 smithi153 ceph-mon[36599]: osdmap e80: 12 total, 12 up, 12 in 2024-06-09T22:15:39.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:39.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:39 smithi104 ceph-mon[32389]: purged_snaps scrub starts 2024-06-09T22:15:39.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:39 smithi104 ceph-mon[32389]: purged_snaps scrub ok 2024-06-09T22:15:39.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:39.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:39 smithi104 ceph-mon[32389]: osd.11 [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515] boot 2024-06-09T22:15:39.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:39 smithi104 ceph-mon[32389]: osdmap e80: 12 total, 12 up, 12 in 2024-06-09T22:15:39.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:39.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:39 smithi194 ceph-mon[36856]: purged_snaps scrub starts 2024-06-09T22:15:39.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:39 smithi194 ceph-mon[36856]: purged_snaps scrub ok 2024-06-09T22:15:39.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:39.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:39 smithi194 ceph-mon[36856]: osd.11 [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515] boot 2024-06-09T22:15:39.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:39 smithi194 ceph-mon[36856]: osdmap e80: 12 total, 12 up, 12 in 2024-06-09T22:15:39.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:15:40.047 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-09T22:15:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:15:40.660 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:40.660 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: osdmap e81: 12 total, 12 up, 12 in 2024-06-09T22:15:40.660 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:40.660 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:40.660 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:40.660 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:40.660 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:40.660 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:40.661 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:40.661 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:40.661 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:40.661 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:40.661 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:40.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:40.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: osdmap e81: 12 total, 12 up, 12 in 2024-06-09T22:15:40.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:40.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:40.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:40.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:40.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:40.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:40.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:40.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:40.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:40.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:40.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:40.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-06-09T22:15:40.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: osdmap e81: 12 total, 12 up, 12 in 2024-06-09T22:15:40.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:40.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:40.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:40.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:40.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:40.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:15:40.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:40.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:40.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:40.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:40.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:41.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:41 smithi104 ceph-mon[32389]: Detected new or changed devices on smithi194 2024-06-09T22:15:41.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:41 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi194 to 4272M 2024-06-09T22:15:41.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:41.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:41.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:41.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:41 smithi153 ceph-mon[36599]: Detected new or changed devices on smithi194 2024-06-09T22:15:41.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:41 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi194 to 4272M 2024-06-09T22:15:41.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:41.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:41.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:41.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:41 smithi194 ceph-mon[36856]: Detected new or changed devices on smithi194 2024-06-09T22:15:41.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:41 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi194 to 4272M 2024-06-09T22:15:41.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:41.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:41.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:42.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.038+0000 7f803940f700 1 -- 172.21.15.104:0/127385723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8034102f60 msgr2=0x7f8034109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:42.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.038+0000 7f803940f700 1 --2- 172.21.15.104:0/127385723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8034102f60 0x7f8034109810 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f802c00b0d0 tx=0x7f802c00b3e0 comp rx=0 tx=0).stop 2024-06-09T22:15:42.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.038+0000 7f803940f700 1 -- 172.21.15.104:0/127385723 shutdown_connections 2024-06-09T22:15:42.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.038+0000 7f803940f700 1 --2- 172.21.15.104:0/127385723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8034102f60 0x7f8034109810 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:42.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.038+0000 7f803940f700 1 --2- 172.21.15.104:0/127385723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80341013c0 0x7f80341017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:42.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.038+0000 7f803940f700 1 --2- 172.21.15.104:0/127385723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80341025c0 0x7f8034102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:42.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.038+0000 7f803940f700 1 -- 172.21.15.104:0/127385723 >> 172.21.15.104:0/127385723 conn(0x7f80340fc960 msgr2=0x7f80340feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:42.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.038+0000 7f803940f700 1 -- 172.21.15.104:0/127385723 shutdown_connections 2024-06-09T22:15:42.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.038+0000 7f803940f700 1 -- 172.21.15.104:0/127385723 wait complete. 2024-06-09T22:15:42.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.038+0000 7f803940f700 1 Processor -- start 2024-06-09T22:15:42.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f803940f700 1 -- start start 2024-06-09T22:15:42.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f803940f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80341013c0 0x7f8034105350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:42.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f803940f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80341025c0 0x7f80341039a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:42.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f803940f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8034102f60 0x7f8034103ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:42.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f803940f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f803410d040 con 0x7f8034102f60 2024-06-09T22:15:42.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f803940f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f803410cec0 con 0x7f80341013c0 2024-06-09T22:15:42.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f803940f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f803410d1c0 con 0x7f80341025c0 2024-06-09T22:15:42.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f80337fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8034102f60 0x7f8034103ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:42.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f80337fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8034102f60 0x7f8034103ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42182/0 (socket says 172.21.15.104:42182) 2024-06-09T22:15:42.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f80337fe700 1 -- 172.21.15.104:0/579220372 learned_addr learned my addr 172.21.15.104:0/579220372 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:15:42.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f802a7fc700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80341025c0 0x7f80341039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:42.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f8032ffd700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80341013c0 0x7f8034105350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:42.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f80337fe700 1 -- 172.21.15.104:0/579220372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80341025c0 msgr2=0x7f80341039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:42.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f80337fe700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80341025c0 0x7f80341039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:42.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f80337fe700 1 -- 172.21.15.104:0/579220372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80341013c0 msgr2=0x7f8034105350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:42.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f80337fe700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80341013c0 0x7f8034105350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:42.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f80337fe700 1 -- 172.21.15.104:0/579220372 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8034104790 con 0x7f8034102f60 2024-06-09T22:15:42.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f802a7fc700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80341025c0 0x7f80341039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:15:42.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f8032ffd700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80341013c0 0x7f8034105350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:15:42.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.039+0000 7f80337fe700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8034102f60 0x7f8034103ee0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f802c006070 tx=0x7f802c007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:42.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.040+0000 7f8030ff9700 1 -- 172.21.15.104:0/579220372 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c021020 con 0x7f8034102f60 2024-06-09T22:15:42.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.040+0000 7f803940f700 1 -- 172.21.15.104:0/579220372 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80341a28a0 con 0x7f8034102f60 2024-06-09T22:15:42.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.040+0000 7f8030ff9700 1 -- 172.21.15.104:0/579220372 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f802c01d950 con 0x7f8034102f60 2024-06-09T22:15:42.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.040+0000 7f8030ff9700 1 -- 172.21.15.104:0/579220372 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c005510 con 0x7f8034102f60 2024-06-09T22:15:42.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.040+0000 7f803940f700 1 -- 172.21.15.104:0/579220372 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80341a2e60 con 0x7f8034102f60 2024-06-09T22:15:42.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.041+0000 7f803940f700 1 -- 172.21.15.104:0/579220372 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8034069180 con 0x7f8034102f60 2024-06-09T22:15:42.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.042+0000 7f8030ff9700 1 -- 172.21.15.104:0/579220372 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f802c024020 con 0x7f8034102f60 2024-06-09T22:15:42.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.042+0000 7f8030ff9700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f80200a2c20 0x7f80200a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:42.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.042+0000 7f8032ffd700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f80200a2c20 0x7f80200a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:42.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.042+0000 7f8030ff9700 1 -- 172.21.15.104:0/579220372 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f802c00f020 con 0x7f8034102f60 2024-06-09T22:15:42.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.043+0000 7f8032ffd700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f80200a2c20 0x7f80200a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8034102420 tx=0x7f8024008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:42.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.047+0000 7f8030ff9700 1 -- 172.21.15.104:0/579220372 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f802c011e50 con 0x7f8034102f60 2024-06-09T22:15:42.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.242+0000 7f803940f700 1 -- 172.21.15.104:0/579220372 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f80341a36c0 con 0x7f8034102f60 2024-06-09T22:15:42.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.242+0000 7f8030ff9700 1 -- 172.21.15.104:0/579220372 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f802c09d2f0 con 0x7f8034102f60 2024-06-09T22:15:42.237 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:15:42.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.244+0000 7f803940f700 1 -- 172.21.15.104:0/579220372 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f80200a2c20 msgr2=0x7f80200a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:42.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.244+0000 7f803940f700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f80200a2c20 0x7f80200a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8034102420 tx=0x7f8024008040 comp rx=0 tx=0).stop 2024-06-09T22:15:42.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.244+0000 7f803940f700 1 -- 172.21.15.104:0/579220372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8034102f60 msgr2=0x7f8034103ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.244+0000 7f803940f700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8034102f60 0x7f8034103ee0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f802c006070 tx=0x7f802c007e40 comp rx=0 tx=0).stop 2024-06-09T22:15:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.244+0000 7f803940f700 1 -- 172.21.15.104:0/579220372 shutdown_connections 2024-06-09T22:15:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.244+0000 7f803940f700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f80200a2c20 0x7f80200a50e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.244+0000 7f803940f700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8034102f60 0x7f8034103ee0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.244+0000 7f803940f700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80341013c0 0x7f8034105350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.244+0000 7f803940f700 1 --2- 172.21.15.104:0/579220372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f80341025c0 0x7f80341039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.244+0000 7f803940f700 1 -- 172.21.15.104:0/579220372 >> 172.21.15.104:0/579220372 conn(0x7f80340fc960 msgr2=0x7f80340fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.245+0000 7f803940f700 1 -- 172.21.15.104:0/579220372 shutdown_connections 2024-06-09T22:15:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:42.245+0000 7f803940f700 1 -- 172.21.15.104:0/579220372 wait complete. 2024-06-09T22:15:42.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:42 smithi104 ceph-mon[32389]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:42.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:42 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/579220372' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-09T22:15:42.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:42 smithi153 ceph-mon[36599]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:42.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:42 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/579220372' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-09T22:15:42.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:42 smithi194 ceph-mon[36856]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:42.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:42 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/579220372' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-09T22:15:42.940 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1717971338,"num_in_osds":12,"osd_in_since":1717971318,"num_remapped_pgs":0} 2024-06-09T22:15:42.941 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:15:43.177 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:15:44.656 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:44 smithi104 ceph-mon[32389]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:44.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:44 smithi153 ceph-mon[36599]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:44.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:44 smithi194 ceph-mon[36856]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:44.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.896+0000 7fca4f2df700 1 -- 172.21.15.104:0/1639230118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca48101620 msgr2=0x7fca48101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:44.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.896+0000 7fca4f2df700 1 --2- 172.21.15.104:0/1639230118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca48101620 0x7fca48101a00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fca380099c0 tx=0x7fca38009cd0 comp rx=0 tx=0).stop 2024-06-09T22:15:44.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.896+0000 7fca4f2df700 1 -- 172.21.15.104:0/1639230118 shutdown_connections 2024-06-09T22:15:44.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.896+0000 7fca4f2df700 1 --2- 172.21.15.104:0/1639230118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca48101620 0x7fca48101a00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:44.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.896+0000 7fca4f2df700 1 --2- 172.21.15.104:0/1639230118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca481066b0 0x7fca4810e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:44.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.896+0000 7fca4f2df700 1 --2- 172.21.15.104:0/1639230118 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca48101fd0 0x7fca48105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:44.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.896+0000 7fca4f2df700 1 -- 172.21.15.104:0/1639230118 >> 172.21.15.104:0/1639230118 conn(0x7fca48078360 msgr2=0x7fca48078770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:44.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.896+0000 7fca4f2df700 1 -- 172.21.15.104:0/1639230118 shutdown_connections 2024-06-09T22:15:44.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.896+0000 7fca4f2df700 1 -- 172.21.15.104:0/1639230118 wait complete. 2024-06-09T22:15:44.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.897+0000 7fca4f2df700 1 Processor -- start 2024-06-09T22:15:44.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.897+0000 7fca4f2df700 1 -- start start 2024-06-09T22:15:44.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.897+0000 7fca4f2df700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca48101620 0x7fca48198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:44.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.897+0000 7fca4f2df700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca48101fd0 0x7fca48199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:44.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.897+0000 7fca4f2df700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca481066b0 0x7fca4819d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:44.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.897+0000 7fca4f2df700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fca48111710 con 0x7fca481066b0 2024-06-09T22:15:44.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.897+0000 7fca4f2df700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fca48111590 con 0x7fca48101620 2024-06-09T22:15:44.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.897+0000 7fca4f2df700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fca48111890 con 0x7fca48101fd0 2024-06-09T22:15:44.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.897+0000 7fca4d87c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca481066b0 0x7fca4819d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:44.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.897+0000 7fca4d87c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca481066b0 0x7fca4819d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42202/0 (socket says 172.21.15.104:42202) 2024-06-09T22:15:44.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.897+0000 7fca4d87c700 1 -- 172.21.15.104:0/582066926 learned_addr learned my addr 172.21.15.104:0/582066926 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:15:44.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca4c87a700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca48101fd0 0x7fca48199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:44.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca4d07b700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca48101620 0x7fca48198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:44.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca4d87c700 1 -- 172.21.15.104:0/582066926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca48101fd0 msgr2=0x7fca48199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:44.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca4d87c700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca48101fd0 0x7fca48199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:44.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca4d87c700 1 -- 172.21.15.104:0/582066926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca48101620 msgr2=0x7fca48198d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:44.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca4d87c700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca48101620 0x7fca48198d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:44.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca4d87c700 1 -- 172.21.15.104:0/582066926 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca4819dd20 con 0x7fca481066b0 2024-06-09T22:15:44.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca4c87a700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca48101fd0 0x7fca48199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:15:44.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca4d87c700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca481066b0 0x7fca4819d5e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fca4000b970 tx=0x7fca4000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:44.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca3e7fc700 1 -- 172.21.15.104:0/582066926 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca40019070 con 0x7fca481066b0 2024-06-09T22:15:44.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca3e7fc700 1 -- 172.21.15.104:0/582066926 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca40014440 con 0x7fca481066b0 2024-06-09T22:15:44.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca4f2df700 1 -- 172.21.15.104:0/582066926 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca4819e010 con 0x7fca481066b0 2024-06-09T22:15:44.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca3e7fc700 1 -- 172.21.15.104:0/582066926 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca400075d0 con 0x7fca481066b0 2024-06-09T22:15:44.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.898+0000 7fca4f2df700 1 -- 172.21.15.104:0/582066926 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca481a5890 con 0x7fca481066b0 2024-06-09T22:15:44.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.899+0000 7fca4f2df700 1 -- 172.21.15.104:0/582066926 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca48069180 con 0x7fca481066b0 2024-06-09T22:15:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.900+0000 7fca3e7fc700 1 -- 172.21.15.104:0/582066926 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fca4001c070 con 0x7fca481066b0 2024-06-09T22:15:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.901+0000 7fca3e7fc700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fca340a2c20 0x7fca340a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.901+0000 7fca4d07b700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fca340a2c20 0x7fca340a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.901+0000 7fca3e7fc700 1 -- 172.21.15.104:0/582066926 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fca400ce4d0 con 0x7fca481066b0 2024-06-09T22:15:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.901+0000 7fca4d07b700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fca340a2c20 0x7fca340a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fca38006010 tx=0x7fca38000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:44.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:44.904+0000 7fca3e7fc700 1 -- 172.21.15.104:0/582066926 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fca4009b8d0 con 0x7fca481066b0 2024-06-09T22:15:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.099+0000 7fca4f2df700 1 -- 172.21.15.104:0/582066926 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca4819a070 con 0x7fca481066b0 2024-06-09T22:15:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.100+0000 7fca3e7fc700 1 -- 172.21.15.104:0/582066926 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18524 (secure 0 0 0) 0x7fca4009b1d0 con 0x7fca481066b0 2024-06-09T22:15:45.095 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:15:45.096 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":81,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:15:39.464742+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:15:45.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.102+0000 7fca4f2df700 1 -- 172.21.15.104:0/582066926 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fca340a2c20 msgr2=0x7fca340a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:45.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.102+0000 7fca4f2df700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fca340a2c20 0x7fca340a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fca38006010 tx=0x7fca38000bc0 comp rx=0 tx=0).stop 2024-06-09T22:15:45.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.102+0000 7fca4f2df700 1 -- 172.21.15.104:0/582066926 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca481066b0 msgr2=0x7fca4819d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:45.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.102+0000 7fca4f2df700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca481066b0 0x7fca4819d5e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fca4000b970 tx=0x7fca4000bd30 comp rx=0 tx=0).stop 2024-06-09T22:15:45.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.102+0000 7fca4f2df700 1 -- 172.21.15.104:0/582066926 shutdown_connections 2024-06-09T22:15:45.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.102+0000 7fca4f2df700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fca340a2c20 0x7fca340a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:45.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.102+0000 7fca4f2df700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca481066b0 0x7fca4819d5e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:45.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.102+0000 7fca4f2df700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca48101620 0x7fca48198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:45.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.102+0000 7fca4f2df700 1 --2- 172.21.15.104:0/582066926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca48101fd0 0x7fca48199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:45.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.102+0000 7fca4f2df700 1 -- 172.21.15.104:0/582066926 >> 172.21.15.104:0/582066926 conn(0x7fca48078360 msgr2=0x7fca48100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:45.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.102+0000 7fca4f2df700 1 -- 172.21.15.104:0/582066926 shutdown_connections 2024-06-09T22:15:45.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:45.102+0000 7fca4f2df700 1 -- 172.21.15.104:0/582066926 wait complete. 2024-06-09T22:15:45.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:45 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/582066926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:15:45.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:45 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/582066926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:15:45.803 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:45 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/582066926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:15:45.823 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-09T22:11:49.547377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-09T22:15:45.823 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-09T22:15:46.057 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:15:46.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:46 smithi153 ceph-mon[36599]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:46.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:46 smithi104 ceph-mon[32389]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:46.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:46 smithi194 ceph-mon[36856]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:47.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.700+0000 7f7f99334700 1 -- 172.21.15.104:0/1529689045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9406ab60 msgr2=0x7f7f9406afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.700+0000 7f7f99334700 1 --2- 172.21.15.104:0/1529689045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9406ab60 0x7f7f9406afe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0099c0 tx=0x7f7f7c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:15:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.700+0000 7f7f99334700 1 -- 172.21.15.104:0/1529689045 shutdown_connections 2024-06-09T22:15:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.700+0000 7f7f99334700 1 --2- 172.21.15.104:0/1529689045 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f9406b520 0x7f7f9410f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.700+0000 7f7f99334700 1 --2- 172.21.15.104:0/1529689045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f941075b0 0x7f7f94107990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.700+0000 7f7f99334700 1 --2- 172.21.15.104:0/1529689045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9406ab60 0x7f7f9406afe0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.700+0000 7f7f99334700 1 -- 172.21.15.104:0/1529689045 >> 172.21.15.104:0/1529689045 conn(0x7f7f94078340 msgr2=0x7f7f94078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.700+0000 7f7f99334700 1 -- 172.21.15.104:0/1529689045 shutdown_connections 2024-06-09T22:15:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.700+0000 7f7f99334700 1 -- 172.21.15.104:0/1529689045 wait complete. 2024-06-09T22:15:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f99334700 1 Processor -- start 2024-06-09T22:15:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f99334700 1 -- start start 2024-06-09T22:15:47.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f99334700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9406ab60 0x7f7f9419e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:47.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f99334700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9406b520 0x7f7f9419ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:47.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f99334700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f941075b0 0x7f7f94198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f99334700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7f94112760 con 0x7f7f9406b520 2024-06-09T22:15:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f99334700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941125e0 con 0x7f7f9406ab60 2024-06-09T22:15:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f99334700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941128e0 con 0x7f7f941075b0 2024-06-09T22:15:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f8a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9406b520 0x7f7f9419ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f8a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9406b520 0x7f7f9419ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58926/0 (socket says 172.21.15.104:58926) 2024-06-09T22:15:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f8a7fc700 1 -- 172.21.15.104:0/3089416609 learned_addr learned my addr 172.21.15.104:0/3089416609 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:15:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f92ffd700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9406ab60 0x7f7f9419e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f8a7fc700 1 -- 172.21.15.104:0/3089416609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f941075b0 msgr2=0x7f7f94198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.701+0000 7f7f937fe700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f941075b0 0x7f7f94198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.702+0000 7f7f8a7fc700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f941075b0 0x7f7f94198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.702+0000 7f7f8a7fc700 1 -- 172.21.15.104:0/3089416609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9406ab60 msgr2=0x7f7f9419e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.702+0000 7f7f8a7fc700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9406ab60 0x7f7f9419e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.702+0000 7f7f8a7fc700 1 -- 172.21.15.104:0/3089416609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f941991c0 con 0x7f7f9406b520 2024-06-09T22:15:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.702+0000 7f7f92ffd700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9406ab60 0x7f7f9419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:15:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.702+0000 7f7f937fe700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f941075b0 0x7f7f94198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:15:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.702+0000 7f7f8a7fc700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9406b520 0x7f7f9419ed90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009fd0 tx=0x7f7f7c003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.702+0000 7f7f90ff9700 1 -- 172.21.15.104:0/3089416609 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c027070 con 0x7f7f9406b520 2024-06-09T22:15:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.702+0000 7f7f90ff9700 1 -- 172.21.15.104:0/3089416609 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f7c004150 con 0x7f7f9406b520 2024-06-09T22:15:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.702+0000 7f7f99334700 1 -- 172.21.15.104:0/3089416609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f94199450 con 0x7f7f9406b520 2024-06-09T22:15:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.702+0000 7f7f90ff9700 1 -- 172.21.15.104:0/3089416609 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c00f460 con 0x7f7f9406b520 2024-06-09T22:15:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.702+0000 7f7f99334700 1 -- 172.21.15.104:0/3089416609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f941a5720 con 0x7f7f9406b520 2024-06-09T22:15:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.703+0000 7f7f99334700 1 -- 172.21.15.104:0/3089416609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f94103be0 con 0x7f7f9406b520 2024-06-09T22:15:47.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.704+0000 7f7f90ff9700 1 -- 172.21.15.104:0/3089416609 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7f7c02a070 con 0x7f7f9406b520 2024-06-09T22:15:47.700 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:47 smithi104 ceph-mon[32389]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:47.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.705+0000 7f7f90ff9700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f800a2c20 0x7f7f800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:47.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.705+0000 7f7f92ffd700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f800a2c20 0x7f7f800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:47.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.705+0000 7f7f90ff9700 1 -- 172.21.15.104:0/3089416609 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7f7c0d02d0 con 0x7f7f9406b520 2024-06-09T22:15:47.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.705+0000 7f7f92ffd700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f800a2c20 0x7f7f800a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7f84005fd0 tx=0x7f7f84005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:47.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.708+0000 7f7f90ff9700 1 -- 172.21.15.104:0/3089416609 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7f7c031070 con 0x7f7f9406b520 2024-06-09T22:15:47.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:47 smithi194 ceph-mon[36856]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:47.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.902+0000 7f7f99334700 1 -- 172.21.15.104:0/3089416609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7f9406a3f0 con 0x7f7f9406b520 2024-06-09T22:15:47.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.903+0000 7f7f90ff9700 1 -- 172.21.15.104:0/3089416609 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f7f7c013070 con 0x7f7f9406b520 2024-06-09T22:15:47.897 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-06-09T22:15:47.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.905+0000 7f7f99334700 1 -- 172.21.15.104:0/3089416609 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f800a2c20 msgr2=0x7f7f800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:47.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.905+0000 7f7f99334700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f800a2c20 0x7f7f800a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7f84005fd0 tx=0x7f7f84005d60 comp rx=0 tx=0).stop 2024-06-09T22:15:47.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.905+0000 7f7f99334700 1 -- 172.21.15.104:0/3089416609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9406b520 msgr2=0x7f7f9419ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:47.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.905+0000 7f7f99334700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9406b520 0x7f7f9419ed90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009fd0 tx=0x7f7f7c003730 comp rx=0 tx=0).stop 2024-06-09T22:15:47.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.905+0000 7f7f99334700 1 -- 172.21.15.104:0/3089416609 shutdown_connections 2024-06-09T22:15:47.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.905+0000 7f7f99334700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f800a2c20 0x7f7f800a50e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:47.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.905+0000 7f7f99334700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f941075b0 0x7f7f94198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:47.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.905+0000 7f7f99334700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9406ab60 0x7f7f9419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:47.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.905+0000 7f7f99334700 1 --2- 172.21.15.104:0/3089416609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9406b520 0x7f7f9419ed90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:47.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.905+0000 7f7f99334700 1 -- 172.21.15.104:0/3089416609 >> 172.21.15.104:0/3089416609 conn(0x7f7f94078340 msgr2=0x7f7f94100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:47.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.905+0000 7f7f99334700 1 -- 172.21.15.104:0/3089416609 shutdown_connections 2024-06-09T22:15:47.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:47.905+0000 7f7f99334700 1 -- 172.21.15.104:0/3089416609 wait complete. 2024-06-09T22:15:47.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:47 smithi153 ceph-mon[36599]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:48.683 INFO:tasks.cephadm:Adding mds.a on smithi104 2024-06-09T22:15:48.683 INFO:tasks.cephadm:Adding mds.d on smithi104 2024-06-09T22:15:48.683 INFO:tasks.cephadm:Adding mds.g on smithi104 2024-06-09T22:15:48.683 INFO:tasks.cephadm:Adding mds.j on smithi104 2024-06-09T22:15:48.683 INFO:tasks.cephadm:Adding mds.b on smithi153 2024-06-09T22:15:48.683 INFO:tasks.cephadm:Adding mds.e on smithi153 2024-06-09T22:15:48.683 INFO:tasks.cephadm:Adding mds.h on smithi153 2024-06-09T22:15:48.683 INFO:tasks.cephadm:Adding mds.k on smithi153 2024-06-09T22:15:48.683 INFO:tasks.cephadm:Adding mds.c on smithi194 2024-06-09T22:15:48.683 INFO:tasks.cephadm:Adding mds.f on smithi194 2024-06-09T22:15:48.684 INFO:tasks.cephadm:Adding mds.i on smithi194 2024-06-09T22:15:48.684 INFO:tasks.cephadm:Adding mds.l on smithi194 2024-06-09T22:15:48.684 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph orch apply mds all '12;smithi104=a;smithi104=d;smithi104=g;smithi104=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l' 2024-06-09T22:15:48.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:48 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3089416609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-09T22:15:48.930 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:15:48.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:48 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3089416609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-09T22:15:49.001 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:48 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3089416609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-09T22:15:49.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:49 smithi194 ceph-mon[36856]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:49.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:49 smithi153 ceph-mon[36599]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:50.001 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:49 smithi104 ceph-mon[32389]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:50.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.601+0000 7f9a661b0700 1 -- 172.21.15.194:0/3791449363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a60103b20 msgr2=0x7f9a60103f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:50.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.601+0000 7f9a661b0700 1 --2- 172.21.15.194:0/3791449363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a60103b20 0x7f9a60103f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000b0a0 tx=0x7f9a5000b3b0 comp rx=0 tx=0).stop 2024-06-09T22:15:50.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.601+0000 7f9a661b0700 1 -- 172.21.15.194:0/3791449363 shutdown_connections 2024-06-09T22:15:50.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.601+0000 7f9a661b0700 1 --2- 172.21.15.194:0/3791449363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a60100f60 0x7f9a6010ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:50.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.601+0000 7f9a661b0700 1 --2- 172.21.15.194:0/3791449363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a60103b20 0x7f9a60103f40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:50.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.601+0000 7f9a661b0700 1 --2- 172.21.15.194:0/3791449363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a601005a0 0x7f9a60100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:50.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.601+0000 7f9a661b0700 1 -- 172.21.15.194:0/3791449363 >> 172.21.15.194:0/3791449363 conn(0x7f9a600fc140 msgr2=0x7f9a600fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:50.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.601+0000 7f9a661b0700 1 -- 172.21.15.194:0/3791449363 shutdown_connections 2024-06-09T22:15:50.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.601+0000 7f9a661b0700 1 -- 172.21.15.194:0/3791449363 wait complete. 2024-06-09T22:15:50.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.601+0000 7f9a661b0700 1 Processor -- start 2024-06-09T22:15:50.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.601+0000 7f9a661b0700 1 -- start start 2024-06-09T22:15:50.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a661b0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a601005a0 0x7f9a60198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:50.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a661b0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a60100f60 0x7f9a60198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:50.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a661b0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a60103b20 0x7f9a6019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:50.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a661b0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6010e180 con 0x7f9a60103b20 2024-06-09T22:15:50.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a661b0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6010e000 con 0x7f9a601005a0 2024-06-09T22:15:50.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a661b0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6010e300 con 0x7f9a60100f60 2024-06-09T22:15:50.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a60100f60 0x7f9a60198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:50.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a60100f60 0x7f9a60198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44318/0 (socket says 172.21.15.194:44318) 2024-06-09T22:15:50.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5effd700 1 -- 172.21.15.194:0/4160307150 learned_addr learned my addr 172.21.15.194:0/4160307150 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5f7fe700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a601005a0 0x7f9a60198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5effd700 1 -- 172.21.15.194:0/4160307150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a601005a0 msgr2=0x7f9a60198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5ffff700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a60103b20 0x7f9a6019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5effd700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a601005a0 0x7f9a60198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5effd700 1 -- 172.21.15.194:0/4160307150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a60103b20 msgr2=0x7f9a6019fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5effd700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a60103b20 0x7f9a6019fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5effd700 1 -- 172.21.15.194:0/4160307150 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a601a0390 con 0x7f9a60100f60 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5ffff700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a60103b20 0x7f9a6019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5effd700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a60100f60 0x7f9a60198c70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400c9a0 tx=0x7f9a5400ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5cff9700 1 -- 172.21.15.194:0/4160307150 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a54004220 con 0x7f9a60100f60 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5cff9700 1 -- 172.21.15.194:0/4160307150 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a54007d40 con 0x7f9a60100f60 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a661b0700 1 -- 172.21.15.194:0/4160307150 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a601a0680 con 0x7f9a60100f60 2024-06-09T22:15:50.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a5cff9700 1 -- 172.21.15.194:0/4160307150 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a54003da0 con 0x7f9a60100f60 2024-06-09T22:15:50.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.602+0000 7f9a661b0700 1 -- 172.21.15.194:0/4160307150 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a601a0c90 con 0x7f9a60100f60 2024-06-09T22:15:50.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.603+0000 7f9a661b0700 1 -- 172.21.15.194:0/4160307150 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a60069180 con 0x7f9a60100f60 2024-06-09T22:15:50.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.604+0000 7f9a5cff9700 1 -- 172.21.15.194:0/4160307150 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9a540043c0 con 0x7f9a60100f60 2024-06-09T22:15:50.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.605+0000 7f9a5cff9700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9a480a2ca0 0x7f9a480a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:50.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.605+0000 7f9a5cff9700 1 -- 172.21.15.194:0/4160307150 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9a540ceaf0 con 0x7f9a60100f60 2024-06-09T22:15:50.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.605+0000 7f9a5f7fe700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9a480a2ca0 0x7f9a480a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:50.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.606+0000 7f9a5f7fe700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9a480a2ca0 0x7f9a480a5160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009040 tx=0x7f9a50006000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:50.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.609+0000 7f9a5cff9700 1 -- 172.21.15.194:0/4160307150 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9a54014030 con 0x7f9a60100f60 2024-06-09T22:15:50.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.812+0000 7f9a661b0700 1 -- 172.21.15.194:0/4160307150 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi104=a;smithi104=d;smithi104=g;smithi104=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}) -- 0x7f9a60062ad0 con 0x7f9a480a2ca0 2024-06-09T22:15:50.814 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.820+0000 7f9a5cff9700 1 -- 172.21.15.194:0/4160307150 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f9a60062ad0 con 0x7f9a480a2ca0 2024-06-09T22:15:50.815 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mds.all update... 2024-06-09T22:15:50.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.822+0000 7f9a661b0700 1 -- 172.21.15.194:0/4160307150 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9a480a2ca0 msgr2=0x7f9a480a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:50.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.822+0000 7f9a661b0700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9a480a2ca0 0x7f9a480a5160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009040 tx=0x7f9a50006000 comp rx=0 tx=0).stop 2024-06-09T22:15:50.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.822+0000 7f9a661b0700 1 -- 172.21.15.194:0/4160307150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a60100f60 msgr2=0x7f9a60198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:50.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.822+0000 7f9a661b0700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a60100f60 0x7f9a60198c70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400c9a0 tx=0x7f9a5400ccb0 comp rx=0 tx=0).stop 2024-06-09T22:15:50.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.822+0000 7f9a661b0700 1 -- 172.21.15.194:0/4160307150 shutdown_connections 2024-06-09T22:15:50.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.822+0000 7f9a661b0700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9a480a2ca0 0x7f9a480a5160 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:50.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.822+0000 7f9a661b0700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a60103b20 0x7f9a6019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:50.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.822+0000 7f9a661b0700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a601005a0 0x7f9a60198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:50.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.822+0000 7f9a661b0700 1 --2- 172.21.15.194:0/4160307150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a60100f60 0x7f9a60198c70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:50.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.822+0000 7f9a661b0700 1 -- 172.21.15.194:0/4160307150 >> 172.21.15.194:0/4160307150 conn(0x7f9a600fc140 msgr2=0x7f9a60106b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:50.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.823+0000 7f9a661b0700 1 -- 172.21.15.194:0/4160307150 shutdown_connections 2024-06-09T22:15:50.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-06-09T22:15:50.823+0000 7f9a661b0700 1 -- 172.21.15.194:0/4160307150 wait complete. 2024-06-09T22:15:51.558 DEBUG:teuthology.orchestra.run.smithi104:mds.a> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.a.service 2024-06-09T22:15:51.561 DEBUG:teuthology.orchestra.run.smithi104:mds.d> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.d.service 2024-06-09T22:15:51.604 DEBUG:teuthology.orchestra.run.smithi104:mds.g> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.g.service 2024-06-09T22:15:51.646 DEBUG:teuthology.orchestra.run.smithi104:mds.j> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.j.service 2024-06-09T22:15:51.689 DEBUG:teuthology.orchestra.run.smithi153:mds.b> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.b.service 2024-06-09T22:15:51.692 DEBUG:teuthology.orchestra.run.smithi153:mds.e> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.e.service 2024-06-09T22:15:51.735 DEBUG:teuthology.orchestra.run.smithi153:mds.h> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.h.service 2024-06-09T22:15:51.780 DEBUG:teuthology.orchestra.run.smithi153:mds.k> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.k.service 2024-06-09T22:15:51.823 DEBUG:teuthology.orchestra.run.smithi194:mds.c> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.c.service 2024-06-09T22:15:51.825 DEBUG:teuthology.orchestra.run.smithi194:mds.f> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.f.service 2024-06-09T22:15:51.868 DEBUG:teuthology.orchestra.run.smithi194:mds.i> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.i.service 2024-06-09T22:15:51.911 DEBUG:teuthology.orchestra.run.smithi194:mds.l> sudo journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.l.service 2024-06-09T22:15:51.954 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-06-09T22:15:51.954 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:15:51.978 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:51 smithi104 ceph-mon[32389]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi104=a;smithi104=d;smithi104=g;smithi104=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:15:51.978 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:51 smithi104 ceph-mon[32389]: Saving service mds.all spec with placement smithi104=a;smithi104=d;smithi104=g;smithi104=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-06-09T22:15:51.978 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:51.978 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:51.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:51.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:51.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:51.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:15:51.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:15:51.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:51 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:51.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:51 smithi104 ceph-mon[32389]: Deploying daemon mds.l on smithi194 2024-06-09T22:15:51.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:51 smithi104 ceph-mon[32389]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:51.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:51 smithi153 ceph-mon[36599]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi104=a;smithi104=d;smithi104=g;smithi104=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:15:51.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:51 smithi153 ceph-mon[36599]: Saving service mds.all spec with placement smithi104=a;smithi104=d;smithi104=g;smithi104=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-06-09T22:15:51.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:51.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:51.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:51.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:51.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:51.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:15:51.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:15:51.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:51 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:51.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:51 smithi153 ceph-mon[36599]: Deploying daemon mds.l on smithi194 2024-06-09T22:15:51.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:51 smithi153 ceph-mon[36599]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:52.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:51 smithi194 ceph-mon[36856]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi104=a;smithi104=d;smithi104=g;smithi104=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:15:52.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:51 smithi194 ceph-mon[36856]: Saving service mds.all spec with placement smithi104=a;smithi104=d;smithi104=g;smithi104=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-06-09T22:15:52.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:52.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:15:52.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:52.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:15:52.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:52.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:15:52.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:15:52.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:51 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:52.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:51 smithi194 ceph-mon[36856]: Deploying daemon mds.l on smithi194 2024-06-09T22:15:52.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:51 smithi194 ceph-mon[36856]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:52.192 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:15:53.554 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:15:53 smithi194 systemd[1]: Starting Ceph mds.l for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:15:53.863 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:15:53 smithi194 podman[66766]: 2024-06-09 22:15:53.490518267 +0000 UTC m=+0.017108170 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:15:53.863 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:15:53 smithi194 podman[66766]: 2024-06-09 22:15:53.615923193 +0000 UTC m=+0.142513101 container create 9373f7a032cae5b48d733c7fb6419c1af3e10da484169c4a206f9e61a07945ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-l, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:15:53.863 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:15:53 smithi194 podman[66766]: 2024-06-09 22:15:53.868938753 +0000 UTC m=+0.395528649 container init 9373f7a032cae5b48d733c7fb6419c1af3e10da484169c4a206f9e61a07945ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-l, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:15:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.897+0000 7fc028de1700 1 -- 172.21.15.104:0/2116850754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc024102030 msgr2=0x7fc024105ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.897+0000 7fc028de1700 1 --2- 172.21.15.104:0/2116850754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc024102030 0x7fc024105ff0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc0180099c0 tx=0x7fc018009cd0 comp rx=0 tx=0).stop 2024-06-09T22:15:53.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.897+0000 7fc028de1700 1 -- 172.21.15.104:0/2116850754 shutdown_connections 2024-06-09T22:15:53.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.897+0000 7fc028de1700 1 --2- 172.21.15.104:0/2116850754 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0241066d0 0x7fc02410e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:53.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.897+0000 7fc028de1700 1 --2- 172.21.15.104:0/2116850754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc024101680 0x7fc024101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:53.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.897+0000 7fc028de1700 1 --2- 172.21.15.104:0/2116850754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc024102030 0x7fc024105ff0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:53.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.897+0000 7fc028de1700 1 -- 172.21.15.104:0/2116850754 >> 172.21.15.104:0/2116850754 conn(0x7fc0240773e0 msgr2=0x7fc0240777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:53.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.897+0000 7fc028de1700 1 -- 172.21.15.104:0/2116850754 shutdown_connections 2024-06-09T22:15:53.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.897+0000 7fc028de1700 1 -- 172.21.15.104:0/2116850754 wait complete. 2024-06-09T22:15:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.898+0000 7fc028de1700 1 Processor -- start 2024-06-09T22:15:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.898+0000 7fc028de1700 1 -- start start 2024-06-09T22:15:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc028de1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc024101680 0x7fc02419e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc028de1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc024102030 0x7fc02419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc028de1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0241066d0 0x7fc024198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc028de1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc024111690 con 0x7fc024102030 2024-06-09T22:15:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc028de1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc024111510 con 0x7fc0241066d0 2024-06-09T22:15:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc028de1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc024111810 con 0x7fc024101680 2024-06-09T22:15:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc021d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc024102030 0x7fc02419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc02259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc024101680 0x7fc02419e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc021d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc024102030 0x7fc02419edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58940/0 (socket says 172.21.15.104:58940) 2024-06-09T22:15:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc021d9b700 1 -- 172.21.15.104:0/3181147177 learned_addr learned my addr 172.21.15.104:0/3181147177 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:15:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc022d9d700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0241066d0 0x7fc024198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc02259c700 1 -- 172.21.15.104:0/3181147177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0241066d0 msgr2=0x7fc024198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc02259c700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0241066d0 0x7fc024198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc02259c700 1 -- 172.21.15.104:0/3181147177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc024102030 msgr2=0x7fc02419edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc02259c700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc024102030 0x7fc02419edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc02259c700 1 -- 172.21.15.104:0/3181147177 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0241991d0 con 0x7fc024101680 2024-06-09T22:15:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc021d9b700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc024102030 0x7fc02419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:15:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc02259c700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc024101680 0x7fc02419e8b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc01400d950 tx=0x7fc01400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc0137fe700 1 -- 172.21.15.104:0/3181147177 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014016070 con 0x7fc024101680 2024-06-09T22:15:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.899+0000 7fc028de1700 1 -- 172.21.15.104:0/3181147177 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0241994c0 con 0x7fc024101680 2024-06-09T22:15:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.900+0000 7fc028de1700 1 -- 172.21.15.104:0/3181147177 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0241a5780 con 0x7fc024101680 2024-06-09T22:15:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.900+0000 7fc028de1700 1 -- 172.21.15.104:0/3181147177 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc024069180 con 0x7fc024101680 2024-06-09T22:15:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.902+0000 7fc0137fe700 1 -- 172.21.15.104:0/3181147177 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc01400fd70 con 0x7fc024101680 2024-06-09T22:15:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.902+0000 7fc0137fe700 1 -- 172.21.15.104:0/3181147177 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014020060 con 0x7fc024101680 2024-06-09T22:15:53.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.904+0000 7fc0137fe700 1 -- 172.21.15.104:0/3181147177 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc014019070 con 0x7fc024101680 2024-06-09T22:15:53.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.905+0000 7fc0137fe700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc00c0a2c20 0x7fc00c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:53.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.905+0000 7fc021d9b700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc00c0a2c20 0x7fc00c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:53.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.905+0000 7fc0137fe700 1 -- 172.21.15.104:0/3181147177 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc0140cea90 con 0x7fc024101680 2024-06-09T22:15:53.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.905+0000 7fc021d9b700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc00c0a2c20 0x7fc00c0a50e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc024107cc0 tx=0x7fc018005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:53.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:53.909+0000 7fc0137fe700 1 -- 172.21.15.104:0/3181147177 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc01409bdd0 con 0x7fc024101680 2024-06-09T22:15:54.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.156+0000 7fc028de1700 1 -- 172.21.15.104:0/3181147177 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc02402dca0 con 0x7fc024101680 2024-06-09T22:15:54.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.156+0000 7fc0137fe700 1 -- 172.21.15.104:0/3181147177 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fc0140860e0 con 0x7fc024101680 2024-06-09T22:15:54.151 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:15:54.151 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-06-09T22:15:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.158+0000 7fc028de1700 1 -- 172.21.15.104:0/3181147177 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc00c0a2c20 msgr2=0x7fc00c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.158+0000 7fc028de1700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc00c0a2c20 0x7fc00c0a50e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc024107cc0 tx=0x7fc018005fb0 comp rx=0 tx=0).stop 2024-06-09T22:15:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.158+0000 7fc028de1700 1 -- 172.21.15.104:0/3181147177 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc024101680 msgr2=0x7fc02419e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.158+0000 7fc028de1700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc024101680 0x7fc02419e8b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc01400d950 tx=0x7fc01400dc60 comp rx=0 tx=0).stop 2024-06-09T22:15:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.159+0000 7fc028de1700 1 -- 172.21.15.104:0/3181147177 shutdown_connections 2024-06-09T22:15:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.159+0000 7fc028de1700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc00c0a2c20 0x7fc00c0a50e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.159+0000 7fc028de1700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0241066d0 0x7fc024198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.159+0000 7fc028de1700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc024101680 0x7fc02419e8b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.159+0000 7fc028de1700 1 --2- 172.21.15.104:0/3181147177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc024102030 0x7fc02419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.159+0000 7fc028de1700 1 -- 172.21.15.104:0/3181147177 >> 172.21.15.104:0/3181147177 conn(0x7fc0240773e0 msgr2=0x7fc02406aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.159+0000 7fc028de1700 1 -- 172.21.15.104:0/3181147177 shutdown_connections 2024-06-09T22:15:54.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:54.159+0000 7fc028de1700 1 -- 172.21.15.104:0/3181147177 wait complete. 2024-06-09T22:15:54.155 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 2 2024-06-09T22:15:54.304 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:15:53 smithi194 podman[66766]: 2024-06-09 22:15:53.872151718 +0000 UTC m=+0.398741604 container start 9373f7a032cae5b48d733c7fb6419c1af3e10da484169c4a206f9e61a07945ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:15:54.304 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:15:53 smithi194 ceph-mds[66785]: -- 172.21.15.194:0/3648938299 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5623ce51da40 con 0x5623ce4f0800 2024-06-09T22:15:54.304 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:15:53 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-l[66781]: starting mds.l at 2024-06-09T22:15:54.304 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:15:53 smithi194 bash[66766]: 9373f7a032cae5b48d733c7fb6419c1af3e10da484169c4a206f9e61a07945ef 2024-06-09T22:15:54.304 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:15:53 smithi194 systemd[1]: Started Ceph mds.l for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:15:54.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:54 smithi104 ceph-mon[32389]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:54.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:54 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] up:boot 2024-06-09T22:15:54.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:54 smithi104 ceph-mon[32389]: fsmap 1 up:standby 2024-06-09T22:15:54.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-09T22:15:54.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:54.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:54.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:54.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:15:54.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:15:54.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:54 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:54.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:54 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3181147177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:15:54.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:54 smithi153 ceph-mon[36599]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:54.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:54 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] up:boot 2024-06-09T22:15:54.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:54 smithi153 ceph-mon[36599]: fsmap 1 up:standby 2024-06-09T22:15:54.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-09T22:15:54.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:54.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:54.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:54.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:15:54.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:15:54.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:54 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:54.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:54 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3181147177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:15:54.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:54 smithi194 ceph-mon[36856]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:54.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:54 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] up:boot 2024-06-09T22:15:54.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:54 smithi194 ceph-mon[36856]: fsmap 1 up:standby 2024-06-09T22:15:54.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-09T22:15:54.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:54.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:54.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:54.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:15:54.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:15:54.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:54 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:54.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:54 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3181147177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:15:55.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:55 smithi104 ceph-mon[32389]: Deploying daemon mds.k on smithi153 2024-06-09T22:15:55.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:55 smithi153 ceph-mon[36599]: Deploying daemon mds.k on smithi153 2024-06-09T22:15:55.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:55 smithi194 ceph-mon[36856]: Deploying daemon mds.k on smithi153 2024-06-09T22:15:56.238 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:15:56 smithi153 systemd[1]: Starting Ceph mds.k for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:15:56.604 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:56 smithi153 ceph-mon[36599]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:56.604 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:15:56 smithi153 podman[65783]: 2024-06-09 22:15:56.290159728 +0000 UTC m=+0.017658176 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:15:56.604 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:15:56 smithi153 podman[65783]: 2024-06-09 22:15:56.399217516 +0000 UTC m=+0.126715951 container create d8e017896f09ecbc0d605ec8dc16da9e3cd702c32cb38c94940624861c03229c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-09T22:15:56.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:56 smithi104 ceph-mon[32389]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:56.790 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:15:56.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:56 smithi194 ceph-mon[36856]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:56.988 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:15:56 smithi153 podman[65783]: 2024-06-09 22:15:56.610169145 +0000 UTC m=+0.337667585 container init d8e017896f09ecbc0d605ec8dc16da9e3cd702c32cb38c94940624861c03229c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-06-09T22:15:56.988 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:15:56 smithi153 podman[65783]: 2024-06-09 22:15:56.613353615 +0000 UTC m=+0.340852056 container start d8e017896f09ecbc0d605ec8dc16da9e3cd702c32cb38c94940624861c03229c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-06-09T22:15:56.988 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:15:56 smithi153 ceph-mds[65802]: -- 172.21.15.153:0/1045576449 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560e73173a40 con 0x560e73146800 2024-06-09T22:15:56.988 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:15:56 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-k[65798]: starting mds.k at 2024-06-09T22:15:56.988 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:15:56 smithi153 bash[65783]: d8e017896f09ecbc0d605ec8dc16da9e3cd702c32cb38c94940624861c03229c 2024-06-09T22:15:56.988 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:15:56 smithi153 systemd[1]: Started Ceph mds.k for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:15:57.049 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:15:57.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:57.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:57.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:57.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:15:57.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:15:57.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:57.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:57 smithi104 ceph-mon[32389]: Deploying daemon mds.j on smithi104 2024-06-09T22:15:57.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:57 smithi104 ceph-mon[32389]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:57.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:57 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.153:6832/3968887191,v1:172.21.15.153:6833/3968887191] up:boot 2024-06-09T22:15:57.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:57 smithi104 ceph-mon[32389]: fsmap 2 up:standby 2024-06-09T22:15:57.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:57 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-09T22:15:57.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:57.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:57.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:57.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:15:57.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:15:57.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:57.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:57 smithi153 ceph-mon[36599]: Deploying daemon mds.j on smithi104 2024-06-09T22:15:57.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:57 smithi153 ceph-mon[36599]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:57.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:57 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.153:6832/3968887191,v1:172.21.15.153:6833/3968887191] up:boot 2024-06-09T22:15:57.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:57 smithi153 ceph-mon[36599]: fsmap 2 up:standby 2024-06-09T22:15:57.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:57 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-09T22:15:58.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:58.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:58.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:15:58.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:15:58.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:15:58.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:15:58.054 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:57 smithi194 ceph-mon[36856]: Deploying daemon mds.j on smithi104 2024-06-09T22:15:58.055 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:57 smithi194 ceph-mon[36856]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:15:58.055 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:57 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.153:6832/3968887191,v1:172.21.15.153:6833/3968887191] up:boot 2024-06-09T22:15:58.055 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:57 smithi194 ceph-mon[36856]: fsmap 2 up:standby 2024-06-09T22:15:58.055 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:57 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-09T22:15:58.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.807+0000 7f9c58edf700 1 -- 172.21.15.104:0/555484167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c54101370 msgr2=0x7f9c541017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:58.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.807+0000 7f9c58edf700 1 --2- 172.21.15.104:0/555484167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c54101370 0x7f9c541017f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9c48009990 tx=0x7f9c48009ca0 comp rx=0 tx=0).stop 2024-06-09T22:15:58.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.808+0000 7f9c58edf700 1 -- 172.21.15.104:0/555484167 shutdown_connections 2024-06-09T22:15:58.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.808+0000 7f9c58edf700 1 --2- 172.21.15.104:0/555484167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c54101d30 0x7f9c5410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:58.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.808+0000 7f9c58edf700 1 --2- 172.21.15.104:0/555484167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c54107420 0x7f9c54107800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:58.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.808+0000 7f9c58edf700 1 --2- 172.21.15.104:0/555484167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c54101370 0x7f9c541017f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:58.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.808+0000 7f9c58edf700 1 -- 172.21.15.104:0/555484167 >> 172.21.15.104:0/555484167 conn(0x7f9c54078340 msgr2=0x7f9c54078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:58.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.808+0000 7f9c58edf700 1 -- 172.21.15.104:0/555484167 shutdown_connections 2024-06-09T22:15:58.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.808+0000 7f9c58edf700 1 -- 172.21.15.104:0/555484167 wait complete. 2024-06-09T22:15:58.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c58edf700 1 Processor -- start 2024-06-09T22:15:58.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c58edf700 1 -- start start 2024-06-09T22:15:58.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c58edf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c54101370 0x7f9c5419d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:58.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c58edf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c54101d30 0x7f9c5419dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c58edf700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c54107420 0x7f9c54197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c58edf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9c54111630 con 0x7f9c54101d30 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c58edf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9c541114b0 con 0x7f9c54101370 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c58edf700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9c541117b0 con 0x7f9c54107420 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c51d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c54101d30 0x7f9c5419dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c51d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c54101d30 0x7f9c5419dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36546/0 (socket says 172.21.15.104:36546) 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c51d9b700 1 -- 172.21.15.104:0/28466156 learned_addr learned my addr 172.21.15.104:0/28466156 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c5259c700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c54101370 0x7f9c5419d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c51d9b700 1 -- 172.21.15.104:0/28466156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c54107420 msgr2=0x7f9c54197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c51d9b700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c54107420 0x7f9c54197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c51d9b700 1 -- 172.21.15.104:0/28466156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c54101370 msgr2=0x7f9c5419d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c51d9b700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c54101370 0x7f9c5419d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.809+0000 7f9c51d9b700 1 -- 172.21.15.104:0/28466156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c54198090 con 0x7f9c54101d30 2024-06-09T22:15:58.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.810+0000 7f9c51d9b700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c54101d30 0x7f9c5419dcb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9c48000c00 tx=0x7f9c480047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:58.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.810+0000 7f9c437fe700 1 -- 172.21.15.104:0/28466156 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c48027070 con 0x7f9c54101d30 2024-06-09T22:15:58.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.810+0000 7f9c58edf700 1 -- 172.21.15.104:0/28466156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c54198320 con 0x7f9c54101d30 2024-06-09T22:15:58.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.810+0000 7f9c437fe700 1 -- 172.21.15.104:0/28466156 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c48004050 con 0x7f9c54101d30 2024-06-09T22:15:58.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.810+0000 7f9c437fe700 1 -- 172.21.15.104:0/28466156 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4800f460 con 0x7f9c54101d30 2024-06-09T22:15:58.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.810+0000 7f9c58edf700 1 -- 172.21.15.104:0/28466156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c541a4640 con 0x7f9c54101d30 2024-06-09T22:15:58.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.811+0000 7f9c58edf700 1 -- 172.21.15.104:0/28466156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c54069180 con 0x7f9c54101d30 2024-06-09T22:15:58.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.812+0000 7f9c437fe700 1 -- 172.21.15.104:0/28466156 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9c4802a070 con 0x7f9c54101d30 2024-06-09T22:15:58.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.812+0000 7f9c437fe700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9c3c0a2c20 0x7f9c3c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:15:58.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.813+0000 7f9c5259c700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9c3c0a2c20 0x7f9c3c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:15:58.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.813+0000 7f9c437fe700 1 -- 172.21.15.104:0/28466156 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9c480cff70 con 0x7f9c54101d30 2024-06-09T22:15:58.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.813+0000 7f9c5259c700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9c3c0a2c20 0x7f9c3c0a50e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c54198a70 tx=0x7f9c4400b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:15:58.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:58.816+0000 7f9c437fe700 1 -- 172.21.15.104:0/28466156 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9c48031070 con 0x7f9c54101d30 2024-06-09T22:15:59.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.058+0000 7f9c58edf700 1 -- 172.21.15.104:0/28466156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c5402dca0 con 0x7f9c54101d30 2024-06-09T22:15:59.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.059+0000 7f9c437fe700 1 -- 172.21.15.104:0/28466156 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f9c48013070 con 0x7f9c54101d30 2024-06-09T22:15:59.055 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:15:59.055 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24362,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-06-09T22:15:59.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.061+0000 7f9c58edf700 1 -- 172.21.15.104:0/28466156 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9c3c0a2c20 msgr2=0x7f9c3c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:59.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.061+0000 7f9c58edf700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9c3c0a2c20 0x7f9c3c0a50e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c54198a70 tx=0x7f9c4400b5b0 comp rx=0 tx=0).stop 2024-06-09T22:15:59.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.061+0000 7f9c58edf700 1 -- 172.21.15.104:0/28466156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c54101d30 msgr2=0x7f9c5419dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:15:59.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.061+0000 7f9c58edf700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c54101d30 0x7f9c5419dcb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9c48000c00 tx=0x7f9c480047b0 comp rx=0 tx=0).stop 2024-06-09T22:15:59.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.061+0000 7f9c58edf700 1 -- 172.21.15.104:0/28466156 shutdown_connections 2024-06-09T22:15:59.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.061+0000 7f9c58edf700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9c3c0a2c20 0x7f9c3c0a50e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:59.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.061+0000 7f9c58edf700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c54107420 0x7f9c54197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:59.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.061+0000 7f9c58edf700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c54101370 0x7f9c5419d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:59.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.061+0000 7f9c58edf700 1 --2- 172.21.15.104:0/28466156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c54101d30 0x7f9c5419dcb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:15:59.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.061+0000 7f9c58edf700 1 -- 172.21.15.104:0/28466156 >> 172.21.15.104:0/28466156 conn(0x7f9c54078340 msgr2=0x7f9c54100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:15:59.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.062+0000 7f9c58edf700 1 -- 172.21.15.104:0/28466156 shutdown_connections 2024-06-09T22:15:59.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:15:59.062+0000 7f9c58edf700 1 -- 172.21.15.104:0/28466156 wait complete. 2024-06-09T22:15:59.058 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 3 2024-06-09T22:15:59.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:15:59 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/28466156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:15:59.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:15:59 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/28466156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:15:59.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:15:59 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/28466156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:00.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:00 smithi153 ceph-mon[36599]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:00.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:00 smithi104 ceph-mon[32389]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:00.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:00 smithi194 ceph-mon[36856]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:01.252 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:16:01 smithi104 systemd[1]: Starting Ceph mds.j for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:16:01.586 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:16:01 smithi104 podman[70263]: 2024-06-09 22:16:01.228296169 +0000 UTC m=+0.017401030 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:16:01.586 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:16:01 smithi104 podman[70263]: 2024-06-09 22:16:01.337825914 +0000 UTC m=+0.126930755 container create b1baca679f74bd6cf4f8d00dcb32fa91f079e70107ef1a1603f1fbd34510d72d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-j, ceph=True, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc) 2024-06-09T22:16:01.587 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:16:01 smithi104 podman[70263]: 2024-06-09 22:16:01.565923714 +0000 UTC m=+0.355028555 container init b1baca679f74bd6cf4f8d00dcb32fa91f079e70107ef1a1603f1fbd34510d72d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-j, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:16:01.587 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:16:01 smithi104 podman[70263]: 2024-06-09 22:16:01.568685297 +0000 UTC m=+0.357790139 container start b1baca679f74bd6cf4f8d00dcb32fa91f079e70107ef1a1603f1fbd34510d72d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-j, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:16:02.002 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:16:01 smithi104 ceph-mds[70282]: -- 172.21.15.104:0/2812155904 <== mon.0 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560d6a147a40 con 0x560d6a11ac00 2024-06-09T22:16:02.002 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:16:01 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-j[70278]: starting mds.j at 2024-06-09T22:16:02.002 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:16:01 smithi104 bash[70263]: b1baca679f74bd6cf4f8d00dcb32fa91f079e70107ef1a1603f1fbd34510d72d 2024-06-09T22:16:02.002 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:16:01 smithi104 systemd[1]: Started Ceph mds.j for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:16:02.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:16:02.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:02 smithi104 ceph-mon[32389]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:02.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:02.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:02.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:02.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:02.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:02.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:02 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:02.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:02 smithi194 ceph-mon[36856]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:02.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:02.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:02.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:02.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:02.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:02.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:02 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:02.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:02 smithi153 ceph-mon[36599]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:02.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:02.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:02.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:02.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:02.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:02.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:02 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:03.519 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:03.643 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:03 smithi104 ceph-mon[32389]: Deploying daemon mds.d on smithi104 2024-06-09T22:16:03.643 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:03 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.104:6834/3517082076,v1:172.21.15.104:6835/3517082076] up:boot 2024-06-09T22:16:03.644 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:03 smithi104 ceph-mon[32389]: fsmap 3 up:standby 2024-06-09T22:16:03.644 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:03 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-09T22:16:03.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:03 smithi153 ceph-mon[36599]: Deploying daemon mds.d on smithi104 2024-06-09T22:16:03.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:03 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.104:6834/3517082076,v1:172.21.15.104:6835/3517082076] up:boot 2024-06-09T22:16:03.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:03 smithi153 ceph-mon[36599]: fsmap 3 up:standby 2024-06-09T22:16:03.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:03 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-09T22:16:03.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:03 smithi194 ceph-mon[36856]: Deploying daemon mds.d on smithi104 2024-06-09T22:16:03.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:03 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.104:6834/3517082076,v1:172.21.15.104:6835/3517082076] up:boot 2024-06-09T22:16:03.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:03 smithi194 ceph-mon[36856]: fsmap 3 up:standby 2024-06-09T22:16:03.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:03 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-09T22:16:04.639 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:04 smithi153 ceph-mon[36599]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:04.653 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:04 smithi104 ceph-mon[32389]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:04.706 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:04 smithi194 ceph-mon[36856]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:05.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.180+0000 7f2048e19700 1 -- 172.21.15.104:0/2333888850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2044101df0 msgr2=0x7f20441021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.180+0000 7f2048e19700 1 --2- 172.21.15.104:0/2333888850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2044101df0 0x7f20441021d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f20340099c0 tx=0x7f2034009cd0 comp rx=0 tx=0).stop 2024-06-09T22:16:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.181+0000 7f2048e19700 1 -- 172.21.15.104:0/2333888850 shutdown_connections 2024-06-09T22:16:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.181+0000 7f2048e19700 1 --2- 172.21.15.104:0/2333888850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2044101df0 0x7f20441021d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.181+0000 7f2048e19700 1 --2- 172.21.15.104:0/2333888850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2044106750 0x7f204410e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.181+0000 7f2048e19700 1 --2- 172.21.15.104:0/2333888850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20441027a0 0x7f2044106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.181+0000 7f2048e19700 1 -- 172.21.15.104:0/2333888850 >> 172.21.15.104:0/2333888850 conn(0x7f2044077a00 msgr2=0x7f20440fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.181+0000 7f2048e19700 1 -- 172.21.15.104:0/2333888850 shutdown_connections 2024-06-09T22:16:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.181+0000 7f2048e19700 1 -- 172.21.15.104:0/2333888850 wait complete. 2024-06-09T22:16:05.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f2048e19700 1 Processor -- start 2024-06-09T22:16:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f2048e19700 1 -- start start 2024-06-09T22:16:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f2048e19700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2044101df0 0x7f2044199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f2048e19700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20441027a0 0x7f2044199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f2048e19700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2044106750 0x7f20441a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f2048e19700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2044111670 con 0x7f2044106750 2024-06-09T22:16:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f2048e19700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f20441114f0 con 0x7f2044101df0 2024-06-09T22:16:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f2048e19700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f20441117f0 con 0x7f20441027a0 2024-06-09T22:16:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f2041d9b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20441027a0 0x7f2044199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f204259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2044101df0 0x7f2044199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f2041d9b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20441027a0 0x7f2044199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:59488/0 (socket says 172.21.15.104:59488) 2024-06-09T22:16:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f2041d9b700 1 -- 172.21.15.104:0/2442903842 learned_addr learned my addr 172.21.15.104:0/2442903842 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.182+0000 7f2042d9d700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2044106750 0x7f20441a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.183+0000 7f2041d9b700 1 -- 172.21.15.104:0/2442903842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2044101df0 msgr2=0x7f2044199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.183+0000 7f2041d9b700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2044101df0 0x7f2044199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.183+0000 7f2041d9b700 1 -- 172.21.15.104:0/2442903842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2044106750 msgr2=0x7f20441a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.183+0000 7f2041d9b700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2044106750 0x7f20441a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.183+0000 7f2041d9b700 1 -- 172.21.15.104:0/2442903842 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20441a4fd0 con 0x7f20441027a0 2024-06-09T22:16:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.183+0000 7f2041d9b700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20441027a0 0x7f2044199f50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f203800c920 tx=0x7f203800cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.183+0000 7f20337fe700 1 -- 172.21.15.104:0/2442903842 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038004a30 con 0x7f20441027a0 2024-06-09T22:16:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.183+0000 7f2048e19700 1 -- 172.21.15.104:0/2442903842 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20441a51a0 con 0x7f20441027a0 2024-06-09T22:16:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.183+0000 7f20337fe700 1 -- 172.21.15.104:0/2442903842 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2038005500 con 0x7f20441027a0 2024-06-09T22:16:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.183+0000 7f20337fe700 1 -- 172.21.15.104:0/2442903842 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038007d00 con 0x7f20441027a0 2024-06-09T22:16:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.183+0000 7f2048e19700 1 -- 172.21.15.104:0/2442903842 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20441a5730 con 0x7f20441027a0 2024-06-09T22:16:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.184+0000 7f2048e19700 1 -- 172.21.15.104:0/2442903842 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2044109990 con 0x7f20441027a0 2024-06-09T22:16:05.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.185+0000 7f20337fe700 1 -- 172.21.15.104:0/2442903842 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f2038005020 con 0x7f20441027a0 2024-06-09T22:16:05.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.186+0000 7f20337fe700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f202c0a2ca0 0x7f202c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:05.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.186+0000 7f20337fe700 1 -- 172.21.15.104:0/2442903842 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f20380ce6f0 con 0x7f20441027a0 2024-06-09T22:16:05.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.186+0000 7f204259c700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f202c0a2ca0 0x7f202c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:05.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.186+0000 7f204259c700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f202c0a2ca0 0x7f202c0a5160 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2034000c00 tx=0x7f2034005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:05.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.190+0000 7f20337fe700 1 -- 172.21.15.104:0/2442903842 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2038014030 con 0x7f20441027a0 2024-06-09T22:16:05.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.441+0000 7f2048e19700 1 -- 172.21.15.104:0/2442903842 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2044069180 con 0x7f20441027a0 2024-06-09T22:16:05.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.442+0000 7f20337fe700 1 -- 172.21.15.104:0/2442903842 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3052 (secure 0 0 0) 0x7f203809ba50 con 0x7f20441027a0 2024-06-09T22:16:05.438 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:16:05.438 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24362,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-06-09T22:16:05.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.444+0000 7f2048e19700 1 -- 172.21.15.104:0/2442903842 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f202c0a2ca0 msgr2=0x7f202c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:05.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.444+0000 7f2048e19700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f202c0a2ca0 0x7f202c0a5160 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2034000c00 tx=0x7f2034005fb0 comp rx=0 tx=0).stop 2024-06-09T22:16:05.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.444+0000 7f2048e19700 1 -- 172.21.15.104:0/2442903842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20441027a0 msgr2=0x7f2044199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:05.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.444+0000 7f2048e19700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20441027a0 0x7f2044199f50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f203800c920 tx=0x7f203800cc30 comp rx=0 tx=0).stop 2024-06-09T22:16:05.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.444+0000 7f2048e19700 1 -- 172.21.15.104:0/2442903842 shutdown_connections 2024-06-09T22:16:05.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.444+0000 7f2048e19700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f202c0a2ca0 0x7f202c0a5160 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:05.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.444+0000 7f2048e19700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2044106750 0x7f20441a4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:05.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.444+0000 7f2048e19700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2044101df0 0x7f2044199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:05.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.444+0000 7f2048e19700 1 --2- 172.21.15.104:0/2442903842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20441027a0 0x7f2044199f50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:05.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.444+0000 7f2048e19700 1 -- 172.21.15.104:0/2442903842 >> 172.21.15.104:0/2442903842 conn(0x7f2044077a00 msgr2=0x7f20440fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:05.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.444+0000 7f2048e19700 1 -- 172.21.15.104:0/2442903842 shutdown_connections 2024-06-09T22:16:05.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:05.444+0000 7f2048e19700 1 -- 172.21.15.104:0/2442903842 wait complete. 2024-06-09T22:16:05.441 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 4 2024-06-09T22:16:06.683 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:06 smithi153 ceph-mon[36599]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:06.683 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:06 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2442903842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:06.683 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:06.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:06.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:06.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:06.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:06.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:06 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:06.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:06 smithi104 ceph-mon[32389]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:06.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:06 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2442903842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:06.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:06.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:06.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:06.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:06.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:06.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:06 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:06.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:06 smithi194 ceph-mon[36856]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:06.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:06 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2442903842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:06.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:06.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:06.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:06.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:06.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:06.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:06 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:07.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:07 smithi153 ceph-mon[36599]: Deploying daemon mds.e on smithi153 2024-06-09T22:16:07.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:07 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.104:6836/3060956100,v1:172.21.15.104:6837/3060956100] up:boot 2024-06-09T22:16:07.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:07 smithi153 ceph-mon[36599]: fsmap 4 up:standby 2024-06-09T22:16:07.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:07 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-09T22:16:07.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:07 smithi104 ceph-mon[32389]: Deploying daemon mds.e on smithi153 2024-06-09T22:16:07.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:07 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.104:6836/3060956100,v1:172.21.15.104:6837/3060956100] up:boot 2024-06-09T22:16:07.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:07 smithi104 ceph-mon[32389]: fsmap 4 up:standby 2024-06-09T22:16:07.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:07 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-09T22:16:07.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:07 smithi194 ceph-mon[36856]: Deploying daemon mds.e on smithi153 2024-06-09T22:16:07.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:07 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.104:6836/3060956100,v1:172.21.15.104:6837/3060956100] up:boot 2024-06-09T22:16:07.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:07 smithi194 ceph-mon[36856]: fsmap 4 up:standby 2024-06-09T22:16:07.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:07 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-09T22:16:07.988 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:16:07 smithi153 systemd[1]: Starting Ceph mds.e for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:16:08.285 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:16:08 smithi153 podman[66096]: 2024-06-09 22:16:07.924302745 +0000 UTC m=+0.017108175 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:16:08.286 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:16:08 smithi153 podman[66096]: 2024-06-09 22:16:08.046345526 +0000 UTC m=+0.139150946 container create c5ae1e6e050f7721c32b7bf9d21a321a58add6a76c57edcfd4ca232e1e551cf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-e, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:16:08.286 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:16:08 smithi153 podman[66096]: 2024-06-09 22:16:08.291312092 +0000 UTC m=+0.384117513 container init c5ae1e6e050f7721c32b7bf9d21a321a58add6a76c57edcfd4ca232e1e551cf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-e, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2) 2024-06-09T22:16:08.343 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:16:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:08.737 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:08 smithi104 ceph-mon[32389]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:08.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:08 smithi153 ceph-mon[36599]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:08.738 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:16:08 smithi153 podman[66096]: 2024-06-09 22:16:08.294005046 +0000 UTC m=+0.386810469 container start c5ae1e6e050f7721c32b7bf9d21a321a58add6a76c57edcfd4ca232e1e551cf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-e, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-09T22:16:08.738 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:16:08 smithi153 ceph-mds[66116]: -- 172.21.15.153:0/2649033579 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55af21f5fa40 con 0x55af21f33000 2024-06-09T22:16:08.738 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:16:08 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-e[66112]: starting mds.e at 2024-06-09T22:16:08.738 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:16:08 smithi153 bash[66096]: c5ae1e6e050f7721c32b7bf9d21a321a58add6a76c57edcfd4ca232e1e551cf3 2024-06-09T22:16:08.738 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:16:08 smithi153 systemd[1]: Started Ceph mds.e for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:16:08.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:08 smithi194 ceph-mon[36856]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:09.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:09 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.153:6834/3075465176,v1:172.21.15.153:6835/3075465176] up:boot 2024-06-09T22:16:09.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:09 smithi153 ceph-mon[36599]: fsmap 5 up:standby 2024-06-09T22:16:09.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:09 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-09T22:16:09.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:09 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:09.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:09 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:09.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:09 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:09.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:09 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:09.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:09 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:09.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:09 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:09.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:09 smithi153 ceph-mon[36599]: Deploying daemon mds.f on smithi194 2024-06-09T22:16:09.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:09 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.153:6834/3075465176,v1:172.21.15.153:6835/3075465176] up:boot 2024-06-09T22:16:09.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:09 smithi104 ceph-mon[32389]: fsmap 5 up:standby 2024-06-09T22:16:09.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:09 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-09T22:16:09.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:09 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:09.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:09 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:09.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:09 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:09.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:09 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:09.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:09 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:09.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:09 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:09.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:09 smithi104 ceph-mon[32389]: Deploying daemon mds.f on smithi194 2024-06-09T22:16:09.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:09 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.153:6834/3075465176,v1:172.21.15.153:6835/3075465176] up:boot 2024-06-09T22:16:09.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:09 smithi194 ceph-mon[36856]: fsmap 5 up:standby 2024-06-09T22:16:09.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:09 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-09T22:16:09.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:09 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:09.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:09 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:09.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:09 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:09.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:09 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:09.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:09 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:09.762 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:09 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:09.762 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:09 smithi194 ceph-mon[36856]: Deploying daemon mds.f on smithi194 2024-06-09T22:16:10.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.357+0000 7f4eff677700 1 -- 172.21.15.104:0/2960865326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef81027a0 msgr2=0x7f4ef8106020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:10.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.357+0000 7f4eff677700 1 --2- 172.21.15.104:0/2960865326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef81027a0 0x7f4ef8106020 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40099c0 tx=0x7f4ef4009cd0 comp rx=0 tx=0).stop 2024-06-09T22:16:10.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.357+0000 7f4eff677700 1 -- 172.21.15.104:0/2960865326 shutdown_connections 2024-06-09T22:16:10.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.357+0000 7f4eff677700 1 --2- 172.21.15.104:0/2960865326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ef8106750 0x7f4ef810e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:10.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.357+0000 7f4eff677700 1 --2- 172.21.15.104:0/2960865326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef8101df0 0x7f4ef81021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:10.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.357+0000 7f4eff677700 1 --2- 172.21.15.104:0/2960865326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef81027a0 0x7f4ef8106020 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:10.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.357+0000 7f4eff677700 1 -- 172.21.15.104:0/2960865326 >> 172.21.15.104:0/2960865326 conn(0x7f4ef8077a00 msgr2=0x7f4ef80fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:10.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.357+0000 7f4eff677700 1 -- 172.21.15.104:0/2960865326 shutdown_connections 2024-06-09T22:16:10.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.357+0000 7f4eff677700 1 -- 172.21.15.104:0/2960865326 wait complete. 2024-06-09T22:16:10.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.358+0000 7f4eff677700 1 Processor -- start 2024-06-09T22:16:10.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.358+0000 7f4eff677700 1 -- start start 2024-06-09T22:16:10.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.358+0000 7f4eff677700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ef8101df0 0x7f4ef819ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:10.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.358+0000 7f4eff677700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef81027a0 0x7f4ef819b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:10.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.358+0000 7f4eff677700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef8106750 0x7f4ef819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:10.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.358+0000 7f4eff677700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef8111660 con 0x7f4ef81027a0 2024-06-09T22:16:10.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.358+0000 7f4eff677700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81114e0 con 0x7f4ef8106750 2024-06-09T22:16:10.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.358+0000 7f4eff677700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81117e0 con 0x7f4ef8101df0 2024-06-09T22:16:10.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.358+0000 7f4efcc12700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef81027a0 0x7f4ef819b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:10.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.358+0000 7f4efd413700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ef8101df0 0x7f4ef819ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:10.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4efcc12700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef81027a0 0x7f4ef819b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37832/0 (socket says 172.21.15.104:37832) 2024-06-09T22:16:10.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4efcc12700 1 -- 172.21.15.104:0/2183920740 learned_addr learned my addr 172.21.15.104:0/2183920740 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:10.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4efd413700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ef8101df0 0x7f4ef819ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:36526/0 (socket says 172.21.15.104:36526) 2024-06-09T22:16:10.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4efdc14700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef8106750 0x7f4ef819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:10.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4efcc12700 1 -- 172.21.15.104:0/2183920740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ef8101df0 msgr2=0x7f4ef819ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:10.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4efcc12700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ef8101df0 0x7f4ef819ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:10.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4efcc12700 1 -- 172.21.15.104:0/2183920740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef8106750 msgr2=0x7f4ef819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:10.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4efcc12700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef8106750 0x7f4ef819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:10.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4efcc12700 1 -- 172.21.15.104:0/2183920740 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef819fd90 con 0x7f4ef81027a0 2024-06-09T22:16:10.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4efd413700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ef8101df0 0x7f4ef819ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:16:10.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4efdc14700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef8106750 0x7f4ef819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:16:10.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4efcc12700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef81027a0 0x7f4ef819b2c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40059d0 tx=0x7f4ef40038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:10.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4eee7fc700 1 -- 172.21.15.104:0/2183920740 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4027070 con 0x7f4ef81027a0 2024-06-09T22:16:10.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4eff677700 1 -- 172.21.15.104:0/2183920740 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef81a0020 con 0x7f4ef81027a0 2024-06-09T22:16:10.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4eee7fc700 1 -- 172.21.15.104:0/2183920740 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ef4004050 con 0x7f4ef81027a0 2024-06-09T22:16:10.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4eff677700 1 -- 172.21.15.104:0/2183920740 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef81a7900 con 0x7f4ef81027a0 2024-06-09T22:16:10.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.359+0000 7f4eee7fc700 1 -- 172.21.15.104:0/2183920740 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4022db0 con 0x7f4ef81027a0 2024-06-09T22:16:10.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.361+0000 7f4eff677700 1 -- 172.21.15.104:0/2183920740 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ef819c0e0 con 0x7f4ef81027a0 2024-06-09T22:16:10.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.362+0000 7f4eee7fc700 1 -- 172.21.15.104:0/2183920740 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4ef402a070 con 0x7f4ef81027a0 2024-06-09T22:16:10.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.362+0000 7f4eee7fc700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4ee40a2ca0 0x7f4ee40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:10.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.362+0000 7f4efd413700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4ee40a2ca0 0x7f4ee40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:10.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.363+0000 7f4eee7fc700 1 -- 172.21.15.104:0/2183920740 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4ef40cf750 con 0x7f4ef81027a0 2024-06-09T22:16:10.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.363+0000 7f4efd413700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4ee40a2ca0 0x7f4ee40a5160 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8005950 tx=0x7f4ee800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:10.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.367+0000 7f4eee7fc700 1 -- 172.21.15.104:0/2183920740 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4ef4013070 con 0x7f4ef81027a0 2024-06-09T22:16:10.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.606+0000 7f4eff677700 1 -- 172.21.15.104:0/2183920740 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ef802dca0 con 0x7f4ef81027a0 2024-06-09T22:16:10.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.607+0000 7f4eee7fc700 1 -- 172.21.15.104:0/2183920740 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4708 (secure 0 0 0) 0x7f4ef409cbe0 con 0x7f4ef81027a0 2024-06-09T22:16:10.602 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:16:10.602 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24362,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24383,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-06-09T22:16:10.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.609+0000 7f4eff677700 1 -- 172.21.15.104:0/2183920740 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4ee40a2ca0 msgr2=0x7f4ee40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:10.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.609+0000 7f4eff677700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4ee40a2ca0 0x7f4ee40a5160 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8005950 tx=0x7f4ee800a300 comp rx=0 tx=0).stop 2024-06-09T22:16:10.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.609+0000 7f4eff677700 1 -- 172.21.15.104:0/2183920740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef81027a0 msgr2=0x7f4ef819b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:10.604 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:10 smithi104 ceph-mon[32389]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:10.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.609+0000 7f4eff677700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef81027a0 0x7f4ef819b2c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40059d0 tx=0x7f4ef40038b0 comp rx=0 tx=0).stop 2024-06-09T22:16:10.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.609+0000 7f4eff677700 1 -- 172.21.15.104:0/2183920740 shutdown_connections 2024-06-09T22:16:10.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.609+0000 7f4eff677700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4ee40a2ca0 0x7f4ee40a5160 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:10.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.609+0000 7f4eff677700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ef8106750 0x7f4ef819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:10.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.609+0000 7f4eff677700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ef8101df0 0x7f4ef819ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:10.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.609+0000 7f4eff677700 1 --2- 172.21.15.104:0/2183920740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef81027a0 0x7f4ef819b2c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:10.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.609+0000 7f4eff677700 1 -- 172.21.15.104:0/2183920740 >> 172.21.15.104:0/2183920740 conn(0x7f4ef8077a00 msgr2=0x7f4ef81055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:10.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.609+0000 7f4eff677700 1 -- 172.21.15.104:0/2183920740 shutdown_connections 2024-06-09T22:16:10.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:10.609+0000 7f4eff677700 1 -- 172.21.15.104:0/2183920740 wait complete. 2024-06-09T22:16:10.606 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 6 2024-06-09T22:16:10.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:10 smithi153 ceph-mon[36599]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:10.765 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:10 smithi194 ceph-mon[36856]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:10.766 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:16:10 smithi194 systemd[1]: Starting Ceph mds.f for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:16:11.054 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:16:10 smithi194 podman[67080]: 2024-06-09 22:16:10.67342684 +0000 UTC m=+0.017010518 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:16:11.054 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:16:10 smithi194 podman[67080]: 2024-06-09 22:16:10.795869333 +0000 UTC m=+0.139453004 container create ea3e90d5e7ea8e91584e570c18ea74d117513d9cf0e55fb07c54584a347c9f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_CLEAN=True, ceph=True, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-09T22:16:11.433 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:16:11 smithi194 podman[67080]: 2024-06-09 22:16:11.066247404 +0000 UTC m=+0.409831075 container init ea3e90d5e7ea8e91584e570c18ea74d117513d9cf0e55fb07c54584a347c9f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-06-09T22:16:11.433 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:16:11 smithi194 podman[67080]: 2024-06-09 22:16:11.068907792 +0000 UTC m=+0.412491471 container start ea3e90d5e7ea8e91584e570c18ea74d117513d9cf0e55fb07c54584a347c9f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-06-09T22:16:11.433 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:16:11 smithi194 ceph-mds[67099]: -- 172.21.15.194:0/1743043511 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56545fb51a40 con 0x56545fb24800 2024-06-09T22:16:11.433 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:16:11 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f[67095]: starting mds.f at 2024-06-09T22:16:11.433 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:16:11 smithi194 bash[67080]: ea3e90d5e7ea8e91584e570c18ea74d117513d9cf0e55fb07c54584a347c9f59 2024-06-09T22:16:11.434 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:16:11 smithi194 systemd[1]: Started Ceph mds.f for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:16:11.548 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:11 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2183920740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:11.548 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:11 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:11.548 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:11 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:11.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:11 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:11.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:11 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:11.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:11 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:11.549 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:11 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:11.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:11 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2183920740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:11.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:11 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:11.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:11 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:11.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:11 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:11.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:11 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:11.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:11 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:11.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:11 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:11.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:11 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2183920740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:11.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:11 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:11.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:11 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:11.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:11 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:11.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:11 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:11.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:11 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:11.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:11 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:12.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:12 smithi153 ceph-mon[36599]: Deploying daemon mds.c on smithi194 2024-06-09T22:16:12.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:12 smithi153 ceph-mon[36599]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:12.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:12 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] up:boot 2024-06-09T22:16:12.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:12 smithi153 ceph-mon[36599]: fsmap 6 up:standby 2024-06-09T22:16:12.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:12 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-09T22:16:12.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:12 smithi104 ceph-mon[32389]: Deploying daemon mds.c on smithi194 2024-06-09T22:16:12.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:12 smithi104 ceph-mon[32389]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:12.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:12 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] up:boot 2024-06-09T22:16:12.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:12 smithi104 ceph-mon[32389]: fsmap 6 up:standby 2024-06-09T22:16:12.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:12 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-09T22:16:12.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:12 smithi194 ceph-mon[36856]: Deploying daemon mds.c on smithi194 2024-06-09T22:16:12.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:12 smithi194 ceph-mon[36856]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:12.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:12 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] up:boot 2024-06-09T22:16:12.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:12 smithi194 ceph-mon[36856]: fsmap 6 up:standby 2024-06-09T22:16:12.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:12 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-09T22:16:13.286 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:16:13.540 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:13.554 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:16:13 smithi194 systemd[1]: Starting Ceph mds.c for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:16:13.883 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:16:13 smithi194 podman[67381]: 2024-06-09 22:16:13.497814742 +0000 UTC m=+0.016926474 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:16:13.883 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:16:13 smithi194 podman[67381]: 2024-06-09 22:16:13.61888964 +0000 UTC m=+0.138001375 container create d0aec5967e0adea7dfbfa62dce793e7f18969e9cd0ea56283db69825f6bf7286 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-c, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:16:13.883 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:16:13 smithi194 podman[67381]: 2024-06-09 22:16:13.888290889 +0000 UTC m=+0.407402601 container init d0aec5967e0adea7dfbfa62dce793e7f18969e9cd0ea56283db69825f6bf7286 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-c, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-09T22:16:14.304 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:16:13 smithi194 podman[67381]: 2024-06-09 22:16:13.891085269 +0000 UTC m=+0.410196993 container start d0aec5967e0adea7dfbfa62dce793e7f18969e9cd0ea56283db69825f6bf7286 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-c, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, io.buildah.version=1.33.5) 2024-06-09T22:16:14.304 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:16:13 smithi194 ceph-mds[67400]: -- 172.21.15.194:0/2789418254 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ca23fc1a40 con 0x55ca23f94800 2024-06-09T22:16:14.305 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:16:13 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-c[67396]: starting mds.c at 2024-06-09T22:16:14.305 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:16:13 smithi194 bash[67381]: d0aec5967e0adea7dfbfa62dce793e7f18969e9cd0ea56283db69825f6bf7286 2024-06-09T22:16:14.305 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:16:13 smithi194 systemd[1]: Started Ceph mds.c for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:16:14.681 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:14 smithi104 ceph-mon[32389]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:14.681 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:14.681 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:14.681 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:14.681 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:14.681 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:14.681 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:14 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:14.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:14 smithi153 ceph-mon[36599]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:14.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:14 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:14.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:14 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:14.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:14 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:14.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:14 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:14.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:14 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:14.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:14 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:14.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:14 smithi194 ceph-mon[36856]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:14.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:14.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:14.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:14.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:14.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:14.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:14 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:15.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.232+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/4272696638 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c68101670 msgr2=0x7f6c68101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:15.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.232+0000 7f6c6d2a3700 1 --2- 172.21.15.104:0/4272696638 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c68101670 0x7f6c68101af0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c009990 tx=0x7f6c5c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:16:15.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.232+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/4272696638 shutdown_connections 2024-06-09T22:16:15.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.232+0000 7f6c6d2a3700 1 --2- 172.21.15.104:0/4272696638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c68102030 0x7f6c6810e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:15.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.232+0000 7f6c6d2a3700 1 --2- 172.21.15.104:0/4272696638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c681074e0 0x7f6c681078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:15.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.232+0000 7f6c6d2a3700 1 --2- 172.21.15.104:0/4272696638 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c68101670 0x7f6c68101af0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:15.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.232+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/4272696638 >> 172.21.15.104:0/4272696638 conn(0x7f6c68078280 msgr2=0x7f6c68078690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:15.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.232+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/4272696638 shutdown_connections 2024-06-09T22:16:15.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.232+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/4272696638 wait complete. 2024-06-09T22:16:15.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.232+0000 7f6c6d2a3700 1 Processor -- start 2024-06-09T22:16:15.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.232+0000 7f6c6d2a3700 1 -- start start 2024-06-09T22:16:15.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c6d2a3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c68101670 0x7f6c68199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:15.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c6d2a3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c68102030 0x7f6c6819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:15.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c6d2a3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c681074e0 0x7f6c681a4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:15.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c6d2a3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6c68111780 con 0x7f6c681074e0 2024-06-09T22:16:15.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c6d2a3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c68111600 con 0x7f6c68101670 2024-06-09T22:16:15.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c6d2a3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6c68111900 con 0x7f6c68102030 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c66ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c68101670 0x7f6c68199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c677fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c681074e0 0x7f6c681a4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c66ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c68101670 0x7f6c68199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:53530/0 (socket says 172.21.15.104:53530) 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c677fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c681074e0 0x7f6c681a4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37848/0 (socket says 172.21.15.104:37848) 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c66ffd700 1 -- 172.21.15.104:0/995456205 learned_addr learned my addr 172.21.15.104:0/995456205 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c667fc700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c68102030 0x7f6c6819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c677fe700 1 -- 172.21.15.104:0/995456205 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c68102030 msgr2=0x7f6c6819a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c677fe700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c68102030 0x7f6c6819a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c677fe700 1 -- 172.21.15.104:0/995456205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c68101670 msgr2=0x7f6c68199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c677fe700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c68101670 0x7f6c68199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c677fe700 1 -- 172.21.15.104:0/995456205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c681a50e0 con 0x7f6c681074e0 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c66ffd700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c68101670 0x7f6c68199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:16:15.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c677fe700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c681074e0 0x7f6c681a4ba0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6c60007f10 tx=0x7f6c6000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:15.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.233+0000 7f6c4ffff700 1 -- 172.21.15.104:0/995456205 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c60004db0 con 0x7f6c681074e0 2024-06-09T22:16:15.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.234+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/995456205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c681a5310 con 0x7f6c681074e0 2024-06-09T22:16:15.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.234+0000 7f6c4ffff700 1 -- 172.21.15.104:0/995456205 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c60009e20 con 0x7f6c681074e0 2024-06-09T22:16:15.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.234+0000 7f6c4ffff700 1 -- 172.21.15.104:0/995456205 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c60015400 con 0x7f6c681074e0 2024-06-09T22:16:15.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.234+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/995456205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c681a5850 con 0x7f6c681074e0 2024-06-09T22:16:15.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.234+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/995456205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c681098c0 con 0x7f6c681074e0 2024-06-09T22:16:15.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.236+0000 7f6c4ffff700 1 -- 172.21.15.104:0/995456205 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6c6000db20 con 0x7f6c681074e0 2024-06-09T22:16:15.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.237+0000 7f6c4ffff700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6c500a2c20 0x7f6c500a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:15.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.237+0000 7f6c66ffd700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6c500a2c20 0x7f6c500a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:15.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.237+0000 7f6c4ffff700 1 -- 172.21.15.104:0/995456205 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6c600ced50 con 0x7f6c681074e0 2024-06-09T22:16:15.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.237+0000 7f6c66ffd700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6c500a2c20 0x7f6c500a50e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6c58005950 tx=0x7f6c5800b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:15.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.241+0000 7f6c4ffff700 1 -- 172.21.15.104:0/995456205 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6c60010070 con 0x7f6c681074e0 2024-06-09T22:16:15.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.482+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/995456205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c6806b1f0 con 0x7f6c681074e0 2024-06-09T22:16:15.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.483+0000 7f6c4ffff700 1 -- 172.21.15.104:0/995456205 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6361 (secure 0 0 0) 0x7f6c6009c150 con 0x7f6c681074e0 2024-06-09T22:16:15.478 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:16:15.478 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24362,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24383,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-06-09T22:16:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.485+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/995456205 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6c500a2c20 msgr2=0x7f6c500a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.485+0000 7f6c6d2a3700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6c500a2c20 0x7f6c500a50e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6c58005950 tx=0x7f6c5800b500 comp rx=0 tx=0).stop 2024-06-09T22:16:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.485+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/995456205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c681074e0 msgr2=0x7f6c681a4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.485+0000 7f6c6d2a3700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c681074e0 0x7f6c681a4ba0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6c60007f10 tx=0x7f6c6000d3b0 comp rx=0 tx=0).stop 2024-06-09T22:16:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.485+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/995456205 shutdown_connections 2024-06-09T22:16:15.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.485+0000 7f6c6d2a3700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6c500a2c20 0x7f6c500a50e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:15.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.485+0000 7f6c6d2a3700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c681074e0 0x7f6c681a4ba0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:15.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.485+0000 7f6c6d2a3700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c68101670 0x7f6c68199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:15.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.485+0000 7f6c6d2a3700 1 --2- 172.21.15.104:0/995456205 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c68102030 0x7f6c6819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:15.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.485+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/995456205 >> 172.21.15.104:0/995456205 conn(0x7f6c68078280 msgr2=0x7f6c68100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:15.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.485+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/995456205 shutdown_connections 2024-06-09T22:16:15.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:15.485+0000 7f6c6d2a3700 1 -- 172.21.15.104:0/995456205 wait complete. 2024-06-09T22:16:15.481 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 8 2024-06-09T22:16:15.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:15 smithi153 ceph-mon[36599]: Deploying daemon mds.h on smithi153 2024-06-09T22:16:15.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:15 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] up:boot 2024-06-09T22:16:15.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:15 smithi153 ceph-mon[36599]: fsmap 7 up:standby 2024-06-09T22:16:15.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:15 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-09T22:16:15.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:15 smithi104 ceph-mon[32389]: Deploying daemon mds.h on smithi153 2024-06-09T22:16:15.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:15 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] up:boot 2024-06-09T22:16:15.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:15 smithi104 ceph-mon[32389]: fsmap 7 up:standby 2024-06-09T22:16:15.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:15 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-09T22:16:15.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:15 smithi194 ceph-mon[36856]: Deploying daemon mds.h on smithi153 2024-06-09T22:16:15.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:15 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] up:boot 2024-06-09T22:16:15.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:15 smithi194 ceph-mon[36856]: fsmap 7 up:standby 2024-06-09T22:16:15.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:15 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-09T22:16:16.454 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:16:16 smithi153 systemd[1]: Starting Ceph mds.h for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:16:16.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:16 smithi153 ceph-mon[36599]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:16.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:16 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/995456205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:16.738 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:16:16 smithi153 podman[66397]: 2024-06-09 22:16:16.445616059 +0000 UTC m=+0.017363680 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:16:16.738 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:16:16 smithi153 podman[66397]: 2024-06-09 22:16:16.578116517 +0000 UTC m=+0.149864157 container create 6a8f04b33aece99c5ed5fbbc063d3bf6aad6a8fd24d82f2ab4678dbcaf29f7cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-h, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-06-09T22:16:16.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:16 smithi104 ceph-mon[32389]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:16.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:16 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/995456205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:16.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:16 smithi194 ceph-mon[36856]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:16.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:16 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/995456205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:17.238 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:16:16 smithi153 podman[66397]: 2024-06-09 22:16:16.83158141 +0000 UTC m=+0.403329027 container init 6a8f04b33aece99c5ed5fbbc063d3bf6aad6a8fd24d82f2ab4678dbcaf29f7cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-06-09T22:16:17.238 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:16:16 smithi153 podman[66397]: 2024-06-09 22:16:16.83421642 +0000 UTC m=+0.405964040 container start 6a8f04b33aece99c5ed5fbbc063d3bf6aad6a8fd24d82f2ab4678dbcaf29f7cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, ceph=True) 2024-06-09T22:16:17.239 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:16:16 smithi153 ceph-mds[66416]: -- 172.21.15.153:0/1227549257 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5619c5db1a40 con 0x5619c5d84800 2024-06-09T22:16:17.239 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:16:16 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-h[66412]: starting mds.h at 2024-06-09T22:16:17.239 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:16:16 smithi153 bash[66397]: 6a8f04b33aece99c5ed5fbbc063d3bf6aad6a8fd24d82f2ab4678dbcaf29f7cc 2024-06-09T22:16:17.239 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:16:16 smithi153 systemd[1]: Started Ceph mds.h for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:16:18.194 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:16:18.223 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:18.223 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:18.223 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:18.223 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:18.223 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:18.224 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:18.224 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:17 smithi104 ceph-mon[32389]: Deploying daemon mds.i on smithi194 2024-06-09T22:16:18.224 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:17 smithi104 ceph-mon[32389]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:18.224 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:17 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.153:6836/3294454895,v1:172.21.15.153:6837/3294454895] up:boot 2024-06-09T22:16:18.224 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:17 smithi104 ceph-mon[32389]: fsmap 8 up:standby 2024-06-09T22:16:18.224 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:17 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-09T22:16:18.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:18.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:18.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:18.238 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:18.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:18.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:18.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:17 smithi153 ceph-mon[36599]: Deploying daemon mds.i on smithi194 2024-06-09T22:16:18.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:17 smithi153 ceph-mon[36599]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:18.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:17 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.153:6836/3294454895,v1:172.21.15.153:6837/3294454895] up:boot 2024-06-09T22:16:18.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:17 smithi153 ceph-mon[36599]: fsmap 8 up:standby 2024-06-09T22:16:18.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:17 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-09T22:16:18.304 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:18.304 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:18.304 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:18.304 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:18.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:18.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:18.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:17 smithi194 ceph-mon[36856]: Deploying daemon mds.i on smithi194 2024-06-09T22:16:18.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:17 smithi194 ceph-mon[36856]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:18.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:17 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.153:6836/3294454895,v1:172.21.15.153:6837/3294454895] up:boot 2024-06-09T22:16:18.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:17 smithi194 ceph-mon[36856]: fsmap 8 up:standby 2024-06-09T22:16:18.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:17 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-09T22:16:18.447 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:19.304 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:16:19 smithi194 systemd[1]: Starting Ceph mds.i for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:16:19.647 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:16:19 smithi194 podman[67684]: 2024-06-09 22:16:19.266635197 +0000 UTC m=+0.017402526 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:16:19.647 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:16:19 smithi194 podman[67684]: 2024-06-09 22:16:19.382910721 +0000 UTC m=+0.133678043 container create 8ffd8f55d181714ee06007c3e3ae3abe9e8b2d682a78e34267a7356e8fdcbf37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, io.buildah.version=1.33.5) 2024-06-09T22:16:19.647 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:16:19 smithi194 podman[67684]: 2024-06-09 22:16:19.652413184 +0000 UTC m=+0.403180513 container init 8ffd8f55d181714ee06007c3e3ae3abe9e8b2d682a78e34267a7356e8fdcbf37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-09T22:16:20.054 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:16:19 smithi194 podman[67684]: 2024-06-09 22:16:19.655259212 +0000 UTC m=+0.406026534 container start 8ffd8f55d181714ee06007c3e3ae3abe9e8b2d682a78e34267a7356e8fdcbf37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:16:20.054 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:16:19 smithi194 ceph-mds[67703]: -- 172.21.15.194:0/1282643043 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d254c89a40 con 0x55d254c5d000 2024-06-09T22:16:20.055 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:16:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i[67699]: starting mds.i at 2024-06-09T22:16:20.055 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:16:19 smithi194 bash[67684]: 8ffd8f55d181714ee06007c3e3ae3abe9e8b2d682a78e34267a7356e8fdcbf37 2024-06-09T22:16:20.055 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:16:19 smithi194 systemd[1]: Started Ceph mds.i for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:16:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.116+0000 7f7dc8ddd700 1 -- 172.21.15.104:0/284993808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dc4106b90 msgr2=0x7f7dc4106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.116+0000 7f7dc8ddd700 1 --2- 172.21.15.104:0/284993808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dc4106b90 0x7f7dc4106f70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7db8009990 tx=0x7f7db8009ca0 comp rx=0 tx=0).stop 2024-06-09T22:16:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.116+0000 7f7dc8ddd700 1 -- 172.21.15.104:0/284993808 shutdown_connections 2024-06-09T22:16:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.116+0000 7f7dc8ddd700 1 --2- 172.21.15.104:0/284993808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dc4106b90 0x7f7dc4106f70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.116+0000 7f7dc8ddd700 1 --2- 172.21.15.104:0/284993808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dc4101510 0x7f7dc410d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.116+0000 7f7dc8ddd700 1 --2- 172.21.15.104:0/284993808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7dc4100b50 0x7f7dc4100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:20.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.116+0000 7f7dc8ddd700 1 -- 172.21.15.104:0/284993808 >> 172.21.15.104:0/284993808 conn(0x7f7dc4077a00 msgr2=0x7f7dc40fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:20.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.116+0000 7f7dc8ddd700 1 -- 172.21.15.104:0/284993808 shutdown_connections 2024-06-09T22:16:20.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.116+0000 7f7dc8ddd700 1 -- 172.21.15.104:0/284993808 wait complete. 2024-06-09T22:16:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc8ddd700 1 Processor -- start 2024-06-09T22:16:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc8ddd700 1 -- start start 2024-06-09T22:16:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc8ddd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7dc4100b50 0x7f7dc40fce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc8ddd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dc4101510 0x7f7dc40fd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc8ddd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dc4106b90 0x7f7dc40fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc8ddd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4110e50 con 0x7f7dc4106b90 2024-06-09T22:16:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc8ddd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4110cd0 con 0x7f7dc4101510 2024-06-09T22:16:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc8ddd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4110fd0 con 0x7f7dc4100b50 2024-06-09T22:16:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7dc4100b50 0x7f7dc40fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dc4106b90 0x7f7dc40fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7dc4100b50 0x7f7dc40fce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:50222/0 (socket says 172.21.15.104:50222) 2024-06-09T22:16:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc259c700 1 -- 172.21.15.104:0/2828196324 learned_addr learned my addr 172.21.15.104:0/2828196324 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dc4106b90 0x7f7dc40fd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40744/0 (socket says 172.21.15.104:40744) 2024-06-09T22:16:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.117+0000 7f7dc1d9b700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dc4101510 0x7f7dc40fd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.118+0000 7f7dc2d9d700 1 -- 172.21.15.104:0/2828196324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7dc4100b50 msgr2=0x7f7dc40fce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.118+0000 7f7dc2d9d700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7dc4100b50 0x7f7dc40fce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.118+0000 7f7dc2d9d700 1 -- 172.21.15.104:0/2828196324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dc4101510 msgr2=0x7f7dc40fd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.118+0000 7f7dc2d9d700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dc4101510 0x7f7dc40fd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.118+0000 7f7dc2d9d700 1 -- 172.21.15.104:0/2828196324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc41a7150 con 0x7f7dc4106b90 2024-06-09T22:16:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.118+0000 7f7dc2d9d700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dc4106b90 0x7f7dc40fd8f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc00ab40 tx=0x7f7dbc00ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.118+0000 7f7db37fe700 1 -- 172.21.15.104:0/2828196324 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc017020 con 0x7f7dc4106b90 2024-06-09T22:16:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.118+0000 7f7db37fe700 1 -- 172.21.15.104:0/2828196324 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dbc004bb0 con 0x7f7dc4106b90 2024-06-09T22:16:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.118+0000 7f7db37fe700 1 -- 172.21.15.104:0/2828196324 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc005260 con 0x7f7dc4106b90 2024-06-09T22:16:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.118+0000 7f7dc8ddd700 1 -- 172.21.15.104:0/2828196324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dc41a7440 con 0x7f7dc4106b90 2024-06-09T22:16:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.118+0000 7f7dc8ddd700 1 -- 172.21.15.104:0/2828196324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dc41a7930 con 0x7f7dc4106b90 2024-06-09T22:16:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.119+0000 7f7db17fa700 1 -- 172.21.15.104:0/2828196324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7da8005330 con 0x7f7dc4106b90 2024-06-09T22:16:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.120+0000 7f7db37fe700 1 -- 172.21.15.104:0/2828196324 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7dbc00c580 con 0x7f7dc4106b90 2024-06-09T22:16:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.121+0000 7f7db37fe700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7dac0a2ca0 0x7f7dac0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.121+0000 7f7dc259c700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7dac0a2ca0 0x7f7dac0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.121+0000 7f7db37fe700 1 -- 172.21.15.104:0/2828196324 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7dbc0ce020 con 0x7f7dc4106b90 2024-06-09T22:16:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.122+0000 7f7dc259c700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7dac0a2ca0 0x7f7dac0a5160 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7db8009750 tx=0x7f7db8005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.126+0000 7f7db37fe700 1 -- 172.21.15.104:0/2828196324 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7dbc021020 con 0x7f7dc4106b90 2024-06-09T22:16:20.389 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:20 smithi104 ceph-mon[32389]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:20.389 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:20.389 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:20.389 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:20.389 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:20.389 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:20.389 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:20 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:20.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.393+0000 7f7db17fa700 1 -- 172.21.15.104:0/2828196324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7da80058f0 con 0x7f7dc4106b90 2024-06-09T22:16:20.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.394+0000 7f7db37fe700 1 -- 172.21.15.104:0/2828196324 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8019 (secure 0 0 0) 0x7f7dbc09b4b0 con 0x7f7dc4106b90 2024-06-09T22:16:20.390 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:16:20.390 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24362,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24383,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24398,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-06-09T22:16:20.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.396+0000 7f7db17fa700 1 -- 172.21.15.104:0/2828196324 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7dac0a2ca0 msgr2=0x7f7dac0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:20.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.396+0000 7f7db17fa700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7dac0a2ca0 0x7f7dac0a5160 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7db8009750 tx=0x7f7db8005a90 comp rx=0 tx=0).stop 2024-06-09T22:16:20.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.396+0000 7f7db17fa700 1 -- 172.21.15.104:0/2828196324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dc4106b90 msgr2=0x7f7dc40fd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:20.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.396+0000 7f7db17fa700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dc4106b90 0x7f7dc40fd8f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc00ab40 tx=0x7f7dbc00ae50 comp rx=0 tx=0).stop 2024-06-09T22:16:20.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.396+0000 7f7db17fa700 1 -- 172.21.15.104:0/2828196324 shutdown_connections 2024-06-09T22:16:20.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.396+0000 7f7db17fa700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7dac0a2ca0 0x7f7dac0a5160 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:20.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.396+0000 7f7db17fa700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dc4106b90 0x7f7dc40fd8f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:20.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.396+0000 7f7db17fa700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7dc4100b50 0x7f7dc40fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:20.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.396+0000 7f7db17fa700 1 --2- 172.21.15.104:0/2828196324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7dc4101510 0x7f7dc40fd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:20.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.396+0000 7f7db17fa700 1 -- 172.21.15.104:0/2828196324 >> 172.21.15.104:0/2828196324 conn(0x7f7dc4077a00 msgr2=0x7f7dc40fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:20.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.396+0000 7f7db17fa700 1 -- 172.21.15.104:0/2828196324 shutdown_connections 2024-06-09T22:16:20.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:20.396+0000 7f7db17fa700 1 -- 172.21.15.104:0/2828196324 wait complete. 2024-06-09T22:16:20.393 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 10 2024-06-09T22:16:20.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:20 smithi153 ceph-mon[36599]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:20.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:20.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:20.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:20.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:20.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:20.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:20 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:20.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:20 smithi194 ceph-mon[36856]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:20.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:20.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:20.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:20.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:20.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:20.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:20 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:21.692 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:21 smithi104 ceph-mon[32389]: Deploying daemon mds.g on smithi104 2024-06-09T22:16:21.693 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:21 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] up:boot 2024-06-09T22:16:21.693 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:21 smithi104 ceph-mon[32389]: fsmap 9 up:standby 2024-06-09T22:16:21.693 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:21 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-09T22:16:21.693 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:21 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2828196324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:21.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:21 smithi153 ceph-mon[36599]: Deploying daemon mds.g on smithi104 2024-06-09T22:16:21.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:21 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] up:boot 2024-06-09T22:16:21.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:21 smithi153 ceph-mon[36599]: fsmap 9 up:standby 2024-06-09T22:16:21.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:21 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-09T22:16:21.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:21 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2828196324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:21.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:21 smithi194 ceph-mon[36856]: Deploying daemon mds.g on smithi104 2024-06-09T22:16:21.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:21 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] up:boot 2024-06-09T22:16:21.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:21 smithi194 ceph-mon[36856]: fsmap 9 up:standby 2024-06-09T22:16:21.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:21 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-09T22:16:21.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:21 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2828196324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:22.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:22 smithi153 ceph-mon[36599]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:22.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:22 smithi104 ceph-mon[32389]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:22.752 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:16:22 smithi104 systemd[1]: Starting Ceph mds.g for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:16:22.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:22 smithi194 ceph-mon[36856]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:23.050 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:16:22 smithi104 podman[71506]: 2024-06-09 22:16:22.68323988 +0000 UTC m=+0.016963671 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:16:23.050 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:16:22 smithi104 podman[71506]: 2024-06-09 22:16:22.803035104 +0000 UTC m=+0.136758894 container create 14860765eeef8b3083e6c5843a2e9d29a20604db94bb1d246d4a6afba8264110 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-g, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:16:23.051 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:16:23 smithi104 podman[71506]: 2024-06-09 22:16:23.05577556 +0000 UTC m=+0.389499341 container init 14860765eeef8b3083e6c5843a2e9d29a20604db94bb1d246d4a6afba8264110 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-3d89acc, ceph=True) 2024-06-09T22:16:23.502 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:16:23 smithi104 podman[71506]: 2024-06-09 22:16:23.058689569 +0000 UTC m=+0.392413370 container start 14860765eeef8b3083e6c5843a2e9d29a20604db94bb1d246d4a6afba8264110 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-06-09T22:16:23.502 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:16:23 smithi104 ceph-mds[71525]: -- 172.21.15.104:0/2683414092 <== mon.0 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56010f57fa40 con 0x56010f552c00 2024-06-09T22:16:23.502 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:16:23 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-g[71521]: starting mds.g at 2024-06-09T22:16:23.502 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:16:23 smithi104 bash[71506]: 14860765eeef8b3083e6c5843a2e9d29a20604db94bb1d246d4a6afba8264110 2024-06-09T22:16:23.502 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:16:23 smithi104 systemd[1]: Started Ceph mds.g for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:16:23.727 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:16:24.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:24 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:24.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:24 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:24.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:24 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:24.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:24 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:24.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:24 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:24.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:24 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:24.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:24 smithi153 ceph-mon[36599]: Deploying daemon mds.a on smithi104 2024-06-09T22:16:24.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:24 smithi153 ceph-mon[36599]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:24.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:24 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.104:6838/647544853,v1:172.21.15.104:6839/647544853] up:boot 2024-06-09T22:16:24.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:24 smithi153 ceph-mon[36599]: fsmap 10 up:standby 2024-06-09T22:16:24.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:24 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-09T22:16:24.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:24 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:24.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:24 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:24.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:24 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:24.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:24 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:24.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:24 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:24.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:24 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:24.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:24 smithi104 ceph-mon[32389]: Deploying daemon mds.a on smithi104 2024-06-09T22:16:24.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:24 smithi104 ceph-mon[32389]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:24.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:24 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.104:6838/647544853,v1:172.21.15.104:6839/647544853] up:boot 2024-06-09T22:16:24.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:24 smithi104 ceph-mon[32389]: fsmap 10 up:standby 2024-06-09T22:16:24.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:24 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-09T22:16:24.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:24 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:24.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:24 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:24.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:24 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:24.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:24 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:24.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:24 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:24.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:24 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:24.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:24 smithi194 ceph-mon[36856]: Deploying daemon mds.a on smithi104 2024-06-09T22:16:24.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:24 smithi194 ceph-mon[36856]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:24.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:24 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.104:6838/647544853,v1:172.21.15.104:6839/647544853] up:boot 2024-06-09T22:16:24.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:24 smithi194 ceph-mon[36856]: fsmap 10 up:standby 2024-06-09T22:16:24.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:24 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-09T22:16:24.907 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:26.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:26 smithi104 ceph-mon[32389]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:26.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:26 smithi153 ceph-mon[36599]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:26.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:26 smithi194 ceph-mon[36856]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:27.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.048+0000 7ff03c910700 1 -- 172.21.15.104:0/3042568085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0381066b0 msgr2=0x7ff03810e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:27.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.048+0000 7ff03c910700 1 --2- 172.21.15.104:0/3042568085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0381066b0 0x7ff03810e280 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff03000b0d0 tx=0x7ff03000b3e0 comp rx=0 tx=0).stop 2024-06-09T22:16:27.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.048+0000 7ff03c910700 1 -- 172.21.15.104:0/3042568085 shutdown_connections 2024-06-09T22:16:27.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.048+0000 7ff03c910700 1 --2- 172.21.15.104:0/3042568085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0381066b0 0x7ff03810e280 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:27.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.048+0000 7ff03c910700 1 --2- 172.21.15.104:0/3042568085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff038101620 0x7ff038101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:27.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.048+0000 7ff03c910700 1 --2- 172.21.15.104:0/3042568085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff038101fd0 0x7ff038105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:27.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.048+0000 7ff03c910700 1 -- 172.21.15.104:0/3042568085 >> 172.21.15.104:0/3042568085 conn(0x7ff038078340 msgr2=0x7ff038078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:27.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.048+0000 7ff03c910700 1 -- 172.21.15.104:0/3042568085 shutdown_connections 2024-06-09T22:16:27.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.048+0000 7ff03c910700 1 -- 172.21.15.104:0/3042568085 wait complete. 2024-06-09T22:16:27.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.049+0000 7ff03c910700 1 Processor -- start 2024-06-09T22:16:27.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.049+0000 7ff03c910700 1 -- start start 2024-06-09T22:16:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.049+0000 7ff03c910700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff038101620 0x7ff038199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.049+0000 7ff03c910700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff038101fd0 0x7ff038199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.049+0000 7ff03c910700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0381066b0 0x7ff0381a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.049+0000 7ff03c910700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff0381116a0 con 0x7ff0381066b0 2024-06-09T22:16:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.049+0000 7ff03c910700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff038111520 con 0x7ff038101fd0 2024-06-09T22:16:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.049+0000 7ff03c910700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff038111820 con 0x7ff038101620 2024-06-09T22:16:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff036d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0381066b0 0x7ff0381a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff036d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0381066b0 0x7ff0381a4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42178/0 (socket says 172.21.15.104:42178) 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff036d9d700 1 -- 172.21.15.104:0/3056823692 learned_addr learned my addr 172.21.15.104:0/3056823692 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff035d9b700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff038101fd0 0x7ff038199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff03659c700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff038101620 0x7ff038199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff036d9d700 1 -- 172.21.15.104:0/3056823692 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff038101620 msgr2=0x7ff038199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff036d9d700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff038101620 0x7ff038199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff036d9d700 1 -- 172.21.15.104:0/3056823692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff038101fd0 msgr2=0x7ff038199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff036d9d700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff038101fd0 0x7ff038199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff036d9d700 1 -- 172.21.15.104:0/3056823692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0381a4fe0 con 0x7ff0381066b0 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff036d9d700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0381066b0 0x7ff0381a4aa0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff030006070 tx=0x7ff030004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff0277fe700 1 -- 172.21.15.104:0/3056823692 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff030021070 con 0x7ff0381066b0 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff0277fe700 1 -- 172.21.15.104:0/3056823692 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff030007b80 con 0x7ff0381066b0 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff03c910700 1 -- 172.21.15.104:0/3056823692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0381a5210 con 0x7ff0381066b0 2024-06-09T22:16:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff0277fe700 1 -- 172.21.15.104:0/3056823692 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0300052a0 con 0x7ff0381066b0 2024-06-09T22:16:27.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.050+0000 7ff03c910700 1 -- 172.21.15.104:0/3056823692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0381a5670 con 0x7ff0381066b0 2024-06-09T22:16:27.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.051+0000 7ff03c910700 1 -- 172.21.15.104:0/3056823692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff038069180 con 0x7ff0381066b0 2024-06-09T22:16:27.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.052+0000 7ff0277fe700 1 -- 172.21.15.104:0/3056823692 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff030024070 con 0x7ff0381066b0 2024-06-09T22:16:27.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.053+0000 7ff0277fe700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff0200a2c20 0x7ff0200a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:27.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.053+0000 7ff03659c700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff0200a2c20 0x7ff0200a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:27.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.053+0000 7ff0277fe700 1 -- 172.21.15.104:0/3056823692 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff0300cfc80 con 0x7ff0381066b0 2024-06-09T22:16:27.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.053+0000 7ff03659c700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff0200a2c20 0x7ff0200a50e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff028005950 tx=0x7ff02800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:27.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.057+0000 7ff0277fe700 1 -- 172.21.15.104:0/3056823692 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff03009d110 con 0x7ff0381066b0 2024-06-09T22:16:27.234 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:16:27 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-a[71945]: starting mds.a at 2024-06-09T22:16:27.234 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:16:27 smithi104 bash[71925]: cdde291a6221532b97b0d9cb4b8206f8307140dcb3b80ac4f3dbcb4ea60af0d9 2024-06-09T22:16:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.309+0000 7ff03c910700 1 -- 172.21.15.104:0/3056823692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff03802de60 con 0x7ff0381066b0 2024-06-09T22:16:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.311+0000 7ff0277fe700 1 -- 172.21.15.104:0/3056823692 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8845 (secure 0 0 0) 0x7ff03009ca10 con 0x7ff0381066b0 2024-06-09T22:16:27.309 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:16:27.309 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24362,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24383,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24398,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-06-09T22:16:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.314+0000 7ff03c910700 1 -- 172.21.15.104:0/3056823692 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff0200a2c20 msgr2=0x7ff0200a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.314+0000 7ff03c910700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff0200a2c20 0x7ff0200a50e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff028005950 tx=0x7ff02800a300 comp rx=0 tx=0).stop 2024-06-09T22:16:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.314+0000 7ff03c910700 1 -- 172.21.15.104:0/3056823692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0381066b0 msgr2=0x7ff0381a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.314+0000 7ff03c910700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0381066b0 0x7ff0381a4aa0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff030006070 tx=0x7ff030004d00 comp rx=0 tx=0).stop 2024-06-09T22:16:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.314+0000 7ff03c910700 1 -- 172.21.15.104:0/3056823692 shutdown_connections 2024-06-09T22:16:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.314+0000 7ff03c910700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff0200a2c20 0x7ff0200a50e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.314+0000 7ff03c910700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0381066b0 0x7ff0381a4aa0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.314+0000 7ff03c910700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff038101620 0x7ff038199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.314+0000 7ff03c910700 1 --2- 172.21.15.104:0/3056823692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff038101fd0 0x7ff038199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.314+0000 7ff03c910700 1 -- 172.21.15.104:0/3056823692 >> 172.21.15.104:0/3056823692 conn(0x7ff038078340 msgr2=0x7ff0380ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.314+0000 7ff03c910700 1 -- 172.21.15.104:0/3056823692 shutdown_connections 2024-06-09T22:16:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:27.314+0000 7ff03c910700 1 -- 172.21.15.104:0/3056823692 wait complete. 2024-06-09T22:16:27.312 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 11 2024-06-09T22:16:27.502 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:16:27 smithi104 systemd[1]: Started Ceph mds.a for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:16:28.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:28 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:28.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:28 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:28.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:28 smithi194 ceph-mon[36856]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:28.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:28 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:28.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:28 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:28.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:28 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:28.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:28 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3056823692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:28.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:28 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:28.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:28 smithi194 ceph-mon[36856]: Deploying daemon mds.b on smithi153 2024-06-09T22:16:28.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:28 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.104:6840/3830332242,v1:172.21.15.104:6841/3830332242] up:boot 2024-06-09T22:16:28.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:28 smithi194 ceph-mon[36856]: fsmap 11 up:standby 2024-06-09T22:16:28.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:28 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-09T22:16:28.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:28 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:28.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:28 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:28.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:28 smithi153 ceph-mon[36599]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:28.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:28 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:28.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:28 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:28.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:28 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:28.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:28 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3056823692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:28.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:28 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:28.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:28 smithi153 ceph-mon[36599]: Deploying daemon mds.b on smithi153 2024-06-09T22:16:28.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:28 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.104:6840/3830332242,v1:172.21.15.104:6841/3830332242] up:boot 2024-06-09T22:16:28.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:28 smithi153 ceph-mon[36599]: fsmap 11 up:standby 2024-06-09T22:16:28.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:28 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-09T22:16:28.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:28 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:28.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:28 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:28.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:28 smithi104 ceph-mon[32389]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:28.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:28 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:28.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:28 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-09T22:16:28.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:28 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-09T22:16:28.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:28 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3056823692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:28.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:28 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:28.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:28 smithi104 ceph-mon[32389]: Deploying daemon mds.b on smithi153 2024-06-09T22:16:28.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:28 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.104:6840/3830332242,v1:172.21.15.104:6841/3830332242] up:boot 2024-06-09T22:16:28.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:28 smithi104 ceph-mon[32389]: fsmap 11 up:standby 2024-06-09T22:16:28.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:28 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-09T22:16:29.655 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:16:29 smithi153 systemd[1]: Starting Ceph mds.b for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:16:29.974 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:16:29 smithi153 podman[66701]: 2024-06-09 22:16:29.562544981 +0000 UTC m=+0.017225184 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:16:29.974 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:16:29 smithi153 podman[66701]: 2024-06-09 22:16:29.72613367 +0000 UTC m=+0.180813865 container create 0c85ecc8056baf0e4c3aefd1c003fcaf4e0138bf6ea97a63d9b96c40c492c84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_CLEAN=True, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-09T22:16:29.974 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:16:29 smithi153 podman[66701]: 2024-06-09 22:16:29.979088784 +0000 UTC m=+0.433768990 container init 0c85ecc8056baf0e4c3aefd1c003fcaf4e0138bf6ea97a63d9b96c40c492c84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-09T22:16:30.025 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:16:30.238 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:16:29 smithi153 podman[66701]: 2024-06-09 22:16:29.981971737 +0000 UTC m=+0.436651947 container start 0c85ecc8056baf0e4c3aefd1c003fcaf4e0138bf6ea97a63d9b96c40c492c84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-09T22:16:30.238 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:16:30 smithi153 ceph-mds[66720]: -- 172.21.15.153:0/3033484101 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5595a4937a40 con 0x5595a490b000 2024-06-09T22:16:30.239 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:16:30 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b[66716]: starting mds.b at 2024-06-09T22:16:30.239 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:16:30 smithi153 bash[66701]: 0c85ecc8056baf0e4c3aefd1c003fcaf4e0138bf6ea97a63d9b96c40c492c84b 2024-06-09T22:16:30.239 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:16:30 smithi153 systemd[1]: Started Ceph mds.b for a948c0ca-26ac-11ef-bc9b-c7b262605968. 2024-06-09T22:16:30.294 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:30.700 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:30 smithi104 ceph-mon[32389]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:30.700 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.700 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.700 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.700 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.700 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.700 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.700 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:16:30.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:30 smithi153 ceph-mon[36599]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:30.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:16:30.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:30 smithi194 ceph-mon[36856]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:30.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:30.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:16:31.411 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:31 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] up:boot 2024-06-09T22:16:31.411 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:31 smithi153 ceph-mon[36599]: fsmap 12 up:standby 2024-06-09T22:16:31.411 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-09T22:16:31.462 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:31 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] up:boot 2024-06-09T22:16:31.462 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:31 smithi104 ceph-mon[32389]: fsmap 12 up:standby 2024-06-09T22:16:31.463 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-09T22:16:31.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:31 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] up:boot 2024-06-09T22:16:31.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:31 smithi194 ceph-mon[36856]: fsmap 12 up:standby 2024-06-09T22:16:31.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-09T22:16:32.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.035+0000 7f13757f0700 1 -- 172.21.15.104:0/2343939255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1370105330 msgr2=0x7f1370105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:32.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.035+0000 7f13757f0700 1 --2- 172.21.15.104:0/2343939255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1370105330 0x7f1370105710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f13600099c0 tx=0x7f1360009cd0 comp rx=0 tx=0).stop 2024-06-09T22:16:32.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.035+0000 7f13757f0700 1 -- 172.21.15.104:0/2343939255 shutdown_connections 2024-06-09T22:16:32.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.035+0000 7f13757f0700 1 --2- 172.21.15.104:0/2343939255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1370105330 0x7f1370105710 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:32.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.035+0000 7f13757f0700 1 --2- 172.21.15.104:0/2343939255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1370107b60 0x7f137010e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:32.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.035+0000 7f13757f0700 1 --2- 172.21.15.104:0/2343939255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13701071a0 0x7f1370107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:32.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.035+0000 7f13757f0700 1 -- 172.21.15.104:0/2343939255 >> 172.21.15.104:0/2343939255 conn(0x7f13700766b0 msgr2=0x7f1370076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:32.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.035+0000 7f13757f0700 1 -- 172.21.15.104:0/2343939255 shutdown_connections 2024-06-09T22:16:32.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.035+0000 7f13757f0700 1 -- 172.21.15.104:0/2343939255 wait complete. 2024-06-09T22:16:32.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.036+0000 7f13757f0700 1 Processor -- start 2024-06-09T22:16:32.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.036+0000 7f13757f0700 1 -- start start 2024-06-09T22:16:32.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f13757f0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1370105330 0x7f137019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:32.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f13757f0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13701071a0 0x7f137019ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:32.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f13757f0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1370107b60 0x7f1370198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:32.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f13757f0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1370111750 con 0x7f1370107b60 2024-06-09T22:16:32.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f13757f0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f13701115d0 con 0x7f13701071a0 2024-06-09T22:16:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f13757f0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f13701118d0 con 0x7f1370105330 2024-06-09T22:16:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f136f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1370107b60 0x7f1370198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f136effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1370105330 0x7f137019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f136e7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13701071a0 0x7f137019ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f136f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1370107b60 0x7f1370198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42212/0 (socket says 172.21.15.104:42212) 2024-06-09T22:16:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f136effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1370105330 0x7f137019e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:36138/0 (socket says 172.21.15.104:36138) 2024-06-09T22:16:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f136f7fe700 1 -- 172.21.15.104:0/2007937098 learned_addr learned my addr 172.21.15.104:0/2007937098 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f136e7fc700 1 -- 172.21.15.104:0/2007937098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1370105330 msgr2=0x7f137019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f136e7fc700 1 --2- 172.21.15.104:0/2007937098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1370105330 0x7f137019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f136e7fc700 1 -- 172.21.15.104:0/2007937098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1370107b60 msgr2=0x7f1370198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f136e7fc700 1 --2- 172.21.15.104:0/2007937098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1370107b60 0x7f1370198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f136e7fc700 1 -- 172.21.15.104:0/2007937098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1370199170 con 0x7f13701071a0 2024-06-09T22:16:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.037+0000 7f136e7fc700 1 --2- 172.21.15.104:0/2007937098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13701071a0 0x7f137019ed90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f136400d750 tx=0x7f136400da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.038+0000 7f1357fff700 1 -- 172.21.15.104:0/2007937098 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364014030 con 0x7f13701071a0 2024-06-09T22:16:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.038+0000 7f13757f0700 1 -- 172.21.15.104:0/2007937098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1370199460 con 0x7f13701071a0 2024-06-09T22:16:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.038+0000 7f1357fff700 1 -- 172.21.15.104:0/2007937098 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1364004500 con 0x7f13701071a0 2024-06-09T22:16:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.038+0000 7f1357fff700 1 -- 172.21.15.104:0/2007937098 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364005400 con 0x7f13701071a0 2024-06-09T22:16:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.038+0000 7f13757f0700 1 -- 172.21.15.104:0/2007937098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13701a5720 con 0x7f13701071a0 2024-06-09T22:16:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.039+0000 7f13757f0700 1 -- 172.21.15.104:0/2007937098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1370069180 con 0x7f13701071a0 2024-06-09T22:16:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.040+0000 7f1357fff700 1 -- 172.21.15.104:0/2007937098 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f1364004750 con 0x7f13701071a0 2024-06-09T22:16:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.041+0000 7f1357fff700 1 --2- 172.21.15.104:0/2007937098 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f13580a2c20 0x7f13580a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.041+0000 7f136effd700 1 --2- 172.21.15.104:0/2007937098 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f13580a2c20 0x7f13580a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.041+0000 7f1357fff700 1 -- 172.21.15.104:0/2007937098 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f13640ce550 con 0x7f13701071a0 2024-06-09T22:16:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.041+0000 7f136effd700 1 --2- 172.21.15.104:0/2007937098 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f13580a2c20 0x7f13580a50e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1360000c00 tx=0x7f1360005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:32.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.044+0000 7f1357fff700 1 -- 172.21.15.104:0/2007937098 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f136409b8b0 con 0x7f13701071a0 2024-06-09T22:16:32.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.288+0000 7f13757f0700 1 -- 172.21.15.104:0/2007937098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13701a5950 con 0x7f13701071a0 2024-06-09T22:16:32.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.289+0000 7f1357fff700 1 -- 172.21.15.104:0/2007937098 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10500 (secure 0 0 0) 0x7f136409b1b0 con 0x7f13701071a0 2024-06-09T22:16:32.285 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:16:32.285 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24362,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24383,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24398,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-06-09T22:16:32.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.292+0000 7f13757f0700 1 -- 172.21.15.104:0/2007937098 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f13580a2c20 msgr2=0x7f13580a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:32.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.292+0000 7f13757f0700 1 --2- 172.21.15.104:0/2007937098 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f13580a2c20 0x7f13580a50e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1360000c00 tx=0x7f1360005fb0 comp rx=0 tx=0).stop 2024-06-09T22:16:32.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.292+0000 7f13757f0700 1 -- 172.21.15.104:0/2007937098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13701071a0 msgr2=0x7f137019ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:32.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.292+0000 7f13757f0700 1 --2- 172.21.15.104:0/2007937098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13701071a0 0x7f137019ed90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f136400d750 tx=0x7f136400da60 comp rx=0 tx=0).stop 2024-06-09T22:16:32.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.292+0000 7f13757f0700 1 -- 172.21.15.104:0/2007937098 shutdown_connections 2024-06-09T22:16:32.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.292+0000 7f13757f0700 1 --2- 172.21.15.104:0/2007937098 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f13580a2c20 0x7f13580a50e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:32.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.292+0000 7f13757f0700 1 --2- 172.21.15.104:0/2007937098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1370107b60 0x7f1370198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:32.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.292+0000 7f13757f0700 1 --2- 172.21.15.104:0/2007937098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1370105330 0x7f137019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:32.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.292+0000 7f13757f0700 1 --2- 172.21.15.104:0/2007937098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13701071a0 0x7f137019ed90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:32.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.292+0000 7f13757f0700 1 -- 172.21.15.104:0/2007937098 >> 172.21.15.104:0/2007937098 conn(0x7f13700766b0 msgr2=0x7f1370100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:32.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.292+0000 7f13757f0700 1 -- 172.21.15.104:0/2007937098 shutdown_connections 2024-06-09T22:16:32.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:32.292+0000 7f13757f0700 1 -- 172.21.15.104:0/2007937098 wait complete. 2024-06-09T22:16:32.288 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 13 2024-06-09T22:16:32.630 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:32 smithi153 ceph-mon[36599]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:32.630 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:32.630 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:32.630 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:32.630 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:32.630 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:32 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2007937098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:32.674 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:32 smithi104 ceph-mon[32389]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:32.674 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:32.674 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:32.674 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:32.674 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:32.674 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:32 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2007937098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:32.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:32 smithi194 ceph-mon[36856]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:32.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:32.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:32.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:32.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:32.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:32 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2007937098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:16:33.004 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-06-09T22:16:33.434 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:34.583 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:34 smithi104 ceph-mon[32389]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:34.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:34 smithi153 ceph-mon[36599]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:34.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:34 smithi194 ceph-mon[36856]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:35.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.117+0000 7f22e101d700 1 -- 172.21.15.104:0/2390733553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22dc108530 msgr2=0x7f22dc108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:35.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.117+0000 7f22e101d700 1 --2- 172.21.15.104:0/2390733553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22dc108530 0x7f22dc108910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009990 tx=0x7f22cc009ca0 comp rx=0 tx=0).stop 2024-06-09T22:16:35.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.117+0000 7f22e101d700 1 -- 172.21.15.104:0/2390733553 shutdown_connections 2024-06-09T22:16:35.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.117+0000 7f22e101d700 1 --2- 172.21.15.104:0/2390733553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22dc108530 0x7f22dc108910 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:35.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.117+0000 7f22e101d700 1 --2- 172.21.15.104:0/2390733553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22dc06b5c0 0x7f22dc10f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:35.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.117+0000 7f22e101d700 1 --2- 172.21.15.104:0/2390733553 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22dc06ac00 0x7f22dc06b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:35.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.117+0000 7f22e101d700 1 -- 172.21.15.104:0/2390733553 >> 172.21.15.104:0/2390733553 conn(0x7f22dc077400 msgr2=0x7f22dc077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:35.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.117+0000 7f22e101d700 1 -- 172.21.15.104:0/2390733553 shutdown_connections 2024-06-09T22:16:35.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.117+0000 7f22e101d700 1 -- 172.21.15.104:0/2390733553 wait complete. 2024-06-09T22:16:35.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.118+0000 7f22e101d700 1 Processor -- start 2024-06-09T22:16:35.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.118+0000 7f22e101d700 1 -- start start 2024-06-09T22:16:35.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22e101d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22dc06ac00 0x7f22dc19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:35.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22e101d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22dc06b5c0 0x7f22dc19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:35.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22e101d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22dc108530 0x7f22dc198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:35.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22e101d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc1127e0 con 0x7f22dc108530 2024-06-09T22:16:35.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22e101d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc112660 con 0x7f22dc06ac00 2024-06-09T22:16:35.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22e101d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc112960 con 0x7f22dc06b5c0 2024-06-09T22:16:35.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22db59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22dc108530 0x7f22dc198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:35.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22db59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22dc108530 0x7f22dc198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42226/0 (socket says 172.21.15.104:42226) 2024-06-09T22:16:35.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22db59e700 1 -- 172.21.15.104:0/612855336 learned_addr learned my addr 172.21.15.104:0/612855336 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:35.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22da59c700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22dc06b5c0 0x7f22dc19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:35.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22dad9d700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22dc06ac00 0x7f22dc19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:35.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22db59e700 1 -- 172.21.15.104:0/612855336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22dc06b5c0 msgr2=0x7f22dc19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:35.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22db59e700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22dc06b5c0 0x7f22dc19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:35.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22db59e700 1 -- 172.21.15.104:0/612855336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22dc06ac00 msgr2=0x7f22dc19e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:35.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22db59e700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22dc06ac00 0x7f22dc19e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:35.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22db59e700 1 -- 172.21.15.104:0/612855336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc199200 con 0x7f22dc108530 2024-06-09T22:16:35.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22db59e700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22dc108530 0x7f22dc198aa0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f22d400b970 tx=0x7f22d400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:35.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.119+0000 7f22c3fff700 1 -- 172.21.15.104:0/612855336 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d4019070 con 0x7f22dc108530 2024-06-09T22:16:35.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.120+0000 7f22c3fff700 1 -- 172.21.15.104:0/612855336 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22d400ce10 con 0x7f22dc108530 2024-06-09T22:16:35.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.120+0000 7f22e101d700 1 -- 172.21.15.104:0/612855336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22dc1994f0 con 0x7f22dc108530 2024-06-09T22:16:35.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.120+0000 7f22c3fff700 1 -- 172.21.15.104:0/612855336 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d4007370 con 0x7f22dc108530 2024-06-09T22:16:35.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.120+0000 7f22e101d700 1 -- 172.21.15.104:0/612855336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22dc1013c0 con 0x7f22dc108530 2024-06-09T22:16:35.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.120+0000 7f22e101d700 1 -- 172.21.15.104:0/612855336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22dc10aa30 con 0x7f22dc108530 2024-06-09T22:16:35.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.122+0000 7f22c3fff700 1 -- 172.21.15.104:0/612855336 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f22d401c070 con 0x7f22dc108530 2024-06-09T22:16:35.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.122+0000 7f22c3fff700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f22c40a2c20 0x7f22c40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:35.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.123+0000 7f22dad9d700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f22c40a2c20 0x7f22c40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:35.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.123+0000 7f22c3fff700 1 -- 172.21.15.104:0/612855336 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f22d40cdf70 con 0x7f22dc108530 2024-06-09T22:16:35.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.123+0000 7f22dad9d700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f22c40a2c20 0x7f22c40a50e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f22cc006010 tx=0x7f22cc00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:35.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.128+0000 7f22c3fff700 1 -- 172.21.15.104:0/612855336 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f22d409b370 con 0x7f22dc108530 2024-06-09T22:16:35.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.340+0000 7f22e101d700 1 -- 172.21.15.104:0/612855336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f22dc199fd0 con 0x7f22dc108530 2024-06-09T22:16:35.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.341+0000 7f22c3fff700 1 -- 172.21.15.104:0/612855336 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f22d409ac70 con 0x7f22dc108530 2024-06-09T22:16:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:16:35.337 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2024-06-09T22:16:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:16:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2024-06-09T22:16:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2024-06-09T22:16:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:16:35.337 INFO:teuthology.orchestra.run.smithi104.stdout:] 2024-06-09T22:16:35.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.343+0000 7f22e101d700 1 -- 172.21.15.104:0/612855336 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f22c40a2c20 msgr2=0x7f22c40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:35.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.343+0000 7f22e101d700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f22c40a2c20 0x7f22c40a50e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f22cc006010 tx=0x7f22cc00b540 comp rx=0 tx=0).stop 2024-06-09T22:16:35.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.343+0000 7f22e101d700 1 -- 172.21.15.104:0/612855336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22dc108530 msgr2=0x7f22dc198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:35.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.343+0000 7f22e101d700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22dc108530 0x7f22dc198aa0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f22d400b970 tx=0x7f22d400bc80 comp rx=0 tx=0).stop 2024-06-09T22:16:35.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.343+0000 7f22e101d700 1 -- 172.21.15.104:0/612855336 shutdown_connections 2024-06-09T22:16:35.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.343+0000 7f22e101d700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f22c40a2c20 0x7f22c40a50e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:35.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.343+0000 7f22e101d700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22dc108530 0x7f22dc198aa0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:35.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.343+0000 7f22e101d700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22dc06ac00 0x7f22dc19e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:35.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.343+0000 7f22e101d700 1 --2- 172.21.15.104:0/612855336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f22dc06b5c0 0x7f22dc19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:35.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.343+0000 7f22e101d700 1 -- 172.21.15.104:0/612855336 >> 172.21.15.104:0/612855336 conn(0x7f22dc077400 msgr2=0x7f22dc0ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:35.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.344+0000 7f22e101d700 1 -- 172.21.15.104:0/612855336 shutdown_connections 2024-06-09T22:16:35.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:35.344+0000 7f22e101d700 1 -- 172.21.15.104:0/612855336 wait complete. 2024-06-09T22:16:35.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:35 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/612855336' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-06-09T22:16:35.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:35 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/612855336' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-06-09T22:16:35.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:35 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/612855336' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-06-09T22:16:35.992 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-09T22:16:35.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-09T22:16:36.241 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:36.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:36 smithi104 ceph-mon[32389]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:36.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:36.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:36.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:36.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:16:36.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:36.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:36 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:16:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:36 smithi153 ceph-mon[36599]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:36 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:36.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:36 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:36 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:36 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:16:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:36 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:36 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:16:36.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:36 smithi194 ceph-mon[36856]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:36.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:36 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:36.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:36 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:36.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:36 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:36.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:36 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:16:36.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:36 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:36.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:36 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:16:37.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.913+0000 7fbd217c7700 1 -- 172.21.15.104:0/259747353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd1c102740 msgr2=0x7fbd1c102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.913+0000 7fbd217c7700 1 --2- 172.21.15.104:0/259747353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd1c102740 0x7fbd1c102b20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0099c0 tx=0x7fbd0c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:16:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.913+0000 7fbd217c7700 1 -- 172.21.15.104:0/259747353 shutdown_connections 2024-06-09T22:16:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.913+0000 7fbd217c7700 1 --2- 172.21.15.104:0/259747353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd1c102740 0x7fbd1c102b20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.913+0000 7fbd217c7700 1 --2- 172.21.15.104:0/259747353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1c107840 0x7fbd1c10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.913+0000 7fbd217c7700 1 --2- 172.21.15.104:0/259747353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd1c1030f0 0x7fbd1c107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.913+0000 7fbd217c7700 1 -- 172.21.15.104:0/259747353 >> 172.21.15.104:0/259747353 conn(0x7fbd1c0795f0 msgr2=0x7fbd1c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.913+0000 7fbd217c7700 1 -- 172.21.15.104:0/259747353 shutdown_connections 2024-06-09T22:16:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.913+0000 7fbd217c7700 1 -- 172.21.15.104:0/259747353 wait complete. 2024-06-09T22:16:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.913+0000 7fbd217c7700 1 Processor -- start 2024-06-09T22:16:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.913+0000 7fbd217c7700 1 -- start start 2024-06-09T22:16:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd217c7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd1c102740 0x7fbd1c19e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd217c7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd1c1030f0 0x7fbd1c19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:37.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd217c7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1c107840 0x7fbd1c198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:37.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd217c7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c112760 con 0x7fbd1c1030f0 2024-06-09T22:16:37.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd217c7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c1125e0 con 0x7fbd1c107840 2024-06-09T22:16:37.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd217c7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c1128e0 con 0x7fbd1c102740 2024-06-09T22:16:37.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd1a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd1c1030f0 0x7fbd1c19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:37.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd1a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd1c1030f0 0x7fbd1c19ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37704/0 (socket says 172.21.15.104:37704) 2024-06-09T22:16:37.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd1a7fc700 1 -- 172.21.15.104:0/1234471141 learned_addr learned my addr 172.21.15.104:0/1234471141 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:37.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd1affd700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd1c102740 0x7fbd1c19e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:37.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd1b7fe700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1c107840 0x7fbd1c198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:37.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd1a7fc700 1 -- 172.21.15.104:0/1234471141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd1c102740 msgr2=0x7fbd1c19e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:37.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd1a7fc700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd1c102740 0x7fbd1c19e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd1a7fc700 1 -- 172.21.15.104:0/1234471141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1c107840 msgr2=0x7fbd1c198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd1a7fc700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1c107840 0x7fbd1c198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd1a7fc700 1 -- 172.21.15.104:0/1234471141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1c199160 con 0x7fbd1c1030f0 2024-06-09T22:16:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd1affd700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd1c102740 0x7fbd1c19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:16:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd1a7fc700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd1c1030f0 0x7fbd1c19ed30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000e9a0 tx=0x7fbd1000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd03fff700 1 -- 172.21.15.104:0/1234471141 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1000ca10 con 0x7fbd1c1030f0 2024-06-09T22:16:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd217c7700 1 -- 172.21.15.104:0/1234471141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd1c199450 con 0x7fbd1c1030f0 2024-06-09T22:16:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd03fff700 1 -- 172.21.15.104:0/1234471141 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd1000cbb0 con 0x7fbd1c1030f0 2024-06-09T22:16:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.914+0000 7fbd03fff700 1 -- 172.21.15.104:0/1234471141 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd10018680 con 0x7fbd1c1030f0 2024-06-09T22:16:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.915+0000 7fbd217c7700 1 -- 172.21.15.104:0/1234471141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd1c1a56c0 con 0x7fbd1c1030f0 2024-06-09T22:16:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.915+0000 7fbd217c7700 1 -- 172.21.15.104:0/1234471141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd1c10aa80 con 0x7fbd1c1030f0 2024-06-09T22:16:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.917+0000 7fbd03fff700 1 -- 172.21.15.104:0/1234471141 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbd10010430 con 0x7fbd1c1030f0 2024-06-09T22:16:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.917+0000 7fbd03fff700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbd040a2c20 0x7fbd040a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.918+0000 7fbd1affd700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbd040a2c20 0x7fbd040a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.918+0000 7fbd03fff700 1 -- 172.21.15.104:0/1234471141 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbd100cede0 con 0x7fbd1c1030f0 2024-06-09T22:16:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.918+0000 7fbd1affd700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbd040a2c20 0x7fbd040a50e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c000c00 tx=0x7fbd0c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:37.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:37.922+0000 7fbd03fff700 1 -- 172.21.15.104:0/1234471141 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbd1009c1e0 con 0x7fbd1c1030f0 2024-06-09T22:16:38.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:38.127+0000 7fbd217c7700 1 -- 172.21.15.104:0/1234471141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fbd1c069180 con 0x7fbd1c1030f0 2024-06-09T22:16:38.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:38 smithi153 ceph-mon[36599]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:38.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:38 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:38.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:38 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:38.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:38 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:38.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:38 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:38.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:38 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1234471141' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-06-09T22:16:38.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:38 smithi104 ceph-mon[32389]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:38.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:38 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:38.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:38 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:38.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:38 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:38.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:38 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:38.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:38 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1234471141' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-06-09T22:16:38.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:38 smithi194 ceph-mon[36856]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:38.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:38 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:38.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:38 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:38.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:38 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:38.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:38 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:38.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:38 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1234471141' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-06-09T22:16:39.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.027+0000 7fbd03fff700 1 -- 172.21.15.104:0/1234471141 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fbd1009bae0 con 0x7fbd1c1030f0 2024-06-09T22:16:39.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.029+0000 7fbd217c7700 1 -- 172.21.15.104:0/1234471141 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbd040a2c20 msgr2=0x7fbd040a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:39.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.029+0000 7fbd217c7700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbd040a2c20 0x7fbd040a50e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c000c00 tx=0x7fbd0c005fb0 comp rx=0 tx=0).stop 2024-06-09T22:16:39.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.029+0000 7fbd217c7700 1 -- 172.21.15.104:0/1234471141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd1c1030f0 msgr2=0x7fbd1c19ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:39.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.029+0000 7fbd217c7700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd1c1030f0 0x7fbd1c19ed30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000e9a0 tx=0x7fbd1000ecb0 comp rx=0 tx=0).stop 2024-06-09T22:16:39.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.029+0000 7fbd217c7700 1 -- 172.21.15.104:0/1234471141 shutdown_connections 2024-06-09T22:16:39.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.029+0000 7fbd217c7700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbd040a2c20 0x7fbd040a50e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:39.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.029+0000 7fbd217c7700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1c107840 0x7fbd1c198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:39.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.029+0000 7fbd217c7700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd1c102740 0x7fbd1c19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:39.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.029+0000 7fbd217c7700 1 --2- 172.21.15.104:0/1234471141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd1c1030f0 0x7fbd1c19ed30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:39.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.029+0000 7fbd217c7700 1 -- 172.21.15.104:0/1234471141 >> 172.21.15.104:0/1234471141 conn(0x7fbd1c0795f0 msgr2=0x7fbd1c101cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:39.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.029+0000 7fbd217c7700 1 -- 172.21.15.104:0/1234471141 shutdown_connections 2024-06-09T22:16:39.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:39.029+0000 7fbd217c7700 1 -- 172.21.15.104:0/1234471141 wait complete. 2024-06-09T22:16:39.026 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2024-06-09T22:16:39.705 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-06-09T22:16:39.940 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:40.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:40.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:40 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1234471141' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-06-09T22:16:40.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:40 smithi194 ceph-mon[36856]: osdmap e82: 12 total, 12 up, 12 in 2024-06-09T22:16:40.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:40.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:40.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:16:40.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:40 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:40.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:40 smithi194 ceph-mon[36856]: pgmap v289: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:40.391 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:40.391 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:40 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1234471141' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-06-09T22:16:40.391 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:40 smithi104 ceph-mon[32389]: osdmap e82: 12 total, 12 up, 12 in 2024-06-09T22:16:40.391 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:40.392 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:40.392 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:16:40.392 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:40 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:40.392 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:40 smithi104 ceph-mon[32389]: pgmap v289: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:40.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:40.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:40 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1234471141' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-06-09T22:16:40.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:40 smithi153 ceph-mon[36599]: osdmap e82: 12 total, 12 up, 12 in 2024-06-09T22:16:40.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:40.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:16:40.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:16:40.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:40 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:16:40.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:40 smithi153 ceph-mon[36599]: pgmap v289: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:41.154 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:41 smithi104 ceph-mon[32389]: osdmap e83: 12 total, 12 up, 12 in 2024-06-09T22:16:41.154 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:41 smithi104 ceph-mon[32389]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-09T22:16:41.304 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:41 smithi194 ceph-mon[36856]: osdmap e83: 12 total, 12 up, 12 in 2024-06-09T22:16:41.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:41 smithi194 ceph-mon[36856]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-09T22:16:41.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:41 smithi153 ceph-mon[36599]: osdmap e83: 12 total, 12 up, 12 in 2024-06-09T22:16:41.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:41 smithi153 ceph-mon[36599]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-09T22:16:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.691+0000 7f2f3d008700 1 -- 172.21.15.104:0/3330480212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f38107400 msgr2=0x7f2f381077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.691+0000 7f2f3d008700 1 --2- 172.21.15.104:0/3330480212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f38107400 0x7f2f381077e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009990 tx=0x7f2f20009ca0 comp rx=0 tx=0).stop 2024-06-09T22:16:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.691+0000 7f2f3d008700 1 -- 172.21.15.104:0/3330480212 shutdown_connections 2024-06-09T22:16:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.691+0000 7f2f3d008700 1 --2- 172.21.15.104:0/3330480212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f38107400 0x7f2f381077e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.691+0000 7f2f3d008700 1 --2- 172.21.15.104:0/3330480212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f38101d60 0x7f2f3810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.691+0000 7f2f3d008700 1 --2- 172.21.15.104:0/3330480212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f381013c0 0x7f2f38101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.691+0000 7f2f3d008700 1 -- 172.21.15.104:0/3330480212 >> 172.21.15.104:0/3330480212 conn(0x7f2f38078330 msgr2=0x7f2f38078740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.691+0000 7f2f3d008700 1 -- 172.21.15.104:0/3330480212 shutdown_connections 2024-06-09T22:16:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.691+0000 7f2f3d008700 1 -- 172.21.15.104:0/3330480212 wait complete. 2024-06-09T22:16:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.692+0000 7f2f3d008700 1 Processor -- start 2024-06-09T22:16:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.692+0000 7f2f3d008700 1 -- start start 2024-06-09T22:16:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.692+0000 7f2f3d008700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f381013c0 0x7f2f38103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.692+0000 7f2f3d008700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f38101d60 0x7f2f381035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.692+0000 7f2f3d008700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f38107400 0x7f2f38103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3d008700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2f38111640 con 0x7f2f38107400 2024-06-09T22:16:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3d008700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2f381114c0 con 0x7f2f381013c0 2024-06-09T22:16:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3d008700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2f381117c0 con 0x7f2f38101d60 2024-06-09T22:16:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f38107400 0x7f2f38103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f38107400 0x7f2f38103af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37720/0 (socket says 172.21.15.104:37720) 2024-06-09T22:16:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3759e700 1 -- 172.21.15.104:0/1256661703 learned_addr learned my addr 172.21.15.104:0/1256661703 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f36d9d700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f381013c0 0x7f2f38103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3659c700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f38101d60 0x7f2f381035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3759e700 1 -- 172.21.15.104:0/1256661703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f38101d60 msgr2=0x7f2f381035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3759e700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f38101d60 0x7f2f381035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3759e700 1 -- 172.21.15.104:0/1256661703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f381013c0 msgr2=0x7f2f38103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3759e700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f381013c0 0x7f2f38103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3759e700 1 -- 172.21.15.104:0/1256661703 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f381a7050 con 0x7f2f38107400 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f36d9d700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f381013c0 0x7f2f38103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3759e700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f38107400 0x7f2f38103af0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2f30007d70 tx=0x7f2f3000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f2ffff700 1 -- 172.21.15.104:0/1256661703 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30016020 con 0x7f2f38107400 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3d008700 1 -- 172.21.15.104:0/1256661703 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f381a7340 con 0x7f2f38107400 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f2ffff700 1 -- 172.21.15.104:0/1256661703 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f3000ad40 con 0x7f2f38107400 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f2ffff700 1 -- 172.21.15.104:0/1256661703 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f300116f0 con 0x7f2f38107400 2024-06-09T22:16:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.693+0000 7f2f3d008700 1 -- 172.21.15.104:0/1256661703 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f381a7830 con 0x7f2f38107400 2024-06-09T22:16:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.694+0000 7f2f3d008700 1 -- 172.21.15.104:0/1256661703 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f38102ab0 con 0x7f2f38107400 2024-06-09T22:16:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.695+0000 7f2f2ffff700 1 -- 172.21.15.104:0/1256661703 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f2f30019020 con 0x7f2f38107400 2024-06-09T22:16:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.696+0000 7f2f2ffff700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2f240a2c20 0x7f2f240a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.696+0000 7f2f36d9d700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2f240a2c20 0x7f2f240a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.696+0000 7f2f2ffff700 1 -- 172.21.15.104:0/1256661703 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7f2f300cf0f0 con 0x7f2f38107400 2024-06-09T22:16:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.697+0000 7f2f36d9d700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2f240a2c20 0x7f2f240a50e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2f20006010 tx=0x7f2f2000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:41.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.700+0000 7f2f2ffff700 1 -- 172.21.15.104:0/1256661703 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2f30020020 con 0x7f2f38107400 2024-06-09T22:16:41.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:41.897+0000 7f2f3d008700 1 -- 172.21.15.104:0/1256661703 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f2f38069180 con 0x7f2f38107400 2024-06-09T22:16:42.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.180+0000 7f2f2ffff700 1 -- 172.21.15.104:0/1256661703 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v85) ==== 171+0+0 (secure 0 0 0) 0x7f2f38069180 con 0x7f2f38107400 2024-06-09T22:16:42.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.182+0000 7f2f3d008700 1 -- 172.21.15.104:0/1256661703 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2f240a2c20 msgr2=0x7f2f240a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:42.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.182+0000 7f2f3d008700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2f240a2c20 0x7f2f240a50e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2f20006010 tx=0x7f2f2000b540 comp rx=0 tx=0).stop 2024-06-09T22:16:42.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.182+0000 7f2f3d008700 1 -- 172.21.15.104:0/1256661703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f38107400 msgr2=0x7f2f38103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:42.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.182+0000 7f2f3d008700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f38107400 0x7f2f38103af0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2f30007d70 tx=0x7f2f3000a340 comp rx=0 tx=0).stop 2024-06-09T22:16:42.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.182+0000 7f2f3d008700 1 -- 172.21.15.104:0/1256661703 shutdown_connections 2024-06-09T22:16:42.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.182+0000 7f2f3d008700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2f240a2c20 0x7f2f240a50e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:42.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.182+0000 7f2f3d008700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f38107400 0x7f2f38103af0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:42.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.182+0000 7f2f3d008700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f381013c0 0x7f2f38103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:42.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.182+0000 7f2f3d008700 1 --2- 172.21.15.104:0/1256661703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f38101d60 0x7f2f381035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:42.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.182+0000 7f2f3d008700 1 -- 172.21.15.104:0/1256661703 >> 172.21.15.104:0/1256661703 conn(0x7f2f38078330 msgr2=0x7f2f381005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:42.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.183+0000 7f2f3d008700 1 -- 172.21.15.104:0/1256661703 shutdown_connections 2024-06-09T22:16:42.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:42.183+0000 7f2f3d008700 1 -- 172.21.15.104:0/1256661703 wait complete. 2024-06-09T22:16:42.179 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2024-06-09T22:16:42.472 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:42 smithi104 ceph-mon[32389]: osdmap e84: 12 total, 12 up, 12 in 2024-06-09T22:16:42.472 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:42 smithi104 ceph-mon[32389]: pgmap v292: 33 pgs: 11 active+clean, 13 creating+peering, 9 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:42.472 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:42 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1256661703' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-06-09T22:16:42.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:42 smithi153 ceph-mon[36599]: osdmap e84: 12 total, 12 up, 12 in 2024-06-09T22:16:42.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:42 smithi153 ceph-mon[36599]: pgmap v292: 33 pgs: 11 active+clean, 13 creating+peering, 9 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:42.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:42 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1256661703' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-06-09T22:16:42.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:42 smithi194 ceph-mon[36856]: osdmap e84: 12 total, 12 up, 12 in 2024-06-09T22:16:42.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:42 smithi194 ceph-mon[36856]: pgmap v292: 33 pgs: 11 active+clean, 13 creating+peering, 9 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:42.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:42 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1256661703' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-06-09T22:16:42.835 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-09T22:16:43.078 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:43.443 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:43 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1256661703' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-06-09T22:16:43.444 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:43 smithi104 ceph-mon[32389]: osdmap e85: 12 total, 12 up, 12 in 2024-06-09T22:16:43.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:43 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1256661703' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-06-09T22:16:43.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:43 smithi153 ceph-mon[36599]: osdmap e85: 12 total, 12 up, 12 in 2024-06-09T22:16:43.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:43 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1256661703' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-06-09T22:16:43.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:43 smithi194 ceph-mon[36856]: osdmap e85: 12 total, 12 up, 12 in 2024-06-09T22:16:44.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:44 smithi153 ceph-mon[36599]: osdmap e86: 12 total, 12 up, 12 in 2024-06-09T22:16:44.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:44 smithi153 ceph-mon[36599]: pgmap v295: 97 pgs: 17 active+clean, 25 creating+peering, 55 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:44.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:44 smithi104 ceph-mon[32389]: osdmap e86: 12 total, 12 up, 12 in 2024-06-09T22:16:44.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:44 smithi104 ceph-mon[32389]: pgmap v295: 97 pgs: 17 active+clean, 25 creating+peering, 55 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:44.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:44 smithi194 ceph-mon[36856]: osdmap e86: 12 total, 12 up, 12 in 2024-06-09T22:16:44.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:44 smithi194 ceph-mon[36856]: pgmap v295: 97 pgs: 17 active+clean, 25 creating+peering, 55 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:44.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.905+0000 7f3f76d82700 1 -- 172.21.15.104:0/1196245494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f70101fd0 msgr2=0x7f3f70105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.905+0000 7f3f76d82700 1 --2- 172.21.15.104:0/1196245494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f70101fd0 0x7f3f70105fd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c0099c0 tx=0x7f3f5c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.905+0000 7f3f76d82700 1 -- 172.21.15.104:0/1196245494 shutdown_connections 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.905+0000 7f3f76d82700 1 --2- 172.21.15.104:0/1196245494 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f701066b0 0x7f3f7010e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.905+0000 7f3f76d82700 1 --2- 172.21.15.104:0/1196245494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f70101620 0x7f3f70101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.905+0000 7f3f76d82700 1 --2- 172.21.15.104:0/1196245494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f70101fd0 0x7f3f70105fd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.905+0000 7f3f76d82700 1 -- 172.21.15.104:0/1196245494 >> 172.21.15.104:0/1196245494 conn(0x7f3f70078340 msgr2=0x7f3f70078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.905+0000 7f3f76d82700 1 -- 172.21.15.104:0/1196245494 shutdown_connections 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.905+0000 7f3f76d82700 1 -- 172.21.15.104:0/1196245494 wait complete. 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.905+0000 7f3f76d82700 1 Processor -- start 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.905+0000 7f3f76d82700 1 -- start start 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.905+0000 7f3f76d82700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f70101620 0x7f3f70103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f76d82700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f70101fd0 0x7f3f701035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:44.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f76d82700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f701066b0 0x7f3f70103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:44.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f76d82700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3f70111670 con 0x7f3f70101fd0 2024-06-09T22:16:44.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f76d82700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3f701114f0 con 0x7f3f70101620 2024-06-09T22:16:44.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f76d82700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3f701117f0 con 0x7f3f701066b0 2024-06-09T22:16:44.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f6ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f70101fd0 0x7f3f701035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:44.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f6ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f70101fd0 0x7f3f701035b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37730/0 (socket says 172.21.15.104:37730) 2024-06-09T22:16:44.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f6ffff700 1 -- 172.21.15.104:0/777761605 learned_addr learned my addr 172.21.15.104:0/777761605 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:44.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f7531f700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f701066b0 0x7f3f70103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:44.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f74b1e700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f70101620 0x7f3f70103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:44.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f6ffff700 1 -- 172.21.15.104:0/777761605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f701066b0 msgr2=0x7f3f70103af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:44.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f6ffff700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f701066b0 0x7f3f70103af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:44.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f6ffff700 1 -- 172.21.15.104:0/777761605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f70101620 msgr2=0x7f3f70103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:44.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f6ffff700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f70101620 0x7f3f70103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:44.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f6ffff700 1 -- 172.21.15.104:0/777761605 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f701a7110 con 0x7f3f70101fd0 2024-06-09T22:16:44.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f74b1e700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f70101620 0x7f3f70103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:16:44.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f6ffff700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f70101fd0 0x7f3f701035b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c0059d0 tx=0x7f3f5c005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:44.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f6dffb700 1 -- 172.21.15.104:0/777761605 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c027070 con 0x7f3f70101fd0 2024-06-09T22:16:44.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f6dffb700 1 -- 172.21.15.104:0/777761605 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f5c0040d0 con 0x7f3f70101fd0 2024-06-09T22:16:44.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f76d82700 1 -- 172.21.15.104:0/777761605 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f701a73a0 con 0x7f3f70101fd0 2024-06-09T22:16:44.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.906+0000 7f3f6dffb700 1 -- 172.21.15.104:0/777761605 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c00f460 con 0x7f3f70101fd0 2024-06-09T22:16:44.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.907+0000 7f3f76d82700 1 -- 172.21.15.104:0/777761605 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f701a77f0 con 0x7f3f70101fd0 2024-06-09T22:16:44.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.907+0000 7f3f76d82700 1 -- 172.21.15.104:0/777761605 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f70069180 con 0x7f3f70101fd0 2024-06-09T22:16:44.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.909+0000 7f3f6dffb700 1 -- 172.21.15.104:0/777761605 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3f5c02a070 con 0x7f3f70101fd0 2024-06-09T22:16:44.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.909+0000 7f3f6dffb700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3f600a2c20 0x7f3f600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:44.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.909+0000 7f3f74b1e700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3f600a2c20 0x7f3f600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:44.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.909+0000 7f3f6dffb700 1 -- 172.21.15.104:0/777761605 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7879+0+0 (secure 0 0 0) 0x7f3f5c0d0510 con 0x7f3f70101fd0 2024-06-09T22:16:44.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.910+0000 7f3f74b1e700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3f600a2c20 0x7f3f600a50e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3f64005fd0 tx=0x7f3f64005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:44.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:44.913+0000 7f3f6dffb700 1 -- 172.21.15.104:0/777761605 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3f5c031020 con 0x7f3f70101fd0 2024-06-09T22:16:45.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.148+0000 7f3f76d82700 1 -- 172.21.15.104:0/777761605 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f3f7002ddc0 con 0x7f3f70101fd0 2024-06-09T22:16:45.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.198+0000 7f3f6dffb700 1 -- 172.21.15.104:0/777761605 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f3f5c013070 con 0x7f3f70101fd0 2024-06-09T22:16:45.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.201+0000 7f3f76d82700 1 -- 172.21.15.104:0/777761605 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3f600a2c20 msgr2=0x7f3f600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:45.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.201+0000 7f3f76d82700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3f600a2c20 0x7f3f600a50e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3f64005fd0 tx=0x7f3f64005dc0 comp rx=0 tx=0).stop 2024-06-09T22:16:45.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.201+0000 7f3f76d82700 1 -- 172.21.15.104:0/777761605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f70101fd0 msgr2=0x7f3f701035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:45.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.201+0000 7f3f76d82700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f70101fd0 0x7f3f701035b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c0059d0 tx=0x7f3f5c005640 comp rx=0 tx=0).stop 2024-06-09T22:16:45.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.201+0000 7f3f76d82700 1 -- 172.21.15.104:0/777761605 shutdown_connections 2024-06-09T22:16:45.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.201+0000 7f3f76d82700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3f600a2c20 0x7f3f600a50e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:45.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.201+0000 7f3f76d82700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f701066b0 0x7f3f70103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:45.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.201+0000 7f3f76d82700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f70101620 0x7f3f70103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:45.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.201+0000 7f3f76d82700 1 --2- 172.21.15.104:0/777761605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f70101fd0 0x7f3f701035b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:45.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.201+0000 7f3f76d82700 1 -- 172.21.15.104:0/777761605 >> 172.21.15.104:0/777761605 conn(0x7f3f70078340 msgr2=0x7f3f7006aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:45.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.201+0000 7f3f76d82700 1 -- 172.21.15.104:0/777761605 shutdown_connections 2024-06-09T22:16:45.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:45.201+0000 7f3f76d82700 1 -- 172.21.15.104:0/777761605 wait complete. 2024-06-09T22:16:45.198 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 2 and data pool 3 2024-06-09T22:16:45.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:45 smithi153 ceph-mon[36599]: osdmap e87: 12 total, 12 up, 12 in 2024-06-09T22:16:45.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:45 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/777761605' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-06-09T22:16:45.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:45 smithi153 ceph-mon[36599]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-06-09T22:16:45.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:45 smithi104 ceph-mon[32389]: osdmap e87: 12 total, 12 up, 12 in 2024-06-09T22:16:45.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:45 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/777761605' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-06-09T22:16:45.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:45 smithi104 ceph-mon[32389]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-06-09T22:16:45.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:45 smithi194 ceph-mon[36856]: osdmap e87: 12 total, 12 up, 12 in 2024-06-09T22:16:45.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:45 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/777761605' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-06-09T22:16:45.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:45 smithi194 ceph-mon[36856]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-06-09T22:16:45.892 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-06-09T22:16:45.892 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-06-09T22:16:46.132 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:46.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:46 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/777761605' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-06-09T22:16:46.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:46 smithi153 ceph-mon[36599]: osdmap e88: 12 total, 12 up, 12 in 2024-06-09T22:16:46.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:46 smithi153 ceph-mon[36599]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-06-09T22:16:46.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:46 smithi153 ceph-mon[36599]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-06-09T22:16:46.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:46 smithi153 ceph-mon[36599]: pgmap v298: 97 pgs: 67 active+clean, 12 creating+peering, 18 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:46.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:46 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/777761605' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-06-09T22:16:46.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:46 smithi104 ceph-mon[32389]: osdmap e88: 12 total, 12 up, 12 in 2024-06-09T22:16:46.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:46 smithi104 ceph-mon[32389]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-06-09T22:16:46.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:46 smithi104 ceph-mon[32389]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-06-09T22:16:46.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:46 smithi104 ceph-mon[32389]: pgmap v298: 97 pgs: 67 active+clean, 12 creating+peering, 18 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:46.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:46 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/777761605' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-06-09T22:16:46.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:46 smithi194 ceph-mon[36856]: osdmap e88: 12 total, 12 up, 12 in 2024-06-09T22:16:46.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:46 smithi194 ceph-mon[36856]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-06-09T22:16:46.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:46 smithi194 ceph-mon[36856]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-06-09T22:16:46.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:46 smithi194 ceph-mon[36856]: pgmap v298: 97 pgs: 67 active+clean, 12 creating+peering, 18 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:47.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:47 smithi153 ceph-mon[36599]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-06-09T22:16:47.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:47 smithi153 ceph-mon[36599]: Cluster is now healthy 2024-06-09T22:16:47.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:47 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] up:active 2024-06-09T22:16:47.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:47 smithi153 ceph-mon[36599]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-09T22:16:47.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:47 smithi153 ceph-mon[36599]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-09T22:16:47.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:47 smithi104 ceph-mon[32389]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-06-09T22:16:47.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:47 smithi104 ceph-mon[32389]: Cluster is now healthy 2024-06-09T22:16:47.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:47 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] up:active 2024-06-09T22:16:47.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:47 smithi104 ceph-mon[32389]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-09T22:16:47.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:47 smithi104 ceph-mon[32389]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-09T22:16:47.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:47 smithi194 ceph-mon[36856]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-06-09T22:16:47.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:47 smithi194 ceph-mon[36856]: Cluster is now healthy 2024-06-09T22:16:47.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:47 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] up:active 2024-06-09T22:16:47.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:47 smithi194 ceph-mon[36856]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-09T22:16:47.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:47 smithi194 ceph-mon[36856]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-09T22:16:47.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.908+0000 7f49082b2700 1 -- 172.21.15.104:0/1145937193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49001073f0 msgr2=0x7f49001077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:47.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.908+0000 7f49082b2700 1 --2- 172.21.15.104:0/1145937193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49001073f0 0x7f49001077d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00b0a0 tx=0x7f48fc00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:16:47.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.908+0000 7f49082b2700 1 -- 172.21.15.104:0/1145937193 shutdown_connections 2024-06-09T22:16:47.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.908+0000 7f49082b2700 1 --2- 172.21.15.104:0/1145937193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49001073f0 0x7f49001077d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:47.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.908+0000 7f49082b2700 1 --2- 172.21.15.104:0/1145937193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4900101d40 0x7f490010e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:47.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.908+0000 7f49082b2700 1 --2- 172.21.15.104:0/1145937193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4900101380 0x7f4900101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:47.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.908+0000 7f49082b2700 1 -- 172.21.15.104:0/1145937193 >> 172.21.15.104:0/1145937193 conn(0x7f4900078300 msgr2=0x7f4900078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:47.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.908+0000 7f49082b2700 1 -- 172.21.15.104:0/1145937193 shutdown_connections 2024-06-09T22:16:47.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.908+0000 7f49082b2700 1 -- 172.21.15.104:0/1145937193 wait complete. 2024-06-09T22:16:47.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.909+0000 7f49082b2700 1 Processor -- start 2024-06-09T22:16:47.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.909+0000 7f49082b2700 1 -- start start 2024-06-09T22:16:47.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.909+0000 7f49082b2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4900101380 0x7f4900199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:47.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.909+0000 7f49082b2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4900101d40 0x7f490019a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:47.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.909+0000 7f49082b2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49001073f0 0x7f49001a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:47.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.909+0000 7f49082b2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4900111690 con 0x7f4900101d40 2024-06-09T22:16:47.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.909+0000 7f49082b2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4900111510 con 0x7f4900101380 2024-06-09T22:16:47.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.909+0000 7f49082b2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4900111810 con 0x7f49001073f0 2024-06-09T22:16:47.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.909+0000 7f490584d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4900101d40 0x7f490019a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:47.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.909+0000 7f490584d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4900101d40 0x7f490019a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36626/0 (socket says 172.21.15.104:36626) 2024-06-09T22:16:47.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.909+0000 7f490584d700 1 -- 172.21.15.104:0/3479736193 learned_addr learned my addr 172.21.15.104:0/3479736193 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:47.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.909+0000 7f490684f700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49001073f0 0x7f49001a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:47.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f490604e700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4900101380 0x7f4900199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:47.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f490584d700 1 -- 172.21.15.104:0/3479736193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49001073f0 msgr2=0x7f49001a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:47.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f490584d700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49001073f0 0x7f49001a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:47.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f490584d700 1 -- 172.21.15.104:0/3479736193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4900101380 msgr2=0x7f4900199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:47.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f490584d700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4900101380 0x7f4900199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:47.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f490584d700 1 -- 172.21.15.104:0/3479736193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49001a5090 con 0x7f4900101d40 2024-06-09T22:16:47.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f490604e700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4900101380 0x7f4900199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:16:47.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f490584d700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4900101d40 0x7f490019a010 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f48f000d980 tx=0x7f48f000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:47.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f48f77fe700 1 -- 172.21.15.104:0/3479736193 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f0017070 con 0x7f4900101d40 2024-06-09T22:16:47.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f49082b2700 1 -- 172.21.15.104:0/3479736193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49001a5320 con 0x7f4900101d40 2024-06-09T22:16:47.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f48f77fe700 1 -- 172.21.15.104:0/3479736193 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48f0004d10 con 0x7f4900101d40 2024-06-09T22:16:47.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f48f77fe700 1 -- 172.21.15.104:0/3479736193 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f00053c0 con 0x7f4900101d40 2024-06-09T22:16:47.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.910+0000 7f49082b2700 1 -- 172.21.15.104:0/3479736193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49001a5830 con 0x7f4900101d40 2024-06-09T22:16:47.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.911+0000 7f49082b2700 1 -- 172.21.15.104:0/3479736193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4900069180 con 0x7f4900101d40 2024-06-09T22:16:47.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.912+0000 7f48f77fe700 1 -- 172.21.15.104:0/3479736193 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f48f0005560 con 0x7f4900101d40 2024-06-09T22:16:47.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.913+0000 7f48f77fe700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f48ec0a2c20 0x7f48ec0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:47.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.913+0000 7f490604e700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f48ec0a2c20 0x7f48ec0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:47.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.913+0000 7f48f77fe700 1 -- 172.21.15.104:0/3479736193 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f48f00cedc0 con 0x7f4900101d40 2024-06-09T22:16:47.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.913+0000 7f490604e700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f48ec0a2c20 0x7f48ec0a50e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009e60 tx=0x7f48fc00b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:47.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:47.917+0000 7f48f77fe700 1 -- 172.21.15.104:0/3479736193 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f48f0020020 con 0x7f4900101d40 2024-06-09T22:16:48.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.132+0000 7f49082b2700 1 -- 172.21.15.104:0/3479736193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f490002dca0 con 0x7f4900101d40 2024-06-09T22:16:48.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.224+0000 7f48f77fe700 1 -- 172.21.15.104:0/3479736193 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v89) ==== 157+0+0 (secure 0 0 0) 0x7f48f009be70 con 0x7f4900101d40 2024-06-09T22:16:48.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.226+0000 7f49082b2700 1 -- 172.21.15.104:0/3479736193 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f48ec0a2c20 msgr2=0x7f48ec0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:48.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.226+0000 7f49082b2700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f48ec0a2c20 0x7f48ec0a50e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009e60 tx=0x7f48fc00b7b0 comp rx=0 tx=0).stop 2024-06-09T22:16:48.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.226+0000 7f49082b2700 1 -- 172.21.15.104:0/3479736193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4900101d40 msgr2=0x7f490019a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:48.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.226+0000 7f49082b2700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4900101d40 0x7f490019a010 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f48f000d980 tx=0x7f48f000dd40 comp rx=0 tx=0).stop 2024-06-09T22:16:48.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.226+0000 7f49082b2700 1 -- 172.21.15.104:0/3479736193 shutdown_connections 2024-06-09T22:16:48.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.226+0000 7f49082b2700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f48ec0a2c20 0x7f48ec0a50e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:48.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.226+0000 7f49082b2700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49001073f0 0x7f49001a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:48.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.226+0000 7f49082b2700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4900101380 0x7f4900199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:48.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.226+0000 7f49082b2700 1 --2- 172.21.15.104:0/3479736193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4900101d40 0x7f490019a010 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:48.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.226+0000 7f49082b2700 1 -- 172.21.15.104:0/3479736193 >> 172.21.15.104:0/3479736193 conn(0x7f4900078300 msgr2=0x7f4900100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:48.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.226+0000 7f49082b2700 1 -- 172.21.15.104:0/3479736193 shutdown_connections 2024-06-09T22:16:48.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:48.226+0000 7f49082b2700 1 -- 172.21.15.104:0/3479736193 wait complete. 2024-06-09T22:16:48.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:48 smithi153 ceph-mon[36599]: pgmap v299: 97 pgs: 85 active+clean, 12 creating+peering; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 399 B/s wr, 5 op/s 2024-06-09T22:16:48.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:48 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3479736193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-06-09T22:16:48.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:48 smithi104 ceph-mon[32389]: pgmap v299: 97 pgs: 85 active+clean, 12 creating+peering; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 399 B/s wr, 5 op/s 2024-06-09T22:16:48.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:48 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3479736193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-06-09T22:16:48.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:48 smithi194 ceph-mon[36856]: pgmap v299: 97 pgs: 85 active+clean, 12 creating+peering; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 399 B/s wr, 5 op/s 2024-06-09T22:16:48.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:48 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3479736193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-06-09T22:16:49.008 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-06-09T22:16:49.251 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:49.252 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:49 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3479736193' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-06-09T22:16:49.252 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:49 smithi104 ceph-mon[32389]: osdmap e89: 12 total, 12 up, 12 in 2024-06-09T22:16:49.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:49 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3479736193' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-06-09T22:16:49.488 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:49 smithi153 ceph-mon[36599]: osdmap e89: 12 total, 12 up, 12 in 2024-06-09T22:16:49.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:49 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3479736193' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-06-09T22:16:49.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:49 smithi194 ceph-mon[36856]: osdmap e89: 12 total, 12 up, 12 in 2024-06-09T22:16:50.489 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:50 smithi153 ceph-mon[36599]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2024-06-09T22:16:50.492 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:50 smithi104 ceph-mon[32389]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2024-06-09T22:16:50.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:50 smithi194 ceph-mon[36856]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2024-06-09T22:16:50.555 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:16:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i[67699]: 2024-06-09T22:16:50.364+0000 7f60c2591700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-09T22:16:51.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.052+0000 7f6ac8e66700 1 -- 172.21.15.104:0/1853463142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ac41013d0 msgr2=0x7f6ac4101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:51.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.052+0000 7f6ac8e66700 1 --2- 172.21.15.104:0/1853463142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ac41013d0 0x7f6ac4101850 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8009990 tx=0x7f6ab8009ca0 comp rx=0 tx=0).stop 2024-06-09T22:16:51.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.052+0000 7f6ac8e66700 1 -- 172.21.15.104:0/1853463142 shutdown_connections 2024-06-09T22:16:51.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.052+0000 7f6ac8e66700 1 --2- 172.21.15.104:0/1853463142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac4101d90 0x7f6ac410e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:51.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.052+0000 7f6ac8e66700 1 --2- 172.21.15.104:0/1853463142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ac4107480 0x7f6ac4107860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:51.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.052+0000 7f6ac8e66700 1 --2- 172.21.15.104:0/1853463142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ac41013d0 0x7f6ac4101850 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:51.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.052+0000 7f6ac8e66700 1 -- 172.21.15.104:0/1853463142 >> 172.21.15.104:0/1853463142 conn(0x7f6ac4078420 msgr2=0x7f6ac4078830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:51.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.052+0000 7f6ac8e66700 1 -- 172.21.15.104:0/1853463142 shutdown_connections 2024-06-09T22:16:51.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.052+0000 7f6ac8e66700 1 -- 172.21.15.104:0/1853463142 wait complete. 2024-06-09T22:16:51.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.052+0000 7f6ac8e66700 1 Processor -- start 2024-06-09T22:16:51.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac8e66700 1 -- start start 2024-06-09T22:16:51.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac8e66700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ac41013d0 0x7f6ac419d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:51.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac8e66700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ac4101d90 0x7f6ac419dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:51.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac8e66700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac4107480 0x7f6ac41977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:51.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac8e66700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41116b0 con 0x7f6ac4101d90 2024-06-09T22:16:51.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac8e66700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac4111530 con 0x7f6ac4107480 2024-06-09T22:16:51.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac8e66700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac4111830 con 0x7f6ac41013d0 2024-06-09T22:16:51.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ac4101d90 0x7f6ac419dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:51.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ac4101d90 0x7f6ac419dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36634/0 (socket says 172.21.15.104:36634) 2024-06-09T22:16:51.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac1d9b700 1 -- 172.21.15.104:0/2739910131 learned_addr learned my addr 172.21.15.104:0/2739910131 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:51.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac2d9d700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac4107480 0x7f6ac41977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:51.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac259c700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ac41013d0 0x7f6ac419d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:51.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac1d9b700 1 -- 172.21.15.104:0/2739910131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ac41013d0 msgr2=0x7f6ac419d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:51.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac1d9b700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ac41013d0 0x7f6ac419d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:51.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac1d9b700 1 -- 172.21.15.104:0/2739910131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac4107480 msgr2=0x7f6ac41977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:51.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac1d9b700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac4107480 0x7f6ac41977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:51.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac1d9b700 1 -- 172.21.15.104:0/2739910131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac4197fd0 con 0x7f6ac4101d90 2024-06-09T22:16:51.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac259c700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ac41013d0 0x7f6ac419d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:16:51.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.053+0000 7f6ac1d9b700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ac4101d90 0x7f6ac419dbb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8000c00 tx=0x7f6ab80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:51.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.054+0000 7f6ab37fe700 1 -- 172.21.15.104:0/2739910131 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8024070 con 0x7f6ac4101d90 2024-06-09T22:16:51.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.054+0000 7f6ac8e66700 1 -- 172.21.15.104:0/2739910131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac4198260 con 0x7f6ac4101d90 2024-06-09T22:16:51.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.054+0000 7f6ab37fe700 1 -- 172.21.15.104:0/2739910131 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ab8003780 con 0x7f6ac4101d90 2024-06-09T22:16:51.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.054+0000 7f6ab37fe700 1 -- 172.21.15.104:0/2739910131 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8020440 con 0x7f6ac4101d90 2024-06-09T22:16:51.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.054+0000 7f6ac8e66700 1 -- 172.21.15.104:0/2739910131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac406abe0 con 0x7f6ac4101d90 2024-06-09T22:16:51.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.055+0000 7f6ac8e66700 1 -- 172.21.15.104:0/2739910131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ac4102ac0 con 0x7f6ac4101d90 2024-06-09T22:16:51.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.056+0000 7f6ab37fe700 1 -- 172.21.15.104:0/2739910131 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6ab8027070 con 0x7f6ac4101d90 2024-06-09T22:16:51.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.056+0000 7f6ab37fe700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6aac0a2cc0 0x7f6aac0a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:51.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.056+0000 7f6ab37fe700 1 -- 172.21.15.104:0/2739910131 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8277+0+0 (secure 0 0 0) 0x7f6ab80d2210 con 0x7f6ac4101d90 2024-06-09T22:16:51.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.056+0000 7f6ac259c700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6aac0a2cc0 0x7f6aac0a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:51.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.057+0000 7f6ac259c700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6aac0a2cc0 0x7f6aac0a5180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400bcf0 tx=0x7f6ab400b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:51.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.061+0000 7f6ab37fe700 1 -- 172.21.15.104:0/2739910131 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6ab8013070 con 0x7f6ac4101d90 2024-06-09T22:16:51.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:51.261+0000 7f6ac8e66700 1 -- 172.21.15.104:0/2739910131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6ac406ae40 con 0x7f6ac4101d90 2024-06-09T22:16:51.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:51 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2739910131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-06-09T22:16:51.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:51 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2739910131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-06-09T22:16:52.002 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:51 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2739910131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-06-09T22:16:52.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.316+0000 7f6ab37fe700 1 -- 172.21.15.104:0/2739910131 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v91) ==== 228+0+0 (secure 0 0 0) 0x7f6ab809f1d0 con 0x7f6ac4101d90 2024-06-09T22:16:52.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.320+0000 7f6ac8e66700 1 -- 172.21.15.104:0/2739910131 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6aac0a2cc0 msgr2=0x7f6aac0a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:52.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.320+0000 7f6ac8e66700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6aac0a2cc0 0x7f6aac0a5180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400bcf0 tx=0x7f6ab400b2c0 comp rx=0 tx=0).stop 2024-06-09T22:16:52.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.320+0000 7f6ac8e66700 1 -- 172.21.15.104:0/2739910131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ac4101d90 msgr2=0x7f6ac419dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:52.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.320+0000 7f6ac8e66700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ac4101d90 0x7f6ac419dbb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8000c00 tx=0x7f6ab80047b0 comp rx=0 tx=0).stop 2024-06-09T22:16:52.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.320+0000 7f6ac8e66700 1 -- 172.21.15.104:0/2739910131 shutdown_connections 2024-06-09T22:16:52.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.320+0000 7f6ac8e66700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6aac0a2cc0 0x7f6aac0a5180 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:52.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.320+0000 7f6ac8e66700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac4107480 0x7f6ac41977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:52.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.320+0000 7f6ac8e66700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ac41013d0 0x7f6ac419d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:52.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.320+0000 7f6ac8e66700 1 --2- 172.21.15.104:0/2739910131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ac4101d90 0x7f6ac419dbb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:52.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.320+0000 7f6ac8e66700 1 -- 172.21.15.104:0/2739910131 >> 172.21.15.104:0/2739910131 conn(0x7f6ac4078420 msgr2=0x7f6ac40ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:52.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.323+0000 7f6ac8e66700 1 -- 172.21.15.104:0/2739910131 shutdown_connections 2024-06-09T22:16:52.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:52.324+0000 7f6ac8e66700 1 -- 172.21.15.104:0/2739910131 wait complete. 2024-06-09T22:16:52.323 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data_ec' created 2024-06-09T22:16:52.657 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:52 smithi104 ceph-mon[32389]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-06-09T22:16:52.658 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:52 smithi104 ceph-mon[32389]: osdmap e90: 12 total, 12 up, 12 in 2024-06-09T22:16:52.658 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:52 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2739910131' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-06-09T22:16:52.658 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:52 smithi104 ceph-mon[32389]: osdmap e91: 12 total, 12 up, 12 in 2024-06-09T22:16:52.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:52 smithi194 ceph-mon[36856]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-06-09T22:16:52.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:52 smithi194 ceph-mon[36856]: osdmap e90: 12 total, 12 up, 12 in 2024-06-09T22:16:52.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:52 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2739910131' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-06-09T22:16:52.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:52 smithi194 ceph-mon[36856]: osdmap e91: 12 total, 12 up, 12 in 2024-06-09T22:16:52.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:52 smithi153 ceph-mon[36599]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-06-09T22:16:52.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:52 smithi153 ceph-mon[36599]: osdmap e90: 12 total, 12 up, 12 in 2024-06-09T22:16:52.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:52 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2739910131' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-06-09T22:16:52.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:52 smithi153 ceph-mon[36599]: osdmap e91: 12 total, 12 up, 12 in 2024-06-09T22:16:53.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-06-09T22:16:53.340 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:54.452 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:54 smithi104 ceph-mon[32389]: pgmap v305: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 2 op/s 2024-06-09T22:16:54.452 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:54 smithi104 ceph-mon[32389]: osdmap e92: 12 total, 12 up, 12 in 2024-06-09T22:16:54.452 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:54 smithi104 ceph-mon[32389]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-09T22:16:54.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:54 smithi153 ceph-mon[36599]: pgmap v305: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 2 op/s 2024-06-09T22:16:54.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:54 smithi153 ceph-mon[36599]: osdmap e92: 12 total, 12 up, 12 in 2024-06-09T22:16:54.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:54 smithi153 ceph-mon[36599]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-09T22:16:54.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:54 smithi194 ceph-mon[36856]: pgmap v305: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 2 op/s 2024-06-09T22:16:54.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:54 smithi194 ceph-mon[36856]: osdmap e92: 12 total, 12 up, 12 in 2024-06-09T22:16:54.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:54 smithi194 ceph-mon[36856]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-09T22:16:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.026+0000 7f0dc57bf700 1 -- 172.21.15.104:0/307448796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0107500 msgr2=0x7f0dc01078e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.026+0000 7f0dc57bf700 1 --2- 172.21.15.104:0/307448796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0107500 0x7f0dc01078e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0db0009990 tx=0x7f0db0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:16:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.026+0000 7f0dc57bf700 1 -- 172.21.15.104:0/307448796 shutdown_connections 2024-06-09T22:16:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.026+0000 7f0dc57bf700 1 --2- 172.21.15.104:0/307448796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0107500 0x7f0dc01078e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.026+0000 7f0dc57bf700 1 --2- 172.21.15.104:0/307448796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc0102070 0x7f0dc010e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.026+0000 7f0dc57bf700 1 --2- 172.21.15.104:0/307448796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dc01016b0 0x7f0dc0101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.026+0000 7f0dc57bf700 1 -- 172.21.15.104:0/307448796 >> 172.21.15.104:0/307448796 conn(0x7f0dc00780f0 msgr2=0x7f0dc0078500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.026+0000 7f0dc57bf700 1 -- 172.21.15.104:0/307448796 shutdown_connections 2024-06-09T22:16:55.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.026+0000 7f0dc57bf700 1 -- 172.21.15.104:0/307448796 wait complete. 2024-06-09T22:16:55.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.026+0000 7f0dc57bf700 1 Processor -- start 2024-06-09T22:16:55.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dc57bf700 1 -- start start 2024-06-09T22:16:55.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dc57bf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc01016b0 0x7f0dc019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:55.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dc57bf700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dc0102070 0x7f0dc019ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:55.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dc57bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0107500 0x7f0dc0198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:55.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dc57bf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc01116c0 con 0x7f0dc0107500 2024-06-09T22:16:55.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dc57bf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc0111540 con 0x7f0dc01016b0 2024-06-09T22:16:55.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dc57bf700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc0111840 con 0x7f0dc0102070 2024-06-09T22:16:55.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dbf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0107500 0x7f0dc0198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:55.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dbf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0107500 0x7f0dc0198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36650/0 (socket says 172.21.15.104:36650) 2024-06-09T22:16:55.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dbf7fe700 1 -- 172.21.15.104:0/2204226845 learned_addr learned my addr 172.21.15.104:0/2204226845 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:55.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dbeffd700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc01016b0 0x7f0dc019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:55.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dbf7fe700 1 -- 172.21.15.104:0/2204226845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dc0102070 msgr2=0x7f0dc019ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:55.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0db67fc700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dc0102070 0x7f0dc019ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:55.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dbf7fe700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dc0102070 0x7f0dc019ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:55.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dbf7fe700 1 -- 172.21.15.104:0/2204226845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc01016b0 msgr2=0x7f0dc019e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:55.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dbf7fe700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc01016b0 0x7f0dc019e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:55.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.027+0000 7f0dbf7fe700 1 -- 172.21.15.104:0/2204226845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc01990d0 con 0x7f0dc0107500 2024-06-09T22:16:55.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.028+0000 7f0dbf7fe700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0107500 0x7f0dc0198970 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0db800dc70 tx=0x7f0db800df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:55.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.028+0000 7f0dbcff9700 1 -- 172.21.15.104:0/2204226845 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db8001e00 con 0x7f0dc0107500 2024-06-09T22:16:55.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.028+0000 7f0dbcff9700 1 -- 172.21.15.104:0/2204226845 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0db8018070 con 0x7f0dc0107500 2024-06-09T22:16:55.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.028+0000 7f0dc57bf700 1 -- 172.21.15.104:0/2204226845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc01993c0 con 0x7f0dc0107500 2024-06-09T22:16:55.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.028+0000 7f0dbcff9700 1 -- 172.21.15.104:0/2204226845 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db80125e0 con 0x7f0dc0107500 2024-06-09T22:16:55.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.028+0000 7f0dc57bf700 1 -- 172.21.15.104:0/2204226845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc0102e20 con 0x7f0dc0107500 2024-06-09T22:16:55.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.029+0000 7f0dc57bf700 1 -- 172.21.15.104:0/2204226845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc0069180 con 0x7f0dc0107500 2024-06-09T22:16:55.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.030+0000 7f0dbcff9700 1 -- 172.21.15.104:0/2204226845 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0db8004ad0 con 0x7f0dc0107500 2024-06-09T22:16:55.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.031+0000 7f0dbcff9700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0dac0a2c20 0x7f0dac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:55.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.031+0000 7f0dbeffd700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0dac0a2c20 0x7f0dac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:55.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.031+0000 7f0dbcff9700 1 -- 172.21.15.104:0/2204226845 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8770+0+0 (secure 0 0 0) 0x7f0db80cf770 con 0x7f0dc0107500 2024-06-09T22:16:55.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.031+0000 7f0dbeffd700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0dac0a2c20 0x7f0dac0a50e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0db000b5c0 tx=0x7f0db0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:55.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.035+0000 7f0dbcff9700 1 -- 172.21.15.104:0/2204226845 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0db809c4b0 con 0x7f0dc0107500 2024-06-09T22:16:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.235+0000 7f0dc57bf700 1 -- 172.21.15.104:0/2204226845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f0dc0103080 con 0x7f0dc0107500 2024-06-09T22:16:55.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.329+0000 7f0dbcff9700 1 -- 172.21.15.104:0/2204226845 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v94) ==== 169+0+0 (secure 0 0 0) 0x7f0db809bdb0 con 0x7f0dc0107500 2024-06-09T22:16:55.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.331+0000 7f0dc57bf700 1 -- 172.21.15.104:0/2204226845 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0dac0a2c20 msgr2=0x7f0dac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.331+0000 7f0dc57bf700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0dac0a2c20 0x7f0dac0a50e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0db000b5c0 tx=0x7f0db0005960 comp rx=0 tx=0).stop 2024-06-09T22:16:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.331+0000 7f0dc57bf700 1 -- 172.21.15.104:0/2204226845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0107500 msgr2=0x7f0dc0198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.331+0000 7f0dc57bf700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0107500 0x7f0dc0198970 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0db800dc70 tx=0x7f0db800df80 comp rx=0 tx=0).stop 2024-06-09T22:16:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.331+0000 7f0dc57bf700 1 -- 172.21.15.104:0/2204226845 shutdown_connections 2024-06-09T22:16:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.331+0000 7f0dc57bf700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0dac0a2c20 0x7f0dac0a50e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.331+0000 7f0dc57bf700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0107500 0x7f0dc0198970 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.331+0000 7f0dc57bf700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc01016b0 0x7f0dc019e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.331+0000 7f0dc57bf700 1 --2- 172.21.15.104:0/2204226845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dc0102070 0x7f0dc019ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.331+0000 7f0dc57bf700 1 -- 172.21.15.104:0/2204226845 >> 172.21.15.104:0/2204226845 conn(0x7f0dc00780f0 msgr2=0x7f0dc0100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.332+0000 7f0dc57bf700 1 -- 172.21.15.104:0/2204226845 shutdown_connections 2024-06-09T22:16:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:55.332+0000 7f0dc57bf700 1 -- 172.21.15.104:0/2204226845 wait complete. 2024-06-09T22:16:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:set pool 4 allow_ec_overwrites to true 2024-06-09T22:16:55.651 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:55 smithi104 ceph-mon[32389]: osdmap e93: 12 total, 12 up, 12 in 2024-06-09T22:16:55.651 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:55 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2204226845' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-06-09T22:16:55.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:55 smithi153 ceph-mon[36599]: osdmap e93: 12 total, 12 up, 12 in 2024-06-09T22:16:55.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:55 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2204226845' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-06-09T22:16:55.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:55 smithi194 ceph-mon[36856]: osdmap e93: 12 total, 12 up, 12 in 2024-06-09T22:16:55.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:55 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2204226845' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-06-09T22:16:55.996 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-06-09T22:16:56.237 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:56.614 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:56 smithi104 ceph-mon[32389]: pgmap v308: 129 pgs: 11 unknown, 118 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-06-09T22:16:56.614 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:56 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2204226845' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-06-09T22:16:56.614 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:56 smithi104 ceph-mon[32389]: osdmap e94: 12 total, 12 up, 12 in 2024-06-09T22:16:56.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:56 smithi153 ceph-mon[36599]: pgmap v308: 129 pgs: 11 unknown, 118 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-06-09T22:16:56.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:56 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2204226845' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-06-09T22:16:56.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:56 smithi153 ceph-mon[36599]: osdmap e94: 12 total, 12 up, 12 in 2024-06-09T22:16:56.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:56 smithi194 ceph-mon[36856]: pgmap v308: 129 pgs: 11 unknown, 118 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-06-09T22:16:56.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:56 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2204226845' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-06-09T22:16:56.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:56 smithi194 ceph-mon[36856]: osdmap e94: 12 total, 12 up, 12 in 2024-06-09T22:16:57.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.998+0000 7f0a899ef700 1 -- 172.21.15.104:0/3906954411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84108500 msgr2=0x7f0a841088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:57.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.998+0000 7f0a899ef700 1 --2- 172.21.15.104:0/3906954411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84108500 0x7f0a841088e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0a74009990 tx=0x7f0a74009ca0 comp rx=0 tx=0).stop 2024-06-09T22:16:57.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.998+0000 7f0a899ef700 1 -- 172.21.15.104:0/3906954411 shutdown_connections 2024-06-09T22:16:57.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.998+0000 7f0a899ef700 1 --2- 172.21.15.104:0/3906954411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84108500 0x7f0a841088e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:57.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.998+0000 7f0a899ef700 1 --2- 172.21.15.104:0/3906954411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a8406ab30 0x7f0a8410f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:57.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.998+0000 7f0a899ef700 1 --2- 172.21.15.104:0/3906954411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8406a170 0x7f0a8406a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:57.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.998+0000 7f0a899ef700 1 -- 172.21.15.104:0/3906954411 >> 172.21.15.104:0/3906954411 conn(0x7f0a84078340 msgr2=0x7f0a84078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:57.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.998+0000 7f0a899ef700 1 -- 172.21.15.104:0/3906954411 shutdown_connections 2024-06-09T22:16:57.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.998+0000 7f0a899ef700 1 -- 172.21.15.104:0/3906954411 wait complete. 2024-06-09T22:16:57.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a899ef700 1 Processor -- start 2024-06-09T22:16:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a899ef700 1 -- start start 2024-06-09T22:16:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a899ef700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8406a170 0x7f0a8419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a899ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a8406ab30 0x7f0a8419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a899ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a84108500 0x7f0a84198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a899ef700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0a84112850 con 0x7f0a8406ab30 2024-06-09T22:16:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a899ef700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a841126d0 con 0x7f0a84108500 2024-06-09T22:16:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a899ef700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0a841129d0 con 0x7f0a8406a170 2024-06-09T22:16:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a827fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a8406ab30 0x7f0a8419ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a827fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a8406ab30 0x7f0a8419ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36270/0 (socket says 172.21.15.104:36270) 2024-06-09T22:16:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a827fc700 1 -- 172.21.15.104:0/409914773 learned_addr learned my addr 172.21.15.104:0/409914773 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:16:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a827fc700 1 -- 172.21.15.104:0/409914773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8406a170 msgr2=0x7f0a8419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a827fc700 1 --2- 172.21.15.104:0/409914773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8406a170 0x7f0a8419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a827fc700 1 -- 172.21.15.104:0/409914773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a84108500 msgr2=0x7f0a84198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a827fc700 1 --2- 172.21.15.104:0/409914773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a84108500 0x7f0a84198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:57.999+0000 7f0a827fc700 1 -- 172.21.15.104:0/409914773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a841992b0 con 0x7f0a8406ab30 2024-06-09T22:16:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.000+0000 7f0a827fc700 1 --2- 172.21.15.104:0/409914773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a8406ab30 0x7f0a8419ee80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00d980 tx=0x7f0a6c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.000+0000 7f0a889ed700 1 -- 172.21.15.104:0/409914773 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c017070 con 0x7f0a8406ab30 2024-06-09T22:16:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.000+0000 7f0a889ed700 1 -- 172.21.15.104:0/409914773 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a6c004d10 con 0x7f0a8406ab30 2024-06-09T22:16:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.000+0000 7f0a899ef700 1 -- 172.21.15.104:0/409914773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a841995a0 con 0x7f0a8406ab30 2024-06-09T22:16:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.000+0000 7f0a889ed700 1 -- 172.21.15.104:0/409914773 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0055c0 con 0x7f0a8406ab30 2024-06-09T22:16:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.000+0000 7f0a899ef700 1 -- 172.21.15.104:0/409914773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a84101390 con 0x7f0a8406ab30 2024-06-09T22:16:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.001+0000 7f0a899ef700 1 -- 172.21.15.104:0/409914773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a84069180 con 0x7f0a8406ab30 2024-06-09T22:16:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.002+0000 7f0a889ed700 1 -- 172.21.15.104:0/409914773 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0a6c004750 con 0x7f0a8406ab30 2024-06-09T22:16:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.003+0000 7f0a889ed700 1 --2- 172.21.15.104:0/409914773 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0a700a2ca0 0x7f0a700a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:16:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.003+0000 7f0a82ffd700 1 --2- 172.21.15.104:0/409914773 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0a700a2ca0 0x7f0a700a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:16:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.003+0000 7f0a889ed700 1 -- 172.21.15.104:0/409914773 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8770+0+0 (secure 0 0 0) 0x7f0a6c0cf530 con 0x7f0a8406ab30 2024-06-09T22:16:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.003+0000 7f0a82ffd700 1 --2- 172.21.15.104:0/409914773 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0a700a2ca0 0x7f0a700a5160 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0a74005f50 tx=0x7f0a74005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:16:58.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.006+0000 7f0a889ed700 1 -- 172.21.15.104:0/409914773 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0a6c020020 con 0x7f0a8406ab30 2024-06-09T22:16:58.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.248+0000 7f0a899ef700 1 -- 172.21.15.104:0/409914773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f0a84199ff0 con 0x7f0a8406ab30 2024-06-09T22:16:58.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.333+0000 7f0a889ed700 1 -- 172.21.15.104:0/409914773 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f0a6c09c2f0 con 0x7f0a8406ab30 2024-06-09T22:16:58.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.336+0000 7f0a899ef700 1 -- 172.21.15.104:0/409914773 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0a700a2ca0 msgr2=0x7f0a700a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:58.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.336+0000 7f0a899ef700 1 --2- 172.21.15.104:0/409914773 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0a700a2ca0 0x7f0a700a5160 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0a74005f50 tx=0x7f0a74005e80 comp rx=0 tx=0).stop 2024-06-09T22:16:58.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.336+0000 7f0a899ef700 1 -- 172.21.15.104:0/409914773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a8406ab30 msgr2=0x7f0a8419ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:16:58.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.336+0000 7f0a899ef700 1 --2- 172.21.15.104:0/409914773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a8406ab30 0x7f0a8419ee80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00d980 tx=0x7f0a6c00dd40 comp rx=0 tx=0).stop 2024-06-09T22:16:58.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.336+0000 7f0a899ef700 1 -- 172.21.15.104:0/409914773 shutdown_connections 2024-06-09T22:16:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.336+0000 7f0a899ef700 1 --2- 172.21.15.104:0/409914773 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0a700a2ca0 0x7f0a700a5160 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.336+0000 7f0a899ef700 1 --2- 172.21.15.104:0/409914773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a84108500 0x7f0a84198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.336+0000 7f0a899ef700 1 --2- 172.21.15.104:0/409914773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a8406a170 0x7f0a8419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.336+0000 7f0a899ef700 1 --2- 172.21.15.104:0/409914773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a8406ab30 0x7f0a8419ee80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:16:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.336+0000 7f0a899ef700 1 -- 172.21.15.104:0/409914773 >> 172.21.15.104:0/409914773 conn(0x7f0a84078340 msgr2=0x7f0a840ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:16:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.336+0000 7f0a899ef700 1 -- 172.21.15.104:0/409914773 shutdown_connections 2024-06-09T22:16:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:16:58.336+0000 7f0a899ef700 1 -- 172.21.15.104:0/409914773 wait complete. 2024-06-09T22:16:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:added data pool 4 to fsmap 2024-06-09T22:16:58.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:58 smithi104 ceph-mon[32389]: pgmap v310: 129 pgs: 2 unknown, 127 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:58.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:58 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/409914773' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-06-09T22:16:58.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:58 smithi153 ceph-mon[36599]: pgmap v310: 129 pgs: 2 unknown, 127 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:58.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:58 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/409914773' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-06-09T22:16:58.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:58 smithi194 ceph-mon[36856]: pgmap v310: 129 pgs: 2 unknown, 127 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:16:58.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:58 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/409914773' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-06-09T22:16:59.026 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:16:59.270 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:16:59.639 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:59 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/409914773' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-06-09T22:16:59.639 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:59 smithi104 ceph-mon[32389]: osdmap e95: 12 total, 12 up, 12 in 2024-06-09T22:16:59.639 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:16:59 smithi104 ceph-mon[32389]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-09T22:16:59.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:59 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/409914773' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-06-09T22:16:59.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:59 smithi153 ceph-mon[36599]: osdmap e95: 12 total, 12 up, 12 in 2024-06-09T22:16:59.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:16:59 smithi153 ceph-mon[36599]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-09T22:16:59.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:59 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/409914773' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-06-09T22:16:59.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:59 smithi194 ceph-mon[36856]: osdmap e95: 12 total, 12 up, 12 in 2024-06-09T22:16:59.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:16:59 smithi194 ceph-mon[36856]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-09T22:17:00.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:00 smithi104 ceph-mon[32389]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:00.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:00 smithi104 ceph-mon[32389]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-09T22:17:00.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:00 smithi104 ceph-mon[32389]: Cluster is now healthy 2024-06-09T22:17:00.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:00 smithi153 ceph-mon[36599]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:00.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:00 smithi153 ceph-mon[36599]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-09T22:17:00.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:00 smithi153 ceph-mon[36599]: Cluster is now healthy 2024-06-09T22:17:00.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:00 smithi194 ceph-mon[36856]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:00.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:00 smithi194 ceph-mon[36856]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-09T22:17:00.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:00 smithi194 ceph-mon[36856]: Cluster is now healthy 2024-06-09T22:17:01.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.051+0000 7f0378a55700 1 -- 172.21.15.104:0/3475009291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03741019e0 msgr2=0x7f0374107120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:01.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.051+0000 7f0378a55700 1 --2- 172.21.15.104:0/3475009291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03741019e0 0x7f0374107120 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f03680099c0 tx=0x7f0368009cd0 comp rx=0 tx=0).stop 2024-06-09T22:17:01.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.051+0000 7f0378a55700 1 -- 172.21.15.104:0/3475009291 shutdown_connections 2024-06-09T22:17:01.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.051+0000 7f0378a55700 1 --2- 172.21.15.104:0/3475009291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0374107850 0x7f037410f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:01.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.051+0000 7f0378a55700 1 --2- 172.21.15.104:0/3475009291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0374101030 0x7f0374101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:01.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.051+0000 7f0378a55700 1 --2- 172.21.15.104:0/3475009291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03741019e0 0x7f0374107120 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:01.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.051+0000 7f0378a55700 1 -- 172.21.15.104:0/3475009291 >> 172.21.15.104:0/3475009291 conn(0x7f03740766d0 msgr2=0x7f0374076ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:01.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.051+0000 7f0378a55700 1 -- 172.21.15.104:0/3475009291 shutdown_connections 2024-06-09T22:17:01.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.051+0000 7f0378a55700 1 -- 172.21.15.104:0/3475009291 wait complete. 2024-06-09T22:17:01.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0378a55700 1 Processor -- start 2024-06-09T22:17:01.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0378a55700 1 -- start start 2024-06-09T22:17:01.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0378a55700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0374101030 0x7f0374198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:01.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0378a55700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03741019e0 0x7f03741992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:01.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0378a55700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0374107850 0x7f037419d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:01.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0378a55700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f03741128a0 con 0x7f03741019e0 2024-06-09T22:17:01.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0378a55700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0374112720 con 0x7f0374101030 2024-06-09T22:17:01.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0378a55700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0374112a20 con 0x7f0374107850 2024-06-09T22:17:01.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0371d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03741019e0 0x7f03741992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:01.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0371d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03741019e0 0x7f03741992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36298/0 (socket says 172.21.15.104:36298) 2024-06-09T22:17:01.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0371d9b700 1 -- 172.21.15.104:0/3532890712 learned_addr learned my addr 172.21.15.104:0/3532890712 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:01.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0371d9b700 1 -- 172.21.15.104:0/3532890712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0374107850 msgr2=0x7f037419d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:01.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0372d9d700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0374107850 0x7f037419d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:01.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f037259c700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0374101030 0x7f0374198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:01.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0371d9b700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0374107850 0x7f037419d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:01.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0371d9b700 1 -- 172.21.15.104:0/3532890712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0374101030 msgr2=0x7f0374198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:01.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0371d9b700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0374101030 0x7f0374198d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:01.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.052+0000 7f0371d9b700 1 -- 172.21.15.104:0/3532890712 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037419dd80 con 0x7f03741019e0 2024-06-09T22:17:01.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.053+0000 7f0371d9b700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03741019e0 0x7f03741992b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0368009fd0 tx=0x7f0368005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:01.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.053+0000 7f03637fe700 1 -- 172.21.15.104:0/3532890712 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368027070 con 0x7f03741019e0 2024-06-09T22:17:01.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.053+0000 7f03637fe700 1 -- 172.21.15.104:0/3532890712 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0368004050 con 0x7f03741019e0 2024-06-09T22:17:01.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.053+0000 7f0378a55700 1 -- 172.21.15.104:0/3532890712 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f037419e010 con 0x7f03741019e0 2024-06-09T22:17:01.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.053+0000 7f03637fe700 1 -- 172.21.15.104:0/3532890712 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368022e00 con 0x7f03741019e0 2024-06-09T22:17:01.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.053+0000 7f0378a55700 1 -- 172.21.15.104:0/3532890712 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03741a58f0 con 0x7f03741019e0 2024-06-09T22:17:01.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.054+0000 7f0378a55700 1 -- 172.21.15.104:0/3532890712 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f037406a6d0 con 0x7f03741019e0 2024-06-09T22:17:01.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.055+0000 7f03637fe700 1 -- 172.21.15.104:0/3532890712 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f036802a070 con 0x7f03741019e0 2024-06-09T22:17:01.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.056+0000 7f03637fe700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f035c0a2c20 0x7f035c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:01.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.056+0000 7f037259c700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f035c0a2c20 0x7f035c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:01.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.056+0000 7f03637fe700 1 -- 172.21.15.104:0/3532890712 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8802+0+0 (secure 0 0 0) 0x7f03680d0650 con 0x7f03741019e0 2024-06-09T22:17:01.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.057+0000 7f037259c700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f035c0a2c20 0x7f035c0a50e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0364006fd0 tx=0x7f0364008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:01.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.060+0000 7f03637fe700 1 -- 172.21.15.104:0/3532890712 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0368013070 con 0x7f03741019e0 2024-06-09T22:17:01.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.297+0000 7f0378a55700 1 -- 172.21.15.104:0/3532890712 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f037410aa90 con 0x7f03741019e0 2024-06-09T22:17:01.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.298+0000 7f03637fe700 1 -- 172.21.15.104:0/3532890712 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11818 (secure 0 0 0) 0x7f036809d400 con 0x7f03741019e0 2024-06-09T22:17:01.294 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:17:01.294 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24362,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24383,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24398,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:16:58.249728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24460,"qdb_cluster":[24460]},"id":1}]} 2024-06-09T22:17:01.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.300+0000 7f0378a55700 1 -- 172.21.15.104:0/3532890712 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f035c0a2c20 msgr2=0x7f035c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:01.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.300+0000 7f0378a55700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f035c0a2c20 0x7f035c0a50e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0364006fd0 tx=0x7f0364008040 comp rx=0 tx=0).stop 2024-06-09T22:17:01.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.300+0000 7f0378a55700 1 -- 172.21.15.104:0/3532890712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03741019e0 msgr2=0x7f03741992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:01.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.300+0000 7f0378a55700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03741019e0 0x7f03741992b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0368009fd0 tx=0x7f0368005f10 comp rx=0 tx=0).stop 2024-06-09T22:17:01.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.300+0000 7f0378a55700 1 -- 172.21.15.104:0/3532890712 shutdown_connections 2024-06-09T22:17:01.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.300+0000 7f0378a55700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f035c0a2c20 0x7f035c0a50e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:01.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.300+0000 7f0378a55700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0374107850 0x7f037419d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:01.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.300+0000 7f0378a55700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0374101030 0x7f0374198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:01.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.300+0000 7f0378a55700 1 --2- 172.21.15.104:0/3532890712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03741019e0 0x7f03741992b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:01.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.300+0000 7f0378a55700 1 -- 172.21.15.104:0/3532890712 >> 172.21.15.104:0/3532890712 conn(0x7f03740766d0 msgr2=0x7f03740fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:01.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.300+0000 7f0378a55700 1 -- 172.21.15.104:0/3532890712 shutdown_connections 2024-06-09T22:17:01.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:01.300+0000 7f0378a55700 1 -- 172.21.15.104:0/3532890712 wait complete. 2024-06-09T22:17:01.297 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 17 2024-06-09T22:17:01.669 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:01 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3532890712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:17:01.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:01 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3532890712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:17:01.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:01 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3532890712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:17:02.044 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:17:02.289 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:02.657 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:02 smithi104 ceph-mon[32389]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:02.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:02 smithi153 ceph-mon[36599]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:02.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:02 smithi194 ceph-mon[36856]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:04.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.208+0000 7f1a40538700 1 -- 172.21.15.104:0/801881698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a38107440 msgr2=0x7f1a38107820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:04.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.208+0000 7f1a40538700 1 --2- 172.21.15.104:0/801881698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a38107440 0x7f1a38107820 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1a28009990 tx=0x7f1a28009ca0 comp rx=0 tx=0).stop 2024-06-09T22:17:04.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.208+0000 7f1a40538700 1 -- 172.21.15.104:0/801881698 shutdown_connections 2024-06-09T22:17:04.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.208+0000 7f1a40538700 1 --2- 172.21.15.104:0/801881698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a38107440 0x7f1a38107820 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:04.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.208+0000 7f1a40538700 1 --2- 172.21.15.104:0/801881698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38101d80 0x7f1a3810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:04.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.208+0000 7f1a40538700 1 --2- 172.21.15.104:0/801881698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a381013c0 0x7f1a38101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:04.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.208+0000 7f1a40538700 1 -- 172.21.15.104:0/801881698 >> 172.21.15.104:0/801881698 conn(0x7f1a38077400 msgr2=0x7f1a38077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:04.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.208+0000 7f1a40538700 1 -- 172.21.15.104:0/801881698 shutdown_connections 2024-06-09T22:17:04.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.208+0000 7f1a40538700 1 -- 172.21.15.104:0/801881698 wait complete. 2024-06-09T22:17:04.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.208+0000 7f1a40538700 1 Processor -- start 2024-06-09T22:17:04.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.208+0000 7f1a40538700 1 -- start start 2024-06-09T22:17:04.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a40538700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a381013c0 0x7f1a3819d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:04.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a40538700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38101d80 0x7f1a3819dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:04.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a40538700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a38107440 0x7f1a38197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:04.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a40538700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1a381116e0 con 0x7f1a38107440 2024-06-09T22:17:04.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a40538700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1a38111560 con 0x7f1a38101d80 2024-06-09T22:17:04.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a40538700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1a38111860 con 0x7f1a381013c0 2024-06-09T22:17:04.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a3ead5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a38107440 0x7f1a38197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:04.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a3ead5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a38107440 0x7f1a38197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36306/0 (socket says 172.21.15.104:36306) 2024-06-09T22:17:04.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a3ead5700 1 -- 172.21.15.104:0/2815357921 learned_addr learned my addr 172.21.15.104:0/2815357921 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:04.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a3e2d4700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a381013c0 0x7f1a3819d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:04.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a3dad3700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38101d80 0x7f1a3819dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a3ead5700 1 -- 172.21.15.104:0/2815357921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a381013c0 msgr2=0x7f1a3819d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a3ead5700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a381013c0 0x7f1a3819d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a3ead5700 1 -- 172.21.15.104:0/2815357921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38101d80 msgr2=0x7f1a3819dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a3ead5700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38101d80 0x7f1a3819dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a3ead5700 1 -- 172.21.15.104:0/2815357921 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a38198180 con 0x7f1a38107440 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a3dad3700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38101d80 0x7f1a3819dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a3ead5700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a38107440 0x7f1a38197a20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000ada0 tx=0x7f1a3000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.209+0000 7f1a2f7fe700 1 -- 172.21.15.104:0/2815357921 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30017020 con 0x7f1a38107440 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.210+0000 7f1a2f7fe700 1 -- 172.21.15.104:0/2815357921 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a3000a0d0 con 0x7f1a38107440 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.210+0000 7f1a40538700 1 -- 172.21.15.104:0/2815357921 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a38198470 con 0x7f1a38107440 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.210+0000 7f1a2f7fe700 1 -- 172.21.15.104:0/2815357921 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30005480 con 0x7f1a38107440 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.210+0000 7f1a40538700 1 -- 172.21.15.104:0/2815357921 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a381a46e0 con 0x7f1a38107440 2024-06-09T22:17:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.211+0000 7f1a40538700 1 -- 172.21.15.104:0/2815357921 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a38109900 con 0x7f1a38107440 2024-06-09T22:17:04.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.212+0000 7f1a2f7fe700 1 -- 172.21.15.104:0/2815357921 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f1a30005620 con 0x7f1a38107440 2024-06-09T22:17:04.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.213+0000 7f1a2f7fe700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1a240a2c20 0x7f1a240a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:04.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.213+0000 7f1a3e2d4700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1a240a2c20 0x7f1a240a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:04.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.213+0000 7f1a2f7fe700 1 -- 172.21.15.104:0/2815357921 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8802+0+0 (secure 0 0 0) 0x7f1a300cf000 con 0x7f1a38107440 2024-06-09T22:17:04.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.213+0000 7f1a3e2d4700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1a240a2c20 0x7f1a240a50e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800b5c0 tx=0x7f1a28005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:04.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.216+0000 7f1a2f7fe700 1 -- 172.21.15.104:0/2815357921 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1a30021020 con 0x7f1a38107440 2024-06-09T22:17:04.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.419+0000 7f1a40538700 1 -- 172.21.15.104:0/2815357921 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a381092e0 con 0x7f1a38107440 2024-06-09T22:17:04.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.420+0000 7f1a2f7fe700 1 -- 172.21.15.104:0/2815357921 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24196 (secure 0 0 0) 0x7f1a3009bdb0 con 0x7f1a38107440 2024-06-09T22:17:04.417 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:17:04.417 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":95,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:16:58.249585+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:17:04.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.422+0000 7f1a40538700 1 -- 172.21.15.104:0/2815357921 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1a240a2c20 msgr2=0x7f1a240a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:04.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.422+0000 7f1a40538700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1a240a2c20 0x7f1a240a50e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800b5c0 tx=0x7f1a28005960 comp rx=0 tx=0).stop 2024-06-09T22:17:04.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.422+0000 7f1a40538700 1 -- 172.21.15.104:0/2815357921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a38107440 msgr2=0x7f1a38197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:04.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.422+0000 7f1a40538700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a38107440 0x7f1a38197a20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000ada0 tx=0x7f1a3000c330 comp rx=0 tx=0).stop 2024-06-09T22:17:04.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.422+0000 7f1a40538700 1 -- 172.21.15.104:0/2815357921 shutdown_connections 2024-06-09T22:17:04.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.422+0000 7f1a40538700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1a240a2c20 0x7f1a240a50e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:04.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.422+0000 7f1a40538700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a38107440 0x7f1a38197a20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:04.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.422+0000 7f1a40538700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a381013c0 0x7f1a3819d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:04.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.422+0000 7f1a40538700 1 --2- 172.21.15.104:0/2815357921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38101d80 0x7f1a3819dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:04.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.422+0000 7f1a40538700 1 -- 172.21.15.104:0/2815357921 >> 172.21.15.104:0/2815357921 conn(0x7f1a38077400 msgr2=0x7f1a38100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:04.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.423+0000 7f1a40538700 1 -- 172.21.15.104:0/2815357921 shutdown_connections 2024-06-09T22:17:04.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:04.423+0000 7f1a40538700 1 -- 172.21.15.104:0/2815357921 wait complete. 2024-06-09T22:17:04.737 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:04 smithi104 ceph-mon[32389]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:04.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:04 smithi153 ceph-mon[36599]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:04.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:04 smithi194 ceph-mon[36856]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:05.087 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:17:05.330 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:05.631 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:05 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2815357921' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:05.721 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:05 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2815357921' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:05.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:05 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2815357921' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:06.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:06 smithi104 ceph-mon[32389]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:06.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:06 smithi153 ceph-mon[36599]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:06.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:06 smithi194 ceph-mon[36856]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:07.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.157+0000 7f1520932700 1 -- 172.21.15.104:0/1667214915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f151c102740 msgr2=0x7f151c102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:07.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.157+0000 7f1520932700 1 --2- 172.21.15.104:0/1667214915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f151c102740 0x7f151c102b20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f150c0099c0 tx=0x7f150c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:17:07.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.158+0000 7f1520932700 1 -- 172.21.15.104:0/1667214915 shutdown_connections 2024-06-09T22:17:07.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.158+0000 7f1520932700 1 --2- 172.21.15.104:0/1667214915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f151c102740 0x7f151c102b20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:07.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.158+0000 7f1520932700 1 --2- 172.21.15.104:0/1667214915 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c107840 0x7f151c10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:07.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.158+0000 7f1520932700 1 --2- 172.21.15.104:0/1667214915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c1030f0 0x7f151c107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:07.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.158+0000 7f1520932700 1 -- 172.21.15.104:0/1667214915 >> 172.21.15.104:0/1667214915 conn(0x7f151c0795f0 msgr2=0x7f151c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:07.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.158+0000 7f1520932700 1 -- 172.21.15.104:0/1667214915 shutdown_connections 2024-06-09T22:17:07.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.158+0000 7f1520932700 1 -- 172.21.15.104:0/1667214915 wait complete. 2024-06-09T22:17:07.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.158+0000 7f1520932700 1 Processor -- start 2024-06-09T22:17:07.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f1520932700 1 -- start start 2024-06-09T22:17:07.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f1520932700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c102740 0x7f151c198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:07.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f1520932700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c1030f0 0x7f151c1992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:07.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f1520932700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f151c107840 0x7f151c19d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:07.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f1520932700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f151c112850 con 0x7f151c107840 2024-06-09T22:17:07.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f1520932700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f151c1126d0 con 0x7f151c1030f0 2024-06-09T22:17:07.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f1520932700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f151c1129d0 con 0x7f151c102740 2024-06-09T22:17:07.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f151ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f151c107840 0x7f151c19d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:07.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f151ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f151c107840 0x7f151c19d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57962/0 (socket says 172.21.15.104:57962) 2024-06-09T22:17:07.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f151ad9d700 1 -- 172.21.15.104:0/1037896046 learned_addr learned my addr 172.21.15.104:0/1037896046 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:07.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f1519d9b700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c1030f0 0x7f151c1992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:07.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f151a59c700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c102740 0x7f151c198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:07.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f151ad9d700 1 -- 172.21.15.104:0/1037896046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c102740 msgr2=0x7f151c198d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:07.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f151ad9d700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c102740 0x7f151c198d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:07.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f151ad9d700 1 -- 172.21.15.104:0/1037896046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c1030f0 msgr2=0x7f151c1992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:07.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f151ad9d700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c1030f0 0x7f151c1992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:07.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f151ad9d700 1 -- 172.21.15.104:0/1037896046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151c19dd70 con 0x7f151c107840 2024-06-09T22:17:07.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f151a59c700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c102740 0x7f151c198d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:17:07.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f1519d9b700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c1030f0 0x7f151c1992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:17:07.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.159+0000 7f151ad9d700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f151c107840 0x7f151c19d630 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f151400b970 tx=0x7f151400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:07.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.160+0000 7f150b7fe700 1 -- 172.21.15.104:0/1037896046 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1514019070 con 0x7f151c107840 2024-06-09T22:17:07.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.160+0000 7f1520932700 1 -- 172.21.15.104:0/1037896046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f151c19e060 con 0x7f151c107840 2024-06-09T22:17:07.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.160+0000 7f150b7fe700 1 -- 172.21.15.104:0/1037896046 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1514014440 con 0x7f151c107840 2024-06-09T22:17:07.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.160+0000 7f150b7fe700 1 -- 172.21.15.104:0/1037896046 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15140075d0 con 0x7f151c107840 2024-06-09T22:17:07.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.160+0000 7f1520932700 1 -- 172.21.15.104:0/1037896046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f151c1a58e0 con 0x7f151c107840 2024-06-09T22:17:07.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.161+0000 7f1520932700 1 -- 172.21.15.104:0/1037896046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f151c10aa80 con 0x7f151c107840 2024-06-09T22:17:07.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.162+0000 7f150b7fe700 1 -- 172.21.15.104:0/1037896046 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f151401c070 con 0x7f151c107840 2024-06-09T22:17:07.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.162+0000 7f150b7fe700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f15040a2c20 0x7f15040a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:07.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.163+0000 7f151a59c700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f15040a2c20 0x7f15040a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:07.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.163+0000 7f150b7fe700 1 -- 172.21.15.104:0/1037896046 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8802+0+0 (secure 0 0 0) 0x7f15140ce710 con 0x7f151c107840 2024-06-09T22:17:07.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.163+0000 7f151a59c700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f15040a2c20 0x7f15040a50e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f150c009750 tx=0x7f150c009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:07.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.167+0000 7f150b7fe700 1 -- 172.21.15.104:0/1037896046 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f151409b430 con 0x7f151c107840 2024-06-09T22:17:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.368+0000 7f1520932700 1 -- 172.21.15.104:0/1037896046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f151c19a0c0 con 0x7f151c107840 2024-06-09T22:17:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.370+0000 7f150b7fe700 1 -- 172.21.15.104:0/1037896046 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24196 (secure 0 0 0) 0x7f151409ad30 con 0x7f151c107840 2024-06-09T22:17:07.366 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:17:07.366 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":95,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:16:58.249585+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:17:07.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.372+0000 7f1520932700 1 -- 172.21.15.104:0/1037896046 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f15040a2c20 msgr2=0x7f15040a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:07.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.372+0000 7f1520932700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f15040a2c20 0x7f15040a50e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f150c009750 tx=0x7f150c009f10 comp rx=0 tx=0).stop 2024-06-09T22:17:07.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.372+0000 7f1520932700 1 -- 172.21.15.104:0/1037896046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f151c107840 msgr2=0x7f151c19d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:07.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.372+0000 7f1520932700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f151c107840 0x7f151c19d630 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f151400b970 tx=0x7f151400bd30 comp rx=0 tx=0).stop 2024-06-09T22:17:07.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.372+0000 7f1520932700 1 -- 172.21.15.104:0/1037896046 shutdown_connections 2024-06-09T22:17:07.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.372+0000 7f1520932700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f15040a2c20 0x7f15040a50e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:07.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.372+0000 7f1520932700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f151c107840 0x7f151c19d630 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:07.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.372+0000 7f1520932700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c102740 0x7f151c198d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:07.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.372+0000 7f1520932700 1 --2- 172.21.15.104:0/1037896046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c1030f0 0x7f151c1992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:07.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.372+0000 7f1520932700 1 -- 172.21.15.104:0/1037896046 >> 172.21.15.104:0/1037896046 conn(0x7f151c0795f0 msgr2=0x7f151c100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:07.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.372+0000 7f1520932700 1 -- 172.21.15.104:0/1037896046 shutdown_connections 2024-06-09T22:17:07.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:07.372+0000 7f1520932700 1 -- 172.21.15.104:0/1037896046 wait complete. 2024-06-09T22:17:07.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:07 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1037896046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:07.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:07 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1037896046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:07.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:07 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1037896046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:08.075 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:17:08.315 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:08.735 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:08 smithi104 ceph-mon[32389]: pgmap v316: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:08.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:08 smithi153 ceph-mon[36599]: pgmap v316: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:08.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:08 smithi194 ceph-mon[36856]: pgmap v316: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.124+0000 7fdf9d158700 1 -- 172.21.15.104:0/2500467675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf98101390 msgr2=0x7fdf98101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.124+0000 7fdf9d158700 1 --2- 172.21.15.104:0/2500467675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf98101390 0x7fdf98101810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c009990 tx=0x7fdf8c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.124+0000 7fdf9d158700 1 -- 172.21.15.104:0/2500467675 shutdown_connections 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.124+0000 7fdf9d158700 1 --2- 172.21.15.104:0/2500467675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98101d50 0x7fdf9810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.124+0000 7fdf9d158700 1 --2- 172.21.15.104:0/2500467675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf98107400 0x7fdf981077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.124+0000 7fdf9d158700 1 --2- 172.21.15.104:0/2500467675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf98101390 0x7fdf98101810 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.124+0000 7fdf9d158700 1 -- 172.21.15.104:0/2500467675 >> 172.21.15.104:0/2500467675 conn(0x7fdf98078340 msgr2=0x7fdf98078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.124+0000 7fdf9d158700 1 -- 172.21.15.104:0/2500467675 shutdown_connections 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.124+0000 7fdf9d158700 1 -- 172.21.15.104:0/2500467675 wait complete. 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.125+0000 7fdf9d158700 1 Processor -- start 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.125+0000 7fdf9d158700 1 -- start start 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.125+0000 7fdf9d158700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf98101390 0x7fdf98199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.125+0000 7fdf9d158700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf98101d50 0x7fdf98199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.125+0000 7fdf9d158700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98107400 0x7fdf981a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf9d158700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdf981116f0 con 0x7fdf98101390 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf9d158700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdf98111570 con 0x7fdf98107400 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf9d158700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdf98111870 con 0x7fdf98101d50 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf96d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf98101390 0x7fdf98199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf96d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf98101390 0x7fdf98199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57972/0 (socket says 172.21.15.104:57972) 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf96d9d700 1 -- 172.21.15.104:0/4021293706 learned_addr learned my addr 172.21.15.104:0/4021293706 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf9759e700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98107400 0x7fdf981a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf9659c700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf98101d50 0x7fdf98199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf96d9d700 1 -- 172.21.15.104:0/4021293706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf98101d50 msgr2=0x7fdf98199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf96d9d700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf98101d50 0x7fdf98199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf96d9d700 1 -- 172.21.15.104:0/4021293706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98107400 msgr2=0x7fdf981a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf96d9d700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98107400 0x7fdf981a4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf96d9d700 1 -- 172.21.15.104:0/4021293706 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf98197880 con 0x7fdf98101390 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf9659c700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf98101d50 0x7fdf98199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:17:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf96d9d700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf98101390 0x7fdf98199a90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800b530 tx=0x7fdf8800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf7ffff700 1 -- 172.21.15.104:0/4021293706 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88015020 con 0x7fdf98101390 2024-06-09T22:17:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf7ffff700 1 -- 172.21.15.104:0/4021293706 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf88011d10 con 0x7fdf98101390 2024-06-09T22:17:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf7ffff700 1 -- 172.21.15.104:0/4021293706 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8800f2e0 con 0x7fdf98101390 2024-06-09T22:17:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf9d158700 1 -- 172.21.15.104:0/4021293706 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf98197b70 con 0x7fdf98101390 2024-06-09T22:17:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.126+0000 7fdf9d158700 1 -- 172.21.15.104:0/4021293706 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf98198060 con 0x7fdf98101390 2024-06-09T22:17:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.127+0000 7fdf9d158700 1 -- 172.21.15.104:0/4021293706 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf98069180 con 0x7fdf98101390 2024-06-09T22:17:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.128+0000 7fdf7ffff700 1 -- 172.21.15.104:0/4021293706 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fdf88018020 con 0x7fdf98101390 2024-06-09T22:17:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.129+0000 7fdf7ffff700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fdf800a2c20 0x7fdf800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.129+0000 7fdf9659c700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fdf800a2c20 0x7fdf800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.129+0000 7fdf7ffff700 1 -- 172.21.15.104:0/4021293706 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8802+0+0 (secure 0 0 0) 0x7fdf880cf0a0 con 0x7fdf98101390 2024-06-09T22:17:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.130+0000 7fdf9659c700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fdf800a2c20 0x7fdf800a50e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c006010 tx=0x7fdf8c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:10.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.133+0000 7fdf7ffff700 1 -- 172.21.15.104:0/4021293706 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdf8801f020 con 0x7fdf98101390 2024-06-09T22:17:10.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.372+0000 7fdf9d158700 1 -- 172.21.15.104:0/4021293706 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf9806a760 con 0x7fdf98101390 2024-06-09T22:17:10.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.373+0000 7fdf7ffff700 1 -- 172.21.15.104:0/4021293706 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11818 (secure 0 0 0) 0x7fdf9806a760 con 0x7fdf98101390 2024-06-09T22:17:10.369 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:17:10.369 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24362,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24383,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24398,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:16:58.249728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24460,"qdb_cluster":[24460]},"id":1}]} 2024-06-09T22:17:10.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.375+0000 7fdf9d158700 1 -- 172.21.15.104:0/4021293706 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fdf800a2c20 msgr2=0x7fdf800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:10.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.375+0000 7fdf9d158700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fdf800a2c20 0x7fdf800a50e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c006010 tx=0x7fdf8c00b540 comp rx=0 tx=0).stop 2024-06-09T22:17:10.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.375+0000 7fdf9d158700 1 -- 172.21.15.104:0/4021293706 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf98101390 msgr2=0x7fdf98199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:10.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.375+0000 7fdf9d158700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf98101390 0x7fdf98199a90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800b530 tx=0x7fdf8800b840 comp rx=0 tx=0).stop 2024-06-09T22:17:10.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.375+0000 7fdf9d158700 1 -- 172.21.15.104:0/4021293706 shutdown_connections 2024-06-09T22:17:10.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.375+0000 7fdf9d158700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fdf800a2c20 0x7fdf800a50e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:10.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.375+0000 7fdf9d158700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf98101390 0x7fdf98199a90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:10.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.375+0000 7fdf9d158700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf98107400 0x7fdf981a4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:10.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.375+0000 7fdf9d158700 1 --2- 172.21.15.104:0/4021293706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdf98101d50 0x7fdf98199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:10.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.375+0000 7fdf9d158700 1 -- 172.21.15.104:0/4021293706 >> 172.21.15.104:0/4021293706 conn(0x7fdf98078340 msgr2=0x7fdf980ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:10.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.375+0000 7fdf9d158700 1 -- 172.21.15.104:0/4021293706 shutdown_connections 2024-06-09T22:17:10.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:10.375+0000 7fdf9d158700 1 -- 172.21.15.104:0/4021293706 wait complete. 2024-06-09T22:17:10.372 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 17 2024-06-09T22:17:10.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:10 smithi153 ceph-mon[36599]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:10.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:10 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/4021293706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:17:10.739 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:10 smithi104 ceph-mon[32389]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:10.739 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:10 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/4021293706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:17:10.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:10 smithi194 ceph-mon[36856]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:10.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:10 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/4021293706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:17:11.234 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:17:11.481 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:12.732 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:12 smithi104 ceph-mon[32389]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:12.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:12 smithi153 ceph-mon[36599]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:12.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:12 smithi194 ceph-mon[36856]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:13.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.292+0000 7f848a055700 1 -- 172.21.15.104:0/2714780391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8484102770 msgr2=0x7f8484102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:13.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.292+0000 7f848a055700 1 --2- 172.21.15.104:0/2714780391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8484102770 0x7f8484102b50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f847c0099c0 tx=0x7f847c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:17:13.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.292+0000 7f848a055700 1 -- 172.21.15.104:0/2714780391 shutdown_connections 2024-06-09T22:17:13.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.292+0000 7f848a055700 1 --2- 172.21.15.104:0/2714780391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8484102770 0x7f8484102b50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:13.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.292+0000 7f848a055700 1 --2- 172.21.15.104:0/2714780391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8484107860 0x7f848410f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:13.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.292+0000 7f848a055700 1 --2- 172.21.15.104:0/2714780391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8484103120 0x7f8484107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.292+0000 7f848a055700 1 -- 172.21.15.104:0/2714780391 >> 172.21.15.104:0/2714780391 conn(0x7f8484079600 msgr2=0x7f8484079a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.292+0000 7f848a055700 1 -- 172.21.15.104:0/2714780391 shutdown_connections 2024-06-09T22:17:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.292+0000 7f848a055700 1 -- 172.21.15.104:0/2714780391 wait complete. 2024-06-09T22:17:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f848a055700 1 Processor -- start 2024-06-09T22:17:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f848a055700 1 -- start start 2024-06-09T22:17:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f848a055700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8484102770 0x7f848419e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f848a055700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8484103120 0x7f848419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f848a055700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8484107860 0x7f8484198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f848a055700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8484112800 con 0x7f8484103120 2024-06-09T22:17:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f848a055700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8484112680 con 0x7f8484107860 2024-06-09T22:17:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f848a055700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8484112980 con 0x7f8484102770 2024-06-09T22:17:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f8482ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8484103120 0x7f848419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f8482ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8484103120 0x7f848419ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57988/0 (socket says 172.21.15.104:57988) 2024-06-09T22:17:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f8482ffd700 1 -- 172.21.15.104:0/970316375 learned_addr learned my addr 172.21.15.104:0/970316375 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f84837fe700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8484102770 0x7f848419e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f8482ffd700 1 -- 172.21.15.104:0/970316375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8484102770 msgr2=0x7f848419e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f8482ffd700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8484102770 0x7f848419e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f8482ffd700 1 -- 172.21.15.104:0/970316375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8484107860 msgr2=0x7f8484198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f8482ffd700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8484107860 0x7f8484198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.293+0000 7f8482ffd700 1 -- 172.21.15.104:0/970316375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84841990a0 con 0x7f8484103120 2024-06-09T22:17:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.294+0000 7f8482ffd700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8484103120 0x7f848419ecc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f846c00c9b0 tx=0x7f846c00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.294+0000 7f8480ff9700 1 -- 172.21.15.104:0/970316375 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c007b00 con 0x7f8484103120 2024-06-09T22:17:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.294+0000 7f848a055700 1 -- 172.21.15.104:0/970316375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8484199390 con 0x7f8484103120 2024-06-09T22:17:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.294+0000 7f8480ff9700 1 -- 172.21.15.104:0/970316375 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f846c007ca0 con 0x7f8484103120 2024-06-09T22:17:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.294+0000 7f8480ff9700 1 -- 172.21.15.104:0/970316375 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c018780 con 0x7f8484103120 2024-06-09T22:17:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.294+0000 7f848a055700 1 -- 172.21.15.104:0/970316375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84841a5650 con 0x7f8484103120 2024-06-09T22:17:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.295+0000 7f848a055700 1 -- 172.21.15.104:0/970316375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8484069180 con 0x7f8484103120 2024-06-09T22:17:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.296+0000 7f8480ff9700 1 -- 172.21.15.104:0/970316375 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f846c00f450 con 0x7f8484103120 2024-06-09T22:17:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.296+0000 7f8480ff9700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f84700a2c20 0x7f84700a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.297+0000 7f84837fe700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f84700a2c20 0x7f84700a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.297+0000 7f8480ff9700 1 -- 172.21.15.104:0/970316375 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8802+0+0 (secure 0 0 0) 0x7f846c0cf2f0 con 0x7f8484103120 2024-06-09T22:17:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.297+0000 7f84837fe700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f84700a2c20 0x7f84700a50e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f847c000c00 tx=0x7f847c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:13.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.301+0000 7f8480ff9700 1 -- 172.21.15.104:0/970316375 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f846c014030 con 0x7f8484103120 2024-06-09T22:17:13.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.501+0000 7f848a055700 1 -- 172.21.15.104:0/970316375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84841a5910 con 0x7f8484103120 2024-06-09T22:17:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.502+0000 7f8480ff9700 1 -- 172.21.15.104:0/970316375 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24196 (secure 0 0 0) 0x7f846c09c010 con 0x7f8484103120 2024-06-09T22:17:13.499 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:17:13.499 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":95,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:16:58.249585+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:17:13.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.505+0000 7f848a055700 1 -- 172.21.15.104:0/970316375 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f84700a2c20 msgr2=0x7f84700a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:13.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.505+0000 7f848a055700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f84700a2c20 0x7f84700a50e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f847c000c00 tx=0x7f847c005fb0 comp rx=0 tx=0).stop 2024-06-09T22:17:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.505+0000 7f848a055700 1 -- 172.21.15.104:0/970316375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8484103120 msgr2=0x7f848419ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.505+0000 7f848a055700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8484103120 0x7f848419ecc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f846c00c9b0 tx=0x7f846c00cd70 comp rx=0 tx=0).stop 2024-06-09T22:17:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.505+0000 7f848a055700 1 -- 172.21.15.104:0/970316375 shutdown_connections 2024-06-09T22:17:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.505+0000 7f848a055700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f84700a2c20 0x7f84700a50e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.505+0000 7f848a055700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8484107860 0x7f8484198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.505+0000 7f848a055700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8484102770 0x7f848419e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.505+0000 7f848a055700 1 --2- 172.21.15.104:0/970316375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8484103120 0x7f848419ecc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.505+0000 7f848a055700 1 -- 172.21.15.104:0/970316375 >> 172.21.15.104:0/970316375 conn(0x7f8484079600 msgr2=0x7f8484100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.505+0000 7f848a055700 1 -- 172.21.15.104:0/970316375 shutdown_connections 2024-06-09T22:17:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:13.505+0000 7f848a055700 1 -- 172.21.15.104:0/970316375 wait complete. 2024-06-09T22:17:14.221 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-09T22:16:45.150445+0000', 'modified': '2024-06-09T22:16:58.249728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24460}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.194:6839/2028005544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6838', 'nonce': 2028005544}, {'type': 'v1', 'addr': '172.21.15.194:6839', 'nonce': 2028005544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24460, 'qdb_cluster': [24460]} 2024-06-09T22:17:14.221 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-09T22:17:14.221 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-06-09T22:17:14.470 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:14.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:14 smithi104 ceph-mon[32389]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:14.502 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:14 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/970316375' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:14.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:14 smithi153 ceph-mon[36599]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:14.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:14 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/970316375' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:14.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:14 smithi194 ceph-mon[36856]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:14.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:14 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/970316375' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:16.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.431+0000 7fefc5b9a700 1 -- 172.21.15.104:0/1082144265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefc01073f0 msgr2=0x7fefc01077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.431+0000 7fefc5b9a700 1 --2- 172.21.15.104:0/1082144265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefc01073f0 0x7fefc01077d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009990 tx=0x7fefb0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.431+0000 7fefc5b9a700 1 -- 172.21.15.104:0/1082144265 shutdown_connections 2024-06-09T22:17:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.431+0000 7fefc5b9a700 1 --2- 172.21.15.104:0/1082144265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefc01073f0 0x7fefc01077d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.431+0000 7fefc5b9a700 1 --2- 172.21.15.104:0/1082144265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc0101d40 0x7fefc010e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.431+0000 7fefc5b9a700 1 --2- 172.21.15.104:0/1082144265 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fefc0101380 0x7fefc0101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.431+0000 7fefc5b9a700 1 -- 172.21.15.104:0/1082144265 >> 172.21.15.104:0/1082144265 conn(0x7fefc0078300 msgr2=0x7fefc0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.431+0000 7fefc5b9a700 1 -- 172.21.15.104:0/1082144265 shutdown_connections 2024-06-09T22:17:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.431+0000 7fefc5b9a700 1 -- 172.21.15.104:0/1082144265 wait complete. 2024-06-09T22:17:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.432+0000 7fefc5b9a700 1 Processor -- start 2024-06-09T22:17:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.432+0000 7fefc5b9a700 1 -- start start 2024-06-09T22:17:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.432+0000 7fefc5b9a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc0101380 0x7fefc019ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.432+0000 7fefc5b9a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefc0101d40 0x7fefc019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.432+0000 7fefc5b9a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fefc01073f0 0x7fefc019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.432+0000 7fefc5b9a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fefc0111690 con 0x7fefc0101d40 2024-06-09T22:17:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.432+0000 7fefc5b9a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fefc0111510 con 0x7fefc0101380 2024-06-09T22:17:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.432+0000 7fefc5b9a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fefc0111810 con 0x7fefc01073f0 2024-06-09T22:17:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.432+0000 7fefb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefc0101d40 0x7fefc019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.432+0000 7fefb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefc0101d40 0x7fefc019b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44624/0 (socket says 172.21.15.104:44624) 2024-06-09T22:17:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.432+0000 7fefb7fff700 1 -- 172.21.15.104:0/989515800 learned_addr learned my addr 172.21.15.104:0/989515800 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefbf7fe700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc0101380 0x7fefc019ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefbffff700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fefc01073f0 0x7fefc019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefb7fff700 1 -- 172.21.15.104:0/989515800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fefc01073f0 msgr2=0x7fefc019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefb7fff700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fefc01073f0 0x7fefc019f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefb7fff700 1 -- 172.21.15.104:0/989515800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc0101380 msgr2=0x7fefc019ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefb7fff700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc0101380 0x7fefc019ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefb7fff700 1 -- 172.21.15.104:0/989515800 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc019fe60 con 0x7fefc0101d40 2024-06-09T22:17:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefb7fff700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefc0101d40 0x7fefc019b390 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fefa800d980 tx=0x7fefa800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefbd7fa700 1 -- 172.21.15.104:0/989515800 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8017070 con 0x7fefc0101d40 2024-06-09T22:17:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefc5b9a700 1 -- 172.21.15.104:0/989515800 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefc01a0150 con 0x7fefc0101d40 2024-06-09T22:17:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefbd7fa700 1 -- 172.21.15.104:0/989515800 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefa8005020 con 0x7fefc0101d40 2024-06-09T22:17:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefbd7fa700 1 -- 172.21.15.104:0/989515800 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8009bc0 con 0x7fefc0101d40 2024-06-09T22:17:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.433+0000 7fefc5b9a700 1 -- 172.21.15.104:0/989515800 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefc01a79d0 con 0x7fefc0101d40 2024-06-09T22:17:16.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.436+0000 7fefbd7fa700 1 -- 172.21.15.104:0/989515800 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fefa80051d0 con 0x7fefc0101d40 2024-06-09T22:17:16.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.437+0000 7fefbd7fa700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fefac0a2c20 0x7fefac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.437+0000 7fefbf7fe700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fefac0a2c20 0x7fefac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.437+0000 7fefbd7fa700 1 -- 172.21.15.104:0/989515800 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8802+0+0 (secure 0 0 0) 0x7fefa80cf1b0 con 0x7fefc0101d40 2024-06-09T22:17:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.437+0000 7fefbf7fe700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fefac0a2c20 0x7fefac0a50e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fefb000b5c0 tx=0x7fefb0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.437+0000 7fefd041db80 1 Processor -- start 2024-06-09T22:17:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.437+0000 7fefd041db80 1 -- start start 2024-06-09T22:17:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefd041db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd416f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefd041db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 0x55dcd4169210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefd041db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd4169750 0x55dcd4169b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefd041db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55dcd405bf10 con 0x55dcd416f830 2024-06-09T22:17:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefd041db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dcd40ba800 con 0x55dcd4169750 2024-06-09T22:17:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefd041db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dcd4022410 con 0x55dcd416ef10 2024-06-09T22:17:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 0x55dcd4169210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 0x55dcd4169210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44640/0 (socket says 172.21.15.104:44640) 2024-06-09T22:17:16.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefb7fff700 1 -- 172.21.15.104:0/1949317717 learned_addr learned my addr 172.21.15.104:0/1949317717 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:16.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefbf7fe700 1 --2- 172.21.15.104:0/1949317717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd416f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefbffff700 1 --2- 172.21.15.104:0/1949317717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd4169750 0x55dcd4169b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefb7fff700 1 -- 172.21.15.104:0/1949317717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 msgr2=0x55dcd416f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:17:16.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefb7fff700 1 --2- 172.21.15.104:0/1949317717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd416f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefb7fff700 1 -- 172.21.15.104:0/1949317717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd4169750 msgr2=0x55dcd4169b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:17:16.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefb7fff700 1 --2- 172.21.15.104:0/1949317717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd4169750 0x55dcd4169b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefb7fff700 1 -- 172.21.15.104:0/1949317717 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcd4009880 con 0x55dcd416f830 2024-06-09T22:17:16.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefb7fff700 1 --2- 172.21.15.104:0/1949317717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 0x55dcd4169210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fefa809b760 tx=0x7fefa809b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef83fd77c6df7482 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:16.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefb5ffb700 1 -- 172.21.15.104:0/1949317717 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa809fb80 con 0x55dcd416f830 2024-06-09T22:17:16.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefb5ffb700 1 -- 172.21.15.104:0/1949317717 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefa80a4440 con 0x55dcd416f830 2024-06-09T22:17:16.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.438+0000 7fefb5ffb700 1 -- 172.21.15.104:0/1949317717 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa80a3fb0 con 0x55dcd416f830 2024-06-09T22:17:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.439+0000 7fefd041db80 1 -- 172.21.15.104:0/1949317717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 msgr2=0x55dcd4169210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.439+0000 7fefd041db80 1 --2- 172.21.15.104:0/1949317717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 0x55dcd4169210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fefa809b760 tx=0x7fefa809b7d0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.439+0000 7fefd041db80 1 -- 172.21.15.104:0/1949317717 shutdown_connections 2024-06-09T22:17:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.439+0000 7fefd041db80 1 --2- 172.21.15.104:0/1949317717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd4169750 0x55dcd4169b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.439+0000 7fefd041db80 1 --2- 172.21.15.104:0/1949317717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd416f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.439+0000 7fefd041db80 1 --2- 172.21.15.104:0/1949317717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 0x55dcd4169210 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.439+0000 7fefd041db80 1 -- 172.21.15.104:0/1949317717 >> 172.21.15.104:0/1949317717 conn(0x55dcd4096030 msgr2=0x55dcd41641d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.439+0000 7fefd041db80 1 -- 172.21.15.104:0/1949317717 shutdown_connections 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.439+0000 7fefd041db80 1 -- 172.21.15.104:0/1949317717 wait complete. 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefd041db80 1 -- start start 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefd041db80 1 Processor -- start 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefd041db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 0x55dcd41755a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefd041db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd41728a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefd041db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd416f830 0x55dcd4172de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefd041db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55dcd405af70 con 0x55dcd4169750 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefd041db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dcd40b67c0 con 0x55dcd416f830 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefd041db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dcd4176210 con 0x55dcd416ef10 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefbf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 0x55dcd41755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefd041db80 10 client.0 _ll_register_callbacks cb 0x55dcd4050560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefbf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 0x55dcd41755a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44652/0 (socket says 172.21.15.104:44652) 2024-06-09T22:17:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefbf7fe700 1 -- 172.21.15.104:0/2368805368 learned_addr learned my addr 172.21.15.104:0/2368805368 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefb7fff700 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd41728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.440+0000 7fefbffff700 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd416f830 0x55dcd4172de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fefbf7fe700 1 -- 172.21.15.104:0/2368805368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 msgr2=0x55dcd41728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fefbf7fe700 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd41728a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fefbf7fe700 1 -- 172.21.15.104:0/2368805368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd416f830 msgr2=0x55dcd4172de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fefbf7fe700 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd416f830 0x55dcd4172de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fefbf7fe700 1 -- 172.21.15.104:0/2368805368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcd4080700 con 0x55dcd4169750 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fefb7fff700 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd41728a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fefbf7fe700 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 0x55dcd41755a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009990 tx=0x7fefb001a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb36c2ff239fe995 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fef9effd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fef9effd700 1 -- 172.21.15.104:0/2368805368 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb0024410 con 0x55dcd4169750 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fef9effd700 1 -- 172.21.15.104:0/2368805368 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8802+0+0 (secure 0 0 0) 0x7fefb00245b0 con 0x55dcd4169750 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fefd041db80 1 -- 172.21.15.104:0/2368805368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefa80a4440 con 0x55dcd4169750 2024-06-09T22:17:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fef9effd700 1 -- 172.21.15.104:0/2368805368 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fefb003e050 con 0x55dcd4169750 2024-06-09T22:17:16.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fefd041db80 10 client.14682 fetch_fsmap learned FSMap version 17 2024-06-09T22:17:16.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fefd041db80 1 -- 172.21.15.104:0/2368805368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55dcd4180990 con 0x55dcd4169750 2024-06-09T22:17:16.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.441+0000 7fef9effd700 1 -- 172.21.15.104:0/2368805368 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefa80a4440 con 0x55dcd4169750 2024-06-09T22:17:16.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.442+0000 7fef9effd700 1 -- 172.21.15.104:0/2368805368 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fefb0030a00 con 0x55dcd4169750 2024-06-09T22:17:16.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.442+0000 7fefd041db80 10 client.14682 fetch_fsmap finished waiting for FSMap version 17 2024-06-09T22:17:16.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.442+0000 7fefd041db80 10 client.14682 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-09T22:17:16.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.442+0000 7fefd041db80 20 client.14682 populate_metadata read hostname 'smithi104' 2024-06-09T22:17:16.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.442+0000 7fefd041db80 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dcd4182fa0 0x55dcd4184410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.442+0000 7fefd041db80 4 client.14682 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:17:16.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.442+0000 7fefd041db80 1 -- 172.21.15.104:0/2368805368 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcd40c7d00 con 0x55dcd4182fa0 2024-06-09T22:17:16.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.442+0000 7fefb7fff700 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dcd4182fa0 0x55dcd4184410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.442+0000 7fefb7fff700 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dcd4182fa0 0x55dcd4184410 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=645d9c0fe9ecba2c server_cookie=b1990a5d268d593e in_seq=0 out_seq=0 2024-06-09T22:17:16.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.442+0000 7fef9effd700 10 client.14682 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:17:16.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.443+0000 7fef9effd700 1 -- 172.21.15.104:0/2368805368 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55dcd40c7d00 con 0x55dcd4182fa0 2024-06-09T22:17:16.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.443+0000 7fef9effd700 10 client.14682 handle_command_reply: tid=0 2024-06-09T22:17:16.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.443+0000 7fefd041db80 1 client.14682 shutdown 2024-06-09T22:17:16.443 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:16 smithi104 ceph-mon[32389]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.443+0000 7fefd041db80 1 -- 172.21.15.104:0/2368805368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 msgr2=0x55dcd41755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.443+0000 7fefd041db80 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 0x55dcd41755a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009990 tx=0x7fefb001a050 comp rx=0 tx=0).stop 2024-06-09T22:17:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.443+0000 7fefd041db80 1 -- 172.21.15.104:0/2368805368 shutdown_connections 2024-06-09T22:17:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.443+0000 7fefd041db80 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dcd4182fa0 0x55dcd4184410 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.443+0000 7fef9effd700 0 client.14682 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:17:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.444+0000 7fefd041db80 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 0x55dcd41755a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.444+0000 7fefd041db80 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd416f830 0x55dcd4172de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.444+0000 7fefd041db80 1 --2- 172.21.15.104:0/2368805368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd41728a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.444+0000 7fefd041db80 1 -- 172.21.15.104:0/2368805368 >> 172.21.15.104:0/2368805368 conn(0x55dcd4096030 msgr2=0x55dcd41636b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.444+0000 7fefd041db80 1 -- 172.21.15.104:0/2368805368 shutdown_connections 2024-06-09T22:17:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.444+0000 7fefd041db80 1 -- 172.21.15.104:0/2368805368 wait complete. 2024-06-09T22:17:16.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.444+0000 7fefd041db80 20 client.14682 trim_cache size 0 max 16384 2024-06-09T22:17:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefd041db80 1 Processor -- start 2024-06-09T22:17:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefd041db80 1 -- start start 2024-06-09T22:17:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefd041db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 0x55dcd4169210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefd041db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd4170320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefd041db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd416f830 0x55dcd4163e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefd041db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55dcd4182750 con 0x55dcd4169750 2024-06-09T22:17:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefd041db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dcd4182e20 con 0x55dcd416f830 2024-06-09T22:17:16.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefd041db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dcd41834f0 con 0x55dcd416ef10 2024-06-09T22:17:16.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefbf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 0x55dcd4169210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefbf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 0x55dcd4169210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44658/0 (socket says 172.21.15.104:44658) 2024-06-09T22:17:16.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefbf7fe700 1 -- 172.21.15.104:0/3803907199 learned_addr learned my addr 172.21.15.104:0/3803907199 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:16.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefbffff700 1 --2- 172.21.15.104:0/3803907199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd416f830 0x55dcd4163e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefb7fff700 1 --2- 172.21.15.104:0/3803907199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd4170320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefbf7fe700 1 -- 172.21.15.104:0/3803907199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 msgr2=0x55dcd4170320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:17:16.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefbf7fe700 1 --2- 172.21.15.104:0/3803907199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd4170320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefbf7fe700 1 -- 172.21.15.104:0/3803907199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd416f830 msgr2=0x55dcd4163e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:17:16.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefbf7fe700 1 --2- 172.21.15.104:0/3803907199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd416f830 0x55dcd4163e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefbf7fe700 1 -- 172.21.15.104:0/3803907199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcd4176210 con 0x55dcd4169750 2024-06-09T22:17:16.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefbffff700 1 --2- 172.21.15.104:0/3803907199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd416f830 0x55dcd4163e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:17:16.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.465+0000 7fefbf7fe700 1 --2- 172.21.15.104:0/3803907199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 0x55dcd4169210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009ce0 tx=0x7fefb0005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69e695fd9bb6b8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:16.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.466+0000 7fefb57fa700 1 -- 172.21.15.104:0/3803907199 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb00177e0 con 0x55dcd4169750 2024-06-09T22:17:16.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.466+0000 7fefb57fa700 1 -- 172.21.15.104:0/3803907199 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb0035070 con 0x55dcd4169750 2024-06-09T22:17:16.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.466+0000 7fefb57fa700 1 -- 172.21.15.104:0/3803907199 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb0030980 con 0x55dcd4169750 2024-06-09T22:17:16.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.466+0000 7fefd041db80 1 -- 172.21.15.104:0/3803907199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 msgr2=0x55dcd4169210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:16.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.466+0000 7fefd041db80 1 --2- 172.21.15.104:0/3803907199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 0x55dcd4169210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009ce0 tx=0x7fefb0005d40 comp rx=0 tx=0).stop 2024-06-09T22:17:16.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.466+0000 7fefd041db80 1 -- 172.21.15.104:0/3803907199 shutdown_connections 2024-06-09T22:17:16.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.466+0000 7fefd041db80 1 --2- 172.21.15.104:0/3803907199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd4169750 0x55dcd4169210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.466+0000 7fefd041db80 1 --2- 172.21.15.104:0/3803907199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd416f830 0x55dcd4163e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.466+0000 7fefd041db80 1 --2- 172.21.15.104:0/3803907199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd4170320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.466+0000 7fefd041db80 1 -- 172.21.15.104:0/3803907199 >> 172.21.15.104:0/3803907199 conn(0x55dcd4096030 msgr2=0x55dcd416fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:16.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.466+0000 7fefd041db80 1 -- 172.21.15.104:0/3803907199 shutdown_connections 2024-06-09T22:17:16.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.466+0000 7fefd041db80 1 -- 172.21.15.104:0/3803907199 wait complete. 2024-06-09T22:17:16.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefd041db80 1 -- start start 2024-06-09T22:17:16.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefd041db80 1 Processor -- start 2024-06-09T22:17:16.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefd041db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd4169750 0x55dcd418dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefd041db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd4163420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefd041db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 0x55dcd4163960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefd041db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55dcd4184290 con 0x55dcd416f830 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefd041db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dcd4183bc0 con 0x55dcd4169750 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefd041db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dcd4184960 con 0x55dcd416ef10 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefd041db80 10 client.0 _ll_register_callbacks cb 0x55dcd4050560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefbf7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd4169750 0x55dcd418dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefbffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 0x55dcd4163960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefbffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 0x55dcd4163960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44668/0 (socket says 172.21.15.104:44668) 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.467+0000 7fefbffff700 1 -- 172.21.15.104:0/1359652451 learned_addr learned my addr 172.21.15.104:0/1359652451 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fefb7fff700 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd4163420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fefbffff700 1 -- 172.21.15.104:0/1359652451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 msgr2=0x55dcd4163420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fefbffff700 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd4163420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fefbffff700 1 -- 172.21.15.104:0/1359652451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd4169750 msgr2=0x55dcd418dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:17:16.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fefbffff700 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd4169750 0x55dcd418dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fefbffff700 1 -- 172.21.15.104:0/1359652451 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcd4164070 con 0x55dcd416f830 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fefb7fff700 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd4163420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fefbffff700 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 0x55dcd4163960 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fefb8007a00 tx=0x7fefb8007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cdf88203b465a2a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fef9effd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fef9effd700 1 -- 172.21.15.104:0/1359652451 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb801c070 con 0x55dcd416f830 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fef9effd700 1 -- 172.21.15.104:0/1359652451 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8802+0+0 (secure 0 0 0) 0x7fefb8016a40 con 0x55dcd416f830 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fefd041db80 1 -- 172.21.15.104:0/1359652451 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefb0035070 con 0x55dcd416f830 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fef9effd700 1 -- 172.21.15.104:0/1359652451 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fefb8016c40 con 0x55dcd416f830 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fefd041db80 10 client.14688 fetch_fsmap learned FSMap version 17 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fefd041db80 1 -- 172.21.15.104:0/1359652451 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55dcd41ee830 con 0x55dcd416f830 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.468+0000 7fef9effd700 1 -- 172.21.15.104:0/1359652451 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb0035070 con 0x55dcd416f830 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.469+0000 7fef9effd700 1 -- 172.21.15.104:0/1359652451 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fefb8021460 con 0x55dcd416f830 2024-06-09T22:17:16.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.469+0000 7fefd041db80 10 client.14688 fetch_fsmap finished waiting for FSMap version 17 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.469+0000 7fefd041db80 10 client.14688 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.469+0000 7fefd041db80 20 client.14688 populate_metadata read hostname 'smithi104' 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.469+0000 7fefd041db80 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dcd4182fa0 0x55dcd41f2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.469+0000 7fefd041db80 4 client.14688 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.469+0000 7fefd041db80 1 -- 172.21.15.104:0/1359652451 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "status"}) -- 0x55dcd40c2280 con 0x55dcd4182fa0 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.469+0000 7fefbf7fe700 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dcd4182fa0 0x55dcd41f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.469+0000 7fefbf7fe700 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dcd4182fa0 0x55dcd41f2130 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b91f15786afba969 server_cookie=ef3b989b26553555 in_seq=0 out_seq=0 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.469+0000 7fef9effd700 10 client.14688 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fef9effd700 1 -- 172.21.15.104:0/1359652451 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55dcd40c2280 con 0x55dcd4182fa0 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fef9effd700 10 client.14688 handle_command_reply: tid=0 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefd041db80 1 client.14688 shutdown 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefd041db80 1 -- 172.21.15.104:0/1359652451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 msgr2=0x55dcd4163960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:16.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefd041db80 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 0x55dcd4163960 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fefb8007a00 tx=0x7fefb8007d10 comp rx=0 tx=0).stop 2024-06-09T22:17:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefd041db80 1 -- 172.21.15.104:0/1359652451 shutdown_connections 2024-06-09T22:17:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefd041db80 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dcd4182fa0 0x55dcd41f2130 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefd041db80 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dcd416f830 0x55dcd4163960 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefd041db80 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcd4169750 0x55dcd418dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefd041db80 1 --2- 172.21.15.104:0/1359652451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd416ef10 0x55dcd4163420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefd041db80 1 -- 172.21.15.104:0/1359652451 >> 172.21.15.104:0/1359652451 conn(0x55dcd4096030 msgr2=0x55dcd41740d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fef9effd700 0 client.14688 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:17:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefd041db80 1 -- 172.21.15.104:0/1359652451 shutdown_connections 2024-06-09T22:17:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefd041db80 1 -- 172.21.15.104:0/1359652451 wait complete. 2024-06-09T22:17:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefd041db80 20 client.14688 trim_cache size 0 max 16384 2024-06-09T22:17:16.471 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:17:16.472 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:17:16.472 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2024-06-09T22:17:16.472 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24460, 2024-06-09T22:17:16.472 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:17:16.472 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:17:16.472 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:17:16.472 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 31.267949494, 2024-06-09T22:17:16.472 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 17, 2024-06-09T22:17:16.472 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 95, 2024-06-09T22:17:16.472 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 94, 2024-06-09T22:17:16.472 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 56.788993574000003 2024-06-09T22:17:16.472 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:17:16.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefc5b9a700 1 -- 172.21.15.104:0/989515800 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fefac0a2c20 msgr2=0x7fefac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:16.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefc5b9a700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fefac0a2c20 0x7fefac0a50e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fefb000b5c0 tx=0x7fefb0005960 comp rx=0 tx=0).stop 2024-06-09T22:17:16.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefc5b9a700 1 -- 172.21.15.104:0/989515800 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefc0101d40 msgr2=0x7fefc019b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:16.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.470+0000 7fefc5b9a700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefc0101d40 0x7fefc019b390 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fefa800d980 tx=0x7fefa800dc90 comp rx=0 tx=0).stop 2024-06-09T22:17:16.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.471+0000 7fefc5b9a700 1 -- 172.21.15.104:0/989515800 shutdown_connections 2024-06-09T22:17:16.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.471+0000 7fefc5b9a700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fefac0a2c20 0x7fefac0a50e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.471+0000 7fefc5b9a700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fefc01073f0 0x7fefc019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.471+0000 7fefc5b9a700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc0101380 0x7fefc019ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.471+0000 7fefc5b9a700 1 --2- 172.21.15.104:0/989515800 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefc0101d40 0x7fefc019b390 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:16.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.471+0000 7fefc5b9a700 1 -- 172.21.15.104:0/989515800 >> 172.21.15.104:0/989515800 conn(0x7fefc0078300 msgr2=0x7fefc006a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:16.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.471+0000 7fefc5b9a700 1 -- 172.21.15.104:0/989515800 shutdown_connections 2024-06-09T22:17:16.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:16.471+0000 7fefc5b9a700 1 -- 172.21.15.104:0/989515800 wait complete. 2024-06-09T22:17:16.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:16 smithi153 ceph-mon[36599]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:16.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:16 smithi194 ceph-mon[36856]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:17.104 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-09T22:17:17.137 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-06-09T22:17:17.137 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-06-09T22:17:17.378 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:18.477 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:18 smithi104 ceph-mon[32389]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:18.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:18 smithi153 ceph-mon[36599]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:18.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:18 smithi194 ceph-mon[36856]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:19.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.019+0000 7f72659d1700 1 -- 172.21.15.104:0/276738049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7260102740 msgr2=0x7f7260102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:19.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.019+0000 7f72659d1700 1 --2- 172.21.15.104:0/276738049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7260102740 0x7f7260102b20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f72480099c0 tx=0x7f7248009cd0 comp rx=0 tx=0).stop 2024-06-09T22:17:19.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.019+0000 7f72659d1700 1 -- 172.21.15.104:0/276738049 shutdown_connections 2024-06-09T22:17:19.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.019+0000 7f72659d1700 1 --2- 172.21.15.104:0/276738049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7260102740 0x7f7260102b20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:19.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.019+0000 7f72659d1700 1 --2- 172.21.15.104:0/276738049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7260107840 0x7f726010f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:19.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.019+0000 7f72659d1700 1 --2- 172.21.15.104:0/276738049 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72601030f0 0x7f7260107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:19.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.019+0000 7f72659d1700 1 -- 172.21.15.104:0/276738049 >> 172.21.15.104:0/276738049 conn(0x7f72600795f0 msgr2=0x7f7260079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:19.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.019+0000 7f72659d1700 1 -- 172.21.15.104:0/276738049 shutdown_connections 2024-06-09T22:17:19.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.019+0000 7f72659d1700 1 -- 172.21.15.104:0/276738049 wait complete. 2024-06-09T22:17:19.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.020+0000 7f72659d1700 1 Processor -- start 2024-06-09T22:17:19.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.020+0000 7f72659d1700 1 -- start start 2024-06-09T22:17:19.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.020+0000 7f72659d1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7260102740 0x7f726019e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:19.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.020+0000 7f72659d1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72601030f0 0x7f726019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:19.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.020+0000 7f72659d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7260107840 0x7f7260198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:19.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.020+0000 7f72659d1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7260112780 con 0x7f7260107840 2024-06-09T22:17:19.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.020+0000 7f72659d1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7260112600 con 0x7f72601030f0 2024-06-09T22:17:19.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.020+0000 7f72659d1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7260112900 con 0x7f7260102740 2024-06-09T22:17:19.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.020+0000 7f725f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7260107840 0x7f7260198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:19.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.020+0000 7f725f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7260107840 0x7f7260198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44680/0 (socket says 172.21.15.104:44680) 2024-06-09T22:17:19.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.020+0000 7f725f7fe700 1 -- 172.21.15.104:0/2361792600 learned_addr learned my addr 172.21.15.104:0/2361792600 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:19.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.020+0000 7f725effd700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7260102740 0x7f726019e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:19.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.021+0000 7f725e7fc700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72601030f0 0x7f726019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:19.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.021+0000 7f725f7fe700 1 -- 172.21.15.104:0/2361792600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7260102740 msgr2=0x7f726019e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:19.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.021+0000 7f725f7fe700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7260102740 0x7f726019e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:19.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.021+0000 7f725f7fe700 1 -- 172.21.15.104:0/2361792600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72601030f0 msgr2=0x7f726019edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:19.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.021+0000 7f725f7fe700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72601030f0 0x7f726019edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:19.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.021+0000 7f725f7fe700 1 -- 172.21.15.104:0/2361792600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72601991a0 con 0x7f7260107840 2024-06-09T22:17:19.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.021+0000 7f725f7fe700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7260107840 0x7f7260198a40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f725800ada0 tx=0x7f725800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:19.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.021+0000 7f72649cf700 1 -- 172.21.15.104:0/2361792600 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7258017020 con 0x7f7260107840 2024-06-09T22:17:19.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.021+0000 7f72659d1700 1 -- 172.21.15.104:0/2361792600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7260199490 con 0x7f7260107840 2024-06-09T22:17:19.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.021+0000 7f72649cf700 1 -- 172.21.15.104:0/2361792600 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f725800a0d0 con 0x7f7260107840 2024-06-09T22:17:19.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.021+0000 7f72649cf700 1 -- 172.21.15.104:0/2361792600 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7258005480 con 0x7f7260107840 2024-06-09T22:17:19.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.021+0000 7f72659d1700 1 -- 172.21.15.104:0/2361792600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72601a5750 con 0x7f7260107840 2024-06-09T22:17:19.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.022+0000 7f72659d1700 1 -- 172.21.15.104:0/2361792600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f726010aa80 con 0x7f7260107840 2024-06-09T22:17:19.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.023+0000 7f72649cf700 1 -- 172.21.15.104:0/2361792600 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7258005620 con 0x7f7260107840 2024-06-09T22:17:19.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.024+0000 7f72649cf700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f724c0a2c20 0x7f724c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:19.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.024+0000 7f725effd700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f724c0a2c20 0x7f724c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:19.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.024+0000 7f72649cf700 1 -- 172.21.15.104:0/2361792600 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8802+0+0 (secure 0 0 0) 0x7f72580cefb0 con 0x7f7260107840 2024-06-09T22:17:19.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.024+0000 7f725effd700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f724c0a2c20 0x7f724c0a50e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7248000c00 tx=0x7f7248005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:19.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.027+0000 7f72649cf700 1 -- 172.21.15.104:0/2361792600 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7258021020 con 0x7f7260107840 2024-06-09T22:17:19.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.269+0000 7f72659d1700 1 -- 172.21.15.104:0/2361792600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7260069180 con 0x7f7260107840 2024-06-09T22:17:19.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.269+0000 7f72649cf700 1 -- 172.21.15.104:0/2361792600 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f725809bd60 con 0x7f7260107840 2024-06-09T22:17:19.265 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-06-09T22:17:19.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.272+0000 7f72659d1700 1 -- 172.21.15.104:0/2361792600 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f724c0a2c20 msgr2=0x7f724c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:19.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.272+0000 7f72659d1700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f724c0a2c20 0x7f724c0a50e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7248000c00 tx=0x7f7248005fb0 comp rx=0 tx=0).stop 2024-06-09T22:17:19.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.272+0000 7f72659d1700 1 -- 172.21.15.104:0/2361792600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7260107840 msgr2=0x7f7260198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:19.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.272+0000 7f72659d1700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7260107840 0x7f7260198a40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f725800ada0 tx=0x7f725800c330 comp rx=0 tx=0).stop 2024-06-09T22:17:19.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.272+0000 7f72659d1700 1 -- 172.21.15.104:0/2361792600 shutdown_connections 2024-06-09T22:17:19.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.272+0000 7f72659d1700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f724c0a2c20 0x7f724c0a50e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:19.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.272+0000 7f72659d1700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7260107840 0x7f7260198a40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:19.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.272+0000 7f72659d1700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7260102740 0x7f726019e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:19.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.272+0000 7f72659d1700 1 --2- 172.21.15.104:0/2361792600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72601030f0 0x7f726019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:19.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.272+0000 7f72659d1700 1 -- 172.21.15.104:0/2361792600 >> 172.21.15.104:0/2361792600 conn(0x7f72600795f0 msgr2=0x7f7260108990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:19.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.272+0000 7f72659d1700 1 -- 172.21.15.104:0/2361792600 shutdown_connections 2024-06-09T22:17:19.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:19.272+0000 7f72659d1700 1 -- 172.21.15.104:0/2361792600 wait complete. 2024-06-09T22:17:19.575 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:19 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2361792600' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-09T22:17:19.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:19 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2361792600' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-09T22:17:19.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:19 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2361792600' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-09T22:17:19.928 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-09T22:17:19.928 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-09T22:17:19.928 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2024-06-09T22:17:19.928 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-06-09T22:17:19.928 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-06-09T22:17:19.929 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-09T22:17:19.929 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-09T22:17:19.929 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-09T22:17:19.929 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-06-09T22:17:19.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:19.929 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2024-06-09T22:17:19.946 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-09T22:17:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-09T22:17:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2024-06-09T22:17:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-06-09T22:17:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2024-06-09T22:17:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-06-09T22:17:19.947 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-09T22:17:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2024-06-09T22:17:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: altname enp7s0f0 2024-06-09T22:17:19.947 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-09T22:17:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2024-06-09T22:17:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: altname enp7s0f1 2024-06-09T22:17:19.947 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-09T22:17:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2024-06-09T22:17:19.947 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-09T22:17:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2024-06-09T22:17:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-09T22:17:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 42205sec preferred_lft 42205sec 2024-06-09T22:17:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::ec4:7aff:fe8f:ce89/64 scope link 2024-06-09T22:17:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-06-09T22:17:19.948 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-06-09T22:17:19.948 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-09T22:17:19.948 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-06-09T22:17:19.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add name ceph-brx type bridge 2024-06-09T22:17:19.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr flush dev ceph-brx 2024-06-09T22:17:19.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set ceph-brx up 2024-06-09T22:17:19.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-06-09T22:17:19.949 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-06-09T22:17:20.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-09T22:17:20.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:17:20.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:20.159 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-09T22:17:20.184 INFO:teuthology.orchestra.run.smithi104.stdout:1 2024-06-09T22:17:20.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:20.185 DEBUG:teuthology.orchestra.run.smithi104:> route 2024-06-09T22:17:20.249 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2024-06-09T22:17:20.249 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-09T22:17:20.249 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-09T22:17:20.249 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-09T22:17:20.249 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-09T22:17:20.250 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-09T22:17:20.250 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-06-09T22:17:20.250 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-09T22:17:20.250 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-09T22:17:20.250 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-09T22:17:20.250 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-06-09T22:17:20.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-09T22:17:20.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:17:20.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:20.557 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2024-06-09T22:17:20.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:20.572 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list-id 2024-06-09T22:17:20.627 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-09T22:17:20.627 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-06-09T22:17:20.627 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-09T22:17:20.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-06-09T22:17:20.628 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-06-09T22:17:20.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-09T22:17:20.725 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:20 smithi104 ceph-mon[32389]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:20.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:20 smithi153 ceph-mon[36599]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:20.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:17:20.746 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-06-09T22:17:20.747 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-09T22:17:20.747 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-06-09T22:17:20.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-06-09T22:17:20.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-06-09T22:17:20.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-06-09T22:17:20.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-06-09T22:17:20.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-06-09T22:17:20.747 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-06-09T22:17:20.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-09T22:17:20.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:20 smithi194 ceph-mon[36856]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:20.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:17:20.871 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-09T22:17:20.871 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-06-09T22:17:20.871 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2024-06-09T22:17:20.871 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set dev brx.0 master ceph-brx 2024-06-09T22:17:20.871 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-06-09T22:17:20.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-09T22:17:20.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:17:20.949 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-06-09T22:17:20.949 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-09T22:17:20.949 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-06-09T22:17:20.968 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-06-09T22:17:20.968 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-09T22:17:20.968 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-06-09T22:17:21.025 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2024-06-09T22:17:21.090 DEBUG:teuthology.orchestra.run.smithi104:> cat /proc/self/mounts | awk '{print $2}' 2024-06-09T22:17:21.147 INFO:teuthology.orchestra.run.smithi104.stdout:/proc 2024-06-09T22:17:21.147 INFO:teuthology.orchestra.run.smithi104.stdout:/sys 2024-06-09T22:17:21.148 INFO:teuthology.orchestra.run.smithi104.stdout:/dev 2024-06-09T22:17:21.148 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/security 2024-06-09T22:17:21.148 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/shm 2024-06-09T22:17:21.148 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/pts 2024-06-09T22:17:21.148 INFO:teuthology.orchestra.run.smithi104.stdout:/run 2024-06-09T22:17:21.148 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/cgroup 2024-06-09T22:17:21.148 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/pstore 2024-06-09T22:17:21.148 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/bpf 2024-06-09T22:17:21.148 INFO:teuthology.orchestra.run.smithi104.stdout:/ 2024-06-09T22:17:21.148 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/selinux 2024-06-09T22:17:21.148 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-06-09T22:17:21.148 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/mqueue 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/hugepages 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/debug 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/tracing 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/fuse/connections 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/config 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-sysctl.service 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/nfs/rpc_pipefs 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/1000 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay 2024-06-09T22:17:21.149 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/0f7e1440791c76273fce7982c525389690bc1862c02cb6d208f1090aac2e191a/merged 2024-06-09T22:17:21.150 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/f22e0e1229910423baf5b457c66c352310b80d7e8f3e6d8ac857ae5f9c053700/merged 2024-06-09T22:17:21.150 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/0 2024-06-09T22:17:21.150 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-06-09T22:17:21.150 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/43e008abbb3482239f1333d7ac09ee5033283f51e3424d812b82c6899dc7b5ad/merged 2024-06-09T22:17:21.150 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/895bbdc72fbf425dc2cbb17558df2f88e4ffa449a49c8a817cf4a2a29f1f3aec/merged 2024-06-09T22:17:21.150 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/58e1873fad927e1d647e4a36ee7ee7dcbf162848e715a812cad36cc563183ca7/merged 2024-06-09T22:17:21.150 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/1ee8996428d2e5914b71e3bbf048ff0c7cd16fd52182359ed9aea3e5507adaec/merged 2024-06-09T22:17:21.150 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/284431a5b2490c43dacf5f86b6acd552dd2df3c112c5fa5a8ad2c58eb3da7ccb/merged 2024-06-09T22:17:21.150 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/17fe93428fdbfaa855a668206ce1696785a69c17f4851999b2ec4f84ea255aac/merged 2024-06-09T22:17:21.150 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/1143e1e2b0f9a8fc44fa95a41bc4899111b2c497799ec95124cff22fea683f41/merged 2024-06-09T22:17:21.150 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/e2382cbe10ebd5f09768e448eef015f26fa6cc63d89958982c917481175eb4c1/merged 2024-06-09T22:17:21.150 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns 2024-06-09T22:17:21.151 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-09T22:17:21.151 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-09T22:17:21.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:21.151 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2024-06-09T22:17:21.203 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-06-09T22:17:21.203 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-06-09T22:17:21.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2024-06-09T22:17:21.269 DEBUG:teuthology.orchestra.run.smithi104:> cat /proc/self/mounts | awk '{print $2}' 2024-06-09T22:17:21.325 INFO:teuthology.orchestra.run.smithi104.stdout:/proc 2024-06-09T22:17:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:/sys 2024-06-09T22:17:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:/dev 2024-06-09T22:17:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/security 2024-06-09T22:17:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/shm 2024-06-09T22:17:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/pts 2024-06-09T22:17:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:/run 2024-06-09T22:17:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/cgroup 2024-06-09T22:17:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/pstore 2024-06-09T22:17:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/bpf 2024-06-09T22:17:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:/ 2024-06-09T22:17:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/selinux 2024-06-09T22:17:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/mqueue 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/hugepages 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/debug 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/tracing 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/fuse/connections 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/config 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-sysctl.service 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/nfs/rpc_pipefs 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/1000 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay 2024-06-09T22:17:21.327 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/0f7e1440791c76273fce7982c525389690bc1862c02cb6d208f1090aac2e191a/merged 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/f22e0e1229910423baf5b457c66c352310b80d7e8f3e6d8ac857ae5f9c053700/merged 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/0 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/43e008abbb3482239f1333d7ac09ee5033283f51e3424d812b82c6899dc7b5ad/merged 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/895bbdc72fbf425dc2cbb17558df2f88e4ffa449a49c8a817cf4a2a29f1f3aec/merged 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/58e1873fad927e1d647e4a36ee7ee7dcbf162848e715a812cad36cc563183ca7/merged 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/1ee8996428d2e5914b71e3bbf048ff0c7cd16fd52182359ed9aea3e5507adaec/merged 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/284431a5b2490c43dacf5f86b6acd552dd2df3c112c5fa5a8ad2c58eb3da7ccb/merged 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/17fe93428fdbfaa855a668206ce1696785a69c17f4851999b2ec4f84ea255aac/merged 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/1143e1e2b0f9a8fc44fa95a41bc4899111b2c497799ec95124cff22fea683f41/merged 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/e2382cbe10ebd5f09768e448eef015f26fa6cc63d89958982c917481175eb4c1/merged 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns 2024-06-09T22:17:21.328 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-09T22:17:21.329 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-09T22:17:21.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:21.329 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2024-06-09T22:17:21.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi104.stderr:2024-06-09T22:17:21.392+0000 7f1e1e725480 -1 init, newargv = 0x55b1a09b4810 newargc=15 2024-06-09T22:17:21.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi104.stderr:ceph-fuse[75847]: starting ceph client 2024-06-09T22:17:21.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi104.stderr:ceph-fuse[75847]: starting fuse 2024-06-09T22:17:22.484 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2024-06-09T22:17:22.516 DEBUG:teuthology.orchestra.run.smithi104:> cat /proc/self/mounts | awk '{print $2}' 2024-06-09T22:17:22.575 INFO:teuthology.orchestra.run.smithi104.stdout:/proc 2024-06-09T22:17:22.575 INFO:teuthology.orchestra.run.smithi104.stdout:/sys 2024-06-09T22:17:22.575 INFO:teuthology.orchestra.run.smithi104.stdout:/dev 2024-06-09T22:17:22.575 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/security 2024-06-09T22:17:22.575 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/shm 2024-06-09T22:17:22.575 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/pts 2024-06-09T22:17:22.575 INFO:teuthology.orchestra.run.smithi104.stdout:/run 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/cgroup 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/pstore 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/bpf 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/ 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/selinux 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/mqueue 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/hugepages 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/debug 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/tracing 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/fuse/connections 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/config 2024-06-09T22:17:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-sysctl.service 2024-06-09T22:17:22.577 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-06-09T22:17:22.577 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/nfs/rpc_pipefs 2024-06-09T22:17:22.577 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/1000 2024-06-09T22:17:22.577 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph 2024-06-09T22:17:22.577 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay 2024-06-09T22:17:22.577 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/0f7e1440791c76273fce7982c525389690bc1862c02cb6d208f1090aac2e191a/merged 2024-06-09T22:17:22.577 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/f22e0e1229910423baf5b457c66c352310b80d7e8f3e6d8ac857ae5f9c053700/merged 2024-06-09T22:17:22.577 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/0 2024-06-09T22:17:22.577 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-06-09T22:17:22.577 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/43e008abbb3482239f1333d7ac09ee5033283f51e3424d812b82c6899dc7b5ad/merged 2024-06-09T22:17:22.577 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/895bbdc72fbf425dc2cbb17558df2f88e4ffa449a49c8a817cf4a2a29f1f3aec/merged 2024-06-09T22:17:22.577 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/58e1873fad927e1d647e4a36ee7ee7dcbf162848e715a812cad36cc563183ca7/merged 2024-06-09T22:17:22.578 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/1ee8996428d2e5914b71e3bbf048ff0c7cd16fd52182359ed9aea3e5507adaec/merged 2024-06-09T22:17:22.578 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/284431a5b2490c43dacf5f86b6acd552dd2df3c112c5fa5a8ad2c58eb3da7ccb/merged 2024-06-09T22:17:22.578 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/17fe93428fdbfaa855a668206ce1696785a69c17f4851999b2ec4f84ea255aac/merged 2024-06-09T22:17:22.578 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/1143e1e2b0f9a8fc44fa95a41bc4899111b2c497799ec95124cff22fea683f41/merged 2024-06-09T22:17:22.578 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/containers/storage/overlay/e2382cbe10ebd5f09768e448eef015f26fa6cc63d89958982c917481175eb4c1/merged 2024-06-09T22:17:22.578 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns 2024-06-09T22:17:22.578 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-09T22:17:22.578 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-09T22:17:22.578 INFO:teuthology.orchestra.run.smithi104.stdout:/home/ubuntu/cephtest/mnt.admin 2024-06-09T22:17:22.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:22.579 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2024-06-09T22:17:22.632 INFO:teuthology.orchestra.run.smithi104.stdout:107 2024-06-09T22:17:22.633 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [107] 2024-06-09T22:17:22.633 DEBUG:teuthology.orchestra.run.smithi104:> sudo stdin-killer -- python3 -c ' 2024-06-09T22:17:22.633 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2024-06-09T22:17:22.633 DEBUG:teuthology.orchestra.run.smithi104:> import re 2024-06-09T22:17:22.633 DEBUG:teuthology.orchestra.run.smithi104:> import os 2024-06-09T22:17:22.633 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2024-06-09T22:17:22.633 DEBUG:teuthology.orchestra.run.smithi104:> 2024-06-09T22:17:22.633 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2024-06-09T22:17:22.633 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-06-09T22:17:22.634 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2024-06-09T22:17:22.635 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2024-06-09T22:17:22.635 DEBUG:teuthology.orchestra.run.smithi104:> return f 2024-06-09T22:17:22.635 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-06-09T22:17:22.635 DEBUG:teuthology.orchestra.run.smithi104:> 2024-06-09T22:17:22.635 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.admin")) 2024-06-09T22:17:22.635 DEBUG:teuthology.orchestra.run.smithi104:> ' 2024-06-09T22:17:22.696 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:22 smithi104 ceph-mon[32389]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:22.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:22 smithi153 ceph-mon[36599]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:22.757 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.admin.75847.asok 2024-06-09T22:17:22.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:17:22.768 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75847.asok 2024-06-09T22:17:22.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:22.768 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75847.asok status 2024-06-09T22:17:22.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:22 smithi194 ceph-mon[36856]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:22.902 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:17:22.902 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)", 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "admin", 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "75847", 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 14700, 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2024-06-09T22:17:22.903 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 14700 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3238064086 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3238064086 2024-06-09T22:17:22.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:17:22.905 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.14700 192.168.0.1:0/3238064086", 2024-06-09T22:17:22.905 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/3238064086", 2024-06-09T22:17:22.905 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2024-06-09T22:17:22.905 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 17, 2024-06-09T22:17:22.905 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 95, 2024-06-09T22:17:22.905 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2024-06-09T22:17:22.905 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2024-06-09T22:17:22.905 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2024-06-09T22:17:22.905 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:17:22.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:22.911 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2024-06-09T22:17:22.911 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-06-09T22:17:22.927 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2024-06-09T22:17:22.927 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-06-09T22:17:22.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:22.927 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-06-09T22:17:22.993 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-06-09T22:17:23.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-09T22:17:23.101 INFO:teuthology.orchestra.run.smithi104.stdout:# file: . 2024-06-09T22:17:23.101 INFO:teuthology.orchestra.run.smithi104.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-06-09T22:17:23.101 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:17:23.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:17:23.107 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:17:23.107 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-09T22:17:23.124 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-06-09T22:17:23.408 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:24.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:24 smithi153 ceph-mon[36599]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-06-09T22:17:24.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:24 smithi104 ceph-mon[32389]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-06-09T22:17:24.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:24 smithi194 ceph-mon[36856]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-06-09T22:17:25.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.139+0000 7fb675431700 1 -- 172.21.15.104:0/283990514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb670108500 msgr2=0x7fb6701088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:25.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.139+0000 7fb675431700 1 --2- 172.21.15.104:0/283990514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb670108500 0x7fb6701088e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb660009990 tx=0x7fb660009ca0 comp rx=0 tx=0).stop 2024-06-09T22:17:25.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.139+0000 7fb675431700 1 -- 172.21.15.104:0/283990514 shutdown_connections 2024-06-09T22:17:25.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.139+0000 7fb675431700 1 --2- 172.21.15.104:0/283990514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb670108500 0x7fb6701088e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:25.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.139+0000 7fb675431700 1 --2- 172.21.15.104:0/283990514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb67006ab30 0x7fb67010f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:25.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.139+0000 7fb675431700 1 --2- 172.21.15.104:0/283990514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb67006a170 0x7fb67006a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:25.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.139+0000 7fb675431700 1 -- 172.21.15.104:0/283990514 >> 172.21.15.104:0/283990514 conn(0x7fb670078340 msgr2=0x7fb670078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:25.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.139+0000 7fb675431700 1 -- 172.21.15.104:0/283990514 shutdown_connections 2024-06-09T22:17:25.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.140+0000 7fb675431700 1 -- 172.21.15.104:0/283990514 wait complete. 2024-06-09T22:17:25.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.140+0000 7fb675431700 1 Processor -- start 2024-06-09T22:17:25.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.140+0000 7fb675431700 1 -- start start 2024-06-09T22:17:25.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.140+0000 7fb675431700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb67006a170 0x7fb67019af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:25.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.140+0000 7fb675431700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67006ab30 0x7fb67019b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:25.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.140+0000 7fb675431700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb670108500 0x7fb67019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:25.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.140+0000 7fb675431700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb670112860 con 0x7fb67006ab30 2024-06-09T22:17:25.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.140+0000 7fb675431700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb6701126e0 con 0x7fb67006a170 2024-06-09T22:17:25.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.140+0000 7fb675431700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb6701129e0 con 0x7fb670108500 2024-06-09T22:17:25.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb66e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67006ab30 0x7fb67019b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:25.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb66e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67006ab30 0x7fb67019b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44698/0 (socket says 172.21.15.104:44698) 2024-06-09T22:17:25.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb66e7fc700 1 -- 172.21.15.104:0/292703408 learned_addr learned my addr 172.21.15.104:0/292703408 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:25.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb66effd700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb67006a170 0x7fb67019af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:25.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb66f7fe700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb670108500 0x7fb67019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb66e7fc700 1 -- 172.21.15.104:0/292703408 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb670108500 msgr2=0x7fb67019f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb66e7fc700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb670108500 0x7fb67019f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb66e7fc700 1 -- 172.21.15.104:0/292703408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb67006a170 msgr2=0x7fb67019af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb66e7fc700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb67006a170 0x7fb67019af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb66e7fc700 1 -- 172.21.15.104:0/292703408 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67019ffa0 con 0x7fb67006ab30 2024-06-09T22:17:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb66f7fe700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb670108500 0x7fb67019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:17:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb66e7fc700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67006ab30 0x7fb67019b4d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb66400b530 tx=0x7fb66400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb657fff700 1 -- 172.21.15.104:0/292703408 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664015020 con 0x7fb67006ab30 2024-06-09T22:17:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb675431700 1 -- 172.21.15.104:0/292703408 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6701a0290 con 0x7fb67006ab30 2024-06-09T22:17:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb657fff700 1 -- 172.21.15.104:0/292703408 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb664011d10 con 0x7fb67006ab30 2024-06-09T22:17:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb657fff700 1 -- 172.21.15.104:0/292703408 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66400f2e0 con 0x7fb67006ab30 2024-06-09T22:17:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.141+0000 7fb675431700 1 -- 172.21.15.104:0/292703408 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6701a7b10 con 0x7fb67006ab30 2024-06-09T22:17:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.142+0000 7fb675431700 1 -- 172.21.15.104:0/292703408 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb670069180 con 0x7fb67006ab30 2024-06-09T22:17:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.143+0000 7fb657fff700 1 -- 172.21.15.104:0/292703408 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb664018020 con 0x7fb67006ab30 2024-06-09T22:17:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.144+0000 7fb657fff700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb6580a2c20 0x7fb6580a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.144+0000 7fb66effd700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb6580a2c20 0x7fb6580a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.144+0000 7fb657fff700 1 -- 172.21.15.104:0/292703408 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8802+0+0 (secure 0 0 0) 0x7fb6640cef40 con 0x7fb67006ab30 2024-06-09T22:17:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.144+0000 7fb66effd700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb6580a2c20 0x7fb6580a50e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb66000b5c0 tx=0x7fb660005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.148+0000 7fb657fff700 1 -- 172.21.15.104:0/292703408 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb66401f020 con 0x7fb67006ab30 2024-06-09T22:17:25.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.367+0000 7fb675431700 1 -- 172.21.15.104:0/292703408 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb6701a7da0 con 0x7fb67006ab30 2024-06-09T22:17:25.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.367+0000 7fb657fff700 1 -- 172.21.15.104:0/292703408 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v95) ==== 81+0+598 (secure 0 0 0) 0x7fb66409bc60 con 0x7fb67006ab30 2024-06-09T22:17:25.364 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6801/85752866 2024-06-10T22:09:25.221731+0000 2024-06-09T22:17:25.364 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/3559678620 2024-06-10T22:08:58.969000+0000 2024-06-09T22:17:25.364 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/3864904957 2024-06-10T22:09:25.221731+0000 2024-06-09T22:17:25.364 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6800/85752866 2024-06-10T22:09:25.221731+0000 2024-06-09T22:17:25.364 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6801/1879478526 2024-06-10T22:08:58.969000+0000 2024-06-09T22:17:25.364 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/2471278860 2024-06-10T22:08:58.969000+0000 2024-06-09T22:17:25.364 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/4119286692 2024-06-10T22:09:25.221731+0000 2024-06-09T22:17:25.364 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/2266338289 2024-06-10T22:08:58.969000+0000 2024-06-09T22:17:25.364 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6800/1879478526 2024-06-10T22:08:58.969000+0000 2024-06-09T22:17:25.364 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/3072042273 2024-06-10T22:09:25.221731+0000 2024-06-09T22:17:25.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.370+0000 7fb675431700 1 -- 172.21.15.104:0/292703408 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb6580a2c20 msgr2=0x7fb6580a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:25.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.370+0000 7fb675431700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb6580a2c20 0x7fb6580a50e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb66000b5c0 tx=0x7fb660005960 comp rx=0 tx=0).stop 2024-06-09T22:17:25.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.370+0000 7fb675431700 1 -- 172.21.15.104:0/292703408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67006ab30 msgr2=0x7fb67019b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:25.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.370+0000 7fb675431700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67006ab30 0x7fb67019b4d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb66400b530 tx=0x7fb66400b840 comp rx=0 tx=0).stop 2024-06-09T22:17:25.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.370+0000 7fb675431700 1 -- 172.21.15.104:0/292703408 shutdown_connections 2024-06-09T22:17:25.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.370+0000 7fb675431700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb6580a2c20 0x7fb6580a50e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:25.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.370+0000 7fb675431700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb670108500 0x7fb67019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:25.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.370+0000 7fb675431700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb67006a170 0x7fb67019af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:25.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.370+0000 7fb675431700 1 --2- 172.21.15.104:0/292703408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67006ab30 0x7fb67019b4d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:25.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.370+0000 7fb675431700 1 -- 172.21.15.104:0/292703408 >> 172.21.15.104:0/292703408 conn(0x7fb670078340 msgr2=0x7fb67010d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:25.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.371+0000 7fb675431700 1 -- 172.21.15.104:0/292703408 shutdown_connections 2024-06-09T22:17:25.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:25.371+0000 7fb675431700 1 -- 172.21.15.104:0/292703408 wait complete. 2024-06-09T22:17:25.368 INFO:teuthology.orchestra.run.smithi104.stderr:listed 10 entries 2024-06-09T22:17:25.737 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-06-09T22:17:25.737 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-06-09T22:17:25.738 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]: dispatch 2024-06-09T22:17:25.738 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-06-09T22:17:25.738 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-06-09T22:17:25.738 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-06-09T22:17:25.738 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-06-09T22:17:25.738 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-06-09T22:17:25.738 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-06-09T22:17:25.738 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-06-09T22:17:25.738 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:25 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/292703408' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-09T22:17:25.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-06-09T22:17:25.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-06-09T22:17:25.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]: dispatch 2024-06-09T22:17:25.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-06-09T22:17:25.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-06-09T22:17:25.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-06-09T22:17:25.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-06-09T22:17:25.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-06-09T22:17:25.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-06-09T22:17:25.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-06-09T22:17:25.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:25 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/292703408' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-09T22:17:25.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-06-09T22:17:25.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-06-09T22:17:25.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]: dispatch 2024-06-09T22:17:25.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-06-09T22:17:25.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-06-09T22:17:25.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-06-09T22:17:25.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-06-09T22:17:25.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-06-09T22:17:25.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-06-09T22:17:25.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-06-09T22:17:25.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:25 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/292703408' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-09T22:17:26.066 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:17:26.066 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-09T22:17:26.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-06-09T22:17:26.363 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:26.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:26 smithi153 ceph-mon[36599]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-06-09T22:17:26.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-06-09T22:17:26.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-06-09T22:17:26.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]': finished 2024-06-09T22:17:26.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-06-09T22:17:26.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-06-09T22:17:26.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-06-09T22:17:26.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-06-09T22:17:26.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-06-09T22:17:26.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-06-09T22:17:26.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-06-09T22:17:26.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:26 smithi153 ceph-mon[36599]: osdmap e96: 12 total, 12 up, 12 in 2024-06-09T22:17:26.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:26 smithi104 ceph-mon[32389]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-06-09T22:17:26.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-06-09T22:17:26.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-06-09T22:17:26.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]': finished 2024-06-09T22:17:26.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-06-09T22:17:26.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-06-09T22:17:26.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-06-09T22:17:26.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-06-09T22:17:26.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-06-09T22:17:26.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-06-09T22:17:26.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-06-09T22:17:26.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:26 smithi104 ceph-mon[32389]: osdmap e96: 12 total, 12 up, 12 in 2024-06-09T22:17:26.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:26 smithi194 ceph-mon[36856]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-06-09T22:17:26.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-06-09T22:17:26.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-06-09T22:17:26.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]': finished 2024-06-09T22:17:26.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-06-09T22:17:26.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-06-09T22:17:26.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-06-09T22:17:26.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-06-09T22:17:26.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-06-09T22:17:26.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-06-09T22:17:26.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-06-09T22:17:26.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:26 smithi194 ceph-mon[36856]: osdmap e96: 12 total, 12 up, 12 in 2024-06-09T22:17:27.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:27 smithi104 ceph-mon[32389]: osdmap e97: 12 total, 12 up, 12 in 2024-06-09T22:17:27.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:27 smithi153 ceph-mon[36599]: osdmap e97: 12 total, 12 up, 12 in 2024-06-09T22:17:27.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:27 smithi194 ceph-mon[36856]: osdmap e97: 12 total, 12 up, 12 in 2024-06-09T22:17:28.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.092+0000 7f9bc7533700 1 -- 172.21.15.104:0/1399874762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc01066d0 msgr2=0x7f9bc010e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:28.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.092+0000 7f9bc7533700 1 --2- 172.21.15.104:0/1399874762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc01066d0 0x7f9bc010e2a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800b0d0 tx=0x7f9bb800b3e0 comp rx=0 tx=0).stop 2024-06-09T22:17:28.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.092+0000 7f9bc7533700 1 -- 172.21.15.104:0/1399874762 shutdown_connections 2024-06-09T22:17:28.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.092+0000 7f9bc7533700 1 --2- 172.21.15.104:0/1399874762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc01066d0 0x7f9bc010e2a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:28.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.092+0000 7f9bc7533700 1 --2- 172.21.15.104:0/1399874762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bc0101680 0x7f9bc0101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:28.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.092+0000 7f9bc7533700 1 --2- 172.21.15.104:0/1399874762 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bc0102030 0x7f9bc0105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:28.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.092+0000 7f9bc7533700 1 -- 172.21.15.104:0/1399874762 >> 172.21.15.104:0/1399874762 conn(0x7f9bc00773e0 msgr2=0x7f9bc00777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:28.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.092+0000 7f9bc7533700 1 -- 172.21.15.104:0/1399874762 shutdown_connections 2024-06-09T22:17:28.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.092+0000 7f9bc7533700 1 -- 172.21.15.104:0/1399874762 wait complete. 2024-06-09T22:17:28.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.093+0000 7f9bc7533700 1 Processor -- start 2024-06-09T22:17:28.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.093+0000 7f9bc7533700 1 -- start start 2024-06-09T22:17:28.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.093+0000 7f9bc7533700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc0101680 0x7f9bc0103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:28.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.093+0000 7f9bc7533700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bc0102030 0x7f9bc01035d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:28.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.093+0000 7f9bc7533700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bc01066d0 0x7f9bc0103b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:28.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.093+0000 7f9bc7533700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc01117b0 con 0x7f9bc0101680 2024-06-09T22:17:28.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.093+0000 7f9bc7533700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc0111630 con 0x7f9bc0102030 2024-06-09T22:17:28.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.093+0000 7f9bc7533700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc0111930 con 0x7f9bc01066d0 2024-06-09T22:17:28.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.093+0000 7f9bc52cf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc0101680 0x7f9bc0103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:28.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.093+0000 7f9bc52cf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc0101680 0x7f9bc0103090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57828/0 (socket says 172.21.15.104:57828) 2024-06-09T22:17:28.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.093+0000 7f9bc52cf700 1 -- 172.21.15.104:0/1465119273 learned_addr learned my addr 172.21.15.104:0/1465119273 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:28.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bc4ace700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bc0102030 0x7f9bc01035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:28.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bc5ad0700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bc01066d0 0x7f9bc0103b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:28.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bc52cf700 1 -- 172.21.15.104:0/1465119273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bc01066d0 msgr2=0x7f9bc0103b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:28.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bc52cf700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bc01066d0 0x7f9bc0103b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:28.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bc52cf700 1 -- 172.21.15.104:0/1465119273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bc0102030 msgr2=0x7f9bc01035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:28.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bc52cf700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bc0102030 0x7f9bc01035d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:28.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bc52cf700 1 -- 172.21.15.104:0/1465119273 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc01a7280 con 0x7f9bc0101680 2024-06-09T22:17:28.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bc4ace700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bc0102030 0x7f9bc01035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:17:28.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bc52cf700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc0101680 0x7f9bc0103090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00b530 tx=0x7f9bbc00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:28.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bb67fc700 1 -- 172.21.15.104:0/1465119273 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc015020 con 0x7f9bc0101680 2024-06-09T22:17:28.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bc7533700 1 -- 172.21.15.104:0/1465119273 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bc01a7570 con 0x7f9bc0101680 2024-06-09T22:17:28.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bb67fc700 1 -- 172.21.15.104:0/1465119273 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bbc011d10 con 0x7f9bc0101680 2024-06-09T22:17:28.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bb67fc700 1 -- 172.21.15.104:0/1465119273 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc010df0 con 0x7f9bc0101680 2024-06-09T22:17:28.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.094+0000 7f9bc7533700 1 -- 172.21.15.104:0/1465119273 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bc01a7ab0 con 0x7f9bc0101680 2024-06-09T22:17:28.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.095+0000 7f9bc7533700 1 -- 172.21.15.104:0/1465119273 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bc0069180 con 0x7f9bc0101680 2024-06-09T22:17:28.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.096+0000 7f9bb67fc700 1 -- 172.21.15.104:0/1465119273 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9bbc018020 con 0x7f9bc0101680 2024-06-09T22:17:28.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.097+0000 7f9bb67fc700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9bac0a2c20 0x7f9bac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:28.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.097+0000 7f9bc4ace700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9bac0a2c20 0x7f9bac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:28.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.097+0000 7f9bb67fc700 1 -- 172.21.15.104:0/1465119273 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7f9bbc0cebb0 con 0x7f9bc0101680 2024-06-09T22:17:28.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.097+0000 7f9bc4ace700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9bac0a2c20 0x7f9bac0a50e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0107be0 tx=0x7f9bb0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:28.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.101+0000 7f9bb67fc700 1 -- 172.21.15.104:0/1465119273 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9bbc01f020 con 0x7f9bc0101680 2024-06-09T22:17:28.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.316+0000 7f9bc7533700 1 -- 172.21.15.104:0/1465119273 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9bc002dca0 con 0x7f9bc0101680 2024-06-09T22:17:28.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.316+0000 7f9bb67fc700 1 -- 172.21.15.104:0/1465119273 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v97) ==== 81+0+598 (secure 0 0 0) 0x7f9bbc09b7b0 con 0x7f9bc0101680 2024-06-09T22:17:28.313 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6801/85752866 2024-06-10T22:09:25.221731+0000 2024-06-09T22:17:28.313 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/3559678620 2024-06-10T22:08:58.969000+0000 2024-06-09T22:17:28.313 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/3864904957 2024-06-10T22:09:25.221731+0000 2024-06-09T22:17:28.313 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6800/85752866 2024-06-10T22:09:25.221731+0000 2024-06-09T22:17:28.313 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6801/1879478526 2024-06-10T22:08:58.969000+0000 2024-06-09T22:17:28.313 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/2471278860 2024-06-10T22:08:58.969000+0000 2024-06-09T22:17:28.313 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/4119286692 2024-06-10T22:09:25.221731+0000 2024-06-09T22:17:28.313 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/2266338289 2024-06-10T22:08:58.969000+0000 2024-06-09T22:17:28.313 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6800/1879478526 2024-06-10T22:08:58.969000+0000 2024-06-09T22:17:28.314 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/3072042273 2024-06-10T22:09:25.221731+0000 2024-06-09T22:17:28.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.319+0000 7f9bc7533700 1 -- 172.21.15.104:0/1465119273 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9bac0a2c20 msgr2=0x7f9bac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:28.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.319+0000 7f9bc7533700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9bac0a2c20 0x7f9bac0a50e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0107be0 tx=0x7f9bb0008040 comp rx=0 tx=0).stop 2024-06-09T22:17:28.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.319+0000 7f9bc7533700 1 -- 172.21.15.104:0/1465119273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc0101680 msgr2=0x7f9bc0103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:28.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.319+0000 7f9bc7533700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc0101680 0x7f9bc0103090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00b530 tx=0x7f9bbc00b8f0 comp rx=0 tx=0).stop 2024-06-09T22:17:28.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.319+0000 7f9bc7533700 1 -- 172.21.15.104:0/1465119273 shutdown_connections 2024-06-09T22:17:28.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.319+0000 7f9bc7533700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9bac0a2c20 0x7f9bac0a50e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:28.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.319+0000 7f9bc7533700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc0101680 0x7f9bc0103090 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:28.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.319+0000 7f9bc7533700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9bc01066d0 0x7f9bc0103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:28.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.319+0000 7f9bc7533700 1 --2- 172.21.15.104:0/1465119273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bc0102030 0x7f9bc01035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:28.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.319+0000 7f9bc7533700 1 -- 172.21.15.104:0/1465119273 >> 172.21.15.104:0/1465119273 conn(0x7f9bc00773e0 msgr2=0x7f9bc006aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:28.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.319+0000 7f9bc7533700 1 -- 172.21.15.104:0/1465119273 shutdown_connections 2024-06-09T22:17:28.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:28.319+0000 7f9bc7533700 1 -- 172.21.15.104:0/1465119273 wait complete. 2024-06-09T22:17:28.317 INFO:teuthology.orchestra.run.smithi104.stderr:listed 10 entries 2024-06-09T22:17:28.688 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:28 smithi104 ceph-mon[32389]: pgmap v328: 129 pgs: 3 peering, 126 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-06-09T22:17:28.688 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:28 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1465119273' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-09T22:17:28.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:28 smithi153 ceph-mon[36599]: pgmap v328: 129 pgs: 3 peering, 126 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-06-09T22:17:28.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:28 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1465119273' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-09T22:17:28.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:28 smithi194 ceph-mon[36856]: pgmap v328: 129 pgs: 3 peering, 126 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-06-09T22:17:28.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:28 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1465119273' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-09T22:17:28.999 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2024-06-09T22:17:28.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:28.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-06-09T22:17:29.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi104.stderr:ceph-fuse[75847]: fuse finished with error 0 and tester_r 0 2024-06-09T22:17:29.027 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:17:30.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:30 smithi153 ceph-mon[36599]: pgmap v329: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-06-09T22:17:30.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:30 smithi104 ceph-mon[32389]: pgmap v329: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-06-09T22:17:30.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:30 smithi194 ceph-mon[36856]: pgmap v329: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-06-09T22:17:32.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:32 smithi153 ceph-mon[36599]: pgmap v330: 129 pgs: 2 peering, 127 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-06-09T22:17:32.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:32 smithi104 ceph-mon[32389]: pgmap v330: 129 pgs: 2 peering, 127 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-06-09T22:17:32.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:32 smithi194 ceph-mon[36856]: pgmap v330: 129 pgs: 2 peering, 127 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-06-09T22:17:34.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:34 smithi153 ceph-mon[36599]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 0 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-06-09T22:17:34.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:34 smithi104 ceph-mon[32389]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 0 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-06-09T22:17:34.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:34 smithi194 ceph-mon[36856]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 0 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-06-09T22:17:35.128 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2024-06-09T22:17:35.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:17:35.129 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-06-09T22:17:35.172 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:17:35.462 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:36.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:36 smithi153 ceph-mon[36599]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 311 B/s wr, 0 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-06-09T22:17:36.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:36 smithi104 ceph-mon[32389]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 311 B/s wr, 0 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-06-09T22:17:36.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:36 smithi194 ceph-mon[36856]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 311 B/s wr, 0 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-06-09T22:17:37.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.068+0000 7fadd8f6a700 1 -- 172.21.15.104:0/4147345342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd4101380 msgr2=0x7fadd4101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:37.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.068+0000 7fadd8f6a700 1 --2- 172.21.15.104:0/4147345342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd4101380 0x7fadd4101800 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009990 tx=0x7fadc8009ca0 comp rx=0 tx=0).stop 2024-06-09T22:17:37.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.068+0000 7fadd8f6a700 1 -- 172.21.15.104:0/4147345342 shutdown_connections 2024-06-09T22:17:37.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.068+0000 7fadd8f6a700 1 --2- 172.21.15.104:0/4147345342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadd4101d40 0x7fadd410e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:37.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.068+0000 7fadd8f6a700 1 --2- 172.21.15.104:0/4147345342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd41073c0 0x7fadd41077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:37.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.068+0000 7fadd8f6a700 1 --2- 172.21.15.104:0/4147345342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd4101380 0x7fadd4101800 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:37.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.068+0000 7fadd8f6a700 1 -- 172.21.15.104:0/4147345342 >> 172.21.15.104:0/4147345342 conn(0x7fadd4078330 msgr2=0x7fadd4078740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:37.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.068+0000 7fadd8f6a700 1 -- 172.21.15.104:0/4147345342 shutdown_connections 2024-06-09T22:17:37.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.068+0000 7fadd8f6a700 1 -- 172.21.15.104:0/4147345342 wait complete. 2024-06-09T22:17:37.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.069+0000 7fadd8f6a700 1 Processor -- start 2024-06-09T22:17:37.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.069+0000 7fadd8f6a700 1 -- start start 2024-06-09T22:17:37.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.069+0000 7fadd8f6a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd4101380 0x7fadd419e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:37.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.069+0000 7fadd8f6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd4101d40 0x7fadd419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.069+0000 7fadd8f6a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadd41073c0 0x7fadd4198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.069+0000 7fadd8f6a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fadd41116e0 con 0x7fadd4101d40 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.069+0000 7fadd8f6a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fadd4111560 con 0x7fadd41073c0 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.069+0000 7fadd8f6a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fadd4111860 con 0x7fadd4101380 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.069+0000 7fadd1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd4101d40 0x7fadd419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd4101d40 0x7fadd419ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52220/0 (socket says 172.21.15.104:52220) 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd1d9b700 1 -- 172.21.15.104:0/1269438879 learned_addr learned my addr 172.21.15.104:0/1269438879 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd259c700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd4101380 0x7fadd419e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd1d9b700 1 -- 172.21.15.104:0/1269438879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd4101380 msgr2=0x7fadd419e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd1d9b700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd4101380 0x7fadd419e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd1d9b700 1 -- 172.21.15.104:0/1269438879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadd41073c0 msgr2=0x7fadd4198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd2d9d700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadd41073c0 0x7fadd4198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:37.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd1d9b700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadd41073c0 0x7fadd4198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:37.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd1d9b700 1 -- 172.21.15.104:0/1269438879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd4199220 con 0x7fadd4101d40 2024-06-09T22:17:37.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd259c700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd4101380 0x7fadd419e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:17:37.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd1d9b700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd4101d40 0x7fadd419ee40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fadc8000c00 tx=0x7fadc8003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:37.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadc37fe700 1 -- 172.21.15.104:0/1269438879 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc8027070 con 0x7fadd4101d40 2024-06-09T22:17:37.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd8f6a700 1 -- 172.21.15.104:0/1269438879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd41994b0 con 0x7fadd4101d40 2024-06-09T22:17:37.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadc37fe700 1 -- 172.21.15.104:0/1269438879 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc8004190 con 0x7fadd4101d40 2024-06-09T22:17:37.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadc37fe700 1 -- 172.21.15.104:0/1269438879 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc800f460 con 0x7fadd4101d40 2024-06-09T22:17:37.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.070+0000 7fadd8f6a700 1 -- 172.21.15.104:0/1269438879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd41a57d0 con 0x7fadd4101d40 2024-06-09T22:17:37.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.071+0000 7fadd8f6a700 1 -- 172.21.15.104:0/1269438879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadd4102a70 con 0x7fadd4101d40 2024-06-09T22:17:37.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.072+0000 7fadc37fe700 1 -- 172.21.15.104:0/1269438879 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fadc802a070 con 0x7fadd4101d40 2024-06-09T22:17:37.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.073+0000 7fadc37fe700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fadbc0a2c20 0x7fadbc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:37.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.073+0000 7fadd259c700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fadbc0a2c20 0x7fadbc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:37.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.073+0000 7fadc37fe700 1 -- 172.21.15.104:0/1269438879 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7fadc80d0730 con 0x7fadd4101d40 2024-06-09T22:17:37.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.074+0000 7fadd259c700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fadbc0a2c20 0x7fadbc0a50e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fadd4199c00 tx=0x7fadc4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:37.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.077+0000 7fadc37fe700 1 -- 172.21.15.104:0/1269438879 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fadc8013070 con 0x7fadd4101d40 2024-06-09T22:17:37.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.274+0000 7fadd8f6a700 1 -- 172.21.15.104:0/1269438879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fadd41a59a0 con 0x7fadd4101d40 2024-06-09T22:17:37.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.276+0000 7fadc37fe700 1 -- 172.21.15.104:0/1269438879 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+24668 (secure 0 0 0) 0x7fadc809d3c0 con 0x7fadd4101d40 2024-06-09T22:17:37.273 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:17:37.273 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":97,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:17:26.456251+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:17:37.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.278+0000 7fadd8f6a700 1 -- 172.21.15.104:0/1269438879 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fadbc0a2c20 msgr2=0x7fadbc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:37.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.278+0000 7fadd8f6a700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fadbc0a2c20 0x7fadbc0a50e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fadd4199c00 tx=0x7fadc4008040 comp rx=0 tx=0).stop 2024-06-09T22:17:37.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.278+0000 7fadd8f6a700 1 -- 172.21.15.104:0/1269438879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd4101d40 msgr2=0x7fadd419ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:37.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.278+0000 7fadd8f6a700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd4101d40 0x7fadd419ee40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fadc8000c00 tx=0x7fadc8003730 comp rx=0 tx=0).stop 2024-06-09T22:17:37.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.278+0000 7fadd8f6a700 1 -- 172.21.15.104:0/1269438879 shutdown_connections 2024-06-09T22:17:37.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.278+0000 7fadd8f6a700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fadbc0a2c20 0x7fadbc0a50e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:37.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.278+0000 7fadd8f6a700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadd41073c0 0x7fadd4198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:37.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.278+0000 7fadd8f6a700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd4101380 0x7fadd419e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:37.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.278+0000 7fadd8f6a700 1 --2- 172.21.15.104:0/1269438879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd4101d40 0x7fadd419ee40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:37.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.278+0000 7fadd8f6a700 1 -- 172.21.15.104:0/1269438879 >> 172.21.15.104:0/1269438879 conn(0x7fadd4078330 msgr2=0x7fadd41006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:37.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.278+0000 7fadd8f6a700 1 -- 172.21.15.104:0/1269438879 shutdown_connections 2024-06-09T22:17:37.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:37.278+0000 7fadd8f6a700 1 -- 172.21.15.104:0/1269438879 wait complete. 2024-06-09T22:17:37.647 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:37 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1269438879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:37.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:37 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1269438879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:37.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:37 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1269438879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:37.984 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:17:38.223 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:38.609 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:38 smithi104 ceph-mon[32389]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 282 B/s wr, 0 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-06-09T22:17:38.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:38 smithi153 ceph-mon[36599]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 282 B/s wr, 0 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-06-09T22:17:38.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:38 smithi194 ceph-mon[36856]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 282 B/s wr, 0 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-06-09T22:17:39.720 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:39 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:17:39.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:39 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:17:39.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.955+0000 7f82a62e0700 1 -- 172.21.15.104:0/3562123494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82a0101380 msgr2=0x7f82a0101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.955+0000 7f82a62e0700 1 --2- 172.21.15.104:0/3562123494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82a0101380 0x7f82a0101800 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8288009990 tx=0x7f8288009ca0 comp rx=0 tx=0).stop 2024-06-09T22:17:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.955+0000 7f82a62e0700 1 -- 172.21.15.104:0/3562123494 shutdown_connections 2024-06-09T22:17:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.955+0000 7f82a62e0700 1 --2- 172.21.15.104:0/3562123494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a0101d40 0x7f82a010e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.955+0000 7f82a62e0700 1 --2- 172.21.15.104:0/3562123494 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82a01073f0 0x7f82a01077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.955+0000 7f82a62e0700 1 --2- 172.21.15.104:0/3562123494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82a0101380 0x7f82a0101800 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.955+0000 7f82a62e0700 1 -- 172.21.15.104:0/3562123494 >> 172.21.15.104:0/3562123494 conn(0x7f82a0078300 msgr2=0x7f82a0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.956+0000 7f82a62e0700 1 -- 172.21.15.104:0/3562123494 shutdown_connections 2024-06-09T22:17:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.956+0000 7f82a62e0700 1 -- 172.21.15.104:0/3562123494 wait complete. 2024-06-09T22:17:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.956+0000 7f82a62e0700 1 Processor -- start 2024-06-09T22:17:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.956+0000 7f82a62e0700 1 -- start start 2024-06-09T22:17:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f82a62e0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82a0101380 0x7f82a019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f82a62e0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a0101d40 0x7f82a019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f82a62e0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82a01073f0 0x7f82a0198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f82a62e0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f82a0111680 con 0x7f82a0101380 2024-06-09T22:17:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f82a62e0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82a0111500 con 0x7f82a0101d40 2024-06-09T22:17:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f82a62e0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f82a0111800 con 0x7f82a01073f0 2024-06-09T22:17:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f829ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82a0101380 0x7f82a019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f829ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82a0101380 0x7f82a019e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52242/0 (socket says 172.21.15.104:52242) 2024-06-09T22:17:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f829ffff700 1 -- 172.21.15.104:0/2872312932 learned_addr learned my addr 172.21.15.104:0/2872312932 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f829f7fe700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a0101d40 0x7f82a019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f82a487d700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82a01073f0 0x7f82a0198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f829ffff700 1 -- 172.21.15.104:0/2872312932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82a01073f0 msgr2=0x7f82a0198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f829ffff700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82a01073f0 0x7f82a0198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f829ffff700 1 -- 172.21.15.104:0/2872312932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a0101d40 msgr2=0x7f82a019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f829ffff700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a0101d40 0x7f82a019ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f829ffff700 1 -- 172.21.15.104:0/2872312932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a0199010 con 0x7f82a0101380 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.957+0000 7f829f7fe700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a0101d40 0x7f82a019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.958+0000 7f829ffff700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82a0101380 0x7f82a019e6b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f829000c970 tx=0x7f829000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.958+0000 7f829d7fa700 1 -- 172.21.15.104:0/2872312932 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82900078d0 con 0x7f82a0101380 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.958+0000 7f829d7fa700 1 -- 172.21.15.104:0/2872312932 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8290018480 con 0x7f82a0101380 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.958+0000 7f829d7fa700 1 -- 172.21.15.104:0/2872312932 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829000f6b0 con 0x7f82a0101380 2024-06-09T22:17:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.958+0000 7f82a62e0700 1 -- 172.21.15.104:0/2872312932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a0199300 con 0x7f82a0101380 2024-06-09T22:17:39.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.958+0000 7f82a62e0700 1 -- 172.21.15.104:0/2872312932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a01a5510 con 0x7f82a0101380 2024-06-09T22:17:39.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.959+0000 7f82a62e0700 1 -- 172.21.15.104:0/2872312932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82a0102aa0 con 0x7f82a0101380 2024-06-09T22:17:39.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.960+0000 7f829d7fa700 1 -- 172.21.15.104:0/2872312932 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8290018630 con 0x7f82a0101380 2024-06-09T22:17:39.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.961+0000 7f829d7fa700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f828c0a2ca0 0x7f828c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:39.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.961+0000 7f829f7fe700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f828c0a2ca0 0x7f828c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:39.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.961+0000 7f829d7fa700 1 -- 172.21.15.104:0/2872312932 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7f82900cf660 con 0x7f82a0101380 2024-06-09T22:17:39.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.962+0000 7f829f7fe700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f828c0a2ca0 0x7f828c0a5160 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f82a006ba50 tx=0x7f8288009f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:39.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:39.965+0000 7f829d7fa700 1 -- 172.21.15.104:0/2872312932 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f829009c320 con 0x7f82a0101380 2024-06-09T22:17:39.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:39 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:17:40.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.184+0000 7f82a62e0700 1 -- 172.21.15.104:0/2872312932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82a0069180 con 0x7f82a0101380 2024-06-09T22:17:40.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.186+0000 7f829d7fa700 1 -- 172.21.15.104:0/2872312932 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+24668 (secure 0 0 0) 0x7f829009bc20 con 0x7f82a0101380 2024-06-09T22:17:40.183 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:17:40.183 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":97,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:17:26.456251+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:17:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.188+0000 7f82a62e0700 1 -- 172.21.15.104:0/2872312932 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f828c0a2ca0 msgr2=0x7f828c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.188+0000 7f82a62e0700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f828c0a2ca0 0x7f828c0a5160 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f82a006ba50 tx=0x7f8288009f00 comp rx=0 tx=0).stop 2024-06-09T22:17:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.188+0000 7f82a62e0700 1 -- 172.21.15.104:0/2872312932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82a0101380 msgr2=0x7f82a019e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.188+0000 7f82a62e0700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82a0101380 0x7f82a019e6b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f829000c970 tx=0x7f829000cc80 comp rx=0 tx=0).stop 2024-06-09T22:17:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.188+0000 7f82a62e0700 1 -- 172.21.15.104:0/2872312932 shutdown_connections 2024-06-09T22:17:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.188+0000 7f82a62e0700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f828c0a2ca0 0x7f828c0a5160 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.188+0000 7f82a62e0700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82a0101380 0x7f82a019e6b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.188+0000 7f82a62e0700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82a01073f0 0x7f82a0198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.188+0000 7f82a62e0700 1 --2- 172.21.15.104:0/2872312932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a0101d40 0x7f82a019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.188+0000 7f82a62e0700 1 -- 172.21.15.104:0/2872312932 >> 172.21.15.104:0/2872312932 conn(0x7f82a0078300 msgr2=0x7f82a0105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.188+0000 7f82a62e0700 1 -- 172.21.15.104:0/2872312932 shutdown_connections 2024-06-09T22:17:40.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:40.188+0000 7f82a62e0700 1 -- 172.21.15.104:0/2872312932 wait complete. 2024-06-09T22:17:40.493 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:40 smithi194 ceph-mon[36856]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-06-09T22:17:40.493 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:40 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2872312932' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:40.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:40 smithi153 ceph-mon[36599]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-06-09T22:17:40.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:40 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2872312932' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:40.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:40 smithi104 ceph-mon[32389]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-06-09T22:17:40.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:40 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2872312932' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:17:41.086 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-06-09T22:17:41.515 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:42.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:42.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:42.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:42.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:41 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:42.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:41 smithi153 ceph-mon[36599]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-06-09T22:17:42.253 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:42.253 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:42.253 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:42.253 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:41 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:42.253 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:41 smithi104 ceph-mon[32389]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-06-09T22:17:42.305 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:42.306 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:42.306 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:42.306 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:41 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:42.306 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:41 smithi194 ceph-mon[36856]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-06-09T22:17:43.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.205+0000 7fc7b82dd700 1 -- 172.21.15.104:0/551303544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b01066b0 msgr2=0x7fc7b010e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:43.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.205+0000 7fc7b82dd700 1 --2- 172.21.15.104:0/551303544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b01066b0 0x7fc7b010e280 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800b0d0 tx=0x7fc7a800b3e0 comp rx=0 tx=0).stop 2024-06-09T22:17:43.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 -- 172.21.15.104:0/551303544 shutdown_connections 2024-06-09T22:17:43.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 --2- 172.21.15.104:0/551303544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b01066b0 0x7fc7b010e280 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:43.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 --2- 172.21.15.104:0/551303544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b0101620 0x7fc7b0101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:43.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 --2- 172.21.15.104:0/551303544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7b0101fd0 0x7fc7b0105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 -- 172.21.15.104:0/551303544 >> 172.21.15.104:0/551303544 conn(0x7fc7b0078360 msgr2=0x7fc7b0078770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 -- 172.21.15.104:0/551303544 shutdown_connections 2024-06-09T22:17:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 -- 172.21.15.104:0/551303544 wait complete. 2024-06-09T22:17:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 Processor -- start 2024-06-09T22:17:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 -- start start 2024-06-09T22:17:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7b0101620 0x7fc7b019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b0101fd0 0x7fc7b019ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b01066b0 0x7fc7b0198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b01116c0 con 0x7fc7b0101fd0 2024-06-09T22:17:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b0111540 con 0x7fc7b01066b0 2024-06-09T22:17:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.206+0000 7fc7b82dd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b0111840 con 0x7fc7b0101620 2024-06-09T22:17:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b5878700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b0101fd0 0x7fc7b019ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b5878700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b0101fd0 0x7fc7b019ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52272/0 (socket says 172.21.15.104:52272) 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b5878700 1 -- 172.21.15.104:0/2509365768 learned_addr learned my addr 172.21.15.104:0/2509365768 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b687a700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b01066b0 0x7fc7b0198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b6079700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7b0101620 0x7fc7b019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b5878700 1 -- 172.21.15.104:0/2509365768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7b0101620 msgr2=0x7fc7b019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b5878700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7b0101620 0x7fc7b019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b5878700 1 -- 172.21.15.104:0/2509365768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b01066b0 msgr2=0x7fc7b0198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b5878700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b01066b0 0x7fc7b0198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b5878700 1 -- 172.21.15.104:0/2509365768 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b01991c0 con 0x7fc7b0101fd0 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b6079700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7b0101620 0x7fc7b019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b5878700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b0101fd0 0x7fc7b019ed90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000c970 tx=0x7fc7a000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7a77fe700 1 -- 172.21.15.104:0/2509365768 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a000ce60 con 0x7fc7b0101fd0 2024-06-09T22:17:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7a77fe700 1 -- 172.21.15.104:0/2509365768 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7a0004830 con 0x7fc7b0101fd0 2024-06-09T22:17:43.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7a77fe700 1 -- 172.21.15.104:0/2509365768 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0005610 con 0x7fc7b0101fd0 2024-06-09T22:17:43.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b82dd700 1 -- 172.21.15.104:0/2509365768 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b01994b0 con 0x7fc7b0101fd0 2024-06-09T22:17:43.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.207+0000 7fc7b82dd700 1 -- 172.21.15.104:0/2509365768 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b01a5720 con 0x7fc7b0101fd0 2024-06-09T22:17:43.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.208+0000 7fc7b82dd700 1 -- 172.21.15.104:0/2509365768 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b0069180 con 0x7fc7b0101fd0 2024-06-09T22:17:43.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.209+0000 7fc7a77fe700 1 -- 172.21.15.104:0/2509365768 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc7a00049e0 con 0x7fc7b0101fd0 2024-06-09T22:17:43.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.210+0000 7fc7a77fe700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc79c0a2c20 0x7fc79c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:43.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.210+0000 7fc7b6079700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc79c0a2c20 0x7fc79c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:43.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.210+0000 7fc7a77fe700 1 -- 172.21.15.104:0/2509365768 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7fc7a00cf300 con 0x7fc7b0101fd0 2024-06-09T22:17:43.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.211+0000 7fc7b6079700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc79c0a2c20 0x7fc79c0a50e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac005fd0 tx=0x7fc7ac005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:43.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.214+0000 7fc7a77fe700 1 -- 172.21.15.104:0/2509365768 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc7a0014030 con 0x7fc7b0101fd0 2024-06-09T22:17:43.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:43.477+0000 7fc7b82dd700 1 -- 172.21.15.104:0/2509365768 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc7b002dca0 con 0x7fc7b0101fd0 2024-06-09T22:17:44.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.393+0000 7fc7a77fe700 1 -- 172.21.15.104:0/2509365768 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fc7a009bf00 con 0x7fc7b0101fd0 2024-06-09T22:17:44.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.396+0000 7fc7b82dd700 1 -- 172.21.15.104:0/2509365768 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc79c0a2c20 msgr2=0x7fc79c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:44.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.396+0000 7fc7b82dd700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc79c0a2c20 0x7fc79c0a50e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac005fd0 tx=0x7fc7ac005f60 comp rx=0 tx=0).stop 2024-06-09T22:17:44.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.396+0000 7fc7b82dd700 1 -- 172.21.15.104:0/2509365768 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b0101fd0 msgr2=0x7fc7b019ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:44.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.396+0000 7fc7b82dd700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b0101fd0 0x7fc7b019ed90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000c970 tx=0x7fc7a000cc80 comp rx=0 tx=0).stop 2024-06-09T22:17:44.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.396+0000 7fc7b82dd700 1 -- 172.21.15.104:0/2509365768 shutdown_connections 2024-06-09T22:17:44.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.396+0000 7fc7b82dd700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc79c0a2c20 0x7fc79c0a50e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:44.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.396+0000 7fc7b82dd700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b01066b0 0x7fc7b0198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:44.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.396+0000 7fc7b82dd700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc7b0101620 0x7fc7b019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:44.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.396+0000 7fc7b82dd700 1 --2- 172.21.15.104:0/2509365768 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7b0101fd0 0x7fc7b019ed90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:44.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.396+0000 7fc7b82dd700 1 -- 172.21.15.104:0/2509365768 >> 172.21.15.104:0/2509365768 conn(0x7fc7b0078360 msgr2=0x7fc7b006ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:44.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.396+0000 7fc7b82dd700 1 -- 172.21.15.104:0/2509365768 shutdown_connections 2024-06-09T22:17:44.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:44.396+0000 7fc7b82dd700 1 -- 172.21.15.104:0/2509365768 wait complete. 2024-06-09T22:17:44.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:44 smithi153 ceph-mon[36599]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-06-09T22:17:44.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:44 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:44.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:44 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:44.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:44 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2509365768' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-06-09T22:17:44.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:44 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:17:44.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:44 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:17:44.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:44 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:44.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:44 smithi104 ceph-mon[32389]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-06-09T22:17:44.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:44 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:44.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:44 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:44.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:44 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2509365768' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-06-09T22:17:44.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:44 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:17:44.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:44 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:17:44.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:44 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:44.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:44 smithi194 ceph-mon[36856]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-06-09T22:17:44.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:44 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:44.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:44 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:44.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:44 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2509365768' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-06-09T22:17:44.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:44 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:17:44.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:44 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:17:44.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:44 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:17:45.112 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-06-09T22:17:45.113 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-06-09T22:17:45.354 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:45.714 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:45 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2509365768' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-06-09T22:17:45.714 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:45 smithi104 ceph-mon[32389]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-09T22:17:45.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:45 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2509365768' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-06-09T22:17:45.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:45 smithi153 ceph-mon[36599]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-09T22:17:45.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:45 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2509365768' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-06-09T22:17:45.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:45 smithi194 ceph-mon[36856]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-06-09T22:17:46.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:46 smithi104 ceph-mon[32389]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:46.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:46 smithi153 ceph-mon[36599]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:46.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:46 smithi194 ceph-mon[36856]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:47.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.098+0000 7ffa71b20700 1 -- 172.21.15.104:0/1927504960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa6c06ab30 msgr2=0x7ffa6c10f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:47.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.098+0000 7ffa71b20700 1 --2- 172.21.15.104:0/1927504960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa6c06ab30 0x7ffa6c10f320 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009990 tx=0x7ffa64009ca0 comp rx=0 tx=0).stop 2024-06-09T22:17:47.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.098+0000 7ffa71b20700 1 -- 172.21.15.104:0/1927504960 shutdown_connections 2024-06-09T22:17:47.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.098+0000 7ffa71b20700 1 --2- 172.21.15.104:0/1927504960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa6c06ab30 0x7ffa6c10f320 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:47.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.098+0000 7ffa71b20700 1 --2- 172.21.15.104:0/1927504960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa6c108500 0x7ffa6c1088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:47.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.098+0000 7ffa71b20700 1 --2- 172.21.15.104:0/1927504960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6c06a170 0x7ffa6c06a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:47.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.098+0000 7ffa71b20700 1 -- 172.21.15.104:0/1927504960 >> 172.21.15.104:0/1927504960 conn(0x7ffa6c078340 msgr2=0x7ffa6c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:47.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.098+0000 7ffa71b20700 1 -- 172.21.15.104:0/1927504960 shutdown_connections 2024-06-09T22:17:47.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.098+0000 7ffa71b20700 1 -- 172.21.15.104:0/1927504960 wait complete. 2024-06-09T22:17:47.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.098+0000 7ffa71b20700 1 Processor -- start 2024-06-09T22:17:47.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa71b20700 1 -- start start 2024-06-09T22:17:47.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa71b20700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa6c06a170 0x7ffa6c19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:47.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa71b20700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa6c06ab30 0x7ffa6c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:47.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa71b20700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6c108500 0x7ffa6c198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:47.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa71b20700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c112860 con 0x7ffa6c06a170 2024-06-09T22:17:47.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa71b20700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c1126e0 con 0x7ffa6c108500 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa71b20700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c1129e0 con 0x7ffa6c06ab30 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa6b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa6c06a170 0x7ffa6c19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa6b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa6c06a170 0x7ffa6c19ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52972/0 (socket says 172.21.15.104:52972) 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa6b7fe700 1 -- 172.21.15.104:0/1669177647 learned_addr learned my addr 172.21.15.104:0/1669177647 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa6bfff700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6c108500 0x7ffa6c198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa6affd700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa6c06ab30 0x7ffa6c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa6b7fe700 1 -- 172.21.15.104:0/1669177647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa6c06ab30 msgr2=0x7ffa6c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa6b7fe700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa6c06ab30 0x7ffa6c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa6b7fe700 1 -- 172.21.15.104:0/1669177647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6c108500 msgr2=0x7ffa6c198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa6b7fe700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6c108500 0x7ffa6c198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa6b7fe700 1 -- 172.21.15.104:0/1669177647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c199460 con 0x7ffa6c06a170 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.099+0000 7ffa6affd700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa6c06ab30 0x7ffa6c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:17:47.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.100+0000 7ffa6b7fe700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa6c06a170 0x7ffa6c19ea60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00ba50 tx=0x7ffa5c00bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:47.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.100+0000 7ffa68ff9700 1 -- 172.21.15.104:0/1669177647 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c019070 con 0x7ffa6c06a170 2024-06-09T22:17:47.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.100+0000 7ffa68ff9700 1 -- 172.21.15.104:0/1669177647 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa5c014440 con 0x7ffa6c06a170 2024-06-09T22:17:47.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.100+0000 7ffa68ff9700 1 -- 172.21.15.104:0/1669177647 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c0075d0 con 0x7ffa6c06a170 2024-06-09T22:17:47.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.100+0000 7ffa71b20700 1 -- 172.21.15.104:0/1669177647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6c199750 con 0x7ffa6c06a170 2024-06-09T22:17:47.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.100+0000 7ffa71b20700 1 -- 172.21.15.104:0/1669177647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa6c1a59e0 con 0x7ffa6c06a170 2024-06-09T22:17:47.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.101+0000 7ffa71b20700 1 -- 172.21.15.104:0/1669177647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa6c10a9e0 con 0x7ffa6c06a170 2024-06-09T22:17:47.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.102+0000 7ffa68ff9700 1 -- 172.21.15.104:0/1669177647 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ffa5c01c070 con 0x7ffa6c06a170 2024-06-09T22:17:47.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.103+0000 7ffa68ff9700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ffa540a2c20 0x7ffa540a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:47.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.103+0000 7ffa6affd700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ffa540a2c20 0x7ffa540a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:47.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.103+0000 7ffa68ff9700 1 -- 172.21.15.104:0/1669177647 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7ffa5c0ce780 con 0x7ffa6c06a170 2024-06-09T22:17:47.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.103+0000 7ffa6affd700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ffa540a2c20 0x7ffa540a50e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c19a280 tx=0x7ffa60008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:47.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.107+0000 7ffa68ff9700 1 -- 172.21.15.104:0/1669177647 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffa5c09b380 con 0x7ffa6c06a170 2024-06-09T22:17:47.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.350+0000 7ffa71b20700 1 -- 172.21.15.104:0/1669177647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7ffa6c199ff0 con 0x7ffa6c06a170 2024-06-09T22:17:47.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.389+0000 7ffa68ff9700 1 -- 172.21.15.104:0/1669177647 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7ffa5c09ac80 con 0x7ffa6c06a170 2024-06-09T22:17:47.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.391+0000 7ffa71b20700 1 -- 172.21.15.104:0/1669177647 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ffa540a2c20 msgr2=0x7ffa540a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:47.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.391+0000 7ffa71b20700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ffa540a2c20 0x7ffa540a50e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c19a280 tx=0x7ffa60008040 comp rx=0 tx=0).stop 2024-06-09T22:17:47.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.391+0000 7ffa71b20700 1 -- 172.21.15.104:0/1669177647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa6c06a170 msgr2=0x7ffa6c19ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:47.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.391+0000 7ffa71b20700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa6c06a170 0x7ffa6c19ea60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00ba50 tx=0x7ffa5c00bd60 comp rx=0 tx=0).stop 2024-06-09T22:17:47.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.391+0000 7ffa71b20700 1 -- 172.21.15.104:0/1669177647 shutdown_connections 2024-06-09T22:17:47.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.391+0000 7ffa71b20700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ffa540a2c20 0x7ffa540a50e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:47.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.391+0000 7ffa71b20700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa6c06a170 0x7ffa6c19ea60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:47.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.391+0000 7ffa71b20700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6c108500 0x7ffa6c198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:47.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.391+0000 7ffa71b20700 1 --2- 172.21.15.104:0/1669177647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa6c06ab30 0x7ffa6c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:47.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.391+0000 7ffa71b20700 1 -- 172.21.15.104:0/1669177647 >> 172.21.15.104:0/1669177647 conn(0x7ffa6c078340 msgr2=0x7ffa6c1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:47.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.391+0000 7ffa71b20700 1 -- 172.21.15.104:0/1669177647 shutdown_connections 2024-06-09T22:17:47.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:47.391+0000 7ffa71b20700 1 -- 172.21.15.104:0/1669177647 wait complete. 2024-06-09T22:17:47.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:47 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1669177647' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-06-09T22:17:47.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:47 smithi153 ceph-mon[36599]: daemon mds.c assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-06-09T22:17:47.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:47 smithi153 ceph-mon[36599]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-06-09T22:17:47.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:47 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1669177647' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-06-09T22:17:47.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:47 smithi153 ceph-mon[36599]: fsmap cephfs:2 {0=i=up:active,1=c=up:creating} 10 up:standby 2024-06-09T22:17:47.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:47 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1669177647' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-06-09T22:17:47.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:47 smithi104 ceph-mon[32389]: daemon mds.c assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-06-09T22:17:47.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:47 smithi104 ceph-mon[32389]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-06-09T22:17:47.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:47 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1669177647' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-06-09T22:17:47.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:47 smithi104 ceph-mon[32389]: fsmap cephfs:2 {0=i=up:active,1=c=up:creating} 10 up:standby 2024-06-09T22:17:47.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:47 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1669177647' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-06-09T22:17:47.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:47 smithi194 ceph-mon[36856]: daemon mds.c assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-06-09T22:17:47.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:47 smithi194 ceph-mon[36856]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-06-09T22:17:47.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:47 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1669177647' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-06-09T22:17:47.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:47 smithi194 ceph-mon[36856]: fsmap cephfs:2 {0=i=up:active,1=c=up:creating} 10 up:standby 2024-06-09T22:17:48.105 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-06-09T22:17:48.346 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:48.705 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:48 smithi104 ceph-mon[32389]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:48.705 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:48 smithi104 ceph-mon[32389]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-06-09T22:17:48.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:48 smithi153 ceph-mon[36599]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:48.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:48 smithi153 ceph-mon[36599]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-06-09T22:17:48.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:48 smithi194 ceph-mon[36856]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:17:48.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:48 smithi194 ceph-mon[36856]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-06-09T22:17:49.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:49 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] up:active 2024-06-09T22:17:49.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:49 smithi104 ceph-mon[32389]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-06-09T22:17:49.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:49 smithi104 ceph-mon[32389]: fsmap cephfs:2 {0=i=up:active,1=c=up:active} 10 up:standby 2024-06-09T22:17:49.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:49 smithi104 ceph-mon[32389]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:creating} 9 up:standby 2024-06-09T22:17:49.504 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:49 smithi104 ceph-mon[32389]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-06-09T22:17:49.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:49 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] up:active 2024-06-09T22:17:49.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:49 smithi153 ceph-mon[36599]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-06-09T22:17:49.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:49 smithi153 ceph-mon[36599]: fsmap cephfs:2 {0=i=up:active,1=c=up:active} 10 up:standby 2024-06-09T22:17:49.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:49 smithi153 ceph-mon[36599]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:creating} 9 up:standby 2024-06-09T22:17:49.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:49 smithi153 ceph-mon[36599]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-06-09T22:17:49.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:49 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] up:active 2024-06-09T22:17:49.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:49 smithi194 ceph-mon[36856]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-06-09T22:17:49.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:49 smithi194 ceph-mon[36856]: fsmap cephfs:2 {0=i=up:active,1=c=up:active} 10 up:standby 2024-06-09T22:17:49.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:49 smithi194 ceph-mon[36856]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:creating} 9 up:standby 2024-06-09T22:17:49.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:49 smithi194 ceph-mon[36856]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-06-09T22:17:50.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.091+0000 7f41575d3700 1 -- 172.21.15.104:0/3598487132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4150101390 msgr2=0x7f4150101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:50.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.091+0000 7f41575d3700 1 --2- 172.21.15.104:0/3598487132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4150101390 0x7f4150101810 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f414c009990 tx=0x7f414c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:17:50.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.091+0000 7f41575d3700 1 -- 172.21.15.104:0/3598487132 shutdown_connections 2024-06-09T22:17:50.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.091+0000 7f41575d3700 1 --2- 172.21.15.104:0/3598487132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4150101d50 0x7f415010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:50.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.091+0000 7f41575d3700 1 --2- 172.21.15.104:0/3598487132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4150107400 0x7f41501077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:50.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.091+0000 7f41575d3700 1 --2- 172.21.15.104:0/3598487132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4150101390 0x7f4150101810 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:50.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.091+0000 7f41575d3700 1 -- 172.21.15.104:0/3598487132 >> 172.21.15.104:0/3598487132 conn(0x7f4150078340 msgr2=0x7f4150078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:50.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.091+0000 7f41575d3700 1 -- 172.21.15.104:0/3598487132 shutdown_connections 2024-06-09T22:17:50.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.091+0000 7f41575d3700 1 -- 172.21.15.104:0/3598487132 wait complete. 2024-06-09T22:17:50.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f41575d3700 1 Processor -- start 2024-06-09T22:17:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f41575d3700 1 -- start start 2024-06-09T22:17:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f41575d3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4150101390 0x7f415019adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f41575d3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4150101d50 0x7f415019b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f41575d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4150107400 0x7f415019f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f41575d3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f41501116f0 con 0x7f4150107400 2024-06-09T22:17:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f41575d3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4150111570 con 0x7f4150101390 2024-06-09T22:17:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f41575d3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4150111870 con 0x7f4150101d50 2024-06-09T22:17:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f4155b70700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4150107400 0x7f415019f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f4155b70700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4150107400 0x7f415019f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52990/0 (socket says 172.21.15.104:52990) 2024-06-09T22:17:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f4155b70700 1 -- 172.21.15.104:0/3221877906 learned_addr learned my addr 172.21.15.104:0/3221877906 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f415536f700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4150101390 0x7f415019adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.092+0000 7f4154b6e700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4150101d50 0x7f415019b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.093+0000 7f4155b70700 1 -- 172.21.15.104:0/3221877906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4150101d50 msgr2=0x7f415019b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.093+0000 7f4155b70700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4150101d50 0x7f415019b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.093+0000 7f4155b70700 1 -- 172.21.15.104:0/3221877906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4150101390 msgr2=0x7f415019adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.093+0000 7f4155b70700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4150101390 0x7f415019adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.093+0000 7f4155b70700 1 -- 172.21.15.104:0/3221877906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415019fe00 con 0x7f4150107400 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.093+0000 7f4155b70700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4150107400 0x7f415019f6c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4148007d70 tx=0x7f414800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.093+0000 7f41427fc700 1 -- 172.21.15.104:0/3221877906 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4148016020 con 0x7f4150107400 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.093+0000 7f41427fc700 1 -- 172.21.15.104:0/3221877906 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f414800ac90 con 0x7f4150107400 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.093+0000 7f41575d3700 1 -- 172.21.15.104:0/3221877906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41501a00f0 con 0x7f4150107400 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.093+0000 7f41427fc700 1 -- 172.21.15.104:0/3221877906 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41480115c0 con 0x7f4150107400 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.093+0000 7f41575d3700 1 -- 172.21.15.104:0/3221877906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41501a7970 con 0x7f4150107400 2024-06-09T22:17:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.094+0000 7f41575d3700 1 -- 172.21.15.104:0/3221877906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4150069180 con 0x7f4150107400 2024-06-09T22:17:50.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.095+0000 7f41427fc700 1 -- 172.21.15.104:0/3221877906 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4148019020 con 0x7f4150107400 2024-06-09T22:17:50.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.096+0000 7f41427fc700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f413c0a2ca0 0x7f413c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:50.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.096+0000 7f415536f700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f413c0a2ca0 0x7f413c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:50.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.096+0000 7f41427fc700 1 -- 172.21.15.104:0/3221877906 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7f41480cf6a0 con 0x7f4150107400 2024-06-09T22:17:50.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.096+0000 7f415536f700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f413c0a2ca0 0x7f413c0a5160 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4144009a10 tx=0x7f4144009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:50.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.099+0000 7f41427fc700 1 -- 172.21.15.104:0/3221877906 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4148020020 con 0x7f4150107400 2024-06-09T22:17:50.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.338+0000 7f41575d3700 1 -- 172.21.15.104:0/3221877906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f41501a7c00 con 0x7f4150107400 2024-06-09T22:17:50.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.430+0000 7f41427fc700 1 -- 172.21.15.104:0/3221877906 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7f414809c2a0 con 0x7f4150107400 2024-06-09T22:17:50.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.432+0000 7f41575d3700 1 -- 172.21.15.104:0/3221877906 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f413c0a2ca0 msgr2=0x7f413c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:50.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.432+0000 7f41575d3700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f413c0a2ca0 0x7f413c0a5160 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4144009a10 tx=0x7f4144009190 comp rx=0 tx=0).stop 2024-06-09T22:17:50.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.432+0000 7f41575d3700 1 -- 172.21.15.104:0/3221877906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4150107400 msgr2=0x7f415019f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:50.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.432+0000 7f41575d3700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4150107400 0x7f415019f6c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4148007d70 tx=0x7f414800a340 comp rx=0 tx=0).stop 2024-06-09T22:17:50.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.432+0000 7f41575d3700 1 -- 172.21.15.104:0/3221877906 shutdown_connections 2024-06-09T22:17:50.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.432+0000 7f41575d3700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f413c0a2ca0 0x7f413c0a5160 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:50.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.432+0000 7f41575d3700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4150107400 0x7f415019f6c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:50.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.432+0000 7f41575d3700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4150101390 0x7f415019adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:50.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.432+0000 7f41575d3700 1 --2- 172.21.15.104:0/3221877906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4150101d50 0x7f415019b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:50.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.432+0000 7f41575d3700 1 -- 172.21.15.104:0/3221877906 >> 172.21.15.104:0/3221877906 conn(0x7f4150078340 msgr2=0x7f415006a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:50.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.432+0000 7f41575d3700 1 -- 172.21.15.104:0/3221877906 shutdown_connections 2024-06-09T22:17:50.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:50.432+0000 7f41575d3700 1 -- 172.21.15.104:0/3221877906 wait complete. 2024-06-09T22:17:50.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:50 smithi153 ceph-mon[36599]: pgmap v339: 129 pgs: 129 active+clean; 582 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-06-09T22:17:50.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:50 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] up:active 2024-06-09T22:17:50.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:50 smithi153 ceph-mon[36599]: daemon mds.b assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-06-09T22:17:50.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:50 smithi153 ceph-mon[36599]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:active} 9 up:standby 2024-06-09T22:17:50.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:50 smithi153 ceph-mon[36599]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=b=up:creating} 8 up:standby 2024-06-09T22:17:50.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:50 smithi153 ceph-mon[36599]: daemon mds.b is now active in filesystem cephfs as rank 3 2024-06-09T22:17:50.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:50 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3221877906' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-06-09T22:17:50.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:50 smithi104 ceph-mon[32389]: pgmap v339: 129 pgs: 129 active+clean; 582 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-06-09T22:17:50.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:50 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] up:active 2024-06-09T22:17:50.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:50 smithi104 ceph-mon[32389]: daemon mds.b assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-06-09T22:17:50.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:50 smithi104 ceph-mon[32389]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:active} 9 up:standby 2024-06-09T22:17:50.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:50 smithi104 ceph-mon[32389]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=b=up:creating} 8 up:standby 2024-06-09T22:17:50.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:50 smithi104 ceph-mon[32389]: daemon mds.b is now active in filesystem cephfs as rank 3 2024-06-09T22:17:50.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:50 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3221877906' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-06-09T22:17:50.805 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:17:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i[67699]: 2024-06-09T22:17:50.367+0000 7f60c2591700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-06-09T22:17:50.806 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:17:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i[67699]: 2024-06-09T22:17:50.367+0000 7f60c2591700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-06-09T22:17:50.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:50 smithi194 ceph-mon[36856]: pgmap v339: 129 pgs: 129 active+clean; 582 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-06-09T22:17:50.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:50 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] up:active 2024-06-09T22:17:50.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:50 smithi194 ceph-mon[36856]: daemon mds.b assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-06-09T22:17:50.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:50 smithi194 ceph-mon[36856]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:active} 9 up:standby 2024-06-09T22:17:50.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:50 smithi194 ceph-mon[36856]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=b=up:creating} 8 up:standby 2024-06-09T22:17:50.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:50 smithi194 ceph-mon[36856]: daemon mds.b is now active in filesystem cephfs as rank 3 2024-06-09T22:17:50.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:50 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3221877906' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-06-09T22:17:51.158 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-06-09T22:17:51.401 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:51.428 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:51 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3221877906' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-06-09T22:17:51.428 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:51 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] up:active 2024-06-09T22:17:51.428 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:51 smithi104 ceph-mon[32389]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-06-09T22:17:51.428 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:51 smithi104 ceph-mon[32389]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-06-09T22:17:51.429 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:51 smithi104 ceph-mon[32389]: Cluster is now healthy 2024-06-09T22:17:51.429 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:51 smithi104 ceph-mon[32389]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=b=up:active} 8 up:standby 2024-06-09T22:17:51.429 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:51 smithi104 ceph-mon[32389]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-06-09T22:17:51.429 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:51 smithi104 ceph-mon[32389]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-06-09T22:17:51.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:51 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3221877906' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-06-09T22:17:51.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:51 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] up:active 2024-06-09T22:17:51.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:51 smithi153 ceph-mon[36599]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-06-09T22:17:51.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:51 smithi153 ceph-mon[36599]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-06-09T22:17:51.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:51 smithi153 ceph-mon[36599]: Cluster is now healthy 2024-06-09T22:17:51.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:51 smithi153 ceph-mon[36599]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=b=up:active} 8 up:standby 2024-06-09T22:17:51.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:51 smithi153 ceph-mon[36599]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-06-09T22:17:51.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:51 smithi153 ceph-mon[36599]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-06-09T22:17:51.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:51 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3221877906' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-06-09T22:17:51.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:51 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] up:active 2024-06-09T22:17:51.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:51 smithi194 ceph-mon[36856]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-06-09T22:17:51.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:51 smithi194 ceph-mon[36856]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-06-09T22:17:51.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:51 smithi194 ceph-mon[36856]: Cluster is now healthy 2024-06-09T22:17:51.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:51 smithi194 ceph-mon[36856]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=b=up:active} 8 up:standby 2024-06-09T22:17:51.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:51 smithi194 ceph-mon[36856]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-06-09T22:17:51.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:51 smithi194 ceph-mon[36856]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-06-09T22:17:52.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:52 smithi153 ceph-mon[36599]: pgmap v340: 129 pgs: 129 active+clean; 585 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2024-06-09T22:17:52.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:52 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] up:active 2024-06-09T22:17:52.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:52 smithi153 ceph-mon[36599]: fsmap cephfs:5 7 up:standby 5 up:active 2024-06-09T22:17:52.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:52 smithi153 ceph-mon[36599]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:17:52.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:52 smithi104 ceph-mon[32389]: pgmap v340: 129 pgs: 129 active+clean; 585 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2024-06-09T22:17:52.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:52 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] up:active 2024-06-09T22:17:52.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:52 smithi104 ceph-mon[32389]: fsmap cephfs:5 7 up:standby 5 up:active 2024-06-09T22:17:52.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:52 smithi104 ceph-mon[32389]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:17:52.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:52 smithi194 ceph-mon[36856]: pgmap v340: 129 pgs: 129 active+clean; 585 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2024-06-09T22:17:52.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:52 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] up:active 2024-06-09T22:17:52.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:52 smithi194 ceph-mon[36856]: fsmap cephfs:5 7 up:standby 5 up:active 2024-06-09T22:17:52.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:52 smithi194 ceph-mon[36856]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:17:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.149+0000 7f72aa31b700 1 -- 172.21.15.104:0/1209137612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a4102740 msgr2=0x7f72a4102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.149+0000 7f72aa31b700 1 --2- 172.21.15.104:0/1209137612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a4102740 0x7f72a4102b20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f72940099c0 tx=0x7f7294009cd0 comp rx=0 tx=0).stop 2024-06-09T22:17:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.149+0000 7f72aa31b700 1 -- 172.21.15.104:0/1209137612 shutdown_connections 2024-06-09T22:17:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.149+0000 7f72aa31b700 1 --2- 172.21.15.104:0/1209137612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a4102740 0x7f72a4102b20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.149+0000 7f72aa31b700 1 --2- 172.21.15.104:0/1209137612 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72a4107840 0x7f72a410f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.149+0000 7f72aa31b700 1 --2- 172.21.15.104:0/1209137612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a41030f0 0x7f72a4107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.149+0000 7f72aa31b700 1 -- 172.21.15.104:0/1209137612 >> 172.21.15.104:0/1209137612 conn(0x7f72a40795f0 msgr2=0x7f72a4079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.149+0000 7f72aa31b700 1 -- 172.21.15.104:0/1209137612 shutdown_connections 2024-06-09T22:17:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.149+0000 7f72aa31b700 1 -- 172.21.15.104:0/1209137612 wait complete. 2024-06-09T22:17:53.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72aa31b700 1 Processor -- start 2024-06-09T22:17:53.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72aa31b700 1 -- start start 2024-06-09T22:17:53.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72aa31b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a4102740 0x7f72a419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:53.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72aa31b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a41030f0 0x7f72a419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:53.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72aa31b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72a4107840 0x7f72a4198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:53.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72aa31b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f72a4112780 con 0x7f72a4102740 2024-06-09T22:17:53.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72aa31b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f72a4112600 con 0x7f72a41030f0 2024-06-09T22:17:53.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72aa31b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f72a4112900 con 0x7f72a4107840 2024-06-09T22:17:53.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72a3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a4102740 0x7f72a419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:53.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72a3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a4102740 0x7f72a419e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53010/0 (socket says 172.21.15.104:53010) 2024-06-09T22:17:53.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72a3fff700 1 -- 172.21.15.104:0/3291128566 learned_addr learned my addr 172.21.15.104:0/3291128566 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f729bfff700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a41030f0 0x7f72a419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72a88b8700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72a4107840 0x7f72a4198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72a3fff700 1 -- 172.21.15.104:0/3291128566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72a4107840 msgr2=0x7f72a4198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72a3fff700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72a4107840 0x7f72a4198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72a3fff700 1 -- 172.21.15.104:0/3291128566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a41030f0 msgr2=0x7f72a419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72a3fff700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a41030f0 0x7f72a419ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.150+0000 7f72a3fff700 1 -- 172.21.15.104:0/3291128566 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a4199160 con 0x7f72a4102740 2024-06-09T22:17:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.151+0000 7f72a3fff700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a4102740 0x7f72a419e840 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7294006010 tx=0x7f7294009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.151+0000 7f72a1ffb700 1 -- 172.21.15.104:0/3291128566 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7294027070 con 0x7f72a4102740 2024-06-09T22:17:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.151+0000 7f72a1ffb700 1 -- 172.21.15.104:0/3291128566 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7294003780 con 0x7f72a4102740 2024-06-09T22:17:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.151+0000 7f72a1ffb700 1 -- 172.21.15.104:0/3291128566 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7294022da0 con 0x7f72a4102740 2024-06-09T22:17:53.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.151+0000 7f72aa31b700 1 -- 172.21.15.104:0/3291128566 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a41993f0 con 0x7f72a4102740 2024-06-09T22:17:53.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.151+0000 7f72aa31b700 1 -- 172.21.15.104:0/3291128566 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a41a5710 con 0x7f72a4102740 2024-06-09T22:17:53.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.152+0000 7f72aa31b700 1 -- 172.21.15.104:0/3291128566 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72a4103bd0 con 0x7f72a4102740 2024-06-09T22:17:53.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.153+0000 7f72a1ffb700 1 -- 172.21.15.104:0/3291128566 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f729402a070 con 0x7f72a4102740 2024-06-09T22:17:53.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.154+0000 7f72a1ffb700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f72900a2c20 0x7f72900a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:53.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.154+0000 7f729bfff700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f72900a2c20 0x7f72900a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:53.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.154+0000 7f72a1ffb700 1 -- 172.21.15.104:0/3291128566 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7f72940d0370 con 0x7f72a4102740 2024-06-09T22:17:53.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.154+0000 7f729bfff700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f72900a2c20 0x7f72900a50e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f72a4102060 tx=0x7f728c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:53.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.158+0000 7f72a1ffb700 1 -- 172.21.15.104:0/3291128566 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7294031080 con 0x7f72a4102740 2024-06-09T22:17:53.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.400+0000 7f72aa31b700 1 -- 172.21.15.104:0/3291128566 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f72a402dca0 con 0x7f72a4102740 2024-06-09T22:17:53.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.448+0000 7f72a1ffb700 1 -- 172.21.15.104:0/3291128566 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v28) ==== 115+0+0 (secure 0 0 0) 0x7f7294013070 con 0x7f72a4102740 2024-06-09T22:17:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.450+0000 7f72aa31b700 1 -- 172.21.15.104:0/3291128566 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f72900a2c20 msgr2=0x7f72900a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.450+0000 7f72aa31b700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f72900a2c20 0x7f72900a50e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f72a4102060 tx=0x7f728c00a300 comp rx=0 tx=0).stop 2024-06-09T22:17:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.450+0000 7f72aa31b700 1 -- 172.21.15.104:0/3291128566 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a4102740 msgr2=0x7f72a419e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.450+0000 7f72aa31b700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a4102740 0x7f72a419e840 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7294006010 tx=0x7f7294009f80 comp rx=0 tx=0).stop 2024-06-09T22:17:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.450+0000 7f72aa31b700 1 -- 172.21.15.104:0/3291128566 shutdown_connections 2024-06-09T22:17:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.450+0000 7f72aa31b700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f72900a2c20 0x7f72900a50e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.450+0000 7f72aa31b700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a4102740 0x7f72a419e840 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.450+0000 7f72aa31b700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72a4107840 0x7f72a4198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.450+0000 7f72aa31b700 1 --2- 172.21.15.104:0/3291128566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a41030f0 0x7f72a419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.450+0000 7f72aa31b700 1 -- 172.21.15.104:0/3291128566 >> 172.21.15.104:0/3291128566 conn(0x7f72a40795f0 msgr2=0x7f72a4100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.450+0000 7f72aa31b700 1 -- 172.21.15.104:0/3291128566 shutdown_connections 2024-06-09T22:17:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:53.450+0000 7f72aa31b700 1 -- 172.21.15.104:0/3291128566 wait complete. 2024-06-09T22:17:53.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:53 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3291128566' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-06-09T22:17:53.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:53 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3291128566' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-06-09T22:17:53.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:53 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3291128566' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-06-09T22:17:54.154 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-06-09T22:17:54.154 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-06-09T22:17:54.392 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:54.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:54 smithi153 ceph-mon[36599]: pgmap v341: 129 pgs: 129 active+clean; 585 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 3.2 KiB/s wr, 11 op/s 2024-06-09T22:17:54.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:54 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3291128566' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-06-09T22:17:54.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:54 smithi153 ceph-mon[36599]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:17:54.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:54 smithi104 ceph-mon[32389]: pgmap v341: 129 pgs: 129 active+clean; 585 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 3.2 KiB/s wr, 11 op/s 2024-06-09T22:17:54.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:54 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3291128566' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-06-09T22:17:54.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:54 smithi104 ceph-mon[32389]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:17:54.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:54 smithi194 ceph-mon[36856]: pgmap v341: 129 pgs: 129 active+clean; 585 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 3.2 KiB/s wr, 11 op/s 2024-06-09T22:17:54.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:54 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3291128566' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-06-09T22:17:54.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:54 smithi194 ceph-mon[36856]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:17:55.805 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:17:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i[67699]: 2024-06-09T22:17:55.368+0000 7f60c2591700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-06-09T22:17:56.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.114+0000 7fc019328700 1 -- 172.21.15.104:0/516305619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0141013a0 msgr2=0x7fc01410f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:56.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.114+0000 7fc019328700 1 --2- 172.21.15.104:0/516305619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0141013a0 0x7fc01410f310 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00b0a0 tx=0x7fc00c00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:17:56.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.114+0000 7fc019328700 1 -- 172.21.15.104:0/516305619 shutdown_connections 2024-06-09T22:17:56.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.114+0000 7fc019328700 1 --2- 172.21.15.104:0/516305619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0141013a0 0x7fc01410f310 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:56.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.114+0000 7fc019328700 1 --2- 172.21.15.104:0/516305619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0141084f0 0x7fc0141088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:56.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.114+0000 7fc019328700 1 --2- 172.21.15.104:0/516305619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0141009e0 0x7fc014100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:56.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.114+0000 7fc019328700 1 -- 172.21.15.104:0/516305619 >> 172.21.15.104:0/516305619 conn(0x7fc014077960 msgr2=0x7fc0140fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:56.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.114+0000 7fc019328700 1 -- 172.21.15.104:0/516305619 shutdown_connections 2024-06-09T22:17:56.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.114+0000 7fc019328700 1 -- 172.21.15.104:0/516305619 wait complete. 2024-06-09T22:17:56.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.114+0000 7fc019328700 1 Processor -- start 2024-06-09T22:17:56.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc019328700 1 -- start start 2024-06-09T22:17:56.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc019328700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0141009e0 0x7fc01419e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:56.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc019328700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0141013a0 0x7fc01419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:56.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc019328700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0141084f0 0x7fc014198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:56.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc019328700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc014112820 con 0x7fc0141084f0 2024-06-09T22:17:56.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc019328700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0141126a0 con 0x7fc0141009e0 2024-06-09T22:17:56.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc019328700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc0141129a0 con 0x7fc0141013a0 2024-06-09T22:17:56.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc0137fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0141084f0 0x7fc014198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:56.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc0137fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0141084f0 0x7fc014198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48534/0 (socket says 172.21.15.104:48534) 2024-06-09T22:17:56.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc0137fe700 1 -- 172.21.15.104:0/36779013 learned_addr learned my addr 172.21.15.104:0/36779013 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:56.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc012ffd700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0141009e0 0x7fc01419e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:56.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc00a7fc700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0141013a0 0x7fc01419ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:56.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc0137fe700 1 -- 172.21.15.104:0/36779013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0141013a0 msgr2=0x7fc01419ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc0137fe700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0141013a0 0x7fc01419ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc0137fe700 1 -- 172.21.15.104:0/36779013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0141009e0 msgr2=0x7fc01419e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc0137fe700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0141009e0 0x7fc01419e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc0137fe700 1 -- 172.21.15.104:0/36779013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0141991a0 con 0x7fc0141084f0 2024-06-09T22:17:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.115+0000 7fc00a7fc700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0141013a0 0x7fc01419ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:17:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.116+0000 7fc0137fe700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0141084f0 0x7fc014198a40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00d010 tx=0x7fc00c00be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.116+0000 7fc010ff9700 1 -- 172.21.15.104:0/36779013 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c021070 con 0x7fc0141084f0 2024-06-09T22:17:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.116+0000 7fc019328700 1 -- 172.21.15.104:0/36779013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc014199430 con 0x7fc0141084f0 2024-06-09T22:17:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.116+0000 7fc010ff9700 1 -- 172.21.15.104:0/36779013 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc00c007d30 con 0x7fc0141084f0 2024-06-09T22:17:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.116+0000 7fc010ff9700 1 -- 172.21.15.104:0/36779013 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c01d450 con 0x7fc0141084f0 2024-06-09T22:17:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.116+0000 7fc019328700 1 -- 172.21.15.104:0/36779013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0141a5700 con 0x7fc0141084f0 2024-06-09T22:17:56.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.117+0000 7fc019328700 1 -- 172.21.15.104:0/36779013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc014069180 con 0x7fc0141084f0 2024-06-09T22:17:56.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.118+0000 7fc010ff9700 1 -- 172.21.15.104:0/36779013 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc00c004120 con 0x7fc0141084f0 2024-06-09T22:17:56.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.119+0000 7fc010ff9700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc0000a2c20 0x7fc0000a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:56.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.119+0000 7fc012ffd700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc0000a2c20 0x7fc0000a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:56.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.119+0000 7fc010ff9700 1 -- 172.21.15.104:0/36779013 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7fc00c00f020 con 0x7fc0141084f0 2024-06-09T22:17:56.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.119+0000 7fc012ffd700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc0000a2c20 0x7fc0000a50e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc004006fd0 tx=0x7fc004008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:56.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.122+0000 7fc010ff9700 1 -- 172.21.15.104:0/36779013 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc00c02a020 con 0x7fc0141084f0 2024-06-09T22:17:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.349+0000 7fc019328700 1 -- 172.21.15.104:0/36779013 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc0140628d0 con 0x7fc0000a2c20 2024-06-09T22:17:56.380 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:17:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:17:56.354+0000 7f15a95b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:17:56.380 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:17:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:17:56.354+0000 7f15a95b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:17:56.380 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:17:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:17:56.354+0000 7f15a95b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:17:56.380 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:17:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:17:56.354+0000 7f15a95b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:17:56.380 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:17:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:17:56.354+0000 7f15a95b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:17:56.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.383+0000 7fc010ff9700 1 -- 172.21.15.104:0/36779013 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc0140628d0 con 0x7fc0000a2c20 2024-06-09T22:17:56.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.386+0000 7fc019328700 1 -- 172.21.15.104:0/36779013 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc0000a2c20 msgr2=0x7fc0000a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:56.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.386+0000 7fc019328700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc0000a2c20 0x7fc0000a50e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc004006fd0 tx=0x7fc004008040 comp rx=0 tx=0).stop 2024-06-09T22:17:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.386+0000 7fc019328700 1 -- 172.21.15.104:0/36779013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0141084f0 msgr2=0x7fc014198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.386+0000 7fc019328700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0141084f0 0x7fc014198a40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00d010 tx=0x7fc00c00be90 comp rx=0 tx=0).stop 2024-06-09T22:17:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.386+0000 7fc019328700 1 -- 172.21.15.104:0/36779013 shutdown_connections 2024-06-09T22:17:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.386+0000 7fc019328700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc0000a2c20 0x7fc0000a50e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.386+0000 7fc019328700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0141084f0 0x7fc014198a40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.386+0000 7fc019328700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0141009e0 0x7fc01419e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.386+0000 7fc019328700 1 --2- 172.21.15.104:0/36779013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0141013a0 0x7fc01419ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.386+0000 7fc019328700 1 -- 172.21.15.104:0/36779013 >> 172.21.15.104:0/36779013 conn(0x7fc014077960 msgr2=0x7fc0140fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.386+0000 7fc019328700 1 -- 172.21.15.104:0/36779013 shutdown_connections 2024-06-09T22:17:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:56.386+0000 7fc019328700 1 -- 172.21.15.104:0/36779013 wait complete. 2024-06-09T22:17:56.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:56 smithi153 ceph-mon[36599]: pgmap v342: 129 pgs: 129 active+clean; 587 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.4 KiB/s wr, 16 op/s 2024-06-09T22:17:56.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:56 smithi104 ceph-mon[32389]: pgmap v342: 129 pgs: 129 active+clean; 587 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.4 KiB/s wr, 16 op/s 2024-06-09T22:17:56.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:56 smithi194 ceph-mon[36856]: pgmap v342: 129 pgs: 129 active+clean; 587 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.4 KiB/s wr, 16 op/s 2024-06-09T22:17:57.032 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-06-09T22:17:57.274 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:17:57.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:57 smithi153 ceph-mon[36599]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:17:57.740 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:57 smithi104 ceph-mon[32389]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:17:57.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:57 smithi194 ceph-mon[36856]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:17:58.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:58 smithi153 ceph-mon[36599]: pgmap v343: 129 pgs: 129 active+clean; 587 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-06-09T22:17:58.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:17:58 smithi153 ceph-mon[36599]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-06-09T22:17:58.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:58 smithi104 ceph-mon[32389]: pgmap v343: 129 pgs: 129 active+clean; 587 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-06-09T22:17:58.752 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:17:58 smithi104 ceph-mon[32389]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-06-09T22:17:58.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:58 smithi194 ceph-mon[36856]: pgmap v343: 129 pgs: 129 active+clean; 587 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-06-09T22:17:58.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:17:58 smithi194 ceph-mon[36856]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-06-09T22:17:58.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.993+0000 7fce51ab3700 1 -- 172.21.15.104:0/2500064740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce4c06abe0 msgr2=0x7fce4c06b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:58.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.993+0000 7fce51ab3700 1 --2- 172.21.15.104:0/2500064740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce4c06abe0 0x7fce4c06b060 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fce40009990 tx=0x7fce40009ca0 comp rx=0 tx=0).stop 2024-06-09T22:17:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.993+0000 7fce51ab3700 1 -- 172.21.15.104:0/2500064740 shutdown_connections 2024-06-09T22:17:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.993+0000 7fce51ab3700 1 --2- 172.21.15.104:0/2500064740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce4c06b5a0 0x7fce4c10f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.993+0000 7fce51ab3700 1 --2- 172.21.15.104:0/2500064740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c108500 0x7fce4c1088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.993+0000 7fce51ab3700 1 --2- 172.21.15.104:0/2500064740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce4c06abe0 0x7fce4c06b060 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.993+0000 7fce51ab3700 1 -- 172.21.15.104:0/2500064740 >> 172.21.15.104:0/2500064740 conn(0x7fce4c078360 msgr2=0x7fce4c078770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.993+0000 7fce51ab3700 1 -- 172.21.15.104:0/2500064740 shutdown_connections 2024-06-09T22:17:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.993+0000 7fce51ab3700 1 -- 172.21.15.104:0/2500064740 wait complete. 2024-06-09T22:17:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.994+0000 7fce51ab3700 1 Processor -- start 2024-06-09T22:17:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.994+0000 7fce51ab3700 1 -- start start 2024-06-09T22:17:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.994+0000 7fce51ab3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce4c06abe0 0x7fce4c19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.994+0000 7fce51ab3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce4c06b5a0 0x7fce4c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.994+0000 7fce51ab3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c108500 0x7fce4c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.994+0000 7fce51ab3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c112830 con 0x7fce4c06abe0 2024-06-09T22:17:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.994+0000 7fce51ab3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c1126b0 con 0x7fce4c108500 2024-06-09T22:17:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.994+0000 7fce51ab3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c1129b0 con 0x7fce4c06b5a0 2024-06-09T22:17:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce4b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce4c06abe0 0x7fce4c19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce4b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce4c06abe0 0x7fce4c19e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48582/0 (socket says 172.21.15.104:48582) 2024-06-09T22:17:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce4b7fe700 1 -- 172.21.15.104:0/1761127199 learned_addr learned my addr 172.21.15.104:0/1761127199 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:17:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce4bfff700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c108500 0x7fce4c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce4affd700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce4c06b5a0 0x7fce4c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce4b7fe700 1 -- 172.21.15.104:0/1761127199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce4c06b5a0 msgr2=0x7fce4c19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce4b7fe700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce4c06b5a0 0x7fce4c19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce4b7fe700 1 -- 172.21.15.104:0/1761127199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c108500 msgr2=0x7fce4c198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce4b7fe700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c108500 0x7fce4c198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce4b7fe700 1 -- 172.21.15.104:0/1761127199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4c199250 con 0x7fce4c06abe0 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce4b7fe700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce4c06abe0 0x7fce4c19e8e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fce3c007e20 tx=0x7fce3c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce48ff9700 1 -- 172.21.15.104:0/1761127199 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c016020 con 0x7fce4c06abe0 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce51ab3700 1 -- 172.21.15.104:0/1761127199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4c199540 con 0x7fce4c06abe0 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce48ff9700 1 -- 172.21.15.104:0/1761127199 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce3c00ac90 con 0x7fce4c06abe0 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce48ff9700 1 -- 172.21.15.104:0/1761127199 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c011620 con 0x7fce4c06abe0 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.995+0000 7fce51ab3700 1 -- 172.21.15.104:0/1761127199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce4c1a57b0 con 0x7fce4c06abe0 2024-06-09T22:17:58.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.996+0000 7fce51ab3700 1 -- 172.21.15.104:0/1761127199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce4c069180 con 0x7fce4c06abe0 2024-06-09T22:17:58.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.997+0000 7fce48ff9700 1 -- 172.21.15.104:0/1761127199 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fce3c019020 con 0x7fce4c06abe0 2024-06-09T22:17:58.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.998+0000 7fce48ff9700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fce340a2cb0 0x7fce340a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:17:58.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.998+0000 7fce4affd700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fce340a2cb0 0x7fce340a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:17:58.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.998+0000 7fce48ff9700 1 -- 172.21.15.104:0/1761127199 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7fce3c0cf0c0 con 0x7fce4c06abe0 2024-06-09T22:17:58.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:58.999+0000 7fce4affd700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fce340a2cb0 0x7fce340a5170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fce40009750 tx=0x7fce40005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:17:58.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.003+0000 7fce48ff9700 1 -- 172.21.15.104:0/1761127199 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fce3c09bc70 con 0x7fce4c06abe0 2024-06-09T22:17:59.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.230+0000 7fce51ab3700 1 -- 172.21.15.104:0/1761127199 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fce4c0628d0 con 0x7fce340a2cb0 2024-06-09T22:17:59.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.263+0000 7fce48ff9700 1 -- 172.21.15.104:0/1761127199 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fce4c0628d0 con 0x7fce340a2cb0 2024-06-09T22:17:59.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.265+0000 7fce51ab3700 1 -- 172.21.15.104:0/1761127199 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fce340a2cb0 msgr2=0x7fce340a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:59.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.265+0000 7fce51ab3700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fce340a2cb0 0x7fce340a5170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fce40009750 tx=0x7fce40005fd0 comp rx=0 tx=0).stop 2024-06-09T22:17:59.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.265+0000 7fce51ab3700 1 -- 172.21.15.104:0/1761127199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce4c06abe0 msgr2=0x7fce4c19e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:17:59.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.265+0000 7fce51ab3700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce4c06abe0 0x7fce4c19e8e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fce3c007e20 tx=0x7fce3c00a340 comp rx=0 tx=0).stop 2024-06-09T22:17:59.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.265+0000 7fce51ab3700 1 -- 172.21.15.104:0/1761127199 shutdown_connections 2024-06-09T22:17:59.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.265+0000 7fce51ab3700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fce340a2cb0 0x7fce340a5170 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:59.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.265+0000 7fce51ab3700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce4c06abe0 0x7fce4c19e8e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:59.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.265+0000 7fce51ab3700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c108500 0x7fce4c198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:59.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.265+0000 7fce51ab3700 1 --2- 172.21.15.104:0/1761127199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce4c06b5a0 0x7fce4c19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:17:59.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.265+0000 7fce51ab3700 1 -- 172.21.15.104:0/1761127199 >> 172.21.15.104:0/1761127199 conn(0x7fce4c078360 msgr2=0x7fce4c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:17:59.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.265+0000 7fce51ab3700 1 -- 172.21.15.104:0/1761127199 shutdown_connections 2024-06-09T22:17:59.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:17:59.265+0000 7fce51ab3700 1 -- 172.21.15.104:0/1761127199 wait complete. 2024-06-09T22:17:59.957 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-06-09T22:18:00.198 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:00.572 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:00 smithi104 ceph-mon[32389]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:18:00.572 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:00 smithi104 ceph-mon[32389]: pgmap v344: 129 pgs: 129 active+clean; 593 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 4.8 KiB/s wr, 21 op/s 2024-06-09T22:18:00.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:00 smithi153 ceph-mon[36599]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:18:00.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:00 smithi153 ceph-mon[36599]: pgmap v344: 129 pgs: 129 active+clean; 593 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 4.8 KiB/s wr, 21 op/s 2024-06-09T22:18:00.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:00 smithi194 ceph-mon[36856]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:18:00.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:00 smithi194 ceph-mon[36856]: pgmap v344: 129 pgs: 129 active+clean; 593 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 4.8 KiB/s wr, 21 op/s 2024-06-09T22:18:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.007+0000 7fe15d266700 1 -- 172.21.15.104:0/4182958843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe158102530 msgr2=0x7fe1581029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.007+0000 7fe15d266700 1 --2- 172.21.15.104:0/4182958843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe158102530 0x7fe1581029b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe1400099c0 tx=0x7fe140009cd0 comp rx=0 tx=0).stop 2024-06-09T22:18:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.007+0000 7fe15d266700 1 -- 172.21.15.104:0/4182958843 shutdown_connections 2024-06-09T22:18:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.007+0000 7fe15d266700 1 --2- 172.21.15.104:0/4182958843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe158102ef0 0x7fe15810f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.007+0000 7fe15d266700 1 --2- 172.21.15.104:0/4182958843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1581085d0 0x7fe1581089b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.007+0000 7fe15d266700 1 --2- 172.21.15.104:0/4182958843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe158102530 0x7fe1581029b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.007+0000 7fe15d266700 1 -- 172.21.15.104:0/4182958843 >> 172.21.15.104:0/4182958843 conn(0x7fe158079640 msgr2=0x7fe158079a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:02.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.007+0000 7fe15d266700 1 -- 172.21.15.104:0/4182958843 shutdown_connections 2024-06-09T22:18:02.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.007+0000 7fe15d266700 1 -- 172.21.15.104:0/4182958843 wait complete. 2024-06-09T22:18:02.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.008+0000 7fe15d266700 1 Processor -- start 2024-06-09T22:18:02.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.008+0000 7fe15d266700 1 -- start start 2024-06-09T22:18:02.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.008+0000 7fe15d266700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe158102530 0x7fe158198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:02.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.008+0000 7fe15d266700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe158102ef0 0x7fe158199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:02.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.008+0000 7fe15d266700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1581085d0 0x7fe15819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:02.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe15d266700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe158112950 con 0x7fe1581085d0 2024-06-09T22:18:02.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe15d266700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe1581127d0 con 0x7fe158102ef0 2024-06-09T22:18:02.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe15d266700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe158112ad0 con 0x7fe158102530 2024-06-09T22:18:02.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe1577fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1581085d0 0x7fe15819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:02.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe1577fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1581085d0 0x7fe15819d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48594/0 (socket says 172.21.15.104:48594) 2024-06-09T22:18:02.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe1577fe700 1 -- 172.21.15.104:0/2633490428 learned_addr learned my addr 172.21.15.104:0/2633490428 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe14e7fc700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe158102ef0 0x7fe158199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe156ffd700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe158102530 0x7fe158198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe1577fe700 1 -- 172.21.15.104:0/2633490428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe158102530 msgr2=0x7fe158198d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe1577fe700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe158102530 0x7fe158198d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe1577fe700 1 -- 172.21.15.104:0/2633490428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe158102ef0 msgr2=0x7fe158199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe1577fe700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe158102ef0 0x7fe158199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe1577fe700 1 -- 172.21.15.104:0/2633490428 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe15819dd30 con 0x7fe1581085d0 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe14e7fc700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe158102ef0 0x7fe158199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe1577fe700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1581085d0 0x7fe15819d5f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe15000b970 tx=0x7fe15000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe154ff9700 1 -- 172.21.15.104:0/2633490428 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe150019070 con 0x7fe1581085d0 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe154ff9700 1 -- 172.21.15.104:0/2633490428 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe150014440 con 0x7fe1581085d0 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe15d266700 1 -- 172.21.15.104:0/2633490428 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe15819e020 con 0x7fe1581085d0 2024-06-09T22:18:02.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe154ff9700 1 -- 172.21.15.104:0/2633490428 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1500075d0 con 0x7fe1581085d0 2024-06-09T22:18:02.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.009+0000 7fe15d266700 1 -- 172.21.15.104:0/2633490428 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1581a58a0 con 0x7fe1581085d0 2024-06-09T22:18:02.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.010+0000 7fe15d266700 1 -- 172.21.15.104:0/2633490428 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe15810aa60 con 0x7fe1581085d0 2024-06-09T22:18:02.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.011+0000 7fe154ff9700 1 -- 172.21.15.104:0/2633490428 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fe15001c070 con 0x7fe1581085d0 2024-06-09T22:18:02.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.012+0000 7fe154ff9700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe1440a2cb0 0x7fe1440a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:02.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.012+0000 7fe156ffd700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe1440a2cb0 0x7fe1440a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:02.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.012+0000 7fe154ff9700 1 -- 172.21.15.104:0/2633490428 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7fe1500ced50 con 0x7fe1581085d0 2024-06-09T22:18:02.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.012+0000 7fe156ffd700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe1440a2cb0 0x7fe1440a5170 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe148006fd0 tx=0x7fe148008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:02.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.015+0000 7fe154ff9700 1 -- 172.21.15.104:0/2633490428 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe15009b900 con 0x7fe1581085d0 2024-06-09T22:18:02.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.243+0000 7fe15d266700 1 -- 172.21.15.104:0/2633490428 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fe15819a080 con 0x7fe1440a2cb0 2024-06-09T22:18:02.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.249+0000 7fe154ff9700 1 -- 172.21.15.104:0/2633490428 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fe15819a080 con 0x7fe1440a2cb0 2024-06-09T22:18:02.246 INFO:teuthology.orchestra.run.smithi104.stdout:/volumes/qa/sv_0/9cc0ba75-ddea-49dc-9f6e-35dfc24d8098 2024-06-09T22:18:02.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.251+0000 7fe15d266700 1 -- 172.21.15.104:0/2633490428 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe1440a2cb0 msgr2=0x7fe1440a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:02.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.251+0000 7fe15d266700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe1440a2cb0 0x7fe1440a5170 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe148006fd0 tx=0x7fe148008040 comp rx=0 tx=0).stop 2024-06-09T22:18:02.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.251+0000 7fe15d266700 1 -- 172.21.15.104:0/2633490428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1581085d0 msgr2=0x7fe15819d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:02.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.251+0000 7fe15d266700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1581085d0 0x7fe15819d5f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe15000b970 tx=0x7fe15000bd30 comp rx=0 tx=0).stop 2024-06-09T22:18:02.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.251+0000 7fe15d266700 1 -- 172.21.15.104:0/2633490428 shutdown_connections 2024-06-09T22:18:02.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.251+0000 7fe15d266700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe1440a2cb0 0x7fe1440a5170 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:02.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.251+0000 7fe15d266700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1581085d0 0x7fe15819d5f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:02.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.251+0000 7fe15d266700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe158102530 0x7fe158198d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:02.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.251+0000 7fe15d266700 1 --2- 172.21.15.104:0/2633490428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe158102ef0 0x7fe158199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:02.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.251+0000 7fe15d266700 1 -- 172.21.15.104:0/2633490428 >> 172.21.15.104:0/2633490428 conn(0x7fe158079640 msgr2=0x7fe158100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:02.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.251+0000 7fe15d266700 1 -- 172.21.15.104:0/2633490428 shutdown_connections 2024-06-09T22:18:02.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:02.251+0000 7fe15d266700 1 -- 172.21.15.104:0/2633490428 wait complete. 2024-06-09T22:18:02.503 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:02 smithi104 ceph-mon[32389]: pgmap v345: 129 pgs: 129 active+clean; 594 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.8 KiB/s wr, 22 op/s 2024-06-09T22:18:02.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:02 smithi153 ceph-mon[36599]: pgmap v345: 129 pgs: 129 active+clean; 594 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.8 KiB/s wr, 22 op/s 2024-06-09T22:18:02.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:02 smithi194 ceph-mon[36856]: pgmap v345: 129 pgs: 129 active+clean; 594 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.8 KiB/s wr, 22 op/s 2024-06-09T22:18:02.975 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-06-09T22:18:03.219 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:03.682 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:03 smithi104 ceph-mon[32389]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:18:03.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:03 smithi153 ceph-mon[36599]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:18:03.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:03 smithi194 ceph-mon[36856]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:18:04.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:04 smithi153 ceph-mon[36599]: pgmap v346: 129 pgs: 129 active+clean; 613 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 3.9 KiB/s wr, 18 op/s 2024-06-09T22:18:04.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:04 smithi104 ceph-mon[32389]: pgmap v346: 129 pgs: 129 active+clean; 613 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 3.9 KiB/s wr, 18 op/s 2024-06-09T22:18:04.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:04 smithi194 ceph-mon[36856]: pgmap v346: 129 pgs: 129 active+clean; 613 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 3.9 KiB/s wr, 18 op/s 2024-06-09T22:18:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.135+0000 7f805338f700 1 -- 172.21.15.104:0/135852297 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f804c1009e0 msgr2=0x7f804c100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.135+0000 7f805338f700 1 --2- 172.21.15.104:0/135852297 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f804c1009e0 0x7f804c100e60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f803c0099c0 tx=0x7f803c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:18:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.135+0000 7f805338f700 1 -- 172.21.15.104:0/135852297 shutdown_connections 2024-06-09T22:18:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.135+0000 7f805338f700 1 --2- 172.21.15.104:0/135852297 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f804c1013a0 0x7f804c10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.135+0000 7f805338f700 1 --2- 172.21.15.104:0/135852297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f804c108560 0x7f804c108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.135+0000 7f805338f700 1 --2- 172.21.15.104:0/135852297 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f804c1009e0 0x7f804c100e60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.135+0000 7f805338f700 1 -- 172.21.15.104:0/135852297 >> 172.21.15.104:0/135852297 conn(0x7f804c077990 msgr2=0x7f804c0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.135+0000 7f805338f700 1 -- 172.21.15.104:0/135852297 shutdown_connections 2024-06-09T22:18:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.135+0000 7f805338f700 1 -- 172.21.15.104:0/135852297 wait complete. 2024-06-09T22:18:05.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805338f700 1 Processor -- start 2024-06-09T22:18:05.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805338f700 1 -- start start 2024-06-09T22:18:05.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805338f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f804c1009e0 0x7f804c101bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:05.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805338f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f804c1013a0 0x7f804c102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:05.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805338f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f804c108560 0x7f804c102640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805338f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f804c1127c0 con 0x7f804c1009e0 2024-06-09T22:18:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805338f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f804c112640 con 0x7f804c1013a0 2024-06-09T22:18:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805338f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f804c112940 con 0x7f804c108560 2024-06-09T22:18:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805112b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f804c1009e0 0x7f804c101bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805112b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f804c1009e0 0x7f804c101bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48624/0 (socket says 172.21.15.104:48624) 2024-06-09T22:18:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805112b700 1 -- 172.21.15.104:0/4058776364 learned_addr learned my addr 172.21.15.104:0/4058776364 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805092a700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f804c1013a0 0x7f804c102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805192c700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f804c108560 0x7f804c102640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805112b700 1 -- 172.21.15.104:0/4058776364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f804c108560 msgr2=0x7f804c102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805112b700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f804c108560 0x7f804c102640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805112b700 1 -- 172.21.15.104:0/4058776364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f804c1013a0 msgr2=0x7f804c102100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805112b700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f804c1013a0 0x7f804c102100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:05.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805112b700 1 -- 172.21.15.104:0/4058776364 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f804c1a7100 con 0x7f804c1009e0 2024-06-09T22:18:05.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.136+0000 7f805112b700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f804c1009e0 0x7f804c101bc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f804800e9c0 tx=0x7f804800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:05.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.137+0000 7f80427fc700 1 -- 172.21.15.104:0/4058776364 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804800cbd0 con 0x7f804c1009e0 2024-06-09T22:18:05.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.137+0000 7f80427fc700 1 -- 172.21.15.104:0/4058776364 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f804800cd70 con 0x7f804c1009e0 2024-06-09T22:18:05.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.137+0000 7f80427fc700 1 -- 172.21.15.104:0/4058776364 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8048010430 con 0x7f804c1009e0 2024-06-09T22:18:05.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.137+0000 7f805338f700 1 -- 172.21.15.104:0/4058776364 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f804c1a73f0 con 0x7f804c1009e0 2024-06-09T22:18:05.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.137+0000 7f805338f700 1 -- 172.21.15.104:0/4058776364 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f804c1a78b0 con 0x7f804c1009e0 2024-06-09T22:18:05.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.138+0000 7f805338f700 1 -- 172.21.15.104:0/4058776364 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f804c10a9f0 con 0x7f804c1009e0 2024-06-09T22:18:05.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.139+0000 7f80427fc700 1 -- 172.21.15.104:0/4058776364 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f8048004750 con 0x7f804c1009e0 2024-06-09T22:18:05.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.139+0000 7f80427fc700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f80380a2cb0 0x7f80380a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:05.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.139+0000 7f805092a700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f80380a2cb0 0x7f80380a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:05.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.139+0000 7f80427fc700 1 -- 172.21.15.104:0/4058776364 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7f80480cf740 con 0x7f804c1009e0 2024-06-09T22:18:05.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.140+0000 7f805092a700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f80380a2cb0 0x7f80380a5170 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f804c103650 tx=0x7f803c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:05.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.144+0000 7f80427fc700 1 -- 172.21.15.104:0/4058776364 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f804809c1e0 con 0x7f804c1009e0 2024-06-09T22:18:05.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.376+0000 7f805338f700 1 -- 172.21.15.104:0/4058776364 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f804c0628d0 con 0x7f80380a2cb0 2024-06-09T22:18:05.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.401+0000 7f80427fc700 1 -- 172.21.15.104:0/4058776364 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f804c0628d0 con 0x7f80380a2cb0 2024-06-09T22:18:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.403+0000 7f805338f700 1 -- 172.21.15.104:0/4058776364 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f80380a2cb0 msgr2=0x7f80380a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.403+0000 7f805338f700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f80380a2cb0 0x7f80380a5170 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f804c103650 tx=0x7f803c005fb0 comp rx=0 tx=0).stop 2024-06-09T22:18:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.403+0000 7f805338f700 1 -- 172.21.15.104:0/4058776364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f804c1009e0 msgr2=0x7f804c101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.403+0000 7f805338f700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f804c1009e0 0x7f804c101bc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f804800e9c0 tx=0x7f804800ecd0 comp rx=0 tx=0).stop 2024-06-09T22:18:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.403+0000 7f805338f700 1 -- 172.21.15.104:0/4058776364 shutdown_connections 2024-06-09T22:18:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.403+0000 7f805338f700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f80380a2cb0 0x7f80380a5170 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:05.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.403+0000 7f805338f700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f804c1009e0 0x7f804c101bc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:05.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.403+0000 7f805338f700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f804c108560 0x7f804c102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:05.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.403+0000 7f805338f700 1 --2- 172.21.15.104:0/4058776364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f804c1013a0 0x7f804c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:05.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.403+0000 7f805338f700 1 -- 172.21.15.104:0/4058776364 >> 172.21.15.104:0/4058776364 conn(0x7f804c077990 msgr2=0x7f804c0fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:05.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.403+0000 7f805338f700 1 -- 172.21.15.104:0/4058776364 shutdown_connections 2024-06-09T22:18:05.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:05.403+0000 7f805338f700 1 -- 172.21.15.104:0/4058776364 wait complete. 2024-06-09T22:18:06.089 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-06-09T22:18:06.328 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:06.707 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:06 smithi104 ceph-mon[32389]: pgmap v347: 129 pgs: 129 active+clean; 613 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 3.7 KiB/s wr, 16 op/s 2024-06-09T22:18:06.707 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:06 smithi104 ceph-mon[32389]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:18:06.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:06 smithi153 ceph-mon[36599]: pgmap v347: 129 pgs: 129 active+clean; 613 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 3.7 KiB/s wr, 16 op/s 2024-06-09T22:18:06.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:06 smithi153 ceph-mon[36599]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:18:06.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:06 smithi194 ceph-mon[36856]: pgmap v347: 129 pgs: 129 active+clean; 613 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 3.7 KiB/s wr, 16 op/s 2024-06-09T22:18:06.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:06 smithi194 ceph-mon[36856]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:18:08.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.166+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1512274851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ef01073f0 msgr2=0x7f0ef01077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:08.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.166+0000 7f0ef4d7c700 1 --2- 172.21.15.104:0/1512274851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ef01073f0 0x7f0ef01077d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0009990 tx=0x7f0ee0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:18:08.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.166+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1512274851 shutdown_connections 2024-06-09T22:18:08.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.166+0000 7f0ef4d7c700 1 --2- 172.21.15.104:0/1512274851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ef01073f0 0x7f0ef01077d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:08.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.166+0000 7f0ef4d7c700 1 --2- 172.21.15.104:0/1512274851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef0101d40 0x7f0ef010e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:08.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.166+0000 7f0ef4d7c700 1 --2- 172.21.15.104:0/1512274851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ef0101380 0x7f0ef0101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:08.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.166+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1512274851 >> 172.21.15.104:0/1512274851 conn(0x7f0ef0078300 msgr2=0x7f0ef0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:08.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.166+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1512274851 shutdown_connections 2024-06-09T22:18:08.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.166+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1512274851 wait complete. 2024-06-09T22:18:08.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0ef4d7c700 1 Processor -- start 2024-06-09T22:18:08.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0ef4d7c700 1 -- start start 2024-06-09T22:18:08.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0ef4d7c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ef0101380 0x7f0ef0199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:08.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0ef4d7c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ef0101d40 0x7f0ef019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:08.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0ef4d7c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef01073f0 0x7f0ef01a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:08.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0ef4d7c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef0111760 con 0x7f0ef0101d40 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0ef4d7c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef01115e0 con 0x7f0ef01073f0 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0ef4d7c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef01118e0 con 0x7f0ef0101380 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0eee59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ef0101380 0x7f0ef0199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0eeed9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef01073f0 0x7f0ef01a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0ee5d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ef0101d40 0x7f0ef019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0eeed9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef01073f0 0x7f0ef01a4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:49272/0 (socket says 172.21.15.104:49272) 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0eee59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ef0101380 0x7f0ef0199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:39512/0 (socket says 172.21.15.104:39512) 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.167+0000 7f0eeed9d700 1 -- 172.21.15.104:0/1606454374 learned_addr learned my addr 172.21.15.104:0/1606454374 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.168+0000 7f0ee5d9b700 1 -- 172.21.15.104:0/1606454374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ef0101380 msgr2=0x7f0ef0199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.168+0000 7f0ee5d9b700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ef0101380 0x7f0ef0199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.168+0000 7f0ee5d9b700 1 -- 172.21.15.104:0/1606454374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef01073f0 msgr2=0x7f0ef01a4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.168+0000 7f0ee5d9b700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef01073f0 0x7f0ef01a4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:08.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.168+0000 7f0ee5d9b700 1 -- 172.21.15.104:0/1606454374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef01a51e0 con 0x7f0ef0101d40 2024-06-09T22:18:08.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.168+0000 7f0eeed9d700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef01073f0 0x7f0ef01a4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:18:08.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.168+0000 7f0ee5d9b700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ef0101d40 0x7f0ef019a160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0ed800b450 tx=0x7f0ed800b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:08.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.168+0000 7f0ee7fff700 1 -- 172.21.15.104:0/1606454374 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8015020 con 0x7f0ef0101d40 2024-06-09T22:18:08.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.168+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1606454374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef01a53b0 con 0x7f0ef0101d40 2024-06-09T22:18:08.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.168+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1606454374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef01a5910 con 0x7f0ef0101d40 2024-06-09T22:18:08.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.168+0000 7f0ee7fff700 1 -- 172.21.15.104:0/1606454374 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed8011d10 con 0x7f0ef0101d40 2024-06-09T22:18:08.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.168+0000 7f0ee7fff700 1 -- 172.21.15.104:0/1606454374 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8010d20 con 0x7f0ef0101d40 2024-06-09T22:18:08.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.169+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1606454374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef0069180 con 0x7f0ef0101d40 2024-06-09T22:18:08.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.170+0000 7f0ee7fff700 1 -- 172.21.15.104:0/1606454374 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f0ed8018020 con 0x7f0ef0101d40 2024-06-09T22:18:08.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.171+0000 7f0ee7fff700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0edc0a2cb0 0x7f0edc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:08.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.171+0000 7f0eee59c700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0edc0a2cb0 0x7f0edc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:08.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.171+0000 7f0ee7fff700 1 -- 172.21.15.104:0/1606454374 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7f0ed80cefb0 con 0x7f0ef0101d40 2024-06-09T22:18:08.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.172+0000 7f0eee59c700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0edc0a2cb0 0x7f0edc0a5170 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000b5c0 tx=0x7f0ee0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:08.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.175+0000 7f0ee7fff700 1 -- 172.21.15.104:0/1606454374 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0ed801f020 con 0x7f0ef0101d40 2024-06-09T22:18:08.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.405+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1606454374 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f0ef00628d0 con 0x7f0edc0a2cb0 2024-06-09T22:18:08.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.411+0000 7f0ee7fff700 1 -- 172.21.15.104:0/1606454374 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0ef00628d0 con 0x7f0edc0a2cb0 2024-06-09T22:18:08.408 INFO:teuthology.orchestra.run.smithi104.stdout:/volumes/qa/sv_1/4df1ea1d-f7bb-4f87-822a-6bafc033026e 2024-06-09T22:18:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.413+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1606454374 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0edc0a2cb0 msgr2=0x7f0edc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.413+0000 7f0ef4d7c700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0edc0a2cb0 0x7f0edc0a5170 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000b5c0 tx=0x7f0ee0005960 comp rx=0 tx=0).stop 2024-06-09T22:18:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.413+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1606454374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ef0101d40 msgr2=0x7f0ef019a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.413+0000 7f0ef4d7c700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ef0101d40 0x7f0ef019a160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0ed800b450 tx=0x7f0ed800b760 comp rx=0 tx=0).stop 2024-06-09T22:18:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.414+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1606454374 shutdown_connections 2024-06-09T22:18:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.414+0000 7f0ef4d7c700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f0edc0a2cb0 0x7f0edc0a5170 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.414+0000 7f0ef4d7c700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef01073f0 0x7f0ef01a4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.414+0000 7f0ef4d7c700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ef0101380 0x7f0ef0199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.414+0000 7f0ef4d7c700 1 --2- 172.21.15.104:0/1606454374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ef0101d40 0x7f0ef019a160 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.414+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1606454374 >> 172.21.15.104:0/1606454374 conn(0x7f0ef0078300 msgr2=0x7f0ef01000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.414+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1606454374 shutdown_connections 2024-06-09T22:18:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:08.414+0000 7f0ef4d7c700 1 -- 172.21.15.104:0/1606454374 wait complete. 2024-06-09T22:18:08.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:08 smithi153 ceph-mon[36599]: pgmap v348: 129 pgs: 129 active+clean; 613 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.6 KiB/s wr, 12 op/s 2024-06-09T22:18:08.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:08 smithi104 ceph-mon[32389]: pgmap v348: 129 pgs: 129 active+clean; 613 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.6 KiB/s wr, 12 op/s 2024-06-09T22:18:08.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:08 smithi194 ceph-mon[36856]: pgmap v348: 129 pgs: 129 active+clean; 613 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.6 KiB/s wr, 12 op/s 2024-06-09T22:18:09.088 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:18:09.328 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:09.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:09 smithi153 ceph-mon[36599]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:18:09.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:09 smithi104 ceph-mon[32389]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:18:09.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:09 smithi194 ceph-mon[36856]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:18:10.576 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:10 smithi104 ceph-mon[32389]: pgmap v349: 129 pgs: 129 active+clean; 630 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-06-09T22:18:10.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:10 smithi153 ceph-mon[36599]: pgmap v349: 129 pgs: 129 active+clean; 630 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-06-09T22:18:10.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:10 smithi194 ceph-mon[36856]: pgmap v349: 129 pgs: 129 active+clean; 630 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-06-09T22:18:11.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.156+0000 7fd8ab24f700 1 -- 172.21.15.104:0/4225800536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8a4107860 msgr2=0x7fd8a410f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:11.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.156+0000 7fd8ab24f700 1 --2- 172.21.15.104:0/4225800536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8a4107860 0x7fd8a410f3e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd89c00b0d0 tx=0x7fd89c00b3e0 comp rx=0 tx=0).stop 2024-06-09T22:18:11.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.156+0000 7fd8ab24f700 1 -- 172.21.15.104:0/4225800536 shutdown_connections 2024-06-09T22:18:11.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.156+0000 7fd8ab24f700 1 --2- 172.21.15.104:0/4225800536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8a4107860 0x7fd8a410f3e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:11.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.156+0000 7fd8ab24f700 1 --2- 172.21.15.104:0/4225800536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8a4102770 0x7fd8a4102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:11.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.156+0000 7fd8ab24f700 1 --2- 172.21.15.104:0/4225800536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a4103120 0x7fd8a4107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:11.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.156+0000 7fd8ab24f700 1 -- 172.21.15.104:0/4225800536 >> 172.21.15.104:0/4225800536 conn(0x7fd8a4079600 msgr2=0x7fd8a4079a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:11.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.156+0000 7fd8ab24f700 1 -- 172.21.15.104:0/4225800536 shutdown_connections 2024-06-09T22:18:11.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.156+0000 7fd8ab24f700 1 -- 172.21.15.104:0/4225800536 wait complete. 2024-06-09T22:18:11.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8ab24f700 1 Processor -- start 2024-06-09T22:18:11.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8ab24f700 1 -- start start 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8ab24f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a4102770 0x7fd8a419e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8ab24f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8a4103120 0x7fd8a419ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8ab24f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8a4107860 0x7fd8a4198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8ab24f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a41127f0 con 0x7fd8a4103120 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8ab24f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a4112670 con 0x7fd8a4102770 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8ab24f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a4112970 con 0x7fd8a4107860 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8a3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8a4103120 0x7fd8a419ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8a8feb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a4102770 0x7fd8a419e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8a3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8a4103120 0x7fd8a419ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44816/0 (socket says 172.21.15.104:44816) 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8a3fff700 1 -- 172.21.15.104:0/578549584 learned_addr learned my addr 172.21.15.104:0/578549584 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.157+0000 7fd8a97ec700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8a4107860 0x7fd8a4198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.158+0000 7fd8a3fff700 1 -- 172.21.15.104:0/578549584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8a4107860 msgr2=0x7fd8a4198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.158+0000 7fd8a3fff700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8a4107860 0x7fd8a4198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:11.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.158+0000 7fd8a3fff700 1 -- 172.21.15.104:0/578549584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a4102770 msgr2=0x7fd8a419e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:11.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.158+0000 7fd8a3fff700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a4102770 0x7fd8a419e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:11.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.158+0000 7fd8a3fff700 1 -- 172.21.15.104:0/578549584 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a4199170 con 0x7fd8a4103120 2024-06-09T22:18:11.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.158+0000 7fd8a8feb700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a4102770 0x7fd8a419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:18:11.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.158+0000 7fd8a3fff700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8a4103120 0x7fd8a419ed90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd89000d950 tx=0x7fd89000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:11.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.158+0000 7fd8a1ffb700 1 -- 172.21.15.104:0/578549584 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890016070 con 0x7fd8a4103120 2024-06-09T22:18:11.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.158+0000 7fd8a1ffb700 1 -- 172.21.15.104:0/578549584 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd89000fd70 con 0x7fd8a4103120 2024-06-09T22:18:11.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.158+0000 7fd8ab24f700 1 -- 172.21.15.104:0/578549584 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a4199460 con 0x7fd8a4103120 2024-06-09T22:18:11.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.158+0000 7fd8a1ffb700 1 -- 172.21.15.104:0/578549584 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89000e610 con 0x7fd8a4103120 2024-06-09T22:18:11.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.158+0000 7fd8ab24f700 1 -- 172.21.15.104:0/578549584 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a41a5720 con 0x7fd8a4103120 2024-06-09T22:18:11.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.159+0000 7fd8ab24f700 1 -- 172.21.15.104:0/578549584 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8a4069180 con 0x7fd8a4103120 2024-06-09T22:18:11.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.161+0000 7fd8a1ffb700 1 -- 172.21.15.104:0/578549584 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fd890019070 con 0x7fd8a4103120 2024-06-09T22:18:11.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.162+0000 7fd8a1ffb700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd8940a2cb0 0x7fd8940a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:11.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.162+0000 7fd8a8feb700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd8940a2cb0 0x7fd8940a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:11.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.162+0000 7fd8a1ffb700 1 -- 172.21.15.104:0/578549584 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7fd8900cef70 con 0x7fd8a4103120 2024-06-09T22:18:11.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.162+0000 7fd8a8feb700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd8940a2cb0 0x7fd8940a5170 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd898005950 tx=0x7fd89800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.165+0000 7fd8a1ffb700 1 -- 172.21.15.104:0/578549584 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd89009ba10 con 0x7fd8a4103120 2024-06-09T22:18:11.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.410+0000 7fd8ab24f700 1 -- 172.21.15.104:0/578549584 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8a402dca0 con 0x7fd8a4103120 2024-06-09T22:18:11.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.411+0000 7fd8a1ffb700 1 -- 172.21.15.104:0/578549584 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7fd89009b310 con 0x7fd8a4103120 2024-06-09T22:18:11.408 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:18:11.409 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:17:53.402012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:18:11.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.413+0000 7fd8ab24f700 1 -- 172.21.15.104:0/578549584 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd8940a2cb0 msgr2=0x7fd8940a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:11.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.413+0000 7fd8ab24f700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd8940a2cb0 0x7fd8940a5170 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd898005950 tx=0x7fd89800a300 comp rx=0 tx=0).stop 2024-06-09T22:18:11.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.413+0000 7fd8ab24f700 1 -- 172.21.15.104:0/578549584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8a4103120 msgr2=0x7fd8a419ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:11.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.413+0000 7fd8ab24f700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8a4103120 0x7fd8a419ed90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd89000d950 tx=0x7fd89000dc60 comp rx=0 tx=0).stop 2024-06-09T22:18:11.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.413+0000 7fd8ab24f700 1 -- 172.21.15.104:0/578549584 shutdown_connections 2024-06-09T22:18:11.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.413+0000 7fd8ab24f700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd8940a2cb0 0x7fd8940a5170 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:11.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.413+0000 7fd8ab24f700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8a4107860 0x7fd8a4198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:11.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.414+0000 7fd8ab24f700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8a4102770 0x7fd8a419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:11.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.414+0000 7fd8ab24f700 1 --2- 172.21.15.104:0/578549584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8a4103120 0x7fd8a419ed90 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:11.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.414+0000 7fd8ab24f700 1 -- 172.21.15.104:0/578549584 >> 172.21.15.104:0/578549584 conn(0x7fd8a4079600 msgr2=0x7fd8a41004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:11.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.414+0000 7fd8ab24f700 1 -- 172.21.15.104:0/578549584 shutdown_connections 2024-06-09T22:18:11.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:11.414+0000 7fd8ab24f700 1 -- 172.21.15.104:0/578549584 wait complete. 2024-06-09T22:18:11.412 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 28 2024-06-09T22:18:11.743 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:11 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/578549584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:18:11.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:11 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/578549584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:18:11.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:11 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/578549584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:18:12.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:18:12.320 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:12.697 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:12 smithi104 ceph-mon[32389]: pgmap v350: 129 pgs: 129 active+clean; 630 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.0 KiB/s wr, 12 op/s 2024-06-09T22:18:12.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:12 smithi194 ceph-mon[36856]: pgmap v350: 129 pgs: 129 active+clean; 630 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.0 KiB/s wr, 12 op/s 2024-06-09T22:18:12.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:12 smithi153 ceph-mon[36599]: pgmap v350: 129 pgs: 129 active+clean; 630 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.0 KiB/s wr, 12 op/s 2024-06-09T22:18:14.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.187+0000 7f6a8636b700 1 -- 172.21.15.104:0/588924919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a80101d40 msgr2=0x7f6a8010e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:14.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.187+0000 7f6a8636b700 1 --2- 172.21.15.104:0/588924919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a80101d40 0x7f6a8010e230 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800b0a0 tx=0x7f6a7800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:18:14.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.187+0000 7f6a8636b700 1 -- 172.21.15.104:0/588924919 shutdown_connections 2024-06-09T22:18:14.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.187+0000 7f6a8636b700 1 --2- 172.21.15.104:0/588924919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a80101d40 0x7f6a8010e230 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:14.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.187+0000 7f6a8636b700 1 --2- 172.21.15.104:0/588924919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a801073f0 0x7f6a801077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:14.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.187+0000 7f6a8636b700 1 --2- 172.21.15.104:0/588924919 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a80101380 0x7f6a80101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:14.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.187+0000 7f6a8636b700 1 -- 172.21.15.104:0/588924919 >> 172.21.15.104:0/588924919 conn(0x7f6a80078300 msgr2=0x7f6a80078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.187+0000 7f6a8636b700 1 -- 172.21.15.104:0/588924919 shutdown_connections 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.187+0000 7f6a8636b700 1 -- 172.21.15.104:0/588924919 wait complete. 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.187+0000 7f6a8636b700 1 Processor -- start 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a8636b700 1 -- start start 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a8636b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80101380 0x7f6a80199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a8636b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a80101d40 0x7f6a8019a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a8636b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a801073f0 0x7f6a801a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a8636b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6a80111760 con 0x7f6a80101d40 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a8636b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a801115e0 con 0x7f6a80101380 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a8636b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6a801118e0 con 0x7f6a801073f0 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a84908700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a801073f0 0x7f6a801a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a7ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80101380 0x7f6a80199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:14.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a77fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a80101d40 0x7f6a8019a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a7ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80101380 0x7f6a80199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:49306/0 (socket says 172.21.15.104:49306) 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a7ffff700 1 -- 172.21.15.104:0/3561533353 learned_addr learned my addr 172.21.15.104:0/3561533353 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a77fff700 1 -- 172.21.15.104:0/3561533353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a801073f0 msgr2=0x7f6a801a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a77fff700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a801073f0 0x7f6a801a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a77fff700 1 -- 172.21.15.104:0/3561533353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80101380 msgr2=0x7f6a80199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a77fff700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80101380 0x7f6a80199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a77fff700 1 -- 172.21.15.104:0/3561533353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a801a50a0 con 0x7f6a80101d40 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.188+0000 7f6a7ffff700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80101380 0x7f6a80199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.189+0000 7f6a77fff700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a80101d40 0x7f6a8019a020 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800b450 tx=0x7f6a6800b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.189+0000 7f6a7dffb700 1 -- 172.21.15.104:0/3561533353 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68015020 con 0x7f6a80101d40 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.189+0000 7f6a8636b700 1 -- 172.21.15.104:0/3561533353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a801a52d0 con 0x7f6a80101d40 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.189+0000 7f6a7dffb700 1 -- 172.21.15.104:0/3561533353 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a68011d10 con 0x7f6a80101d40 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.189+0000 7f6a7dffb700 1 -- 172.21.15.104:0/3561533353 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68010d20 con 0x7f6a80101d40 2024-06-09T22:18:14.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.189+0000 7f6a8636b700 1 -- 172.21.15.104:0/3561533353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a801a5790 con 0x7f6a80101d40 2024-06-09T22:18:14.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.190+0000 7f6a8636b700 1 -- 172.21.15.104:0/3561533353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a80109960 con 0x7f6a80101d40 2024-06-09T22:18:14.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.191+0000 7f6a7dffb700 1 -- 172.21.15.104:0/3561533353 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f6a68018020 con 0x7f6a80101d40 2024-06-09T22:18:14.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.191+0000 7f6a7dffb700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6a6c0a2cb0 0x7f6a6c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:14.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.192+0000 7f6a7ffff700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6a6c0a2cb0 0x7f6a6c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:14.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.192+0000 7f6a7dffb700 1 -- 172.21.15.104:0/3561533353 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7f6a680ceab0 con 0x7f6a80101d40 2024-06-09T22:18:14.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.192+0000 7f6a7ffff700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6a6c0a2cb0 0x7f6a6c0a5170 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009ea0 tx=0x7f6a70009290 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:14.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.196+0000 7f6a7dffb700 1 -- 172.21.15.104:0/3561533353 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6a6801f020 con 0x7f6a80101d40 2024-06-09T22:18:14.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.403+0000 7f6a8636b700 1 -- 172.21.15.104:0/3561533353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6a8006b000 con 0x7f6a80101d40 2024-06-09T22:18:14.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.405+0000 7f6a7dffb700 1 -- 172.21.15.104:0/3561533353 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+24668 (secure 0 0 0) 0x7f6a6809b660 con 0x7f6a80101d40 2024-06-09T22:18:14.402 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:18:14.402 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":97,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:17:26.456251+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:18:14.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.407+0000 7f6a8636b700 1 -- 172.21.15.104:0/3561533353 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6a6c0a2cb0 msgr2=0x7f6a6c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:14.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.407+0000 7f6a8636b700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6a6c0a2cb0 0x7f6a6c0a5170 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009ea0 tx=0x7f6a70009290 comp rx=0 tx=0).stop 2024-06-09T22:18:14.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.407+0000 7f6a8636b700 1 -- 172.21.15.104:0/3561533353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a80101d40 msgr2=0x7f6a8019a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:14.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.407+0000 7f6a8636b700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a80101d40 0x7f6a8019a020 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800b450 tx=0x7f6a6800b760 comp rx=0 tx=0).stop 2024-06-09T22:18:14.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.407+0000 7f6a8636b700 1 -- 172.21.15.104:0/3561533353 shutdown_connections 2024-06-09T22:18:14.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.407+0000 7f6a8636b700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6a6c0a2cb0 0x7f6a6c0a5170 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:14.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.407+0000 7f6a8636b700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a801073f0 0x7f6a801a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:14.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.407+0000 7f6a8636b700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a80101380 0x7f6a80199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:14.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.407+0000 7f6a8636b700 1 --2- 172.21.15.104:0/3561533353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a80101d40 0x7f6a8019a020 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:14.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.407+0000 7f6a8636b700 1 -- 172.21.15.104:0/3561533353 >> 172.21.15.104:0/3561533353 conn(0x7f6a80078300 msgr2=0x7f6a800ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:14.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.407+0000 7f6a8636b700 1 -- 172.21.15.104:0/3561533353 shutdown_connections 2024-06-09T22:18:14.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:14.407+0000 7f6a8636b700 1 -- 172.21.15.104:0/3561533353 wait complete. 2024-06-09T22:18:14.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:14 smithi104 ceph-mon[32389]: pgmap v351: 129 pgs: 129 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-06-09T22:18:14.530 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:14 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3561533353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:18:14.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:14 smithi194 ceph-mon[36856]: pgmap v351: 129 pgs: 129 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-06-09T22:18:14.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:14 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3561533353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:18:14.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:14 smithi153 ceph-mon[36599]: pgmap v351: 129 pgs: 129 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-06-09T22:18:14.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:14 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3561533353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:18:15.206 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-09T22:18:15.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-09T22:18:15.461 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:16.721 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:16 smithi104 ceph-mon[32389]: pgmap v352: 129 pgs: 129 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-06-09T22:18:16.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:16 smithi194 ceph-mon[36856]: pgmap v352: 129 pgs: 129 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-06-09T22:18:16.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:16 smithi153 ceph-mon[36599]: pgmap v352: 129 pgs: 129 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-06-09T22:18:17.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.262+0000 7f97deeca700 1 -- 172.21.15.104:0/1995633538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d8101390 msgr2=0x7f97d8101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:17.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.262+0000 7f97deeca700 1 --2- 172.21.15.104:0/1995633538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d8101390 0x7f97d8101810 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f97c800b0a0 tx=0x7f97c800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:18:17.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.262+0000 7f97deeca700 1 -- 172.21.15.104:0/1995633538 shutdown_connections 2024-06-09T22:18:17.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.262+0000 7f97deeca700 1 --2- 172.21.15.104:0/1995633538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97d8101d50 0x7f97d810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:17.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.262+0000 7f97deeca700 1 --2- 172.21.15.104:0/1995633538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97d8107400 0x7f97d81077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:17.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.262+0000 7f97deeca700 1 --2- 172.21.15.104:0/1995633538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d8101390 0x7f97d8101810 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:17.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.262+0000 7f97deeca700 1 -- 172.21.15.104:0/1995633538 >> 172.21.15.104:0/1995633538 conn(0x7f97d8078340 msgr2=0x7f97d8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:17.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.262+0000 7f97deeca700 1 -- 172.21.15.104:0/1995633538 shutdown_connections 2024-06-09T22:18:17.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.262+0000 7f97deeca700 1 -- 172.21.15.104:0/1995633538 wait complete. 2024-06-09T22:18:17.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.263+0000 7f97deeca700 1 Processor -- start 2024-06-09T22:18:17.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.263+0000 7f97deeca700 1 -- start start 2024-06-09T22:18:17.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.263+0000 7f97deeca700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d8101390 0x7f97d819d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:17.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.263+0000 7f97deeca700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97d8101d50 0x7f97d819dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:17.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.263+0000 7f97deeca700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97d8107400 0x7f97d81979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:17.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.263+0000 7f97deeca700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f97d8111740 con 0x7f97d8101390 2024-06-09T22:18:17.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.263+0000 7f97deeca700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97d81115c0 con 0x7f97d8107400 2024-06-09T22:18:17.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.263+0000 7f97deeca700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f97d81118c0 con 0x7f97d8101d50 2024-06-09T22:18:17.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97dcc66700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d8101390 0x7f97d819d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:17.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97dcc66700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d8101390 0x7f97d819d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60406/0 (socket says 172.21.15.104:60406) 2024-06-09T22:18:17.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97dcc66700 1 -- 172.21.15.104:0/422814021 learned_addr learned my addr 172.21.15.104:0/422814021 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:17.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97d7fff700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97d8101d50 0x7f97d819dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:17.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97dd467700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97d8107400 0x7f97d81979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:17.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97dcc66700 1 -- 172.21.15.104:0/422814021 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97d8101d50 msgr2=0x7f97d819dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:17.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97dcc66700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97d8101d50 0x7f97d819dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:17.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97dcc66700 1 -- 172.21.15.104:0/422814021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97d8107400 msgr2=0x7f97d81979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:17.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97dcc66700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97d8107400 0x7f97d81979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:17.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97dcc66700 1 -- 172.21.15.104:0/422814021 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d8198120 con 0x7f97d8101390 2024-06-09T22:18:17.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97dcc66700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d8101390 0x7f97d819d7b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c9b0 tx=0x7f97d000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:17.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97d5ffb700 1 -- 172.21.15.104:0/422814021 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0007b00 con 0x7f97d8101390 2024-06-09T22:18:17.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97deeca700 1 -- 172.21.15.104:0/422814021 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d8198410 con 0x7f97d8101390 2024-06-09T22:18:17.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97d5ffb700 1 -- 172.21.15.104:0/422814021 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97d000ce50 con 0x7f97d8101390 2024-06-09T22:18:17.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97d5ffb700 1 -- 172.21.15.104:0/422814021 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0018660 con 0x7f97d8101390 2024-06-09T22:18:17.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.264+0000 7f97deeca700 1 -- 172.21.15.104:0/422814021 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d81a4680 con 0x7f97d8101390 2024-06-09T22:18:17.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.265+0000 7f97deeca700 1 -- 172.21.15.104:0/422814021 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d8069180 con 0x7f97d8101390 2024-06-09T22:18:17.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.266+0000 7f97d5ffb700 1 -- 172.21.15.104:0/422814021 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f97d000f450 con 0x7f97d8101390 2024-06-09T22:18:17.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.267+0000 7f97d5ffb700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f97c40a2cb0 0x7f97c40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:17.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.267+0000 7f97d7fff700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f97c40a2cb0 0x7f97c40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:17.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.267+0000 7f97d5ffb700 1 -- 172.21.15.104:0/422814021 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7f97d00cf460 con 0x7f97d8101390 2024-06-09T22:18:17.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.268+0000 7f97d7fff700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f97c40a2cb0 0x7f97c40a5170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f97c800b070 tx=0x7f97c8006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:17.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.272+0000 7f97d5ffb700 1 -- 172.21.15.104:0/422814021 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f97d0014030 con 0x7f97d8101390 2024-06-09T22:18:17.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.464+0000 7f97deeca700 1 -- 172.21.15.104:0/422814021 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f97d80628d0 con 0x7f97c40a2cb0 2024-06-09T22:18:17.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.474+0000 7f97d5ffb700 1 -- 172.21.15.104:0/422814021 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340663 (secure 0 0 0) 0x7f97d80628d0 con 0x7f97c40a2cb0 2024-06-09T22:18:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:18:17.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.477+0000 7f97deeca700 1 -- 172.21.15.104:0/422814021 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f97c40a2cb0 msgr2=0x7f97c40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:17.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.477+0000 7f97deeca700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f97c40a2cb0 0x7f97c40a5170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f97c800b070 tx=0x7f97c8006210 comp rx=0 tx=0).stop 2024-06-09T22:18:17.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.477+0000 7f97deeca700 1 -- 172.21.15.104:0/422814021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d8101390 msgr2=0x7f97d819d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:17.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.477+0000 7f97deeca700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d8101390 0x7f97d819d7b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c9b0 tx=0x7f97d000ccc0 comp rx=0 tx=0).stop 2024-06-09T22:18:17.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.477+0000 7f97deeca700 1 -- 172.21.15.104:0/422814021 shutdown_connections 2024-06-09T22:18:17.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.477+0000 7f97deeca700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f97c40a2cb0 0x7f97c40a5170 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:17.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.477+0000 7f97deeca700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d8101390 0x7f97d819d7b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:17.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.477+0000 7f97deeca700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97d8107400 0x7f97d81979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:17.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.477+0000 7f97deeca700 1 --2- 172.21.15.104:0/422814021 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97d8101d50 0x7f97d819dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:17.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.477+0000 7f97deeca700 1 -- 172.21.15.104:0/422814021 >> 172.21.15.104:0/422814021 conn(0x7f97d8078340 msgr2=0x7f97d80ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:17.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.477+0000 7f97deeca700 1 -- 172.21.15.104:0/422814021 shutdown_connections 2024-06-09T22:18:17.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:17.477+0000 7f97deeca700 1 -- 172.21.15.104:0/422814021 wait complete. 2024-06-09T22:18:17.480 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-06-09T22:18:18.250 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-06-09T22:18:17.330677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654866,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":408,"num_read_kb":326,"num_write":428,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":350,"ondisk_log_size":350,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378948,"kb_used_data":13156,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562876,"statfs":{"total":1151940427776,"available":1151552385024,"internally_reserved":0,"allocated":13471744,"data_stored":8561138,"data_compressed":424191,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27584,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":88,"num_write":21,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003335"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470110+0000","last_change":"2024-06-09T22:16:43.212998+0000","last_active":"2024-06-09T22:17:26.470110+0000","last_peered":"2024-06-09T22:17:26.470110+0000","last_clean":"2024-06-09T22:17:26.470110+0000","last_became_active":"2024-06-09T22:16:43.212614+0000","last_became_peered":"2024-06-09T22:16:43.212614+0000","last_unstale":"2024-06-09T22:17:26.470110+0000","last_undegraded":"2024-06-09T22:17:26.470110+0000","last_fullsized":"2024-06-09T22:17:26.470110+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:22.728486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467914+0000","last_change":"2024-06-09T22:16:43.208746+0000","last_active":"2024-06-09T22:17:26.467914+0000","last_peered":"2024-06-09T22:17:26.467914+0000","last_clean":"2024-06-09T22:17:26.467914+0000","last_became_active":"2024-06-09T22:16:43.208458+0000","last_became_peered":"2024-06-09T22:16:43.208458+0000","last_unstale":"2024-06-09T22:17:26.467914+0000","last_undegraded":"2024-06-09T22:17:26.467914+0000","last_fullsized":"2024-06-09T22:17:26.467914+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:24:35.350449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467803+0000","last_change":"2024-06-09T22:16:43.200770+0000","last_active":"2024-06-09T22:17:26.467803+0000","last_peered":"2024-06-09T22:17:26.467803+0000","last_clean":"2024-06-09T22:17:26.467803+0000","last_became_active":"2024-06-09T22:16:43.200449+0000","last_became_peered":"2024-06-09T22:16:43.200449+0000","last_unstale":"2024-06-09T22:17:26.467803+0000","last_undegraded":"2024-06-09T22:17:26.467803+0000","last_fullsized":"2024-06-09T22:17:26.467803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:51:09.017260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696787+0000","last_change":"2024-06-09T22:16:43.203844+0000","last_active":"2024-06-09T22:17:26.696787+0000","last_peered":"2024-06-09T22:17:26.696787+0000","last_clean":"2024-06-09T22:17:26.696787+0000","last_became_active":"2024-06-09T22:16:43.203431+0000","last_became_peered":"2024-06-09T22:16:43.203431+0000","last_unstale":"2024-06-09T22:17:26.696787+0000","last_undegraded":"2024-06-09T22:17:26.696787+0000","last_fullsized":"2024-06-09T22:17:26.696787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:51:36.913312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468202+0000","last_change":"2024-06-09T22:16:43.207597+0000","last_active":"2024-06-09T22:17:26.468202+0000","last_peered":"2024-06-09T22:17:26.468202+0000","last_clean":"2024-06-09T22:17:26.468202+0000","last_became_active":"2024-06-09T22:16:43.207266+0000","last_became_peered":"2024-06-09T22:16:43.207266+0000","last_unstale":"2024-06-09T22:17:26.468202+0000","last_undegraded":"2024-06-09T22:17:26.468202+0000","last_fullsized":"2024-06-09T22:17:26.468202+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:47:05.558403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.474121+0000","last_change":"2024-06-09T22:17:26.474121+0000","last_active":"2024-06-09T22:17:26.474121+0000","last_peered":"2024-06-09T22:17:26.474121+0000","last_clean":"2024-06-09T22:17:26.474121+0000","last_became_active":"2024-06-09T22:17:26.473619+0000","last_became_peered":"2024-06-09T22:17:26.473619+0000","last_unstale":"2024-06-09T22:17:26.474121+0000","last_undegraded":"2024-06-09T22:17:26.474121+0000","last_fullsized":"2024-06-09T22:17:26.474121+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:56:42.171235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.472272+0000","last_change":"2024-06-09T22:16:43.208815+0000","last_active":"2024-06-09T22:17:26.472272+0000","last_peered":"2024-06-09T22:17:26.472272+0000","last_clean":"2024-06-09T22:17:26.472272+0000","last_became_active":"2024-06-09T22:16:43.208433+0000","last_became_peered":"2024-06-09T22:16:43.208433+0000","last_unstale":"2024-06-09T22:17:26.472272+0000","last_undegraded":"2024-06-09T22:17:26.472272+0000","last_fullsized":"2024-06-09T22:17:26.472272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:36:26.585037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470482+0000","last_change":"2024-06-09T22:16:43.210195+0000","last_active":"2024-06-09T22:17:26.470482+0000","last_peered":"2024-06-09T22:17:26.470482+0000","last_clean":"2024-06-09T22:17:26.470482+0000","last_became_active":"2024-06-09T22:16:43.209892+0000","last_became_peered":"2024-06-09T22:16:43.209892+0000","last_unstale":"2024-06-09T22:17:26.470482+0000","last_undegraded":"2024-06-09T22:17:26.470482+0000","last_fullsized":"2024-06-09T22:17:26.470482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:44.088212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772888+0000","last_change":"2024-06-09T22:16:43.202517+0000","last_active":"2024-06-09T22:17:26.772888+0000","last_peered":"2024-06-09T22:17:26.772888+0000","last_clean":"2024-06-09T22:17:26.772888+0000","last_became_active":"2024-06-09T22:16:43.201796+0000","last_became_peered":"2024-06-09T22:16:43.201796+0000","last_unstale":"2024-06-09T22:17:26.772888+0000","last_undegraded":"2024-06-09T22:17:26.772888+0000","last_fullsized":"2024-06-09T22:17:26.772888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:30.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470410+0000","last_change":"2024-06-09T22:16:43.212732+0000","last_active":"2024-06-09T22:17:26.470410+0000","last_peered":"2024-06-09T22:17:26.470410+0000","last_clean":"2024-06-09T22:17:26.470410+0000","last_became_active":"2024-06-09T22:16:43.212048+0000","last_became_peered":"2024-06-09T22:16:43.212048+0000","last_unstale":"2024-06-09T22:17:26.470410+0000","last_undegraded":"2024-06-09T22:17:26.470410+0000","last_fullsized":"2024-06-09T22:17:26.470410+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:47.383291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473150+0000","last_change":"2024-06-09T22:16:43.203778+0000","last_active":"2024-06-09T22:17:26.473150+0000","last_peered":"2024-06-09T22:17:26.473150+0000","last_clean":"2024-06-09T22:17:26.473150+0000","last_became_active":"2024-06-09T22:16:43.202954+0000","last_became_peered":"2024-06-09T22:16:43.202954+0000","last_unstale":"2024-06-09T22:17:26.473150+0000","last_undegraded":"2024-06-09T22:17:26.473150+0000","last_fullsized":"2024-06-09T22:17:26.473150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:56:53.708645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696798+0000","last_change":"2024-06-09T22:16:43.211425+0000","last_active":"2024-06-09T22:17:26.696798+0000","last_peered":"2024-06-09T22:17:26.696798+0000","last_clean":"2024-06-09T22:17:26.696798+0000","last_became_active":"2024-06-09T22:16:43.211001+0000","last_became_peered":"2024-06-09T22:16:43.211001+0000","last_unstale":"2024-06-09T22:17:26.696798+0000","last_undegraded":"2024-06-09T22:17:26.696798+0000","last_fullsized":"2024-06-09T22:17:26.696798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:20:59.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473445+0000","last_change":"2024-06-09T22:16:43.208334+0000","last_active":"2024-06-09T22:17:26.473445+0000","last_peered":"2024-06-09T22:17:26.473445+0000","last_clean":"2024-06-09T22:17:26.473445+0000","last_became_active":"2024-06-09T22:16:43.207993+0000","last_became_peered":"2024-06-09T22:16:43.207993+0000","last_unstale":"2024-06-09T22:17:26.473445+0000","last_undegraded":"2024-06-09T22:17:26.473445+0000","last_fullsized":"2024-06-09T22:17:26.473445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:07:05.740748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467715+0000","last_change":"2024-06-09T22:16:43.207211+0000","last_active":"2024-06-09T22:17:26.467715+0000","last_peered":"2024-06-09T22:17:26.467715+0000","last_clean":"2024-06-09T22:17:26.467715+0000","last_became_active":"2024-06-09T22:16:43.206889+0000","last_became_peered":"2024-06-09T22:16:43.206889+0000","last_unstale":"2024-06-09T22:17:26.467715+0000","last_undegraded":"2024-06-09T22:17:26.467715+0000","last_fullsized":"2024-06-09T22:17:26.467715+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:25:47.461030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467752+0000","last_change":"2024-06-09T22:16:43.202202+0000","last_active":"2024-06-09T22:17:26.467752+0000","last_peered":"2024-06-09T22:17:26.467752+0000","last_clean":"2024-06-09T22:17:26.467752+0000","last_became_active":"2024-06-09T22:16:43.201834+0000","last_became_peered":"2024-06-09T22:16:43.201834+0000","last_unstale":"2024-06-09T22:17:26.467752+0000","last_undegraded":"2024-06-09T22:17:26.467752+0000","last_fullsized":"2024-06-09T22:17:26.467752+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:31:51.296627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772170+0000","last_change":"2024-06-09T22:16:43.200637+0000","last_active":"2024-06-09T22:17:26.772170+0000","last_peered":"2024-06-09T22:17:26.772170+0000","last_clean":"2024-06-09T22:17:26.772170+0000","last_became_active":"2024-06-09T22:16:43.200268+0000","last_became_peered":"2024-06-09T22:16:43.200268+0000","last_unstale":"2024-06-09T22:17:26.772170+0000","last_undegraded":"2024-06-09T22:17:26.772170+0000","last_fullsized":"2024-06-09T22:17:26.772170+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:25:59.881453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469876+0000","last_change":"2024-06-09T22:16:43.209212+0000","last_active":"2024-06-09T22:17:26.469876+0000","last_peered":"2024-06-09T22:17:26.469876+0000","last_clean":"2024-06-09T22:17:26.469876+0000","last_became_active":"2024-06-09T22:16:43.207789+0000","last_became_peered":"2024-06-09T22:16:43.207789+0000","last_unstale":"2024-06-09T22:17:26.469876+0000","last_undegraded":"2024-06-09T22:17:26.469876+0000","last_fullsized":"2024-06-09T22:17:26.469876+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:17.316595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.465745+0000","last_change":"2024-06-09T22:16:43.210935+0000","last_active":"2024-06-09T22:17:26.465745+0000","last_peered":"2024-06-09T22:17:26.465745+0000","last_clean":"2024-06-09T22:17:26.465745+0000","last_became_active":"2024-06-09T22:16:43.210631+0000","last_became_peered":"2024-06-09T22:16:43.210631+0000","last_unstale":"2024-06-09T22:17:26.465745+0000","last_undegraded":"2024-06-09T22:17:26.465745+0000","last_fullsized":"2024-06-09T22:17:26.465745+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:22:58.887313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470400+0000","last_change":"2024-06-09T22:16:43.198872+0000","last_active":"2024-06-09T22:17:26.470400+0000","last_peered":"2024-06-09T22:17:26.470400+0000","last_clean":"2024-06-09T22:17:26.470400+0000","last_became_active":"2024-06-09T22:16:43.198498+0000","last_became_peered":"2024-06-09T22:16:43.198498+0000","last_unstale":"2024-06-09T22:17:26.470400+0000","last_undegraded":"2024-06-09T22:17:26.470400+0000","last_fullsized":"2024-06-09T22:17:26.470400+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:25:28.853048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470043+0000","last_change":"2024-06-09T22:16:53.343656+0000","last_active":"2024-06-09T22:17:26.470043+0000","last_peered":"2024-06-09T22:17:26.470043+0000","last_clean":"2024-06-09T22:17:26.470043+0000","last_became_active":"2024-06-09T22:16:53.343281+0000","last_became_peered":"2024-06-09T22:16:53.343281+0000","last_unstale":"2024-06-09T22:17:26.470043+0000","last_undegraded":"2024-06-09T22:17:26.470043+0000","last_fullsized":"2024-06-09T22:17:26.470043+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:40.961842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"97'13","reported_seq":77,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:14.482483+0000","last_change":"2024-06-09T22:16:40.040145+0000","last_active":"2024-06-09T22:18:14.482483+0000","last_peered":"2024-06-09T22:18:14.482483+0000","last_clean":"2024-06-09T22:18:14.482483+0000","last_became_active":"2024-06-09T22:16:40.039787+0000","last_became_peered":"2024-06-09T22:16:40.039787+0000","last_unstale":"2024-06-09T22:18:14.482483+0000","last_undegraded":"2024-06-09T22:18:14.482483+0000","last_fullsized":"2024-06-09T22:18:14.482483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:52:32.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.774053+0000","last_change":"2024-06-09T22:17:26.774053+0000","last_active":"2024-06-09T22:17:26.774053+0000","last_peered":"2024-06-09T22:17:26.774053+0000","last_clean":"2024-06-09T22:17:26.774053+0000","last_became_active":"2024-06-09T22:17:26.773786+0000","last_became_peered":"2024-06-09T22:17:26.773786+0000","last_unstale":"2024-06-09T22:17:26.774053+0000","last_undegraded":"2024-06-09T22:17:26.774053+0000","last_fullsized":"2024-06-09T22:17:26.774053+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:36.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696042+0000","last_change":"2024-06-09T22:16:53.332048+0000","last_active":"2024-06-09T22:17:26.696042+0000","last_peered":"2024-06-09T22:17:26.696042+0000","last_clean":"2024-06-09T22:17:26.696042+0000","last_became_active":"2024-06-09T22:16:53.330683+0000","last_became_peered":"2024-06-09T22:16:53.330683+0000","last_unstale":"2024-06-09T22:17:26.696042+0000","last_undegraded":"2024-06-09T22:17:26.696042+0000","last_fullsized":"2024-06-09T22:17:26.696042+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:32:23.219061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"97'2","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470379+0000","last_change":"2024-06-09T22:16:40.070193+0000","last_active":"2024-06-09T22:17:26.470379+0000","last_peered":"2024-06-09T22:17:26.470379+0000","last_clean":"2024-06-09T22:17:26.470379+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:17:26.470379+0000","last_undegraded":"2024-06-09T22:17:26.470379+0000","last_fullsized":"2024-06-09T22:17:26.470379+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:19:19.787552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467804+0000","last_change":"2024-06-09T22:16:43.199814+0000","last_active":"2024-06-09T22:17:26.467804+0000","last_peered":"2024-06-09T22:17:26.467804+0000","last_clean":"2024-06-09T22:17:26.467804+0000","last_became_active":"2024-06-09T22:16:43.199487+0000","last_became_peered":"2024-06-09T22:16:43.199487+0000","last_unstale":"2024-06-09T22:17:26.467804+0000","last_undegraded":"2024-06-09T22:17:26.467804+0000","last_fullsized":"2024-06-09T22:17:26.467804+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:55.672050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696360+0000","last_change":"2024-06-09T22:16:53.336253+0000","last_active":"2024-06-09T22:17:26.696360+0000","last_peered":"2024-06-09T22:17:26.696360+0000","last_clean":"2024-06-09T22:17:26.696360+0000","last_became_active":"2024-06-09T22:16:53.335869+0000","last_became_peered":"2024-06-09T22:16:53.335869+0000","last_unstale":"2024-06-09T22:17:26.696360+0000","last_undegraded":"2024-06-09T22:17:26.696360+0000","last_fullsized":"2024-06-09T22:17:26.696360+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:28:12.894878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"97'12","reported_seq":53,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:51.448960+0000","last_change":"2024-06-09T22:16:40.045365+0000","last_active":"2024-06-09T22:17:51.448960+0000","last_peered":"2024-06-09T22:17:51.448960+0000","last_clean":"2024-06-09T22:17:51.448960+0000","last_became_active":"2024-06-09T22:16:40.044989+0000","last_became_peered":"2024-06-09T22:16:40.044989+0000","last_unstale":"2024-06-09T22:17:51.448960+0000","last_undegraded":"2024-06-09T22:17:51.448960+0000","last_fullsized":"2024-06-09T22:17:51.448960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:01:05.850170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772394+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:17:26.772394+0000","last_peered":"2024-06-09T22:17:26.772394+0000","last_clean":"2024-06-09T22:17:26.772394+0000","last_became_active":"2024-06-09T22:16:43.203999+0000","last_became_peered":"2024-06-09T22:16:43.203999+0000","last_unstale":"2024-06-09T22:17:26.772394+0000","last_undegraded":"2024-06-09T22:17:26.772394+0000","last_fullsized":"2024-06-09T22:17:26.772394+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:37:28.362655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696514+0000","last_change":"2024-06-09T22:16:53.333565+0000","last_active":"2024-06-09T22:17:26.696514+0000","last_peered":"2024-06-09T22:17:26.696514+0000","last_clean":"2024-06-09T22:17:26.696514+0000","last_became_active":"2024-06-09T22:16:53.332921+0000","last_became_peered":"2024-06-09T22:16:53.332921+0000","last_unstale":"2024-06-09T22:17:26.696514+0000","last_undegraded":"2024-06-09T22:17:26.696514+0000","last_fullsized":"2024-06-09T22:17:26.696514+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:32.085617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"97'5","reported_seq":46,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.447829+0000","last_change":"2024-06-09T22:16:40.039766+0000","last_active":"2024-06-09T22:17:50.447829+0000","last_peered":"2024-06-09T22:17:50.447829+0000","last_clean":"2024-06-09T22:17:50.447829+0000","last_became_active":"2024-06-09T22:16:40.039329+0000","last_became_peered":"2024-06-09T22:16:40.039329+0000","last_unstale":"2024-06-09T22:17:50.447829+0000","last_undegraded":"2024-06-09T22:17:50.447829+0000","last_fullsized":"2024-06-09T22:17:50.447829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:53.673890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.773335+0000","last_change":"2024-06-09T22:16:43.201200+0000","last_active":"2024-06-09T22:17:26.773335+0000","last_peered":"2024-06-09T22:17:26.773335+0000","last_clean":"2024-06-09T22:17:26.773335+0000","last_became_active":"2024-06-09T22:16:43.200830+0000","last_became_peered":"2024-06-09T22:16:43.200830+0000","last_unstale":"2024-06-09T22:17:26.773335+0000","last_undegraded":"2024-06-09T22:17:26.773335+0000","last_fullsized":"2024-06-09T22:17:26.773335+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:38:52.093173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469698+0000","last_change":"2024-06-09T22:16:53.339811+0000","last_active":"2024-06-09T22:17:26.469698+0000","last_peered":"2024-06-09T22:17:26.469698+0000","last_clean":"2024-06-09T22:17:26.469698+0000","last_became_active":"2024-06-09T22:16:53.339305+0000","last_became_peered":"2024-06-09T22:16:53.339305+0000","last_unstale":"2024-06-09T22:17:26.469698+0000","last_undegraded":"2024-06-09T22:17:26.469698+0000","last_fullsized":"2024-06-09T22:17:26.469698+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:08:25.800245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"97'6","reported_seq":72,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:13.483336+0000","last_change":"2024-06-09T22:16:40.044159+0000","last_active":"2024-06-09T22:18:13.483336+0000","last_peered":"2024-06-09T22:18:13.483336+0000","last_clean":"2024-06-09T22:18:13.483336+0000","last_became_active":"2024-06-09T22:16:40.043839+0000","last_became_peered":"2024-06-09T22:16:40.043839+0000","last_unstale":"2024-06-09T22:18:13.483336+0000","last_undegraded":"2024-06-09T22:18:13.483336+0000","last_fullsized":"2024-06-09T22:18:13.483336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:22:09.339219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468655+0000","last_change":"2024-06-09T22:16:43.202015+0000","last_active":"2024-06-09T22:17:26.468655+0000","last_peered":"2024-06-09T22:17:26.468655+0000","last_clean":"2024-06-09T22:17:26.468655+0000","last_became_active":"2024-06-09T22:16:43.201689+0000","last_became_peered":"2024-06-09T22:16:43.201689+0000","last_unstale":"2024-06-09T22:17:26.468655+0000","last_undegraded":"2024-06-09T22:17:26.468655+0000","last_fullsized":"2024-06-09T22:17:26.468655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:35:12.180399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470051+0000","last_change":"2024-06-09T22:16:53.339000+0000","last_active":"2024-06-09T22:17:26.470051+0000","last_peered":"2024-06-09T22:17:26.470051+0000","last_clean":"2024-06-09T22:17:26.470051+0000","last_became_active":"2024-06-09T22:16:53.338556+0000","last_became_peered":"2024-06-09T22:16:53.338556+0000","last_unstale":"2024-06-09T22:17:26.470051+0000","last_undegraded":"2024-06-09T22:17:26.470051+0000","last_fullsized":"2024-06-09T22:17:26.470051+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:32:42.458481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"97'7","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:49.441383+0000","last_change":"2024-06-09T22:17:26.698204+0000","last_active":"2024-06-09T22:17:49.441383+0000","last_peered":"2024-06-09T22:17:49.441383+0000","last_clean":"2024-06-09T22:17:49.441383+0000","last_became_active":"2024-06-09T22:17:26.697716+0000","last_became_peered":"2024-06-09T22:17:26.697716+0000","last_unstale":"2024-06-09T22:17:49.441383+0000","last_undegraded":"2024-06-09T22:17:49.441383+0000","last_fullsized":"2024-06-09T22:17:49.441383+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:36:37.724459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.773176+0000","last_change":"2024-06-09T22:16:43.198917+0000","last_active":"2024-06-09T22:17:26.773176+0000","last_peered":"2024-06-09T22:17:26.773176+0000","last_clean":"2024-06-09T22:17:26.773176+0000","last_became_active":"2024-06-09T22:16:43.198567+0000","last_became_peered":"2024-06-09T22:16:43.198567+0000","last_unstale":"2024-06-09T22:17:26.773176+0000","last_undegraded":"2024-06-09T22:17:26.773176+0000","last_fullsized":"2024-06-09T22:17:26.773176+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:34.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469768+0000","last_change":"2024-06-09T22:16:53.342203+0000","last_active":"2024-06-09T22:17:26.469768+0000","last_peered":"2024-06-09T22:17:26.469768+0000","last_clean":"2024-06-09T22:17:26.469768+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:17:26.469768+0000","last_undegraded":"2024-06-09T22:17:26.469768+0000","last_fullsized":"2024-06-09T22:17:26.469768+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:41:41.491961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"97'3","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:48.428754+0000","last_change":"2024-06-09T22:16:40.038684+0000","last_active":"2024-06-09T22:17:48.428754+0000","last_peered":"2024-06-09T22:17:48.428754+0000","last_clean":"2024-06-09T22:17:48.428754+0000","last_became_active":"2024-06-09T22:16:40.038386+0000","last_became_peered":"2024-06-09T22:16:40.038386+0000","last_unstale":"2024-06-09T22:17:48.428754+0000","last_undegraded":"2024-06-09T22:17:48.428754+0000","last_fullsized":"2024-06-09T22:17:48.428754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:59.188758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470596+0000","last_change":"2024-06-09T22:16:43.198837+0000","last_active":"2024-06-09T22:17:26.470596+0000","last_peered":"2024-06-09T22:17:26.470596+0000","last_clean":"2024-06-09T22:17:26.470596+0000","last_became_active":"2024-06-09T22:16:43.198515+0000","last_became_peered":"2024-06-09T22:16:43.198515+0000","last_unstale":"2024-06-09T22:17:26.470596+0000","last_undegraded":"2024-06-09T22:17:26.470596+0000","last_fullsized":"2024-06-09T22:17:26.470596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:36:13.863202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467878+0000","last_change":"2024-06-09T22:16:53.342688+0000","last_active":"2024-06-09T22:17:26.467878+0000","last_peered":"2024-06-09T22:17:26.467878+0000","last_clean":"2024-06-09T22:17:26.467878+0000","last_became_active":"2024-06-09T22:16:53.342314+0000","last_became_peered":"2024-06-09T22:16:53.342314+0000","last_unstale":"2024-06-09T22:17:26.467878+0000","last_undegraded":"2024-06-09T22:17:26.467878+0000","last_fullsized":"2024-06-09T22:17:26.467878+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:51:35.801587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"97'3","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.447410+0000","last_change":"2024-06-09T22:16:40.041831+0000","last_active":"2024-06-09T22:17:50.447410+0000","last_peered":"2024-06-09T22:17:50.447410+0000","last_clean":"2024-06-09T22:17:50.447410+0000","last_became_active":"2024-06-09T22:16:40.041484+0000","last_became_peered":"2024-06-09T22:16:40.041484+0000","last_unstale":"2024-06-09T22:17:50.447410+0000","last_undegraded":"2024-06-09T22:17:50.447410+0000","last_fullsized":"2024-06-09T22:17:50.447410+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:10:19.829357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.465273+0000","last_change":"2024-06-09T22:16:43.210769+0000","last_active":"2024-06-09T22:17:26.465273+0000","last_peered":"2024-06-09T22:17:26.465273+0000","last_clean":"2024-06-09T22:17:26.465273+0000","last_became_active":"2024-06-09T22:16:43.210429+0000","last_became_peered":"2024-06-09T22:16:43.210429+0000","last_unstale":"2024-06-09T22:17:26.465273+0000","last_undegraded":"2024-06-09T22:17:26.465273+0000","last_fullsized":"2024-06-09T22:17:26.465273+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:12:28.788830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469973+0000","last_change":"2024-06-09T22:16:53.336536+0000","last_active":"2024-06-09T22:17:26.469973+0000","last_peered":"2024-06-09T22:17:26.469973+0000","last_clean":"2024-06-09T22:17:26.469973+0000","last_became_active":"2024-06-09T22:16:53.335178+0000","last_became_peered":"2024-06-09T22:16:53.335178+0000","last_unstale":"2024-06-09T22:17:26.469973+0000","last_undegraded":"2024-06-09T22:17:26.469973+0000","last_fullsized":"2024-06-09T22:17:26.469973+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:57:19.476483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"97'4","reported_seq":85,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:12.481334+0000","last_change":"2024-06-09T22:16:40.038484+0000","last_active":"2024-06-09T22:18:12.481334+0000","last_peered":"2024-06-09T22:18:12.481334+0000","last_clean":"2024-06-09T22:18:12.481334+0000","last_became_active":"2024-06-09T22:16:40.038084+0000","last_became_peered":"2024-06-09T22:16:40.038084+0000","last_unstale":"2024-06-09T22:18:12.481334+0000","last_undegraded":"2024-06-09T22:18:12.481334+0000","last_fullsized":"2024-06-09T22:18:12.481334+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:19.733107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":44,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772859+0000","last_change":"2024-06-09T22:16:43.205137+0000","last_active":"2024-06-09T22:17:26.772859+0000","last_peered":"2024-06-09T22:17:26.772859+0000","last_clean":"2024-06-09T22:17:26.772859+0000","last_became_active":"2024-06-09T22:16:43.204499+0000","last_became_peered":"2024-06-09T22:16:43.204499+0000","last_unstale":"2024-06-09T22:17:26.772859+0000","last_undegraded":"2024-06-09T22:17:26.772859+0000","last_fullsized":"2024-06-09T22:17:26.772859+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:15.718493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.465220+0000","last_change":"2024-06-09T22:16:53.344260+0000","last_active":"2024-06-09T22:17:26.465220+0000","last_peered":"2024-06-09T22:17:26.465220+0000","last_clean":"2024-06-09T22:17:26.465220+0000","last_became_active":"2024-06-09T22:16:53.343840+0000","last_became_peered":"2024-06-09T22:16:53.343840+0000","last_unstale":"2024-06-09T22:17:26.465220+0000","last_undegraded":"2024-06-09T22:17:26.465220+0000","last_fullsized":"2024-06-09T22:17:26.465220+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:40:23.305509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"97'4","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.446862+0000","last_change":"2024-06-09T22:16:40.042660+0000","last_active":"2024-06-09T22:17:50.446862+0000","last_peered":"2024-06-09T22:17:50.446862+0000","last_clean":"2024-06-09T22:17:50.446862+0000","last_became_active":"2024-06-09T22:16:40.041353+0000","last_became_peered":"2024-06-09T22:16:40.041353+0000","last_unstale":"2024-06-09T22:17:50.446862+0000","last_undegraded":"2024-06-09T22:17:50.446862+0000","last_fullsized":"2024-06-09T22:17:50.446862+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:34:59.064827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468069+0000","last_change":"2024-06-09T22:16:43.205083+0000","last_active":"2024-06-09T22:17:26.468069+0000","last_peered":"2024-06-09T22:17:26.468069+0000","last_clean":"2024-06-09T22:17:26.468069+0000","last_became_active":"2024-06-09T22:16:43.204600+0000","last_became_peered":"2024-06-09T22:16:43.204600+0000","last_unstale":"2024-06-09T22:17:26.468069+0000","last_undegraded":"2024-06-09T22:17:26.468069+0000","last_fullsized":"2024-06-09T22:17:26.468069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:05:12.555465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696558+0000","last_change":"2024-06-09T22:16:53.336169+0000","last_active":"2024-06-09T22:17:26.696558+0000","last_peered":"2024-06-09T22:17:26.696558+0000","last_clean":"2024-06-09T22:17:26.696558+0000","last_became_active":"2024-06-09T22:16:53.335657+0000","last_became_peered":"2024-06-09T22:16:53.335657+0000","last_unstale":"2024-06-09T22:17:26.696558+0000","last_undegraded":"2024-06-09T22:17:26.696558+0000","last_fullsized":"2024-06-09T22:17:26.696558+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:53:21.282767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"97'5","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:49.440446+0000","last_change":"2024-06-09T22:16:40.070568+0000","last_active":"2024-06-09T22:17:49.440446+0000","last_peered":"2024-06-09T22:17:49.440446+0000","last_clean":"2024-06-09T22:17:49.440446+0000","last_became_active":"2024-06-09T22:16:40.070205+0000","last_became_peered":"2024-06-09T22:16:40.070205+0000","last_unstale":"2024-06-09T22:17:49.440446+0000","last_undegraded":"2024-06-09T22:17:49.440446+0000","last_fullsized":"2024-06-09T22:17:49.440446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.449965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772574+0000","last_change":"2024-06-09T22:16:43.205219+0000","last_active":"2024-06-09T22:17:26.772574+0000","last_peered":"2024-06-09T22:17:26.772574+0000","last_clean":"2024-06-09T22:17:26.772574+0000","last_became_active":"2024-06-09T22:16:43.203467+0000","last_became_peered":"2024-06-09T22:16:43.203467+0000","last_unstale":"2024-06-09T22:17:26.772574+0000","last_undegraded":"2024-06-09T22:17:26.772574+0000","last_fullsized":"2024-06-09T22:17:26.772574+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:22.784034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.465239+0000","last_change":"2024-06-09T22:16:53.338990+0000","last_active":"2024-06-09T22:17:26.465239+0000","last_peered":"2024-06-09T22:17:26.465239+0000","last_clean":"2024-06-09T22:17:26.465239+0000","last_became_active":"2024-06-09T22:16:53.338568+0000","last_became_peered":"2024-06-09T22:16:53.338568+0000","last_unstale":"2024-06-09T22:17:26.465239+0000","last_undegraded":"2024-06-09T22:17:26.465239+0000","last_fullsized":"2024-06-09T22:17:26.465239+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:46:39.814043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"97'31","reported_seq":119,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:12.482548+0000","last_change":"2024-06-09T22:16:40.044404+0000","last_active":"2024-06-09T22:18:12.482548+0000","last_peered":"2024-06-09T22:18:12.482548+0000","last_clean":"2024-06-09T22:18:12.482548+0000","last_became_active":"2024-06-09T22:16:40.044044+0000","last_became_peered":"2024-06-09T22:16:40.044044+0000","last_unstale":"2024-06-09T22:18:12.482548+0000","last_undegraded":"2024-06-09T22:18:12.482548+0000","last_fullsized":"2024-06-09T22:18:12.482548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:50:27.038759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":81,"num_write":33,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469338+0000","last_change":"2024-06-09T22:16:43.206963+0000","last_active":"2024-06-09T22:17:26.469338+0000","last_peered":"2024-06-09T22:17:26.469338+0000","last_clean":"2024-06-09T22:17:26.469338+0000","last_became_active":"2024-06-09T22:16:43.206628+0000","last_became_peered":"2024-06-09T22:16:43.206628+0000","last_unstale":"2024-06-09T22:17:26.469338+0000","last_undegraded":"2024-06-09T22:17:26.469338+0000","last_fullsized":"2024-06-09T22:17:26.469338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:33:39.912236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.773641+0000","last_change":"2024-06-09T22:16:53.339332+0000","last_active":"2024-06-09T22:17:26.773641+0000","last_peered":"2024-06-09T22:17:26.773641+0000","last_clean":"2024-06-09T22:17:26.773641+0000","last_became_active":"2024-06-09T22:16:53.338819+0000","last_became_peered":"2024-06-09T22:16:53.338819+0000","last_unstale":"2024-06-09T22:17:26.773641+0000","last_undegraded":"2024-06-09T22:17:26.773641+0000","last_fullsized":"2024-06-09T22:17:26.773641+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:57:41.142142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"97'9","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:51.449071+0000","last_change":"2024-06-09T22:17:26.478256+0000","last_active":"2024-06-09T22:17:51.449071+0000","last_peered":"2024-06-09T22:17:51.449071+0000","last_clean":"2024-06-09T22:17:51.449071+0000","last_became_active":"2024-06-09T22:17:26.472264+0000","last_became_peered":"2024-06-09T22:17:26.472264+0000","last_unstale":"2024-06-09T22:17:51.449071+0000","last_undegraded":"2024-06-09T22:17:51.449071+0000","last_fullsized":"2024-06-09T22:17:51.449071+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:14:19.919841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473163+0000","last_change":"2024-06-09T22:16:43.204484+0000","last_active":"2024-06-09T22:17:26.473163+0000","last_peered":"2024-06-09T22:17:26.473163+0000","last_clean":"2024-06-09T22:17:26.473163+0000","last_became_active":"2024-06-09T22:16:43.204014+0000","last_became_peered":"2024-06-09T22:16:43.204014+0000","last_unstale":"2024-06-09T22:17:26.473163+0000","last_undegraded":"2024-06-09T22:17:26.473163+0000","last_fullsized":"2024-06-09T22:17:26.473163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:55:12.528231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.475179+0000","last_change":"2024-06-09T22:17:26.475179+0000","last_active":"2024-06-09T22:17:26.475179+0000","last_peered":"2024-06-09T22:17:26.475179+0000","last_clean":"2024-06-09T22:17:26.475179+0000","last_became_active":"2024-06-09T22:17:26.474517+0000","last_became_peered":"2024-06-09T22:17:26.474517+0000","last_unstale":"2024-06-09T22:17:26.475179+0000","last_undegraded":"2024-06-09T22:17:26.475179+0000","last_fullsized":"2024-06-09T22:17:26.475179+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:26:02.387830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"97'5","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.447706+0000","last_change":"2024-06-09T22:16:40.043066+0000","last_active":"2024-06-09T22:17:50.447706+0000","last_peered":"2024-06-09T22:17:50.447706+0000","last_clean":"2024-06-09T22:17:50.447706+0000","last_became_active":"2024-06-09T22:16:40.042619+0000","last_became_peered":"2024-06-09T22:16:40.042619+0000","last_unstale":"2024-06-09T22:17:50.447706+0000","last_undegraded":"2024-06-09T22:17:50.447706+0000","last_fullsized":"2024-06-09T22:17:50.447706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:29.689614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.475566+0000","last_change":"2024-06-09T22:17:26.475566+0000","last_active":"2024-06-09T22:17:26.475566+0000","last_peered":"2024-06-09T22:17:26.475566+0000","last_clean":"2024-06-09T22:17:26.475566+0000","last_became_active":"2024-06-09T22:17:26.475037+0000","last_became_peered":"2024-06-09T22:17:26.475037+0000","last_unstale":"2024-06-09T22:17:26.475566+0000","last_undegraded":"2024-06-09T22:17:26.475566+0000","last_fullsized":"2024-06-09T22:17:26.475566+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:33.075144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.475111+0000","last_change":"2024-06-09T22:17:26.475111+0000","last_active":"2024-06-09T22:17:26.475111+0000","last_peered":"2024-06-09T22:17:26.475111+0000","last_clean":"2024-06-09T22:17:26.475111+0000","last_became_active":"2024-06-09T22:17:26.474563+0000","last_became_peered":"2024-06-09T22:17:26.474563+0000","last_unstale":"2024-06-09T22:17:26.475111+0000","last_undegraded":"2024-06-09T22:17:26.475111+0000","last_fullsized":"2024-06-09T22:17:26.475111+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:09:08.458761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"97'6","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:13.481969+0000","last_change":"2024-06-09T22:16:40.040538+0000","last_active":"2024-06-09T22:18:13.481969+0000","last_peered":"2024-06-09T22:18:13.481969+0000","last_clean":"2024-06-09T22:18:13.481969+0000","last_became_active":"2024-06-09T22:16:40.039896+0000","last_became_peered":"2024-06-09T22:16:40.039896+0000","last_unstale":"2024-06-09T22:18:13.481969+0000","last_undegraded":"2024-06-09T22:18:13.481969+0000","last_fullsized":"2024-06-09T22:18:13.481969+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:00:33.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":978,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.475568+0000","last_change":"2024-06-09T22:17:26.475568+0000","last_active":"2024-06-09T22:17:26.475568+0000","last_peered":"2024-06-09T22:17:26.475568+0000","last_clean":"2024-06-09T22:17:26.475568+0000","last_became_active":"2024-06-09T22:17:26.475163+0000","last_became_peered":"2024-06-09T22:17:26.475163+0000","last_unstale":"2024-06-09T22:17:26.475568+0000","last_undegraded":"2024-06-09T22:17:26.475568+0000","last_fullsized":"2024-06-09T22:17:26.475568+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:20.865164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473986+0000","last_change":"2024-06-09T22:16:53.338201+0000","last_active":"2024-06-09T22:17:26.473986+0000","last_peered":"2024-06-09T22:17:26.473986+0000","last_clean":"2024-06-09T22:17:26.473986+0000","last_became_active":"2024-06-09T22:16:53.337816+0000","last_became_peered":"2024-06-09T22:16:53.337816+0000","last_unstale":"2024-06-09T22:17:26.473986+0000","last_undegraded":"2024-06-09T22:17:26.473986+0000","last_fullsized":"2024-06-09T22:17:26.473986+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:35:51.861135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"97'17","reported_seq":58,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.448956+0000","last_change":"2024-06-09T22:16:40.070770+0000","last_active":"2024-06-09T22:17:50.448956+0000","last_peered":"2024-06-09T22:17:50.448956+0000","last_clean":"2024-06-09T22:17:50.448956+0000","last_became_active":"2024-06-09T22:16:40.070398+0000","last_became_peered":"2024-06-09T22:16:40.070398+0000","last_unstale":"2024-06-09T22:17:50.448956+0000","last_undegraded":"2024-06-09T22:17:50.448956+0000","last_fullsized":"2024-06-09T22:17:50.448956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:04:58.273601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772781+0000","last_change":"2024-06-09T22:16:43.199246+0000","last_active":"2024-06-09T22:17:26.772781+0000","last_peered":"2024-06-09T22:17:26.772781+0000","last_clean":"2024-06-09T22:17:26.772781+0000","last_became_active":"2024-06-09T22:16:43.198006+0000","last_became_peered":"2024-06-09T22:16:43.198006+0000","last_unstale":"2024-06-09T22:17:26.772781+0000","last_undegraded":"2024-06-09T22:17:26.772781+0000","last_fullsized":"2024-06-09T22:17:26.772781+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:30:55.836628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772670+0000","last_change":"2024-06-09T22:16:53.342217+0000","last_active":"2024-06-09T22:17:26.772670+0000","last_peered":"2024-06-09T22:17:26.772670+0000","last_clean":"2024-06-09T22:17:26.772670+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:17:26.772670+0000","last_undegraded":"2024-06-09T22:17:26.772670+0000","last_fullsized":"2024-06-09T22:17:26.772670+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:25:28.635091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"97'2","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.423931+0000","last_change":"2024-06-09T22:16:40.039841+0000","last_active":"2024-06-09T22:17:50.423931+0000","last_peered":"2024-06-09T22:17:50.423931+0000","last_clean":"2024-06-09T22:17:50.423931+0000","last_became_active":"2024-06-09T22:16:40.039416+0000","last_became_peered":"2024-06-09T22:16:40.039416+0000","last_unstale":"2024-06-09T22:17:50.423931+0000","last_undegraded":"2024-06-09T22:17:50.423931+0000","last_fullsized":"2024-06-09T22:17:50.423931+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:27:36.213839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469582+0000","last_change":"2024-06-09T22:16:43.214363+0000","last_active":"2024-06-09T22:17:26.469582+0000","last_peered":"2024-06-09T22:17:26.469582+0000","last_clean":"2024-06-09T22:17:26.469582+0000","last_became_active":"2024-06-09T22:16:43.214015+0000","last_became_peered":"2024-06-09T22:16:43.214015+0000","last_unstale":"2024-06-09T22:17:26.469582+0000","last_undegraded":"2024-06-09T22:17:26.469582+0000","last_fullsized":"2024-06-09T22:17:26.469582+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:13:06.924468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468869+0000","last_change":"2024-06-09T22:16:53.342151+0000","last_active":"2024-06-09T22:17:26.468869+0000","last_peered":"2024-06-09T22:17:26.468869+0000","last_clean":"2024-06-09T22:17:26.468869+0000","last_became_active":"2024-06-09T22:16:53.341743+0000","last_became_peered":"2024-06-09T22:16:53.341743+0000","last_unstale":"2024-06-09T22:17:26.468869+0000","last_undegraded":"2024-06-09T22:17:26.468869+0000","last_fullsized":"2024-06-09T22:17:26.468869+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:39:44.793301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"97'1","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473153+0000","last_change":"2024-06-09T22:17:26.472397+0000","last_active":"2024-06-09T22:17:26.473153+0000","last_peered":"2024-06-09T22:17:26.473153+0000","last_clean":"2024-06-09T22:17:26.473153+0000","last_became_active":"2024-06-09T22:17:26.471950+0000","last_became_peered":"2024-06-09T22:17:26.471950+0000","last_unstale":"2024-06-09T22:17:26.473153+0000","last_undegraded":"2024-06-09T22:17:26.473153+0000","last_fullsized":"2024-06-09T22:17:26.473153+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:35:57.481510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469807+0000","last_change":"2024-06-09T22:16:43.214976+0000","last_active":"2024-06-09T22:17:26.469807+0000","last_peered":"2024-06-09T22:17:26.469807+0000","last_clean":"2024-06-09T22:17:26.469807+0000","last_became_active":"2024-06-09T22:16:43.214592+0000","last_became_peered":"2024-06-09T22:16:43.214592+0000","last_unstale":"2024-06-09T22:17:26.469807+0000","last_undegraded":"2024-06-09T22:17:26.469807+0000","last_fullsized":"2024-06-09T22:17:26.469807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:16:21.981144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473670+0000","last_change":"2024-06-09T22:16:53.342721+0000","last_active":"2024-06-09T22:17:26.473670+0000","last_peered":"2024-06-09T22:17:26.473670+0000","last_clean":"2024-06-09T22:17:26.473670+0000","last_became_active":"2024-06-09T22:16:53.342342+0000","last_became_peered":"2024-06-09T22:16:53.342342+0000","last_unstale":"2024-06-09T22:17:26.473670+0000","last_undegraded":"2024-06-09T22:17:26.473670+0000","last_fullsized":"2024-06-09T22:17:26.473670+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:22:07.469873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"97'5","reported_seq":70,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:13.481881+0000","last_change":"2024-06-09T22:16:40.041307+0000","last_active":"2024-06-09T22:18:13.481881+0000","last_peered":"2024-06-09T22:18:13.481881+0000","last_clean":"2024-06-09T22:18:13.481881+0000","last_became_active":"2024-06-09T22:16:40.040819+0000","last_became_peered":"2024-06-09T22:16:40.040819+0000","last_unstale":"2024-06-09T22:18:13.481881+0000","last_undegraded":"2024-06-09T22:18:13.481881+0000","last_fullsized":"2024-06-09T22:18:13.481881+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:14:29.658293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469809+0000","last_change":"2024-06-09T22:16:43.215719+0000","last_active":"2024-06-09T22:17:26.469809+0000","last_peered":"2024-06-09T22:17:26.469809+0000","last_clean":"2024-06-09T22:17:26.469809+0000","last_became_active":"2024-06-09T22:16:43.215356+0000","last_became_peered":"2024-06-09T22:16:43.215356+0000","last_unstale":"2024-06-09T22:17:26.469809+0000","last_undegraded":"2024-06-09T22:17:26.469809+0000","last_fullsized":"2024-06-09T22:17:26.469809+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:15:07.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473615+0000","last_change":"2024-06-09T22:16:53.342948+0000","last_active":"2024-06-09T22:17:26.473615+0000","last_peered":"2024-06-09T22:17:26.473615+0000","last_clean":"2024-06-09T22:17:26.473615+0000","last_became_active":"2024-06-09T22:16:53.342352+0000","last_became_peered":"2024-06-09T22:16:53.342352+0000","last_unstale":"2024-06-09T22:17:26.473615+0000","last_undegraded":"2024-06-09T22:17:26.473615+0000","last_fullsized":"2024-06-09T22:17:26.473615+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:43:27.453789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772973+0000","last_change":"2024-06-09T22:16:43.204897+0000","last_active":"2024-06-09T22:17:26.772973+0000","last_peered":"2024-06-09T22:17:26.772973+0000","last_clean":"2024-06-09T22:17:26.772973+0000","last_became_active":"2024-06-09T22:16:43.204470+0000","last_became_peered":"2024-06-09T22:16:43.204470+0000","last_unstale":"2024-06-09T22:17:26.772973+0000","last_undegraded":"2024-06-09T22:17:26.772973+0000","last_fullsized":"2024-06-09T22:17:26.772973+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:56.681392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"97'5","reported_seq":70,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:13.481399+0000","last_change":"2024-06-09T22:16:40.070079+0000","last_active":"2024-06-09T22:18:13.481399+0000","last_peered":"2024-06-09T22:18:13.481399+0000","last_clean":"2024-06-09T22:18:13.481399+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:18:13.481399+0000","last_undegraded":"2024-06-09T22:18:13.481399+0000","last_fullsized":"2024-06-09T22:18:13.481399+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:00:45.132139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"97'1","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473014+0000","last_change":"2024-06-09T22:16:53.338891+0000","last_active":"2024-06-09T22:17:26.473014+0000","last_peered":"2024-06-09T22:17:26.473014+0000","last_clean":"2024-06-09T22:17:26.473014+0000","last_became_active":"2024-06-09T22:16:53.338526+0000","last_became_peered":"2024-06-09T22:16:53.338526+0000","last_unstale":"2024-06-09T22:17:26.473014+0000","last_undegraded":"2024-06-09T22:17:26.473014+0000","last_fullsized":"2024-06-09T22:17:26.473014+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:34:37.004407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470574+0000","last_change":"2024-06-09T22:16:43.199234+0000","last_active":"2024-06-09T22:17:26.470574+0000","last_peered":"2024-06-09T22:17:26.470574+0000","last_clean":"2024-06-09T22:17:26.470574+0000","last_became_active":"2024-06-09T22:16:43.192012+0000","last_became_peered":"2024-06-09T22:16:43.192012+0000","last_unstale":"2024-06-09T22:17:26.470574+0000","last_undegraded":"2024-06-09T22:17:26.470574+0000","last_fullsized":"2024-06-09T22:17:26.470574+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:17.429236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"97'6","reported_seq":47,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:51.450603+0000","last_change":"2024-06-09T22:16:40.041970+0000","last_active":"2024-06-09T22:17:51.450603+0000","last_peered":"2024-06-09T22:17:51.450603+0000","last_clean":"2024-06-09T22:17:51.450603+0000","last_became_active":"2024-06-09T22:16:40.040371+0000","last_became_peered":"2024-06-09T22:16:40.040371+0000","last_unstale":"2024-06-09T22:17:51.450603+0000","last_undegraded":"2024-06-09T22:17:51.450603+0000","last_fullsized":"2024-06-09T22:17:51.450603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:52:31.210423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":374,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.472832+0000","last_change":"2024-06-09T22:15:15.367397+0000","last_active":"2024-06-09T22:17:26.472832+0000","last_peered":"2024-06-09T22:17:26.472832+0000","last_clean":"2024-06-09T22:17:26.472832+0000","last_became_active":"2024-06-09T22:15:15.366974+0000","last_became_peered":"2024-06-09T22:15:15.366974+0000","last_unstale":"2024-06-09T22:17:26.472832+0000","last_undegraded":"2024-06-09T22:17:26.472832+0000","last_fullsized":"2024-06-09T22:17:26.472832+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_clean_scrub_stamp":"2024-06-09T22:11:49.886056+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:24.018031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.773504+0000","last_change":"2024-06-09T22:16:53.342249+0000","last_active":"2024-06-09T22:17:26.773504+0000","last_peered":"2024-06-09T22:17:26.773504+0000","last_clean":"2024-06-09T22:17:26.773504+0000","last_became_active":"2024-06-09T22:16:53.341779+0000","last_became_peered":"2024-06-09T22:16:53.341779+0000","last_unstale":"2024-06-09T22:17:26.773504+0000","last_undegraded":"2024-06-09T22:17:26.773504+0000","last_fullsized":"2024-06-09T22:17:26.773504+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:21:21.101578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.472826+0000","last_change":"2024-06-09T22:16:43.200778+0000","last_active":"2024-06-09T22:17:26.472826+0000","last_peered":"2024-06-09T22:17:26.472826+0000","last_clean":"2024-06-09T22:17:26.472826+0000","last_became_active":"2024-06-09T22:16:43.200490+0000","last_became_peered":"2024-06-09T22:16:43.200490+0000","last_unstale":"2024-06-09T22:17:26.472826+0000","last_undegraded":"2024-06-09T22:17:26.472826+0000","last_fullsized":"2024-06-09T22:17:26.472826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:20:17.530204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"97'4","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:48.428568+0000","last_change":"2024-06-09T22:16:40.041774+0000","last_active":"2024-06-09T22:17:48.428568+0000","last_peered":"2024-06-09T22:17:48.428568+0000","last_clean":"2024-06-09T22:17:48.428568+0000","last_became_active":"2024-06-09T22:16:40.041085+0000","last_became_peered":"2024-06-09T22:16:40.041085+0000","last_unstale":"2024-06-09T22:17:48.428568+0000","last_undegraded":"2024-06-09T22:17:48.428568+0000","last_fullsized":"2024-06-09T22:17:48.428568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:09:41.140083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468191+0000","last_change":"2024-06-09T22:16:53.336895+0000","last_active":"2024-06-09T22:17:26.468191+0000","last_peered":"2024-06-09T22:17:26.468191+0000","last_clean":"2024-06-09T22:17:26.468191+0000","last_became_active":"2024-06-09T22:16:53.336497+0000","last_became_peered":"2024-06-09T22:16:53.336497+0000","last_unstale":"2024-06-09T22:17:26.468191+0000","last_undegraded":"2024-06-09T22:17:26.468191+0000","last_fullsized":"2024-06-09T22:17:26.468191+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:29:52.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"97'4","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.448185+0000","last_change":"2024-06-09T22:16:40.070909+0000","last_active":"2024-06-09T22:17:50.448185+0000","last_peered":"2024-06-09T22:17:50.448185+0000","last_clean":"2024-06-09T22:17:50.448185+0000","last_became_active":"2024-06-09T22:16:40.070556+0000","last_became_peered":"2024-06-09T22:16:40.070556+0000","last_unstale":"2024-06-09T22:17:50.448185+0000","last_undegraded":"2024-06-09T22:17:50.448185+0000","last_fullsized":"2024-06-09T22:17:50.448185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:59:17.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473784+0000","last_change":"2024-06-09T22:16:43.205951+0000","last_active":"2024-06-09T22:17:26.473784+0000","last_peered":"2024-06-09T22:17:26.473784+0000","last_clean":"2024-06-09T22:17:26.473784+0000","last_became_active":"2024-06-09T22:16:43.205594+0000","last_became_peered":"2024-06-09T22:16:43.205594+0000","last_unstale":"2024-06-09T22:17:26.473784+0000","last_undegraded":"2024-06-09T22:17:26.473784+0000","last_fullsized":"2024-06-09T22:17:26.473784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:17.956406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469938+0000","last_change":"2024-06-09T22:16:53.343292+0000","last_active":"2024-06-09T22:17:26.469938+0000","last_peered":"2024-06-09T22:17:26.469938+0000","last_clean":"2024-06-09T22:17:26.469938+0000","last_became_active":"2024-06-09T22:16:53.342900+0000","last_became_peered":"2024-06-09T22:16:53.342900+0000","last_unstale":"2024-06-09T22:17:26.469938+0000","last_undegraded":"2024-06-09T22:17:26.469938+0000","last_fullsized":"2024-06-09T22:17:26.469938+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:08:07.328250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"97'15","reported_seq":53,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:12.220326+0000","last_change":"2024-06-09T22:16:40.039417+0000","last_active":"2024-06-09T22:18:12.220326+0000","last_peered":"2024-06-09T22:18:12.220326+0000","last_clean":"2024-06-09T22:18:12.220326+0000","last_became_active":"2024-06-09T22:16:40.039099+0000","last_became_peered":"2024-06-09T22:16:40.039099+0000","last_unstale":"2024-06-09T22:18:12.220326+0000","last_undegraded":"2024-06-09T22:18:12.220326+0000","last_fullsized":"2024-06-09T22:18:12.220326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:22:59.284442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470377+0000","last_change":"2024-06-09T22:16:43.201285+0000","last_active":"2024-06-09T22:17:26.470377+0000","last_peered":"2024-06-09T22:17:26.470377+0000","last_clean":"2024-06-09T22:17:26.470377+0000","last_became_active":"2024-06-09T22:16:43.200993+0000","last_became_peered":"2024-06-09T22:16:43.200993+0000","last_unstale":"2024-06-09T22:17:26.470377+0000","last_undegraded":"2024-06-09T22:17:26.470377+0000","last_fullsized":"2024-06-09T22:17:26.470377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:15:26.491588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470670+0000","last_change":"2024-06-09T22:16:53.334088+0000","last_active":"2024-06-09T22:17:26.470670+0000","last_peered":"2024-06-09T22:17:26.470670+0000","last_clean":"2024-06-09T22:17:26.470670+0000","last_became_active":"2024-06-09T22:16:53.333597+0000","last_became_peered":"2024-06-09T22:16:53.333597+0000","last_unstale":"2024-06-09T22:17:26.470670+0000","last_undegraded":"2024-06-09T22:17:26.470670+0000","last_fullsized":"2024-06-09T22:17:26.470670+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:50:22.041816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"97'7","reported_seq":46,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.448170+0000","last_change":"2024-06-09T22:16:40.364982+0000","last_active":"2024-06-09T22:17:50.448170+0000","last_peered":"2024-06-09T22:17:50.448170+0000","last_clean":"2024-06-09T22:17:50.448170+0000","last_became_active":"2024-06-09T22:16:40.364585+0000","last_became_peered":"2024-06-09T22:16:40.364585+0000","last_unstale":"2024-06-09T22:17:50.448170+0000","last_undegraded":"2024-06-09T22:17:50.448170+0000","last_fullsized":"2024-06-09T22:17:50.448170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:28.777950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.474491+0000","last_change":"2024-06-09T22:17:26.474491+0000","last_active":"2024-06-09T22:17:26.474491+0000","last_peered":"2024-06-09T22:17:26.474491+0000","last_clean":"2024-06-09T22:17:26.474491+0000","last_became_active":"2024-06-09T22:17:26.473975+0000","last_became_peered":"2024-06-09T22:17:26.473975+0000","last_unstale":"2024-06-09T22:17:26.474491+0000","last_undegraded":"2024-06-09T22:17:26.474491+0000","last_fullsized":"2024-06-09T22:17:26.474491+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:13:46.998595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696646+0000","last_change":"2024-06-09T22:16:53.342273+0000","last_active":"2024-06-09T22:17:26.696646+0000","last_peered":"2024-06-09T22:17:26.696646+0000","last_clean":"2024-06-09T22:17:26.696646+0000","last_became_active":"2024-06-09T22:16:53.341827+0000","last_became_peered":"2024-06-09T22:16:53.341827+0000","last_unstale":"2024-06-09T22:17:26.696646+0000","last_undegraded":"2024-06-09T22:17:26.696646+0000","last_fullsized":"2024-06-09T22:17:26.696646+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:23:26.099737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"97'2","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:47.400625+0000","last_change":"2024-06-09T22:16:40.070498+0000","last_active":"2024-06-09T22:17:47.400625+0000","last_peered":"2024-06-09T22:17:47.400625+0000","last_clean":"2024-06-09T22:17:47.400625+0000","last_became_active":"2024-06-09T22:16:40.070144+0000","last_became_peered":"2024-06-09T22:16:40.070144+0000","last_unstale":"2024-06-09T22:17:47.400625+0000","last_undegraded":"2024-06-09T22:17:47.400625+0000","last_fullsized":"2024-06-09T22:17:47.400625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:52:22.214832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468077+0000","last_change":"2024-06-09T22:16:43.210448+0000","last_active":"2024-06-09T22:17:26.468077+0000","last_peered":"2024-06-09T22:17:26.468077+0000","last_clean":"2024-06-09T22:17:26.468077+0000","last_became_active":"2024-06-09T22:16:43.209980+0000","last_became_peered":"2024-06-09T22:16:43.209980+0000","last_unstale":"2024-06-09T22:17:26.468077+0000","last_undegraded":"2024-06-09T22:17:26.468077+0000","last_fullsized":"2024-06-09T22:17:26.468077+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:58:15.691207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.465998+0000","last_change":"2024-06-09T22:16:53.340644+0000","last_active":"2024-06-09T22:17:26.465998+0000","last_peered":"2024-06-09T22:17:26.465998+0000","last_clean":"2024-06-09T22:17:26.465998+0000","last_became_active":"2024-06-09T22:16:53.339771+0000","last_became_peered":"2024-06-09T22:16:53.339771+0000","last_unstale":"2024-06-09T22:17:26.465998+0000","last_undegraded":"2024-06-09T22:17:26.465998+0000","last_fullsized":"2024-06-09T22:17:26.465998+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:45:23.281387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"97'7","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.447669+0000","last_change":"2024-06-09T22:16:40.040461+0000","last_active":"2024-06-09T22:17:50.447669+0000","last_peered":"2024-06-09T22:17:50.447669+0000","last_clean":"2024-06-09T22:17:50.447669+0000","last_became_active":"2024-06-09T22:16:40.040168+0000","last_became_peered":"2024-06-09T22:16:40.040168+0000","last_unstale":"2024-06-09T22:17:50.447669+0000","last_undegraded":"2024-06-09T22:17:50.447669+0000","last_fullsized":"2024-06-09T22:17:50.447669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:28.936631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467671+0000","last_change":"2024-06-09T22:16:43.208779+0000","last_active":"2024-06-09T22:17:26.467671+0000","last_peered":"2024-06-09T22:17:26.467671+0000","last_clean":"2024-06-09T22:17:26.467671+0000","last_became_active":"2024-06-09T22:16:43.208136+0000","last_became_peered":"2024-06-09T22:16:43.208136+0000","last_unstale":"2024-06-09T22:17:26.467671+0000","last_undegraded":"2024-06-09T22:17:26.467671+0000","last_fullsized":"2024-06-09T22:17:26.467671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:22:39.721069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"97'1","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696723+0000","last_change":"2024-06-09T22:16:53.331241+0000","last_active":"2024-06-09T22:17:26.696723+0000","last_peered":"2024-06-09T22:17:26.696723+0000","last_clean":"2024-06-09T22:17:26.696723+0000","last_became_active":"2024-06-09T22:16:53.329589+0000","last_became_peered":"2024-06-09T22:16:53.329589+0000","last_unstale":"2024-06-09T22:17:26.696723+0000","last_undegraded":"2024-06-09T22:17:26.696723+0000","last_fullsized":"2024-06-09T22:17:26.696723+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:56:21.319711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"97'3","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:49.441888+0000","last_change":"2024-06-09T22:16:40.042491+0000","last_active":"2024-06-09T22:17:49.441888+0000","last_peered":"2024-06-09T22:17:49.441888+0000","last_clean":"2024-06-09T22:17:49.441888+0000","last_became_active":"2024-06-09T22:16:40.041846+0000","last_became_peered":"2024-06-09T22:16:40.041846+0000","last_unstale":"2024-06-09T22:17:49.441888+0000","last_undegraded":"2024-06-09T22:17:49.441888+0000","last_fullsized":"2024-06-09T22:17:49.441888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:59.144003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470789+0000","last_change":"2024-06-09T22:16:43.209195+0000","last_active":"2024-06-09T22:17:26.470789+0000","last_peered":"2024-06-09T22:17:26.470789+0000","last_clean":"2024-06-09T22:17:26.470789+0000","last_became_active":"2024-06-09T22:16:43.208797+0000","last_became_peered":"2024-06-09T22:16:43.208797+0000","last_unstale":"2024-06-09T22:17:26.470789+0000","last_undegraded":"2024-06-09T22:17:26.470789+0000","last_fullsized":"2024-06-09T22:17:26.470789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:05:31.689900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696746+0000","last_change":"2024-06-09T22:16:53.342173+0000","last_active":"2024-06-09T22:17:26.696746+0000","last_peered":"2024-06-09T22:17:26.696746+0000","last_clean":"2024-06-09T22:17:26.696746+0000","last_became_active":"2024-06-09T22:16:53.341712+0000","last_became_peered":"2024-06-09T22:16:53.341712+0000","last_unstale":"2024-06-09T22:17:26.696746+0000","last_undegraded":"2024-06-09T22:17:26.696746+0000","last_fullsized":"2024-06-09T22:17:26.696746+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:53:12.574764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"97'4","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:47.401751+0000","last_change":"2024-06-09T22:16:40.044238+0000","last_active":"2024-06-09T22:17:47.401751+0000","last_peered":"2024-06-09T22:17:47.401751+0000","last_clean":"2024-06-09T22:17:47.401751+0000","last_became_active":"2024-06-09T22:16:40.043857+0000","last_became_peered":"2024-06-09T22:16:40.043857+0000","last_unstale":"2024-06-09T22:17:47.401751+0000","last_undegraded":"2024-06-09T22:17:47.401751+0000","last_fullsized":"2024-06-09T22:17:47.401751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:44:45.743307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.472875+0000","last_change":"2024-06-09T22:16:43.202363+0000","last_active":"2024-06-09T22:17:26.472875+0000","last_peered":"2024-06-09T22:17:26.472875+0000","last_clean":"2024-06-09T22:17:26.472875+0000","last_became_active":"2024-06-09T22:16:43.202029+0000","last_became_peered":"2024-06-09T22:16:43.202029+0000","last_unstale":"2024-06-09T22:17:26.472875+0000","last_undegraded":"2024-06-09T22:17:26.472875+0000","last_fullsized":"2024-06-09T22:17:26.472875+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:11:19.513055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772267+0000","last_change":"2024-06-09T22:16:53.333837+0000","last_active":"2024-06-09T22:17:26.772267+0000","last_peered":"2024-06-09T22:17:26.772267+0000","last_clean":"2024-06-09T22:17:26.772267+0000","last_became_active":"2024-06-09T22:16:53.333362+0000","last_became_peered":"2024-06-09T22:16:53.333362+0000","last_unstale":"2024-06-09T22:17:26.772267+0000","last_undegraded":"2024-06-09T22:17:26.772267+0000","last_fullsized":"2024-06-09T22:17:26.772267+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:47:04.463040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"97'6","reported_seq":70,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:12.482226+0000","last_change":"2024-06-09T22:16:40.364834+0000","last_active":"2024-06-09T22:18:12.482226+0000","last_peered":"2024-06-09T22:18:12.482226+0000","last_clean":"2024-06-09T22:18:12.482226+0000","last_became_active":"2024-06-09T22:16:40.364433+0000","last_became_peered":"2024-06-09T22:16:40.364433+0000","last_unstale":"2024-06-09T22:18:12.482226+0000","last_undegraded":"2024-06-09T22:18:12.482226+0000","last_fullsized":"2024-06-09T22:18:12.482226+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:43:01.967911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473314+0000","last_change":"2024-06-09T22:16:43.201183+0000","last_active":"2024-06-09T22:17:26.473314+0000","last_peered":"2024-06-09T22:17:26.473314+0000","last_clean":"2024-06-09T22:17:26.473314+0000","last_became_active":"2024-06-09T22:16:43.200774+0000","last_became_peered":"2024-06-09T22:16:43.200774+0000","last_unstale":"2024-06-09T22:17:26.473314+0000","last_undegraded":"2024-06-09T22:17:26.473314+0000","last_fullsized":"2024-06-09T22:17:26.473314+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:07:55.772066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467752+0000","last_change":"2024-06-09T22:16:53.345820+0000","last_active":"2024-06-09T22:17:26.467752+0000","last_peered":"2024-06-09T22:17:26.467752+0000","last_clean":"2024-06-09T22:17:26.467752+0000","last_became_active":"2024-06-09T22:16:53.345449+0000","last_became_peered":"2024-06-09T22:16:53.345449+0000","last_unstale":"2024-06-09T22:17:26.467752+0000","last_undegraded":"2024-06-09T22:17:26.467752+0000","last_fullsized":"2024-06-09T22:17:26.467752+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:47:54.478520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"97'3","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.445679+0000","last_change":"2024-06-09T22:16:40.040582+0000","last_active":"2024-06-09T22:17:50.445679+0000","last_peered":"2024-06-09T22:17:50.445679+0000","last_clean":"2024-06-09T22:17:50.445679+0000","last_became_active":"2024-06-09T22:16:40.040152+0000","last_became_peered":"2024-06-09T22:16:40.040152+0000","last_unstale":"2024-06-09T22:17:50.445679+0000","last_undegraded":"2024-06-09T22:17:50.445679+0000","last_fullsized":"2024-06-09T22:17:50.445679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:50:00.878096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.472792+0000","last_change":"2024-06-09T22:16:43.206211+0000","last_active":"2024-06-09T22:17:26.472792+0000","last_peered":"2024-06-09T22:17:26.472792+0000","last_clean":"2024-06-09T22:17:26.472792+0000","last_became_active":"2024-06-09T22:16:43.205814+0000","last_became_peered":"2024-06-09T22:16:43.205814+0000","last_unstale":"2024-06-09T22:17:26.472792+0000","last_undegraded":"2024-06-09T22:17:26.472792+0000","last_fullsized":"2024-06-09T22:17:26.472792+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:20:19.058475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772465+0000","last_change":"2024-06-09T22:16:53.337695+0000","last_active":"2024-06-09T22:17:26.772465+0000","last_peered":"2024-06-09T22:17:26.772465+0000","last_clean":"2024-06-09T22:17:26.772465+0000","last_became_active":"2024-06-09T22:16:53.334872+0000","last_became_peered":"2024-06-09T22:16:53.334872+0000","last_unstale":"2024-06-09T22:17:26.772465+0000","last_undegraded":"2024-06-09T22:17:26.772465+0000","last_fullsized":"2024-06-09T22:17:26.772465+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:50:36.617736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"97'5","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.445165+0000","last_change":"2024-06-09T22:16:40.036603+0000","last_active":"2024-06-09T22:17:50.445165+0000","last_peered":"2024-06-09T22:17:50.445165+0000","last_clean":"2024-06-09T22:17:50.445165+0000","last_became_active":"2024-06-09T22:16:40.036274+0000","last_became_peered":"2024-06-09T22:16:40.036274+0000","last_unstale":"2024-06-09T22:17:50.445165+0000","last_undegraded":"2024-06-09T22:17:50.445165+0000","last_fullsized":"2024-06-09T22:17:50.445165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:31.652897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473418+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:17:26.473418+0000","last_peered":"2024-06-09T22:17:26.473418+0000","last_clean":"2024-06-09T22:17:26.473418+0000","last_became_active":"2024-06-09T22:16:43.204197+0000","last_became_peered":"2024-06-09T22:16:43.204197+0000","last_unstale":"2024-06-09T22:17:26.473418+0000","last_undegraded":"2024-06-09T22:17:26.473418+0000","last_fullsized":"2024-06-09T22:17:26.473418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:38:49.850120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468056+0000","last_change":"2024-06-09T22:16:43.208477+0000","last_active":"2024-06-09T22:17:26.468056+0000","last_peered":"2024-06-09T22:17:26.468056+0000","last_clean":"2024-06-09T22:17:26.468056+0000","last_became_active":"2024-06-09T22:16:43.206018+0000","last_became_peered":"2024-06-09T22:16:43.206018+0000","last_unstale":"2024-06-09T22:17:26.468056+0000","last_undegraded":"2024-06-09T22:17:26.468056+0000","last_fullsized":"2024-06-09T22:17:26.468056+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:19:49.870825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469938+0000","last_change":"2024-06-09T22:16:43.206809+0000","last_active":"2024-06-09T22:17:26.469938+0000","last_peered":"2024-06-09T22:17:26.469938+0000","last_clean":"2024-06-09T22:17:26.469938+0000","last_became_active":"2024-06-09T22:16:43.206175+0000","last_became_peered":"2024-06-09T22:16:43.206175+0000","last_unstale":"2024-06-09T22:17:26.469938+0000","last_undegraded":"2024-06-09T22:17:26.469938+0000","last_fullsized":"2024-06-09T22:17:26.469938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:39:22.502266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469427+0000","last_change":"2024-06-09T22:16:43.202163+0000","last_active":"2024-06-09T22:17:26.469427+0000","last_peered":"2024-06-09T22:17:26.469427+0000","last_clean":"2024-06-09T22:17:26.469427+0000","last_became_active":"2024-06-09T22:16:43.201839+0000","last_became_peered":"2024-06-09T22:16:43.201839+0000","last_unstale":"2024-06-09T22:17:26.469427+0000","last_undegraded":"2024-06-09T22:17:26.469427+0000","last_fullsized":"2024-06-09T22:17:26.469427+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:47:46.723289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468006+0000","last_change":"2024-06-09T22:16:43.202734+0000","last_active":"2024-06-09T22:17:26.468006+0000","last_peered":"2024-06-09T22:17:26.468006+0000","last_clean":"2024-06-09T22:17:26.468006+0000","last_became_active":"2024-06-09T22:16:43.201638+0000","last_became_peered":"2024-06-09T22:16:43.201638+0000","last_unstale":"2024-06-09T22:17:26.468006+0000","last_undegraded":"2024-06-09T22:17:26.468006+0000","last_fullsized":"2024-06-09T22:17:26.468006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:16.360985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473784+0000","last_change":"2024-06-09T22:16:43.199113+0000","last_active":"2024-06-09T22:17:26.473784+0000","last_peered":"2024-06-09T22:17:26.473784+0000","last_clean":"2024-06-09T22:17:26.473784+0000","last_became_active":"2024-06-09T22:16:43.198804+0000","last_became_peered":"2024-06-09T22:16:43.198804+0000","last_unstale":"2024-06-09T22:17:26.473784+0000","last_undegraded":"2024-06-09T22:17:26.473784+0000","last_fullsized":"2024-06-09T22:17:26.473784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:18:08.999477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469398+0000","last_change":"2024-06-09T22:16:43.204445+0000","last_active":"2024-06-09T22:17:26.469398+0000","last_peered":"2024-06-09T22:17:26.469398+0000","last_clean":"2024-06-09T22:17:26.469398+0000","last_became_active":"2024-06-09T22:16:43.203081+0000","last_became_peered":"2024-06-09T22:16:43.203081+0000","last_unstale":"2024-06-09T22:17:26.469398+0000","last_undegraded":"2024-06-09T22:17:26.469398+0000","last_fullsized":"2024-06-09T22:17:26.469398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:39.303813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470194+0000","last_change":"2024-06-09T22:16:43.211402+0000","last_active":"2024-06-09T22:17:26.470194+0000","last_peered":"2024-06-09T22:17:26.470194+0000","last_clean":"2024-06-09T22:17:26.470194+0000","last_became_active":"2024-06-09T22:16:43.210871+0000","last_became_peered":"2024-06-09T22:16:43.210871+0000","last_unstale":"2024-06-09T22:17:26.470194+0000","last_undegraded":"2024-06-09T22:17:26.470194+0000","last_fullsized":"2024-06-09T22:17:26.470194+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:07:21.467617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470506+0000","last_change":"2024-06-09T22:16:43.201159+0000","last_active":"2024-06-09T22:17:26.470506+0000","last_peered":"2024-06-09T22:17:26.470506+0000","last_clean":"2024-06-09T22:17:26.470506+0000","last_became_active":"2024-06-09T22:16:43.200850+0000","last_became_peered":"2024-06-09T22:16:43.200850+0000","last_unstale":"2024-06-09T22:17:26.470506+0000","last_undegraded":"2024-06-09T22:17:26.470506+0000","last_fullsized":"2024-06-09T22:17:26.470506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:20.515309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772774+0000","last_change":"2024-06-09T22:16:43.196830+0000","last_active":"2024-06-09T22:17:26.772774+0000","last_peered":"2024-06-09T22:17:26.772774+0000","last_clean":"2024-06-09T22:17:26.772774+0000","last_became_active":"2024-06-09T22:16:43.195682+0000","last_became_peered":"2024-06-09T22:16:43.195682+0000","last_unstale":"2024-06-09T22:17:26.772774+0000","last_undegraded":"2024-06-09T22:17:26.772774+0000","last_fullsized":"2024-06-09T22:17:26.772774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:34:18.770235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470144+0000","last_change":"2024-06-09T22:16:43.201959+0000","last_active":"2024-06-09T22:17:26.470144+0000","last_peered":"2024-06-09T22:17:26.470144+0000","last_clean":"2024-06-09T22:17:26.470144+0000","last_became_active":"2024-06-09T22:16:43.201669+0000","last_became_peered":"2024-06-09T22:16:43.201669+0000","last_unstale":"2024-06-09T22:17:26.470144+0000","last_undegraded":"2024-06-09T22:17:26.470144+0000","last_fullsized":"2024-06-09T22:17:26.470144+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:48:57.348041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772524+0000","last_change":"2024-06-09T22:16:43.201823+0000","last_active":"2024-06-09T22:17:26.772524+0000","last_peered":"2024-06-09T22:17:26.772524+0000","last_clean":"2024-06-09T22:17:26.772524+0000","last_became_active":"2024-06-09T22:16:43.201434+0000","last_became_peered":"2024-06-09T22:16:43.201434+0000","last_unstale":"2024-06-09T22:17:26.772524+0000","last_undegraded":"2024-06-09T22:17:26.772524+0000","last_fullsized":"2024-06-09T22:17:26.772524+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:56:13.540895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473264+0000","last_change":"2024-06-09T22:16:43.197734+0000","last_active":"2024-06-09T22:17:26.473264+0000","last_peered":"2024-06-09T22:17:26.473264+0000","last_clean":"2024-06-09T22:17:26.473264+0000","last_became_active":"2024-06-09T22:16:43.197442+0000","last_became_peered":"2024-06-09T22:16:43.197442+0000","last_unstale":"2024-06-09T22:17:26.473264+0000","last_undegraded":"2024-06-09T22:17:26.473264+0000","last_fullsized":"2024-06-09T22:17:26.473264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:35:40.720718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470366+0000","last_change":"2024-06-09T22:16:43.210375+0000","last_active":"2024-06-09T22:17:26.470366+0000","last_peered":"2024-06-09T22:17:26.470366+0000","last_clean":"2024-06-09T22:17:26.470366+0000","last_became_active":"2024-06-09T22:16:43.209647+0000","last_became_peered":"2024-06-09T22:16:43.209647+0000","last_unstale":"2024-06-09T22:17:26.470366+0000","last_undegraded":"2024-06-09T22:17:26.470366+0000","last_fullsized":"2024-06-09T22:17:26.470366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:11:57.593066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247891,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64264,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":272,"num_read_kb":207,"num_write":245,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":201510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":552059,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645351,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32368,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712784,"statfs":{"total":95995035648,"available":95961890816,"internally_reserved":0,"allocated":1220608,"data_stored":1195479,"data_compressed":62260,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":555627,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":551909,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1150976,"data_stored":1148795,"data_compressed":62260,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1077248,"data_stored":547035,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36456,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708696,"statfs":{"total":95995035648,"available":95957704704,"internally_reserved":0,"allocated":1146880,"data_stored":613455,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1101824,"data_stored":548881,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1150976,"data_stored":605669,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":546743,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1085440,"data_stored":551499,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1159168,"data_stored":1143987,"data_compressed":62260,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34243,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34243,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":59356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":58798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-09T22:18:18.255 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-09T22:18:18.499 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:18.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:18 smithi104 ceph-mon[32389]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-06-09T22:18:18.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:18 smithi104 ceph-mon[32389]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:18:18.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:18 smithi194 ceph-mon[36856]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-06-09T22:18:18.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:18 smithi194 ceph-mon[36856]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:18:18.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:18 smithi153 ceph-mon[36599]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-06-09T22:18:18.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:18 smithi153 ceph-mon[36599]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:18:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.288+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/754972142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8108650 msgr2=0x7fc8f8108a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.288+0000 7fc8fd7b6700 1 --2- 172.21.15.104:0/754972142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8108650 0x7fc8f8108a30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0009990 tx=0x7fc8e0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:18:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.288+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/754972142 shutdown_connections 2024-06-09T22:18:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.288+0000 7fc8fd7b6700 1 --2- 172.21.15.104:0/754972142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8108650 0x7fc8f8108a30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.288+0000 7fc8fd7b6700 1 --2- 172.21.15.104:0/754972142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f8103170 0x7fc8f810f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.288+0000 7fc8fd7b6700 1 --2- 172.21.15.104:0/754972142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8f81027b0 0x7fc8f8102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.288+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/754972142 >> 172.21.15.104:0/754972142 conn(0x7fc8f807a420 msgr2=0x7fc8f807a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.288+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/754972142 shutdown_connections 2024-06-09T22:18:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.288+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/754972142 wait complete. 2024-06-09T22:18:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.289+0000 7fc8fd7b6700 1 Processor -- start 2024-06-09T22:18:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.289+0000 7fc8fd7b6700 1 -- start start 2024-06-09T22:18:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.289+0000 7fc8fd7b6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f81027b0 0x7fc8f8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.289+0000 7fc8fd7b6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8f8103170 0x7fc8f8074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.289+0000 7fc8fd7b6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8108650 0x7fc8f8075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.289+0000 7fc8fd7b6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f81128c0 con 0x7fc8f8108650 2024-06-09T22:18:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.289+0000 7fc8fd7b6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f8112740 con 0x7fc8f81027b0 2024-06-09T22:18:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.289+0000 7fc8fd7b6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f8112a40 con 0x7fc8f8103170 2024-06-09T22:18:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8f77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8108650 0x7fc8f8075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8f77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8108650 0x7fc8f8075210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60428/0 (socket says 172.21.15.104:60428) 2024-06-09T22:18:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8f77fe700 1 -- 172.21.15.104:0/676343807 learned_addr learned my addr 172.21.15.104:0/676343807 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8f6ffd700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f81027b0 0x7fc8f8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8f67fc700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8f8103170 0x7fc8f8074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8f77fe700 1 -- 172.21.15.104:0/676343807 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8f8103170 msgr2=0x7fc8f8074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8f77fe700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8f8103170 0x7fc8f8074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8f77fe700 1 -- 172.21.15.104:0/676343807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f81027b0 msgr2=0x7fc8f8074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8f77fe700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f81027b0 0x7fc8f8074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8f77fe700 1 -- 172.21.15.104:0/676343807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8f81a7290 con 0x7fc8f8108650 2024-06-09T22:18:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8f77fe700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8108650 0x7fc8f8075210 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0007f10 tx=0x7fc8f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8effff700 1 -- 172.21.15.104:0/676343807 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f000ddd0 con 0x7fc8f8108650 2024-06-09T22:18:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/676343807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8f81a7580 con 0x7fc8f8108650 2024-06-09T22:18:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8effff700 1 -- 172.21.15.104:0/676343807 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8f0004020 con 0x7fc8f8108650 2024-06-09T22:18:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8effff700 1 -- 172.21.15.104:0/676343807 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0015660 con 0x7fc8f8108650 2024-06-09T22:18:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.290+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/676343807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8f81a7a70 con 0x7fc8f8108650 2024-06-09T22:18:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.291+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/676343807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8f810aaa0 con 0x7fc8f8108650 2024-06-09T22:18:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.292+0000 7fc8effff700 1 -- 172.21.15.104:0/676343807 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fc8f00041d0 con 0x7fc8f8108650 2024-06-09T22:18:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.293+0000 7fc8effff700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc8e40a2cb0 0x7fc8e40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.293+0000 7fc8f6ffd700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc8e40a2cb0 0x7fc8e40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.293+0000 7fc8effff700 1 -- 172.21.15.104:0/676343807 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7fc8f00cf6a0 con 0x7fc8f8108650 2024-06-09T22:18:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.293+0000 7fc8f6ffd700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc8e40a2cb0 0x7fc8e40a5170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000b5c0 tx=0x7fc8e0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.297+0000 7fc8effff700 1 -- 172.21.15.104:0/676343807 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc8f0010070 con 0x7fc8f8108650 2024-06-09T22:18:20.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.490+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/676343807 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc8f80628d0 con 0x7fc8e40a2cb0 2024-06-09T22:18:20.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.500+0000 7fc8effff700 1 -- 172.21.15.104:0/676343807 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340664 (secure 0 0 0) 0x7fc8f80628d0 con 0x7fc8e40a2cb0 2024-06-09T22:18:20.501 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:18:20.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.503+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/676343807 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc8e40a2cb0 msgr2=0x7fc8e40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:20.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.503+0000 7fc8fd7b6700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc8e40a2cb0 0x7fc8e40a5170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000b5c0 tx=0x7fc8e0005960 comp rx=0 tx=0).stop 2024-06-09T22:18:20.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.503+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/676343807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8108650 msgr2=0x7fc8f8075210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:20.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.503+0000 7fc8fd7b6700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8108650 0x7fc8f8075210 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0007f10 tx=0x7fc8f000d3b0 comp rx=0 tx=0).stop 2024-06-09T22:18:20.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.503+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/676343807 shutdown_connections 2024-06-09T22:18:20.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.503+0000 7fc8fd7b6700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc8e40a2cb0 0x7fc8e40a5170 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:20.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.503+0000 7fc8fd7b6700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8108650 0x7fc8f8075210 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:20.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.503+0000 7fc8fd7b6700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f81027b0 0x7fc8f8074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:20.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.503+0000 7fc8fd7b6700 1 --2- 172.21.15.104:0/676343807 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8f8103170 0x7fc8f8074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:20.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.503+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/676343807 >> 172.21.15.104:0/676343807 conn(0x7fc8f807a420 msgr2=0x7fc8f8101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:20.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.503+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/676343807 shutdown_connections 2024-06-09T22:18:20.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:20.503+0000 7fc8fd7b6700 1 -- 172.21.15.104:0/676343807 wait complete. 2024-06-09T22:18:20.504 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-06-09T22:18:20.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:20 smithi104 ceph-mon[32389]: pgmap v354: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-06-09T22:18:20.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:20 smithi194 ceph-mon[36856]: pgmap v354: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-06-09T22:18:20.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:20 smithi153 ceph-mon[36599]: pgmap v354: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-06-09T22:18:21.234 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-06-09T22:18:19.331213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663584,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":434,"num_read_kb":350,"num_write":433,"num_write_kb":3086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":355,"ondisk_log_size":355,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378964,"kb_used_data":13172,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562860,"statfs":{"total":1151940427776,"available":1151552368640,"internally_reserved":0,"allocated":13488128,"data_stored":8578574,"data_compressed":424191,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36302,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":128,"num_read_kb":102,"num_write":22,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003304"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470110+0000","last_change":"2024-06-09T22:16:43.212998+0000","last_active":"2024-06-09T22:17:26.470110+0000","last_peered":"2024-06-09T22:17:26.470110+0000","last_clean":"2024-06-09T22:17:26.470110+0000","last_became_active":"2024-06-09T22:16:43.212614+0000","last_became_peered":"2024-06-09T22:16:43.212614+0000","last_unstale":"2024-06-09T22:17:26.470110+0000","last_undegraded":"2024-06-09T22:17:26.470110+0000","last_fullsized":"2024-06-09T22:17:26.470110+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:22.728486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467914+0000","last_change":"2024-06-09T22:16:43.208746+0000","last_active":"2024-06-09T22:17:26.467914+0000","last_peered":"2024-06-09T22:17:26.467914+0000","last_clean":"2024-06-09T22:17:26.467914+0000","last_became_active":"2024-06-09T22:16:43.208458+0000","last_became_peered":"2024-06-09T22:16:43.208458+0000","last_unstale":"2024-06-09T22:17:26.467914+0000","last_undegraded":"2024-06-09T22:17:26.467914+0000","last_fullsized":"2024-06-09T22:17:26.467914+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:24:35.350449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467803+0000","last_change":"2024-06-09T22:16:43.200770+0000","last_active":"2024-06-09T22:17:26.467803+0000","last_peered":"2024-06-09T22:17:26.467803+0000","last_clean":"2024-06-09T22:17:26.467803+0000","last_became_active":"2024-06-09T22:16:43.200449+0000","last_became_peered":"2024-06-09T22:16:43.200449+0000","last_unstale":"2024-06-09T22:17:26.467803+0000","last_undegraded":"2024-06-09T22:17:26.467803+0000","last_fullsized":"2024-06-09T22:17:26.467803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:51:09.017260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696787+0000","last_change":"2024-06-09T22:16:43.203844+0000","last_active":"2024-06-09T22:17:26.696787+0000","last_peered":"2024-06-09T22:17:26.696787+0000","last_clean":"2024-06-09T22:17:26.696787+0000","last_became_active":"2024-06-09T22:16:43.203431+0000","last_became_peered":"2024-06-09T22:16:43.203431+0000","last_unstale":"2024-06-09T22:17:26.696787+0000","last_undegraded":"2024-06-09T22:17:26.696787+0000","last_fullsized":"2024-06-09T22:17:26.696787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:51:36.913312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468202+0000","last_change":"2024-06-09T22:16:43.207597+0000","last_active":"2024-06-09T22:17:26.468202+0000","last_peered":"2024-06-09T22:17:26.468202+0000","last_clean":"2024-06-09T22:17:26.468202+0000","last_became_active":"2024-06-09T22:16:43.207266+0000","last_became_peered":"2024-06-09T22:16:43.207266+0000","last_unstale":"2024-06-09T22:17:26.468202+0000","last_undegraded":"2024-06-09T22:17:26.468202+0000","last_fullsized":"2024-06-09T22:17:26.468202+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:47:05.558403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.474121+0000","last_change":"2024-06-09T22:17:26.474121+0000","last_active":"2024-06-09T22:17:26.474121+0000","last_peered":"2024-06-09T22:17:26.474121+0000","last_clean":"2024-06-09T22:17:26.474121+0000","last_became_active":"2024-06-09T22:17:26.473619+0000","last_became_peered":"2024-06-09T22:17:26.473619+0000","last_unstale":"2024-06-09T22:17:26.474121+0000","last_undegraded":"2024-06-09T22:17:26.474121+0000","last_fullsized":"2024-06-09T22:17:26.474121+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:56:42.171235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.472272+0000","last_change":"2024-06-09T22:16:43.208815+0000","last_active":"2024-06-09T22:17:26.472272+0000","last_peered":"2024-06-09T22:17:26.472272+0000","last_clean":"2024-06-09T22:17:26.472272+0000","last_became_active":"2024-06-09T22:16:43.208433+0000","last_became_peered":"2024-06-09T22:16:43.208433+0000","last_unstale":"2024-06-09T22:17:26.472272+0000","last_undegraded":"2024-06-09T22:17:26.472272+0000","last_fullsized":"2024-06-09T22:17:26.472272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:36:26.585037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470482+0000","last_change":"2024-06-09T22:16:43.210195+0000","last_active":"2024-06-09T22:17:26.470482+0000","last_peered":"2024-06-09T22:17:26.470482+0000","last_clean":"2024-06-09T22:17:26.470482+0000","last_became_active":"2024-06-09T22:16:43.209892+0000","last_became_peered":"2024-06-09T22:16:43.209892+0000","last_unstale":"2024-06-09T22:17:26.470482+0000","last_undegraded":"2024-06-09T22:17:26.470482+0000","last_fullsized":"2024-06-09T22:17:26.470482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:44.088212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772888+0000","last_change":"2024-06-09T22:16:43.202517+0000","last_active":"2024-06-09T22:17:26.772888+0000","last_peered":"2024-06-09T22:17:26.772888+0000","last_clean":"2024-06-09T22:17:26.772888+0000","last_became_active":"2024-06-09T22:16:43.201796+0000","last_became_peered":"2024-06-09T22:16:43.201796+0000","last_unstale":"2024-06-09T22:17:26.772888+0000","last_undegraded":"2024-06-09T22:17:26.772888+0000","last_fullsized":"2024-06-09T22:17:26.772888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:30.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470410+0000","last_change":"2024-06-09T22:16:43.212732+0000","last_active":"2024-06-09T22:17:26.470410+0000","last_peered":"2024-06-09T22:17:26.470410+0000","last_clean":"2024-06-09T22:17:26.470410+0000","last_became_active":"2024-06-09T22:16:43.212048+0000","last_became_peered":"2024-06-09T22:16:43.212048+0000","last_unstale":"2024-06-09T22:17:26.470410+0000","last_undegraded":"2024-06-09T22:17:26.470410+0000","last_fullsized":"2024-06-09T22:17:26.470410+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:47.383291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473150+0000","last_change":"2024-06-09T22:16:43.203778+0000","last_active":"2024-06-09T22:17:26.473150+0000","last_peered":"2024-06-09T22:17:26.473150+0000","last_clean":"2024-06-09T22:17:26.473150+0000","last_became_active":"2024-06-09T22:16:43.202954+0000","last_became_peered":"2024-06-09T22:16:43.202954+0000","last_unstale":"2024-06-09T22:17:26.473150+0000","last_undegraded":"2024-06-09T22:17:26.473150+0000","last_fullsized":"2024-06-09T22:17:26.473150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:56:53.708645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696798+0000","last_change":"2024-06-09T22:16:43.211425+0000","last_active":"2024-06-09T22:17:26.696798+0000","last_peered":"2024-06-09T22:17:26.696798+0000","last_clean":"2024-06-09T22:17:26.696798+0000","last_became_active":"2024-06-09T22:16:43.211001+0000","last_became_peered":"2024-06-09T22:16:43.211001+0000","last_unstale":"2024-06-09T22:17:26.696798+0000","last_undegraded":"2024-06-09T22:17:26.696798+0000","last_fullsized":"2024-06-09T22:17:26.696798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:20:59.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473445+0000","last_change":"2024-06-09T22:16:43.208334+0000","last_active":"2024-06-09T22:17:26.473445+0000","last_peered":"2024-06-09T22:17:26.473445+0000","last_clean":"2024-06-09T22:17:26.473445+0000","last_became_active":"2024-06-09T22:16:43.207993+0000","last_became_peered":"2024-06-09T22:16:43.207993+0000","last_unstale":"2024-06-09T22:17:26.473445+0000","last_undegraded":"2024-06-09T22:17:26.473445+0000","last_fullsized":"2024-06-09T22:17:26.473445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:07:05.740748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467715+0000","last_change":"2024-06-09T22:16:43.207211+0000","last_active":"2024-06-09T22:17:26.467715+0000","last_peered":"2024-06-09T22:17:26.467715+0000","last_clean":"2024-06-09T22:17:26.467715+0000","last_became_active":"2024-06-09T22:16:43.206889+0000","last_became_peered":"2024-06-09T22:16:43.206889+0000","last_unstale":"2024-06-09T22:17:26.467715+0000","last_undegraded":"2024-06-09T22:17:26.467715+0000","last_fullsized":"2024-06-09T22:17:26.467715+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:25:47.461030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467752+0000","last_change":"2024-06-09T22:16:43.202202+0000","last_active":"2024-06-09T22:17:26.467752+0000","last_peered":"2024-06-09T22:17:26.467752+0000","last_clean":"2024-06-09T22:17:26.467752+0000","last_became_active":"2024-06-09T22:16:43.201834+0000","last_became_peered":"2024-06-09T22:16:43.201834+0000","last_unstale":"2024-06-09T22:17:26.467752+0000","last_undegraded":"2024-06-09T22:17:26.467752+0000","last_fullsized":"2024-06-09T22:17:26.467752+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:31:51.296627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772170+0000","last_change":"2024-06-09T22:16:43.200637+0000","last_active":"2024-06-09T22:17:26.772170+0000","last_peered":"2024-06-09T22:17:26.772170+0000","last_clean":"2024-06-09T22:17:26.772170+0000","last_became_active":"2024-06-09T22:16:43.200268+0000","last_became_peered":"2024-06-09T22:16:43.200268+0000","last_unstale":"2024-06-09T22:17:26.772170+0000","last_undegraded":"2024-06-09T22:17:26.772170+0000","last_fullsized":"2024-06-09T22:17:26.772170+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:25:59.881453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469876+0000","last_change":"2024-06-09T22:16:43.209212+0000","last_active":"2024-06-09T22:17:26.469876+0000","last_peered":"2024-06-09T22:17:26.469876+0000","last_clean":"2024-06-09T22:17:26.469876+0000","last_became_active":"2024-06-09T22:16:43.207789+0000","last_became_peered":"2024-06-09T22:16:43.207789+0000","last_unstale":"2024-06-09T22:17:26.469876+0000","last_undegraded":"2024-06-09T22:17:26.469876+0000","last_fullsized":"2024-06-09T22:17:26.469876+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:17.316595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.465745+0000","last_change":"2024-06-09T22:16:43.210935+0000","last_active":"2024-06-09T22:17:26.465745+0000","last_peered":"2024-06-09T22:17:26.465745+0000","last_clean":"2024-06-09T22:17:26.465745+0000","last_became_active":"2024-06-09T22:16:43.210631+0000","last_became_peered":"2024-06-09T22:16:43.210631+0000","last_unstale":"2024-06-09T22:17:26.465745+0000","last_undegraded":"2024-06-09T22:17:26.465745+0000","last_fullsized":"2024-06-09T22:17:26.465745+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:22:58.887313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470400+0000","last_change":"2024-06-09T22:16:43.198872+0000","last_active":"2024-06-09T22:17:26.470400+0000","last_peered":"2024-06-09T22:17:26.470400+0000","last_clean":"2024-06-09T22:17:26.470400+0000","last_became_active":"2024-06-09T22:16:43.198498+0000","last_became_peered":"2024-06-09T22:16:43.198498+0000","last_unstale":"2024-06-09T22:17:26.470400+0000","last_undegraded":"2024-06-09T22:17:26.470400+0000","last_fullsized":"2024-06-09T22:17:26.470400+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:25:28.853048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470043+0000","last_change":"2024-06-09T22:16:53.343656+0000","last_active":"2024-06-09T22:17:26.470043+0000","last_peered":"2024-06-09T22:17:26.470043+0000","last_clean":"2024-06-09T22:17:26.470043+0000","last_became_active":"2024-06-09T22:16:53.343281+0000","last_became_peered":"2024-06-09T22:16:53.343281+0000","last_unstale":"2024-06-09T22:17:26.470043+0000","last_undegraded":"2024-06-09T22:17:26.470043+0000","last_fullsized":"2024-06-09T22:17:26.470043+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:40.961842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"97'13","reported_seq":77,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:14.482483+0000","last_change":"2024-06-09T22:16:40.040145+0000","last_active":"2024-06-09T22:18:14.482483+0000","last_peered":"2024-06-09T22:18:14.482483+0000","last_clean":"2024-06-09T22:18:14.482483+0000","last_became_active":"2024-06-09T22:16:40.039787+0000","last_became_peered":"2024-06-09T22:16:40.039787+0000","last_unstale":"2024-06-09T22:18:14.482483+0000","last_undegraded":"2024-06-09T22:18:14.482483+0000","last_fullsized":"2024-06-09T22:18:14.482483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:52:32.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.774053+0000","last_change":"2024-06-09T22:17:26.774053+0000","last_active":"2024-06-09T22:17:26.774053+0000","last_peered":"2024-06-09T22:17:26.774053+0000","last_clean":"2024-06-09T22:17:26.774053+0000","last_became_active":"2024-06-09T22:17:26.773786+0000","last_became_peered":"2024-06-09T22:17:26.773786+0000","last_unstale":"2024-06-09T22:17:26.774053+0000","last_undegraded":"2024-06-09T22:17:26.774053+0000","last_fullsized":"2024-06-09T22:17:26.774053+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:36.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696042+0000","last_change":"2024-06-09T22:16:53.332048+0000","last_active":"2024-06-09T22:17:26.696042+0000","last_peered":"2024-06-09T22:17:26.696042+0000","last_clean":"2024-06-09T22:17:26.696042+0000","last_became_active":"2024-06-09T22:16:53.330683+0000","last_became_peered":"2024-06-09T22:16:53.330683+0000","last_unstale":"2024-06-09T22:17:26.696042+0000","last_undegraded":"2024-06-09T22:17:26.696042+0000","last_fullsized":"2024-06-09T22:17:26.696042+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:32:23.219061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"97'2","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470379+0000","last_change":"2024-06-09T22:16:40.070193+0000","last_active":"2024-06-09T22:17:26.470379+0000","last_peered":"2024-06-09T22:17:26.470379+0000","last_clean":"2024-06-09T22:17:26.470379+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:17:26.470379+0000","last_undegraded":"2024-06-09T22:17:26.470379+0000","last_fullsized":"2024-06-09T22:17:26.470379+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:19:19.787552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467804+0000","last_change":"2024-06-09T22:16:43.199814+0000","last_active":"2024-06-09T22:17:26.467804+0000","last_peered":"2024-06-09T22:17:26.467804+0000","last_clean":"2024-06-09T22:17:26.467804+0000","last_became_active":"2024-06-09T22:16:43.199487+0000","last_became_peered":"2024-06-09T22:16:43.199487+0000","last_unstale":"2024-06-09T22:17:26.467804+0000","last_undegraded":"2024-06-09T22:17:26.467804+0000","last_fullsized":"2024-06-09T22:17:26.467804+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:55.672050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696360+0000","last_change":"2024-06-09T22:16:53.336253+0000","last_active":"2024-06-09T22:17:26.696360+0000","last_peered":"2024-06-09T22:17:26.696360+0000","last_clean":"2024-06-09T22:17:26.696360+0000","last_became_active":"2024-06-09T22:16:53.335869+0000","last_became_peered":"2024-06-09T22:16:53.335869+0000","last_unstale":"2024-06-09T22:17:26.696360+0000","last_undegraded":"2024-06-09T22:17:26.696360+0000","last_fullsized":"2024-06-09T22:17:26.696360+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:28:12.894878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"97'12","reported_seq":53,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:51.448960+0000","last_change":"2024-06-09T22:16:40.045365+0000","last_active":"2024-06-09T22:17:51.448960+0000","last_peered":"2024-06-09T22:17:51.448960+0000","last_clean":"2024-06-09T22:17:51.448960+0000","last_became_active":"2024-06-09T22:16:40.044989+0000","last_became_peered":"2024-06-09T22:16:40.044989+0000","last_unstale":"2024-06-09T22:17:51.448960+0000","last_undegraded":"2024-06-09T22:17:51.448960+0000","last_fullsized":"2024-06-09T22:17:51.448960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:01:05.850170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772394+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:17:26.772394+0000","last_peered":"2024-06-09T22:17:26.772394+0000","last_clean":"2024-06-09T22:17:26.772394+0000","last_became_active":"2024-06-09T22:16:43.203999+0000","last_became_peered":"2024-06-09T22:16:43.203999+0000","last_unstale":"2024-06-09T22:17:26.772394+0000","last_undegraded":"2024-06-09T22:17:26.772394+0000","last_fullsized":"2024-06-09T22:17:26.772394+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:37:28.362655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696514+0000","last_change":"2024-06-09T22:16:53.333565+0000","last_active":"2024-06-09T22:17:26.696514+0000","last_peered":"2024-06-09T22:17:26.696514+0000","last_clean":"2024-06-09T22:17:26.696514+0000","last_became_active":"2024-06-09T22:16:53.332921+0000","last_became_peered":"2024-06-09T22:16:53.332921+0000","last_unstale":"2024-06-09T22:17:26.696514+0000","last_undegraded":"2024-06-09T22:17:26.696514+0000","last_fullsized":"2024-06-09T22:17:26.696514+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:32.085617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"97'5","reported_seq":46,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.447829+0000","last_change":"2024-06-09T22:16:40.039766+0000","last_active":"2024-06-09T22:17:50.447829+0000","last_peered":"2024-06-09T22:17:50.447829+0000","last_clean":"2024-06-09T22:17:50.447829+0000","last_became_active":"2024-06-09T22:16:40.039329+0000","last_became_peered":"2024-06-09T22:16:40.039329+0000","last_unstale":"2024-06-09T22:17:50.447829+0000","last_undegraded":"2024-06-09T22:17:50.447829+0000","last_fullsized":"2024-06-09T22:17:50.447829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:53.673890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.773335+0000","last_change":"2024-06-09T22:16:43.201200+0000","last_active":"2024-06-09T22:17:26.773335+0000","last_peered":"2024-06-09T22:17:26.773335+0000","last_clean":"2024-06-09T22:17:26.773335+0000","last_became_active":"2024-06-09T22:16:43.200830+0000","last_became_peered":"2024-06-09T22:16:43.200830+0000","last_unstale":"2024-06-09T22:17:26.773335+0000","last_undegraded":"2024-06-09T22:17:26.773335+0000","last_fullsized":"2024-06-09T22:17:26.773335+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:38:52.093173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469698+0000","last_change":"2024-06-09T22:16:53.339811+0000","last_active":"2024-06-09T22:17:26.469698+0000","last_peered":"2024-06-09T22:17:26.469698+0000","last_clean":"2024-06-09T22:17:26.469698+0000","last_became_active":"2024-06-09T22:16:53.339305+0000","last_became_peered":"2024-06-09T22:16:53.339305+0000","last_unstale":"2024-06-09T22:17:26.469698+0000","last_undegraded":"2024-06-09T22:17:26.469698+0000","last_fullsized":"2024-06-09T22:17:26.469698+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:08:25.800245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"97'6","reported_seq":72,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:13.483336+0000","last_change":"2024-06-09T22:16:40.044159+0000","last_active":"2024-06-09T22:18:13.483336+0000","last_peered":"2024-06-09T22:18:13.483336+0000","last_clean":"2024-06-09T22:18:13.483336+0000","last_became_active":"2024-06-09T22:16:40.043839+0000","last_became_peered":"2024-06-09T22:16:40.043839+0000","last_unstale":"2024-06-09T22:18:13.483336+0000","last_undegraded":"2024-06-09T22:18:13.483336+0000","last_fullsized":"2024-06-09T22:18:13.483336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:22:09.339219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468655+0000","last_change":"2024-06-09T22:16:43.202015+0000","last_active":"2024-06-09T22:17:26.468655+0000","last_peered":"2024-06-09T22:17:26.468655+0000","last_clean":"2024-06-09T22:17:26.468655+0000","last_became_active":"2024-06-09T22:16:43.201689+0000","last_became_peered":"2024-06-09T22:16:43.201689+0000","last_unstale":"2024-06-09T22:17:26.468655+0000","last_undegraded":"2024-06-09T22:17:26.468655+0000","last_fullsized":"2024-06-09T22:17:26.468655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:35:12.180399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470051+0000","last_change":"2024-06-09T22:16:53.339000+0000","last_active":"2024-06-09T22:17:26.470051+0000","last_peered":"2024-06-09T22:17:26.470051+0000","last_clean":"2024-06-09T22:17:26.470051+0000","last_became_active":"2024-06-09T22:16:53.338556+0000","last_became_peered":"2024-06-09T22:16:53.338556+0000","last_unstale":"2024-06-09T22:17:26.470051+0000","last_undegraded":"2024-06-09T22:17:26.470051+0000","last_fullsized":"2024-06-09T22:17:26.470051+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:32:42.458481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"97'7","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:49.441383+0000","last_change":"2024-06-09T22:17:26.698204+0000","last_active":"2024-06-09T22:17:49.441383+0000","last_peered":"2024-06-09T22:17:49.441383+0000","last_clean":"2024-06-09T22:17:49.441383+0000","last_became_active":"2024-06-09T22:17:26.697716+0000","last_became_peered":"2024-06-09T22:17:26.697716+0000","last_unstale":"2024-06-09T22:17:49.441383+0000","last_undegraded":"2024-06-09T22:17:49.441383+0000","last_fullsized":"2024-06-09T22:17:49.441383+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:36:37.724459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.773176+0000","last_change":"2024-06-09T22:16:43.198917+0000","last_active":"2024-06-09T22:17:26.773176+0000","last_peered":"2024-06-09T22:17:26.773176+0000","last_clean":"2024-06-09T22:17:26.773176+0000","last_became_active":"2024-06-09T22:16:43.198567+0000","last_became_peered":"2024-06-09T22:16:43.198567+0000","last_unstale":"2024-06-09T22:17:26.773176+0000","last_undegraded":"2024-06-09T22:17:26.773176+0000","last_fullsized":"2024-06-09T22:17:26.773176+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:34.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469768+0000","last_change":"2024-06-09T22:16:53.342203+0000","last_active":"2024-06-09T22:17:26.469768+0000","last_peered":"2024-06-09T22:17:26.469768+0000","last_clean":"2024-06-09T22:17:26.469768+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:17:26.469768+0000","last_undegraded":"2024-06-09T22:17:26.469768+0000","last_fullsized":"2024-06-09T22:17:26.469768+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:41:41.491961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"97'3","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:48.428754+0000","last_change":"2024-06-09T22:16:40.038684+0000","last_active":"2024-06-09T22:17:48.428754+0000","last_peered":"2024-06-09T22:17:48.428754+0000","last_clean":"2024-06-09T22:17:48.428754+0000","last_became_active":"2024-06-09T22:16:40.038386+0000","last_became_peered":"2024-06-09T22:16:40.038386+0000","last_unstale":"2024-06-09T22:17:48.428754+0000","last_undegraded":"2024-06-09T22:17:48.428754+0000","last_fullsized":"2024-06-09T22:17:48.428754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:59.188758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470596+0000","last_change":"2024-06-09T22:16:43.198837+0000","last_active":"2024-06-09T22:17:26.470596+0000","last_peered":"2024-06-09T22:17:26.470596+0000","last_clean":"2024-06-09T22:17:26.470596+0000","last_became_active":"2024-06-09T22:16:43.198515+0000","last_became_peered":"2024-06-09T22:16:43.198515+0000","last_unstale":"2024-06-09T22:17:26.470596+0000","last_undegraded":"2024-06-09T22:17:26.470596+0000","last_fullsized":"2024-06-09T22:17:26.470596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:36:13.863202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467878+0000","last_change":"2024-06-09T22:16:53.342688+0000","last_active":"2024-06-09T22:17:26.467878+0000","last_peered":"2024-06-09T22:17:26.467878+0000","last_clean":"2024-06-09T22:17:26.467878+0000","last_became_active":"2024-06-09T22:16:53.342314+0000","last_became_peered":"2024-06-09T22:16:53.342314+0000","last_unstale":"2024-06-09T22:17:26.467878+0000","last_undegraded":"2024-06-09T22:17:26.467878+0000","last_fullsized":"2024-06-09T22:17:26.467878+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:51:35.801587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"97'3","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.447410+0000","last_change":"2024-06-09T22:16:40.041831+0000","last_active":"2024-06-09T22:17:50.447410+0000","last_peered":"2024-06-09T22:17:50.447410+0000","last_clean":"2024-06-09T22:17:50.447410+0000","last_became_active":"2024-06-09T22:16:40.041484+0000","last_became_peered":"2024-06-09T22:16:40.041484+0000","last_unstale":"2024-06-09T22:17:50.447410+0000","last_undegraded":"2024-06-09T22:17:50.447410+0000","last_fullsized":"2024-06-09T22:17:50.447410+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:10:19.829357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.465273+0000","last_change":"2024-06-09T22:16:43.210769+0000","last_active":"2024-06-09T22:17:26.465273+0000","last_peered":"2024-06-09T22:17:26.465273+0000","last_clean":"2024-06-09T22:17:26.465273+0000","last_became_active":"2024-06-09T22:16:43.210429+0000","last_became_peered":"2024-06-09T22:16:43.210429+0000","last_unstale":"2024-06-09T22:17:26.465273+0000","last_undegraded":"2024-06-09T22:17:26.465273+0000","last_fullsized":"2024-06-09T22:17:26.465273+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:12:28.788830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469973+0000","last_change":"2024-06-09T22:16:53.336536+0000","last_active":"2024-06-09T22:17:26.469973+0000","last_peered":"2024-06-09T22:17:26.469973+0000","last_clean":"2024-06-09T22:17:26.469973+0000","last_became_active":"2024-06-09T22:16:53.335178+0000","last_became_peered":"2024-06-09T22:16:53.335178+0000","last_unstale":"2024-06-09T22:17:26.469973+0000","last_undegraded":"2024-06-09T22:17:26.469973+0000","last_fullsized":"2024-06-09T22:17:26.469973+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:57:19.476483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"97'4","reported_seq":95,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:17.488126+0000","last_change":"2024-06-09T22:16:40.038484+0000","last_active":"2024-06-09T22:18:17.488126+0000","last_peered":"2024-06-09T22:18:17.488126+0000","last_clean":"2024-06-09T22:18:17.488126+0000","last_became_active":"2024-06-09T22:16:40.038084+0000","last_became_peered":"2024-06-09T22:16:40.038084+0000","last_unstale":"2024-06-09T22:18:17.488126+0000","last_undegraded":"2024-06-09T22:18:17.488126+0000","last_fullsized":"2024-06-09T22:18:17.488126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:19.733107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":54,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772859+0000","last_change":"2024-06-09T22:16:43.205137+0000","last_active":"2024-06-09T22:17:26.772859+0000","last_peered":"2024-06-09T22:17:26.772859+0000","last_clean":"2024-06-09T22:17:26.772859+0000","last_became_active":"2024-06-09T22:16:43.204499+0000","last_became_peered":"2024-06-09T22:16:43.204499+0000","last_unstale":"2024-06-09T22:17:26.772859+0000","last_undegraded":"2024-06-09T22:17:26.772859+0000","last_fullsized":"2024-06-09T22:17:26.772859+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:15.718493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.465220+0000","last_change":"2024-06-09T22:16:53.344260+0000","last_active":"2024-06-09T22:17:26.465220+0000","last_peered":"2024-06-09T22:17:26.465220+0000","last_clean":"2024-06-09T22:17:26.465220+0000","last_became_active":"2024-06-09T22:16:53.343840+0000","last_became_peered":"2024-06-09T22:16:53.343840+0000","last_unstale":"2024-06-09T22:17:26.465220+0000","last_undegraded":"2024-06-09T22:17:26.465220+0000","last_fullsized":"2024-06-09T22:17:26.465220+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:40:23.305509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"97'4","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.446862+0000","last_change":"2024-06-09T22:16:40.042660+0000","last_active":"2024-06-09T22:17:50.446862+0000","last_peered":"2024-06-09T22:17:50.446862+0000","last_clean":"2024-06-09T22:17:50.446862+0000","last_became_active":"2024-06-09T22:16:40.041353+0000","last_became_peered":"2024-06-09T22:16:40.041353+0000","last_unstale":"2024-06-09T22:17:50.446862+0000","last_undegraded":"2024-06-09T22:17:50.446862+0000","last_fullsized":"2024-06-09T22:17:50.446862+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:34:59.064827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468069+0000","last_change":"2024-06-09T22:16:43.205083+0000","last_active":"2024-06-09T22:17:26.468069+0000","last_peered":"2024-06-09T22:17:26.468069+0000","last_clean":"2024-06-09T22:17:26.468069+0000","last_became_active":"2024-06-09T22:16:43.204600+0000","last_became_peered":"2024-06-09T22:16:43.204600+0000","last_unstale":"2024-06-09T22:17:26.468069+0000","last_undegraded":"2024-06-09T22:17:26.468069+0000","last_fullsized":"2024-06-09T22:17:26.468069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:05:12.555465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696558+0000","last_change":"2024-06-09T22:16:53.336169+0000","last_active":"2024-06-09T22:17:26.696558+0000","last_peered":"2024-06-09T22:17:26.696558+0000","last_clean":"2024-06-09T22:17:26.696558+0000","last_became_active":"2024-06-09T22:16:53.335657+0000","last_became_peered":"2024-06-09T22:16:53.335657+0000","last_unstale":"2024-06-09T22:17:26.696558+0000","last_undegraded":"2024-06-09T22:17:26.696558+0000","last_fullsized":"2024-06-09T22:17:26.696558+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:53:21.282767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"97'5","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:49.440446+0000","last_change":"2024-06-09T22:16:40.070568+0000","last_active":"2024-06-09T22:17:49.440446+0000","last_peered":"2024-06-09T22:17:49.440446+0000","last_clean":"2024-06-09T22:17:49.440446+0000","last_became_active":"2024-06-09T22:16:40.070205+0000","last_became_peered":"2024-06-09T22:16:40.070205+0000","last_unstale":"2024-06-09T22:17:49.440446+0000","last_undegraded":"2024-06-09T22:17:49.440446+0000","last_fullsized":"2024-06-09T22:17:49.440446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.449965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772574+0000","last_change":"2024-06-09T22:16:43.205219+0000","last_active":"2024-06-09T22:17:26.772574+0000","last_peered":"2024-06-09T22:17:26.772574+0000","last_clean":"2024-06-09T22:17:26.772574+0000","last_became_active":"2024-06-09T22:16:43.203467+0000","last_became_peered":"2024-06-09T22:16:43.203467+0000","last_unstale":"2024-06-09T22:17:26.772574+0000","last_undegraded":"2024-06-09T22:17:26.772574+0000","last_fullsized":"2024-06-09T22:17:26.772574+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:22.784034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.465239+0000","last_change":"2024-06-09T22:16:53.338990+0000","last_active":"2024-06-09T22:17:26.465239+0000","last_peered":"2024-06-09T22:17:26.465239+0000","last_clean":"2024-06-09T22:17:26.465239+0000","last_became_active":"2024-06-09T22:16:53.338568+0000","last_became_peered":"2024-06-09T22:16:53.338568+0000","last_unstale":"2024-06-09T22:17:26.465239+0000","last_undegraded":"2024-06-09T22:17:26.465239+0000","last_fullsized":"2024-06-09T22:17:26.465239+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:46:39.814043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"97'36","reported_seq":135,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:17.488585+0000","last_change":"2024-06-09T22:16:40.044404+0000","last_active":"2024-06-09T22:18:17.488585+0000","last_peered":"2024-06-09T22:18:17.488585+0000","last_clean":"2024-06-09T22:18:17.488585+0000","last_became_active":"2024-06-09T22:16:40.044044+0000","last_became_peered":"2024-06-09T22:16:40.044044+0000","last_unstale":"2024-06-09T22:18:17.488585+0000","last_undegraded":"2024-06-09T22:18:17.488585+0000","last_fullsized":"2024-06-09T22:18:17.488585+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:50:27.038759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65390,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":62,"num_read_kb":95,"num_write":38,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469338+0000","last_change":"2024-06-09T22:16:43.206963+0000","last_active":"2024-06-09T22:17:26.469338+0000","last_peered":"2024-06-09T22:17:26.469338+0000","last_clean":"2024-06-09T22:17:26.469338+0000","last_became_active":"2024-06-09T22:16:43.206628+0000","last_became_peered":"2024-06-09T22:16:43.206628+0000","last_unstale":"2024-06-09T22:17:26.469338+0000","last_undegraded":"2024-06-09T22:17:26.469338+0000","last_fullsized":"2024-06-09T22:17:26.469338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:33:39.912236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.773641+0000","last_change":"2024-06-09T22:16:53.339332+0000","last_active":"2024-06-09T22:17:26.773641+0000","last_peered":"2024-06-09T22:17:26.773641+0000","last_clean":"2024-06-09T22:17:26.773641+0000","last_became_active":"2024-06-09T22:16:53.338819+0000","last_became_peered":"2024-06-09T22:16:53.338819+0000","last_unstale":"2024-06-09T22:17:26.773641+0000","last_undegraded":"2024-06-09T22:17:26.773641+0000","last_fullsized":"2024-06-09T22:17:26.773641+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:57:41.142142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"97'9","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:51.449071+0000","last_change":"2024-06-09T22:17:26.478256+0000","last_active":"2024-06-09T22:17:51.449071+0000","last_peered":"2024-06-09T22:17:51.449071+0000","last_clean":"2024-06-09T22:17:51.449071+0000","last_became_active":"2024-06-09T22:17:26.472264+0000","last_became_peered":"2024-06-09T22:17:26.472264+0000","last_unstale":"2024-06-09T22:17:51.449071+0000","last_undegraded":"2024-06-09T22:17:51.449071+0000","last_fullsized":"2024-06-09T22:17:51.449071+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:14:19.919841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473163+0000","last_change":"2024-06-09T22:16:43.204484+0000","last_active":"2024-06-09T22:17:26.473163+0000","last_peered":"2024-06-09T22:17:26.473163+0000","last_clean":"2024-06-09T22:17:26.473163+0000","last_became_active":"2024-06-09T22:16:43.204014+0000","last_became_peered":"2024-06-09T22:16:43.204014+0000","last_unstale":"2024-06-09T22:17:26.473163+0000","last_undegraded":"2024-06-09T22:17:26.473163+0000","last_fullsized":"2024-06-09T22:17:26.473163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:55:12.528231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.475179+0000","last_change":"2024-06-09T22:17:26.475179+0000","last_active":"2024-06-09T22:17:26.475179+0000","last_peered":"2024-06-09T22:17:26.475179+0000","last_clean":"2024-06-09T22:17:26.475179+0000","last_became_active":"2024-06-09T22:17:26.474517+0000","last_became_peered":"2024-06-09T22:17:26.474517+0000","last_unstale":"2024-06-09T22:17:26.475179+0000","last_undegraded":"2024-06-09T22:17:26.475179+0000","last_fullsized":"2024-06-09T22:17:26.475179+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:26:02.387830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"97'5","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.447706+0000","last_change":"2024-06-09T22:16:40.043066+0000","last_active":"2024-06-09T22:17:50.447706+0000","last_peered":"2024-06-09T22:17:50.447706+0000","last_clean":"2024-06-09T22:17:50.447706+0000","last_became_active":"2024-06-09T22:16:40.042619+0000","last_became_peered":"2024-06-09T22:16:40.042619+0000","last_unstale":"2024-06-09T22:17:50.447706+0000","last_undegraded":"2024-06-09T22:17:50.447706+0000","last_fullsized":"2024-06-09T22:17:50.447706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:29.689614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.475566+0000","last_change":"2024-06-09T22:17:26.475566+0000","last_active":"2024-06-09T22:17:26.475566+0000","last_peered":"2024-06-09T22:17:26.475566+0000","last_clean":"2024-06-09T22:17:26.475566+0000","last_became_active":"2024-06-09T22:17:26.475037+0000","last_became_peered":"2024-06-09T22:17:26.475037+0000","last_unstale":"2024-06-09T22:17:26.475566+0000","last_undegraded":"2024-06-09T22:17:26.475566+0000","last_fullsized":"2024-06-09T22:17:26.475566+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:33.075144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.475111+0000","last_change":"2024-06-09T22:17:26.475111+0000","last_active":"2024-06-09T22:17:26.475111+0000","last_peered":"2024-06-09T22:17:26.475111+0000","last_clean":"2024-06-09T22:17:26.475111+0000","last_became_active":"2024-06-09T22:17:26.474563+0000","last_became_peered":"2024-06-09T22:17:26.474563+0000","last_unstale":"2024-06-09T22:17:26.475111+0000","last_undegraded":"2024-06-09T22:17:26.475111+0000","last_fullsized":"2024-06-09T22:17:26.475111+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:09:08.458761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"97'6","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:13.481969+0000","last_change":"2024-06-09T22:16:40.040538+0000","last_active":"2024-06-09T22:18:13.481969+0000","last_peered":"2024-06-09T22:18:13.481969+0000","last_clean":"2024-06-09T22:18:13.481969+0000","last_became_active":"2024-06-09T22:16:40.039896+0000","last_became_peered":"2024-06-09T22:16:40.039896+0000","last_unstale":"2024-06-09T22:18:13.481969+0000","last_undegraded":"2024-06-09T22:18:13.481969+0000","last_fullsized":"2024-06-09T22:18:13.481969+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:00:33.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":978,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.475568+0000","last_change":"2024-06-09T22:17:26.475568+0000","last_active":"2024-06-09T22:17:26.475568+0000","last_peered":"2024-06-09T22:17:26.475568+0000","last_clean":"2024-06-09T22:17:26.475568+0000","last_became_active":"2024-06-09T22:17:26.475163+0000","last_became_peered":"2024-06-09T22:17:26.475163+0000","last_unstale":"2024-06-09T22:17:26.475568+0000","last_undegraded":"2024-06-09T22:17:26.475568+0000","last_fullsized":"2024-06-09T22:17:26.475568+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:20.865164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473986+0000","last_change":"2024-06-09T22:16:53.338201+0000","last_active":"2024-06-09T22:17:26.473986+0000","last_peered":"2024-06-09T22:17:26.473986+0000","last_clean":"2024-06-09T22:17:26.473986+0000","last_became_active":"2024-06-09T22:16:53.337816+0000","last_became_peered":"2024-06-09T22:16:53.337816+0000","last_unstale":"2024-06-09T22:17:26.473986+0000","last_undegraded":"2024-06-09T22:17:26.473986+0000","last_fullsized":"2024-06-09T22:17:26.473986+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:35:51.861135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"97'17","reported_seq":58,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.448956+0000","last_change":"2024-06-09T22:16:40.070770+0000","last_active":"2024-06-09T22:17:50.448956+0000","last_peered":"2024-06-09T22:17:50.448956+0000","last_clean":"2024-06-09T22:17:50.448956+0000","last_became_active":"2024-06-09T22:16:40.070398+0000","last_became_peered":"2024-06-09T22:16:40.070398+0000","last_unstale":"2024-06-09T22:17:50.448956+0000","last_undegraded":"2024-06-09T22:17:50.448956+0000","last_fullsized":"2024-06-09T22:17:50.448956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:04:58.273601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772781+0000","last_change":"2024-06-09T22:16:43.199246+0000","last_active":"2024-06-09T22:17:26.772781+0000","last_peered":"2024-06-09T22:17:26.772781+0000","last_clean":"2024-06-09T22:17:26.772781+0000","last_became_active":"2024-06-09T22:16:43.198006+0000","last_became_peered":"2024-06-09T22:16:43.198006+0000","last_unstale":"2024-06-09T22:17:26.772781+0000","last_undegraded":"2024-06-09T22:17:26.772781+0000","last_fullsized":"2024-06-09T22:17:26.772781+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:30:55.836628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772670+0000","last_change":"2024-06-09T22:16:53.342217+0000","last_active":"2024-06-09T22:17:26.772670+0000","last_peered":"2024-06-09T22:17:26.772670+0000","last_clean":"2024-06-09T22:17:26.772670+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:17:26.772670+0000","last_undegraded":"2024-06-09T22:17:26.772670+0000","last_fullsized":"2024-06-09T22:17:26.772670+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:25:28.635091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"97'2","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.423931+0000","last_change":"2024-06-09T22:16:40.039841+0000","last_active":"2024-06-09T22:17:50.423931+0000","last_peered":"2024-06-09T22:17:50.423931+0000","last_clean":"2024-06-09T22:17:50.423931+0000","last_became_active":"2024-06-09T22:16:40.039416+0000","last_became_peered":"2024-06-09T22:16:40.039416+0000","last_unstale":"2024-06-09T22:17:50.423931+0000","last_undegraded":"2024-06-09T22:17:50.423931+0000","last_fullsized":"2024-06-09T22:17:50.423931+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:27:36.213839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469582+0000","last_change":"2024-06-09T22:16:43.214363+0000","last_active":"2024-06-09T22:17:26.469582+0000","last_peered":"2024-06-09T22:17:26.469582+0000","last_clean":"2024-06-09T22:17:26.469582+0000","last_became_active":"2024-06-09T22:16:43.214015+0000","last_became_peered":"2024-06-09T22:16:43.214015+0000","last_unstale":"2024-06-09T22:17:26.469582+0000","last_undegraded":"2024-06-09T22:17:26.469582+0000","last_fullsized":"2024-06-09T22:17:26.469582+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:13:06.924468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468869+0000","last_change":"2024-06-09T22:16:53.342151+0000","last_active":"2024-06-09T22:17:26.468869+0000","last_peered":"2024-06-09T22:17:26.468869+0000","last_clean":"2024-06-09T22:17:26.468869+0000","last_became_active":"2024-06-09T22:16:53.341743+0000","last_became_peered":"2024-06-09T22:16:53.341743+0000","last_unstale":"2024-06-09T22:17:26.468869+0000","last_undegraded":"2024-06-09T22:17:26.468869+0000","last_fullsized":"2024-06-09T22:17:26.468869+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:39:44.793301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"97'1","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473153+0000","last_change":"2024-06-09T22:17:26.472397+0000","last_active":"2024-06-09T22:17:26.473153+0000","last_peered":"2024-06-09T22:17:26.473153+0000","last_clean":"2024-06-09T22:17:26.473153+0000","last_became_active":"2024-06-09T22:17:26.471950+0000","last_became_peered":"2024-06-09T22:17:26.471950+0000","last_unstale":"2024-06-09T22:17:26.473153+0000","last_undegraded":"2024-06-09T22:17:26.473153+0000","last_fullsized":"2024-06-09T22:17:26.473153+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:35:57.481510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469807+0000","last_change":"2024-06-09T22:16:43.214976+0000","last_active":"2024-06-09T22:17:26.469807+0000","last_peered":"2024-06-09T22:17:26.469807+0000","last_clean":"2024-06-09T22:17:26.469807+0000","last_became_active":"2024-06-09T22:16:43.214592+0000","last_became_peered":"2024-06-09T22:16:43.214592+0000","last_unstale":"2024-06-09T22:17:26.469807+0000","last_undegraded":"2024-06-09T22:17:26.469807+0000","last_fullsized":"2024-06-09T22:17:26.469807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:16:21.981144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473670+0000","last_change":"2024-06-09T22:16:53.342721+0000","last_active":"2024-06-09T22:17:26.473670+0000","last_peered":"2024-06-09T22:17:26.473670+0000","last_clean":"2024-06-09T22:17:26.473670+0000","last_became_active":"2024-06-09T22:16:53.342342+0000","last_became_peered":"2024-06-09T22:16:53.342342+0000","last_unstale":"2024-06-09T22:17:26.473670+0000","last_undegraded":"2024-06-09T22:17:26.473670+0000","last_fullsized":"2024-06-09T22:17:26.473670+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:22:07.469873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"97'5","reported_seq":70,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:13.481881+0000","last_change":"2024-06-09T22:16:40.041307+0000","last_active":"2024-06-09T22:18:13.481881+0000","last_peered":"2024-06-09T22:18:13.481881+0000","last_clean":"2024-06-09T22:18:13.481881+0000","last_became_active":"2024-06-09T22:16:40.040819+0000","last_became_peered":"2024-06-09T22:16:40.040819+0000","last_unstale":"2024-06-09T22:18:13.481881+0000","last_undegraded":"2024-06-09T22:18:13.481881+0000","last_fullsized":"2024-06-09T22:18:13.481881+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:14:29.658293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469809+0000","last_change":"2024-06-09T22:16:43.215719+0000","last_active":"2024-06-09T22:17:26.469809+0000","last_peered":"2024-06-09T22:17:26.469809+0000","last_clean":"2024-06-09T22:17:26.469809+0000","last_became_active":"2024-06-09T22:16:43.215356+0000","last_became_peered":"2024-06-09T22:16:43.215356+0000","last_unstale":"2024-06-09T22:17:26.469809+0000","last_undegraded":"2024-06-09T22:17:26.469809+0000","last_fullsized":"2024-06-09T22:17:26.469809+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:15:07.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473615+0000","last_change":"2024-06-09T22:16:53.342948+0000","last_active":"2024-06-09T22:17:26.473615+0000","last_peered":"2024-06-09T22:17:26.473615+0000","last_clean":"2024-06-09T22:17:26.473615+0000","last_became_active":"2024-06-09T22:16:53.342352+0000","last_became_peered":"2024-06-09T22:16:53.342352+0000","last_unstale":"2024-06-09T22:17:26.473615+0000","last_undegraded":"2024-06-09T22:17:26.473615+0000","last_fullsized":"2024-06-09T22:17:26.473615+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:43:27.453789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772973+0000","last_change":"2024-06-09T22:16:43.204897+0000","last_active":"2024-06-09T22:17:26.772973+0000","last_peered":"2024-06-09T22:17:26.772973+0000","last_clean":"2024-06-09T22:17:26.772973+0000","last_became_active":"2024-06-09T22:16:43.204470+0000","last_became_peered":"2024-06-09T22:16:43.204470+0000","last_unstale":"2024-06-09T22:17:26.772973+0000","last_undegraded":"2024-06-09T22:17:26.772973+0000","last_fullsized":"2024-06-09T22:17:26.772973+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:56.681392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"97'5","reported_seq":70,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:13.481399+0000","last_change":"2024-06-09T22:16:40.070079+0000","last_active":"2024-06-09T22:18:13.481399+0000","last_peered":"2024-06-09T22:18:13.481399+0000","last_clean":"2024-06-09T22:18:13.481399+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:18:13.481399+0000","last_undegraded":"2024-06-09T22:18:13.481399+0000","last_fullsized":"2024-06-09T22:18:13.481399+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:00:45.132139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"97'1","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473014+0000","last_change":"2024-06-09T22:16:53.338891+0000","last_active":"2024-06-09T22:17:26.473014+0000","last_peered":"2024-06-09T22:17:26.473014+0000","last_clean":"2024-06-09T22:17:26.473014+0000","last_became_active":"2024-06-09T22:16:53.338526+0000","last_became_peered":"2024-06-09T22:16:53.338526+0000","last_unstale":"2024-06-09T22:17:26.473014+0000","last_undegraded":"2024-06-09T22:17:26.473014+0000","last_fullsized":"2024-06-09T22:17:26.473014+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:34:37.004407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470574+0000","last_change":"2024-06-09T22:16:43.199234+0000","last_active":"2024-06-09T22:17:26.470574+0000","last_peered":"2024-06-09T22:17:26.470574+0000","last_clean":"2024-06-09T22:17:26.470574+0000","last_became_active":"2024-06-09T22:16:43.192012+0000","last_became_peered":"2024-06-09T22:16:43.192012+0000","last_unstale":"2024-06-09T22:17:26.470574+0000","last_undegraded":"2024-06-09T22:17:26.470574+0000","last_fullsized":"2024-06-09T22:17:26.470574+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:17.429236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"97'6","reported_seq":47,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:51.450603+0000","last_change":"2024-06-09T22:16:40.041970+0000","last_active":"2024-06-09T22:17:51.450603+0000","last_peered":"2024-06-09T22:17:51.450603+0000","last_clean":"2024-06-09T22:17:51.450603+0000","last_became_active":"2024-06-09T22:16:40.040371+0000","last_became_peered":"2024-06-09T22:16:40.040371+0000","last_unstale":"2024-06-09T22:17:51.450603+0000","last_undegraded":"2024-06-09T22:17:51.450603+0000","last_fullsized":"2024-06-09T22:17:51.450603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:52:31.210423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":374,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.472832+0000","last_change":"2024-06-09T22:15:15.367397+0000","last_active":"2024-06-09T22:17:26.472832+0000","last_peered":"2024-06-09T22:17:26.472832+0000","last_clean":"2024-06-09T22:17:26.472832+0000","last_became_active":"2024-06-09T22:15:15.366974+0000","last_became_peered":"2024-06-09T22:15:15.366974+0000","last_unstale":"2024-06-09T22:17:26.472832+0000","last_undegraded":"2024-06-09T22:17:26.472832+0000","last_fullsized":"2024-06-09T22:17:26.472832+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_clean_scrub_stamp":"2024-06-09T22:11:49.886056+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:24.018031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.773504+0000","last_change":"2024-06-09T22:16:53.342249+0000","last_active":"2024-06-09T22:17:26.773504+0000","last_peered":"2024-06-09T22:17:26.773504+0000","last_clean":"2024-06-09T22:17:26.773504+0000","last_became_active":"2024-06-09T22:16:53.341779+0000","last_became_peered":"2024-06-09T22:16:53.341779+0000","last_unstale":"2024-06-09T22:17:26.773504+0000","last_undegraded":"2024-06-09T22:17:26.773504+0000","last_fullsized":"2024-06-09T22:17:26.773504+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:21:21.101578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.472826+0000","last_change":"2024-06-09T22:16:43.200778+0000","last_active":"2024-06-09T22:17:26.472826+0000","last_peered":"2024-06-09T22:17:26.472826+0000","last_clean":"2024-06-09T22:17:26.472826+0000","last_became_active":"2024-06-09T22:16:43.200490+0000","last_became_peered":"2024-06-09T22:16:43.200490+0000","last_unstale":"2024-06-09T22:17:26.472826+0000","last_undegraded":"2024-06-09T22:17:26.472826+0000","last_fullsized":"2024-06-09T22:17:26.472826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:20:17.530204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"97'4","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:48.428568+0000","last_change":"2024-06-09T22:16:40.041774+0000","last_active":"2024-06-09T22:17:48.428568+0000","last_peered":"2024-06-09T22:17:48.428568+0000","last_clean":"2024-06-09T22:17:48.428568+0000","last_became_active":"2024-06-09T22:16:40.041085+0000","last_became_peered":"2024-06-09T22:16:40.041085+0000","last_unstale":"2024-06-09T22:17:48.428568+0000","last_undegraded":"2024-06-09T22:17:48.428568+0000","last_fullsized":"2024-06-09T22:17:48.428568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:09:41.140083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468191+0000","last_change":"2024-06-09T22:16:53.336895+0000","last_active":"2024-06-09T22:17:26.468191+0000","last_peered":"2024-06-09T22:17:26.468191+0000","last_clean":"2024-06-09T22:17:26.468191+0000","last_became_active":"2024-06-09T22:16:53.336497+0000","last_became_peered":"2024-06-09T22:16:53.336497+0000","last_unstale":"2024-06-09T22:17:26.468191+0000","last_undegraded":"2024-06-09T22:17:26.468191+0000","last_fullsized":"2024-06-09T22:17:26.468191+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:29:52.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"97'4","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.448185+0000","last_change":"2024-06-09T22:16:40.070909+0000","last_active":"2024-06-09T22:17:50.448185+0000","last_peered":"2024-06-09T22:17:50.448185+0000","last_clean":"2024-06-09T22:17:50.448185+0000","last_became_active":"2024-06-09T22:16:40.070556+0000","last_became_peered":"2024-06-09T22:16:40.070556+0000","last_unstale":"2024-06-09T22:17:50.448185+0000","last_undegraded":"2024-06-09T22:17:50.448185+0000","last_fullsized":"2024-06-09T22:17:50.448185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:59:17.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473784+0000","last_change":"2024-06-09T22:16:43.205951+0000","last_active":"2024-06-09T22:17:26.473784+0000","last_peered":"2024-06-09T22:17:26.473784+0000","last_clean":"2024-06-09T22:17:26.473784+0000","last_became_active":"2024-06-09T22:16:43.205594+0000","last_became_peered":"2024-06-09T22:16:43.205594+0000","last_unstale":"2024-06-09T22:17:26.473784+0000","last_undegraded":"2024-06-09T22:17:26.473784+0000","last_fullsized":"2024-06-09T22:17:26.473784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:17.956406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469938+0000","last_change":"2024-06-09T22:16:53.343292+0000","last_active":"2024-06-09T22:17:26.469938+0000","last_peered":"2024-06-09T22:17:26.469938+0000","last_clean":"2024-06-09T22:17:26.469938+0000","last_became_active":"2024-06-09T22:16:53.342900+0000","last_became_peered":"2024-06-09T22:16:53.342900+0000","last_unstale":"2024-06-09T22:17:26.469938+0000","last_undegraded":"2024-06-09T22:17:26.469938+0000","last_fullsized":"2024-06-09T22:17:26.469938+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:08:07.328250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"97'15","reported_seq":53,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:12.220326+0000","last_change":"2024-06-09T22:16:40.039417+0000","last_active":"2024-06-09T22:18:12.220326+0000","last_peered":"2024-06-09T22:18:12.220326+0000","last_clean":"2024-06-09T22:18:12.220326+0000","last_became_active":"2024-06-09T22:16:40.039099+0000","last_became_peered":"2024-06-09T22:16:40.039099+0000","last_unstale":"2024-06-09T22:18:12.220326+0000","last_undegraded":"2024-06-09T22:18:12.220326+0000","last_fullsized":"2024-06-09T22:18:12.220326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:22:59.284442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470377+0000","last_change":"2024-06-09T22:16:43.201285+0000","last_active":"2024-06-09T22:17:26.470377+0000","last_peered":"2024-06-09T22:17:26.470377+0000","last_clean":"2024-06-09T22:17:26.470377+0000","last_became_active":"2024-06-09T22:16:43.200993+0000","last_became_peered":"2024-06-09T22:16:43.200993+0000","last_unstale":"2024-06-09T22:17:26.470377+0000","last_undegraded":"2024-06-09T22:17:26.470377+0000","last_fullsized":"2024-06-09T22:17:26.470377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:15:26.491588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470670+0000","last_change":"2024-06-09T22:16:53.334088+0000","last_active":"2024-06-09T22:17:26.470670+0000","last_peered":"2024-06-09T22:17:26.470670+0000","last_clean":"2024-06-09T22:17:26.470670+0000","last_became_active":"2024-06-09T22:16:53.333597+0000","last_became_peered":"2024-06-09T22:16:53.333597+0000","last_unstale":"2024-06-09T22:17:26.470670+0000","last_undegraded":"2024-06-09T22:17:26.470670+0000","last_fullsized":"2024-06-09T22:17:26.470670+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:50:22.041816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"97'7","reported_seq":46,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.448170+0000","last_change":"2024-06-09T22:16:40.364982+0000","last_active":"2024-06-09T22:17:50.448170+0000","last_peered":"2024-06-09T22:17:50.448170+0000","last_clean":"2024-06-09T22:17:50.448170+0000","last_became_active":"2024-06-09T22:16:40.364585+0000","last_became_peered":"2024-06-09T22:16:40.364585+0000","last_unstale":"2024-06-09T22:17:50.448170+0000","last_undegraded":"2024-06-09T22:17:50.448170+0000","last_fullsized":"2024-06-09T22:17:50.448170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:28.777950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.474491+0000","last_change":"2024-06-09T22:17:26.474491+0000","last_active":"2024-06-09T22:17:26.474491+0000","last_peered":"2024-06-09T22:17:26.474491+0000","last_clean":"2024-06-09T22:17:26.474491+0000","last_became_active":"2024-06-09T22:17:26.473975+0000","last_became_peered":"2024-06-09T22:17:26.473975+0000","last_unstale":"2024-06-09T22:17:26.474491+0000","last_undegraded":"2024-06-09T22:17:26.474491+0000","last_fullsized":"2024-06-09T22:17:26.474491+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:13:46.998595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696646+0000","last_change":"2024-06-09T22:16:53.342273+0000","last_active":"2024-06-09T22:17:26.696646+0000","last_peered":"2024-06-09T22:17:26.696646+0000","last_clean":"2024-06-09T22:17:26.696646+0000","last_became_active":"2024-06-09T22:16:53.341827+0000","last_became_peered":"2024-06-09T22:16:53.341827+0000","last_unstale":"2024-06-09T22:17:26.696646+0000","last_undegraded":"2024-06-09T22:17:26.696646+0000","last_fullsized":"2024-06-09T22:17:26.696646+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:23:26.099737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"97'2","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:47.400625+0000","last_change":"2024-06-09T22:16:40.070498+0000","last_active":"2024-06-09T22:17:47.400625+0000","last_peered":"2024-06-09T22:17:47.400625+0000","last_clean":"2024-06-09T22:17:47.400625+0000","last_became_active":"2024-06-09T22:16:40.070144+0000","last_became_peered":"2024-06-09T22:16:40.070144+0000","last_unstale":"2024-06-09T22:17:47.400625+0000","last_undegraded":"2024-06-09T22:17:47.400625+0000","last_fullsized":"2024-06-09T22:17:47.400625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:52:22.214832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468077+0000","last_change":"2024-06-09T22:16:43.210448+0000","last_active":"2024-06-09T22:17:26.468077+0000","last_peered":"2024-06-09T22:17:26.468077+0000","last_clean":"2024-06-09T22:17:26.468077+0000","last_became_active":"2024-06-09T22:16:43.209980+0000","last_became_peered":"2024-06-09T22:16:43.209980+0000","last_unstale":"2024-06-09T22:17:26.468077+0000","last_undegraded":"2024-06-09T22:17:26.468077+0000","last_fullsized":"2024-06-09T22:17:26.468077+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:58:15.691207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.465998+0000","last_change":"2024-06-09T22:16:53.340644+0000","last_active":"2024-06-09T22:17:26.465998+0000","last_peered":"2024-06-09T22:17:26.465998+0000","last_clean":"2024-06-09T22:17:26.465998+0000","last_became_active":"2024-06-09T22:16:53.339771+0000","last_became_peered":"2024-06-09T22:16:53.339771+0000","last_unstale":"2024-06-09T22:17:26.465998+0000","last_undegraded":"2024-06-09T22:17:26.465998+0000","last_fullsized":"2024-06-09T22:17:26.465998+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:45:23.281387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"97'7","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.447669+0000","last_change":"2024-06-09T22:16:40.040461+0000","last_active":"2024-06-09T22:17:50.447669+0000","last_peered":"2024-06-09T22:17:50.447669+0000","last_clean":"2024-06-09T22:17:50.447669+0000","last_became_active":"2024-06-09T22:16:40.040168+0000","last_became_peered":"2024-06-09T22:16:40.040168+0000","last_unstale":"2024-06-09T22:17:50.447669+0000","last_undegraded":"2024-06-09T22:17:50.447669+0000","last_fullsized":"2024-06-09T22:17:50.447669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:28.936631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467671+0000","last_change":"2024-06-09T22:16:43.208779+0000","last_active":"2024-06-09T22:17:26.467671+0000","last_peered":"2024-06-09T22:17:26.467671+0000","last_clean":"2024-06-09T22:17:26.467671+0000","last_became_active":"2024-06-09T22:16:43.208136+0000","last_became_peered":"2024-06-09T22:16:43.208136+0000","last_unstale":"2024-06-09T22:17:26.467671+0000","last_undegraded":"2024-06-09T22:17:26.467671+0000","last_fullsized":"2024-06-09T22:17:26.467671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:22:39.721069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"97'1","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696723+0000","last_change":"2024-06-09T22:16:53.331241+0000","last_active":"2024-06-09T22:17:26.696723+0000","last_peered":"2024-06-09T22:17:26.696723+0000","last_clean":"2024-06-09T22:17:26.696723+0000","last_became_active":"2024-06-09T22:16:53.329589+0000","last_became_peered":"2024-06-09T22:16:53.329589+0000","last_unstale":"2024-06-09T22:17:26.696723+0000","last_undegraded":"2024-06-09T22:17:26.696723+0000","last_fullsized":"2024-06-09T22:17:26.696723+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:56:21.319711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"97'3","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:49.441888+0000","last_change":"2024-06-09T22:16:40.042491+0000","last_active":"2024-06-09T22:17:49.441888+0000","last_peered":"2024-06-09T22:17:49.441888+0000","last_clean":"2024-06-09T22:17:49.441888+0000","last_became_active":"2024-06-09T22:16:40.041846+0000","last_became_peered":"2024-06-09T22:16:40.041846+0000","last_unstale":"2024-06-09T22:17:49.441888+0000","last_undegraded":"2024-06-09T22:17:49.441888+0000","last_fullsized":"2024-06-09T22:17:49.441888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:59.144003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470789+0000","last_change":"2024-06-09T22:16:43.209195+0000","last_active":"2024-06-09T22:17:26.470789+0000","last_peered":"2024-06-09T22:17:26.470789+0000","last_clean":"2024-06-09T22:17:26.470789+0000","last_became_active":"2024-06-09T22:16:43.208797+0000","last_became_peered":"2024-06-09T22:16:43.208797+0000","last_unstale":"2024-06-09T22:17:26.470789+0000","last_undegraded":"2024-06-09T22:17:26.470789+0000","last_fullsized":"2024-06-09T22:17:26.470789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:05:31.689900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.696746+0000","last_change":"2024-06-09T22:16:53.342173+0000","last_active":"2024-06-09T22:17:26.696746+0000","last_peered":"2024-06-09T22:17:26.696746+0000","last_clean":"2024-06-09T22:17:26.696746+0000","last_became_active":"2024-06-09T22:16:53.341712+0000","last_became_peered":"2024-06-09T22:16:53.341712+0000","last_unstale":"2024-06-09T22:17:26.696746+0000","last_undegraded":"2024-06-09T22:17:26.696746+0000","last_fullsized":"2024-06-09T22:17:26.696746+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:53:12.574764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"97'4","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:47.401751+0000","last_change":"2024-06-09T22:16:40.044238+0000","last_active":"2024-06-09T22:17:47.401751+0000","last_peered":"2024-06-09T22:17:47.401751+0000","last_clean":"2024-06-09T22:17:47.401751+0000","last_became_active":"2024-06-09T22:16:40.043857+0000","last_became_peered":"2024-06-09T22:16:40.043857+0000","last_unstale":"2024-06-09T22:17:47.401751+0000","last_undegraded":"2024-06-09T22:17:47.401751+0000","last_fullsized":"2024-06-09T22:17:47.401751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:44:45.743307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.472875+0000","last_change":"2024-06-09T22:16:43.202363+0000","last_active":"2024-06-09T22:17:26.472875+0000","last_peered":"2024-06-09T22:17:26.472875+0000","last_clean":"2024-06-09T22:17:26.472875+0000","last_became_active":"2024-06-09T22:16:43.202029+0000","last_became_peered":"2024-06-09T22:16:43.202029+0000","last_unstale":"2024-06-09T22:17:26.472875+0000","last_undegraded":"2024-06-09T22:17:26.472875+0000","last_fullsized":"2024-06-09T22:17:26.472875+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:11:19.513055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772267+0000","last_change":"2024-06-09T22:16:53.333837+0000","last_active":"2024-06-09T22:17:26.772267+0000","last_peered":"2024-06-09T22:17:26.772267+0000","last_clean":"2024-06-09T22:17:26.772267+0000","last_became_active":"2024-06-09T22:16:53.333362+0000","last_became_peered":"2024-06-09T22:16:53.333362+0000","last_unstale":"2024-06-09T22:17:26.772267+0000","last_undegraded":"2024-06-09T22:17:26.772267+0000","last_fullsized":"2024-06-09T22:17:26.772267+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:47:04.463040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"97'6","reported_seq":75,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:18:17.488985+0000","last_change":"2024-06-09T22:16:40.364834+0000","last_active":"2024-06-09T22:18:17.488985+0000","last_peered":"2024-06-09T22:18:17.488985+0000","last_clean":"2024-06-09T22:18:17.488985+0000","last_became_active":"2024-06-09T22:16:40.364433+0000","last_became_peered":"2024-06-09T22:16:40.364433+0000","last_unstale":"2024-06-09T22:18:17.488985+0000","last_undegraded":"2024-06-09T22:18:17.488985+0000","last_fullsized":"2024-06-09T22:18:17.488985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:43:01.967911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473314+0000","last_change":"2024-06-09T22:16:43.201183+0000","last_active":"2024-06-09T22:17:26.473314+0000","last_peered":"2024-06-09T22:17:26.473314+0000","last_clean":"2024-06-09T22:17:26.473314+0000","last_became_active":"2024-06-09T22:16:43.200774+0000","last_became_peered":"2024-06-09T22:16:43.200774+0000","last_unstale":"2024-06-09T22:17:26.473314+0000","last_undegraded":"2024-06-09T22:17:26.473314+0000","last_fullsized":"2024-06-09T22:17:26.473314+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:07:55.772066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.467752+0000","last_change":"2024-06-09T22:16:53.345820+0000","last_active":"2024-06-09T22:17:26.467752+0000","last_peered":"2024-06-09T22:17:26.467752+0000","last_clean":"2024-06-09T22:17:26.467752+0000","last_became_active":"2024-06-09T22:16:53.345449+0000","last_became_peered":"2024-06-09T22:16:53.345449+0000","last_unstale":"2024-06-09T22:17:26.467752+0000","last_undegraded":"2024-06-09T22:17:26.467752+0000","last_fullsized":"2024-06-09T22:17:26.467752+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:47:54.478520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"97'3","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.445679+0000","last_change":"2024-06-09T22:16:40.040582+0000","last_active":"2024-06-09T22:17:50.445679+0000","last_peered":"2024-06-09T22:17:50.445679+0000","last_clean":"2024-06-09T22:17:50.445679+0000","last_became_active":"2024-06-09T22:16:40.040152+0000","last_became_peered":"2024-06-09T22:16:40.040152+0000","last_unstale":"2024-06-09T22:17:50.445679+0000","last_undegraded":"2024-06-09T22:17:50.445679+0000","last_fullsized":"2024-06-09T22:17:50.445679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:50:00.878096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.472792+0000","last_change":"2024-06-09T22:16:43.206211+0000","last_active":"2024-06-09T22:17:26.472792+0000","last_peered":"2024-06-09T22:17:26.472792+0000","last_clean":"2024-06-09T22:17:26.472792+0000","last_became_active":"2024-06-09T22:16:43.205814+0000","last_became_peered":"2024-06-09T22:16:43.205814+0000","last_unstale":"2024-06-09T22:17:26.472792+0000","last_undegraded":"2024-06-09T22:17:26.472792+0000","last_fullsized":"2024-06-09T22:17:26.472792+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:20:19.058475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772465+0000","last_change":"2024-06-09T22:16:53.337695+0000","last_active":"2024-06-09T22:17:26.772465+0000","last_peered":"2024-06-09T22:17:26.772465+0000","last_clean":"2024-06-09T22:17:26.772465+0000","last_became_active":"2024-06-09T22:16:53.334872+0000","last_became_peered":"2024-06-09T22:16:53.334872+0000","last_unstale":"2024-06-09T22:17:26.772465+0000","last_undegraded":"2024-06-09T22:17:26.772465+0000","last_fullsized":"2024-06-09T22:17:26.772465+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:50:36.617736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"97'5","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:50.445165+0000","last_change":"2024-06-09T22:16:40.036603+0000","last_active":"2024-06-09T22:17:50.445165+0000","last_peered":"2024-06-09T22:17:50.445165+0000","last_clean":"2024-06-09T22:17:50.445165+0000","last_became_active":"2024-06-09T22:16:40.036274+0000","last_became_peered":"2024-06-09T22:16:40.036274+0000","last_unstale":"2024-06-09T22:17:50.445165+0000","last_undegraded":"2024-06-09T22:17:50.445165+0000","last_fullsized":"2024-06-09T22:17:50.445165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:31.652897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473418+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:17:26.473418+0000","last_peered":"2024-06-09T22:17:26.473418+0000","last_clean":"2024-06-09T22:17:26.473418+0000","last_became_active":"2024-06-09T22:16:43.204197+0000","last_became_peered":"2024-06-09T22:16:43.204197+0000","last_unstale":"2024-06-09T22:17:26.473418+0000","last_undegraded":"2024-06-09T22:17:26.473418+0000","last_fullsized":"2024-06-09T22:17:26.473418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:38:49.850120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468056+0000","last_change":"2024-06-09T22:16:43.208477+0000","last_active":"2024-06-09T22:17:26.468056+0000","last_peered":"2024-06-09T22:17:26.468056+0000","last_clean":"2024-06-09T22:17:26.468056+0000","last_became_active":"2024-06-09T22:16:43.206018+0000","last_became_peered":"2024-06-09T22:16:43.206018+0000","last_unstale":"2024-06-09T22:17:26.468056+0000","last_undegraded":"2024-06-09T22:17:26.468056+0000","last_fullsized":"2024-06-09T22:17:26.468056+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:19:49.870825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469938+0000","last_change":"2024-06-09T22:16:43.206809+0000","last_active":"2024-06-09T22:17:26.469938+0000","last_peered":"2024-06-09T22:17:26.469938+0000","last_clean":"2024-06-09T22:17:26.469938+0000","last_became_active":"2024-06-09T22:16:43.206175+0000","last_became_peered":"2024-06-09T22:16:43.206175+0000","last_unstale":"2024-06-09T22:17:26.469938+0000","last_undegraded":"2024-06-09T22:17:26.469938+0000","last_fullsized":"2024-06-09T22:17:26.469938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:39:22.502266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469427+0000","last_change":"2024-06-09T22:16:43.202163+0000","last_active":"2024-06-09T22:17:26.469427+0000","last_peered":"2024-06-09T22:17:26.469427+0000","last_clean":"2024-06-09T22:17:26.469427+0000","last_became_active":"2024-06-09T22:16:43.201839+0000","last_became_peered":"2024-06-09T22:16:43.201839+0000","last_unstale":"2024-06-09T22:17:26.469427+0000","last_undegraded":"2024-06-09T22:17:26.469427+0000","last_fullsized":"2024-06-09T22:17:26.469427+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:47:46.723289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.468006+0000","last_change":"2024-06-09T22:16:43.202734+0000","last_active":"2024-06-09T22:17:26.468006+0000","last_peered":"2024-06-09T22:17:26.468006+0000","last_clean":"2024-06-09T22:17:26.468006+0000","last_became_active":"2024-06-09T22:16:43.201638+0000","last_became_peered":"2024-06-09T22:16:43.201638+0000","last_unstale":"2024-06-09T22:17:26.468006+0000","last_undegraded":"2024-06-09T22:17:26.468006+0000","last_fullsized":"2024-06-09T22:17:26.468006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:16.360985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473784+0000","last_change":"2024-06-09T22:16:43.199113+0000","last_active":"2024-06-09T22:17:26.473784+0000","last_peered":"2024-06-09T22:17:26.473784+0000","last_clean":"2024-06-09T22:17:26.473784+0000","last_became_active":"2024-06-09T22:16:43.198804+0000","last_became_peered":"2024-06-09T22:16:43.198804+0000","last_unstale":"2024-06-09T22:17:26.473784+0000","last_undegraded":"2024-06-09T22:17:26.473784+0000","last_fullsized":"2024-06-09T22:17:26.473784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:18:08.999477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.469398+0000","last_change":"2024-06-09T22:16:43.204445+0000","last_active":"2024-06-09T22:17:26.469398+0000","last_peered":"2024-06-09T22:17:26.469398+0000","last_clean":"2024-06-09T22:17:26.469398+0000","last_became_active":"2024-06-09T22:16:43.203081+0000","last_became_peered":"2024-06-09T22:16:43.203081+0000","last_unstale":"2024-06-09T22:17:26.469398+0000","last_undegraded":"2024-06-09T22:17:26.469398+0000","last_fullsized":"2024-06-09T22:17:26.469398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:39.303813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470194+0000","last_change":"2024-06-09T22:16:43.211402+0000","last_active":"2024-06-09T22:17:26.470194+0000","last_peered":"2024-06-09T22:17:26.470194+0000","last_clean":"2024-06-09T22:17:26.470194+0000","last_became_active":"2024-06-09T22:16:43.210871+0000","last_became_peered":"2024-06-09T22:16:43.210871+0000","last_unstale":"2024-06-09T22:17:26.470194+0000","last_undegraded":"2024-06-09T22:17:26.470194+0000","last_fullsized":"2024-06-09T22:17:26.470194+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:07:21.467617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470506+0000","last_change":"2024-06-09T22:16:43.201159+0000","last_active":"2024-06-09T22:17:26.470506+0000","last_peered":"2024-06-09T22:17:26.470506+0000","last_clean":"2024-06-09T22:17:26.470506+0000","last_became_active":"2024-06-09T22:16:43.200850+0000","last_became_peered":"2024-06-09T22:16:43.200850+0000","last_unstale":"2024-06-09T22:17:26.470506+0000","last_undegraded":"2024-06-09T22:17:26.470506+0000","last_fullsized":"2024-06-09T22:17:26.470506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:20.515309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772774+0000","last_change":"2024-06-09T22:16:43.196830+0000","last_active":"2024-06-09T22:17:26.772774+0000","last_peered":"2024-06-09T22:17:26.772774+0000","last_clean":"2024-06-09T22:17:26.772774+0000","last_became_active":"2024-06-09T22:16:43.195682+0000","last_became_peered":"2024-06-09T22:16:43.195682+0000","last_unstale":"2024-06-09T22:17:26.772774+0000","last_undegraded":"2024-06-09T22:17:26.772774+0000","last_fullsized":"2024-06-09T22:17:26.772774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:34:18.770235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470144+0000","last_change":"2024-06-09T22:16:43.201959+0000","last_active":"2024-06-09T22:17:26.470144+0000","last_peered":"2024-06-09T22:17:26.470144+0000","last_clean":"2024-06-09T22:17:26.470144+0000","last_became_active":"2024-06-09T22:16:43.201669+0000","last_became_peered":"2024-06-09T22:16:43.201669+0000","last_unstale":"2024-06-09T22:17:26.470144+0000","last_undegraded":"2024-06-09T22:17:26.470144+0000","last_fullsized":"2024-06-09T22:17:26.470144+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:48:57.348041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.772524+0000","last_change":"2024-06-09T22:16:43.201823+0000","last_active":"2024-06-09T22:17:26.772524+0000","last_peered":"2024-06-09T22:17:26.772524+0000","last_clean":"2024-06-09T22:17:26.772524+0000","last_became_active":"2024-06-09T22:16:43.201434+0000","last_became_peered":"2024-06-09T22:16:43.201434+0000","last_unstale":"2024-06-09T22:17:26.772524+0000","last_undegraded":"2024-06-09T22:17:26.772524+0000","last_fullsized":"2024-06-09T22:17:26.772524+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:56:13.540895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.473264+0000","last_change":"2024-06-09T22:16:43.197734+0000","last_active":"2024-06-09T22:17:26.473264+0000","last_peered":"2024-06-09T22:17:26.473264+0000","last_clean":"2024-06-09T22:17:26.473264+0000","last_became_active":"2024-06-09T22:16:43.197442+0000","last_became_peered":"2024-06-09T22:16:43.197442+0000","last_unstale":"2024-06-09T22:17:26.473264+0000","last_undegraded":"2024-06-09T22:17:26.473264+0000","last_fullsized":"2024-06-09T22:17:26.473264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:35:40.720718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-06-09T22:17:26.470366+0000","last_change":"2024-06-09T22:16:43.210375+0000","last_active":"2024-06-09T22:17:26.470366+0000","last_peered":"2024-06-09T22:17:26.470366+0000","last_clean":"2024-06-09T22:17:26.470366+0000","last_became_active":"2024-06-09T22:16:43.209647+0000","last_became_peered":"2024-06-09T22:16:43.209647+0000","last_unstale":"2024-06-09T22:17:26.470366+0000","last_undegraded":"2024-06-09T22:17:26.470366+0000","last_fullsized":"2024-06-09T22:17:26.470366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:11:57.593066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247891,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72982,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":298,"num_read_kb":231,"num_write":250,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":218946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":552059,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645352,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32376,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712776,"statfs":{"total":95995035648,"available":95961882624,"internally_reserved":0,"allocated":1228800,"data_stored":1204197,"data_compressed":62260,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":555627,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":551909,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1150976,"data_stored":1148795,"data_compressed":62260,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1077248,"data_stored":547035,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36456,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708696,"statfs":{"total":95995035648,"available":95957704704,"internally_reserved":0,"allocated":1146880,"data_stored":613455,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1101824,"data_stored":548881,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1159168,"data_stored":614387,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":546743,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1085440,"data_stored":551499,"data_compressed":26379,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1159168,"data_stored":1143987,"data_compressed":62260,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34243,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34243,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":67516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-09T22:18:21.241 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-09T22:18:21.241 INFO:tasks.cephadm:Setting up client nodes... 2024-06-09T22:18:21.241 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-09T22:18:21.485 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:21.771 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:21 smithi104 ceph-mon[32389]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:18:21.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:21 smithi194 ceph-mon[36856]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:18:21.989 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:21 smithi153 ceph-mon[36599]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:18:22.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:22 smithi104 ceph-mon[32389]: pgmap v355: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-06-09T22:18:22.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:22 smithi194 ceph-mon[36856]: pgmap v355: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-06-09T22:18:22.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:22 smithi153 ceph-mon[36599]: pgmap v355: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-06-09T22:18:23.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.325+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/4046761639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb8102510 msgr2=0x7fbdb8102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:23.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.325+0000 7fbdbe3a8700 1 --2- 172.21.15.104:0/4046761639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb8102510 0x7fbdb8102930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbda80099c0 tx=0x7fbda8009cd0 comp rx=0 tx=0).stop 2024-06-09T22:18:23.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.325+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/4046761639 shutdown_connections 2024-06-09T22:18:23.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.325+0000 7fbdbe3a8700 1 --2- 172.21.15.104:0/4046761639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb8102510 0x7fbdb8102930 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:23.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.325+0000 7fbdbe3a8700 1 --2- 172.21.15.104:0/4046761639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdb81040b0 0x7fbdb810a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:23.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.325+0000 7fbdbe3a8700 1 --2- 172.21.15.104:0/4046761639 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdb8103710 0x7fbdb8103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:23.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.325+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/4046761639 >> 172.21.15.104:0/4046761639 conn(0x7fbdb80fdab0 msgr2=0x7fbdb80ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:23.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.325+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/4046761639 shutdown_connections 2024-06-09T22:18:23.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.325+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/4046761639 wait complete. 2024-06-09T22:18:23.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.325+0000 7fbdbe3a8700 1 Processor -- start 2024-06-09T22:18:23.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.325+0000 7fbdbe3a8700 1 -- start start 2024-06-09T22:18:23.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdbe3a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb8102510 0x7fbdb81987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:23.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdbe3a8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdb8103710 0x7fbdb8198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:23.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdbe3a8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdb81040b0 0x7fbdb819fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:23.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdbe3a8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb810e1d0 con 0x7fbdb8102510 2024-06-09T22:18:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdbe3a8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb810e050 con 0x7fbdb81040b0 2024-06-09T22:18:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdbe3a8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb810e350 con 0x7fbdb8103710 2024-06-09T22:18:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb8102510 0x7fbdb81987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb8102510 0x7fbdb81987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60446/0 (socket says 172.21.15.104:60446) 2024-06-09T22:18:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdb7fff700 1 -- 172.21.15.104:0/3804132422 learned_addr learned my addr 172.21.15.104:0/3804132422 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdb7fff700 1 -- 172.21.15.104:0/3804132422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdb8103710 msgr2=0x7fbdb8198d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdb7fff700 1 --2- 172.21.15.104:0/3804132422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdb8103710 0x7fbdb8198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdb7fff700 1 -- 172.21.15.104:0/3804132422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdb81040b0 msgr2=0x7fbdb819fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdb7fff700 1 --2- 172.21.15.104:0/3804132422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdb81040b0 0x7fbdb819fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdb7fff700 1 -- 172.21.15.104:0/3804132422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb81a0440 con 0x7fbdb8102510 2024-06-09T22:18:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdb7fff700 1 --2- 172.21.15.104:0/3804132422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb8102510 0x7fbdb81987e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbda8005e80 tx=0x7fbda8003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdb5ffb700 1 -- 172.21.15.104:0/3804132422 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda8027070 con 0x7fbdb8102510 2024-06-09T22:18:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/3804132422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdb81a06d0 con 0x7fbdb8102510 2024-06-09T22:18:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdb5ffb700 1 -- 172.21.15.104:0/3804132422 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbda8022470 con 0x7fbdb8102510 2024-06-09T22:18:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdb5ffb700 1 -- 172.21.15.104:0/3804132422 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda800f6c0 con 0x7fbdb8102510 2024-06-09T22:18:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.326+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/3804132422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdb81a0c80 con 0x7fbdb8102510 2024-06-09T22:18:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.327+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/3804132422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdb8050a00 con 0x7fbdb8102510 2024-06-09T22:18:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.329+0000 7fbdb5ffb700 1 -- 172.21.15.104:0/3804132422 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fbda802a070 con 0x7fbdb8102510 2024-06-09T22:18:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.329+0000 7fbdb5ffb700 1 --2- 172.21.15.104:0/3804132422 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbda40a2d60 0x7fbda40a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.329+0000 7fbdaffff700 1 --2- 172.21.15.104:0/3804132422 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbda40a2d60 0x7fbda40a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.329+0000 7fbdb5ffb700 1 -- 172.21.15.104:0/3804132422 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9092+0+0 (secure 0 0 0) 0x7fbda80d0850 con 0x7fbdb8102510 2024-06-09T22:18:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.330+0000 7fbdaffff700 1 --2- 172.21.15.104:0/3804132422 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbda40a2d60 0x7fbda40a5220 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbda00097b0 tx=0x7fbda0006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:23.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.333+0000 7fbdb5ffb700 1 -- 172.21.15.104:0/3804132422 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbda8013070 con 0x7fbdb8102510 2024-06-09T22:18:23.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.612+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/3804132422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fbdb802dd70 con 0x7fbdb8102510 2024-06-09T22:18:23.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.614+0000 7fbdb5ffb700 1 -- 172.21.15.104:0/3804132422 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fbda809d380 con 0x7fbdb8102510 2024-06-09T22:18:23.611 INFO:teuthology.orchestra.run.smithi104.stdout:[client.0] 2024-06-09T22:18:23.612 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQAvKmZmYbyaJBAAyQBO+TYD9nj7VelrQGro0A== 2024-06-09T22:18:23.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.616+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/3804132422 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbda40a2d60 msgr2=0x7fbda40a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:23.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.616+0000 7fbdbe3a8700 1 --2- 172.21.15.104:0/3804132422 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbda40a2d60 0x7fbda40a5220 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbda00097b0 tx=0x7fbda0006d20 comp rx=0 tx=0).stop 2024-06-09T22:18:23.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.616+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/3804132422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb8102510 msgr2=0x7fbdb81987e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:23.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.616+0000 7fbdbe3a8700 1 --2- 172.21.15.104:0/3804132422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb8102510 0x7fbdb81987e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbda8005e80 tx=0x7fbda8003680 comp rx=0 tx=0).stop 2024-06-09T22:18:23.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.617+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/3804132422 shutdown_connections 2024-06-09T22:18:23.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.617+0000 7fbdbe3a8700 1 --2- 172.21.15.104:0/3804132422 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbda40a2d60 0x7fbda40a5220 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:23.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.617+0000 7fbdbe3a8700 1 --2- 172.21.15.104:0/3804132422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb8102510 0x7fbdb81987e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:23.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.617+0000 7fbdbe3a8700 1 --2- 172.21.15.104:0/3804132422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdb81040b0 0x7fbdb819fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:23.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.617+0000 7fbdbe3a8700 1 --2- 172.21.15.104:0/3804132422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdb8103710 0x7fbdb8198d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:23.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.617+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/3804132422 >> 172.21.15.104:0/3804132422 conn(0x7fbdb80fdab0 msgr2=0x7fbdb8106a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:23.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.617+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/3804132422 shutdown_connections 2024-06-09T22:18:23.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:23.617+0000 7fbdbe3a8700 1 -- 172.21.15.104:0/3804132422 wait complete. 2024-06-09T22:18:23.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:23 smithi194 ceph-mon[36856]: pgmap v356: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-06-09T22:18:23.930 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:23 smithi104 ceph-mon[32389]: pgmap v356: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-06-09T22:18:23.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:23 smithi153 ceph-mon[36599]: pgmap v356: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-06-09T22:18:24.388 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:18:24.388 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-09T22:18:24.388 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-09T22:18:24.427 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-09T22:18:24.427 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-09T22:18:24.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-06-09T22:18:24.706 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:24.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:24 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3804132422' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-09T22:18:24.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:24 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3804132422' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-09T22:18:24.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:24 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3804132422' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-09T22:18:24.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:24 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3804132422' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-09T22:18:24.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:24 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3804132422' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-09T22:18:24.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:24 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3804132422' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-09T22:18:25.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:25 smithi194 ceph-mon[36856]: pgmap v357: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 10 op/s 2024-06-09T22:18:25.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 10]}]: dispatch 2024-06-09T22:18:25.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:25 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 8]}]: dispatch 2024-06-09T22:18:25.828 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:25 smithi104 ceph-mon[32389]: pgmap v357: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 10 op/s 2024-06-09T22:18:25.828 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 10]}]: dispatch 2024-06-09T22:18:25.828 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:25 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 8]}]: dispatch 2024-06-09T22:18:25.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:25 smithi153 ceph-mon[36599]: pgmap v357: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 10 op/s 2024-06-09T22:18:25.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 10]}]: dispatch 2024-06-09T22:18:25.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:25 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 8]}]: dispatch 2024-06-09T22:18:26.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.596+0000 7f397345e700 1 -- 172.21.15.104:0/633375747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c1030f0 msgr2=0x7f396c107110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:26.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.596+0000 7f397345e700 1 --2- 172.21.15.104:0/633375747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c1030f0 0x7f396c107110 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f395c0099c0 tx=0x7f395c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:18:26.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.596+0000 7f397345e700 1 -- 172.21.15.104:0/633375747 shutdown_connections 2024-06-09T22:18:26.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.596+0000 7f397345e700 1 --2- 172.21.15.104:0/633375747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f396c107840 0x7f396c10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:26.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.596+0000 7f397345e700 1 --2- 172.21.15.104:0/633375747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c102740 0x7f396c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:26.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.596+0000 7f397345e700 1 --2- 172.21.15.104:0/633375747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c1030f0 0x7f396c107110 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:26.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.596+0000 7f397345e700 1 -- 172.21.15.104:0/633375747 >> 172.21.15.104:0/633375747 conn(0x7f396c0795f0 msgr2=0x7f396c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.597+0000 7f397345e700 1 -- 172.21.15.104:0/633375747 shutdown_connections 2024-06-09T22:18:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.597+0000 7f397345e700 1 -- 172.21.15.104:0/633375747 wait complete. 2024-06-09T22:18:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.597+0000 7f397345e700 1 Processor -- start 2024-06-09T22:18:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.597+0000 7f397345e700 1 -- start start 2024-06-09T22:18:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.597+0000 7f397345e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f396c102740 0x7f396c19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.597+0000 7f397345e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c1030f0 0x7f396c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.597+0000 7f397345e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c107840 0x7f396c198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.597+0000 7f397345e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f396c1127f0 con 0x7f396c1030f0 2024-06-09T22:18:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.597+0000 7f397345e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f396c112670 con 0x7f396c107840 2024-06-09T22:18:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.597+0000 7f397345e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f396c112970 con 0x7f396c102740 2024-06-09T22:18:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39709f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c1030f0 0x7f396c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39709f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c1030f0 0x7f396c19edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59442/0 (socket says 172.21.15.104:59442) 2024-06-09T22:18:26.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39709f9700 1 -- 172.21.15.104:0/1346149121 learned_addr learned my addr 172.21.15.104:0/1346149121 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:26.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39711fa700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f396c102740 0x7f396c19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:26.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39719fb700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c107840 0x7f396c198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:26.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39709f9700 1 -- 172.21.15.104:0/1346149121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f396c102740 msgr2=0x7f396c19e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:26.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39709f9700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f396c102740 0x7f396c19e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:26.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39709f9700 1 -- 172.21.15.104:0/1346149121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c107840 msgr2=0x7f396c198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:26.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39709f9700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c107840 0x7f396c198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:26.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39709f9700 1 -- 172.21.15.104:0/1346149121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c199190 con 0x7f396c1030f0 2024-06-09T22:18:26.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39709f9700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c1030f0 0x7f396c19edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f395c0059d0 tx=0x7f395c0038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:26.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39627fc700 1 -- 172.21.15.104:0/1346149121 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c027070 con 0x7f396c1030f0 2024-06-09T22:18:26.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39627fc700 1 -- 172.21.15.104:0/1346149121 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f395c004050 con 0x7f396c1030f0 2024-06-09T22:18:26.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f397345e700 1 -- 172.21.15.104:0/1346149121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396c199420 con 0x7f396c1030f0 2024-06-09T22:18:26.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f39627fc700 1 -- 172.21.15.104:0/1346149121 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c022db0 con 0x7f396c1030f0 2024-06-09T22:18:26.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.598+0000 7f397345e700 1 -- 172.21.15.104:0/1346149121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396c1a5740 con 0x7f396c1030f0 2024-06-09T22:18:26.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.599+0000 7f397345e700 1 -- 172.21.15.104:0/1346149121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396c069180 con 0x7f396c1030f0 2024-06-09T22:18:26.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.600+0000 7f39627fc700 1 -- 172.21.15.104:0/1346149121 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f395c02a070 con 0x7f396c1030f0 2024-06-09T22:18:26.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.601+0000 7f39627fc700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f39580a2cb0 0x7f39580a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:26.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.601+0000 7f39711fa700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f39580a2cb0 0x7f39580a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:26.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.601+0000 7f39627fc700 1 -- 172.21.15.104:0/1346149121 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f395c0d02e0 con 0x7f396c1030f0 2024-06-09T22:18:26.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.601+0000 7f39711fa700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f39580a2cb0 0x7f39580a5170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f396c199b70 tx=0x7f3968009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:26.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.605+0000 7f39627fc700 1 -- 172.21.15.104:0/1346149121 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f395c05e850 con 0x7f396c1030f0 2024-06-09T22:18:26.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 10]}]': finished 2024-06-09T22:18:26.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:26 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 8]}]': finished 2024-06-09T22:18:26.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:26 smithi104 ceph-mon[32389]: osdmap e98: 12 total, 12 up, 12 in 2024-06-09T22:18:26.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 10]}]': finished 2024-06-09T22:18:26.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:26 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 8]}]': finished 2024-06-09T22:18:26.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:26 smithi194 ceph-mon[36856]: osdmap e98: 12 total, 12 up, 12 in 2024-06-09T22:18:26.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.867+0000 7f397345e700 1 -- 172.21.15.104:0/1346149121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f396c10aa80 con 0x7f396c1030f0 2024-06-09T22:18:26.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.872+0000 7f39627fc700 1 -- 172.21.15.104:0/1346149121 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263207 (secure 0 0 0) 0x7f395c013070 con 0x7f396c1030f0 2024-06-09T22:18:26.872 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:18:26.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.874+0000 7f397345e700 1 -- 172.21.15.104:0/1346149121 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f39580a2cb0 msgr2=0x7f39580a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:26.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.874+0000 7f397345e700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f39580a2cb0 0x7f39580a5170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f396c199b70 tx=0x7f3968009190 comp rx=0 tx=0).stop 2024-06-09T22:18:26.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.874+0000 7f397345e700 1 -- 172.21.15.104:0/1346149121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c1030f0 msgr2=0x7f396c19edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:26.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.874+0000 7f397345e700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c1030f0 0x7f396c19edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f395c0059d0 tx=0x7f395c0038b0 comp rx=0 tx=0).stop 2024-06-09T22:18:26.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.874+0000 7f397345e700 1 -- 172.21.15.104:0/1346149121 shutdown_connections 2024-06-09T22:18:26.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.874+0000 7f397345e700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f39580a2cb0 0x7f39580a5170 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:26.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.874+0000 7f397345e700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c107840 0x7f396c198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:26.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.874+0000 7f397345e700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f396c102740 0x7f396c19e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:26.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.874+0000 7f397345e700 1 --2- 172.21.15.104:0/1346149121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f396c1030f0 0x7f396c19edb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:26.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.874+0000 7f397345e700 1 -- 172.21.15.104:0/1346149121 >> 172.21.15.104:0/1346149121 conn(0x7f396c0795f0 msgr2=0x7f396c108990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:26.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.875+0000 7f397345e700 1 -- 172.21.15.104:0/1346149121 shutdown_connections 2024-06-09T22:18:26.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:26.875+0000 7f397345e700 1 -- 172.21.15.104:0/1346149121 wait complete. 2024-06-09T22:18:26.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 10]}]': finished 2024-06-09T22:18:26.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:26 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 8]}]': finished 2024-06-09T22:18:26.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:26 smithi153 ceph-mon[36599]: osdmap e98: 12 total, 12 up, 12 in 2024-06-09T22:18:27.680 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1906048760},{"type":"v1","addr":"172.21.15.104:6801","nonce":1906048760}]},"active_addr":"172.21.15.104:6801/1906048760","active_change":"2024-06-09T22:09:25.221837+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24115,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.104:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":517954187}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":2389242747}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":357628765}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":167857460}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":3780581862}]}]} 2024-06-09T22:18:27.685 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-09T22:18:27.685 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-09T22:18:27.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:18:27.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:27 smithi194 ceph-mon[36856]: osdmap e99: 12 total, 12 up, 12 in 2024-06-09T22:18:27.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:27 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1346149121' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-09T22:18:27.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:27 smithi194 ceph-mon[36856]: pgmap v360: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-06-09T22:18:27.925 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:27.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:27 smithi153 ceph-mon[36599]: osdmap e99: 12 total, 12 up, 12 in 2024-06-09T22:18:27.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:27 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1346149121' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-09T22:18:27.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:27 smithi153 ceph-mon[36599]: pgmap v360: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-06-09T22:18:28.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:27 smithi104 ceph-mon[32389]: osdmap e99: 12 total, 12 up, 12 in 2024-06-09T22:18:28.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:27 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1346149121' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-09T22:18:28.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:27 smithi104 ceph-mon[32389]: pgmap v360: 129 pgs: 129 active+clean; 648 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-06-09T22:18:29.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.646+0000 7f66be426700 1 -- 172.21.15.104:0/1951597615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66b8102900 msgr2=0x7f66b810e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:29.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.646+0000 7f66be426700 1 --2- 172.21.15.104:0/1951597615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66b8102900 0x7f66b810e310 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f66b000b0d0 tx=0x7f66b000b3e0 comp rx=0 tx=0).stop 2024-06-09T22:18:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.646+0000 7f66be426700 1 -- 172.21.15.104:0/1951597615 shutdown_connections 2024-06-09T22:18:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.646+0000 7f66be426700 1 --2- 172.21.15.104:0/1951597615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66b8102900 0x7f66b810e310 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.646+0000 7f66be426700 1 --2- 172.21.15.104:0/1951597615 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66b8107660 0x7f66b8107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.646+0000 7f66be426700 1 --2- 172.21.15.104:0/1951597615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66b8101f40 0x7f66b81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.646+0000 7f66be426700 1 -- 172.21.15.104:0/1951597615 >> 172.21.15.104:0/1951597615 conn(0x7f66b80766b0 msgr2=0x7f66b8076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.646+0000 7f66be426700 1 -- 172.21.15.104:0/1951597615 shutdown_connections 2024-06-09T22:18:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.646+0000 7f66be426700 1 -- 172.21.15.104:0/1951597615 wait complete. 2024-06-09T22:18:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66be426700 1 Processor -- start 2024-06-09T22:18:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66be426700 1 -- start start 2024-06-09T22:18:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66be426700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66b8101f40 0x7f66b819d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66be426700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66b8102900 0x7f66b819dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66be426700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66b8107660 0x7f66b8197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66be426700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f66b81117b0 con 0x7f66b8107660 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66be426700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f66b8111630 con 0x7f66b8102900 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66be426700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f66b8111930 con 0x7f66b8101f40 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66bc9c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66b8107660 0x7f66b8197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66bc9c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66b8107660 0x7f66b8197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59460/0 (socket says 172.21.15.104:59460) 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66bc9c3700 1 -- 172.21.15.104:0/211868377 learned_addr learned my addr 172.21.15.104:0/211868377 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66b7fff700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66b8101f40 0x7f66b819d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66affff700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66b8102900 0x7f66b819dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66bc9c3700 1 -- 172.21.15.104:0/211868377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66b8101f40 msgr2=0x7f66b819d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66bc9c3700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66b8101f40 0x7f66b819d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66bc9c3700 1 -- 172.21.15.104:0/211868377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66b8102900 msgr2=0x7f66b819dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66bc9c3700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66b8102900 0x7f66b819dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.647+0000 7f66bc9c3700 1 -- 172.21.15.104:0/211868377 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b81980d0 con 0x7f66b8107660 2024-06-09T22:18:29.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.648+0000 7f66bc9c3700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66b8107660 0x7f66b8197970 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f66b000d010 tx=0x7f66b0004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:29.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.648+0000 7f66b5ffb700 1 -- 172.21.15.104:0/211868377 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0021020 con 0x7f66b8107660 2024-06-09T22:18:29.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.648+0000 7f66b5ffb700 1 -- 172.21.15.104:0/211868377 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66b0007b80 con 0x7f66b8107660 2024-06-09T22:18:29.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.648+0000 7f66be426700 1 -- 172.21.15.104:0/211868377 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b8198360 con 0x7f66b8107660 2024-06-09T22:18:29.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.648+0000 7f66b5ffb700 1 -- 172.21.15.104:0/211868377 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b00052a0 con 0x7f66b8107660 2024-06-09T22:18:29.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.648+0000 7f66be426700 1 -- 172.21.15.104:0/211868377 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b81a4680 con 0x7f66b8107660 2024-06-09T22:18:29.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.649+0000 7f66be426700 1 -- 172.21.15.104:0/211868377 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b8069180 con 0x7f66b8107660 2024-06-09T22:18:29.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.650+0000 7f66b5ffb700 1 -- 172.21.15.104:0/211868377 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f66b0024020 con 0x7f66b8107660 2024-06-09T22:18:29.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.651+0000 7f66b5ffb700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f66a40a2d30 0x7f66a40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:29.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.651+0000 7f66b7fff700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f66a40a2d30 0x7f66a40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:29.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.651+0000 7f66b5ffb700 1 -- 172.21.15.104:0/211868377 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f66b000f020 con 0x7f66b8107660 2024-06-09T22:18:29.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.651+0000 7f66b7fff700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f66a40a2d30 0x7f66a40a51f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f66b8198ab0 tx=0x7f66a800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:29.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.654+0000 7f66b5ffb700 1 -- 172.21.15.104:0/211868377 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f66b009d570 con 0x7f66b8107660 2024-06-09T22:18:29.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.849+0000 7f66be426700 1 -- 172.21.15.104:0/211868377 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f66b81a4850 con 0x7f66b8107660 2024-06-09T22:18:29.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.851+0000 7f66b5ffb700 1 -- 172.21.15.104:0/211868377 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+24762 (secure 0 0 0) 0x7f66b009ce70 con 0x7f66b8107660 2024-06-09T22:18:29.848 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:18:29.848 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":99,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:18:26.525714+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:18:29.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.853+0000 7f66be426700 1 -- 172.21.15.104:0/211868377 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f66a40a2d30 msgr2=0x7f66a40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:29.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.853+0000 7f66be426700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f66a40a2d30 0x7f66a40a51f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f66b8198ab0 tx=0x7f66a800b410 comp rx=0 tx=0).stop 2024-06-09T22:18:29.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.853+0000 7f66be426700 1 -- 172.21.15.104:0/211868377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66b8107660 msgr2=0x7f66b8197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:29.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.853+0000 7f66be426700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66b8107660 0x7f66b8197970 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f66b000d010 tx=0x7f66b0004ce0 comp rx=0 tx=0).stop 2024-06-09T22:18:29.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.853+0000 7f66be426700 1 -- 172.21.15.104:0/211868377 shutdown_connections 2024-06-09T22:18:29.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.853+0000 7f66be426700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f66a40a2d30 0x7f66a40a51f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:29.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.853+0000 7f66be426700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66b8107660 0x7f66b8197970 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:29.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.853+0000 7f66be426700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66b8101f40 0x7f66b819d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:29.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.853+0000 7f66be426700 1 --2- 172.21.15.104:0/211868377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66b8102900 0x7f66b819dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:29.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.853+0000 7f66be426700 1 -- 172.21.15.104:0/211868377 >> 172.21.15.104:0/211868377 conn(0x7f66b80766b0 msgr2=0x7f66b80fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:29.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.853+0000 7f66be426700 1 -- 172.21.15.104:0/211868377 shutdown_connections 2024-06-09T22:18:29.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:29.853+0000 7f66be426700 1 -- 172.21.15.104:0/211868377 wait complete. 2024-06-09T22:18:30.504 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:30 smithi104 ceph-mon[32389]: pgmap v361: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 383 B/s wr, 10 op/s 2024-06-09T22:18:30.504 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:30 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/211868377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:18:30.603 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-09T22:18:30.603 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:18:30.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:30 smithi153 ceph-mon[36599]: pgmap v361: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 383 B/s wr, 10 op/s 2024-06-09T22:18:30.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:30 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/211868377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:18:30.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:30 smithi194 ceph-mon[36856]: pgmap v361: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 383 B/s wr, 10 op/s 2024-06-09T22:18:30.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:30 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/211868377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:18:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:32.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.605+0000 7f31d9fc4700 1 -- 172.21.15.104:0/838845553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d41084f0 msgr2=0x7f31d41088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:32.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.605+0000 7f31d9fc4700 1 --2- 172.21.15.104:0/838845553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d41084f0 0x7f31d41088d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009990 tx=0x7f31c4009ca0 comp rx=0 tx=0).stop 2024-06-09T22:18:32.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.605+0000 7f31d9fc4700 1 -- 172.21.15.104:0/838845553 shutdown_connections 2024-06-09T22:18:32.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.605+0000 7f31d9fc4700 1 --2- 172.21.15.104:0/838845553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d41084f0 0x7f31d41088d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:32.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.605+0000 7f31d9fc4700 1 --2- 172.21.15.104:0/838845553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31d4102e80 0x7f31d410f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:32.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.605+0000 7f31d9fc4700 1 --2- 172.21.15.104:0/838845553 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31d41024e0 0x7f31d4102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:32.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.605+0000 7f31d9fc4700 1 -- 172.21.15.104:0/838845553 >> 172.21.15.104:0/838845553 conn(0x7f31d4079530 msgr2=0x7f31d4079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:32.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.605+0000 7f31d9fc4700 1 -- 172.21.15.104:0/838845553 shutdown_connections 2024-06-09T22:18:32.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.605+0000 7f31d9fc4700 1 -- 172.21.15.104:0/838845553 wait complete. 2024-06-09T22:18:32.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.605+0000 7f31d9fc4700 1 Processor -- start 2024-06-09T22:18:32.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.605+0000 7f31d9fc4700 1 -- start start 2024-06-09T22:18:32.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d9fc4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31d41024e0 0x7f31d4074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:32.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d9fc4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31d4102e80 0x7f31d4074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:32.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d9fc4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d41084f0 0x7f31d4075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:32.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d9fc4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f31d4112750 con 0x7f31d41084f0 2024-06-09T22:18:32.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d9fc4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f31d41125d0 con 0x7f31d41024e0 2024-06-09T22:18:32.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d9fc4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f31d41128d0 con 0x7f31d4102e80 2024-06-09T22:18:32.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d41084f0 0x7f31d4075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:32.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d41084f0 0x7f31d4075210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59474/0 (socket says 172.21.15.104:59474) 2024-06-09T22:18:32.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d3fff700 1 -- 172.21.15.104:0/2270316638 learned_addr learned my addr 172.21.15.104:0/2270316638 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:32.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d2ffd700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31d4102e80 0x7f31d4074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:32.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d37fe700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31d41024e0 0x7f31d4074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:32.605 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:32 smithi104 ceph-mon[32389]: pgmap v362: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 383 B/s wr, 9 op/s 2024-06-09T22:18:32.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d3fff700 1 -- 172.21.15.104:0/2270316638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31d4102e80 msgr2=0x7f31d4074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:32.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d3fff700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31d4102e80 0x7f31d4074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:32.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d3fff700 1 -- 172.21.15.104:0/2270316638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31d41024e0 msgr2=0x7f31d4074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:32.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d3fff700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31d41024e0 0x7f31d4074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:32.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d3fff700 1 -- 172.21.15.104:0/2270316638 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d41a71b0 con 0x7f31d41084f0 2024-06-09T22:18:32.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.606+0000 7f31d3fff700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d41084f0 0x7f31d4075210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f31cc007ee0 tx=0x7f31cc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:32.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.607+0000 7f31d0ff9700 1 -- 172.21.15.104:0/2270316638 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc004bb0 con 0x7f31d41084f0 2024-06-09T22:18:32.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.607+0000 7f31d9fc4700 1 -- 172.21.15.104:0/2270316638 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d41a74a0 con 0x7f31d41084f0 2024-06-09T22:18:32.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.607+0000 7f31d9fc4700 1 -- 172.21.15.104:0/2270316638 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d41a7a30 con 0x7f31d41084f0 2024-06-09T22:18:32.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.607+0000 7f31d0ff9700 1 -- 172.21.15.104:0/2270316638 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31cc004d50 con 0x7f31d41084f0 2024-06-09T22:18:32.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.607+0000 7f31d0ff9700 1 -- 172.21.15.104:0/2270316638 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc005960 con 0x7f31d41084f0 2024-06-09T22:18:32.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.608+0000 7f31d9fc4700 1 -- 172.21.15.104:0/2270316638 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31c0005330 con 0x7f31d41084f0 2024-06-09T22:18:32.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.609+0000 7f31d0ff9700 1 -- 172.21.15.104:0/2270316638 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f31cc01a020 con 0x7f31d41084f0 2024-06-09T22:18:32.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.609+0000 7f31d0ff9700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f31bc0a2d30 0x7f31bc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:32.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.610+0000 7f31d37fe700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f31bc0a2d30 0x7f31bc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:32.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.610+0000 7f31d0ff9700 1 -- 172.21.15.104:0/2270316638 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f31cc0cea30 con 0x7f31d41084f0 2024-06-09T22:18:32.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.610+0000 7f31d37fe700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f31bc0a2d30 0x7f31bc0a51f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f31c400b5c0 tx=0x7f31c4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:32.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.614+0000 7f31d0ff9700 1 -- 172.21.15.104:0/2270316638 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f31cc010070 con 0x7f31d41084f0 2024-06-09T22:18:32.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:32 smithi153 ceph-mon[36599]: pgmap v362: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 383 B/s wr, 9 op/s 2024-06-09T22:18:32.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:32 smithi194 ceph-mon[36856]: pgmap v362: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 383 B/s wr, 9 op/s 2024-06-09T22:18:32.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.809+0000 7f31d9fc4700 1 -- 172.21.15.104:0/2270316638 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31c00058f0 con 0x7f31d41084f0 2024-06-09T22:18:32.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.811+0000 7f31d0ff9700 1 -- 172.21.15.104:0/2270316638 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+24762 (secure 0 0 0) 0x7f31cc09b4a0 con 0x7f31d41084f0 2024-06-09T22:18:32.809 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:18:32.809 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":99,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:18:26.525714+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:18:32.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.813+0000 7f31d9fc4700 1 -- 172.21.15.104:0/2270316638 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f31bc0a2d30 msgr2=0x7f31bc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:32.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.813+0000 7f31d9fc4700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f31bc0a2d30 0x7f31bc0a51f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f31c400b5c0 tx=0x7f31c4005960 comp rx=0 tx=0).stop 2024-06-09T22:18:32.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.813+0000 7f31d9fc4700 1 -- 172.21.15.104:0/2270316638 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d41084f0 msgr2=0x7f31d4075210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:32.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.813+0000 7f31d9fc4700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d41084f0 0x7f31d4075210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f31cc007ee0 tx=0x7f31cc00d3b0 comp rx=0 tx=0).stop 2024-06-09T22:18:32.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.813+0000 7f31d9fc4700 1 -- 172.21.15.104:0/2270316638 shutdown_connections 2024-06-09T22:18:32.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.813+0000 7f31d9fc4700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f31bc0a2d30 0x7f31bc0a51f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:32.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.813+0000 7f31d9fc4700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d41084f0 0x7f31d4075210 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:32.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.813+0000 7f31d9fc4700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31d41024e0 0x7f31d4074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:32.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.813+0000 7f31d9fc4700 1 --2- 172.21.15.104:0/2270316638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31d4102e80 0x7f31d4074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:32.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.813+0000 7f31d9fc4700 1 -- 172.21.15.104:0/2270316638 >> 172.21.15.104:0/2270316638 conn(0x7f31d4079530 msgr2=0x7f31d4100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:32.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.813+0000 7f31d9fc4700 1 -- 172.21.15.104:0/2270316638 shutdown_connections 2024-06-09T22:18:32.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:32.813+0000 7f31d9fc4700 1 -- 172.21.15.104:0/2270316638 wait complete. 2024-06-09T22:18:33.615 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:33 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2270316638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:18:33.647 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-09T22:18:33.647 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-09T22:18:33.648 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-09T22:18:33.648 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-09T22:18:33.648 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-09T22:18:33.649 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-09T22:18:33.649 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-09T22:18:33.650 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-09T22:18:33.650 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-06-09T22:18:33.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-06-09T22:18:33.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-06-09T22:18:33.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-06-09T22:18:33.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:33 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2270316638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:18:33.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:33 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2270316638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:18:34.284 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:34.686 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:34 smithi104 ceph-mon[32389]: pgmap v363: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 383 B/s wr, 10 op/s 2024-06-09T22:18:34.707 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:34 smithi194 ceph-mon[36856]: pgmap v363: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 383 B/s wr, 10 op/s 2024-06-09T22:18:34.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:34 smithi153 ceph-mon[36599]: pgmap v363: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 383 B/s wr, 10 op/s 2024-06-09T22:18:35.464 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:35.471 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:35.471 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:35.472 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:35.477 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:35.477 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:35.478 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:35.487 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:35.496 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:35.500 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:35.501 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:18:36.710 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:36 smithi104 ceph-mon[32389]: pgmap v364: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 313 B/s wr, 9 op/s 2024-06-09T22:18:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:36 smithi153 ceph-mon[36599]: pgmap v364: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 313 B/s wr, 9 op/s 2024-06-09T22:18:36.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:36 smithi194 ceph-mon[36856]: pgmap v364: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 313 B/s wr, 9 op/s 2024-06-09T22:18:38.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:38 smithi153 ceph-mon[36599]: pgmap v365: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 284 B/s wr, 9 op/s 2024-06-09T22:18:38.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:38 smithi104 ceph-mon[32389]: pgmap v365: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 284 B/s wr, 9 op/s 2024-06-09T22:18:38.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:38 smithi194 ceph-mon[36856]: pgmap v365: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 284 B/s wr, 9 op/s 2024-06-09T22:18:40.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.441+0000 7f31a5afd700 1 -- 172.21.15.104:0/1611356790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a0101390 msgr2=0x7f31a0101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:40.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.441+0000 7f31a5afd700 1 --2- 172.21.15.104:0/1611356790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a0101390 0x7f31a0101810 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3194009990 tx=0x7f3194009ca0 comp rx=0 tx=0).stop 2024-06-09T22:18:40.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.441+0000 7f31a5afd700 1 -- 172.21.15.104:0/1611356790 shutdown_connections 2024-06-09T22:18:40.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.441+0000 7f31a5afd700 1 --2- 172.21.15.104:0/1611356790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31a0101d50 0x7f31a010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:40.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.441+0000 7f31a5afd700 1 --2- 172.21.15.104:0/1611356790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a0107400 0x7f31a01077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:40.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.441+0000 7f31a5afd700 1 --2- 172.21.15.104:0/1611356790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a0101390 0x7f31a0101810 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:40.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.441+0000 7f31a5afd700 1 -- 172.21.15.104:0/1611356790 >> 172.21.15.104:0/1611356790 conn(0x7f31a0078340 msgr2=0x7f31a0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:40.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.441+0000 7f31a5afd700 1 -- 172.21.15.104:0/1611356790 shutdown_connections 2024-06-09T22:18:40.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.441+0000 7f31a5afd700 1 -- 172.21.15.104:0/1611356790 wait complete. 2024-06-09T22:18:40.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f31a5afd700 1 Processor -- start 2024-06-09T22:18:40.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f31a5afd700 1 -- start start 2024-06-09T22:18:40.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f31a5afd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31a0101390 0x7f31a0197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:40.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f31a5afd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a0101d50 0x7f31a0197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:40.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f31a5afd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a0107400 0x7f31a0198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:40.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f31a5afd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f31a01116c0 con 0x7f31a0107400 2024-06-09T22:18:40.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f31a5afd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0111540 con 0x7f31a0101390 2024-06-09T22:18:40.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f31a5afd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0111840 con 0x7f31a0101d50 2024-06-09T22:18:40.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f319ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a0107400 0x7f31a0198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:40.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f319f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31a0101390 0x7f31a0197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:40.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f319ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a0107400 0x7f31a0198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50408/0 (socket says 172.21.15.104:50408) 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f319ffff700 1 -- 172.21.15.104:0/239512479 learned_addr learned my addr 172.21.15.104:0/239512479 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.442+0000 7f319effd700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a0101d50 0x7f31a0197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.443+0000 7f319f7fe700 1 -- 172.21.15.104:0/239512479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a0101d50 msgr2=0x7f31a0197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.443+0000 7f319f7fe700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31a0101d50 0x7f31a0197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.443+0000 7f319f7fe700 1 -- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a0107400 msgr2=0x7f31a0198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.443+0000 7f319f7fe700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a0107400 0x7f31a0198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.443+0000 7f319f7fe700 1 -- 172.21.15.104:0/239512479 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a01a7120 con 0x7f31a0101390 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.443+0000 7f319ffff700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a0107400 0x7f31a0198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.443+0000 7f319f7fe700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31a0101390 0x7f31a0197910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f319000d980 tx=0x7f319000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.443+0000 7f319cff9700 1 -- 172.21.15.104:0/239512479 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190017070 con 0x7f31a0101390 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.443+0000 7f31a5afd700 1 -- 172.21.15.104:0/239512479 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a01a7410 con 0x7f31a0101390 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.443+0000 7f319cff9700 1 -- 172.21.15.104:0/239512479 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f319000fd70 con 0x7f31a0101390 2024-06-09T22:18:40.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.443+0000 7f319cff9700 1 -- 172.21.15.104:0/239512479 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319000e3b0 con 0x7f31a0101390 2024-06-09T22:18:40.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.443+0000 7f31a5afd700 1 -- 172.21.15.104:0/239512479 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a01a7950 con 0x7f31a0101390 2024-06-09T22:18:40.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.444+0000 7f31a5afd700 1 -- 172.21.15.104:0/239512479 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f31a0062c20 con 0x7f31a0101390 2024-06-09T22:18:40.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.445+0000 7f319cff9700 1 -- 172.21.15.104:0/239512479 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f319001a070 con 0x7f31a0101390 2024-06-09T22:18:40.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.446+0000 7f319cff9700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f31880a2cb0 0x7f31880a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:40.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.446+0000 7f319effd700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f31880a2cb0 0x7f31880a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:40.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.446+0000 7f319cff9700 1 -- 172.21.15.104:0/239512479 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f31900cf100 con 0x7f31a0101390 2024-06-09T22:18:40.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.446+0000 7f319cff9700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340] conn(0x7f31880adf10 0x7f31880b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:40.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.446+0000 7f319cff9700 1 -- 172.21.15.104:0/239512479 --> [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3190023e70 con 0x7f31880adf10 2024-06-09T22:18:40.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.446+0000 7f319cff9700 1 -- 172.21.15.104:0/239512479 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f31900cf550 con 0x7f31a0101390 2024-06-09T22:18:40.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.446+0000 7f319ffff700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340] conn(0x7f31880adf10 0x7f31880b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:40.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.446+0000 7f319effd700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f31880a2cb0 0x7f31880a5170 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f31a0199a70 tx=0x7f3194005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:40.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.447+0000 7f319ffff700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340] conn(0x7f31880adf10 0x7f31880b03b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:40.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.447+0000 7f319cff9700 1 -- 172.21.15.104:0/239512479 <== osd.3 v2:172.21.15.104:6810/3003003340 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3190023e70 con 0x7f31880adf10 2024-06-09T22:18:40.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.463+0000 7f31a5afd700 1 -- 172.21.15.104:0/239512479 --> [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f31a00630a0 con 0x7f31880adf10 2024-06-09T22:18:40.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.464+0000 7f319cff9700 1 -- 172.21.15.104:0/239512479 <== osd.3 v2:172.21.15.104:6810/3003003340 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f31a00630a0 con 0x7f31880adf10 2024-06-09T22:18:40.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.464+0000 7f31a5afd700 1 -- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340] conn(0x7f31880adf10 msgr2=0x7f31880b03b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:40.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.464+0000 7f31a5afd700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340] conn(0x7f31880adf10 0x7f31880b03b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:40.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.464+0000 7f31a5afd700 1 -- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f31880a2cb0 msgr2=0x7f31880a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:40.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.464+0000 7f31a5afd700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f31880a2cb0 0x7f31880a5170 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f31a0199a70 tx=0x7f3194005fd0 comp rx=0 tx=0).stop 2024-06-09T22:18:40.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.464+0000 7f31a5afd700 1 -- 172.21.15.104:0/239512479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31a0101390 msgr2=0x7f31a0197910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:40.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.464+0000 7f31a5afd700 1 --2- 172.21.15.104:0/239512479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31a0101390 0x7f31a0197910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f319000d980 tx=0x7f319000dc90 comp rx=0 tx=0).stop 2024-06-09T22:18:40.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.464+0000 7f319ffff700 1 -- 172.21.15.104:0/239512479 reap_dead start 2024-06-09T22:18:40.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.464+0000 7f31a5afd700 1 -- 172.21.15.104:0/239512479 shutdown_connections 2024-06-09T22:18:40.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.464+0000 7f31a5afd700 1 -- 172.21.15.104:0/239512479 >> 172.21.15.104:0/239512479 conn(0x7f31a0078340 msgr2=0x7f31a00fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:40.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.464+0000 7f31a5afd700 1 -- 172.21.15.104:0/239512479 shutdown_connections 2024-06-09T22:18:40.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:40.464+0000 7f31a5afd700 1 -- 172.21.15.104:0/239512479 wait complete. 2024-06-09T22:18:40.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:40 smithi153 ceph-mon[36599]: pgmap v366: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-06-09T22:18:40.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:40 smithi104 ceph-mon[32389]: pgmap v366: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-06-09T22:18:40.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:40 smithi194 ceph-mon[36856]: pgmap v366: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-06-09T22:18:42.572 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:42 smithi104 ceph-mon[32389]: pgmap v367: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:18:42.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:42 smithi153 ceph-mon[36599]: pgmap v367: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:18:42.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:42 smithi194 ceph-mon[36856]: pgmap v367: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:18:44.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:44 smithi153 ceph-mon[36599]: pgmap v368: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:18:44.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:44 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:18:44.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:44 smithi104 ceph-mon[32389]: pgmap v368: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:18:44.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:44 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:18:44.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:44 smithi194 ceph-mon[36856]: pgmap v368: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:18:44.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:44 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:18:46.396 INFO:teuthology.orchestra.run.smithi104.stdout:107374182480 2024-06-09T22:18:46.397 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-09T22:18:46.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:46 smithi104 ceph-mon[32389]: pgmap v369: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:18:46.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:46 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:18:46.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:46 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:18:46.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:46 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:18:46.620 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:46 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:18:46.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:46 smithi153 ceph-mon[36599]: pgmap v369: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:18:46.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:46 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:18:46.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:46 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:18:46.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:46 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:18:46.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:46 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:18:46.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:46 smithi194 ceph-mon[36856]: pgmap v369: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:18:46.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:46 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:18:46.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:46 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:18:46.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:46 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:18:46.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:46 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:18:48.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.375+0000 7f26addb1700 1 -- 172.21.15.104:0/772068342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26a8107400 msgr2=0x7f26a81077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:48.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.375+0000 7f26addb1700 1 --2- 172.21.15.104:0/772068342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26a8107400 0x7f26a81077e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2698009990 tx=0x7f2698009ca0 comp rx=0 tx=0).stop 2024-06-09T22:18:48.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.375+0000 7f26addb1700 1 -- 172.21.15.104:0/772068342 shutdown_connections 2024-06-09T22:18:48.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.375+0000 7f26addb1700 1 --2- 172.21.15.104:0/772068342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26a8107400 0x7f26a81077e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:48.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.375+0000 7f26addb1700 1 --2- 172.21.15.104:0/772068342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26a8101d50 0x7f26a810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:48.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.375+0000 7f26addb1700 1 --2- 172.21.15.104:0/772068342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26a8101390 0x7f26a8101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:48.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.375+0000 7f26addb1700 1 -- 172.21.15.104:0/772068342 >> 172.21.15.104:0/772068342 conn(0x7f26a8078340 msgr2=0x7f26a8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:48.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.375+0000 7f26addb1700 1 -- 172.21.15.104:0/772068342 shutdown_connections 2024-06-09T22:18:48.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.375+0000 7f26addb1700 1 -- 172.21.15.104:0/772068342 wait complete. 2024-06-09T22:18:48.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26addb1700 1 Processor -- start 2024-06-09T22:18:48.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26addb1700 1 -- start start 2024-06-09T22:18:48.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26addb1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26a8101390 0x7f26a819e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:48.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26addb1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26a8101d50 0x7f26a819ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:48.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26addb1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26a8107400 0x7f26a8198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:48.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26addb1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f26a8111670 con 0x7f26a8101390 2024-06-09T22:18:48.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26addb1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f26a81114f0 con 0x7f26a8101d50 2024-06-09T22:18:48.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26addb1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f26a81117f0 con 0x7f26a8107400 2024-06-09T22:18:48.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26a77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26a8101390 0x7f26a819e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:48.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f269ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26a8101d50 0x7f26a819ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:48.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26a77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26a8101390 0x7f26a819e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33330/0 (socket says 172.21.15.104:33330) 2024-06-09T22:18:48.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f269ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26a8101d50 0x7f26a819ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:50690/0 (socket says 172.21.15.104:50690) 2024-06-09T22:18:48.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26a77fe700 1 -- 172.21.15.104:0/2639610907 learned_addr learned my addr 172.21.15.104:0/2639610907 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:48.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26a7fff700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26a8107400 0x7f26a8198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:48.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26a77fe700 1 -- 172.21.15.104:0/2639610907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26a8107400 msgr2=0x7f26a8198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:48.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26a77fe700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26a8107400 0x7f26a8198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:48.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26a77fe700 1 -- 172.21.15.104:0/2639610907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26a8101d50 msgr2=0x7f26a819ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:48.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26a77fe700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26a8101d50 0x7f26a819ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:48.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.376+0000 7f26a77fe700 1 -- 172.21.15.104:0/2639610907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a8199180 con 0x7f26a8101390 2024-06-09T22:18:48.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.377+0000 7f26a77fe700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26a8101390 0x7f26a819e810 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f26980038b0 tx=0x7f2698009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:48.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.377+0000 7f26a57fa700 1 -- 172.21.15.104:0/2639610907 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2698027070 con 0x7f26a8101390 2024-06-09T22:18:48.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.377+0000 7f26addb1700 1 -- 172.21.15.104:0/2639610907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a8199410 con 0x7f26a8101390 2024-06-09T22:18:48.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.377+0000 7f26a57fa700 1 -- 172.21.15.104:0/2639610907 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2698004050 con 0x7f26a8101390 2024-06-09T22:18:48.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.377+0000 7f26a57fa700 1 -- 172.21.15.104:0/2639610907 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2698022cd0 con 0x7f26a8101390 2024-06-09T22:18:48.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.377+0000 7f26addb1700 1 -- 172.21.15.104:0/2639610907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a81a56e0 con 0x7f26a8101390 2024-06-09T22:18:48.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.378+0000 7f26addb1700 1 -- 172.21.15.104:0/2639610907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f26a8105500 con 0x7f26a8101390 2024-06-09T22:18:48.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.379+0000 7f26a57fa700 1 -- 172.21.15.104:0/2639610907 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f269802a070 con 0x7f26a8101390 2024-06-09T22:18:48.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.379+0000 7f26a57fa700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f26940a2cb0 0x7f26940a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:48.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.380+0000 7f269ffff700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f26940a2cb0 0x7f26940a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:48.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.380+0000 7f26a57fa700 1 -- 172.21.15.104:0/2639610907 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f26980d0ed0 con 0x7f26a8101390 2024-06-09T22:18:48.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.380+0000 7f26a57fa700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024] conn(0x7f26940adf10 0x7f26940b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:48.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.380+0000 7f26a57fa700 1 -- 172.21.15.104:0/2639610907 --> [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2698010ec0 con 0x7f26940adf10 2024-06-09T22:18:48.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.380+0000 7f26a57fa700 1 -- 172.21.15.104:0/2639610907 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f26980d1320 con 0x7f26a8101390 2024-06-09T22:18:48.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.380+0000 7f269ffff700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f26940a2cb0 0x7f26940a5170 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f26a8078940 tx=0x7f269000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:48.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.380+0000 7f26a7fff700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024] conn(0x7f26940adf10 0x7f26940b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:48.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.380+0000 7f26a7fff700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024] conn(0x7f26940adf10 0x7f26940b03b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:48.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.381+0000 7f26a57fa700 1 -- 172.21.15.104:0/2639610907 <== osd.7 v2:172.21.15.153:6816/1289060024 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2698010ec0 con 0x7f26940adf10 2024-06-09T22:18:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.404+0000 7f26addb1700 1 -- 172.21.15.104:0/2639610907 --> [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f26a8102ab0 con 0x7f26940adf10 2024-06-09T22:18:48.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.405+0000 7f26a57fa700 1 -- 172.21.15.104:0/2639610907 <== osd.7 v2:172.21.15.153:6816/1289060024 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f26a8102ab0 con 0x7f26940adf10 2024-06-09T22:18:48.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.405+0000 7f26addb1700 1 -- 172.21.15.104:0/2639610907 >> [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024] conn(0x7f26940adf10 msgr2=0x7f26940b03b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:48.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.405+0000 7f26addb1700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024] conn(0x7f26940adf10 0x7f26940b03b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:48.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.405+0000 7f26addb1700 1 -- 172.21.15.104:0/2639610907 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f26940a2cb0 msgr2=0x7f26940a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:48.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.405+0000 7f26addb1700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f26940a2cb0 0x7f26940a5170 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f26a8078940 tx=0x7f269000b410 comp rx=0 tx=0).stop 2024-06-09T22:18:48.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.405+0000 7f26addb1700 1 -- 172.21.15.104:0/2639610907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26a8101390 msgr2=0x7f26a819e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:48.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.405+0000 7f26addb1700 1 --2- 172.21.15.104:0/2639610907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26a8101390 0x7f26a819e810 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f26980038b0 tx=0x7f2698009f50 comp rx=0 tx=0).stop 2024-06-09T22:18:48.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.405+0000 7f26a7fff700 1 -- 172.21.15.104:0/2639610907 reap_dead start 2024-06-09T22:18:48.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.405+0000 7f26addb1700 1 -- 172.21.15.104:0/2639610907 shutdown_connections 2024-06-09T22:18:48.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.405+0000 7f26addb1700 1 -- 172.21.15.104:0/2639610907 >> 172.21.15.104:0/2639610907 conn(0x7f26a8078340 msgr2=0x7f26a8100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:48.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.405+0000 7f26addb1700 1 -- 172.21.15.104:0/2639610907 shutdown_connections 2024-06-09T22:18:48.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:48.406+0000 7f26addb1700 1 -- 172.21.15.104:0/2639610907 wait complete. 2024-06-09T22:18:48.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:48 smithi153 ceph-mon[36599]: pgmap v370: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:18:48.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:48 smithi104 ceph-mon[32389]: pgmap v370: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:18:48.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:48 smithi194 ceph-mon[36856]: pgmap v370: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:18:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.530+0000 7fb6dea7a700 1 -- 172.21.15.104:0/417145933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6d8102740 msgr2=0x7fb6d8102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.530+0000 7fb6dea7a700 1 --2- 172.21.15.104:0/417145933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6d8102740 0x7fb6d8102b20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb6c800b0d0 tx=0x7fb6c800b3e0 comp rx=0 tx=0).stop 2024-06-09T22:18:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.530+0000 7fb6dea7a700 1 -- 172.21.15.104:0/417145933 shutdown_connections 2024-06-09T22:18:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.530+0000 7fb6dea7a700 1 --2- 172.21.15.104:0/417145933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6d8102740 0x7fb6d8102b20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.530+0000 7fb6dea7a700 1 --2- 172.21.15.104:0/417145933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d8107840 0x7fb6d810f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.530+0000 7fb6dea7a700 1 --2- 172.21.15.104:0/417145933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb6d81030f0 0x7fb6d8107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.530+0000 7fb6dea7a700 1 -- 172.21.15.104:0/417145933 >> 172.21.15.104:0/417145933 conn(0x7fb6d80795f0 msgr2=0x7fb6d8079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.531+0000 7fb6dea7a700 1 -- 172.21.15.104:0/417145933 shutdown_connections 2024-06-09T22:18:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.531+0000 7fb6dea7a700 1 -- 172.21.15.104:0/417145933 wait complete. 2024-06-09T22:18:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.531+0000 7fb6dea7a700 1 Processor -- start 2024-06-09T22:18:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.531+0000 7fb6dea7a700 1 -- start start 2024-06-09T22:18:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.531+0000 7fb6dea7a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6d8102740 0x7fb6d819e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.531+0000 7fb6dea7a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb6d81030f0 0x7fb6d819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.531+0000 7fb6dea7a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d8107840 0x7fb6d8198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.531+0000 7fb6dea7a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d81127d0 con 0x7fb6d8102740 2024-06-09T22:18:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.531+0000 7fb6dea7a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d8112650 con 0x7fb6d8107840 2024-06-09T22:18:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.531+0000 7fb6dea7a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d8112950 con 0x7fb6d81030f0 2024-06-09T22:18:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.531+0000 7fb6dc816700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6d8102740 0x7fb6d819e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6dc816700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6d8102740 0x7fb6d819e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33384/0 (socket says 172.21.15.104:33384) 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6dc816700 1 -- 172.21.15.104:0/2757247804 learned_addr learned my addr 172.21.15.104:0/2757247804 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6d7fff700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb6d81030f0 0x7fb6d819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6dd017700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d8107840 0x7fb6d8198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6dc816700 1 -- 172.21.15.104:0/2757247804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb6d81030f0 msgr2=0x7fb6d819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6dc816700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb6d81030f0 0x7fb6d819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6dc816700 1 -- 172.21.15.104:0/2757247804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d8107840 msgr2=0x7fb6d8198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6dc816700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d8107840 0x7fb6d8198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6dc816700 1 -- 172.21.15.104:0/2757247804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d81991b0 con 0x7fb6d8102740 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6dc816700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6d8102740 0x7fb6d819e890 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb6c800b820 tx=0x7fb6c8004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6d5ffb700 1 -- 172.21.15.104:0/2757247804 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8021020 con 0x7fb6d8102740 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6d5ffb700 1 -- 172.21.15.104:0/2757247804 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6c8007c00 con 0x7fb6d8102740 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6d5ffb700 1 -- 172.21.15.104:0/2757247804 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80053c0 con 0x7fb6d8102740 2024-06-09T22:18:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6dea7a700 1 -- 172.21.15.104:0/2757247804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d8199440 con 0x7fb6d8102740 2024-06-09T22:18:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.532+0000 7fb6dea7a700 1 -- 172.21.15.104:0/2757247804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d81a5760 con 0x7fb6d8102740 2024-06-09T22:18:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.533+0000 7fb6dea7a700 1 -- 172.21.15.104:0/2757247804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb6d8062c20 con 0x7fb6d8102740 2024-06-09T22:18:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.534+0000 7fb6d5ffb700 1 -- 172.21.15.104:0/2757247804 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fb6c8024020 con 0x7fb6d8102740 2024-06-09T22:18:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.535+0000 7fb6d5ffb700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb6c40a2cb0 0x7fb6c40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.535+0000 7fb6d7fff700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb6c40a2cb0 0x7fb6c40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.535+0000 7fb6d5ffb700 1 -- 172.21.15.104:0/2757247804 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fb6c800f020 con 0x7fb6d8102740 2024-06-09T22:18:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.535+0000 7fb6d5ffb700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655] conn(0x7fb6c40adf10 0x7fb6c40b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.535+0000 7fb6d5ffb700 1 -- 172.21.15.104:0/2757247804 --> [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6c802fe50 con 0x7fb6c40adf10 2024-06-09T22:18:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.535+0000 7fb6d5ffb700 1 -- 172.21.15.104:0/2757247804 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fb6c80d11e0 con 0x7fb6d8102740 2024-06-09T22:18:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.535+0000 7fb6d7fff700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb6c40a2cb0 0x7fb6c40a5170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8102020 tx=0x7fb6c0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.536+0000 7fb6dd017700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655] conn(0x7fb6c40adf10 0x7fb6c40b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.536+0000 7fb6dd017700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655] conn(0x7fb6c40adf10 0x7fb6c40b03b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:50.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.537+0000 7fb6d5ffb700 1 -- 172.21.15.104:0/2757247804 <== osd.10 v2:172.21.15.153:6824/1184232655 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb6c802fe50 con 0x7fb6c40adf10 2024-06-09T22:18:50.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.553+0000 7fb6dea7a700 1 -- 172.21.15.104:0/2757247804 --> [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb6d80630a0 con 0x7fb6c40adf10 2024-06-09T22:18:50.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.553+0000 7fb6d5ffb700 1 -- 172.21.15.104:0/2757247804 <== osd.10 v2:172.21.15.153:6824/1184232655 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb6d80630a0 con 0x7fb6c40adf10 2024-06-09T22:18:50.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.554+0000 7fb6dea7a700 1 -- 172.21.15.104:0/2757247804 >> [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655] conn(0x7fb6c40adf10 msgr2=0x7fb6c40b03b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:50.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.554+0000 7fb6dea7a700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655] conn(0x7fb6c40adf10 0x7fb6c40b03b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:50.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.554+0000 7fb6dea7a700 1 -- 172.21.15.104:0/2757247804 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb6c40a2cb0 msgr2=0x7fb6c40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:50.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.554+0000 7fb6dea7a700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb6c40a2cb0 0x7fb6c40a5170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8102020 tx=0x7fb6c0009190 comp rx=0 tx=0).stop 2024-06-09T22:18:50.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.554+0000 7fb6dea7a700 1 -- 172.21.15.104:0/2757247804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6d8102740 msgr2=0x7fb6d819e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:50.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.554+0000 7fb6dea7a700 1 --2- 172.21.15.104:0/2757247804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6d8102740 0x7fb6d819e890 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb6c800b820 tx=0x7fb6c8004e20 comp rx=0 tx=0).stop 2024-06-09T22:18:50.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.554+0000 7fb6dd017700 1 -- 172.21.15.104:0/2757247804 reap_dead start 2024-06-09T22:18:50.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.554+0000 7fb6dea7a700 1 -- 172.21.15.104:0/2757247804 shutdown_connections 2024-06-09T22:18:50.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.554+0000 7fb6dea7a700 1 -- 172.21.15.104:0/2757247804 >> 172.21.15.104:0/2757247804 conn(0x7fb6d80795f0 msgr2=0x7fb6d8100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:50.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.554+0000 7fb6dea7a700 1 -- 172.21.15.104:0/2757247804 shutdown_connections 2024-06-09T22:18:50.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.554+0000 7fb6dea7a700 1 -- 172.21.15.104:0/2757247804 wait complete. 2024-06-09T22:18:50.664 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:50 smithi104 ceph-mon[32389]: pgmap v371: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:18:50.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:50 smithi153 ceph-mon[36599]: pgmap v371: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:18:50.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:50 smithi194 ceph-mon[36856]: pgmap v371: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:18:51.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.981+0000 7faf6ebdf700 1 -- 172.21.15.104:0/22636145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf68103120 msgr2=0x7faf68107130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.981+0000 7faf6ebdf700 1 --2- 172.21.15.104:0/22636145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf68103120 0x7faf68107130 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7faf540099c0 tx=0x7faf54009cd0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 -- 172.21.15.104:0/22636145 shutdown_connections 2024-06-09T22:18:51.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 --2- 172.21.15.104:0/22636145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf68107860 0x7faf6810f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 --2- 172.21.15.104:0/22636145 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf68102770 0x7faf68102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 --2- 172.21.15.104:0/22636145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf68103120 0x7faf68107130 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 -- 172.21.15.104:0/22636145 >> 172.21.15.104:0/22636145 conn(0x7faf68079600 msgr2=0x7faf68079a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:51.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 -- 172.21.15.104:0/22636145 shutdown_connections 2024-06-09T22:18:51.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 -- 172.21.15.104:0/22636145 wait complete. 2024-06-09T22:18:51.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 Processor -- start 2024-06-09T22:18:51.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 -- start start 2024-06-09T22:18:51.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf68102770 0x7faf6819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf68103120 0x7faf6819ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf68107860 0x7faf68198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7faf68112770 con 0x7faf68103120 2024-06-09T22:18:51.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf681125f0 con 0x7faf68107860 2024-06-09T22:18:51.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.982+0000 7faf6ebdf700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faf681128f0 con 0x7faf68102770 2024-06-09T22:18:51.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf67fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf68103120 0x7faf6819ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf67fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf68103120 0x7faf6819ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33398/0 (socket says 172.21.15.104:33398) 2024-06-09T22:18:51.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf67fff700 1 -- 172.21.15.104:0/171638217 learned_addr learned my addr 172.21.15.104:0/171638217 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:51.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf6d17c700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf68107860 0x7faf68198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf6c97b700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf68102770 0x7faf6819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf67fff700 1 -- 172.21.15.104:0/171638217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf68102770 msgr2=0x7faf6819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf67fff700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf68102770 0x7faf6819e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf67fff700 1 -- 172.21.15.104:0/171638217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf68107860 msgr2=0x7faf68198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf67fff700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf68107860 0x7faf68198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf67fff700 1 -- 172.21.15.104:0/171638217 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf681990f0 con 0x7faf68103120 2024-06-09T22:18:51.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf6d17c700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf68107860 0x7faf68198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:18:51.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf67fff700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf68103120 0x7faf6819ecc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7faf54006010 tx=0x7faf54005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:51.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf65ffb700 1 -- 172.21.15.104:0/171638217 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf54027070 con 0x7faf68103120 2024-06-09T22:18:51.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf65ffb700 1 -- 172.21.15.104:0/171638217 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf540040d0 con 0x7faf68103120 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf6ebdf700 1 -- 172.21.15.104:0/171638217 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf68199380 con 0x7faf68103120 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf65ffb700 1 -- 172.21.15.104:0/171638217 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5400f460 con 0x7faf68103120 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.983+0000 7faf6ebdf700 1 -- 172.21.15.104:0/171638217 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf681a5650 con 0x7faf68103120 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.984+0000 7faf6ebdf700 1 -- 172.21.15.104:0/171638217 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faf68062c20 con 0x7faf68103120 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.986+0000 7faf65ffb700 1 -- 172.21.15.104:0/171638217 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7faf5402a070 con 0x7faf68103120 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.986+0000 7faf65ffb700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7faf580a2cb0 0x7faf580a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.986+0000 7faf6c97b700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7faf580a2cb0 0x7faf580a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.986+0000 7faf65ffb700 1 -- 172.21.15.104:0/171638217 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7faf540d0f10 con 0x7faf68103120 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.986+0000 7faf65ffb700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515] conn(0x7faf580adf10 0x7faf580b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.986+0000 7faf65ffb700 1 -- 172.21.15.104:0/171638217 --> [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf580b0a80 con 0x7faf580adf10 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.986+0000 7faf65ffb700 1 -- 172.21.15.104:0/171638217 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7faf540d1360 con 0x7faf68103120 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.987+0000 7faf6c97b700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7faf580a2cb0 0x7faf580a5170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faf5c005ea0 tx=0x7faf5c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:51.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.987+0000 7faf6d17c700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515] conn(0x7faf580adf10 0x7faf580b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.987+0000 7faf6d17c700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515] conn(0x7faf580adf10 0x7faf580b03b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.988+0000 7faf65ffb700 1 -- 172.21.15.104:0/171638217 <== osd.11 v2:172.21.15.194:6824/2145259515 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf580b0a80 con 0x7faf580adf10 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.005+0000 7faf6ebdf700 1 -- 172.21.15.104:0/171638217 --> [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faf680630a0 con 0x7faf580adf10 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.005+0000 7faf65ffb700 1 -- 172.21.15.104:0/171638217 <== osd.11 v2:172.21.15.194:6824/2145259515 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7faf680630a0 con 0x7faf580adf10 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.006+0000 7faf6ebdf700 1 -- 172.21.15.104:0/171638217 >> [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515] conn(0x7faf580adf10 msgr2=0x7faf580b03b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.006+0000 7faf6ebdf700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515] conn(0x7faf580adf10 0x7faf580b03b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.006+0000 7faf6ebdf700 1 -- 172.21.15.104:0/171638217 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7faf580a2cb0 msgr2=0x7faf580a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.006+0000 7faf6ebdf700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7faf580a2cb0 0x7faf580a5170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faf5c005ea0 tx=0x7faf5c005e30 comp rx=0 tx=0).stop 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.006+0000 7faf6ebdf700 1 -- 172.21.15.104:0/171638217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf68103120 msgr2=0x7faf6819ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.006+0000 7faf6ebdf700 1 --2- 172.21.15.104:0/171638217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf68103120 0x7faf6819ecc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7faf54006010 tx=0x7faf54005640 comp rx=0 tx=0).stop 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.006+0000 7faf6d17c700 1 -- 172.21.15.104:0/171638217 reap_dead start 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.006+0000 7faf6ebdf700 1 -- 172.21.15.104:0/171638217 shutdown_connections 2024-06-09T22:18:51.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.006+0000 7faf6ebdf700 1 -- 172.21.15.104:0/171638217 >> 172.21.15.104:0/171638217 conn(0x7faf68079600 msgr2=0x7faf681004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:51.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.006+0000 7faf6ebdf700 1 -- 172.21.15.104:0/171638217 shutdown_connections 2024-06-09T22:18:51.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:51.006+0000 7faf6ebdf700 1 -- 172.21.15.104:0/171638217 wait complete. 2024-06-09T22:18:51.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.403+0000 7fdc226d4700 1 -- 172.21.15.104:0/1228189558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc1c102540 msgr2=0x7fdc1c1029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.403+0000 7fdc226d4700 1 --2- 172.21.15.104:0/1228189558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc1c102540 0x7fdc1c1029a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009990 tx=0x7fdc10009ca0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.403+0000 7fdc226d4700 1 -- 172.21.15.104:0/1228189558 shutdown_connections 2024-06-09T22:18:51.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.403+0000 7fdc226d4700 1 --2- 172.21.15.104:0/1228189558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc1c102ee0 0x7fdc1c10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.403+0000 7fdc226d4700 1 --2- 172.21.15.104:0/1228189558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c107450 0x7fdc1c107830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.403+0000 7fdc226d4700 1 --2- 172.21.15.104:0/1228189558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc1c102540 0x7fdc1c1029a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.403+0000 7fdc226d4700 1 -- 172.21.15.104:0/1228189558 >> 172.21.15.104:0/1228189558 conn(0x7fdc1c0795f0 msgr2=0x7fdc1c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:51.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.403+0000 7fdc226d4700 1 -- 172.21.15.104:0/1228189558 shutdown_connections 2024-06-09T22:18:51.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.403+0000 7fdc226d4700 1 -- 172.21.15.104:0/1228189558 wait complete. 2024-06-09T22:18:51.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.403+0000 7fdc226d4700 1 Processor -- start 2024-06-09T22:18:51.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.403+0000 7fdc226d4700 1 -- start start 2024-06-09T22:18:51.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc226d4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c102540 0x7fdc1c19ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc226d4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc1c102ee0 0x7fdc1c19b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc226d4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc1c107450 0x7fdc1c19f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc226d4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1127e0 con 0x7fdc1c107450 2024-06-09T22:18:51.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc226d4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c112660 con 0x7fdc1c102ee0 2024-06-09T22:18:51.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc226d4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c112960 con 0x7fdc1c102540 2024-06-09T22:18:51.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc20c71700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc1c107450 0x7fdc1c19f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc20c71700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc1c107450 0x7fdc1c19f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33356/0 (socket says 172.21.15.104:33356) 2024-06-09T22:18:51.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc20c71700 1 -- 172.21.15.104:0/1650553150 learned_addr learned my addr 172.21.15.104:0/1650553150 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:51.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc1b7fe700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc1c102ee0 0x7fdc1c19b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc1bfff700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c102540 0x7fdc1c19ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc20c71700 1 -- 172.21.15.104:0/1650553150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c102540 msgr2=0x7fdc1c19ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc20c71700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c102540 0x7fdc1c19ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc20c71700 1 -- 172.21.15.104:0/1650553150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc1c102ee0 msgr2=0x7fdc1c19b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc20c71700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc1c102ee0 0x7fdc1c19b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc20c71700 1 -- 172.21.15.104:0/1650553150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c19fe90 con 0x7fdc1c107450 2024-06-09T22:18:51.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.404+0000 7fdc20c71700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc1c107450 0x7fdc1c19f750 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdc14007f10 tx=0x7fdc1400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:51.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.405+0000 7fdc197fa700 1 -- 172.21.15.104:0/1650553150 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14009e20 con 0x7fdc1c107450 2024-06-09T22:18:51.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.405+0000 7fdc197fa700 1 -- 172.21.15.104:0/1650553150 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc14004bb0 con 0x7fdc1c107450 2024-06-09T22:18:51.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.405+0000 7fdc226d4700 1 -- 172.21.15.104:0/1650553150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1c1a0180 con 0x7fdc1c107450 2024-06-09T22:18:51.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.405+0000 7fdc197fa700 1 -- 172.21.15.104:0/1650553150 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14005960 con 0x7fdc1c107450 2024-06-09T22:18:51.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.405+0000 7fdc226d4700 1 -- 172.21.15.104:0/1650553150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc1c1a7a00 con 0x7fdc1c107450 2024-06-09T22:18:51.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.406+0000 7fdc226d4700 1 -- 172.21.15.104:0/1650553150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdc1c100e10 con 0x7fdc1c107450 2024-06-09T22:18:51.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.407+0000 7fdc197fa700 1 -- 172.21.15.104:0/1650553150 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fdc1401a020 con 0x7fdc1c107450 2024-06-09T22:18:51.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.407+0000 7fdc197fa700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fdc040a2cb0 0x7fdc040a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.407+0000 7fdc1bfff700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fdc040a2cb0 0x7fdc040a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.407+0000 7fdc197fa700 1 -- 172.21.15.104:0/1650553150 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fdc140cf2b0 con 0x7fdc1c107450 2024-06-09T22:18:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.408+0000 7fdc197fa700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494] conn(0x7fdc040adf10 0x7fdc040b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.408+0000 7fdc197fa700 1 -- 172.21.15.104:0/1650553150 --> [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc040b0a80 con 0x7fdc040adf10 2024-06-09T22:18:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.408+0000 7fdc197fa700 1 -- 172.21.15.104:0/1650553150 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fdc140cf700 con 0x7fdc1c107450 2024-06-09T22:18:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.408+0000 7fdc1bfff700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fdc040a2cb0 0x7fdc040a5170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c005950 tx=0x7fdc0c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.408+0000 7fdc1b7fe700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494] conn(0x7fdc040adf10 0x7fdc040b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.408+0000 7fdc1b7fe700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494] conn(0x7fdc040adf10 0x7fdc040b03b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.409+0000 7fdc197fa700 1 -- 172.21.15.104:0/1650553150 <== osd.5 v2:172.21.15.194:6808/3138023494 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc040b0a80 con 0x7fdc040adf10 2024-06-09T22:18:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.425+0000 7fdc226d4700 1 -- 172.21.15.104:0/1650553150 --> [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdc1c103c00 con 0x7fdc040adf10 2024-06-09T22:18:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.426+0000 7fdc197fa700 1 -- 172.21.15.104:0/1650553150 <== osd.5 v2:172.21.15.194:6808/3138023494 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdc1c103c00 con 0x7fdc040adf10 2024-06-09T22:18:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.426+0000 7fdc226d4700 1 -- 172.21.15.104:0/1650553150 >> [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494] conn(0x7fdc040adf10 msgr2=0x7fdc040b03b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.426+0000 7fdc226d4700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494] conn(0x7fdc040adf10 0x7fdc040b03b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.426+0000 7fdc226d4700 1 -- 172.21.15.104:0/1650553150 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fdc040a2cb0 msgr2=0x7fdc040a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.426+0000 7fdc226d4700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fdc040a2cb0 0x7fdc040a5170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c005950 tx=0x7fdc0c00a300 comp rx=0 tx=0).stop 2024-06-09T22:18:51.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.426+0000 7fdc226d4700 1 -- 172.21.15.104:0/1650553150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc1c107450 msgr2=0x7fdc1c19f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.426+0000 7fdc226d4700 1 --2- 172.21.15.104:0/1650553150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc1c107450 0x7fdc1c19f750 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdc14007f10 tx=0x7fdc1400d3b0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.426+0000 7fdc20c71700 1 -- 172.21.15.104:0/1650553150 reap_dead start 2024-06-09T22:18:51.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.426+0000 7fdc226d4700 1 -- 172.21.15.104:0/1650553150 shutdown_connections 2024-06-09T22:18:51.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.426+0000 7fdc226d4700 1 -- 172.21.15.104:0/1650553150 >> 172.21.15.104:0/1650553150 conn(0x7fdc1c0795f0 msgr2=0x7fdc1c1003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:51.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.427+0000 7fdc226d4700 1 -- 172.21.15.104:0/1650553150 shutdown_connections 2024-06-09T22:18:51.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:49.427+0000 7fdc226d4700 1 -- 172.21.15.104:0/1650553150 wait complete. 2024-06-09T22:18:51.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.417+0000 7ff3edf21700 1 -- 172.21.15.104:0/3864279231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3e806b5c0 msgr2=0x7ff3e810f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.417+0000 7ff3edf21700 1 --2- 172.21.15.104:0/3864279231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3e806b5c0 0x7ff3e810f370 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000b0a0 tx=0x7ff3e000b3b0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.417+0000 7ff3edf21700 1 -- 172.21.15.104:0/3864279231 shutdown_connections 2024-06-09T22:18:51.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.417+0000 7ff3edf21700 1 --2- 172.21.15.104:0/3864279231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3e806b5c0 0x7ff3e810f370 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.417+0000 7ff3edf21700 1 --2- 172.21.15.104:0/3864279231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e8108530 0x7ff3e8108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.417+0000 7ff3edf21700 1 --2- 172.21.15.104:0/3864279231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3e806ac00 0x7ff3e806b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.417+0000 7ff3edf21700 1 -- 172.21.15.104:0/3864279231 >> 172.21.15.104:0/3864279231 conn(0x7ff3e8077400 msgr2=0x7ff3e8077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:51.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.417+0000 7ff3edf21700 1 -- 172.21.15.104:0/3864279231 shutdown_connections 2024-06-09T22:18:51.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.417+0000 7ff3edf21700 1 -- 172.21.15.104:0/3864279231 wait complete. 2024-06-09T22:18:51.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.417+0000 7ff3edf21700 1 Processor -- start 2024-06-09T22:18:51.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3edf21700 1 -- start start 2024-06-09T22:18:51.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3edf21700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e806ac00 0x7ff3e819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3edf21700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3e806b5c0 0x7ff3e819ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3edf21700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3e8108530 0x7ff3e8198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3edf21700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8112760 con 0x7ff3e8108530 2024-06-09T22:18:51.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3edf21700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e81125e0 con 0x7ff3e806ac00 2024-06-09T22:18:51.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3edf21700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e81128e0 con 0x7ff3e806b5c0 2024-06-09T22:18:51.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3e7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3e8108530 0x7ff3e8198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3e7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3e8108530 0x7ff3e8198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33370/0 (socket says 172.21.15.104:33370) 2024-06-09T22:18:51.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3e7fff700 1 -- 172.21.15.104:0/647777881 learned_addr learned my addr 172.21.15.104:0/647777881 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:51.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3e6ffd700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3e806b5c0 0x7ff3e819ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3e77fe700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e806ac00 0x7ff3e819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3e7fff700 1 -- 172.21.15.104:0/647777881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3e806b5c0 msgr2=0x7ff3e819ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3e7fff700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3e806b5c0 0x7ff3e819ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3e7fff700 1 -- 172.21.15.104:0/647777881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e806ac00 msgr2=0x7ff3e819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3e7fff700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e806ac00 0x7ff3e819e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3e7fff700 1 -- 172.21.15.104:0/647777881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e81990a0 con 0x7ff3e8108530 2024-06-09T22:18:51.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.418+0000 7ff3e7fff700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3e8108530 0x7ff3e8198940 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00062a0 tx=0x7ff3e0004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:51.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.419+0000 7ff3e4ff9700 1 -- 172.21.15.104:0/647777881 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0021070 con 0x7ff3e8108530 2024-06-09T22:18:51.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.419+0000 7ff3e4ff9700 1 -- 172.21.15.104:0/647777881 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3e0007c80 con 0x7ff3e8108530 2024-06-09T22:18:51.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.419+0000 7ff3edf21700 1 -- 172.21.15.104:0/647777881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e8199330 con 0x7ff3e8108530 2024-06-09T22:18:51.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.419+0000 7ff3e4ff9700 1 -- 172.21.15.104:0/647777881 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e001d450 con 0x7ff3e8108530 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.419+0000 7ff3edf21700 1 -- 172.21.15.104:0/647777881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e81a5650 con 0x7ff3e8108530 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.419+0000 7ff3edf21700 1 -- 172.21.15.104:0/647777881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff3e8100890 con 0x7ff3e8108530 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.421+0000 7ff3e4ff9700 1 -- 172.21.15.104:0/647777881 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff3e0004000 con 0x7ff3e8108530 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.421+0000 7ff3e4ff9700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff3d00a2d30 0x7ff3d00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.422+0000 7ff3e4ff9700 1 -- 172.21.15.104:0/647777881 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7ff3e000f020 con 0x7ff3e8108530 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.422+0000 7ff3e77fe700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff3d00a2d30 0x7ff3d00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.422+0000 7ff3e4ff9700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419] conn(0x7ff3d00adf90 0x7ff3d00b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.422+0000 7ff3e4ff9700 1 -- 172.21.15.104:0/647777881 --> [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3e001eec0 con 0x7ff3d00adf90 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.422+0000 7ff3e4ff9700 1 -- 172.21.15.104:0/647777881 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7ff3e00d1290 con 0x7ff3e8108530 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.422+0000 7ff3e6ffd700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419] conn(0x7ff3d00adf90 0x7ff3d00b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.422+0000 7ff3e77fe700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff3d00a2d30 0x7ff3d00a51f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8199a80 tx=0x7ff3dc00b4d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.422+0000 7ff3e6ffd700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419] conn(0x7ff3d00adf90 0x7ff3d00b0430 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.423+0000 7ff3e4ff9700 1 -- 172.21.15.104:0/647777881 <== osd.6 v2:172.21.15.104:6818/4097161419 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3e001eec0 con 0x7ff3d00adf90 2024-06-09T22:18:51.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.440+0000 7ff3edf21700 1 -- 172.21.15.104:0/647777881 --> [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff3e8103bf0 con 0x7ff3d00adf90 2024-06-09T22:18:51.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.440+0000 7ff3e4ff9700 1 -- 172.21.15.104:0/647777881 <== osd.6 v2:172.21.15.104:6818/4097161419 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff3e8103bf0 con 0x7ff3d00adf90 2024-06-09T22:18:51.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.441+0000 7ff3edf21700 1 -- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419] conn(0x7ff3d00adf90 msgr2=0x7ff3d00b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.441+0000 7ff3edf21700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419] conn(0x7ff3d00adf90 0x7ff3d00b0430 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.441+0000 7ff3edf21700 1 -- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff3d00a2d30 msgr2=0x7ff3d00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.441+0000 7ff3edf21700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff3d00a2d30 0x7ff3d00a51f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8199a80 tx=0x7ff3dc00b4d0 comp rx=0 tx=0).stop 2024-06-09T22:18:51.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.441+0000 7ff3edf21700 1 -- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3e8108530 msgr2=0x7ff3e8198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:51.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.441+0000 7ff3edf21700 1 --2- 172.21.15.104:0/647777881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3e8108530 0x7ff3e8198940 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00062a0 tx=0x7ff3e0004e20 comp rx=0 tx=0).stop 2024-06-09T22:18:51.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.441+0000 7ff3e7fff700 1 -- 172.21.15.104:0/647777881 reap_dead start 2024-06-09T22:18:51.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.441+0000 7ff3edf21700 1 -- 172.21.15.104:0/647777881 shutdown_connections 2024-06-09T22:18:51.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.441+0000 7ff3edf21700 1 -- 172.21.15.104:0/647777881 >> 172.21.15.104:0/647777881 conn(0x7ff3e8077400 msgr2=0x7ff3e80ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:51.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.441+0000 7ff3edf21700 1 -- 172.21.15.104:0/647777881 shutdown_connections 2024-06-09T22:18:51.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:50.441+0000 7ff3edf21700 1 -- 172.21.15.104:0/647777881 wait complete. 2024-06-09T22:18:52.504 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:52 smithi104 ceph-mon[32389]: pgmap v372: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-06-09T22:18:52.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:52 smithi153 ceph-mon[36599]: pgmap v372: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-06-09T22:18:52.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:52 smithi194 ceph-mon[36856]: pgmap v372: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-06-09T22:18:53.441 INFO:teuthology.orchestra.run.smithi104.stdout:163208757320 2024-06-09T22:18:53.441 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-09T22:18:53.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.651+0000 7f6ef573d700 1 -- 172.21.15.104:0/2184483633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef006a170 msgr2=0x7f6ef006a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:53.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.651+0000 7f6ef573d700 1 --2- 172.21.15.104:0/2184483633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef006a170 0x7f6ef006a5f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4009990 tx=0x7f6ee4009ca0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.651+0000 7f6ef573d700 1 -- 172.21.15.104:0/2184483633 shutdown_connections 2024-06-09T22:18:53.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.651+0000 7f6ef573d700 1 --2- 172.21.15.104:0/2184483633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ef006ab30 0x7f6ef010f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.651+0000 7f6ef573d700 1 --2- 172.21.15.104:0/2184483633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ef0108500 0x7f6ef01088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.651+0000 7f6ef573d700 1 --2- 172.21.15.104:0/2184483633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef006a170 0x7f6ef006a5f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.651+0000 7f6ef573d700 1 -- 172.21.15.104:0/2184483633 >> 172.21.15.104:0/2184483633 conn(0x7f6ef0078340 msgr2=0x7f6ef0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.651+0000 7f6ef573d700 1 -- 172.21.15.104:0/2184483633 shutdown_connections 2024-06-09T22:18:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.651+0000 7f6ef573d700 1 -- 172.21.15.104:0/2184483633 wait complete. 2024-06-09T22:18:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.651+0000 7f6ef573d700 1 Processor -- start 2024-06-09T22:18:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6ef573d700 1 -- start start 2024-06-09T22:18:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6ef573d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef006a170 0x7f6ef019e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6ef573d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ef006ab30 0x7f6ef019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6ef573d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ef0108500 0x7f6ef0198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6ef573d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef0112760 con 0x7f6ef006a170 2024-06-09T22:18:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6ef573d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef01125e0 con 0x7f6ef006ab30 2024-06-09T22:18:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6ef573d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef01128e0 con 0x7f6ef0108500 2024-06-09T22:18:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6eeeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef006a170 0x7f6ef019e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6eeeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef006a170 0x7f6ef019e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33412/0 (socket says 172.21.15.104:33412) 2024-06-09T22:18:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6eeeffd700 1 -- 172.21.15.104:0/2166084911 learned_addr learned my addr 172.21.15.104:0/2166084911 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6eee7fc700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ef006ab30 0x7f6ef019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6eef7fe700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ef0108500 0x7f6ef0198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6eeeffd700 1 -- 172.21.15.104:0/2166084911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ef0108500 msgr2=0x7f6ef0198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6eeeffd700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ef0108500 0x7f6ef0198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6eeeffd700 1 -- 172.21.15.104:0/2166084911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ef006ab30 msgr2=0x7f6ef019edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6eeeffd700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ef006ab30 0x7f6ef019edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6eeeffd700 1 -- 172.21.15.104:0/2166084911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef01991a0 con 0x7f6ef006a170 2024-06-09T22:18:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.652+0000 7f6eeeffd700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef006a170 0x7f6ef019e880 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000d980 tx=0x7f6ee000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.653+0000 7f6ed7fff700 1 -- 172.21.15.104:0/2166084911 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0016070 con 0x7f6ef006a170 2024-06-09T22:18:53.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.653+0000 7f6ef573d700 1 -- 172.21.15.104:0/2166084911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef0199490 con 0x7f6ef006a170 2024-06-09T22:18:53.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.653+0000 7f6ed7fff700 1 -- 172.21.15.104:0/2166084911 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ee0004d10 con 0x7f6ef006a170 2024-06-09T22:18:53.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.653+0000 7f6ed7fff700 1 -- 172.21.15.104:0/2166084911 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00055f0 con 0x7f6ef006a170 2024-06-09T22:18:53.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.653+0000 7f6ef573d700 1 -- 172.21.15.104:0/2166084911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef01a5750 con 0x7f6ef006a170 2024-06-09T22:18:53.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.654+0000 7f6ef573d700 1 -- 172.21.15.104:0/2166084911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6ef0062c20 con 0x7f6ef006a170 2024-06-09T22:18:53.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.655+0000 7f6ed7fff700 1 -- 172.21.15.104:0/2166084911 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f6ee0004750 con 0x7f6ef006a170 2024-06-09T22:18:53.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.655+0000 7f6ed7fff700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6ed80a2cb0 0x7f6ed80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:53.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.655+0000 7f6eee7fc700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6ed80a2cb0 0x7f6ed80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:53.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.655+0000 7f6ed7fff700 1 -- 172.21.15.104:0/2166084911 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f6ee00cfa10 con 0x7f6ef006a170 2024-06-09T22:18:53.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.656+0000 7f6ed7fff700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505] conn(0x7f6ed80adf10 0x7f6ed80b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:53.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.656+0000 7f6ed7fff700 1 -- 172.21.15.104:0/2166084911 --> [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ee0015ec0 con 0x7f6ed80adf10 2024-06-09T22:18:53.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.656+0000 7f6ed7fff700 1 -- 172.21.15.104:0/2166084911 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f6ee00cfe60 con 0x7f6ef006a170 2024-06-09T22:18:53.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.656+0000 7f6eef7fe700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505] conn(0x7f6ed80adf10 0x7f6ed80b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:53.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.656+0000 7f6eee7fc700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6ed80a2cb0 0x7f6ed80a5170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400b2e0 tx=0x7f6ee4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:53.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.656+0000 7f6eef7fe700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505] conn(0x7f6ed80adf10 0x7f6ed80b03b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:53.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.657+0000 7f6ed7fff700 1 -- 172.21.15.104:0/2166084911 <== osd.0 v2:172.21.15.104:6802/2202624505 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ee0015ec0 con 0x7f6ed80adf10 2024-06-09T22:18:53.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.682+0000 7f6ef573d700 1 -- 172.21.15.104:0/2166084911 --> [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6ef00630a0 con 0x7f6ed80adf10 2024-06-09T22:18:53.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.682+0000 7f6ed7fff700 1 -- 172.21.15.104:0/2166084911 <== osd.0 v2:172.21.15.104:6802/2202624505 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6ef00630a0 con 0x7f6ed80adf10 2024-06-09T22:18:53.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.683+0000 7f6ef573d700 1 -- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505] conn(0x7f6ed80adf10 msgr2=0x7f6ed80b03b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:53.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.683+0000 7f6ef573d700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505] conn(0x7f6ed80adf10 0x7f6ed80b03b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.683+0000 7f6ef573d700 1 -- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6ed80a2cb0 msgr2=0x7f6ed80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:53.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.683+0000 7f6ef573d700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6ed80a2cb0 0x7f6ed80a5170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400b2e0 tx=0x7f6ee4005960 comp rx=0 tx=0).stop 2024-06-09T22:18:53.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.683+0000 7f6ef573d700 1 -- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef006a170 msgr2=0x7f6ef019e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:53.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.683+0000 7f6ef573d700 1 --2- 172.21.15.104:0/2166084911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef006a170 0x7f6ef019e880 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000d980 tx=0x7f6ee000dc90 comp rx=0 tx=0).stop 2024-06-09T22:18:53.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.683+0000 7f6eef7fe700 1 -- 172.21.15.104:0/2166084911 reap_dead start 2024-06-09T22:18:53.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.683+0000 7f6ef573d700 1 -- 172.21.15.104:0/2166084911 shutdown_connections 2024-06-09T22:18:53.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.683+0000 7f6ef573d700 1 -- 172.21.15.104:0/2166084911 >> 172.21.15.104:0/2166084911 conn(0x7f6ef0078340 msgr2=0x7f6ef0100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:53.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.683+0000 7f6ef573d700 1 -- 172.21.15.104:0/2166084911 shutdown_connections 2024-06-09T22:18:53.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.683+0000 7f6ef573d700 1 -- 172.21.15.104:0/2166084911 wait complete. 2024-06-09T22:18:53.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.890+0000 7fabe8531700 1 -- 172.21.15.104:0/3743509245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabe01077b0 msgr2=0x7fabe010f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:53.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.890+0000 7fabe8531700 1 --2- 172.21.15.104:0/3743509245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabe01077b0 0x7fabe010f3c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fabd800b0d0 tx=0x7fabd800b3e0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.890+0000 7fabe8531700 1 -- 172.21.15.104:0/3743509245 shutdown_connections 2024-06-09T22:18:53.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.890+0000 7fabe8531700 1 --2- 172.21.15.104:0/3743509245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabe01077b0 0x7fabe010f3c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.890+0000 7fabe8531700 1 --2- 172.21.15.104:0/3743509245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe006ae60 0x7fabe006b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.890+0000 7fabe8531700 1 --2- 172.21.15.104:0/3743509245 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabe006b810 0x7fabe01070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.890+0000 7fabe8531700 1 -- 172.21.15.104:0/3743509245 >> 172.21.15.104:0/3743509245 conn(0x7fabe00773e0 msgr2=0x7fabe00777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:53.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.890+0000 7fabe8531700 1 -- 172.21.15.104:0/3743509245 shutdown_connections 2024-06-09T22:18:53.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.890+0000 7fabe8531700 1 -- 172.21.15.104:0/3743509245 wait complete. 2024-06-09T22:18:53.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe8531700 1 Processor -- start 2024-06-09T22:18:53.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe8531700 1 -- start start 2024-06-09T22:18:53.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe8531700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabe006ae60 0x7fabe019e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:53.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe8531700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabe006b810 0x7fabe019ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:53.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe8531700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe01077b0 0x7fabe0198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:53.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe8531700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fabe0112850 con 0x7fabe006ae60 2024-06-09T22:18:53.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe8531700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fabe01126d0 con 0x7fabe01077b0 2024-06-09T22:18:53.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe8531700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fabe01129d0 con 0x7fabe006b810 2024-06-09T22:18:53.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe62cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabe006ae60 0x7fabe019e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:53.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe62cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabe006ae60 0x7fabe019e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33432/0 (socket says 172.21.15.104:33432) 2024-06-09T22:18:53.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe62cd700 1 -- 172.21.15.104:0/1925545593 learned_addr learned my addr 172.21.15.104:0/1925545593 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:53.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe6ace700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe01077b0 0x7fabe0198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:53.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe5acc700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabe006b810 0x7fabe019ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe62cd700 1 -- 172.21.15.104:0/1925545593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabe006b810 msgr2=0x7fabe019ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe62cd700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fabe006b810 0x7fabe019ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe62cd700 1 -- 172.21.15.104:0/1925545593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe01077b0 msgr2=0x7fabe0198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.891+0000 7fabe62cd700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe01077b0 0x7fabe0198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.892+0000 7fabe62cd700 1 -- 172.21.15.104:0/1925545593 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe0199210 con 0x7fabe006ae60 2024-06-09T22:18:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.892+0000 7fabe62cd700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabe006ae60 0x7fabe019e8a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fabd000d950 tx=0x7fabd000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.892+0000 7fabd77fe700 1 -- 172.21.15.104:0/1925545593 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd0016070 con 0x7fabe006ae60 2024-06-09T22:18:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.892+0000 7fabe8531700 1 -- 172.21.15.104:0/1925545593 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe0199500 con 0x7fabe006ae60 2024-06-09T22:18:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.892+0000 7fabd77fe700 1 -- 172.21.15.104:0/1925545593 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabd000fd70 con 0x7fabe006ae60 2024-06-09T22:18:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.892+0000 7fabd77fe700 1 -- 172.21.15.104:0/1925545593 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd000e610 con 0x7fabe006ae60 2024-06-09T22:18:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.892+0000 7fabe8531700 1 -- 172.21.15.104:0/1925545593 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe01a5770 con 0x7fabe006ae60 2024-06-09T22:18:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.893+0000 7fabe8531700 1 -- 172.21.15.104:0/1925545593 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fabe0062c20 con 0x7fabe006ae60 2024-06-09T22:18:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.894+0000 7fabd77fe700 1 -- 172.21.15.104:0/1925545593 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fabd0019070 con 0x7fabe006ae60 2024-06-09T22:18:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.895+0000 7fabd77fe700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fabcc0a2d30 0x7fabcc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.895+0000 7fabe5acc700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fabcc0a2d30 0x7fabcc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.895+0000 7fabd77fe700 1 -- 172.21.15.104:0/1925545593 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fabd00cecd0 con 0x7fabe006ae60 2024-06-09T22:18:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.895+0000 7fabd77fe700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981] conn(0x7fabcc0adf90 0x7fabcc0b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.895+0000 7fabd77fe700 1 -- 172.21.15.104:0/1925545593 --> [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fabd0013ec0 con 0x7fabcc0adf90 2024-06-09T22:18:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.895+0000 7fabd77fe700 1 -- 172.21.15.104:0/1925545593 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fabd00d1390 con 0x7fabe006ae60 2024-06-09T22:18:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.895+0000 7fabe5acc700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fabcc0a2d30 0x7fabcc0a51f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fabe0108c00 tx=0x7fabdc00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.895+0000 7fabe6ace700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981] conn(0x7fabcc0adf90 0x7fabcc0b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.896+0000 7fabe6ace700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981] conn(0x7fabcc0adf90 0x7fabcc0b0430 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.896+0000 7fabd77fe700 1 -- 172.21.15.104:0/1925545593 <== osd.4 v2:172.21.15.153:6808/1237651981 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fabd0013ec0 con 0x7fabcc0adf90 2024-06-09T22:18:53.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.912+0000 7fabe8531700 1 -- 172.21.15.104:0/1925545593 --> [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fabe0063050 con 0x7fabcc0adf90 2024-06-09T22:18:53.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.913+0000 7fabd77fe700 1 -- 172.21.15.104:0/1925545593 <== osd.4 v2:172.21.15.153:6808/1237651981 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fabe0063050 con 0x7fabcc0adf90 2024-06-09T22:18:53.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.913+0000 7fabe8531700 1 -- 172.21.15.104:0/1925545593 >> [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981] conn(0x7fabcc0adf90 msgr2=0x7fabcc0b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:53.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.913+0000 7fabe8531700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981] conn(0x7fabcc0adf90 0x7fabcc0b0430 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.913+0000 7fabe8531700 1 -- 172.21.15.104:0/1925545593 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fabcc0a2d30 msgr2=0x7fabcc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:53.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.913+0000 7fabe8531700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fabcc0a2d30 0x7fabcc0a51f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fabe0108c00 tx=0x7fabdc00b3c0 comp rx=0 tx=0).stop 2024-06-09T22:18:53.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.913+0000 7fabe8531700 1 -- 172.21.15.104:0/1925545593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabe006ae60 msgr2=0x7fabe019e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:53.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.913+0000 7fabe8531700 1 --2- 172.21.15.104:0/1925545593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabe006ae60 0x7fabe019e8a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fabd000d950 tx=0x7fabd000dc60 comp rx=0 tx=0).stop 2024-06-09T22:18:53.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.913+0000 7fabe6ace700 1 -- 172.21.15.104:0/1925545593 reap_dead start 2024-06-09T22:18:53.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.913+0000 7fabe8531700 1 -- 172.21.15.104:0/1925545593 shutdown_connections 2024-06-09T22:18:53.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.913+0000 7fabe8531700 1 -- 172.21.15.104:0/1925545593 >> 172.21.15.104:0/1925545593 conn(0x7fabe00773e0 msgr2=0x7fabe00ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:53.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.913+0000 7fabe8531700 1 -- 172.21.15.104:0/1925545593 shutdown_connections 2024-06-09T22:18:53.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:53.913+0000 7fabe8531700 1 -- 172.21.15.104:0/1925545593 wait complete. 2024-06-09T22:18:54.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:54 smithi153 ceph-mon[36599]: pgmap v373: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:18:54.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:54 smithi104 ceph-mon[32389]: pgmap v373: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:18:54.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:54 smithi194 ceph-mon[36856]: pgmap v373: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:18:55.248 INFO:teuthology.orchestra.run.smithi104.stdout:219043332157 2024-06-09T22:18:55.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-09T22:18:55.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.924+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/1751290952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd4107390 msgr2=0x7f8cd4107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:55.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.924+0000 7f8cd9f5d700 1 --2- 172.21.15.104:0/1751290952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd4107390 0x7f8cd4107770 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40099c0 tx=0x7f8cc4009cd0 comp rx=0 tx=0).stop 2024-06-09T22:18:55.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.924+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/1751290952 shutdown_connections 2024-06-09T22:18:55.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.924+0000 7f8cd9f5d700 1 --2- 172.21.15.104:0/1751290952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd4107390 0x7f8cd4107770 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:55.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.924+0000 7f8cd9f5d700 1 --2- 172.21.15.104:0/1751290952 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cd406b5b0 0x7f8cd410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:55.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.924+0000 7f8cd9f5d700 1 --2- 172.21.15.104:0/1751290952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd406abf0 0x7f8cd406b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:55.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.924+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/1751290952 >> 172.21.15.104:0/1751290952 conn(0x7f8cd4078340 msgr2=0x7f8cd4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:55.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.924+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/1751290952 shutdown_connections 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.924+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/1751290952 wait complete. 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.924+0000 7f8cd9f5d700 1 Processor -- start 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd9f5d700 1 -- start start 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd9f5d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd406abf0 0x7f8cd419e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd9f5d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd406b5b0 0x7f8cd419eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd9f5d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cd4107390 0x7f8cd4198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd9f5d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd4112770 con 0x7f8cd406b5b0 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd9f5d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd41125f0 con 0x7f8cd406abf0 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd9f5d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd41128f0 con 0x7f8cd4107390 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd406b5b0 0x7f8cd419eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd406b5b0 0x7f8cd419eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33464/0 (socket says 172.21.15.104:33464) 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd2ffd700 1 -- 172.21.15.104:0/3289935995 learned_addr learned my addr 172.21.15.104:0/3289935995 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd37fe700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd406abf0 0x7f8cd419e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:55.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd3fff700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cd4107390 0x7f8cd4198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd2ffd700 1 -- 172.21.15.104:0/3289935995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cd4107390 msgr2=0x7f8cd4198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd2ffd700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cd4107390 0x7f8cd4198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd2ffd700 1 -- 172.21.15.104:0/3289935995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd406abf0 msgr2=0x7f8cd419e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd2ffd700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd406abf0 0x7f8cd419e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd2ffd700 1 -- 172.21.15.104:0/3289935995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd41991d0 con 0x7f8cd406b5b0 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd37fe700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd406abf0 0x7f8cd419e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.925+0000 7f8cd3fff700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cd4107390 0x7f8cd4198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.926+0000 7f8cd2ffd700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd406b5b0 0x7f8cd419eda0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8cc800c920 tx=0x7f8cc800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.926+0000 7f8cd0ff9700 1 -- 172.21.15.104:0/3289935995 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8007b00 con 0x7f8cd406b5b0 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.926+0000 7f8cd0ff9700 1 -- 172.21.15.104:0/3289935995 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cc8004d10 con 0x7f8cd406b5b0 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.926+0000 7f8cd0ff9700 1 -- 172.21.15.104:0/3289935995 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc800f450 con 0x7f8cd406b5b0 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.926+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/3289935995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cd41994c0 con 0x7f8cd406b5b0 2024-06-09T22:18:55.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.926+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/3289935995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cd41a5730 con 0x7f8cd406b5b0 2024-06-09T22:18:55.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.927+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/3289935995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8cd4062be0 con 0x7f8cd406b5b0 2024-06-09T22:18:55.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.928+0000 7f8cd0ff9700 1 -- 172.21.15.104:0/3289935995 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f8cc8004750 con 0x7f8cd406b5b0 2024-06-09T22:18:55.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.928+0000 7f8cd0ff9700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f8cbc0a2d30 0x7f8cbc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:55.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.929+0000 7f8cd37fe700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f8cbc0a2d30 0x7f8cbc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:55.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.929+0000 7f8cd0ff9700 1 -- 172.21.15.104:0/3289935995 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f8cc80cf6e0 con 0x7f8cd406b5b0 2024-06-09T22:18:55.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.929+0000 7f8cd0ff9700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084] conn(0x7f8cbc0adf90 0x7f8cbc0b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:55.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.929+0000 7f8cd0ff9700 1 -- 172.21.15.104:0/3289935995 --> [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8cc8012ec0 con 0x7f8cbc0adf90 2024-06-09T22:18:55.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.929+0000 7f8cd0ff9700 1 -- 172.21.15.104:0/3289935995 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f8cc80cfb30 con 0x7f8cd406b5b0 2024-06-09T22:18:55.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.929+0000 7f8cd37fe700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f8cbc0a2d30 0x7f8cbc0a51f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400b580 tx=0x7f8cc4003780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:55.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.929+0000 7f8cd3fff700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084] conn(0x7f8cbc0adf90 0x7f8cbc0b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.930+0000 7f8cd3fff700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084] conn(0x7f8cbc0adf90 0x7f8cbc0b0430 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.931+0000 7f8cd0ff9700 1 -- 172.21.15.104:0/3289935995 <== osd.2 v2:172.21.15.194:6800/120935084 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8cc8012ec0 con 0x7f8cbc0adf90 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.947+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/3289935995 --> [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8cd4062fe0 con 0x7f8cbc0adf90 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.947+0000 7f8cd0ff9700 1 -- 172.21.15.104:0/3289935995 <== osd.2 v2:172.21.15.194:6800/120935084 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8cd4062fe0 con 0x7f8cbc0adf90 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.948+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/3289935995 >> [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084] conn(0x7f8cbc0adf90 msgr2=0x7f8cbc0b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.948+0000 7f8cd9f5d700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084] conn(0x7f8cbc0adf90 0x7f8cbc0b0430 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.948+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/3289935995 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f8cbc0a2d30 msgr2=0x7f8cbc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.948+0000 7f8cd9f5d700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f8cbc0a2d30 0x7f8cbc0a51f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400b580 tx=0x7f8cc4003780 comp rx=0 tx=0).stop 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.948+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/3289935995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd406b5b0 msgr2=0x7f8cd419eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.948+0000 7f8cd9f5d700 1 --2- 172.21.15.104:0/3289935995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd406b5b0 0x7f8cd419eda0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8cc800c920 tx=0x7f8cc800cce0 comp rx=0 tx=0).stop 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.948+0000 7f8cd3fff700 1 -- 172.21.15.104:0/3289935995 reap_dead start 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.948+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/3289935995 shutdown_connections 2024-06-09T22:18:55.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.948+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/3289935995 >> 172.21.15.104:0/3289935995 conn(0x7f8cd4078340 msgr2=0x7f8cd40ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:55.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.948+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/3289935995 shutdown_connections 2024-06-09T22:18:55.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.948+0000 7f8cd9f5d700 1 -- 172.21.15.104:0/3289935995 wait complete. 2024-06-09T22:18:56.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.346+0000 7fbb4515a700 1 -- 172.21.15.104:0/2575113046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb40101fd0 msgr2=0x7fbb40105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:56.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.346+0000 7fbb4515a700 1 --2- 172.21.15.104:0/2575113046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb40101fd0 0x7fbb40105fd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbb340099c0 tx=0x7fbb34009cd0 comp rx=0 tx=0).stop 2024-06-09T22:18:56.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.346+0000 7fbb4515a700 1 -- 172.21.15.104:0/2575113046 shutdown_connections 2024-06-09T22:18:56.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.346+0000 7fbb4515a700 1 --2- 172.21.15.104:0/2575113046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb401066b0 0x7fbb4010e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:56.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.346+0000 7fbb4515a700 1 --2- 172.21.15.104:0/2575113046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb40101620 0x7fbb40101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:56.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.346+0000 7fbb4515a700 1 --2- 172.21.15.104:0/2575113046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb40101fd0 0x7fbb40105fd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:56.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.346+0000 7fbb4515a700 1 -- 172.21.15.104:0/2575113046 >> 172.21.15.104:0/2575113046 conn(0x7fbb40078360 msgr2=0x7fbb40078770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:56.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.346+0000 7fbb4515a700 1 -- 172.21.15.104:0/2575113046 shutdown_connections 2024-06-09T22:18:56.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.346+0000 7fbb4515a700 1 -- 172.21.15.104:0/2575113046 wait complete. 2024-06-09T22:18:56.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.346+0000 7fbb4515a700 1 Processor -- start 2024-06-09T22:18:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb4515a700 1 -- start start 2024-06-09T22:18:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb4515a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb40101620 0x7fbb40199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb4515a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb40101fd0 0x7fbb4019a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb4515a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb401066b0 0x7fbb401a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb4515a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbb401116f0 con 0x7fbb40101fd0 2024-06-09T22:18:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb4515a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbb40111570 con 0x7fbb401066b0 2024-06-09T22:18:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb4515a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbb40111870 con 0x7fbb40101620 2024-06-09T22:18:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb3e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb40101fd0 0x7fbb4019a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb3e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb40101fd0 0x7fbb4019a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38864/0 (socket says 172.21.15.104:38864) 2024-06-09T22:18:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb3e59c700 1 -- 172.21.15.104:0/2695972881 learned_addr learned my addr 172.21.15.104:0/2695972881 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb3f59e700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb401066b0 0x7fbb401a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:56.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb3ed9d700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb40101620 0x7fbb40199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:56.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb3e59c700 1 -- 172.21.15.104:0/2695972881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb40101620 msgr2=0x7fbb40199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:56.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb3e59c700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb40101620 0x7fbb40199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:56.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb3e59c700 1 -- 172.21.15.104:0/2695972881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb401066b0 msgr2=0x7fbb401a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:56.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb3e59c700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb401066b0 0x7fbb401a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:56.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb3e59c700 1 -- 172.21.15.104:0/2695972881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb401a5090 con 0x7fbb40101fd0 2024-06-09T22:18:56.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.347+0000 7fbb3e59c700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb40101fd0 0x7fbb4019a010 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbb340059d0 tx=0x7fbb340038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:56.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.348+0000 7fbb27fff700 1 -- 172.21.15.104:0/2695972881 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb34027070 con 0x7fbb40101fd0 2024-06-09T22:18:56.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.348+0000 7fbb4515a700 1 -- 172.21.15.104:0/2695972881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb401a5260 con 0x7fbb40101fd0 2024-06-09T22:18:56.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.348+0000 7fbb27fff700 1 -- 172.21.15.104:0/2695972881 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb34004050 con 0x7fbb40101fd0 2024-06-09T22:18:56.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.348+0000 7fbb27fff700 1 -- 172.21.15.104:0/2695972881 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb34022cf0 con 0x7fbb40101fd0 2024-06-09T22:18:56.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.348+0000 7fbb4515a700 1 -- 172.21.15.104:0/2695972881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb401a5710 con 0x7fbb40101fd0 2024-06-09T22:18:56.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.348+0000 7fbb4515a700 1 -- 172.21.15.104:0/2695972881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbb40062ba0 con 0x7fbb40101fd0 2024-06-09T22:18:56.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.350+0000 7fbb27fff700 1 -- 172.21.15.104:0/2695972881 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fbb3402a070 con 0x7fbb40101fd0 2024-06-09T22:18:56.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.350+0000 7fbb27fff700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbb280a2cb0 0x7fbb280a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:56.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.351+0000 7fbb3ed9d700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbb280a2cb0 0x7fbb280a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:56.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.351+0000 7fbb27fff700 1 -- 172.21.15.104:0/2695972881 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fbb340d10f0 con 0x7fbb40101fd0 2024-06-09T22:18:56.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.351+0000 7fbb27fff700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298] conn(0x7fbb280adf10 0x7fbb280b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:56.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.351+0000 7fbb27fff700 1 -- 172.21.15.104:0/2695972881 --> [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb280b0a80 con 0x7fbb280adf10 2024-06-09T22:18:56.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.351+0000 7fbb27fff700 1 -- 172.21.15.104:0/2695972881 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fbb340d1540 con 0x7fbb40101fd0 2024-06-09T22:18:56.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.351+0000 7fbb3f59e700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298] conn(0x7fbb280adf10 0x7fbb280b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:56.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.351+0000 7fbb3ed9d700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbb280a2cb0 0x7fbb280a5170 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009a10 tx=0x7fbb30009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:56.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.351+0000 7fbb3f59e700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298] conn(0x7fbb280adf10 0x7fbb280b03b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:56.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.352+0000 7fbb27fff700 1 -- 172.21.15.104:0/2695972881 <== osd.9 v2:172.21.15.104:6826/3652005298 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbb280b0a80 con 0x7fbb280adf10 2024-06-09T22:18:56.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.368+0000 7fbb4515a700 1 -- 172.21.15.104:0/2695972881 --> [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbb40062fd0 con 0x7fbb280adf10 2024-06-09T22:18:56.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.368+0000 7fbb27fff700 1 -- 172.21.15.104:0/2695972881 <== osd.9 v2:172.21.15.104:6826/3652005298 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbb40062fd0 con 0x7fbb280adf10 2024-06-09T22:18:56.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.369+0000 7fbb4515a700 1 -- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298] conn(0x7fbb280adf10 msgr2=0x7fbb280b03b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:56.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.369+0000 7fbb4515a700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298] conn(0x7fbb280adf10 0x7fbb280b03b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:56.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.369+0000 7fbb4515a700 1 -- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbb280a2cb0 msgr2=0x7fbb280a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:56.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.369+0000 7fbb4515a700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbb280a2cb0 0x7fbb280a5170 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009a10 tx=0x7fbb30009190 comp rx=0 tx=0).stop 2024-06-09T22:18:56.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.369+0000 7fbb4515a700 1 -- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb40101fd0 msgr2=0x7fbb4019a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:56.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.369+0000 7fbb4515a700 1 --2- 172.21.15.104:0/2695972881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb40101fd0 0x7fbb4019a010 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbb340059d0 tx=0x7fbb340038b0 comp rx=0 tx=0).stop 2024-06-09T22:18:56.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.369+0000 7fbb3f59e700 1 -- 172.21.15.104:0/2695972881 reap_dead start 2024-06-09T22:18:56.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.369+0000 7fbb4515a700 1 -- 172.21.15.104:0/2695972881 shutdown_connections 2024-06-09T22:18:56.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.369+0000 7fbb4515a700 1 -- 172.21.15.104:0/2695972881 >> 172.21.15.104:0/2695972881 conn(0x7fbb40078360 msgr2=0x7fbb400ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:56.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.369+0000 7fbb4515a700 1 -- 172.21.15.104:0/2695972881 shutdown_connections 2024-06-09T22:18:56.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:56.369+0000 7fbb4515a700 1 -- 172.21.15.104:0/2695972881 wait complete. 2024-06-09T22:18:56.483 INFO:teuthology.orchestra.run.smithi104.stdout:197568495684 2024-06-09T22:18:56.483 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-09T22:18:56.504 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:56 smithi104 ceph-mon[32389]: pgmap v374: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 10 op/s 2024-06-09T22:18:56.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:56 smithi153 ceph-mon[36599]: pgmap v374: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 10 op/s 2024-06-09T22:18:56.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:56 smithi194 ceph-mon[36856]: pgmap v374: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 10 op/s 2024-06-09T22:18:57.205 INFO:teuthology.orchestra.run.smithi104.stdout:287762808886 2024-06-09T22:18:57.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-06-09T22:18:57.630 INFO:teuthology.orchestra.run.smithi104.stdout:34359738466 2024-06-09T22:18:57.630 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-09T22:18:57.997 INFO:teuthology.orchestra.run.smithi104.stdout:343597383722 2024-06-09T22:18:57.997 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-06-09T22:18:58.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.236+0000 7fc84d71d700 1 -- 172.21.15.104:0/816562497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848107400 msgr2=0x7fc8481077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:58.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.236+0000 7fc84d71d700 1 --2- 172.21.15.104:0/816562497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848107400 0x7fc8481077e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc838009990 tx=0x7fc838009ca0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.236+0000 7fc84d71d700 1 -- 172.21.15.104:0/816562497 shutdown_connections 2024-06-09T22:18:58.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.236+0000 7fc84d71d700 1 --2- 172.21.15.104:0/816562497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848107400 0x7fc8481077e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.236+0000 7fc84d71d700 1 --2- 172.21.15.104:0/816562497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc848101d50 0x7fc84810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.236+0000 7fc84d71d700 1 --2- 172.21.15.104:0/816562497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc848101390 0x7fc848101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.236+0000 7fc84d71d700 1 -- 172.21.15.104:0/816562497 >> 172.21.15.104:0/816562497 conn(0x7fc848078340 msgr2=0x7fc848078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:58.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.237+0000 7fc84d71d700 1 -- 172.21.15.104:0/816562497 shutdown_connections 2024-06-09T22:18:58.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.237+0000 7fc84d71d700 1 -- 172.21.15.104:0/816562497 wait complete. 2024-06-09T22:18:58.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.238+0000 7fc84d71d700 1 Processor -- start 2024-06-09T22:18:58.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.238+0000 7fc84d71d700 1 -- start start 2024-06-09T22:18:58.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.238+0000 7fc84d71d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc848101390 0x7fc84819d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:58.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.238+0000 7fc84d71d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc848101d50 0x7fc84819dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:58.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.238+0000 7fc84d71d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848107400 0x7fc848197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:58.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.238+0000 7fc84d71d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc8481116f0 con 0x7fc848107400 2024-06-09T22:18:58.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.238+0000 7fc84d71d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc848111570 con 0x7fc848101d50 2024-06-09T22:18:58.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.238+0000 7fc84d71d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc848111870 con 0x7fc848101390 2024-06-09T22:18:58.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.238+0000 7fc8477fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848107400 0x7fc848197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:58.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.238+0000 7fc8477fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848107400 0x7fc848197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38874/0 (socket says 172.21.15.104:38874) 2024-06-09T22:18:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.238+0000 7fc8477fe700 1 -- 172.21.15.104:0/797253062 learned_addr learned my addr 172.21.15.104:0/797253062 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.238+0000 7fc8467fc700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc848101d50 0x7fc84819dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc846ffd700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc848101390 0x7fc84819d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc8477fe700 1 -- 172.21.15.104:0/797253062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc848101390 msgr2=0x7fc84819d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc8477fe700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc848101390 0x7fc84819d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc8477fe700 1 -- 172.21.15.104:0/797253062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc848101d50 msgr2=0x7fc84819dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc8477fe700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc848101d50 0x7fc84819dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc8477fe700 1 -- 172.21.15.104:0/797253062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc848198170 con 0x7fc848107400 2024-06-09T22:18:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc8467fc700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc848101d50 0x7fc84819dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:18:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc8477fe700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848107400 0x7fc848197a10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc840007f10 tx=0x7fc84000f5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc82ffff700 1 -- 172.21.15.104:0/797253062 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840016030 con 0x7fc848107400 2024-06-09T22:18:58.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc82ffff700 1 -- 172.21.15.104:0/797253062 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc84000fd30 con 0x7fc848107400 2024-06-09T22:18:58.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc82ffff700 1 -- 172.21.15.104:0/797253062 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840005020 con 0x7fc848107400 2024-06-09T22:18:58.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc84d71d700 1 -- 172.21.15.104:0/797253062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc848198460 con 0x7fc848107400 2024-06-09T22:18:58.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.239+0000 7fc84d71d700 1 -- 172.21.15.104:0/797253062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc84806a170 con 0x7fc848107400 2024-06-09T22:18:58.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.240+0000 7fc84d71d700 1 -- 172.21.15.104:0/797253062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc84806a6e0 con 0x7fc848107400 2024-06-09T22:18:58.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.242+0000 7fc82ffff700 1 -- 172.21.15.104:0/797253062 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fc840012460 con 0x7fc848107400 2024-06-09T22:18:58.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.243+0000 7fc82ffff700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc8300a2d30 0x7fc8300a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:58.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.243+0000 7fc82ffff700 1 -- 172.21.15.104:0/797253062 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fc8400d1b10 con 0x7fc848107400 2024-06-09T22:18:58.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.243+0000 7fc846ffd700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc8300a2d30 0x7fc8300a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:58.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.243+0000 7fc82ffff700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097] conn(0x7fc8300adf90 0x7fc8300b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:58.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.243+0000 7fc82ffff700 1 -- 172.21.15.104:0/797253062 --> [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc840013e50 con 0x7fc8300adf90 2024-06-09T22:18:58.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.243+0000 7fc82ffff700 1 -- 172.21.15.104:0/797253062 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fc8400d1f60 con 0x7fc848107400 2024-06-09T22:18:58.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.243+0000 7fc8467fc700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097] conn(0x7fc8300adf90 0x7fc8300b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:58.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.243+0000 7fc846ffd700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc8300a2d30 0x7fc8300a51f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc838006010 tx=0x7fc838005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:58.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.244+0000 7fc8467fc700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097] conn(0x7fc8300adf90 0x7fc8300b0430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:58.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.245+0000 7fc82ffff700 1 -- 172.21.15.104:0/797253062 <== osd.1 v2:172.21.15.153:6800/628783097 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc840013e50 con 0x7fc8300adf90 2024-06-09T22:18:58.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.266+0000 7fc84d71d700 1 -- 172.21.15.104:0/797253062 --> [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc84806ab60 con 0x7fc8300adf90 2024-06-09T22:18:58.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.267+0000 7fc82ffff700 1 -- 172.21.15.104:0/797253062 <== osd.1 v2:172.21.15.153:6800/628783097 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc84806ab60 con 0x7fc8300adf90 2024-06-09T22:18:58.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.267+0000 7fc84d71d700 1 -- 172.21.15.104:0/797253062 >> [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097] conn(0x7fc8300adf90 msgr2=0x7fc8300b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:58.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.267+0000 7fc84d71d700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097] conn(0x7fc8300adf90 0x7fc8300b0430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.268+0000 7fc84d71d700 1 -- 172.21.15.104:0/797253062 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc8300a2d30 msgr2=0x7fc8300a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:58.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.268+0000 7fc84d71d700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fc8300a2d30 0x7fc8300a51f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc838006010 tx=0x7fc838005c00 comp rx=0 tx=0).stop 2024-06-09T22:18:58.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.268+0000 7fc84d71d700 1 -- 172.21.15.104:0/797253062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848107400 msgr2=0x7fc848197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:58.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.268+0000 7fc84d71d700 1 --2- 172.21.15.104:0/797253062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848107400 0x7fc848197a10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc840007f10 tx=0x7fc84000f5c0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.268+0000 7fc8477fe700 1 -- 172.21.15.104:0/797253062 reap_dead start 2024-06-09T22:18:58.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.268+0000 7fc84d71d700 1 -- 172.21.15.104:0/797253062 shutdown_connections 2024-06-09T22:18:58.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.268+0000 7fc84d71d700 1 -- 172.21.15.104:0/797253062 >> 172.21.15.104:0/797253062 conn(0x7fc848078340 msgr2=0x7fc8481000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:58.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.268+0000 7fc84d71d700 1 -- 172.21.15.104:0/797253062 shutdown_connections 2024-06-09T22:18:58.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:58.268+0000 7fc84d71d700 1 -- 172.21.15.104:0/797253062 wait complete. 2024-06-09T22:18:58.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.122+0000 7f8914fae700 1 -- 172.21.15.104:0/2768816307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89101017f0 msgr2=0x7f891010f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:58.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.122+0000 7f8914fae700 1 --2- 172.21.15.104:0/2768816307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89101017f0 0x7f891010f260 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f890800b0a0 tx=0x7f890800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.122+0000 7f8914fae700 1 -- 172.21.15.104:0/2768816307 shutdown_connections 2024-06-09T22:18:58.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.122+0000 7f8914fae700 1 --2- 172.21.15.104:0/2768816307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89101017f0 0x7f891010f260 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.122+0000 7f8914fae700 1 --2- 172.21.15.104:0/2768816307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8910107550 0x7f8910107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.122+0000 7f8914fae700 1 --2- 172.21.15.104:0/2768816307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8910100e30 0x7f89101012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.122+0000 7f8914fae700 1 -- 172.21.15.104:0/2768816307 >> 172.21.15.104:0/2768816307 conn(0x7f89100766b0 msgr2=0x7f8910076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:58.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.123+0000 7f8914fae700 1 -- 172.21.15.104:0/2768816307 shutdown_connections 2024-06-09T22:18:58.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.123+0000 7f8914fae700 1 -- 172.21.15.104:0/2768816307 wait complete. 2024-06-09T22:18:58.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.123+0000 7f8914fae700 1 Processor -- start 2024-06-09T22:18:58.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.123+0000 7f8914fae700 1 -- start start 2024-06-09T22:18:58.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.123+0000 7f8914fae700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8910100e30 0x7f891019e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:58.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.123+0000 7f8914fae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89101017f0 0x7f891019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:58.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.123+0000 7f8914fae700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8910107550 0x7f8910198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:58.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.123+0000 7f8914fae700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8910112850 con 0x7f89101017f0 2024-06-09T22:18:58.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.123+0000 7f8914fae700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f89101126d0 con 0x7f8910100e30 2024-06-09T22:18:58.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.123+0000 7f8914fae700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f89101129d0 con 0x7f8910107550 2024-06-09T22:18:58.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f890dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89101017f0 0x7f891019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:58.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f890dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89101017f0 0x7f891019edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33446/0 (socket says 172.21.15.104:33446) 2024-06-09T22:18:58.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f890dd9b700 1 -- 172.21.15.104:0/4126851563 learned_addr learned my addr 172.21.15.104:0/4126851563 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:18:58.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f890ed9d700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8910107550 0x7f8910198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:58.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f890e59c700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8910100e30 0x7f891019e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:58.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f890dd9b700 1 -- 172.21.15.104:0/4126851563 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8910107550 msgr2=0x7f8910198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:58.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f890dd9b700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8910107550 0x7f8910198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f890dd9b700 1 -- 172.21.15.104:0/4126851563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8910100e30 msgr2=0x7f891019e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:58.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f890dd9b700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8910100e30 0x7f891019e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f890dd9b700 1 -- 172.21.15.104:0/4126851563 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89101991d0 con 0x7f89101017f0 2024-06-09T22:18:58.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f890e59c700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8910100e30 0x7f891019e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:18:58.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f890dd9b700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89101017f0 0x7f891019edf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f890000c970 tx=0x7f890000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:58.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f88ff7fe700 1 -- 172.21.15.104:0/4126851563 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8900007aa0 con 0x7f89101017f0 2024-06-09T22:18:58.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f8914fae700 1 -- 172.21.15.104:0/4126851563 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89101994c0 con 0x7f89101017f0 2024-06-09T22:18:58.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f88ff7fe700 1 -- 172.21.15.104:0/4126851563 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8900004d10 con 0x7f89101017f0 2024-06-09T22:18:58.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f88ff7fe700 1 -- 172.21.15.104:0/4126851563 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890000f450 con 0x7f89101017f0 2024-06-09T22:18:58.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.124+0000 7f8914fae700 1 -- 172.21.15.104:0/4126851563 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89101a5780 con 0x7f89101017f0 2024-06-09T22:18:58.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.125+0000 7f8914fae700 1 -- 172.21.15.104:0/4126851563 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8910100230 con 0x7f89101017f0 2024-06-09T22:18:58.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.126+0000 7f88ff7fe700 1 -- 172.21.15.104:0/4126851563 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f8900004750 con 0x7f89101017f0 2024-06-09T22:18:58.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.127+0000 7f88ff7fe700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f88f80a2cb0 0x7f88f80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:58.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.127+0000 7f88ff7fe700 1 -- 172.21.15.104:0/4126851563 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f89000cf760 con 0x7f89101017f0 2024-06-09T22:18:58.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.127+0000 7f890e59c700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f88f80a2cb0 0x7f88f80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:58.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.127+0000 7f88ff7fe700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300] conn(0x7f88f80adf10 0x7f88f80b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:18:58.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.127+0000 7f88ff7fe700 1 -- 172.21.15.104:0/4126851563 --> [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8900016ec0 con 0x7f88f80adf10 2024-06-09T22:18:58.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.127+0000 7f88ff7fe700 1 -- 172.21.15.104:0/4126851563 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f89000cfbb0 con 0x7f89101017f0 2024-06-09T22:18:58.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.127+0000 7f890e59c700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f88f80a2cb0 0x7f88f80a5170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8910199bb0 tx=0x7f8904008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:58.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.128+0000 7f890ed9d700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300] conn(0x7f88f80adf10 0x7f88f80b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:18:58.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.128+0000 7f890ed9d700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300] conn(0x7f88f80adf10 0x7f88f80b03b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:18:58.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.129+0000 7f88ff7fe700 1 -- 172.21.15.104:0/4126851563 <== osd.8 v2:172.21.15.194:6816/2851208300 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8900016ec0 con 0x7f88f80adf10 2024-06-09T22:18:58.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.145+0000 7f8914fae700 1 -- 172.21.15.104:0/4126851563 --> [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f891006a6e0 con 0x7f88f80adf10 2024-06-09T22:18:58.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.145+0000 7f88ff7fe700 1 -- 172.21.15.104:0/4126851563 <== osd.8 v2:172.21.15.194:6816/2851208300 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f891006a6e0 con 0x7f88f80adf10 2024-06-09T22:18:58.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.146+0000 7f8914fae700 1 -- 172.21.15.104:0/4126851563 >> [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300] conn(0x7f88f80adf10 msgr2=0x7f88f80b03b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:58.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.146+0000 7f8914fae700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300] conn(0x7f88f80adf10 0x7f88f80b03b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:18:58.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.146+0000 7f8914fae700 1 -- 172.21.15.104:0/4126851563 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f88f80a2cb0 msgr2=0x7f88f80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:58.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.146+0000 7f8914fae700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f88f80a2cb0 0x7f88f80a5170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8910199bb0 tx=0x7f8904008040 comp rx=0 tx=0).stop 2024-06-09T22:18:58.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.146+0000 7f8914fae700 1 -- 172.21.15.104:0/4126851563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89101017f0 msgr2=0x7f891019edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:18:58.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.146+0000 7f8914fae700 1 --2- 172.21.15.104:0/4126851563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89101017f0 0x7f891019edf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f890000c970 tx=0x7f890000cc80 comp rx=0 tx=0).stop 2024-06-09T22:18:58.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.146+0000 7f890ed9d700 1 -- 172.21.15.104:0/4126851563 reap_dead start 2024-06-09T22:18:58.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.146+0000 7f8914fae700 1 -- 172.21.15.104:0/4126851563 shutdown_connections 2024-06-09T22:18:58.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.146+0000 7f8914fae700 1 -- 172.21.15.104:0/4126851563 >> 172.21.15.104:0/4126851563 conn(0x7f89100766b0 msgr2=0x7f89100ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:18:58.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.146+0000 7f8914fae700 1 -- 172.21.15.104:0/4126851563 shutdown_connections 2024-06-09T22:18:58.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:18:54.146+0000 7f8914fae700 1 -- 172.21.15.104:0/4126851563 wait complete. 2024-06-09T22:18:58.504 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:18:58 smithi104 ceph-mon[32389]: pgmap v375: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:18:58.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:18:58 smithi153 ceph-mon[36599]: pgmap v375: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:18:58.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:18:58 smithi194 ceph-mon[36856]: pgmap v375: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:18:58.931 INFO:teuthology.orchestra.run.smithi104.stdout:309237645359 2024-06-09T22:18:58.931 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-06-09T22:18:59.772 INFO:teuthology.orchestra.run.smithi104.stdout:55834574942 2024-06-09T22:18:59.772 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-09T22:19:00.168 INFO:teuthology.orchestra.run.smithi104.stdout:253403070522 2024-06-09T22:19:00.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-06-09T22:19:00.615 INFO:teuthology.orchestra.run.smithi104.stdout:141733920846 2024-06-09T22:19:00.616 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-09T22:19:00.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:00 smithi153 ceph-mon[36599]: pgmap v376: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:19:00.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:00 smithi104 ceph-mon[32389]: pgmap v376: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:19:00.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:00 smithi194 ceph-mon[36856]: pgmap v376: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:19:01.105 INFO:teuthology.orchestra.run.smithi104.stdout:77309411416 2024-06-09T22:19:01.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-09T22:19:01.197 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:01.526 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:01.527 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:01.529 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:01.530 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:01.533 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:01.673 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:01.674 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:01.676 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:02.504 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:02 smithi104 ceph-mon[32389]: pgmap v377: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:19:02.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:02 smithi153 ceph-mon[36599]: pgmap v377: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:19:02.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:02 smithi194 ceph-mon[36856]: pgmap v377: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:19:04.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:04 smithi153 ceph-mon[36599]: pgmap v378: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-06-09T22:19:04.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:04 smithi104 ceph-mon[32389]: pgmap v378: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-06-09T22:19:04.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:04 smithi194 ceph-mon[36856]: pgmap v378: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-06-09T22:19:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:06.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:06 smithi153 ceph-mon[36599]: pgmap v379: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:06.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:06 smithi104 ceph-mon[32389]: pgmap v379: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:06.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:06 smithi194 ceph-mon[36856]: pgmap v379: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:08.617 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:08 smithi104 ceph-mon[32389]: pgmap v380: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:19:08.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:08 smithi153 ceph-mon[36599]: pgmap v380: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:19:08.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:08 smithi194 ceph-mon[36856]: pgmap v380: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.446+0000 7fb7ad263700 1 -- 172.21.15.104:0/800660925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a806aa00 msgr2=0x7fb7a806ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.446+0000 7fb7ad263700 1 --2- 172.21.15.104:0/800660925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a806aa00 0x7fb7a806ae80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009990 tx=0x7fb79c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.446+0000 7fb7ad263700 1 -- 172.21.15.104:0/800660925 shutdown_connections 2024-06-09T22:19:10.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.446+0000 7fb7ad263700 1 --2- 172.21.15.104:0/800660925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a806b3c0 0x7fb7a810f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:10.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.446+0000 7fb7ad263700 1 --2- 172.21.15.104:0/800660925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7a81084f0 0x7fb7a81088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:10.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.446+0000 7fb7ad263700 1 --2- 172.21.15.104:0/800660925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a806aa00 0x7fb7a806ae80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:10.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.446+0000 7fb7ad263700 1 -- 172.21.15.104:0/800660925 >> 172.21.15.104:0/800660925 conn(0x7fb7a8078300 msgr2=0x7fb7a8078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:10.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.446+0000 7fb7ad263700 1 -- 172.21.15.104:0/800660925 shutdown_connections 2024-06-09T22:19:10.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.446+0000 7fb7ad263700 1 -- 172.21.15.104:0/800660925 wait complete. 2024-06-09T22:19:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.447+0000 7fb7ad263700 1 Processor -- start 2024-06-09T22:19:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.447+0000 7fb7ad263700 1 -- start start 2024-06-09T22:19:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7ad263700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a806aa00 0x7fb7a819e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7ad263700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a806b3c0 0x7fb7a819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7ad263700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7a81084f0 0x7fb7a8198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7ad263700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a8112750 con 0x7fb7a806b3c0 2024-06-09T22:19:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7ad263700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a81125d0 con 0x7fb7a806aa00 2024-06-09T22:19:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7ad263700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a81128d0 con 0x7fb7a81084f0 2024-06-09T22:19:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7a67fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a806b3c0 0x7fb7a819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7a67fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a806b3c0 0x7fb7a819edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36564/0 (socket says 172.21.15.104:36564) 2024-06-09T22:19:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7a67fc700 1 -- 172.21.15.104:0/865758974 learned_addr learned my addr 172.21.15.104:0/865758974 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7a77fe700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7a81084f0 0x7fb7a8198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7a6ffd700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a806aa00 0x7fb7a819e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7a67fc700 1 -- 172.21.15.104:0/865758974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7a81084f0 msgr2=0x7fb7a8198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7a67fc700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7a81084f0 0x7fb7a8198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7a67fc700 1 -- 172.21.15.104:0/865758974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a806aa00 msgr2=0x7fb7a819e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7a67fc700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a806aa00 0x7fb7a819e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7a67fc700 1 -- 172.21.15.104:0/865758974 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a8199190 con 0x7fb7a806b3c0 2024-06-09T22:19:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7a6ffd700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a806aa00 0x7fb7a819e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:19:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.448+0000 7fb7a67fc700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a806b3c0 0x7fb7a819edb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb79c000c00 tx=0x7fb79c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.449+0000 7fb78ffff700 1 -- 172.21.15.104:0/865758974 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79c027070 con 0x7fb7a806b3c0 2024-06-09T22:19:10.449 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:10.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.449+0000 7fb7ad263700 1 -- 172.21.15.104:0/865758974 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a8199420 con 0x7fb7a806b3c0 2024-06-09T22:19:10.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.449+0000 7fb78ffff700 1 -- 172.21.15.104:0/865758974 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb79c003780 con 0x7fb7a806b3c0 2024-06-09T22:19:10.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.449+0000 7fb78ffff700 1 -- 172.21.15.104:0/865758974 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79c022dd0 con 0x7fb7a806b3c0 2024-06-09T22:19:10.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.449+0000 7fb7ad263700 1 -- 172.21.15.104:0/865758974 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a81a5740 con 0x7fb7a806b3c0 2024-06-09T22:19:10.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.450+0000 7fb7ad263700 1 -- 172.21.15.104:0/865758974 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7a8103bb0 con 0x7fb7a806b3c0 2024-06-09T22:19:10.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.451+0000 7fb78ffff700 1 -- 172.21.15.104:0/865758974 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fb79c02a070 con 0x7fb7a806b3c0 2024-06-09T22:19:10.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.452+0000 7fb78ffff700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb7900a2cb0 0x7fb7900a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:10.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.452+0000 7fb78ffff700 1 -- 172.21.15.104:0/865758974 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fb79c0d05e0 con 0x7fb7a806b3c0 2024-06-09T22:19:10.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.452+0000 7fb7a6ffd700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb7900a2cb0 0x7fb7900a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:10.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.453+0000 7fb7a6ffd700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb7900a2cb0 0x7fb7900a5170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb798005e10 tx=0x7fb798005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:10.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.456+0000 7fb78ffff700 1 -- 172.21.15.104:0/865758974 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb79c013070 con 0x7fb7a806b3c0 2024-06-09T22:19:10.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.659+0000 7fb7ad263700 1 -- 172.21.15.104:0/865758974 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fb7a802dca0 con 0x7fb7a806b3c0 2024-06-09T22:19:10.657 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:10 smithi104 ceph-mon[32389]: pgmap v381: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:19:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.660+0000 7fb78ffff700 1 -- 172.21.15.104:0/865758974 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb79c09d170 con 0x7fb7a806b3c0 2024-06-09T22:19:10.658 INFO:teuthology.orchestra.run.smithi104.stdout:219043332161 2024-06-09T22:19:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.662+0000 7fb7ad263700 1 -- 172.21.15.104:0/865758974 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb7900a2cb0 msgr2=0x7fb7900a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.662+0000 7fb7ad263700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb7900a2cb0 0x7fb7900a5170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb798005e10 tx=0x7fb798005d60 comp rx=0 tx=0).stop 2024-06-09T22:19:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.662+0000 7fb7ad263700 1 -- 172.21.15.104:0/865758974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a806b3c0 msgr2=0x7fb7a819edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.662+0000 7fb7ad263700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a806b3c0 0x7fb7a819edb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb79c000c00 tx=0x7fb79c0047b0 comp rx=0 tx=0).stop 2024-06-09T22:19:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.662+0000 7fb7ad263700 1 -- 172.21.15.104:0/865758974 shutdown_connections 2024-06-09T22:19:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.662+0000 7fb7ad263700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb7900a2cb0 0x7fb7900a5170 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.662+0000 7fb7ad263700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7a81084f0 0x7fb7a8198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.662+0000 7fb7ad263700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a806aa00 0x7fb7a819e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.662+0000 7fb7ad263700 1 --2- 172.21.15.104:0/865758974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a806b3c0 0x7fb7a819edb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.662+0000 7fb7ad263700 1 -- 172.21.15.104:0/865758974 >> 172.21.15.104:0/865758974 conn(0x7fb7a8078300 msgr2=0x7fb7a8100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.663+0000 7fb7ad263700 1 -- 172.21.15.104:0/865758974 shutdown_connections 2024-06-09T22:19:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:10.663+0000 7fb7ad263700 1 -- 172.21.15.104:0/865758974 wait complete. 2024-06-09T22:19:10.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:10 smithi153 ceph-mon[36599]: pgmap v381: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:19:10.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:10 smithi194 ceph-mon[36856]: pgmap v381: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:19:11.651 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:11 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/865758974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-09T22:19:11.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:11 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/865758974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-09T22:19:11.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:11 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/865758974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-09T22:19:12.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.087+0000 7f95e9b65700 1 -- 172.21.15.104:0/2575648140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e4101390 msgr2=0x7f95e4101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:12.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.087+0000 7f95e9b65700 1 --2- 172.21.15.104:0/2575648140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e4101390 0x7f95e4101810 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f95d4009990 tx=0x7f95d4009ca0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.087+0000 7f95e9b65700 1 -- 172.21.15.104:0/2575648140 shutdown_connections 2024-06-09T22:19:12.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.087+0000 7f95e9b65700 1 --2- 172.21.15.104:0/2575648140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e4101d50 0x7f95e410e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.087+0000 7f95e9b65700 1 --2- 172.21.15.104:0/2575648140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95e4107400 0x7f95e41077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.087+0000 7f95e9b65700 1 --2- 172.21.15.104:0/2575648140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e4101390 0x7f95e4101810 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.087+0000 7f95e9b65700 1 -- 172.21.15.104:0/2575648140 >> 172.21.15.104:0/2575648140 conn(0x7f95e4078340 msgr2=0x7f95e4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:12.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.087+0000 7f95e9b65700 1 -- 172.21.15.104:0/2575648140 shutdown_connections 2024-06-09T22:19:12.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.087+0000 7f95e9b65700 1 -- 172.21.15.104:0/2575648140 wait complete. 2024-06-09T22:19:12.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.088+0000 7f95e9b65700 1 Processor -- start 2024-06-09T22:19:12.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.088+0000 7f95e9b65700 1 -- start start 2024-06-09T22:19:12.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.088+0000 7f95e9b65700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95e4101390 0x7f95e4103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:12.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.088+0000 7f95e9b65700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e4101d50 0x7f95e41035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:12.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.088+0000 7f95e9b65700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e4107400 0x7f95e4103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:12.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.088+0000 7f95e9b65700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f95e41116c0 con 0x7f95e4107400 2024-06-09T22:19:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.088+0000 7f95e9b65700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95e4111540 con 0x7f95e4101d50 2024-06-09T22:19:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.088+0000 7f95e9b65700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f95e4111840 con 0x7f95e4101390 2024-06-09T22:19:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.088+0000 7f95e3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e4107400 0x7f95e4103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.088+0000 7f95e2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e4101d50 0x7f95e41035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.088+0000 7f95e3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e4107400 0x7f95e4103af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36600/0 (socket says 172.21.15.104:36600) 2024-06-09T22:19:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.088+0000 7f95e3fff700 1 -- 172.21.15.104:0/230346704 learned_addr learned my addr 172.21.15.104:0/230346704 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.089+0000 7f95e37fe700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95e4101390 0x7f95e4103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.089+0000 7f95e3fff700 1 -- 172.21.15.104:0/230346704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95e4101390 msgr2=0x7f95e4103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.089+0000 7f95e3fff700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95e4101390 0x7f95e4103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.089+0000 7f95e3fff700 1 -- 172.21.15.104:0/230346704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e4101d50 msgr2=0x7f95e41035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.089+0000 7f95e3fff700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e4101d50 0x7f95e41035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.089+0000 7f95e3fff700 1 -- 172.21.15.104:0/230346704 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e41a70b0 con 0x7f95e4107400 2024-06-09T22:19:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.089+0000 7f95e3fff700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e4107400 0x7f95e4103af0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f95dc007d70 tx=0x7f95dc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.089+0000 7f95e0ff9700 1 -- 172.21.15.104:0/230346704 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc016020 con 0x7f95e4107400 2024-06-09T22:19:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.089+0000 7f95e0ff9700 1 -- 172.21.15.104:0/230346704 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95dc00ac90 con 0x7f95e4107400 2024-06-09T22:19:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.089+0000 7f95e0ff9700 1 -- 172.21.15.104:0/230346704 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc0115c0 con 0x7f95e4107400 2024-06-09T22:19:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.089+0000 7f95e9b65700 1 -- 172.21.15.104:0/230346704 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e41a73a0 con 0x7f95e4107400 2024-06-09T22:19:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.089+0000 7f95e9b65700 1 -- 172.21.15.104:0/230346704 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e41a7890 con 0x7f95e4107400 2024-06-09T22:19:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.090+0000 7f95e9b65700 1 -- 172.21.15.104:0/230346704 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e4069180 con 0x7f95e4107400 2024-06-09T22:19:12.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.092+0000 7f95e0ff9700 1 -- 172.21.15.104:0/230346704 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f95dc019020 con 0x7f95e4107400 2024-06-09T22:19:12.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.093+0000 7f95e0ff9700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f95d00a2cb0 0x7f95d00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:12.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.093+0000 7f95e37fe700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f95d00a2cb0 0x7f95d00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:12.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.093+0000 7f95e0ff9700 1 -- 172.21.15.104:0/230346704 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f95dc0cf330 con 0x7f95e4107400 2024-06-09T22:19:12.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.094+0000 7f95e37fe700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f95d00a2cb0 0x7f95d00a5170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f95cc00abd0 tx=0x7f95cc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:12.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.098+0000 7f95e0ff9700 1 -- 172.21.15.104:0/230346704 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f95dc020020 con 0x7f95e4107400 2024-06-09T22:19:12.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.959+0000 7f7820b44700 1 -- 172.21.15.104:0/3899586119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f781c107660 msgr2=0x7f781c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:12.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.959+0000 7f7820b44700 1 --2- 172.21.15.104:0/3899586119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f781c107660 0x7f781c107a40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7810009990 tx=0x7f7810009ca0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.959+0000 7f7820b44700 1 -- 172.21.15.104:0/3899586119 shutdown_connections 2024-06-09T22:19:12.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.959+0000 7f7820b44700 1 --2- 172.21.15.104:0/3899586119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f781c107660 0x7f781c107a40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.959+0000 7f7820b44700 1 --2- 172.21.15.104:0/3899586119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f781c1024b0 0x7f781c10e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.959+0000 7f7820b44700 1 --2- 172.21.15.104:0/3899586119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f781c101af0 0x7f781c101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.959+0000 7f7820b44700 1 -- 172.21.15.104:0/3899586119 >> 172.21.15.104:0/3899586119 conn(0x7f781c077990 msgr2=0x7f781c0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:12.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.959+0000 7f7820b44700 1 -- 172.21.15.104:0/3899586119 shutdown_connections 2024-06-09T22:19:12.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.959+0000 7f7820b44700 1 -- 172.21.15.104:0/3899586119 wait complete. 2024-06-09T22:19:12.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.960+0000 7f7820b44700 1 Processor -- start 2024-06-09T22:19:12.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.960+0000 7f7820b44700 1 -- start start 2024-06-09T22:19:12.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7820b44700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f781c101af0 0x7f781c197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:12.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7820b44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f781c1024b0 0x7f781c198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:12.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7820b44700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f781c107660 0x7f781c19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:12.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7820b44700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f781c1119c0 con 0x7f781c1024b0 2024-06-09T22:19:12.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7820b44700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f781c111840 con 0x7f781c101af0 2024-06-09T22:19:12.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7820b44700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f781c111b40 con 0x7f781c107660 2024-06-09T22:19:12.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7819d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f781c1024b0 0x7f781c198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7819d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f781c1024b0 0x7f781c198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36586/0 (socket says 172.21.15.104:36586) 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7819d9b700 1 -- 172.21.15.104:0/3451990245 learned_addr learned my addr 172.21.15.104:0/3451990245 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f781ad9d700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f781c107660 0x7f781c19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f781a59c700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f781c101af0 0x7f781c197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7819d9b700 1 -- 172.21.15.104:0/3451990245 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f781c107660 msgr2=0x7f781c19c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7819d9b700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f781c107660 0x7f781c19c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7819d9b700 1 -- 172.21.15.104:0/3451990245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f781c101af0 msgr2=0x7f781c197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7819d9b700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f781c101af0 0x7f781c197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7819d9b700 1 -- 172.21.15.104:0/3451990245 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f781c19ce60 con 0x7f781c1024b0 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7819d9b700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f781c1024b0 0x7f781c198390 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f780c00b530 tx=0x7f780c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f780b7fe700 1 -- 172.21.15.104:0/3451990245 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c015020 con 0x7f781c1024b0 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7820b44700 1 -- 172.21.15.104:0/3451990245 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f781c19d150 con 0x7f781c1024b0 2024-06-09T22:19:12.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f780b7fe700 1 -- 172.21.15.104:0/3451990245 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f780c011d10 con 0x7f781c1024b0 2024-06-09T22:19:12.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f780b7fe700 1 -- 172.21.15.104:0/3451990245 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c010e50 con 0x7f781c1024b0 2024-06-09T22:19:12.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.961+0000 7f7820b44700 1 -- 172.21.15.104:0/3451990245 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f781c1a4a50 con 0x7f781c1024b0 2024-06-09T22:19:12.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.962+0000 7f7820b44700 1 -- 172.21.15.104:0/3451990245 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f781c069180 con 0x7f781c1024b0 2024-06-09T22:19:12.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.963+0000 7f780b7fe700 1 -- 172.21.15.104:0/3451990245 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f780c018020 con 0x7f781c1024b0 2024-06-09T22:19:12.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.964+0000 7f780b7fe700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f78040a2e50 0x7f78040a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:12.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.964+0000 7f781a59c700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f78040a2e50 0x7f78040a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:12.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.964+0000 7f780b7fe700 1 -- 172.21.15.104:0/3451990245 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f780c0cecd0 con 0x7f781c1024b0 2024-06-09T22:19:12.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.965+0000 7f781a59c700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f78040a2e50 0x7f78040a5310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7810005cd0 tx=0x7f7810005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:12.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:11.969+0000 7f780b7fe700 1 -- 172.21.15.104:0/3451990245 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f780c01f020 con 0x7f781c1024b0 2024-06-09T22:19:12.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.175+0000 7f7820b44700 1 -- 172.21.15.104:0/3451990245 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f781c06a9e0 con 0x7f781c1024b0 2024-06-09T22:19:12.304 INFO:teuthology.orchestra.run.smithi104.stdout:55834574945 2024-06-09T22:19:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.176+0000 7f780b7fe700 1 -- 172.21.15.104:0/3451990245 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f780c09b850 con 0x7f781c1024b0 2024-06-09T22:19:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.178+0000 7f7820b44700 1 -- 172.21.15.104:0/3451990245 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f78040a2e50 msgr2=0x7f78040a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.178+0000 7f7820b44700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f78040a2e50 0x7f78040a5310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7810005cd0 tx=0x7f7810005c40 comp rx=0 tx=0).stop 2024-06-09T22:19:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.178+0000 7f7820b44700 1 -- 172.21.15.104:0/3451990245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f781c1024b0 msgr2=0x7f781c198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.178+0000 7f7820b44700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f781c1024b0 0x7f781c198390 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f780c00b530 tx=0x7f780c00b8f0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.178+0000 7f7820b44700 1 -- 172.21.15.104:0/3451990245 shutdown_connections 2024-06-09T22:19:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.178+0000 7f7820b44700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f78040a2e50 0x7f78040a5310 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.178+0000 7f7820b44700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f781c107660 0x7f781c19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.178+0000 7f7820b44700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f781c101af0 0x7f781c197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.178+0000 7f7820b44700 1 --2- 172.21.15.104:0/3451990245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f781c1024b0 0x7f781c198390 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.178+0000 7f7820b44700 1 -- 172.21.15.104:0/3451990245 >> 172.21.15.104:0/3451990245 conn(0x7f781c077990 msgr2=0x7f781c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.178+0000 7f7820b44700 1 -- 172.21.15.104:0/3451990245 shutdown_connections 2024-06-09T22:19:12.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.178+0000 7f7820b44700 1 -- 172.21.15.104:0/3451990245 wait complete. 2024-06-09T22:19:12.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.315+0000 7f95e9b65700 1 -- 172.21.15.104:0/230346704 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f95e41a7bf0 con 0x7f95e4107400 2024-06-09T22:19:12.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.315+0000 7f95e0ff9700 1 -- 172.21.15.104:0/230346704 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f95dc09bf50 con 0x7f95e4107400 2024-06-09T22:19:12.313 INFO:teuthology.orchestra.run.smithi104.stdout:197568495688 2024-06-09T22:19:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.317+0000 7f95e9b65700 1 -- 172.21.15.104:0/230346704 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f95d00a2cb0 msgr2=0x7f95d00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.317+0000 7f95e9b65700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f95d00a2cb0 0x7f95d00a5170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f95cc00abd0 tx=0x7f95cc00a380 comp rx=0 tx=0).stop 2024-06-09T22:19:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.317+0000 7f95e9b65700 1 -- 172.21.15.104:0/230346704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e4107400 msgr2=0x7f95e4103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.317+0000 7f95e9b65700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e4107400 0x7f95e4103af0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f95dc007d70 tx=0x7f95dc00a340 comp rx=0 tx=0).stop 2024-06-09T22:19:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.318+0000 7f95e9b65700 1 -- 172.21.15.104:0/230346704 shutdown_connections 2024-06-09T22:19:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.318+0000 7f95e9b65700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f95d00a2cb0 0x7f95d00a5170 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.318+0000 7f95e9b65700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e4107400 0x7f95e4103af0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.318+0000 7f95e9b65700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95e4101390 0x7f95e4103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.318+0000 7f95e9b65700 1 --2- 172.21.15.104:0/230346704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95e4101d50 0x7f95e41035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:12.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.318+0000 7f95e9b65700 1 -- 172.21.15.104:0/230346704 >> 172.21.15.104:0/230346704 conn(0x7f95e4078340 msgr2=0x7f95e4105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:12.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.318+0000 7f95e9b65700 1 -- 172.21.15.104:0/230346704 shutdown_connections 2024-06-09T22:19:12.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:12.318+0000 7f95e9b65700 1 -- 172.21.15.104:0/230346704 wait complete. 2024-06-09T22:19:12.694 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:12 smithi104 ceph-mon[32389]: pgmap v382: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:19:12.694 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:12 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3451990245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-09T22:19:12.694 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:12 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/230346704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-09T22:19:12.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:12 smithi153 ceph-mon[36599]: pgmap v382: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:19:12.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:12 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3451990245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-09T22:19:12.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:12 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/230346704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-09T22:19:12.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:12 smithi194 ceph-mon[36856]: pgmap v382: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:19:12.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:12 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3451990245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-09T22:19:12.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:12 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/230346704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-09T22:19:14.601 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:14 smithi104 ceph-mon[32389]: pgmap v383: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:19:14.629 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495684 got 197568495688 for osd.6 2024-06-09T22:19:14.630 DEBUG:teuthology.parallel:result is None 2024-06-09T22:19:14.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:14 smithi153 ceph-mon[36599]: pgmap v383: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:19:14.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:14 smithi194 ceph-mon[36856]: pgmap v383: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:19:15.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.350+0000 7f4e977fe700 1 -- 172.21.15.104:0/905291186 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84004230 con 0x7f4e9806a170 2024-06-09T22:19:15.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.350+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/905291186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9806a170 msgr2=0x7f4e9806a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:15.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.350+0000 7f4e9f0ad700 1 --2- 172.21.15.104:0/905291186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9806a170 0x7f4e9806a5f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009990 tx=0x7f4e84009ca0 comp rx=0 tx=0).stop 2024-06-09T22:19:15.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.350+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/905291186 shutdown_connections 2024-06-09T22:19:15.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.350+0000 7f4e9f0ad700 1 --2- 172.21.15.104:0/905291186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e9806ab30 0x7f4e9810f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:15.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.350+0000 7f4e9f0ad700 1 --2- 172.21.15.104:0/905291186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e98108500 0x7f4e981088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:15.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.350+0000 7f4e9f0ad700 1 --2- 172.21.15.104:0/905291186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9806a170 0x7f4e9806a5f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:15.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.350+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/905291186 >> 172.21.15.104:0/905291186 conn(0x7f4e98078340 msgr2=0x7f4e98078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:15.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.351+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/905291186 shutdown_connections 2024-06-09T22:19:15.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.351+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/905291186 wait complete. 2024-06-09T22:19:15.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.351+0000 7f4e9f0ad700 1 Processor -- start 2024-06-09T22:19:15.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.352+0000 7f4e9f0ad700 1 -- start start 2024-06-09T22:19:15.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.352+0000 7f4e9f0ad700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e9806ab30 0x7f4e98198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:15.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.352+0000 7f4e9f0ad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e98108500 0x7f4e98199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:15.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.352+0000 7f4e9f0ad700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9819d7e0 0x7f4e9819dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:15.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.352+0000 7f4e9f0ad700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4e981128a0 con 0x7f4e9819d7e0 2024-06-09T22:19:15.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.352+0000 7f4e9f0ad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e98112720 con 0x7f4e98108500 2024-06-09T22:19:15.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.352+0000 7f4e9f0ad700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e98112a20 con 0x7f4e9806ab30 2024-06-09T22:19:15.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.352+0000 7f4e97fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e98108500 0x7f4e98199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:15.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.352+0000 7f4e97fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e98108500 0x7f4e98199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:37156/0 (socket says 172.21.15.104:37156) 2024-06-09T22:19:15.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.352+0000 7f4e97fff700 1 -- 172.21.15.104:0/1473257556 learned_addr learned my addr 172.21.15.104:0/1473257556 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:15.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.352+0000 7f4e9ce49700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e9806ab30 0x7f4e98198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:15.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.352+0000 7f4e9d64a700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9819d7e0 0x7f4e9819dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:15.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.353+0000 7f4e97fff700 1 -- 172.21.15.104:0/1473257556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e9806ab30 msgr2=0x7f4e98198f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:15.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.353+0000 7f4e97fff700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e9806ab30 0x7f4e98198f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:15.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.353+0000 7f4e97fff700 1 -- 172.21.15.104:0/1473257556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9819d7e0 msgr2=0x7f4e9819dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:15.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.353+0000 7f4e97fff700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9819d7e0 0x7f4e9819dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:15.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.353+0000 7f4e97fff700 1 -- 172.21.15.104:0/1473257556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9819e300 con 0x7f4e98108500 2024-06-09T22:19:15.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.353+0000 7f4e97fff700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e98108500 0x7f4e98199450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400b2e0 tx=0x7f4e8400ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:15.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.353+0000 7f4e95ffb700 1 -- 172.21.15.104:0/1473257556 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84026070 con 0x7f4e98108500 2024-06-09T22:19:15.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.353+0000 7f4e95ffb700 1 -- 172.21.15.104:0/1473257556 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e84003d70 con 0x7f4e98108500 2024-06-09T22:19:15.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.353+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/1473257556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e9819e590 con 0x7f4e98108500 2024-06-09T22:19:15.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.353+0000 7f4e95ffb700 1 -- 172.21.15.104:0/1473257556 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84021410 con 0x7f4e98108500 2024-06-09T22:19:15.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.354+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/1473257556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e981015b0 con 0x7f4e98108500 2024-06-09T22:19:15.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.355+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/1473257556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e98069180 con 0x7f4e98108500 2024-06-09T22:19:15.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.356+0000 7f4e95ffb700 1 -- 172.21.15.104:0/1473257556 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f4e84029070 con 0x7f4e98108500 2024-06-09T22:19:15.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.357+0000 7f4e95ffb700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4e880a2cb0 0x7f4e880a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:15.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.357+0000 7f4e95ffb700 1 -- 172.21.15.104:0/1473257556 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f4e840d0770 con 0x7f4e98108500 2024-06-09T22:19:15.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.357+0000 7f4e9ce49700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4e880a2cb0 0x7f4e880a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:15.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.358+0000 7f4e9ce49700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4e880a2cb0 0x7f4e880a5170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c009c60 tx=0x7f4e8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:15.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.361+0000 7f4e95ffb700 1 -- 172.21.15.104:0/1473257556 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4e84030080 con 0x7f4e98108500 2024-06-09T22:19:15.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.574+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/1473257556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f4e981017e0 con 0x7f4e98108500 2024-06-09T22:19:15.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.574+0000 7f4e95ffb700 1 -- 172.21.15.104:0/1473257556 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4e84013070 con 0x7f4e98108500 2024-06-09T22:19:15.573 INFO:teuthology.orchestra.run.smithi104.stdout:107374182487 2024-06-09T22:19:15.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.576+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/1473257556 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4e880a2cb0 msgr2=0x7f4e880a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.577+0000 7f4e9f0ad700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4e880a2cb0 0x7f4e880a5170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c009c60 tx=0x7f4e8c009260 comp rx=0 tx=0).stop 2024-06-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.577+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/1473257556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e98108500 msgr2=0x7f4e98199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.577+0000 7f4e9f0ad700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e98108500 0x7f4e98199450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400b2e0 tx=0x7f4e8400ba90 comp rx=0 tx=0).stop 2024-06-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.577+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/1473257556 shutdown_connections 2024-06-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.577+0000 7f4e9f0ad700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f4e880a2cb0 0x7f4e880a5170 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.577+0000 7f4e9f0ad700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9819d7e0 0x7f4e9819dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.577+0000 7f4e9f0ad700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e9806ab30 0x7f4e98198f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.577+0000 7f4e9f0ad700 1 --2- 172.21.15.104:0/1473257556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e98108500 0x7f4e98199450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.577+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/1473257556 >> 172.21.15.104:0/1473257556 conn(0x7f4e98078340 msgr2=0x7f4e98109240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.577+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/1473257556 shutdown_connections 2024-06-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:15.577+0000 7f4e9f0ad700 1 -- 172.21.15.104:0/1473257556 wait complete. 2024-06-09T22:19:16.260 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574942 got 55834574945 for osd.1 2024-06-09T22:19:16.260 DEBUG:teuthology.parallel:result is None 2024-06-09T22:19:16.710 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182480 got 107374182487 for osd.3 2024-06-09T22:19:16.710 DEBUG:teuthology.parallel:result is None 2024-06-09T22:19:16.737 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:16 smithi104 ceph-mon[32389]: pgmap v384: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:16.737 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:16 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1473257556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-09T22:19:16.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:16 smithi153 ceph-mon[36599]: pgmap v384: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:16.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:16 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1473257556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-09T22:19:16.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:16 smithi194 ceph-mon[36856]: pgmap v384: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:16.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:16 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1473257556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-09T22:19:17.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.280+0000 7f78fa1a8700 1 -- 172.21.15.104:0/2026893213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78f4101d30 msgr2=0x7f78f410e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:17.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.280+0000 7f78fa1a8700 1 --2- 172.21.15.104:0/2026893213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78f4101d30 0x7f78f410e210 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f78ec00b0a0 tx=0x7f78ec00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:19:17.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.280+0000 7f78fa1a8700 1 -- 172.21.15.104:0/2026893213 shutdown_connections 2024-06-09T22:19:17.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.280+0000 7f78fa1a8700 1 --2- 172.21.15.104:0/2026893213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78f4101d30 0x7f78f410e210 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:17.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.280+0000 7f78fa1a8700 1 --2- 172.21.15.104:0/2026893213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f41073d0 0x7f78f41077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:17.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.280+0000 7f78fa1a8700 1 --2- 172.21.15.104:0/2026893213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78f4101390 0x7f78f41017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:17.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.280+0000 7f78fa1a8700 1 -- 172.21.15.104:0/2026893213 >> 172.21.15.104:0/2026893213 conn(0x7f78f40782c0 msgr2=0x7f78f40786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:17.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.280+0000 7f78fa1a8700 1 -- 172.21.15.104:0/2026893213 shutdown_connections 2024-06-09T22:19:17.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.280+0000 7f78fa1a8700 1 -- 172.21.15.104:0/2026893213 wait complete. 2024-06-09T22:19:17.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.281+0000 7f78fa1a8700 1 Processor -- start 2024-06-09T22:19:17.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.281+0000 7f78fa1a8700 1 -- start start 2024-06-09T22:19:17.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.281+0000 7f78fa1a8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78f4101390 0x7f78f419d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:17.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.281+0000 7f78fa1a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78f4101d30 0x7f78f419dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:17.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.281+0000 7f78fa1a8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f41073d0 0x7f78f41978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:17.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.281+0000 7f78fa1a8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f78f41116d0 con 0x7f78f4101d30 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.281+0000 7f78fa1a8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78f4111550 con 0x7f78f41073d0 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.281+0000 7f78fa1a8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f78f4111850 con 0x7f78f4101390 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78f4101d30 0x7f78f419dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78f4101d30 0x7f78f419dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50014/0 (socket says 172.21.15.104:50014) 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f2ffd700 1 -- 172.21.15.104:0/1519855548 learned_addr learned my addr 172.21.15.104:0/1519855548 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f37fe700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78f4101390 0x7f78f419d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f3fff700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f41073d0 0x7f78f41978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f2ffd700 1 -- 172.21.15.104:0/1519855548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78f4101390 msgr2=0x7f78f419d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f2ffd700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78f4101390 0x7f78f419d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f2ffd700 1 -- 172.21.15.104:0/1519855548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f41073d0 msgr2=0x7f78f41978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f2ffd700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f41073d0 0x7f78f41978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f2ffd700 1 -- 172.21.15.104:0/1519855548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f4198010 con 0x7f78f4101d30 2024-06-09T22:19:17.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f2ffd700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78f4101d30 0x7f78f419dbe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f78e400b530 tx=0x7f78e400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:17.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f0ff9700 1 -- 172.21.15.104:0/1519855548 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4015020 con 0x7f78f4101d30 2024-06-09T22:19:17.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f0ff9700 1 -- 172.21.15.104:0/1519855548 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e4011d10 con 0x7f78f4101d30 2024-06-09T22:19:17.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78fa1a8700 1 -- 172.21.15.104:0/1519855548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f4198300 con 0x7f78f4101d30 2024-06-09T22:19:17.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78f0ff9700 1 -- 172.21.15.104:0/1519855548 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e400f2e0 con 0x7f78f4101d30 2024-06-09T22:19:17.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.282+0000 7f78fa1a8700 1 -- 172.21.15.104:0/1519855548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f41a4570 con 0x7f78f4101d30 2024-06-09T22:19:17.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.283+0000 7f78fa1a8700 1 -- 172.21.15.104:0/1519855548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f4069180 con 0x7f78f4101d30 2024-06-09T22:19:17.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.284+0000 7f78f0ff9700 1 -- 172.21.15.104:0/1519855548 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f78e4018020 con 0x7f78f4101d30 2024-06-09T22:19:17.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.285+0000 7f78f0ff9700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f78e00a2cb0 0x7f78e00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:17.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.285+0000 7f78f37fe700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f78e00a2cb0 0x7f78e00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:17.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.285+0000 7f78f0ff9700 1 -- 172.21.15.104:0/1519855548 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f78e40cf290 con 0x7f78f4101d30 2024-06-09T22:19:17.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.286+0000 7f78f37fe700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f78e00a2cb0 0x7f78e00a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f78f40ffba0 tx=0x7f78dc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:17.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.291+0000 7f78f0ff9700 1 -- 172.21.15.104:0/1519855548 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f78e401f020 con 0x7f78f4101d30 2024-06-09T22:19:17.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.499+0000 7f78fa1a8700 1 -- 172.21.15.104:0/1519855548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f78f402dca0 con 0x7f78f4101d30 2024-06-09T22:19:17.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.499+0000 7f78f0ff9700 1 -- 172.21.15.104:0/1519855548 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f78e409be10 con 0x7f78f4101d30 2024-06-09T22:19:17.497 INFO:teuthology.orchestra.run.smithi104.stdout:309237645364 2024-06-09T22:19:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.502+0000 7f78fa1a8700 1 -- 172.21.15.104:0/1519855548 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f78e00a2cb0 msgr2=0x7f78e00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.502+0000 7f78fa1a8700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f78e00a2cb0 0x7f78e00a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f78f40ffba0 tx=0x7f78dc00a380 comp rx=0 tx=0).stop 2024-06-09T22:19:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.502+0000 7f78fa1a8700 1 -- 172.21.15.104:0/1519855548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78f4101d30 msgr2=0x7f78f419dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.502+0000 7f78fa1a8700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78f4101d30 0x7f78f419dbe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f78e400b530 tx=0x7f78e400b840 comp rx=0 tx=0).stop 2024-06-09T22:19:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.502+0000 7f78fa1a8700 1 -- 172.21.15.104:0/1519855548 shutdown_connections 2024-06-09T22:19:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.502+0000 7f78fa1a8700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f78e00a2cb0 0x7f78e00a5170 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.502+0000 7f78fa1a8700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f41073d0 0x7f78f41978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.502+0000 7f78fa1a8700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78f4101390 0x7f78f419d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.502+0000 7f78fa1a8700 1 --2- 172.21.15.104:0/1519855548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78f4101d30 0x7f78f419dbe0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.502+0000 7f78fa1a8700 1 -- 172.21.15.104:0/1519855548 >> 172.21.15.104:0/1519855548 conn(0x7f78f40782c0 msgr2=0x7f78f40ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.502+0000 7f78fa1a8700 1 -- 172.21.15.104:0/1519855548 shutdown_connections 2024-06-09T22:19:17.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:17.502+0000 7f78fa1a8700 1 -- 172.21.15.104:0/1519855548 wait complete. 2024-06-09T22:19:18.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.412+0000 7ff487d97700 1 -- 172.21.15.104:0/1053166308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff48006b5c0 msgr2=0x7ff48010f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:18.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.412+0000 7ff487d97700 1 --2- 172.21.15.104:0/1053166308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff48006b5c0 0x7ff48010f370 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff47800b0a0 tx=0x7ff47800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.412+0000 7ff487d97700 1 -- 172.21.15.104:0/1053166308 shutdown_connections 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.412+0000 7ff487d97700 1 --2- 172.21.15.104:0/1053166308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff48006b5c0 0x7ff48010f370 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.412+0000 7ff487d97700 1 --2- 172.21.15.104:0/1053166308 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff480108530 0x7ff480108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.412+0000 7ff487d97700 1 --2- 172.21.15.104:0/1053166308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48006ac00 0x7ff48006b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.412+0000 7ff487d97700 1 -- 172.21.15.104:0/1053166308 >> 172.21.15.104:0/1053166308 conn(0x7ff480077400 msgr2=0x7ff480077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.412+0000 7ff487d97700 1 -- 172.21.15.104:0/1053166308 shutdown_connections 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.412+0000 7ff487d97700 1 -- 172.21.15.104:0/1053166308 wait complete. 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff487d97700 1 Processor -- start 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff487d97700 1 -- start start 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff487d97700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff48006ac00 0x7ff480101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff487d97700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48006b5c0 0x7ff4801019d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff487d97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff480108530 0x7ff480101f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff487d97700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff4801127e0 con 0x7ff480108530 2024-06-09T22:19:18.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff487d97700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff480112660 con 0x7ff48006b5c0 2024-06-09T22:19:18.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff487d97700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff480112960 con 0x7ff48006ac00 2024-06-09T22:19:18.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff486334700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff480108530 0x7ff480101f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:18.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff486334700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff480108530 0x7ff480101f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50038/0 (socket says 172.21.15.104:50038) 2024-06-09T22:19:18.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff486334700 1 -- 172.21.15.104:0/1826288867 learned_addr learned my addr 172.21.15.104:0/1826288867 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:18.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff485332700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48006b5c0 0x7ff4801019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:18.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff485b33700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff48006ac00 0x7ff480101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:18.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff486334700 1 -- 172.21.15.104:0/1826288867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff48006ac00 msgr2=0x7ff480101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:18.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff486334700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff48006ac00 0x7ff480101490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff486334700 1 -- 172.21.15.104:0/1826288867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48006b5c0 msgr2=0x7ff4801019d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:18.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff486334700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48006b5c0 0x7ff4801019d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.413+0000 7ff486334700 1 -- 172.21.15.104:0/1826288867 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4801a7210 con 0x7ff480108530 2024-06-09T22:19:18.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.414+0000 7ff486334700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff480108530 0x7ff480101f10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff4780062a0 tx=0x7ff47800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:18.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.414+0000 7ff476ffd700 1 -- 172.21.15.104:0/1826288867 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff478021070 con 0x7ff480108530 2024-06-09T22:19:18.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.414+0000 7ff476ffd700 1 -- 172.21.15.104:0/1826288867 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff478007c80 con 0x7ff480108530 2024-06-09T22:19:18.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.414+0000 7ff487d97700 1 -- 172.21.15.104:0/1826288867 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4801a74a0 con 0x7ff480108530 2024-06-09T22:19:18.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.414+0000 7ff476ffd700 1 -- 172.21.15.104:0/1826288867 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff478005420 con 0x7ff480108530 2024-06-09T22:19:18.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.414+0000 7ff487d97700 1 -- 172.21.15.104:0/1826288867 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4801a7950 con 0x7ff480108530 2024-06-09T22:19:18.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.415+0000 7ff487d97700 1 -- 172.21.15.104:0/1826288867 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff48010aa30 con 0x7ff480108530 2024-06-09T22:19:18.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.416+0000 7ff476ffd700 1 -- 172.21.15.104:0/1826288867 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff478024070 con 0x7ff480108530 2024-06-09T22:19:18.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.416+0000 7ff476ffd700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff46c0a2cb0 0x7ff46c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:18.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.417+0000 7ff485b33700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff46c0a2cb0 0x7ff46c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:18.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.417+0000 7ff476ffd700 1 -- 172.21.15.104:0/1826288867 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7ff47800f020 con 0x7ff480108530 2024-06-09T22:19:18.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.417+0000 7ff485b33700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff46c0a2cb0 0x7ff46c0a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009a10 tx=0x7ff47c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:18.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.421+0000 7ff476ffd700 1 -- 172.21.15.104:0/1826288867 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff47802b030 con 0x7ff480108530 2024-06-09T22:19:18.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.625+0000 7ff487d97700 1 -- 172.21.15.104:0/1826288867 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff480102970 con 0x7ff480108530 2024-06-09T22:19:18.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.626+0000 7ff476ffd700 1 -- 172.21.15.104:0/1826288867 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff47809d360 con 0x7ff480108530 2024-06-09T22:19:18.624 INFO:teuthology.orchestra.run.smithi104.stdout:141733920851 2024-06-09T22:19:18.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.629+0000 7ff487d97700 1 -- 172.21.15.104:0/1826288867 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff46c0a2cb0 msgr2=0x7ff46c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:18.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.629+0000 7ff487d97700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff46c0a2cb0 0x7ff46c0a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009a10 tx=0x7ff47c009190 comp rx=0 tx=0).stop 2024-06-09T22:19:18.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.629+0000 7ff487d97700 1 -- 172.21.15.104:0/1826288867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff480108530 msgr2=0x7ff480101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:18.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.629+0000 7ff487d97700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff480108530 0x7ff480101f10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff4780062a0 tx=0x7ff47800be60 comp rx=0 tx=0).stop 2024-06-09T22:19:18.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.629+0000 7ff487d97700 1 -- 172.21.15.104:0/1826288867 shutdown_connections 2024-06-09T22:19:18.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.629+0000 7ff487d97700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff46c0a2cb0 0x7ff46c0a5170 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.629+0000 7ff487d97700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff480108530 0x7ff480101f10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.629+0000 7ff487d97700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff48006ac00 0x7ff480101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.629+0000 7ff487d97700 1 --2- 172.21.15.104:0/1826288867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48006b5c0 0x7ff4801019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.629+0000 7ff487d97700 1 -- 172.21.15.104:0/1826288867 >> 172.21.15.104:0/1826288867 conn(0x7ff480077400 msgr2=0x7ff480100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:18.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.630+0000 7ff487d97700 1 -- 172.21.15.104:0/1826288867 shutdown_connections 2024-06-09T22:19:18.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.630+0000 7ff487d97700 1 -- 172.21.15.104:0/1826288867 wait complete. 2024-06-09T22:19:18.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.729+0000 7f295cf21700 1 -- 172.21.15.104:0/2988392715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2958107400 msgr2=0x7f29581077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:18.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.729+0000 7f295cf21700 1 --2- 172.21.15.104:0/2988392715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2958107400 0x7f29581077e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f295000b0a0 tx=0x7f295000b3b0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.729+0000 7f295cf21700 1 -- 172.21.15.104:0/2988392715 shutdown_connections 2024-06-09T22:19:18.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.729+0000 7f295cf21700 1 --2- 172.21.15.104:0/2988392715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2958107400 0x7f29581077e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.729+0000 7f295cf21700 1 --2- 172.21.15.104:0/2988392715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2958101d50 0x7f295810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.729+0000 7f295cf21700 1 --2- 172.21.15.104:0/2988392715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2958101390 0x7f2958101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.729+0000 7f295cf21700 1 -- 172.21.15.104:0/2988392715 >> 172.21.15.104:0/2988392715 conn(0x7f2958078340 msgr2=0x7f2958078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:18.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.729+0000 7f295cf21700 1 -- 172.21.15.104:0/2988392715 shutdown_connections 2024-06-09T22:19:18.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.729+0000 7f295cf21700 1 -- 172.21.15.104:0/2988392715 wait complete. 2024-06-09T22:19:18.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.730+0000 7f295cf21700 1 Processor -- start 2024-06-09T22:19:18.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.730+0000 7f295cf21700 1 -- start start 2024-06-09T22:19:18.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.730+0000 7f295cf21700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2958101390 0x7f2958199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:18.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.730+0000 7f295cf21700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2958101d50 0x7f295819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:18.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.730+0000 7f295cf21700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2958107400 0x7f29581a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:18.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.730+0000 7f295cf21700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f29581116f0 con 0x7f2958101d50 2024-06-09T22:19:18.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.730+0000 7f295cf21700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2958111570 con 0x7f2958107400 2024-06-09T22:19:18.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.730+0000 7f295cf21700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2958111870 con 0x7f2958101390 2024-06-09T22:19:18.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.730+0000 7f2955d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2958101d50 0x7f295819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.730+0000 7f2955d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2958101d50 0x7f295819a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50046/0 (socket says 172.21.15.104:50046) 2024-06-09T22:19:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.730+0000 7f2955d9b700 1 -- 172.21.15.104:0/468199826 learned_addr learned my addr 172.21.15.104:0/468199826 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f2956d9d700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2958107400 0x7f29581a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f295659c700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2958101390 0x7f2958199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f2955d9b700 1 -- 172.21.15.104:0/468199826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2958101390 msgr2=0x7f2958199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f2955d9b700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2958101390 0x7f2958199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f2955d9b700 1 -- 172.21.15.104:0/468199826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2958107400 msgr2=0x7f29581a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f2955d9b700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2958107400 0x7f29581a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f2955d9b700 1 -- 172.21.15.104:0/468199826 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29581a5090 con 0x7f2958101d50 2024-06-09T22:19:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f2955d9b700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2958101d50 0x7f295819a010 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f294000d950 tx=0x7f294000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f294f7fe700 1 -- 172.21.15.104:0/468199826 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2940016070 con 0x7f2958101d50 2024-06-09T22:19:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f294f7fe700 1 -- 172.21.15.104:0/468199826 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2940009e20 con 0x7f2958101d50 2024-06-09T22:19:18.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f295cf21700 1 -- 172.21.15.104:0/468199826 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29581a52c0 con 0x7f2958101d50 2024-06-09T22:19:18.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f294f7fe700 1 -- 172.21.15.104:0/468199826 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294000f620 con 0x7f2958101d50 2024-06-09T22:19:18.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.731+0000 7f295cf21700 1 -- 172.21.15.104:0/468199826 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29581a57d0 con 0x7f2958101d50 2024-06-09T22:19:18.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.732+0000 7f295cf21700 1 -- 172.21.15.104:0/468199826 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2958069180 con 0x7f2958101d50 2024-06-09T22:19:18.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.733+0000 7f294f7fe700 1 -- 172.21.15.104:0/468199826 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2940019070 con 0x7f2958101d50 2024-06-09T22:19:18.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.734+0000 7f294f7fe700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f29440a2cb0 0x7f29440a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:18.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.734+0000 7f295659c700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f29440a2cb0 0x7f29440a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:18.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.734+0000 7f294f7fe700 1 -- 172.21.15.104:0/468199826 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f29400cf110 con 0x7f2958101d50 2024-06-09T22:19:18.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.735+0000 7f295659c700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f29440a2cb0 0x7f29440a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2950006130 tx=0x7f29500060a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:18.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.739+0000 7f294f7fe700 1 -- 172.21.15.104:0/468199826 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f294009bb80 con 0x7f2958101d50 2024-06-09T22:19:18.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:18 smithi153 ceph-mon[36599]: pgmap v385: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:19:18.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:18 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1519855548' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-09T22:19:18.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:18 smithi104 ceph-mon[32389]: pgmap v385: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:19:18.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:18 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1519855548' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-09T22:19:18.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:18 smithi194 ceph-mon[36856]: pgmap v385: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:19:18.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:18 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1519855548' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-09T22:19:18.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.939+0000 7f295cf21700 1 -- 172.21.15.104:0/468199826 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f295802dca0 con 0x7f2958101d50 2024-06-09T22:19:18.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.940+0000 7f294f7fe700 1 -- 172.21.15.104:0/468199826 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f294009b480 con 0x7f2958101d50 2024-06-09T22:19:18.938 INFO:teuthology.orchestra.run.smithi104.stdout:343597383727 2024-06-09T22:19:18.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.942+0000 7f295cf21700 1 -- 172.21.15.104:0/468199826 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f29440a2cb0 msgr2=0x7f29440a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:18.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.942+0000 7f295cf21700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f29440a2cb0 0x7f29440a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2950006130 tx=0x7f29500060a0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.942+0000 7f295cf21700 1 -- 172.21.15.104:0/468199826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2958101d50 msgr2=0x7f295819a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:18.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.942+0000 7f295cf21700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2958101d50 0x7f295819a010 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f294000d950 tx=0x7f294000dc60 comp rx=0 tx=0).stop 2024-06-09T22:19:18.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.942+0000 7f295cf21700 1 -- 172.21.15.104:0/468199826 shutdown_connections 2024-06-09T22:19:18.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.942+0000 7f295cf21700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f29440a2cb0 0x7f29440a5170 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.942+0000 7f295cf21700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2958107400 0x7f29581a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.942+0000 7f295cf21700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2958101390 0x7f2958199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.942+0000 7f295cf21700 1 --2- 172.21.15.104:0/468199826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2958101d50 0x7f295819a010 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:18.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.942+0000 7f295cf21700 1 -- 172.21.15.104:0/468199826 >> 172.21.15.104:0/468199826 conn(0x7f2958078340 msgr2=0x7f29580fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:18.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.942+0000 7f295cf21700 1 -- 172.21.15.104:0/468199826 shutdown_connections 2024-06-09T22:19:18.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:18.942+0000 7f295cf21700 1 -- 172.21.15.104:0/468199826 wait complete. 2024-06-09T22:19:19.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:19 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1826288867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-09T22:19:19.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:19 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/468199826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-09T22:19:19.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:19 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1826288867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-09T22:19:19.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:19 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/468199826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-09T22:19:19.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:19 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1826288867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-09T22:19:19.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:19 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/468199826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-09T22:19:20.741 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:20 smithi104 ceph-mon[32389]: pgmap v386: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:19:20.775 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920846 got 141733920851 for osd.4 2024-06-09T22:19:20.775 DEBUG:teuthology.parallel:result is None 2024-06-09T22:19:20.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:20 smithi194 ceph-mon[36856]: pgmap v386: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:19:20.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:20 smithi153 ceph-mon[36599]: pgmap v386: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:19:21.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.311+0000 7ff40761f700 1 -- 172.21.15.104:0/1729289355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff400107400 msgr2=0x7ff4001077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:21.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.311+0000 7ff40761f700 1 --2- 172.21.15.104:0/1729289355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff400107400 0x7ff4001077e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc009990 tx=0x7ff3fc009ca0 comp rx=0 tx=0).stop 2024-06-09T22:19:21.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 -- 172.21.15.104:0/1729289355 shutdown_connections 2024-06-09T22:19:21.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 --2- 172.21.15.104:0/1729289355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff400107400 0x7ff4001077e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:21.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 --2- 172.21.15.104:0/1729289355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff400101d60 0x7ff40010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:21.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 --2- 172.21.15.104:0/1729289355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4001013c0 0x7ff400101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:21.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 -- 172.21.15.104:0/1729289355 >> 172.21.15.104:0/1729289355 conn(0x7ff400078330 msgr2=0x7ff400078740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:21.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 -- 172.21.15.104:0/1729289355 shutdown_connections 2024-06-09T22:19:21.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 -- 172.21.15.104:0/1729289355 wait complete. 2024-06-09T22:19:21.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 Processor -- start 2024-06-09T22:19:21.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 -- start start 2024-06-09T22:19:21.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4001013c0 0x7ff400199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:21.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff400101d60 0x7ff40019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:21.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff400107400 0x7ff4001a4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:21.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff4001116c0 con 0x7ff400101d60 2024-06-09T22:19:21.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff400111540 con 0x7ff4001013c0 2024-06-09T22:19:21.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.312+0000 7ff40761f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff400111840 con 0x7ff400107400 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff404bba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff400101d60 0x7ff40019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff404bba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff400101d60 0x7ff40019a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50068/0 (socket says 172.21.15.104:50068) 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff404bba700 1 -- 172.21.15.104:0/328604856 learned_addr learned my addr 172.21.15.104:0/328604856 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff405bbc700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff400107400 0x7ff4001a4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff4053bb700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4001013c0 0x7ff400199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff404bba700 1 -- 172.21.15.104:0/328604856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff400107400 msgr2=0x7ff4001a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff404bba700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff400107400 0x7ff4001a4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff404bba700 1 -- 172.21.15.104:0/328604856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4001013c0 msgr2=0x7ff400199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff404bba700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4001013c0 0x7ff400199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff404bba700 1 -- 172.21.15.104:0/328604856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4001a50e0 con 0x7ff400101d60 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff4053bb700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4001013c0 0x7ff400199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff404bba700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff400101d60 0x7ff40019a060 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000c920 tx=0x7ff3f000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:21.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff3f67fc700 1 -- 172.21.15.104:0/328604856 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0007b00 con 0x7ff400101d60 2024-06-09T22:19:21.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff3f67fc700 1 -- 172.21.15.104:0/328604856 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3f0004d10 con 0x7ff400101d60 2024-06-09T22:19:21.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff40761f700 1 -- 172.21.15.104:0/328604856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4001a5310 con 0x7ff400101d60 2024-06-09T22:19:21.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff3f67fc700 1 -- 172.21.15.104:0/328604856 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f000f450 con 0x7ff400101d60 2024-06-09T22:19:21.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.313+0000 7ff40761f700 1 -- 172.21.15.104:0/328604856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4001a5820 con 0x7ff400101d60 2024-06-09T22:19:21.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.314+0000 7ff40761f700 1 -- 172.21.15.104:0/328604856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff400102ab0 con 0x7ff400101d60 2024-06-09T22:19:21.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.315+0000 7ff3f67fc700 1 -- 172.21.15.104:0/328604856 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff3f0004750 con 0x7ff400101d60 2024-06-09T22:19:21.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.316+0000 7ff3f67fc700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff3ec0a2cb0 0x7ff3ec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:21.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.316+0000 7ff4053bb700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff3ec0a2cb0 0x7ff3ec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:21.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.316+0000 7ff3f67fc700 1 -- 172.21.15.104:0/328604856 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7ff3f00cf570 con 0x7ff400101d60 2024-06-09T22:19:21.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.316+0000 7ff4053bb700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff3ec0a2cb0 0x7ff3ec0a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc000c00 tx=0x7ff3fc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:21.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.319+0000 7ff3f67fc700 1 -- 172.21.15.104:0/328604856 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff3f0014030 con 0x7ff400101d60 2024-06-09T22:19:21.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.525+0000 7ff40761f700 1 -- 172.21.15.104:0/328604856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7ff40002dca0 con 0x7ff400101d60 2024-06-09T22:19:21.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.526+0000 7ff3f67fc700 1 -- 172.21.15.104:0/328604856 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff3f009bfe0 con 0x7ff400101d60 2024-06-09T22:19:21.523 INFO:teuthology.orchestra.run.smithi104.stdout:287762808890 2024-06-09T22:19:21.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.528+0000 7ff40761f700 1 -- 172.21.15.104:0/328604856 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff3ec0a2cb0 msgr2=0x7ff3ec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:21.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.528+0000 7ff40761f700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff3ec0a2cb0 0x7ff3ec0a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc000c00 tx=0x7ff3fc005fb0 comp rx=0 tx=0).stop 2024-06-09T22:19:21.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.528+0000 7ff40761f700 1 -- 172.21.15.104:0/328604856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff400101d60 msgr2=0x7ff40019a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:21.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.528+0000 7ff40761f700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff400101d60 0x7ff40019a060 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000c920 tx=0x7ff3f000cce0 comp rx=0 tx=0).stop 2024-06-09T22:19:21.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.528+0000 7ff40761f700 1 -- 172.21.15.104:0/328604856 shutdown_connections 2024-06-09T22:19:21.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.528+0000 7ff40761f700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff3ec0a2cb0 0x7ff3ec0a5170 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:21.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.528+0000 7ff40761f700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff400107400 0x7ff4001a4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:21.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.528+0000 7ff40761f700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4001013c0 0x7ff400199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:21.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.528+0000 7ff40761f700 1 --2- 172.21.15.104:0/328604856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff400101d60 0x7ff40019a060 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:21.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.528+0000 7ff40761f700 1 -- 172.21.15.104:0/328604856 >> 172.21.15.104:0/328604856 conn(0x7ff400078330 msgr2=0x7ff400100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:21.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.528+0000 7ff40761f700 1 -- 172.21.15.104:0/328604856 shutdown_connections 2024-06-09T22:19:21.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:21.528+0000 7ff40761f700 1 -- 172.21.15.104:0/328604856 wait complete. 2024-06-09T22:19:22.591 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:22 smithi104 ceph-mon[32389]: pgmap v387: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:19:22.591 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:22 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/328604856' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-09T22:19:22.620 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645359 got 309237645364 for osd.10 2024-06-09T22:19:22.620 DEBUG:teuthology.parallel:result is None 2024-06-09T22:19:22.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:22 smithi194 ceph-mon[36856]: pgmap v387: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:19:22.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:22 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/328604856' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-09T22:19:22.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.824+0000 7f1708a26700 1 -- 172.21.15.104:0/1481283943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17041066d0 msgr2=0x7f170410e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:22.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.824+0000 7f1708a26700 1 --2- 172.21.15.104:0/1481283943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17041066d0 0x7f170410e2a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00b0d0 tx=0x7f16fc00b3e0 comp rx=0 tx=0).stop 2024-06-09T22:19:22.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.825+0000 7f1708a26700 1 -- 172.21.15.104:0/1481283943 shutdown_connections 2024-06-09T22:19:22.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.825+0000 7f1708a26700 1 --2- 172.21.15.104:0/1481283943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17041066d0 0x7f170410e2a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:22.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.825+0000 7f1708a26700 1 --2- 172.21.15.104:0/1481283943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1704101680 0x7f1704101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:22.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.825+0000 7f1708a26700 1 --2- 172.21.15.104:0/1481283943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1704102030 0x7f1704105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:22.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.825+0000 7f1708a26700 1 -- 172.21.15.104:0/1481283943 >> 172.21.15.104:0/1481283943 conn(0x7f17040773e0 msgr2=0x7f17040777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:22.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.825+0000 7f1708a26700 1 -- 172.21.15.104:0/1481283943 shutdown_connections 2024-06-09T22:19:22.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.825+0000 7f1708a26700 1 -- 172.21.15.104:0/1481283943 wait complete. 2024-06-09T22:19:22.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.825+0000 7f1708a26700 1 Processor -- start 2024-06-09T22:19:22.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.825+0000 7f1708a26700 1 -- start start 2024-06-09T22:19:22.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1708a26700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1704101680 0x7f1704199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:22.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1708a26700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1704102030 0x7f1704199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:22.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1708a26700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17041066d0 0x7f17041a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:22.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1708a26700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1704111750 con 0x7f1704102030 2024-06-09T22:19:22.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1708a26700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17041115d0 con 0x7f1704101680 2024-06-09T22:19:22.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1708a26700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f17041118d0 con 0x7f17041066d0 2024-06-09T22:19:22.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1701d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1704102030 0x7f1704199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:22.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1701d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1704102030 0x7f1704199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50084/0 (socket says 172.21.15.104:50084) 2024-06-09T22:19:22.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1701d9b700 1 -- 172.21.15.104:0/2737344862 learned_addr learned my addr 172.21.15.104:0/2737344862 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:22.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f170259c700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1704101680 0x7f1704199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:22.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1702d9d700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17041066d0 0x7f17041a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:22.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1701d9b700 1 -- 172.21.15.104:0/2737344862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17041066d0 msgr2=0x7f17041a4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:22.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1701d9b700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17041066d0 0x7f17041a4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:22.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1701d9b700 1 -- 172.21.15.104:0/2737344862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1704101680 msgr2=0x7f1704199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:22.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1701d9b700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1704101680 0x7f1704199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:22.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1701d9b700 1 -- 172.21.15.104:0/2737344862 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17041a5070 con 0x7f1704102030 2024-06-09T22:19:22.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1701d9b700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1704102030 0x7f1704199ff0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f16f800b500 tx=0x7f16f800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:22.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f16f37fe700 1 -- 172.21.15.104:0/2737344862 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f8015020 con 0x7f1704102030 2024-06-09T22:19:22.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f1708a26700 1 -- 172.21.15.104:0/2737344862 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17041a5240 con 0x7f1704102030 2024-06-09T22:19:22.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.826+0000 7f16f37fe700 1 -- 172.21.15.104:0/2737344862 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16f8011d10 con 0x7f1704102030 2024-06-09T22:19:22.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.827+0000 7f16f37fe700 1 -- 172.21.15.104:0/2737344862 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f800f2e0 con 0x7f1704102030 2024-06-09T22:19:22.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.827+0000 7f1708a26700 1 -- 172.21.15.104:0/2737344862 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17041a5780 con 0x7f1704102030 2024-06-09T22:19:22.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.827+0000 7f1708a26700 1 -- 172.21.15.104:0/2737344862 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1704069180 con 0x7f1704102030 2024-06-09T22:19:22.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.829+0000 7f16f37fe700 1 -- 172.21.15.104:0/2737344862 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f16f8018020 con 0x7f1704102030 2024-06-09T22:19:22.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.829+0000 7f16f37fe700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f16ec0a2cb0 0x7f16ec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:22.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.829+0000 7f170259c700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f16ec0a2cb0 0x7f16ec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:22.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.829+0000 7f16f37fe700 1 -- 172.21.15.104:0/2737344862 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f16f80cf0e0 con 0x7f1704102030 2024-06-09T22:19:22.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.830+0000 7f170259c700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f16ec0a2cb0 0x7f16ec0a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005950 tx=0x7f16f4005eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:22.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:22.833+0000 7f16f37fe700 1 -- 172.21.15.104:0/2737344862 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f16f801f020 con 0x7f1704102030 2024-06-09T22:19:22.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:22 smithi153 ceph-mon[36599]: pgmap v387: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:19:22.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:22 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/328604856' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-09T22:19:23.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.031+0000 7f1708a26700 1 -- 172.21.15.104:0/2737344862 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f170406b1c0 con 0x7f1704102030 2024-06-09T22:19:23.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.031+0000 7f16f37fe700 1 -- 172.21.15.104:0/2737344862 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f16f809bc60 con 0x7f1704102030 2024-06-09T22:19:23.029 INFO:teuthology.orchestra.run.smithi104.stdout:163208757327 2024-06-09T22:19:23.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.033+0000 7f1708a26700 1 -- 172.21.15.104:0/2737344862 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f16ec0a2cb0 msgr2=0x7f16ec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:23.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.033+0000 7f1708a26700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f16ec0a2cb0 0x7f16ec0a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005950 tx=0x7f16f4005eb0 comp rx=0 tx=0).stop 2024-06-09T22:19:23.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.033+0000 7f1708a26700 1 -- 172.21.15.104:0/2737344862 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1704102030 msgr2=0x7f1704199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:23.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.033+0000 7f1708a26700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1704102030 0x7f1704199ff0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f16f800b500 tx=0x7f16f800b8c0 comp rx=0 tx=0).stop 2024-06-09T22:19:23.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.033+0000 7f1708a26700 1 -- 172.21.15.104:0/2737344862 shutdown_connections 2024-06-09T22:19:23.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.033+0000 7f1708a26700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f16ec0a2cb0 0x7f16ec0a5170 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:23.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.033+0000 7f1708a26700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17041066d0 0x7f17041a4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:23.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.033+0000 7f1708a26700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1704101680 0x7f1704199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:23.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.033+0000 7f1708a26700 1 --2- 172.21.15.104:0/2737344862 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1704102030 0x7f1704199ff0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:23.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.033+0000 7f1708a26700 1 -- 172.21.15.104:0/2737344862 >> 172.21.15.104:0/2737344862 conn(0x7f17040773e0 msgr2=0x7f17040ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:23.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.033+0000 7f1708a26700 1 -- 172.21.15.104:0/2737344862 shutdown_connections 2024-06-09T22:19:23.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:23.034+0000 7f1708a26700 1 -- 172.21.15.104:0/2737344862 wait complete. 2024-06-09T22:19:23.395 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:23.421 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332157 got 219043332161 for osd.7 2024-06-09T22:19:23.422 DEBUG:teuthology.parallel:result is None 2024-06-09T22:19:23.668 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:23 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2737344862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-09T22:19:23.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:23 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2737344862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-09T22:19:23.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:23 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2737344862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-09T22:19:24.323 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808886 got 287762808890 for osd.9 2024-06-09T22:19:24.323 DEBUG:teuthology.parallel:result is None 2024-06-09T22:19:24.568 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:24 smithi104 ceph-mon[32389]: pgmap v388: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:19:24.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:24 smithi194 ceph-mon[36856]: pgmap v388: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:19:24.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:24 smithi153 ceph-mon[36599]: pgmap v388: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:19:25.271 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757320 got 163208757327 for osd.5 2024-06-09T22:19:25.272 DEBUG:teuthology.parallel:result is None 2024-06-09T22:19:25.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.450+0000 7f3e6b896700 1 -- 172.21.15.104:0/705785716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e64107880 msgr2=0x7f3e6410f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:25.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.450+0000 7f3e6b896700 1 --2- 172.21.15.104:0/705785716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e64107880 0x7f3e6410f400 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c00b0d0 tx=0x7f3e5c00b3e0 comp rx=0 tx=0).stop 2024-06-09T22:19:25.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.450+0000 7f3e6b896700 1 -- 172.21.15.104:0/705785716 shutdown_connections 2024-06-09T22:19:25.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.450+0000 7f3e6b896700 1 --2- 172.21.15.104:0/705785716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e64107880 0x7f3e6410f400 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:25.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.450+0000 7f3e6b896700 1 --2- 172.21.15.104:0/705785716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64100c40 0x7f3e64101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:25.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.450+0000 7f3e6b896700 1 --2- 172.21.15.104:0/705785716 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e641015f0 0x7f3e64107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:25.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.450+0000 7f3e6b896700 1 -- 172.21.15.104:0/705785716 >> 172.21.15.104:0/705785716 conn(0x7f3e64077990 msgr2=0x7f3e640fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:25.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.450+0000 7f3e6b896700 1 -- 172.21.15.104:0/705785716 shutdown_connections 2024-06-09T22:19:25.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.450+0000 7f3e6b896700 1 -- 172.21.15.104:0/705785716 wait complete. 2024-06-09T22:19:25.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e6b896700 1 Processor -- start 2024-06-09T22:19:25.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e6b896700 1 -- start start 2024-06-09T22:19:25.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e6b896700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64100c40 0x7f3e64101bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e6b896700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e641015f0 0x7f3e64102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e6b896700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e64107880 0x7f3e64102640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e6b896700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3e641127a0 con 0x7f3e641015f0 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e6b896700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e64112620 con 0x7f3e64100c40 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e6b896700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3e64112920 con 0x7f3e64107880 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e68e31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e641015f0 0x7f3e64102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e68e31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e641015f0 0x7f3e64102100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50100/0 (socket says 172.21.15.104:50100) 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e68e31700 1 -- 172.21.15.104:0/1116666017 learned_addr learned my addr 172.21.15.104:0/1116666017 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e69632700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64100c40 0x7f3e64101bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e69e33700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e64107880 0x7f3e64102640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e68e31700 1 -- 172.21.15.104:0/1116666017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e64107880 msgr2=0x7f3e64102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e68e31700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e64107880 0x7f3e64102640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e68e31700 1 -- 172.21.15.104:0/1116666017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64100c40 msgr2=0x7f3e64101bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:25.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e68e31700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64100c40 0x7f3e64101bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:25.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e68e31700 1 -- 172.21.15.104:0/1116666017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e641a7080 con 0x7f3e641015f0 2024-06-09T22:19:25.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.451+0000 7f3e68e31700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e641015f0 0x7f3e64102100 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400d980 tx=0x7f3e5400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:25.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.452+0000 7f3e5a7fc700 1 -- 172.21.15.104:0/1116666017 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e54016070 con 0x7f3e641015f0 2024-06-09T22:19:25.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.452+0000 7f3e5a7fc700 1 -- 172.21.15.104:0/1116666017 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e54004d10 con 0x7f3e641015f0 2024-06-09T22:19:25.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.452+0000 7f3e6b896700 1 -- 172.21.15.104:0/1116666017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e641a7370 con 0x7f3e641015f0 2024-06-09T22:19:25.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.452+0000 7f3e5a7fc700 1 -- 172.21.15.104:0/1116666017 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e540055f0 con 0x7f3e641015f0 2024-06-09T22:19:25.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.452+0000 7f3e6b896700 1 -- 172.21.15.104:0/1116666017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e641a78b0 con 0x7f3e641015f0 2024-06-09T22:19:25.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.453+0000 7f3e6b896700 1 -- 172.21.15.104:0/1116666017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e64069180 con 0x7f3e641015f0 2024-06-09T22:19:25.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.455+0000 7f3e5a7fc700 1 -- 172.21.15.104:0/1116666017 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f3e54004750 con 0x7f3e641015f0 2024-06-09T22:19:25.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.456+0000 7f3e5a7fc700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3e500a2d30 0x7f3e500a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:25.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.456+0000 7f3e69632700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3e500a2d30 0x7f3e500a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:25.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.456+0000 7f3e5a7fc700 1 -- 172.21.15.104:0/1116666017 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f3e540cf8b0 con 0x7f3e641015f0 2024-06-09T22:19:25.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.456+0000 7f3e69632700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3e500a2d30 0x7f3e500a51f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3e60005950 tx=0x7f3e6000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:25.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.459+0000 7f3e5a7fc700 1 -- 172.21.15.104:0/1116666017 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3e5401f020 con 0x7f3e641015f0 2024-06-09T22:19:25.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.527+0000 7f3e5a7fc700 1 -- 172.21.15.104:0/1116666017 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3e54098230 con 0x7f3e641015f0 2024-06-09T22:19:25.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.692+0000 7f3e6b896700 1 -- 172.21.15.104:0/1116666017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f3e6402de60 con 0x7f3e641015f0 2024-06-09T22:19:25.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.693+0000 7f3e5a7fc700 1 -- 172.21.15.104:0/1116666017 <== mon.0 v2:172.21.15.104:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3e540c6df0 con 0x7f3e641015f0 2024-06-09T22:19:25.690 INFO:teuthology.orchestra.run.smithi104.stdout:34359738472 2024-06-09T22:19:25.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.695+0000 7f3e6b896700 1 -- 172.21.15.104:0/1116666017 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3e500a2d30 msgr2=0x7f3e500a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:25.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.695+0000 7f3e6b896700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3e500a2d30 0x7f3e500a51f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3e60005950 tx=0x7f3e6000b410 comp rx=0 tx=0).stop 2024-06-09T22:19:25.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.695+0000 7f3e6b896700 1 -- 172.21.15.104:0/1116666017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e641015f0 msgr2=0x7f3e64102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:25.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.695+0000 7f3e6b896700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e641015f0 0x7f3e64102100 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400d980 tx=0x7f3e5400dc90 comp rx=0 tx=0).stop 2024-06-09T22:19:25.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.695+0000 7f3e6b896700 1 -- 172.21.15.104:0/1116666017 shutdown_connections 2024-06-09T22:19:25.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.695+0000 7f3e6b896700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3e500a2d30 0x7f3e500a51f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:25.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.695+0000 7f3e6b896700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e64107880 0x7f3e64102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:25.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.695+0000 7f3e6b896700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64100c40 0x7f3e64101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:25.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.695+0000 7f3e6b896700 1 --2- 172.21.15.104:0/1116666017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e641015f0 0x7f3e64102100 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:25.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.695+0000 7f3e6b896700 1 -- 172.21.15.104:0/1116666017 >> 172.21.15.104:0/1116666017 conn(0x7f3e64077990 msgr2=0x7f3e64105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:25.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.695+0000 7f3e6b896700 1 -- 172.21.15.104:0/1116666017 shutdown_connections 2024-06-09T22:19:25.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:25.695+0000 7f3e6b896700 1 -- 172.21.15.104:0/1116666017 wait complete. 2024-06-09T22:19:26.477 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738466 got 34359738472 for osd.0 2024-06-09T22:19:26.478 DEBUG:teuthology.parallel:result is None 2024-06-09T22:19:26.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.671+0000 7fbddbde6700 1 -- 172.21.15.104:0/2228698067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdd4107840 msgr2=0x7fbdd410f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:26.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.671+0000 7fbddbde6700 1 --2- 172.21.15.104:0/2228698067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdd4107840 0x7fbdd410f3c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc00b0d0 tx=0x7fbdcc00b3e0 comp rx=0 tx=0).stop 2024-06-09T22:19:26.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.671+0000 7fbddbde6700 1 -- 172.21.15.104:0/2228698067 shutdown_connections 2024-06-09T22:19:26.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.671+0000 7fbddbde6700 1 --2- 172.21.15.104:0/2228698067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdd4107840 0x7fbdd410f3c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:26.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.671+0000 7fbddbde6700 1 --2- 172.21.15.104:0/2228698067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdd4102770 0x7fbdd4102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:26.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.671+0000 7fbddbde6700 1 --2- 172.21.15.104:0/2228698067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd4103120 0x7fbdd4107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:26.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.671+0000 7fbddbde6700 1 -- 172.21.15.104:0/2228698067 >> 172.21.15.104:0/2228698067 conn(0x7fbdd4079500 msgr2=0x7fbdd4079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:26.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.671+0000 7fbddbde6700 1 -- 172.21.15.104:0/2228698067 shutdown_connections 2024-06-09T22:19:26.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.671+0000 7fbddbde6700 1 -- 172.21.15.104:0/2228698067 wait complete. 2024-06-09T22:19:26.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.672+0000 7fbddbde6700 1 Processor -- start 2024-06-09T22:19:26.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.672+0000 7fbddbde6700 1 -- start start 2024-06-09T22:19:26.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.672+0000 7fbddbde6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdd4102770 0x7fbdd419e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:26.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.672+0000 7fbddbde6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdd4103120 0x7fbdd419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:26.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.672+0000 7fbddbde6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd4107840 0x7fbdd4198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:26.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.672+0000 7fbddbde6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd4112780 con 0x7fbdd4102770 2024-06-09T22:19:26.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.672+0000 7fbddbde6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd4112600 con 0x7fbdd4107840 2024-06-09T22:19:26.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.672+0000 7fbddbde6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd4112900 con 0x7fbdd4103120 2024-06-09T22:19:26.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.672+0000 7fbdd9b82700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdd4102770 0x7fbdd419e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:26.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.672+0000 7fbdd9b82700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdd4102770 0x7fbdd419e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38452/0 (socket says 172.21.15.104:38452) 2024-06-09T22:19:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.672+0000 7fbdd9b82700 1 -- 172.21.15.104:0/1506986978 learned_addr learned my addr 172.21.15.104:0/1506986978 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.672+0000 7fbdd9381700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdd4103120 0x7fbdd419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.673+0000 7fbdda383700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd4107840 0x7fbdd4198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.673+0000 7fbdd9b82700 1 -- 172.21.15.104:0/1506986978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdd4103120 msgr2=0x7fbdd419ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.673+0000 7fbdd9b82700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdd4103120 0x7fbdd419ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.673+0000 7fbdd9b82700 1 -- 172.21.15.104:0/1506986978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd4107840 msgr2=0x7fbdd4198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.673+0000 7fbdd9b82700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd4107840 0x7fbdd4198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.673+0000 7fbdd9b82700 1 -- 172.21.15.104:0/1506986978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd41990a0 con 0x7fbdd4102770 2024-06-09T22:19:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.673+0000 7fbdd9b82700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdd4102770 0x7fbdd419e780 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbdd000d980 tx=0x7fbdd000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.673+0000 7fbdcaffd700 1 -- 172.21.15.104:0/1506986978 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0017070 con 0x7fbdd4102770 2024-06-09T22:19:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.673+0000 7fbdcaffd700 1 -- 172.21.15.104:0/1506986978 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdd0005020 con 0x7fbdd4102770 2024-06-09T22:19:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.673+0000 7fbddbde6700 1 -- 172.21.15.104:0/1506986978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd4199390 con 0x7fbdd4102770 2024-06-09T22:19:26.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.673+0000 7fbdcaffd700 1 -- 172.21.15.104:0/1506986978 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0009bc0 con 0x7fbdd4102770 2024-06-09T22:19:26.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.673+0000 7fbddbde6700 1 -- 172.21.15.104:0/1506986978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdd40746c0 con 0x7fbdd4102770 2024-06-09T22:19:26.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.674+0000 7fbddbde6700 1 -- 172.21.15.104:0/1506986978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdd4069180 con 0x7fbdd4102770 2024-06-09T22:19:26.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.675+0000 7fbdcaffd700 1 -- 172.21.15.104:0/1506986978 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fbdd00051d0 con 0x7fbdd4102770 2024-06-09T22:19:26.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.676+0000 7fbdcaffd700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbdc00a2c20 0x7fbdc00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:26.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.676+0000 7fbdd9381700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbdc00a2c20 0x7fbdc00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:26.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.676+0000 7fbdcaffd700 1 -- 172.21.15.104:0/1506986978 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fbdd00cf300 con 0x7fbdd4102770 2024-06-09T22:19:26.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.676+0000 7fbdd9381700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbdc00a2c20 0x7fbdc00a50e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4102010 tx=0x7fbdc400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:26.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.680+0000 7fbdcaffd700 1 -- 172.21.15.104:0/1506986978 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbdd0020020 con 0x7fbdd4102770 2024-06-09T22:19:26.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:26 smithi104 ceph-mon[32389]: pgmap v389: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:26.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:26 smithi104 ceph-mon[32389]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-06-09T22:19:26.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:26 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1116666017' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-09T22:19:26.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:26 smithi194 ceph-mon[36856]: pgmap v389: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:26.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:26 smithi194 ceph-mon[36856]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-06-09T22:19:26.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:26 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1116666017' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-09T22:19:26.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.891+0000 7fbddbde6700 1 -- 172.21.15.104:0/1506986978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fbdd402dca0 con 0x7fbdd4102770 2024-06-09T22:19:26.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.892+0000 7fbdcaffd700 1 -- 172.21.15.104:0/1506986978 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbdd009bed0 con 0x7fbdd4102770 2024-06-09T22:19:26.889 INFO:teuthology.orchestra.run.smithi104.stdout:253403070528 2024-06-09T22:19:26.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.894+0000 7fbddbde6700 1 -- 172.21.15.104:0/1506986978 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbdc00a2c20 msgr2=0x7fbdc00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:26.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.894+0000 7fbddbde6700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbdc00a2c20 0x7fbdc00a50e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4102010 tx=0x7fbdc400b410 comp rx=0 tx=0).stop 2024-06-09T22:19:26.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.894+0000 7fbddbde6700 1 -- 172.21.15.104:0/1506986978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdd4102770 msgr2=0x7fbdd419e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:26.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.894+0000 7fbddbde6700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdd4102770 0x7fbdd419e780 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbdd000d980 tx=0x7fbdd000dc90 comp rx=0 tx=0).stop 2024-06-09T22:19:26.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.894+0000 7fbddbde6700 1 -- 172.21.15.104:0/1506986978 shutdown_connections 2024-06-09T22:19:26.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.894+0000 7fbddbde6700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fbdc00a2c20 0x7fbdc00a50e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:26.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.894+0000 7fbddbde6700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdd4102770 0x7fbdd419e780 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:26.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.894+0000 7fbddbde6700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd4107840 0x7fbdd4198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:26.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.894+0000 7fbddbde6700 1 --2- 172.21.15.104:0/1506986978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdd4103120 0x7fbdd419ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:26.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.894+0000 7fbddbde6700 1 -- 172.21.15.104:0/1506986978 >> 172.21.15.104:0/1506986978 conn(0x7fbdd4079500 msgr2=0x7fbdd41006b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:26.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.894+0000 7fbddbde6700 1 -- 172.21.15.104:0/1506986978 shutdown_connections 2024-06-09T22:19:26.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:26.894+0000 7fbddbde6700 1 -- 172.21.15.104:0/1506986978 wait complete. 2024-06-09T22:19:26.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:26 smithi153 ceph-mon[36599]: pgmap v389: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:26.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:26 smithi153 ceph-mon[36599]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-06-09T22:19:26.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:26 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1116666017' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-09T22:19:27.594 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383722 got 343597383727 for osd.11 2024-06-09T22:19:27.594 DEBUG:teuthology.parallel:result is None 2024-06-09T22:19:27.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:27 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1506986978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-09T22:19:27.881 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:27 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1506986978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-09T22:19:27.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:27 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1506986978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-09T22:19:28.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:28 smithi104 ceph-mon[32389]: pgmap v390: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-06-09T22:19:28.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:28 smithi194 ceph-mon[36856]: pgmap v390: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-06-09T22:19:28.825 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070522 got 253403070528 for osd.8 2024-06-09T22:19:28.826 DEBUG:teuthology.parallel:result is None 2024-06-09T22:19:28.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:28 smithi153 ceph-mon[36599]: pgmap v390: 129 pgs: 129 active+clean; 650 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-06-09T22:19:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.324+0000 7f7887bf0700 1 -- 172.21.15.104:0/496413492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788006ab30 msgr2=0x7f788010f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.324+0000 7f7887bf0700 1 --2- 172.21.15.104:0/496413492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788006ab30 0x7f788010f320 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f787800b0a0 tx=0x7f787800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:19:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.324+0000 7f7887bf0700 1 -- 172.21.15.104:0/496413492 shutdown_connections 2024-06-09T22:19:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.324+0000 7f7887bf0700 1 --2- 172.21.15.104:0/496413492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788006ab30 0x7f788010f320 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.324+0000 7f7887bf0700 1 --2- 172.21.15.104:0/496413492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7880108500 0x7f78801088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.324+0000 7f7887bf0700 1 --2- 172.21.15.104:0/496413492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f788006a170 0x7f788006a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.324+0000 7f7887bf0700 1 -- 172.21.15.104:0/496413492 >> 172.21.15.104:0/496413492 conn(0x7f7880078340 msgr2=0x7f7880078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:29.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.324+0000 7f7887bf0700 1 -- 172.21.15.104:0/496413492 shutdown_connections 2024-06-09T22:19:29.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.324+0000 7f7887bf0700 1 -- 172.21.15.104:0/496413492 wait complete. 2024-06-09T22:19:29.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f7887bf0700 1 Processor -- start 2024-06-09T22:19:29.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f7887bf0700 1 -- start start 2024-06-09T22:19:29.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f7887bf0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788006a170 0x7f788019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:29.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f7887bf0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f788006ab30 0x7f788019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:29.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f7887bf0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7880108500 0x7f78801989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:29.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f7887bf0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7880112780 con 0x7f788006a170 2024-06-09T22:19:29.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f7887bf0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7880112600 con 0x7f7880108500 2024-06-09T22:19:29.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f7887bf0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7880112900 con 0x7f788006ab30 2024-06-09T22:19:29.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f788598c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788006a170 0x7f788019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:29.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f788598c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788006a170 0x7f788019e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38462/0 (socket says 172.21.15.104:38462) 2024-06-09T22:19:29.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f788598c700 1 -- 172.21.15.104:0/114699051 learned_addr learned my addr 172.21.15.104:0/114699051 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:29.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f788618d700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7880108500 0x7f78801989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:29.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f788518b700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f788006ab30 0x7f788019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:29.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f788598c700 1 -- 172.21.15.104:0/114699051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f788006ab30 msgr2=0x7f788019ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:29.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f788598c700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f788006ab30 0x7f788019ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:29.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.325+0000 7f788598c700 1 -- 172.21.15.104:0/114699051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7880108500 msgr2=0x7f78801989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:29.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.326+0000 7f788598c700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7880108500 0x7f78801989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:29.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.326+0000 7f788598c700 1 -- 172.21.15.104:0/114699051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7880199100 con 0x7f788006a170 2024-06-09T22:19:29.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.326+0000 7f788618d700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7880108500 0x7f78801989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:19:29.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.326+0000 7f788598c700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788006a170 0x7f788019e790 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f787c00c9b0 tx=0x7f787c00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:29.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.326+0000 7f7876ffd700 1 -- 172.21.15.104:0/114699051 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c007b00 con 0x7f788006a170 2024-06-09T22:19:29.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.326+0000 7f7887bf0700 1 -- 172.21.15.104:0/114699051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78801993f0 con 0x7f788006a170 2024-06-09T22:19:29.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.326+0000 7f7876ffd700 1 -- 172.21.15.104:0/114699051 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f787c007ca0 con 0x7f788006a170 2024-06-09T22:19:29.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.326+0000 7f7876ffd700 1 -- 172.21.15.104:0/114699051 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c018780 con 0x7f788006a170 2024-06-09T22:19:29.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.326+0000 7f7887bf0700 1 -- 172.21.15.104:0/114699051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78801a5660 con 0x7f788006a170 2024-06-09T22:19:29.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.327+0000 7f7887bf0700 1 -- 172.21.15.104:0/114699051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7880069180 con 0x7f788006a170 2024-06-09T22:19:29.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.328+0000 7f7876ffd700 1 -- 172.21.15.104:0/114699051 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f787c00f450 con 0x7f788006a170 2024-06-09T22:19:29.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.329+0000 7f7876ffd700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f786c0a2ca0 0x7f786c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:29.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.329+0000 7f788518b700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f786c0a2ca0 0x7f786c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:29.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.329+0000 7f7876ffd700 1 -- 172.21.15.104:0/114699051 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f787c0cf360 con 0x7f788006a170 2024-06-09T22:19:29.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.329+0000 7f788518b700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f786c0a2ca0 0x7f786c0a5160 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7880078910 tx=0x7f7870005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:29.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.333+0000 7f7876ffd700 1 -- 172.21.15.104:0/114699051 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f787c014030 con 0x7f788006a170 2024-06-09T22:19:29.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.542+0000 7f7887bf0700 1 -- 172.21.15.104:0/114699051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f7880199e80 con 0x7f788006a170 2024-06-09T22:19:29.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.543+0000 7f7876ffd700 1 -- 172.21.15.104:0/114699051 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f787c09bf30 con 0x7f788006a170 2024-06-09T22:19:29.541 INFO:teuthology.orchestra.run.smithi104.stdout:77309411423 2024-06-09T22:19:29.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.545+0000 7f7887bf0700 1 -- 172.21.15.104:0/114699051 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f786c0a2ca0 msgr2=0x7f786c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:29.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.545+0000 7f7887bf0700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f786c0a2ca0 0x7f786c0a5160 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7880078910 tx=0x7f7870005cf0 comp rx=0 tx=0).stop 2024-06-09T22:19:29.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.545+0000 7f7887bf0700 1 -- 172.21.15.104:0/114699051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788006a170 msgr2=0x7f788019e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:29.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.545+0000 7f7887bf0700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788006a170 0x7f788019e790 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f787c00c9b0 tx=0x7f787c00cd70 comp rx=0 tx=0).stop 2024-06-09T22:19:29.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.545+0000 7f7887bf0700 1 -- 172.21.15.104:0/114699051 shutdown_connections 2024-06-09T22:19:29.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.545+0000 7f7887bf0700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f786c0a2ca0 0x7f786c0a5160 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:29.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.545+0000 7f7887bf0700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788006a170 0x7f788019e790 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:29.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.545+0000 7f7887bf0700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7880108500 0x7f78801989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:29.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.545+0000 7f7887bf0700 1 --2- 172.21.15.104:0/114699051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f788006ab30 0x7f788019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:29.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.545+0000 7f7887bf0700 1 -- 172.21.15.104:0/114699051 >> 172.21.15.104:0/114699051 conn(0x7f7880078340 msgr2=0x7f78800fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:29.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.545+0000 7f7887bf0700 1 -- 172.21.15.104:0/114699051 shutdown_connections 2024-06-09T22:19:29.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:29.545+0000 7f7887bf0700 1 -- 172.21.15.104:0/114699051 wait complete. 2024-06-09T22:19:30.326 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411416 got 77309411423 for osd.2 2024-06-09T22:19:30.327 DEBUG:teuthology.parallel:result is None 2024-06-09T22:19:30.327 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-09T22:19:30.327 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-09T22:19:30.577 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:30.699 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:30 smithi104 ceph-mon[32389]: pgmap v391: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 11 op/s 2024-06-09T22:19:30.699 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:19:30.699 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:30 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:19:30.699 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:30 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/114699051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-09T22:19:30.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:30 smithi153 ceph-mon[36599]: pgmap v391: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 11 op/s 2024-06-09T22:19:30.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:19:30.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:30 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:19:30.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:30 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/114699051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-09T22:19:30.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:30 smithi194 ceph-mon[36856]: pgmap v391: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 11 op/s 2024-06-09T22:19:30.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:19:30.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:30 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:19:30.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:30 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/114699051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-09T22:19:31.740 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:19:31.740 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:19:31.740 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:19:31.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:19:31.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:19:31.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:19:31.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:19:31.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:19:31.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:19:32.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.339+0000 7f76ce2b5700 1 -- 172.21.15.104:0/4121809925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c810ef70 msgr2=0x7f76c8111360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:32.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.339+0000 7f76ce2b5700 1 --2- 172.21.15.104:0/4121809925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c810ef70 0x7f76c8111360 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f76c000b0a0 tx=0x7f76c000b3b0 comp rx=0 tx=0).stop 2024-06-09T22:19:32.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.339+0000 7f76ce2b5700 1 -- 172.21.15.104:0/4121809925 shutdown_connections 2024-06-09T22:19:32.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.339+0000 7f76ce2b5700 1 --2- 172.21.15.104:0/4121809925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c810ef70 0x7f76c8111360 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:32.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.339+0000 7f76ce2b5700 1 --2- 172.21.15.104:0/4121809925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76c806abe0 0x7f76c806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:32.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.339+0000 7f76ce2b5700 1 --2- 172.21.15.104:0/4121809925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76c806b500 0x7f76c810ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:32.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.339+0000 7f76ce2b5700 1 -- 172.21.15.104:0/4121809925 >> 172.21.15.104:0/4121809925 conn(0x7f76c8078360 msgr2=0x7f76c8078770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:32.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.339+0000 7f76ce2b5700 1 -- 172.21.15.104:0/4121809925 shutdown_connections 2024-06-09T22:19:32.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.339+0000 7f76ce2b5700 1 -- 172.21.15.104:0/4121809925 wait complete. 2024-06-09T22:19:32.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.340+0000 7f76ce2b5700 1 Processor -- start 2024-06-09T22:19:32.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.340+0000 7f76ce2b5700 1 -- start start 2024-06-09T22:19:32.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.340+0000 7f76ce2b5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76c806abe0 0x7f76c819aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:32.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.340+0000 7f76ce2b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806b500 0x7f76c819b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:32.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.340+0000 7f76ce2b5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76c810ef70 0x7f76c819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:32.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.340+0000 7f76ce2b5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f76c81149a0 con 0x7f76c806b500 2024-06-09T22:19:32.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.340+0000 7f76ce2b5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f76c8114820 con 0x7f76c806abe0 2024-06-09T22:19:32.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.340+0000 7f76ce2b5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f76c8114b20 con 0x7f76c810ef70 2024-06-09T22:19:32.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.340+0000 7f76c77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806b500 0x7f76c819b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:32.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.340+0000 7f76c77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806b500 0x7f76c819b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38488/0 (socket says 172.21.15.104:38488) 2024-06-09T22:19:32.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.340+0000 7f76c77fe700 1 -- 172.21.15.104:0/679948272 learned_addr learned my addr 172.21.15.104:0/679948272 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:32.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76c7fff700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76c806abe0 0x7f76c819aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:32.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76cc852700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76c810ef70 0x7f76c819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:32.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76c77fe700 1 -- 172.21.15.104:0/679948272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76c810ef70 msgr2=0x7f76c819f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76c77fe700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76c810ef70 0x7f76c819f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76c77fe700 1 -- 172.21.15.104:0/679948272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76c806abe0 msgr2=0x7f76c819aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76c77fe700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76c806abe0 0x7f76c819aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76c77fe700 1 -- 172.21.15.104:0/679948272 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c819fe70 con 0x7f76c806b500 2024-06-09T22:19:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76c7fff700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76c806abe0 0x7f76c819aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:19:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76c77fe700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806b500 0x7f76c819b3f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00c970 tx=0x7f76bc00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76c57fa700 1 -- 172.21.15.104:0/679948272 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc007aa0 con 0x7f76c806b500 2024-06-09T22:19:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76ce2b5700 1 -- 172.21.15.104:0/679948272 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76c81a0160 con 0x7f76c806b500 2024-06-09T22:19:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76c57fa700 1 -- 172.21.15.104:0/679948272 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76bc004d10 con 0x7f76c806b500 2024-06-09T22:19:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76c57fa700 1 -- 172.21.15.104:0/679948272 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc005690 con 0x7f76c806b500 2024-06-09T22:19:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.341+0000 7f76ce2b5700 1 -- 172.21.15.104:0/679948272 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c81a79e0 con 0x7f76c806b500 2024-06-09T22:19:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.342+0000 7f76ce2b5700 1 -- 172.21.15.104:0/679948272 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76c8069180 con 0x7f76c806b500 2024-06-09T22:19:32.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.343+0000 7f76c57fa700 1 -- 172.21.15.104:0/679948272 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f76bc004750 con 0x7f76c806b500 2024-06-09T22:19:32.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.344+0000 7f76c57fa700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f76b00a2c20 0x7f76b00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:32.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.344+0000 7f76c7fff700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f76b00a2c20 0x7f76b00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:32.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.344+0000 7f76c57fa700 1 -- 172.21.15.104:0/679948272 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f76bc0cf1f0 con 0x7f76c806b500 2024-06-09T22:19:32.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.344+0000 7f76c7fff700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f76b00a2c20 0x7f76b00a50e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f76b8005950 tx=0x7f76b80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:32.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.348+0000 7f76c57fa700 1 -- 172.21.15.104:0/679948272 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f76bc09be50 con 0x7f76c806b500 2024-06-09T22:19:32.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.544+0000 7f76ce2b5700 1 -- 172.21.15.104:0/679948272 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f76c819c1c0 con 0x7f76b00a2c20 2024-06-09T22:19:32.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.554+0000 7f76c57fa700 1 -- 172.21.15.104:0/679948272 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340901 (secure 0 0 0) 0x7f76c819c1c0 con 0x7f76b00a2c20 2024-06-09T22:19:32.556 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:19:32.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.557+0000 7f76ce2b5700 1 -- 172.21.15.104:0/679948272 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f76b00a2c20 msgr2=0x7f76b00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:32.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.557+0000 7f76ce2b5700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f76b00a2c20 0x7f76b00a50e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f76b8005950 tx=0x7f76b80058e0 comp rx=0 tx=0).stop 2024-06-09T22:19:32.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.557+0000 7f76ce2b5700 1 -- 172.21.15.104:0/679948272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806b500 msgr2=0x7f76c819b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:32.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.557+0000 7f76ce2b5700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806b500 0x7f76c819b3f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00c970 tx=0x7f76bc00cd30 comp rx=0 tx=0).stop 2024-06-09T22:19:32.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.557+0000 7f76ce2b5700 1 -- 172.21.15.104:0/679948272 shutdown_connections 2024-06-09T22:19:32.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.557+0000 7f76ce2b5700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f76b00a2c20 0x7f76b00a50e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:32.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.557+0000 7f76ce2b5700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76c810ef70 0x7f76c819f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:32.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.557+0000 7f76ce2b5700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76c806abe0 0x7f76c819aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:32.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.557+0000 7f76ce2b5700 1 --2- 172.21.15.104:0/679948272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806b500 0x7f76c819b3f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:32.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.557+0000 7f76ce2b5700 1 -- 172.21.15.104:0/679948272 >> 172.21.15.104:0/679948272 conn(0x7f76c8078360 msgr2=0x7f76c81103c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:32.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.557+0000 7f76ce2b5700 1 -- 172.21.15.104:0/679948272 shutdown_connections 2024-06-09T22:19:32.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:32.557+0000 7f76ce2b5700 1 -- 172.21.15.104:0/679948272 wait complete. 2024-06-09T22:19:32.560 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-06-09T22:19:32.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:32 smithi153 ceph-mon[36599]: pgmap v392: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-06-09T22:19:32.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:32 smithi104 ceph-mon[32389]: pgmap v392: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-06-09T22:19:32.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:32 smithi194 ceph-mon[36856]: pgmap v392: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-06-09T22:19:33.381 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-06-09T22:19:31.350807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666753,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1158,"num_read_kb":712,"num_write":445,"num_write_kb":3098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":866,"ondisk_log_size":866,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379264,"kb_used_data":13472,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562560,"statfs":{"total":1151940427776,"available":1151552061440,"internally_reserved":0,"allocated":13795328,"data_stored":8813369,"data_compressed":495303,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":124,"num_read_kb":59,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002970"},"pg_stats":[{"pgid":"3.3f","version":"99'8","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.337332+0000","last_change":"2024-06-09T22:16:43.212998+0000","last_active":"2024-06-09T22:19:25.337332+0000","last_peered":"2024-06-09T22:19:25.337332+0000","last_clean":"2024-06-09T22:19:25.337332+0000","last_became_active":"2024-06-09T22:16:43.212614+0000","last_became_peered":"2024-06-09T22:16:43.212614+0000","last_unstale":"2024-06-09T22:19:25.337332+0000","last_undegraded":"2024-06-09T22:19:25.337332+0000","last_fullsized":"2024-06-09T22:19:25.337332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:22.728486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"99'4","reported_seq":42,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.337169+0000","last_change":"2024-06-09T22:16:43.208746+0000","last_active":"2024-06-09T22:19:25.337169+0000","last_peered":"2024-06-09T22:19:25.337169+0000","last_clean":"2024-06-09T22:19:25.337169+0000","last_became_active":"2024-06-09T22:16:43.208458+0000","last_became_peered":"2024-06-09T22:16:43.208458+0000","last_unstale":"2024-06-09T22:19:25.337169+0000","last_undegraded":"2024-06-09T22:19:25.337169+0000","last_fullsized":"2024-06-09T22:19:25.337169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:24:35.350449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"99'7","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.343331+0000","last_change":"2024-06-09T22:16:43.200770+0000","last_active":"2024-06-09T22:19:25.343331+0000","last_peered":"2024-06-09T22:19:25.343331+0000","last_clean":"2024-06-09T22:19:25.343331+0000","last_became_active":"2024-06-09T22:16:43.200449+0000","last_became_peered":"2024-06-09T22:16:43.200449+0000","last_unstale":"2024-06-09T22:19:25.343331+0000","last_undegraded":"2024-06-09T22:19:25.343331+0000","last_fullsized":"2024-06-09T22:19:25.343331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:51:09.017260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"99'8","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.329609+0000","last_change":"2024-06-09T22:16:43.203844+0000","last_active":"2024-06-09T22:19:25.329609+0000","last_peered":"2024-06-09T22:19:25.329609+0000","last_clean":"2024-06-09T22:19:25.329609+0000","last_became_active":"2024-06-09T22:16:43.203431+0000","last_became_peered":"2024-06-09T22:16:43.203431+0000","last_unstale":"2024-06-09T22:19:25.329609+0000","last_undegraded":"2024-06-09T22:19:25.329609+0000","last_fullsized":"2024-06-09T22:19:25.329609+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:51:36.913312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"99'4","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.330178+0000","last_change":"2024-06-09T22:16:43.207597+0000","last_active":"2024-06-09T22:19:25.330178+0000","last_peered":"2024-06-09T22:19:25.330178+0000","last_clean":"2024-06-09T22:19:25.330178+0000","last_became_active":"2024-06-09T22:16:43.207266+0000","last_became_peered":"2024-06-09T22:16:43.207266+0000","last_unstale":"2024-06-09T22:19:25.330178+0000","last_undegraded":"2024-06-09T22:19:25.330178+0000","last_fullsized":"2024-06-09T22:19:25.330178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:47:05.558403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"99'6","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.333457+0000","last_change":"2024-06-09T22:17:26.474121+0000","last_active":"2024-06-09T22:19:25.333457+0000","last_peered":"2024-06-09T22:19:25.333457+0000","last_clean":"2024-06-09T22:19:25.333457+0000","last_became_active":"2024-06-09T22:17:26.473619+0000","last_became_peered":"2024-06-09T22:17:26.473619+0000","last_unstale":"2024-06-09T22:19:25.333457+0000","last_undegraded":"2024-06-09T22:19:25.333457+0000","last_fullsized":"2024-06-09T22:19:25.333457+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:56:42.171235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"99'5","reported_seq":44,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.340581+0000","last_change":"2024-06-09T22:16:43.208815+0000","last_active":"2024-06-09T22:19:25.340581+0000","last_peered":"2024-06-09T22:19:25.340581+0000","last_clean":"2024-06-09T22:19:25.340581+0000","last_became_active":"2024-06-09T22:16:43.208433+0000","last_became_peered":"2024-06-09T22:16:43.208433+0000","last_unstale":"2024-06-09T22:19:25.340581+0000","last_undegraded":"2024-06-09T22:19:25.340581+0000","last_fullsized":"2024-06-09T22:19:25.340581+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:36:26.585037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"99'9","reported_seq":47,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.334575+0000","last_change":"2024-06-09T22:16:43.210195+0000","last_active":"2024-06-09T22:19:25.334575+0000","last_peered":"2024-06-09T22:19:25.334575+0000","last_clean":"2024-06-09T22:19:25.334575+0000","last_became_active":"2024-06-09T22:16:43.209892+0000","last_became_peered":"2024-06-09T22:16:43.209892+0000","last_unstale":"2024-06-09T22:19:25.334575+0000","last_undegraded":"2024-06-09T22:19:25.334575+0000","last_fullsized":"2024-06-09T22:19:25.334575+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:44.088212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"99'11","reported_seq":47,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.340921+0000","last_change":"2024-06-09T22:16:43.202517+0000","last_active":"2024-06-09T22:19:25.340921+0000","last_peered":"2024-06-09T22:19:25.340921+0000","last_clean":"2024-06-09T22:19:25.340921+0000","last_became_active":"2024-06-09T22:16:43.201796+0000","last_became_peered":"2024-06-09T22:16:43.201796+0000","last_unstale":"2024-06-09T22:19:25.340921+0000","last_undegraded":"2024-06-09T22:19:25.340921+0000","last_fullsized":"2024-06-09T22:19:25.340921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:30.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"99'16","reported_seq":52,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.331100+0000","last_change":"2024-06-09T22:16:43.212732+0000","last_active":"2024-06-09T22:19:25.331100+0000","last_peered":"2024-06-09T22:19:25.331100+0000","last_clean":"2024-06-09T22:19:25.331100+0000","last_became_active":"2024-06-09T22:16:43.212048+0000","last_became_peered":"2024-06-09T22:16:43.212048+0000","last_unstale":"2024-06-09T22:19:25.331100+0000","last_undegraded":"2024-06-09T22:19:25.331100+0000","last_fullsized":"2024-06-09T22:19:25.331100+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:47.383291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"99'6","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.335545+0000","last_change":"2024-06-09T22:16:43.203778+0000","last_active":"2024-06-09T22:19:25.335545+0000","last_peered":"2024-06-09T22:19:25.335545+0000","last_clean":"2024-06-09T22:19:25.335545+0000","last_became_active":"2024-06-09T22:16:43.202954+0000","last_became_peered":"2024-06-09T22:16:43.202954+0000","last_unstale":"2024-06-09T22:19:25.335545+0000","last_undegraded":"2024-06-09T22:19:25.335545+0000","last_fullsized":"2024-06-09T22:19:25.335545+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:56:53.708645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"99'7","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.325125+0000","last_change":"2024-06-09T22:16:43.211425+0000","last_active":"2024-06-09T22:19:25.325125+0000","last_peered":"2024-06-09T22:19:25.325125+0000","last_clean":"2024-06-09T22:19:25.325125+0000","last_became_active":"2024-06-09T22:16:43.211001+0000","last_became_peered":"2024-06-09T22:16:43.211001+0000","last_unstale":"2024-06-09T22:19:25.325125+0000","last_undegraded":"2024-06-09T22:19:25.325125+0000","last_fullsized":"2024-06-09T22:19:25.325125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:20:59.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"99'10","reported_seq":49,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.341452+0000","last_change":"2024-06-09T22:16:43.208334+0000","last_active":"2024-06-09T22:19:25.341452+0000","last_peered":"2024-06-09T22:19:25.341452+0000","last_clean":"2024-06-09T22:19:25.341452+0000","last_became_active":"2024-06-09T22:16:43.207993+0000","last_became_peered":"2024-06-09T22:16:43.207993+0000","last_unstale":"2024-06-09T22:19:25.341452+0000","last_undegraded":"2024-06-09T22:19:25.341452+0000","last_fullsized":"2024-06-09T22:19:25.341452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:07:05.740748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"99'7","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.335591+0000","last_change":"2024-06-09T22:16:43.207211+0000","last_active":"2024-06-09T22:19:25.335591+0000","last_peered":"2024-06-09T22:19:25.335591+0000","last_clean":"2024-06-09T22:19:25.335591+0000","last_became_active":"2024-06-09T22:16:43.206889+0000","last_became_peered":"2024-06-09T22:16:43.206889+0000","last_unstale":"2024-06-09T22:19:25.335591+0000","last_undegraded":"2024-06-09T22:19:25.335591+0000","last_fullsized":"2024-06-09T22:19:25.335591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:25:47.461030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"99'7","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.338216+0000","last_change":"2024-06-09T22:16:43.202202+0000","last_active":"2024-06-09T22:19:25.338216+0000","last_peered":"2024-06-09T22:19:25.338216+0000","last_clean":"2024-06-09T22:19:25.338216+0000","last_became_active":"2024-06-09T22:16:43.201834+0000","last_became_peered":"2024-06-09T22:16:43.201834+0000","last_unstale":"2024-06-09T22:19:25.338216+0000","last_undegraded":"2024-06-09T22:19:25.338216+0000","last_fullsized":"2024-06-09T22:19:25.338216+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:31:51.296627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"99'9","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.342020+0000","last_change":"2024-06-09T22:16:43.200637+0000","last_active":"2024-06-09T22:19:25.342020+0000","last_peered":"2024-06-09T22:19:25.342020+0000","last_clean":"2024-06-09T22:19:25.342020+0000","last_became_active":"2024-06-09T22:16:43.200268+0000","last_became_peered":"2024-06-09T22:16:43.200268+0000","last_unstale":"2024-06-09T22:19:25.342020+0000","last_undegraded":"2024-06-09T22:19:25.342020+0000","last_fullsized":"2024-06-09T22:19:25.342020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:25:59.881453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"99'8","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.339163+0000","last_change":"2024-06-09T22:16:43.209212+0000","last_active":"2024-06-09T22:19:25.339163+0000","last_peered":"2024-06-09T22:19:25.339163+0000","last_clean":"2024-06-09T22:19:25.339163+0000","last_became_active":"2024-06-09T22:16:43.207789+0000","last_became_peered":"2024-06-09T22:16:43.207789+0000","last_unstale":"2024-06-09T22:19:25.339163+0000","last_undegraded":"2024-06-09T22:19:25.339163+0000","last_fullsized":"2024-06-09T22:19:25.339163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:17.316595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"99'10","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.326694+0000","last_change":"2024-06-09T22:16:43.210935+0000","last_active":"2024-06-09T22:19:25.326694+0000","last_peered":"2024-06-09T22:19:25.326694+0000","last_clean":"2024-06-09T22:19:25.326694+0000","last_became_active":"2024-06-09T22:16:43.210631+0000","last_became_peered":"2024-06-09T22:16:43.210631+0000","last_unstale":"2024-06-09T22:19:25.326694+0000","last_undegraded":"2024-06-09T22:19:25.326694+0000","last_fullsized":"2024-06-09T22:19:25.326694+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:22:58.887313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"99'7","reported_seq":44,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.322868+0000","last_change":"2024-06-09T22:16:43.198872+0000","last_active":"2024-06-09T22:19:25.322868+0000","last_peered":"2024-06-09T22:19:25.322868+0000","last_clean":"2024-06-09T22:19:25.322868+0000","last_became_active":"2024-06-09T22:16:43.198498+0000","last_became_peered":"2024-06-09T22:16:43.198498+0000","last_unstale":"2024-06-09T22:19:25.322868+0000","last_undegraded":"2024-06-09T22:19:25.322868+0000","last_fullsized":"2024-06-09T22:19:25.322868+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:25:28.853048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537676+0000","last_change":"2024-06-09T22:16:53.343656+0000","last_active":"2024-06-09T22:18:26.537676+0000","last_peered":"2024-06-09T22:18:26.537676+0000","last_clean":"2024-06-09T22:18:26.537676+0000","last_became_active":"2024-06-09T22:16:53.343281+0000","last_became_peered":"2024-06-09T22:16:53.343281+0000","last_unstale":"2024-06-09T22:18:26.537676+0000","last_undegraded":"2024-06-09T22:18:26.537676+0000","last_fullsized":"2024-06-09T22:18:26.537676+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:40.961842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"97'13","reported_seq":152,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:24.577574+0000","last_change":"2024-06-09T22:16:40.040145+0000","last_active":"2024-06-09T22:19:24.577574+0000","last_peered":"2024-06-09T22:19:24.577574+0000","last_clean":"2024-06-09T22:19:24.577574+0000","last_became_active":"2024-06-09T22:16:40.039787+0000","last_became_peered":"2024-06-09T22:16:40.039787+0000","last_unstale":"2024-06-09T22:19:24.577574+0000","last_undegraded":"2024-06-09T22:19:24.577574+0000","last_fullsized":"2024-06-09T22:19:24.577574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:52:32.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":96,"num_read_kb":96,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"99'5","reported_seq":21,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.323678+0000","last_change":"2024-06-09T22:17:26.774053+0000","last_active":"2024-06-09T22:19:25.323678+0000","last_peered":"2024-06-09T22:19:25.323678+0000","last_clean":"2024-06-09T22:19:25.323678+0000","last_became_active":"2024-06-09T22:17:26.773786+0000","last_became_peered":"2024-06-09T22:17:26.773786+0000","last_unstale":"2024-06-09T22:19:25.323678+0000","last_undegraded":"2024-06-09T22:19:25.323678+0000","last_fullsized":"2024-06-09T22:19:25.323678+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:36.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695534+0000","last_change":"2024-06-09T22:16:53.332048+0000","last_active":"2024-06-09T22:18:26.695534+0000","last_peered":"2024-06-09T22:18:26.695534+0000","last_clean":"2024-06-09T22:18:26.695534+0000","last_became_active":"2024-06-09T22:16:53.330683+0000","last_became_peered":"2024-06-09T22:16:53.330683+0000","last_unstale":"2024-06-09T22:18:26.695534+0000","last_undegraded":"2024-06-09T22:18:26.695534+0000","last_fullsized":"2024-06-09T22:18:26.695534+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:32:23.219061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"97'2","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.697538+0000","last_change":"2024-06-09T22:16:40.070193+0000","last_active":"2024-06-09T22:18:26.697538+0000","last_peered":"2024-06-09T22:18:26.697538+0000","last_clean":"2024-06-09T22:18:26.697538+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:18:26.697538+0000","last_undegraded":"2024-06-09T22:18:26.697538+0000","last_fullsized":"2024-06-09T22:18:26.697538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:19:19.787552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"99'6","reported_seq":44,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.342761+0000","last_change":"2024-06-09T22:16:43.199814+0000","last_active":"2024-06-09T22:19:25.342761+0000","last_peered":"2024-06-09T22:19:25.342761+0000","last_clean":"2024-06-09T22:19:25.342761+0000","last_became_active":"2024-06-09T22:16:43.199487+0000","last_became_peered":"2024-06-09T22:16:43.199487+0000","last_unstale":"2024-06-09T22:19:25.342761+0000","last_undegraded":"2024-06-09T22:19:25.342761+0000","last_fullsized":"2024-06-09T22:19:25.342761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:55.672050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696106+0000","last_change":"2024-06-09T22:16:53.336253+0000","last_active":"2024-06-09T22:18:26.696106+0000","last_peered":"2024-06-09T22:18:26.696106+0000","last_clean":"2024-06-09T22:18:26.696106+0000","last_became_active":"2024-06-09T22:16:53.335869+0000","last_became_peered":"2024-06-09T22:16:53.335869+0000","last_unstale":"2024-06-09T22:18:26.696106+0000","last_undegraded":"2024-06-09T22:18:26.696106+0000","last_fullsized":"2024-06-09T22:18:26.696106+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:28:12.894878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"97'12","reported_seq":58,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535881+0000","last_change":"2024-06-09T22:16:40.045365+0000","last_active":"2024-06-09T22:18:26.535881+0000","last_peered":"2024-06-09T22:18:26.535881+0000","last_clean":"2024-06-09T22:18:26.535881+0000","last_became_active":"2024-06-09T22:16:40.044989+0000","last_became_peered":"2024-06-09T22:16:40.044989+0000","last_unstale":"2024-06-09T22:18:26.535881+0000","last_undegraded":"2024-06-09T22:18:26.535881+0000","last_fullsized":"2024-06-09T22:18:26.535881+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:01:05.850170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"99'9","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.339362+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:19:25.339362+0000","last_peered":"2024-06-09T22:19:25.339362+0000","last_clean":"2024-06-09T22:19:25.339362+0000","last_became_active":"2024-06-09T22:16:43.203999+0000","last_became_peered":"2024-06-09T22:16:43.203999+0000","last_unstale":"2024-06-09T22:19:25.339362+0000","last_undegraded":"2024-06-09T22:19:25.339362+0000","last_fullsized":"2024-06-09T22:19:25.339362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:37:28.362655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695532+0000","last_change":"2024-06-09T22:16:53.333565+0000","last_active":"2024-06-09T22:18:26.695532+0000","last_peered":"2024-06-09T22:18:26.695532+0000","last_clean":"2024-06-09T22:18:26.695532+0000","last_became_active":"2024-06-09T22:16:53.332921+0000","last_became_peered":"2024-06-09T22:16:53.332921+0000","last_unstale":"2024-06-09T22:18:26.695532+0000","last_undegraded":"2024-06-09T22:18:26.695532+0000","last_fullsized":"2024-06-09T22:18:26.695532+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:32.085617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"97'5","reported_seq":51,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535750+0000","last_change":"2024-06-09T22:16:40.039766+0000","last_active":"2024-06-09T22:18:26.535750+0000","last_peered":"2024-06-09T22:18:26.535750+0000","last_clean":"2024-06-09T22:18:26.535750+0000","last_became_active":"2024-06-09T22:16:40.039329+0000","last_became_peered":"2024-06-09T22:16:40.039329+0000","last_unstale":"2024-06-09T22:18:26.535750+0000","last_undegraded":"2024-06-09T22:18:26.535750+0000","last_fullsized":"2024-06-09T22:18:26.535750+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:53.673890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"99'15","reported_seq":52,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.344212+0000","last_change":"2024-06-09T22:16:43.201200+0000","last_active":"2024-06-09T22:19:25.344212+0000","last_peered":"2024-06-09T22:19:25.344212+0000","last_clean":"2024-06-09T22:19:25.344212+0000","last_became_active":"2024-06-09T22:16:43.200830+0000","last_became_peered":"2024-06-09T22:16:43.200830+0000","last_unstale":"2024-06-09T22:19:25.344212+0000","last_undegraded":"2024-06-09T22:19:25.344212+0000","last_fullsized":"2024-06-09T22:19:25.344212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:38:52.093173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537838+0000","last_change":"2024-06-09T22:16:53.339811+0000","last_active":"2024-06-09T22:18:26.537838+0000","last_peered":"2024-06-09T22:18:26.537838+0000","last_clean":"2024-06-09T22:18:26.537838+0000","last_became_active":"2024-06-09T22:16:53.339305+0000","last_became_peered":"2024-06-09T22:16:53.339305+0000","last_unstale":"2024-06-09T22:18:26.537838+0000","last_undegraded":"2024-06-09T22:18:26.537838+0000","last_fullsized":"2024-06-09T22:18:26.537838+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:08:25.800245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"99'7","reported_seq":154,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:28.586104+0000","last_change":"2024-06-09T22:16:40.044159+0000","last_active":"2024-06-09T22:19:28.586104+0000","last_peered":"2024-06-09T22:19:28.586104+0000","last_clean":"2024-06-09T22:19:28.586104+0000","last_became_active":"2024-06-09T22:16:40.043839+0000","last_became_peered":"2024-06-09T22:16:40.043839+0000","last_unstale":"2024-06-09T22:19:28.586104+0000","last_undegraded":"2024-06-09T22:19:28.586104+0000","last_fullsized":"2024-06-09T22:19:28.586104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:22:09.339219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1153,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"99'7","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.334318+0000","last_change":"2024-06-09T22:16:43.202015+0000","last_active":"2024-06-09T22:19:25.334318+0000","last_peered":"2024-06-09T22:19:25.334318+0000","last_clean":"2024-06-09T22:19:25.334318+0000","last_became_active":"2024-06-09T22:16:43.201689+0000","last_became_peered":"2024-06-09T22:16:43.201689+0000","last_unstale":"2024-06-09T22:19:25.334318+0000","last_undegraded":"2024-06-09T22:19:25.334318+0000","last_fullsized":"2024-06-09T22:19:25.334318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:35:12.180399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":36,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.699126+0000","last_change":"2024-06-09T22:18:26.698600+0000","last_active":"2024-06-09T22:18:26.699126+0000","last_peered":"2024-06-09T22:18:26.699126+0000","last_clean":"2024-06-09T22:18:26.699126+0000","last_became_active":"2024-06-09T22:18:26.698201+0000","last_became_peered":"2024-06-09T22:18:26.698201+0000","last_unstale":"2024-06-09T22:18:26.699126+0000","last_undegraded":"2024-06-09T22:18:26.699126+0000","last_fullsized":"2024-06-09T22:18:26.699126+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:55:13.513557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"97'7","reported_seq":55,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.697193+0000","last_change":"2024-06-09T22:17:26.698204+0000","last_active":"2024-06-09T22:18:26.697193+0000","last_peered":"2024-06-09T22:18:26.697193+0000","last_clean":"2024-06-09T22:18:26.697193+0000","last_became_active":"2024-06-09T22:17:26.697716+0000","last_became_peered":"2024-06-09T22:17:26.697716+0000","last_unstale":"2024-06-09T22:18:26.697193+0000","last_undegraded":"2024-06-09T22:18:26.697193+0000","last_fullsized":"2024-06-09T22:18:26.697193+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:36:37.724459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"99'8","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.342285+0000","last_change":"2024-06-09T22:16:43.198917+0000","last_active":"2024-06-09T22:19:25.342285+0000","last_peered":"2024-06-09T22:19:25.342285+0000","last_clean":"2024-06-09T22:19:25.342285+0000","last_became_active":"2024-06-09T22:16:43.198567+0000","last_became_peered":"2024-06-09T22:16:43.198567+0000","last_unstale":"2024-06-09T22:19:25.342285+0000","last_undegraded":"2024-06-09T22:19:25.342285+0000","last_fullsized":"2024-06-09T22:19:25.342285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:34.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696208+0000","last_change":"2024-06-09T22:16:53.342203+0000","last_active":"2024-06-09T22:18:26.696208+0000","last_peered":"2024-06-09T22:18:26.696208+0000","last_clean":"2024-06-09T22:18:26.696208+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:18:26.696208+0000","last_undegraded":"2024-06-09T22:18:26.696208+0000","last_fullsized":"2024-06-09T22:18:26.696208+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:41:41.491961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"97'3","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695926+0000","last_change":"2024-06-09T22:16:40.038684+0000","last_active":"2024-06-09T22:18:26.695926+0000","last_peered":"2024-06-09T22:18:26.695926+0000","last_clean":"2024-06-09T22:18:26.695926+0000","last_became_active":"2024-06-09T22:16:40.038386+0000","last_became_peered":"2024-06-09T22:16:40.038386+0000","last_unstale":"2024-06-09T22:18:26.695926+0000","last_undegraded":"2024-06-09T22:18:26.695926+0000","last_fullsized":"2024-06-09T22:18:26.695926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:59.188758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"99'8","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.330417+0000","last_change":"2024-06-09T22:16:43.198837+0000","last_active":"2024-06-09T22:19:25.330417+0000","last_peered":"2024-06-09T22:19:25.330417+0000","last_clean":"2024-06-09T22:19:25.330417+0000","last_became_active":"2024-06-09T22:16:43.198515+0000","last_became_peered":"2024-06-09T22:16:43.198515+0000","last_unstale":"2024-06-09T22:19:25.330417+0000","last_undegraded":"2024-06-09T22:19:25.330417+0000","last_fullsized":"2024-06-09T22:19:25.330417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:36:13.863202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.534258+0000","last_change":"2024-06-09T22:16:53.342688+0000","last_active":"2024-06-09T22:18:26.534258+0000","last_peered":"2024-06-09T22:18:26.534258+0000","last_clean":"2024-06-09T22:18:26.534258+0000","last_became_active":"2024-06-09T22:16:53.342314+0000","last_became_peered":"2024-06-09T22:16:53.342314+0000","last_unstale":"2024-06-09T22:18:26.534258+0000","last_undegraded":"2024-06-09T22:18:26.534258+0000","last_fullsized":"2024-06-09T22:18:26.534258+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:51:35.801587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"97'3","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695265+0000","last_change":"2024-06-09T22:16:40.041831+0000","last_active":"2024-06-09T22:18:26.695265+0000","last_peered":"2024-06-09T22:18:26.695265+0000","last_clean":"2024-06-09T22:18:26.695265+0000","last_became_active":"2024-06-09T22:16:40.041484+0000","last_became_peered":"2024-06-09T22:16:40.041484+0000","last_unstale":"2024-06-09T22:18:26.695265+0000","last_undegraded":"2024-06-09T22:18:26.695265+0000","last_fullsized":"2024-06-09T22:18:26.695265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:10:19.829357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"99'11","reported_seq":47,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.330343+0000","last_change":"2024-06-09T22:16:43.210769+0000","last_active":"2024-06-09T22:19:25.330343+0000","last_peered":"2024-06-09T22:19:25.330343+0000","last_clean":"2024-06-09T22:19:25.330343+0000","last_became_active":"2024-06-09T22:16:43.210429+0000","last_became_peered":"2024-06-09T22:16:43.210429+0000","last_unstale":"2024-06-09T22:19:25.330343+0000","last_undegraded":"2024-06-09T22:19:25.330343+0000","last_fullsized":"2024-06-09T22:19:25.330343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:12:28.788830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695595+0000","last_change":"2024-06-09T22:16:53.336536+0000","last_active":"2024-06-09T22:18:26.695595+0000","last_peered":"2024-06-09T22:18:26.695595+0000","last_clean":"2024-06-09T22:18:26.695595+0000","last_became_active":"2024-06-09T22:16:53.335178+0000","last_became_peered":"2024-06-09T22:16:53.335178+0000","last_unstale":"2024-06-09T22:18:26.695595+0000","last_undegraded":"2024-06-09T22:18:26.695595+0000","last_fullsized":"2024-06-09T22:18:26.695595+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:57:19.476483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"99'6","reported_seq":242,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:27.583650+0000","last_change":"2024-06-09T22:16:40.038484+0000","last_active":"2024-06-09T22:19:27.583650+0000","last_peered":"2024-06-09T22:19:27.583650+0000","last_clean":"2024-06-09T22:19:27.583650+0000","last_became_active":"2024-06-09T22:16:40.038084+0000","last_became_peered":"2024-06-09T22:16:40.038084+0000","last_unstale":"2024-06-09T22:19:27.583650+0000","last_undegraded":"2024-06-09T22:19:27.583650+0000","last_fullsized":"2024-06-09T22:19:27.583650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:19.733107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":194,"num_read_kb":194,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"99'10","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.339912+0000","last_change":"2024-06-09T22:16:43.205137+0000","last_active":"2024-06-09T22:19:25.339912+0000","last_peered":"2024-06-09T22:19:25.339912+0000","last_clean":"2024-06-09T22:19:25.339912+0000","last_became_active":"2024-06-09T22:16:43.204499+0000","last_became_peered":"2024-06-09T22:16:43.204499+0000","last_unstale":"2024-06-09T22:19:25.339912+0000","last_undegraded":"2024-06-09T22:19:25.339912+0000","last_fullsized":"2024-06-09T22:19:25.339912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:15.718493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695908+0000","last_change":"2024-06-09T22:16:53.344260+0000","last_active":"2024-06-09T22:18:26.695908+0000","last_peered":"2024-06-09T22:18:26.695908+0000","last_clean":"2024-06-09T22:18:26.695908+0000","last_became_active":"2024-06-09T22:16:53.343840+0000","last_became_peered":"2024-06-09T22:16:53.343840+0000","last_unstale":"2024-06-09T22:18:26.695908+0000","last_undegraded":"2024-06-09T22:18:26.695908+0000","last_fullsized":"2024-06-09T22:18:26.695908+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:40:23.305509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"97'4","reported_seq":49,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.538210+0000","last_change":"2024-06-09T22:16:40.042660+0000","last_active":"2024-06-09T22:18:26.538210+0000","last_peered":"2024-06-09T22:18:26.538210+0000","last_clean":"2024-06-09T22:18:26.538210+0000","last_became_active":"2024-06-09T22:16:40.041353+0000","last_became_peered":"2024-06-09T22:16:40.041353+0000","last_unstale":"2024-06-09T22:18:26.538210+0000","last_undegraded":"2024-06-09T22:18:26.538210+0000","last_fullsized":"2024-06-09T22:18:26.538210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:34:59.064827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"99'9","reported_seq":47,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.339075+0000","last_change":"2024-06-09T22:16:43.205083+0000","last_active":"2024-06-09T22:19:25.339075+0000","last_peered":"2024-06-09T22:19:25.339075+0000","last_clean":"2024-06-09T22:19:25.339075+0000","last_became_active":"2024-06-09T22:16:43.204600+0000","last_became_peered":"2024-06-09T22:16:43.204600+0000","last_unstale":"2024-06-09T22:19:25.339075+0000","last_undegraded":"2024-06-09T22:19:25.339075+0000","last_fullsized":"2024-06-09T22:19:25.339075+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:05:12.555465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696388+0000","last_change":"2024-06-09T22:16:53.336169+0000","last_active":"2024-06-09T22:18:26.696388+0000","last_peered":"2024-06-09T22:18:26.696388+0000","last_clean":"2024-06-09T22:18:26.696388+0000","last_became_active":"2024-06-09T22:16:53.335657+0000","last_became_peered":"2024-06-09T22:16:53.335657+0000","last_unstale":"2024-06-09T22:18:26.696388+0000","last_undegraded":"2024-06-09T22:18:26.696388+0000","last_fullsized":"2024-06-09T22:18:26.696388+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:53:21.282767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"97'5","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695835+0000","last_change":"2024-06-09T22:16:40.070568+0000","last_active":"2024-06-09T22:18:26.695835+0000","last_peered":"2024-06-09T22:18:26.695835+0000","last_clean":"2024-06-09T22:18:26.695835+0000","last_became_active":"2024-06-09T22:16:40.070205+0000","last_became_peered":"2024-06-09T22:16:40.070205+0000","last_unstale":"2024-06-09T22:18:26.695835+0000","last_undegraded":"2024-06-09T22:18:26.695835+0000","last_fullsized":"2024-06-09T22:18:26.695835+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.449965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"99'11","reported_seq":47,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.334059+0000","last_change":"2024-06-09T22:16:43.205219+0000","last_active":"2024-06-09T22:19:25.334059+0000","last_peered":"2024-06-09T22:19:25.334059+0000","last_clean":"2024-06-09T22:19:25.334059+0000","last_became_active":"2024-06-09T22:16:43.203467+0000","last_became_peered":"2024-06-09T22:16:43.203467+0000","last_unstale":"2024-06-09T22:19:25.334059+0000","last_undegraded":"2024-06-09T22:19:25.334059+0000","last_fullsized":"2024-06-09T22:19:25.334059+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:22.784034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696569+0000","last_change":"2024-06-09T22:16:53.338990+0000","last_active":"2024-06-09T22:18:26.696569+0000","last_peered":"2024-06-09T22:18:26.696569+0000","last_clean":"2024-06-09T22:18:26.696569+0000","last_became_active":"2024-06-09T22:16:53.338568+0000","last_became_peered":"2024-06-09T22:16:53.338568+0000","last_unstale":"2024-06-09T22:18:26.696569+0000","last_undegraded":"2024-06-09T22:18:26.696569+0000","last_fullsized":"2024-06-09T22:18:26.696569+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:46:39.814043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"99'40","reported_seq":286,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:27.583804+0000","last_change":"2024-06-09T22:16:40.044404+0000","last_active":"2024-06-09T22:19:27.583804+0000","last_peered":"2024-06-09T22:19:27.583804+0000","last_clean":"2024-06-09T22:19:27.583804+0000","last_became_active":"2024-06-09T22:16:40.044044+0000","last_became_peered":"2024-06-09T22:16:40.044044+0000","last_unstale":"2024-06-09T22:19:27.583804+0000","last_undegraded":"2024-06-09T22:19:27.583804+0000","last_fullsized":"2024-06-09T22:19:27.583804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:50:27.038759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68136,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":204,"num_read_kb":169,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"99'6","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.333782+0000","last_change":"2024-06-09T22:16:43.206963+0000","last_active":"2024-06-09T22:19:25.333782+0000","last_peered":"2024-06-09T22:19:25.333782+0000","last_clean":"2024-06-09T22:19:25.333782+0000","last_became_active":"2024-06-09T22:16:43.206628+0000","last_became_peered":"2024-06-09T22:16:43.206628+0000","last_unstale":"2024-06-09T22:19:25.333782+0000","last_undegraded":"2024-06-09T22:19:25.333782+0000","last_fullsized":"2024-06-09T22:19:25.333782+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:33:39.912236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696283+0000","last_change":"2024-06-09T22:16:53.339332+0000","last_active":"2024-06-09T22:18:26.696283+0000","last_peered":"2024-06-09T22:18:26.696283+0000","last_clean":"2024-06-09T22:18:26.696283+0000","last_became_active":"2024-06-09T22:16:53.338819+0000","last_became_peered":"2024-06-09T22:16:53.338819+0000","last_unstale":"2024-06-09T22:18:26.696283+0000","last_undegraded":"2024-06-09T22:18:26.696283+0000","last_fullsized":"2024-06-09T22:18:26.696283+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:57:41.142142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"97'9","reported_seq":30,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696016+0000","last_change":"2024-06-09T22:17:26.478256+0000","last_active":"2024-06-09T22:18:26.696016+0000","last_peered":"2024-06-09T22:18:26.696016+0000","last_clean":"2024-06-09T22:18:26.696016+0000","last_became_active":"2024-06-09T22:17:26.472264+0000","last_became_peered":"2024-06-09T22:17:26.472264+0000","last_unstale":"2024-06-09T22:18:26.696016+0000","last_undegraded":"2024-06-09T22:18:26.696016+0000","last_fullsized":"2024-06-09T22:18:26.696016+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:14:19.919841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"99'9","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.336506+0000","last_change":"2024-06-09T22:16:43.204484+0000","last_active":"2024-06-09T22:19:25.336506+0000","last_peered":"2024-06-09T22:19:25.336506+0000","last_clean":"2024-06-09T22:19:25.336506+0000","last_became_active":"2024-06-09T22:16:43.204014+0000","last_became_peered":"2024-06-09T22:16:43.204014+0000","last_unstale":"2024-06-09T22:19:25.336506+0000","last_undegraded":"2024-06-09T22:19:25.336506+0000","last_fullsized":"2024-06-09T22:19:25.336506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:55:12.528231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":36,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695616+0000","last_change":"2024-06-09T22:17:26.475179+0000","last_active":"2024-06-09T22:18:26.695616+0000","last_peered":"2024-06-09T22:18:26.695616+0000","last_clean":"2024-06-09T22:18:26.695616+0000","last_became_active":"2024-06-09T22:17:26.474517+0000","last_became_peered":"2024-06-09T22:17:26.474517+0000","last_unstale":"2024-06-09T22:18:26.695616+0000","last_undegraded":"2024-06-09T22:18:26.695616+0000","last_fullsized":"2024-06-09T22:18:26.695616+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:26:02.387830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"97'5","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695661+0000","last_change":"2024-06-09T22:16:40.043066+0000","last_active":"2024-06-09T22:18:26.695661+0000","last_peered":"2024-06-09T22:18:26.695661+0000","last_clean":"2024-06-09T22:18:26.695661+0000","last_became_active":"2024-06-09T22:16:40.042619+0000","last_became_peered":"2024-06-09T22:16:40.042619+0000","last_unstale":"2024-06-09T22:18:26.695661+0000","last_undegraded":"2024-06-09T22:18:26.695661+0000","last_fullsized":"2024-06-09T22:18:26.695661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:29.689614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"99'15","reported_seq":60,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.334548+0000","last_change":"2024-06-09T22:17:26.475566+0000","last_active":"2024-06-09T22:19:25.334548+0000","last_peered":"2024-06-09T22:19:25.334548+0000","last_clean":"2024-06-09T22:19:25.334548+0000","last_became_active":"2024-06-09T22:17:26.475037+0000","last_became_peered":"2024-06-09T22:17:26.475037+0000","last_unstale":"2024-06-09T22:19:25.334548+0000","last_undegraded":"2024-06-09T22:19:25.334548+0000","last_fullsized":"2024-06-09T22:19:25.334548+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:33.075144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535654+0000","last_change":"2024-06-09T22:17:26.475111+0000","last_active":"2024-06-09T22:18:26.535654+0000","last_peered":"2024-06-09T22:18:26.535654+0000","last_clean":"2024-06-09T22:18:26.535654+0000","last_became_active":"2024-06-09T22:17:26.474563+0000","last_became_peered":"2024-06-09T22:17:26.474563+0000","last_unstale":"2024-06-09T22:18:26.535654+0000","last_undegraded":"2024-06-09T22:18:26.535654+0000","last_fullsized":"2024-06-09T22:18:26.535654+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:09:08.458761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"99'7","reported_seq":153,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:28.585533+0000","last_change":"2024-06-09T22:16:40.040538+0000","last_active":"2024-06-09T22:19:28.585533+0000","last_peered":"2024-06-09T22:19:28.585533+0000","last_clean":"2024-06-09T22:19:28.585533+0000","last_became_active":"2024-06-09T22:16:40.039896+0000","last_became_peered":"2024-06-09T22:16:40.039896+0000","last_unstale":"2024-06-09T22:19:28.585533+0000","last_undegraded":"2024-06-09T22:19:28.585533+0000","last_fullsized":"2024-06-09T22:19:28.585533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:00:33.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1119,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"99'6","reported_seq":51,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.336900+0000","last_change":"2024-06-09T22:17:26.475568+0000","last_active":"2024-06-09T22:19:25.336900+0000","last_peered":"2024-06-09T22:19:25.336900+0000","last_clean":"2024-06-09T22:19:25.336900+0000","last_became_active":"2024-06-09T22:17:26.475163+0000","last_became_peered":"2024-06-09T22:17:26.475163+0000","last_unstale":"2024-06-09T22:19:25.336900+0000","last_undegraded":"2024-06-09T22:19:25.336900+0000","last_fullsized":"2024-06-09T22:19:25.336900+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:20.865164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696860+0000","last_change":"2024-06-09T22:16:53.338201+0000","last_active":"2024-06-09T22:18:26.696860+0000","last_peered":"2024-06-09T22:18:26.696860+0000","last_clean":"2024-06-09T22:18:26.696860+0000","last_became_active":"2024-06-09T22:16:53.337816+0000","last_became_peered":"2024-06-09T22:16:53.337816+0000","last_unstale":"2024-06-09T22:18:26.696860+0000","last_undegraded":"2024-06-09T22:18:26.696860+0000","last_fullsized":"2024-06-09T22:18:26.696860+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:35:51.861135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"97'17","reported_seq":63,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535122+0000","last_change":"2024-06-09T22:16:40.070770+0000","last_active":"2024-06-09T22:18:26.535122+0000","last_peered":"2024-06-09T22:18:26.535122+0000","last_clean":"2024-06-09T22:18:26.535122+0000","last_became_active":"2024-06-09T22:16:40.070398+0000","last_became_peered":"2024-06-09T22:16:40.070398+0000","last_unstale":"2024-06-09T22:18:26.535122+0000","last_undegraded":"2024-06-09T22:18:26.535122+0000","last_fullsized":"2024-06-09T22:18:26.535122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:04:58.273601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"99'6","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.341776+0000","last_change":"2024-06-09T22:16:43.199246+0000","last_active":"2024-06-09T22:19:25.341776+0000","last_peered":"2024-06-09T22:19:25.341776+0000","last_clean":"2024-06-09T22:19:25.341776+0000","last_became_active":"2024-06-09T22:16:43.198006+0000","last_became_peered":"2024-06-09T22:16:43.198006+0000","last_unstale":"2024-06-09T22:19:25.341776+0000","last_undegraded":"2024-06-09T22:19:25.341776+0000","last_fullsized":"2024-06-09T22:19:25.341776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:30:55.836628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696277+0000","last_change":"2024-06-09T22:16:53.342217+0000","last_active":"2024-06-09T22:18:26.696277+0000","last_peered":"2024-06-09T22:18:26.696277+0000","last_clean":"2024-06-09T22:18:26.696277+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:18:26.696277+0000","last_undegraded":"2024-06-09T22:18:26.696277+0000","last_fullsized":"2024-06-09T22:18:26.696277+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:25:28.635091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"97'2","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535854+0000","last_change":"2024-06-09T22:16:40.039841+0000","last_active":"2024-06-09T22:18:26.535854+0000","last_peered":"2024-06-09T22:18:26.535854+0000","last_clean":"2024-06-09T22:18:26.535854+0000","last_became_active":"2024-06-09T22:16:40.039416+0000","last_became_peered":"2024-06-09T22:16:40.039416+0000","last_unstale":"2024-06-09T22:18:26.535854+0000","last_undegraded":"2024-06-09T22:18:26.535854+0000","last_fullsized":"2024-06-09T22:18:26.535854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:27:36.213839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"99'5","reported_seq":42,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.329021+0000","last_change":"2024-06-09T22:16:43.214363+0000","last_active":"2024-06-09T22:19:25.329021+0000","last_peered":"2024-06-09T22:19:25.329021+0000","last_clean":"2024-06-09T22:19:25.329021+0000","last_became_active":"2024-06-09T22:16:43.214015+0000","last_became_peered":"2024-06-09T22:16:43.214015+0000","last_unstale":"2024-06-09T22:19:25.329021+0000","last_undegraded":"2024-06-09T22:19:25.329021+0000","last_fullsized":"2024-06-09T22:19:25.329021+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:13:06.924468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535334+0000","last_change":"2024-06-09T22:16:53.342151+0000","last_active":"2024-06-09T22:18:26.535334+0000","last_peered":"2024-06-09T22:18:26.535334+0000","last_clean":"2024-06-09T22:18:26.535334+0000","last_became_active":"2024-06-09T22:16:53.341743+0000","last_became_peered":"2024-06-09T22:16:53.341743+0000","last_unstale":"2024-06-09T22:18:26.535334+0000","last_undegraded":"2024-06-09T22:18:26.535334+0000","last_fullsized":"2024-06-09T22:18:26.535334+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:39:44.793301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"97'1","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.697415+0000","last_change":"2024-06-09T22:17:26.472397+0000","last_active":"2024-06-09T22:18:26.697415+0000","last_peered":"2024-06-09T22:18:26.697415+0000","last_clean":"2024-06-09T22:18:26.697415+0000","last_became_active":"2024-06-09T22:17:26.471950+0000","last_became_peered":"2024-06-09T22:17:26.471950+0000","last_unstale":"2024-06-09T22:18:26.697415+0000","last_undegraded":"2024-06-09T22:18:26.697415+0000","last_fullsized":"2024-06-09T22:18:26.697415+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:35:57.481510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"99'4","reported_seq":40,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.331770+0000","last_change":"2024-06-09T22:16:43.214976+0000","last_active":"2024-06-09T22:19:25.331770+0000","last_peered":"2024-06-09T22:19:25.331770+0000","last_clean":"2024-06-09T22:19:25.331770+0000","last_became_active":"2024-06-09T22:16:43.214592+0000","last_became_peered":"2024-06-09T22:16:43.214592+0000","last_unstale":"2024-06-09T22:19:25.331770+0000","last_undegraded":"2024-06-09T22:19:25.331770+0000","last_fullsized":"2024-06-09T22:19:25.331770+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:16:21.981144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696417+0000","last_change":"2024-06-09T22:16:53.342721+0000","last_active":"2024-06-09T22:18:26.696417+0000","last_peered":"2024-06-09T22:18:26.696417+0000","last_clean":"2024-06-09T22:18:26.696417+0000","last_became_active":"2024-06-09T22:16:53.342342+0000","last_became_peered":"2024-06-09T22:16:53.342342+0000","last_unstale":"2024-06-09T22:18:26.696417+0000","last_undegraded":"2024-06-09T22:18:26.696417+0000","last_fullsized":"2024-06-09T22:18:26.696417+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:22:07.469873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"97'5","reported_seq":149,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:27.582221+0000","last_change":"2024-06-09T22:16:40.041307+0000","last_active":"2024-06-09T22:19:27.582221+0000","last_peered":"2024-06-09T22:19:27.582221+0000","last_clean":"2024-06-09T22:19:27.582221+0000","last_became_active":"2024-06-09T22:16:40.040819+0000","last_became_peered":"2024-06-09T22:16:40.040819+0000","last_unstale":"2024-06-09T22:19:27.582221+0000","last_undegraded":"2024-06-09T22:19:27.582221+0000","last_fullsized":"2024-06-09T22:19:27.582221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:14:29.658293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"99'7","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.328428+0000","last_change":"2024-06-09T22:16:43.215719+0000","last_active":"2024-06-09T22:19:25.328428+0000","last_peered":"2024-06-09T22:19:25.328428+0000","last_clean":"2024-06-09T22:19:25.328428+0000","last_became_active":"2024-06-09T22:16:43.215356+0000","last_became_peered":"2024-06-09T22:16:43.215356+0000","last_unstale":"2024-06-09T22:19:25.328428+0000","last_undegraded":"2024-06-09T22:19:25.328428+0000","last_fullsized":"2024-06-09T22:19:25.328428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:15:07.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696648+0000","last_change":"2024-06-09T22:16:53.342948+0000","last_active":"2024-06-09T22:18:26.696648+0000","last_peered":"2024-06-09T22:18:26.696648+0000","last_clean":"2024-06-09T22:18:26.696648+0000","last_became_active":"2024-06-09T22:16:53.342352+0000","last_became_peered":"2024-06-09T22:16:53.342352+0000","last_unstale":"2024-06-09T22:18:26.696648+0000","last_undegraded":"2024-06-09T22:18:26.696648+0000","last_fullsized":"2024-06-09T22:18:26.696648+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:43:27.453789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"99'8","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.342038+0000","last_change":"2024-06-09T22:16:43.204897+0000","last_active":"2024-06-09T22:19:25.342038+0000","last_peered":"2024-06-09T22:19:25.342038+0000","last_clean":"2024-06-09T22:19:25.342038+0000","last_became_active":"2024-06-09T22:16:43.204470+0000","last_became_peered":"2024-06-09T22:16:43.204470+0000","last_unstale":"2024-06-09T22:19:25.342038+0000","last_undegraded":"2024-06-09T22:19:25.342038+0000","last_fullsized":"2024-06-09T22:19:25.342038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:56.681392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"99'6","reported_seq":151,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:28.584823+0000","last_change":"2024-06-09T22:16:40.070079+0000","last_active":"2024-06-09T22:19:28.584823+0000","last_peered":"2024-06-09T22:19:28.584823+0000","last_clean":"2024-06-09T22:19:28.584823+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:19:28.584823+0000","last_undegraded":"2024-06-09T22:19:28.584823+0000","last_fullsized":"2024-06-09T22:19:28.584823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:00:45.132139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":100,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"97'1","reported_seq":30,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537957+0000","last_change":"2024-06-09T22:16:53.338891+0000","last_active":"2024-06-09T22:18:26.537957+0000","last_peered":"2024-06-09T22:18:26.537957+0000","last_clean":"2024-06-09T22:18:26.537957+0000","last_became_active":"2024-06-09T22:16:53.338526+0000","last_became_peered":"2024-06-09T22:16:53.338526+0000","last_unstale":"2024-06-09T22:18:26.537957+0000","last_undegraded":"2024-06-09T22:18:26.537957+0000","last_fullsized":"2024-06-09T22:18:26.537957+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:34:37.004407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"99'8","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.332163+0000","last_change":"2024-06-09T22:16:43.199234+0000","last_active":"2024-06-09T22:19:25.332163+0000","last_peered":"2024-06-09T22:19:25.332163+0000","last_clean":"2024-06-09T22:19:25.332163+0000","last_became_active":"2024-06-09T22:16:43.192012+0000","last_became_peered":"2024-06-09T22:16:43.192012+0000","last_unstale":"2024-06-09T22:19:25.332163+0000","last_undegraded":"2024-06-09T22:19:25.332163+0000","last_fullsized":"2024-06-09T22:19:25.332163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:17.429236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"97'6","reported_seq":52,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696236+0000","last_change":"2024-06-09T22:16:40.041970+0000","last_active":"2024-06-09T22:18:26.696236+0000","last_peered":"2024-06-09T22:18:26.696236+0000","last_clean":"2024-06-09T22:18:26.696236+0000","last_became_active":"2024-06-09T22:16:40.040371+0000","last_became_peered":"2024-06-09T22:16:40.040371+0000","last_unstale":"2024-06-09T22:18:26.696236+0000","last_undegraded":"2024-06-09T22:18:26.696236+0000","last_fullsized":"2024-06-09T22:18:26.696236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:52:31.210423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":378,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537270+0000","last_change":"2024-06-09T22:15:15.367397+0000","last_active":"2024-06-09T22:18:26.537270+0000","last_peered":"2024-06-09T22:18:26.537270+0000","last_clean":"2024-06-09T22:18:26.537270+0000","last_became_active":"2024-06-09T22:15:15.366974+0000","last_became_peered":"2024-06-09T22:15:15.366974+0000","last_unstale":"2024-06-09T22:18:26.537270+0000","last_undegraded":"2024-06-09T22:18:26.537270+0000","last_fullsized":"2024-06-09T22:18:26.537270+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_clean_scrub_stamp":"2024-06-09T22:11:49.886056+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:24.018031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695822+0000","last_change":"2024-06-09T22:16:53.342249+0000","last_active":"2024-06-09T22:18:26.695822+0000","last_peered":"2024-06-09T22:18:26.695822+0000","last_clean":"2024-06-09T22:18:26.695822+0000","last_became_active":"2024-06-09T22:16:53.341779+0000","last_became_peered":"2024-06-09T22:16:53.341779+0000","last_unstale":"2024-06-09T22:18:26.695822+0000","last_undegraded":"2024-06-09T22:18:26.695822+0000","last_fullsized":"2024-06-09T22:18:26.695822+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:21:21.101578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"99'5","reported_seq":44,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.332653+0000","last_change":"2024-06-09T22:16:43.200778+0000","last_active":"2024-06-09T22:19:25.332653+0000","last_peered":"2024-06-09T22:19:25.332653+0000","last_clean":"2024-06-09T22:19:25.332653+0000","last_became_active":"2024-06-09T22:16:43.200490+0000","last_became_peered":"2024-06-09T22:16:43.200490+0000","last_unstale":"2024-06-09T22:19:25.332653+0000","last_undegraded":"2024-06-09T22:19:25.332653+0000","last_fullsized":"2024-06-09T22:19:25.332653+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:20:17.530204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"97'4","reported_seq":49,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537575+0000","last_change":"2024-06-09T22:16:40.041774+0000","last_active":"2024-06-09T22:18:26.537575+0000","last_peered":"2024-06-09T22:18:26.537575+0000","last_clean":"2024-06-09T22:18:26.537575+0000","last_became_active":"2024-06-09T22:16:40.041085+0000","last_became_peered":"2024-06-09T22:16:40.041085+0000","last_unstale":"2024-06-09T22:18:26.537575+0000","last_undegraded":"2024-06-09T22:18:26.537575+0000","last_fullsized":"2024-06-09T22:18:26.537575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:09:41.140083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696364+0000","last_change":"2024-06-09T22:16:53.336895+0000","last_active":"2024-06-09T22:18:26.696364+0000","last_peered":"2024-06-09T22:18:26.696364+0000","last_clean":"2024-06-09T22:18:26.696364+0000","last_became_active":"2024-06-09T22:16:53.336497+0000","last_became_peered":"2024-06-09T22:16:53.336497+0000","last_unstale":"2024-06-09T22:18:26.696364+0000","last_undegraded":"2024-06-09T22:18:26.696364+0000","last_fullsized":"2024-06-09T22:18:26.696364+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:29:52.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"97'4","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535749+0000","last_change":"2024-06-09T22:16:40.070909+0000","last_active":"2024-06-09T22:18:26.535749+0000","last_peered":"2024-06-09T22:18:26.535749+0000","last_clean":"2024-06-09T22:18:26.535749+0000","last_became_active":"2024-06-09T22:16:40.070556+0000","last_became_peered":"2024-06-09T22:16:40.070556+0000","last_unstale":"2024-06-09T22:18:26.535749+0000","last_undegraded":"2024-06-09T22:18:26.535749+0000","last_fullsized":"2024-06-09T22:18:26.535749+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:59:17.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"99'9","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.324693+0000","last_change":"2024-06-09T22:16:43.205951+0000","last_active":"2024-06-09T22:19:25.324693+0000","last_peered":"2024-06-09T22:19:25.324693+0000","last_clean":"2024-06-09T22:19:25.324693+0000","last_became_active":"2024-06-09T22:16:43.205594+0000","last_became_peered":"2024-06-09T22:16:43.205594+0000","last_unstale":"2024-06-09T22:19:25.324693+0000","last_undegraded":"2024-06-09T22:19:25.324693+0000","last_fullsized":"2024-06-09T22:19:25.324693+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:17.956406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695289+0000","last_change":"2024-06-09T22:16:53.343292+0000","last_active":"2024-06-09T22:18:26.695289+0000","last_peered":"2024-06-09T22:18:26.695289+0000","last_clean":"2024-06-09T22:18:26.695289+0000","last_became_active":"2024-06-09T22:16:53.342900+0000","last_became_peered":"2024-06-09T22:16:53.342900+0000","last_unstale":"2024-06-09T22:18:26.695289+0000","last_undegraded":"2024-06-09T22:18:26.695289+0000","last_fullsized":"2024-06-09T22:18:26.695289+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:08:07.328250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"99'16","reported_seq":59,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:51.368666+0000","last_change":"2024-06-09T22:16:40.039417+0000","last_active":"2024-06-09T22:18:51.368666+0000","last_peered":"2024-06-09T22:18:51.368666+0000","last_clean":"2024-06-09T22:18:51.368666+0000","last_became_active":"2024-06-09T22:16:40.039099+0000","last_became_peered":"2024-06-09T22:16:40.039099+0000","last_unstale":"2024-06-09T22:18:51.368666+0000","last_undegraded":"2024-06-09T22:18:51.368666+0000","last_fullsized":"2024-06-09T22:18:51.368666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:22:59.284442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"99'5","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.333403+0000","last_change":"2024-06-09T22:16:43.201285+0000","last_active":"2024-06-09T22:19:25.333403+0000","last_peered":"2024-06-09T22:19:25.333403+0000","last_clean":"2024-06-09T22:19:25.333403+0000","last_became_active":"2024-06-09T22:16:43.200993+0000","last_became_peered":"2024-06-09T22:16:43.200993+0000","last_unstale":"2024-06-09T22:19:25.333403+0000","last_undegraded":"2024-06-09T22:19:25.333403+0000","last_fullsized":"2024-06-09T22:19:25.333403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:15:26.491588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537462+0000","last_change":"2024-06-09T22:16:53.334088+0000","last_active":"2024-06-09T22:18:26.537462+0000","last_peered":"2024-06-09T22:18:26.537462+0000","last_clean":"2024-06-09T22:18:26.537462+0000","last_became_active":"2024-06-09T22:16:53.333597+0000","last_became_peered":"2024-06-09T22:16:53.333597+0000","last_unstale":"2024-06-09T22:18:26.537462+0000","last_undegraded":"2024-06-09T22:18:26.537462+0000","last_fullsized":"2024-06-09T22:18:26.537462+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:50:22.041816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"97'7","reported_seq":51,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696271+0000","last_change":"2024-06-09T22:16:40.364982+0000","last_active":"2024-06-09T22:18:26.696271+0000","last_peered":"2024-06-09T22:18:26.696271+0000","last_clean":"2024-06-09T22:18:26.696271+0000","last_became_active":"2024-06-09T22:16:40.364585+0000","last_became_peered":"2024-06-09T22:16:40.364585+0000","last_unstale":"2024-06-09T22:18:26.696271+0000","last_undegraded":"2024-06-09T22:18:26.696271+0000","last_fullsized":"2024-06-09T22:18:26.696271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:28.777950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"99'7","reported_seq":51,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.340802+0000","last_change":"2024-06-09T22:17:26.474491+0000","last_active":"2024-06-09T22:19:25.340802+0000","last_peered":"2024-06-09T22:19:25.340802+0000","last_clean":"2024-06-09T22:19:25.340802+0000","last_became_active":"2024-06-09T22:17:26.473975+0000","last_became_peered":"2024-06-09T22:17:26.473975+0000","last_unstale":"2024-06-09T22:19:25.340802+0000","last_undegraded":"2024-06-09T22:19:25.340802+0000","last_fullsized":"2024-06-09T22:19:25.340802+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:13:46.998595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696429+0000","last_change":"2024-06-09T22:16:53.342273+0000","last_active":"2024-06-09T22:18:26.696429+0000","last_peered":"2024-06-09T22:18:26.696429+0000","last_clean":"2024-06-09T22:18:26.696429+0000","last_became_active":"2024-06-09T22:16:53.341827+0000","last_became_peered":"2024-06-09T22:16:53.341827+0000","last_unstale":"2024-06-09T22:18:26.696429+0000","last_undegraded":"2024-06-09T22:18:26.696429+0000","last_fullsized":"2024-06-09T22:18:26.696429+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:23:26.099737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"97'2","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695616+0000","last_change":"2024-06-09T22:16:40.070498+0000","last_active":"2024-06-09T22:18:26.695616+0000","last_peered":"2024-06-09T22:18:26.695616+0000","last_clean":"2024-06-09T22:18:26.695616+0000","last_became_active":"2024-06-09T22:16:40.070144+0000","last_became_peered":"2024-06-09T22:16:40.070144+0000","last_unstale":"2024-06-09T22:18:26.695616+0000","last_undegraded":"2024-06-09T22:18:26.695616+0000","last_fullsized":"2024-06-09T22:18:26.695616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:52:22.214832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"99'2","reported_seq":41,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.312986+0000","last_change":"2024-06-09T22:16:43.210448+0000","last_active":"2024-06-09T22:19:25.312986+0000","last_peered":"2024-06-09T22:19:25.312986+0000","last_clean":"2024-06-09T22:19:25.312986+0000","last_became_active":"2024-06-09T22:16:43.209980+0000","last_became_peered":"2024-06-09T22:16:43.209980+0000","last_unstale":"2024-06-09T22:19:25.312986+0000","last_undegraded":"2024-06-09T22:19:25.312986+0000","last_fullsized":"2024-06-09T22:19:25.312986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:58:15.691207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.697676+0000","last_change":"2024-06-09T22:16:53.340644+0000","last_active":"2024-06-09T22:18:26.697676+0000","last_peered":"2024-06-09T22:18:26.697676+0000","last_clean":"2024-06-09T22:18:26.697676+0000","last_became_active":"2024-06-09T22:16:53.339771+0000","last_became_peered":"2024-06-09T22:16:53.339771+0000","last_unstale":"2024-06-09T22:18:26.697676+0000","last_undegraded":"2024-06-09T22:18:26.697676+0000","last_fullsized":"2024-06-09T22:18:26.697676+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:45:23.281387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"97'7","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695861+0000","last_change":"2024-06-09T22:16:40.040461+0000","last_active":"2024-06-09T22:18:26.695861+0000","last_peered":"2024-06-09T22:18:26.695861+0000","last_clean":"2024-06-09T22:18:26.695861+0000","last_became_active":"2024-06-09T22:16:40.040168+0000","last_became_peered":"2024-06-09T22:16:40.040168+0000","last_unstale":"2024-06-09T22:18:26.695861+0000","last_undegraded":"2024-06-09T22:18:26.695861+0000","last_fullsized":"2024-06-09T22:18:26.695861+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:28.936631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"99'9","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.331262+0000","last_change":"2024-06-09T22:16:43.208779+0000","last_active":"2024-06-09T22:19:25.331262+0000","last_peered":"2024-06-09T22:19:25.331262+0000","last_clean":"2024-06-09T22:19:25.331262+0000","last_became_active":"2024-06-09T22:16:43.208136+0000","last_became_peered":"2024-06-09T22:16:43.208136+0000","last_unstale":"2024-06-09T22:19:25.331262+0000","last_undegraded":"2024-06-09T22:19:25.331262+0000","last_fullsized":"2024-06-09T22:19:25.331262+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:22:39.721069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"97'1","reported_seq":30,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695777+0000","last_change":"2024-06-09T22:16:53.331241+0000","last_active":"2024-06-09T22:18:26.695777+0000","last_peered":"2024-06-09T22:18:26.695777+0000","last_clean":"2024-06-09T22:18:26.695777+0000","last_became_active":"2024-06-09T22:16:53.329589+0000","last_became_peered":"2024-06-09T22:16:53.329589+0000","last_unstale":"2024-06-09T22:18:26.695777+0000","last_undegraded":"2024-06-09T22:18:26.695777+0000","last_fullsized":"2024-06-09T22:18:26.695777+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:56:21.319711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"97'3","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537743+0000","last_change":"2024-06-09T22:16:40.042491+0000","last_active":"2024-06-09T22:18:26.537743+0000","last_peered":"2024-06-09T22:18:26.537743+0000","last_clean":"2024-06-09T22:18:26.537743+0000","last_became_active":"2024-06-09T22:16:40.041846+0000","last_became_peered":"2024-06-09T22:16:40.041846+0000","last_unstale":"2024-06-09T22:18:26.537743+0000","last_undegraded":"2024-06-09T22:18:26.537743+0000","last_fullsized":"2024-06-09T22:18:26.537743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:59.144003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"99'8","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.341076+0000","last_change":"2024-06-09T22:16:43.209195+0000","last_active":"2024-06-09T22:19:25.341076+0000","last_peered":"2024-06-09T22:19:25.341076+0000","last_clean":"2024-06-09T22:19:25.341076+0000","last_became_active":"2024-06-09T22:16:43.208797+0000","last_became_peered":"2024-06-09T22:16:43.208797+0000","last_unstale":"2024-06-09T22:19:25.341076+0000","last_undegraded":"2024-06-09T22:19:25.341076+0000","last_fullsized":"2024-06-09T22:19:25.341076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:05:31.689900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696403+0000","last_change":"2024-06-09T22:16:53.342173+0000","last_active":"2024-06-09T22:18:26.696403+0000","last_peered":"2024-06-09T22:18:26.696403+0000","last_clean":"2024-06-09T22:18:26.696403+0000","last_became_active":"2024-06-09T22:16:53.341712+0000","last_became_peered":"2024-06-09T22:16:53.341712+0000","last_unstale":"2024-06-09T22:18:26.696403+0000","last_undegraded":"2024-06-09T22:18:26.696403+0000","last_fullsized":"2024-06-09T22:18:26.696403+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:53:12.574764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"97'4","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535214+0000","last_change":"2024-06-09T22:16:40.044238+0000","last_active":"2024-06-09T22:18:26.535214+0000","last_peered":"2024-06-09T22:18:26.535214+0000","last_clean":"2024-06-09T22:18:26.535214+0000","last_became_active":"2024-06-09T22:16:40.043857+0000","last_became_peered":"2024-06-09T22:16:40.043857+0000","last_unstale":"2024-06-09T22:18:26.535214+0000","last_undegraded":"2024-06-09T22:18:26.535214+0000","last_fullsized":"2024-06-09T22:18:26.535214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:44:45.743307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"99'5","reported_seq":44,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.341149+0000","last_change":"2024-06-09T22:16:43.202363+0000","last_active":"2024-06-09T22:19:25.341149+0000","last_peered":"2024-06-09T22:19:25.341149+0000","last_clean":"2024-06-09T22:19:25.341149+0000","last_became_active":"2024-06-09T22:16:43.202029+0000","last_became_peered":"2024-06-09T22:16:43.202029+0000","last_unstale":"2024-06-09T22:19:25.341149+0000","last_undegraded":"2024-06-09T22:19:25.341149+0000","last_fullsized":"2024-06-09T22:19:25.341149+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:11:19.513055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696098+0000","last_change":"2024-06-09T22:16:53.333837+0000","last_active":"2024-06-09T22:18:26.696098+0000","last_peered":"2024-06-09T22:18:26.696098+0000","last_clean":"2024-06-09T22:18:26.696098+0000","last_became_active":"2024-06-09T22:16:53.333362+0000","last_became_peered":"2024-06-09T22:16:53.333362+0000","last_unstale":"2024-06-09T22:18:26.696098+0000","last_undegraded":"2024-06-09T22:18:26.696098+0000","last_fullsized":"2024-06-09T22:18:26.696098+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:47:04.463040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"99'7","reported_seq":152,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:27.584512+0000","last_change":"2024-06-09T22:16:40.364834+0000","last_active":"2024-06-09T22:19:27.584512+0000","last_peered":"2024-06-09T22:19:27.584512+0000","last_clean":"2024-06-09T22:19:27.584512+0000","last_became_active":"2024-06-09T22:16:40.364433+0000","last_became_peered":"2024-06-09T22:16:40.364433+0000","last_unstale":"2024-06-09T22:19:27.584512+0000","last_undegraded":"2024-06-09T22:19:27.584512+0000","last_fullsized":"2024-06-09T22:19:27.584512+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:43:01.967911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"99'7","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.335979+0000","last_change":"2024-06-09T22:16:43.201183+0000","last_active":"2024-06-09T22:19:25.335979+0000","last_peered":"2024-06-09T22:19:25.335979+0000","last_clean":"2024-06-09T22:19:25.335979+0000","last_became_active":"2024-06-09T22:16:43.200774+0000","last_became_peered":"2024-06-09T22:16:43.200774+0000","last_unstale":"2024-06-09T22:19:25.335979+0000","last_undegraded":"2024-06-09T22:19:25.335979+0000","last_fullsized":"2024-06-09T22:19:25.335979+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:07:55.772066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":36,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.699557+0000","last_change":"2024-06-09T22:18:26.699190+0000","last_active":"2024-06-09T22:18:26.699557+0000","last_peered":"2024-06-09T22:18:26.699557+0000","last_clean":"2024-06-09T22:18:26.699557+0000","last_became_active":"2024-06-09T22:18:26.698614+0000","last_became_peered":"2024-06-09T22:18:26.698614+0000","last_unstale":"2024-06-09T22:18:26.699557+0000","last_undegraded":"2024-06-09T22:18:26.699557+0000","last_fullsized":"2024-06-09T22:18:26.699557+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:45.343605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,1,4],"acting":[0,10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"97'3","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696079+0000","last_change":"2024-06-09T22:16:40.040582+0000","last_active":"2024-06-09T22:18:26.696079+0000","last_peered":"2024-06-09T22:18:26.696079+0000","last_clean":"2024-06-09T22:18:26.696079+0000","last_became_active":"2024-06-09T22:16:40.040152+0000","last_became_peered":"2024-06-09T22:16:40.040152+0000","last_unstale":"2024-06-09T22:18:26.696079+0000","last_undegraded":"2024-06-09T22:18:26.696079+0000","last_fullsized":"2024-06-09T22:18:26.696079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:50:00.878096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"99'6","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.340841+0000","last_change":"2024-06-09T22:16:43.206211+0000","last_active":"2024-06-09T22:19:25.340841+0000","last_peered":"2024-06-09T22:19:25.340841+0000","last_clean":"2024-06-09T22:19:25.340841+0000","last_became_active":"2024-06-09T22:16:43.205814+0000","last_became_peered":"2024-06-09T22:16:43.205814+0000","last_unstale":"2024-06-09T22:19:25.340841+0000","last_undegraded":"2024-06-09T22:19:25.340841+0000","last_fullsized":"2024-06-09T22:19:25.340841+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:20:19.058475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695056+0000","last_change":"2024-06-09T22:16:53.337695+0000","last_active":"2024-06-09T22:18:26.695056+0000","last_peered":"2024-06-09T22:18:26.695056+0000","last_clean":"2024-06-09T22:18:26.695056+0000","last_became_active":"2024-06-09T22:16:53.334872+0000","last_became_peered":"2024-06-09T22:16:53.334872+0000","last_unstale":"2024-06-09T22:18:26.695056+0000","last_undegraded":"2024-06-09T22:18:26.695056+0000","last_fullsized":"2024-06-09T22:18:26.695056+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:50:36.617736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"97'5","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696936+0000","last_change":"2024-06-09T22:16:40.036603+0000","last_active":"2024-06-09T22:18:26.696936+0000","last_peered":"2024-06-09T22:18:26.696936+0000","last_clean":"2024-06-09T22:18:26.696936+0000","last_became_active":"2024-06-09T22:16:40.036274+0000","last_became_peered":"2024-06-09T22:16:40.036274+0000","last_unstale":"2024-06-09T22:18:26.696936+0000","last_undegraded":"2024-06-09T22:18:26.696936+0000","last_fullsized":"2024-06-09T22:18:26.696936+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:31.652897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"99'9","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.336962+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:19:25.336962+0000","last_peered":"2024-06-09T22:19:25.336962+0000","last_clean":"2024-06-09T22:19:25.336962+0000","last_became_active":"2024-06-09T22:16:43.204197+0000","last_became_peered":"2024-06-09T22:16:43.204197+0000","last_unstale":"2024-06-09T22:19:25.336962+0000","last_undegraded":"2024-06-09T22:19:25.336962+0000","last_fullsized":"2024-06-09T22:19:25.336962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:38:49.850120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"99'9","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.337895+0000","last_change":"2024-06-09T22:16:43.208477+0000","last_active":"2024-06-09T22:19:25.337895+0000","last_peered":"2024-06-09T22:19:25.337895+0000","last_clean":"2024-06-09T22:19:25.337895+0000","last_became_active":"2024-06-09T22:16:43.206018+0000","last_became_peered":"2024-06-09T22:16:43.206018+0000","last_unstale":"2024-06-09T22:19:25.337895+0000","last_undegraded":"2024-06-09T22:19:25.337895+0000","last_fullsized":"2024-06-09T22:19:25.337895+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:19:49.870825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"99'12","reported_seq":49,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.343127+0000","last_change":"2024-06-09T22:16:43.206809+0000","last_active":"2024-06-09T22:19:25.343127+0000","last_peered":"2024-06-09T22:19:25.343127+0000","last_clean":"2024-06-09T22:19:25.343127+0000","last_became_active":"2024-06-09T22:16:43.206175+0000","last_became_peered":"2024-06-09T22:16:43.206175+0000","last_unstale":"2024-06-09T22:19:25.343127+0000","last_undegraded":"2024-06-09T22:19:25.343127+0000","last_fullsized":"2024-06-09T22:19:25.343127+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:39:22.502266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"99'8","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.340310+0000","last_change":"2024-06-09T22:16:43.202163+0000","last_active":"2024-06-09T22:19:25.340310+0000","last_peered":"2024-06-09T22:19:25.340310+0000","last_clean":"2024-06-09T22:19:25.340310+0000","last_became_active":"2024-06-09T22:16:43.201839+0000","last_became_peered":"2024-06-09T22:16:43.201839+0000","last_unstale":"2024-06-09T22:19:25.340310+0000","last_undegraded":"2024-06-09T22:19:25.340310+0000","last_fullsized":"2024-06-09T22:19:25.340310+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:47:46.723289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"99'7","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.339893+0000","last_change":"2024-06-09T22:16:43.202734+0000","last_active":"2024-06-09T22:19:25.339893+0000","last_peered":"2024-06-09T22:19:25.339893+0000","last_clean":"2024-06-09T22:19:25.339893+0000","last_became_active":"2024-06-09T22:16:43.201638+0000","last_became_peered":"2024-06-09T22:16:43.201638+0000","last_unstale":"2024-06-09T22:19:25.339893+0000","last_undegraded":"2024-06-09T22:19:25.339893+0000","last_fullsized":"2024-06-09T22:19:25.339893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:16.360985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"99'4","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.339155+0000","last_change":"2024-06-09T22:16:43.199113+0000","last_active":"2024-06-09T22:19:25.339155+0000","last_peered":"2024-06-09T22:19:25.339155+0000","last_clean":"2024-06-09T22:19:25.339155+0000","last_became_active":"2024-06-09T22:16:43.198804+0000","last_became_peered":"2024-06-09T22:16:43.198804+0000","last_unstale":"2024-06-09T22:19:25.339155+0000","last_undegraded":"2024-06-09T22:19:25.339155+0000","last_fullsized":"2024-06-09T22:19:25.339155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:18:08.999477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"99'6","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.326942+0000","last_change":"2024-06-09T22:16:43.204445+0000","last_active":"2024-06-09T22:19:25.326942+0000","last_peered":"2024-06-09T22:19:25.326942+0000","last_clean":"2024-06-09T22:19:25.326942+0000","last_became_active":"2024-06-09T22:16:43.203081+0000","last_became_peered":"2024-06-09T22:16:43.203081+0000","last_unstale":"2024-06-09T22:19:25.326942+0000","last_undegraded":"2024-06-09T22:19:25.326942+0000","last_fullsized":"2024-06-09T22:19:25.326942+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:39.303813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"99'15","reported_seq":52,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.333117+0000","last_change":"2024-06-09T22:16:43.211402+0000","last_active":"2024-06-09T22:19:25.333117+0000","last_peered":"2024-06-09T22:19:25.333117+0000","last_clean":"2024-06-09T22:19:25.333117+0000","last_became_active":"2024-06-09T22:16:43.210871+0000","last_became_peered":"2024-06-09T22:16:43.210871+0000","last_unstale":"2024-06-09T22:19:25.333117+0000","last_undegraded":"2024-06-09T22:19:25.333117+0000","last_fullsized":"2024-06-09T22:19:25.333117+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:07:21.467617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"99'2","reported_seq":40,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.328895+0000","last_change":"2024-06-09T22:16:43.201159+0000","last_active":"2024-06-09T22:19:25.328895+0000","last_peered":"2024-06-09T22:19:25.328895+0000","last_clean":"2024-06-09T22:19:25.328895+0000","last_became_active":"2024-06-09T22:16:43.200850+0000","last_became_peered":"2024-06-09T22:16:43.200850+0000","last_unstale":"2024-06-09T22:19:25.328895+0000","last_undegraded":"2024-06-09T22:19:25.328895+0000","last_fullsized":"2024-06-09T22:19:25.328895+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:20.515309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"99'6","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.336981+0000","last_change":"2024-06-09T22:16:43.196830+0000","last_active":"2024-06-09T22:19:25.336981+0000","last_peered":"2024-06-09T22:19:25.336981+0000","last_clean":"2024-06-09T22:19:25.336981+0000","last_became_active":"2024-06-09T22:16:43.195682+0000","last_became_peered":"2024-06-09T22:16:43.195682+0000","last_unstale":"2024-06-09T22:19:25.336981+0000","last_undegraded":"2024-06-09T22:19:25.336981+0000","last_fullsized":"2024-06-09T22:19:25.336981+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:34:18.770235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"99'8","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.331017+0000","last_change":"2024-06-09T22:16:43.201959+0000","last_active":"2024-06-09T22:19:25.331017+0000","last_peered":"2024-06-09T22:19:25.331017+0000","last_clean":"2024-06-09T22:19:25.331017+0000","last_became_active":"2024-06-09T22:16:43.201669+0000","last_became_peered":"2024-06-09T22:16:43.201669+0000","last_unstale":"2024-06-09T22:19:25.331017+0000","last_undegraded":"2024-06-09T22:19:25.331017+0000","last_fullsized":"2024-06-09T22:19:25.331017+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:48:57.348041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"99'10","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.341873+0000","last_change":"2024-06-09T22:16:43.201823+0000","last_active":"2024-06-09T22:19:25.341873+0000","last_peered":"2024-06-09T22:19:25.341873+0000","last_clean":"2024-06-09T22:19:25.341873+0000","last_became_active":"2024-06-09T22:16:43.201434+0000","last_became_peered":"2024-06-09T22:16:43.201434+0000","last_unstale":"2024-06-09T22:19:25.341873+0000","last_undegraded":"2024-06-09T22:19:25.341873+0000","last_fullsized":"2024-06-09T22:19:25.341873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:56:13.540895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"99'11","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.340132+0000","last_change":"2024-06-09T22:16:43.197734+0000","last_active":"2024-06-09T22:19:25.340132+0000","last_peered":"2024-06-09T22:19:25.340132+0000","last_clean":"2024-06-09T22:19:25.340132+0000","last_became_active":"2024-06-09T22:16:43.197442+0000","last_became_peered":"2024-06-09T22:16:43.197442+0000","last_unstale":"2024-06-09T22:19:25.340132+0000","last_undegraded":"2024-06-09T22:19:25.340132+0000","last_fullsized":"2024-06-09T22:19:25.340132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:35:40.720718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"99'8","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.335947+0000","last_change":"2024-06-09T22:16:43.210375+0000","last_active":"2024-06-09T22:19:25.335947+0000","last_peered":"2024-06-09T22:19:25.335947+0000","last_clean":"2024-06-09T22:19:25.335947+0000","last_became_active":"2024-06-09T22:16:43.209647+0000","last_became_peered":"2024-06-09T22:16:43.209647+0000","last_unstale":"2024-06-09T22:19:25.335947+0000","last_undegraded":"2024-06-09T22:19:25.335947+0000","last_fullsized":"2024-06-09T22:19:25.335947+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:11:57.593066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247891,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":76151,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1022,"num_read_kb":593,"num_write":262,"num_write_kb":158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":228453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383730,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":570974,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645367,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32404,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712748,"statfs":{"total":95995035648,"available":95961853952,"internally_reserved":0,"allocated":1257472,"data_stored":1225858,"data_compressed":68186,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808892,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":574401,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070530,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1122304,"data_stored":570683,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332166,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1175552,"data_stored":1167851,"data_compressed":68186,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495692,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1101824,"data_stored":565809,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36484,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708668,"statfs":{"total":95995035648,"available":95957676032,"internally_reserved":0,"allocated":1175552,"data_stored":634975,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1126400,"data_stored":567796,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1187840,"data_stored":636189,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182490,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":565517,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920854,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1110016,"data_stored":570414,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757329,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32332,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712820,"statfs":{"total":95995035648,"available":95961927680,"internally_reserved":0,"allocated":1183744,"data_stored":1162902,"data_compressed":68186,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34243,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34243,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":70403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-09T22:19:33.387 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-09T22:19:33.631 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:34.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:34 smithi153 ceph-mon[36599]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:19:34.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:34 smithi153 ceph-mon[36599]: pgmap v393: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 767 B/s wr, 11 op/s 2024-06-09T22:19:34.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:34 smithi104 ceph-mon[32389]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:19:34.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:34 smithi104 ceph-mon[32389]: pgmap v393: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 767 B/s wr, 11 op/s 2024-06-09T22:19:34.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:34 smithi194 ceph-mon[36856]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:19:34.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:34 smithi194 ceph-mon[36856]: pgmap v393: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 767 B/s wr, 11 op/s 2024-06-09T22:19:35.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.357+0000 7f08fae5c700 1 -- 172.21.15.104:0/1869607793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08f41024e0 msgr2=0x7f08f4102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:35.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.357+0000 7f08fae5c700 1 --2- 172.21.15.104:0/1869607793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08f41024e0 0x7f08f4102940 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009990 tx=0x7f08e0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:19:35.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.357+0000 7f08fae5c700 1 -- 172.21.15.104:0/1869607793 shutdown_connections 2024-06-09T22:19:35.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.357+0000 7f08fae5c700 1 --2- 172.21.15.104:0/1869607793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08f4102e80 0x7f08f410f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:35.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.357+0000 7f08fae5c700 1 --2- 172.21.15.104:0/1869607793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f08f41084f0 0x7f08f41088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:35.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.357+0000 7f08fae5c700 1 --2- 172.21.15.104:0/1869607793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08f41024e0 0x7f08f4102940 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:35.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.358+0000 7f08fae5c700 1 -- 172.21.15.104:0/1869607793 >> 172.21.15.104:0/1869607793 conn(0x7f08f4079530 msgr2=0x7f08f4079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:35.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.358+0000 7f08fae5c700 1 -- 172.21.15.104:0/1869607793 shutdown_connections 2024-06-09T22:19:35.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.358+0000 7f08fae5c700 1 -- 172.21.15.104:0/1869607793 wait complete. 2024-06-09T22:19:35.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.358+0000 7f08fae5c700 1 Processor -- start 2024-06-09T22:19:35.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.358+0000 7f08fae5c700 1 -- start start 2024-06-09T22:19:35.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.358+0000 7f08fae5c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f08f41024e0 0x7f08f4074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:35.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.358+0000 7f08fae5c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08f4102e80 0x7f08f4074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:35.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.358+0000 7f08fae5c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08f41084f0 0x7f08f4075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.358+0000 7f08fae5c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f08f4112750 con 0x7f08f4102e80 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.358+0000 7f08fae5c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08f41125d0 con 0x7f08f41084f0 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.358+0000 7f08fae5c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f08f41128d0 con 0x7f08f41024e0 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.358+0000 7f08f3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08f4102e80 0x7f08f4074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08f4102e80 0x7f08f4074cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38498/0 (socket says 172.21.15.104:38498) 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f3fff700 1 -- 172.21.15.104:0/541287796 learned_addr learned my addr 172.21.15.104:0/541287796 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f8bf8700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f08f41024e0 0x7f08f4074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f93f9700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08f41084f0 0x7f08f4075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f3fff700 1 -- 172.21.15.104:0/541287796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f08f41024e0 msgr2=0x7f08f4074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f3fff700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f08f41024e0 0x7f08f4074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f3fff700 1 -- 172.21.15.104:0/541287796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08f41084f0 msgr2=0x7f08f4075210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f3fff700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08f41084f0 0x7f08f4075210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f3fff700 1 -- 172.21.15.104:0/541287796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f41a7190 con 0x7f08f4102e80 2024-06-09T22:19:35.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f93f9700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08f41084f0 0x7f08f4075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:19:35.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f3fff700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08f4102e80 0x7f08f4074cd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f08e0000c00 tx=0x7f08e0003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:35.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f1ffb700 1 -- 172.21.15.104:0/541287796 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e0027070 con 0x7f08f4102e80 2024-06-09T22:19:35.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f1ffb700 1 -- 172.21.15.104:0/541287796 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08e0004110 con 0x7f08f4102e80 2024-06-09T22:19:35.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08f1ffb700 1 -- 172.21.15.104:0/541287796 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e000f460 con 0x7f08f4102e80 2024-06-09T22:19:35.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.359+0000 7f08fae5c700 1 -- 172.21.15.104:0/541287796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08f41a7420 con 0x7f08f4102e80 2024-06-09T22:19:35.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.360+0000 7f08fae5c700 1 -- 172.21.15.104:0/541287796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08f41a7820 con 0x7f08f4102e80 2024-06-09T22:19:35.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.360+0000 7f08fae5c700 1 -- 172.21.15.104:0/541287796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08f4069180 con 0x7f08f4102e80 2024-06-09T22:19:35.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.362+0000 7f08f1ffb700 1 -- 172.21.15.104:0/541287796 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f08e002a070 con 0x7f08f4102e80 2024-06-09T22:19:35.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.362+0000 7f08f1ffb700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f08e40a2ca0 0x7f08e40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:35.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.363+0000 7f08f8bf8700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f08e40a2ca0 0x7f08e40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:35.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.363+0000 7f08f1ffb700 1 -- 172.21.15.104:0/541287796 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f08e00d02d0 con 0x7f08f4102e80 2024-06-09T22:19:35.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.363+0000 7f08f8bf8700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f08e40a2ca0 0x7f08e40a5160 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f08e8005950 tx=0x7f08e800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:35.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.367+0000 7f08f1ffb700 1 -- 172.21.15.104:0/541287796 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f08e005e850 con 0x7f08f4102e80 2024-06-09T22:19:35.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.561+0000 7f08fae5c700 1 -- 172.21.15.104:0/541287796 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f08f40628d0 con 0x7f08e40a2ca0 2024-06-09T22:19:35.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.570+0000 7f08f1ffb700 1 -- 172.21.15.104:0/541287796 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340792 (secure 0 0 0) 0x7f08f40628d0 con 0x7f08e40a2ca0 2024-06-09T22:19:35.575 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:19:35.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.573+0000 7f08fae5c700 1 -- 172.21.15.104:0/541287796 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f08e40a2ca0 msgr2=0x7f08e40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:35.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.573+0000 7f08fae5c700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f08e40a2ca0 0x7f08e40a5160 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f08e8005950 tx=0x7f08e800a300 comp rx=0 tx=0).stop 2024-06-09T22:19:35.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.573+0000 7f08fae5c700 1 -- 172.21.15.104:0/541287796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08f4102e80 msgr2=0x7f08f4074cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:35.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.573+0000 7f08fae5c700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08f4102e80 0x7f08f4074cd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f08e0000c00 tx=0x7f08e0003680 comp rx=0 tx=0).stop 2024-06-09T22:19:35.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.574+0000 7f08fae5c700 1 -- 172.21.15.104:0/541287796 shutdown_connections 2024-06-09T22:19:35.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.574+0000 7f08fae5c700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f08e40a2ca0 0x7f08e40a5160 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:35.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.574+0000 7f08fae5c700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08f41084f0 0x7f08f4075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:35.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.574+0000 7f08fae5c700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f08f41024e0 0x7f08f4074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:35.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.574+0000 7f08fae5c700 1 --2- 172.21.15.104:0/541287796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08f4102e80 0x7f08f4074cd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:35.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.574+0000 7f08fae5c700 1 -- 172.21.15.104:0/541287796 >> 172.21.15.104:0/541287796 conn(0x7f08f4079530 msgr2=0x7f08f4100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:35.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.574+0000 7f08fae5c700 1 -- 172.21.15.104:0/541287796 shutdown_connections 2024-06-09T22:19:35.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:35.574+0000 7f08fae5c700 1 -- 172.21.15.104:0/541287796 wait complete. 2024-06-09T22:19:35.578 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-06-09T22:19:36.390 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-06-09T22:19:35.352128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666753,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1203,"num_read_kb":732,"num_write":445,"num_write_kb":3098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":866,"ondisk_log_size":866,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379264,"kb_used_data":13472,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562560,"statfs":{"total":1151940427776,"available":1151552061440,"internally_reserved":0,"allocated":13795328,"data_stored":8813369,"data_compressed":495303,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":124,"num_read_kb":54,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003282"},"pg_stats":[{"pgid":"3.3f","version":"99'8","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.337332+0000","last_change":"2024-06-09T22:16:43.212998+0000","last_active":"2024-06-09T22:19:25.337332+0000","last_peered":"2024-06-09T22:19:25.337332+0000","last_clean":"2024-06-09T22:19:25.337332+0000","last_became_active":"2024-06-09T22:16:43.212614+0000","last_became_peered":"2024-06-09T22:16:43.212614+0000","last_unstale":"2024-06-09T22:19:25.337332+0000","last_undegraded":"2024-06-09T22:19:25.337332+0000","last_fullsized":"2024-06-09T22:19:25.337332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:22.728486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"99'4","reported_seq":42,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.337169+0000","last_change":"2024-06-09T22:16:43.208746+0000","last_active":"2024-06-09T22:19:25.337169+0000","last_peered":"2024-06-09T22:19:25.337169+0000","last_clean":"2024-06-09T22:19:25.337169+0000","last_became_active":"2024-06-09T22:16:43.208458+0000","last_became_peered":"2024-06-09T22:16:43.208458+0000","last_unstale":"2024-06-09T22:19:25.337169+0000","last_undegraded":"2024-06-09T22:19:25.337169+0000","last_fullsized":"2024-06-09T22:19:25.337169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:24:35.350449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"99'7","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.343331+0000","last_change":"2024-06-09T22:16:43.200770+0000","last_active":"2024-06-09T22:19:25.343331+0000","last_peered":"2024-06-09T22:19:25.343331+0000","last_clean":"2024-06-09T22:19:25.343331+0000","last_became_active":"2024-06-09T22:16:43.200449+0000","last_became_peered":"2024-06-09T22:16:43.200449+0000","last_unstale":"2024-06-09T22:19:25.343331+0000","last_undegraded":"2024-06-09T22:19:25.343331+0000","last_fullsized":"2024-06-09T22:19:25.343331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:51:09.017260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"99'8","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.329609+0000","last_change":"2024-06-09T22:16:43.203844+0000","last_active":"2024-06-09T22:19:25.329609+0000","last_peered":"2024-06-09T22:19:25.329609+0000","last_clean":"2024-06-09T22:19:25.329609+0000","last_became_active":"2024-06-09T22:16:43.203431+0000","last_became_peered":"2024-06-09T22:16:43.203431+0000","last_unstale":"2024-06-09T22:19:25.329609+0000","last_undegraded":"2024-06-09T22:19:25.329609+0000","last_fullsized":"2024-06-09T22:19:25.329609+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:51:36.913312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"99'4","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.330178+0000","last_change":"2024-06-09T22:16:43.207597+0000","last_active":"2024-06-09T22:19:25.330178+0000","last_peered":"2024-06-09T22:19:25.330178+0000","last_clean":"2024-06-09T22:19:25.330178+0000","last_became_active":"2024-06-09T22:16:43.207266+0000","last_became_peered":"2024-06-09T22:16:43.207266+0000","last_unstale":"2024-06-09T22:19:25.330178+0000","last_undegraded":"2024-06-09T22:19:25.330178+0000","last_fullsized":"2024-06-09T22:19:25.330178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:47:05.558403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"99'6","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.333457+0000","last_change":"2024-06-09T22:17:26.474121+0000","last_active":"2024-06-09T22:19:25.333457+0000","last_peered":"2024-06-09T22:19:25.333457+0000","last_clean":"2024-06-09T22:19:25.333457+0000","last_became_active":"2024-06-09T22:17:26.473619+0000","last_became_peered":"2024-06-09T22:17:26.473619+0000","last_unstale":"2024-06-09T22:19:25.333457+0000","last_undegraded":"2024-06-09T22:19:25.333457+0000","last_fullsized":"2024-06-09T22:19:25.333457+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:56:42.171235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"99'5","reported_seq":44,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.340581+0000","last_change":"2024-06-09T22:16:43.208815+0000","last_active":"2024-06-09T22:19:25.340581+0000","last_peered":"2024-06-09T22:19:25.340581+0000","last_clean":"2024-06-09T22:19:25.340581+0000","last_became_active":"2024-06-09T22:16:43.208433+0000","last_became_peered":"2024-06-09T22:16:43.208433+0000","last_unstale":"2024-06-09T22:19:25.340581+0000","last_undegraded":"2024-06-09T22:19:25.340581+0000","last_fullsized":"2024-06-09T22:19:25.340581+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:36:26.585037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"99'9","reported_seq":47,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.334575+0000","last_change":"2024-06-09T22:16:43.210195+0000","last_active":"2024-06-09T22:19:25.334575+0000","last_peered":"2024-06-09T22:19:25.334575+0000","last_clean":"2024-06-09T22:19:25.334575+0000","last_became_active":"2024-06-09T22:16:43.209892+0000","last_became_peered":"2024-06-09T22:16:43.209892+0000","last_unstale":"2024-06-09T22:19:25.334575+0000","last_undegraded":"2024-06-09T22:19:25.334575+0000","last_fullsized":"2024-06-09T22:19:25.334575+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:44.088212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"99'11","reported_seq":47,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.340921+0000","last_change":"2024-06-09T22:16:43.202517+0000","last_active":"2024-06-09T22:19:25.340921+0000","last_peered":"2024-06-09T22:19:25.340921+0000","last_clean":"2024-06-09T22:19:25.340921+0000","last_became_active":"2024-06-09T22:16:43.201796+0000","last_became_peered":"2024-06-09T22:16:43.201796+0000","last_unstale":"2024-06-09T22:19:25.340921+0000","last_undegraded":"2024-06-09T22:19:25.340921+0000","last_fullsized":"2024-06-09T22:19:25.340921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:30.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"99'16","reported_seq":52,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.331100+0000","last_change":"2024-06-09T22:16:43.212732+0000","last_active":"2024-06-09T22:19:25.331100+0000","last_peered":"2024-06-09T22:19:25.331100+0000","last_clean":"2024-06-09T22:19:25.331100+0000","last_became_active":"2024-06-09T22:16:43.212048+0000","last_became_peered":"2024-06-09T22:16:43.212048+0000","last_unstale":"2024-06-09T22:19:25.331100+0000","last_undegraded":"2024-06-09T22:19:25.331100+0000","last_fullsized":"2024-06-09T22:19:25.331100+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:47.383291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"99'6","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.335545+0000","last_change":"2024-06-09T22:16:43.203778+0000","last_active":"2024-06-09T22:19:25.335545+0000","last_peered":"2024-06-09T22:19:25.335545+0000","last_clean":"2024-06-09T22:19:25.335545+0000","last_became_active":"2024-06-09T22:16:43.202954+0000","last_became_peered":"2024-06-09T22:16:43.202954+0000","last_unstale":"2024-06-09T22:19:25.335545+0000","last_undegraded":"2024-06-09T22:19:25.335545+0000","last_fullsized":"2024-06-09T22:19:25.335545+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:56:53.708645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"99'7","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.325125+0000","last_change":"2024-06-09T22:16:43.211425+0000","last_active":"2024-06-09T22:19:25.325125+0000","last_peered":"2024-06-09T22:19:25.325125+0000","last_clean":"2024-06-09T22:19:25.325125+0000","last_became_active":"2024-06-09T22:16:43.211001+0000","last_became_peered":"2024-06-09T22:16:43.211001+0000","last_unstale":"2024-06-09T22:19:25.325125+0000","last_undegraded":"2024-06-09T22:19:25.325125+0000","last_fullsized":"2024-06-09T22:19:25.325125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:20:59.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"99'10","reported_seq":49,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.341452+0000","last_change":"2024-06-09T22:16:43.208334+0000","last_active":"2024-06-09T22:19:25.341452+0000","last_peered":"2024-06-09T22:19:25.341452+0000","last_clean":"2024-06-09T22:19:25.341452+0000","last_became_active":"2024-06-09T22:16:43.207993+0000","last_became_peered":"2024-06-09T22:16:43.207993+0000","last_unstale":"2024-06-09T22:19:25.341452+0000","last_undegraded":"2024-06-09T22:19:25.341452+0000","last_fullsized":"2024-06-09T22:19:25.341452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:07:05.740748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"99'7","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.335591+0000","last_change":"2024-06-09T22:16:43.207211+0000","last_active":"2024-06-09T22:19:25.335591+0000","last_peered":"2024-06-09T22:19:25.335591+0000","last_clean":"2024-06-09T22:19:25.335591+0000","last_became_active":"2024-06-09T22:16:43.206889+0000","last_became_peered":"2024-06-09T22:16:43.206889+0000","last_unstale":"2024-06-09T22:19:25.335591+0000","last_undegraded":"2024-06-09T22:19:25.335591+0000","last_fullsized":"2024-06-09T22:19:25.335591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:25:47.461030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"99'7","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.338216+0000","last_change":"2024-06-09T22:16:43.202202+0000","last_active":"2024-06-09T22:19:25.338216+0000","last_peered":"2024-06-09T22:19:25.338216+0000","last_clean":"2024-06-09T22:19:25.338216+0000","last_became_active":"2024-06-09T22:16:43.201834+0000","last_became_peered":"2024-06-09T22:16:43.201834+0000","last_unstale":"2024-06-09T22:19:25.338216+0000","last_undegraded":"2024-06-09T22:19:25.338216+0000","last_fullsized":"2024-06-09T22:19:25.338216+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:31:51.296627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"99'9","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.342020+0000","last_change":"2024-06-09T22:16:43.200637+0000","last_active":"2024-06-09T22:19:25.342020+0000","last_peered":"2024-06-09T22:19:25.342020+0000","last_clean":"2024-06-09T22:19:25.342020+0000","last_became_active":"2024-06-09T22:16:43.200268+0000","last_became_peered":"2024-06-09T22:16:43.200268+0000","last_unstale":"2024-06-09T22:19:25.342020+0000","last_undegraded":"2024-06-09T22:19:25.342020+0000","last_fullsized":"2024-06-09T22:19:25.342020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:25:59.881453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"99'8","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.339163+0000","last_change":"2024-06-09T22:16:43.209212+0000","last_active":"2024-06-09T22:19:25.339163+0000","last_peered":"2024-06-09T22:19:25.339163+0000","last_clean":"2024-06-09T22:19:25.339163+0000","last_became_active":"2024-06-09T22:16:43.207789+0000","last_became_peered":"2024-06-09T22:16:43.207789+0000","last_unstale":"2024-06-09T22:19:25.339163+0000","last_undegraded":"2024-06-09T22:19:25.339163+0000","last_fullsized":"2024-06-09T22:19:25.339163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:17.316595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"99'10","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.326694+0000","last_change":"2024-06-09T22:16:43.210935+0000","last_active":"2024-06-09T22:19:25.326694+0000","last_peered":"2024-06-09T22:19:25.326694+0000","last_clean":"2024-06-09T22:19:25.326694+0000","last_became_active":"2024-06-09T22:16:43.210631+0000","last_became_peered":"2024-06-09T22:16:43.210631+0000","last_unstale":"2024-06-09T22:19:25.326694+0000","last_undegraded":"2024-06-09T22:19:25.326694+0000","last_fullsized":"2024-06-09T22:19:25.326694+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:22:58.887313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"99'7","reported_seq":44,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.322868+0000","last_change":"2024-06-09T22:16:43.198872+0000","last_active":"2024-06-09T22:19:25.322868+0000","last_peered":"2024-06-09T22:19:25.322868+0000","last_clean":"2024-06-09T22:19:25.322868+0000","last_became_active":"2024-06-09T22:16:43.198498+0000","last_became_peered":"2024-06-09T22:16:43.198498+0000","last_unstale":"2024-06-09T22:19:25.322868+0000","last_undegraded":"2024-06-09T22:19:25.322868+0000","last_fullsized":"2024-06-09T22:19:25.322868+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:25:28.853048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537676+0000","last_change":"2024-06-09T22:16:53.343656+0000","last_active":"2024-06-09T22:18:26.537676+0000","last_peered":"2024-06-09T22:18:26.537676+0000","last_clean":"2024-06-09T22:18:26.537676+0000","last_became_active":"2024-06-09T22:16:53.343281+0000","last_became_peered":"2024-06-09T22:16:53.343281+0000","last_unstale":"2024-06-09T22:18:26.537676+0000","last_undegraded":"2024-06-09T22:18:26.537676+0000","last_fullsized":"2024-06-09T22:18:26.537676+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:40.961842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"97'13","reported_seq":157,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:29.584224+0000","last_change":"2024-06-09T22:16:40.040145+0000","last_active":"2024-06-09T22:19:29.584224+0000","last_peered":"2024-06-09T22:19:29.584224+0000","last_clean":"2024-06-09T22:19:29.584224+0000","last_became_active":"2024-06-09T22:16:40.039787+0000","last_became_peered":"2024-06-09T22:16:40.039787+0000","last_unstale":"2024-06-09T22:19:29.584224+0000","last_undegraded":"2024-06-09T22:19:29.584224+0000","last_fullsized":"2024-06-09T22:19:29.584224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:52:32.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":101,"num_read_kb":101,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"99'5","reported_seq":21,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.323678+0000","last_change":"2024-06-09T22:17:26.774053+0000","last_active":"2024-06-09T22:19:25.323678+0000","last_peered":"2024-06-09T22:19:25.323678+0000","last_clean":"2024-06-09T22:19:25.323678+0000","last_became_active":"2024-06-09T22:17:26.773786+0000","last_became_peered":"2024-06-09T22:17:26.773786+0000","last_unstale":"2024-06-09T22:19:25.323678+0000","last_undegraded":"2024-06-09T22:19:25.323678+0000","last_fullsized":"2024-06-09T22:19:25.323678+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:36.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695534+0000","last_change":"2024-06-09T22:16:53.332048+0000","last_active":"2024-06-09T22:18:26.695534+0000","last_peered":"2024-06-09T22:18:26.695534+0000","last_clean":"2024-06-09T22:18:26.695534+0000","last_became_active":"2024-06-09T22:16:53.330683+0000","last_became_peered":"2024-06-09T22:16:53.330683+0000","last_unstale":"2024-06-09T22:18:26.695534+0000","last_undegraded":"2024-06-09T22:18:26.695534+0000","last_fullsized":"2024-06-09T22:18:26.695534+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:32:23.219061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"97'2","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.697538+0000","last_change":"2024-06-09T22:16:40.070193+0000","last_active":"2024-06-09T22:18:26.697538+0000","last_peered":"2024-06-09T22:18:26.697538+0000","last_clean":"2024-06-09T22:18:26.697538+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:18:26.697538+0000","last_undegraded":"2024-06-09T22:18:26.697538+0000","last_fullsized":"2024-06-09T22:18:26.697538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:19:19.787552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"99'6","reported_seq":44,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.342761+0000","last_change":"2024-06-09T22:16:43.199814+0000","last_active":"2024-06-09T22:19:25.342761+0000","last_peered":"2024-06-09T22:19:25.342761+0000","last_clean":"2024-06-09T22:19:25.342761+0000","last_became_active":"2024-06-09T22:16:43.199487+0000","last_became_peered":"2024-06-09T22:16:43.199487+0000","last_unstale":"2024-06-09T22:19:25.342761+0000","last_undegraded":"2024-06-09T22:19:25.342761+0000","last_fullsized":"2024-06-09T22:19:25.342761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:55.672050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696106+0000","last_change":"2024-06-09T22:16:53.336253+0000","last_active":"2024-06-09T22:18:26.696106+0000","last_peered":"2024-06-09T22:18:26.696106+0000","last_clean":"2024-06-09T22:18:26.696106+0000","last_became_active":"2024-06-09T22:16:53.335869+0000","last_became_peered":"2024-06-09T22:16:53.335869+0000","last_unstale":"2024-06-09T22:18:26.696106+0000","last_undegraded":"2024-06-09T22:18:26.696106+0000","last_fullsized":"2024-06-09T22:18:26.696106+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:28:12.894878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"97'12","reported_seq":58,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535881+0000","last_change":"2024-06-09T22:16:40.045365+0000","last_active":"2024-06-09T22:18:26.535881+0000","last_peered":"2024-06-09T22:18:26.535881+0000","last_clean":"2024-06-09T22:18:26.535881+0000","last_became_active":"2024-06-09T22:16:40.044989+0000","last_became_peered":"2024-06-09T22:16:40.044989+0000","last_unstale":"2024-06-09T22:18:26.535881+0000","last_undegraded":"2024-06-09T22:18:26.535881+0000","last_fullsized":"2024-06-09T22:18:26.535881+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:01:05.850170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"99'9","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.339362+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:19:25.339362+0000","last_peered":"2024-06-09T22:19:25.339362+0000","last_clean":"2024-06-09T22:19:25.339362+0000","last_became_active":"2024-06-09T22:16:43.203999+0000","last_became_peered":"2024-06-09T22:16:43.203999+0000","last_unstale":"2024-06-09T22:19:25.339362+0000","last_undegraded":"2024-06-09T22:19:25.339362+0000","last_fullsized":"2024-06-09T22:19:25.339362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:37:28.362655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695532+0000","last_change":"2024-06-09T22:16:53.333565+0000","last_active":"2024-06-09T22:18:26.695532+0000","last_peered":"2024-06-09T22:18:26.695532+0000","last_clean":"2024-06-09T22:18:26.695532+0000","last_became_active":"2024-06-09T22:16:53.332921+0000","last_became_peered":"2024-06-09T22:16:53.332921+0000","last_unstale":"2024-06-09T22:18:26.695532+0000","last_undegraded":"2024-06-09T22:18:26.695532+0000","last_fullsized":"2024-06-09T22:18:26.695532+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:32.085617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"97'5","reported_seq":51,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535750+0000","last_change":"2024-06-09T22:16:40.039766+0000","last_active":"2024-06-09T22:18:26.535750+0000","last_peered":"2024-06-09T22:18:26.535750+0000","last_clean":"2024-06-09T22:18:26.535750+0000","last_became_active":"2024-06-09T22:16:40.039329+0000","last_became_peered":"2024-06-09T22:16:40.039329+0000","last_unstale":"2024-06-09T22:18:26.535750+0000","last_undegraded":"2024-06-09T22:18:26.535750+0000","last_fullsized":"2024-06-09T22:18:26.535750+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:53.673890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"99'15","reported_seq":52,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.344212+0000","last_change":"2024-06-09T22:16:43.201200+0000","last_active":"2024-06-09T22:19:25.344212+0000","last_peered":"2024-06-09T22:19:25.344212+0000","last_clean":"2024-06-09T22:19:25.344212+0000","last_became_active":"2024-06-09T22:16:43.200830+0000","last_became_peered":"2024-06-09T22:16:43.200830+0000","last_unstale":"2024-06-09T22:19:25.344212+0000","last_undegraded":"2024-06-09T22:19:25.344212+0000","last_fullsized":"2024-06-09T22:19:25.344212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:38:52.093173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537838+0000","last_change":"2024-06-09T22:16:53.339811+0000","last_active":"2024-06-09T22:18:26.537838+0000","last_peered":"2024-06-09T22:18:26.537838+0000","last_clean":"2024-06-09T22:18:26.537838+0000","last_became_active":"2024-06-09T22:16:53.339305+0000","last_became_peered":"2024-06-09T22:16:53.339305+0000","last_unstale":"2024-06-09T22:18:26.537838+0000","last_undegraded":"2024-06-09T22:18:26.537838+0000","last_fullsized":"2024-06-09T22:18:26.537838+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:08:25.800245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"99'7","reported_seq":159,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:33.592869+0000","last_change":"2024-06-09T22:16:40.044159+0000","last_active":"2024-06-09T22:19:33.592869+0000","last_peered":"2024-06-09T22:19:33.592869+0000","last_clean":"2024-06-09T22:19:33.592869+0000","last_became_active":"2024-06-09T22:16:40.043839+0000","last_became_peered":"2024-06-09T22:16:40.043839+0000","last_unstale":"2024-06-09T22:19:33.592869+0000","last_undegraded":"2024-06-09T22:19:33.592869+0000","last_fullsized":"2024-06-09T22:19:33.592869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:22:09.339219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1153,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"99'7","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.334318+0000","last_change":"2024-06-09T22:16:43.202015+0000","last_active":"2024-06-09T22:19:25.334318+0000","last_peered":"2024-06-09T22:19:25.334318+0000","last_clean":"2024-06-09T22:19:25.334318+0000","last_became_active":"2024-06-09T22:16:43.201689+0000","last_became_peered":"2024-06-09T22:16:43.201689+0000","last_unstale":"2024-06-09T22:19:25.334318+0000","last_undegraded":"2024-06-09T22:19:25.334318+0000","last_fullsized":"2024-06-09T22:19:25.334318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:35:12.180399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":36,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.699126+0000","last_change":"2024-06-09T22:18:26.698600+0000","last_active":"2024-06-09T22:18:26.699126+0000","last_peered":"2024-06-09T22:18:26.699126+0000","last_clean":"2024-06-09T22:18:26.699126+0000","last_became_active":"2024-06-09T22:18:26.698201+0000","last_became_peered":"2024-06-09T22:18:26.698201+0000","last_unstale":"2024-06-09T22:18:26.699126+0000","last_undegraded":"2024-06-09T22:18:26.699126+0000","last_fullsized":"2024-06-09T22:18:26.699126+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:55:13.513557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"97'7","reported_seq":55,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.697193+0000","last_change":"2024-06-09T22:17:26.698204+0000","last_active":"2024-06-09T22:18:26.697193+0000","last_peered":"2024-06-09T22:18:26.697193+0000","last_clean":"2024-06-09T22:18:26.697193+0000","last_became_active":"2024-06-09T22:17:26.697716+0000","last_became_peered":"2024-06-09T22:17:26.697716+0000","last_unstale":"2024-06-09T22:18:26.697193+0000","last_undegraded":"2024-06-09T22:18:26.697193+0000","last_fullsized":"2024-06-09T22:18:26.697193+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:36:37.724459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"99'8","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.342285+0000","last_change":"2024-06-09T22:16:43.198917+0000","last_active":"2024-06-09T22:19:25.342285+0000","last_peered":"2024-06-09T22:19:25.342285+0000","last_clean":"2024-06-09T22:19:25.342285+0000","last_became_active":"2024-06-09T22:16:43.198567+0000","last_became_peered":"2024-06-09T22:16:43.198567+0000","last_unstale":"2024-06-09T22:19:25.342285+0000","last_undegraded":"2024-06-09T22:19:25.342285+0000","last_fullsized":"2024-06-09T22:19:25.342285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:34.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696208+0000","last_change":"2024-06-09T22:16:53.342203+0000","last_active":"2024-06-09T22:18:26.696208+0000","last_peered":"2024-06-09T22:18:26.696208+0000","last_clean":"2024-06-09T22:18:26.696208+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:18:26.696208+0000","last_undegraded":"2024-06-09T22:18:26.696208+0000","last_fullsized":"2024-06-09T22:18:26.696208+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:41:41.491961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"97'3","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695926+0000","last_change":"2024-06-09T22:16:40.038684+0000","last_active":"2024-06-09T22:18:26.695926+0000","last_peered":"2024-06-09T22:18:26.695926+0000","last_clean":"2024-06-09T22:18:26.695926+0000","last_became_active":"2024-06-09T22:16:40.038386+0000","last_became_peered":"2024-06-09T22:16:40.038386+0000","last_unstale":"2024-06-09T22:18:26.695926+0000","last_undegraded":"2024-06-09T22:18:26.695926+0000","last_fullsized":"2024-06-09T22:18:26.695926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:59.188758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"99'8","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.330417+0000","last_change":"2024-06-09T22:16:43.198837+0000","last_active":"2024-06-09T22:19:25.330417+0000","last_peered":"2024-06-09T22:19:25.330417+0000","last_clean":"2024-06-09T22:19:25.330417+0000","last_became_active":"2024-06-09T22:16:43.198515+0000","last_became_peered":"2024-06-09T22:16:43.198515+0000","last_unstale":"2024-06-09T22:19:25.330417+0000","last_undegraded":"2024-06-09T22:19:25.330417+0000","last_fullsized":"2024-06-09T22:19:25.330417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:36:13.863202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.534258+0000","last_change":"2024-06-09T22:16:53.342688+0000","last_active":"2024-06-09T22:18:26.534258+0000","last_peered":"2024-06-09T22:18:26.534258+0000","last_clean":"2024-06-09T22:18:26.534258+0000","last_became_active":"2024-06-09T22:16:53.342314+0000","last_became_peered":"2024-06-09T22:16:53.342314+0000","last_unstale":"2024-06-09T22:18:26.534258+0000","last_undegraded":"2024-06-09T22:18:26.534258+0000","last_fullsized":"2024-06-09T22:18:26.534258+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:51:35.801587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"97'3","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695265+0000","last_change":"2024-06-09T22:16:40.041831+0000","last_active":"2024-06-09T22:18:26.695265+0000","last_peered":"2024-06-09T22:18:26.695265+0000","last_clean":"2024-06-09T22:18:26.695265+0000","last_became_active":"2024-06-09T22:16:40.041484+0000","last_became_peered":"2024-06-09T22:16:40.041484+0000","last_unstale":"2024-06-09T22:18:26.695265+0000","last_undegraded":"2024-06-09T22:18:26.695265+0000","last_fullsized":"2024-06-09T22:18:26.695265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:10:19.829357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"99'11","reported_seq":47,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.330343+0000","last_change":"2024-06-09T22:16:43.210769+0000","last_active":"2024-06-09T22:19:25.330343+0000","last_peered":"2024-06-09T22:19:25.330343+0000","last_clean":"2024-06-09T22:19:25.330343+0000","last_became_active":"2024-06-09T22:16:43.210429+0000","last_became_peered":"2024-06-09T22:16:43.210429+0000","last_unstale":"2024-06-09T22:19:25.330343+0000","last_undegraded":"2024-06-09T22:19:25.330343+0000","last_fullsized":"2024-06-09T22:19:25.330343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:12:28.788830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695595+0000","last_change":"2024-06-09T22:16:53.336536+0000","last_active":"2024-06-09T22:18:26.695595+0000","last_peered":"2024-06-09T22:18:26.695595+0000","last_clean":"2024-06-09T22:18:26.695595+0000","last_became_active":"2024-06-09T22:16:53.335178+0000","last_became_peered":"2024-06-09T22:16:53.335178+0000","last_unstale":"2024-06-09T22:18:26.695595+0000","last_undegraded":"2024-06-09T22:18:26.695595+0000","last_fullsized":"2024-06-09T22:18:26.695595+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:57:19.476483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"99'6","reported_seq":252,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:32.590973+0000","last_change":"2024-06-09T22:16:40.038484+0000","last_active":"2024-06-09T22:19:32.590973+0000","last_peered":"2024-06-09T22:19:32.590973+0000","last_clean":"2024-06-09T22:19:32.590973+0000","last_became_active":"2024-06-09T22:16:40.038084+0000","last_became_peered":"2024-06-09T22:16:40.038084+0000","last_unstale":"2024-06-09T22:19:32.590973+0000","last_undegraded":"2024-06-09T22:19:32.590973+0000","last_fullsized":"2024-06-09T22:19:32.590973+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:19.733107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":204,"num_read_kb":204,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"99'10","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.339912+0000","last_change":"2024-06-09T22:16:43.205137+0000","last_active":"2024-06-09T22:19:25.339912+0000","last_peered":"2024-06-09T22:19:25.339912+0000","last_clean":"2024-06-09T22:19:25.339912+0000","last_became_active":"2024-06-09T22:16:43.204499+0000","last_became_peered":"2024-06-09T22:16:43.204499+0000","last_unstale":"2024-06-09T22:19:25.339912+0000","last_undegraded":"2024-06-09T22:19:25.339912+0000","last_fullsized":"2024-06-09T22:19:25.339912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:15.718493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695908+0000","last_change":"2024-06-09T22:16:53.344260+0000","last_active":"2024-06-09T22:18:26.695908+0000","last_peered":"2024-06-09T22:18:26.695908+0000","last_clean":"2024-06-09T22:18:26.695908+0000","last_became_active":"2024-06-09T22:16:53.343840+0000","last_became_peered":"2024-06-09T22:16:53.343840+0000","last_unstale":"2024-06-09T22:18:26.695908+0000","last_undegraded":"2024-06-09T22:18:26.695908+0000","last_fullsized":"2024-06-09T22:18:26.695908+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:40:23.305509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"97'4","reported_seq":49,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.538210+0000","last_change":"2024-06-09T22:16:40.042660+0000","last_active":"2024-06-09T22:18:26.538210+0000","last_peered":"2024-06-09T22:18:26.538210+0000","last_clean":"2024-06-09T22:18:26.538210+0000","last_became_active":"2024-06-09T22:16:40.041353+0000","last_became_peered":"2024-06-09T22:16:40.041353+0000","last_unstale":"2024-06-09T22:18:26.538210+0000","last_undegraded":"2024-06-09T22:18:26.538210+0000","last_fullsized":"2024-06-09T22:18:26.538210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:34:59.064827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"99'9","reported_seq":47,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.339075+0000","last_change":"2024-06-09T22:16:43.205083+0000","last_active":"2024-06-09T22:19:25.339075+0000","last_peered":"2024-06-09T22:19:25.339075+0000","last_clean":"2024-06-09T22:19:25.339075+0000","last_became_active":"2024-06-09T22:16:43.204600+0000","last_became_peered":"2024-06-09T22:16:43.204600+0000","last_unstale":"2024-06-09T22:19:25.339075+0000","last_undegraded":"2024-06-09T22:19:25.339075+0000","last_fullsized":"2024-06-09T22:19:25.339075+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:05:12.555465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696388+0000","last_change":"2024-06-09T22:16:53.336169+0000","last_active":"2024-06-09T22:18:26.696388+0000","last_peered":"2024-06-09T22:18:26.696388+0000","last_clean":"2024-06-09T22:18:26.696388+0000","last_became_active":"2024-06-09T22:16:53.335657+0000","last_became_peered":"2024-06-09T22:16:53.335657+0000","last_unstale":"2024-06-09T22:18:26.696388+0000","last_undegraded":"2024-06-09T22:18:26.696388+0000","last_fullsized":"2024-06-09T22:18:26.696388+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:53:21.282767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"97'5","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695835+0000","last_change":"2024-06-09T22:16:40.070568+0000","last_active":"2024-06-09T22:18:26.695835+0000","last_peered":"2024-06-09T22:18:26.695835+0000","last_clean":"2024-06-09T22:18:26.695835+0000","last_became_active":"2024-06-09T22:16:40.070205+0000","last_became_peered":"2024-06-09T22:16:40.070205+0000","last_unstale":"2024-06-09T22:18:26.695835+0000","last_undegraded":"2024-06-09T22:18:26.695835+0000","last_fullsized":"2024-06-09T22:18:26.695835+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.449965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"99'11","reported_seq":47,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.334059+0000","last_change":"2024-06-09T22:16:43.205219+0000","last_active":"2024-06-09T22:19:25.334059+0000","last_peered":"2024-06-09T22:19:25.334059+0000","last_clean":"2024-06-09T22:19:25.334059+0000","last_became_active":"2024-06-09T22:16:43.203467+0000","last_became_peered":"2024-06-09T22:16:43.203467+0000","last_unstale":"2024-06-09T22:19:25.334059+0000","last_undegraded":"2024-06-09T22:19:25.334059+0000","last_fullsized":"2024-06-09T22:19:25.334059+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:22.784034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696569+0000","last_change":"2024-06-09T22:16:53.338990+0000","last_active":"2024-06-09T22:18:26.696569+0000","last_peered":"2024-06-09T22:18:26.696569+0000","last_clean":"2024-06-09T22:18:26.696569+0000","last_became_active":"2024-06-09T22:16:53.338568+0000","last_became_peered":"2024-06-09T22:16:53.338568+0000","last_unstale":"2024-06-09T22:18:26.696569+0000","last_undegraded":"2024-06-09T22:18:26.696569+0000","last_fullsized":"2024-06-09T22:18:26.696569+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:46:39.814043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"99'40","reported_seq":296,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:32.590722+0000","last_change":"2024-06-09T22:16:40.044404+0000","last_active":"2024-06-09T22:19:32.590722+0000","last_peered":"2024-06-09T22:19:32.590722+0000","last_clean":"2024-06-09T22:19:32.590722+0000","last_became_active":"2024-06-09T22:16:40.044044+0000","last_became_peered":"2024-06-09T22:16:40.044044+0000","last_unstale":"2024-06-09T22:19:32.590722+0000","last_undegraded":"2024-06-09T22:19:32.590722+0000","last_fullsized":"2024-06-09T22:19:32.590722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:50:27.038759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68136,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":214,"num_read_kb":174,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"99'6","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.333782+0000","last_change":"2024-06-09T22:16:43.206963+0000","last_active":"2024-06-09T22:19:25.333782+0000","last_peered":"2024-06-09T22:19:25.333782+0000","last_clean":"2024-06-09T22:19:25.333782+0000","last_became_active":"2024-06-09T22:16:43.206628+0000","last_became_peered":"2024-06-09T22:16:43.206628+0000","last_unstale":"2024-06-09T22:19:25.333782+0000","last_undegraded":"2024-06-09T22:19:25.333782+0000","last_fullsized":"2024-06-09T22:19:25.333782+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:33:39.912236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696283+0000","last_change":"2024-06-09T22:16:53.339332+0000","last_active":"2024-06-09T22:18:26.696283+0000","last_peered":"2024-06-09T22:18:26.696283+0000","last_clean":"2024-06-09T22:18:26.696283+0000","last_became_active":"2024-06-09T22:16:53.338819+0000","last_became_peered":"2024-06-09T22:16:53.338819+0000","last_unstale":"2024-06-09T22:18:26.696283+0000","last_undegraded":"2024-06-09T22:18:26.696283+0000","last_fullsized":"2024-06-09T22:18:26.696283+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:57:41.142142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"97'9","reported_seq":30,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696016+0000","last_change":"2024-06-09T22:17:26.478256+0000","last_active":"2024-06-09T22:18:26.696016+0000","last_peered":"2024-06-09T22:18:26.696016+0000","last_clean":"2024-06-09T22:18:26.696016+0000","last_became_active":"2024-06-09T22:17:26.472264+0000","last_became_peered":"2024-06-09T22:17:26.472264+0000","last_unstale":"2024-06-09T22:18:26.696016+0000","last_undegraded":"2024-06-09T22:18:26.696016+0000","last_fullsized":"2024-06-09T22:18:26.696016+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:14:19.919841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"99'9","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.336506+0000","last_change":"2024-06-09T22:16:43.204484+0000","last_active":"2024-06-09T22:19:25.336506+0000","last_peered":"2024-06-09T22:19:25.336506+0000","last_clean":"2024-06-09T22:19:25.336506+0000","last_became_active":"2024-06-09T22:16:43.204014+0000","last_became_peered":"2024-06-09T22:16:43.204014+0000","last_unstale":"2024-06-09T22:19:25.336506+0000","last_undegraded":"2024-06-09T22:19:25.336506+0000","last_fullsized":"2024-06-09T22:19:25.336506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:55:12.528231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":36,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695616+0000","last_change":"2024-06-09T22:17:26.475179+0000","last_active":"2024-06-09T22:18:26.695616+0000","last_peered":"2024-06-09T22:18:26.695616+0000","last_clean":"2024-06-09T22:18:26.695616+0000","last_became_active":"2024-06-09T22:17:26.474517+0000","last_became_peered":"2024-06-09T22:17:26.474517+0000","last_unstale":"2024-06-09T22:18:26.695616+0000","last_undegraded":"2024-06-09T22:18:26.695616+0000","last_fullsized":"2024-06-09T22:18:26.695616+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:26:02.387830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"97'5","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695661+0000","last_change":"2024-06-09T22:16:40.043066+0000","last_active":"2024-06-09T22:18:26.695661+0000","last_peered":"2024-06-09T22:18:26.695661+0000","last_clean":"2024-06-09T22:18:26.695661+0000","last_became_active":"2024-06-09T22:16:40.042619+0000","last_became_peered":"2024-06-09T22:16:40.042619+0000","last_unstale":"2024-06-09T22:18:26.695661+0000","last_undegraded":"2024-06-09T22:18:26.695661+0000","last_fullsized":"2024-06-09T22:18:26.695661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:29.689614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"99'15","reported_seq":60,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.334548+0000","last_change":"2024-06-09T22:17:26.475566+0000","last_active":"2024-06-09T22:19:25.334548+0000","last_peered":"2024-06-09T22:19:25.334548+0000","last_clean":"2024-06-09T22:19:25.334548+0000","last_became_active":"2024-06-09T22:17:26.475037+0000","last_became_peered":"2024-06-09T22:17:26.475037+0000","last_unstale":"2024-06-09T22:19:25.334548+0000","last_undegraded":"2024-06-09T22:19:25.334548+0000","last_fullsized":"2024-06-09T22:19:25.334548+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:33.075144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535654+0000","last_change":"2024-06-09T22:17:26.475111+0000","last_active":"2024-06-09T22:18:26.535654+0000","last_peered":"2024-06-09T22:18:26.535654+0000","last_clean":"2024-06-09T22:18:26.535654+0000","last_became_active":"2024-06-09T22:17:26.474563+0000","last_became_peered":"2024-06-09T22:17:26.474563+0000","last_unstale":"2024-06-09T22:18:26.535654+0000","last_undegraded":"2024-06-09T22:18:26.535654+0000","last_fullsized":"2024-06-09T22:18:26.535654+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:09:08.458761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"99'7","reported_seq":158,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:33.591802+0000","last_change":"2024-06-09T22:16:40.040538+0000","last_active":"2024-06-09T22:19:33.591802+0000","last_peered":"2024-06-09T22:19:33.591802+0000","last_clean":"2024-06-09T22:19:33.591802+0000","last_became_active":"2024-06-09T22:16:40.039896+0000","last_became_peered":"2024-06-09T22:16:40.039896+0000","last_unstale":"2024-06-09T22:19:33.591802+0000","last_undegraded":"2024-06-09T22:19:33.591802+0000","last_fullsized":"2024-06-09T22:19:33.591802+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:00:33.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1119,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"99'6","reported_seq":51,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.336900+0000","last_change":"2024-06-09T22:17:26.475568+0000","last_active":"2024-06-09T22:19:25.336900+0000","last_peered":"2024-06-09T22:19:25.336900+0000","last_clean":"2024-06-09T22:19:25.336900+0000","last_became_active":"2024-06-09T22:17:26.475163+0000","last_became_peered":"2024-06-09T22:17:26.475163+0000","last_unstale":"2024-06-09T22:19:25.336900+0000","last_undegraded":"2024-06-09T22:19:25.336900+0000","last_fullsized":"2024-06-09T22:19:25.336900+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:20.865164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696860+0000","last_change":"2024-06-09T22:16:53.338201+0000","last_active":"2024-06-09T22:18:26.696860+0000","last_peered":"2024-06-09T22:18:26.696860+0000","last_clean":"2024-06-09T22:18:26.696860+0000","last_became_active":"2024-06-09T22:16:53.337816+0000","last_became_peered":"2024-06-09T22:16:53.337816+0000","last_unstale":"2024-06-09T22:18:26.696860+0000","last_undegraded":"2024-06-09T22:18:26.696860+0000","last_fullsized":"2024-06-09T22:18:26.696860+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:35:51.861135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"97'17","reported_seq":63,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535122+0000","last_change":"2024-06-09T22:16:40.070770+0000","last_active":"2024-06-09T22:18:26.535122+0000","last_peered":"2024-06-09T22:18:26.535122+0000","last_clean":"2024-06-09T22:18:26.535122+0000","last_became_active":"2024-06-09T22:16:40.070398+0000","last_became_peered":"2024-06-09T22:16:40.070398+0000","last_unstale":"2024-06-09T22:18:26.535122+0000","last_undegraded":"2024-06-09T22:18:26.535122+0000","last_fullsized":"2024-06-09T22:18:26.535122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:04:58.273601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"99'6","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.341776+0000","last_change":"2024-06-09T22:16:43.199246+0000","last_active":"2024-06-09T22:19:25.341776+0000","last_peered":"2024-06-09T22:19:25.341776+0000","last_clean":"2024-06-09T22:19:25.341776+0000","last_became_active":"2024-06-09T22:16:43.198006+0000","last_became_peered":"2024-06-09T22:16:43.198006+0000","last_unstale":"2024-06-09T22:19:25.341776+0000","last_undegraded":"2024-06-09T22:19:25.341776+0000","last_fullsized":"2024-06-09T22:19:25.341776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:30:55.836628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696277+0000","last_change":"2024-06-09T22:16:53.342217+0000","last_active":"2024-06-09T22:18:26.696277+0000","last_peered":"2024-06-09T22:18:26.696277+0000","last_clean":"2024-06-09T22:18:26.696277+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:18:26.696277+0000","last_undegraded":"2024-06-09T22:18:26.696277+0000","last_fullsized":"2024-06-09T22:18:26.696277+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:25:28.635091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"97'2","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535854+0000","last_change":"2024-06-09T22:16:40.039841+0000","last_active":"2024-06-09T22:18:26.535854+0000","last_peered":"2024-06-09T22:18:26.535854+0000","last_clean":"2024-06-09T22:18:26.535854+0000","last_became_active":"2024-06-09T22:16:40.039416+0000","last_became_peered":"2024-06-09T22:16:40.039416+0000","last_unstale":"2024-06-09T22:18:26.535854+0000","last_undegraded":"2024-06-09T22:18:26.535854+0000","last_fullsized":"2024-06-09T22:18:26.535854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:27:36.213839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"99'5","reported_seq":42,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.329021+0000","last_change":"2024-06-09T22:16:43.214363+0000","last_active":"2024-06-09T22:19:25.329021+0000","last_peered":"2024-06-09T22:19:25.329021+0000","last_clean":"2024-06-09T22:19:25.329021+0000","last_became_active":"2024-06-09T22:16:43.214015+0000","last_became_peered":"2024-06-09T22:16:43.214015+0000","last_unstale":"2024-06-09T22:19:25.329021+0000","last_undegraded":"2024-06-09T22:19:25.329021+0000","last_fullsized":"2024-06-09T22:19:25.329021+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:13:06.924468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535334+0000","last_change":"2024-06-09T22:16:53.342151+0000","last_active":"2024-06-09T22:18:26.535334+0000","last_peered":"2024-06-09T22:18:26.535334+0000","last_clean":"2024-06-09T22:18:26.535334+0000","last_became_active":"2024-06-09T22:16:53.341743+0000","last_became_peered":"2024-06-09T22:16:53.341743+0000","last_unstale":"2024-06-09T22:18:26.535334+0000","last_undegraded":"2024-06-09T22:18:26.535334+0000","last_fullsized":"2024-06-09T22:18:26.535334+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:39:44.793301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"97'1","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.697415+0000","last_change":"2024-06-09T22:17:26.472397+0000","last_active":"2024-06-09T22:18:26.697415+0000","last_peered":"2024-06-09T22:18:26.697415+0000","last_clean":"2024-06-09T22:18:26.697415+0000","last_became_active":"2024-06-09T22:17:26.471950+0000","last_became_peered":"2024-06-09T22:17:26.471950+0000","last_unstale":"2024-06-09T22:18:26.697415+0000","last_undegraded":"2024-06-09T22:18:26.697415+0000","last_fullsized":"2024-06-09T22:18:26.697415+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:35:57.481510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"99'4","reported_seq":40,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.331770+0000","last_change":"2024-06-09T22:16:43.214976+0000","last_active":"2024-06-09T22:19:25.331770+0000","last_peered":"2024-06-09T22:19:25.331770+0000","last_clean":"2024-06-09T22:19:25.331770+0000","last_became_active":"2024-06-09T22:16:43.214592+0000","last_became_peered":"2024-06-09T22:16:43.214592+0000","last_unstale":"2024-06-09T22:19:25.331770+0000","last_undegraded":"2024-06-09T22:19:25.331770+0000","last_fullsized":"2024-06-09T22:19:25.331770+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:16:21.981144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696417+0000","last_change":"2024-06-09T22:16:53.342721+0000","last_active":"2024-06-09T22:18:26.696417+0000","last_peered":"2024-06-09T22:18:26.696417+0000","last_clean":"2024-06-09T22:18:26.696417+0000","last_became_active":"2024-06-09T22:16:53.342342+0000","last_became_peered":"2024-06-09T22:16:53.342342+0000","last_unstale":"2024-06-09T22:18:26.696417+0000","last_undegraded":"2024-06-09T22:18:26.696417+0000","last_fullsized":"2024-06-09T22:18:26.696417+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:22:07.469873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"97'5","reported_seq":154,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:32.589034+0000","last_change":"2024-06-09T22:16:40.041307+0000","last_active":"2024-06-09T22:19:32.589034+0000","last_peered":"2024-06-09T22:19:32.589034+0000","last_clean":"2024-06-09T22:19:32.589034+0000","last_became_active":"2024-06-09T22:16:40.040819+0000","last_became_peered":"2024-06-09T22:16:40.040819+0000","last_unstale":"2024-06-09T22:19:32.589034+0000","last_undegraded":"2024-06-09T22:19:32.589034+0000","last_fullsized":"2024-06-09T22:19:32.589034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:14:29.658293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":104,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"99'7","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.328428+0000","last_change":"2024-06-09T22:16:43.215719+0000","last_active":"2024-06-09T22:19:25.328428+0000","last_peered":"2024-06-09T22:19:25.328428+0000","last_clean":"2024-06-09T22:19:25.328428+0000","last_became_active":"2024-06-09T22:16:43.215356+0000","last_became_peered":"2024-06-09T22:16:43.215356+0000","last_unstale":"2024-06-09T22:19:25.328428+0000","last_undegraded":"2024-06-09T22:19:25.328428+0000","last_fullsized":"2024-06-09T22:19:25.328428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:15:07.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696648+0000","last_change":"2024-06-09T22:16:53.342948+0000","last_active":"2024-06-09T22:18:26.696648+0000","last_peered":"2024-06-09T22:18:26.696648+0000","last_clean":"2024-06-09T22:18:26.696648+0000","last_became_active":"2024-06-09T22:16:53.342352+0000","last_became_peered":"2024-06-09T22:16:53.342352+0000","last_unstale":"2024-06-09T22:18:26.696648+0000","last_undegraded":"2024-06-09T22:18:26.696648+0000","last_fullsized":"2024-06-09T22:18:26.696648+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:43:27.453789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"99'8","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.342038+0000","last_change":"2024-06-09T22:16:43.204897+0000","last_active":"2024-06-09T22:19:25.342038+0000","last_peered":"2024-06-09T22:19:25.342038+0000","last_clean":"2024-06-09T22:19:25.342038+0000","last_became_active":"2024-06-09T22:16:43.204470+0000","last_became_peered":"2024-06-09T22:16:43.204470+0000","last_unstale":"2024-06-09T22:19:25.342038+0000","last_undegraded":"2024-06-09T22:19:25.342038+0000","last_fullsized":"2024-06-09T22:19:25.342038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:56.681392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"99'6","reported_seq":151,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:28.584823+0000","last_change":"2024-06-09T22:16:40.070079+0000","last_active":"2024-06-09T22:19:28.584823+0000","last_peered":"2024-06-09T22:19:28.584823+0000","last_clean":"2024-06-09T22:19:28.584823+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:19:28.584823+0000","last_undegraded":"2024-06-09T22:19:28.584823+0000","last_fullsized":"2024-06-09T22:19:28.584823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:00:45.132139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":100,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"97'1","reported_seq":30,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537957+0000","last_change":"2024-06-09T22:16:53.338891+0000","last_active":"2024-06-09T22:18:26.537957+0000","last_peered":"2024-06-09T22:18:26.537957+0000","last_clean":"2024-06-09T22:18:26.537957+0000","last_became_active":"2024-06-09T22:16:53.338526+0000","last_became_peered":"2024-06-09T22:16:53.338526+0000","last_unstale":"2024-06-09T22:18:26.537957+0000","last_undegraded":"2024-06-09T22:18:26.537957+0000","last_fullsized":"2024-06-09T22:18:26.537957+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:34:37.004407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"99'8","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.332163+0000","last_change":"2024-06-09T22:16:43.199234+0000","last_active":"2024-06-09T22:19:25.332163+0000","last_peered":"2024-06-09T22:19:25.332163+0000","last_clean":"2024-06-09T22:19:25.332163+0000","last_became_active":"2024-06-09T22:16:43.192012+0000","last_became_peered":"2024-06-09T22:16:43.192012+0000","last_unstale":"2024-06-09T22:19:25.332163+0000","last_undegraded":"2024-06-09T22:19:25.332163+0000","last_fullsized":"2024-06-09T22:19:25.332163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:17.429236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"97'6","reported_seq":52,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696236+0000","last_change":"2024-06-09T22:16:40.041970+0000","last_active":"2024-06-09T22:18:26.696236+0000","last_peered":"2024-06-09T22:18:26.696236+0000","last_clean":"2024-06-09T22:18:26.696236+0000","last_became_active":"2024-06-09T22:16:40.040371+0000","last_became_peered":"2024-06-09T22:16:40.040371+0000","last_unstale":"2024-06-09T22:18:26.696236+0000","last_undegraded":"2024-06-09T22:18:26.696236+0000","last_fullsized":"2024-06-09T22:18:26.696236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:52:31.210423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":378,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537270+0000","last_change":"2024-06-09T22:15:15.367397+0000","last_active":"2024-06-09T22:18:26.537270+0000","last_peered":"2024-06-09T22:18:26.537270+0000","last_clean":"2024-06-09T22:18:26.537270+0000","last_became_active":"2024-06-09T22:15:15.366974+0000","last_became_peered":"2024-06-09T22:15:15.366974+0000","last_unstale":"2024-06-09T22:18:26.537270+0000","last_undegraded":"2024-06-09T22:18:26.537270+0000","last_fullsized":"2024-06-09T22:18:26.537270+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_clean_scrub_stamp":"2024-06-09T22:11:49.886056+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:24.018031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695822+0000","last_change":"2024-06-09T22:16:53.342249+0000","last_active":"2024-06-09T22:18:26.695822+0000","last_peered":"2024-06-09T22:18:26.695822+0000","last_clean":"2024-06-09T22:18:26.695822+0000","last_became_active":"2024-06-09T22:16:53.341779+0000","last_became_peered":"2024-06-09T22:16:53.341779+0000","last_unstale":"2024-06-09T22:18:26.695822+0000","last_undegraded":"2024-06-09T22:18:26.695822+0000","last_fullsized":"2024-06-09T22:18:26.695822+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:21:21.101578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"99'5","reported_seq":44,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.332653+0000","last_change":"2024-06-09T22:16:43.200778+0000","last_active":"2024-06-09T22:19:25.332653+0000","last_peered":"2024-06-09T22:19:25.332653+0000","last_clean":"2024-06-09T22:19:25.332653+0000","last_became_active":"2024-06-09T22:16:43.200490+0000","last_became_peered":"2024-06-09T22:16:43.200490+0000","last_unstale":"2024-06-09T22:19:25.332653+0000","last_undegraded":"2024-06-09T22:19:25.332653+0000","last_fullsized":"2024-06-09T22:19:25.332653+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:20:17.530204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"97'4","reported_seq":49,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537575+0000","last_change":"2024-06-09T22:16:40.041774+0000","last_active":"2024-06-09T22:18:26.537575+0000","last_peered":"2024-06-09T22:18:26.537575+0000","last_clean":"2024-06-09T22:18:26.537575+0000","last_became_active":"2024-06-09T22:16:40.041085+0000","last_became_peered":"2024-06-09T22:16:40.041085+0000","last_unstale":"2024-06-09T22:18:26.537575+0000","last_undegraded":"2024-06-09T22:18:26.537575+0000","last_fullsized":"2024-06-09T22:18:26.537575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:09:41.140083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696364+0000","last_change":"2024-06-09T22:16:53.336895+0000","last_active":"2024-06-09T22:18:26.696364+0000","last_peered":"2024-06-09T22:18:26.696364+0000","last_clean":"2024-06-09T22:18:26.696364+0000","last_became_active":"2024-06-09T22:16:53.336497+0000","last_became_peered":"2024-06-09T22:16:53.336497+0000","last_unstale":"2024-06-09T22:18:26.696364+0000","last_undegraded":"2024-06-09T22:18:26.696364+0000","last_fullsized":"2024-06-09T22:18:26.696364+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:29:52.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"97'4","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535749+0000","last_change":"2024-06-09T22:16:40.070909+0000","last_active":"2024-06-09T22:18:26.535749+0000","last_peered":"2024-06-09T22:18:26.535749+0000","last_clean":"2024-06-09T22:18:26.535749+0000","last_became_active":"2024-06-09T22:16:40.070556+0000","last_became_peered":"2024-06-09T22:16:40.070556+0000","last_unstale":"2024-06-09T22:18:26.535749+0000","last_undegraded":"2024-06-09T22:18:26.535749+0000","last_fullsized":"2024-06-09T22:18:26.535749+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:59:17.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"99'9","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.324693+0000","last_change":"2024-06-09T22:16:43.205951+0000","last_active":"2024-06-09T22:19:25.324693+0000","last_peered":"2024-06-09T22:19:25.324693+0000","last_clean":"2024-06-09T22:19:25.324693+0000","last_became_active":"2024-06-09T22:16:43.205594+0000","last_became_peered":"2024-06-09T22:16:43.205594+0000","last_unstale":"2024-06-09T22:19:25.324693+0000","last_undegraded":"2024-06-09T22:19:25.324693+0000","last_fullsized":"2024-06-09T22:19:25.324693+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:17.956406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695289+0000","last_change":"2024-06-09T22:16:53.343292+0000","last_active":"2024-06-09T22:18:26.695289+0000","last_peered":"2024-06-09T22:18:26.695289+0000","last_clean":"2024-06-09T22:18:26.695289+0000","last_became_active":"2024-06-09T22:16:53.342900+0000","last_became_peered":"2024-06-09T22:16:53.342900+0000","last_unstale":"2024-06-09T22:18:26.695289+0000","last_undegraded":"2024-06-09T22:18:26.695289+0000","last_fullsized":"2024-06-09T22:18:26.695289+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:08:07.328250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"99'16","reported_seq":59,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:51.368666+0000","last_change":"2024-06-09T22:16:40.039417+0000","last_active":"2024-06-09T22:18:51.368666+0000","last_peered":"2024-06-09T22:18:51.368666+0000","last_clean":"2024-06-09T22:18:51.368666+0000","last_became_active":"2024-06-09T22:16:40.039099+0000","last_became_peered":"2024-06-09T22:16:40.039099+0000","last_unstale":"2024-06-09T22:18:51.368666+0000","last_undegraded":"2024-06-09T22:18:51.368666+0000","last_fullsized":"2024-06-09T22:18:51.368666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:22:59.284442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"99'5","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.333403+0000","last_change":"2024-06-09T22:16:43.201285+0000","last_active":"2024-06-09T22:19:25.333403+0000","last_peered":"2024-06-09T22:19:25.333403+0000","last_clean":"2024-06-09T22:19:25.333403+0000","last_became_active":"2024-06-09T22:16:43.200993+0000","last_became_peered":"2024-06-09T22:16:43.200993+0000","last_unstale":"2024-06-09T22:19:25.333403+0000","last_undegraded":"2024-06-09T22:19:25.333403+0000","last_fullsized":"2024-06-09T22:19:25.333403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:15:26.491588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537462+0000","last_change":"2024-06-09T22:16:53.334088+0000","last_active":"2024-06-09T22:18:26.537462+0000","last_peered":"2024-06-09T22:18:26.537462+0000","last_clean":"2024-06-09T22:18:26.537462+0000","last_became_active":"2024-06-09T22:16:53.333597+0000","last_became_peered":"2024-06-09T22:16:53.333597+0000","last_unstale":"2024-06-09T22:18:26.537462+0000","last_undegraded":"2024-06-09T22:18:26.537462+0000","last_fullsized":"2024-06-09T22:18:26.537462+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:50:22.041816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"97'7","reported_seq":51,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696271+0000","last_change":"2024-06-09T22:16:40.364982+0000","last_active":"2024-06-09T22:18:26.696271+0000","last_peered":"2024-06-09T22:18:26.696271+0000","last_clean":"2024-06-09T22:18:26.696271+0000","last_became_active":"2024-06-09T22:16:40.364585+0000","last_became_peered":"2024-06-09T22:16:40.364585+0000","last_unstale":"2024-06-09T22:18:26.696271+0000","last_undegraded":"2024-06-09T22:18:26.696271+0000","last_fullsized":"2024-06-09T22:18:26.696271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:28.777950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"99'7","reported_seq":51,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.340802+0000","last_change":"2024-06-09T22:17:26.474491+0000","last_active":"2024-06-09T22:19:25.340802+0000","last_peered":"2024-06-09T22:19:25.340802+0000","last_clean":"2024-06-09T22:19:25.340802+0000","last_became_active":"2024-06-09T22:17:26.473975+0000","last_became_peered":"2024-06-09T22:17:26.473975+0000","last_unstale":"2024-06-09T22:19:25.340802+0000","last_undegraded":"2024-06-09T22:19:25.340802+0000","last_fullsized":"2024-06-09T22:19:25.340802+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:13:46.998595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696429+0000","last_change":"2024-06-09T22:16:53.342273+0000","last_active":"2024-06-09T22:18:26.696429+0000","last_peered":"2024-06-09T22:18:26.696429+0000","last_clean":"2024-06-09T22:18:26.696429+0000","last_became_active":"2024-06-09T22:16:53.341827+0000","last_became_peered":"2024-06-09T22:16:53.341827+0000","last_unstale":"2024-06-09T22:18:26.696429+0000","last_undegraded":"2024-06-09T22:18:26.696429+0000","last_fullsized":"2024-06-09T22:18:26.696429+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:23:26.099737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"97'2","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695616+0000","last_change":"2024-06-09T22:16:40.070498+0000","last_active":"2024-06-09T22:18:26.695616+0000","last_peered":"2024-06-09T22:18:26.695616+0000","last_clean":"2024-06-09T22:18:26.695616+0000","last_became_active":"2024-06-09T22:16:40.070144+0000","last_became_peered":"2024-06-09T22:16:40.070144+0000","last_unstale":"2024-06-09T22:18:26.695616+0000","last_undegraded":"2024-06-09T22:18:26.695616+0000","last_fullsized":"2024-06-09T22:18:26.695616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:52:22.214832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"99'2","reported_seq":41,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.312986+0000","last_change":"2024-06-09T22:16:43.210448+0000","last_active":"2024-06-09T22:19:25.312986+0000","last_peered":"2024-06-09T22:19:25.312986+0000","last_clean":"2024-06-09T22:19:25.312986+0000","last_became_active":"2024-06-09T22:16:43.209980+0000","last_became_peered":"2024-06-09T22:16:43.209980+0000","last_unstale":"2024-06-09T22:19:25.312986+0000","last_undegraded":"2024-06-09T22:19:25.312986+0000","last_fullsized":"2024-06-09T22:19:25.312986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:58:15.691207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.697676+0000","last_change":"2024-06-09T22:16:53.340644+0000","last_active":"2024-06-09T22:18:26.697676+0000","last_peered":"2024-06-09T22:18:26.697676+0000","last_clean":"2024-06-09T22:18:26.697676+0000","last_became_active":"2024-06-09T22:16:53.339771+0000","last_became_peered":"2024-06-09T22:16:53.339771+0000","last_unstale":"2024-06-09T22:18:26.697676+0000","last_undegraded":"2024-06-09T22:18:26.697676+0000","last_fullsized":"2024-06-09T22:18:26.697676+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:45:23.281387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"97'7","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695861+0000","last_change":"2024-06-09T22:16:40.040461+0000","last_active":"2024-06-09T22:18:26.695861+0000","last_peered":"2024-06-09T22:18:26.695861+0000","last_clean":"2024-06-09T22:18:26.695861+0000","last_became_active":"2024-06-09T22:16:40.040168+0000","last_became_peered":"2024-06-09T22:16:40.040168+0000","last_unstale":"2024-06-09T22:18:26.695861+0000","last_undegraded":"2024-06-09T22:18:26.695861+0000","last_fullsized":"2024-06-09T22:18:26.695861+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:28.936631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"99'9","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.331262+0000","last_change":"2024-06-09T22:16:43.208779+0000","last_active":"2024-06-09T22:19:25.331262+0000","last_peered":"2024-06-09T22:19:25.331262+0000","last_clean":"2024-06-09T22:19:25.331262+0000","last_became_active":"2024-06-09T22:16:43.208136+0000","last_became_peered":"2024-06-09T22:16:43.208136+0000","last_unstale":"2024-06-09T22:19:25.331262+0000","last_undegraded":"2024-06-09T22:19:25.331262+0000","last_fullsized":"2024-06-09T22:19:25.331262+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:22:39.721069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"97'1","reported_seq":30,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695777+0000","last_change":"2024-06-09T22:16:53.331241+0000","last_active":"2024-06-09T22:18:26.695777+0000","last_peered":"2024-06-09T22:18:26.695777+0000","last_clean":"2024-06-09T22:18:26.695777+0000","last_became_active":"2024-06-09T22:16:53.329589+0000","last_became_peered":"2024-06-09T22:16:53.329589+0000","last_unstale":"2024-06-09T22:18:26.695777+0000","last_undegraded":"2024-06-09T22:18:26.695777+0000","last_fullsized":"2024-06-09T22:18:26.695777+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:56:21.319711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"97'3","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.537743+0000","last_change":"2024-06-09T22:16:40.042491+0000","last_active":"2024-06-09T22:18:26.537743+0000","last_peered":"2024-06-09T22:18:26.537743+0000","last_clean":"2024-06-09T22:18:26.537743+0000","last_became_active":"2024-06-09T22:16:40.041846+0000","last_became_peered":"2024-06-09T22:16:40.041846+0000","last_unstale":"2024-06-09T22:18:26.537743+0000","last_undegraded":"2024-06-09T22:18:26.537743+0000","last_fullsized":"2024-06-09T22:18:26.537743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:59.144003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"99'8","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.341076+0000","last_change":"2024-06-09T22:16:43.209195+0000","last_active":"2024-06-09T22:19:25.341076+0000","last_peered":"2024-06-09T22:19:25.341076+0000","last_clean":"2024-06-09T22:19:25.341076+0000","last_became_active":"2024-06-09T22:16:43.208797+0000","last_became_peered":"2024-06-09T22:16:43.208797+0000","last_unstale":"2024-06-09T22:19:25.341076+0000","last_undegraded":"2024-06-09T22:19:25.341076+0000","last_fullsized":"2024-06-09T22:19:25.341076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:05:31.689900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696403+0000","last_change":"2024-06-09T22:16:53.342173+0000","last_active":"2024-06-09T22:18:26.696403+0000","last_peered":"2024-06-09T22:18:26.696403+0000","last_clean":"2024-06-09T22:18:26.696403+0000","last_became_active":"2024-06-09T22:16:53.341712+0000","last_became_peered":"2024-06-09T22:16:53.341712+0000","last_unstale":"2024-06-09T22:18:26.696403+0000","last_undegraded":"2024-06-09T22:18:26.696403+0000","last_fullsized":"2024-06-09T22:18:26.696403+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:53:12.574764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"97'4","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.535214+0000","last_change":"2024-06-09T22:16:40.044238+0000","last_active":"2024-06-09T22:18:26.535214+0000","last_peered":"2024-06-09T22:18:26.535214+0000","last_clean":"2024-06-09T22:18:26.535214+0000","last_became_active":"2024-06-09T22:16:40.043857+0000","last_became_peered":"2024-06-09T22:16:40.043857+0000","last_unstale":"2024-06-09T22:18:26.535214+0000","last_undegraded":"2024-06-09T22:18:26.535214+0000","last_fullsized":"2024-06-09T22:18:26.535214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:44:45.743307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"99'5","reported_seq":44,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.341149+0000","last_change":"2024-06-09T22:16:43.202363+0000","last_active":"2024-06-09T22:19:25.341149+0000","last_peered":"2024-06-09T22:19:25.341149+0000","last_clean":"2024-06-09T22:19:25.341149+0000","last_became_active":"2024-06-09T22:16:43.202029+0000","last_became_peered":"2024-06-09T22:16:43.202029+0000","last_unstale":"2024-06-09T22:19:25.341149+0000","last_undegraded":"2024-06-09T22:19:25.341149+0000","last_fullsized":"2024-06-09T22:19:25.341149+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:11:19.513055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696098+0000","last_change":"2024-06-09T22:16:53.333837+0000","last_active":"2024-06-09T22:18:26.696098+0000","last_peered":"2024-06-09T22:18:26.696098+0000","last_clean":"2024-06-09T22:18:26.696098+0000","last_became_active":"2024-06-09T22:16:53.333362+0000","last_became_peered":"2024-06-09T22:16:53.333362+0000","last_unstale":"2024-06-09T22:18:26.696098+0000","last_undegraded":"2024-06-09T22:18:26.696098+0000","last_fullsized":"2024-06-09T22:18:26.696098+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:47:04.463040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"99'7","reported_seq":157,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:32.591615+0000","last_change":"2024-06-09T22:16:40.364834+0000","last_active":"2024-06-09T22:19:32.591615+0000","last_peered":"2024-06-09T22:19:32.591615+0000","last_clean":"2024-06-09T22:19:32.591615+0000","last_became_active":"2024-06-09T22:16:40.364433+0000","last_became_peered":"2024-06-09T22:16:40.364433+0000","last_unstale":"2024-06-09T22:19:32.591615+0000","last_undegraded":"2024-06-09T22:19:32.591615+0000","last_fullsized":"2024-06-09T22:19:32.591615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:43:01.967911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":107,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"99'7","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.335979+0000","last_change":"2024-06-09T22:16:43.201183+0000","last_active":"2024-06-09T22:19:25.335979+0000","last_peered":"2024-06-09T22:19:25.335979+0000","last_clean":"2024-06-09T22:19:25.335979+0000","last_became_active":"2024-06-09T22:16:43.200774+0000","last_became_peered":"2024-06-09T22:16:43.200774+0000","last_unstale":"2024-06-09T22:19:25.335979+0000","last_undegraded":"2024-06-09T22:19:25.335979+0000","last_fullsized":"2024-06-09T22:19:25.335979+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:07:55.772066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":36,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.699557+0000","last_change":"2024-06-09T22:18:26.699190+0000","last_active":"2024-06-09T22:18:26.699557+0000","last_peered":"2024-06-09T22:18:26.699557+0000","last_clean":"2024-06-09T22:18:26.699557+0000","last_became_active":"2024-06-09T22:18:26.698614+0000","last_became_peered":"2024-06-09T22:18:26.698614+0000","last_unstale":"2024-06-09T22:18:26.699557+0000","last_undegraded":"2024-06-09T22:18:26.699557+0000","last_fullsized":"2024-06-09T22:18:26.699557+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:45.343605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,1,4],"acting":[0,10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"97'3","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696079+0000","last_change":"2024-06-09T22:16:40.040582+0000","last_active":"2024-06-09T22:18:26.696079+0000","last_peered":"2024-06-09T22:18:26.696079+0000","last_clean":"2024-06-09T22:18:26.696079+0000","last_became_active":"2024-06-09T22:16:40.040152+0000","last_became_peered":"2024-06-09T22:16:40.040152+0000","last_unstale":"2024-06-09T22:18:26.696079+0000","last_undegraded":"2024-06-09T22:18:26.696079+0000","last_fullsized":"2024-06-09T22:18:26.696079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:50:00.878096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"99'6","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.340841+0000","last_change":"2024-06-09T22:16:43.206211+0000","last_active":"2024-06-09T22:19:25.340841+0000","last_peered":"2024-06-09T22:19:25.340841+0000","last_clean":"2024-06-09T22:19:25.340841+0000","last_became_active":"2024-06-09T22:16:43.205814+0000","last_became_peered":"2024-06-09T22:16:43.205814+0000","last_unstale":"2024-06-09T22:19:25.340841+0000","last_undegraded":"2024-06-09T22:19:25.340841+0000","last_fullsized":"2024-06-09T22:19:25.340841+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:20:19.058475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.695056+0000","last_change":"2024-06-09T22:16:53.337695+0000","last_active":"2024-06-09T22:18:26.695056+0000","last_peered":"2024-06-09T22:18:26.695056+0000","last_clean":"2024-06-09T22:18:26.695056+0000","last_became_active":"2024-06-09T22:16:53.334872+0000","last_became_peered":"2024-06-09T22:16:53.334872+0000","last_unstale":"2024-06-09T22:18:26.695056+0000","last_undegraded":"2024-06-09T22:18:26.695056+0000","last_fullsized":"2024-06-09T22:18:26.695056+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:50:36.617736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"97'5","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:18:26.696936+0000","last_change":"2024-06-09T22:16:40.036603+0000","last_active":"2024-06-09T22:18:26.696936+0000","last_peered":"2024-06-09T22:18:26.696936+0000","last_clean":"2024-06-09T22:18:26.696936+0000","last_became_active":"2024-06-09T22:16:40.036274+0000","last_became_peered":"2024-06-09T22:16:40.036274+0000","last_unstale":"2024-06-09T22:18:26.696936+0000","last_undegraded":"2024-06-09T22:18:26.696936+0000","last_fullsized":"2024-06-09T22:18:26.696936+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:31.652897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"99'9","reported_seq":48,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.336962+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:19:25.336962+0000","last_peered":"2024-06-09T22:19:25.336962+0000","last_clean":"2024-06-09T22:19:25.336962+0000","last_became_active":"2024-06-09T22:16:43.204197+0000","last_became_peered":"2024-06-09T22:16:43.204197+0000","last_unstale":"2024-06-09T22:19:25.336962+0000","last_undegraded":"2024-06-09T22:19:25.336962+0000","last_fullsized":"2024-06-09T22:19:25.336962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:38:49.850120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"99'9","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.337895+0000","last_change":"2024-06-09T22:16:43.208477+0000","last_active":"2024-06-09T22:19:25.337895+0000","last_peered":"2024-06-09T22:19:25.337895+0000","last_clean":"2024-06-09T22:19:25.337895+0000","last_became_active":"2024-06-09T22:16:43.206018+0000","last_became_peered":"2024-06-09T22:16:43.206018+0000","last_unstale":"2024-06-09T22:19:25.337895+0000","last_undegraded":"2024-06-09T22:19:25.337895+0000","last_fullsized":"2024-06-09T22:19:25.337895+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:19:49.870825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"99'12","reported_seq":49,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.343127+0000","last_change":"2024-06-09T22:16:43.206809+0000","last_active":"2024-06-09T22:19:25.343127+0000","last_peered":"2024-06-09T22:19:25.343127+0000","last_clean":"2024-06-09T22:19:25.343127+0000","last_became_active":"2024-06-09T22:16:43.206175+0000","last_became_peered":"2024-06-09T22:16:43.206175+0000","last_unstale":"2024-06-09T22:19:25.343127+0000","last_undegraded":"2024-06-09T22:19:25.343127+0000","last_fullsized":"2024-06-09T22:19:25.343127+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:39:22.502266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"99'8","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.340310+0000","last_change":"2024-06-09T22:16:43.202163+0000","last_active":"2024-06-09T22:19:25.340310+0000","last_peered":"2024-06-09T22:19:25.340310+0000","last_clean":"2024-06-09T22:19:25.340310+0000","last_became_active":"2024-06-09T22:16:43.201839+0000","last_became_peered":"2024-06-09T22:16:43.201839+0000","last_unstale":"2024-06-09T22:19:25.340310+0000","last_undegraded":"2024-06-09T22:19:25.340310+0000","last_fullsized":"2024-06-09T22:19:25.340310+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:47:46.723289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"99'7","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.339893+0000","last_change":"2024-06-09T22:16:43.202734+0000","last_active":"2024-06-09T22:19:25.339893+0000","last_peered":"2024-06-09T22:19:25.339893+0000","last_clean":"2024-06-09T22:19:25.339893+0000","last_became_active":"2024-06-09T22:16:43.201638+0000","last_became_peered":"2024-06-09T22:16:43.201638+0000","last_unstale":"2024-06-09T22:19:25.339893+0000","last_undegraded":"2024-06-09T22:19:25.339893+0000","last_fullsized":"2024-06-09T22:19:25.339893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:16.360985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"99'4","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.339155+0000","last_change":"2024-06-09T22:16:43.199113+0000","last_active":"2024-06-09T22:19:25.339155+0000","last_peered":"2024-06-09T22:19:25.339155+0000","last_clean":"2024-06-09T22:19:25.339155+0000","last_became_active":"2024-06-09T22:16:43.198804+0000","last_became_peered":"2024-06-09T22:16:43.198804+0000","last_unstale":"2024-06-09T22:19:25.339155+0000","last_undegraded":"2024-06-09T22:19:25.339155+0000","last_fullsized":"2024-06-09T22:19:25.339155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:18:08.999477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"99'6","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.326942+0000","last_change":"2024-06-09T22:16:43.204445+0000","last_active":"2024-06-09T22:19:25.326942+0000","last_peered":"2024-06-09T22:19:25.326942+0000","last_clean":"2024-06-09T22:19:25.326942+0000","last_became_active":"2024-06-09T22:16:43.203081+0000","last_became_peered":"2024-06-09T22:16:43.203081+0000","last_unstale":"2024-06-09T22:19:25.326942+0000","last_undegraded":"2024-06-09T22:19:25.326942+0000","last_fullsized":"2024-06-09T22:19:25.326942+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:39.303813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"99'15","reported_seq":52,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.333117+0000","last_change":"2024-06-09T22:16:43.211402+0000","last_active":"2024-06-09T22:19:25.333117+0000","last_peered":"2024-06-09T22:19:25.333117+0000","last_clean":"2024-06-09T22:19:25.333117+0000","last_became_active":"2024-06-09T22:16:43.210871+0000","last_became_peered":"2024-06-09T22:16:43.210871+0000","last_unstale":"2024-06-09T22:19:25.333117+0000","last_undegraded":"2024-06-09T22:19:25.333117+0000","last_fullsized":"2024-06-09T22:19:25.333117+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:07:21.467617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"99'2","reported_seq":40,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.328895+0000","last_change":"2024-06-09T22:16:43.201159+0000","last_active":"2024-06-09T22:19:25.328895+0000","last_peered":"2024-06-09T22:19:25.328895+0000","last_clean":"2024-06-09T22:19:25.328895+0000","last_became_active":"2024-06-09T22:16:43.200850+0000","last_became_peered":"2024-06-09T22:16:43.200850+0000","last_unstale":"2024-06-09T22:19:25.328895+0000","last_undegraded":"2024-06-09T22:19:25.328895+0000","last_fullsized":"2024-06-09T22:19:25.328895+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:20.515309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"99'6","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.336981+0000","last_change":"2024-06-09T22:16:43.196830+0000","last_active":"2024-06-09T22:19:25.336981+0000","last_peered":"2024-06-09T22:19:25.336981+0000","last_clean":"2024-06-09T22:19:25.336981+0000","last_became_active":"2024-06-09T22:16:43.195682+0000","last_became_peered":"2024-06-09T22:16:43.195682+0000","last_unstale":"2024-06-09T22:19:25.336981+0000","last_undegraded":"2024-06-09T22:19:25.336981+0000","last_fullsized":"2024-06-09T22:19:25.336981+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:34:18.770235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"99'8","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.331017+0000","last_change":"2024-06-09T22:16:43.201959+0000","last_active":"2024-06-09T22:19:25.331017+0000","last_peered":"2024-06-09T22:19:25.331017+0000","last_clean":"2024-06-09T22:19:25.331017+0000","last_became_active":"2024-06-09T22:16:43.201669+0000","last_became_peered":"2024-06-09T22:16:43.201669+0000","last_unstale":"2024-06-09T22:19:25.331017+0000","last_undegraded":"2024-06-09T22:19:25.331017+0000","last_fullsized":"2024-06-09T22:19:25.331017+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:48:57.348041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"99'10","reported_seq":46,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.341873+0000","last_change":"2024-06-09T22:16:43.201823+0000","last_active":"2024-06-09T22:19:25.341873+0000","last_peered":"2024-06-09T22:19:25.341873+0000","last_clean":"2024-06-09T22:19:25.341873+0000","last_became_active":"2024-06-09T22:16:43.201434+0000","last_became_peered":"2024-06-09T22:16:43.201434+0000","last_unstale":"2024-06-09T22:19:25.341873+0000","last_undegraded":"2024-06-09T22:19:25.341873+0000","last_fullsized":"2024-06-09T22:19:25.341873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:56:13.540895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"99'11","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.340132+0000","last_change":"2024-06-09T22:16:43.197734+0000","last_active":"2024-06-09T22:19:25.340132+0000","last_peered":"2024-06-09T22:19:25.340132+0000","last_clean":"2024-06-09T22:19:25.340132+0000","last_became_active":"2024-06-09T22:16:43.197442+0000","last_became_peered":"2024-06-09T22:16:43.197442+0000","last_unstale":"2024-06-09T22:19:25.340132+0000","last_undegraded":"2024-06-09T22:19:25.340132+0000","last_fullsized":"2024-06-09T22:19:25.340132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:35:40.720718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"99'8","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-06-09T22:19:25.335947+0000","last_change":"2024-06-09T22:16:43.210375+0000","last_active":"2024-06-09T22:19:25.335947+0000","last_peered":"2024-06-09T22:19:25.335947+0000","last_clean":"2024-06-09T22:19:25.335947+0000","last_became_active":"2024-06-09T22:16:43.209647+0000","last_became_peered":"2024-06-09T22:16:43.209647+0000","last_unstale":"2024-06-09T22:19:25.335947+0000","last_undegraded":"2024-06-09T22:19:25.335947+0000","last_fullsized":"2024-06-09T22:19:25.335947+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:11:57.593066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247891,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":76151,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1067,"num_read_kb":613,"num_write":262,"num_write_kb":158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":228453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383731,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":570974,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645368,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32404,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712748,"statfs":{"total":95995035648,"available":95961853952,"internally_reserved":0,"allocated":1257472,"data_stored":1225858,"data_compressed":68186,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808893,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":574401,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070530,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1122304,"data_stored":570683,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332167,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1175552,"data_stored":1167851,"data_compressed":68186,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495692,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1101824,"data_stored":565809,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36484,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708668,"statfs":{"total":95995035648,"available":95957676032,"internally_reserved":0,"allocated":1175552,"data_stored":634975,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1126400,"data_stored":567796,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1187840,"data_stored":636189,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182491,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":565517,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920855,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1110016,"data_stored":570414,"data_compressed":32305,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757330,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32332,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712820,"statfs":{"total":95995035648,"available":95961927680,"internally_reserved":0,"allocated":1183744,"data_stored":1162902,"data_compressed":68186,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34243,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34243,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":70403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-09T22:19:36.395 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-09T22:19:36.395 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-09T22:19:36.395 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-09T22:19:36.395 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-09T22:19:36.641 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:36.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:36 smithi153 ceph-mon[36599]: pgmap v394: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 11 op/s 2024-06-09T22:19:36.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:36 smithi104 ceph-mon[32389]: pgmap v394: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 11 op/s 2024-06-09T22:19:36.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:36 smithi194 ceph-mon[36856]: pgmap v394: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 11 op/s 2024-06-09T22:19:37.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:37 smithi153 ceph-mon[36599]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:19:37.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:37 smithi104 ceph-mon[32389]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:19:37.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:37 smithi194 ceph-mon[36856]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:19:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.543+0000 7feb6be67700 1 -- 172.21.15.104:0/2401894935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb64101fd0 msgr2=0x7feb64105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.543+0000 7feb6be67700 1 --2- 172.21.15.104:0/2401894935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb64101fd0 0x7feb64105fd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7feb540099c0 tx=0x7feb54009cd0 comp rx=0 tx=0).stop 2024-06-09T22:19:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.543+0000 7feb6be67700 1 -- 172.21.15.104:0/2401894935 shutdown_connections 2024-06-09T22:19:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.543+0000 7feb6be67700 1 --2- 172.21.15.104:0/2401894935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb641066b0 0x7feb6410e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.543+0000 7feb6be67700 1 --2- 172.21.15.104:0/2401894935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb64101620 0x7feb64101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.543+0000 7feb6be67700 1 --2- 172.21.15.104:0/2401894935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb64101fd0 0x7feb64105fd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.543+0000 7feb6be67700 1 -- 172.21.15.104:0/2401894935 >> 172.21.15.104:0/2401894935 conn(0x7feb64078360 msgr2=0x7feb64078770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.543+0000 7feb6be67700 1 -- 172.21.15.104:0/2401894935 shutdown_connections 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.543+0000 7feb6be67700 1 -- 172.21.15.104:0/2401894935 wait complete. 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb6be67700 1 Processor -- start 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb6be67700 1 -- start start 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb6be67700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb64101620 0x7feb6419e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb6be67700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb64101fd0 0x7feb6419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb6be67700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb641066b0 0x7feb64198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb6be67700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feb641116c0 con 0x7feb64101620 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb6be67700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb64111540 con 0x7feb64101fd0 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb6be67700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feb64111840 con 0x7feb641066b0 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb69c03700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb64101620 0x7feb6419e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb69c03700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb64101620 0x7feb6419e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55360/0 (socket says 172.21.15.104:55360) 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb69c03700 1 -- 172.21.15.104:0/3662880103 learned_addr learned my addr 172.21.15.104:0/3662880103 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb69402700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb64101fd0 0x7feb6419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.544+0000 7feb6a404700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb641066b0 0x7feb64198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.545+0000 7feb69c03700 1 -- 172.21.15.104:0/3662880103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb641066b0 msgr2=0x7feb64198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.545+0000 7feb69c03700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb641066b0 0x7feb64198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.545+0000 7feb69c03700 1 -- 172.21.15.104:0/3662880103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb64101fd0 msgr2=0x7feb6419ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.545+0000 7feb69c03700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb64101fd0 0x7feb6419ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.545+0000 7feb69c03700 1 -- 172.21.15.104:0/3662880103 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb64199200 con 0x7feb64101620 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.545+0000 7feb6a404700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb641066b0 0x7feb64198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.545+0000 7feb69c03700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb64101620 0x7feb6419e8e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7feb6000d980 tx=0x7feb6000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.545+0000 7feb5affd700 1 -- 172.21.15.104:0/3662880103 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb60016070 con 0x7feb64101620 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.545+0000 7feb5affd700 1 -- 172.21.15.104:0/3662880103 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb60004d10 con 0x7feb64101620 2024-06-09T22:19:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.545+0000 7feb6be67700 1 -- 172.21.15.104:0/3662880103 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb641994f0 con 0x7feb64101620 2024-06-09T22:19:38.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.545+0000 7feb5affd700 1 -- 172.21.15.104:0/3662880103 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb600055f0 con 0x7feb64101620 2024-06-09T22:19:38.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.545+0000 7feb6be67700 1 -- 172.21.15.104:0/3662880103 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb641a57b0 con 0x7feb64101620 2024-06-09T22:19:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.546+0000 7feb6be67700 1 -- 172.21.15.104:0/3662880103 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb64069180 con 0x7feb64101620 2024-06-09T22:19:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.547+0000 7feb5affd700 1 -- 172.21.15.104:0/3662880103 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7feb60004750 con 0x7feb64101620 2024-06-09T22:19:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.548+0000 7feb5affd700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7feb500a2c20 0x7feb500a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.548+0000 7feb69402700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7feb500a2c20 0x7feb500a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.548+0000 7feb5affd700 1 -- 172.21.15.104:0/3662880103 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7feb600cf200 con 0x7feb64101620 2024-06-09T22:19:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.548+0000 7feb69402700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7feb500a2c20 0x7feb500a50e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feb54000c00 tx=0x7feb54005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:38.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.552+0000 7feb5affd700 1 -- 172.21.15.104:0/3662880103 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7feb6001f020 con 0x7feb64101620 2024-06-09T22:19:38.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:38 smithi153 ceph-mon[36599]: pgmap v395: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 10 op/s 2024-06-09T22:19:38.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:38 smithi104 ceph-mon[32389]: pgmap v395: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 10 op/s 2024-06-09T22:19:38.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.793+0000 7feb6be67700 1 -- 172.21.15.104:0/3662880103 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7feb6402dca0 con 0x7feb64101620 2024-06-09T22:19:38.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.793+0000 7feb5affd700 1 -- 172.21.15.104:0/3662880103 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7feb6009bdd0 con 0x7feb64101620 2024-06-09T22:19:38.791 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:19:38.791 INFO:teuthology.orchestra.run.smithi104.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-09T22:19:38.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.795+0000 7feb6be67700 1 -- 172.21.15.104:0/3662880103 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7feb500a2c20 msgr2=0x7feb500a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:38.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.795+0000 7feb6be67700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7feb500a2c20 0x7feb500a50e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feb54000c00 tx=0x7feb54005fb0 comp rx=0 tx=0).stop 2024-06-09T22:19:38.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.795+0000 7feb6be67700 1 -- 172.21.15.104:0/3662880103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb64101620 msgr2=0x7feb6419e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:38.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.795+0000 7feb6be67700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb64101620 0x7feb6419e8e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7feb6000d980 tx=0x7feb6000dc90 comp rx=0 tx=0).stop 2024-06-09T22:19:38.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.795+0000 7feb6be67700 1 -- 172.21.15.104:0/3662880103 shutdown_connections 2024-06-09T22:19:38.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.795+0000 7feb6be67700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7feb500a2c20 0x7feb500a50e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:38.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.795+0000 7feb6be67700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb64101620 0x7feb6419e8e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:38.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.795+0000 7feb6be67700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb641066b0 0x7feb64198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:38.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.795+0000 7feb6be67700 1 --2- 172.21.15.104:0/3662880103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb64101fd0 0x7feb6419ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:38.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.795+0000 7feb6be67700 1 -- 172.21.15.104:0/3662880103 >> 172.21.15.104:0/3662880103 conn(0x7feb64078360 msgr2=0x7feb6406aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:38.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.796+0000 7feb6be67700 1 -- 172.21.15.104:0/3662880103 shutdown_connections 2024-06-09T22:19:38.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:38.796+0000 7feb6be67700 1 -- 172.21.15.104:0/3662880103 wait complete. 2024-06-09T22:19:38.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:38 smithi194 ceph-mon[36856]: pgmap v395: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 10 op/s 2024-06-09T22:19:39.558 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-06-09T22:19:39.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:19:39.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:39 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3662880103' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-09T22:19:39.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:39 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3662880103' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-09T22:19:39.803 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:39.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:39 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3662880103' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-09T22:19:40.505 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:40 smithi104 ceph-mon[32389]: pgmap v396: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-06-09T22:19:40.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:40 smithi153 ceph-mon[36599]: pgmap v396: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-06-09T22:19:40.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:40 smithi194 ceph-mon[36856]: pgmap v396: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-06-09T22:19:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.556+0000 7f483bf98700 1 -- 172.21.15.104:0/3376486372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48341030f0 msgr2=0x7f4834107100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.556+0000 7f483bf98700 1 --2- 172.21.15.104:0/3376486372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48341030f0 0x7f4834107100 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f48240099c0 tx=0x7f4824009cd0 comp rx=0 tx=0).stop 2024-06-09T22:19:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.556+0000 7f483bf98700 1 -- 172.21.15.104:0/3376486372 shutdown_connections 2024-06-09T22:19:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.556+0000 7f483bf98700 1 --2- 172.21.15.104:0/3376486372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4834107830 0x7f483410f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.556+0000 7f483bf98700 1 --2- 172.21.15.104:0/3376486372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4834102740 0x7f4834102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.556+0000 7f483bf98700 1 --2- 172.21.15.104:0/3376486372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48341030f0 0x7f4834107100 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.556+0000 7f483bf98700 1 -- 172.21.15.104:0/3376486372 >> 172.21.15.104:0/3376486372 conn(0x7f48340795b0 msgr2=0x7f48340799c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.556+0000 7f483bf98700 1 -- 172.21.15.104:0/3376486372 shutdown_connections 2024-06-09T22:19:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.556+0000 7f483bf98700 1 -- 172.21.15.104:0/3376486372 wait complete. 2024-06-09T22:19:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f483bf98700 1 Processor -- start 2024-06-09T22:19:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f483bf98700 1 -- start start 2024-06-09T22:19:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f483bf98700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4834102740 0x7f483419e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f483bf98700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48341030f0 0x7f483419eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f483bf98700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4834107830 0x7f4834198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f483bf98700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f48341127c0 con 0x7f4834107830 2024-06-09T22:19:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f483bf98700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4834112640 con 0x7f48341030f0 2024-06-09T22:19:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f483bf98700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4834112940 con 0x7f4834102740 2024-06-09T22:19:41.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f483a535700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4834107830 0x7f4834198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:41.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f483a535700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4834107830 0x7f4834198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55380/0 (socket says 172.21.15.104:55380) 2024-06-09T22:19:41.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f483a535700 1 -- 172.21.15.104:0/1579222660 learned_addr learned my addr 172.21.15.104:0/1579222660 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:41.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f4839d34700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4834102740 0x7f483419e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:41.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.557+0000 7f4839533700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48341030f0 0x7f483419eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:41.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.558+0000 7f483a535700 1 -- 172.21.15.104:0/1579222660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4834102740 msgr2=0x7f483419e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:41.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.558+0000 7f483a535700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4834102740 0x7f483419e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:41.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.558+0000 7f483a535700 1 -- 172.21.15.104:0/1579222660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48341030f0 msgr2=0x7f483419eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:41.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.558+0000 7f483a535700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48341030f0 0x7f483419eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:41.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.558+0000 7f483a535700 1 -- 172.21.15.104:0/1579222660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4834199180 con 0x7f4834107830 2024-06-09T22:19:41.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.558+0000 7f483a535700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4834107830 0x7f4834198a20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f482c00b970 tx=0x7f482c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:41.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.558+0000 7f482affd700 1 -- 172.21.15.104:0/1579222660 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482c019070 con 0x7f4834107830 2024-06-09T22:19:41.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.558+0000 7f482affd700 1 -- 172.21.15.104:0/1579222660 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f482c00ce10 con 0x7f4834107830 2024-06-09T22:19:41.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.558+0000 7f483bf98700 1 -- 172.21.15.104:0/1579222660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4834199470 con 0x7f4834107830 2024-06-09T22:19:41.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.558+0000 7f482affd700 1 -- 172.21.15.104:0/1579222660 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482c007370 con 0x7f4834107830 2024-06-09T22:19:41.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.558+0000 7f483bf98700 1 -- 172.21.15.104:0/1579222660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48341a5730 con 0x7f4834107830 2024-06-09T22:19:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.559+0000 7f483bf98700 1 -- 172.21.15.104:0/1579222660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4834069180 con 0x7f4834107830 2024-06-09T22:19:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.561+0000 7f482affd700 1 -- 172.21.15.104:0/1579222660 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f482c01c070 con 0x7f4834107830 2024-06-09T22:19:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.562+0000 7f482affd700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f48200a2c20 0x7f48200a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.562+0000 7f4839d34700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f48200a2c20 0x7f48200a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.562+0000 7f482affd700 1 -- 172.21.15.104:0/1579222660 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f482c0cec80 con 0x7f4834107830 2024-06-09T22:19:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.562+0000 7f4839d34700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f48200a2c20 0x7f48200a50e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4834199b60 tx=0x7f483000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:41.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.565+0000 7f482affd700 1 -- 172.21.15.104:0/1579222660 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f482c09b850 con 0x7f4834107830 2024-06-09T22:19:41.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.807+0000 7f483bf98700 1 -- 172.21.15.104:0/1579222660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48341a59c0 con 0x7f4834107830 2024-06-09T22:19:41.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.807+0000 7f482affd700 1 -- 172.21.15.104:0/1579222660 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7f482c09b150 con 0x7f4834107830 2024-06-09T22:19:41.805 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:19:41.805 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:17:53.402012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:19:41.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.810+0000 7f483bf98700 1 -- 172.21.15.104:0/1579222660 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f48200a2c20 msgr2=0x7f48200a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:41.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.810+0000 7f483bf98700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f48200a2c20 0x7f48200a50e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4834199b60 tx=0x7f483000b410 comp rx=0 tx=0).stop 2024-06-09T22:19:41.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.810+0000 7f483bf98700 1 -- 172.21.15.104:0/1579222660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4834107830 msgr2=0x7f4834198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:41.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.810+0000 7f483bf98700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4834107830 0x7f4834198a20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f482c00b970 tx=0x7f482c00bc80 comp rx=0 tx=0).stop 2024-06-09T22:19:41.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.810+0000 7f483bf98700 1 -- 172.21.15.104:0/1579222660 shutdown_connections 2024-06-09T22:19:41.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.810+0000 7f483bf98700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f48200a2c20 0x7f48200a50e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:41.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.810+0000 7f483bf98700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4834107830 0x7f4834198a20 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:41.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.810+0000 7f483bf98700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4834102740 0x7f483419e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:41.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.810+0000 7f483bf98700 1 --2- 172.21.15.104:0/1579222660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48341030f0 0x7f483419eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:41.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.810+0000 7f483bf98700 1 -- 172.21.15.104:0/1579222660 >> 172.21.15.104:0/1579222660 conn(0x7f48340795b0 msgr2=0x7f4834100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:41.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.810+0000 7f483bf98700 1 -- 172.21.15.104:0/1579222660 shutdown_connections 2024-06-09T22:19:41.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:41.810+0000 7f483bf98700 1 -- 172.21.15.104:0/1579222660 wait complete. 2024-06-09T22:19:41.809 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 28 2024-06-09T22:19:42.611 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:19:42.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:42 smithi153 ceph-mon[36599]: pgmap v397: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-09T22:19:42.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:42 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1579222660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:19:42.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:42 smithi104 ceph-mon[32389]: pgmap v397: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-09T22:19:42.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:42 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1579222660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:19:42.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:42 smithi194 ceph-mon[36856]: pgmap v397: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-06-09T22:19:42.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:42 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1579222660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:19:42.853 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:44.557 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:44 smithi104 ceph-mon[32389]: pgmap v398: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:19:44.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.561+0000 7f89beca7700 1 -- 172.21.15.104:0/4042131332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89b806b5c0 msgr2=0x7f89b810f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:44.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.561+0000 7f89beca7700 1 --2- 172.21.15.104:0/4042131332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89b806b5c0 0x7f89b810f370 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f89b000b0a0 tx=0x7f89b000b3b0 comp rx=0 tx=0).stop 2024-06-09T22:19:44.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.561+0000 7f89beca7700 1 -- 172.21.15.104:0/4042131332 shutdown_connections 2024-06-09T22:19:44.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.561+0000 7f89beca7700 1 --2- 172.21.15.104:0/4042131332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89b806b5c0 0x7f89b810f370 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:44.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.561+0000 7f89beca7700 1 --2- 172.21.15.104:0/4042131332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89b8108530 0x7f89b8108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:44.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.561+0000 7f89beca7700 1 --2- 172.21.15.104:0/4042131332 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89b806ac00 0x7f89b806b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:44.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.561+0000 7f89beca7700 1 -- 172.21.15.104:0/4042131332 >> 172.21.15.104:0/4042131332 conn(0x7f89b8077400 msgr2=0x7f89b8077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:44.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.561+0000 7f89beca7700 1 -- 172.21.15.104:0/4042131332 shutdown_connections 2024-06-09T22:19:44.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.561+0000 7f89beca7700 1 -- 172.21.15.104:0/4042131332 wait complete. 2024-06-09T22:19:44.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89beca7700 1 Processor -- start 2024-06-09T22:19:44.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89beca7700 1 -- start start 2024-06-09T22:19:44.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89beca7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89b806ac00 0x7f89b819e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:44.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89beca7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89b806b5c0 0x7f89b819ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:44.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89beca7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89b8108530 0x7f89b8198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:44.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89beca7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f89b8112780 con 0x7f89b8108530 2024-06-09T22:19:44.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89beca7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f89b8112600 con 0x7f89b806b5c0 2024-06-09T22:19:44.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89beca7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f89b8112900 con 0x7f89b806ac00 2024-06-09T22:19:44.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89bd244700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89b8108530 0x7f89b8198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89bd244700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89b8108530 0x7f89b8198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55402/0 (socket says 172.21.15.104:55402) 2024-06-09T22:19:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89bd244700 1 -- 172.21.15.104:0/341106902 learned_addr learned my addr 172.21.15.104:0/341106902 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89b7fff700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89b806b5c0 0x7f89b819ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89bca43700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89b806ac00 0x7f89b819e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89bd244700 1 -- 172.21.15.104:0/341106902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89b806ac00 msgr2=0x7f89b819e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89bd244700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89b806ac00 0x7f89b819e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89bd244700 1 -- 172.21.15.104:0/341106902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89b806b5c0 msgr2=0x7f89b819ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89bd244700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89b806b5c0 0x7f89b819ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.562+0000 7f89bd244700 1 -- 172.21.15.104:0/341106902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b81991e0 con 0x7f89b8108530 2024-06-09T22:19:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.563+0000 7f89bd244700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89b8108530 0x7f89b8198a80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f89b000b780 tx=0x7f89b0004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.563+0000 7f89b5ffb700 1 -- 172.21.15.104:0/341106902 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0021070 con 0x7f89b8108530 2024-06-09T22:19:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.563+0000 7f89b5ffb700 1 -- 172.21.15.104:0/341106902 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89b0007b80 con 0x7f89b8108530 2024-06-09T22:19:44.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.563+0000 7f89beca7700 1 -- 172.21.15.104:0/341106902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b8199470 con 0x7f89b8108530 2024-06-09T22:19:44.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.563+0000 7f89b5ffb700 1 -- 172.21.15.104:0/341106902 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b00052a0 con 0x7f89b8108530 2024-06-09T22:19:44.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.563+0000 7f89beca7700 1 -- 172.21.15.104:0/341106902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b81a5790 con 0x7f89b8108530 2024-06-09T22:19:44.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.564+0000 7f89beca7700 1 -- 172.21.15.104:0/341106902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89b810aa30 con 0x7f89b8108530 2024-06-09T22:19:44.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.565+0000 7f89b5ffb700 1 -- 172.21.15.104:0/341106902 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f89b0024070 con 0x7f89b8108530 2024-06-09T22:19:44.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.566+0000 7f89b5ffb700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f89a80a2ca0 0x7f89a80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:44.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.566+0000 7f89bca43700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f89a80a2ca0 0x7f89a80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:44.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.566+0000 7f89b5ffb700 1 -- 172.21.15.104:0/341106902 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f89b000f020 con 0x7f89b8108530 2024-06-09T22:19:44.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.566+0000 7f89bca43700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f89a80a2ca0 0x7f89a80a5160 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f89b8199bc0 tx=0x7f89ac00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:44.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.570+0000 7f89b5ffb700 1 -- 172.21.15.104:0/341106902 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f89b002b020 con 0x7f89b8108530 2024-06-09T22:19:44.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:44 smithi153 ceph-mon[36599]: pgmap v398: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:19:44.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.809+0000 7f89beca7700 1 -- 172.21.15.104:0/341106902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89b8069180 con 0x7f89b8108530 2024-06-09T22:19:44.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:44 smithi194 ceph-mon[36856]: pgmap v398: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:19:44.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.809+0000 7f89b5ffb700 1 -- 172.21.15.104:0/341106902 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7f89b009d480 con 0x7f89b8108530 2024-06-09T22:19:44.808 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:19:44.808 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:17:53.402012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:19:44.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.811+0000 7f89beca7700 1 -- 172.21.15.104:0/341106902 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f89a80a2ca0 msgr2=0x7f89a80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:44.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.811+0000 7f89beca7700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f89a80a2ca0 0x7f89a80a5160 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f89b8199bc0 tx=0x7f89ac00b3c0 comp rx=0 tx=0).stop 2024-06-09T22:19:44.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.812+0000 7f89beca7700 1 -- 172.21.15.104:0/341106902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89b8108530 msgr2=0x7f89b8198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:44.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.812+0000 7f89beca7700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89b8108530 0x7f89b8198a80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f89b000b780 tx=0x7f89b0004d00 comp rx=0 tx=0).stop 2024-06-09T22:19:44.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.812+0000 7f89beca7700 1 -- 172.21.15.104:0/341106902 shutdown_connections 2024-06-09T22:19:44.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.812+0000 7f89beca7700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f89a80a2ca0 0x7f89a80a5160 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:44.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.812+0000 7f89beca7700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89b8108530 0x7f89b8198a80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:44.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.812+0000 7f89beca7700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89b806ac00 0x7f89b819e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:44.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.812+0000 7f89beca7700 1 --2- 172.21.15.104:0/341106902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89b806b5c0 0x7f89b819ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:44.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.812+0000 7f89beca7700 1 -- 172.21.15.104:0/341106902 >> 172.21.15.104:0/341106902 conn(0x7f89b8077400 msgr2=0x7f89b8100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:44.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.812+0000 7f89beca7700 1 -- 172.21.15.104:0/341106902 shutdown_connections 2024-06-09T22:19:44.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:44.812+0000 7f89beca7700 1 -- 172.21.15.104:0/341106902 wait complete. 2024-06-09T22:19:44.811 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 28 2024-06-09T22:19:45.631 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:19:45.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:45 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/341106902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:19:45.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:45 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/341106902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:19:45.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:45 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/341106902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:19:45.878 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:46.504 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:46 smithi104 ceph-mon[32389]: pgmap v399: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:46.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:46 smithi153 ceph-mon[36599]: pgmap v399: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:46.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:46 smithi194 ceph-mon[36856]: pgmap v399: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:47.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.547+0000 7fd40c771700 1 -- 172.21.15.104:0/1365941882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd404108500 msgr2=0x7fd4041088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:47.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.547+0000 7fd40c771700 1 --2- 172.21.15.104:0/1365941882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd404108500 0x7fd4041088e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd400009990 tx=0x7fd400009ca0 comp rx=0 tx=0).stop 2024-06-09T22:19:47.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.547+0000 7fd40c771700 1 -- 172.21.15.104:0/1365941882 shutdown_connections 2024-06-09T22:19:47.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.547+0000 7fd40c771700 1 --2- 172.21.15.104:0/1365941882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd404108500 0x7fd4041088e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:47.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.547+0000 7fd40c771700 1 --2- 172.21.15.104:0/1365941882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd40406ab30 0x7fd40410f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:47.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.547+0000 7fd40c771700 1 --2- 172.21.15.104:0/1365941882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd40406a170 0x7fd40406a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:47.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.547+0000 7fd40c771700 1 -- 172.21.15.104:0/1365941882 >> 172.21.15.104:0/1365941882 conn(0x7fd404078340 msgr2=0x7fd404078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:47.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.547+0000 7fd40c771700 1 -- 172.21.15.104:0/1365941882 shutdown_connections 2024-06-09T22:19:47.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.547+0000 7fd40c771700 1 -- 172.21.15.104:0/1365941882 wait complete. 2024-06-09T22:19:47.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd40c771700 1 Processor -- start 2024-06-09T22:19:47.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd40c771700 1 -- start start 2024-06-09T22:19:47.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd40c771700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd40406a170 0x7fd40419e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:47.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd40c771700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd40406ab30 0x7fd40419ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:47.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd40c771700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd404108500 0x7fd4041989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:47.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd40c771700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd404112780 con 0x7fd404108500 2024-06-09T22:19:47.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd40c771700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd404112600 con 0x7fd40406ab30 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd40c771700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd404112900 con 0x7fd40406a170 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd40ad0e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd404108500 0x7fd4041989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd409d0c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd40406ab30 0x7fd40419ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd40ad0e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd404108500 0x7fd4041989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54752/0 (socket says 172.21.15.104:54752) 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd409d0c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd40406ab30 0x7fd40419ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:60632/0 (socket says 172.21.15.104:60632) 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd40ad0e700 1 -- 172.21.15.104:0/1104279524 learned_addr learned my addr 172.21.15.104:0/1104279524 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.548+0000 7fd40a50d700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd40406a170 0x7fd40419e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.549+0000 7fd40ad0e700 1 -- 172.21.15.104:0/1104279524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd40406a170 msgr2=0x7fd40419e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.549+0000 7fd40ad0e700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd40406a170 0x7fd40419e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.549+0000 7fd40ad0e700 1 -- 172.21.15.104:0/1104279524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd40406ab30 msgr2=0x7fd40419ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.549+0000 7fd40ad0e700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd40406ab30 0x7fd40419ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.549+0000 7fd40ad0e700 1 -- 172.21.15.104:0/1104279524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd404199100 con 0x7fd404108500 2024-06-09T22:19:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.549+0000 7fd40ad0e700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd404108500 0x7fd4041989a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc007d70 tx=0x7fd3fc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:47.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.549+0000 7fd3fb7fe700 1 -- 172.21.15.104:0/1104279524 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc016020 con 0x7fd404108500 2024-06-09T22:19:47.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.549+0000 7fd40c771700 1 -- 172.21.15.104:0/1104279524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4041993f0 con 0x7fd404108500 2024-06-09T22:19:47.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.549+0000 7fd3fb7fe700 1 -- 172.21.15.104:0/1104279524 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3fc00ac90 con 0x7fd404108500 2024-06-09T22:19:47.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.549+0000 7fd3fb7fe700 1 -- 172.21.15.104:0/1104279524 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0115c0 con 0x7fd404108500 2024-06-09T22:19:47.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.549+0000 7fd40c771700 1 -- 172.21.15.104:0/1104279524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4041a56b0 con 0x7fd404108500 2024-06-09T22:19:47.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.550+0000 7fd40c771700 1 -- 172.21.15.104:0/1104279524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd40410a9e0 con 0x7fd404108500 2024-06-09T22:19:47.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.552+0000 7fd3fb7fe700 1 -- 172.21.15.104:0/1104279524 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd3fc019020 con 0x7fd404108500 2024-06-09T22:19:47.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.553+0000 7fd3fb7fe700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd3f00a2c20 0x7fd3f00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:47.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.553+0000 7fd40a50d700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd3f00a2c20 0x7fd3f00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:47.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.553+0000 7fd3fb7fe700 1 -- 172.21.15.104:0/1104279524 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fd3fc0cf6f0 con 0x7fd404108500 2024-06-09T22:19:47.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.553+0000 7fd40a50d700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd3f00a2c20 0x7fd3f00a50e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd40000b5c0 tx=0x7fd400005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:47.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.556+0000 7fd3fb7fe700 1 -- 172.21.15.104:0/1104279524 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd3fc020020 con 0x7fd404108500 2024-06-09T22:19:47.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.752+0000 7fd40c771700 1 -- 172.21.15.104:0/1104279524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd40410a3c0 con 0x7fd404108500 2024-06-09T22:19:47.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.754+0000 7fd3fb7fe700 1 -- 172.21.15.104:0/1104279524 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+24762 (secure 0 0 0) 0x7fd3fc09c2c0 con 0x7fd404108500 2024-06-09T22:19:47.752 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:19:47.753 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":99,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:18:26.525714+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:19:47.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.757+0000 7fd40c771700 1 -- 172.21.15.104:0/1104279524 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd3f00a2c20 msgr2=0x7fd3f00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:47.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.757+0000 7fd40c771700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd3f00a2c20 0x7fd3f00a50e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd40000b5c0 tx=0x7fd400005960 comp rx=0 tx=0).stop 2024-06-09T22:19:47.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.757+0000 7fd40c771700 1 -- 172.21.15.104:0/1104279524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd404108500 msgr2=0x7fd4041989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:47.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.757+0000 7fd40c771700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd404108500 0x7fd4041989a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc007d70 tx=0x7fd3fc00a340 comp rx=0 tx=0).stop 2024-06-09T22:19:47.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.757+0000 7fd40c771700 1 -- 172.21.15.104:0/1104279524 shutdown_connections 2024-06-09T22:19:47.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.757+0000 7fd40c771700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd3f00a2c20 0x7fd3f00a50e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:47.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.757+0000 7fd40c771700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd404108500 0x7fd4041989a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:47.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.757+0000 7fd40c771700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd40406a170 0x7fd40419e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:47.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.757+0000 7fd40c771700 1 --2- 172.21.15.104:0/1104279524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd40406ab30 0x7fd40419ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:47.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.757+0000 7fd40c771700 1 -- 172.21.15.104:0/1104279524 >> 172.21.15.104:0/1104279524 conn(0x7fd404078340 msgr2=0x7fd4040ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:47.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.757+0000 7fd40c771700 1 -- 172.21.15.104:0/1104279524 shutdown_connections 2024-06-09T22:19:47.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:47.757+0000 7fd40c771700 1 -- 172.21.15.104:0/1104279524 wait complete. 2024-06-09T22:19:48.469 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:19:48.713 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:48.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:48 smithi153 ceph-mon[36599]: pgmap v400: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:19:48.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:48 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1104279524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:19:48.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:48 smithi104 ceph-mon[32389]: pgmap v400: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:19:48.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:48 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1104279524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:19:48.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:48 smithi194 ceph-mon[36856]: pgmap v400: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:19:48.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:48 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1104279524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:19:50.482 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:50 smithi104 ceph-mon[32389]: pgmap v401: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:19:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.488+0000 7f1b89270700 1 -- 172.21.15.104:0/397629953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b84101d50 msgr2=0x7f1b8410e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.488+0000 7f1b89270700 1 --2- 172.21.15.104:0/397629953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b84101d50 0x7f1b8410e240 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00b0a0 tx=0x7f1b7c00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:19:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.488+0000 7f1b89270700 1 -- 172.21.15.104:0/397629953 shutdown_connections 2024-06-09T22:19:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.488+0000 7f1b89270700 1 --2- 172.21.15.104:0/397629953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b84101d50 0x7f1b8410e240 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:50.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.488+0000 7f1b89270700 1 --2- 172.21.15.104:0/397629953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84107400 0x7f1b841077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:50.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.488+0000 7f1b89270700 1 --2- 172.21.15.104:0/397629953 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b84101390 0x7f1b84101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:50.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.488+0000 7f1b89270700 1 -- 172.21.15.104:0/397629953 >> 172.21.15.104:0/397629953 conn(0x7f1b84078340 msgr2=0x7f1b84078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:50.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.488+0000 7f1b89270700 1 -- 172.21.15.104:0/397629953 shutdown_connections 2024-06-09T22:19:50.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.488+0000 7f1b89270700 1 -- 172.21.15.104:0/397629953 wait complete. 2024-06-09T22:19:50.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b89270700 1 Processor -- start 2024-06-09T22:19:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b89270700 1 -- start start 2024-06-09T22:19:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b89270700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b84101390 0x7f1b8419d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b89270700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84101d50 0x7f1b8419dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b89270700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b84107400 0x7f1b84197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b89270700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1b841116a0 con 0x7f1b84101390 2024-06-09T22:19:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b89270700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1b84111520 con 0x7f1b84101d50 2024-06-09T22:19:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b89270700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1b84111820 con 0x7f1b84107400 2024-06-09T22:19:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b82ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b84101390 0x7f1b8419d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b82ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b84101390 0x7f1b8419d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54770/0 (socket says 172.21.15.104:54770) 2024-06-09T22:19:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b82ffd700 1 -- 172.21.15.104:0/2672475358 learned_addr learned my addr 172.21.15.104:0/2672475358 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b827fc700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84101d50 0x7f1b8419dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.489+0000 7f1b837fe700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b84107400 0x7f1b84197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b82ffd700 1 -- 172.21.15.104:0/2672475358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b84107400 msgr2=0x7f1b84197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b82ffd700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b84107400 0x7f1b84197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b82ffd700 1 -- 172.21.15.104:0/2672475358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84101d50 msgr2=0x7f1b8419dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b82ffd700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84101d50 0x7f1b8419dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b82ffd700 1 -- 172.21.15.104:0/2672475358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b841980e0 con 0x7f1b84101390 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b837fe700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b84107400 0x7f1b84197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b827fc700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84101d50 0x7f1b8419dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b82ffd700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b84101390 0x7f1b8419d7c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400c920 tx=0x7f1b7400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b6bfff700 1 -- 172.21.15.104:0/2672475358 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b74007b00 con 0x7f1b84101390 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b6bfff700 1 -- 172.21.15.104:0/2672475358 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b74004d10 con 0x7f1b84101390 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b89270700 1 -- 172.21.15.104:0/2672475358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b841983d0 con 0x7f1b84101390 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b6bfff700 1 -- 172.21.15.104:0/2672475358 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7400f450 con 0x7f1b84101390 2024-06-09T22:19:50.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.490+0000 7f1b89270700 1 -- 172.21.15.104:0/2672475358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b841a4690 con 0x7f1b84101390 2024-06-09T22:19:50.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.491+0000 7f1b89270700 1 -- 172.21.15.104:0/2672475358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b84069180 con 0x7f1b84101390 2024-06-09T22:19:50.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.493+0000 7f1b6bfff700 1 -- 172.21.15.104:0/2672475358 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1b74004750 con 0x7f1b84101390 2024-06-09T22:19:50.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.493+0000 7f1b6bfff700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1b6c0a2c20 0x7f1b6c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:50.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.493+0000 7f1b827fc700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1b6c0a2c20 0x7f1b6c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:50.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.493+0000 7f1b6bfff700 1 -- 172.21.15.104:0/2672475358 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f1b740cf380 con 0x7f1b84101390 2024-06-09T22:19:50.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.494+0000 7f1b827fc700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1b6c0a2c20 0x7f1b6c0a50e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b840788b0 tx=0x7f1b7800b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:50.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.498+0000 7f1b6bfff700 1 -- 172.21.15.104:0/2672475358 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1b74014030 con 0x7f1b84101390 2024-06-09T22:19:50.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.737+0000 7f1b89270700 1 -- 172.21.15.104:0/2672475358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b84198eb0 con 0x7f1b84101390 2024-06-09T22:19:50.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.738+0000 7f1b6bfff700 1 -- 172.21.15.104:0/2672475358 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7f1b7409bf50 con 0x7f1b84101390 2024-06-09T22:19:50.736 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:19:50.737 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:17:53.402012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:19:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.740+0000 7f1b89270700 1 -- 172.21.15.104:0/2672475358 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1b6c0a2c20 msgr2=0x7f1b6c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.740+0000 7f1b89270700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1b6c0a2c20 0x7f1b6c0a50e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b840788b0 tx=0x7f1b7800b2c0 comp rx=0 tx=0).stop 2024-06-09T22:19:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.740+0000 7f1b89270700 1 -- 172.21.15.104:0/2672475358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b84101390 msgr2=0x7f1b8419d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.740+0000 7f1b89270700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b84101390 0x7f1b8419d7c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400c920 tx=0x7f1b7400cce0 comp rx=0 tx=0).stop 2024-06-09T22:19:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.740+0000 7f1b89270700 1 -- 172.21.15.104:0/2672475358 shutdown_connections 2024-06-09T22:19:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.740+0000 7f1b89270700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1b6c0a2c20 0x7f1b6c0a50e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.740+0000 7f1b89270700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b84101390 0x7f1b8419d7c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.740+0000 7f1b89270700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b84107400 0x7f1b84197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.740+0000 7f1b89270700 1 --2- 172.21.15.104:0/2672475358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b84101d50 0x7f1b8419dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:50.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.740+0000 7f1b89270700 1 -- 172.21.15.104:0/2672475358 >> 172.21.15.104:0/2672475358 conn(0x7f1b84078340 msgr2=0x7f1b840ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:50.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.740+0000 7f1b89270700 1 -- 172.21.15.104:0/2672475358 shutdown_connections 2024-06-09T22:19:50.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:50.740+0000 7f1b89270700 1 -- 172.21.15.104:0/2672475358 wait complete. 2024-06-09T22:19:50.741 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 28 2024-06-09T22:19:50.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:50 smithi153 ceph-mon[36599]: pgmap v401: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:19:50.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:50 smithi194 ceph-mon[36856]: pgmap v401: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:19:51.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:51 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2672475358' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:19:51.497 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-06-09T22:16:45.150445+0000', 'modified': '2024-06-09T22:17:53.402012+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24460, 'mds_1': 24448, 'mds_2': 24442, 'mds_3': 24419, 'mds_4': 24403}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14571': {'gid': 14571, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6839/647544853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6838', 'nonce': 647544853}, {'type': 'v1', 'addr': '172.21.15.104:6839', 'nonce': 647544853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14583': {'gid': 14583, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6841/3830332242', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6840', 'nonce': 3830332242}, {'type': 'v1', 'addr': '172.21.15.104:6841', 'nonce': 3830332242}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24362': {'gid': 24362, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6833/3968887191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6832', 'nonce': 3968887191}, {'type': 'v1', 'addr': '172.21.15.153:6833', 'nonce': 3968887191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24383': {'gid': 24383, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6835/3075465176', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 3075465176}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 3075465176}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24398': {'gid': 24398, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6837/3294454895', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 3294454895}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 3294454895}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24403': {'gid': 24403, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.194:6833/3284873174', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6832', 'nonce': 3284873174}, {'type': 'v1', 'addr': '172.21.15.194:6833', 'nonce': 3284873174}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24419': {'gid': 24419, 'name': 'b', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.153:6839/779682823', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 779682823}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 779682823}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 26, 'addr': '172.21.15.194:6835/2995819396', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6834', 'nonce': 2995819396}, {'type': 'v1', 'addr': '172.21.15.194:6835', 'nonce': 2995819396}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24448': {'gid': 24448, 'name': 'c', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 25, 'addr': '172.21.15.194:6837/464494707', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6836', 'nonce': 464494707}, {'type': 'v1', 'addr': '172.21.15.194:6837', 'nonce': 464494707}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.194:6839/2028005544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6838', 'nonce': 2028005544}, {'type': 'v1', 'addr': '172.21.15.194:6839', 'nonce': 2028005544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24460, 'qdb_cluster': [24403, 24419, 24442, 24448, 24460]} 2024-06-09T22:19:51.497 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-06-09T22:19:51.498 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.l status 2024-06-09T22:19:51.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:51 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2672475358' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:19:51.745 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:51.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:51 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2672475358' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:19:52.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:52 smithi153 ceph-mon[36599]: pgmap v402: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:19:52.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:52 smithi104 ceph-mon[32389]: pgmap v402: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:19:52.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:52 smithi194 ceph-mon[36856]: pgmap v402: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:19:53.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.496+0000 7f439f59e700 1 -- 172.21.15.104:0/2164777251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a0101520 msgr2=0x7f43a0101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:53.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.496+0000 7f439f59e700 1 --2- 172.21.15.104:0/2164777251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a0101520 0x7f43a0101900 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f43880099c0 tx=0x7f4388009cd0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.496+0000 7f439f59e700 1 -- 172.21.15.104:0/2164777251 shutdown_connections 2024-06-09T22:19:53.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.496+0000 7f439f59e700 1 --2- 172.21.15.104:0/2164777251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a0101520 0x7f43a0101900 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.496+0000 7f439f59e700 1 --2- 172.21.15.104:0/2164777251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43a0106600 0x7f43a010e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.496+0000 7f439f59e700 1 --2- 172.21.15.104:0/2164777251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43a0101ed0 0x7f43a0105f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.496+0000 7f439f59e700 1 -- 172.21.15.104:0/2164777251 >> 172.21.15.104:0/2164777251 conn(0x7f43a00fcdb0 msgr2=0x7f43a00ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:53.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.496+0000 7f439f59e700 1 -- 172.21.15.104:0/2164777251 shutdown_connections 2024-06-09T22:19:53.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.496+0000 7f439f59e700 1 -- 172.21.15.104:0/2164777251 wait complete. 2024-06-09T22:19:53.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.497+0000 7f439f59e700 1 Processor -- start 2024-06-09T22:19:53.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.497+0000 7f439f59e700 1 -- start start 2024-06-09T22:19:53.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.497+0000 7f439f59e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43a0101520 0x7f43a01999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.497+0000 7f439f59e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43a0101ed0 0x7f43a0199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.497+0000 7f439f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a0106600 0x7f43a01a4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.497+0000 7f439f59e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f43a0111660 con 0x7f43a0106600 2024-06-09T22:19:53.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.497+0000 7f439f59e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f43a01114e0 con 0x7f43a0101520 2024-06-09T22:19:53.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.497+0000 7f439f59e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f43a01117e0 con 0x7f43a0101ed0 2024-06-09T22:19:53.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.497+0000 7f439ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a0106600 0x7f43a01a4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.497+0000 7f439ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a0106600 0x7f43a01a4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54774/0 (socket says 172.21.15.104:54774) 2024-06-09T22:19:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.497+0000 7f439ed9d700 1 -- 172.21.15.104:0/1207842663 learned_addr learned my addr 172.21.15.104:0/1207842663 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f439dd9b700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43a0101ed0 0x7f43a0199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f439e59c700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43a0101520 0x7f43a01999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f439ed9d700 1 -- 172.21.15.104:0/1207842663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43a0101ed0 msgr2=0x7f43a0199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f439ed9d700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43a0101ed0 0x7f43a0199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f439ed9d700 1 -- 172.21.15.104:0/1207842663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43a0101520 msgr2=0x7f43a01999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f439ed9d700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43a0101520 0x7f43a01999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f439ed9d700 1 -- 172.21.15.104:0/1207842663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a01a4f80 con 0x7f43a0106600 2024-06-09T22:19:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f439dd9b700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43a0101ed0 0x7f43a0199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:19:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f439ed9d700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a0106600 0x7f43a01a4a40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4398007ee0 tx=0x7f439800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f43977fe700 1 -- 172.21.15.104:0/1207842663 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439800dbd0 con 0x7f43a0106600 2024-06-09T22:19:53.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f43977fe700 1 -- 172.21.15.104:0/1207842663 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f439800dd70 con 0x7f43a0106600 2024-06-09T22:19:53.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f439f59e700 1 -- 172.21.15.104:0/1207842663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a01a51b0 con 0x7f43a0106600 2024-06-09T22:19:53.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f43977fe700 1 -- 172.21.15.104:0/1207842663 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4398005a60 con 0x7f43a0106600 2024-06-09T22:19:53.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.498+0000 7f439f59e700 1 -- 172.21.15.104:0/1207842663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a01a56c0 con 0x7f43a0106600 2024-06-09T22:19:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.500+0000 7f43977fe700 1 -- 172.21.15.104:0/1207842663 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f439801a020 con 0x7f43a0106600 2024-06-09T22:19:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.501+0000 7f43977fe700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f438c0a2c20 0x7f438c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.501+0000 7f439e59c700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f438c0a2c20 0x7f438c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.501+0000 7f43977fe700 1 -- 172.21.15.104:0/1207842663 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f43980ceaf0 con 0x7f43a0106600 2024-06-09T22:19:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.501+0000 7f439e59c700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f438c0a2c20 0x7f438c0a50e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4388000c00 tx=0x7f4388005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.502+0000 7f43af683b80 1 Processor -- start 2024-06-09T22:19:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.502+0000 7f43af683b80 1 -- start start 2024-06-09T22:19:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.502+0000 7f43af683b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dc8a0 0x5595641dcc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.502+0000 7f43af683b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.502+0000 7f43af683b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 0x5595641d7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.502+0000 7f43af683b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5595641287c0 con 0x5595641dd1c0 2024-06-09T22:19:53.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.502+0000 7f43af683b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5595640d06d0 con 0x5595641dc8a0 2024-06-09T22:19:53.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.502+0000 7f43af683b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559564124780 con 0x5595641d7030 2024-06-09T22:19:53.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.502+0000 7f439dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.502+0000 7f439dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54776/0 (socket says 172.21.15.104:54776) 2024-06-09T22:19:53.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.502+0000 7f439dd9b700 1 -- 172.21.15.104:0/4210439644 learned_addr learned my addr 172.21.15.104:0/4210439644 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:53.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.502+0000 7f439e59c700 1 --2- 172.21.15.104:0/4210439644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dc8a0 0x5595641dcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.503+0000 7f439ed9d700 1 --2- 172.21.15.104:0/4210439644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 0x5595641d7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.503+0000 7f439dd9b700 1 -- 172.21.15.104:0/4210439644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 msgr2=0x5595641d7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:53.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.503+0000 7f439dd9b700 1 --2- 172.21.15.104:0/4210439644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 0x5595641d7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.503+0000 7f439dd9b700 1 -- 172.21.15.104:0/4210439644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dc8a0 msgr2=0x5595641dcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:53.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.503+0000 7f439dd9b700 1 --2- 172.21.15.104:0/4210439644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dc8a0 0x5595641dcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.503+0000 7f439dd9b700 1 -- 172.21.15.104:0/4210439644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595640c6f00 con 0x5595641dd1c0 2024-06-09T22:19:53.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.503+0000 7f439dd9b700 1 --2- 172.21.15.104:0/4210439644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d6af0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4390007830 tx=0x7f4390007b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5175308b03b88355 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:53.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.503+0000 7f4394ff9700 1 -- 172.21.15.104:0/4210439644 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43900125f0 con 0x5595641dd1c0 2024-06-09T22:19:53.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.503+0000 7f4394ff9700 1 -- 172.21.15.104:0/4210439644 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4390018070 con 0x5595641dd1c0 2024-06-09T22:19:53.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.503+0000 7f4394ff9700 1 -- 172.21.15.104:0/4210439644 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4390013bb0 con 0x5595641dd1c0 2024-06-09T22:19:53.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.504+0000 7f43af683b80 1 -- 172.21.15.104:0/4210439644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 msgr2=0x5595641d6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:53.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.504+0000 7f43af683b80 1 --2- 172.21.15.104:0/4210439644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d6af0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4390007830 tx=0x7f4390007b40 comp rx=0 tx=0).stop 2024-06-09T22:19:53.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.504+0000 7f43af683b80 1 -- 172.21.15.104:0/4210439644 shutdown_connections 2024-06-09T22:19:53.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.504+0000 7f43af683b80 1 --2- 172.21.15.104:0/4210439644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 0x5595641d7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.504+0000 7f43af683b80 1 --2- 172.21.15.104:0/4210439644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dc8a0 0x5595641dcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.504+0000 7f43af683b80 1 --2- 172.21.15.104:0/4210439644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d6af0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.504+0000 7f43af683b80 1 -- 172.21.15.104:0/4210439644 >> 172.21.15.104:0/4210439644 conn(0x559564101c70 msgr2=0x5595641d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.504+0000 7f43af683b80 1 -- 172.21.15.104:0/4210439644 shutdown_connections 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.504+0000 7f43af683b80 1 -- 172.21.15.104:0/4210439644 wait complete. 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.504+0000 7f43af683b80 1 -- start start 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.504+0000 7f43af683b80 1 Processor -- start 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f43af683b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 0x5595641e2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f43af683b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dc8a0 0x5595641e01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f43af683b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dd1c0 0x5595641e06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f43af683b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5595641e3b80 con 0x5595641dc8a0 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f43af683b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5595640c8ba0 con 0x5595641dd1c0 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f43af683b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5595641e3d00 con 0x5595641d7030 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f43af683b80 10 client.0 _ll_register_callbacks cb 0x5595640c28d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f439dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dc8a0 0x5595641e01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f439dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dc8a0 0x5595641e01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54784/0 (socket says 172.21.15.104:54784) 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f439dd9b700 1 -- 172.21.15.104:0/3693880473 learned_addr learned my addr 172.21.15.104:0/3693880473 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f439ed9d700 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dd1c0 0x5595641e06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f439e59c700 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 0x5595641e2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f439dd9b700 1 -- 172.21.15.104:0/3693880473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 msgr2=0x5595641e2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f439dd9b700 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 0x5595641e2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f439dd9b700 1 -- 172.21.15.104:0/3693880473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dd1c0 msgr2=0x5595641e06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f439dd9b700 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dd1c0 0x5595641e06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f439dd9b700 1 -- 172.21.15.104:0/3693880473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559564077880 con 0x5595641dc8a0 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f439dd9b700 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dc8a0 0x5595641e01a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f43900048a0 tx=0x7f43900136b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5009ff7d40a26b7a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f4385ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f4385ffb700 1 -- 172.21.15.104:0/3693880473 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4390013ce0 con 0x5595641dc8a0 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f4385ffb700 1 -- 172.21.15.104:0/3693880473 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f43900125f0 con 0x5595641dc8a0 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.505+0000 7f43af683b80 1 -- 172.21.15.104:0/3693880473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4390018070 con 0x5595641dc8a0 2024-06-09T22:19:53.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.506+0000 7f4385ffb700 1 -- 172.21.15.104:0/3693880473 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4390030050 con 0x5595641dc8a0 2024-06-09T22:19:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.506+0000 7f43af683b80 10 client.15030 fetch_fsmap learned FSMap version 28 2024-06-09T22:19:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.506+0000 7f43af683b80 1 -- 172.21.15.104:0/3693880473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595641ee480 con 0x5595641dc8a0 2024-06-09T22:19:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.506+0000 7f4385ffb700 1 -- 172.21.15.104:0/3693880473 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4390018070 con 0x5595641dc8a0 2024-06-09T22:19:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.506+0000 7f4385ffb700 1 -- 172.21.15.104:0/3693880473 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4390027820 con 0x5595641dc8a0 2024-06-09T22:19:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.506+0000 7f43af683b80 10 client.15030 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:19:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.506+0000 7f43af683b80 10 client.15030 resolve_mds: resolved name 'l' to daemon mds.l 2024-06-09T22:19:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.506+0000 7f43af683b80 20 client.15030 populate_metadata read hostname 'smithi104' 2024-06-09T22:19:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.506+0000 7f43af683b80 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5595641f06a0 0x5595641f1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.506+0000 7f43af683b80 4 client.15030 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:19:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.506+0000 7f43af683b80 1 -- 172.21.15.104:0/3693880473 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559564135cc0 con 0x5595641f06a0 2024-06-09T22:19:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.506+0000 7f439e59c700 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5595641f06a0 0x5595641f1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.507+0000 7f439e59c700 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5595641f06a0 0x5595641f1b10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a62cab4ee49567d5 server_cookie=52ce82a49c8d4c96 in_seq=0 out_seq=0 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.507+0000 7f4385ffb700 10 client.15030 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f4385ffb700 1 -- 172.21.15.104:0/3693880473 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x559564135cc0 con 0x5595641f06a0 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f4385ffb700 10 client.15030 handle_command_reply: tid=0 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f43af683b80 1 client.15030 shutdown 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f43af683b80 1 -- 172.21.15.104:0/3693880473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dc8a0 msgr2=0x5595641e01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f43af683b80 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dc8a0 0x5595641e01a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f43900048a0 tx=0x7f43900136b0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f43af683b80 1 -- 172.21.15.104:0/3693880473 shutdown_connections 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f43af683b80 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5595641f06a0 0x5595641f1b10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f43af683b80 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dd1c0 0x5595641e06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f43af683b80 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 0x5595641e2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f43af683b80 1 --2- 172.21.15.104:0/3693880473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dc8a0 0x5595641e01a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f43af683b80 1 -- 172.21.15.104:0/3693880473 >> 172.21.15.104:0/3693880473 conn(0x559564101c70 msgr2=0x5595641d5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f4385ffb700 0 client.15030 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:19:53.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f43af683b80 1 -- 172.21.15.104:0/3693880473 shutdown_connections 2024-06-09T22:19:53.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f43af683b80 1 -- 172.21.15.104:0/3693880473 wait complete. 2024-06-09T22:19:53.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.508+0000 7f43af683b80 20 client.15030 trim_cache size 0 max 16384 2024-06-09T22:19:53.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.529+0000 7f43af683b80 1 Processor -- start 2024-06-09T22:19:53.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.529+0000 7f43af683b80 1 -- start start 2024-06-09T22:19:53.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.529+0000 7f43af683b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 0x5595641d6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.529+0000 7f43af683b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dc8a0 0x5595641fb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.529+0000 7f43af683b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.529+0000 7f43af683b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5595641efe50 con 0x5595641dd1c0 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.529+0000 7f43af683b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5595641f0520 con 0x5595641dc8a0 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.529+0000 7f43af683b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5595641f0bf0 con 0x5595641d7030 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f439ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f439ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54798/0 (socket says 172.21.15.104:54798) 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f439ed9d700 1 -- 172.21.15.104:0/4093671113 learned_addr learned my addr 172.21.15.104:0/4093671113 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f439dd9b700 1 --2- 172.21.15.104:0/4093671113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dc8a0 0x5595641fb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f439e59c700 1 --2- 172.21.15.104:0/4093671113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 0x5595641d6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f439ed9d700 1 -- 172.21.15.104:0/4093671113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 msgr2=0x5595641d6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f439ed9d700 1 --2- 172.21.15.104:0/4093671113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 0x5595641d6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f439ed9d700 1 -- 172.21.15.104:0/4093671113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dc8a0 msgr2=0x5595641fb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:53.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f439ed9d700 1 --2- 172.21.15.104:0/4093671113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dc8a0 0x5595641fb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f439ed9d700 1 -- 172.21.15.104:0/4093671113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595641d29c0 con 0x5595641dd1c0 2024-06-09T22:19:53.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f439ed9d700 1 --2- 172.21.15.104:0/4093671113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d1490 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f43980cc5d0 tx=0x7f439800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b7562b7a1af62d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:53.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f4387fff700 1 -- 172.21.15.104:0/4093671113 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439809afc0 con 0x5595641dd1c0 2024-06-09T22:19:53.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f4387fff700 1 -- 172.21.15.104:0/4093671113 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f439809f370 con 0x5595641dd1c0 2024-06-09T22:19:53.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.530+0000 7f4387fff700 1 -- 172.21.15.104:0/4093671113 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4398022250 con 0x5595641dd1c0 2024-06-09T22:19:53.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.531+0000 7f43af683b80 1 -- 172.21.15.104:0/4093671113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 msgr2=0x5595641d1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:53.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.531+0000 7f43af683b80 1 --2- 172.21.15.104:0/4093671113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d1490 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f43980cc5d0 tx=0x7f439800b040 comp rx=0 tx=0).stop 2024-06-09T22:19:53.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.531+0000 7f43af683b80 1 -- 172.21.15.104:0/4093671113 shutdown_connections 2024-06-09T22:19:53.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.531+0000 7f43af683b80 1 --2- 172.21.15.104:0/4093671113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d1490 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.531+0000 7f43af683b80 1 --2- 172.21.15.104:0/4093671113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641d7030 0x5595641d6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.531+0000 7f43af683b80 1 --2- 172.21.15.104:0/4093671113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641dc8a0 0x5595641fb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.531+0000 7f43af683b80 1 -- 172.21.15.104:0/4093671113 >> 172.21.15.104:0/4093671113 conn(0x559564101c70 msgr2=0x5595641dea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:53.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.531+0000 7f43af683b80 1 -- 172.21.15.104:0/4093671113 shutdown_connections 2024-06-09T22:19:53.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.531+0000 7f43af683b80 1 -- 172.21.15.104:0/4093671113 wait complete. 2024-06-09T22:19:53.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.531+0000 7f43af683b80 1 -- start start 2024-06-09T22:19:53.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.531+0000 7f43af683b80 1 Processor -- start 2024-06-09T22:19:53.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f43af683b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641d7030 0x5595641fb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f43af683b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641dc8a0 0x5595641d5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f43af683b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f43af683b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5595641f1990 con 0x5595641dd1c0 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f43af683b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5595641f12c0 con 0x5595641d7030 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f43af683b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5595641f2060 con 0x5595641dc8a0 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f43af683b80 10 client.0 _ll_register_callbacks cb 0x5595640c28d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f439ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f439ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54800/0 (socket says 172.21.15.104:54800) 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f439e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641d7030 0x5595641fb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f439ed9d700 1 -- 172.21.15.104:0/20676015 learned_addr learned my addr 172.21.15.104:0/20676015 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f439dd9b700 1 --2- 172.21.15.104:0/20676015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641dc8a0 0x5595641d5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f439ed9d700 1 -- 172.21.15.104:0/20676015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641dc8a0 msgr2=0x5595641d5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f439ed9d700 1 --2- 172.21.15.104:0/20676015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641dc8a0 0x5595641d5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f439ed9d700 1 -- 172.21.15.104:0/20676015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641d7030 msgr2=0x5595641fb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f439ed9d700 1 --2- 172.21.15.104:0/20676015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641d7030 0x5595641fb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f439ed9d700 1 -- 172.21.15.104:0/20676015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595641d6810 con 0x5595641dd1c0 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f439ed9d700 1 --2- 172.21.15.104:0/20676015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d6020 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f439809af00 tx=0x7f439809b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ec2a93e60f687d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f4385ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f4385ffb700 1 -- 172.21.15.104:0/20676015 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4398022250 con 0x5595641dd1c0 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f4385ffb700 1 -- 172.21.15.104:0/20676015 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f43980223f0 con 0x5595641dd1c0 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.532+0000 7f43af683b80 1 -- 172.21.15.104:0/20676015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f439809f370 con 0x5595641dd1c0 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.533+0000 7f4385ffb700 1 -- 172.21.15.104:0/20676015 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f439809b220 con 0x5595641dd1c0 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.533+0000 7f43af683b80 10 client.15036 fetch_fsmap learned FSMap version 28 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.533+0000 7f43af683b80 1 -- 172.21.15.104:0/20676015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55956425ff30 con 0x5595641dd1c0 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.533+0000 7f4385ffb700 1 -- 172.21.15.104:0/20676015 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f439809f370 con 0x5595641dd1c0 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.533+0000 7f4385ffb700 1 -- 172.21.15.104:0/20676015 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f439809d070 con 0x5595641dd1c0 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.533+0000 7f43af683b80 10 client.15036 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.533+0000 7f43af683b80 10 client.15036 resolve_mds: resolved name 'l' to daemon mds.l 2024-06-09T22:19:53.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.533+0000 7f43af683b80 20 client.15036 populate_metadata read hostname 'smithi104' 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.533+0000 7f43af683b80 1 --2- 172.21.15.104:0/20676015 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5595642644d0 0x5595642631e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.533+0000 7f43af683b80 4 client.15036 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.533+0000 7f43af683b80 1 -- 172.21.15.104:0/20676015 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "status"}) -- 0x559564130240 con 0x5595642644d0 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.533+0000 7f439e59c700 1 --2- 172.21.15.104:0/20676015 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5595642644d0 0x5595642631e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f439e59c700 1 --2- 172.21.15.104:0/20676015 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5595642644d0 0x5595642631e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b8563556e639ba6c server_cookie=c46bd229f3b9981c in_seq=0 out_seq=0 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f4385ffb700 10 client.15036 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f4385ffb700 1 -- 172.21.15.104:0/20676015 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x559564130240 con 0x5595642644d0 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f4385ffb700 10 client.15036 handle_command_reply: tid=0 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f43af683b80 1 client.15036 shutdown 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f43af683b80 1 -- 172.21.15.104:0/20676015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 msgr2=0x5595641d6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f43af683b80 1 --2- 172.21.15.104:0/20676015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d6020 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f439809af00 tx=0x7f439809b7f0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f43af683b80 1 -- 172.21.15.104:0/20676015 shutdown_connections 2024-06-09T22:19:53.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f43af683b80 1 --2- 172.21.15.104:0/20676015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5595641dd1c0 0x5595641d6020 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f43af683b80 1 --2- 172.21.15.104:0/20676015 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5595642644d0 0x5595642631e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f43af683b80 1 --2- 172.21.15.104:0/20676015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5595641d7030 0x5595641fb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f43af683b80 1 --2- 172.21.15.104:0/20676015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5595641dc8a0 0x5595641d5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f43af683b80 1 -- 172.21.15.104:0/20676015 >> 172.21.15.104:0/20676015 conn(0x559564101c70 msgr2=0x5595641dd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:53.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f4385ffb700 0 client.15036 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:19:53.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f43af683b80 1 -- 172.21.15.104:0/20676015 shutdown_connections 2024-06-09T22:19:53.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f43af683b80 1 -- 172.21.15.104:0/20676015 wait complete. 2024-06-09T22:19:53.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.534+0000 7f43af683b80 20 client.15036 trim_cache size 0 max 16384 2024-06-09T22:19:53.539 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:19:53.539 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:19:53.540 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 4, 2024-06-09T22:19:53.540 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24403, 2024-06-09T22:19:53.540 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:19:53.540 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:19:53.540 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:19:53.540 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 123.097821432, 2024-06-09T22:19:53.540 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 28, 2024-06-09T22:19:53.540 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 99, 2024-06-09T22:19:53.540 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 97, 2024-06-09T22:19:53.540 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 239.637744029 2024-06-09T22:19:53.540 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:19:53.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.535+0000 7f439f59e700 1 -- 172.21.15.104:0/1207842663 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f438c0a2c20 msgr2=0x7f438c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:53.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.535+0000 7f439f59e700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f438c0a2c20 0x7f438c0a50e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4388000c00 tx=0x7f4388005fb0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.535+0000 7f439f59e700 1 -- 172.21.15.104:0/1207842663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a0106600 msgr2=0x7f43a01a4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:53.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.535+0000 7f439f59e700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a0106600 0x7f43a01a4a40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4398007ee0 tx=0x7f439800d3b0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.535+0000 7f439f59e700 1 -- 172.21.15.104:0/1207842663 shutdown_connections 2024-06-09T22:19:53.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.535+0000 7f439f59e700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f438c0a2c20 0x7f438c0a50e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.535+0000 7f439f59e700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a0106600 0x7f43a01a4a40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.535+0000 7f439f59e700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43a0101520 0x7f43a01999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.535+0000 7f439f59e700 1 --2- 172.21.15.104:0/1207842663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43a0101ed0 0x7f43a0199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:53.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.535+0000 7f439f59e700 1 -- 172.21.15.104:0/1207842663 >> 172.21.15.104:0/1207842663 conn(0x7f43a00fcdb0 msgr2=0x7f43a00feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:53.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.535+0000 7f439f59e700 1 -- 172.21.15.104:0/1207842663 shutdown_connections 2024-06-09T22:19:53.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:53.535+0000 7f439f59e700 1 -- 172.21.15.104:0/1207842663 wait complete. 2024-06-09T22:19:54.243 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-06-09T22:19:54.487 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:54.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:54 smithi153 ceph-mon[36599]: pgmap v403: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:19:54.753 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:54 smithi104 ceph-mon[32389]: pgmap v403: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:19:54.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:54 smithi194 ceph-mon[36856]: pgmap v403: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:19:56.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.291+0000 7f44c4036700 1 -- 172.21.15.104:0/1696523070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44bc1030f0 msgr2=0x7f44bc107110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:56.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.291+0000 7f44c4036700 1 --2- 172.21.15.104:0/1696523070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44bc1030f0 0x7f44bc107110 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0099c0 tx=0x7f44ac009cd0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.291+0000 7f44c4036700 1 -- 172.21.15.104:0/1696523070 shutdown_connections 2024-06-09T22:19:56.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.291+0000 7f44c4036700 1 --2- 172.21.15.104:0/1696523070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc107840 0x7f44bc10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.291+0000 7f44c4036700 1 --2- 172.21.15.104:0/1696523070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44bc102740 0x7f44bc102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.291+0000 7f44c4036700 1 --2- 172.21.15.104:0/1696523070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44bc1030f0 0x7f44bc107110 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.291+0000 7f44c4036700 1 -- 172.21.15.104:0/1696523070 >> 172.21.15.104:0/1696523070 conn(0x7f44bc0795f0 msgr2=0x7f44bc079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:56.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.291+0000 7f44c4036700 1 -- 172.21.15.104:0/1696523070 shutdown_connections 2024-06-09T22:19:56.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.291+0000 7f44c4036700 1 -- 172.21.15.104:0/1696523070 wait complete. 2024-06-09T22:19:56.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.291+0000 7f44c4036700 1 Processor -- start 2024-06-09T22:19:56.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c4036700 1 -- start start 2024-06-09T22:19:56.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c4036700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44bc102740 0x7f44bc19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c4036700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc1030f0 0x7f44bc19b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c4036700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44bc107840 0x7f44bc19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c4036700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc1127f0 con 0x7f44bc107840 2024-06-09T22:19:56.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c4036700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc112670 con 0x7f44bc102740 2024-06-09T22:19:56.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c4036700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc112970 con 0x7f44bc1030f0 2024-06-09T22:19:56.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c25d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44bc107840 0x7f44bc19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c25d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44bc107840 0x7f44bc19f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60628/0 (socket says 172.21.15.104:60628) 2024-06-09T22:19:56.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c25d3700 1 -- 172.21.15.104:0/874984197 learned_addr learned my addr 172.21.15.104:0/874984197 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:56.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44bc102740 0x7f44bc19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c15d1700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc1030f0 0x7f44bc19b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c25d3700 1 -- 172.21.15.104:0/874984197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc1030f0 msgr2=0x7f44bc19b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:56.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c25d3700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc1030f0 0x7f44bc19b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c25d3700 1 -- 172.21.15.104:0/874984197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44bc102740 msgr2=0x7f44bc19ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:56.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c25d3700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44bc102740 0x7f44bc19ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c25d3700 1 -- 172.21.15.104:0/874984197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44bc19fe30 con 0x7f44bc107840 2024-06-09T22:19:56.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44c25d3700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44bc107840 0x7f44bc19f6f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f44b4007ee0 tx=0x7f44b400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:56.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.292+0000 7f44b2ffd700 1 -- 172.21.15.104:0/874984197 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b400dbd0 con 0x7f44bc107840 2024-06-09T22:19:56.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.293+0000 7f44b2ffd700 1 -- 172.21.15.104:0/874984197 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44b400dd70 con 0x7f44bc107840 2024-06-09T22:19:56.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.293+0000 7f44c4036700 1 -- 172.21.15.104:0/874984197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44bc1a0120 con 0x7f44bc107840 2024-06-09T22:19:56.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.293+0000 7f44b2ffd700 1 -- 172.21.15.104:0/874984197 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b4005a60 con 0x7f44bc107840 2024-06-09T22:19:56.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.293+0000 7f44c4036700 1 -- 172.21.15.104:0/874984197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44bc1a7a20 con 0x7f44bc107840 2024-06-09T22:19:56.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.295+0000 7f44b2ffd700 1 -- 172.21.15.104:0/874984197 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f44b401a020 con 0x7f44bc107840 2024-06-09T22:19:56.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.295+0000 7f44b2ffd700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f44a80a2ca0 0x7f44a80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.296+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f44a80a2ca0 0x7f44a80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.296+0000 7f44b2ffd700 1 -- 172.21.15.104:0/874984197 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f44b40cea60 con 0x7f44bc107840 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.296+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f44a80a2ca0 0x7f44a80a5160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f44b8009a10 tx=0x7f44b8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.296+0000 7f44ce8b9b80 1 Processor -- start 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44ce8b9b80 1 -- start start 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44ce8b9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 0x561a746fc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44ce8b9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746fcf10 0x561a746f6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44ce8b9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746f6ed0 0x561a746f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44ce8b9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561a746487c0 con 0x561a746fc5f0 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44ce8b9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561a745f06d0 con 0x561a746fcf10 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44ce8b9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561a74644780 con 0x561a746f6ed0 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44c1dd2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 0x561a746fc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44c1dd2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 0x561a746fc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60644/0 (socket says 172.21.15.104:60644) 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44c1dd2700 1 -- 172.21.15.104:0/2212311115 learned_addr learned my addr 172.21.15.104:0/2212311115 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:56.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44c15d1700 1 --2- 172.21.15.104:0/2212311115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746fcf10 0x561a746f6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44c1dd2700 1 -- 172.21.15.104:0/2212311115 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746f6ed0 msgr2=0x561a746f72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:56.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44c25d3700 1 --2- 172.21.15.104:0/2212311115 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746f6ed0 0x561a746f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/2212311115 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746f6ed0 0x561a746f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44c1dd2700 1 -- 172.21.15.104:0/2212311115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746fcf10 msgr2=0x561a746f6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:56.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/2212311115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746fcf10 0x561a746f6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44c1dd2700 1 -- 172.21.15.104:0/2212311115 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a745e6f00 con 0x561a746fc5f0 2024-06-09T22:19:56.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/2212311115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 0x561a746fc9d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f44b800ea00 tx=0x7f44b8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848ce3bfe947275c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:56.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.297+0000 7f44a7fff700 1 -- 172.21.15.104:0/2212311115 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b801b040 con 0x561a746fc5f0 2024-06-09T22:19:56.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.298+0000 7f44a7fff700 1 -- 172.21.15.104:0/2212311115 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44b8024070 con 0x561a746fc5f0 2024-06-09T22:19:56.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.298+0000 7f44a7fff700 1 -- 172.21.15.104:0/2212311115 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b8013810 con 0x561a746fc5f0 2024-06-09T22:19:56.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.298+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2212311115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 msgr2=0x561a746fc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:56.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.298+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/2212311115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 0x561a746fc9d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f44b800ea00 tx=0x7f44b8011000 comp rx=0 tx=0).stop 2024-06-09T22:19:56.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.298+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2212311115 shutdown_connections 2024-06-09T22:19:56.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.298+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/2212311115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 0x561a746fc9d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.298+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/2212311115 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746f6ed0 0x561a746f72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.298+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/2212311115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746fcf10 0x561a746f6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.298+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2212311115 >> 172.21.15.104:0/2212311115 conn(0x561a74621c70 msgr2=0x561a746f1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:56.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.298+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2212311115 shutdown_connections 2024-06-09T22:19:56.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.298+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2212311115 wait complete. 2024-06-09T22:19:56.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.299+0000 7f44ce8b9b80 1 -- start start 2024-06-09T22:19:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.299+0000 7f44ce8b9b80 1 Processor -- start 2024-06-09T22:19:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.299+0000 7f44ce8b9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746f6ed0 0x561a74702c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.299+0000 7f44ce8b9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fc5f0 0x561a746fff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.299+0000 7f44ce8b9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746fcf10 0x561a747004a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.299+0000 7f44ce8b9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561a74703980 con 0x561a746f6ed0 2024-06-09T22:19:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.299+0000 7f44ce8b9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561a745e8ba0 con 0x561a746fcf10 2024-06-09T22:19:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.299+0000 7f44ce8b9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561a74703b00 con 0x561a746fc5f0 2024-06-09T22:19:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.299+0000 7f44ce8b9b80 10 client.0 _ll_register_callbacks cb 0x561a745e28d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:19:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.299+0000 7f44c1dd2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746f6ed0 0x561a74702c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.299+0000 7f44c1dd2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746f6ed0 0x561a74702c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60646/0 (socket says 172.21.15.104:60646) 2024-06-09T22:19:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.299+0000 7f44c1dd2700 1 -- 172.21.15.104:0/620998565 learned_addr learned my addr 172.21.15.104:0/620998565 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44c25d3700 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746fcf10 0x561a747004a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44c15d1700 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fc5f0 0x561a746fff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44c1dd2700 1 -- 172.21.15.104:0/620998565 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fc5f0 msgr2=0x561a746fff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fc5f0 0x561a746fff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44c1dd2700 1 -- 172.21.15.104:0/620998565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746fcf10 msgr2=0x561a747004a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746fcf10 0x561a747004a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44c1dd2700 1 -- 172.21.15.104:0/620998565 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a74597880 con 0x561a746f6ed0 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746f6ed0 0x561a74702c60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f44b800edc0 tx=0x7f44b8015a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9de94dd20bc000 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44a57fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44a57fa700 1 -- 172.21.15.104:0/620998565 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b802d070 con 0x561a746f6ed0 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44a57fa700 1 -- 172.21.15.104:0/620998565 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f44b8013900 con 0x561a746f6ed0 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/620998565 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44b8024070 con 0x561a746f6ed0 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44a57fa700 1 -- 172.21.15.104:0/620998565 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f44b803b050 con 0x561a746f6ed0 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44ce8b9b80 10 client.15048 fetch_fsmap learned FSMap version 28 2024-06-09T22:19:56.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/620998565 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561a7470e290 con 0x561a746f6ed0 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.300+0000 7f44a57fa700 1 -- 172.21.15.104:0/620998565 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44b8024070 con 0x561a746f6ed0 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.301+0000 7f44a57fa700 1 -- 172.21.15.104:0/620998565 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f44b801e030 con 0x561a746f6ed0 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.301+0000 7f44ce8b9b80 10 client.15048 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.301+0000 7f44ce8b9b80 10 client.15048 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.301+0000 7f44ce8b9b80 20 client.15048 populate_metadata read hostname 'smithi104' 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.301+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x561a747104c0 0x561a74711930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.301+0000 7f44ce8b9b80 4 client.15048 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.301+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/620998565 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a74655cc0 con 0x561a747104c0 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.301+0000 7f44c15d1700 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x561a747104c0 0x561a74711930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.301+0000 7f44a57fa700 10 client.15048 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.301+0000 7f44c15d1700 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x561a747104c0 0x561a74711930 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=999583a4e797e5c7 server_cookie=2701995397ecb7f2 in_seq=0 out_seq=0 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.302+0000 7f44a57fa700 1 -- 172.21.15.104:0/620998565 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x561a74655cc0 con 0x561a747104c0 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.302+0000 7f44a57fa700 10 client.15048 handle_command_reply: tid=0 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.302+0000 7f44ce8b9b80 1 client.15048 shutdown 2024-06-09T22:19:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.302+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/620998565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746f6ed0 msgr2=0x561a74702c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:56.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.302+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746f6ed0 0x561a74702c60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f44b800edc0 tx=0x7f44b8015a20 comp rx=0 tx=0).stop 2024-06-09T22:19:56.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.302+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/620998565 shutdown_connections 2024-06-09T22:19:56.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.302+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x561a747104c0 0x561a74711930 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.302+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746f6ed0 0x561a74702c60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.302+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746fcf10 0x561a747004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.303+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/620998565 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fc5f0 0x561a746fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.302+0000 7f44a57fa700 0 client.15048 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:19:56.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.303+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/620998565 >> 172.21.15.104:0/620998565 conn(0x561a74621c70 msgr2=0x561a746f5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:56.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.303+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/620998565 shutdown_connections 2024-06-09T22:19:56.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.303+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/620998565 wait complete. 2024-06-09T22:19:56.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.303+0000 7f44ce8b9b80 20 client.15048 trim_cache size 0 max 16384 2024-06-09T22:19:56.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.323+0000 7f44ce8b9b80 1 Processor -- start 2024-06-09T22:19:56.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.323+0000 7f44ce8b9b80 1 -- start start 2024-06-09T22:19:56.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44ce8b9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746f6ed0 0x561a746f6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44ce8b9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fc5f0 0x561a746fd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44ce8b9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fcf10 0x561a746fd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44ce8b9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561a7470fc70 con 0x561a746fcf10 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44ce8b9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561a74710340 con 0x561a746f6ed0 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44ce8b9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561a74710a10 con 0x561a746fc5f0 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44c25d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fcf10 0x561a746fd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44c25d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fcf10 0x561a746fd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60650/0 (socket says 172.21.15.104:60650) 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44c25d3700 1 -- 172.21.15.104:0/3125444376 learned_addr learned my addr 172.21.15.104:0/3125444376 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44c25d3700 1 -- 172.21.15.104:0/3125444376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fc5f0 msgr2=0x561a746fd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44c15d1700 1 --2- 172.21.15.104:0/3125444376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fc5f0 0x561a746fd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/3125444376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746f6ed0 0x561a746f6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44c25d3700 1 --2- 172.21.15.104:0/3125444376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fc5f0 0x561a746fd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44c25d3700 1 -- 172.21.15.104:0/3125444376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746f6ed0 msgr2=0x561a746f6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:56.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44c25d3700 1 --2- 172.21.15.104:0/3125444376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746f6ed0 0x561a746f6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44c25d3700 1 -- 172.21.15.104:0/3125444376 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a746fe4d0 con 0x561a746fcf10 2024-06-09T22:19:56.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/3125444376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746f6ed0 0x561a746f6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:19:56.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44c25d3700 1 --2- 172.21.15.104:0/3125444376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fcf10 0x561a746fd830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f44b409ac30 tx=0x7f44b40ce040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912cb8a4d96c2f27 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:56.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44a77fe700 1 -- 172.21.15.104:0/3125444376 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b409f2e0 con 0x561a746fcf10 2024-06-09T22:19:56.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44a77fe700 1 -- 172.21.15.104:0/3125444376 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44b40a3ba0 con 0x561a746fcf10 2024-06-09T22:19:56.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.324+0000 7f44a77fe700 1 -- 172.21.15.104:0/3125444376 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b409b190 con 0x561a746fcf10 2024-06-09T22:19:56.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.325+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/3125444376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fcf10 msgr2=0x561a746fd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:56.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.325+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/3125444376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fcf10 0x561a746fd830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f44b409ac30 tx=0x7f44b40ce040 comp rx=0 tx=0).stop 2024-06-09T22:19:56.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.325+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/3125444376 shutdown_connections 2024-06-09T22:19:56.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.325+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/3125444376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fcf10 0x561a746fd830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.325+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/3125444376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746f6ed0 0x561a746f6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.325+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/3125444376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fc5f0 0x561a746fd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.325+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/3125444376 >> 172.21.15.104:0/3125444376 conn(0x561a74621c70 msgr2=0x561a746fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:56.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.325+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/3125444376 shutdown_connections 2024-06-09T22:19:56.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.325+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/3125444376 wait complete. 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.325+0000 7f44ce8b9b80 1 -- start start 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.325+0000 7f44ce8b9b80 1 Processor -- start 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44ce8b9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746f6ed0 0x561a746f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44ce8b9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 0x561a746f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44ce8b9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fcf10 0x561a746f6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44ce8b9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561a747117b0 con 0x561a746fc5f0 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44ce8b9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561a747110e0 con 0x561a746f6ed0 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44ce8b9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561a74711e80 con 0x561a746fcf10 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44ce8b9b80 10 client.0 _ll_register_callbacks cb 0x561a745e28d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44c15d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 0x561a746f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44c15d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 0x561a746f62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60664/0 (socket says 172.21.15.104:60664) 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44c15d1700 1 -- 172.21.15.104:0/2617619210 learned_addr learned my addr 172.21.15.104:0/2617619210 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746f6ed0 0x561a746f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44c25d3700 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fcf10 0x561a746f6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44c15d1700 1 -- 172.21.15.104:0/2617619210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fcf10 msgr2=0x561a746f6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44c15d1700 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fcf10 0x561a746f6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44c15d1700 1 -- 172.21.15.104:0/2617619210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746f6ed0 msgr2=0x561a746f5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44c15d1700 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746f6ed0 0x561a746f5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44c15d1700 1 -- 172.21.15.104:0/2617619210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a74719430 con 0x561a746fc5f0 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.326+0000 7f44c25d3700 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fcf10 0x561a746f6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44c15d1700 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 0x561a746f62c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0098e0 tx=0x7f44ac005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48046f05164595c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44a57fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44a57fa700 1 -- 172.21.15.104:0/2617619210 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ac003d70 con 0x561a746fc5f0 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44a57fa700 1 -- 172.21.15.104:0/2617619210 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f44ac0050b0 con 0x561a746fc5f0 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2617619210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44b40a3ba0 con 0x561a746fc5f0 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44a57fa700 1 -- 172.21.15.104:0/2617619210 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f44ac03a050 con 0x561a746fc5f0 2024-06-09T22:19:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44ce8b9b80 10 client.15054 fetch_fsmap learned FSMap version 28 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2617619210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561a7477c0f0 con 0x561a746fc5f0 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44a57fa700 1 -- 172.21.15.104:0/2617619210 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44b40a3ba0 con 0x561a746fc5f0 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44a57fa700 1 -- 172.21.15.104:0/2617619210 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f44ac035030 con 0x561a746fc5f0 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44ce8b9b80 10 client.15054 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44ce8b9b80 10 client.15054 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44ce8b9b80 20 client.15054 populate_metadata read hostname 'smithi104' 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x561a74780d30 0x561a7477fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44ce8b9b80 4 client.15054 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.327+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2617619210 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "status"}) -- 0x561a74650240 con 0x561a74780d30 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x561a74780d30 0x561a7477fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44c1dd2700 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x561a74780d30 0x561a7477fa40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=721204d431af9e88 server_cookie=4c406a9301295c4e in_seq=0 out_seq=0 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44a57fa700 10 client.15054 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44a57fa700 1 -- 172.21.15.104:0/2617619210 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561a74650240 con 0x561a74780d30 2024-06-09T22:19:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44a57fa700 10 client.15054 handle_command_reply: tid=0 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44ce8b9b80 1 client.15054 shutdown 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2617619210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 msgr2=0x561a746f62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 0x561a746f62c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0098e0 tx=0x7f44ac005c00 comp rx=0 tx=0).stop 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2617619210 shutdown_connections 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561a746fcf10 0x561a746f6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a746f6ed0 0x561a746f5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x561a74780d30 0x561a7477fa40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44ce8b9b80 1 --2- 172.21.15.104:0/2617619210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561a746fc5f0 0x561a746f62c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44a57fa700 0 client.15054 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2617619210 >> 172.21.15.104:0/2617619210 conn(0x561a74621c70 msgr2=0x561a746fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2617619210 shutdown_connections 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.328+0000 7f44ce8b9b80 1 -- 172.21.15.104:0/2617619210 wait complete. 2024-06-09T22:19:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44ce8b9b80 20 client.15054 trim_cache size 0 max 16384 2024-06-09T22:19:56.333 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:19:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:19:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 3, 2024-06-09T22:19:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24419, 2024-06-09T22:19:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:19:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:19:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:19:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 126.897637667, 2024-06-09T22:19:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 28, 2024-06-09T22:19:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 99, 2024-06-09T22:19:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 97, 2024-06-09T22:19:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 206.32209480099999 2024-06-09T22:19:56.334 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:19:56.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44c4036700 1 -- 172.21.15.104:0/874984197 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f44a80a2ca0 msgr2=0x7f44a80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:56.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44c4036700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f44a80a2ca0 0x7f44a80a5160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f44b8009a10 tx=0x7f44b8009190 comp rx=0 tx=0).stop 2024-06-09T22:19:56.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44c4036700 1 -- 172.21.15.104:0/874984197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44bc107840 msgr2=0x7f44bc19f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:56.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44c4036700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44bc107840 0x7f44bc19f6f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f44b4007ee0 tx=0x7f44b400d3b0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44c4036700 1 -- 172.21.15.104:0/874984197 shutdown_connections 2024-06-09T22:19:56.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44c4036700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f44a80a2ca0 0x7f44a80a5160 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44c4036700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44bc107840 0x7f44bc19f6f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44c4036700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44bc102740 0x7f44bc19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44c4036700 1 --2- 172.21.15.104:0/874984197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc1030f0 0x7f44bc19b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:56.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44c4036700 1 -- 172.21.15.104:0/874984197 >> 172.21.15.104:0/874984197 conn(0x7f44bc0795f0 msgr2=0x7f44bc108990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:56.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44c4036700 1 -- 172.21.15.104:0/874984197 shutdown_connections 2024-06-09T22:19:56.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:56.329+0000 7f44c4036700 1 -- 172.21.15.104:0/874984197 wait complete. 2024-06-09T22:19:56.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:56 smithi153 ceph-mon[36599]: pgmap v404: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:56.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:56 smithi104 ceph-mon[32389]: pgmap v404: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:56.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:56 smithi194 ceph-mon[36856]: pgmap v404: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:19:57.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.f status 2024-06-09T22:19:57.376 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:19:58.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:19:58 smithi153 ceph-mon[36599]: pgmap v405: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:19:58.751 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:19:58 smithi104 ceph-mon[32389]: pgmap v405: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:19:58.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:19:58 smithi194 ceph-mon[36856]: pgmap v405: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:19:59.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.124+0000 7fe2c58d0700 1 -- 172.21.15.104:0/3982669761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c0101390 msgr2=0x7fe2c0101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:59.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.124+0000 7fe2c58d0700 1 --2- 172.21.15.104:0/3982669761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c0101390 0x7fe2c0101810 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009990 tx=0x7fe2b4009ca0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.124+0000 7fe2c58d0700 1 -- 172.21.15.104:0/3982669761 shutdown_connections 2024-06-09T22:19:59.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.124+0000 7fe2c58d0700 1 --2- 172.21.15.104:0/3982669761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe2c0101d50 0x7fe2c010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.124+0000 7fe2c58d0700 1 --2- 172.21.15.104:0/3982669761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2c0107400 0x7fe2c01077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.124+0000 7fe2c58d0700 1 --2- 172.21.15.104:0/3982669761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c0101390 0x7fe2c0101810 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.124+0000 7fe2c58d0700 1 -- 172.21.15.104:0/3982669761 >> 172.21.15.104:0/3982669761 conn(0x7fe2c0078340 msgr2=0x7fe2c0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:59.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.125+0000 7fe2c58d0700 1 -- 172.21.15.104:0/3982669761 shutdown_connections 2024-06-09T22:19:59.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.125+0000 7fe2c58d0700 1 -- 172.21.15.104:0/3982669761 wait complete. 2024-06-09T22:19:59.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.125+0000 7fe2c58d0700 1 Processor -- start 2024-06-09T22:19:59.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.125+0000 7fe2c58d0700 1 -- start start 2024-06-09T22:19:59.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2c58d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c0101390 0x7fe2c019e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2c58d0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2c0101d50 0x7fe2c019ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2c58d0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe2c0107400 0x7fe2c0198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2c58d0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c0111720 con 0x7fe2c0101390 2024-06-09T22:19:59.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2c58d0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c01115a0 con 0x7fe2c0101d50 2024-06-09T22:19:59.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2c58d0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c01118a0 con 0x7fe2c0107400 2024-06-09T22:19:59.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c0101390 0x7fe2c019e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c0101390 0x7fe2c019e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60680/0 (socket says 172.21.15.104:60680) 2024-06-09T22:19:59.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2beffd700 1 -- 172.21.15.104:0/487705825 learned_addr learned my addr 172.21.15.104:0/487705825 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:59.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2be7fc700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2c0101d50 0x7fe2c019ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2bf7fe700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe2c0107400 0x7fe2c0198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2beffd700 1 -- 172.21.15.104:0/487705825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe2c0107400 msgr2=0x7fe2c0198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:59.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2beffd700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe2c0107400 0x7fe2c0198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2beffd700 1 -- 172.21.15.104:0/487705825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2c0101d50 msgr2=0x7fe2c019ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:59.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2beffd700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2c0101d50 0x7fe2c019ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2beffd700 1 -- 172.21.15.104:0/487705825 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2c0199280 con 0x7fe2c0101390 2024-06-09T22:19:59.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2beffd700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c0101390 0x7fe2c019e910 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000b530 tx=0x7fe2b000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:59.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2c48ce700 1 -- 172.21.15.104:0/487705825 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0015020 con 0x7fe2c0101390 2024-06-09T22:19:59.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2c58d0700 1 -- 172.21.15.104:0/487705825 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2c0199570 con 0x7fe2c0101390 2024-06-09T22:19:59.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2c48ce700 1 -- 172.21.15.104:0/487705825 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2b0011d10 con 0x7fe2c0101390 2024-06-09T22:19:59.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2c48ce700 1 -- 172.21.15.104:0/487705825 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b000f2e0 con 0x7fe2c0101390 2024-06-09T22:19:59.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.126+0000 7fe2c58d0700 1 -- 172.21.15.104:0/487705825 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2c01a57e0 con 0x7fe2c0101390 2024-06-09T22:19:59.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.129+0000 7fe2c48ce700 1 -- 172.21.15.104:0/487705825 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fe2b0018020 con 0x7fe2c0101390 2024-06-09T22:19:59.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.129+0000 7fe2c48ce700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe2a80a2c20 0x7fe2a80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.129+0000 7fe2be7fc700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe2a80a2c20 0x7fe2a80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.129+0000 7fe2c48ce700 1 -- 172.21.15.104:0/487705825 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fe2b00cec30 con 0x7fe2c0101390 2024-06-09T22:19:59.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2be7fc700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe2a80a2c20 0x7fe2a80a50e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400b5c0 tx=0x7fe2b4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:59.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2d0153b80 1 Processor -- start 2024-06-09T22:19:59.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2d0153b80 1 -- start start 2024-06-09T22:19:59.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2d0153b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f115f0 0x564495f119d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2d0153b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f11f10 0x564495f0b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2d0153b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f0bed0 0x564495f0c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2d0153b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564495e5d7c0 con 0x564495f115f0 2024-06-09T22:19:59.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2d0153b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564495e056d0 con 0x564495f11f10 2024-06-09T22:19:59.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2d0153b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564495e59780 con 0x564495f0bed0 2024-06-09T22:19:59.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f115f0 0x564495f119d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f115f0 0x564495f119d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60688/0 (socket says 172.21.15.104:60688) 2024-06-09T22:19:59.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2beffd700 1 -- 172.21.15.104:0/884187440 learned_addr learned my addr 172.21.15.104:0/884187440 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:59.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2bf7fe700 1 --2- 172.21.15.104:0/884187440 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f0bed0 0x564495f0c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.130+0000 7fe2be7fc700 1 --2- 172.21.15.104:0/884187440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f11f10 0x564495f0b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.131+0000 7fe2beffd700 1 -- 172.21.15.104:0/884187440 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f0bed0 msgr2=0x564495f0c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:59.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.131+0000 7fe2beffd700 1 --2- 172.21.15.104:0/884187440 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f0bed0 0x564495f0c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.131+0000 7fe2beffd700 1 -- 172.21.15.104:0/884187440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f11f10 msgr2=0x564495f0b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.131+0000 7fe2beffd700 1 --2- 172.21.15.104:0/884187440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f11f10 0x564495f0b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.131+0000 7fe2beffd700 1 -- 172.21.15.104:0/884187440 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564495dfbf00 con 0x564495f115f0 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.131+0000 7fe2beffd700 1 --2- 172.21.15.104:0/884187440 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f115f0 0x564495f119d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0097560 tx=0x7fe2b009aa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee59db32e143acc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.131+0000 7fe2a5ffb700 1 -- 172.21.15.104:0/884187440 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b009b100 con 0x564495f115f0 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.131+0000 7fe2a5ffb700 1 -- 172.21.15.104:0/884187440 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2b009f4b0 con 0x564495f115f0 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.131+0000 7fe2a5ffb700 1 -- 172.21.15.104:0/884187440 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b009f160 con 0x564495f115f0 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.132+0000 7fe2d0153b80 1 -- 172.21.15.104:0/884187440 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f115f0 msgr2=0x564495f119d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.132+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/884187440 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f115f0 0x564495f119d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0097560 tx=0x7fe2b009aa60 comp rx=0 tx=0).stop 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.132+0000 7fe2d0153b80 1 -- 172.21.15.104:0/884187440 shutdown_connections 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.132+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/884187440 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f115f0 0x564495f119d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.132+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/884187440 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f0bed0 0x564495f0c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.132+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/884187440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f11f10 0x564495f0b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.132+0000 7fe2d0153b80 1 -- 172.21.15.104:0/884187440 >> 172.21.15.104:0/884187440 conn(0x564495e36c70 msgr2=0x564495f06800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:59.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.132+0000 7fe2d0153b80 1 -- 172.21.15.104:0/884187440 shutdown_connections 2024-06-09T22:19:59.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.132+0000 7fe2d0153b80 1 -- 172.21.15.104:0/884187440 wait complete. 2024-06-09T22:19:59.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.132+0000 7fe2d0153b80 1 -- start start 2024-06-09T22:19:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.132+0000 7fe2d0153b80 1 Processor -- start 2024-06-09T22:19:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2d0153b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f17c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2d0153b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f115f0 0x564495f14f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2d0153b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f11f10 0x564495f15490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2d0153b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564495f1cd90 con 0x564495f0bed0 2024-06-09T22:19:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2d0153b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564495dfdba0 con 0x564495f11f10 2024-06-09T22:19:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2d0153b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564495f1cf10 con 0x564495f115f0 2024-06-09T22:19:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2d0153b80 10 client.0 _ll_register_callbacks cb 0x564495df78d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:19:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f17c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f17c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60694/0 (socket says 172.21.15.104:60694) 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2beffd700 1 -- 172.21.15.104:0/1511774813 learned_addr learned my addr 172.21.15.104:0/1511774813 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2be7fc700 1 --2- 172.21.15.104:0/1511774813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f115f0 0x564495f14f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2beffd700 1 -- 172.21.15.104:0/1511774813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f115f0 msgr2=0x564495f14f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2beffd700 1 --2- 172.21.15.104:0/1511774813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f115f0 0x564495f14f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2beffd700 1 -- 172.21.15.104:0/1511774813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f11f10 msgr2=0x564495f15490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2beffd700 1 --2- 172.21.15.104:0/1511774813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f11f10 0x564495f15490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2beffd700 1 -- 172.21.15.104:0/1511774813 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564495dac880 con 0x564495f0bed0 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2beffd700 1 --2- 172.21.15.104:0/1511774813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f17c50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe2b009adb0 tx=0x7fe2b009b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1bd38539487b5bb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe29effd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe29effd700 1 -- 172.21.15.104:0/1511774813 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b009f160 con 0x564495f0bed0 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe29effd700 1 -- 172.21.15.104:0/1511774813 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fe2b00bb0b0 con 0x564495f0bed0 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe2d0153b80 1 -- 172.21.15.104:0/1511774813 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2b009f4b0 con 0x564495f0bed0 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.133+0000 7fe29effd700 1 -- 172.21.15.104:0/1511774813 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe2b00660d0 con 0x564495f0bed0 2024-06-09T22:19:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.134+0000 7fe2d0153b80 10 client.15066 fetch_fsmap learned FSMap version 28 2024-06-09T22:19:59.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.134+0000 7fe2d0153b80 1 -- 172.21.15.104:0/1511774813 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564495f1cd90 con 0x564495f0bed0 2024-06-09T22:19:59.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.134+0000 7fe29effd700 1 -- 172.21.15.104:0/1511774813 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2b009f4b0 con 0x564495f0bed0 2024-06-09T22:19:59.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.134+0000 7fe29effd700 1 -- 172.21.15.104:0/1511774813 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe2b00a38e0 con 0x564495f0bed0 2024-06-09T22:19:59.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.134+0000 7fe2d0153b80 10 client.15066 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:19:59.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.134+0000 7fe2d0153b80 10 client.15066 resolve_mds: resolved name 'f' to daemon mds.f 2024-06-09T22:19:59.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.134+0000 7fe2d0153b80 20 client.15066 populate_metadata read hostname 'smithi104' 2024-06-09T22:19:59.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.134+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/1511774813 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564495f290f0 0x564495f2abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.134+0000 7fe2d0153b80 4 client.15066 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:19:59.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.134+0000 7fe2d0153b80 1 -- 172.21.15.104:0/1511774813 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564495e6acc0 con 0x564495f290f0 2024-06-09T22:19:59.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.134+0000 7fe2be7fc700 1 --2- 172.21.15.104:0/1511774813 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564495f290f0 0x564495f2abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.135+0000 7fe29effd700 10 client.15066 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.135+0000 7fe2be7fc700 1 --2- 172.21.15.104:0/1511774813 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564495f290f0 0x564495f2abc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5cbdd985753196d1 server_cookie=c9149ef8ad527c14 in_seq=0 out_seq=0 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.135+0000 7fe29effd700 1 -- 172.21.15.104:0/1511774813 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x564495e6acc0 con 0x564495f290f0 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.135+0000 7fe29effd700 10 client.15066 handle_command_reply: tid=0 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe2d0153b80 1 client.15066 shutdown 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe2d0153b80 1 -- 172.21.15.104:0/1511774813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 msgr2=0x564495f17c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/1511774813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f17c50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe2b009adb0 tx=0x7fe2b009b1b0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe2d0153b80 1 -- 172.21.15.104:0/1511774813 shutdown_connections 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/1511774813 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564495f290f0 0x564495f2abc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/1511774813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f17c50 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/1511774813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f11f10 0x564495f15490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/1511774813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f115f0 0x564495f14f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe2d0153b80 1 -- 172.21.15.104:0/1511774813 >> 172.21.15.104:0/1511774813 conn(0x564495e36c70 msgr2=0x564495f065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:59.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe29effd700 0 client.15066 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:19:59.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe2d0153b80 1 -- 172.21.15.104:0/1511774813 shutdown_connections 2024-06-09T22:19:59.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe2d0153b80 1 -- 172.21.15.104:0/1511774813 wait complete. 2024-06-09T22:19:59.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.136+0000 7fe2d0153b80 20 client.15066 trim_cache size 0 max 16384 2024-06-09T22:19:59.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.157+0000 7fe2d0153b80 1 Processor -- start 2024-06-09T22:19:59.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.157+0000 7fe2d0153b80 1 -- start start 2024-06-09T22:19:59.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.157+0000 7fe2d0153b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f0b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.157+0000 7fe2d0153b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f115f0 0x564495f12a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2d0153b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f11f10 0x564495effdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2d0153b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564495f28f70 con 0x564495f0bed0 2024-06-09T22:19:59.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2d0153b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564495f295d0 con 0x564495f115f0 2024-06-09T22:19:59.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2d0153b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564495f29ca0 con 0x564495f11f10 2024-06-09T22:19:59.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f0b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f0b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60698/0 (socket says 172.21.15.104:60698) 2024-06-09T22:19:59.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2beffd700 1 -- 172.21.15.104:0/706782554 learned_addr learned my addr 172.21.15.104:0/706782554 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:59.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2bf7fe700 1 --2- 172.21.15.104:0/706782554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f11f10 0x564495effdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2beffd700 1 -- 172.21.15.104:0/706782554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f11f10 msgr2=0x564495effdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:59.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2beffd700 1 --2- 172.21.15.104:0/706782554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f11f10 0x564495effdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2beffd700 1 -- 172.21.15.104:0/706782554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f115f0 msgr2=0x564495f12a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2beffd700 1 --2- 172.21.15.104:0/706782554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f115f0 0x564495f12a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2beffd700 1 -- 172.21.15.104:0/706782554 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564495f134c0 con 0x564495f0bed0 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2bf7fe700 1 --2- 172.21.15.104:0/706782554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f11f10 0x564495effdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2beffd700 1 --2- 172.21.15.104:0/706782554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f0b990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00993b0 tx=0x7fe2b0010a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5459500a15c9ab4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2a57fa700 1 -- 172.21.15.104:0/706782554 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0084690 con 0x564495f0bed0 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2a57fa700 1 -- 172.21.15.104:0/706782554 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2b009f4b0 con 0x564495f0bed0 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.158+0000 7fe2a57fa700 1 -- 172.21.15.104:0/706782554 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b009aa60 con 0x564495f0bed0 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.159+0000 7fe2d0153b80 1 -- 172.21.15.104:0/706782554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 msgr2=0x564495f0b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.159+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/706782554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f0b990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00993b0 tx=0x7fe2b0010a10 comp rx=0 tx=0).stop 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.159+0000 7fe2d0153b80 1 -- 172.21.15.104:0/706782554 shutdown_connections 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.159+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/706782554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f0b990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.159+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/706782554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f11f10 0x564495effdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.159+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/706782554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f115f0 0x564495f12a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.159+0000 7fe2d0153b80 1 -- 172.21.15.104:0/706782554 >> 172.21.15.104:0/706782554 conn(0x564495e36c70 msgr2=0x564495f122f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:59.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.159+0000 7fe2d0153b80 1 -- 172.21.15.104:0/706782554 shutdown_connections 2024-06-09T22:19:59.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.159+0000 7fe2d0153b80 1 -- 172.21.15.104:0/706782554 wait complete. 2024-06-09T22:19:59.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.159+0000 7fe2d0153b80 1 -- start start 2024-06-09T22:19:59.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2d0153b80 1 Processor -- start 2024-06-09T22:19:59.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2d0153b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f34800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2d0153b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f115f0 0x564495f0a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2d0153b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f11f10 0x564495f0aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2d0153b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564495f2aa40 con 0x564495f0bed0 2024-06-09T22:19:59.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2d0153b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564495f2a370 con 0x564495f115f0 2024-06-09T22:19:59.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2d0153b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564495f2b110 con 0x564495f11f10 2024-06-09T22:19:59.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2d0153b80 10 client.0 _ll_register_callbacks cb 0x564495df78d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:19:59.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f34800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f34800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60700/0 (socket says 172.21.15.104:60700) 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2beffd700 1 -- 172.21.15.104:0/3610764294 learned_addr learned my addr 172.21.15.104:0/3610764294 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2be7fc700 1 --2- 172.21.15.104:0/3610764294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f115f0 0x564495f0a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2beffd700 1 -- 172.21.15.104:0/3610764294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f11f10 msgr2=0x564495f0aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2beffd700 1 --2- 172.21.15.104:0/3610764294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f11f10 0x564495f0aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2beffd700 1 -- 172.21.15.104:0/3610764294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f115f0 msgr2=0x564495f0a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2beffd700 1 --2- 172.21.15.104:0/3610764294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f115f0 0x564495f0a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2beffd700 1 -- 172.21.15.104:0/3610764294 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564495f0b700 con 0x564495f0bed0 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe2beffd700 1 --2- 172.21.15.104:0/3610764294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f34800 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00a3840 tx=0x7fe2b009ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd34721727c272f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.160+0000 7fe29effd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe29effd700 1 -- 172.21.15.104:0/3610764294 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b009aa60 con 0x564495f0bed0 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe29effd700 1 -- 172.21.15.104:0/3610764294 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fe2b009b360 con 0x564495f0bed0 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe2d0153b80 1 -- 172.21.15.104:0/3610764294 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2b009f4b0 con 0x564495f0bed0 2024-06-09T22:19:59.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe29effd700 1 -- 172.21.15.104:0/3610764294 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe2b00a3d70 con 0x564495f0bed0 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe2d0153b80 10 client.15072 fetch_fsmap learned FSMap version 28 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe2d0153b80 1 -- 172.21.15.104:0/3610764294 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564495f0b700 con 0x564495f0bed0 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe29effd700 1 -- 172.21.15.104:0/3610764294 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2b009f4b0 con 0x564495f0bed0 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe29effd700 1 -- 172.21.15.104:0/3610764294 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe2b0096fe0 con 0x564495f0bed0 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe2d0153b80 10 client.15072 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe2d0153b80 10 client.15072 resolve_mds: resolved name 'f' to daemon mds.f 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe2d0153b80 20 client.15072 populate_metadata read hostname 'smithi104' 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/3610764294 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564495f99530 0x564495f98240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe2d0153b80 4 client.15072 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe2d0153b80 1 -- 172.21.15.104:0/3610764294 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "status"}) -- 0x564495e65240 con 0x564495f99530 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.161+0000 7fe2be7fc700 1 --2- 172.21.15.104:0/3610764294 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564495f99530 0x564495f98240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe29effd700 10 client.15072 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe2be7fc700 1 --2- 172.21.15.104:0/3610764294 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564495f99530 0x564495f98240 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c10465a48b4be587 server_cookie=5c739921e0c80f42 in_seq=0 out_seq=0 2024-06-09T22:19:59.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe29effd700 1 -- 172.21.15.104:0/3610764294 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564495e65240 con 0x564495f99530 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe29effd700 10 client.15072 handle_command_reply: tid=0 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe2d0153b80 1 client.15072 shutdown 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe2d0153b80 1 -- 172.21.15.104:0/3610764294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 msgr2=0x564495f34800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/3610764294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f34800 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00a3840 tx=0x7fe2b009ad60 comp rx=0 tx=0).stop 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe2d0153b80 1 -- 172.21.15.104:0/3610764294 shutdown_connections 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/3610764294 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564495f99530 0x564495f98240 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/3610764294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564495f0bed0 0x564495f34800 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/3610764294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564495f11f10 0x564495f0aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe29effd700 0 client.15072 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe2d0153b80 1 --2- 172.21.15.104:0/3610764294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564495f115f0 0x564495f0a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe2d0153b80 1 -- 172.21.15.104:0/3610764294 >> 172.21.15.104:0/3610764294 conn(0x564495e36c70 msgr2=0x564495effdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe2d0153b80 1 -- 172.21.15.104:0/3610764294 shutdown_connections 2024-06-09T22:19:59.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.162+0000 7fe2d0153b80 1 -- 172.21.15.104:0/3610764294 wait complete. 2024-06-09T22:19:59.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2d0153b80 20 client.15072 trim_cache size 0 max 16384 2024-06-09T22:19:59.167 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:19:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:19:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 2, 2024-06-09T22:19:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24442, 2024-06-09T22:19:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:19:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:19:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:19:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 130.74413453299999, 2024-06-09T22:19:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 28, 2024-06-09T22:19:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 99, 2024-06-09T22:19:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 97, 2024-06-09T22:19:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 228.06934908299999 2024-06-09T22:19:59.168 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:19:59.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2c58d0700 1 -- 172.21.15.104:0/487705825 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe2a80a2c20 msgr2=0x7fe2a80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:59.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2c58d0700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe2a80a2c20 0x7fe2a80a50e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400b5c0 tx=0x7fe2b4005960 comp rx=0 tx=0).stop 2024-06-09T22:19:59.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2c58d0700 1 -- 172.21.15.104:0/487705825 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c0101390 msgr2=0x7fe2c019e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:19:59.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2c58d0700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c0101390 0x7fe2c019e910 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000b530 tx=0x7fe2b000b840 comp rx=0 tx=0).stop 2024-06-09T22:19:59.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2c58d0700 1 -- 172.21.15.104:0/487705825 shutdown_connections 2024-06-09T22:19:59.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2c58d0700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe2a80a2c20 0x7fe2a80a50e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2c58d0700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2c0101390 0x7fe2c019e910 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2c58d0700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe2c0107400 0x7fe2c0198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2c58d0700 1 --2- 172.21.15.104:0/487705825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2c0101d50 0x7fe2c019ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:19:59.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2c58d0700 1 -- 172.21.15.104:0/487705825 >> 172.21.15.104:0/487705825 conn(0x7fe2c0078340 msgr2=0x7fe2c0100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:19:59.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2c58d0700 1 -- 172.21.15.104:0/487705825 shutdown_connections 2024-06-09T22:19:59.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:19:59.163+0000 7fe2c58d0700 1 -- 172.21.15.104:0/487705825 wait complete. 2024-06-09T22:19:59.956 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.c status 2024-06-09T22:20:00.200 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:00.589 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:00 smithi104 ceph-mon[32389]: pgmap v406: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:00.589 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:00 smithi104 ceph-mon[32389]: overall HEALTH_OK 2024-06-09T22:20:00.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:00 smithi153 ceph-mon[36599]: pgmap v406: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:00.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:00 smithi153 ceph-mon[36599]: overall HEALTH_OK 2024-06-09T22:20:00.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:00 smithi194 ceph-mon[36856]: pgmap v406: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:00.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:00 smithi194 ceph-mon[36856]: overall HEALTH_OK 2024-06-09T22:20:01.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.935+0000 7f1058665700 1 -- 172.21.15.104:0/4024329290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1050101510 msgr2=0x7f105010d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.935+0000 7f1058665700 1 --2- 172.21.15.104:0/4024329290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1050101510 0x7f105010d9d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f10480099c0 tx=0x7f1048009cd0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.935+0000 7f1058665700 1 -- 172.21.15.104:0/4024329290 shutdown_connections 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.935+0000 7f1058665700 1 --2- 172.21.15.104:0/4024329290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1050101510 0x7f105010d9d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.935+0000 7f1058665700 1 --2- 172.21.15.104:0/4024329290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1050106c00 0x7f1050106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.935+0000 7f1058665700 1 --2- 172.21.15.104:0/4024329290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1050100b50 0x7f1050100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.935+0000 7f1058665700 1 -- 172.21.15.104:0/4024329290 >> 172.21.15.104:0/4024329290 conn(0x7f1050077990 msgr2=0x7f10500fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.935+0000 7f1058665700 1 -- 172.21.15.104:0/4024329290 shutdown_connections 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.935+0000 7f1058665700 1 -- 172.21.15.104:0/4024329290 wait complete. 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.936+0000 7f1058665700 1 Processor -- start 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.936+0000 7f1058665700 1 -- start start 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.936+0000 7f1058665700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1050100b50 0x7f105019d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.936+0000 7f1058665700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1050101510 0x7f105019dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.936+0000 7f1058665700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1050106c00 0x7f1050197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1058665700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1050111730 con 0x7f1050100b50 2024-06-09T22:20:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1058665700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f10501115b0 con 0x7f1050106c00 2024-06-09T22:20:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1058665700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f10501118b0 con 0x7f1050101510 2024-06-09T22:20:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1056401700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1050100b50 0x7f105019d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1056401700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1050100b50 0x7f105019d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60724/0 (socket says 172.21.15.104:60724) 2024-06-09T22:20:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1056401700 1 -- 172.21.15.104:0/1122153284 learned_addr learned my addr 172.21.15.104:0/1122153284 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1056c02700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1050106c00 0x7f1050197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1055c00700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1050101510 0x7f105019dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1056401700 1 -- 172.21.15.104:0/1122153284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1050101510 msgr2=0x7f105019dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1056401700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1050101510 0x7f105019dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1056401700 1 -- 172.21.15.104:0/1122153284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1050106c00 msgr2=0x7f1050197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1056401700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1050106c00 0x7f1050197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1056401700 1 -- 172.21.15.104:0/1122153284 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1050198030 con 0x7f1050100b50 2024-06-09T22:20:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1055c00700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1050101510 0x7f105019dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1056401700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1050100b50 0x7f105019d6d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f10400060b0 tx=0x7f104000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f104f7fe700 1 -- 172.21.15.104:0/1122153284 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104001a070 con 0x7f1050100b50 2024-06-09T22:20:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f104f7fe700 1 -- 172.21.15.104:0/1122153284 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f104000ddd0 con 0x7f1050100b50 2024-06-09T22:20:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1058665700 1 -- 172.21.15.104:0/1122153284 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1050198320 con 0x7f1050100b50 2024-06-09T22:20:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f104f7fe700 1 -- 172.21.15.104:0/1122153284 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10400147a0 con 0x7f1050100b50 2024-06-09T22:20:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.937+0000 7f1058665700 1 -- 172.21.15.104:0/1122153284 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10501a4530 con 0x7f1050100b50 2024-06-09T22:20:01.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.940+0000 7f104f7fe700 1 -- 172.21.15.104:0/1122153284 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f104001d070 con 0x7f1050100b50 2024-06-09T22:20:01.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.940+0000 7f104f7fe700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f10380a2c20 0x7f10380a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.940+0000 7f1055c00700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f10380a2c20 0x7f10380a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.940+0000 7f104f7fe700 1 -- 172.21.15.104:0/1122153284 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f10400cec00 con 0x7f1050100b50 2024-06-09T22:20:01.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.941+0000 7f1055c00700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f10380a2c20 0x7f10380a50e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1044005fd0 tx=0x7f1044005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:01.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.941+0000 7f1062ee8b80 1 Processor -- start 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.941+0000 7f1062ee8b80 1 -- start start 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.941+0000 7f1062ee8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40120 0x55fa05d40500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.941+0000 7f1062ee8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40a40 0x55fa05d3a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.941+0000 7f1062ee8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d3ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.941+0000 7f1062ee8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fa05c2af10 con 0x55fa05d40a40 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.941+0000 7f1062ee8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fa05c89800 con 0x55fa05d3a960 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.941+0000 7f1062ee8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fa05bf1410 con 0x55fa05d40120 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f1055c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40a40 0x55fa05d3a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f1055c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40a40 0x55fa05d3a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60726/0 (socket says 172.21.15.104:60726) 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f1055c00700 1 -- 172.21.15.104:0/4203885341 learned_addr learned my addr 172.21.15.104:0/4203885341 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f1056401700 1 --2- 172.21.15.104:0/4203885341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40120 0x55fa05d40500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f1056c02700 1 --2- 172.21.15.104:0/4203885341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d3ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f1055c00700 1 -- 172.21.15.104:0/4203885341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40120 msgr2=0x55fa05d40500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:01.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f1055c00700 1 --2- 172.21.15.104:0/4203885341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40120 0x55fa05d40500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f1055c00700 1 -- 172.21.15.104:0/4203885341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 msgr2=0x55fa05d3ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f1055c00700 1 --2- 172.21.15.104:0/4203885341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d3ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f1055c00700 1 -- 172.21.15.104:0/4203885341 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa05bd8880 con 0x55fa05d40a40 2024-06-09T22:20:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f1055c00700 1 --2- 172.21.15.104:0/4203885341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40a40 0x55fa05d3a420 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f104400ff00 tx=0x7f1044012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=777bf04c75cc5e6b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f104cff9700 1 -- 172.21.15.104:0/4203885341 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044022070 con 0x55fa05d40a40 2024-06-09T22:20:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f104cff9700 1 -- 172.21.15.104:0/4203885341 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1044011c60 con 0x55fa05d40a40 2024-06-09T22:20:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.942+0000 7f104cff9700 1 -- 172.21.15.104:0/4203885341 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104401d5f0 con 0x55fa05d40a40 2024-06-09T22:20:01.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.943+0000 7f1062ee8b80 1 -- 172.21.15.104:0/4203885341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40a40 msgr2=0x55fa05d3a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.943+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/4203885341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40a40 0x55fa05d3a420 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f104400ff00 tx=0x7f1044012000 comp rx=0 tx=0).stop 2024-06-09T22:20:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.943+0000 7f1062ee8b80 1 -- 172.21.15.104:0/4203885341 shutdown_connections 2024-06-09T22:20:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.943+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/4203885341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d3ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.943+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/4203885341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40120 0x55fa05d40500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.943+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/4203885341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40a40 0x55fa05d3a420 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.943+0000 7f1062ee8b80 1 -- 172.21.15.104:0/4203885341 >> 172.21.15.104:0/4203885341 conn(0x55fa05c65030 msgr2=0x55fa05d331d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.943+0000 7f1062ee8b80 1 -- 172.21.15.104:0/4203885341 shutdown_connections 2024-06-09T22:20:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.943+0000 7f1062ee8b80 1 -- 172.21.15.104:0/4203885341 wait complete. 2024-06-09T22:20:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.943+0000 7f1062ee8b80 1 -- start start 2024-06-09T22:20:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1062ee8b80 1 Processor -- start 2024-06-09T22:20:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1062ee8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d46790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1062ee8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d43a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1062ee8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40a40 0x55fa05d43fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1062ee8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fa05c29f70 con 0x55fa05d40120 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1062ee8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fa05c857c0 con 0x55fa05d3a960 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1062ee8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fa05d47430 con 0x55fa05d40a40 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1062ee8b80 10 client.0 _ll_register_callbacks cb 0x55fa05c1f560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1055c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d43a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1055c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d43a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60730/0 (socket says 172.21.15.104:60730) 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1055c00700 1 -- 172.21.15.104:0/2674956276 learned_addr learned my addr 172.21.15.104:0/2674956276 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1056401700 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d46790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1056c02700 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40a40 0x55fa05d43fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1055c00700 1 -- 172.21.15.104:0/2674956276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40a40 msgr2=0x55fa05d43fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1055c00700 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40a40 0x55fa05d43fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1055c00700 1 -- 172.21.15.104:0/2674956276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 msgr2=0x55fa05d46790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1055c00700 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d46790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1055c00700 1 -- 172.21.15.104:0/2674956276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa05c4f700 con 0x55fa05d40120 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.944+0000 7f1056c02700 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40a40 0x55fa05d43fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1055c00700 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d43a90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f104401ee90 tx=0x7f1044011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a865234270eec server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1035ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1035ffb700 1 -- 172.21.15.104:0/2674956276 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044022070 con 0x55fa05d40120 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1035ffb700 1 -- 172.21.15.104:0/2674956276 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f1044005480 con 0x55fa05d40120 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2674956276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1044011c60 con 0x55fa05d40120 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1035ffb700 1 -- 172.21.15.104:0/2674956276 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f104403c050 con 0x55fa05d40120 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1062ee8b80 10 client.15084 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2674956276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fa05d51b90 con 0x55fa05d40120 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1035ffb700 1 -- 172.21.15.104:0/2674956276 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1044011c60 con 0x55fa05d40120 2024-06-09T22:20:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1035ffb700 1 -- 172.21.15.104:0/2674956276 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f104401e780 con 0x55fa05d40120 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1062ee8b80 10 client.15084 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1062ee8b80 10 client.15084 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1062ee8b80 20 client.15084 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55fa05d53df0 0x55fa05d55260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1062ee8b80 4 client.15084 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.945+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2674956276 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa05c96d00 con 0x55fa05d53df0 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.946+0000 7f1056401700 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55fa05d53df0 0x55fa05d55260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.946+0000 7f1035ffb700 10 client.15084 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.946+0000 7f1056401700 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55fa05d53df0 0x55fa05d55260 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=42123ebdef6265a5 server_cookie=70b95d23f0401447 in_seq=0 out_seq=0 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1035ffb700 1 -- 172.21.15.104:0/2674956276 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55fa05c96d00 con 0x55fa05d53df0 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1035ffb700 10 client.15084 handle_command_reply: tid=0 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1062ee8b80 1 client.15084 shutdown 2024-06-09T22:20:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2674956276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 msgr2=0x55fa05d43a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d43a90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f104401ee90 tx=0x7f1044011ec0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2674956276 shutdown_connections 2024-06-09T22:20:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55fa05d53df0 0x55fa05d55260 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40a40 0x55fa05d43fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1035ffb700 0 client.15084 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:20:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d46790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/2674956276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d43a90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2674956276 >> 172.21.15.104:0/2674956276 conn(0x55fa05c65030 msgr2=0x55fa05d326b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2674956276 shutdown_connections 2024-06-09T22:20:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.947+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2674956276 wait complete. 2024-06-09T22:20:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.948+0000 7f1062ee8b80 20 client.15084 trim_cache size 0 max 16384 2024-06-09T22:20:01.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1062ee8b80 1 Processor -- start 2024-06-09T22:20:01.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1062ee8b80 1 -- start start 2024-06-09T22:20:01.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1062ee8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d3a960 0x55fa05d44020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1062ee8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d415f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1062ee8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d40a40 0x55fa05d462d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1062ee8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fa05d535a0 con 0x55fa05d40120 2024-06-09T22:20:01.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1062ee8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fa05d53c70 con 0x55fa05d40a40 2024-06-09T22:20:01.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1062ee8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fa05d54340 con 0x55fa05d3a960 2024-06-09T22:20:01.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1055c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1055c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d415f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60744/0 (socket says 172.21.15.104:60744) 2024-06-09T22:20:01.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1055c00700 1 -- 172.21.15.104:0/587112546 learned_addr learned my addr 172.21.15.104:0/587112546 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:01.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1056c02700 1 --2- 172.21.15.104:0/587112546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d40a40 0x55fa05d462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1055c00700 1 -- 172.21.15.104:0/587112546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d3a960 msgr2=0x55fa05d44020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:01.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1055c00700 1 --2- 172.21.15.104:0/587112546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d3a960 0x55fa05d44020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1055c00700 1 -- 172.21.15.104:0/587112546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d40a40 msgr2=0x55fa05d462d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:01.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1055c00700 1 --2- 172.21.15.104:0/587112546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d40a40 0x55fa05d462d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1055c00700 1 -- 172.21.15.104:0/587112546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa05d47430 con 0x55fa05d40120 2024-06-09T22:20:01.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1055c00700 1 --2- 172.21.15.104:0/587112546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d415f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f104400bb80 tx=0x7f1044005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ee7cf5a0e58ddd server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:01.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.969+0000 7f1037fff700 1 -- 172.21.15.104:0/587112546 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044015070 con 0x55fa05d40120 2024-06-09T22:20:01.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.970+0000 7f1037fff700 1 -- 172.21.15.104:0/587112546 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1044011c20 con 0x55fa05d40120 2024-06-09T22:20:01.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.970+0000 7f1037fff700 1 -- 172.21.15.104:0/587112546 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044022040 con 0x55fa05d40120 2024-06-09T22:20:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.970+0000 7f1062ee8b80 1 -- 172.21.15.104:0/587112546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 msgr2=0x55fa05d415f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.970+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/587112546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d415f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f104400bb80 tx=0x7f1044005b20 comp rx=0 tx=0).stop 2024-06-09T22:20:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.970+0000 7f1062ee8b80 1 -- 172.21.15.104:0/587112546 shutdown_connections 2024-06-09T22:20:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.970+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/587112546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d40a40 0x55fa05d462d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.970+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/587112546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d3a960 0x55fa05d44020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.970+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/587112546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d415f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.970+0000 7f1062ee8b80 1 -- 172.21.15.104:0/587112546 >> 172.21.15.104:0/587112546 conn(0x55fa05c65030 msgr2=0x55fa05d40ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.970+0000 7f1062ee8b80 1 -- 172.21.15.104:0/587112546 shutdown_connections 2024-06-09T22:20:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.970+0000 7f1062ee8b80 1 -- 172.21.15.104:0/587112546 wait complete. 2024-06-09T22:20:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1062ee8b80 1 -- start start 2024-06-09T22:20:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1062ee8b80 1 Processor -- start 2024-06-09T22:20:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1062ee8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d5ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1062ee8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d31ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1062ee8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40a40 0x55fa05d32400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1062ee8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fa05d550e0 con 0x55fa05d40120 2024-06-09T22:20:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1062ee8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fa05d54a10 con 0x55fa05d3a960 2024-06-09T22:20:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1062ee8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fa05d557b0 con 0x55fa05d40a40 2024-06-09T22:20:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1062ee8b80 10 client.0 _ll_register_callbacks cb 0x55fa05c1f560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1055c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d31ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1055c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d31ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60758/0 (socket says 172.21.15.104:60758) 2024-06-09T22:20:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1055c00700 1 -- 172.21.15.104:0/2706142898 learned_addr learned my addr 172.21.15.104:0/2706142898 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1056401700 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d5ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1055c00700 1 -- 172.21.15.104:0/2706142898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40a40 msgr2=0x55fa05d32400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1056c02700 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40a40 0x55fa05d32400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1055c00700 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40a40 0x55fa05d32400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1055c00700 1 -- 172.21.15.104:0/2706142898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 msgr2=0x55fa05d5ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1055c00700 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d5ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1055c00700 1 -- 172.21.15.104:0/2706142898 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa05d32b30 con 0x55fa05d40120 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1056401700 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d5ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1055c00700 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d31ec0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f104401ec70 tx=0x7f104400ffa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea801fdad95a08fb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.971+0000 7f1035ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1035ffb700 1 -- 172.21.15.104:0/2706142898 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044015030 con 0x55fa05d40120 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1035ffb700 1 -- 172.21.15.104:0/2706142898 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f104402e040 con 0x55fa05d40120 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2706142898 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1044011c20 con 0x55fa05d40120 2024-06-09T22:20:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1035ffb700 1 -- 172.21.15.104:0/2706142898 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1044055050 con 0x55fa05d40120 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1062ee8b80 10 client.15090 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2706142898 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fa05dc15c0 con 0x55fa05d40120 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1035ffb700 1 -- 172.21.15.104:0/2706142898 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1044011c20 con 0x55fa05d40120 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1035ffb700 1 -- 172.21.15.104:0/2706142898 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1044050030 con 0x55fa05d40120 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1062ee8b80 10 client.15090 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1062ee8b80 10 client.15090 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1062ee8b80 20 client.15090 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55fa05d53df0 0x55fa05dc4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1062ee8b80 4 client.15090 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2706142898 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "status"}) -- 0x55fa05c91280 con 0x55fa05d53df0 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.972+0000 7f1056401700 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55fa05d53df0 0x55fa05dc4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1035ffb700 10 client.15090 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:20:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1056401700 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55fa05d53df0 0x55fa05dc4a10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2070d8b09f819918 server_cookie=f3681ab421c59fd7 in_seq=0 out_seq=0 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1035ffb700 1 -- 172.21.15.104:0/2706142898 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55fa05c91280 con 0x55fa05d53df0 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1035ffb700 10 client.15090 handle_command_reply: tid=0 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1062ee8b80 1 client.15090 shutdown 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2706142898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 msgr2=0x55fa05d31ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d31ec0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f104401ec70 tx=0x7f104400ffa0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2706142898 shutdown_connections 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55fa05d53df0 0x55fa05dc4a10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa05d40a40 0x55fa05d32400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa05d3a960 0x55fa05d5ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1062ee8b80 1 --2- 172.21.15.104:0/2706142898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa05d40120 0x55fa05d31ec0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2706142898 >> 172.21.15.104:0/2706142898 conn(0x55fa05c65030 msgr2=0x55fa05d3ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1035ffb700 0 client.15090 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:20:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2706142898 shutdown_connections 2024-06-09T22:20:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.973+0000 7f1062ee8b80 1 -- 172.21.15.104:0/2706142898 wait complete. 2024-06-09T22:20:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1062ee8b80 20 client.15090 trim_cache size 0 max 16384 2024-06-09T22:20:01.982 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:20:01.982 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:20:01.982 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 1, 2024-06-09T22:20:01.982 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24448, 2024-06-09T22:20:01.982 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:20:01.982 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:20:01.982 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:20:01.983 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 134.582767316, 2024-06-09T22:20:01.983 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 28, 2024-06-09T22:20:01.983 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 99, 2024-06-09T22:20:01.983 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 97, 2024-06-09T22:20:01.983 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 228.05763803100001 2024-06-09T22:20:01.983 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:20:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1058665700 1 -- 172.21.15.104:0/1122153284 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f10380a2c20 msgr2=0x7f10380a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1058665700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f10380a2c20 0x7f10380a50e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1044005fd0 tx=0x7f1044005f60 comp rx=0 tx=0).stop 2024-06-09T22:20:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1058665700 1 -- 172.21.15.104:0/1122153284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1050100b50 msgr2=0x7f105019d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1058665700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1050100b50 0x7f105019d6d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f10400060b0 tx=0x7f104000d3b0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1058665700 1 -- 172.21.15.104:0/1122153284 shutdown_connections 2024-06-09T22:20:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1058665700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f10380a2c20 0x7f10380a50e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1058665700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1050100b50 0x7f105019d6d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1058665700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1050106c00 0x7f1050197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1058665700 1 --2- 172.21.15.104:0/1122153284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1050101510 0x7f105019dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1058665700 1 -- 172.21.15.104:0/1122153284 >> 172.21.15.104:0/1122153284 conn(0x7f1050077990 msgr2=0x7f10500fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1058665700 1 -- 172.21.15.104:0/1122153284 shutdown_connections 2024-06-09T22:20:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:01.974+0000 7f1058665700 1 -- 172.21.15.104:0/1122153284 wait complete. 2024-06-09T22:20:02.686 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:02 smithi104 ceph-mon[32389]: pgmap v407: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:02.715 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-06-09T22:20:02.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:02 smithi194 ceph-mon[36856]: pgmap v407: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:02.954 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:02.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:02 smithi153 ceph-mon[36599]: pgmap v407: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:04.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.687+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/3466891867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bb4101390 msgr2=0x7f2bb41017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:04.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.687+0000 7f2bbc0bd700 1 --2- 172.21.15.104:0/3466891867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bb4101390 0x7f2bb41017f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0009990 tx=0x7f2bb0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.687+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/3466891867 shutdown_connections 2024-06-09T22:20:04.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.687+0000 7f2bbc0bd700 1 --2- 172.21.15.104:0/3466891867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bb4101d30 0x7f2bb410e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.687+0000 7f2bbc0bd700 1 --2- 172.21.15.104:0/3466891867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bb41073d0 0x7f2bb41077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.687+0000 7f2bbc0bd700 1 --2- 172.21.15.104:0/3466891867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bb4101390 0x7f2bb41017f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.687+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/3466891867 >> 172.21.15.104:0/3466891867 conn(0x7f2bb40782c0 msgr2=0x7f2bb40786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:04.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.687+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/3466891867 shutdown_connections 2024-06-09T22:20:04.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.688+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/3466891867 wait complete. 2024-06-09T22:20:04.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bbc0bd700 1 Processor -- start 2024-06-09T22:20:04.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bbc0bd700 1 -- start start 2024-06-09T22:20:04.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bbc0bd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bb4101390 0x7f2bb419e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bbc0bd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bb4101d30 0x7f2bb419ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bbc0bd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bb41073d0 0x7f2bb4198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bbc0bd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb41116d0 con 0x7f2bb41073d0 2024-06-09T22:20:04.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bbc0bd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb4111550 con 0x7f2bb4101d30 2024-06-09T22:20:04.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bbc0bd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb4111850 con 0x7f2bb4101390 2024-06-09T22:20:04.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bba65a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bb41073d0 0x7f2bb4198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bba65a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bb41073d0 0x7f2bb4198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60784/0 (socket says 172.21.15.104:60784) 2024-06-09T22:20:04.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bba65a700 1 -- 172.21.15.104:0/1435946388 learned_addr learned my addr 172.21.15.104:0/1435946388 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:04.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bb9658700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bb4101d30 0x7f2bb419ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bb4101390 0x7f2bb419e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bba65a700 1 -- 172.21.15.104:0/1435946388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bb4101390 msgr2=0x7f2bb419e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:04.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bba65a700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bb4101390 0x7f2bb419e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bba65a700 1 -- 172.21.15.104:0/1435946388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bb4101d30 msgr2=0x7f2bb419ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:04.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bba65a700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bb4101d30 0x7f2bb419ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bba65a700 1 -- 172.21.15.104:0/1435946388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb41991e0 con 0x7f2bb41073d0 2024-06-09T22:20:04.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.689+0000 7f2bb9658700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bb4101d30 0x7f2bb419ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:04.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.690+0000 7f2bba65a700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bb41073d0 0x7f2bb4198a80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00b970 tx=0x7f2bac00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:04.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.690+0000 7f2baaffd700 1 -- 172.21.15.104:0/1435946388 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac019070 con 0x7f2bb41073d0 2024-06-09T22:20:04.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.690+0000 7f2baaffd700 1 -- 172.21.15.104:0/1435946388 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bac014440 con 0x7f2bb41073d0 2024-06-09T22:20:04.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.690+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/1435946388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb41994d0 con 0x7f2bb41073d0 2024-06-09T22:20:04.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.690+0000 7f2baaffd700 1 -- 172.21.15.104:0/1435946388 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac0075d0 con 0x7f2bb41073d0 2024-06-09T22:20:04.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.690+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/1435946388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb41a5790 con 0x7f2bb41073d0 2024-06-09T22:20:04.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.692+0000 7f2baaffd700 1 -- 172.21.15.104:0/1435946388 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f2bac01c070 con 0x7f2bb41073d0 2024-06-09T22:20:04.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.692+0000 7f2baaffd700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2ba00a2c20 0x7f2ba00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.693+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2ba00a2c20 0x7f2ba00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.693+0000 7f2baaffd700 1 -- 172.21.15.104:0/1435946388 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f2bac0ce840 con 0x7f2bb41073d0 2024-06-09T22:20:04.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.693+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2ba00a2c20 0x7f2ba00a50e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4005950 tx=0x7f2ba40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:04.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.693+0000 7f2bc6940b80 1 Processor -- start 2024-06-09T22:20:04.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.693+0000 7f2bc6940b80 1 -- start start 2024-06-09T22:20:04.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bc6940b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 0x559bfebba9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bc6940b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 0x559bfebb4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bc6940b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebb4ed0 0x559bfebb52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bc6940b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559bfeb067c0 con 0x559bfebbaf10 2024-06-09T22:20:04.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bc6940b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559bfeaae6d0 con 0x559bfebba5f0 2024-06-09T22:20:04.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bc6940b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559bfeb02780 con 0x559bfebb4ed0 2024-06-09T22:20:04.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bb9658700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 0x559bfebb4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bb9658700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 0x559bfebb4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60794/0 (socket says 172.21.15.104:60794) 2024-06-09T22:20:04.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bb9658700 1 -- 172.21.15.104:0/2149533792 learned_addr learned my addr 172.21.15.104:0/2149533792 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:04.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/2149533792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 0x559bfebba9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bb9658700 1 -- 172.21.15.104:0/2149533792 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebb4ed0 msgr2=0x559bfebb52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:04.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bb9658700 1 --2- 172.21.15.104:0/2149533792 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebb4ed0 0x559bfebb52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bb9658700 1 -- 172.21.15.104:0/2149533792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 msgr2=0x559bfebba9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:04.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bb9658700 1 --2- 172.21.15.104:0/2149533792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 0x559bfebba9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bb9658700 1 -- 172.21.15.104:0/2149533792 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bfeaa4f00 con 0x559bfebbaf10 2024-06-09T22:20:04.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2bb9658700 1 --2- 172.21.15.104:0/2149533792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 0x559bfebb4990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0006010 tx=0x7f2bb0009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecd2a3143e8a9c7 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:04.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2b9ffff700 1 -- 172.21.15.104:0/2149533792 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0027070 con 0x559bfebbaf10 2024-06-09T22:20:04.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2b9ffff700 1 -- 172.21.15.104:0/2149533792 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bb00055b0 con 0x559bfebbaf10 2024-06-09T22:20:04.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.694+0000 7f2b9ffff700 1 -- 172.21.15.104:0/2149533792 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb000bd30 con 0x559bfebbaf10 2024-06-09T22:20:04.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.695+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2149533792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 msgr2=0x559bfebb4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:04.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.695+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2149533792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 0x559bfebb4990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0006010 tx=0x7f2bb0009ee0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.695+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2149533792 shutdown_connections 2024-06-09T22:20:04.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.695+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2149533792 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebb4ed0 0x559bfebb52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.695+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2149533792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 0x559bfebba9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.695+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2149533792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 0x559bfebb4990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.695+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2149533792 >> 172.21.15.104:0/2149533792 conn(0x559bfeadfc70 msgr2=0x559bfebaf800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:04.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.695+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2149533792 shutdown_connections 2024-06-09T22:20:04.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.695+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2149533792 wait complete. 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bc6940b80 1 -- start start 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bc6940b80 1 Processor -- start 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bc6940b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebb4ed0 0x559bfebc0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bc6940b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebba5f0 0x559bfebbdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bc6940b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 0x559bfebbe510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bc6940b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559bfebc1920 con 0x559bfebbaf10 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bc6940b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559bfeaa6ba0 con 0x559bfebb4ed0 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bc6940b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559bfebc1aa0 con 0x559bfebba5f0 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bc6940b80 10 client.0 _ll_register_callbacks cb 0x559bfeaa08d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bba65a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 0x559bfebbe510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bba65a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 0x559bfebbe510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60796/0 (socket says 172.21.15.104:60796) 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bba65a700 1 -- 172.21.15.104:0/2066010048 learned_addr learned my addr 172.21.15.104:0/2066010048 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bb9658700 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebba5f0 0x559bfebbdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.696+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebb4ed0 0x559bfebc0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2bba65a700 1 -- 172.21.15.104:0/2066010048 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebba5f0 msgr2=0x559bfebbdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2bba65a700 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebba5f0 0x559bfebbdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2bba65a700 1 -- 172.21.15.104:0/2066010048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebb4ed0 msgr2=0x559bfebc0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2bba65a700 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebb4ed0 0x559bfebc0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2bba65a700 1 -- 172.21.15.104:0/2066010048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bfea55880 con 0x559bfebbaf10 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2bba65a700 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 0x559bfebbe510 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2bac09aca0 tx=0x7f2bac0148d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d96d1681af624f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2b9d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2b9d7fa700 1 -- 172.21.15.104:0/2066010048 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac0a34f0 con 0x559bfebbaf10 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2b9d7fa700 1 -- 172.21.15.104:0/2066010048 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f2bac0a3690 con 0x559bfebbaf10 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2066010048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bb00055b0 con 0x559bfebbaf10 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2b9d7fa700 1 -- 172.21.15.104:0/2066010048 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2bac079b80 con 0x559bfebbaf10 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2bc6940b80 10 client.15102 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:04.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2066010048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559bfebcc330 con 0x559bfebbaf10 2024-06-09T22:20:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.697+0000 7f2b9d7fa700 1 -- 172.21.15.104:0/2066010048 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bb00055b0 con 0x559bfebbaf10 2024-06-09T22:20:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.698+0000 7f2b9d7fa700 1 -- 172.21.15.104:0/2066010048 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2bac09a160 con 0x559bfebbaf10 2024-06-09T22:20:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.698+0000 7f2bc6940b80 10 client.15102 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.698+0000 7f2bc6940b80 10 client.15102 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-09T22:20:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.698+0000 7f2bc6940b80 20 client.15102 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.698+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559bfebce820 0x559bfebcfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.698+0000 7f2bc6940b80 4 client.15102 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:20:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.698+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2066010048 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559bfeb13cc0 con 0x559bfebce820 2024-06-09T22:20:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.698+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559bfebce820 0x559bfebcfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.698+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559bfebce820 0x559bfebcfc90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baee3c80f1d64d39 server_cookie=3f1029ca800f7805 in_seq=0 out_seq=0 2024-06-09T22:20:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.698+0000 7f2b9d7fa700 10 client.15102 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:20:04.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.699+0000 7f2b9d7fa700 1 -- 172.21.15.104:0/2066010048 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x559bfeb13cc0 con 0x559bfebce820 2024-06-09T22:20:04.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.699+0000 7f2b9d7fa700 10 client.15102 handle_command_reply: tid=0 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.699+0000 7f2bc6940b80 1 client.15102 shutdown 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.699+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2066010048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 msgr2=0x559bfebbe510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.699+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 0x559bfebbe510 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2bac09aca0 tx=0x7f2bac0148d0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.699+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2066010048 shutdown_connections 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.699+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559bfebce820 0x559bfebcfc90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.699+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebbaf10 0x559bfebbe510 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.699+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebb4ed0 0x559bfebc0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.699+0000 7f2b9d7fa700 0 client.15102 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.699+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2066010048 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebba5f0 0x559bfebbdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.699+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2066010048 >> 172.21.15.104:0/2066010048 conn(0x559bfeadfc70 msgr2=0x559bfebb3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.700+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2066010048 shutdown_connections 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.700+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2066010048 wait complete. 2024-06-09T22:20:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.700+0000 7f2bc6940b80 20 client.15102 trim_cache size 0 max 16384 2024-06-09T22:20:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.720+0000 7f2bc6940b80 1 Processor -- start 2024-06-09T22:20:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.720+0000 7f2bc6940b80 1 -- start start 2024-06-09T22:20:04.719 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:04 smithi104 ceph-mon[32389]: pgmap v408: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:04.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bc6940b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 0x559bfebb4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bc6940b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 0x559bfebbdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bc6940b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebbaf10 0x559bfebbe4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bc6940b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559bfebcdfd0 con 0x559bfebb4ed0 2024-06-09T22:20:04.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bc6940b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559bfebce6a0 con 0x559bfebba5f0 2024-06-09T22:20:04.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bc6940b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559bfebced70 con 0x559bfebbaf10 2024-06-09T22:20:04.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bb9e59700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 0x559bfebb4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bb9e59700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 0x559bfebb4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60802/0 (socket says 172.21.15.104:60802) 2024-06-09T22:20:04.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bb9e59700 1 -- 172.21.15.104:0/1381124538 learned_addr learned my addr 172.21.15.104:0/1381124538 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:04.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bb9658700 1 --2- 172.21.15.104:0/1381124538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 0x559bfebbdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bb9e59700 1 -- 172.21.15.104:0/1381124538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebbaf10 msgr2=0x559bfebbe4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:04.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bba65a700 1 --2- 172.21.15.104:0/1381124538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebbaf10 0x559bfebbe4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/1381124538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebbaf10 0x559bfebbe4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bb9e59700 1 -- 172.21.15.104:0/1381124538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 msgr2=0x559bfebbdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:04.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/1381124538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 0x559bfebbdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bb9e59700 1 -- 172.21.15.104:0/1381124538 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bfebbc540 con 0x559bfebb4ed0 2024-06-09T22:20:04.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/1381124538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 0x559bfebb4990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x559bfeb7a250 tx=0x7f2ba4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39874bc9dce504b3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:04.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.721+0000 7f2b9f7fe700 1 -- 172.21.15.104:0/1381124538 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba402e070 con 0x559bfebb4ed0 2024-06-09T22:20:04.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.722+0000 7f2b9f7fe700 1 -- 172.21.15.104:0/1381124538 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ba4020db0 con 0x559bfebb4ed0 2024-06-09T22:20:04.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.722+0000 7f2b9f7fe700 1 -- 172.21.15.104:0/1381124538 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4011970 con 0x559bfebb4ed0 2024-06-09T22:20:04.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.722+0000 7f2bc6940b80 1 -- 172.21.15.104:0/1381124538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 msgr2=0x559bfebb4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.722+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/1381124538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 0x559bfebb4990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x559bfeb7a250 tx=0x7f2ba4015000 comp rx=0 tx=0).stop 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.722+0000 7f2bc6940b80 1 -- 172.21.15.104:0/1381124538 shutdown_connections 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.722+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/1381124538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 0x559bfebb4990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.722+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/1381124538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebbaf10 0x559bfebbe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.722+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/1381124538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 0x559bfebbdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.722+0000 7f2bc6940b80 1 -- 172.21.15.104:0/1381124538 >> 172.21.15.104:0/1381124538 conn(0x559bfeadfc70 msgr2=0x559bfebbc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.722+0000 7f2bc6940b80 1 -- 172.21.15.104:0/1381124538 shutdown_connections 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.722+0000 7f2bc6940b80 1 -- 172.21.15.104:0/1381124538 wait complete. 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.723+0000 7f2bc6940b80 1 -- start start 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.723+0000 7f2bc6940b80 1 Processor -- start 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.723+0000 7f2bc6940b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 0x559bfebd92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.723+0000 7f2bc6940b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 0x559bfebd9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.723+0000 7f2bc6940b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebbaf10 0x559bfebd40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.723+0000 7f2bc6940b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559bfebcfb10 con 0x559bfebb4ed0 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.723+0000 7f2bc6940b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559bfebcf440 con 0x559bfebba5f0 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.723+0000 7f2bc6940b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559bfebd01e0 con 0x559bfebbaf10 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.723+0000 7f2bc6940b80 10 client.0 _ll_register_callbacks cb 0x559bfeaa08d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bb9e59700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 0x559bfebd92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bb9e59700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 0x559bfebd92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60808/0 (socket says 172.21.15.104:60808) 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bb9e59700 1 -- 172.21.15.104:0/2361483176 learned_addr learned my addr 172.21.15.104:0/2361483176 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bb9658700 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 0x559bfebd9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bba65a700 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebbaf10 0x559bfebd40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bb9e59700 1 -- 172.21.15.104:0/2361483176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebbaf10 msgr2=0x559bfebd40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebbaf10 0x559bfebd40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bb9e59700 1 -- 172.21.15.104:0/2361483176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 msgr2=0x559bfebd9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:04.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 0x559bfebd9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bb9e59700 1 -- 172.21.15.104:0/2361483176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bfebd4700 con 0x559bfebb4ed0 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bba65a700 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebbaf10 0x559bfebd40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bb9e59700 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 0x559bfebd92f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2ba402c040 tx=0x7f2ba4005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8c2dde71ffe4ca server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2b9d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2b9d7fa700 1 -- 172.21.15.104:0/2361483176 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba402e040 con 0x559bfebb4ed0 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2b9d7fa700 1 -- 172.21.15.104:0/2361483176 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f2ba4005520 con 0x559bfebb4ed0 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2361483176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ba4020db0 con 0x559bfebb4ed0 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2b9d7fa700 1 -- 172.21.15.104:0/2361483176 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2ba4040050 con 0x559bfebb4ed0 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.724+0000 7f2bc6940b80 10 client.15108 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.725+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2361483176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559bfec3ee00 con 0x559bfebb4ed0 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.725+0000 7f2b9d7fa700 1 -- 172.21.15.104:0/2361483176 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ba4020db0 con 0x559bfebb4ed0 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.725+0000 7f2b9d7fa700 1 -- 172.21.15.104:0/2361483176 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2ba4011a00 con 0x559bfebb4ed0 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.725+0000 7f2bc6940b80 10 client.15108 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:04.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.725+0000 7f2bc6940b80 10 client.15108 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-09T22:20:04.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.725+0000 7f2bc6940b80 20 client.15108 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:04.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.725+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559bfec436a0 0x559bfec423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:04.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.725+0000 7f2bc6940b80 4 client.15108 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:20:04.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.725+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2361483176 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "status"}) -- 0x559bfeb0e240 con 0x559bfec436a0 2024-06-09T22:20:04.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.725+0000 7f2bb9658700 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559bfec436a0 0x559bfec423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:04.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.726+0000 7f2bb9658700 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559bfec436a0 0x559bfec423b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18c6ea81f380aaa5 server_cookie=83a246a0170e9cd1 in_seq=0 out_seq=0 2024-06-09T22:20:04.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.726+0000 7f2b9d7fa700 10 client.15108 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:20:04.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.726+0000 7f2b9d7fa700 1 -- 172.21.15.104:0/2361483176 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559bfeb0e240 con 0x559bfec436a0 2024-06-09T22:20:04.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.726+0000 7f2b9d7fa700 10 client.15108 handle_command_reply: tid=0 2024-06-09T22:20:04.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.726+0000 7f2bc6940b80 1 client.15108 shutdown 2024-06-09T22:20:04.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.726+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2361483176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 msgr2=0x559bfebd92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:04.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.726+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 0x559bfebd92f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2ba402c040 tx=0x7f2ba4005040 comp rx=0 tx=0).stop 2024-06-09T22:20:04.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2361483176 shutdown_connections 2024-06-09T22:20:04.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559bfec436a0 0x559bfec423b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559bfebb4ed0 0x559bfebd92f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2b9d7fa700 0 client.15108 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:20:04.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559bfebbaf10 0x559bfebd40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bc6940b80 1 --2- 172.21.15.104:0/2361483176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559bfebba5f0 0x559bfebd9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2361483176 >> 172.21.15.104:0/2361483176 conn(0x559bfeadfc70 msgr2=0x559bfebbdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:04.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2361483176 shutdown_connections 2024-06-09T22:20:04.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bc6940b80 1 -- 172.21.15.104:0/2361483176 wait complete. 2024-06-09T22:20:04.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bc6940b80 20 client.15108 trim_cache size 0 max 16384 2024-06-09T22:20:04.731 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:20:04.732 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:20:04.732 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2024-06-09T22:20:04.732 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24460, 2024-06-09T22:20:04.732 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:20:04.732 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:20:04.732 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:20:04.732 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 199.52567988300001, 2024-06-09T22:20:04.732 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 28, 2024-06-09T22:20:04.732 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 99, 2024-06-09T22:20:04.732 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 97, 2024-06-09T22:20:04.732 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 225.046732214 2024-06-09T22:20:04.732 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:20:04.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/1435946388 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2ba00a2c20 msgr2=0x7f2ba00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:04.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bbc0bd700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2ba00a2c20 0x7f2ba00a50e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4005950 tx=0x7f2ba40058e0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/1435946388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bb41073d0 msgr2=0x7f2bb4198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:04.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bbc0bd700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bb41073d0 0x7f2bb4198a80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00b970 tx=0x7f2bac00bd30 comp rx=0 tx=0).stop 2024-06-09T22:20:04.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/1435946388 shutdown_connections 2024-06-09T22:20:04.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bbc0bd700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f2ba00a2c20 0x7f2ba00a50e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bbc0bd700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bb41073d0 0x7f2bb4198a80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bbc0bd700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bb4101390 0x7f2bb419e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bbc0bd700 1 --2- 172.21.15.104:0/1435946388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bb4101d30 0x7f2bb419ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:04.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.727+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/1435946388 >> 172.21.15.104:0/1435946388 conn(0x7f2bb40782c0 msgr2=0x7f2bb41055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:04.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.728+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/1435946388 shutdown_connections 2024-06-09T22:20:04.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:04.728+0000 7f2bbc0bd700 1 -- 172.21.15.104:0/1435946388 wait complete. 2024-06-09T22:20:04.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:04 smithi153 ceph-mon[36599]: pgmap v408: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:04.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:04 smithi194 ceph-mon[36856]: pgmap v408: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:05.431 INFO:tasks.cephadm:Setup complete, yielding 2024-06-09T22:20:05.432 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-09T22:20:05.444 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi104.front.sepia.ceph.com 2024-06-09T22:20:05.444 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch status' 2024-06-09T22:20:05.683 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:06.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:06 smithi104 ceph-mon[32389]: pgmap v409: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:06.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:06 smithi194 ceph-mon[36856]: pgmap v409: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:06.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:06 smithi153 ceph-mon[36599]: pgmap v409: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:07.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.457+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/2290247168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cc81025e0 msgr2=0x7f3cc8102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:07.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.457+0000 7f3ccdb9d700 1 --2- 172.21.15.104:0/2290247168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cc81025e0 0x7f3cc8102a40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc0099c0 tx=0x7f3cbc009cd0 comp rx=0 tx=0).stop 2024-06-09T22:20:07.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.457+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/2290247168 shutdown_connections 2024-06-09T22:20:07.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.457+0000 7f3ccdb9d700 1 --2- 172.21.15.104:0/2290247168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc8102f80 0x7f3cc8109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:07.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.457+0000 7f3ccdb9d700 1 --2- 172.21.15.104:0/2290247168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3cc81013e0 0x7f3cc8101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:07.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.457+0000 7f3ccdb9d700 1 --2- 172.21.15.104:0/2290247168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cc81025e0 0x7f3cc8102a40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:07.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.457+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/2290247168 >> 172.21.15.104:0/2290247168 conn(0x7f3cc80fc9c0 msgr2=0x7f3cc80fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:07.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.457+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/2290247168 shutdown_connections 2024-06-09T22:20:07.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.457+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/2290247168 wait complete. 2024-06-09T22:20:07.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.458+0000 7f3ccdb9d700 1 Processor -- start 2024-06-09T22:20:07.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.458+0000 7f3ccdb9d700 1 -- start start 2024-06-09T22:20:07.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.458+0000 7f3ccdb9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc81013e0 0x7f3cc819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:07.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.458+0000 7f3ccdb9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3cc81025e0 0x7f3cc819ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:07.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.458+0000 7f3ccdb9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cc8102f80 0x7f3cc81a1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:07.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.458+0000 7f3ccdb9d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc810cfd0 con 0x7f3cc8102f80 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.458+0000 7f3ccdb9d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc810ce50 con 0x7f3cc81013e0 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.458+0000 7f3ccdb9d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc810d150 con 0x7f3cc81025e0 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.458+0000 7f3cc7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cc8102f80 0x7f3cc81a1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cc8102f80 0x7f3cc81a1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46326/0 (socket says 172.21.15.104:46326) 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc7fff700 1 -- 172.21.15.104:0/4281363286 learned_addr learned my addr 172.21.15.104:0/4281363286 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc77fe700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc81013e0 0x7f3cc819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc6ffd700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3cc81025e0 0x7f3cc819ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc7fff700 1 -- 172.21.15.104:0/4281363286 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3cc81025e0 msgr2=0x7f3cc819ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc7fff700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3cc81025e0 0x7f3cc819ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc7fff700 1 -- 172.21.15.104:0/4281363286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc81013e0 msgr2=0x7f3cc819a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc7fff700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc81013e0 0x7f3cc819a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc7fff700 1 -- 172.21.15.104:0/4281363286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc81a2540 con 0x7f3cc8102f80 2024-06-09T22:20:07.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc7fff700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cc8102f80 0x7f3cc81a1ea0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0007f10 tx=0x7f3cc000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:07.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc4ff9700 1 -- 172.21.15.104:0/4281363286 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0004bb0 con 0x7f3cc8102f80 2024-06-09T22:20:07.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc4ff9700 1 -- 172.21.15.104:0/4281363286 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cc0004d50 con 0x7f3cc8102f80 2024-06-09T22:20:07.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/4281363286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cc81a2830 con 0x7f3cc8102f80 2024-06-09T22:20:07.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3cc4ff9700 1 -- 172.21.15.104:0/4281363286 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc00059f0 con 0x7f3cc8102f80 2024-06-09T22:20:07.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.459+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/4281363286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cc81a2d20 con 0x7f3cc8102f80 2024-06-09T22:20:07.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.460+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/4281363286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cc8069180 con 0x7f3cc8102f80 2024-06-09T22:20:07.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.461+0000 7f3cc4ff9700 1 -- 172.21.15.104:0/4281363286 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3cc001a020 con 0x7f3cc8102f80 2024-06-09T22:20:07.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.462+0000 7f3cc4ff9700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3cb00a2c20 0x7f3cb00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:07.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.462+0000 7f3cc77fe700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3cb00a2c20 0x7f3cb00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:07.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.462+0000 7f3cc4ff9700 1 -- 172.21.15.104:0/4281363286 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f3cc00cf190 con 0x7f3cc8102f80 2024-06-09T22:20:07.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.463+0000 7f3cc77fe700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3cb00a2c20 0x7f3cb00a50e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8102440 tx=0x7f3cb800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:07.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.466+0000 7f3cc4ff9700 1 -- 172.21.15.104:0/4281363286 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3cc0010070 con 0x7f3cc8102f80 2024-06-09T22:20:07.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.681+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/4281363286 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f3cc80feb90 con 0x7f3cb00a2c20 2024-06-09T22:20:07.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.682+0000 7f3cc4ff9700 1 -- 172.21.15.104:0/4281363286 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f3cc80feb90 con 0x7f3cb00a2c20 2024-06-09T22:20:07.681 INFO:teuthology.orchestra.run.smithi104.stdout:Backend: cephadm 2024-06-09T22:20:07.681 INFO:teuthology.orchestra.run.smithi104.stdout:Available: Yes 2024-06-09T22:20:07.681 INFO:teuthology.orchestra.run.smithi104.stdout:Paused: No 2024-06-09T22:20:07.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.684+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/4281363286 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3cb00a2c20 msgr2=0x7f3cb00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:07.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.684+0000 7f3ccdb9d700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3cb00a2c20 0x7f3cb00a50e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8102440 tx=0x7f3cb800b410 comp rx=0 tx=0).stop 2024-06-09T22:20:07.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.684+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/4281363286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cc8102f80 msgr2=0x7f3cc81a1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:07.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.684+0000 7f3ccdb9d700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cc8102f80 0x7f3cc81a1ea0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0007f10 tx=0x7f3cc000d3b0 comp rx=0 tx=0).stop 2024-06-09T22:20:07.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.685+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/4281363286 shutdown_connections 2024-06-09T22:20:07.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.685+0000 7f3ccdb9d700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f3cb00a2c20 0x7f3cb00a50e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:07.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.685+0000 7f3ccdb9d700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cc8102f80 0x7f3cc81a1ea0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:07.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.685+0000 7f3ccdb9d700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc81013e0 0x7f3cc819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:07.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.685+0000 7f3ccdb9d700 1 --2- 172.21.15.104:0/4281363286 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3cc81025e0 0x7f3cc819ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:07.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.685+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/4281363286 >> 172.21.15.104:0/4281363286 conn(0x7f3cc80fc9c0 msgr2=0x7f3cc80fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:07.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.685+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/4281363286 shutdown_connections 2024-06-09T22:20:07.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:07.685+0000 7f3ccdb9d700 1 -- 172.21.15.104:0/4281363286 wait complete. 2024-06-09T22:20:08.496 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ps' 2024-06-09T22:20:08.738 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:08.754 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:08 smithi104 ceph-mon[32389]: pgmap v410: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:08.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:08 smithi194 ceph-mon[36856]: pgmap v410: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:08.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:08 smithi153 ceph-mon[36599]: pgmap v410: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:09.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:09 smithi194 ceph-mon[36856]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:09.907 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:09 smithi104 ceph-mon[32389]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:09.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:09 smithi153 ceph-mon[36599]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:10.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.502+0000 7f28e5b22700 1 -- 172.21.15.104:0/3406876119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0103870 msgr2=0x7f28e0103cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:10.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.502+0000 7f28e5b22700 1 --2- 172.21.15.104:0/3406876119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0103870 0x7f28e0103cd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f28c8009990 tx=0x7f28c8009ca0 comp rx=0 tx=0).stop 2024-06-09T22:20:10.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.502+0000 7f28e5b22700 1 -- 172.21.15.104:0/3406876119 shutdown_connections 2024-06-09T22:20:10.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.502+0000 7f28e5b22700 1 --2- 172.21.15.104:0/3406876119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28e0104210 0x7f28e010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:10.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.502+0000 7f28e5b22700 1 --2- 172.21.15.104:0/3406876119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0100890 0x7f28e0100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:10.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.502+0000 7f28e5b22700 1 --2- 172.21.15.104:0/3406876119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0103870 0x7f28e0103cd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:10.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.502+0000 7f28e5b22700 1 -- 172.21.15.104:0/3406876119 >> 172.21.15.104:0/3406876119 conn(0x7f28e00fc1f0 msgr2=0x7f28e00fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:10.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.502+0000 7f28e5b22700 1 -- 172.21.15.104:0/3406876119 shutdown_connections 2024-06-09T22:20:10.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.502+0000 7f28e5b22700 1 -- 172.21.15.104:0/3406876119 wait complete. 2024-06-09T22:20:10.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.502+0000 7f28e5b22700 1 Processor -- start 2024-06-09T22:20:10.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28e5b22700 1 -- start start 2024-06-09T22:20:10.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28e5b22700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0100890 0x7f28e01987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:10.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28e5b22700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0103870 0x7f28e0198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:10.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28e5b22700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28e0104210 0x7f28e019fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:10.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28e5b22700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010e240 con 0x7f28e0103870 2024-06-09T22:20:10.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28e5b22700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010e0c0 con 0x7f28e0100890 2024-06-09T22:20:10.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28e5b22700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010e3c0 con 0x7f28e0104210 2024-06-09T22:20:10.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28deffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0103870 0x7f28e0198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:10.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28deffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0103870 0x7f28e0198d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46348/0 (socket says 172.21.15.104:46348) 2024-06-09T22:20:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28deffd700 1 -- 172.21.15.104:0/663021404 learned_addr learned my addr 172.21.15.104:0/663021404 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28df7fe700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0100890 0x7f28e01987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28dffff700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28e0104210 0x7f28e019fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28deffd700 1 -- 172.21.15.104:0/663021404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28e0104210 msgr2=0x7f28e019fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28deffd700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28e0104210 0x7f28e019fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28deffd700 1 -- 172.21.15.104:0/663021404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0100890 msgr2=0x7f28e01987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28deffd700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0100890 0x7f28e01987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28deffd700 1 -- 172.21.15.104:0/663021404 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e01a0450 con 0x7f28e0103870 2024-06-09T22:20:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28df7fe700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0100890 0x7f28e01987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.503+0000 7f28deffd700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0103870 0x7f28e0198d30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f28c8004790 tx=0x7f28c8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.504+0000 7f28dcff9700 1 -- 172.21.15.104:0/663021404 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8027070 con 0x7f28e0103870 2024-06-09T22:20:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.504+0000 7f28dcff9700 1 -- 172.21.15.104:0/663021404 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c8004210 con 0x7f28e0103870 2024-06-09T22:20:10.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.504+0000 7f28dcff9700 1 -- 172.21.15.104:0/663021404 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c800f460 con 0x7f28e0103870 2024-06-09T22:20:10.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.504+0000 7f28e5b22700 1 -- 172.21.15.104:0/663021404 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e01a06e0 con 0x7f28e0103870 2024-06-09T22:20:10.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.504+0000 7f28e5b22700 1 -- 172.21.15.104:0/663021404 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e01a0b30 con 0x7f28e0103870 2024-06-09T22:20:10.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.505+0000 7f28e5b22700 1 -- 172.21.15.104:0/663021404 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e0069180 con 0x7f28e0103870 2024-06-09T22:20:10.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.506+0000 7f28dcff9700 1 -- 172.21.15.104:0/663021404 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f28c802a070 con 0x7f28e0103870 2024-06-09T22:20:10.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.506+0000 7f28dcff9700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f28cc0a2c20 0x7f28cc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:10.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.506+0000 7f28df7fe700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f28cc0a2c20 0x7f28cc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:10.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.507+0000 7f28dcff9700 1 -- 172.21.15.104:0/663021404 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f28c80d01b0 con 0x7f28e0103870 2024-06-09T22:20:10.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.507+0000 7f28df7fe700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f28cc0a2c20 0x7f28cc0a50e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f28e006a9a0 tx=0x7f28d0009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:10.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.511+0000 7f28dcff9700 1 -- 172.21.15.104:0/663021404 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f28c8013070 con 0x7f28e0103870 2024-06-09T22:20:10.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.719+0000 7f28e5b22700 1 -- 172.21.15.104:0/663021404 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f28e00628d0 con 0x7f28cc0a2c20 2024-06-09T22:20:10.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.732+0000 7f28dcff9700 1 -- 172.21.15.104:0/663021404 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f28e00628d0 con 0x7f28cc0a2c20 2024-06-09T22:20:10.730 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mds.a smithi104 running (3m) 41s ago 3m 18.1M - 19.0.0-2946-g3d89accf b045ac545f82 cdde291a6221 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mds.b smithi153 running (3m) 84s ago 3m 18.0M - 19.0.0-2946-g3d89accf b045ac545f82 0c85ecc8056b 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mds.c smithi194 running (3m) 84s ago 3m 20.6M - 19.0.0-2946-g3d89accf b045ac545f82 d0aec5967e0a 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mds.d smithi104 running (4m) 41s ago 4m 13.6M - 19.0.0-2946-g3d89accf b045ac545f82 5cc820ab0504 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mds.e smithi153 running (4m) 84s ago 4m 17.3M - 19.0.0-2946-g3d89accf b045ac545f82 c5ae1e6e050f 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mds.f smithi194 running (4m) 84s ago 3m 18.6M - 19.0.0-2946-g3d89accf b045ac545f82 ea3e90d5e7ea 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mds.g smithi104 running (3m) 41s ago 3m 18.0M - 19.0.0-2946-g3d89accf b045ac545f82 14860765eeef 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mds.h smithi153 running (3m) 84s ago 3m 20.6M - 19.0.0-2946-g3d89accf b045ac545f82 6a8f04b33aec 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mds.i smithi194 running (3m) 84s ago 3m 26.1M - 19.0.0-2946-g3d89accf b045ac545f82 8ffd8f55d181 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mds.j smithi104 running (4m) 41s ago 4m 15.2M - 19.0.0-2946-g3d89accf b045ac545f82 b1baca679f74 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mds.k smithi153 running (4m) 84s ago 4m 17.4M - 19.0.0-2946-g3d89accf b045ac545f82 d8e017896f09 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mds.l smithi194 running (4m) 84s ago 4m 17.0M - 19.0.0-2946-g3d89accf b045ac545f82 9373f7a032ca 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.x smithi104 *:9283,8765 running (11m) 41s ago 11m 518M - 19.0.0-2946-g3d89accf b045ac545f82 b6c0d3ce81db 2024-06-09T22:20:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.y smithi153 *:8443,8765 running (9m) 84s ago 9m 453M - 19.0.0-2946-g3d89accf b045ac545f82 0827a1ba0af3 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.z smithi194 *:8443,8765 running (9m) 84s ago 9m 456M - 19.0.0-2946-g3d89accf b045ac545f82 0b98115dc341 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:mon.a smithi104 running (11m) 41s ago 11m 51.1M 2048M 19.0.0-2946-g3d89accf b045ac545f82 2cb3d4ded8f6 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:mon.b smithi153 running (9m) 84s ago 9m 39.7M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7de2111927b6 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:mon.c smithi194 running (10m) 84s ago 10m 39.3M 2048M 19.0.0-2946-g3d89accf b045ac545f82 1e622cac1cc7 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (9m) 41s ago 9m 57.8M 4272M 19.0.0-2946-g3d89accf b045ac545f82 00b495df162d 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi153 running (8m) 84s ago 8m 58.2M 4272M 19.0.0-2946-g3d89accf b045ac545f82 3f6c64c59e84 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi194 running (8m) 84s ago 8m 49.6M 4272M 19.0.0-2946-g3d89accf b045ac545f82 3aacd8b0174f 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (8m) 41s ago 8m 47.7M 4272M 19.0.0-2946-g3d89accf b045ac545f82 04dcef22b256 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi153 running (7m) 84s ago 7m 55.7M 4272M 19.0.0-2946-g3d89accf b045ac545f82 26a600117843 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi194 running (7m) 84s ago 7m 56.4M 4272M 19.0.0-2946-g3d89accf b045ac545f82 44790d227297 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi104 running (6m) 41s ago 6m 53.4M 4272M 19.0.0-2946-g3d89accf b045ac545f82 da309d072339 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi153 running (6m) 84s ago 6m 55.7M 4272M 19.0.0-2946-g3d89accf b045ac545f82 85308217697b 2024-06-09T22:20:10.732 INFO:teuthology.orchestra.run.smithi104.stdout:osd.8 smithi194 running (5m) 84s ago 5m 57.4M 4272M 19.0.0-2946-g3d89accf b045ac545f82 42996772a59e 2024-06-09T22:20:10.733 INFO:teuthology.orchestra.run.smithi104.stdout:osd.9 smithi104 running (5m) 41s ago 5m 43.1M 4272M 19.0.0-2946-g3d89accf b045ac545f82 336722146f56 2024-06-09T22:20:10.733 INFO:teuthology.orchestra.run.smithi104.stdout:osd.10 smithi153 running (5m) 84s ago 5m 56.2M 4272M 19.0.0-2946-g3d89accf b045ac545f82 12f75f292eea 2024-06-09T22:20:10.733 INFO:teuthology.orchestra.run.smithi104.stdout:osd.11 smithi194 running (4m) 84s ago 4m 42.9M 4272M 19.0.0-2946-g3d89accf b045ac545f82 4e38c921712f 2024-06-09T22:20:10.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.734+0000 7f28e5b22700 1 -- 172.21.15.104:0/663021404 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f28cc0a2c20 msgr2=0x7f28cc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:10.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.734+0000 7f28e5b22700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f28cc0a2c20 0x7f28cc0a50e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f28e006a9a0 tx=0x7f28d0009570 comp rx=0 tx=0).stop 2024-06-09T22:20:10.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.734+0000 7f28e5b22700 1 -- 172.21.15.104:0/663021404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0103870 msgr2=0x7f28e0198d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:10.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.734+0000 7f28e5b22700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0103870 0x7f28e0198d30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f28c8004790 tx=0x7f28c8009580 comp rx=0 tx=0).stop 2024-06-09T22:20:10.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.735+0000 7f28e5b22700 1 -- 172.21.15.104:0/663021404 shutdown_connections 2024-06-09T22:20:10.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.735+0000 7f28e5b22700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f28cc0a2c20 0x7f28cc0a50e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:10.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.735+0000 7f28e5b22700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28e0104210 0x7f28e019fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:10.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.735+0000 7f28e5b22700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0100890 0x7f28e01987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:10.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.735+0000 7f28e5b22700 1 --2- 172.21.15.104:0/663021404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0103870 0x7f28e0198d30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:10.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.735+0000 7f28e5b22700 1 -- 172.21.15.104:0/663021404 >> 172.21.15.104:0/663021404 conn(0x7f28e00fc1f0 msgr2=0x7f28e0106b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:10.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.735+0000 7f28e5b22700 1 -- 172.21.15.104:0/663021404 shutdown_connections 2024-06-09T22:20:10.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:10.735+0000 7f28e5b22700 1 -- 172.21.15.104:0/663021404 wait complete. 2024-06-09T22:20:10.744 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:10 smithi104 ceph-mon[32389]: pgmap v411: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:10.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:10 smithi194 ceph-mon[36856]: pgmap v411: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:10.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:10 smithi153 ceph-mon[36599]: pgmap v411: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:11.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ls' 2024-06-09T22:20:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:12.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:12 smithi194 ceph-mon[36856]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:12.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:12 smithi194 ceph-mon[36856]: pgmap v412: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:12.934 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:12 smithi104 ceph-mon[32389]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:12.934 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:12 smithi104 ceph-mon[32389]: pgmap v412: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:12.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:12 smithi153 ceph-mon[36599]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:12.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:12 smithi153 ceph-mon[36599]: pgmap v412: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:13.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.503+0000 7f8f3214f700 1 -- 172.21.15.104:0/3502914964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f2c1024e0 msgr2=0x7f8f2c102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.503+0000 7f8f3214f700 1 --2- 172.21.15.104:0/3502914964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f2c1024e0 0x7f8f2c102900 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c0099c0 tx=0x7f8f1c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:20:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.503+0000 7f8f3214f700 1 -- 172.21.15.104:0/3502914964 shutdown_connections 2024-06-09T22:20:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.503+0000 7f8f3214f700 1 --2- 172.21.15.104:0/3502914964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f2c1024e0 0x7f8f2c102900 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.503+0000 7f8f3214f700 1 --2- 172.21.15.104:0/3502914964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f2c104080 0x7f8f2c10a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.503+0000 7f8f3214f700 1 --2- 172.21.15.104:0/3502914964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f2c1036e0 0x7f8f2c103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.503+0000 7f8f3214f700 1 -- 172.21.15.104:0/3502914964 >> 172.21.15.104:0/3502914964 conn(0x7f8f2c0fda80 msgr2=0x7f8f2c0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.503+0000 7f8f3214f700 1 -- 172.21.15.104:0/3502914964 shutdown_connections 2024-06-09T22:20:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.503+0000 7f8f3214f700 1 -- 172.21.15.104:0/3502914964 wait complete. 2024-06-09T22:20:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.504+0000 7f8f3214f700 1 Processor -- start 2024-06-09T22:20:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.504+0000 7f8f3214f700 1 -- start start 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.504+0000 7f8f3214f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f2c1024e0 0x7f8f2c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.504+0000 7f8f3214f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f2c1036e0 0x7f8f2c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f3214f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f2c104080 0x7f8f2c19fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f3214f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2c10e130 con 0x7f8f2c1024e0 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f3214f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2c10dfb0 con 0x7f8f2c1036e0 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f3214f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2c10e2b0 con 0x7f8f2c104080 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f2b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f2c1024e0 0x7f8f2c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f2b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f2c1024e0 0x7f8f2c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46356/0 (socket says 172.21.15.104:46356) 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f2b7fe700 1 -- 172.21.15.104:0/1419365121 learned_addr learned my addr 172.21.15.104:0/1419365121 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f2affd700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f2c1036e0 0x7f8f2c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f2b7fe700 1 -- 172.21.15.104:0/1419365121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f2c104080 msgr2=0x7f8f2c19fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f2b7fe700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f2c104080 0x7f8f2c19fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f2b7fe700 1 -- 172.21.15.104:0/1419365121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f2c1036e0 msgr2=0x7f8f2c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:13.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f2b7fe700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f2c1036e0 0x7f8f2c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:13.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f2b7fe700 1 -- 172.21.15.104:0/1419365121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2c1a03b0 con 0x7f8f2c1024e0 2024-06-09T22:20:13.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f2affd700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f2c1036e0 0x7f8f2c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:13.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f2b7fe700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f2c1024e0 0x7f8f2c198750 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009510 tx=0x7f8f1c009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:13.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f28ff9700 1 -- 172.21.15.104:0/1419365121 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c027070 con 0x7f8f2c1024e0 2024-06-09T22:20:13.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f3214f700 1 -- 172.21.15.104:0/1419365121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f2c1a0640 con 0x7f8f2c1024e0 2024-06-09T22:20:13.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f28ff9700 1 -- 172.21.15.104:0/1419365121 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f1c004150 con 0x7f8f2c1024e0 2024-06-09T22:20:13.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f28ff9700 1 -- 172.21.15.104:0/1419365121 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c00f460 con 0x7f8f2c1024e0 2024-06-09T22:20:13.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.505+0000 7f8f3214f700 1 -- 172.21.15.104:0/1419365121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f2c1a0bf0 con 0x7f8f2c1024e0 2024-06-09T22:20:13.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.506+0000 7f8f3214f700 1 -- 172.21.15.104:0/1419365121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f2c069180 con 0x7f8f2c1024e0 2024-06-09T22:20:13.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.507+0000 7f8f28ff9700 1 -- 172.21.15.104:0/1419365121 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8f1c02a070 con 0x7f8f2c1024e0 2024-06-09T22:20:13.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.508+0000 7f8f28ff9700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f8f140a2c20 0x7f8f140a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:13.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.508+0000 7f8f2affd700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f8f140a2c20 0x7f8f140a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:13.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.508+0000 7f8f28ff9700 1 -- 172.21.15.104:0/1419365121 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f8f1c0d0ab0 con 0x7f8f2c1024e0 2024-06-09T22:20:13.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.508+0000 7f8f2affd700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f8f140a2c20 0x7f8f140a50e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8f200099d0 tx=0x7f8f20008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:13.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.512+0000 7f8f28ff9700 1 -- 172.21.15.104:0/1419365121 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8f1c031070 con 0x7f8f2c1024e0 2024-06-09T22:20:13.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.715+0000 7f8f3214f700 1 -- 172.21.15.104:0/1419365121 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f8f2c0628d0 con 0x7f8f140a2c20 2024-06-09T22:20:13.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.721+0000 7f8f28ff9700 1 -- 172.21.15.104:0/1419365121 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f8f2c0628d0 con 0x7f8f140a2c20 2024-06-09T22:20:13.719 INFO:teuthology.orchestra.run.smithi104.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-06-09T22:20:13.719 INFO:teuthology.orchestra.run.smithi104.stdout:mds.all 12/12 87s ago 4m smithi104=a;smithi104=d;smithi104=g;smithi104=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-06-09T22:20:13.720 INFO:teuthology.orchestra.run.smithi104.stdout:mgr 3/3 87s ago 9m smithi104=x;smithi153=y;smithi194=z;count:3 2024-06-09T22:20:13.720 INFO:teuthology.orchestra.run.smithi104.stdout:mon 3/3 87s ago 10m smithi104:172.21.15.104=a;smithi153:172.21.15.153=b;smithi194:172.21.15.194=c;count:3 2024-06-09T22:20:13.720 INFO:teuthology.orchestra.run.smithi104.stdout:osd 12 87s ago - 2024-06-09T22:20:13.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.723+0000 7f8f3214f700 1 -- 172.21.15.104:0/1419365121 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f8f140a2c20 msgr2=0x7f8f140a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:13.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.723+0000 7f8f3214f700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f8f140a2c20 0x7f8f140a50e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8f200099d0 tx=0x7f8f20008040 comp rx=0 tx=0).stop 2024-06-09T22:20:13.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.723+0000 7f8f3214f700 1 -- 172.21.15.104:0/1419365121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f2c1024e0 msgr2=0x7f8f2c198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:13.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.723+0000 7f8f3214f700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f2c1024e0 0x7f8f2c198750 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009510 tx=0x7f8f1c009f10 comp rx=0 tx=0).stop 2024-06-09T22:20:13.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.723+0000 7f8f3214f700 1 -- 172.21.15.104:0/1419365121 shutdown_connections 2024-06-09T22:20:13.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.723+0000 7f8f3214f700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f8f140a2c20 0x7f8f140a50e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:13.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.723+0000 7f8f3214f700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f2c1024e0 0x7f8f2c198750 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:13.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.723+0000 7f8f3214f700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f2c104080 0x7f8f2c19fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:13.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.723+0000 7f8f3214f700 1 --2- 172.21.15.104:0/1419365121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f2c1036e0 0x7f8f2c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:13.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.723+0000 7f8f3214f700 1 -- 172.21.15.104:0/1419365121 >> 172.21.15.104:0/1419365121 conn(0x7f8f2c0fda80 msgr2=0x7f8f2c106a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:13.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.724+0000 7f8f3214f700 1 -- 172.21.15.104:0/1419365121 shutdown_connections 2024-06-09T22:20:13.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:13.724+0000 7f8f3214f700 1 -- 172.21.15.104:0/1419365121 wait complete. 2024-06-09T22:20:14.455 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch host ls' 2024-06-09T22:20:14.707 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:14.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:14 smithi104 ceph-mon[32389]: pgmap v413: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:14.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:14 smithi194 ceph-mon[36856]: pgmap v413: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:14.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:14 smithi153 ceph-mon[36599]: pgmap v413: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:15.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:15 smithi104 ceph-mon[32389]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:15.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:15 smithi194 ceph-mon[36856]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:15.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:15 smithi153 ceph-mon[36599]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:16.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.471+0000 7fe0ecbac700 1 -- 172.21.15.104:0/2536161649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0e81013f0 msgr2=0x7fe0e8101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:16.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.471+0000 7fe0ecbac700 1 --2- 172.21.15.104:0/2536161649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0e81013f0 0x7fe0e8101810 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe0d80099c0 tx=0x7fe0d8009cd0 comp rx=0 tx=0).stop 2024-06-09T22:20:16.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.471+0000 7fe0ecbac700 1 -- 172.21.15.104:0/2536161649 shutdown_connections 2024-06-09T22:20:16.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.471+0000 7fe0ecbac700 1 --2- 172.21.15.104:0/2536161649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0e81013f0 0x7fe0e8101810 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:16.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.471+0000 7fe0ecbac700 1 --2- 172.21.15.104:0/2536161649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e8102f90 0x7fe0e8109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:16.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.471+0000 7fe0ecbac700 1 --2- 172.21.15.104:0/2536161649 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0e81025f0 0x7fe0e8102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:16.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.471+0000 7fe0ecbac700 1 -- 172.21.15.104:0/2536161649 >> 172.21.15.104:0/2536161649 conn(0x7fe0e80fc990 msgr2=0x7fe0e80fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:16.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.471+0000 7fe0ecbac700 1 -- 172.21.15.104:0/2536161649 shutdown_connections 2024-06-09T22:20:16.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.471+0000 7fe0ecbac700 1 -- 172.21.15.104:0/2536161649 wait complete. 2024-06-09T22:20:16.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0ecbac700 1 Processor -- start 2024-06-09T22:20:16.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0ecbac700 1 -- start start 2024-06-09T22:20:16.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0ecbac700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0e81013f0 0x7fe0e806acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:16.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0ecbac700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0e81025f0 0x7fe0e819d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:16.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0ecbac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e8102f90 0x7fe0e819fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0ecbac700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e810cfd0 con 0x7fe0e81025f0 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0ecbac700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e810ce50 con 0x7fe0e8102f90 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0ecbac700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e810d150 con 0x7fe0e81013f0 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0dffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0e81025f0 0x7fe0e819d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0dffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0e81025f0 0x7fe0e819d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40872/0 (socket says 172.21.15.104:40872) 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0dffff700 1 -- 172.21.15.104:0/1953523492 learned_addr learned my addr 172.21.15.104:0/1953523492 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0e659c700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0e81013f0 0x7fe0e806acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0e6d9d700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e8102f90 0x7fe0e819fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0dffff700 1 -- 172.21.15.104:0/1953523492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0e81013f0 msgr2=0x7fe0e806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0dffff700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0e81013f0 0x7fe0e806acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0dffff700 1 -- 172.21.15.104:0/1953523492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e8102f90 msgr2=0x7fe0e819fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0dffff700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e8102f90 0x7fe0e819fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.472+0000 7fe0dffff700 1 -- 172.21.15.104:0/1953523492 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e81a0380 con 0x7fe0e81025f0 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.473+0000 7fe0dffff700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0e81025f0 0x7fe0e819d470 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000d980 tx=0x7fe0d000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.473+0000 7fe0df7fe700 1 -- 172.21.15.104:0/1953523492 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0017070 con 0x7fe0e81025f0 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.473+0000 7fe0df7fe700 1 -- 172.21.15.104:0/1953523492 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0d0005020 con 0x7fe0e81025f0 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.473+0000 7fe0ecbac700 1 -- 172.21.15.104:0/1953523492 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e81a0670 con 0x7fe0e81025f0 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.473+0000 7fe0df7fe700 1 -- 172.21.15.104:0/1953523492 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d000fbf0 con 0x7fe0e81025f0 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.473+0000 7fe0ecbac700 1 -- 172.21.15.104:0/1953523492 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e81a0bb0 con 0x7fe0e81025f0 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.474+0000 7fe0ecbac700 1 -- 172.21.15.104:0/1953523492 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e8069180 con 0x7fe0e81025f0 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.475+0000 7fe0df7fe700 1 -- 172.21.15.104:0/1953523492 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fe0d00051d0 con 0x7fe0e81025f0 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.476+0000 7fe0df7fe700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe0d40a2ca0 0x7fe0d40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.476+0000 7fe0e659c700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe0d40a2ca0 0x7fe0d40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.476+0000 7fe0df7fe700 1 -- 172.21.15.104:0/1953523492 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fe0d00cf190 con 0x7fe0e81025f0 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.476+0000 7fe0e659c700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe0d40a2ca0 0x7fe0d40a5160 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8000c00 tx=0x7fe0d8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:16.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.480+0000 7fe0df7fe700 1 -- 172.21.15.104:0/1953523492 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe0d0020020 con 0x7fe0e81025f0 2024-06-09T22:20:16.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.686+0000 7fe0ecbac700 1 -- 172.21.15.104:0/1953523492 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fe0e80628d0 con 0x7fe0d40a2ca0 2024-06-09T22:20:16.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.687+0000 7fe0df7fe700 1 -- 172.21.15.104:0/1953523492 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fe0e80628d0 con 0x7fe0d40a2ca0 2024-06-09T22:20:16.685 INFO:teuthology.orchestra.run.smithi104.stdout:HOST ADDR LABELS STATUS 2024-06-09T22:20:16.685 INFO:teuthology.orchestra.run.smithi104.stdout:smithi104 172.21.15.104 2024-06-09T22:20:16.685 INFO:teuthology.orchestra.run.smithi104.stdout:smithi153 172.21.15.153 2024-06-09T22:20:16.685 INFO:teuthology.orchestra.run.smithi104.stdout:smithi194 172.21.15.194 2024-06-09T22:20:16.685 INFO:teuthology.orchestra.run.smithi104.stdout:3 hosts in cluster 2024-06-09T22:20:16.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.689+0000 7fe0ecbac700 1 -- 172.21.15.104:0/1953523492 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe0d40a2ca0 msgr2=0x7fe0d40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:16.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.689+0000 7fe0ecbac700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe0d40a2ca0 0x7fe0d40a5160 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8000c00 tx=0x7fe0d8005fb0 comp rx=0 tx=0).stop 2024-06-09T22:20:16.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.689+0000 7fe0ecbac700 1 -- 172.21.15.104:0/1953523492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0e81025f0 msgr2=0x7fe0e819d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:16.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.689+0000 7fe0ecbac700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0e81025f0 0x7fe0e819d470 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000d980 tx=0x7fe0d000dc90 comp rx=0 tx=0).stop 2024-06-09T22:20:16.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.689+0000 7fe0ecbac700 1 -- 172.21.15.104:0/1953523492 shutdown_connections 2024-06-09T22:20:16.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.689+0000 7fe0ecbac700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe0d40a2ca0 0x7fe0d40a5160 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:16.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.689+0000 7fe0ecbac700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e8102f90 0x7fe0e819fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:16.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.689+0000 7fe0ecbac700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0e81013f0 0x7fe0e806acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:16.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.689+0000 7fe0ecbac700 1 --2- 172.21.15.104:0/1953523492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0e81025f0 0x7fe0e819d470 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:16.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.689+0000 7fe0ecbac700 1 -- 172.21.15.104:0/1953523492 >> 172.21.15.104:0/1953523492 conn(0x7fe0e80fc990 msgr2=0x7fe0e81058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:16.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.689+0000 7fe0ecbac700 1 -- 172.21.15.104:0/1953523492 shutdown_connections 2024-06-09T22:20:16.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:16.689+0000 7fe0ecbac700 1 -- 172.21.15.104:0/1953523492 wait complete. 2024-06-09T22:20:16.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:16 smithi104 ceph-mon[32389]: pgmap v414: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:16.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:16 smithi194 ceph-mon[36856]: pgmap v414: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:16.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:16 smithi153 ceph-mon[36599]: pgmap v414: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:17.408 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch device ls' 2024-06-09T22:20:17.653 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:18.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:18 smithi104 ceph-mon[32389]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:18.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:18 smithi104 ceph-mon[32389]: pgmap v415: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:18.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:18 smithi194 ceph-mon[36856]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:18.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:18 smithi194 ceph-mon[36856]: pgmap v415: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:18.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:18 smithi153 ceph-mon[36599]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:18.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:18 smithi153 ceph-mon[36599]: pgmap v415: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.329+0000 7f90f555a700 1 -- 172.21.15.104:0/3147930614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f01007a0 msgr2=0x7f90f0100bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.329+0000 7f90f555a700 1 --2- 172.21.15.104:0/3147930614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f01007a0 0x7f90f0100bc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009990 tx=0x7f90e0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:20:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.329+0000 7f90f555a700 1 -- 172.21.15.104:0/3147930614 shutdown_connections 2024-06-09T22:20:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.329+0000 7f90f555a700 1 --2- 172.21.15.104:0/3147930614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f01007a0 0x7f90f0100bc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.329+0000 7f90f555a700 1 --2- 172.21.15.104:0/3147930614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104120 0x7f90f010a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.329+0000 7f90f555a700 1 --2- 172.21.15.104:0/3147930614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f0103780 0x7f90f0103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.329+0000 7f90f555a700 1 -- 172.21.15.104:0/3147930614 >> 172.21.15.104:0/3147930614 conn(0x7f90f00fc120 msgr2=0x7f90f00fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.329+0000 7f90f555a700 1 -- 172.21.15.104:0/3147930614 shutdown_connections 2024-06-09T22:20:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.329+0000 7f90f555a700 1 -- 172.21.15.104:0/3147930614 wait complete. 2024-06-09T22:20:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.329+0000 7f90f555a700 1 Processor -- start 2024-06-09T22:20:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90f555a700 1 -- start start 2024-06-09T22:20:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90f555a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f01007a0 0x7f90f0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90f555a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f0103780 0x7f90f0198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90f555a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104120 0x7f90f019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90f555a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f90f010e0b0 con 0x7f90f0103780 2024-06-09T22:20:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90f555a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f90f010df30 con 0x7f90f01007a0 2024-06-09T22:20:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90f555a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f90f010e230 con 0x7f90f0104120 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90e67fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f0103780 0x7f90f0198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90e67fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f0103780 0x7f90f0198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40894/0 (socket says 172.21.15.104:40894) 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90e67fc700 1 -- 172.21.15.104:0/3901242259 learned_addr learned my addr 172.21.15.104:0/3901242259 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90eeffd700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f01007a0 0x7f90f0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90ef7fe700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104120 0x7f90f019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90e67fc700 1 -- 172.21.15.104:0/3901242259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104120 msgr2=0x7f90f019fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90e67fc700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104120 0x7f90f019fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90e67fc700 1 -- 172.21.15.104:0/3901242259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f01007a0 msgr2=0x7f90f0198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90e67fc700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f01007a0 0x7f90f0198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90e67fc700 1 -- 172.21.15.104:0/3901242259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f01a03a0 con 0x7f90f0103780 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.330+0000 7f90ef7fe700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104120 0x7f90f019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.331+0000 7f90e67fc700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f0103780 0x7f90f0198c80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f90d800d9b0 tx=0x7f90d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.331+0000 7f90ecff9700 1 -- 172.21.15.104:0/3901242259 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d8016070 con 0x7f90f0103780 2024-06-09T22:20:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.331+0000 7f90f555a700 1 -- 172.21.15.104:0/3901242259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90f01a0690 con 0x7f90f0103780 2024-06-09T22:20:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.331+0000 7f90ecff9700 1 -- 172.21.15.104:0/3901242259 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90d8004d10 con 0x7f90f0103780 2024-06-09T22:20:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.331+0000 7f90ecff9700 1 -- 172.21.15.104:0/3901242259 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d8005630 con 0x7f90f0103780 2024-06-09T22:20:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.331+0000 7f90f555a700 1 -- 172.21.15.104:0/3901242259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90f01a0bd0 con 0x7f90f0103780 2024-06-09T22:20:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.332+0000 7f90f555a700 1 -- 172.21.15.104:0/3901242259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90f0069180 con 0x7f90f0103780 2024-06-09T22:20:19.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.333+0000 7f90ecff9700 1 -- 172.21.15.104:0/3901242259 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f90d8004750 con 0x7f90f0103780 2024-06-09T22:20:19.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.334+0000 7f90ecff9700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f90dc0a2c20 0x7f90dc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:19.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.334+0000 7f90eeffd700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f90dc0a2c20 0x7f90dc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:19.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.334+0000 7f90ecff9700 1 -- 172.21.15.104:0/3901242259 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f90d80cf270 con 0x7f90f0103780 2024-06-09T22:20:19.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.334+0000 7f90eeffd700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f90dc0a2c20 0x7f90dc0a50e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f90e0003890 tx=0x7f90e0003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:19.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.338+0000 7f90ecff9700 1 -- 172.21.15.104:0/3901242259 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f90d801f020 con 0x7f90f0103780 2024-06-09T22:20:19.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.547+0000 7f90f555a700 1 -- 172.21.15.104:0/3901242259 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f90f00628d0 con 0x7f90dc0a2c20 2024-06-09T22:20:19.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.549+0000 7f90ecff9700 1 -- 172.21.15.104:0/3901242259 <== mgr.14152 v2:172.21.15.104:6800/1906048760 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f90f00628d0 con 0x7f90dc0a2c20 2024-06-09T22:20:19.546 INFO:teuthology.orchestra.run.smithi104.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-06-09T22:20:19.546 INFO:teuthology.orchestra.run.smithi104.stdout:smithi104 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NW400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-06-09T22:20:19.547 INFO:teuthology.orchestra.run.smithi104.stdout:smithi153 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300MV400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-06-09T22:20:19.547 INFO:teuthology.orchestra.run.smithi104.stdout:smithi194 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6240002A400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-06-09T22:20:19.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.551+0000 7f90f555a700 1 -- 172.21.15.104:0/3901242259 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f90dc0a2c20 msgr2=0x7f90dc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:19.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.551+0000 7f90f555a700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f90dc0a2c20 0x7f90dc0a50e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f90e0003890 tx=0x7f90e0003960 comp rx=0 tx=0).stop 2024-06-09T22:20:19.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.551+0000 7f90f555a700 1 -- 172.21.15.104:0/3901242259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f0103780 msgr2=0x7f90f0198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:19.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.551+0000 7f90f555a700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f0103780 0x7f90f0198c80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f90d800d9b0 tx=0x7f90d800dcc0 comp rx=0 tx=0).stop 2024-06-09T22:20:19.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.551+0000 7f90f555a700 1 -- 172.21.15.104:0/3901242259 shutdown_connections 2024-06-09T22:20:19.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.551+0000 7f90f555a700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f90dc0a2c20 0x7f90dc0a50e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:19.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.551+0000 7f90f555a700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104120 0x7f90f019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:19.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.551+0000 7f90f555a700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f01007a0 0x7f90f0198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:19.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.551+0000 7f90f555a700 1 --2- 172.21.15.104:0/3901242259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f0103780 0x7f90f0198c80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:19.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.551+0000 7f90f555a700 1 -- 172.21.15.104:0/3901242259 >> 172.21.15.104:0/3901242259 conn(0x7f90f00fc120 msgr2=0x7f90f0106b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:19.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.552+0000 7f90f555a700 1 -- 172.21.15.104:0/3901242259 shutdown_connections 2024-06-09T22:20:19.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:19.552+0000 7f90f555a700 1 -- 172.21.15.104:0/3901242259 wait complete. 2024-06-09T22:20:19.926 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:19 smithi104 ceph-mon[32389]: pgmap v416: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:19.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:19 smithi153 ceph-mon[36599]: pgmap v416: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:20.057 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:19 smithi194 ceph-mon[36856]: pgmap v416: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:20.260 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-09T22:20:20.269 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi104.front.sepia.ceph.com 2024-06-09T22:20:20.269 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- bash -c 'ceph fs dump' 2024-06-09T22:20:20.514 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:20.873 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:20 smithi104 ceph-mon[32389]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:20.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:20 smithi153 ceph-mon[36599]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:21.057 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:20 smithi194 ceph-mon[36856]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:21.956 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:21 smithi104 ceph-mon[32389]: pgmap v417: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:21.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:21 smithi153 ceph-mon[36599]: pgmap v417: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:22.057 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:21 smithi194 ceph-mon[36856]: pgmap v417: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:22.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.204+0000 7f30d2a4c700 1 -- 172.21.15.104:0/3845240062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30cc102610 msgr2=0x7f30cc102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:22.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.204+0000 7f30d2a4c700 1 --2- 172.21.15.104:0/3845240062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30cc102610 0x7f30cc102a70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0099c0 tx=0x7f30bc009cd0 comp rx=0 tx=0).stop 2024-06-09T22:20:22.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.204+0000 7f30d2a4c700 1 -- 172.21.15.104:0/3845240062 shutdown_connections 2024-06-09T22:20:22.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.204+0000 7f30d2a4c700 1 --2- 172.21.15.104:0/3845240062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30cc102fb0 0x7f30cc109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:22.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.204+0000 7f30d2a4c700 1 --2- 172.21.15.104:0/3845240062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30cc101410 0x7f30cc101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:22.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.204+0000 7f30d2a4c700 1 --2- 172.21.15.104:0/3845240062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30cc102610 0x7f30cc102a70 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:22.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.204+0000 7f30d2a4c700 1 -- 172.21.15.104:0/3845240062 >> 172.21.15.104:0/3845240062 conn(0x7f30cc0fc9f0 msgr2=0x7f30cc0fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:22.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.204+0000 7f30d2a4c700 1 -- 172.21.15.104:0/3845240062 shutdown_connections 2024-06-09T22:20:22.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.204+0000 7f30d2a4c700 1 -- 172.21.15.104:0/3845240062 wait complete. 2024-06-09T22:20:22.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.204+0000 7f30d2a4c700 1 Processor -- start 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d2a4c700 1 -- start start 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d2a4c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30cc101410 0x7f30cc198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d2a4c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30cc102610 0x7f30cc198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d2a4c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30cc102fb0 0x7f30cc19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d2a4c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc10cff0 con 0x7f30cc102fb0 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d2a4c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc10ce70 con 0x7f30cc101410 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d2a4c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc10d170 con 0x7f30cc102610 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d0fe9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30cc102fb0 0x7f30cc19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d0fe9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30cc102fb0 0x7f30cc19fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40914/0 (socket says 172.21.15.104:40914) 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d0fe9700 1 -- 172.21.15.104:0/1402523723 learned_addr learned my addr 172.21.15.104:0/1402523723 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30cbfff700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30cc101410 0x7f30cc198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30cb7fe700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30cc102610 0x7f30cc198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:22.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d0fe9700 1 -- 172.21.15.104:0/1402523723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30cc102610 msgr2=0x7f30cc198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d0fe9700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30cc102610 0x7f30cc198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d0fe9700 1 -- 172.21.15.104:0/1402523723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30cc101410 msgr2=0x7f30cc198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d0fe9700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30cc101410 0x7f30cc198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d0fe9700 1 -- 172.21.15.104:0/1402523723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30cc1a0340 con 0x7f30cc102fb0 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30cb7fe700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30cc102610 0x7f30cc198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.205+0000 7f30d0fe9700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30cc102fb0 0x7f30cc19fca0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f30c400ef60 tx=0x7f30c400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.206+0000 7f30c97fa700 1 -- 172.21.15.104:0/1402523723 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c40092e0 con 0x7f30cc102fb0 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.206+0000 7f30c97fa700 1 -- 172.21.15.104:0/1402523723 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30c4004bb0 con 0x7f30cc102fb0 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.206+0000 7f30d2a4c700 1 -- 172.21.15.104:0/1402523723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30cc1a05d0 con 0x7f30cc102fb0 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.206+0000 7f30c97fa700 1 -- 172.21.15.104:0/1402523723 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4004280 con 0x7f30cc102fb0 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.206+0000 7f30d2a4c700 1 -- 172.21.15.104:0/1402523723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30cc1a0b10 con 0x7f30cc102fb0 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.207+0000 7f30d2a4c700 1 -- 172.21.15.104:0/1402523723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30cc069180 con 0x7f30cc102fb0 2024-06-09T22:20:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.208+0000 7f30c97fa700 1 -- 172.21.15.104:0/1402523723 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f30c401b030 con 0x7f30cc102fb0 2024-06-09T22:20:22.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.209+0000 7f30c97fa700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f30b40a2c20 0x7f30b40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:22.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.209+0000 7f30cbfff700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f30b40a2c20 0x7f30b40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:22.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.209+0000 7f30c97fa700 1 -- 172.21.15.104:0/1402523723 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f30c40cf830 con 0x7f30cc102fb0 2024-06-09T22:20:22.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.209+0000 7f30cbfff700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f30b40a2c20 0x7f30b40a50e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f30cc102470 tx=0x7f30c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.213+0000 7f30c97fa700 1 -- 172.21.15.104:0/1402523723 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f30c4022020 con 0x7f30cc102fb0 2024-06-09T22:20:22.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.451+0000 7f30d2a4c700 1 -- 172.21.15.104:0/1402523723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f30cc1a0f30 con 0x7f30cc102fb0 2024-06-09T22:20:22.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.452+0000 7f30c97fa700 1 -- 172.21.15.104:0/1402523723 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3106 (secure 0 0 0) 0x7f30c4010040 con 0x7f30cc102fb0 2024-06-09T22:20:22.450 INFO:teuthology.orchestra.run.smithi104.stdout:e28 2024-06-09T22:20:22.450 INFO:teuthology.orchestra.run.smithi104.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-06-09T22:20:22.450 INFO:teuthology.orchestra.run.smithi104.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:legacy client fscid: 1 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:Filesystem 'cephfs' (1) 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:fs_name cephfs 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:epoch 28 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:created 2024-06-09T22:16:45.150445+0000 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:modified 2024-06-09T22:17:53.402012+0000 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:tableserver 0 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:root 0 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:session_timeout 300 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:session_autoclose 300 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:max_file_size 1099511627776 2024-06-09T22:20:22.451 INFO:teuthology.orchestra.run.smithi104.stdout:max_xattr_size 65536 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:required_client_features {} 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:last_failure 0 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:last_failure_osd_epoch 0 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:max_mds 5 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:in 0,1,2,3,4 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:up {0=24460,1=24448,2=24442,3=24419,4=24403} 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:failed 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:damaged 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:stopped 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:data_pools [3,4] 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:metadata_pool 2 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:inline_data disabled 2024-06-09T22:20:22.452 INFO:teuthology.orchestra.run.smithi104.stdout:balancer 2024-06-09T22:20:22.453 INFO:teuthology.orchestra.run.smithi104.stdout:bal_rank_mask -1 2024-06-09T22:20:22.453 INFO:teuthology.orchestra.run.smithi104.stdout:standby_count_wanted 0 2024-06-09T22:20:22.453 INFO:teuthology.orchestra.run.smithi104.stdout:qdb_cluster leader: 24460 members: 24403,24419,24442,24448,24460 2024-06-09T22:20:22.453 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.i{0:24460} state up:active seq 8 addr [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] compat {c=[1],r=[1],i=[1fff]}] 2024-06-09T22:20:22.453 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.h{0:24398} state up:standby-replay seq 1 addr [v2:172.21.15.153:6836/3294454895,v1:172.21.15.153:6837/3294454895] compat {c=[1],r=[1],i=[1fff]}] 2024-06-09T22:20:22.453 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.c{1:24448} state up:active seq 25 addr [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] compat {c=[1],r=[1],i=[1fff]}] 2024-06-09T22:20:22.453 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.e{1:24383} state up:standby-replay seq 1 addr [v2:172.21.15.153:6834/3075465176,v1:172.21.15.153:6835/3075465176] compat {c=[1],r=[1],i=[1fff]}] 2024-06-09T22:20:22.453 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.f{2:24442} state up:active seq 26 addr [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] compat {c=[1],r=[1],i=[1fff]}] 2024-06-09T22:20:22.453 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.k{2:24362} state up:standby-replay seq 1 addr [v2:172.21.15.153:6832/3968887191,v1:172.21.15.153:6833/3968887191] compat {c=[1],r=[1],i=[1fff]}] 2024-06-09T22:20:22.453 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.b{3:24419} state up:active seq 21 addr [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] compat {c=[1],r=[1],i=[1fff]}] 2024-06-09T22:20:22.453 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.a{3:14583} state up:standby-replay seq 1 addr [v2:172.21.15.104:6840/3830332242,v1:172.21.15.104:6841/3830332242] compat {c=[1],r=[1],i=[1fff]}] 2024-06-09T22:20:22.453 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.l{4:24403} state up:active seq 31 addr [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] compat {c=[1],r=[1],i=[1fff]}] 2024-06-09T22:20:22.454 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.g{4:14571} state up:standby-replay seq 1 addr [v2:172.21.15.104:6838/647544853,v1:172.21.15.104:6839/647544853] compat {c=[1],r=[1],i=[1fff]}] 2024-06-09T22:20:22.454 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:20:22.454 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:20:22.454 INFO:teuthology.orchestra.run.smithi104.stdout:Standby daemons: 2024-06-09T22:20:22.454 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:20:22.454 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.j{-1:14523} state up:standby seq 1 addr [v2:172.21.15.104:6834/3517082076,v1:172.21.15.104:6835/3517082076] compat {c=[1],r=[1],i=[1fff]}] 2024-06-09T22:20:22.454 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.d{-1:14535} state up:standby seq 1 addr [v2:172.21.15.104:6836/3060956100,v1:172.21.15.104:6837/3060956100] compat {c=[1],r=[1],i=[1fff]}] 2024-06-09T22:20:22.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.455+0000 7f30d2a4c700 1 -- 172.21.15.104:0/1402523723 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f30b40a2c20 msgr2=0x7f30b40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:22.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.455+0000 7f30d2a4c700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f30b40a2c20 0x7f30b40a50e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f30cc102470 tx=0x7f30c0009260 comp rx=0 tx=0).stop 2024-06-09T22:20:22.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.455+0000 7f30d2a4c700 1 -- 172.21.15.104:0/1402523723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30cc102fb0 msgr2=0x7f30cc19fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:22.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.455+0000 7f30d2a4c700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30cc102fb0 0x7f30cc19fca0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f30c400ef60 tx=0x7f30c400ef90 comp rx=0 tx=0).stop 2024-06-09T22:20:22.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.455+0000 7f30d2a4c700 1 -- 172.21.15.104:0/1402523723 shutdown_connections 2024-06-09T22:20:22.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.455+0000 7f30d2a4c700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f30b40a2c20 0x7f30b40a50e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:22.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.455+0000 7f30d2a4c700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30cc102fb0 0x7f30cc19fca0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:22.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.455+0000 7f30d2a4c700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30cc101410 0x7f30cc198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:22.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.455+0000 7f30d2a4c700 1 --2- 172.21.15.104:0/1402523723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30cc102610 0x7f30cc198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:22.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.455+0000 7f30d2a4c700 1 -- 172.21.15.104:0/1402523723 >> 172.21.15.104:0/1402523723 conn(0x7f30cc0fc9f0 msgr2=0x7f30cc0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:22.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.455+0000 7f30d2a4c700 1 -- 172.21.15.104:0/1402523723 shutdown_connections 2024-06-09T22:20:22.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:22.455+0000 7f30d2a4c700 1 -- 172.21.15.104:0/1402523723 wait complete. 2024-06-09T22:20:22.456 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 28 2024-06-09T22:20:22.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:22 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1402523723' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-06-09T22:20:22.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:22 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1402523723' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-06-09T22:20:23.057 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:22 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1402523723' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-06-09T22:20:23.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- bash -c 'ceph osd dump' 2024-06-09T22:20:23.417 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:23.824 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:23 smithi104 ceph-mon[32389]: pgmap v418: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:23.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:23 smithi153 ceph-mon[36599]: pgmap v418: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:24.057 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:23 smithi194 ceph-mon[36856]: pgmap v418: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:25.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.104+0000 7f9a399b1700 1 -- 172.21.15.104:0/4166855040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a34102f50 msgr2=0x7f9a34105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.104+0000 7f9a399b1700 1 --2- 172.21.15.104:0/4166855040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a34102f50 0x7f9a34105340 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00b0a0 tx=0x7f9a2c00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.104+0000 7f9a399b1700 1 -- 172.21.15.104:0/4166855040 shutdown_connections 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.104+0000 7f9a399b1700 1 --2- 172.21.15.104:0/4166855040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a34102f50 0x7f9a34105340 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.104+0000 7f9a399b1700 1 --2- 172.21.15.104:0/4166855040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a341081b0 0x7f9a3410a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.104+0000 7f9a399b1700 1 --2- 172.21.15.104:0/4166855040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a34105880 0x7f9a34107c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.104+0000 7f9a399b1700 1 -- 172.21.15.104:0/4166855040 >> 172.21.15.104:0/4166855040 conn(0x7f9a340fc960 msgr2=0x7f9a340feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.104+0000 7f9a399b1700 1 -- 172.21.15.104:0/4166855040 shutdown_connections 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.104+0000 7f9a399b1700 1 -- 172.21.15.104:0/4166855040 wait complete. 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a399b1700 1 Processor -- start 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a399b1700 1 -- start start 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a399b1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a34102f50 0x7f9a341028b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a399b1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34105880 0x7f9a34100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a399b1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a341081b0 0x7f9a34101440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a399b1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9a3410e090 con 0x7f9a34102f50 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a399b1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a3410df10 con 0x7f9a34105880 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a399b1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9a3410e210 con 0x7f9a341081b0 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a32ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a34102f50 0x7f9a341028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a32ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a34102f50 0x7f9a341028b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40928/0 (socket says 172.21.15.104:40928) 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a32ffd700 1 -- 172.21.15.104:0/1562379232 learned_addr learned my addr 172.21.15.104:0/1562379232 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a327fc700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34105880 0x7f9a34100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a337fe700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a341081b0 0x7f9a34101440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a32ffd700 1 -- 172.21.15.104:0/1562379232 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a341081b0 msgr2=0x7f9a34101440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a32ffd700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a341081b0 0x7f9a34101440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a32ffd700 1 -- 172.21.15.104:0/1562379232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34105880 msgr2=0x7f9a34100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a32ffd700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34105880 0x7f9a34100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.105+0000 7f9a32ffd700 1 -- 172.21.15.104:0/1562379232 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a34101cf0 con 0x7f9a34102f50 2024-06-09T22:20:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.106+0000 7f9a32ffd700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a34102f50 0x7f9a341028b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00d010 tx=0x7f9a2c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.106+0000 7f9a389af700 1 -- 172.21.15.104:0/1562379232 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c021070 con 0x7f9a34102f50 2024-06-09T22:20:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.106+0000 7f9a399b1700 1 -- 172.21.15.104:0/1562379232 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a341a27d0 con 0x7f9a34102f50 2024-06-09T22:20:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.106+0000 7f9a389af700 1 -- 172.21.15.104:0/1562379232 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a2c004020 con 0x7f9a34102f50 2024-06-09T22:20:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.106+0000 7f9a389af700 1 -- 172.21.15.104:0/1562379232 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c01d6b0 con 0x7f9a34102f50 2024-06-09T22:20:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.106+0000 7f9a399b1700 1 -- 172.21.15.104:0/1562379232 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a341a2c80 con 0x7f9a34102f50 2024-06-09T22:20:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.107+0000 7f9a399b1700 1 -- 172.21.15.104:0/1562379232 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a34069180 con 0x7f9a34102f50 2024-06-09T22:20:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.109+0000 7f9a389af700 1 -- 172.21.15.104:0/1562379232 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f9a2c0041d0 con 0x7f9a34102f50 2024-06-09T22:20:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.109+0000 7f9a389af700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9a200a2c20 0x7f9a200a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.109+0000 7f9a389af700 1 -- 172.21.15.104:0/1562379232 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f9a2c00f020 con 0x7f9a34102f50 2024-06-09T22:20:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.110+0000 7f9a327fc700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9a200a2c20 0x7f9a200a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.110+0000 7f9a327fc700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9a200a2c20 0x7f9a200a50e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c005950 tx=0x7f9a1c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:25.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.113+0000 7f9a389af700 1 -- 172.21.15.104:0/1562379232 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9a2c02a020 con 0x7f9a34102f50 2024-06-09T22:20:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.308+0000 7f9a399b1700 1 -- 172.21.15.104:0/1562379232 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f9a3402dca0 con 0x7f9a34102f50 2024-06-09T22:20:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.309+0000 7f9a389af700 1 -- 172.21.15.104:0/1562379232 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v99) ==== 56+0+5683 (secure 0 0 0) 0x7f9a2c09d740 con 0x7f9a34102f50 2024-06-09T22:20:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:epoch 99 2024-06-09T22:20:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:20:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:created 2024-06-09T22:08:31.471440+0000 2024-06-09T22:20:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:modified 2024-06-09T22:18:26.525714+0000 2024-06-09T22:20:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-06-09T22:20:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:crush_version 27 2024-06-09T22:20:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:full_ratio 0.95 2024-06-09T22:20:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:backfillfull_ratio 0.9 2024-06-09T22:20:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:nearfull_ratio 0.85 2024-06-09T22:20:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:require_min_compat_client luminous 2024-06-09T22:20:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:min_compat_client luminous 2024-06-09T22:20:25.308 INFO:teuthology.orchestra.run.smithi104.stdout:require_osd_release squid 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:stretch_mode_enabled false 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 88 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 88 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 95 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:max_osd 12 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 up in weight 1 up_from 8 up_thru 98 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6802/2202624505,v1:172.21.15.104:6803/2202624505] [v2:172.21.15.104:6804/2202624505,v1:172.21.15.104:6805/2202624505] exists,up 5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 up in weight 1 up_from 13 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6800/628783097,v1:172.21.15.153:6801/628783097] [v2:172.21.15.153:6802/628783097,v1:172.21.15.153:6803/628783097] exists,up aa718c73-5826-43d9-9b5b-c99ddec256da 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 up in weight 1 up_from 18 up_thru 98 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6800/120935084,v1:172.21.15.194:6801/120935084] [v2:172.21.15.194:6802/120935084,v1:172.21.15.194:6803/120935084] exists,up 35204639-9a6e-485d-a3b7-2d2330406b3e 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 up in weight 1 up_from 25 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6810/3003003340,v1:172.21.15.104:6811/3003003340] [v2:172.21.15.104:6812/3003003340,v1:172.21.15.104:6813/3003003340] exists,up 3acd0acc-fcf7-4122-a939-d961bfc03c0b 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 up in weight 1 up_from 33 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6808/1237651981,v1:172.21.15.153:6809/1237651981] [v2:172.21.15.153:6810/1237651981,v1:172.21.15.153:6811/1237651981] exists,up e0d2ceee-535e-48d8-90e7-5c5edb3eebe4 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 up in weight 1 up_from 38 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6808/3138023494,v1:172.21.15.194:6809/3138023494] [v2:172.21.15.194:6810/3138023494,v1:172.21.15.194:6811/3138023494] exists,up 49a8983d-15aa-47ef-83b9-72ea75fe4505 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 up in weight 1 up_from 46 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6818/4097161419,v1:172.21.15.104:6819/4097161419] [v2:172.21.15.104:6820/4097161419,v1:172.21.15.104:6821/4097161419] exists,up 08563350-3ded-4ff6-a214-12d9242f984a 2024-06-09T22:20:25.309 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 up in weight 1 up_from 51 up_thru 96 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6816/1289060024,v1:172.21.15.153:6817/1289060024] [v2:172.21.15.153:6818/1289060024,v1:172.21.15.153:6819/1289060024] exists,up b23baa2c-c27d-43a3-9820-8626c26d3f5f 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:osd.8 up in weight 1 up_from 59 up_thru 96 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6816/2851208300,v1:172.21.15.194:6817/2851208300] [v2:172.21.15.194:6818/2851208300,v1:172.21.15.194:6819/2851208300] exists,up 8beddca3-bc69-4ef9-955b-f0296e0f28bc 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:osd.9 up in weight 1 up_from 67 up_thru 96 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6826/3652005298,v1:172.21.15.104:6827/3652005298] [v2:172.21.15.104:6828/3652005298,v1:172.21.15.104:6829/3652005298] exists,up d82b0c48-b957-4209-afc9-020ded569c2c 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:osd.10 up in weight 1 up_from 72 up_thru 96 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6824/1184232655,v1:172.21.15.153:6825/1184232655] [v2:172.21.15.153:6826/1184232655,v1:172.21.15.153:6827/1184232655] exists,up 98aa6a7b-baac-4816-bf0c-fa0b01931b86 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:osd.11 up in weight 1 up_from 80 up_thru 96 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6824/2145259515,v1:172.21.15.194:6825/2145259515] [v2:172.21.15.194:6826/2145259515,v1:172.21.15.194:6827/2145259515] exists,up f107b3c6-8206-4cd0-bc41-5cc8eb8f179d 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:pg_upmap_items 2.4 [2,11] 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:pg_upmap_items 2.13 [2,11] 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:pg_upmap_items 2.1a [8,4] 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:pg_upmap_items 3.6 [4,6] 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:pg_upmap_items 3.10 [5,9] 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:pg_upmap_items 3.13 [4,9] 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:pg_upmap_items 3.1e [3,9] 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:pg_upmap_items 3.3a [4,8] 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:pg_upmap_items 4.b [9,10] 2024-06-09T22:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:pg_upmap_items 4.14 [9,8] 2024-06-09T22:20:25.311 INFO:teuthology.orchestra.run.smithi104.stdout:pg_upmap_items 4.17 [9,8] 2024-06-09T22:20:25.311 INFO:teuthology.orchestra.run.smithi104.stdout:pg_upmap_items 4.1c [5,8] 2024-06-09T22:20:25.311 INFO:teuthology.orchestra.run.smithi104.stdout:blocklist 172.21.15.104:6801/85752866 expires 2024-06-10T22:09:25.221731+0000 2024-06-09T22:20:25.311 INFO:teuthology.orchestra.run.smithi104.stdout:blocklist 172.21.15.104:0/3559678620 expires 2024-06-10T22:08:58.969000+0000 2024-06-09T22:20:25.311 INFO:teuthology.orchestra.run.smithi104.stdout:blocklist 172.21.15.104:0/3864904957 expires 2024-06-10T22:09:25.221731+0000 2024-06-09T22:20:25.311 INFO:teuthology.orchestra.run.smithi104.stdout:blocklist 172.21.15.104:6800/85752866 expires 2024-06-10T22:09:25.221731+0000 2024-06-09T22:20:25.311 INFO:teuthology.orchestra.run.smithi104.stdout:blocklist 172.21.15.104:6801/1879478526 expires 2024-06-10T22:08:58.969000+0000 2024-06-09T22:20:25.311 INFO:teuthology.orchestra.run.smithi104.stdout:blocklist 172.21.15.104:0/2471278860 expires 2024-06-10T22:08:58.969000+0000 2024-06-09T22:20:25.311 INFO:teuthology.orchestra.run.smithi104.stdout:blocklist 172.21.15.104:0/4119286692 expires 2024-06-10T22:09:25.221731+0000 2024-06-09T22:20:25.311 INFO:teuthology.orchestra.run.smithi104.stdout:blocklist 172.21.15.104:0/2266338289 expires 2024-06-10T22:08:58.969000+0000 2024-06-09T22:20:25.311 INFO:teuthology.orchestra.run.smithi104.stdout:blocklist 172.21.15.104:6800/1879478526 expires 2024-06-10T22:08:58.969000+0000 2024-06-09T22:20:25.311 INFO:teuthology.orchestra.run.smithi104.stdout:blocklist 172.21.15.104:0/3072042273 expires 2024-06-10T22:09:25.221731+0000 2024-06-09T22:20:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.311+0000 7f9a399b1700 1 -- 172.21.15.104:0/1562379232 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9a200a2c20 msgr2=0x7f9a200a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.311+0000 7f9a399b1700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9a200a2c20 0x7f9a200a50e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c005950 tx=0x7f9a1c00a300 comp rx=0 tx=0).stop 2024-06-09T22:20:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.311+0000 7f9a399b1700 1 -- 172.21.15.104:0/1562379232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a34102f50 msgr2=0x7f9a341028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.311+0000 7f9a399b1700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a34102f50 0x7f9a341028b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00d010 tx=0x7f9a2c00be80 comp rx=0 tx=0).stop 2024-06-09T22:20:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.311+0000 7f9a399b1700 1 -- 172.21.15.104:0/1562379232 shutdown_connections 2024-06-09T22:20:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.311+0000 7f9a399b1700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f9a200a2c20 0x7f9a200a50e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.311+0000 7f9a399b1700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a34102f50 0x7f9a341028b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.311+0000 7f9a399b1700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a341081b0 0x7f9a34101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.311+0000 7f9a399b1700 1 --2- 172.21.15.104:0/1562379232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34105880 0x7f9a34100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.311+0000 7f9a399b1700 1 -- 172.21.15.104:0/1562379232 >> 172.21.15.104:0/1562379232 conn(0x7f9a340fc960 msgr2=0x7f9a34103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.312+0000 7f9a399b1700 1 -- 172.21.15.104:0/1562379232 shutdown_connections 2024-06-09T22:20:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:25.312+0000 7f9a399b1700 1 -- 172.21.15.104:0/1562379232 wait complete. 2024-06-09T22:20:25.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:25 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1562379232' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-09T22:20:25.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:25 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1562379232' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-09T22:20:25.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:25 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1562379232' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-09T22:20:26.017 INFO:teuthology.run_tasks:Running task fs.ready... 2024-06-09T22:20:26.029 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:20:26.270 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:26.717 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:26 smithi104 ceph-mon[32389]: pgmap v419: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:26.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:26 smithi153 ceph-mon[36599]: pgmap v419: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:26.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:26 smithi194 ceph-mon[36856]: pgmap v419: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:27.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.988+0000 7ff71a6a5700 1 -- 172.21.15.104:0/1207544960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff714101650 msgr2=0x7ff714101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.988+0000 7ff71a6a5700 1 --2- 172.21.15.104:0/1207544960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff714101650 0x7ff714101a30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff704009920 tx=0x7ff704009c30 comp rx=0 tx=0).stop 2024-06-09T22:20:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.988+0000 7ff71a6a5700 1 -- 172.21.15.104:0/1207544960 shutdown_connections 2024-06-09T22:20:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.988+0000 7ff71a6a5700 1 --2- 172.21.15.104:0/1207544960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff714101650 0x7ff714101a30 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.988+0000 7ff71a6a5700 1 --2- 172.21.15.104:0/1207544960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7141066a0 0x7ff71410e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.988+0000 7ff71a6a5700 1 --2- 172.21.15.104:0/1207544960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff714102000 0x7ff714105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.988+0000 7ff71a6a5700 1 -- 172.21.15.104:0/1207544960 >> 172.21.15.104:0/1207544960 conn(0x7ff7140773e0 msgr2=0x7ff7140777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.988+0000 7ff71a6a5700 1 -- 172.21.15.104:0/1207544960 shutdown_connections 2024-06-09T22:20:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.988+0000 7ff71a6a5700 1 -- 172.21.15.104:0/1207544960 wait complete. 2024-06-09T22:20:27.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.988+0000 7ff71a6a5700 1 Processor -- start 2024-06-09T22:20:27.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff71a6a5700 1 -- start start 2024-06-09T22:20:27.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff71a6a5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff714101650 0x7ff714199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:27.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff71a6a5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff714102000 0x7ff71419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:27.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff71a6a5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7141066a0 0x7ff7141a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:27.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff71a6a5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff7141116b0 con 0x7ff714102000 2024-06-09T22:20:27.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff71a6a5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff714111530 con 0x7ff714101650 2024-06-09T22:20:27.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff71a6a5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff714111830 con 0x7ff7141066a0 2024-06-09T22:20:27.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff7137fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff714102000 0x7ff71419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:27.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff7137fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff714102000 0x7ff71419a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46336/0 (socket says 172.21.15.104:46336) 2024-06-09T22:20:27.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff7137fe700 1 -- 172.21.15.104:0/2264688729 learned_addr learned my addr 172.21.15.104:0/2264688729 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:27.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff713fff700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff714101650 0x7ff714199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:27.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff718c42700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7141066a0 0x7ff7141a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:27.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff7137fe700 1 -- 172.21.15.104:0/2264688729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7141066a0 msgr2=0x7ff7141a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:27.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff7137fe700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7141066a0 0x7ff7141a4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:27.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff7137fe700 1 -- 172.21.15.104:0/2264688729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff714101650 msgr2=0x7ff714199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:27.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.989+0000 7ff7137fe700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff714101650 0x7ff714199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:27.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.990+0000 7ff7137fe700 1 -- 172.21.15.104:0/2264688729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7141a50f0 con 0x7ff714102000 2024-06-09T22:20:27.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.990+0000 7ff7137fe700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff714102000 0x7ff71419a070 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff70800e970 tx=0x7ff70800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:27.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.990+0000 7ff7117fa700 1 -- 172.21.15.104:0/2264688729 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff708004a30 con 0x7ff714102000 2024-06-09T22:20:27.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.990+0000 7ff71a6a5700 1 -- 172.21.15.104:0/2264688729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7141a5320 con 0x7ff714102000 2024-06-09T22:20:27.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.990+0000 7ff7117fa700 1 -- 172.21.15.104:0/2264688729 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff70800ccc0 con 0x7ff714102000 2024-06-09T22:20:27.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.990+0000 7ff7117fa700 1 -- 172.21.15.104:0/2264688729 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff708005650 con 0x7ff714102000 2024-06-09T22:20:27.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.990+0000 7ff71a6a5700 1 -- 172.21.15.104:0/2264688729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7141a58b0 con 0x7ff714102000 2024-06-09T22:20:27.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.991+0000 7ff71a6a5700 1 -- 172.21.15.104:0/2264688729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff714069180 con 0x7ff714102000 2024-06-09T22:20:27.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.992+0000 7ff7117fa700 1 -- 172.21.15.104:0/2264688729 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff708004bd0 con 0x7ff714102000 2024-06-09T22:20:27.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.993+0000 7ff7117fa700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff6fc0a2c20 0x7ff6fc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:27.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.993+0000 7ff713fff700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff6fc0a2c20 0x7ff6fc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:27.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.993+0000 7ff7117fa700 1 -- 172.21.15.104:0/2264688729 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7ff7080cf5c0 con 0x7ff714102000 2024-06-09T22:20:27.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.993+0000 7ff713fff700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff6fc0a2c20 0x7ff6fc0a50e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff70400c010 tx=0x7ff704019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:27.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:27.997+0000 7ff7117fa700 1 -- 172.21.15.104:0/2264688729 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff70809c190 con 0x7ff714102000 2024-06-09T22:20:28.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.234+0000 7ff71a6a5700 1 -- 172.21.15.104:0/2264688729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff71402de60 con 0x7ff714102000 2024-06-09T22:20:28.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.235+0000 7ff7117fa700 1 -- 172.21.15.104:0/2264688729 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7ff70809ba90 con 0x7ff714102000 2024-06-09T22:20:28.234 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:20:28.234 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:17:53.402012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:20:28.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.237+0000 7ff71a6a5700 1 -- 172.21.15.104:0/2264688729 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff6fc0a2c20 msgr2=0x7ff6fc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:28.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.237+0000 7ff71a6a5700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff6fc0a2c20 0x7ff6fc0a50e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff70400c010 tx=0x7ff704019040 comp rx=0 tx=0).stop 2024-06-09T22:20:28.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.237+0000 7ff71a6a5700 1 -- 172.21.15.104:0/2264688729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff714102000 msgr2=0x7ff71419a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:28.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.237+0000 7ff71a6a5700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff714102000 0x7ff71419a070 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff70800e970 tx=0x7ff70800ec80 comp rx=0 tx=0).stop 2024-06-09T22:20:28.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.237+0000 7ff71a6a5700 1 -- 172.21.15.104:0/2264688729 shutdown_connections 2024-06-09T22:20:28.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.237+0000 7ff71a6a5700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7ff6fc0a2c20 0x7ff6fc0a50e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:28.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.237+0000 7ff71a6a5700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7141066a0 0x7ff7141a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:28.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.237+0000 7ff71a6a5700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff714101650 0x7ff714199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:28.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.237+0000 7ff71a6a5700 1 --2- 172.21.15.104:0/2264688729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff714102000 0x7ff71419a070 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:28.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.237+0000 7ff71a6a5700 1 -- 172.21.15.104:0/2264688729 >> 172.21.15.104:0/2264688729 conn(0x7ff7140773e0 msgr2=0x7ff714105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:28.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.237+0000 7ff71a6a5700 1 -- 172.21.15.104:0/2264688729 shutdown_connections 2024-06-09T22:20:28.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:28.237+0000 7ff71a6a5700 1 -- 172.21.15.104:0/2264688729 wait complete. 2024-06-09T22:20:28.237 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 28 2024-06-09T22:20:28.505 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:28 smithi104 ceph-mon[32389]: pgmap v420: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:28.505 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:28 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2264688729' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:20:28.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:28 smithi153 ceph-mon[36599]: pgmap v420: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:28.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:28 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2264688729' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:20:28.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:28 smithi194 ceph-mon[36856]: pgmap v420: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:28.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:28 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2264688729' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:20:28.909 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:20:29.146 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:30.654 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:30 smithi104 ceph-mon[32389]: pgmap v421: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:30.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:30 smithi153 ceph-mon[36599]: pgmap v421: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:30.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:30 smithi194 ceph-mon[36856]: pgmap v421: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:30.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.908+0000 7fe6c0942700 1 -- 172.21.15.104:0/1538616649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6bc1027a0 msgr2=0x7fe6bc106020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:30.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.908+0000 7fe6c0942700 1 --2- 172.21.15.104:0/1538616649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6bc1027a0 0x7fe6bc106020 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00099c0 tx=0x7fe6b0009cd0 comp rx=0 tx=0).stop 2024-06-09T22:20:30.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.908+0000 7fe6c0942700 1 -- 172.21.15.104:0/1538616649 shutdown_connections 2024-06-09T22:20:30.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.908+0000 7fe6c0942700 1 --2- 172.21.15.104:0/1538616649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6bc106750 0x7fe6bc10e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:30.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.908+0000 7fe6c0942700 1 --2- 172.21.15.104:0/1538616649 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6bc101df0 0x7fe6bc1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:30.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.908+0000 7fe6c0942700 1 --2- 172.21.15.104:0/1538616649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6bc1027a0 0x7fe6bc106020 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:30.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.908+0000 7fe6c0942700 1 -- 172.21.15.104:0/1538616649 >> 172.21.15.104:0/1538616649 conn(0x7fe6bc077a00 msgr2=0x7fe6bc0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:30.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.908+0000 7fe6c0942700 1 -- 172.21.15.104:0/1538616649 shutdown_connections 2024-06-09T22:20:30.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.908+0000 7fe6c0942700 1 -- 172.21.15.104:0/1538616649 wait complete. 2024-06-09T22:20:30.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.909+0000 7fe6c0942700 1 Processor -- start 2024-06-09T22:20:30.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.909+0000 7fe6c0942700 1 -- start start 2024-06-09T22:20:30.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6c0942700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6bc101df0 0x7fe6bc19e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:30.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6c0942700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6bc1027a0 0x7fe6bc19ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:30.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6c0942700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6bc106750 0x7fe6bc198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:30.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6c0942700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc111660 con 0x7fe6bc106750 2024-06-09T22:20:30.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6c0942700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc1114e0 con 0x7fe6bc1027a0 2024-06-09T22:20:30.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6c0942700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc1117e0 con 0x7fe6bc101df0 2024-06-09T22:20:30.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6bad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6bc106750 0x7fe6bc198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:30.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6bad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6bc106750 0x7fe6bc198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46350/0 (socket says 172.21.15.104:46350) 2024-06-09T22:20:30.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6bad9d700 1 -- 172.21.15.104:0/1005467997 learned_addr learned my addr 172.21.15.104:0/1005467997 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:30.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6ba59c700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6bc101df0 0x7fe6bc19e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:30.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6b9d9b700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6bc1027a0 0x7fe6bc19ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:30.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6bad9d700 1 -- 172.21.15.104:0/1005467997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6bc101df0 msgr2=0x7fe6bc19e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:30.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6bad9d700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6bc101df0 0x7fe6bc19e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:30.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6bad9d700 1 -- 172.21.15.104:0/1005467997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6bc1027a0 msgr2=0x7fe6bc19ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:30.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6bad9d700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6bc1027a0 0x7fe6bc19ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:30.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6bad9d700 1 -- 172.21.15.104:0/1005467997 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6bc1990d0 con 0x7fe6bc106750 2024-06-09T22:20:30.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6ba59c700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6bc101df0 0x7fe6bc19e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:20:30.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6b9d9b700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6bc1027a0 0x7fe6bc19ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:30.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.910+0000 7fe6bad9d700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6bc106750 0x7fe6bc198970 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400ef30 tx=0x7fe6b400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:30.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.911+0000 7fe6ab7fe700 1 -- 172.21.15.104:0/1005467997 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40092e0 con 0x7fe6bc106750 2024-06-09T22:20:30.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.911+0000 7fe6c0942700 1 -- 172.21.15.104:0/1005467997 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6bc199360 con 0x7fe6bc106750 2024-06-09T22:20:30.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.911+0000 7fe6c0942700 1 -- 172.21.15.104:0/1005467997 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6bc0fcda0 con 0x7fe6bc106750 2024-06-09T22:20:30.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.911+0000 7fe6ab7fe700 1 -- 172.21.15.104:0/1005467997 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b4004bb0 con 0x7fe6bc106750 2024-06-09T22:20:30.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.911+0000 7fe6ab7fe700 1 -- 172.21.15.104:0/1005467997 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4004280 con 0x7fe6bc106750 2024-06-09T22:20:30.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.912+0000 7fe6c0942700 1 -- 172.21.15.104:0/1005467997 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe69c005330 con 0x7fe6bc106750 2024-06-09T22:20:30.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.913+0000 7fe6ab7fe700 1 -- 172.21.15.104:0/1005467997 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fe6b401b030 con 0x7fe6bc106750 2024-06-09T22:20:30.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.913+0000 7fe6ab7fe700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe6a40a2ca0 0x7fe6a40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:30.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.914+0000 7fe6ba59c700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe6a40a2ca0 0x7fe6a40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:30.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.914+0000 7fe6ab7fe700 1 -- 172.21.15.104:0/1005467997 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fe6b40cedb0 con 0x7fe6bc106750 2024-06-09T22:20:30.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.914+0000 7fe6ba59c700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe6a40a2ca0 0x7fe6a40a5160 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc199ab0 tx=0x7fe6ac012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:30.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:30.918+0000 7fe6ab7fe700 1 -- 172.21.15.104:0/1005467997 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe6b4022020 con 0x7fe6bc106750 2024-06-09T22:20:31.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.169+0000 7fe6c0942700 1 -- 172.21.15.104:0/1005467997 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe69c005620 con 0x7fe6bc106750 2024-06-09T22:20:31.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.170+0000 7fe6ab7fe700 1 -- 172.21.15.104:0/1005467997 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7fe6b4010040 con 0x7fe6bc106750 2024-06-09T22:20:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:20:31.169 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:17:53.402012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:20:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.172+0000 7fe6c0942700 1 -- 172.21.15.104:0/1005467997 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe6a40a2ca0 msgr2=0x7fe6a40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.172+0000 7fe6c0942700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe6a40a2ca0 0x7fe6a40a5160 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc199ab0 tx=0x7fe6ac012040 comp rx=0 tx=0).stop 2024-06-09T22:20:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.172+0000 7fe6c0942700 1 -- 172.21.15.104:0/1005467997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6bc106750 msgr2=0x7fe6bc198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.172+0000 7fe6c0942700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6bc106750 0x7fe6bc198970 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400ef30 tx=0x7fe6b400c3a0 comp rx=0 tx=0).stop 2024-06-09T22:20:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.172+0000 7fe6c0942700 1 -- 172.21.15.104:0/1005467997 shutdown_connections 2024-06-09T22:20:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.172+0000 7fe6c0942700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe6a40a2ca0 0x7fe6a40a5160 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.172+0000 7fe6c0942700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6bc106750 0x7fe6bc198970 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.172+0000 7fe6c0942700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6bc101df0 0x7fe6bc19e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:31.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.172+0000 7fe6c0942700 1 --2- 172.21.15.104:0/1005467997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6bc1027a0 0x7fe6bc19ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:31.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.172+0000 7fe6c0942700 1 -- 172.21.15.104:0/1005467997 >> 172.21.15.104:0/1005467997 conn(0x7fe6bc077a00 msgr2=0x7fe6bc104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:31.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.172+0000 7fe6c0942700 1 -- 172.21.15.104:0/1005467997 shutdown_connections 2024-06-09T22:20:31.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:31.172+0000 7fe6c0942700 1 -- 172.21.15.104:0/1005467997 wait complete. 2024-06-09T22:20:31.172 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 28 2024-06-09T22:20:31.497 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:20:31.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:31 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1005467997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:20:31.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.498 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:31 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:20:31.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:31 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1005467997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:20:31.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:31 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:20:31.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:31 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1005467997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:20:31.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:31 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:31.837 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:20:32.086 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:32.459 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:32 smithi104 ceph-mon[32389]: pgmap v422: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:32.459 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:20:32.459 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:20:32.459 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:32 smithi104 ceph-mon[32389]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:32.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:32 smithi153 ceph-mon[36599]: pgmap v422: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:32.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:20:32.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:20:32.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:32 smithi153 ceph-mon[36599]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:32.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:32 smithi194 ceph-mon[36856]: pgmap v422: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:32.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:20:32.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:20:32.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:32 smithi194 ceph-mon[36856]: from='mgr.14152 172.21.15.104:0/2129580690' entity='mgr.x' 2024-06-09T22:20:33.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.775+0000 7fb4fa00e700 1 -- 172.21.15.104:0/2915759855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f4101d30 msgr2=0x7fb4f410e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:33.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.775+0000 7fb4fa00e700 1 --2- 172.21.15.104:0/2915759855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f4101d30 0x7fb4f410e1d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00b0a0 tx=0x7fb4ec00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:20:33.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.775+0000 7fb4fa00e700 1 -- 172.21.15.104:0/2915759855 shutdown_connections 2024-06-09T22:20:33.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.775+0000 7fb4fa00e700 1 --2- 172.21.15.104:0/2915759855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f4101d30 0x7fb4f410e1d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:33.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.775+0000 7fb4fa00e700 1 --2- 172.21.15.104:0/2915759855 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb4f4107420 0x7fb4f4107800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:33.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.775+0000 7fb4fa00e700 1 --2- 172.21.15.104:0/2915759855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f4101370 0x7fb4f41017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:33.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.775+0000 7fb4fa00e700 1 -- 172.21.15.104:0/2915759855 >> 172.21.15.104:0/2915759855 conn(0x7fb4f4078340 msgr2=0x7fb4f4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:33.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.775+0000 7fb4fa00e700 1 -- 172.21.15.104:0/2915759855 shutdown_connections 2024-06-09T22:20:33.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.775+0000 7fb4fa00e700 1 -- 172.21.15.104:0/2915759855 wait complete. 2024-06-09T22:20:33.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.776+0000 7fb4fa00e700 1 Processor -- start 2024-06-09T22:20:33.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.776+0000 7fb4fa00e700 1 -- start start 2024-06-09T22:20:33.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.776+0000 7fb4fa00e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f4101370 0x7fb4f4197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:33.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4fa00e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb4f4101d30 0x7fb4f4197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:33.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4fa00e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f4107420 0x7fb4f41982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:33.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4fa00e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f4111700 con 0x7fb4f4107420 2024-06-09T22:20:33.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4fa00e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f4111580 con 0x7fb4f4101370 2024-06-09T22:20:33.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4fa00e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f4111880 con 0x7fb4f4101d30 2024-06-09T22:20:33.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f4107420 0x7fb4f41982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:33.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f4107420 0x7fb4f41982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46360/0 (socket says 172.21.15.104:46360) 2024-06-09T22:20:33.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f3fff700 1 -- 172.21.15.104:0/744185831 learned_addr learned my addr 172.21.15.104:0/744185831 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:33.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f2ffd700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb4f4101d30 0x7fb4f4197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:33.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f37fe700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f4101370 0x7fb4f4197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:33.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f3fff700 1 -- 172.21.15.104:0/744185831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb4f4101d30 msgr2=0x7fb4f4197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:33.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f3fff700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb4f4101d30 0x7fb4f4197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:33.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f3fff700 1 -- 172.21.15.104:0/744185831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f4101370 msgr2=0x7fb4f4197870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:33.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f3fff700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f4101370 0x7fb4f4197870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:33.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f3fff700 1 -- 172.21.15.104:0/744185831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f41a70b0 con 0x7fb4f4107420 2024-06-09T22:20:33.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f3fff700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f4107420 0x7fb4f41982f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00d010 tx=0x7fb4ec004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:33.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f0ff9700 1 -- 172.21.15.104:0/744185831 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec021070 con 0x7fb4f4107420 2024-06-09T22:20:33.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4fa00e700 1 -- 172.21.15.104:0/744185831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f41a7340 con 0x7fb4f4107420 2024-06-09T22:20:33.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f0ff9700 1 -- 172.21.15.104:0/744185831 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4ec007c00 con 0x7fb4f4107420 2024-06-09T22:20:33.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4f0ff9700 1 -- 172.21.15.104:0/744185831 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec01d450 con 0x7fb4f4107420 2024-06-09T22:20:33.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.777+0000 7fb4fa00e700 1 -- 172.21.15.104:0/744185831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f41a78d0 con 0x7fb4f4107420 2024-06-09T22:20:33.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.778+0000 7fb4fa00e700 1 -- 172.21.15.104:0/744185831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f4109770 con 0x7fb4f4107420 2024-06-09T22:20:33.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.779+0000 7fb4f0ff9700 1 -- 172.21.15.104:0/744185831 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb4ec004000 con 0x7fb4f4107420 2024-06-09T22:20:33.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.780+0000 7fb4f0ff9700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb4dc0a2c20 0x7fb4dc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:33.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.780+0000 7fb4f37fe700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb4dc0a2c20 0x7fb4dc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:33.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.781+0000 7fb4f0ff9700 1 -- 172.21.15.104:0/744185831 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fb4ec00f020 con 0x7fb4f4107420 2024-06-09T22:20:33.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.781+0000 7fb4f37fe700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb4dc0a2c20 0x7fb4dc0a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4006fd0 tx=0x7fb4e4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:33.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.784+0000 7fb4f0ff9700 1 -- 172.21.15.104:0/744185831 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb4ec02a020 con 0x7fb4f4107420 2024-06-09T22:20:33.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.983+0000 7fb4fa00e700 1 -- 172.21.15.104:0/744185831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4f406b180 con 0x7fb4f4107420 2024-06-09T22:20:33.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.985+0000 7fb4f0ff9700 1 -- 172.21.15.104:0/744185831 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+24762 (secure 0 0 0) 0x7fb4ec007db0 con 0x7fb4f4107420 2024-06-09T22:20:33.984 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:20:33.984 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":99,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:18:26.525714+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:20:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.987+0000 7fb4fa00e700 1 -- 172.21.15.104:0/744185831 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb4dc0a2c20 msgr2=0x7fb4dc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.987+0000 7fb4fa00e700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb4dc0a2c20 0x7fb4dc0a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4006fd0 tx=0x7fb4e4008040 comp rx=0 tx=0).stop 2024-06-09T22:20:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.987+0000 7fb4fa00e700 1 -- 172.21.15.104:0/744185831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f4107420 msgr2=0x7fb4f41982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.987+0000 7fb4fa00e700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f4107420 0x7fb4f41982f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00d010 tx=0x7fb4ec004d50 comp rx=0 tx=0).stop 2024-06-09T22:20:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.987+0000 7fb4fa00e700 1 -- 172.21.15.104:0/744185831 shutdown_connections 2024-06-09T22:20:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.987+0000 7fb4fa00e700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb4dc0a2c20 0x7fb4dc0a50e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.987+0000 7fb4fa00e700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4f4107420 0x7fb4f41982f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.987+0000 7fb4fa00e700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f4101370 0x7fb4f4197870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.987+0000 7fb4fa00e700 1 --2- 172.21.15.104:0/744185831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb4f4101d30 0x7fb4f4197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:33.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.987+0000 7fb4fa00e700 1 -- 172.21.15.104:0/744185831 >> 172.21.15.104:0/744185831 conn(0x7fb4f4078340 msgr2=0x7fb4f4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:33.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.987+0000 7fb4fa00e700 1 -- 172.21.15.104:0/744185831 shutdown_connections 2024-06-09T22:20:33.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:33.987+0000 7fb4fa00e700 1 -- 172.21.15.104:0/744185831 wait complete. 2024-06-09T22:20:34.618 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:34 smithi104 ceph-mon[32389]: pgmap v423: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:34.619 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:34 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/744185831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:20:34.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-06-09T22:16:45.150445+0000', 'modified': '2024-06-09T22:17:53.402012+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24460, 'mds_1': 24448, 'mds_2': 24442, 'mds_3': 24419, 'mds_4': 24403}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14571': {'gid': 14571, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6839/647544853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6838', 'nonce': 647544853}, {'type': 'v1', 'addr': '172.21.15.104:6839', 'nonce': 647544853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14583': {'gid': 14583, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6841/3830332242', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6840', 'nonce': 3830332242}, {'type': 'v1', 'addr': '172.21.15.104:6841', 'nonce': 3830332242}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24362': {'gid': 24362, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6833/3968887191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6832', 'nonce': 3968887191}, {'type': 'v1', 'addr': '172.21.15.153:6833', 'nonce': 3968887191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24383': {'gid': 24383, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6835/3075465176', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 3075465176}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 3075465176}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24398': {'gid': 24398, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6837/3294454895', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 3294454895}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 3294454895}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24403': {'gid': 24403, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.194:6833/3284873174', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6832', 'nonce': 3284873174}, {'type': 'v1', 'addr': '172.21.15.194:6833', 'nonce': 3284873174}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24419': {'gid': 24419, 'name': 'b', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.153:6839/779682823', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 779682823}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 779682823}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 26, 'addr': '172.21.15.194:6835/2995819396', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6834', 'nonce': 2995819396}, {'type': 'v1', 'addr': '172.21.15.194:6835', 'nonce': 2995819396}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24448': {'gid': 24448, 'name': 'c', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 25, 'addr': '172.21.15.194:6837/464494707', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6836', 'nonce': 464494707}, {'type': 'v1', 'addr': '172.21.15.194:6837', 'nonce': 464494707}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.194:6839/2028005544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6838', 'nonce': 2028005544}, {'type': 'v1', 'addr': '172.21.15.194:6839', 'nonce': 2028005544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24460, 'qdb_cluster': [24403, 24419, 24442, 24448, 24460]} 2024-06-09T22:20:34.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-06-09T22:20:34.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.l status 2024-06-09T22:20:34.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:34 smithi153 ceph-mon[36599]: pgmap v423: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:34.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:34 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/744185831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:20:34.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:34 smithi194 ceph-mon[36856]: pgmap v423: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:34.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:34 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/744185831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:20:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:36.637 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:36 smithi104 ceph-mon[32389]: pgmap v424: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:36.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.638+0000 7f1cbe366700 1 -- 172.21.15.104:0/3769964308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1cb8103150 msgr2=0x7f1cb8107140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:36.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.638+0000 7f1cbe366700 1 --2- 172.21.15.104:0/3769964308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1cb8103150 0x7f1cb8107140 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0099c0 tx=0x7f1cac009cd0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.638+0000 7f1cbe366700 1 -- 172.21.15.104:0/3769964308 shutdown_connections 2024-06-09T22:20:36.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.638+0000 7f1cbe366700 1 --2- 172.21.15.104:0/3769964308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cb8107870 0x7f1cb810f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.638+0000 7f1cbe366700 1 --2- 172.21.15.104:0/3769964308 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cb81027a0 0x7f1cb8102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.638+0000 7f1cbe366700 1 --2- 172.21.15.104:0/3769964308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1cb8103150 0x7f1cb8107140 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.638+0000 7f1cbe366700 1 -- 172.21.15.104:0/3769964308 >> 172.21.15.104:0/3769964308 conn(0x7f1cb8079520 msgr2=0x7f1cb8079930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:36.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.638+0000 7f1cbe366700 1 -- 172.21.15.104:0/3769964308 shutdown_connections 2024-06-09T22:20:36.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.638+0000 7f1cbe366700 1 -- 172.21.15.104:0/3769964308 wait complete. 2024-06-09T22:20:36.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.639+0000 7f1cbe366700 1 Processor -- start 2024-06-09T22:20:36.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.639+0000 7f1cbe366700 1 -- start start 2024-06-09T22:20:36.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.639+0000 7f1cbe366700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1cb81027a0 0x7f1cb8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.639+0000 7f1cbe366700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cb8103150 0x7f1cb8074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.639+0000 7f1cbe366700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cb8107870 0x7f1cb8075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.639+0000 7f1cbe366700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb8112780 con 0x7f1cb81027a0 2024-06-09T22:20:36.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.639+0000 7f1cbe366700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb8112600 con 0x7f1cb8107870 2024-06-09T22:20:36.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.639+0000 7f1cbe366700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb8112900 con 0x7f1cb8103150 2024-06-09T22:20:36.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1cb81027a0 0x7f1cb8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1cb81027a0 0x7f1cb8074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53108/0 (socket says 172.21.15.104:53108) 2024-06-09T22:20:36.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb7fff700 1 -- 172.21.15.104:0/640931398 learned_addr learned my addr 172.21.15.104:0/640931398 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:36.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cb8103150 0x7f1cb8074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb7fff700 1 -- 172.21.15.104:0/640931398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cb8103150 msgr2=0x7f1cb8074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:36.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb7fff700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cb8103150 0x7f1cb8074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cbc903700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cb8107870 0x7f1cb8075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb7fff700 1 -- 172.21.15.104:0/640931398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cb8107870 msgr2=0x7f1cb8075210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:36.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb7fff700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cb8107870 0x7f1cb8075210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb7fff700 1 -- 172.21.15.104:0/640931398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb81a7160 con 0x7f1cb81027a0 2024-06-09T22:20:36.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cb8103150 0x7f1cb8074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:36.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb7fff700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1cb81027a0 0x7f1cb8074790 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800c920 tx=0x7f1ca800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:36.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb57fa700 1 -- 172.21.15.104:0/640931398 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8007b00 con 0x7f1cb81027a0 2024-06-09T22:20:36.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cbe366700 1 -- 172.21.15.104:0/640931398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb81a7450 con 0x7f1cb81027a0 2024-06-09T22:20:36.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb57fa700 1 -- 172.21.15.104:0/640931398 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ca8004d10 con 0x7f1cb81027a0 2024-06-09T22:20:36.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cb57fa700 1 -- 172.21.15.104:0/640931398 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca80056e0 con 0x7f1cb81027a0 2024-06-09T22:20:36.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.640+0000 7f1cbe366700 1 -- 172.21.15.104:0/640931398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb81a79b0 con 0x7f1cb81027a0 2024-06-09T22:20:36.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.643+0000 7f1cb57fa700 1 -- 172.21.15.104:0/640931398 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1ca8007ca0 con 0x7f1cb81027a0 2024-06-09T22:20:36.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.643+0000 7f1cb57fa700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1ca00a2c20 0x7f1ca00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.643+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1ca00a2c20 0x7f1ca00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.644+0000 7f1cb57fa700 1 -- 172.21.15.104:0/640931398 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f1ca80cf040 con 0x7f1cb81027a0 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.644+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1ca00a2c20 0x7f1ca00a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8076220 tx=0x7f1cac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.644+0000 7f1cc8be9b80 1 Processor -- start 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.644+0000 7f1cc8be9b80 1 -- start start 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.644+0000 7f1cc8be9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db8375f0 0x5649db8379d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.644+0000 7f1cc8be9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db837f10 0x5649db831990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.644+0000 7f1cc8be9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db831ed0 0x5649db8322f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.644+0000 7f1cc8be9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5649db7837c0 con 0x5649db837f10 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.644+0000 7f1cc8be9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5649db72b6d0 con 0x5649db8375f0 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.644+0000 7f1cc8be9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5649db77f780 con 0x5649db831ed0 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.644+0000 7f1cb77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db837f10 0x5649db831990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1cb77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db837f10 0x5649db831990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53124/0 (socket says 172.21.15.104:53124) 2024-06-09T22:20:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1cb77fe700 1 -- 172.21.15.104:0/1751429418 learned_addr learned my addr 172.21.15.104:0/1751429418 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1cbc903700 1 --2- 172.21.15.104:0/1751429418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db831ed0 0x5649db8322f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1cb7fff700 1 --2- 172.21.15.104:0/1751429418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db8375f0 0x5649db8379d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1cb77fe700 1 -- 172.21.15.104:0/1751429418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db831ed0 msgr2=0x5649db8322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/1751429418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db831ed0 0x5649db8322f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1cb77fe700 1 -- 172.21.15.104:0/1751429418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db8375f0 msgr2=0x5649db8379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/1751429418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db8375f0 0x5649db8379d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1cb77fe700 1 -- 172.21.15.104:0/1751429418 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649db721f00 con 0x5649db837f10 2024-06-09T22:20:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/1751429418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db837f10 0x5649db831990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1cac009750 tx=0x7f1cac01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb350a60c1d0366 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1c9e7fc700 1 -- 172.21.15.104:0/1751429418 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac003960 con 0x5649db837f10 2024-06-09T22:20:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1c9e7fc700 1 -- 172.21.15.104:0/1751429418 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cac005070 con 0x5649db837f10 2024-06-09T22:20:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.645+0000 7f1c9e7fc700 1 -- 172.21.15.104:0/1751429418 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac00f760 con 0x5649db837f10 2024-06-09T22:20:36.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.646+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/1751429418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db837f10 msgr2=0x5649db831990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:36.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.646+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/1751429418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db837f10 0x5649db831990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1cac009750 tx=0x7f1cac01f000 comp rx=0 tx=0).stop 2024-06-09T22:20:36.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.646+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/1751429418 shutdown_connections 2024-06-09T22:20:36.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.646+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/1751429418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db831ed0 0x5649db8322f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.646+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/1751429418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db8375f0 0x5649db8379d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.646+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/1751429418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db837f10 0x5649db831990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.646+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/1751429418 >> 172.21.15.104:0/1751429418 conn(0x5649db75cc70 msgr2=0x5649db82c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:36.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.646+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/1751429418 shutdown_connections 2024-06-09T22:20:36.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.646+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/1751429418 wait complete. 2024-06-09T22:20:36.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.646+0000 7f1cc8be9b80 1 -- start start 2024-06-09T22:20:36.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cc8be9b80 1 Processor -- start 2024-06-09T22:20:36.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cc8be9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db831ed0 0x5649db83dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cc8be9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 0x5649db83afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cc8be9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db837f10 0x5649db83b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cc8be9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5649db83e920 con 0x5649db8375f0 2024-06-09T22:20:36.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cc8be9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5649db723ba0 con 0x5649db831ed0 2024-06-09T22:20:36.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cc8be9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5649db83eaa0 con 0x5649db837f10 2024-06-09T22:20:36.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cc8be9b80 10 client.0 _ll_register_callbacks cb 0x5649db71d8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:36.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cb77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 0x5649db83afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cb77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 0x5649db83afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53126/0 (socket says 172.21.15.104:53126) 2024-06-09T22:20:36.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cb77fe700 1 -- 172.21.15.104:0/2959427854 learned_addr learned my addr 172.21.15.104:0/2959427854 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cb7fff700 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db831ed0 0x5649db83dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cbc903700 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db837f10 0x5649db83b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cb77fe700 1 -- 172.21.15.104:0/2959427854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db837f10 msgr2=0x5649db83b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db837f10 0x5649db83b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cb77fe700 1 -- 172.21.15.104:0/2959427854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db831ed0 msgr2=0x5649db83dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db831ed0 0x5649db83dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cb77fe700 1 -- 172.21.15.104:0/2959427854 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649db6d2880 con 0x5649db8375f0 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cb7fff700 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db831ed0 0x5649db83dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 0x5649db83afe0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1cac009d10 tx=0x7f1cac003d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993c6b28364f1bef server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.647+0000 7f1c977fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1c977fe700 1 -- 172.21.15.104:0/2959427854 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac034070 con 0x5649db8375f0 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1c977fe700 1 -- 172.21.15.104:0/2959427854 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f1cac00f670 con 0x5649db8375f0 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2959427854 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cac005070 con 0x5649db8375f0 2024-06-09T22:20:36.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1c977fe700 1 -- 172.21.15.104:0/2959427854 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1cac00f870 con 0x5649db8375f0 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1cc8be9b80 10 client.15180 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2959427854 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5649db849280 con 0x5649db8375f0 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1c977fe700 1 -- 172.21.15.104:0/2959427854 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cac005070 con 0x5649db8375f0 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1c977fe700 1 -- 172.21.15.104:0/2959427854 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1cac030960 con 0x5649db8375f0 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1cc8be9b80 10 client.15180 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1cc8be9b80 10 client.15180 resolve_mds: resolved name 'l' to daemon mds.l 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1cc8be9b80 20 client.15180 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5649db84a5d0 0x5649db84c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1cc8be9b80 4 client.15180 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.648+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2959427854 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649db790cc0 con 0x5649db84a5d0 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.649+0000 7f1cb7fff700 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5649db84a5d0 0x5649db84c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.649+0000 7f1c977fe700 10 client.15180 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:20:36.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.649+0000 7f1cb7fff700 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5649db84a5d0 0x5649db84c730 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7f8618a2fb60532b server_cookie=64a076dcb336e0b3 in_seq=0 out_seq=0 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1c977fe700 1 -- 172.21.15.104:0/2959427854 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5649db790cc0 con 0x5649db84a5d0 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1c977fe700 10 client.15180 handle_command_reply: tid=0 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1cc8be9b80 1 client.15180 shutdown 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2959427854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 msgr2=0x5649db83afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 0x5649db83afe0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1cac009d10 tx=0x7f1cac003d90 comp rx=0 tx=0).stop 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2959427854 shutdown_connections 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5649db84a5d0 0x5649db84c730 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db837f10 0x5649db83b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db831ed0 0x5649db83dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/2959427854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 0x5649db83afe0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2959427854 >> 172.21.15.104:0/2959427854 conn(0x5649db75cc70 msgr2=0x5649db82c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1c977fe700 0 client.15180 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2959427854 shutdown_connections 2024-06-09T22:20:36.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2959427854 wait complete. 2024-06-09T22:20:36.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.650+0000 7f1cc8be9b80 20 client.15180 trim_cache size 0 max 16384 2024-06-09T22:20:36.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.671+0000 7f1cc8be9b80 1 Processor -- start 2024-06-09T22:20:36.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.671+0000 7f1cc8be9b80 1 -- start start 2024-06-09T22:20:36.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.671+0000 7f1cc8be9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db831ed0 0x5649db831990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.671+0000 7f1cc8be9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 0x5649db83b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.671+0000 7f1cc8be9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db837f10 0x5649db855b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.671+0000 7f1cc8be9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5649db84aa70 con 0x5649db8375f0 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.671+0000 7f1cc8be9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5649db84b140 con 0x5649db837f10 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.671+0000 7f1cc8be9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5649db84b810 con 0x5649db831ed0 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.671+0000 7f1cb77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 0x5649db83b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.671+0000 7f1cb77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 0x5649db83b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53140/0 (socket says 172.21.15.104:53140) 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.671+0000 7f1cb77fe700 1 -- 172.21.15.104:0/4011090979 learned_addr learned my addr 172.21.15.104:0/4011090979 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.672+0000 7f1cbc903700 1 --2- 172.21.15.104:0/4011090979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db837f10 0x5649db855b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.672+0000 7f1cb7fff700 1 --2- 172.21.15.104:0/4011090979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db831ed0 0x5649db831990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.672+0000 7f1cb77fe700 1 -- 172.21.15.104:0/4011090979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db831ed0 msgr2=0x5649db831990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.672+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/4011090979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db831ed0 0x5649db831990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.672+0000 7f1cb77fe700 1 -- 172.21.15.104:0/4011090979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db837f10 msgr2=0x5649db855b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:36.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.672+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/4011090979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db837f10 0x5649db855b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.672+0000 7f1cb77fe700 1 -- 172.21.15.104:0/4011090979 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649db84af60 con 0x5649db8375f0 2024-06-09T22:20:36.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.672+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/4011090979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 0x5649db83b550 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0098e0 tx=0x7f1cac009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f0c9e906eca28c5 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:36.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.672+0000 7f1c9dffb700 1 -- 172.21.15.104:0/4011090979 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac034040 con 0x5649db8375f0 2024-06-09T22:20:36.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.672+0000 7f1c9dffb700 1 -- 172.21.15.104:0/4011090979 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cac003760 con 0x5649db8375f0 2024-06-09T22:20:36.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.672+0000 7f1c9dffb700 1 -- 172.21.15.104:0/4011090979 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac030770 con 0x5649db8375f0 2024-06-09T22:20:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.673+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/4011090979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 msgr2=0x5649db83b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.673+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/4011090979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 0x5649db83b550 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0098e0 tx=0x7f1cac009310 comp rx=0 tx=0).stop 2024-06-09T22:20:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.673+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/4011090979 shutdown_connections 2024-06-09T22:20:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.673+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/4011090979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db837f10 0x5649db855b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.673+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/4011090979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db831ed0 0x5649db831990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.673+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/4011090979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db8375f0 0x5649db83b550 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.673+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/4011090979 >> 172.21.15.104:0/4011090979 conn(0x5649db75cc70 msgr2=0x5649db8398d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.673+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/4011090979 shutdown_connections 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.673+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/4011090979 wait complete. 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.673+0000 7f1cc8be9b80 1 -- start start 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.673+0000 7f1cc8be9b80 1 Processor -- start 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cc8be9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db831ed0 0x5649db856190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cc8be9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db8375f0 0x5649db83a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cc8be9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db837f10 0x5649db83aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cc8be9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5649db84c5b0 con 0x5649db831ed0 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cc8be9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5649db84bee0 con 0x5649db837f10 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cc8be9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5649db84cc80 con 0x5649db8375f0 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cc8be9b80 10 client.0 _ll_register_callbacks cb 0x5649db71d8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db831ed0 0x5649db856190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db831ed0 0x5649db856190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53150/0 (socket says 172.21.15.104:53150) 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cb7fff700 1 -- 172.21.15.104:0/2493942782 learned_addr learned my addr 172.21.15.104:0/2493942782 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db8375f0 0x5649db83a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cbc903700 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db837f10 0x5649db83aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cb7fff700 1 -- 172.21.15.104:0/2493942782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db8375f0 msgr2=0x5649db83a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cb7fff700 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db8375f0 0x5649db83a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cb7fff700 1 -- 172.21.15.104:0/2493942782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db837f10 msgr2=0x5649db83aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cb7fff700 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db837f10 0x5649db83aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cb7fff700 1 -- 172.21.15.104:0/2493942782 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649db83b130 con 0x5649db831ed0 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cb7fff700 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db831ed0 0x5649db856190 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1ca809b4a0 tx=0x7f1ca80cc680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8d59ee4be26ef3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1c977fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1c977fe700 1 -- 172.21.15.104:0/2493942782 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca801d040 con 0x5649db831ed0 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1c977fe700 1 -- 172.21.15.104:0/2493942782 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f1ca80b4290 con 0x5649db831ed0 2024-06-09T22:20:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.674+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2493942782 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cac003760 con 0x5649db831ed0 2024-06-09T22:20:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.675+0000 7f1c977fe700 1 -- 172.21.15.104:0/2493942782 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ca809f8c0 con 0x5649db831ed0 2024-06-09T22:20:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.675+0000 7f1cc8be9b80 10 client.15186 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.675+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2493942782 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5649db8b6b50 con 0x5649db831ed0 2024-06-09T22:20:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.675+0000 7f1c977fe700 1 -- 172.21.15.104:0/2493942782 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cac003760 con 0x5649db831ed0 2024-06-09T22:20:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.675+0000 7f1c977fe700 1 -- 172.21.15.104:0/2493942782 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1ca80a3cf0 con 0x5649db831ed0 2024-06-09T22:20:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.675+0000 7f1cc8be9b80 10 client.15186 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.675+0000 7f1cc8be9b80 10 client.15186 resolve_mds: resolved name 'l' to daemon mds.l 2024-06-09T22:20:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.675+0000 7f1cc8be9b80 20 client.15186 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.675+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5649db8bb570 0x5649db8ba280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.675+0000 7f1cc8be9b80 4 client.15186 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:20:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.675+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2493942782 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "status"}) -- 0x5649db78b240 con 0x5649db8bb570 2024-06-09T22:20:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.675+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5649db8bb570 0x5649db8ba280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1cb77fe700 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5649db8bb570 0x5649db8ba280 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=222f7a3d2d4732ff server_cookie=c15455748a1ea7f9 in_seq=0 out_seq=0 2024-06-09T22:20:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1c977fe700 10 client.15186 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:20:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1c977fe700 1 -- 172.21.15.104:0/2493942782 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5649db78b240 con 0x5649db8bb570 2024-06-09T22:20:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1c977fe700 10 client.15186 handle_command_reply: tid=0 2024-06-09T22:20:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1cc8be9b80 1 client.15186 shutdown 2024-06-09T22:20:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2493942782 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db831ed0 msgr2=0x5649db856190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db831ed0 0x5649db856190 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1ca809b4a0 tx=0x7f1ca80cc680 comp rx=0 tx=0).stop 2024-06-09T22:20:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2493942782 shutdown_connections 2024-06-09T22:20:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649db831ed0 0x5649db856190 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5649db8bb570 0x5649db8ba280 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1c977fe700 0 client.15186 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:20:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649db837f10 0x5649db83aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1cc8be9b80 1 --2- 172.21.15.104:0/2493942782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649db8375f0 0x5649db83a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2493942782 >> 172.21.15.104:0/2493942782 conn(0x5649db75cc70 msgr2=0x5649db838330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2493942782 shutdown_connections 2024-06-09T22:20:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.676+0000 7f1cc8be9b80 1 -- 172.21.15.104:0/2493942782 wait complete. 2024-06-09T22:20:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cc8be9b80 20 client.15186 trim_cache size 0 max 16384 2024-06-09T22:20:36.680 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:20:36.680 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:20:36.681 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 4, 2024-06-09T22:20:36.681 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24403, 2024-06-09T22:20:36.681 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:20:36.681 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:20:36.681 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:20:36.681 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 166.24004938600001, 2024-06-09T22:20:36.681 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 28, 2024-06-09T22:20:36.681 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 99, 2024-06-09T22:20:36.681 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 97, 2024-06-09T22:20:36.681 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 282.77997303000001 2024-06-09T22:20:36.681 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:20:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cbe366700 1 -- 172.21.15.104:0/640931398 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1ca00a2c20 msgr2=0x7f1ca00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cbe366700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1ca00a2c20 0x7f1ca00a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8076220 tx=0x7f1cac005fd0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cbe366700 1 -- 172.21.15.104:0/640931398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1cb81027a0 msgr2=0x7f1cb8074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cbe366700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1cb81027a0 0x7f1cb8074790 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800c920 tx=0x7f1ca800cce0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cbe366700 1 -- 172.21.15.104:0/640931398 shutdown_connections 2024-06-09T22:20:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cbe366700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f1ca00a2c20 0x7f1ca00a50e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cbe366700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1cb81027a0 0x7f1cb8074790 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cbe366700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cb8107870 0x7f1cb8075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cbe366700 1 --2- 172.21.15.104:0/640931398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cb8103150 0x7f1cb8074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cbe366700 1 -- 172.21.15.104:0/640931398 >> 172.21.15.104:0/640931398 conn(0x7f1cb8079520 msgr2=0x7f1cb8100610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cbe366700 1 -- 172.21.15.104:0/640931398 shutdown_connections 2024-06-09T22:20:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:36.677+0000 7f1cbe366700 1 -- 172.21.15.104:0/640931398 wait complete. 2024-06-09T22:20:36.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:36 smithi153 ceph-mon[36599]: pgmap v424: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:36.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:36 smithi194 ceph-mon[36856]: pgmap v424: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:37.496 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-06-09T22:20:37.736 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:38.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:38 smithi153 ceph-mon[36599]: pgmap v425: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:38.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:38 smithi104 ceph-mon[32389]: pgmap v425: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:38.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:38 smithi194 ceph-mon[36856]: pgmap v425: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:39.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.656+0000 7f6a448a5700 1 -- 172.21.15.104:0/252263044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a40101fd0 msgr2=0x7f6a40105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:39.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.656+0000 7f6a448a5700 1 --2- 172.21.15.104:0/252263044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a40101fd0 0x7f6a40105fd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6a300099c0 tx=0x7f6a30009cd0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.656+0000 7f6a448a5700 1 -- 172.21.15.104:0/252263044 shutdown_connections 2024-06-09T22:20:39.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.656+0000 7f6a448a5700 1 --2- 172.21.15.104:0/252263044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a401066b0 0x7f6a4010e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.656+0000 7f6a448a5700 1 --2- 172.21.15.104:0/252263044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a40101620 0x7f6a40101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.656+0000 7f6a448a5700 1 --2- 172.21.15.104:0/252263044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a40101fd0 0x7f6a40105fd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.656+0000 7f6a448a5700 1 -- 172.21.15.104:0/252263044 >> 172.21.15.104:0/252263044 conn(0x7f6a40078360 msgr2=0x7f6a40078770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:39.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.656+0000 7f6a448a5700 1 -- 172.21.15.104:0/252263044 shutdown_connections 2024-06-09T22:20:39.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.656+0000 7f6a448a5700 1 -- 172.21.15.104:0/252263044 wait complete. 2024-06-09T22:20:39.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.656+0000 7f6a448a5700 1 Processor -- start 2024-06-09T22:20:39.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a448a5700 1 -- start start 2024-06-09T22:20:39.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a448a5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a40101620 0x7f6a4019e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a448a5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a40101fd0 0x7f6a4019ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a448a5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a401066b0 0x7f6a401989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a448a5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6a40111670 con 0x7f6a401066b0 2024-06-09T22:20:39.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a448a5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401114f0 con 0x7f6a40101620 2024-06-09T22:20:39.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a448a5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401117f0 con 0x7f6a40101fd0 2024-06-09T22:20:39.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a3ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a401066b0 0x7f6a401989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a3ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a401066b0 0x7f6a401989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53172/0 (socket says 172.21.15.104:53172) 2024-06-09T22:20:39.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a3ed9d700 1 -- 172.21.15.104:0/752226141 learned_addr learned my addr 172.21.15.104:0/752226141 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:39.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a3dd9b700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a40101fd0 0x7f6a4019ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a3e59c700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a40101620 0x7f6a4019e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a3ed9d700 1 -- 172.21.15.104:0/752226141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a40101fd0 msgr2=0x7f6a4019ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:39.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a3ed9d700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a40101fd0 0x7f6a4019ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a3ed9d700 1 -- 172.21.15.104:0/752226141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a40101620 msgr2=0x7f6a4019e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:39.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a3ed9d700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a40101620 0x7f6a4019e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a3ed9d700 1 -- 172.21.15.104:0/752226141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a40199100 con 0x7f6a401066b0 2024-06-09T22:20:39.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.657+0000 7f6a3ed9d700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a401066b0 0x7f6a401989a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6a38007d70 tx=0x7f6a3800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:39.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.658+0000 7f6a2f7fe700 1 -- 172.21.15.104:0/752226141 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a38016020 con 0x7f6a401066b0 2024-06-09T22:20:39.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.658+0000 7f6a448a5700 1 -- 172.21.15.104:0/752226141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a401993f0 con 0x7f6a401066b0 2024-06-09T22:20:39.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.658+0000 7f6a2f7fe700 1 -- 172.21.15.104:0/752226141 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a3800ad40 con 0x7f6a401066b0 2024-06-09T22:20:39.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.658+0000 7f6a2f7fe700 1 -- 172.21.15.104:0/752226141 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a38011600 con 0x7f6a401066b0 2024-06-09T22:20:39.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.658+0000 7f6a448a5700 1 -- 172.21.15.104:0/752226141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a401a56b0 con 0x7f6a401066b0 2024-06-09T22:20:39.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.660+0000 7f6a2f7fe700 1 -- 172.21.15.104:0/752226141 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f6a38019020 con 0x7f6a401066b0 2024-06-09T22:20:39.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.661+0000 7f6a2f7fe700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6a280a2ca0 0x7f6a280a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.661+0000 7f6a3e59c700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6a280a2ca0 0x7f6a280a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.661+0000 7f6a2f7fe700 1 -- 172.21.15.104:0/752226141 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f6a380cf280 con 0x7f6a401066b0 2024-06-09T22:20:39.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.661+0000 7f6a3e59c700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6a280a2ca0 0x7f6a280a5160 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6a34005ea0 tx=0x7f6a34005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:39.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.661+0000 7f6a4f128b80 1 Processor -- start 2024-06-09T22:20:39.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a4f128b80 1 -- start start 2024-06-09T22:20:39.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a4f128b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 0x55b4a0bab360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a4f128b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bab8a0 0x55b4a0baf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a4f128b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a4f128b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b4a0af87c0 con 0x55b4a0baaf80 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a4f128b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b4a0aa06d0 con 0x55b4a0bab8a0 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a4f128b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b4a0af4780 con 0x55b4a0bafbb0 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a3e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 0x55b4a0bab360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a3e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 0x55b4a0bab360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53184/0 (socket says 172.21.15.104:53184) 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a3e59c700 1 -- 172.21.15.104:0/1580465798 learned_addr learned my addr 172.21.15.104:0/1580465798 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a3ed9d700 1 --2- 172.21.15.104:0/1580465798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a3dd9b700 1 --2- 172.21.15.104:0/1580465798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bab8a0 0x55b4a0baf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a3e59c700 1 -- 172.21.15.104:0/1580465798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bafbb0 msgr2=0x55b4a0bb0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a3e59c700 1 --2- 172.21.15.104:0/1580465798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a3e59c700 1 -- 172.21.15.104:0/1580465798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bab8a0 msgr2=0x55b4a0baf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a3e59c700 1 --2- 172.21.15.104:0/1580465798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bab8a0 0x55b4a0baf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.662+0000 7f6a3e59c700 1 -- 172.21.15.104:0/1580465798 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4a0a96f00 con 0x55b4a0baaf80 2024-06-09T22:20:39.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.663+0000 7f6a3e59c700 1 --2- 172.21.15.104:0/1580465798 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 0x55b4a0bab360 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400a520 tx=0x7f6a34011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f79316c9d537272 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:39.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.663+0000 7f6a2cff9700 1 -- 172.21.15.104:0/1580465798 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34021070 con 0x55b4a0baaf80 2024-06-09T22:20:39.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.663+0000 7f6a2cff9700 1 -- 172.21.15.104:0/1580465798 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a34010c20 con 0x55b4a0baaf80 2024-06-09T22:20:39.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.663+0000 7f6a2cff9700 1 -- 172.21.15.104:0/1580465798 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3400f620 con 0x55b4a0baaf80 2024-06-09T22:20:39.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.663+0000 7f6a4f128b80 1 -- 172.21.15.104:0/1580465798 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 msgr2=0x55b4a0bab360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:39.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.663+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/1580465798 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 0x55b4a0bab360 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400a520 tx=0x7f6a34011000 comp rx=0 tx=0).stop 2024-06-09T22:20:39.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.663+0000 7f6a4f128b80 1 -- 172.21.15.104:0/1580465798 shutdown_connections 2024-06-09T22:20:39.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.663+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/1580465798 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 0x55b4a0bab360 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/1580465798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/1580465798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bab8a0 0x55b4a0baf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 -- 172.21.15.104:0/1580465798 >> 172.21.15.104:0/1580465798 conn(0x55b4a0ad1c70 msgr2=0x55b4a0ba1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:39.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 -- 172.21.15.104:0/1580465798 shutdown_connections 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 -- 172.21.15.104:0/1580465798 wait complete. 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 -- start start 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 Processor -- start 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0baaf80 0x55b4a0bb7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bb4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b4a0bb7d70 con 0x55b4a0bab8a0 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b4a0a98ba0 con 0x55b4a0bafbb0 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b4a0bb7ef0 con 0x55b4a0baaf80 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a4f128b80 10 client.0 _ll_register_callbacks cb 0x55b4a0a928d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a3dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bb4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a3dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bb4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53190/0 (socket says 172.21.15.104:53190) 2024-06-09T22:20:39.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a3e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0baaf80 0x55b4a0bb7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a3dd9b700 1 -- 172.21.15.104:0/2013364263 learned_addr learned my addr 172.21.15.104:0/2013364263 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:39.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.664+0000 7f6a3ed9d700 1 --2- 172.21.15.104:0/2013364263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a3dd9b700 1 -- 172.21.15.104:0/2013364263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0baaf80 msgr2=0x55b4a0bb7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:39.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a3dd9b700 1 --2- 172.21.15.104:0/2013364263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0baaf80 0x55b4a0bb7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a3dd9b700 1 -- 172.21.15.104:0/2013364263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bafbb0 msgr2=0x55b4a0bb4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:39.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a3dd9b700 1 --2- 172.21.15.104:0/2013364263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a3dd9b700 1 -- 172.21.15.104:0/2013364263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4a0a47880 con 0x55b4a0bab8a0 2024-06-09T22:20:39.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a3dd9b700 1 --2- 172.21.15.104:0/2013364263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bb4320 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6a300099c0 tx=0x7f6a30005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863d6ba8bbdaad72 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:39.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a25ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:39.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a25ffb700 1 -- 172.21.15.104:0/2013364263 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30028060 con 0x55b4a0bab8a0 2024-06-09T22:20:39.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a25ffb700 1 -- 172.21.15.104:0/2013364263 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f6a300054d0 con 0x55b4a0bab8a0 2024-06-09T22:20:39.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a4f128b80 1 -- 172.21.15.104:0/2013364263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a34010c20 con 0x55b4a0bab8a0 2024-06-09T22:20:39.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a25ffb700 1 -- 172.21.15.104:0/2013364263 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6a3002f050 con 0x55b4a0bab8a0 2024-06-09T22:20:39.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a4f128b80 10 client.15198 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:39.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a4f128b80 1 -- 172.21.15.104:0/2013364263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b4a0bc25d0 con 0x55b4a0bab8a0 2024-06-09T22:20:39.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.665+0000 7f6a25ffb700 1 -- 172.21.15.104:0/2013364263 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a34010c20 con 0x55b4a0bab8a0 2024-06-09T22:20:39.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.666+0000 7f6a25ffb700 1 -- 172.21.15.104:0/2013364263 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6a3000f5f0 con 0x55b4a0bab8a0 2024-06-09T22:20:39.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.666+0000 7f6a4f128b80 10 client.15198 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:39.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.666+0000 7f6a4f128b80 10 client.15198 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-09T22:20:39.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.666+0000 7f6a4f128b80 20 client.15198 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:39.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.666+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/2013364263 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55b4a0bc4bf0 0x55b4a0bc6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.666+0000 7f6a4f128b80 4 client.15198 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:20:39.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.666+0000 7f6a4f128b80 1 -- 172.21.15.104:0/2013364263 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4a0b05cc0 con 0x55b4a0bc4bf0 2024-06-09T22:20:39.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.666+0000 7f6a3e59c700 1 --2- 172.21.15.104:0/2013364263 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55b4a0bc4bf0 0x55b4a0bc6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.666+0000 7f6a25ffb700 10 client.15198 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:20:39.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.666+0000 7f6a3e59c700 1 --2- 172.21.15.104:0/2013364263 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55b4a0bc4bf0 0x55b4a0bc6060 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=524d229f4304fba0 server_cookie=74572f49be6848f9 in_seq=0 out_seq=0 2024-06-09T22:20:39.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.667+0000 7f6a25ffb700 1 -- 172.21.15.104:0/2013364263 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b4a0b05cc0 con 0x55b4a0bc4bf0 2024-06-09T22:20:39.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.667+0000 7f6a25ffb700 10 client.15198 handle_command_reply: tid=0 2024-06-09T22:20:39.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.667+0000 7f6a4f128b80 1 client.15198 shutdown 2024-06-09T22:20:39.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.667+0000 7f6a4f128b80 1 -- 172.21.15.104:0/2013364263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bab8a0 msgr2=0x55b4a0bb4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:39.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.667+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/2013364263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bb4320 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6a300099c0 tx=0x7f6a30005c80 comp rx=0 tx=0).stop 2024-06-09T22:20:39.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.667+0000 7f6a4f128b80 1 -- 172.21.15.104:0/2013364263 shutdown_connections 2024-06-09T22:20:39.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.667+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/2013364263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.667+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/2013364263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0baaf80 0x55b4a0bb7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.667+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/2013364263 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55b4a0bc4bf0 0x55b4a0bc6060 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.667+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/2013364263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bb4320 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.667+0000 7f6a4f128b80 1 -- 172.21.15.104:0/2013364263 >> 172.21.15.104:0/2013364263 conn(0x55b4a0ad1c70 msgr2=0x55b4a0ba15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:39.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.667+0000 7f6a25ffb700 0 client.15198 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:20:39.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.668+0000 7f6a4f128b80 1 -- 172.21.15.104:0/2013364263 shutdown_connections 2024-06-09T22:20:39.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.668+0000 7f6a4f128b80 1 -- 172.21.15.104:0/2013364263 wait complete. 2024-06-09T22:20:39.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.668+0000 7f6a4f128b80 20 client.15198 trim_cache size 0 max 16384 2024-06-09T22:20:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a4f128b80 1 Processor -- start 2024-06-09T22:20:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a4f128b80 1 -- start start 2024-06-09T22:20:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a4f128b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0baaf80 0x55b4a0bb4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a4f128b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bb18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a4f128b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a4f128b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b4a0bc43a0 con 0x55b4a0bafbb0 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a4f128b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b4a0bc4a70 con 0x55b4a0baaf80 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a4f128b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b4a0bc5140 con 0x55b4a0bab8a0 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a3ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a3ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53196/0 (socket says 172.21.15.104:53196) 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a3ed9d700 1 -- 172.21.15.104:0/3398071937 learned_addr learned my addr 172.21.15.104:0/3398071937 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a3dd9b700 1 --2- 172.21.15.104:0/3398071937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bb18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a3ed9d700 1 -- 172.21.15.104:0/3398071937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bab8a0 msgr2=0x55b4a0bb18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a3ed9d700 1 --2- 172.21.15.104:0/3398071937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bb18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a3e59c700 1 --2- 172.21.15.104:0/3398071937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0baaf80 0x55b4a0bb4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a3ed9d700 1 -- 172.21.15.104:0/3398071937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0baaf80 msgr2=0x55b4a0bb4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:39.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a3ed9d700 1 --2- 172.21.15.104:0/3398071937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0baaf80 0x55b4a0bb4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a3ed9d700 1 -- 172.21.15.104:0/3398071937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4a0bb2330 con 0x55b4a0bafbb0 2024-06-09T22:20:39.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.689+0000 7f6a3ed9d700 1 --2- 172.21.15.104:0/3398071937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb1df0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6a380cc9f0 tx=0x7f6a38086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b58b2982b09c26a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:39.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.690+0000 7f6a27fff700 1 -- 172.21.15.104:0/3398071937 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a380b3070 con 0x55b4a0bafbb0 2024-06-09T22:20:39.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.690+0000 7f6a27fff700 1 -- 172.21.15.104:0/3398071937 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a3809b750 con 0x55b4a0bafbb0 2024-06-09T22:20:39.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.690+0000 7f6a27fff700 1 -- 172.21.15.104:0/3398071937 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a380a3f30 con 0x55b4a0bafbb0 2024-06-09T22:20:39.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.690+0000 7f6a4f128b80 1 -- 172.21.15.104:0/3398071937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bafbb0 msgr2=0x55b4a0bb1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:39.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.690+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/3398071937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb1df0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6a380cc9f0 tx=0x7f6a38086040 comp rx=0 tx=0).stop 2024-06-09T22:20:39.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.690+0000 7f6a4f128b80 1 -- 172.21.15.104:0/3398071937 shutdown_connections 2024-06-09T22:20:39.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.690+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/3398071937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bb1df0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.690+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/3398071937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0baaf80 0x55b4a0bb4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.690+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/3398071937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bb18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.690+0000 7f6a4f128b80 1 -- 172.21.15.104:0/3398071937 >> 172.21.15.104:0/3398071937 conn(0x55b4a0ad1c70 msgr2=0x55b4a0bb2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:39.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.690+0000 7f6a4f128b80 1 -- 172.21.15.104:0/3398071937 shutdown_connections 2024-06-09T22:20:39.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.690+0000 7f6a4f128b80 1 -- 172.21.15.104:0/3398071937 wait complete. 2024-06-09T22:20:39.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a4f128b80 1 -- start start 2024-06-09T22:20:39.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a4f128b80 1 Processor -- start 2024-06-09T22:20:39.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a4f128b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 0x55b4a0badb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a4f128b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bcf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a4f128b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bcfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a4f128b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b4a0bc5ee0 con 0x55b4a0baaf80 2024-06-09T22:20:39.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a4f128b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b4a0bc5810 con 0x55b4a0bab8a0 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a4f128b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b4a0bc65b0 con 0x55b4a0bafbb0 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a4f128b80 10 client.0 _ll_register_callbacks cb 0x55b4a0a928d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a3e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 0x55b4a0badb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a3e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 0x55b4a0badb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53208/0 (socket says 172.21.15.104:53208) 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a3e59c700 1 -- 172.21.15.104:0/473114579 learned_addr learned my addr 172.21.15.104:0/473114579 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a3ed9d700 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bcfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a3dd9b700 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bcf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a3e59c700 1 -- 172.21.15.104:0/473114579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bafbb0 msgr2=0x55b4a0bcfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a3e59c700 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bcfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a3e59c700 1 -- 172.21.15.104:0/473114579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bab8a0 msgr2=0x55b4a0bcf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a3e59c700 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bcf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a3e59c700 1 -- 172.21.15.104:0/473114579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4a0bd0090 con 0x55b4a0baaf80 2024-06-09T22:20:39.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a3dd9b700 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bcf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.691+0000 7f6a3ed9d700 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bcfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a3e59c700 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 0x55b4a0badb60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6a34013040 tx=0x7f6a3400e790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878edd04ca678764 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a25ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a25ffb700 1 -- 172.21.15.104:0/473114579 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34021040 con 0x55b4a0baaf80 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a4f128b80 1 -- 172.21.15.104:0/473114579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a3809b750 con 0x55b4a0baaf80 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a25ffb700 1 -- 172.21.15.104:0/473114579 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f6a34005250 con 0x55b4a0baaf80 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a25ffb700 1 -- 172.21.15.104:0/473114579 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6a34042050 con 0x55b4a0baaf80 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a4f128b80 10 client.15204 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a4f128b80 1 -- 172.21.15.104:0/473114579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b4a0c30540 con 0x55b4a0baaf80 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a25ffb700 1 -- 172.21.15.104:0/473114579 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a3809b750 con 0x55b4a0baaf80 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a25ffb700 1 -- 172.21.15.104:0/473114579 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6a3401d450 con 0x55b4a0baaf80 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a4f128b80 10 client.15204 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:39.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a4f128b80 10 client.15204 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a4f128b80 20 client.15204 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55b4a0c34dc0 0x55b4a0c33ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a4f128b80 4 client.15204 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.692+0000 7f6a4f128b80 1 -- 172.21.15.104:0/473114579 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "status"}) -- 0x55b4a0b00240 con 0x55b4a0c34dc0 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a3dd9b700 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55b4a0c34dc0 0x55b4a0c33ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a25ffb700 10 client.15204 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a3dd9b700 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55b4a0c34dc0 0x55b4a0c33ad0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e119363a780d13cb server_cookie=c946530c2ad6846a in_seq=0 out_seq=0 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a25ffb700 1 -- 172.21.15.104:0/473114579 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b4a0b00240 con 0x55b4a0c34dc0 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a25ffb700 10 client.15204 handle_command_reply: tid=0 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a4f128b80 1 client.15204 shutdown 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a4f128b80 1 -- 172.21.15.104:0/473114579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 msgr2=0x55b4a0badb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 0x55b4a0badb60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6a34013040 tx=0x7f6a3400e790 comp rx=0 tx=0).stop 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a4f128b80 1 -- 172.21.15.104:0/473114579 shutdown_connections 2024-06-09T22:20:39.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55b4a0c34dc0 0x55b4a0c33ad0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b4a0baaf80 0x55b4a0badb60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a25ffb700 0 client.15204 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:20:39.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b4a0bafbb0 0x55b4a0bcfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a4f128b80 1 --2- 172.21.15.104:0/473114579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4a0bab8a0 0x55b4a0bcf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.693+0000 7f6a4f128b80 1 -- 172.21.15.104:0/473114579 >> 172.21.15.104:0/473114579 conn(0x55b4a0ad1c70 msgr2=0x55b4a0bb2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:39.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a4f128b80 1 -- 172.21.15.104:0/473114579 shutdown_connections 2024-06-09T22:20:39.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a4f128b80 1 -- 172.21.15.104:0/473114579 wait complete. 2024-06-09T22:20:39.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a4f128b80 20 client.15204 trim_cache size 0 max 16384 2024-06-09T22:20:39.700 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:20:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:20:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 3, 2024-06-09T22:20:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24419, 2024-06-09T22:20:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:20:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:20:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:20:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 170.262758951, 2024-06-09T22:20:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 28, 2024-06-09T22:20:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 99, 2024-06-09T22:20:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 97, 2024-06-09T22:20:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 249.68721514200001 2024-06-09T22:20:39.701 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:20:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a448a5700 1 -- 172.21.15.104:0/752226141 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6a280a2ca0 msgr2=0x7f6a280a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a448a5700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6a280a2ca0 0x7f6a280a5160 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6a34005ea0 tx=0x7f6a34005e30 comp rx=0 tx=0).stop 2024-06-09T22:20:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a448a5700 1 -- 172.21.15.104:0/752226141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a401066b0 msgr2=0x7f6a401989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a448a5700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a401066b0 0x7f6a401989a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6a38007d70 tx=0x7f6a3800a340 comp rx=0 tx=0).stop 2024-06-09T22:20:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a448a5700 1 -- 172.21.15.104:0/752226141 shutdown_connections 2024-06-09T22:20:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a448a5700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f6a280a2ca0 0x7f6a280a5160 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a448a5700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a401066b0 0x7f6a401989a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a448a5700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a40101620 0x7f6a4019e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a448a5700 1 --2- 172.21.15.104:0/752226141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a40101fd0 0x7f6a4019ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a448a5700 1 -- 172.21.15.104:0/752226141 >> 172.21.15.104:0/752226141 conn(0x7f6a40078360 msgr2=0x7f6a401051b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a448a5700 1 -- 172.21.15.104:0/752226141 shutdown_connections 2024-06-09T22:20:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:39.694+0000 7f6a448a5700 1 -- 172.21.15.104:0/752226141 wait complete. 2024-06-09T22:20:40.559 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.f status 2024-06-09T22:20:40.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:40 smithi153 ceph-mon[36599]: pgmap v426: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:40.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:40 smithi104 ceph-mon[32389]: pgmap v426: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:40.800 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:40.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:40 smithi194 ceph-mon[36856]: pgmap v426: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.430+0000 7fe3ca24d700 1 -- 172.21.15.104:0/2941326543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3c4108500 msgr2=0x7fe3c41088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.430+0000 7fe3ca24d700 1 --2- 172.21.15.104:0/2941326543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3c4108500 0x7fe3c41088e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009990 tx=0x7fe3b4009ca0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.430+0000 7fe3ca24d700 1 -- 172.21.15.104:0/2941326543 shutdown_connections 2024-06-09T22:20:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.430+0000 7fe3ca24d700 1 --2- 172.21.15.104:0/2941326543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3c4108500 0x7fe3c41088e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.430+0000 7fe3ca24d700 1 --2- 172.21.15.104:0/2941326543 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c406ab30 0x7fe3c410f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.430+0000 7fe3ca24d700 1 --2- 172.21.15.104:0/2941326543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3c406a170 0x7fe3c406a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.430+0000 7fe3ca24d700 1 -- 172.21.15.104:0/2941326543 >> 172.21.15.104:0/2941326543 conn(0x7fe3c4078340 msgr2=0x7fe3c4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.430+0000 7fe3ca24d700 1 -- 172.21.15.104:0/2941326543 shutdown_connections 2024-06-09T22:20:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.430+0000 7fe3ca24d700 1 -- 172.21.15.104:0/2941326543 wait complete. 2024-06-09T22:20:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.430+0000 7fe3ca24d700 1 Processor -- start 2024-06-09T22:20:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3ca24d700 1 -- start start 2024-06-09T22:20:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3ca24d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3c406a170 0x7fe3c419e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3ca24d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c406ab30 0x7fe3c419ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3ca24d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3c4108500 0x7fe3c4198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3ca24d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c4112780 con 0x7fe3c4108500 2024-06-09T22:20:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3ca24d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c4112600 con 0x7fe3c406a170 2024-06-09T22:20:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3ca24d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c4112900 con 0x7fe3c406ab30 2024-06-09T22:20:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3c4108500 0x7fe3c4198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3c4108500 0x7fe3c4198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53224/0 (socket says 172.21.15.104:53224) 2024-06-09T22:20:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3c3fff700 1 -- 172.21.15.104:0/3587771568 learned_addr learned my addr 172.21.15.104:0/3587771568 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3c406a170 0x7fe3c419e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3c2ffd700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c406ab30 0x7fe3c419ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3c3fff700 1 -- 172.21.15.104:0/3587771568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c406ab30 msgr2=0x7fe3c419ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3c3fff700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c406ab30 0x7fe3c419ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3c3fff700 1 -- 172.21.15.104:0/3587771568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3c406a170 msgr2=0x7fe3c419e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3c3fff700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3c406a170 0x7fe3c419e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3c3fff700 1 -- 172.21.15.104:0/3587771568 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c4199240 con 0x7fe3c4108500 2024-06-09T22:20:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3c406a170 0x7fe3c419e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:20:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.431+0000 7fe3c2ffd700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c406ab30 0x7fe3c419ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.432+0000 7fe3c3fff700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3c4108500 0x7fe3c4198ae0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc007ee0 tx=0x7fe3bc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.432+0000 7fe3c0ff9700 1 -- 172.21.15.104:0/3587771568 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc00dbd0 con 0x7fe3c4108500 2024-06-09T22:20:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.432+0000 7fe3ca24d700 1 -- 172.21.15.104:0/3587771568 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c4199530 con 0x7fe3c4108500 2024-06-09T22:20:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.432+0000 7fe3c0ff9700 1 -- 172.21.15.104:0/3587771568 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3bc00dd70 con 0x7fe3c4108500 2024-06-09T22:20:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.432+0000 7fe3c0ff9700 1 -- 172.21.15.104:0/3587771568 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc005a60 con 0x7fe3c4108500 2024-06-09T22:20:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.432+0000 7fe3ca24d700 1 -- 172.21.15.104:0/3587771568 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c4101390 con 0x7fe3c4108500 2024-06-09T22:20:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.434+0000 7fe3c0ff9700 1 -- 172.21.15.104:0/3587771568 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fe3bc01a020 con 0x7fe3c4108500 2024-06-09T22:20:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.435+0000 7fe3c0ff9700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe3b00a2c20 0x7fe3b00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.435+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe3b00a2c20 0x7fe3b00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.435+0000 7fe3c0ff9700 1 -- 172.21.15.104:0/3587771568 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fe3bc0cea50 con 0x7fe3c4108500 2024-06-09T22:20:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.435+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe3b00a2c20 0x7fe3b00a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4107f80 tx=0x7fe3b4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3d4ad0b80 1 Processor -- start 2024-06-09T22:20:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3d4ad0b80 1 -- start start 2024-06-09T22:20:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3d4ad0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 0x55e6df8909d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3d4ad0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 0x55e6df88a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3d4ad0b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df88b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3d4ad0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e6df7dc7c0 con 0x55e6df8905f0 2024-06-09T22:20:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3d4ad0b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e6df7846d0 con 0x55e6df88aed0 2024-06-09T22:20:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3d4ad0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e6df7d8780 con 0x55e6df890f10 2024-06-09T22:20:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3c37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 0x55e6df8909d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3c37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 0x55e6df8909d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53234/0 (socket says 172.21.15.104:53234) 2024-06-09T22:20:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3c37fe700 1 -- 172.21.15.104:0/1960414744 learned_addr learned my addr 172.21.15.104:0/1960414744 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3c3fff700 1 --2- 172.21.15.104:0/1960414744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df88b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3c2ffd700 1 --2- 172.21.15.104:0/1960414744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 0x55e6df88a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3c37fe700 1 -- 172.21.15.104:0/1960414744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 msgr2=0x55e6df88a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/1960414744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 0x55e6df88a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3c37fe700 1 -- 172.21.15.104:0/1960414744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 msgr2=0x55e6df88b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/1960414744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df88b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.436+0000 7fe3c37fe700 1 -- 172.21.15.104:0/1960414744 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6df77af00 con 0x55e6df8905f0 2024-06-09T22:20:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.437+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/1960414744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 0x55e6df8909d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4015000 tx=0x7fe3b4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e2624055b97c66 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.437+0000 7fe3b9ffb700 1 -- 172.21.15.104:0/1960414744 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b400be10 con 0x55e6df8905f0 2024-06-09T22:20:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.437+0000 7fe3b9ffb700 1 -- 172.21.15.104:0/1960414744 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b400fb50 con 0x55e6df8905f0 2024-06-09T22:20:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.437+0000 7fe3b9ffb700 1 -- 172.21.15.104:0/1960414744 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b402f660 con 0x55e6df8905f0 2024-06-09T22:20:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.437+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/1960414744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 msgr2=0x55e6df8909d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.437+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/1960414744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 0x55e6df8909d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4015000 tx=0x7fe3b4020000 comp rx=0 tx=0).stop 2024-06-09T22:20:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.438+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/1960414744 shutdown_connections 2024-06-09T22:20:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.438+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/1960414744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 0x55e6df8909d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.438+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/1960414744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df88b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.438+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/1960414744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 0x55e6df88a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.438+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/1960414744 >> 172.21.15.104:0/1960414744 conn(0x55e6df7b5c70 msgr2=0x55e6df885800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.438+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/1960414744 shutdown_connections 2024-06-09T22:20:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.438+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/1960414744 wait complete. 2024-06-09T22:20:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.438+0000 7fe3d4ad0b80 1 -- start start 2024-06-09T22:20:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.438+0000 7fe3d4ad0b80 1 Processor -- start 2024-06-09T22:20:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3d4ad0b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df896c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3d4ad0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 0x55e6df893f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3d4ad0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 0x55e6df8944b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3d4ad0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e6df8978c0 con 0x55e6df8905f0 2024-06-09T22:20:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3d4ad0b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e6df77cba0 con 0x55e6df88aed0 2024-06-09T22:20:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3d4ad0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e6df897a40 con 0x55e6df890f10 2024-06-09T22:20:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3d4ad0b80 10 client.0 _ll_register_callbacks cb 0x55e6df7768d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3c2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 0x55e6df893f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3c2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 0x55e6df893f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53244/0 (socket says 172.21.15.104:53244) 2024-06-09T22:20:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3c2ffd700 1 -- 172.21.15.104:0/4170270981 learned_addr learned my addr 172.21.15.104:0/4170270981 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df896c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3c3fff700 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 0x55e6df8944b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3c2ffd700 1 -- 172.21.15.104:0/4170270981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 msgr2=0x55e6df8944b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3c2ffd700 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 0x55e6df8944b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3c2ffd700 1 -- 172.21.15.104:0/4170270981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 msgr2=0x55e6df896c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3c2ffd700 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df896c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3c2ffd700 1 -- 172.21.15.104:0/4170270981 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6df72b880 con 0x55e6df8905f0 2024-06-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3c2ffd700 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 0x55e6df893f70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac010700 tx=0x7fe3ac010a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3373c9f19fcfac62 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3a2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3a2ffd700 1 -- 172.21.15.104:0/4170270981 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac004230 con 0x55e6df8905f0 2024-06-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3a2ffd700 1 -- 172.21.15.104:0/4170270981 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fe3ac0043d0 con 0x55e6df8905f0 2024-06-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.439+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4170270981 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3b400fb50 con 0x55e6df8905f0 2024-06-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.440+0000 7fe3a2ffd700 1 -- 172.21.15.104:0/4170270981 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe3ac024050 con 0x55e6df8905f0 2024-06-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.440+0000 7fe3d4ad0b80 10 client.15216 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.440+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4170270981 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e6df8a2220 con 0x55e6df8905f0 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.440+0000 7fe3a2ffd700 1 -- 172.21.15.104:0/4170270981 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b400fb50 con 0x55e6df8905f0 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.440+0000 7fe3a2ffd700 1 -- 172.21.15.104:0/4170270981 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe3ac011a00 con 0x55e6df8905f0 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.440+0000 7fe3d4ad0b80 10 client.15216 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.440+0000 7fe3d4ad0b80 10 client.15216 resolve_mds: resolved name 'f' to daemon mds.f 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.440+0000 7fe3d4ad0b80 20 client.15216 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.440+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55e6df8a4850 0x55e6df8a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.440+0000 7fe3d4ad0b80 4 client.15216 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.440+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4170270981 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6df7e9cc0 con 0x55e6df8a4850 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.441+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55e6df8a4850 0x55e6df8a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.441+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55e6df8a4850 0x55e6df8a5cc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f904d402f3e6a36f server_cookie=7785c7c401e7928 in_seq=0 out_seq=0 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.441+0000 7fe3a2ffd700 10 client.15216 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:20:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3a2ffd700 1 -- 172.21.15.104:0/4170270981 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55e6df7e9cc0 con 0x55e6df8a4850 2024-06-09T22:20:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3a2ffd700 10 client.15216 handle_command_reply: tid=0 2024-06-09T22:20:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3d4ad0b80 1 client.15216 shutdown 2024-06-09T22:20:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4170270981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 msgr2=0x55e6df893f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 0x55e6df893f70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac010700 tx=0x7fe3ac010a10 comp rx=0 tx=0).stop 2024-06-09T22:20:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4170270981 shutdown_connections 2024-06-09T22:20:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55e6df8a4850 0x55e6df8a5cc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3a2ffd700 0 client.15216 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:20:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 0x55e6df8944b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df896c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/4170270981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df8905f0 0x55e6df893f70 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4170270981 >> 172.21.15.104:0/4170270981 conn(0x55e6df7b5c70 msgr2=0x55e6df889980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4170270981 shutdown_connections 2024-06-09T22:20:42.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.442+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4170270981 wait complete. 2024-06-09T22:20:42.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.443+0000 7fe3d4ad0b80 20 client.15216 trim_cache size 0 max 16384 2024-06-09T22:20:42.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3d4ad0b80 1 Processor -- start 2024-06-09T22:20:42.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3d4ad0b80 1 -- start start 2024-06-09T22:20:42.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3d4ad0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df88aed0 0x55e6df88a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3d4ad0b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df8905f0 0x55e6df891ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3d4ad0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 0x55e6df8af120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3d4ad0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e6df8a4000 con 0x55e6df88aed0 2024-06-09T22:20:42.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3d4ad0b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e6df8a46d0 con 0x55e6df8905f0 2024-06-09T22:20:42.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3d4ad0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e6df8a4da0 con 0x55e6df890f10 2024-06-09T22:20:42.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3c37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df88aed0 0x55e6df88a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3c37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df88aed0 0x55e6df88a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53254/0 (socket says 172.21.15.104:53254) 2024-06-09T22:20:42.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3c37fe700 1 -- 172.21.15.104:0/4139153940 learned_addr learned my addr 172.21.15.104:0/4139153940 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:42.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3c3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 0x55e6df8af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.464+0000 7fe3c2ffd700 1 --2- 172.21.15.104:0/4139153940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df8905f0 0x55e6df891ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.465+0000 7fe3c37fe700 1 -- 172.21.15.104:0/4139153940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 msgr2=0x55e6df8af120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:42.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.465+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/4139153940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 0x55e6df8af120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.465+0000 7fe3c37fe700 1 -- 172.21.15.104:0/4139153940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df8905f0 msgr2=0x55e6df891ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:42.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.465+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/4139153940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df8905f0 0x55e6df891ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.465+0000 7fe3c37fe700 1 -- 172.21.15.104:0/4139153940 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6df8924e0 con 0x55e6df88aed0 2024-06-09T22:20:42.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.465+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/4139153940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df88aed0 0x55e6df88a990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe3b400bde0 tx=0x7fe3b4026040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e47e144e930e0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:42.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.467+0000 7fe3b97fa700 1 -- 172.21.15.104:0/4139153940 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b40244d0 con 0x55e6df88aed0 2024-06-09T22:20:42.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.467+0000 7fe3b97fa700 1 -- 172.21.15.104:0/4139153940 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b4024b50 con 0x55e6df88aed0 2024-06-09T22:20:42.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.467+0000 7fe3b97fa700 1 -- 172.21.15.104:0/4139153940 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b40383f0 con 0x55e6df88aed0 2024-06-09T22:20:42.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.468+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4139153940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df88aed0 msgr2=0x55e6df88a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:42.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.468+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/4139153940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df88aed0 0x55e6df88a990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe3b400bde0 tx=0x7fe3b4026040 comp rx=0 tx=0).stop 2024-06-09T22:20:42.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.468+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4139153940 shutdown_connections 2024-06-09T22:20:42.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.468+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/4139153940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df88aed0 0x55e6df88a990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.468+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/4139153940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df890f10 0x55e6df8af120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.468+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/4139153940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df8905f0 0x55e6df891ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.468+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4139153940 >> 172.21.15.104:0/4139153940 conn(0x55e6df7b5c70 msgr2=0x55e6df8913b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:42.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.468+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4139153940 shutdown_connections 2024-06-09T22:20:42.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.468+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/4139153940 wait complete. 2024-06-09T22:20:42.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.468+0000 7fe3d4ad0b80 1 -- start start 2024-06-09T22:20:42.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.468+0000 7fe3d4ad0b80 1 Processor -- start 2024-06-09T22:20:42.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3d4ad0b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df8af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3d4ad0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df8905f0 0x55e6df8a8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3d4ad0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df890f10 0x55e6df8a9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3d4ad0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e6df8a5b40 con 0x55e6df890f10 2024-06-09T22:20:42.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3d4ad0b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e6df8a5470 con 0x55e6df88aed0 2024-06-09T22:20:42.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3d4ad0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e6df8a6210 con 0x55e6df8905f0 2024-06-09T22:20:42.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3d4ad0b80 10 client.0 _ll_register_callbacks cb 0x55e6df7768d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:42.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df890f10 0x55e6df8a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df890f10 0x55e6df8a9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53260/0 (socket says 172.21.15.104:53260) 2024-06-09T22:20:42.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3c3fff700 1 -- 172.21.15.104:0/2820118127 learned_addr learned my addr 172.21.15.104:0/2820118127 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:42.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df8af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3c2ffd700 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df8905f0 0x55e6df8a8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3c3fff700 1 -- 172.21.15.104:0/2820118127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df8905f0 msgr2=0x55e6df8a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:42.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3c3fff700 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df8905f0 0x55e6df8a8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3c3fff700 1 -- 172.21.15.104:0/2820118127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 msgr2=0x55e6df8af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:42.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3c3fff700 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df8af9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3c3fff700 1 -- 172.21.15.104:0/2820118127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6df8ad500 con 0x55e6df890f10 2024-06-09T22:20:42.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df8af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:42.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3c3fff700 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df890f10 0x55e6df8a9430 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc0074d0 tx=0x7fe3bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c40a4033ee45a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:42.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3a2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:42.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3a2ffd700 1 -- 172.21.15.104:0/2820118127 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc0a3b90 con 0x55e6df890f10 2024-06-09T22:20:42.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3a2ffd700 1 -- 172.21.15.104:0/2820118127 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fe3bc022250 con 0x55e6df890f10 2024-06-09T22:20:42.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.469+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/2820118127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3b4024b50 con 0x55e6df890f10 2024-06-09T22:20:42.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.470+0000 7fe3a2ffd700 1 -- 172.21.15.104:0/2820118127 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe3bc022450 con 0x55e6df890f10 2024-06-09T22:20:42.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.470+0000 7fe3d4ad0b80 10 client.15222 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:42.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.470+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/2820118127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e6df910350 con 0x55e6df890f10 2024-06-09T22:20:42.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.470+0000 7fe3a2ffd700 1 -- 172.21.15.104:0/2820118127 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b4024b50 con 0x55e6df890f10 2024-06-09T22:20:42.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.470+0000 7fe3a2ffd700 1 -- 172.21.15.104:0/2820118127 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe3bc0b9b60 con 0x55e6df890f10 2024-06-09T22:20:42.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.470+0000 7fe3d4ad0b80 10 client.15222 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:42.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.470+0000 7fe3d4ad0b80 10 client.15222 resolve_mds: resolved name 'f' to daemon mds.f 2024-06-09T22:20:42.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.470+0000 7fe3d4ad0b80 20 client.15222 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:42.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.470+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55e6df8a4850 0x55e6df913c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:42.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.470+0000 7fe3d4ad0b80 4 client.15222 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:20:42.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.470+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/2820118127 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "status"}) -- 0x55e6df7e4240 con 0x55e6df8a4850 2024-06-09T22:20:42.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.470+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55e6df8a4850 0x55e6df913c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:42.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3a2ffd700 10 client.15222 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:20:42.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3c37fe700 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55e6df8a4850 0x55e6df913c40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d76ae74679435b55 server_cookie=85ce890c5dd71643 in_seq=0 out_seq=0 2024-06-09T22:20:42.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3a2ffd700 1 -- 172.21.15.104:0/2820118127 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55e6df7e4240 con 0x55e6df8a4850 2024-06-09T22:20:42.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3a2ffd700 10 client.15222 handle_command_reply: tid=0 2024-06-09T22:20:42.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3d4ad0b80 1 client.15222 shutdown 2024-06-09T22:20:42.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/2820118127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df890f10 msgr2=0x55e6df8a9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:42.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df890f10 0x55e6df8a9430 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc0074d0 tx=0x7fe3bc00b040 comp rx=0 tx=0).stop 2024-06-09T22:20:42.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/2820118127 shutdown_connections 2024-06-09T22:20:42.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55e6df8a4850 0x55e6df913c40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e6df890f10 0x55e6df8a9430 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e6df88aed0 0x55e6df8af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3a2ffd700 0 client.15222 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:20:42.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3d4ad0b80 1 --2- 172.21.15.104:0/2820118127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e6df8905f0 0x55e6df8a8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/2820118127 >> 172.21.15.104:0/2820118127 conn(0x55e6df7b5c70 msgr2=0x55e6df87edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:42.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/2820118127 shutdown_connections 2024-06-09T22:20:42.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3d4ad0b80 1 -- 172.21.15.104:0/2820118127 wait complete. 2024-06-09T22:20:42.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.471+0000 7fe3d4ad0b80 20 client.15222 trim_cache size 0 max 16384 2024-06-09T22:20:42.477 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:20:42.477 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:20:42.477 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 2, 2024-06-09T22:20:42.477 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24442, 2024-06-09T22:20:42.477 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:20:42.477 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:20:42.478 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:20:42.478 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 174.05300611300001, 2024-06-09T22:20:42.478 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 28, 2024-06-09T22:20:42.478 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 99, 2024-06-09T22:20:42.478 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 97, 2024-06-09T22:20:42.478 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 271.37822020900001 2024-06-09T22:20:42.478 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:20:42.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.472+0000 7fe3ca24d700 1 -- 172.21.15.104:0/3587771568 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe3b00a2c20 msgr2=0x7fe3b00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:42.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.472+0000 7fe3ca24d700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe3b00a2c20 0x7fe3b00a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4107f80 tx=0x7fe3b4005960 comp rx=0 tx=0).stop 2024-06-09T22:20:42.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.472+0000 7fe3ca24d700 1 -- 172.21.15.104:0/3587771568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3c4108500 msgr2=0x7fe3c4198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:42.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.472+0000 7fe3ca24d700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3c4108500 0x7fe3c4198ae0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc007ee0 tx=0x7fe3bc00d3b0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.472+0000 7fe3ca24d700 1 -- 172.21.15.104:0/3587771568 shutdown_connections 2024-06-09T22:20:42.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.472+0000 7fe3ca24d700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fe3b00a2c20 0x7fe3b00a50e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.472+0000 7fe3ca24d700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3c4108500 0x7fe3c4198ae0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.472+0000 7fe3ca24d700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3c406a170 0x7fe3c419e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.472+0000 7fe3ca24d700 1 --2- 172.21.15.104:0/3587771568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c406ab30 0x7fe3c419ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:42.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.472+0000 7fe3ca24d700 1 -- 172.21.15.104:0/3587771568 >> 172.21.15.104:0/3587771568 conn(0x7fe3c4078340 msgr2=0x7fe3c4100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:42.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.472+0000 7fe3ca24d700 1 -- 172.21.15.104:0/3587771568 shutdown_connections 2024-06-09T22:20:42.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:42.472+0000 7fe3ca24d700 1 -- 172.21.15.104:0/3587771568 wait complete. 2024-06-09T22:20:42.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:42 smithi153 ceph-mon[36599]: pgmap v427: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:42.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:42 smithi104 ceph-mon[32389]: pgmap v427: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:42.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:42 smithi194 ceph-mon[36856]: pgmap v427: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-06-09T22:20:43.306 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.c status 2024-06-09T22:20:43.549 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:44.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:44 smithi153 ceph-mon[36599]: pgmap v428: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:44.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:44 smithi104 ceph-mon[32389]: pgmap v428: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:44.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:44 smithi194 ceph-mon[36856]: pgmap v428: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-06-09T22:20:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.395+0000 7fa330479700 1 -- 172.21.15.104:0/1509237323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa328102ee0 msgr2=0x7fa32810f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.395+0000 7fa330479700 1 --2- 172.21.15.104:0/1509237323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa328102ee0 0x7fa32810f3c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa32000b0a0 tx=0x7fa32000b3b0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.395+0000 7fa330479700 1 -- 172.21.15.104:0/1509237323 shutdown_connections 2024-06-09T22:20:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.395+0000 7fa330479700 1 --2- 172.21.15.104:0/1509237323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa328102ee0 0x7fa32810f3c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.395+0000 7fa330479700 1 --2- 172.21.15.104:0/1509237323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa328107450 0x7fa328107830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.395+0000 7fa330479700 1 --2- 172.21.15.104:0/1509237323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa328102540 0x7fa3281029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.395+0000 7fa330479700 1 -- 172.21.15.104:0/1509237323 >> 172.21.15.104:0/1509237323 conn(0x7fa3280795f0 msgr2=0x7fa328079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.395+0000 7fa330479700 1 -- 172.21.15.104:0/1509237323 shutdown_connections 2024-06-09T22:20:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.395+0000 7fa330479700 1 -- 172.21.15.104:0/1509237323 wait complete. 2024-06-09T22:20:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa330479700 1 Processor -- start 2024-06-09T22:20:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa330479700 1 -- start start 2024-06-09T22:20:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa330479700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa328102540 0x7fa32819e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa330479700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa328102ee0 0x7fa32819ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa330479700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa328107450 0x7fa328198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa330479700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa3281127e0 con 0x7fa328102ee0 2024-06-09T22:20:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa330479700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa328112660 con 0x7fa328102540 2024-06-09T22:20:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa330479700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa328112960 con 0x7fa328107450 2024-06-09T22:20:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa32da14700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa328102ee0 0x7fa32819ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa32da14700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa328102ee0 0x7fa32819ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53276/0 (socket says 172.21.15.104:53276) 2024-06-09T22:20:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa32da14700 1 -- 172.21.15.104:0/2010383622 learned_addr learned my addr 172.21.15.104:0/2010383622 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa32e215700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa328102540 0x7fa32819e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.396+0000 7fa32ea16700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa328107450 0x7fa328198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.397+0000 7fa32da14700 1 -- 172.21.15.104:0/2010383622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa328107450 msgr2=0x7fa328198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.397+0000 7fa32da14700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa328107450 0x7fa328198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.397+0000 7fa32da14700 1 -- 172.21.15.104:0/2010383622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa328102540 msgr2=0x7fa32819e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.397+0000 7fa32da14700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa328102540 0x7fa32819e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.397+0000 7fa32da14700 1 -- 172.21.15.104:0/2010383622 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa328199180 con 0x7fa328102ee0 2024-06-09T22:20:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.397+0000 7fa32da14700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa328102ee0 0x7fa32819ed50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00e9a0 tx=0x7fa31c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.397+0000 7fa31b7fe700 1 -- 172.21.15.104:0/2010383622 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c00cbd0 con 0x7fa328102ee0 2024-06-09T22:20:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.397+0000 7fa31b7fe700 1 -- 172.21.15.104:0/2010383622 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa31c00cd70 con 0x7fa328102ee0 2024-06-09T22:20:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.397+0000 7fa330479700 1 -- 172.21.15.104:0/2010383622 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa328199470 con 0x7fa328102ee0 2024-06-09T22:20:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.397+0000 7fa31b7fe700 1 -- 172.21.15.104:0/2010383622 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c018750 con 0x7fa328102ee0 2024-06-09T22:20:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.397+0000 7fa330479700 1 -- 172.21.15.104:0/2010383622 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3281a56e0 con 0x7fa328102ee0 2024-06-09T22:20:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.399+0000 7fa31b7fe700 1 -- 172.21.15.104:0/2010383622 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa31c010430 con 0x7fa328102ee0 2024-06-09T22:20:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.400+0000 7fa31b7fe700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa3140a2c20 0x7fa3140a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.400+0000 7fa32e215700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa3140a2c20 0x7fa3140a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.400+0000 7fa31b7fe700 1 -- 172.21.15.104:0/2010383622 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fa31c0cf690 con 0x7fa328102ee0 2024-06-09T22:20:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa32e215700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa3140a2c20 0x7fa3140a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa328102510 tx=0x7fa32400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa33acfcb80 1 Processor -- start 2024-06-09T22:20:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa33acfcb80 1 -- start start 2024-06-09T22:20:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa33acfcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 0x564989de12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa33acfcb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de1830 0x564989ddb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa33acfcb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989ddbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa33acfcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564989ccdf10 con 0x564989de0f10 2024-06-09T22:20:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa33acfcb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564989d2c800 con 0x564989ddb750 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa33acfcb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564989c94410 con 0x564989de1830 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa32e215700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 0x564989de12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa32e215700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 0x564989de12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53280/0 (socket says 172.21.15.104:53280) 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa32e215700 1 -- 172.21.15.104:0/628447081 learned_addr learned my addr 172.21.15.104:0/628447081 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.401+0000 7fa32da14700 1 --2- 172.21.15.104:0/628447081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de1830 0x564989ddb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.402+0000 7fa32ea16700 1 --2- 172.21.15.104:0/628447081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989ddbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.402+0000 7fa32e215700 1 -- 172.21.15.104:0/628447081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de1830 msgr2=0x564989ddb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.402+0000 7fa32e215700 1 --2- 172.21.15.104:0/628447081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de1830 0x564989ddb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.402+0000 7fa32e215700 1 -- 172.21.15.104:0/628447081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 msgr2=0x564989ddbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.402+0000 7fa32e215700 1 --2- 172.21.15.104:0/628447081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989ddbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.402+0000 7fa32e215700 1 -- 172.21.15.104:0/628447081 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564989c7b880 con 0x564989de0f10 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.402+0000 7fa32e215700 1 --2- 172.21.15.104:0/628447081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 0x564989de12f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa324005c50 tx=0x7fa324013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c417158a7eb5732a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.402+0000 7fa318d70700 1 -- 172.21.15.104:0/628447081 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa324011db0 con 0x564989de0f10 2024-06-09T22:20:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.402+0000 7fa318d70700 1 -- 172.21.15.104:0/628447081 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa324010e70 con 0x564989de0f10 2024-06-09T22:20:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.402+0000 7fa318d70700 1 -- 172.21.15.104:0/628447081 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32401c770 con 0x564989de0f10 2024-06-09T22:20:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.403+0000 7fa33acfcb80 1 -- 172.21.15.104:0/628447081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 msgr2=0x564989de12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.403+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/628447081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 0x564989de12f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa324005c50 tx=0x7fa324013000 comp rx=0 tx=0).stop 2024-06-09T22:20:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.403+0000 7fa33acfcb80 1 -- 172.21.15.104:0/628447081 shutdown_connections 2024-06-09T22:20:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.403+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/628447081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 0x564989de12f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.403+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/628447081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989ddbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.403+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/628447081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de1830 0x564989ddb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.403+0000 7fa33acfcb80 1 -- 172.21.15.104:0/628447081 >> 172.21.15.104:0/628447081 conn(0x564989d08030 msgr2=0x564989dd61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.403+0000 7fa33acfcb80 1 -- 172.21.15.104:0/628447081 shutdown_connections 2024-06-09T22:20:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.403+0000 7fa33acfcb80 1 -- 172.21.15.104:0/628447081 wait complete. 2024-06-09T22:20:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.403+0000 7fa33acfcb80 1 -- start start 2024-06-09T22:20:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.403+0000 7fa33acfcb80 1 Processor -- start 2024-06-09T22:20:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa33acfcb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989de7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa33acfcb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de0f10 0x564989de4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa33acfcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de1830 0x564989de4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa33acfcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564989cccf70 con 0x564989de1830 2024-06-09T22:20:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa33acfcb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564989d287c0 con 0x564989ddb750 2024-06-09T22:20:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa33acfcb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564989de8210 con 0x564989de0f10 2024-06-09T22:20:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa33acfcb80 10 client.0 _ll_register_callbacks cb 0x564989cc2560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32ea16700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de1830 0x564989de4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32ea16700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de1830 0x564989de4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53288/0 (socket says 172.21.15.104:53288) 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32ea16700 1 -- 172.21.15.104:0/785824015 learned_addr learned my addr 172.21.15.104:0/785824015 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32e215700 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989de7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32da14700 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de0f10 0x564989de4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32ea16700 1 -- 172.21.15.104:0/785824015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de0f10 msgr2=0x564989de4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32ea16700 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de0f10 0x564989de4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32ea16700 1 -- 172.21.15.104:0/785824015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 msgr2=0x564989de7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32ea16700 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989de7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32ea16700 1 -- 172.21.15.104:0/785824015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564989cf2700 con 0x564989de1830 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32da14700 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de0f10 0x564989de4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32e215700 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989de7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:20:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa32ea16700 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de1830 0x564989de4db0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa32000bcc0 tx=0x7fa32000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c8724d6c72198d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.404+0000 7fa311ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa311ffb700 1 -- 172.21.15.104:0/785824015 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3200074e0 con 0x564989de1830 2024-06-09T22:20:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa311ffb700 1 -- 172.21.15.104:0/785824015 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fa320007680 con 0x564989de1830 2024-06-09T22:20:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa33acfcb80 1 -- 172.21.15.104:0/785824015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa324010e70 con 0x564989de1830 2024-06-09T22:20:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa311ffb700 1 -- 172.21.15.104:0/785824015 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa32002e050 con 0x564989de1830 2024-06-09T22:20:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa33acfcb80 10 client.15234 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa33acfcb80 1 -- 172.21.15.104:0/785824015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564989df2a20 con 0x564989de1830 2024-06-09T22:20:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa311ffb700 1 -- 172.21.15.104:0/785824015 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa324010e70 con 0x564989de1830 2024-06-09T22:20:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa311ffb700 1 -- 172.21.15.104:0/785824015 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa320011070 con 0x564989de1830 2024-06-09T22:20:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa33acfcb80 10 client.15234 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa33acfcb80 10 client.15234 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-09T22:20:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa33acfcb80 20 client.15234 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x564989df5070 0x564989df64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa33acfcb80 4 client.15234 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:20:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.405+0000 7fa33acfcb80 1 -- 172.21.15.104:0/785824015 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564989d39d00 con 0x564989df5070 2024-06-09T22:20:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.406+0000 7fa32e215700 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x564989df5070 0x564989df64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.406+0000 7fa311ffb700 10 client.15234 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:20:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.406+0000 7fa32e215700 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x564989df5070 0x564989df64e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c7cce296192f9ecb server_cookie=27b924cfb95b7e86 in_seq=0 out_seq=0 2024-06-09T22:20:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa311ffb700 1 -- 172.21.15.104:0/785824015 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x564989d39d00 con 0x564989df5070 2024-06-09T22:20:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa311ffb700 10 client.15234 handle_command_reply: tid=0 2024-06-09T22:20:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa33acfcb80 1 client.15234 shutdown 2024-06-09T22:20:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa33acfcb80 1 -- 172.21.15.104:0/785824015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de1830 msgr2=0x564989de4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de1830 0x564989de4db0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa32000bcc0 tx=0x7fa32000bda0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa33acfcb80 1 -- 172.21.15.104:0/785824015 shutdown_connections 2024-06-09T22:20:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x564989df5070 0x564989df64e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de1830 0x564989de4db0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989de7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/785824015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de0f10 0x564989de4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa33acfcb80 1 -- 172.21.15.104:0/785824015 >> 172.21.15.104:0/785824015 conn(0x564989d08030 msgr2=0x564989dd56b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa311ffb700 0 client.15234 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:20:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa33acfcb80 1 -- 172.21.15.104:0/785824015 shutdown_connections 2024-06-09T22:20:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa33acfcb80 1 -- 172.21.15.104:0/785824015 wait complete. 2024-06-09T22:20:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.407+0000 7fa33acfcb80 20 client.15234 trim_cache size 0 max 16384 2024-06-09T22:20:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.428+0000 7fa33acfcb80 1 Processor -- start 2024-06-09T22:20:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa33acfcb80 1 -- start start 2024-06-09T22:20:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa33acfcb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989ddb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa33acfcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 0x564989de4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa33acfcb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de1830 0x564989dd5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa33acfcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564989df4820 con 0x564989de0f10 2024-06-09T22:20:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa33acfcb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564989df4ef0 con 0x564989ddb750 2024-06-09T22:20:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa33acfcb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564989df55c0 con 0x564989de1830 2024-06-09T22:20:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa32da14700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 0x564989de4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa32da14700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 0x564989de4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53300/0 (socket says 172.21.15.104:53300) 2024-06-09T22:20:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa32da14700 1 -- 172.21.15.104:0/2213887690 learned_addr learned my addr 172.21.15.104:0/2213887690 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa32e215700 1 --2- 172.21.15.104:0/2213887690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989ddb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa32ea16700 1 --2- 172.21.15.104:0/2213887690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de1830 0x564989dd5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa32da14700 1 -- 172.21.15.104:0/2213887690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de1830 msgr2=0x564989dd5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa32da14700 1 --2- 172.21.15.104:0/2213887690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de1830 0x564989dd5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa32da14700 1 -- 172.21.15.104:0/2213887690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 msgr2=0x564989ddb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:45.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa32da14700 1 --2- 172.21.15.104:0/2213887690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989ddb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.429+0000 7fa32da14700 1 -- 172.21.15.104:0/2213887690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564989de5750 con 0x564989de0f10 2024-06-09T22:20:45.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.430+0000 7fa32da14700 1 --2- 172.21.15.104:0/2213887690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 0x564989de4e00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa31c098530 tx=0x7fa31c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4e66fd635ed19f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:45.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.430+0000 7fa313fff700 1 -- 172.21.15.104:0/2213887690 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c09bb60 con 0x564989de0f10 2024-06-09T22:20:45.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.430+0000 7fa313fff700 1 -- 172.21.15.104:0/2213887690 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa31c0b3070 con 0x564989de0f10 2024-06-09T22:20:45.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.430+0000 7fa313fff700 1 -- 172.21.15.104:0/2213887690 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c09bdc0 con 0x564989de0f10 2024-06-09T22:20:45.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.430+0000 7fa33acfcb80 1 -- 172.21.15.104:0/2213887690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 msgr2=0x564989de4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:45.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.430+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/2213887690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 0x564989de4e00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa31c098530 tx=0x7fa31c009840 comp rx=0 tx=0).stop 2024-06-09T22:20:45.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 -- 172.21.15.104:0/2213887690 shutdown_connections 2024-06-09T22:20:45.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/2213887690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989de1830 0x564989dd5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/2213887690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989ddb750 0x564989ddb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/2213887690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989de0f10 0x564989de4e00 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 -- 172.21.15.104:0/2213887690 >> 172.21.15.104:0/2213887690 conn(0x564989d08030 msgr2=0x564989ddea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 -- 172.21.15.104:0/2213887690 shutdown_connections 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 -- 172.21.15.104:0/2213887690 wait complete. 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 -- start start 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 Processor -- start 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989ddb750 0x564989dffe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989de1830 0x564989dd5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989dd5760 0x564989dd5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564989df6360 con 0x564989ddb750 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564989df5c90 con 0x564989de1830 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564989df6a30 con 0x564989dd5760 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa33acfcb80 10 client.0 _ll_register_callbacks cb 0x564989cc2560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa32e215700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989ddb750 0x564989dffe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa32e215700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989ddb750 0x564989dffe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53312/0 (socket says 172.21.15.104:53312) 2024-06-09T22:20:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.431+0000 7fa32e215700 1 -- 172.21.15.104:0/1658793487 learned_addr learned my addr 172.21.15.104:0/1658793487 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa32ea16700 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989dd5760 0x564989dd5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa32da14700 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989de1830 0x564989dd5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa32e215700 1 -- 172.21.15.104:0/1658793487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989dd5760 msgr2=0x564989dd5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa32e215700 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989dd5760 0x564989dd5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa32e215700 1 -- 172.21.15.104:0/1658793487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989de1830 msgr2=0x564989dd5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa32e215700 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989de1830 0x564989dd5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa32e215700 1 -- 172.21.15.104:0/1658793487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564989dfd940 con 0x564989ddb750 2024-06-09T22:20:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa32e215700 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989ddb750 0x564989dffe20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa324011a00 tx=0x7fa324010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6d69db8acea8fb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa311ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa311ffb700 1 -- 172.21.15.104:0/1658793487 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32401e830 con 0x564989ddb750 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa311ffb700 1 -- 172.21.15.104:0/1658793487 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fa32401e9d0 con 0x564989ddb750 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa33acfcb80 1 -- 172.21.15.104:0/1658793487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa31c0b3070 con 0x564989ddb750 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa311ffb700 1 -- 172.21.15.104:0/1658793487 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa324010e70 con 0x564989ddb750 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa33acfcb80 10 client.15240 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.432+0000 7fa33acfcb80 1 -- 172.21.15.104:0/1658793487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564989e64f20 con 0x564989ddb750 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.433+0000 7fa311ffb700 1 -- 172.21.15.104:0/1658793487 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa31c0b3070 con 0x564989ddb750 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.433+0000 7fa311ffb700 1 -- 172.21.15.104:0/1658793487 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa324003b00 con 0x564989ddb750 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.433+0000 7fa33acfcb80 10 client.15240 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.433+0000 7fa33acfcb80 10 client.15240 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.433+0000 7fa33acfcb80 20 client.15240 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.433+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x564989e699b0 0x564989e686c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.433+0000 7fa33acfcb80 4 client.15240 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.433+0000 7fa33acfcb80 1 -- 172.21.15.104:0/1658793487 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "status"}) -- 0x564989d34280 con 0x564989e699b0 2024-06-09T22:20:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.433+0000 7fa32da14700 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x564989e699b0 0x564989e686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa311ffb700 10 client.15240 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:20:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa32da14700 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x564989e699b0 0x564989e686c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=57039877fe13ee6 server_cookie=efbba6a8528a2e40 in_seq=0 out_seq=0 2024-06-09T22:20:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa311ffb700 1 -- 172.21.15.104:0/1658793487 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x564989d34280 con 0x564989e699b0 2024-06-09T22:20:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa311ffb700 10 client.15240 handle_command_reply: tid=0 2024-06-09T22:20:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa33acfcb80 1 client.15240 shutdown 2024-06-09T22:20:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa33acfcb80 1 -- 172.21.15.104:0/1658793487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989ddb750 msgr2=0x564989dffe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989ddb750 0x564989dffe20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa324011a00 tx=0x7fa324010860 comp rx=0 tx=0).stop 2024-06-09T22:20:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa33acfcb80 1 -- 172.21.15.104:0/1658793487 shutdown_connections 2024-06-09T22:20:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x564989e699b0 0x564989e686c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564989ddb750 0x564989dffe20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa311ffb700 0 client.15240 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:20:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564989dd5760 0x564989dd5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa33acfcb80 1 --2- 172.21.15.104:0/1658793487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564989de1830 0x564989dd5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa33acfcb80 1 -- 172.21.15.104:0/1658793487 >> 172.21.15.104:0/1658793487 conn(0x564989d08030 msgr2=0x564989de3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa33acfcb80 1 -- 172.21.15.104:0/1658793487 shutdown_connections 2024-06-09T22:20:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa33acfcb80 1 -- 172.21.15.104:0/1658793487 wait complete. 2024-06-09T22:20:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.434+0000 7fa33acfcb80 20 client.15240 trim_cache size 0 max 16384 2024-06-09T22:20:45.439 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:20:45.439 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:20:45.439 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 1, 2024-06-09T22:20:45.439 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24448, 2024-06-09T22:20:45.439 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:20:45.440 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:20:45.440 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:20:45.440 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 178.043478072, 2024-06-09T22:20:45.440 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 28, 2024-06-09T22:20:45.440 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 99, 2024-06-09T22:20:45.440 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 97, 2024-06-09T22:20:45.440 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 271.51834879 2024-06-09T22:20:45.440 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:20:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.435+0000 7fa330479700 1 -- 172.21.15.104:0/2010383622 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa3140a2c20 msgr2=0x7fa3140a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.435+0000 7fa330479700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa3140a2c20 0x7fa3140a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa328102510 tx=0x7fa32400b410 comp rx=0 tx=0).stop 2024-06-09T22:20:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.435+0000 7fa330479700 1 -- 172.21.15.104:0/2010383622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa328102ee0 msgr2=0x7fa32819ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.435+0000 7fa330479700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa328102ee0 0x7fa32819ed50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00e9a0 tx=0x7fa31c00ecb0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.435+0000 7fa330479700 1 -- 172.21.15.104:0/2010383622 shutdown_connections 2024-06-09T22:20:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.435+0000 7fa330479700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa3140a2c20 0x7fa3140a50e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.435+0000 7fa330479700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa328107450 0x7fa328198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.435+0000 7fa330479700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa328102540 0x7fa32819e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.435+0000 7fa330479700 1 --2- 172.21.15.104:0/2010383622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa328102ee0 0x7fa32819ed50 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.435+0000 7fa330479700 1 -- 172.21.15.104:0/2010383622 >> 172.21.15.104:0/2010383622 conn(0x7fa3280795f0 msgr2=0x7fa328100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.435+0000 7fa330479700 1 -- 172.21.15.104:0/2010383622 shutdown_connections 2024-06-09T22:20:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:45.435+0000 7fa330479700 1 -- 172.21.15.104:0/2010383622 wait complete. 2024-06-09T22:20:46.310 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-06-09T22:20:46.563 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:46.685 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:46 smithi104 ceph-mon[32389]: pgmap v429: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:46.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:46 smithi153 ceph-mon[36599]: pgmap v429: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:46.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:46 smithi194 ceph-mon[36856]: pgmap v429: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-06-09T22:20:48.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.392+0000 7f0245276700 1 -- 172.21.15.104:0/3585045218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0240102740 msgr2=0x7f0240102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:48.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.392+0000 7f0245276700 1 --2- 172.21.15.104:0/3585045218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0240102740 0x7f0240102b20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f02300099c0 tx=0x7f0230009cd0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.392+0000 7f0245276700 1 -- 172.21.15.104:0/3585045218 shutdown_connections 2024-06-09T22:20:48.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.392+0000 7f0245276700 1 --2- 172.21.15.104:0/3585045218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0240102740 0x7f0240102b20 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.392+0000 7f0245276700 1 --2- 172.21.15.104:0/3585045218 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0240107840 0x7f024010f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.392+0000 7f0245276700 1 --2- 172.21.15.104:0/3585045218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02401030f0 0x7f0240107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.392+0000 7f0245276700 1 -- 172.21.15.104:0/3585045218 >> 172.21.15.104:0/3585045218 conn(0x7f02400795f0 msgr2=0x7f0240079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:48.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.392+0000 7f0245276700 1 -- 172.21.15.104:0/3585045218 shutdown_connections 2024-06-09T22:20:48.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.392+0000 7f0245276700 1 -- 172.21.15.104:0/3585045218 wait complete. 2024-06-09T22:20:48.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.393+0000 7f0245276700 1 Processor -- start 2024-06-09T22:20:48.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.393+0000 7f0245276700 1 -- start start 2024-06-09T22:20:48.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.393+0000 7f0245276700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0240102740 0x7f024019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.393+0000 7f0245276700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02401030f0 0x7f024019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f0245276700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0240107840 0x7f02401989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f0245276700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0240112760 con 0x7f0240107840 2024-06-09T22:20:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f0245276700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02401125e0 con 0x7f02401030f0 2024-06-09T22:20:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f0245276700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f02401128e0 con 0x7f0240102740 2024-06-09T22:20:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f023effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0240102740 0x7f024019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f023f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0240107840 0x7f02401989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f023effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0240102740 0x7f024019e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:41190/0 (socket says 172.21.15.104:41190) 2024-06-09T22:20:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f023effd700 1 -- 172.21.15.104:0/3739662127 learned_addr learned my addr 172.21.15.104:0/3739662127 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f023e7fc700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02401030f0 0x7f024019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f023f7fe700 1 -- 172.21.15.104:0/3739662127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0240102740 msgr2=0x7f024019e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f023f7fe700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0240102740 0x7f024019e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f023f7fe700 1 -- 172.21.15.104:0/3739662127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02401030f0 msgr2=0x7f024019ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:48.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f023f7fe700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02401030f0 0x7f024019ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f023f7fe700 1 -- 172.21.15.104:0/3739662127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0240199140 con 0x7f0240107840 2024-06-09T22:20:48.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f023f7fe700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0240107840 0x7f02401989e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f023800ab40 tx=0x7f023800ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:48.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f0227fff700 1 -- 172.21.15.104:0/3739662127 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238017020 con 0x7f0240107840 2024-06-09T22:20:48.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f0245276700 1 -- 172.21.15.104:0/3739662127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0240199430 con 0x7f0240107840 2024-06-09T22:20:48.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f0227fff700 1 -- 172.21.15.104:0/3739662127 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0238004bb0 con 0x7f0240107840 2024-06-09T22:20:48.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f0227fff700 1 -- 172.21.15.104:0/3739662127 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238005260 con 0x7f0240107840 2024-06-09T22:20:48.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.394+0000 7f0245276700 1 -- 172.21.15.104:0/3739662127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02401a56f0 con 0x7f0240107840 2024-06-09T22:20:48.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.397+0000 7f0227fff700 1 -- 172.21.15.104:0/3739662127 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f023800c580 con 0x7f0240107840 2024-06-09T22:20:48.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.398+0000 7f0227fff700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f02280a2c20 0x7f02280a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.398+0000 7f023effd700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f02280a2c20 0x7f02280a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.398+0000 7f0227fff700 1 -- 172.21.15.104:0/3739662127 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f02380ce9f0 con 0x7f0240107840 2024-06-09T22:20:48.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.398+0000 7f023effd700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f02280a2c20 0x7f02280a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0230009990 tx=0x7f0230005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:48.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.399+0000 7f024faf9b80 1 Processor -- start 2024-06-09T22:20:48.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.399+0000 7f024faf9b80 1 -- start start 2024-06-09T22:20:48.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.399+0000 7f024faf9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b947f80 0x55fa3b948360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.399+0000 7f024faf9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b9488a0 0x55fa3b94c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.399+0000 7f024faf9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b94d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.399+0000 7f024faf9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fa3b8957c0 con 0x55fa3b94cbe0 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.399+0000 7f024faf9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fa3b83d6d0 con 0x55fa3b947f80 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.399+0000 7f024faf9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fa3b891780 con 0x55fa3b9488a0 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f023f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b94d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f023f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b94d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54670/0 (socket says 172.21.15.104:54670) 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f023f7fe700 1 -- 172.21.15.104:0/2446522797 learned_addr learned my addr 172.21.15.104:0/2446522797 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f023effd700 1 --2- 172.21.15.104:0/2446522797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b947f80 0x55fa3b948360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f023e7fc700 1 --2- 172.21.15.104:0/2446522797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b9488a0 0x55fa3b94c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f023f7fe700 1 -- 172.21.15.104:0/2446522797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b9488a0 msgr2=0x55fa3b94c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f023f7fe700 1 --2- 172.21.15.104:0/2446522797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b9488a0 0x55fa3b94c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f023f7fe700 1 -- 172.21.15.104:0/2446522797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b947f80 msgr2=0x55fa3b948360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:48.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f023f7fe700 1 --2- 172.21.15.104:0/2446522797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b947f80 0x55fa3b948360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f023f7fe700 1 -- 172.21.15.104:0/2446522797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa3b833f00 con 0x55fa3b94cbe0 2024-06-09T22:20:48.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f023f7fe700 1 --2- 172.21.15.104:0/2446522797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b94d040 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0238097350 tx=0x7f023800a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d20552ad6effbbc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:48.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f02257fa700 1 -- 172.21.15.104:0/2446522797 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023809f2a0 con 0x55fa3b94cbe0 2024-06-09T22:20:48.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f02257fa700 1 -- 172.21.15.104:0/2446522797 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02380a3ad0 con 0x55fa3b94cbe0 2024-06-09T22:20:48.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.400+0000 7f02257fa700 1 -- 172.21.15.104:0/2446522797 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023809b3b0 con 0x55fa3b94cbe0 2024-06-09T22:20:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.401+0000 7f024faf9b80 1 -- 172.21.15.104:0/2446522797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 msgr2=0x55fa3b94d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.401+0000 7f024faf9b80 1 --2- 172.21.15.104:0/2446522797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b94d040 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0238097350 tx=0x7f023800a2e0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.401+0000 7f024faf9b80 1 -- 172.21.15.104:0/2446522797 shutdown_connections 2024-06-09T22:20:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.401+0000 7f024faf9b80 1 --2- 172.21.15.104:0/2446522797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b94d040 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.401+0000 7f024faf9b80 1 --2- 172.21.15.104:0/2446522797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b947f80 0x55fa3b948360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.401+0000 7f024faf9b80 1 --2- 172.21.15.104:0/2446522797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b9488a0 0x55fa3b94c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.401+0000 7f024faf9b80 1 -- 172.21.15.104:0/2446522797 >> 172.21.15.104:0/2446522797 conn(0x55fa3b86ec70 msgr2=0x55fa3b93e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.401+0000 7f024faf9b80 1 -- 172.21.15.104:0/2446522797 shutdown_connections 2024-06-09T22:20:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.401+0000 7f024faf9b80 1 -- 172.21.15.104:0/2446522797 wait complete. 2024-06-09T22:20:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f024faf9b80 1 -- start start 2024-06-09T22:20:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f024faf9b80 1 Processor -- start 2024-06-09T22:20:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f024faf9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b947f80 0x55fa3b954120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f024faf9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b9488a0 0x55fa3b951490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f024faf9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b9519d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f024faf9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fa3b954d80 con 0x55fa3b94cbe0 2024-06-09T22:20:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f024faf9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fa3b835ba0 con 0x55fa3b9488a0 2024-06-09T22:20:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f024faf9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fa3b954f00 con 0x55fa3b947f80 2024-06-09T22:20:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f024faf9b80 10 client.0 _ll_register_callbacks cb 0x55fa3b82f8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f023f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b9519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f023f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b9519d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54678/0 (socket says 172.21.15.104:54678) 2024-06-09T22:20:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f023f7fe700 1 -- 172.21.15.104:0/3808494029 learned_addr learned my addr 172.21.15.104:0/3808494029 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f023e7fc700 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b9488a0 0x55fa3b951490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.402+0000 7f023effd700 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b947f80 0x55fa3b954120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f023f7fe700 1 -- 172.21.15.104:0/3808494029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b947f80 msgr2=0x55fa3b954120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f023f7fe700 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b947f80 0x55fa3b954120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f023f7fe700 1 -- 172.21.15.104:0/3808494029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b9488a0 msgr2=0x55fa3b951490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f023f7fe700 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b9488a0 0x55fa3b951490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f023f7fe700 1 -- 172.21.15.104:0/3808494029 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa3b7e4880 con 0x55fa3b94cbe0 2024-06-09T22:20:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f023e7fc700 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b9488a0 0x55fa3b951490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f023f7fe700 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b9519d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f023800a0a0 tx=0x7f023800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a79e6bb084ebfdd server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f0222ff5700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f0222ff5700 1 -- 172.21.15.104:0/3808494029 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023809f2a0 con 0x55fa3b94cbe0 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f0222ff5700 1 -- 172.21.15.104:0/3808494029 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f023809b150 con 0x55fa3b94cbe0 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f024faf9b80 1 -- 172.21.15.104:0/3808494029 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02380a3ad0 con 0x55fa3b94cbe0 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f0222ff5700 1 -- 172.21.15.104:0/3808494029 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0238081050 con 0x55fa3b94cbe0 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f024faf9b80 10 client.15252 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.403+0000 7f024faf9b80 1 -- 172.21.15.104:0/3808494029 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fa3b95f760 con 0x55fa3b94cbe0 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.404+0000 7f0222ff5700 1 -- 172.21.15.104:0/3808494029 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02380a3ad0 con 0x55fa3b94cbe0 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.404+0000 7f0222ff5700 1 -- 172.21.15.104:0/3808494029 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0238012070 con 0x55fa3b94cbe0 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.404+0000 7f024faf9b80 10 client.15252 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.404+0000 7f024faf9b80 10 client.15252 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.404+0000 7f024faf9b80 20 client.15252 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.404+0000 7f024faf9b80 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fa3b961930 0x55fa3b962da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.404+0000 7f024faf9b80 4 client.15252 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.404+0000 7f024faf9b80 1 -- 172.21.15.104:0/3808494029 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa3b8a2cc0 con 0x55fa3b961930 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.404+0000 7f023effd700 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fa3b961930 0x55fa3b962da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.405+0000 7f0222ff5700 10 client.15252 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.405+0000 7f023effd700 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fa3b961930 0x55fa3b962da0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ede290f6b3afd0e6 server_cookie=e5ad354a065a44d in_seq=0 out_seq=0 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.405+0000 7f0222ff5700 1 -- 172.21.15.104:0/3808494029 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55fa3b8a2cc0 con 0x55fa3b961930 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.405+0000 7f0222ff5700 10 client.15252 handle_command_reply: tid=0 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.405+0000 7f024faf9b80 1 client.15252 shutdown 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.406+0000 7f024faf9b80 1 -- 172.21.15.104:0/3808494029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 msgr2=0x55fa3b9519d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.406+0000 7f024faf9b80 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b9519d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f023800a0a0 tx=0x7f023800cbe0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.406+0000 7f024faf9b80 1 -- 172.21.15.104:0/3808494029 shutdown_connections 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.406+0000 7f024faf9b80 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fa3b961930 0x55fa3b962da0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.406+0000 7f024faf9b80 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b9519d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.406+0000 7f024faf9b80 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b947f80 0x55fa3b954120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.406+0000 7f024faf9b80 1 --2- 172.21.15.104:0/3808494029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b9488a0 0x55fa3b951490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.406+0000 7f024faf9b80 1 -- 172.21.15.104:0/3808494029 >> 172.21.15.104:0/3808494029 conn(0x55fa3b86ec70 msgr2=0x55fa3b942980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.406+0000 7f0222ff5700 0 client.15252 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:20:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.406+0000 7f024faf9b80 1 -- 172.21.15.104:0/3808494029 shutdown_connections 2024-06-09T22:20:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.406+0000 7f024faf9b80 1 -- 172.21.15.104:0/3808494029 wait complete. 2024-06-09T22:20:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.406+0000 7f024faf9b80 20 client.15252 trim_cache size 0 max 16384 2024-06-09T22:20:48.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.427+0000 7f024faf9b80 1 Processor -- start 2024-06-09T22:20:48.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.427+0000 7f024faf9b80 1 -- start start 2024-06-09T22:20:48.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f024faf9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b947f80 0x55fa3b9432d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f024faf9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b9488a0 0x55fa3b951a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f024faf9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b937db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f024faf9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fa3b9610e0 con 0x55fa3b94cbe0 2024-06-09T22:20:48.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f024faf9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fa3b9617b0 con 0x55fa3b9488a0 2024-06-09T22:20:48.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f024faf9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fa3b961e80 con 0x55fa3b947f80 2024-06-09T22:20:48.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f023f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b937db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f023f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b937db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54682/0 (socket says 172.21.15.104:54682) 2024-06-09T22:20:48.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f023f7fe700 1 -- 172.21.15.104:0/3277758814 learned_addr learned my addr 172.21.15.104:0/3277758814 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:48.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f023effd700 1 --2- 172.21.15.104:0/3277758814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b947f80 0x55fa3b9432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f023f7fe700 1 -- 172.21.15.104:0/3277758814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b947f80 msgr2=0x55fa3b9432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:48.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f023f7fe700 1 --2- 172.21.15.104:0/3277758814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b947f80 0x55fa3b9432d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f023e7fc700 1 --2- 172.21.15.104:0/3277758814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b9488a0 0x55fa3b951a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f023f7fe700 1 -- 172.21.15.104:0/3277758814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b9488a0 msgr2=0x55fa3b951a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:48.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f023f7fe700 1 --2- 172.21.15.104:0/3277758814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b9488a0 0x55fa3b951a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f023f7fe700 1 -- 172.21.15.104:0/3277758814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa3b962370 con 0x55fa3b94cbe0 2024-06-09T22:20:48.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f023f7fe700 1 --2- 172.21.15.104:0/3277758814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b937db0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0238097350 tx=0x7f0238084390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade902e781da0cff server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:48.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f0224ff9700 1 -- 172.21.15.104:0/3277758814 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023809f2a0 con 0x55fa3b94cbe0 2024-06-09T22:20:48.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f0224ff9700 1 -- 172.21.15.104:0/3277758814 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02380a3ad0 con 0x55fa3b94cbe0 2024-06-09T22:20:48.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.428+0000 7f0224ff9700 1 -- 172.21.15.104:0/3277758814 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023809b3b0 con 0x55fa3b94cbe0 2024-06-09T22:20:48.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.429+0000 7f024faf9b80 1 -- 172.21.15.104:0/3277758814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 msgr2=0x55fa3b937db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:48.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.429+0000 7f024faf9b80 1 --2- 172.21.15.104:0/3277758814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b937db0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0238097350 tx=0x7f0238084390 comp rx=0 tx=0).stop 2024-06-09T22:20:48.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.429+0000 7f024faf9b80 1 -- 172.21.15.104:0/3277758814 shutdown_connections 2024-06-09T22:20:48.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.429+0000 7f024faf9b80 1 --2- 172.21.15.104:0/3277758814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b94cbe0 0x55fa3b937db0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.429+0000 7f024faf9b80 1 --2- 172.21.15.104:0/3277758814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b947f80 0x55fa3b9432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.429+0000 7f024faf9b80 1 --2- 172.21.15.104:0/3277758814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b9488a0 0x55fa3b951a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.429+0000 7f024faf9b80 1 -- 172.21.15.104:0/3277758814 >> 172.21.15.104:0/3277758814 conn(0x55fa3b86ec70 msgr2=0x55fa3b94fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:48.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.429+0000 7f024faf9b80 1 -- 172.21.15.104:0/3277758814 shutdown_connections 2024-06-09T22:20:48.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.429+0000 7f024faf9b80 1 -- 172.21.15.104:0/3277758814 wait complete. 2024-06-09T22:20:48.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.429+0000 7f024faf9b80 1 -- start start 2024-06-09T22:20:48.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.429+0000 7f024faf9b80 1 Processor -- start 2024-06-09T22:20:48.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f024faf9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b947f80 0x55fa3b9509f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f024faf9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b9488a0 0x55fa3b950f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f024faf9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b94cbe0 0x55fa3b951490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f024faf9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fa3b962c20 con 0x55fa3b9488a0 2024-06-09T22:20:48.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f024faf9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fa3b962550 con 0x55fa3b947f80 2024-06-09T22:20:48.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f024faf9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fa3b9632f0 con 0x55fa3b94cbe0 2024-06-09T22:20:48.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f024faf9b80 10 client.0 _ll_register_callbacks cb 0x55fa3b82f8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:20:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f023e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b9488a0 0x55fa3b950f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f023e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b9488a0 0x55fa3b950f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54696/0 (socket says 172.21.15.104:54696) 2024-06-09T22:20:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f023e7fc700 1 -- 172.21.15.104:0/946702560 learned_addr learned my addr 172.21.15.104:0/946702560 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f023f7fe700 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b94cbe0 0x55fa3b951490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f023effd700 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b947f80 0x55fa3b9509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f023e7fc700 1 -- 172.21.15.104:0/946702560 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b94cbe0 msgr2=0x55fa3b951490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f023e7fc700 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b94cbe0 0x55fa3b951490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f023e7fc700 1 -- 172.21.15.104:0/946702560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b947f80 msgr2=0x55fa3b9509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:20:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f023e7fc700 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b947f80 0x55fa3b9509f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f023e7fc700 1 -- 172.21.15.104:0/946702560 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa3b96b510 con 0x55fa3b9488a0 2024-06-09T22:20:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f023effd700 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b947f80 0x55fa3b9509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:20:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f023e7fc700 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b9488a0 0x55fa3b950f30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0240102020 tx=0x7f0234003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95fdc0a8b4f29140 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f0222ff5700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:20:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f0222ff5700 1 -- 172.21.15.104:0/946702560 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0234003e40 con 0x55fa3b9488a0 2024-06-09T22:20:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f024faf9b80 1 -- 172.21.15.104:0/946702560 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02380a3ad0 con 0x55fa3b9488a0 2024-06-09T22:20:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.430+0000 7f0222ff5700 1 -- 172.21.15.104:0/946702560 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f0234004830 con 0x55fa3b9488a0 2024-06-09T22:20:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.431+0000 7f0222ff5700 1 -- 172.21.15.104:0/946702560 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0234026050 con 0x55fa3b9488a0 2024-06-09T22:20:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.431+0000 7f024faf9b80 10 client.15258 fetch_fsmap learned FSMap version 28 2024-06-09T22:20:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.431+0000 7f024faf9b80 1 -- 172.21.15.104:0/946702560 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fa3b96c8a0 con 0x55fa3b9488a0 2024-06-09T22:20:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.431+0000 7f0222ff5700 1 -- 172.21.15.104:0/946702560 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02380a3ad0 con 0x55fa3b9488a0 2024-06-09T22:20:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.431+0000 7f0222ff5700 1 -- 172.21.15.104:0/946702560 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f023401d920 con 0x55fa3b9488a0 2024-06-09T22:20:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.431+0000 7f024faf9b80 10 client.15258 fetch_fsmap finished waiting for FSMap version 28 2024-06-09T22:20:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.431+0000 7f024faf9b80 10 client.15258 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.431+0000 7f024faf9b80 20 client.15258 populate_metadata read hostname 'smithi104' 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.431+0000 7f024faf9b80 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fa3b9d15d0 0x55fa3b9d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.431+0000 7f024faf9b80 4 client.15258 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.431+0000 7f024faf9b80 1 -- 172.21.15.104:0/946702560 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "status"}) -- 0x55fa3b89d240 con 0x55fa3b9d15d0 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.431+0000 7f023effd700 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fa3b9d15d0 0x55fa3b9d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f023effd700 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fa3b9d15d0 0x55fa3b9d02e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f27629c88d8d784c server_cookie=6d95f6c5eaa25fe3 in_seq=0 out_seq=0 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f0222ff5700 10 client.15258 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f0222ff5700 1 -- 172.21.15.104:0/946702560 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fa3b89d240 con 0x55fa3b9d15d0 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f0222ff5700 10 client.15258 handle_command_reply: tid=0 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f024faf9b80 1 client.15258 shutdown 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f024faf9b80 1 -- 172.21.15.104:0/946702560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b9488a0 msgr2=0x55fa3b950f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f024faf9b80 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b9488a0 0x55fa3b950f30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0240102020 tx=0x7f0234003960 comp rx=0 tx=0).stop 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f024faf9b80 1 -- 172.21.15.104:0/946702560 shutdown_connections 2024-06-09T22:20:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f024faf9b80 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fa3b9d15d0 0x55fa3b9d02e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f024faf9b80 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fa3b94cbe0 0x55fa3b951490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f0222ff5700 0 client.15258 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:20:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f024faf9b80 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa3b947f80 0x55fa3b9509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f024faf9b80 1 --2- 172.21.15.104:0/946702560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fa3b9488a0 0x55fa3b950f30 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f024faf9b80 1 -- 172.21.15.104:0/946702560 >> 172.21.15.104:0/946702560 conn(0x55fa3b86ec70 msgr2=0x55fa3b94fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f024faf9b80 1 -- 172.21.15.104:0/946702560 shutdown_connections 2024-06-09T22:20:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f024faf9b80 1 -- 172.21.15.104:0/946702560 wait complete. 2024-06-09T22:20:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.432+0000 7f024faf9b80 20 client.15258 trim_cache size 0 max 16384 2024-06-09T22:20:48.438 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:20:48.438 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:20:48.438 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2024-06-09T22:20:48.439 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24460, 2024-06-09T22:20:48.439 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:20:48.439 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:20:48.439 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:20:48.439 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 243.23125627600001, 2024-06-09T22:20:48.439 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 28, 2024-06-09T22:20:48.439 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 99, 2024-06-09T22:20:48.439 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 97, 2024-06-09T22:20:48.439 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 268.75229806599998 2024-06-09T22:20:48.439 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:20:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.433+0000 7f0245276700 1 -- 172.21.15.104:0/3739662127 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f02280a2c20 msgr2=0x7f02280a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.433+0000 7f0245276700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f02280a2c20 0x7f02280a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0230009990 tx=0x7f0230005960 comp rx=0 tx=0).stop 2024-06-09T22:20:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.433+0000 7f0245276700 1 -- 172.21.15.104:0/3739662127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0240107840 msgr2=0x7f02401989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.433+0000 7f0245276700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0240107840 0x7f02401989e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f023800ab40 tx=0x7f023800ae50 comp rx=0 tx=0).stop 2024-06-09T22:20:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.433+0000 7f0245276700 1 -- 172.21.15.104:0/3739662127 shutdown_connections 2024-06-09T22:20:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.433+0000 7f0245276700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f02280a2c20 0x7f02280a50e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.433+0000 7f0245276700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0240107840 0x7f02401989e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.433+0000 7f0245276700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0240102740 0x7f024019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.433+0000 7f0245276700 1 --2- 172.21.15.104:0/3739662127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02401030f0 0x7f024019ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.433+0000 7f0245276700 1 -- 172.21.15.104:0/3739662127 >> 172.21.15.104:0/3739662127 conn(0x7f02400795f0 msgr2=0x7f0240100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.433+0000 7f0245276700 1 -- 172.21.15.104:0/3739662127 shutdown_connections 2024-06-09T22:20:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:48.433+0000 7f0245276700 1 -- 172.21.15.104:0/3739662127 wait complete. 2024-06-09T22:20:48.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:48 smithi153 ceph-mon[36599]: pgmap v430: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:48.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:48 smithi104 ceph-mon[32389]: pgmap v430: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:48.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:48 smithi194 ceph-mon[36856]: pgmap v430: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-06-09T22:20:49.137 INFO:teuthology.run_tasks:Running task sequential... 2024-06-09T22:20:49.147 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-06-09T22:20:49.147 INFO:teuthology.task.sequential:In sequential, running task print... 2024-06-09T22:20:49.147 INFO:teuthology.task.print:Enabling mgr modules 2024-06-09T22:20:49.147 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-06-09T22:20:49.148 INFO:teuthology.task.exec:Executing custom commands... 2024-06-09T22:20:49.148 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi104.front.sepia.ceph.com 2024-06-09T22:20:49.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-06-09T22:20:49.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.254+0000 7fb8787a9640 1 -- 172.21.15.104:0/514005199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb870101f20 msgr2=0x7fb870102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:49.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.254+0000 7fb8787a9640 1 --2- 172.21.15.104:0/514005199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb870101f20 0x7fb870102340 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009900 tx=0x7fb86c02a410 comp rx=0 tx=0).stop 2024-06-09T22:20:49.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.254+0000 7fb8787a9640 1 -- 172.21.15.104:0/514005199 shutdown_connections 2024-06-09T22:20:49.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.254+0000 7fb8787a9640 1 --2- 172.21.15.104:0/514005199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb870101530 0x7fb870108dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:49.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.254+0000 7fb8787a9640 1 --2- 172.21.15.104:0/514005199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb870100b70 0x7fb870100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:49.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.254+0000 7fb8787a9640 1 --2- 172.21.15.104:0/514005199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb870101f20 0x7fb870102340 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:49.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.254+0000 7fb8787a9640 1 -- 172.21.15.104:0/514005199 >> 172.21.15.104:0/514005199 conn(0x7fb8700fb600 msgr2=0x7fb8700fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:49.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.254+0000 7fb8787a9640 1 -- 172.21.15.104:0/514005199 shutdown_connections 2024-06-09T22:20:49.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.254+0000 7fb8787a9640 1 -- 172.21.15.104:0/514005199 wait complete. 2024-06-09T22:20:49.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb8787a9640 1 Processor -- start 2024-06-09T22:20:49.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb8787a9640 1 -- start start 2024-06-09T22:20:49.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb8787a9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb870100b70 0x7fb870196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:49.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb8787a9640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb870101530 0x7fb870197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:49.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb8787a9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb870101f20 0x7fb87019e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:49.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb8787a9640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb87010c5a0 con 0x7fb870101f20 2024-06-09T22:20:49.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb8787a9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb87010c420 con 0x7fb870100b70 2024-06-09T22:20:49.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb8787a9640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb87010c720 con 0x7fb870101530 2024-06-09T22:20:49.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb876d46640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb870101f20 0x7fb87019e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb876d46640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb870101f20 0x7fb87019e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54716/0 (socket says 172.21.15.104:54716) 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb876d46640 1 -- 172.21.15.104:0/2672998347 learned_addr learned my addr 172.21.15.104:0/2672998347 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb875d44640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb870101530 0x7fb870197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb876545640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb870100b70 0x7fb870196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb876d46640 1 -- 172.21.15.104:0/2672998347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb870101530 msgr2=0x7fb870197100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb876d46640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb870101530 0x7fb870197100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb876d46640 1 -- 172.21.15.104:0/2672998347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb870100b70 msgr2=0x7fb870196bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb876d46640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb870100b70 0x7fb870196bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.255+0000 7fb876d46640 1 -- 172.21.15.104:0/2672998347 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87019e7e0 con 0x7fb870101f20 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.256+0000 7fb875d44640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb870101530 0x7fb870197100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.256+0000 7fb876545640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb870100b70 0x7fb870196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.256+0000 7fb876d46640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb870101f20 0x7fb87019e170 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb86800eaa0 tx=0x7fb86800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:49.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.256+0000 7fb8677fe640 1 -- 172.21.15.104:0/2672998347 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86800cdb0 con 0x7fb870101f20 2024-06-09T22:20:49.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.256+0000 7fb8677fe640 1 -- 172.21.15.104:0/2672998347 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb868005020 con 0x7fb870101f20 2024-06-09T22:20:49.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.256+0000 7fb8787a9640 1 -- 172.21.15.104:0/2672998347 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87019ead0 con 0x7fb870101f20 2024-06-09T22:20:49.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.256+0000 7fb8677fe640 1 -- 172.21.15.104:0/2672998347 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8680116b0 con 0x7fb870101f20 2024-06-09T22:20:49.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.256+0000 7fb8787a9640 1 -- 172.21.15.104:0/2672998347 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87019efd0 con 0x7fb870101f20 2024-06-09T22:20:49.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.257+0000 7fb8787a9640 1 -- 172.21.15.104:0/2672998347 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb870107c50 con 0x7fb870101f20 2024-06-09T22:20:49.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.258+0000 7fb8677fe640 1 -- 172.21.15.104:0/2672998347 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb868005560 con 0x7fb870101f20 2024-06-09T22:20:49.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.259+0000 7fb8677fe640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb85c0a2c30 0x7fb85c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:49.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.259+0000 7fb876545640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb85c0a2c30 0x7fb85c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:49.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.259+0000 7fb8677fe640 1 -- 172.21.15.104:0/2672998347 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fb8680cfac0 con 0x7fb870101f20 2024-06-09T22:20:49.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.259+0000 7fb876545640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb85c0a2c30 0x7fb85c0a50f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb86c000c00 tx=0x7fb86c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:49.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.262+0000 7fb8677fe640 1 -- 172.21.15.104:0/2672998347 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb868016030 con 0x7fb870101f20 2024-06-09T22:20:49.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:49.489+0000 7fb8787a9640 1 -- 172.21.15.104:0/2672998347 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fb87002d530 con 0x7fb870101f20 2024-06-09T22:20:50.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.484+0000 7fb8677fe640 1 -- 172.21.15.104:0/2672998347 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fb86809c690 con 0x7fb870101f20 2024-06-09T22:20:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.487+0000 7fb8787a9640 1 -- 172.21.15.104:0/2672998347 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb85c0a2c30 msgr2=0x7fb85c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.487+0000 7fb8787a9640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb85c0a2c30 0x7fb85c0a50f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb86c000c00 tx=0x7fb86c031400 comp rx=0 tx=0).stop 2024-06-09T22:20:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.487+0000 7fb8787a9640 1 -- 172.21.15.104:0/2672998347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb870101f20 msgr2=0x7fb87019e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.487+0000 7fb8787a9640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb870101f20 0x7fb87019e170 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb86800eaa0 tx=0x7fb86800ef60 comp rx=0 tx=0).stop 2024-06-09T22:20:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.487+0000 7fb8787a9640 1 -- 172.21.15.104:0/2672998347 shutdown_connections 2024-06-09T22:20:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.487+0000 7fb8787a9640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fb85c0a2c30 0x7fb85c0a50f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.487+0000 7fb8787a9640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb870101f20 0x7fb87019e170 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.487+0000 7fb8787a9640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb870101530 0x7fb870197100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:50.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.487+0000 7fb8787a9640 1 --2- 172.21.15.104:0/2672998347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb870100b70 0x7fb870196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:50.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.487+0000 7fb8787a9640 1 -- 172.21.15.104:0/2672998347 >> 172.21.15.104:0/2672998347 conn(0x7fb8700fb600 msgr2=0x7fb870104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:50.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.487+0000 7fb8787a9640 1 -- 172.21.15.104:0/2672998347 shutdown_connections 2024-06-09T22:20:50.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:50.487+0000 7fb8787a9640 1 -- 172.21.15.104:0/2672998347 wait complete. 2024-06-09T22:20:50.507 INFO:teuthology.run_tasks:Running task kclient... 2024-06-09T22:20:50.521 INFO:tasks.kclient:Mounting kernel clients... 2024-06-09T22:20:50.521 INFO:tasks.kclient:config is {'client.0': {}} 2024-06-09T22:20:50.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:20:50.522 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2024-06-09T22:20:50.538 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-06-09T22:20:50.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:20:50.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-09T22:20:50.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:20:50.604 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link delete ceph-brx 2024-06-09T22:20:50.708 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-06-09T22:20:50.708 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-09T22:20:50.708 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-09T22:20:50.709 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-06-09T22:20:50.732 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:50 smithi104 ceph-mon[32389]: pgmap v431: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:50.732 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:50 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2672998347' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-06-09T22:20:50.732 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: ignoring --setuser ceph since I am not root 2024-06-09T22:20:50.732 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: ignoring --setgroup ceph since I am not root 2024-06-09T22:20:50.733 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:50.680+0000 7f2df01bf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-09T22:20:50.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:50 smithi153 ceph-mon[36599]: pgmap v431: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:50.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:50 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2672998347' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-06-09T22:20:50.741 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: ignoring --setuser ceph since I am not root 2024-06-09T22:20:50.742 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: ignoring --setgroup ceph since I am not root 2024-06-09T22:20:50.742 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:50 smithi153 ceph-mgr[38024]: -- 172.21.15.153:0/1822944810 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b023f225a0 con 0x55b023ef6800 2024-06-09T22:20:50.742 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:50.691+0000 7fae33814200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-09T22:20:50.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:50 smithi194 ceph-mon[36856]: pgmap v431: 129 pgs: 129 active+clean; 651 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-06-09T22:20:50.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:50 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2672998347' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-06-09T22:20:50.808 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: ignoring --setuser ceph since I am not root 2024-06-09T22:20:50.808 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: ignoring --setgroup ceph since I am not root 2024-06-09T22:20:50.808 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:50 smithi194 ceph-mgr[38651]: -- 172.21.15.194:0/1322972803 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a8a7ddc5a0 con 0x55a8a7db0800 2024-06-09T22:20:50.808 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:50.669+0000 7f8227133200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-09T22:20:50.959 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:20:50.991 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:50.873+0000 7f2df01bf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:20:50.991 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:50.992+0000 7f2df01bf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-09T22:20:51.241 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:50.878+0000 7fae33814200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:20:51.241 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:50.982+0000 7fae33814200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-09T22:20:51.267 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:50.853+0000 7f8227133200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:20:51.267 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:50.955+0000 7f8227133200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-09T22:20:51.267 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:51 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:51.269+0000 7f8227133200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-09T22:20:51.505 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:51 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:51.314+0000 7f2df01bf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-09T22:20:51.505 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:51 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:51.401+0000 7f2df01bf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-09T22:20:51.505 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:51 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2672998347' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-06-09T22:20:51.505 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:51 smithi104 ceph-mon[32389]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-06-09T22:20:51.557 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:51 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2672998347' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-06-09T22:20:51.557 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:51 smithi194 ceph-mon[36856]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-06-09T22:20:51.558 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:51 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:51.355+0000 7f8227133200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-09T22:20:51.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:51 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2672998347' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-06-09T22:20:51.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:51 smithi153 ceph-mon[36599]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-06-09T22:20:51.742 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:51 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:51.297+0000 7fae33814200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-09T22:20:51.742 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:51 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:51.384+0000 7fae33814200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-09T22:20:52.756 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:52.618+0000 7f2df01bf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-09T22:20:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.779+0000 7f7f56411700 1 -- 172.21.15.104:0/3570222466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f501085f0 msgr2=0x7f7f501089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.779+0000 7f7f56411700 1 --2- 172.21.15.104:0/3570222466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f501085f0 0x7f7f501089d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7f40009990 tx=0x7f7f40009ca0 comp rx=0 tx=0).stop 2024-06-09T22:20:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.779+0000 7f7f56411700 1 -- 172.21.15.104:0/3570222466 shutdown_connections 2024-06-09T22:20:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.779+0000 7f7f56411700 1 --2- 172.21.15.104:0/3570222466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f501085f0 0x7f7f501089d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.779+0000 7f7f56411700 1 --2- 172.21.15.104:0/3570222466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f50102ea0 0x7f7f5010f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.779+0000 7f7f56411700 1 --2- 172.21.15.104:0/3570222466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f501024e0 0x7f7f50102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.779+0000 7f7f56411700 1 -- 172.21.15.104:0/3570222466 >> 172.21.15.104:0/3570222466 conn(0x7f7f50079640 msgr2=0x7f7f50079a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.779+0000 7f7f56411700 1 -- 172.21.15.104:0/3570222466 shutdown_connections 2024-06-09T22:20:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.779+0000 7f7f56411700 1 -- 172.21.15.104:0/3570222466 wait complete. 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.779+0000 7f7f56411700 1 Processor -- start 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.779+0000 7f7f56411700 1 -- start start 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f56411700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f501024e0 0x7f7f5019e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f56411700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f50102ea0 0x7f7f5019ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f56411700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f501085f0 0x7f7f501989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f56411700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7f501127b0 con 0x7f7f501085f0 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f56411700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f50112630 con 0x7f7f501024e0 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f56411700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f50112930 con 0x7f7f50102ea0 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f549ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f501085f0 0x7f7f501989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f4f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f50102ea0 0x7f7f5019ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f549ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f501085f0 0x7f7f501989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54766/0 (socket says 172.21.15.104:54766) 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f4f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f50102ea0 0x7f7f5019ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:41260/0 (socket says 172.21.15.104:41260) 2024-06-09T22:20:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f549ae700 1 -- 172.21.15.104:0/469580413 learned_addr learned my addr 172.21.15.104:0/469580413 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f4ffff700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f501024e0 0x7f7f5019e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f549ae700 1 -- 172.21.15.104:0/469580413 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f50102ea0 msgr2=0x7f7f5019ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f549ae700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f50102ea0 0x7f7f5019ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f549ae700 1 -- 172.21.15.104:0/469580413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f501024e0 msgr2=0x7f7f5019e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f549ae700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f501024e0 0x7f7f5019e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f549ae700 1 -- 172.21.15.104:0/469580413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f50199150 con 0x7f7f501085f0 2024-06-09T22:20:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f4f7fe700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f50102ea0 0x7f7f5019ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:20:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.780+0000 7f7f549ae700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f501085f0 0x7f7f501989f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800efc0 tx=0x7f7f4800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.781+0000 7f7f4d7fa700 1 -- 172.21.15.104:0/469580413 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f480092e0 con 0x7f7f501085f0 2024-06-09T22:20:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.781+0000 7f7f56411700 1 -- 172.21.15.104:0/469580413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f501993e0 con 0x7f7f501085f0 2024-06-09T22:20:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.781+0000 7f7f4d7fa700 1 -- 172.21.15.104:0/469580413 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f48004bb0 con 0x7f7f501085f0 2024-06-09T22:20:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.781+0000 7f7f4d7fa700 1 -- 172.21.15.104:0/469580413 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48004020 con 0x7f7f501085f0 2024-06-09T22:20:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.781+0000 7f7f56411700 1 -- 172.21.15.104:0/469580413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f501a56b0 con 0x7f7f501085f0 2024-06-09T22:20:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.782+0000 7f7f56411700 1 -- 172.21.15.104:0/469580413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f50069180 con 0x7f7f501085f0 2024-06-09T22:20:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.783+0000 7f7f4d7fa700 1 -- 172.21.15.104:0/469580413 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f7f4801b030 con 0x7f7f501085f0 2024-06-09T22:20:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.784+0000 7f7f4d7fa700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f380a2c70 0x7f7f380a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.784+0000 7f7f4ffff700 1 -- 172.21.15.104:0/469580413 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f380a2c70 msgr2=0x7f7f380a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1906048760 2024-06-09T22:20:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.784+0000 7f7f4d7fa700 1 -- 172.21.15.104:0/469580413 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7f7f480cf7e0 con 0x7f7f501085f0 2024-06-09T22:20:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.784+0000 7f7f4ffff700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f380a2c70 0x7f7f380a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:20:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.788+0000 7f7f4d7fa700 1 -- 172.21.15.104:0/469580413 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7f48022020 con 0x7f7f501085f0 2024-06-09T22:20:52.807 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:52 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:52.544+0000 7f8227133200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-09T22:20:52.807 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:52 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:52.714+0000 7f8227133200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-09T22:20:52.917 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:52 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:52.575+0000 7fae33814200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-09T22:20:52.917 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:52 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:52.747+0000 7fae33814200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-09T22:20:52.918 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:52 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:52.919+0000 7fae33814200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-09T22:20:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.985+0000 7f7f4ffff700 1 -- 172.21.15.104:0/469580413 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f380a2c70 msgr2=0x7f7f380a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1906048760 2024-06-09T22:20:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:52.985+0000 7f7f4ffff700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f380a2c70 0x7f7f380a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:20:53.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.029+0000 7f7f56411700 1 -- 172.21.15.104:0/469580413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7f5002dca0 con 0x7f7f501085f0 2024-06-09T22:20:53.029 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:52.797+0000 7f2df01bf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-09T22:20:53.029 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:52.968+0000 7f2df01bf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-09T22:20:53.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.030+0000 7f7f4d7fa700 1 -- 172.21.15.104:0/469580413 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7f7f48010040 con 0x7f7f501085f0 2024-06-09T22:20:53.029 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-06-09T22:20:53.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.032+0000 7f7f56411700 1 -- 172.21.15.104:0/469580413 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f380a2c70 msgr2=0x7f7f380a5130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-09T22:20:53.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.032+0000 7f7f56411700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f380a2c70 0x7f7f380a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:53.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.032+0000 7f7f56411700 1 -- 172.21.15.104:0/469580413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f501085f0 msgr2=0x7f7f501989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:53.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.032+0000 7f7f56411700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f501085f0 0x7f7f501989f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800efc0 tx=0x7f7f4800c3a0 comp rx=0 tx=0).stop 2024-06-09T22:20:53.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.033+0000 7f7f56411700 1 -- 172.21.15.104:0/469580413 shutdown_connections 2024-06-09T22:20:53.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.033+0000 7f7f56411700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7f7f380a2c70 0x7f7f380a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:53.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.033+0000 7f7f56411700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f501085f0 0x7f7f501989f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:53.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.033+0000 7f7f56411700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f501024e0 0x7f7f5019e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:53.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.033+0000 7f7f56411700 1 --2- 172.21.15.104:0/469580413 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f50102ea0 0x7f7f5019ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:53.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.033+0000 7f7f56411700 1 -- 172.21.15.104:0/469580413 >> 172.21.15.104:0/469580413 conn(0x7f7f50079640 msgr2=0x7f7f50100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:53.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.033+0000 7f7f56411700 1 -- 172.21.15.104:0/469580413 shutdown_connections 2024-06-09T22:20:53.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:53.033+0000 7f7f56411700 1 -- 172.21.15.104:0/469580413 wait complete. 2024-06-09T22:20:53.176 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:53 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/469580413' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-09T22:20:53.177 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:52 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:52.886+0000 7f8227133200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-09T22:20:53.208 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:53 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/469580413' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-09T22:20:53.370 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:53 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/469580413' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-09T22:20:53.371 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:53 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:53.274+0000 7f2df01bf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-09T22:20:53.476 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:53 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:53.178+0000 7f8227133200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-09T22:20:53.476 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:53 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:53.270+0000 7f8227133200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-09T22:20:53.476 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:53 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:53.478+0000 7f8227133200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-09T22:20:53.491 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:53 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:53.210+0000 7fae33814200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-09T22:20:53.491 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:53 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:53.304+0000 7fae33814200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-09T22:20:53.755 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:53 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:53.372+0000 7f2df01bf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-09T22:20:53.755 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:53 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:53.579+0000 7f2df01bf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-09T22:20:53.807 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:53 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:53.673+0000 7f8227133200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:20:53.859 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-06-09T22:20:53.859 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-09T22:20:53.859 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2024-06-09T22:20:53.859 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-09T22:20:53.859 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-09T22:20:53.859 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-06-09T22:20:53.859 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-09T22:20:53.859 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-09T22:20:53.859 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-09T22:20:53.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:20:53.860 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2024-06-09T22:20:53.917 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-09T22:20:53.917 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-09T22:20:53.917 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2024-06-09T22:20:53.917 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-06-09T22:20:53.917 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2024-06-09T22:20:53.917 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-06-09T22:20:53.917 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-09T22:20:53.917 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2024-06-09T22:20:53.918 INFO:teuthology.orchestra.run.smithi104.stdout: altname enp7s0f0 2024-06-09T22:20:53.918 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-09T22:20:53.918 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2024-06-09T22:20:53.918 INFO:teuthology.orchestra.run.smithi104.stdout: altname enp7s0f1 2024-06-09T22:20:53.918 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-09T22:20:53.918 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2024-06-09T22:20:53.918 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-09T22:20:53.918 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2024-06-09T22:20:53.918 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-09T22:20:53.918 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41991sec preferred_lft 41991sec 2024-06-09T22:20:53.918 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::ec4:7aff:fe8f:ce89/64 scope link 2024-06-09T22:20:53.918 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-06-09T22:20:53.919 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-06-09T22:20:53.919 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-09T22:20:53.919 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-06-09T22:20:53.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add name ceph-brx type bridge 2024-06-09T22:20:53.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr flush dev ceph-brx 2024-06-09T22:20:53.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set ceph-brx up 2024-06-09T22:20:53.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-06-09T22:20:53.919 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-06-09T22:20:53.991 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:53 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:53.511+0000 7fae33814200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-09T22:20:53.991 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:53 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:53.707+0000 7fae33814200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:20:54.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-09T22:20:54.030 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:53 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:53.776+0000 7f2df01bf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:20:54.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:20:54.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:20:54.066 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-09T22:20:54.089 INFO:teuthology.orchestra.run.smithi104.stdout:1 2024-06-09T22:20:54.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:20:54.091 DEBUG:teuthology.orchestra.run.smithi104:> route 2024-06-09T22:20:54.147 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2024-06-09T22:20:54.147 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-09T22:20:54.147 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-09T22:20:54.148 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-09T22:20:54.148 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-09T22:20:54.148 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-09T22:20:54.148 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-06-09T22:20:54.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-09T22:20:54.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-09T22:20:54.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-09T22:20:54.148 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-06-09T22:20:54.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-09T22:20:54.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:20:54.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:20:54.279 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2024-06-09T22:20:54.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:20:54.295 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list-id 2024-06-09T22:20:54.351 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-09T22:20:54.351 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-06-09T22:20:54.351 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-09T22:20:54.351 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-06-09T22:20:54.351 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-06-09T22:20:54.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-09T22:20:54.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:20:54.471 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-06-09T22:20:54.471 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-09T22:20:54.471 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-06-09T22:20:54.471 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-06-09T22:20:54.471 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-06-09T22:20:54.472 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-06-09T22:20:54.472 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-06-09T22:20:54.472 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-06-09T22:20:54.472 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-06-09T22:20:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-09T22:20:54.557 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:54 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:54.278+0000 7f8227133200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-09T22:20:54.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:20:54.590 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-09T22:20:54.590 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-06-09T22:20:54.590 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2024-06-09T22:20:54.590 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set dev brx.0 master ceph-brx 2024-06-09T22:20:54.590 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-06-09T22:20:54.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-09T22:20:54.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:20:54.668 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-09T22:20:54.668 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-09T22:20:54.684 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-09T22:20:54.684 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-09T22:20:54.684 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-09T22:20:54.742 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-06-09T22:20:54.742 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-06-09T22:20:54.742 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/4df1ea1d-f7bb-4f87-822a-6bafc033026e 2024-06-09T22:20:54.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:20:54.742 DEBUG:teuthology.orchestra.run.smithi104:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/4df1ea1d-f7bb-4f87-822a-6bafc033026e /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-06-09T22:20:54.743 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:54 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:54.317+0000 7fae33814200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-09T22:20:54.755 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:54.416+0000 7f2df01bf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-09T22:20:54.798 INFO:teuthology.orchestra.run.smithi104.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-06-09T22:20:54.798 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync". 2024-06-09T22:20:54.798 INFO:teuthology.orchestra.run.smithi104.stdout:invalid new device string format 2024-06-09T22:20:54.961 INFO:teuthology.orchestra.run.smithi104.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-06-09T22:20:54.961 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync". 2024-06-09T22:20:54.961 INFO:teuthology.orchestra.run.smithi104.stdout:invalid new device string format 2024-06-09T22:20:54.961 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: resolved to: "172.21.15.104:3300,172.21.15.153:3300,172.21.15.194:3300" 2024-06-09T22:20:54.961 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.104:3300,172.21.15.153:3300,172.21.15.194:3300:/volumes/qa/sv_1/4df1ea1d-f7bb-4f87-822a-6bafc033026e 2024-06-09T22:20:54.961 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,key=0,fsid=a948c0ca-26ac-11ef-bc9b-c7b262605968" will pass to kernel 2024-06-09T22:20:54.962 INFO:teuthology.orchestra.run.smithi104.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-09T22:20:54.962 INFO:teuthology.orchestra.run.smithi104.stdout: You just mounted a file system that supports labels which does not 2024-06-09T22:20:54.962 INFO:teuthology.orchestra.run.smithi104.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-09T22:20:54.962 INFO:teuthology.orchestra.run.smithi104.stdout: applications will generate AVC messages and not be allowed access to 2024-06-09T22:20:54.962 INFO:teuthology.orchestra.run.smithi104.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-09T22:20:54.964 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-09T22:20:54.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:20:54.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-09T22:20:54.993 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-09T22:20:55.002 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:54 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:54.704+0000 7f8227133200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-09T22:20:55.002 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:54 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:54.802+0000 7f8227133200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-09T22:20:55.003 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:55.004+0000 7f8227133200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-09T22:20:55.043 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:54 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:54.744+0000 7fae33814200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-09T22:20:55.043 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:54 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:54.842+0000 7fae33814200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-09T22:20:55.044 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:55 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:55.045+0000 7fae33814200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-09T22:20:55.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-09T22:20:55.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:20:55.091 INFO:teuthology.orchestra.run.smithi104.stdout:client24922 2024-06-09T22:20:55.091 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:20:55.091 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-09T22:20:55.108 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-09T22:20:55.198 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:54.879+0000 7f2df01bf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-09T22:20:55.199 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:54.984+0000 7f2df01bf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-09T22:20:55.199 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:55 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:55.200+0000 7f2df01bf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-09T22:20:55.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-09T22:20:55.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:20:55.207 INFO:teuthology.orchestra.run.smithi104.stdout:client24922 2024-06-09T22:20:55.207 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-09T22:20:55.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-09T22:20:55.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:20:55.265 INFO:teuthology.orchestra.run.smithi104.stdout:client24922 2024-06-09T22:20:55.265 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968.client24922/status') 2024-06-09T22:20:55.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-09T22:20:55.324 INFO:teuthology.orchestra.run.smithi104.stdout:instance: client.24922 (3)172.21.15.104:0/3788201202 2024-06-09T22:20:55.325 INFO:teuthology.orchestra.run.smithi104.stdout:blocklisted: false 2024-06-09T22:20:55.325 INFO:teuthology.orchestra.run.smithi104.stderr:0+1 records in 2024-06-09T22:20:55.325 INFO:teuthology.orchestra.run.smithi104.stderr:0+1 records out 2024-06-09T22:20:55.325 INFO:teuthology.orchestra.run.smithi104.stderr:72 bytes copied, 2.6001e-05 s, 2.8 MB/s 2024-06-09T22:20:55.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-09T22:20:55.333 INFO:teuthology.run_tasks:Running task exec... 2024-06-09T22:20:55.340 INFO:teuthology.task.exec:Executing custom commands... 2024-06-09T22:20:55.341 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi104.front.sepia.ceph.com 2024-06-09T22:20:55.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-06-09T22:20:55.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.449+0000 7fa7f6b29640 1 -- 172.21.15.104:0/2475558804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7f01024b0 msgr2=0x7fa7f0108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:55.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.449+0000 7fa7f6b29640 1 --2- 172.21.15.104:0/2475558804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7f01024b0 0x7fa7f0108d40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8009ca0 tx=0x7fa7e802a8c0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.449+0000 7fa7f6b29640 1 -- 172.21.15.104:0/2475558804 shutdown_connections 2024-06-09T22:20:55.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.449+0000 7fa7f6b29640 1 --2- 172.21.15.104:0/2475558804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7f01024b0 0x7fa7f0108d40 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.449+0000 7fa7f6b29640 1 --2- 172.21.15.104:0/2475558804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7f0101af0 0x7fa7f0101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.449+0000 7fa7f6b29640 1 --2- 172.21.15.104:0/2475558804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7f0100900 0x7fa7f0100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.449+0000 7fa7f6b29640 1 -- 172.21.15.104:0/2475558804 >> 172.21.15.104:0/2475558804 conn(0x7fa7f00fbe70 msgr2=0x7fa7f00fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:55.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.449+0000 7fa7f6b29640 1 -- 172.21.15.104:0/2475558804 shutdown_connections 2024-06-09T22:20:55.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.449+0000 7fa7f6b29640 1 -- 172.21.15.104:0/2475558804 wait complete. 2024-06-09T22:20:55.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f6b29640 1 Processor -- start 2024-06-09T22:20:55.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f6b29640 1 -- start start 2024-06-09T22:20:55.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f6b29640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7f0100900 0x7fa7f0196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f6b29640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7f0101af0 0x7fa7f0197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f6b29640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7f01024b0 0x7fa7f019e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f6b29640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f010c580 con 0x7fa7f0100900 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f6b29640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f010c400 con 0x7fa7f01024b0 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f6b29640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f010c700 con 0x7fa7f0101af0 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f48c5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7f0100900 0x7fa7f0196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f48c5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7f0100900 0x7fa7f0196b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54786/0 (socket says 172.21.15.104:54786) 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f48c5640 1 -- 172.21.15.104:0/3001135386 learned_addr learned my addr 172.21.15.104:0/3001135386 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f50c6640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7f01024b0 0x7fa7f019e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7effff640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7f0101af0 0x7fa7f0197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f48c5640 1 -- 172.21.15.104:0/3001135386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7f0101af0 msgr2=0x7fa7f0197070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f48c5640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7f0101af0 0x7fa7f0197070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f48c5640 1 -- 172.21.15.104:0/3001135386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7f01024b0 msgr2=0x7fa7f019e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:55.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f48c5640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7f01024b0 0x7fa7f019e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.450+0000 7fa7f48c5640 1 -- 172.21.15.104:0/3001135386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7f019e750 con 0x7fa7f0100900 2024-06-09T22:20:55.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.451+0000 7fa7f48c5640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7f0100900 0x7fa7f0196b30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa7e400e9c0 tx=0x7fa7e400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:55.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.451+0000 7fa7edffb640 1 -- 172.21.15.104:0/3001135386 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e400cd80 con 0x7fa7f0100900 2024-06-09T22:20:55.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.451+0000 7fa7f6b29640 1 -- 172.21.15.104:0/3001135386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7f019ea40 con 0x7fa7f0100900 2024-06-09T22:20:55.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.451+0000 7fa7edffb640 1 -- 172.21.15.104:0/3001135386 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7e40035f0 con 0x7fa7f0100900 2024-06-09T22:20:55.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.451+0000 7fa7edffb640 1 -- 172.21.15.104:0/3001135386 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4010690 con 0x7fa7f0100900 2024-06-09T22:20:55.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.451+0000 7fa7f6b29640 1 -- 172.21.15.104:0/3001135386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7f019ef90 con 0x7fa7f0100900 2024-06-09T22:20:55.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.452+0000 7fa7f6b29640 1 -- 172.21.15.104:0/3001135386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7f0107bc0 con 0x7fa7f0100900 2024-06-09T22:20:55.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.453+0000 7fa7edffb640 1 -- 172.21.15.104:0/3001135386 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7fa7e40037a0 con 0x7fa7f0100900 2024-06-09T22:20:55.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.453+0000 7fa7edffb640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7e00a2c80 0x7fa7e00a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:55.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.453+0000 7fa7effff640 1 -- 172.21.15.104:0/3001135386 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7e00a2c80 msgr2=0x7fa7e00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1906048760 2024-06-09T22:20:55.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.454+0000 7fa7edffb640 1 -- 172.21.15.104:0/3001135386 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fa7e40cf800 con 0x7fa7f0100900 2024-06-09T22:20:55.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.454+0000 7fa7effff640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7e00a2c80 0x7fa7e00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:20:55.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.458+0000 7fa7edffb640 1 -- 172.21.15.104:0/3001135386 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa7e409c3d0 con 0x7fa7f0100900 2024-06-09T22:20:55.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.655+0000 7fa7effff640 1 -- 172.21.15.104:0/3001135386 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7e00a2c80 msgr2=0x7fa7e00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1906048760 2024-06-09T22:20:55.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.655+0000 7fa7effff640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7e00a2c80 0x7fa7e00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:20:55.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.679+0000 7fa7f6b29640 1 -- 172.21.15.104:0/3001135386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fa7f002d2e0 con 0x7fa7f0100900 2024-06-09T22:20:55.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.731+0000 7fa7edffb640 1 -- 172.21.15.104:0/3001135386 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7fa7f002d2e0 con 0x7fa7f0100900 2024-06-09T22:20:55.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.733+0000 7fa7f6b29640 1 -- 172.21.15.104:0/3001135386 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7e00a2c80 msgr2=0x7fa7e00a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-09T22:20:55.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.733+0000 7fa7f6b29640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7e00a2c80 0x7fa7e00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.733+0000 7fa7f6b29640 1 -- 172.21.15.104:0/3001135386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7f0100900 msgr2=0x7fa7f0196b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:55.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.733+0000 7fa7f6b29640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7f0100900 0x7fa7f0196b30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa7e400e9c0 tx=0x7fa7e400ee80 comp rx=0 tx=0).stop 2024-06-09T22:20:55.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.733+0000 7fa7f6b29640 1 -- 172.21.15.104:0/3001135386 shutdown_connections 2024-06-09T22:20:55.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.733+0000 7fa7f6b29640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fa7e00a2c80 0x7fa7e00a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.733+0000 7fa7f6b29640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7f01024b0 0x7fa7f019e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.733+0000 7fa7f6b29640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7f0101af0 0x7fa7f0197070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.733+0000 7fa7f6b29640 1 --2- 172.21.15.104:0/3001135386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7f0100900 0x7fa7f0196b30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.733+0000 7fa7f6b29640 1 -- 172.21.15.104:0/3001135386 >> 172.21.15.104:0/3001135386 conn(0x7fa7f00fbe70 msgr2=0x7fa7f0104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:55.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.733+0000 7fa7f6b29640 1 -- 172.21.15.104:0/3001135386 shutdown_connections 2024-06-09T22:20:55.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.733+0000 7fa7f6b29640 1 -- 172.21.15.104:0/3001135386 wait complete. 2024-06-09T22:20:55.743 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-06-09T22:20:55.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:55 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3001135386' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-06-09T22:20:55.808 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:55.534+0000 7f8227133200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-09T22:20:55.808 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:55.628+0000 7f8227133200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-09T22:20:55.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.850+0000 7fd7f8bce640 1 -- 172.21.15.104:0/743489024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7f4100900 msgr2=0x7fd7f4100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:55.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.850+0000 7fd7f8bce640 1 --2- 172.21.15.104:0/743489024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7f4100900 0x7fd7f4100d20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009900 tx=0x7fd7e402a410 comp rx=0 tx=0).stop 2024-06-09T22:20:55.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.850+0000 7fd7f8bce640 1 -- 172.21.15.104:0/743489024 shutdown_connections 2024-06-09T22:20:55.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.850+0000 7fd7f8bce640 1 --2- 172.21.15.104:0/743489024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7f41024b0 0x7fd7f4108d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.850+0000 7fd7f8bce640 1 --2- 172.21.15.104:0/743489024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7f4101af0 0x7fd7f4101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.850+0000 7fd7f8bce640 1 --2- 172.21.15.104:0/743489024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7f4100900 0x7fd7f4100d20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.850+0000 7fd7f8bce640 1 -- 172.21.15.104:0/743489024 >> 172.21.15.104:0/743489024 conn(0x7fd7f40fbe70 msgr2=0x7fd7f40fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.850+0000 7fd7f8bce640 1 -- 172.21.15.104:0/743489024 shutdown_connections 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.850+0000 7fd7f8bce640 1 -- 172.21.15.104:0/743489024 wait complete. 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7f8bce640 1 Processor -- start 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7f8bce640 1 -- start start 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7f8bce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7f4100900 0x7fd7f4197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7f8bce640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7f4101af0 0x7fd7f4198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7f8bce640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7f41024b0 0x7fd7f419f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7f8bce640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f410c570 con 0x7fd7f4101af0 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7f8bce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f410c3f0 con 0x7fd7f4100900 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7f8bce640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f410c6f0 con 0x7fd7f41024b0 2024-06-09T22:20:55.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7ebfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7f4101af0 0x7fd7f4198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:55.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7ebfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7f4101af0 0x7fd7f4198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54800/0 (socket says 172.21.15.104:54800) 2024-06-09T22:20:55.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7ebfff640 1 -- 172.21.15.104:0/2504280672 learned_addr learned my addr 172.21.15.104:0/2504280672 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:55.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7f259c640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7f4100900 0x7fd7f4197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:55.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.851+0000 7fd7f2d9d640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7f41024b0 0x7fd7f419f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:55.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.852+0000 7fd7ebfff640 1 -- 172.21.15.104:0/2504280672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7f41024b0 msgr2=0x7fd7f419f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:55.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.852+0000 7fd7ebfff640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7f41024b0 0x7fd7f419f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.852+0000 7fd7ebfff640 1 -- 172.21.15.104:0/2504280672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7f4100900 msgr2=0x7fd7f4197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:55.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.852+0000 7fd7ebfff640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7f4100900 0x7fd7f4197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:55.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.852+0000 7fd7ebfff640 1 -- 172.21.15.104:0/2504280672 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f419f930 con 0x7fd7f4101af0 2024-06-09T22:20:55.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.852+0000 7fd7ebfff640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7f4101af0 0x7fd7f4198260 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc00e9f0 tx=0x7fd7dc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:55.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.852+0000 7fd7eb7fe640 1 -- 172.21.15.104:0/2504280672 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc00cd80 con 0x7fd7f4101af0 2024-06-09T22:20:55.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.852+0000 7fd7eb7fe640 1 -- 172.21.15.104:0/2504280672 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7dc005500 con 0x7fd7f4101af0 2024-06-09T22:20:55.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.852+0000 7fd7f8bce640 1 -- 172.21.15.104:0/2504280672 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f419fc20 con 0x7fd7f4101af0 2024-06-09T22:20:55.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.852+0000 7fd7eb7fe640 1 -- 172.21.15.104:0/2504280672 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc010690 con 0x7fd7f4101af0 2024-06-09T22:20:55.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.852+0000 7fd7f8bce640 1 -- 172.21.15.104:0/2504280672 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f41a00f0 con 0x7fd7f4101af0 2024-06-09T22:20:55.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.853+0000 7fd7f8bce640 1 -- 172.21.15.104:0/2504280672 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f41a02c0 con 0x7fd7f4101af0 2024-06-09T22:20:55.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.854+0000 7fd7eb7fe640 1 -- 172.21.15.104:0/2504280672 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7fd7dc0046c0 con 0x7fd7f4101af0 2024-06-09T22:20:55.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.855+0000 7fd7eb7fe640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd7e00a2c80 0x7fd7e00a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:55.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.855+0000 7fd7f259c640 1 -- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd7e00a2c80 msgr2=0x7fd7e00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1906048760 2024-06-09T22:20:55.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.855+0000 7fd7eb7fe640 1 -- 172.21.15.104:0/2504280672 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9150+0+0 (secure 0 0 0) 0x7fd7dc0cf8a0 con 0x7fd7f4101af0 2024-06-09T22:20:55.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.855+0000 7fd7f259c640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd7e00a2c80 0x7fd7e00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:20:55.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:55.859+0000 7fd7eb7fe640 1 -- 172.21.15.104:0/2504280672 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd7dc09c470 con 0x7fd7f4101af0 2024-06-09T22:20:55.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:55 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3001135386' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-06-09T22:20:55.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:55 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:55.576+0000 7fae33814200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-09T22:20:55.914 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:55 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:55.669+0000 7fae33814200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-09T22:20:56.005 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:55 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3001135386' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-06-09T22:20:56.005 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:55 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:55.749+0000 7f2df01bf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-09T22:20:56.005 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:55 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:55.849+0000 7f2df01bf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-09T22:20:56.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:56.056+0000 7fd7f259c640 1 -- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd7e00a2c80 msgr2=0x7fd7e00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1906048760 2024-06-09T22:20:56.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:56.056+0000 7fd7f259c640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd7e00a2c80 0x7fd7e00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:20:56.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:56.068+0000 7fd7f8bce640 1 -- 172.21.15.104:0/2504280672 --> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fd7f40fe010 con 0x7fd7e00a2c80 2024-06-09T22:20:56.219 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:55.874+0000 7f8227133200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-09T22:20:56.219 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:56 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:56.008+0000 7f8227133200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-09T22:20:56.219 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:56 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:56.104+0000 7f8227133200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-09T22:20:56.219 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:56 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:56.220+0000 7f8227133200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-09T22:20:56.241 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:55 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:55.915+0000 7fae33814200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-09T22:20:56.241 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:56 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:56.050+0000 7fae33814200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-09T22:20:56.241 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:56 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:56.147+0000 7fae33814200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-09T22:20:56.453 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:56.105+0000 7f2df01bf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-09T22:20:56.453 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:56.240+0000 7f2df01bf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-09T22:20:56.454 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:56.336+0000 7f2df01bf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-09T22:20:56.454 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:56.454+0000 7f2df01bf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-09T22:20:56.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:56.457+0000 7fd7f259c640 1 -- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd7e00a2c80 msgr2=0x7fd7e00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1906048760 2024-06-09T22:20:56.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:56.457+0000 7fd7f259c640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd7e00a2c80 0x7fd7e00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-09T22:20:56.567 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:56 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:56.265+0000 7fae33814200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-09T22:20:56.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:56 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3001135386' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-06-09T22:20:56.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:56 smithi194 ceph-mon[36856]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:20:56.808 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:56 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:56.527+0000 7f8227133200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-09T22:20:56.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:56 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3001135386' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-06-09T22:20:56.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:56 smithi153 ceph-mon[36599]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:20:56.992 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:56 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:56.569+0000 7fae33814200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-09T22:20:57.005 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:56.757+0000 7f2df01bf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-09T22:20:57.005 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:56 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3001135386' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-06-09T22:20:57.005 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:56 smithi104 ceph-mon[32389]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:20:57.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:57.258+0000 7fd7f259c640 1 -- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd7e00a2c80 msgr2=0x7fd7e00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/1906048760 2024-06-09T22:20:57.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:57.258+0000 7fd7f259c640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd7e00a2c80 0x7fd7e00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-09T22:20:57.491 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:57.177+0000 7fae33814200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-09T22:20:57.491 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y[38020]: 2024-06-09T22:20:57.397+0000 7fae33814200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-09T22:20:57.557 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:57.132+0000 7f8227133200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-09T22:20:57.557 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z[38599]: 2024-06-09T22:20:57.351+0000 7f8227133200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-09T22:20:57.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:57.600+0000 7fd7eb7fe640 1 -- 172.21.15.104:0/2504280672 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7fd7dc098350 con 0x7fd7f4101af0 2024-06-09T22:20:57.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:57.601+0000 7fd7eb7fe640 1 -- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd7e00a2c80 msgr2=0x7fd7e00a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-09T22:20:57.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:57.601+0000 7fd7eb7fe640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/1906048760,v1:172.21.15.104:6801/1906048760] conn(0x7fd7e00a2c80 0x7fd7e00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:57.734 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:57.366+0000 7f2df01bf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-09T22:20:57.734 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:57.588+0000 7f2df01bf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-09T22:20:57.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: Standby manager daemon z restarted 2024-06-09T22:20:57.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: Standby manager daemon z started 2024-06-09T22:20:57.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.194:0/2766076228' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-09T22:20:57.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.194:0/2766076228' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-09T22:20:57.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.194:0/2766076228' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.194:0/2766076228' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: Standby manager daemon y restarted 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: Standby manager daemon y started 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.153:0/831168851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.153:0/831168851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.153:0/831168851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.? 172.21.15.153:0/831168851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: Active manager daemon x restarted 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: Activating manager daemon x 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: osdmap e100: 12 total, 12 up, 12 in 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: mgrmap e20: x(active, starting, since 0.0123485s), standbys: y, z 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:20:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:20:57.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:20:57.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-09T22:20:57.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-09T22:20:57.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-09T22:20:57.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-09T22:20:57.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-09T22:20:57.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-09T22:20:57.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-09T22:20:57.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-09T22:20:57.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-09T22:20:57.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-09T22:20:57.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-09T22:20:57.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-09T22:20:57.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-09T22:20:57.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-09T22:20:57.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-09T22:20:57.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:20:57.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:20:57.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-09T22:20:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: Manager daemon x is now available 2024-06-09T22:20:57.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:20:57.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-09T22:20:57.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:57 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-09T22:20:58.005 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:57.839+0000 7f2d52abf700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.006 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:57.839+0000 7f2d52abf700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.006 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:57.839+0000 7f2d52abf700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.006 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:57.839+0000 7f2d52abf700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.006 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:57.839+0000 7f2d52abf700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.006 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:57.840+0000 7f2d4fab9700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.006 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:57.840+0000 7f2d4fab9700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.006 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:57.840+0000 7f2d4fab9700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.006 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:57.840+0000 7f2d4fab9700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.006 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:57.840+0000 7f2d4fab9700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: Standby manager daemon z restarted 2024-06-09T22:20:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: Standby manager daemon z started 2024-06-09T22:20:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.194:0/2766076228' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-09T22:20:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.194:0/2766076228' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-09T22:20:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.194:0/2766076228' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-09T22:20:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.194:0/2766076228' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-09T22:20:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: Standby manager daemon y restarted 2024-06-09T22:20:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: Standby manager daemon y started 2024-06-09T22:20:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.153:0/831168851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-09T22:20:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.153:0/831168851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-09T22:20:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.153:0/831168851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-09T22:20:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.? 172.21.15.153:0/831168851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: Active manager daemon x restarted 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: Activating manager daemon x 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: osdmap e100: 12 total, 12 up, 12 in 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: mgrmap e20: x(active, starting, since 0.0123485s), standbys: y, z 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-09T22:20:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-09T22:20:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-09T22:20:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-09T22:20:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-09T22:20:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-09T22:20:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-09T22:20:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-09T22:20:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-09T22:20:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-09T22:20:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-09T22:20:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:20:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-09T22:20:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-09T22:20:58.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: Manager daemon x is now available 2024-06-09T22:20:58.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:20:58.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-09T22:20:58.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:57 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-09T22:20:58.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: Standby manager daemon z restarted 2024-06-09T22:20:58.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: Standby manager daemon z started 2024-06-09T22:20:58.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.194:0/2766076228' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-09T22:20:58.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.194:0/2766076228' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-09T22:20:58.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.194:0/2766076228' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-09T22:20:58.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.194:0/2766076228' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-09T22:20:58.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: Standby manager daemon y restarted 2024-06-09T22:20:58.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: Standby manager daemon y started 2024-06-09T22:20:58.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.153:0/831168851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-09T22:20:58.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.153:0/831168851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-09T22:20:58.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.153:0/831168851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-09T22:20:58.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.? 172.21.15.153:0/831168851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-09T22:20:58.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: Active manager daemon x restarted 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: Activating manager daemon x 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: osdmap e100: 12 total, 12 up, 12 in 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: mgrmap e20: x(active, starting, since 0.0123485s), standbys: y, z 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-09T22:20:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-09T22:20:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-09T22:20:58.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-09T22:20:58.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-09T22:20:58.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-09T22:20:58.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-09T22:20:58.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-09T22:20:58.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-09T22:20:58.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-09T22:20:58.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-09T22:20:58.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: Manager daemon x is now available 2024-06-09T22:20:58.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:20:58.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-09T22:20:58.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:57 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-09T22:20:58.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.605+0000 7fd7eb7fe640 1 -- 172.21.15.104:0/2504280672 <== mon.0 v2:172.21.15.104:3300/0 8 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7fd7dc09bfd0 con 0x7fd7f4101af0 2024-06-09T22:20:58.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.606+0000 7fd7eb7fe640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd7e006ad60 0x7fd7e006b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:58.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.606+0000 7fd7eb7fe640 1 -- 172.21.15.104:0/2504280672 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fd7dc015e80 con 0x7fd7e006ad60 2024-06-09T22:20:58.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.611+0000 7fd7f259c640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd7e006ad60 0x7fd7e006b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:58.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.613+0000 7fd7f259c640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd7e006ad60 0x7fd7e006b140 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4005ab0 tx=0x7fd7e4000bc0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:58.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.634+0000 7fd7eb7fe640 1 -- 172.21.15.104:0/2504280672 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fd7dc015e80 con 0x7fd7e006ad60 2024-06-09T22:20:58.633 INFO:teuthology.orchestra.run.smithi104.stdout:{} 2024-06-09T22:20:58.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.636+0000 7fd7f8bce640 1 -- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd7e006ad60 msgr2=0x7fd7e006b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:58.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.636+0000 7fd7f8bce640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd7e006ad60 0x7fd7e006b140 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4005ab0 tx=0x7fd7e4000bc0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.636+0000 7fd7f8bce640 1 -- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7f4101af0 msgr2=0x7fd7f4198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:58.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.636+0000 7fd7f8bce640 1 --2- 172.21.15.104:0/2504280672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7f4101af0 0x7fd7f4198260 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc00e9f0 tx=0x7fd7dc00edf0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.636+0000 7fd7f2d9d640 1 -- 172.21.15.104:0/2504280672 reap_dead start 2024-06-09T22:20:58.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.636+0000 7fd7f8bce640 1 -- 172.21.15.104:0/2504280672 shutdown_connections 2024-06-09T22:20:58.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.636+0000 7fd7f8bce640 1 -- 172.21.15.104:0/2504280672 >> 172.21.15.104:0/2504280672 conn(0x7fd7f40fbe70 msgr2=0x7fd7f40fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:58.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.636+0000 7fd7f8bce640 1 -- 172.21.15.104:0/2504280672 shutdown_connections 2024-06-09T22:20:58.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.636+0000 7fd7f8bce640 1 -- 172.21.15.104:0/2504280672 wait complete. 2024-06-09T22:20:58.646 INFO:teuthology.run_tasks:Running task check-counter... 2024-06-09T22:20:58.658 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-06-09T22:20:58.678 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-09T22:20:58.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.785+0000 7f6f68d34640 1 -- 172.21.15.104:0/2125117919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f641035c0 msgr2=0x7f6f64109e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:58.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.785+0000 7f6f68d34640 1 --2- 172.21.15.104:0/2125117919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f641035c0 0x7f6f64109e50 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c009ca0 tx=0x7f6f5c02a8c0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.785+0000 7f6f68d34640 1 -- 172.21.15.104:0/2125117919 shutdown_connections 2024-06-09T22:20:58.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.785+0000 7f6f68d34640 1 --2- 172.21.15.104:0/2125117919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f641035c0 0x7f6f64109e50 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.785+0000 7f6f68d34640 1 --2- 172.21.15.104:0/2125117919 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f64102c00 0x7f6f64103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.785+0000 7f6f68d34640 1 --2- 172.21.15.104:0/2125117919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f64101a10 0x7f6f64101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.785+0000 7f6f68d34640 1 -- 172.21.15.104:0/2125117919 >> 172.21.15.104:0/2125117919 conn(0x7f6f640fcfa0 msgr2=0x7f6f640ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:58.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.785+0000 7f6f68d34640 1 -- 172.21.15.104:0/2125117919 shutdown_connections 2024-06-09T22:20:58.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.785+0000 7f6f68d34640 1 -- 172.21.15.104:0/2125117919 wait complete. 2024-06-09T22:20:58.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f68d34640 1 Processor -- start 2024-06-09T22:20:58.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f68d34640 1 -- start start 2024-06-09T22:20:58.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f68d34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f64101a10 0x7f6f64075750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:58.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f68d34640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f64102c00 0x7f6f64073d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:58.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f68d34640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f641035c0 0x7f6f640742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:58.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f68d34640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6410d690 con 0x7f6f64102c00 2024-06-09T22:20:58.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f68d34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6410d510 con 0x7f6f64101a10 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f68d34640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6410d810 con 0x7f6f641035c0 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f61d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f64102c00 0x7f6f64073d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f61d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f64102c00 0x7f6f64073d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44286/0 (socket says 172.21.15.104:44286) 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f61d9b640 1 -- 172.21.15.104:0/1238614844 learned_addr learned my addr 172.21.15.104:0/1238614844 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f6259c640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f64101a10 0x7f6f64075750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.786+0000 7f6f62d9d640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f641035c0 0x7f6f640742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.787+0000 7f6f61d9b640 1 -- 172.21.15.104:0/1238614844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f641035c0 msgr2=0x7f6f640742d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.787+0000 7f6f61d9b640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f641035c0 0x7f6f640742d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.787+0000 7f6f61d9b640 1 -- 172.21.15.104:0/1238614844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f64101a10 msgr2=0x7f6f64075750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.787+0000 7f6f61d9b640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f64101a10 0x7f6f64075750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.787+0000 7f6f61d9b640 1 -- 172.21.15.104:0/1238614844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f64074b60 con 0x7f6f64102c00 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.787+0000 7f6f61d9b640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f64102c00 0x7f6f64073d90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00e9f0 tx=0x7f6f4c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:58.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.787+0000 7f6f5b7fe640 1 -- 172.21.15.104:0/1238614844 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c00cd80 con 0x7f6f64102c00 2024-06-09T22:20:58.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.787+0000 7f6f68d34640 1 -- 172.21.15.104:0/1238614844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f641a1e20 con 0x7f6f64102c00 2024-06-09T22:20:58.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.787+0000 7f6f5b7fe640 1 -- 172.21.15.104:0/1238614844 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f4c005500 con 0x7f6f64102c00 2024-06-09T22:20:58.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.787+0000 7f6f5b7fe640 1 -- 172.21.15.104:0/1238614844 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c010690 con 0x7f6f64102c00 2024-06-09T22:20:58.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.787+0000 7f6f68d34640 1 -- 172.21.15.104:0/1238614844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f641a2230 con 0x7f6f64102c00 2024-06-09T22:20:58.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.788+0000 7f6f68d34640 1 -- 172.21.15.104:0/1238614844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f64108e00 con 0x7f6f64102c00 2024-06-09T22:20:58.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.789+0000 7f6f5b7fe640 1 -- 172.21.15.104:0/1238614844 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f6f4c0046c0 con 0x7f6f64102c00 2024-06-09T22:20:58.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.790+0000 7f6f5b7fe640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6f500a2990 0x7f6f500a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:58.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.790+0000 7f6f6259c640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6f500a2990 0x7f6f500a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:58.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.790+0000 7f6f5b7fe640 1 -- 172.21.15.104:0/1238614844 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6f4c0cf550 con 0x7f6f64102c00 2024-06-09T22:20:58.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.790+0000 7f6f6259c640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6f500a2990 0x7f6f500a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6f64102a60 tx=0x7f6f54005f90 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:58.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.794+0000 7f6f5b7fe640 1 -- 172.21.15.104:0/1238614844 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6f4c09c310 con 0x7f6f64102c00 2024-06-09T22:20:58.870 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:58 smithi104 ceph-mon[32389]: [09/Jun/2024:22:20:58] ENGINE Bus STARTING 2024-06-09T22:20:58.870 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:58 smithi104 ceph-mon[32389]: [09/Jun/2024:22:20:58] ENGINE Serving on https://172.21.15.104:7150 2024-06-09T22:20:58.870 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:58 smithi104 ceph-mon[32389]: [09/Jun/2024:22:20:58] ENGINE Serving on http://172.21.15.104:8765 2024-06-09T22:20:58.870 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:58 smithi104 ceph-mon[32389]: [09/Jun/2024:22:20:58] ENGINE Bus STARTED 2024-06-09T22:20:58.871 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:58 smithi104 ceph-mon[32389]: mgrmap e21: x(active, since 1.01852s), standbys: y, z 2024-06-09T22:20:58.871 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:58.619+0000 7f2d4cab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.871 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:58.619+0000 7f2d4cab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.871 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:58.619+0000 7f2d4cab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.871 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:58.619+0000 7f2d4cab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.871 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:20:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:20:58.619+0000 7f2d4cab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:20:58.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.977+0000 7f6f68d34640 1 -- 172.21.15.104:0/1238614844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f640626e0 con 0x7f6f64102c00 2024-06-09T22:20:58.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.979+0000 7f6f5b7fe640 1 -- 172.21.15.104:0/1238614844 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+25144 (secure 0 0 0) 0x7f6f4c09bc10 con 0x7f6f64102c00 2024-06-09T22:20:58.977 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:20:58.978 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":100,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:20:57.590817+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:0/2389242747":"2024-06-10T22:20:57.590784+0000","172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6801/1906048760":"2024-06-10T22:20:57.590784+0000","172.21.15.104:6800/1906048760":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/517954187":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/167857460":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/357628765":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:20:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.981+0000 7f6f68d34640 1 -- 172.21.15.104:0/1238614844 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6f500a2990 msgr2=0x7f6f500a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.981+0000 7f6f68d34640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6f500a2990 0x7f6f500a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6f64102a60 tx=0x7f6f54005f90 comp rx=0 tx=0).stop 2024-06-09T22:20:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.981+0000 7f6f68d34640 1 -- 172.21.15.104:0/1238614844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f64102c00 msgr2=0x7f6f64073d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.981+0000 7f6f68d34640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f64102c00 0x7f6f64073d90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00e9f0 tx=0x7f6f4c00edf0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.981+0000 7f6f68d34640 1 -- 172.21.15.104:0/1238614844 shutdown_connections 2024-06-09T22:20:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.981+0000 7f6f68d34640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6f500a2990 0x7f6f500a4e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.981+0000 7f6f68d34640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f641035c0 0x7f6f640742d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.981+0000 7f6f68d34640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f64102c00 0x7f6f64073d90 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.981+0000 7f6f68d34640 1 --2- 172.21.15.104:0/1238614844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f64101a10 0x7f6f64075750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.981+0000 7f6f68d34640 1 -- 172.21.15.104:0/1238614844 >> 172.21.15.104:0/1238614844 conn(0x7f6f640fcfa0 msgr2=0x7f6f64105fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.981+0000 7f6f68d34640 1 -- 172.21.15.104:0/1238614844 shutdown_connections 2024-06-09T22:20:58.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:58.981+0000 7f6f68d34640 1 -- 172.21.15.104:0/1238614844 wait complete. 2024-06-09T22:20:58.992 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-09T22:11:49.547377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-09T22:16:38.369246+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-09T22:16:42.128973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-09T22:16:51.549770+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-09T22:20:58.993 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-09T22:20:58.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:58 smithi153 ceph-mon[36599]: [09/Jun/2024:22:20:58] ENGINE Bus STARTING 2024-06-09T22:20:58.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:58 smithi153 ceph-mon[36599]: [09/Jun/2024:22:20:58] ENGINE Serving on https://172.21.15.104:7150 2024-06-09T22:20:58.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:58 smithi153 ceph-mon[36599]: [09/Jun/2024:22:20:58] ENGINE Serving on http://172.21.15.104:8765 2024-06-09T22:20:58.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:58 smithi153 ceph-mon[36599]: [09/Jun/2024:22:20:58] ENGINE Bus STARTED 2024-06-09T22:20:58.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:58 smithi153 ceph-mon[36599]: mgrmap e21: x(active, since 1.01852s), standbys: y, z 2024-06-09T22:20:59.057 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:58 smithi194 ceph-mon[36856]: [09/Jun/2024:22:20:58] ENGINE Bus STARTING 2024-06-09T22:20:59.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:58 smithi194 ceph-mon[36856]: [09/Jun/2024:22:20:58] ENGINE Serving on https://172.21.15.104:7150 2024-06-09T22:20:59.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:58 smithi194 ceph-mon[36856]: [09/Jun/2024:22:20:58] ENGINE Serving on http://172.21.15.104:8765 2024-06-09T22:20:59.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:58 smithi194 ceph-mon[36856]: [09/Jun/2024:22:20:58] ENGINE Bus STARTED 2024-06-09T22:20:59.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:58 smithi194 ceph-mon[36856]: mgrmap e21: x(active, since 1.01852s), standbys: y, z 2024-06-09T22:20:59.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.100+0000 7f1404590640 1 -- 172.21.15.104:0/1519113002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13fc102490 msgr2=0x7f13fc108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.100+0000 7f1404590640 1 --2- 172.21.15.104:0/1519113002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13fc102490 0x7f13fc108d20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f13f4009ca0 tx=0x7f13f402a8c0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 -- 172.21.15.104:0/1519113002 shutdown_connections 2024-06-09T22:20:59.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 --2- 172.21.15.104:0/1519113002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13fc102490 0x7f13fc108d20 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 --2- 172.21.15.104:0/1519113002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13fc101ad0 0x7f13fc101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 --2- 172.21.15.104:0/1519113002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13fc1008e0 0x7f13fc100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 -- 172.21.15.104:0/1519113002 >> 172.21.15.104:0/1519113002 conn(0x7f13fc0fbe90 msgr2=0x7f13fc0fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:59.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 -- 172.21.15.104:0/1519113002 shutdown_connections 2024-06-09T22:20:59.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 -- 172.21.15.104:0/1519113002 wait complete. 2024-06-09T22:20:59.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 Processor -- start 2024-06-09T22:20:59.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 -- start start 2024-06-09T22:20:59.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13fc1008e0 0x7f13fc06ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:59.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13fc101ad0 0x7f13fc19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:59.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13fc102490 0x7f13fc19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:59.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f13fc10c560 con 0x7f13fc101ad0 2024-06-09T22:20:59.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f13fc10c3e0 con 0x7f13fc102490 2024-06-09T22:20:59.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1404590640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f13fc10c6e0 con 0x7f13fc1008e0 2024-06-09T22:20:59.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1401b2b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13fc101ad0 0x7f13fc19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:59.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1401b2b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13fc101ad0 0x7f13fc19c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44302/0 (socket says 172.21.15.104:44302) 2024-06-09T22:20:59.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f1401b2b640 1 -- 172.21.15.104:0/861008950 learned_addr learned my addr 172.21.15.104:0/861008950 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:59.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.101+0000 7f140232c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13fc1008e0 0x7f13fc06ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:59.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.102+0000 7f1402b2d640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13fc102490 0x7f13fc19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:59.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.102+0000 7f1401b2b640 1 -- 172.21.15.104:0/861008950 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13fc1008e0 msgr2=0x7f13fc06ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.102+0000 7f1401b2b640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13fc1008e0 0x7f13fc06ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.102+0000 7f1401b2b640 1 -- 172.21.15.104:0/861008950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13fc102490 msgr2=0x7f13fc19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.102+0000 7f1401b2b640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13fc102490 0x7f13fc19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.102+0000 7f1401b2b640 1 -- 172.21.15.104:0/861008950 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13fc19f860 con 0x7f13fc101ad0 2024-06-09T22:20:59.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.102+0000 7f1401b2b640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13fc101ad0 0x7f13fc19c930 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f13f000e9f0 tx=0x7f13f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:59.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.102+0000 7f13ef7fe640 1 -- 172.21.15.104:0/861008950 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f000cd80 con 0x7f13fc101ad0 2024-06-09T22:20:59.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.102+0000 7f13ef7fe640 1 -- 172.21.15.104:0/861008950 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13f0005500 con 0x7f13fc101ad0 2024-06-09T22:20:59.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.102+0000 7f1404590640 1 -- 172.21.15.104:0/861008950 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13fc19faf0 con 0x7f13fc101ad0 2024-06-09T22:20:59.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.102+0000 7f13ef7fe640 1 -- 172.21.15.104:0/861008950 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f0010690 con 0x7f13fc101ad0 2024-06-09T22:20:59.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.102+0000 7f1404590640 1 -- 172.21.15.104:0/861008950 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13fc19fff0 con 0x7f13fc101ad0 2024-06-09T22:20:59.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.103+0000 7f1404590640 1 -- 172.21.15.104:0/861008950 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13fc107ba0 con 0x7f13fc101ad0 2024-06-09T22:20:59.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.104+0000 7f13ef7fe640 1 -- 172.21.15.104:0/861008950 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f13f00046c0 con 0x7f13fc101ad0 2024-06-09T22:20:59.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.105+0000 7f13ef7fe640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13e80a2990 0x7f13e80a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:59.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.105+0000 7f13ef7fe640 1 -- 172.21.15.104:0/861008950 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f13f00cf5d0 con 0x7f13fc101ad0 2024-06-09T22:20:59.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.105+0000 7f140232c640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13e80a2990 0x7f13e80a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:59.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.106+0000 7f140232c640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13e80a2990 0x7f13e80a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f13fc101930 tx=0x7f13f8005f90 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:59.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.110+0000 7f13ef7fe640 1 -- 172.21.15.104:0/861008950 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f13f009c390 con 0x7f13fc101ad0 2024-06-09T22:20:59.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.291+0000 7f1404590640 1 -- 172.21.15.104:0/861008950 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f13fc0625d0 con 0x7f13fc101ad0 2024-06-09T22:20:59.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.292+0000 7f13ef7fe640 1 -- 172.21.15.104:0/861008950 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) ==== 93+0+10 (secure 0 0 0) 0x7f13f009bc90 con 0x7f13fc101ad0 2024-06-09T22:20:59.290 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-06-09T22:20:59.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.294+0000 7f1404590640 1 -- 172.21.15.104:0/861008950 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13e80a2990 msgr2=0x7f13e80a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.294+0000 7f1404590640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13e80a2990 0x7f13e80a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f13fc101930 tx=0x7f13f8005f90 comp rx=0 tx=0).stop 2024-06-09T22:20:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.294+0000 7f1404590640 1 -- 172.21.15.104:0/861008950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13fc101ad0 msgr2=0x7f13fc19c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.294+0000 7f1404590640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13fc101ad0 0x7f13fc19c930 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f13f000e9f0 tx=0x7f13f000edf0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.294+0000 7f1404590640 1 -- 172.21.15.104:0/861008950 shutdown_connections 2024-06-09T22:20:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.294+0000 7f1404590640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13e80a2990 0x7f13e80a4e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.294+0000 7f1404590640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13fc102490 0x7f13fc19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.294+0000 7f1404590640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13fc101ad0 0x7f13fc19c930 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.294+0000 7f1404590640 1 --2- 172.21.15.104:0/861008950 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13fc1008e0 0x7f13fc06ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.294+0000 7f1404590640 1 -- 172.21.15.104:0/861008950 >> 172.21.15.104:0/861008950 conn(0x7f13fc0fbe90 msgr2=0x7f13fc104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.294+0000 7f1404590640 1 -- 172.21.15.104:0/861008950 shutdown_connections 2024-06-09T22:20:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.294+0000 7f1404590640 1 -- 172.21.15.104:0/861008950 wait complete. 2024-06-09T22:20:59.303 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-09T22:20:59.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.419+0000 7f460a431640 1 -- 172.21.15.104:0/906099737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4604102480 msgr2=0x7f4604108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.419+0000 7f460a431640 1 --2- 172.21.15.104:0/906099737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4604102480 0x7f4604108d10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f45fc009c20 tx=0x7f45fc02c8c0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.419+0000 7f460a431640 1 -- 172.21.15.104:0/906099737 shutdown_connections 2024-06-09T22:20:59.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.419+0000 7f460a431640 1 --2- 172.21.15.104:0/906099737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4604102480 0x7f4604108d10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.419+0000 7f460a431640 1 --2- 172.21.15.104:0/906099737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4604101ac0 0x7f4604101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.419+0000 7f460a431640 1 --2- 172.21.15.104:0/906099737 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f46041008d0 0x7f4604100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.420+0000 7f460a431640 1 -- 172.21.15.104:0/906099737 >> 172.21.15.104:0/906099737 conn(0x7f46040fbe40 msgr2=0x7f46040fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:59.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.420+0000 7f460a431640 1 -- 172.21.15.104:0/906099737 shutdown_connections 2024-06-09T22:20:59.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.420+0000 7f460a431640 1 -- 172.21.15.104:0/906099737 wait complete. 2024-06-09T22:20:59.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.420+0000 7f460a431640 1 Processor -- start 2024-06-09T22:20:59.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.420+0000 7f460a431640 1 -- start start 2024-06-09T22:20:59.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f460a431640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46041008d0 0x7f4604195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:59.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f460a431640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4604101ac0 0x7f4604195fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:59.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f460a431640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4604102480 0x7f460419d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:59.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f460a431640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f460410b410 con 0x7f46041008d0 2024-06-09T22:20:59.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f460a431640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f460410b290 con 0x7f4604101ac0 2024-06-09T22:20:59.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f460a431640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f460410b590 con 0x7f4604102480 2024-06-09T22:20:59.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f4603fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46041008d0 0x7f4604195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:59.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f4603fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46041008d0 0x7f4604195a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44322/0 (socket says 172.21.15.104:44322) 2024-06-09T22:20:59.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f4603fff640 1 -- 172.21.15.104:0/2540662000 learned_addr learned my addr 172.21.15.104:0/2540662000 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:59.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f46037fe640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4604101ac0 0x7f4604195fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:59.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f46089ce640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4604102480 0x7f460419d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:59.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f4603fff640 1 -- 172.21.15.104:0/2540662000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4604102480 msgr2=0x7f460419d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f4603fff640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4604102480 0x7f460419d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f4603fff640 1 -- 172.21.15.104:0/2540662000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4604101ac0 msgr2=0x7f4604195fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f4603fff640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4604101ac0 0x7f4604195fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f4603fff640 1 -- 172.21.15.104:0/2540662000 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f460419d6a0 con 0x7f46041008d0 2024-06-09T22:20:59.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.421+0000 7f4603fff640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46041008d0 0x7f4604195a80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f45f400e9c0 tx=0x7f45f400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:59.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.422+0000 7f46017fa640 1 -- 172.21.15.104:0/2540662000 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f400ccc0 con 0x7f46041008d0 2024-06-09T22:20:59.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.422+0000 7f46017fa640 1 -- 172.21.15.104:0/2540662000 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45f40035f0 con 0x7f46041008d0 2024-06-09T22:20:59.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.422+0000 7f460a431640 1 -- 172.21.15.104:0/2540662000 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f460419d990 con 0x7f46041008d0 2024-06-09T22:20:59.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.422+0000 7f46017fa640 1 -- 172.21.15.104:0/2540662000 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f400ce60 con 0x7f46041008d0 2024-06-09T22:20:59.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.422+0000 7f460a431640 1 -- 172.21.15.104:0/2540662000 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f460419df30 con 0x7f46041008d0 2024-06-09T22:20:59.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.423+0000 7f460a431640 1 -- 172.21.15.104:0/2540662000 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4604068c10 con 0x7f46041008d0 2024-06-09T22:20:59.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.424+0000 7f46017fa640 1 -- 172.21.15.104:0/2540662000 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f45f40037a0 con 0x7f46041008d0 2024-06-09T22:20:59.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.425+0000 7f46017fa640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45ec0a2a10 0x7f45ec0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:59.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.425+0000 7f46037fe640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45ec0a2a10 0x7f45ec0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:59.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.425+0000 7f46017fa640 1 -- 172.21.15.104:0/2540662000 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f45f40cf3e0 con 0x7f46041008d0 2024-06-09T22:20:59.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.426+0000 7f46037fe640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45ec0a2a10 0x7f45ec0a4ed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4604196f90 tx=0x7f45f8009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:59.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.430+0000 7f46017fa640 1 -- 172.21.15.104:0/2540662000 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f45f409c180 con 0x7f46041008d0 2024-06-09T22:20:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.620+0000 7f460a431640 1 -- 172.21.15.104:0/2540662000 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f46040625a0 con 0x7f46041008d0 2024-06-09T22:20:59.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.621+0000 7f46017fa640 1 -- 172.21.15.104:0/2540662000 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) ==== 104+0+11 (secure 0 0 0) 0x7f45f409ba80 con 0x7f46041008d0 2024-06-09T22:20:59.620 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-06-09T22:20:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.624+0000 7f460a431640 1 -- 172.21.15.104:0/2540662000 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45ec0a2a10 msgr2=0x7f45ec0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.624+0000 7f460a431640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45ec0a2a10 0x7f45ec0a4ed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4604196f90 tx=0x7f45f8009260 comp rx=0 tx=0).stop 2024-06-09T22:20:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.624+0000 7f460a431640 1 -- 172.21.15.104:0/2540662000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46041008d0 msgr2=0x7f4604195a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.624+0000 7f460a431640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46041008d0 0x7f4604195a80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f45f400e9c0 tx=0x7f45f400edc0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.624+0000 7f460a431640 1 -- 172.21.15.104:0/2540662000 shutdown_connections 2024-06-09T22:20:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.624+0000 7f460a431640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45ec0a2a10 0x7f45ec0a4ed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.624+0000 7f460a431640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4604102480 0x7f460419d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.624+0000 7f460a431640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4604101ac0 0x7f4604195fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.624+0000 7f460a431640 1 --2- 172.21.15.104:0/2540662000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46041008d0 0x7f4604195a80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.624+0000 7f460a431640 1 -- 172.21.15.104:0/2540662000 >> 172.21.15.104:0/2540662000 conn(0x7f46040fbe40 msgr2=0x7f4604104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.624+0000 7f460a431640 1 -- 172.21.15.104:0/2540662000 shutdown_connections 2024-06-09T22:20:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.624+0000 7f460a431640 1 -- 172.21.15.104:0/2540662000 wait complete. 2024-06-09T22:20:59.635 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-09T22:20:59.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.762+0000 7f75087b7640 1 -- 172.21.15.104:0/506795935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75001008e0 msgr2=0x7f7500100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.762+0000 7f75087b7640 1 --2- 172.21.15.104:0/506795935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75001008e0 0x7f7500100d00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009900 tx=0x7f74fc02a400 comp rx=0 tx=0).stop 2024-06-09T22:20:59.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.762+0000 7f75087b7640 1 -- 172.21.15.104:0/506795935 shutdown_connections 2024-06-09T22:20:59.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.762+0000 7f75087b7640 1 --2- 172.21.15.104:0/506795935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7500102490 0x7f7500108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.762+0000 7f75087b7640 1 --2- 172.21.15.104:0/506795935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7500101ad0 0x7f7500101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.762+0000 7f75087b7640 1 --2- 172.21.15.104:0/506795935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75001008e0 0x7f7500100d00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.762+0000 7f75087b7640 1 -- 172.21.15.104:0/506795935 >> 172.21.15.104:0/506795935 conn(0x7f75000fbe90 msgr2=0x7f75000fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:59.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.762+0000 7f75087b7640 1 -- 172.21.15.104:0/506795935 shutdown_connections 2024-06-09T22:20:59.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.762+0000 7f75087b7640 1 -- 172.21.15.104:0/506795935 wait complete. 2024-06-09T22:20:59.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f75087b7640 1 Processor -- start 2024-06-09T22:20:59.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f75087b7640 1 -- start start 2024-06-09T22:20:59.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f75087b7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75001008e0 0x7f7500197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:59.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f75087b7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7500101ad0 0x7f7500198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:59.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f75087b7640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7500102490 0x7f750019f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:59.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f75087b7640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f750010d630 con 0x7f75001008e0 2024-06-09T22:20:59.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f75087b7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f750010d4b0 con 0x7f7500101ad0 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f75087b7640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f750010d7b0 con 0x7f7500102490 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f7506553640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75001008e0 0x7f7500197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f7506553640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75001008e0 0x7f7500197d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44376/0 (socket says 172.21.15.104:44376) 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f7506553640 1 -- 172.21.15.104:0/422527928 learned_addr learned my addr 172.21.15.104:0/422527928 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f7506d54640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7500102490 0x7f750019f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.763+0000 7f7505d52640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7500101ad0 0x7f7500198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.764+0000 7f7506d54640 1 -- 172.21.15.104:0/422527928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7500101ad0 msgr2=0x7f7500198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.764+0000 7f7506d54640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7500101ad0 0x7f7500198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.764+0000 7f7506d54640 1 -- 172.21.15.104:0/422527928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75001008e0 msgr2=0x7f7500197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.764+0000 7f7506d54640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75001008e0 0x7f7500197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.764+0000 7f7506d54640 1 -- 172.21.15.104:0/422527928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750019f950 con 0x7f7500102490 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.764+0000 7f7506553640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75001008e0 0x7f7500197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:20:59.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.764+0000 7f7506d54640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7500102490 0x7f750019f2b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f74f000d680 tx=0x7f74f000db40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:59.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.764+0000 7f74f77fe640 1 -- 172.21.15.104:0/422527928 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0017070 con 0x7f7500102490 2024-06-09T22:20:59.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.764+0000 7f74f77fe640 1 -- 172.21.15.104:0/422527928 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f0009820 con 0x7f7500102490 2024-06-09T22:20:59.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.764+0000 7f75087b7640 1 -- 172.21.15.104:0/422527928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750019fc40 con 0x7f7500102490 2024-06-09T22:20:59.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.764+0000 7f74f77fe640 1 -- 172.21.15.104:0/422527928 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0004e00 con 0x7f7500102490 2024-06-09T22:20:59.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.764+0000 7f75087b7640 1 -- 172.21.15.104:0/422527928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75001a0140 con 0x7f7500102490 2024-06-09T22:20:59.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.765+0000 7f75087b7640 1 -- 172.21.15.104:0/422527928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7500107cd0 con 0x7f7500102490 2024-06-09T22:20:59.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.767+0000 7f74f77fe640 1 -- 172.21.15.104:0/422527928 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 22) ==== 138005+0+0 (secure 0 0 0) 0x7f74f00099d0 con 0x7f7500102490 2024-06-09T22:20:59.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.767+0000 7f74f77fe640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f74ec0a2d90 0x7f74ec0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:20:59.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.768+0000 7f7506553640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f74ec0a2d90 0x7f74ec0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:20:59.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.768+0000 7f74f77fe640 1 -- 172.21.15.104:0/422527928 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f74f00cf230 con 0x7f7500102490 2024-06-09T22:20:59.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.768+0000 7f7506553640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f74ec0a2d90 0x7f74ec0a5250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0314e0 tx=0x7f74fc000bc0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:20:59.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.772+0000 7f74f77fe640 1 -- 172.21.15.104:0/422527928 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f74f009b320 con 0x7f7500102490 2024-06-09T22:20:59.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.967+0000 7f75087b7640 1 -- 172.21.15.104:0/422527928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f750002d530 con 0x7f7500102490 2024-06-09T22:20:59.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.967+0000 7f74f77fe640 1 -- 172.21.15.104:0/422527928 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) ==== 100+0+11 (secure 0 0 0) 0x7f74f009f620 con 0x7f7500102490 2024-06-09T22:20:59.966 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-06-09T22:20:59.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.969+0000 7f75087b7640 1 -- 172.21.15.104:0/422527928 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f74ec0a2d90 msgr2=0x7f74ec0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.969+0000 7f75087b7640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f74ec0a2d90 0x7f74ec0a5250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0314e0 tx=0x7f74fc000bc0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.969+0000 7f75087b7640 1 -- 172.21.15.104:0/422527928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7500102490 msgr2=0x7f750019f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:20:59.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.969+0000 7f75087b7640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7500102490 0x7f750019f2b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f74f000d680 tx=0x7f74f000db40 comp rx=0 tx=0).stop 2024-06-09T22:20:59.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.970+0000 7f75087b7640 1 -- 172.21.15.104:0/422527928 shutdown_connections 2024-06-09T22:20:59.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.970+0000 7f75087b7640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f74ec0a2d90 0x7f74ec0a5250 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.970+0000 7f75087b7640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7500102490 0x7f750019f2b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.970+0000 7f75087b7640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7500101ad0 0x7f7500198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.970+0000 7f75087b7640 1 --2- 172.21.15.104:0/422527928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75001008e0 0x7f7500197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:20:59.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.970+0000 7f75087b7640 1 -- 172.21.15.104:0/422527928 >> 172.21.15.104:0/422527928 conn(0x7f75000fbe90 msgr2=0x7f7500104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:20:59.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.970+0000 7f75087b7640 1 -- 172.21.15.104:0/422527928 shutdown_connections 2024-06-09T22:20:59.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:20:59.970+0000 7f75087b7640 1 -- 172.21.15.104:0/422527928 wait complete. 2024-06-09T22:20:59.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:59 smithi104 ceph-mon[32389]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:59.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:59 smithi104 ceph-mon[32389]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:20:59.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:59 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1238614844' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:20:59.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:59 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/861008950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-09T22:20:59.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:59 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2540662000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-09T22:20:59.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:59 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:20:59.979 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:20:59 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:20:59.980 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-09T22:20:59.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:59 smithi153 ceph-mon[36599]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:20:59.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:59 smithi153 ceph-mon[36599]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:20:59.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:59 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1238614844' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:20:59.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:59 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/861008950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-09T22:20:59.991 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:59 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2540662000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-09T22:20:59.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:59 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:20:59.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:20:59 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:00.057 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:59 smithi194 ceph-mon[36856]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:00.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:59 smithi194 ceph-mon[36856]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:21:00.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:59 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1238614844' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:21:00.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:59 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/861008950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-09T22:21:00.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:59 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2540662000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-09T22:21:00.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:59 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:00.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:20:59 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:00.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.086+0000 7f29c79e7640 1 -- 172.21.15.104:0/3921765022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c0100c30 msgr2=0x7f29c0101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:00.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.086+0000 7f29c79e7640 1 --2- 172.21.15.104:0/3921765022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c0100c30 0x7f29c0101090 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009900 tx=0x7f29b002a480 comp rx=0 tx=0).stop 2024-06-09T22:21:00.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.086+0000 7f29c79e7640 1 -- 172.21.15.104:0/3921765022 shutdown_connections 2024-06-09T22:21:00.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.086+0000 7f29c79e7640 1 --2- 172.21.15.104:0/3921765022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c01015d0 0x7f29c0101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.086+0000 7f29c79e7640 1 --2- 172.21.15.104:0/3921765022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c0100c30 0x7f29c0101090 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.086+0000 7f29c79e7640 1 --2- 172.21.15.104:0/3921765022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c01002d0 0x7f29c01006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.087+0000 7f29c79e7640 1 -- 172.21.15.104:0/3921765022 >> 172.21.15.104:0/3921765022 conn(0x7f29c00fbe50 msgr2=0x7f29c00fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.087+0000 7f29c79e7640 1 -- 172.21.15.104:0/3921765022 shutdown_connections 2024-06-09T22:21:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.087+0000 7f29c79e7640 1 -- 172.21.15.104:0/3921765022 wait complete. 2024-06-09T22:21:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.087+0000 7f29c79e7640 1 Processor -- start 2024-06-09T22:21:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.087+0000 7f29c79e7640 1 -- start start 2024-06-09T22:21:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.087+0000 7f29c79e7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c01002d0 0x7f29c0199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.087+0000 7f29c79e7640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c0100c30 0x7f29c019a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.087+0000 7f29c79e7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c01015d0 0x7f29c01a14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.087+0000 7f29c79e7640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f29c010f840 con 0x7f29c01015d0 2024-06-09T22:21:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.087+0000 7f29c79e7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29c010f6c0 con 0x7f29c01002d0 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.087+0000 7f29c79e7640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f29c010f9c0 con 0x7f29c0100c30 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c5f84640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c01015d0 0x7f29c01a14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c5f84640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c01015d0 0x7f29c01a14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44396/0 (socket says 172.21.15.104:44396) 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c5f84640 1 -- 172.21.15.104:0/3720052257 learned_addr learned my addr 172.21.15.104:0/3720052257 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c5783640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c01002d0 0x7f29c0199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c4f82640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c0100c30 0x7f29c019a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c5f84640 1 -- 172.21.15.104:0/3720052257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c0100c30 msgr2=0x7f29c019a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c5f84640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c0100c30 0x7f29c019a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c5f84640 1 -- 172.21.15.104:0/3720052257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c01002d0 msgr2=0x7f29c0199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c5f84640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c01002d0 0x7f29c0199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c5f84640 1 -- 172.21.15.104:0/3720052257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c01a1b30 con 0x7f29c01015d0 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c5f84640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c01015d0 0x7f29c01a14c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f29b800eaa0 tx=0x7f29b800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:00.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29b67fc640 1 -- 172.21.15.104:0/3720052257 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b800cdb0 con 0x7f29c01015d0 2024-06-09T22:21:00.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c79e7640 1 -- 172.21.15.104:0/3720052257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c01a1e20 con 0x7f29c01015d0 2024-06-09T22:21:00.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29b67fc640 1 -- 172.21.15.104:0/3720052257 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29b8005020 con 0x7f29c01015d0 2024-06-09T22:21:00.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29b67fc640 1 -- 172.21.15.104:0/3720052257 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b80116b0 con 0x7f29c01015d0 2024-06-09T22:21:00.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.088+0000 7f29c79e7640 1 -- 172.21.15.104:0/3720052257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c01a2290 con 0x7f29c01015d0 2024-06-09T22:21:00.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.089+0000 7f29c79e7640 1 -- 172.21.15.104:0/3720052257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c010b210 con 0x7f29c01015d0 2024-06-09T22:21:00.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.090+0000 7f29b67fc640 1 -- 172.21.15.104:0/3720052257 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 22) ==== 138005+0+0 (secure 0 0 0) 0x7f29b8005560 con 0x7f29c01015d0 2024-06-09T22:21:00.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.091+0000 7f29b67fc640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f29ac0a2d90 0x7f29ac0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:00.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.091+0000 7f29c5783640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f29ac0a2d90 0x7f29ac0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:00.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.091+0000 7f29b67fc640 1 -- 172.21.15.104:0/3720052257 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f29b80cfea0 con 0x7f29c01015d0 2024-06-09T22:21:00.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.092+0000 7f29c5783640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f29ac0a2d90 0x7f29ac0a5250 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009bd0 tx=0x7f29bc005f90 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:00.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.099+0000 7f29b67fc640 1 -- 172.21.15.104:0/3720052257 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f29b8016030 con 0x7f29c01015d0 2024-06-09T22:21:00.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.284+0000 7f29c79e7640 1 -- 172.21.15.104:0/3720052257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f29c002d470 con 0x7f29c01015d0 2024-06-09T22:21:00.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.285+0000 7f29b67fc640 1 -- 172.21.15.104:0/3720052257 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v100) ==== 103+0+11 (secure 0 0 0) 0x7f29b809bf90 con 0x7f29c01015d0 2024-06-09T22:21:00.284 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-06-09T22:21:00.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.287+0000 7f29c79e7640 1 -- 172.21.15.104:0/3720052257 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f29ac0a2d90 msgr2=0x7f29ac0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:00.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.287+0000 7f29c79e7640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f29ac0a2d90 0x7f29ac0a5250 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009bd0 tx=0x7f29bc005f90 comp rx=0 tx=0).stop 2024-06-09T22:21:00.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.287+0000 7f29c79e7640 1 -- 172.21.15.104:0/3720052257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c01015d0 msgr2=0x7f29c01a14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:00.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.287+0000 7f29c79e7640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c01015d0 0x7f29c01a14c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f29b800eaa0 tx=0x7f29b800ef60 comp rx=0 tx=0).stop 2024-06-09T22:21:00.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.287+0000 7f29c79e7640 1 -- 172.21.15.104:0/3720052257 shutdown_connections 2024-06-09T22:21:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.287+0000 7f29c79e7640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f29ac0a2d90 0x7f29ac0a5250 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.287+0000 7f29c79e7640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c01015d0 0x7f29c01a14c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.287+0000 7f29c79e7640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c0100c30 0x7f29c019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.287+0000 7f29c79e7640 1 --2- 172.21.15.104:0/3720052257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c01002d0 0x7f29c0199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.287+0000 7f29c79e7640 1 -- 172.21.15.104:0/3720052257 >> 172.21.15.104:0/3720052257 conn(0x7f29c00fbe50 msgr2=0x7f29c00fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.287+0000 7f29c79e7640 1 -- 172.21.15.104:0/3720052257 shutdown_connections 2024-06-09T22:21:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:00.287+0000 7f29c79e7640 1 -- 172.21.15.104:0/3720052257 wait complete. 2024-06-09T22:21:00.297 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-06-09T22:21:00.297 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:21:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:00.940 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:00 smithi104 ceph-mon[32389]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:21:00.940 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:00 smithi104 ceph-mon[32389]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-06-09T22:21:00.940 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:00 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:00.940 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:00 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:00.941 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:00 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:00.941 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:00 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:00.941 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:00 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/422527928' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-09T22:21:00.941 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:00 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3720052257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-06-09T22:21:01.057 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:00 smithi194 ceph-mon[36856]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:21:01.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:00 smithi194 ceph-mon[36856]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-06-09T22:21:01.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:00 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:01.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:00 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:01.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:00 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:01.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:00 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:01.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:00 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/422527928' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-09T22:21:01.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:00 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3720052257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-06-09T22:21:01.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:00 smithi153 ceph-mon[36599]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:21:01.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:00 smithi153 ceph-mon[36599]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-06-09T22:21:01.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:00 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:01.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:00 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:01.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:00 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:01.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:00 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:01.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:00 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/422527928' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-09T22:21:01.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:00 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3720052257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-06-09T22:21:02.002 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:02.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:02.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:02.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:02.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-06-09T22:21:02.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: Unable to set osd_memory_target on smithi153 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-06-09T22:21:02.004 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.004 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-06-09T22:21:02.004 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:01 smithi104 ceph-mon[32389]: Unable to set osd_memory_target on smithi194 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-06-09T22:21:02.241 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:02.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:02.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:02.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:02.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-06-09T22:21:02.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: Unable to set osd_memory_target on smithi153 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-06-09T22:21:02.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-06-09T22:21:02.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:01 smithi153 ceph-mon[36599]: Unable to set osd_memory_target on smithi194 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-06-09T22:21:02.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:02.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:02.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:02.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:02.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-06-09T22:21:02.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: Unable to set osd_memory_target on smithi153 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-06-09T22:21:02.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:02.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-06-09T22:21:02.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:01 smithi194 ceph-mon[36856]: Unable to set osd_memory_target on smithi194 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-06-09T22:21:02.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.407+0000 7fba66c51700 1 -- 172.21.15.104:0/1336723065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba60108500 msgr2=0x7fba601088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:02.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.407+0000 7fba66c51700 1 --2- 172.21.15.104:0/1336723065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba60108500 0x7fba601088e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fba54009990 tx=0x7fba54009ca0 comp rx=0 tx=0).stop 2024-06-09T22:21:02.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.407+0000 7fba66c51700 1 -- 172.21.15.104:0/1336723065 shutdown_connections 2024-06-09T22:21:02.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.407+0000 7fba66c51700 1 --2- 172.21.15.104:0/1336723065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba60108500 0x7fba601088e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:02.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.407+0000 7fba66c51700 1 --2- 172.21.15.104:0/1336723065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba6006ab30 0x7fba6010f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.407+0000 7fba66c51700 1 --2- 172.21.15.104:0/1336723065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6006a170 0x7fba6006a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.407+0000 7fba66c51700 1 -- 172.21.15.104:0/1336723065 >> 172.21.15.104:0/1336723065 conn(0x7fba60078340 msgr2=0x7fba60078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.407+0000 7fba66c51700 1 -- 172.21.15.104:0/1336723065 shutdown_connections 2024-06-09T22:21:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.407+0000 7fba66c51700 1 -- 172.21.15.104:0/1336723065 wait complete. 2024-06-09T22:21:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.407+0000 7fba66c51700 1 Processor -- start 2024-06-09T22:21:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.407+0000 7fba66c51700 1 -- start start 2024-06-09T22:21:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba66c51700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6006a170 0x7fba60101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba66c51700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba6006ab30 0x7fba601019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba66c51700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba60108500 0x7fba60101ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba66c51700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fba601127d0 con 0x7fba6006ab30 2024-06-09T22:21:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba66c51700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba60112650 con 0x7fba6006a170 2024-06-09T22:21:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba66c51700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fba60112950 con 0x7fba60108500 2024-06-09T22:21:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba5ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba6006ab30 0x7fba601019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba5ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba6006ab30 0x7fba601019a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44402/0 (socket says 172.21.15.104:44402) 2024-06-09T22:21:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba5ffff700 1 -- 172.21.15.104:0/2571622697 learned_addr learned my addr 172.21.15.104:0/2571622697 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba649ed700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6006a170 0x7fba60101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba651ee700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba60108500 0x7fba60101ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba5ffff700 1 -- 172.21.15.104:0/2571622697 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba60108500 msgr2=0x7fba60101ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba5ffff700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba60108500 0x7fba60101ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba5ffff700 1 -- 172.21.15.104:0/2571622697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6006a170 msgr2=0x7fba60101460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba5ffff700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6006a170 0x7fba60101460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba5ffff700 1 -- 172.21.15.104:0/2571622697 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba601a7210 con 0x7fba6006ab30 2024-06-09T22:21:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba649ed700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6006a170 0x7fba60101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:21:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.408+0000 7fba5ffff700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba6006ab30 0x7fba601019a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00e9a0 tx=0x7fba4c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:02.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.409+0000 7fba5dffb700 1 -- 172.21.15.104:0/2571622697 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00ca10 con 0x7fba6006ab30 2024-06-09T22:21:02.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.409+0000 7fba66c51700 1 -- 172.21.15.104:0/2571622697 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba601a7500 con 0x7fba6006ab30 2024-06-09T22:21:02.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.409+0000 7fba5dffb700 1 -- 172.21.15.104:0/2571622697 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba4c00cbb0 con 0x7fba6006ab30 2024-06-09T22:21:02.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.409+0000 7fba5dffb700 1 -- 172.21.15.104:0/2571622697 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c018680 con 0x7fba6006ab30 2024-06-09T22:21:02.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.409+0000 7fba66c51700 1 -- 172.21.15.104:0/2571622697 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba601a7a40 con 0x7fba6006ab30 2024-06-09T22:21:02.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.410+0000 7fba66c51700 1 -- 172.21.15.104:0/2571622697 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba6010a9e0 con 0x7fba6006ab30 2024-06-09T22:21:02.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.411+0000 7fba5dffb700 1 -- 172.21.15.104:0/2571622697 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7fba4c010430 con 0x7fba6006ab30 2024-06-09T22:21:02.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.412+0000 7fba5dffb700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fba500a30b0 0x7fba500a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:02.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.412+0000 7fba649ed700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fba500a30b0 0x7fba500a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:02.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.412+0000 7fba5dffb700 1 -- 172.21.15.104:0/2571622697 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fba4c018420 con 0x7fba6006ab30 2024-06-09T22:21:02.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.412+0000 7fba649ed700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fba500a30b0 0x7fba500a5570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fba5400b2e0 tx=0x7fba54005fd0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.420+0000 7fba5dffb700 1 -- 172.21.15.104:0/2571622697 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fba4c09c0e0 con 0x7fba6006ab30 2024-06-09T22:21:02.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.717+0000 7fba66c51700 1 -- 172.21.15.104:0/2571622697 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba60069180 con 0x7fba6006ab30 2024-06-09T22:21:02.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.718+0000 7fba5dffb700 1 -- 172.21.15.104:0/2571622697 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11953 (secure 0 0 0) 0x7fba4c09b9e0 con 0x7fba6006ab30 2024-06-09T22:21:02.717 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:02.717 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:20:55.680750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24460,24448,24442,24403,24419]},"id":1}]} 2024-06-09T22:21:02.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.720+0000 7fba66c51700 1 -- 172.21.15.104:0/2571622697 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fba500a30b0 msgr2=0x7fba500a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:02.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.720+0000 7fba66c51700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fba500a30b0 0x7fba500a5570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fba5400b2e0 tx=0x7fba54005fd0 comp rx=0 tx=0).stop 2024-06-09T22:21:02.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.720+0000 7fba66c51700 1 -- 172.21.15.104:0/2571622697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba6006ab30 msgr2=0x7fba601019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:02.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.720+0000 7fba66c51700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba6006ab30 0x7fba601019a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00e9a0 tx=0x7fba4c00ecb0 comp rx=0 tx=0).stop 2024-06-09T22:21:02.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.720+0000 7fba66c51700 1 -- 172.21.15.104:0/2571622697 shutdown_connections 2024-06-09T22:21:02.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.720+0000 7fba66c51700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fba500a30b0 0x7fba500a5570 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:02.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.720+0000 7fba66c51700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba60108500 0x7fba60101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:02.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.720+0000 7fba66c51700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6006a170 0x7fba60101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:02.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.720+0000 7fba66c51700 1 --2- 172.21.15.104:0/2571622697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba6006ab30 0x7fba601019a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:02.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.720+0000 7fba66c51700 1 -- 172.21.15.104:0/2571622697 >> 172.21.15.104:0/2571622697 conn(0x7fba60078340 msgr2=0x7fba60100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:02.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.721+0000 7fba66c51700 1 -- 172.21.15.104:0/2571622697 shutdown_connections 2024-06-09T22:21:02.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:02.721+0000 7fba66c51700 1 -- 172.21.15.104:0/2571622697 wait complete. 2024-06-09T22:21:02.720 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 29 2024-06-09T22:21:03.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:21:03.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-06-09T22:21:03.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:03.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:03.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:03.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:03.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:03.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:03.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi104 to 176.1M 2024-06-09T22:21:03.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: Unable to set osd_memory_target on smithi104 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-09T22:21:03.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:03.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:03.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: Updating smithi104:/etc/ceph/ceph.conf 2024-06-09T22:21:03.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: Updating smithi153:/etc/ceph/ceph.conf 2024-06-09T22:21:03.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: Updating smithi194:/etc/ceph/ceph.conf 2024-06-09T22:21:03.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: Updating smithi104:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:21:03.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: Updating smithi153:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:21:03.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: Updating smithi194:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:21:03.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:21:03.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:21:03.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:21:03.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:02 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2571622697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:03.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:21:03.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-06-09T22:21:03.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:03.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:03.242 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:03.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:03.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:03.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:03.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi104 to 176.1M 2024-06-09T22:21:03.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: Unable to set osd_memory_target on smithi104 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-09T22:21:03.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:03.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:03.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: Updating smithi104:/etc/ceph/ceph.conf 2024-06-09T22:21:03.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: Updating smithi153:/etc/ceph/ceph.conf 2024-06-09T22:21:03.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: Updating smithi194:/etc/ceph/ceph.conf 2024-06-09T22:21:03.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: Updating smithi104:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:21:03.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: Updating smithi153:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:21:03.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: Updating smithi194:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:21:03.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:21:03.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:21:03.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:21:03.246 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:02 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2571622697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:03.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:21:03.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-06-09T22:21:03.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:03.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:03.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:03.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:03.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:03.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:21:03.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi104 to 176.1M 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: Unable to set osd_memory_target on smithi104 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: Updating smithi104:/etc/ceph/ceph.conf 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: Updating smithi153:/etc/ceph/ceph.conf 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: Updating smithi194:/etc/ceph/ceph.conf 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: Updating smithi104:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: Updating smithi153:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: Updating smithi194:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-06-09T22:21:03.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:02 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2571622697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:03.550 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-06-09T22:21:03.550 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-06-09T22:21:03.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-09T22:21:03.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.656+0000 7f449635a640 1 -- 172.21.15.104:0/1477988944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f449010e280 msgr2=0x7f4490110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:03.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.656+0000 7f449635a640 1 --2- 172.21.15.104:0/1477988944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f449010e280 0x7f4490110670 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4488009ca0 tx=0x7f448802a950 comp rx=0 tx=0).stop 2024-06-09T22:21:03.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.656+0000 7f449635a640 1 -- 172.21.15.104:0/1477988944 shutdown_connections 2024-06-09T22:21:03.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.656+0000 7f449635a640 1 --2- 172.21.15.104:0/1477988944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f449010e280 0x7f4490110670 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:03.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.656+0000 7f449635a640 1 --2- 172.21.15.104:0/1477988944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44900745c0 0x7f449010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.656+0000 7f449635a640 1 --2- 172.21.15.104:0/1477988944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4490073ca0 0x7f4490074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.656+0000 7f449635a640 1 -- 172.21.15.104:0/1477988944 >> 172.21.15.104:0/1477988944 conn(0x7f4490078e30 msgr2=0x7f4490079240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.656+0000 7f449635a640 1 -- 172.21.15.104:0/1477988944 shutdown_connections 2024-06-09T22:21:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.656+0000 7f449635a640 1 -- 172.21.15.104:0/1477988944 wait complete. 2024-06-09T22:21:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.657+0000 7f449635a640 1 Processor -- start 2024-06-09T22:21:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.657+0000 7f449635a640 1 -- start start 2024-06-09T22:21:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.657+0000 7f449635a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4490073ca0 0x7f449019a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.657+0000 7f449635a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44900745c0 0x7f449019a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.657+0000 7f449635a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f449010e280 0x7f449019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.657+0000 7f449635a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4490113fb0 con 0x7f449010e280 2024-06-09T22:21:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.657+0000 7f449635a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4490113e30 con 0x7f44900745c0 2024-06-09T22:21:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.657+0000 7f449635a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4490114130 con 0x7f4490073ca0 2024-06-09T22:21:03.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.657+0000 7f44948f7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f449010e280 0x7f449019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:03.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.657+0000 7f44948f7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f449010e280 0x7f449019ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44412/0 (socket says 172.21.15.104:44412) 2024-06-09T22:21:03.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.657+0000 7f44948f7640 1 -- 172.21.15.104:0/3598675144 learned_addr learned my addr 172.21.15.104:0/3598675144 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:03.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.657+0000 7f448ffff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4490073ca0 0x7f449019a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:03.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f448f7fe640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44900745c0 0x7f449019a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:03.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f44948f7640 1 -- 172.21.15.104:0/3598675144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4490073ca0 msgr2=0x7f449019a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:03.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f44948f7640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4490073ca0 0x7f449019a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:03.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f44948f7640 1 -- 172.21.15.104:0/3598675144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44900745c0 msgr2=0x7f449019a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:03.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f44948f7640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44900745c0 0x7f449019a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:03.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f44948f7640 1 -- 172.21.15.104:0/3598675144 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f449019f360 con 0x7f449010e280 2024-06-09T22:21:03.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f448f7fe640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44900745c0 0x7f449019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:21:03.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f44948f7640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f449010e280 0x7f449019ec30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4488004af0 tx=0x7f448802f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:03.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f448d7fa640 1 -- 172.21.15.104:0/3598675144 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488034040 con 0x7f449010e280 2024-06-09T22:21:03.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f449635a640 1 -- 172.21.15.104:0/3598675144 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f449019f5f0 con 0x7f449010e280 2024-06-09T22:21:03.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f448d7fa640 1 -- 172.21.15.104:0/3598675144 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4488030920 con 0x7f449010e280 2024-06-09T22:21:03.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f448d7fa640 1 -- 172.21.15.104:0/3598675144 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44880047b0 con 0x7f449010e280 2024-06-09T22:21:03.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.658+0000 7f449635a640 1 -- 172.21.15.104:0/3598675144 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44901a6ee0 con 0x7f449010e280 2024-06-09T22:21:03.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.659+0000 7f449635a640 1 -- 172.21.15.104:0/3598675144 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4490068c10 con 0x7f449010e280 2024-06-09T22:21:03.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.660+0000 7f448d7fa640 1 -- 172.21.15.104:0/3598675144 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f4488037020 con 0x7f449010e280 2024-06-09T22:21:03.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.661+0000 7f448d7fa640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44780a2e20 0x7f44780a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:03.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.661+0000 7f448ffff640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44780a2e20 0x7f44780a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:03.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.661+0000 7f448d7fa640 1 -- 172.21.15.104:0/3598675144 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f44880f0750 con 0x7f449010e280 2024-06-09T22:21:03.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.661+0000 7f448ffff640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44780a2e20 0x7f44780a52e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4480009bd0 tx=0x7f4480005f90 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:03.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.665+0000 7f448d7fa640 1 -- 172.21.15.104:0/3598675144 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f44880bcec0 con 0x7f449010e280 2024-06-09T22:21:03.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.836+0000 7f449635a640 1 -- 172.21.15.104:0/3598675144 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4490100070 con 0x7f44780a2e20 2024-06-09T22:21:03.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.846+0000 7f448d7fa640 1 -- 172.21.15.104:0/3598675144 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339859 (secure 0 0 0) 0x7f4490100070 con 0x7f44780a2e20 2024-06-09T22:21:03.849 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:03.851 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-06-09T22:21:03.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.850+0000 7f449635a640 1 -- 172.21.15.104:0/3598675144 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44780a2e20 msgr2=0x7f44780a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:03.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.850+0000 7f449635a640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44780a2e20 0x7f44780a52e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4480009bd0 tx=0x7f4480005f90 comp rx=0 tx=0).stop 2024-06-09T22:21:03.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.850+0000 7f449635a640 1 -- 172.21.15.104:0/3598675144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f449010e280 msgr2=0x7f449019ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:03.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.850+0000 7f449635a640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f449010e280 0x7f449019ec30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4488004af0 tx=0x7f448802f5b0 comp rx=0 tx=0).stop 2024-06-09T22:21:03.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.850+0000 7f449635a640 1 -- 172.21.15.104:0/3598675144 shutdown_connections 2024-06-09T22:21:03.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.850+0000 7f449635a640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44780a2e20 0x7f44780a52e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:03.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.850+0000 7f449635a640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f449010e280 0x7f449019ec30 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:03.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.850+0000 7f449635a640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44900745c0 0x7f449019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:03.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.850+0000 7f449635a640 1 --2- 172.21.15.104:0/3598675144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4490073ca0 0x7f449019a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:03.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.850+0000 7f449635a640 1 -- 172.21.15.104:0/3598675144 >> 172.21.15.104:0/3598675144 conn(0x7f4490078e30 msgr2=0x7f44900ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:03.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.850+0000 7f449635a640 1 -- 172.21.15.104:0/3598675144 shutdown_connections 2024-06-09T22:21:03.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:03.850+0000 7f449635a640 1 -- 172.21.15.104:0/3598675144 wait complete. 2024-06-09T22:21:03.860 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-06-09T22:21:03.609423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1198059,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2151,"num_read_kb":1248,"num_write":570,"num_write_kb":4421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":951,"ondisk_log_size":951,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383732,"kb_used_data":13740,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558092,"statfs":{"total":1151940427776,"available":1151547486208,"internally_reserved":0,"allocated":14069760,"data_stored":10388717,"data_compressed":600501,"data_compressed_allocated":815104,"data_compressed_original":4374528,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.989045"},"pg_stats":[{"pgid":"3.3f","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710445+0000","last_change":"2024-06-09T22:16:43.212998+0000","last_active":"2024-06-09T22:20:57.710445+0000","last_peered":"2024-06-09T22:20:57.710445+0000","last_clean":"2024-06-09T22:20:57.710445+0000","last_became_active":"2024-06-09T22:16:43.212614+0000","last_became_peered":"2024-06-09T22:16:43.212614+0000","last_unstale":"2024-06-09T22:20:57.710445+0000","last_undegraded":"2024-06-09T22:20:57.710445+0000","last_fullsized":"2024-06-09T22:20:57.710445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:22.728486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"99'4","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711160+0000","last_change":"2024-06-09T22:16:43.208746+0000","last_active":"2024-06-09T22:20:57.711160+0000","last_peered":"2024-06-09T22:20:57.711160+0000","last_clean":"2024-06-09T22:20:57.711160+0000","last_became_active":"2024-06-09T22:16:43.208458+0000","last_became_peered":"2024-06-09T22:16:43.208458+0000","last_unstale":"2024-06-09T22:20:57.711160+0000","last_undegraded":"2024-06-09T22:20:57.711160+0000","last_fullsized":"2024-06-09T22:20:57.711160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:24:35.350449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710878+0000","last_change":"2024-06-09T22:16:43.200770+0000","last_active":"2024-06-09T22:20:57.710878+0000","last_peered":"2024-06-09T22:20:57.710878+0000","last_clean":"2024-06-09T22:20:57.710878+0000","last_became_active":"2024-06-09T22:16:43.200449+0000","last_became_peered":"2024-06-09T22:16:43.200449+0000","last_unstale":"2024-06-09T22:20:57.710878+0000","last_undegraded":"2024-06-09T22:20:57.710878+0000","last_fullsized":"2024-06-09T22:20:57.710878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:51:09.017260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703458+0000","last_change":"2024-06-09T22:16:43.203844+0000","last_active":"2024-06-09T22:20:57.703458+0000","last_peered":"2024-06-09T22:20:57.703458+0000","last_clean":"2024-06-09T22:20:57.703458+0000","last_became_active":"2024-06-09T22:16:43.203431+0000","last_became_peered":"2024-06-09T22:16:43.203431+0000","last_unstale":"2024-06-09T22:20:57.703458+0000","last_undegraded":"2024-06-09T22:20:57.703458+0000","last_fullsized":"2024-06-09T22:20:57.703458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:51:36.913312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"99'4","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694538+0000","last_change":"2024-06-09T22:16:43.207597+0000","last_active":"2024-06-09T22:20:57.694538+0000","last_peered":"2024-06-09T22:20:57.694538+0000","last_clean":"2024-06-09T22:20:57.694538+0000","last_became_active":"2024-06-09T22:16:43.207266+0000","last_became_peered":"2024-06-09T22:16:43.207266+0000","last_unstale":"2024-06-09T22:20:57.694538+0000","last_undegraded":"2024-06-09T22:20:57.694538+0000","last_fullsized":"2024-06-09T22:20:57.694538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:47:05.558403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"99'6","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702988+0000","last_change":"2024-06-09T22:17:26.474121+0000","last_active":"2024-06-09T22:20:57.702988+0000","last_peered":"2024-06-09T22:20:57.702988+0000","last_clean":"2024-06-09T22:20:57.702988+0000","last_became_active":"2024-06-09T22:17:26.473619+0000","last_became_peered":"2024-06-09T22:17:26.473619+0000","last_unstale":"2024-06-09T22:20:57.702988+0000","last_undegraded":"2024-06-09T22:20:57.702988+0000","last_fullsized":"2024-06-09T22:20:57.702988+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:56:42.171235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"99'5","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596592+0000","last_change":"2024-06-09T22:16:43.208815+0000","last_active":"2024-06-09T22:20:57.596592+0000","last_peered":"2024-06-09T22:20:57.596592+0000","last_clean":"2024-06-09T22:20:57.596592+0000","last_became_active":"2024-06-09T22:16:43.208433+0000","last_became_peered":"2024-06-09T22:16:43.208433+0000","last_unstale":"2024-06-09T22:20:57.596592+0000","last_undegraded":"2024-06-09T22:20:57.596592+0000","last_fullsized":"2024-06-09T22:20:57.596592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:36:26.585037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694783+0000","last_change":"2024-06-09T22:16:43.210195+0000","last_active":"2024-06-09T22:20:57.694783+0000","last_peered":"2024-06-09T22:20:57.694783+0000","last_clean":"2024-06-09T22:20:57.694783+0000","last_became_active":"2024-06-09T22:16:43.209892+0000","last_became_peered":"2024-06-09T22:16:43.209892+0000","last_unstale":"2024-06-09T22:20:57.694783+0000","last_undegraded":"2024-06-09T22:20:57.694783+0000","last_fullsized":"2024-06-09T22:20:57.694783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:44.088212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710618+0000","last_change":"2024-06-09T22:16:43.202517+0000","last_active":"2024-06-09T22:20:57.710618+0000","last_peered":"2024-06-09T22:20:57.710618+0000","last_clean":"2024-06-09T22:20:57.710618+0000","last_became_active":"2024-06-09T22:16:43.201796+0000","last_became_peered":"2024-06-09T22:16:43.201796+0000","last_unstale":"2024-06-09T22:20:57.710618+0000","last_undegraded":"2024-06-09T22:20:57.710618+0000","last_fullsized":"2024-06-09T22:20:57.710618+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:30.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"99'16","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703409+0000","last_change":"2024-06-09T22:16:43.212732+0000","last_active":"2024-06-09T22:20:57.703409+0000","last_peered":"2024-06-09T22:20:57.703409+0000","last_clean":"2024-06-09T22:20:57.703409+0000","last_became_active":"2024-06-09T22:16:43.212048+0000","last_became_peered":"2024-06-09T22:16:43.212048+0000","last_unstale":"2024-06-09T22:20:57.703409+0000","last_undegraded":"2024-06-09T22:20:57.703409+0000","last_fullsized":"2024-06-09T22:20:57.703409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:47.383291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597025+0000","last_change":"2024-06-09T22:16:43.203778+0000","last_active":"2024-06-09T22:20:57.597025+0000","last_peered":"2024-06-09T22:20:57.597025+0000","last_clean":"2024-06-09T22:20:57.597025+0000","last_became_active":"2024-06-09T22:16:43.202954+0000","last_became_peered":"2024-06-09T22:16:43.202954+0000","last_unstale":"2024-06-09T22:20:57.597025+0000","last_undegraded":"2024-06-09T22:20:57.597025+0000","last_fullsized":"2024-06-09T22:20:57.597025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:56:53.708645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703526+0000","last_change":"2024-06-09T22:16:43.211425+0000","last_active":"2024-06-09T22:20:57.703526+0000","last_peered":"2024-06-09T22:20:57.703526+0000","last_clean":"2024-06-09T22:20:57.703526+0000","last_became_active":"2024-06-09T22:16:43.211001+0000","last_became_peered":"2024-06-09T22:16:43.211001+0000","last_unstale":"2024-06-09T22:20:57.703526+0000","last_undegraded":"2024-06-09T22:20:57.703526+0000","last_fullsized":"2024-06-09T22:20:57.703526+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:20:59.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"99'10","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595794+0000","last_change":"2024-06-09T22:16:43.208334+0000","last_active":"2024-06-09T22:20:57.595794+0000","last_peered":"2024-06-09T22:20:57.595794+0000","last_clean":"2024-06-09T22:20:57.595794+0000","last_became_active":"2024-06-09T22:16:43.207993+0000","last_became_peered":"2024-06-09T22:16:43.207993+0000","last_unstale":"2024-06-09T22:20:57.595794+0000","last_undegraded":"2024-06-09T22:20:57.595794+0000","last_fullsized":"2024-06-09T22:20:57.595794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:07:05.740748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710461+0000","last_change":"2024-06-09T22:16:43.207211+0000","last_active":"2024-06-09T22:20:57.710461+0000","last_peered":"2024-06-09T22:20:57.710461+0000","last_clean":"2024-06-09T22:20:57.710461+0000","last_became_active":"2024-06-09T22:16:43.206889+0000","last_became_peered":"2024-06-09T22:16:43.206889+0000","last_unstale":"2024-06-09T22:20:57.710461+0000","last_undegraded":"2024-06-09T22:20:57.710461+0000","last_fullsized":"2024-06-09T22:20:57.710461+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:25:47.461030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710404+0000","last_change":"2024-06-09T22:16:43.202202+0000","last_active":"2024-06-09T22:20:57.710404+0000","last_peered":"2024-06-09T22:20:57.710404+0000","last_clean":"2024-06-09T22:20:57.710404+0000","last_became_active":"2024-06-09T22:16:43.201834+0000","last_became_peered":"2024-06-09T22:16:43.201834+0000","last_unstale":"2024-06-09T22:20:57.710404+0000","last_undegraded":"2024-06-09T22:20:57.710404+0000","last_fullsized":"2024-06-09T22:20:57.710404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:31:51.296627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"99'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709927+0000","last_change":"2024-06-09T22:16:43.200637+0000","last_active":"2024-06-09T22:20:57.709927+0000","last_peered":"2024-06-09T22:20:57.709927+0000","last_clean":"2024-06-09T22:20:57.709927+0000","last_became_active":"2024-06-09T22:16:43.200268+0000","last_became_peered":"2024-06-09T22:16:43.200268+0000","last_unstale":"2024-06-09T22:20:57.709927+0000","last_undegraded":"2024-06-09T22:20:57.709927+0000","last_fullsized":"2024-06-09T22:20:57.709927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:25:59.881453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710229+0000","last_change":"2024-06-09T22:16:43.209212+0000","last_active":"2024-06-09T22:20:57.710229+0000","last_peered":"2024-06-09T22:20:57.710229+0000","last_clean":"2024-06-09T22:20:57.710229+0000","last_became_active":"2024-06-09T22:16:43.207789+0000","last_became_peered":"2024-06-09T22:16:43.207789+0000","last_unstale":"2024-06-09T22:20:57.710229+0000","last_undegraded":"2024-06-09T22:20:57.710229+0000","last_fullsized":"2024-06-09T22:20:57.710229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:17.316595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692756+0000","last_change":"2024-06-09T22:16:43.210935+0000","last_active":"2024-06-09T22:20:57.692756+0000","last_peered":"2024-06-09T22:20:57.692756+0000","last_clean":"2024-06-09T22:20:57.692756+0000","last_became_active":"2024-06-09T22:16:43.210631+0000","last_became_peered":"2024-06-09T22:16:43.210631+0000","last_unstale":"2024-06-09T22:20:57.692756+0000","last_undegraded":"2024-06-09T22:20:57.692756+0000","last_fullsized":"2024-06-09T22:20:57.692756+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:22:58.887313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"99'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597630+0000","last_change":"2024-06-09T22:16:43.198872+0000","last_active":"2024-06-09T22:20:57.597630+0000","last_peered":"2024-06-09T22:20:57.597630+0000","last_clean":"2024-06-09T22:20:57.597630+0000","last_became_active":"2024-06-09T22:16:43.198498+0000","last_became_peered":"2024-06-09T22:16:43.198498+0000","last_unstale":"2024-06-09T22:20:57.597630+0000","last_undegraded":"2024-06-09T22:20:57.597630+0000","last_fullsized":"2024-06-09T22:20:57.597630+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:25:28.853048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597223+0000","last_change":"2024-06-09T22:16:53.343656+0000","last_active":"2024-06-09T22:20:57.597223+0000","last_peered":"2024-06-09T22:20:57.597223+0000","last_clean":"2024-06-09T22:20:57.597223+0000","last_became_active":"2024-06-09T22:16:53.343281+0000","last_became_peered":"2024-06-09T22:16:53.343281+0000","last_unstale":"2024-06-09T22:20:57.597223+0000","last_undegraded":"2024-06-09T22:20:57.597223+0000","last_fullsized":"2024-06-09T22:20:57.597223+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:40.961842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"97'13","reported_seq":247,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.700687+0000","last_change":"2024-06-09T22:16:40.040145+0000","last_active":"2024-06-09T22:20:57.700687+0000","last_peered":"2024-06-09T22:20:57.700687+0000","last_clean":"2024-06-09T22:20:57.700687+0000","last_became_active":"2024-06-09T22:16:40.039787+0000","last_became_peered":"2024-06-09T22:16:40.039787+0000","last_unstale":"2024-06-09T22:20:57.700687+0000","last_undegraded":"2024-06-09T22:20:57.700687+0000","last_fullsized":"2024-06-09T22:20:57.700687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:52:32.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":188,"num_read_kb":188,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"99'5","reported_seq":23,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692731+0000","last_change":"2024-06-09T22:17:26.774053+0000","last_active":"2024-06-09T22:20:57.692731+0000","last_peered":"2024-06-09T22:20:57.692731+0000","last_clean":"2024-06-09T22:20:57.692731+0000","last_became_active":"2024-06-09T22:17:26.773786+0000","last_became_peered":"2024-06-09T22:17:26.773786+0000","last_unstale":"2024-06-09T22:20:57.692731+0000","last_undegraded":"2024-06-09T22:20:57.692731+0000","last_fullsized":"2024-06-09T22:20:57.692731+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:36.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703244+0000","last_change":"2024-06-09T22:16:53.332048+0000","last_active":"2024-06-09T22:20:57.703244+0000","last_peered":"2024-06-09T22:20:57.703244+0000","last_clean":"2024-06-09T22:20:57.703244+0000","last_became_active":"2024-06-09T22:16:53.330683+0000","last_became_peered":"2024-06-09T22:16:53.330683+0000","last_unstale":"2024-06-09T22:20:57.703244+0000","last_undegraded":"2024-06-09T22:20:57.703244+0000","last_fullsized":"2024-06-09T22:20:57.703244+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:32:23.219061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"97'2","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696082+0000","last_change":"2024-06-09T22:16:40.070193+0000","last_active":"2024-06-09T22:20:57.696082+0000","last_peered":"2024-06-09T22:20:57.696082+0000","last_clean":"2024-06-09T22:20:57.696082+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:20:57.696082+0000","last_undegraded":"2024-06-09T22:20:57.696082+0000","last_fullsized":"2024-06-09T22:20:57.696082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:19:19.787552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711015+0000","last_change":"2024-06-09T22:16:43.199814+0000","last_active":"2024-06-09T22:20:57.711015+0000","last_peered":"2024-06-09T22:20:57.711015+0000","last_clean":"2024-06-09T22:20:57.711015+0000","last_became_active":"2024-06-09T22:16:43.199487+0000","last_became_peered":"2024-06-09T22:16:43.199487+0000","last_unstale":"2024-06-09T22:20:57.711015+0000","last_undegraded":"2024-06-09T22:20:57.711015+0000","last_fullsized":"2024-06-09T22:20:57.711015+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:55.672050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703543+0000","last_change":"2024-06-09T22:16:53.336253+0000","last_active":"2024-06-09T22:20:57.703543+0000","last_peered":"2024-06-09T22:20:57.703543+0000","last_clean":"2024-06-09T22:20:57.703543+0000","last_became_active":"2024-06-09T22:16:53.335869+0000","last_became_peered":"2024-06-09T22:16:53.335869+0000","last_unstale":"2024-06-09T22:20:57.703543+0000","last_undegraded":"2024-06-09T22:20:57.703543+0000","last_fullsized":"2024-06-09T22:20:57.703543+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:28:12.894878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"97'12","reported_seq":60,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695155+0000","last_change":"2024-06-09T22:16:40.045365+0000","last_active":"2024-06-09T22:20:57.695155+0000","last_peered":"2024-06-09T22:20:57.695155+0000","last_clean":"2024-06-09T22:20:57.695155+0000","last_became_active":"2024-06-09T22:16:40.044989+0000","last_became_peered":"2024-06-09T22:16:40.044989+0000","last_unstale":"2024-06-09T22:20:57.695155+0000","last_undegraded":"2024-06-09T22:20:57.695155+0000","last_fullsized":"2024-06-09T22:20:57.695155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:01:05.850170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"99'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710115+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:20:57.710115+0000","last_peered":"2024-06-09T22:20:57.710115+0000","last_clean":"2024-06-09T22:20:57.710115+0000","last_became_active":"2024-06-09T22:16:43.203999+0000","last_became_peered":"2024-06-09T22:16:43.203999+0000","last_unstale":"2024-06-09T22:20:57.710115+0000","last_undegraded":"2024-06-09T22:20:57.710115+0000","last_fullsized":"2024-06-09T22:20:57.710115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:37:28.362655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703277+0000","last_change":"2024-06-09T22:16:53.333565+0000","last_active":"2024-06-09T22:20:57.703277+0000","last_peered":"2024-06-09T22:20:57.703277+0000","last_clean":"2024-06-09T22:20:57.703277+0000","last_became_active":"2024-06-09T22:16:53.332921+0000","last_became_peered":"2024-06-09T22:16:53.332921+0000","last_unstale":"2024-06-09T22:20:57.703277+0000","last_undegraded":"2024-06-09T22:20:57.703277+0000","last_fullsized":"2024-06-09T22:20:57.703277+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:32.085617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"97'5","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694916+0000","last_change":"2024-06-09T22:16:40.039766+0000","last_active":"2024-06-09T22:20:57.694916+0000","last_peered":"2024-06-09T22:20:57.694916+0000","last_clean":"2024-06-09T22:20:57.694916+0000","last_became_active":"2024-06-09T22:16:40.039329+0000","last_became_peered":"2024-06-09T22:16:40.039329+0000","last_unstale":"2024-06-09T22:20:57.694916+0000","last_undegraded":"2024-06-09T22:20:57.694916+0000","last_fullsized":"2024-06-09T22:20:57.694916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:53.673890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"99'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703385+0000","last_change":"2024-06-09T22:16:43.201200+0000","last_active":"2024-06-09T22:20:57.703385+0000","last_peered":"2024-06-09T22:20:57.703385+0000","last_clean":"2024-06-09T22:20:57.703385+0000","last_became_active":"2024-06-09T22:16:43.200830+0000","last_became_peered":"2024-06-09T22:16:43.200830+0000","last_unstale":"2024-06-09T22:20:57.703385+0000","last_undegraded":"2024-06-09T22:20:57.703385+0000","last_fullsized":"2024-06-09T22:20:57.703385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:38:52.093173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597461+0000","last_change":"2024-06-09T22:16:53.339811+0000","last_active":"2024-06-09T22:20:57.597461+0000","last_peered":"2024-06-09T22:20:57.597461+0000","last_clean":"2024-06-09T22:20:57.597461+0000","last_became_active":"2024-06-09T22:16:53.339305+0000","last_became_peered":"2024-06-09T22:16:53.339305+0000","last_unstale":"2024-06-09T22:20:57.597461+0000","last_undegraded":"2024-06-09T22:20:57.597461+0000","last_fullsized":"2024-06-09T22:20:57.597461+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:08:25.800245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"100'8","reported_seq":245,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.704478+0000","last_change":"2024-06-09T22:16:40.044159+0000","last_active":"2024-06-09T22:20:57.704478+0000","last_peered":"2024-06-09T22:20:57.704478+0000","last_clean":"2024-06-09T22:20:57.704478+0000","last_became_active":"2024-06-09T22:16:40.043839+0000","last_became_peered":"2024-06-09T22:16:40.043839+0000","last_unstale":"2024-06-09T22:20:57.704478+0000","last_undegraded":"2024-06-09T22:20:57.704478+0000","last_fullsized":"2024-06-09T22:20:57.704478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:22:09.339219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1543,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":191,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694583+0000","last_change":"2024-06-09T22:16:43.202015+0000","last_active":"2024-06-09T22:20:57.694583+0000","last_peered":"2024-06-09T22:20:57.694583+0000","last_clean":"2024-06-09T22:20:57.694583+0000","last_became_active":"2024-06-09T22:16:43.201689+0000","last_became_peered":"2024-06-09T22:16:43.201689+0000","last_unstale":"2024-06-09T22:20:57.694583+0000","last_undegraded":"2024-06-09T22:20:57.694583+0000","last_fullsized":"2024-06-09T22:20:57.694583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:35:12.180399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597147+0000","last_change":"2024-06-09T22:18:26.698600+0000","last_active":"2024-06-09T22:20:57.597147+0000","last_peered":"2024-06-09T22:20:57.597147+0000","last_clean":"2024-06-09T22:20:57.597147+0000","last_became_active":"2024-06-09T22:18:26.698201+0000","last_became_peered":"2024-06-09T22:18:26.698201+0000","last_unstale":"2024-06-09T22:20:57.597147+0000","last_undegraded":"2024-06-09T22:20:57.597147+0000","last_fullsized":"2024-06-09T22:20:57.597147+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:55:13.513557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"97'7","reported_seq":57,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702844+0000","last_change":"2024-06-09T22:17:26.698204+0000","last_active":"2024-06-09T22:20:57.702844+0000","last_peered":"2024-06-09T22:20:57.702844+0000","last_clean":"2024-06-09T22:20:57.702844+0000","last_became_active":"2024-06-09T22:17:26.697716+0000","last_became_peered":"2024-06-09T22:17:26.697716+0000","last_unstale":"2024-06-09T22:20:57.702844+0000","last_undegraded":"2024-06-09T22:20:57.702844+0000","last_fullsized":"2024-06-09T22:20:57.702844+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:36:37.724459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703554+0000","last_change":"2024-06-09T22:16:43.198917+0000","last_active":"2024-06-09T22:20:57.703554+0000","last_peered":"2024-06-09T22:20:57.703554+0000","last_clean":"2024-06-09T22:20:57.703554+0000","last_became_active":"2024-06-09T22:16:43.198567+0000","last_became_peered":"2024-06-09T22:16:43.198567+0000","last_unstale":"2024-06-09T22:20:57.703554+0000","last_undegraded":"2024-06-09T22:20:57.703554+0000","last_fullsized":"2024-06-09T22:20:57.703554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:34.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710203+0000","last_change":"2024-06-09T22:16:53.342203+0000","last_active":"2024-06-09T22:20:57.710203+0000","last_peered":"2024-06-09T22:20:57.710203+0000","last_clean":"2024-06-09T22:20:57.710203+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:20:57.710203+0000","last_undegraded":"2024-06-09T22:20:57.710203+0000","last_fullsized":"2024-06-09T22:20:57.710203+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:41:41.491961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"97'3","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595514+0000","last_change":"2024-06-09T22:16:40.038684+0000","last_active":"2024-06-09T22:20:57.595514+0000","last_peered":"2024-06-09T22:20:57.595514+0000","last_clean":"2024-06-09T22:20:57.595514+0000","last_became_active":"2024-06-09T22:16:40.038386+0000","last_became_peered":"2024-06-09T22:16:40.038386+0000","last_unstale":"2024-06-09T22:20:57.595514+0000","last_undegraded":"2024-06-09T22:20:57.595514+0000","last_fullsized":"2024-06-09T22:20:57.595514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:59.188758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694679+0000","last_change":"2024-06-09T22:16:43.198837+0000","last_active":"2024-06-09T22:20:57.694679+0000","last_peered":"2024-06-09T22:20:57.694679+0000","last_clean":"2024-06-09T22:20:57.694679+0000","last_became_active":"2024-06-09T22:16:43.198515+0000","last_became_peered":"2024-06-09T22:16:43.198515+0000","last_unstale":"2024-06-09T22:20:57.694679+0000","last_undegraded":"2024-06-09T22:20:57.694679+0000","last_fullsized":"2024-06-09T22:20:57.694679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:36:13.863202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711076+0000","last_change":"2024-06-09T22:16:53.342688+0000","last_active":"2024-06-09T22:20:57.711076+0000","last_peered":"2024-06-09T22:20:57.711076+0000","last_clean":"2024-06-09T22:20:57.711076+0000","last_became_active":"2024-06-09T22:16:53.342314+0000","last_became_peered":"2024-06-09T22:16:53.342314+0000","last_unstale":"2024-06-09T22:20:57.711076+0000","last_undegraded":"2024-06-09T22:20:57.711076+0000","last_fullsized":"2024-06-09T22:20:57.711076+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:51:35.801587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"97'3","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703264+0000","last_change":"2024-06-09T22:16:40.041831+0000","last_active":"2024-06-09T22:20:57.703264+0000","last_peered":"2024-06-09T22:20:57.703264+0000","last_clean":"2024-06-09T22:20:57.703264+0000","last_became_active":"2024-06-09T22:16:40.041484+0000","last_became_peered":"2024-06-09T22:16:40.041484+0000","last_unstale":"2024-06-09T22:20:57.703264+0000","last_undegraded":"2024-06-09T22:20:57.703264+0000","last_fullsized":"2024-06-09T22:20:57.703264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:10:19.829357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692213+0000","last_change":"2024-06-09T22:16:43.210769+0000","last_active":"2024-06-09T22:20:57.692213+0000","last_peered":"2024-06-09T22:20:57.692213+0000","last_clean":"2024-06-09T22:20:57.692213+0000","last_became_active":"2024-06-09T22:16:43.210429+0000","last_became_peered":"2024-06-09T22:16:43.210429+0000","last_unstale":"2024-06-09T22:20:57.692213+0000","last_undegraded":"2024-06-09T22:20:57.692213+0000","last_fullsized":"2024-06-09T22:20:57.692213+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:12:28.788830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694708+0000","last_change":"2024-06-09T22:16:53.336536+0000","last_active":"2024-06-09T22:20:57.694708+0000","last_peered":"2024-06-09T22:20:57.694708+0000","last_clean":"2024-06-09T22:20:57.694708+0000","last_became_active":"2024-06-09T22:16:53.335178+0000","last_became_peered":"2024-06-09T22:16:53.335178+0000","last_unstale":"2024-06-09T22:20:57.694708+0000","last_undegraded":"2024-06-09T22:20:57.694708+0000","last_fullsized":"2024-06-09T22:20:57.694708+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:57:19.476483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'7","reported_seq":425,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.704124+0000","last_change":"2024-06-09T22:16:40.038484+0000","last_active":"2024-06-09T22:20:57.704124+0000","last_peered":"2024-06-09T22:20:57.704124+0000","last_clean":"2024-06-09T22:20:57.704124+0000","last_became_active":"2024-06-09T22:16:40.038084+0000","last_became_peered":"2024-06-09T22:16:40.038084+0000","last_unstale":"2024-06-09T22:20:57.704124+0000","last_undegraded":"2024-06-09T22:20:57.704124+0000","last_fullsized":"2024-06-09T22:20:57.704124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:19.733107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":374,"num_read_kb":374,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710525+0000","last_change":"2024-06-09T22:16:43.205137+0000","last_active":"2024-06-09T22:20:57.710525+0000","last_peered":"2024-06-09T22:20:57.710525+0000","last_clean":"2024-06-09T22:20:57.710525+0000","last_became_active":"2024-06-09T22:16:43.204499+0000","last_became_peered":"2024-06-09T22:16:43.204499+0000","last_unstale":"2024-06-09T22:20:57.710525+0000","last_undegraded":"2024-06-09T22:20:57.710525+0000","last_fullsized":"2024-06-09T22:20:57.710525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:15.718493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692194+0000","last_change":"2024-06-09T22:16:53.344260+0000","last_active":"2024-06-09T22:20:57.692194+0000","last_peered":"2024-06-09T22:20:57.692194+0000","last_clean":"2024-06-09T22:20:57.692194+0000","last_became_active":"2024-06-09T22:16:53.343840+0000","last_became_peered":"2024-06-09T22:16:53.343840+0000","last_unstale":"2024-06-09T22:20:57.692194+0000","last_undegraded":"2024-06-09T22:20:57.692194+0000","last_fullsized":"2024-06-09T22:20:57.692194+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:40:23.305509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"97'4","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596747+0000","last_change":"2024-06-09T22:16:40.042660+0000","last_active":"2024-06-09T22:20:57.596747+0000","last_peered":"2024-06-09T22:20:57.596747+0000","last_clean":"2024-06-09T22:20:57.596747+0000","last_became_active":"2024-06-09T22:16:40.041353+0000","last_became_peered":"2024-06-09T22:16:40.041353+0000","last_unstale":"2024-06-09T22:20:57.596747+0000","last_undegraded":"2024-06-09T22:20:57.596747+0000","last_fullsized":"2024-06-09T22:20:57.596747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:34:59.064827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711158+0000","last_change":"2024-06-09T22:16:43.205083+0000","last_active":"2024-06-09T22:20:57.711158+0000","last_peered":"2024-06-09T22:20:57.711158+0000","last_clean":"2024-06-09T22:20:57.711158+0000","last_became_active":"2024-06-09T22:16:43.204600+0000","last_became_peered":"2024-06-09T22:16:43.204600+0000","last_unstale":"2024-06-09T22:20:57.711158+0000","last_undegraded":"2024-06-09T22:20:57.711158+0000","last_fullsized":"2024-06-09T22:20:57.711158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:05:12.555465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703699+0000","last_change":"2024-06-09T22:16:53.336169+0000","last_active":"2024-06-09T22:20:57.703699+0000","last_peered":"2024-06-09T22:20:57.703699+0000","last_clean":"2024-06-09T22:20:57.703699+0000","last_became_active":"2024-06-09T22:16:53.335657+0000","last_became_peered":"2024-06-09T22:16:53.335657+0000","last_unstale":"2024-06-09T22:20:57.703699+0000","last_undegraded":"2024-06-09T22:20:57.703699+0000","last_fullsized":"2024-06-09T22:20:57.703699+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:53:21.282767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703313+0000","last_change":"2024-06-09T22:16:40.070568+0000","last_active":"2024-06-09T22:20:57.703313+0000","last_peered":"2024-06-09T22:20:57.703313+0000","last_clean":"2024-06-09T22:20:57.703313+0000","last_became_active":"2024-06-09T22:16:40.070205+0000","last_became_peered":"2024-06-09T22:16:40.070205+0000","last_unstale":"2024-06-09T22:20:57.703313+0000","last_undegraded":"2024-06-09T22:20:57.703313+0000","last_fullsized":"2024-06-09T22:20:57.703313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.449965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710336+0000","last_change":"2024-06-09T22:16:43.205219+0000","last_active":"2024-06-09T22:20:57.710336+0000","last_peered":"2024-06-09T22:20:57.710336+0000","last_clean":"2024-06-09T22:20:57.710336+0000","last_became_active":"2024-06-09T22:16:43.203467+0000","last_became_peered":"2024-06-09T22:16:43.203467+0000","last_unstale":"2024-06-09T22:20:57.710336+0000","last_undegraded":"2024-06-09T22:20:57.710336+0000","last_fullsized":"2024-06-09T22:20:57.710336+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:22.784034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692165+0000","last_change":"2024-06-09T22:16:53.338990+0000","last_active":"2024-06-09T22:20:57.692165+0000","last_peered":"2024-06-09T22:20:57.692165+0000","last_clean":"2024-06-09T22:20:57.692165+0000","last_became_active":"2024-06-09T22:16:53.338568+0000","last_became_peered":"2024-06-09T22:16:53.338568+0000","last_unstale":"2024-06-09T22:20:57.692165+0000","last_undegraded":"2024-06-09T22:20:57.692165+0000","last_fullsized":"2024-06-09T22:20:57.692165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:46:39.814043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"100'44","reported_seq":473,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.846818+0000","last_change":"2024-06-09T22:16:40.044404+0000","last_active":"2024-06-09T22:20:57.846818+0000","last_peered":"2024-06-09T22:20:57.846818+0000","last_clean":"2024-06-09T22:20:57.846818+0000","last_became_active":"2024-06-09T22:16:40.044044+0000","last_became_peered":"2024-06-09T22:16:40.044044+0000","last_unstale":"2024-06-09T22:20:57.846818+0000","last_undegraded":"2024-06-09T22:20:57.846818+0000","last_fullsized":"2024-06-09T22:20:57.846818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:50:27.038759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73538,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":385,"num_read_kb":260,"num_write":46,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694903+0000","last_change":"2024-06-09T22:16:43.206963+0000","last_active":"2024-06-09T22:20:57.694903+0000","last_peered":"2024-06-09T22:20:57.694903+0000","last_clean":"2024-06-09T22:20:57.694903+0000","last_became_active":"2024-06-09T22:16:43.206628+0000","last_became_peered":"2024-06-09T22:16:43.206628+0000","last_unstale":"2024-06-09T22:20:57.694903+0000","last_undegraded":"2024-06-09T22:20:57.694903+0000","last_fullsized":"2024-06-09T22:20:57.694903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:33:39.912236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703845+0000","last_change":"2024-06-09T22:16:53.339332+0000","last_active":"2024-06-09T22:20:57.703845+0000","last_peered":"2024-06-09T22:20:57.703845+0000","last_clean":"2024-06-09T22:20:57.703845+0000","last_became_active":"2024-06-09T22:16:53.338819+0000","last_became_peered":"2024-06-09T22:16:53.338819+0000","last_unstale":"2024-06-09T22:20:57.703845+0000","last_undegraded":"2024-06-09T22:20:57.703845+0000","last_fullsized":"2024-06-09T22:20:57.703845+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:57:41.142142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"97'9","reported_seq":33,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709996+0000","last_change":"2024-06-09T22:17:26.478256+0000","last_active":"2024-06-09T22:20:57.709996+0000","last_peered":"2024-06-09T22:20:57.709996+0000","last_clean":"2024-06-09T22:20:57.709996+0000","last_became_active":"2024-06-09T22:17:26.472264+0000","last_became_peered":"2024-06-09T22:17:26.472264+0000","last_unstale":"2024-06-09T22:20:57.709996+0000","last_undegraded":"2024-06-09T22:20:57.709996+0000","last_fullsized":"2024-06-09T22:20:57.709996+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:14:19.919841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"99'9","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595647+0000","last_change":"2024-06-09T22:16:43.204484+0000","last_active":"2024-06-09T22:20:57.595647+0000","last_peered":"2024-06-09T22:20:57.595647+0000","last_clean":"2024-06-09T22:20:57.595647+0000","last_became_active":"2024-06-09T22:16:43.204014+0000","last_became_peered":"2024-06-09T22:16:43.204014+0000","last_unstale":"2024-06-09T22:20:57.595647+0000","last_undegraded":"2024-06-09T22:20:57.595647+0000","last_fullsized":"2024-06-09T22:20:57.595647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:55:12.528231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710227+0000","last_change":"2024-06-09T22:17:26.475179+0000","last_active":"2024-06-09T22:20:57.710227+0000","last_peered":"2024-06-09T22:20:57.710227+0000","last_clean":"2024-06-09T22:20:57.710227+0000","last_became_active":"2024-06-09T22:17:26.474517+0000","last_became_peered":"2024-06-09T22:17:26.474517+0000","last_unstale":"2024-06-09T22:20:57.710227+0000","last_undegraded":"2024-06-09T22:20:57.710227+0000","last_fullsized":"2024-06-09T22:20:57.710227+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:26:02.387830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703306+0000","last_change":"2024-06-09T22:16:40.043066+0000","last_active":"2024-06-09T22:20:57.703306+0000","last_peered":"2024-06-09T22:20:57.703306+0000","last_clean":"2024-06-09T22:20:57.703306+0000","last_became_active":"2024-06-09T22:16:40.042619+0000","last_became_peered":"2024-06-09T22:16:40.042619+0000","last_unstale":"2024-06-09T22:20:57.703306+0000","last_undegraded":"2024-06-09T22:20:57.703306+0000","last_fullsized":"2024-06-09T22:20:57.703306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:29.689614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"99'15","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597173+0000","last_change":"2024-06-09T22:17:26.475566+0000","last_active":"2024-06-09T22:20:57.597173+0000","last_peered":"2024-06-09T22:20:57.597173+0000","last_clean":"2024-06-09T22:20:57.597173+0000","last_became_active":"2024-06-09T22:17:26.475037+0000","last_became_peered":"2024-06-09T22:17:26.475037+0000","last_unstale":"2024-06-09T22:20:57.597173+0000","last_undegraded":"2024-06-09T22:20:57.597173+0000","last_fullsized":"2024-06-09T22:20:57.597173+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:33.075144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694959+0000","last_change":"2024-06-09T22:17:26.475111+0000","last_active":"2024-06-09T22:20:57.694959+0000","last_peered":"2024-06-09T22:20:57.694959+0000","last_clean":"2024-06-09T22:20:57.694959+0000","last_became_active":"2024-06-09T22:17:26.474563+0000","last_became_peered":"2024-06-09T22:17:26.474563+0000","last_unstale":"2024-06-09T22:20:57.694959+0000","last_undegraded":"2024-06-09T22:20:57.694959+0000","last_fullsized":"2024-06-09T22:20:57.694959+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:09:08.458761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"100'9","reported_seq":246,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703821+0000","last_change":"2024-06-09T22:16:40.040538+0000","last_active":"2024-06-09T22:20:57.703821+0000","last_peered":"2024-06-09T22:20:57.703821+0000","last_clean":"2024-06-09T22:20:57.703821+0000","last_became_active":"2024-06-09T22:16:40.039896+0000","last_became_peered":"2024-06-09T22:16:40.039896+0000","last_unstale":"2024-06-09T22:20:57.703821+0000","last_undegraded":"2024-06-09T22:20:57.703821+0000","last_fullsized":"2024-06-09T22:20:57.703821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:00:33.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1817,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":6,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"99'6","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596912+0000","last_change":"2024-06-09T22:17:26.475568+0000","last_active":"2024-06-09T22:20:57.596912+0000","last_peered":"2024-06-09T22:20:57.596912+0000","last_clean":"2024-06-09T22:20:57.596912+0000","last_became_active":"2024-06-09T22:17:26.475163+0000","last_became_peered":"2024-06-09T22:17:26.475163+0000","last_unstale":"2024-06-09T22:20:57.596912+0000","last_undegraded":"2024-06-09T22:20:57.596912+0000","last_fullsized":"2024-06-09T22:20:57.596912+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:20.865164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595970+0000","last_change":"2024-06-09T22:16:53.338201+0000","last_active":"2024-06-09T22:20:57.595970+0000","last_peered":"2024-06-09T22:20:57.595970+0000","last_clean":"2024-06-09T22:20:57.595970+0000","last_became_active":"2024-06-09T22:16:53.337816+0000","last_became_peered":"2024-06-09T22:16:53.337816+0000","last_unstale":"2024-06-09T22:20:57.595970+0000","last_undegraded":"2024-06-09T22:20:57.595970+0000","last_fullsized":"2024-06-09T22:20:57.595970+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:35:51.861135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"97'17","reported_seq":65,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694467+0000","last_change":"2024-06-09T22:16:40.070770+0000","last_active":"2024-06-09T22:20:57.694467+0000","last_peered":"2024-06-09T22:20:57.694467+0000","last_clean":"2024-06-09T22:20:57.694467+0000","last_became_active":"2024-06-09T22:16:40.070398+0000","last_became_peered":"2024-06-09T22:16:40.070398+0000","last_unstale":"2024-06-09T22:20:57.694467+0000","last_undegraded":"2024-06-09T22:20:57.694467+0000","last_fullsized":"2024-06-09T22:20:57.694467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:04:58.273601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"99'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703349+0000","last_change":"2024-06-09T22:16:43.199246+0000","last_active":"2024-06-09T22:20:57.703349+0000","last_peered":"2024-06-09T22:20:57.703349+0000","last_clean":"2024-06-09T22:20:57.703349+0000","last_became_active":"2024-06-09T22:16:43.198006+0000","last_became_peered":"2024-06-09T22:16:43.198006+0000","last_unstale":"2024-06-09T22:20:57.703349+0000","last_undegraded":"2024-06-09T22:20:57.703349+0000","last_fullsized":"2024-06-09T22:20:57.703349+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:30:55.836628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703251+0000","last_change":"2024-06-09T22:16:53.342217+0000","last_active":"2024-06-09T22:20:57.703251+0000","last_peered":"2024-06-09T22:20:57.703251+0000","last_clean":"2024-06-09T22:20:57.703251+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:20:57.703251+0000","last_undegraded":"2024-06-09T22:20:57.703251+0000","last_fullsized":"2024-06-09T22:20:57.703251+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:25:28.635091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"100'4","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695066+0000","last_change":"2024-06-09T22:16:40.039841+0000","last_active":"2024-06-09T22:20:57.695066+0000","last_peered":"2024-06-09T22:20:57.695066+0000","last_clean":"2024-06-09T22:20:57.695066+0000","last_became_active":"2024-06-09T22:16:40.039416+0000","last_became_peered":"2024-06-09T22:16:40.039416+0000","last_unstale":"2024-06-09T22:20:57.695066+0000","last_undegraded":"2024-06-09T22:20:57.695066+0000","last_fullsized":"2024-06-09T22:20:57.695066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:27:36.213839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"99'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710082+0000","last_change":"2024-06-09T22:16:43.214363+0000","last_active":"2024-06-09T22:20:57.710082+0000","last_peered":"2024-06-09T22:20:57.710082+0000","last_clean":"2024-06-09T22:20:57.710082+0000","last_became_active":"2024-06-09T22:16:43.214015+0000","last_became_peered":"2024-06-09T22:16:43.214015+0000","last_unstale":"2024-06-09T22:20:57.710082+0000","last_undegraded":"2024-06-09T22:20:57.710082+0000","last_fullsized":"2024-06-09T22:20:57.710082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:13:06.924468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694673+0000","last_change":"2024-06-09T22:16:53.342151+0000","last_active":"2024-06-09T22:20:57.694673+0000","last_peered":"2024-06-09T22:20:57.694673+0000","last_clean":"2024-06-09T22:20:57.694673+0000","last_became_active":"2024-06-09T22:16:53.341743+0000","last_became_peered":"2024-06-09T22:16:53.341743+0000","last_unstale":"2024-06-09T22:20:57.694673+0000","last_undegraded":"2024-06-09T22:20:57.694673+0000","last_fullsized":"2024-06-09T22:20:57.694673+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:39:44.793301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"97'1","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692594+0000","last_change":"2024-06-09T22:17:26.472397+0000","last_active":"2024-06-09T22:20:57.692594+0000","last_peered":"2024-06-09T22:20:57.692594+0000","last_clean":"2024-06-09T22:20:57.692594+0000","last_became_active":"2024-06-09T22:17:26.471950+0000","last_became_peered":"2024-06-09T22:17:26.471950+0000","last_unstale":"2024-06-09T22:20:57.692594+0000","last_undegraded":"2024-06-09T22:20:57.692594+0000","last_fullsized":"2024-06-09T22:20:57.692594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:35:57.481510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"99'4","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703072+0000","last_change":"2024-06-09T22:16:43.214976+0000","last_active":"2024-06-09T22:20:57.703072+0000","last_peered":"2024-06-09T22:20:57.703072+0000","last_clean":"2024-06-09T22:20:57.703072+0000","last_became_active":"2024-06-09T22:16:43.214592+0000","last_became_peered":"2024-06-09T22:16:43.214592+0000","last_unstale":"2024-06-09T22:20:57.703072+0000","last_undegraded":"2024-06-09T22:20:57.703072+0000","last_fullsized":"2024-06-09T22:20:57.703072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:16:21.981144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595597+0000","last_change":"2024-06-09T22:16:53.342721+0000","last_active":"2024-06-09T22:20:57.595597+0000","last_peered":"2024-06-09T22:20:57.595597+0000","last_clean":"2024-06-09T22:20:57.595597+0000","last_became_active":"2024-06-09T22:16:53.342342+0000","last_became_peered":"2024-06-09T22:16:53.342342+0000","last_unstale":"2024-06-09T22:20:57.595597+0000","last_undegraded":"2024-06-09T22:20:57.595597+0000","last_fullsized":"2024-06-09T22:20:57.595597+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:22:07.469873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"97'5","reported_seq":241,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596811+0000","last_change":"2024-06-09T22:16:40.041307+0000","last_active":"2024-06-09T22:20:57.596811+0000","last_peered":"2024-06-09T22:20:57.596811+0000","last_clean":"2024-06-09T22:20:57.596811+0000","last_became_active":"2024-06-09T22:16:40.040819+0000","last_became_peered":"2024-06-09T22:16:40.040819+0000","last_unstale":"2024-06-09T22:20:57.596811+0000","last_undegraded":"2024-06-09T22:20:57.596811+0000","last_fullsized":"2024-06-09T22:20:57.596811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:14:29.658293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"99'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702925+0000","last_change":"2024-06-09T22:16:43.215719+0000","last_active":"2024-06-09T22:20:57.702925+0000","last_peered":"2024-06-09T22:20:57.702925+0000","last_clean":"2024-06-09T22:20:57.702925+0000","last_became_active":"2024-06-09T22:16:43.215356+0000","last_became_peered":"2024-06-09T22:16:43.215356+0000","last_unstale":"2024-06-09T22:20:57.702925+0000","last_undegraded":"2024-06-09T22:20:57.702925+0000","last_fullsized":"2024-06-09T22:20:57.702925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:15:07.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596001+0000","last_change":"2024-06-09T22:16:53.342948+0000","last_active":"2024-06-09T22:20:57.596001+0000","last_peered":"2024-06-09T22:20:57.596001+0000","last_clean":"2024-06-09T22:20:57.596001+0000","last_became_active":"2024-06-09T22:16:53.342352+0000","last_became_peered":"2024-06-09T22:16:53.342352+0000","last_unstale":"2024-06-09T22:20:57.596001+0000","last_undegraded":"2024-06-09T22:20:57.596001+0000","last_fullsized":"2024-06-09T22:20:57.596001+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:43:27.453789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703458+0000","last_change":"2024-06-09T22:16:43.204897+0000","last_active":"2024-06-09T22:20:57.703458+0000","last_peered":"2024-06-09T22:20:57.703458+0000","last_clean":"2024-06-09T22:20:57.703458+0000","last_became_active":"2024-06-09T22:16:43.204470+0000","last_became_peered":"2024-06-09T22:16:43.204470+0000","last_unstale":"2024-06-09T22:20:57.703458+0000","last_undegraded":"2024-06-09T22:20:57.703458+0000","last_fullsized":"2024-06-09T22:20:57.703458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:56.681392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"99'7","reported_seq":248,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:01.708116+0000","last_change":"2024-06-09T22:16:40.070079+0000","last_active":"2024-06-09T22:21:01.708116+0000","last_peered":"2024-06-09T22:21:01.708116+0000","last_clean":"2024-06-09T22:21:01.708116+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:21:01.708116+0000","last_undegraded":"2024-06-09T22:21:01.708116+0000","last_fullsized":"2024-06-09T22:21:01.708116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:00:45.132139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":193,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"97'1","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596821+0000","last_change":"2024-06-09T22:16:53.338891+0000","last_active":"2024-06-09T22:20:57.596821+0000","last_peered":"2024-06-09T22:20:57.596821+0000","last_clean":"2024-06-09T22:20:57.596821+0000","last_became_active":"2024-06-09T22:16:53.338526+0000","last_became_peered":"2024-06-09T22:16:53.338526+0000","last_unstale":"2024-06-09T22:20:57.596821+0000","last_undegraded":"2024-06-09T22:20:57.596821+0000","last_fullsized":"2024-06-09T22:20:57.596821+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:34:37.004407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695269+0000","last_change":"2024-06-09T22:16:43.199234+0000","last_active":"2024-06-09T22:20:57.695269+0000","last_peered":"2024-06-09T22:20:57.695269+0000","last_clean":"2024-06-09T22:20:57.695269+0000","last_became_active":"2024-06-09T22:16:43.192012+0000","last_became_peered":"2024-06-09T22:16:43.192012+0000","last_unstale":"2024-06-09T22:20:57.695269+0000","last_undegraded":"2024-06-09T22:20:57.695269+0000","last_fullsized":"2024-06-09T22:20:57.695269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:17.429236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"97'6","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703645+0000","last_change":"2024-06-09T22:16:40.041970+0000","last_active":"2024-06-09T22:20:57.703645+0000","last_peered":"2024-06-09T22:20:57.703645+0000","last_clean":"2024-06-09T22:20:57.703645+0000","last_became_active":"2024-06-09T22:16:40.040371+0000","last_became_peered":"2024-06-09T22:16:40.040371+0000","last_unstale":"2024-06-09T22:20:57.703645+0000","last_undegraded":"2024-06-09T22:20:57.703645+0000","last_fullsized":"2024-06-09T22:20:57.703645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:52:31.210423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":381,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596577+0000","last_change":"2024-06-09T22:15:15.367397+0000","last_active":"2024-06-09T22:20:57.596577+0000","last_peered":"2024-06-09T22:20:57.596577+0000","last_clean":"2024-06-09T22:20:57.596577+0000","last_became_active":"2024-06-09T22:15:15.366974+0000","last_became_peered":"2024-06-09T22:15:15.366974+0000","last_unstale":"2024-06-09T22:20:57.596577+0000","last_undegraded":"2024-06-09T22:20:57.596577+0000","last_fullsized":"2024-06-09T22:20:57.596577+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_clean_scrub_stamp":"2024-06-09T22:11:49.886056+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:24.018031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703964+0000","last_change":"2024-06-09T22:16:53.342249+0000","last_active":"2024-06-09T22:20:57.703964+0000","last_peered":"2024-06-09T22:20:57.703964+0000","last_clean":"2024-06-09T22:20:57.703964+0000","last_became_active":"2024-06-09T22:16:53.341779+0000","last_became_peered":"2024-06-09T22:16:53.341779+0000","last_unstale":"2024-06-09T22:20:57.703964+0000","last_undegraded":"2024-06-09T22:20:57.703964+0000","last_fullsized":"2024-06-09T22:20:57.703964+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:21:21.101578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"99'5","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595425+0000","last_change":"2024-06-09T22:16:43.200778+0000","last_active":"2024-06-09T22:20:57.595425+0000","last_peered":"2024-06-09T22:20:57.595425+0000","last_clean":"2024-06-09T22:20:57.595425+0000","last_became_active":"2024-06-09T22:16:43.200490+0000","last_became_peered":"2024-06-09T22:16:43.200490+0000","last_unstale":"2024-06-09T22:20:57.595425+0000","last_undegraded":"2024-06-09T22:20:57.595425+0000","last_fullsized":"2024-06-09T22:20:57.595425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:20:17.530204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"97'4","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596731+0000","last_change":"2024-06-09T22:16:40.041774+0000","last_active":"2024-06-09T22:20:57.596731+0000","last_peered":"2024-06-09T22:20:57.596731+0000","last_clean":"2024-06-09T22:20:57.596731+0000","last_became_active":"2024-06-09T22:16:40.041085+0000","last_became_peered":"2024-06-09T22:16:40.041085+0000","last_unstale":"2024-06-09T22:20:57.596731+0000","last_undegraded":"2024-06-09T22:20:57.596731+0000","last_fullsized":"2024-06-09T22:20:57.596731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:09:41.140083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710062+0000","last_change":"2024-06-09T22:16:53.336895+0000","last_active":"2024-06-09T22:20:57.710062+0000","last_peered":"2024-06-09T22:20:57.710062+0000","last_clean":"2024-06-09T22:20:57.710062+0000","last_became_active":"2024-06-09T22:16:53.336497+0000","last_became_peered":"2024-06-09T22:16:53.336497+0000","last_unstale":"2024-06-09T22:20:57.710062+0000","last_undegraded":"2024-06-09T22:20:57.710062+0000","last_fullsized":"2024-06-09T22:20:57.710062+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:29:52.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"97'4","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695101+0000","last_change":"2024-06-09T22:16:40.070909+0000","last_active":"2024-06-09T22:20:57.695101+0000","last_peered":"2024-06-09T22:20:57.695101+0000","last_clean":"2024-06-09T22:20:57.695101+0000","last_became_active":"2024-06-09T22:16:40.070556+0000","last_became_peered":"2024-06-09T22:16:40.070556+0000","last_unstale":"2024-06-09T22:20:57.695101+0000","last_undegraded":"2024-06-09T22:20:57.695101+0000","last_fullsized":"2024-06-09T22:20:57.695101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:59:17.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"99'9","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595793+0000","last_change":"2024-06-09T22:16:43.205951+0000","last_active":"2024-06-09T22:20:57.595793+0000","last_peered":"2024-06-09T22:20:57.595793+0000","last_clean":"2024-06-09T22:20:57.595793+0000","last_became_active":"2024-06-09T22:16:43.205594+0000","last_became_peered":"2024-06-09T22:16:43.205594+0000","last_unstale":"2024-06-09T22:20:57.595793+0000","last_undegraded":"2024-06-09T22:20:57.595793+0000","last_fullsized":"2024-06-09T22:20:57.595793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:17.956406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694724+0000","last_change":"2024-06-09T22:16:53.343292+0000","last_active":"2024-06-09T22:20:57.694724+0000","last_peered":"2024-06-09T22:20:57.694724+0000","last_clean":"2024-06-09T22:20:57.694724+0000","last_became_active":"2024-06-09T22:16:53.342900+0000","last_became_peered":"2024-06-09T22:16:53.342900+0000","last_unstale":"2024-06-09T22:20:57.694724+0000","last_undegraded":"2024-06-09T22:20:57.694724+0000","last_fullsized":"2024-06-09T22:20:57.694724+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:08:07.328250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"100'51","reported_seq":120,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.812147+0000","last_change":"2024-06-09T22:16:40.039417+0000","last_active":"2024-06-09T22:20:57.812147+0000","last_peered":"2024-06-09T22:20:57.812147+0000","last_clean":"2024-06-09T22:20:57.812147+0000","last_became_active":"2024-06-09T22:16:40.039099+0000","last_became_peered":"2024-06-09T22:16:40.039099+0000","last_unstale":"2024-06-09T22:20:57.812147+0000","last_undegraded":"2024-06-09T22:20:57.812147+0000","last_fullsized":"2024-06-09T22:20:57.812147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:22:59.284442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":90,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"99'5","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696131+0000","last_change":"2024-06-09T22:16:43.201285+0000","last_active":"2024-06-09T22:20:57.696131+0000","last_peered":"2024-06-09T22:20:57.696131+0000","last_clean":"2024-06-09T22:20:57.696131+0000","last_became_active":"2024-06-09T22:16:43.200993+0000","last_became_peered":"2024-06-09T22:16:43.200993+0000","last_unstale":"2024-06-09T22:20:57.696131+0000","last_undegraded":"2024-06-09T22:20:57.696131+0000","last_fullsized":"2024-06-09T22:20:57.696131+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:15:26.491588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596798+0000","last_change":"2024-06-09T22:16:53.334088+0000","last_active":"2024-06-09T22:20:57.596798+0000","last_peered":"2024-06-09T22:20:57.596798+0000","last_clean":"2024-06-09T22:20:57.596798+0000","last_became_active":"2024-06-09T22:16:53.333597+0000","last_became_peered":"2024-06-09T22:16:53.333597+0000","last_unstale":"2024-06-09T22:20:57.596798+0000","last_undegraded":"2024-06-09T22:20:57.596798+0000","last_fullsized":"2024-06-09T22:20:57.596798+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:50:22.041816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"97'7","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703700+0000","last_change":"2024-06-09T22:16:40.364982+0000","last_active":"2024-06-09T22:20:57.703700+0000","last_peered":"2024-06-09T22:20:57.703700+0000","last_clean":"2024-06-09T22:20:57.703700+0000","last_became_active":"2024-06-09T22:16:40.364585+0000","last_became_peered":"2024-06-09T22:16:40.364585+0000","last_unstale":"2024-06-09T22:20:57.703700+0000","last_undegraded":"2024-06-09T22:20:57.703700+0000","last_fullsized":"2024-06-09T22:20:57.703700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:28.777950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"99'7","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711181+0000","last_change":"2024-06-09T22:17:26.474491+0000","last_active":"2024-06-09T22:20:57.711181+0000","last_peered":"2024-06-09T22:20:57.711181+0000","last_clean":"2024-06-09T22:20:57.711181+0000","last_became_active":"2024-06-09T22:17:26.473975+0000","last_became_peered":"2024-06-09T22:17:26.473975+0000","last_unstale":"2024-06-09T22:20:57.711181+0000","last_undegraded":"2024-06-09T22:20:57.711181+0000","last_fullsized":"2024-06-09T22:20:57.711181+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:13:46.998595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703384+0000","last_change":"2024-06-09T22:16:53.342273+0000","last_active":"2024-06-09T22:20:57.703384+0000","last_peered":"2024-06-09T22:20:57.703384+0000","last_clean":"2024-06-09T22:20:57.703384+0000","last_became_active":"2024-06-09T22:16:53.341827+0000","last_became_peered":"2024-06-09T22:16:53.341827+0000","last_unstale":"2024-06-09T22:20:57.703384+0000","last_undegraded":"2024-06-09T22:20:57.703384+0000","last_fullsized":"2024-06-09T22:20:57.703384+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:23:26.099737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"97'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709982+0000","last_change":"2024-06-09T22:16:40.070498+0000","last_active":"2024-06-09T22:20:57.709982+0000","last_peered":"2024-06-09T22:20:57.709982+0000","last_clean":"2024-06-09T22:20:57.709982+0000","last_became_active":"2024-06-09T22:16:40.070144+0000","last_became_peered":"2024-06-09T22:16:40.070144+0000","last_unstale":"2024-06-09T22:20:57.709982+0000","last_undegraded":"2024-06-09T22:20:57.709982+0000","last_fullsized":"2024-06-09T22:20:57.709982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:52:22.214832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"99'2","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694467+0000","last_change":"2024-06-09T22:16:43.210448+0000","last_active":"2024-06-09T22:20:57.694467+0000","last_peered":"2024-06-09T22:20:57.694467+0000","last_clean":"2024-06-09T22:20:57.694467+0000","last_became_active":"2024-06-09T22:16:43.209980+0000","last_became_peered":"2024-06-09T22:16:43.209980+0000","last_unstale":"2024-06-09T22:20:57.694467+0000","last_undegraded":"2024-06-09T22:20:57.694467+0000","last_fullsized":"2024-06-09T22:20:57.694467+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:58:15.691207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692864+0000","last_change":"2024-06-09T22:16:53.340644+0000","last_active":"2024-06-09T22:20:57.692864+0000","last_peered":"2024-06-09T22:20:57.692864+0000","last_clean":"2024-06-09T22:20:57.692864+0000","last_became_active":"2024-06-09T22:16:53.339771+0000","last_became_peered":"2024-06-09T22:16:53.339771+0000","last_unstale":"2024-06-09T22:20:57.692864+0000","last_undegraded":"2024-06-09T22:20:57.692864+0000","last_fullsized":"2024-06-09T22:20:57.692864+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:45:23.281387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"97'7","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694859+0000","last_change":"2024-06-09T22:16:40.040461+0000","last_active":"2024-06-09T22:20:57.694859+0000","last_peered":"2024-06-09T22:20:57.694859+0000","last_clean":"2024-06-09T22:20:57.694859+0000","last_became_active":"2024-06-09T22:16:40.040168+0000","last_became_peered":"2024-06-09T22:16:40.040168+0000","last_unstale":"2024-06-09T22:20:57.694859+0000","last_undegraded":"2024-06-09T22:20:57.694859+0000","last_fullsized":"2024-06-09T22:20:57.694859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:28.936631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"99'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709897+0000","last_change":"2024-06-09T22:16:43.208779+0000","last_active":"2024-06-09T22:20:57.709897+0000","last_peered":"2024-06-09T22:20:57.709897+0000","last_clean":"2024-06-09T22:20:57.709897+0000","last_became_active":"2024-06-09T22:16:43.208136+0000","last_became_peered":"2024-06-09T22:16:43.208136+0000","last_unstale":"2024-06-09T22:20:57.709897+0000","last_undegraded":"2024-06-09T22:20:57.709897+0000","last_fullsized":"2024-06-09T22:20:57.709897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:22:39.721069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"97'1","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703384+0000","last_change":"2024-06-09T22:16:53.331241+0000","last_active":"2024-06-09T22:20:57.703384+0000","last_peered":"2024-06-09T22:20:57.703384+0000","last_clean":"2024-06-09T22:20:57.703384+0000","last_became_active":"2024-06-09T22:16:53.329589+0000","last_became_peered":"2024-06-09T22:16:53.329589+0000","last_unstale":"2024-06-09T22:20:57.703384+0000","last_undegraded":"2024-06-09T22:20:57.703384+0000","last_fullsized":"2024-06-09T22:20:57.703384+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:56:21.319711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"100'4","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596918+0000","last_change":"2024-06-09T22:16:40.042491+0000","last_active":"2024-06-09T22:20:57.596918+0000","last_peered":"2024-06-09T22:20:57.596918+0000","last_clean":"2024-06-09T22:20:57.596918+0000","last_became_active":"2024-06-09T22:16:40.041846+0000","last_became_peered":"2024-06-09T22:16:40.041846+0000","last_unstale":"2024-06-09T22:20:57.596918+0000","last_undegraded":"2024-06-09T22:20:57.596918+0000","last_fullsized":"2024-06-09T22:20:57.596918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:59.144003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695844+0000","last_change":"2024-06-09T22:16:43.209195+0000","last_active":"2024-06-09T22:20:57.695844+0000","last_peered":"2024-06-09T22:20:57.695844+0000","last_clean":"2024-06-09T22:20:57.695844+0000","last_became_active":"2024-06-09T22:16:43.208797+0000","last_became_peered":"2024-06-09T22:16:43.208797+0000","last_unstale":"2024-06-09T22:20:57.695844+0000","last_undegraded":"2024-06-09T22:20:57.695844+0000","last_fullsized":"2024-06-09T22:20:57.695844+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:05:31.689900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703290+0000","last_change":"2024-06-09T22:16:53.342173+0000","last_active":"2024-06-09T22:20:57.703290+0000","last_peered":"2024-06-09T22:20:57.703290+0000","last_clean":"2024-06-09T22:20:57.703290+0000","last_became_active":"2024-06-09T22:16:53.341712+0000","last_became_peered":"2024-06-09T22:16:53.341712+0000","last_unstale":"2024-06-09T22:20:57.703290+0000","last_undegraded":"2024-06-09T22:20:57.703290+0000","last_fullsized":"2024-06-09T22:20:57.703290+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:53:12.574764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"97'4","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694569+0000","last_change":"2024-06-09T22:16:40.044238+0000","last_active":"2024-06-09T22:20:57.694569+0000","last_peered":"2024-06-09T22:20:57.694569+0000","last_clean":"2024-06-09T22:20:57.694569+0000","last_became_active":"2024-06-09T22:16:40.043857+0000","last_became_peered":"2024-06-09T22:16:40.043857+0000","last_unstale":"2024-06-09T22:20:57.694569+0000","last_undegraded":"2024-06-09T22:20:57.694569+0000","last_fullsized":"2024-06-09T22:20:57.694569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:44:45.743307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"99'5","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596886+0000","last_change":"2024-06-09T22:16:43.202363+0000","last_active":"2024-06-09T22:20:57.596886+0000","last_peered":"2024-06-09T22:20:57.596886+0000","last_clean":"2024-06-09T22:20:57.596886+0000","last_became_active":"2024-06-09T22:16:43.202029+0000","last_became_peered":"2024-06-09T22:16:43.202029+0000","last_unstale":"2024-06-09T22:20:57.596886+0000","last_undegraded":"2024-06-09T22:20:57.596886+0000","last_fullsized":"2024-06-09T22:20:57.596886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:11:19.513055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710013+0000","last_change":"2024-06-09T22:16:53.333837+0000","last_active":"2024-06-09T22:20:57.710013+0000","last_peered":"2024-06-09T22:20:57.710013+0000","last_clean":"2024-06-09T22:20:57.710013+0000","last_became_active":"2024-06-09T22:16:53.333362+0000","last_became_peered":"2024-06-09T22:16:53.333362+0000","last_unstale":"2024-06-09T22:20:57.710013+0000","last_undegraded":"2024-06-09T22:20:57.710013+0000","last_fullsized":"2024-06-09T22:20:57.710013+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:47:04.463040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"99'7","reported_seq":243,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597416+0000","last_change":"2024-06-09T22:16:40.364834+0000","last_active":"2024-06-09T22:20:57.597416+0000","last_peered":"2024-06-09T22:20:57.597416+0000","last_clean":"2024-06-09T22:20:57.597416+0000","last_became_active":"2024-06-09T22:16:40.364433+0000","last_became_peered":"2024-06-09T22:16:40.364433+0000","last_unstale":"2024-06-09T22:20:57.597416+0000","last_undegraded":"2024-06-09T22:20:57.597416+0000","last_fullsized":"2024-06-09T22:20:57.597416+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:43:01.967911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":191,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597174+0000","last_change":"2024-06-09T22:16:43.201183+0000","last_active":"2024-06-09T22:20:57.597174+0000","last_peered":"2024-06-09T22:20:57.597174+0000","last_clean":"2024-06-09T22:20:57.597174+0000","last_became_active":"2024-06-09T22:16:43.200774+0000","last_became_peered":"2024-06-09T22:16:43.200774+0000","last_unstale":"2024-06-09T22:20:57.597174+0000","last_undegraded":"2024-06-09T22:20:57.597174+0000","last_fullsized":"2024-06-09T22:20:57.597174+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:07:55.772066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710583+0000","last_change":"2024-06-09T22:18:26.699190+0000","last_active":"2024-06-09T22:20:57.710583+0000","last_peered":"2024-06-09T22:20:57.710583+0000","last_clean":"2024-06-09T22:20:57.710583+0000","last_became_active":"2024-06-09T22:18:26.698614+0000","last_became_peered":"2024-06-09T22:18:26.698614+0000","last_unstale":"2024-06-09T22:20:57.710583+0000","last_undegraded":"2024-06-09T22:20:57.710583+0000","last_fullsized":"2024-06-09T22:20:57.710583+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:45.343605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,1,4],"acting":[0,10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"100'41","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.804410+0000","last_change":"2024-06-09T22:16:40.040582+0000","last_active":"2024-06-09T22:20:57.804410+0000","last_peered":"2024-06-09T22:20:57.804410+0000","last_clean":"2024-06-09T22:20:57.804410+0000","last_became_active":"2024-06-09T22:16:40.040152+0000","last_became_peered":"2024-06-09T22:16:40.040152+0000","last_unstale":"2024-06-09T22:20:57.804410+0000","last_undegraded":"2024-06-09T22:20:57.804410+0000","last_fullsized":"2024-06-09T22:20:57.804410+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:50:00.878096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596909+0000","last_change":"2024-06-09T22:16:43.206211+0000","last_active":"2024-06-09T22:20:57.596909+0000","last_peered":"2024-06-09T22:20:57.596909+0000","last_clean":"2024-06-09T22:20:57.596909+0000","last_became_active":"2024-06-09T22:16:43.205814+0000","last_became_peered":"2024-06-09T22:16:43.205814+0000","last_unstale":"2024-06-09T22:20:57.596909+0000","last_undegraded":"2024-06-09T22:20:57.596909+0000","last_fullsized":"2024-06-09T22:20:57.596909+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:20:19.058475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703043+0000","last_change":"2024-06-09T22:16:53.337695+0000","last_active":"2024-06-09T22:20:57.703043+0000","last_peered":"2024-06-09T22:20:57.703043+0000","last_clean":"2024-06-09T22:20:57.703043+0000","last_became_active":"2024-06-09T22:16:53.334872+0000","last_became_peered":"2024-06-09T22:16:53.334872+0000","last_unstale":"2024-06-09T22:20:57.703043+0000","last_undegraded":"2024-06-09T22:20:57.703043+0000","last_fullsized":"2024-06-09T22:20:57.703043+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:50:36.617736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596073+0000","last_change":"2024-06-09T22:16:40.036603+0000","last_active":"2024-06-09T22:20:57.596073+0000","last_peered":"2024-06-09T22:20:57.596073+0000","last_clean":"2024-06-09T22:20:57.596073+0000","last_became_active":"2024-06-09T22:16:40.036274+0000","last_became_peered":"2024-06-09T22:16:40.036274+0000","last_unstale":"2024-06-09T22:20:57.596073+0000","last_undegraded":"2024-06-09T22:20:57.596073+0000","last_fullsized":"2024-06-09T22:20:57.596073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:31.652897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"99'9","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596989+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:20:57.596989+0000","last_peered":"2024-06-09T22:20:57.596989+0000","last_clean":"2024-06-09T22:20:57.596989+0000","last_became_active":"2024-06-09T22:16:43.204197+0000","last_became_peered":"2024-06-09T22:16:43.204197+0000","last_unstale":"2024-06-09T22:20:57.596989+0000","last_undegraded":"2024-06-09T22:20:57.596989+0000","last_fullsized":"2024-06-09T22:20:57.596989+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:38:49.850120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710013+0000","last_change":"2024-06-09T22:16:43.208477+0000","last_active":"2024-06-09T22:20:57.710013+0000","last_peered":"2024-06-09T22:20:57.710013+0000","last_clean":"2024-06-09T22:20:57.710013+0000","last_became_active":"2024-06-09T22:16:43.206018+0000","last_became_peered":"2024-06-09T22:16:43.206018+0000","last_unstale":"2024-06-09T22:20:57.710013+0000","last_undegraded":"2024-06-09T22:20:57.710013+0000","last_fullsized":"2024-06-09T22:20:57.710013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:19:49.870825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"99'12","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597095+0000","last_change":"2024-06-09T22:16:43.206809+0000","last_active":"2024-06-09T22:20:57.597095+0000","last_peered":"2024-06-09T22:20:57.597095+0000","last_clean":"2024-06-09T22:20:57.597095+0000","last_became_active":"2024-06-09T22:16:43.206175+0000","last_became_peered":"2024-06-09T22:16:43.206175+0000","last_unstale":"2024-06-09T22:20:57.597095+0000","last_undegraded":"2024-06-09T22:20:57.597095+0000","last_fullsized":"2024-06-09T22:20:57.597095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:39:22.502266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596879+0000","last_change":"2024-06-09T22:16:43.202163+0000","last_active":"2024-06-09T22:20:57.596879+0000","last_peered":"2024-06-09T22:20:57.596879+0000","last_clean":"2024-06-09T22:20:57.596879+0000","last_became_active":"2024-06-09T22:16:43.201839+0000","last_became_peered":"2024-06-09T22:16:43.201839+0000","last_unstale":"2024-06-09T22:20:57.596879+0000","last_undegraded":"2024-06-09T22:20:57.596879+0000","last_fullsized":"2024-06-09T22:20:57.596879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:47:46.723289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710813+0000","last_change":"2024-06-09T22:16:43.202734+0000","last_active":"2024-06-09T22:20:57.710813+0000","last_peered":"2024-06-09T22:20:57.710813+0000","last_clean":"2024-06-09T22:20:57.710813+0000","last_became_active":"2024-06-09T22:16:43.201638+0000","last_became_peered":"2024-06-09T22:16:43.201638+0000","last_unstale":"2024-06-09T22:20:57.710813+0000","last_undegraded":"2024-06-09T22:20:57.710813+0000","last_fullsized":"2024-06-09T22:20:57.710813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:16.360985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"99'4","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595768+0000","last_change":"2024-06-09T22:16:43.199113+0000","last_active":"2024-06-09T22:20:57.595768+0000","last_peered":"2024-06-09T22:20:57.595768+0000","last_clean":"2024-06-09T22:20:57.595768+0000","last_became_active":"2024-06-09T22:16:43.198804+0000","last_became_peered":"2024-06-09T22:16:43.198804+0000","last_unstale":"2024-06-09T22:20:57.595768+0000","last_undegraded":"2024-06-09T22:20:57.595768+0000","last_fullsized":"2024-06-09T22:20:57.595768+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:18:08.999477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"99'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710339+0000","last_change":"2024-06-09T22:16:43.204445+0000","last_active":"2024-06-09T22:20:57.710339+0000","last_peered":"2024-06-09T22:20:57.710339+0000","last_clean":"2024-06-09T22:20:57.710339+0000","last_became_active":"2024-06-09T22:16:43.203081+0000","last_became_peered":"2024-06-09T22:16:43.203081+0000","last_unstale":"2024-06-09T22:20:57.710339+0000","last_undegraded":"2024-06-09T22:20:57.710339+0000","last_fullsized":"2024-06-09T22:20:57.710339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:39.303813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"99'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597592+0000","last_change":"2024-06-09T22:16:43.211402+0000","last_active":"2024-06-09T22:20:57.597592+0000","last_peered":"2024-06-09T22:20:57.597592+0000","last_clean":"2024-06-09T22:20:57.597592+0000","last_became_active":"2024-06-09T22:16:43.210871+0000","last_became_peered":"2024-06-09T22:16:43.210871+0000","last_unstale":"2024-06-09T22:20:57.597592+0000","last_undegraded":"2024-06-09T22:20:57.597592+0000","last_fullsized":"2024-06-09T22:20:57.597592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:07:21.467617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"99'2","reported_seq":42,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696237+0000","last_change":"2024-06-09T22:16:43.201159+0000","last_active":"2024-06-09T22:20:57.696237+0000","last_peered":"2024-06-09T22:20:57.696237+0000","last_clean":"2024-06-09T22:20:57.696237+0000","last_became_active":"2024-06-09T22:16:43.200850+0000","last_became_peered":"2024-06-09T22:16:43.200850+0000","last_unstale":"2024-06-09T22:20:57.696237+0000","last_undegraded":"2024-06-09T22:20:57.696237+0000","last_fullsized":"2024-06-09T22:20:57.696237+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:20.515309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"99'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703213+0000","last_change":"2024-06-09T22:16:43.196830+0000","last_active":"2024-06-09T22:20:57.703213+0000","last_peered":"2024-06-09T22:20:57.703213+0000","last_clean":"2024-06-09T22:20:57.703213+0000","last_became_active":"2024-06-09T22:16:43.195682+0000","last_became_peered":"2024-06-09T22:16:43.195682+0000","last_unstale":"2024-06-09T22:20:57.703213+0000","last_undegraded":"2024-06-09T22:20:57.703213+0000","last_fullsized":"2024-06-09T22:20:57.703213+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:34:18.770235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694921+0000","last_change":"2024-06-09T22:16:43.201959+0000","last_active":"2024-06-09T22:20:57.694921+0000","last_peered":"2024-06-09T22:20:57.694921+0000","last_clean":"2024-06-09T22:20:57.694921+0000","last_became_active":"2024-06-09T22:16:43.201669+0000","last_became_peered":"2024-06-09T22:16:43.201669+0000","last_unstale":"2024-06-09T22:20:57.694921+0000","last_undegraded":"2024-06-09T22:20:57.694921+0000","last_fullsized":"2024-06-09T22:20:57.694921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:48:57.348041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710198+0000","last_change":"2024-06-09T22:16:43.201823+0000","last_active":"2024-06-09T22:20:57.710198+0000","last_peered":"2024-06-09T22:20:57.710198+0000","last_clean":"2024-06-09T22:20:57.710198+0000","last_became_active":"2024-06-09T22:16:43.201434+0000","last_became_peered":"2024-06-09T22:16:43.201434+0000","last_unstale":"2024-06-09T22:20:57.710198+0000","last_undegraded":"2024-06-09T22:20:57.710198+0000","last_fullsized":"2024-06-09T22:20:57.710198+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:56:13.540895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"99'11","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595624+0000","last_change":"2024-06-09T22:16:43.197734+0000","last_active":"2024-06-09T22:20:57.595624+0000","last_peered":"2024-06-09T22:20:57.595624+0000","last_clean":"2024-06-09T22:20:57.595624+0000","last_became_active":"2024-06-09T22:16:43.197442+0000","last_became_peered":"2024-06-09T22:16:43.197442+0000","last_unstale":"2024-06-09T22:20:57.595624+0000","last_undegraded":"2024-06-09T22:20:57.595624+0000","last_fullsized":"2024-06-09T22:20:57.595624+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:35:40.720718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710339+0000","last_change":"2024-06-09T22:16:43.210375+0000","last_active":"2024-06-09T22:20:57.710339+0000","last_peered":"2024-06-09T22:20:57.710339+0000","last_clean":"2024-06-09T22:20:57.710339+0000","last_became_active":"2024-06-09T22:16:43.209647+0000","last_became_peered":"2024-06-09T22:16:43.209647+0000","last_unstale":"2024-06-09T22:20:57.710339+0000","last_undegraded":"2024-06-09T22:20:57.710339+0000","last_fullsized":"2024-06-09T22:20:57.710339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:11:57.593066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107643,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":607457,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2015,"num_read_kb":1129,"num_write":387,"num_write_kb":1481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":737280,"data_stored":1822761,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383747,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":580840,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645384,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32424,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712728,"statfs":{"total":95995035648,"available":95961833472,"internally_reserved":0,"allocated":1273856,"data_stored":1236852,"data_compressed":71140,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808909,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1167360,"data_stored":977483,"data_compressed":54405,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070546,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1159168,"data_stored":973765,"data_compressed":54405,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332183,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32340,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712812,"statfs":{"total":95995035648,"available":95961919488,"internally_reserved":0,"allocated":1187840,"data_stored":1178025,"data_compressed":71140,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495710,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1114112,"data_stored":576065,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36504,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708648,"statfs":{"total":95995035648,"available":95957655552,"internally_reserved":0,"allocated":1191936,"data_stored":650243,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708700,"statfs":{"total":95995035648,"available":95957708800,"internally_reserved":0,"allocated":1138688,"data_stored":577662,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":651847,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182507,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1171456,"data_stored":1100199,"data_compressed":60561,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920871,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32276,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712876,"statfs":{"total":95995035648,"available":95961985024,"internally_reserved":0,"allocated":1122304,"data_stored":580978,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757346,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36524,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708628,"statfs":{"total":95995035648,"available":95957635072,"internally_reserved":0,"allocated":1216512,"data_stored":1304758,"data_compressed":77296,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":75290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":76894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135341,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":208103,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-09T22:21:03.865 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-09T22:21:04.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.002+0000 7f7d320f5640 1 -- 172.21.15.104:0/757024744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c1008f0 msgr2=0x7f7d2c100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:04.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.002+0000 7f7d320f5640 1 --2- 172.21.15.104:0/757024744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c1008f0 0x7f7d2c100d70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7d20009930 tx=0x7f7d2002a420 comp rx=0 tx=0).stop 2024-06-09T22:21:04.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.002+0000 7f7d320f5640 1 -- 172.21.15.104:0/757024744 shutdown_connections 2024-06-09T22:21:04.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.002+0000 7f7d320f5640 1 --2- 172.21.15.104:0/757024744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d2c1012b0 0x7f7d2c10d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.002+0000 7f7d320f5640 1 --2- 172.21.15.104:0/757024744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c1008f0 0x7f7d2c100d70 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.002+0000 7f7d320f5640 1 --2- 172.21.15.104:0/757024744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d2c1068b0 0x7f7d2c106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.002+0000 7f7d320f5640 1 -- 172.21.15.104:0/757024744 >> 172.21.15.104:0/757024744 conn(0x7f7d2c077a00 msgr2=0x7f7d2c077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:04.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.002+0000 7f7d320f5640 1 -- 172.21.15.104:0/757024744 shutdown_connections 2024-06-09T22:21:04.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.002+0000 7f7d320f5640 1 -- 172.21.15.104:0/757024744 wait complete. 2024-06-09T22:21:04.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.002+0000 7f7d320f5640 1 Processor -- start 2024-06-09T22:21:04.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d320f5640 1 -- start start 2024-06-09T22:21:04.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d320f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d2c1008f0 0x7f7d2c197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:04.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d320f5640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d2c1012b0 0x7f7d2c197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:04.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d320f5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c1068b0 0x7f7d2c19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:04.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d320f5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2c110cc0 con 0x7f7d2c1068b0 2024-06-09T22:21:04.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d320f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2c110b40 con 0x7f7d2c1008f0 2024-06-09T22:21:04.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d320f5640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2c110e40 con 0x7f7d2c1012b0 2024-06-09T22:21:04.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d2bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c1068b0 0x7f7d2c19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:04.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d2bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c1068b0 0x7f7d2c19bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44436/0 (socket says 172.21.15.104:44436) 2024-06-09T22:21:04.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d2bfff640 1 -- 172.21.15.104:0/714932913 learned_addr learned my addr 172.21.15.104:0/714932913 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:04.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d2affd640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d2c1012b0 0x7f7d2c197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:04.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d2b7fe640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d2c1008f0 0x7f7d2c197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:04.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d2bfff640 1 -- 172.21.15.104:0/714932913 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d2c1012b0 msgr2=0x7f7d2c197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:04.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d2bfff640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d2c1012b0 0x7f7d2c197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d2bfff640 1 -- 172.21.15.104:0/714932913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d2c1008f0 msgr2=0x7f7d2c197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:04.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d2bfff640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d2c1008f0 0x7f7d2c197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d2bfff640 1 -- 172.21.15.104:0/714932913 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2c19c1f0 con 0x7f7d2c1068b0 2024-06-09T22:21:04.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d2b7fe640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d2c1008f0 0x7f7d2c197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:04.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.003+0000 7f7d2bfff640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c1068b0 0x7f7d2c19bac0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400ea60 tx=0x7f7d2400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:04.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.004+0000 7f7d28ff9640 1 -- 172.21.15.104:0/714932913 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2400ce40 con 0x7f7d2c1068b0 2024-06-09T22:21:04.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.004+0000 7f7d28ff9640 1 -- 172.21.15.104:0/714932913 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d24005020 con 0x7f7d2c1068b0 2024-06-09T22:21:04.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.004+0000 7f7d320f5640 1 -- 172.21.15.104:0/714932913 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d2c19c4e0 con 0x7f7d2c1068b0 2024-06-09T22:21:04.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.004+0000 7f7d28ff9640 1 -- 172.21.15.104:0/714932913 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24010690 con 0x7f7d2c1068b0 2024-06-09T22:21:04.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.004+0000 7f7d320f5640 1 -- 172.21.15.104:0/714932913 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d2c1a3d70 con 0x7f7d2c1068b0 2024-06-09T22:21:04.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.005+0000 7f7d320f5640 1 -- 172.21.15.104:0/714932913 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d2c068c10 con 0x7f7d2c1068b0 2024-06-09T22:21:04.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.006+0000 7f7d28ff9640 1 -- 172.21.15.104:0/714932913 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f7d24005520 con 0x7f7d2c1068b0 2024-06-09T22:21:04.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.006+0000 7f7d28ff9640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7d140a2e20 0x7f7d140a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:04.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.006+0000 7f7d28ff9640 1 -- 172.21.15.104:0/714932913 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7d240cffb0 con 0x7f7d2c1068b0 2024-06-09T22:21:04.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.006+0000 7f7d2b7fe640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7d140a2e20 0x7f7d140a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:04.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.007+0000 7f7d2b7fe640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7d140a2e20 0x7f7d140a52e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009ab0 tx=0x7f7d1c0094b0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:04.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.011+0000 7f7d28ff9640 1 -- 172.21.15.104:0/714932913 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7d2409c610 con 0x7f7d2c1068b0 2024-06-09T22:21:04.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.191+0000 7f7d320f5640 1 -- 172.21.15.104:0/714932913 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7d2c108ea0 con 0x7f7d140a2e20 2024-06-09T22:21:04.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.200+0000 7f7d28ff9640 1 -- 172.21.15.104:0/714932913 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339859 (secure 0 0 0) 0x7f7d1c00ea30 con 0x7f7d140a2e20 2024-06-09T22:21:04.206 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:04.208 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-06-09T22:21:04.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.203+0000 7f7d320f5640 1 -- 172.21.15.104:0/714932913 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7d140a2e20 msgr2=0x7f7d140a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:04.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.203+0000 7f7d320f5640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7d140a2e20 0x7f7d140a52e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009ab0 tx=0x7f7d1c0094b0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.203+0000 7f7d320f5640 1 -- 172.21.15.104:0/714932913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c1068b0 msgr2=0x7f7d2c19bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:04.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.203+0000 7f7d320f5640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c1068b0 0x7f7d2c19bac0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400ea60 tx=0x7f7d2400ef20 comp rx=0 tx=0).stop 2024-06-09T22:21:04.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.203+0000 7f7d320f5640 1 -- 172.21.15.104:0/714932913 shutdown_connections 2024-06-09T22:21:04.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.203+0000 7f7d320f5640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7d140a2e20 0x7f7d140a52e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.203+0000 7f7d320f5640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c1068b0 0x7f7d2c19bac0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.203+0000 7f7d320f5640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d2c1012b0 0x7f7d2c197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.203+0000 7f7d320f5640 1 --2- 172.21.15.104:0/714932913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d2c1008f0 0x7f7d2c197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.204+0000 7f7d320f5640 1 -- 172.21.15.104:0/714932913 >> 172.21.15.104:0/714932913 conn(0x7f7d2c077a00 msgr2=0x7f7d2c103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:04.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.204+0000 7f7d320f5640 1 -- 172.21.15.104:0/714932913 shutdown_connections 2024-06-09T22:21:04.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.204+0000 7f7d320f5640 1 -- 172.21.15.104:0/714932913 wait complete. 2024-06-09T22:21:04.214 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: Updating smithi104:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-09T22:21:04.214 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: Updating smithi153:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-09T22:21:04.214 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: Updating smithi194:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-09T22:21:04.214 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.214 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.215 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.215 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.215 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.215 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.215 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.215 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:21:04.216 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:04.216 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:04.216 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:04 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.217 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-06-09T22:21:03.609423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1198059,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2151,"num_read_kb":1248,"num_write":570,"num_write_kb":4421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":951,"ondisk_log_size":951,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383732,"kb_used_data":13740,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558092,"statfs":{"total":1151940427776,"available":1151547486208,"internally_reserved":0,"allocated":14069760,"data_stored":10388717,"data_compressed":600501,"data_compressed_allocated":815104,"data_compressed_original":4374528,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.989045"},"pg_stats":[{"pgid":"3.3f","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710445+0000","last_change":"2024-06-09T22:16:43.212998+0000","last_active":"2024-06-09T22:20:57.710445+0000","last_peered":"2024-06-09T22:20:57.710445+0000","last_clean":"2024-06-09T22:20:57.710445+0000","last_became_active":"2024-06-09T22:16:43.212614+0000","last_became_peered":"2024-06-09T22:16:43.212614+0000","last_unstale":"2024-06-09T22:20:57.710445+0000","last_undegraded":"2024-06-09T22:20:57.710445+0000","last_fullsized":"2024-06-09T22:20:57.710445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:22.728486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"99'4","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711160+0000","last_change":"2024-06-09T22:16:43.208746+0000","last_active":"2024-06-09T22:20:57.711160+0000","last_peered":"2024-06-09T22:20:57.711160+0000","last_clean":"2024-06-09T22:20:57.711160+0000","last_became_active":"2024-06-09T22:16:43.208458+0000","last_became_peered":"2024-06-09T22:16:43.208458+0000","last_unstale":"2024-06-09T22:20:57.711160+0000","last_undegraded":"2024-06-09T22:20:57.711160+0000","last_fullsized":"2024-06-09T22:20:57.711160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:24:35.350449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710878+0000","last_change":"2024-06-09T22:16:43.200770+0000","last_active":"2024-06-09T22:20:57.710878+0000","last_peered":"2024-06-09T22:20:57.710878+0000","last_clean":"2024-06-09T22:20:57.710878+0000","last_became_active":"2024-06-09T22:16:43.200449+0000","last_became_peered":"2024-06-09T22:16:43.200449+0000","last_unstale":"2024-06-09T22:20:57.710878+0000","last_undegraded":"2024-06-09T22:20:57.710878+0000","last_fullsized":"2024-06-09T22:20:57.710878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:51:09.017260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703458+0000","last_change":"2024-06-09T22:16:43.203844+0000","last_active":"2024-06-09T22:20:57.703458+0000","last_peered":"2024-06-09T22:20:57.703458+0000","last_clean":"2024-06-09T22:20:57.703458+0000","last_became_active":"2024-06-09T22:16:43.203431+0000","last_became_peered":"2024-06-09T22:16:43.203431+0000","last_unstale":"2024-06-09T22:20:57.703458+0000","last_undegraded":"2024-06-09T22:20:57.703458+0000","last_fullsized":"2024-06-09T22:20:57.703458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:51:36.913312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"99'4","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694538+0000","last_change":"2024-06-09T22:16:43.207597+0000","last_active":"2024-06-09T22:20:57.694538+0000","last_peered":"2024-06-09T22:20:57.694538+0000","last_clean":"2024-06-09T22:20:57.694538+0000","last_became_active":"2024-06-09T22:16:43.207266+0000","last_became_peered":"2024-06-09T22:16:43.207266+0000","last_unstale":"2024-06-09T22:20:57.694538+0000","last_undegraded":"2024-06-09T22:20:57.694538+0000","last_fullsized":"2024-06-09T22:20:57.694538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:47:05.558403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"99'6","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702988+0000","last_change":"2024-06-09T22:17:26.474121+0000","last_active":"2024-06-09T22:20:57.702988+0000","last_peered":"2024-06-09T22:20:57.702988+0000","last_clean":"2024-06-09T22:20:57.702988+0000","last_became_active":"2024-06-09T22:17:26.473619+0000","last_became_peered":"2024-06-09T22:17:26.473619+0000","last_unstale":"2024-06-09T22:20:57.702988+0000","last_undegraded":"2024-06-09T22:20:57.702988+0000","last_fullsized":"2024-06-09T22:20:57.702988+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:56:42.171235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"99'5","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596592+0000","last_change":"2024-06-09T22:16:43.208815+0000","last_active":"2024-06-09T22:20:57.596592+0000","last_peered":"2024-06-09T22:20:57.596592+0000","last_clean":"2024-06-09T22:20:57.596592+0000","last_became_active":"2024-06-09T22:16:43.208433+0000","last_became_peered":"2024-06-09T22:16:43.208433+0000","last_unstale":"2024-06-09T22:20:57.596592+0000","last_undegraded":"2024-06-09T22:20:57.596592+0000","last_fullsized":"2024-06-09T22:20:57.596592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:36:26.585037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694783+0000","last_change":"2024-06-09T22:16:43.210195+0000","last_active":"2024-06-09T22:20:57.694783+0000","last_peered":"2024-06-09T22:20:57.694783+0000","last_clean":"2024-06-09T22:20:57.694783+0000","last_became_active":"2024-06-09T22:16:43.209892+0000","last_became_peered":"2024-06-09T22:16:43.209892+0000","last_unstale":"2024-06-09T22:20:57.694783+0000","last_undegraded":"2024-06-09T22:20:57.694783+0000","last_fullsized":"2024-06-09T22:20:57.694783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:44.088212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710618+0000","last_change":"2024-06-09T22:16:43.202517+0000","last_active":"2024-06-09T22:20:57.710618+0000","last_peered":"2024-06-09T22:20:57.710618+0000","last_clean":"2024-06-09T22:20:57.710618+0000","last_became_active":"2024-06-09T22:16:43.201796+0000","last_became_peered":"2024-06-09T22:16:43.201796+0000","last_unstale":"2024-06-09T22:20:57.710618+0000","last_undegraded":"2024-06-09T22:20:57.710618+0000","last_fullsized":"2024-06-09T22:20:57.710618+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:30.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"99'16","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703409+0000","last_change":"2024-06-09T22:16:43.212732+0000","last_active":"2024-06-09T22:20:57.703409+0000","last_peered":"2024-06-09T22:20:57.703409+0000","last_clean":"2024-06-09T22:20:57.703409+0000","last_became_active":"2024-06-09T22:16:43.212048+0000","last_became_peered":"2024-06-09T22:16:43.212048+0000","last_unstale":"2024-06-09T22:20:57.703409+0000","last_undegraded":"2024-06-09T22:20:57.703409+0000","last_fullsized":"2024-06-09T22:20:57.703409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:47.383291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597025+0000","last_change":"2024-06-09T22:16:43.203778+0000","last_active":"2024-06-09T22:20:57.597025+0000","last_peered":"2024-06-09T22:20:57.597025+0000","last_clean":"2024-06-09T22:20:57.597025+0000","last_became_active":"2024-06-09T22:16:43.202954+0000","last_became_peered":"2024-06-09T22:16:43.202954+0000","last_unstale":"2024-06-09T22:20:57.597025+0000","last_undegraded":"2024-06-09T22:20:57.597025+0000","last_fullsized":"2024-06-09T22:20:57.597025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:56:53.708645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703526+0000","last_change":"2024-06-09T22:16:43.211425+0000","last_active":"2024-06-09T22:20:57.703526+0000","last_peered":"2024-06-09T22:20:57.703526+0000","last_clean":"2024-06-09T22:20:57.703526+0000","last_became_active":"2024-06-09T22:16:43.211001+0000","last_became_peered":"2024-06-09T22:16:43.211001+0000","last_unstale":"2024-06-09T22:20:57.703526+0000","last_undegraded":"2024-06-09T22:20:57.703526+0000","last_fullsized":"2024-06-09T22:20:57.703526+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:20:59.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"99'10","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595794+0000","last_change":"2024-06-09T22:16:43.208334+0000","last_active":"2024-06-09T22:20:57.595794+0000","last_peered":"2024-06-09T22:20:57.595794+0000","last_clean":"2024-06-09T22:20:57.595794+0000","last_became_active":"2024-06-09T22:16:43.207993+0000","last_became_peered":"2024-06-09T22:16:43.207993+0000","last_unstale":"2024-06-09T22:20:57.595794+0000","last_undegraded":"2024-06-09T22:20:57.595794+0000","last_fullsized":"2024-06-09T22:20:57.595794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:07:05.740748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710461+0000","last_change":"2024-06-09T22:16:43.207211+0000","last_active":"2024-06-09T22:20:57.710461+0000","last_peered":"2024-06-09T22:20:57.710461+0000","last_clean":"2024-06-09T22:20:57.710461+0000","last_became_active":"2024-06-09T22:16:43.206889+0000","last_became_peered":"2024-06-09T22:16:43.206889+0000","last_unstale":"2024-06-09T22:20:57.710461+0000","last_undegraded":"2024-06-09T22:20:57.710461+0000","last_fullsized":"2024-06-09T22:20:57.710461+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:25:47.461030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710404+0000","last_change":"2024-06-09T22:16:43.202202+0000","last_active":"2024-06-09T22:20:57.710404+0000","last_peered":"2024-06-09T22:20:57.710404+0000","last_clean":"2024-06-09T22:20:57.710404+0000","last_became_active":"2024-06-09T22:16:43.201834+0000","last_became_peered":"2024-06-09T22:16:43.201834+0000","last_unstale":"2024-06-09T22:20:57.710404+0000","last_undegraded":"2024-06-09T22:20:57.710404+0000","last_fullsized":"2024-06-09T22:20:57.710404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:31:51.296627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"99'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709927+0000","last_change":"2024-06-09T22:16:43.200637+0000","last_active":"2024-06-09T22:20:57.709927+0000","last_peered":"2024-06-09T22:20:57.709927+0000","last_clean":"2024-06-09T22:20:57.709927+0000","last_became_active":"2024-06-09T22:16:43.200268+0000","last_became_peered":"2024-06-09T22:16:43.200268+0000","last_unstale":"2024-06-09T22:20:57.709927+0000","last_undegraded":"2024-06-09T22:20:57.709927+0000","last_fullsized":"2024-06-09T22:20:57.709927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:25:59.881453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710229+0000","last_change":"2024-06-09T22:16:43.209212+0000","last_active":"2024-06-09T22:20:57.710229+0000","last_peered":"2024-06-09T22:20:57.710229+0000","last_clean":"2024-06-09T22:20:57.710229+0000","last_became_active":"2024-06-09T22:16:43.207789+0000","last_became_peered":"2024-06-09T22:16:43.207789+0000","last_unstale":"2024-06-09T22:20:57.710229+0000","last_undegraded":"2024-06-09T22:20:57.710229+0000","last_fullsized":"2024-06-09T22:20:57.710229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:17.316595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692756+0000","last_change":"2024-06-09T22:16:43.210935+0000","last_active":"2024-06-09T22:20:57.692756+0000","last_peered":"2024-06-09T22:20:57.692756+0000","last_clean":"2024-06-09T22:20:57.692756+0000","last_became_active":"2024-06-09T22:16:43.210631+0000","last_became_peered":"2024-06-09T22:16:43.210631+0000","last_unstale":"2024-06-09T22:20:57.692756+0000","last_undegraded":"2024-06-09T22:20:57.692756+0000","last_fullsized":"2024-06-09T22:20:57.692756+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:22:58.887313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"99'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597630+0000","last_change":"2024-06-09T22:16:43.198872+0000","last_active":"2024-06-09T22:20:57.597630+0000","last_peered":"2024-06-09T22:20:57.597630+0000","last_clean":"2024-06-09T22:20:57.597630+0000","last_became_active":"2024-06-09T22:16:43.198498+0000","last_became_peered":"2024-06-09T22:16:43.198498+0000","last_unstale":"2024-06-09T22:20:57.597630+0000","last_undegraded":"2024-06-09T22:20:57.597630+0000","last_fullsized":"2024-06-09T22:20:57.597630+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:25:28.853048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597223+0000","last_change":"2024-06-09T22:16:53.343656+0000","last_active":"2024-06-09T22:20:57.597223+0000","last_peered":"2024-06-09T22:20:57.597223+0000","last_clean":"2024-06-09T22:20:57.597223+0000","last_became_active":"2024-06-09T22:16:53.343281+0000","last_became_peered":"2024-06-09T22:16:53.343281+0000","last_unstale":"2024-06-09T22:20:57.597223+0000","last_undegraded":"2024-06-09T22:20:57.597223+0000","last_fullsized":"2024-06-09T22:20:57.597223+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:40.961842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"97'13","reported_seq":247,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.700687+0000","last_change":"2024-06-09T22:16:40.040145+0000","last_active":"2024-06-09T22:20:57.700687+0000","last_peered":"2024-06-09T22:20:57.700687+0000","last_clean":"2024-06-09T22:20:57.700687+0000","last_became_active":"2024-06-09T22:16:40.039787+0000","last_became_peered":"2024-06-09T22:16:40.039787+0000","last_unstale":"2024-06-09T22:20:57.700687+0000","last_undegraded":"2024-06-09T22:20:57.700687+0000","last_fullsized":"2024-06-09T22:20:57.700687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:52:32.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":188,"num_read_kb":188,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"99'5","reported_seq":23,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692731+0000","last_change":"2024-06-09T22:17:26.774053+0000","last_active":"2024-06-09T22:20:57.692731+0000","last_peered":"2024-06-09T22:20:57.692731+0000","last_clean":"2024-06-09T22:20:57.692731+0000","last_became_active":"2024-06-09T22:17:26.773786+0000","last_became_peered":"2024-06-09T22:17:26.773786+0000","last_unstale":"2024-06-09T22:20:57.692731+0000","last_undegraded":"2024-06-09T22:20:57.692731+0000","last_fullsized":"2024-06-09T22:20:57.692731+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:36.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703244+0000","last_change":"2024-06-09T22:16:53.332048+0000","last_active":"2024-06-09T22:20:57.703244+0000","last_peered":"2024-06-09T22:20:57.703244+0000","last_clean":"2024-06-09T22:20:57.703244+0000","last_became_active":"2024-06-09T22:16:53.330683+0000","last_became_peered":"2024-06-09T22:16:53.330683+0000","last_unstale":"2024-06-09T22:20:57.703244+0000","last_undegraded":"2024-06-09T22:20:57.703244+0000","last_fullsized":"2024-06-09T22:20:57.703244+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:32:23.219061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"97'2","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696082+0000","last_change":"2024-06-09T22:16:40.070193+0000","last_active":"2024-06-09T22:20:57.696082+0000","last_peered":"2024-06-09T22:20:57.696082+0000","last_clean":"2024-06-09T22:20:57.696082+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:20:57.696082+0000","last_undegraded":"2024-06-09T22:20:57.696082+0000","last_fullsized":"2024-06-09T22:20:57.696082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:19:19.787552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711015+0000","last_change":"2024-06-09T22:16:43.199814+0000","last_active":"2024-06-09T22:20:57.711015+0000","last_peered":"2024-06-09T22:20:57.711015+0000","last_clean":"2024-06-09T22:20:57.711015+0000","last_became_active":"2024-06-09T22:16:43.199487+0000","last_became_peered":"2024-06-09T22:16:43.199487+0000","last_unstale":"2024-06-09T22:20:57.711015+0000","last_undegraded":"2024-06-09T22:20:57.711015+0000","last_fullsized":"2024-06-09T22:20:57.711015+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:55.672050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703543+0000","last_change":"2024-06-09T22:16:53.336253+0000","last_active":"2024-06-09T22:20:57.703543+0000","last_peered":"2024-06-09T22:20:57.703543+0000","last_clean":"2024-06-09T22:20:57.703543+0000","last_became_active":"2024-06-09T22:16:53.335869+0000","last_became_peered":"2024-06-09T22:16:53.335869+0000","last_unstale":"2024-06-09T22:20:57.703543+0000","last_undegraded":"2024-06-09T22:20:57.703543+0000","last_fullsized":"2024-06-09T22:20:57.703543+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:28:12.894878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"97'12","reported_seq":60,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695155+0000","last_change":"2024-06-09T22:16:40.045365+0000","last_active":"2024-06-09T22:20:57.695155+0000","last_peered":"2024-06-09T22:20:57.695155+0000","last_clean":"2024-06-09T22:20:57.695155+0000","last_became_active":"2024-06-09T22:16:40.044989+0000","last_became_peered":"2024-06-09T22:16:40.044989+0000","last_unstale":"2024-06-09T22:20:57.695155+0000","last_undegraded":"2024-06-09T22:20:57.695155+0000","last_fullsized":"2024-06-09T22:20:57.695155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:01:05.850170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"99'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710115+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:20:57.710115+0000","last_peered":"2024-06-09T22:20:57.710115+0000","last_clean":"2024-06-09T22:20:57.710115+0000","last_became_active":"2024-06-09T22:16:43.203999+0000","last_became_peered":"2024-06-09T22:16:43.203999+0000","last_unstale":"2024-06-09T22:20:57.710115+0000","last_undegraded":"2024-06-09T22:20:57.710115+0000","last_fullsized":"2024-06-09T22:20:57.710115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:37:28.362655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703277+0000","last_change":"2024-06-09T22:16:53.333565+0000","last_active":"2024-06-09T22:20:57.703277+0000","last_peered":"2024-06-09T22:20:57.703277+0000","last_clean":"2024-06-09T22:20:57.703277+0000","last_became_active":"2024-06-09T22:16:53.332921+0000","last_became_peered":"2024-06-09T22:16:53.332921+0000","last_unstale":"2024-06-09T22:20:57.703277+0000","last_undegraded":"2024-06-09T22:20:57.703277+0000","last_fullsized":"2024-06-09T22:20:57.703277+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:32.085617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"97'5","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694916+0000","last_change":"2024-06-09T22:16:40.039766+0000","last_active":"2024-06-09T22:20:57.694916+0000","last_peered":"2024-06-09T22:20:57.694916+0000","last_clean":"2024-06-09T22:20:57.694916+0000","last_became_active":"2024-06-09T22:16:40.039329+0000","last_became_peered":"2024-06-09T22:16:40.039329+0000","last_unstale":"2024-06-09T22:20:57.694916+0000","last_undegraded":"2024-06-09T22:20:57.694916+0000","last_fullsized":"2024-06-09T22:20:57.694916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:53.673890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"99'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703385+0000","last_change":"2024-06-09T22:16:43.201200+0000","last_active":"2024-06-09T22:20:57.703385+0000","last_peered":"2024-06-09T22:20:57.703385+0000","last_clean":"2024-06-09T22:20:57.703385+0000","last_became_active":"2024-06-09T22:16:43.200830+0000","last_became_peered":"2024-06-09T22:16:43.200830+0000","last_unstale":"2024-06-09T22:20:57.703385+0000","last_undegraded":"2024-06-09T22:20:57.703385+0000","last_fullsized":"2024-06-09T22:20:57.703385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:38:52.093173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597461+0000","last_change":"2024-06-09T22:16:53.339811+0000","last_active":"2024-06-09T22:20:57.597461+0000","last_peered":"2024-06-09T22:20:57.597461+0000","last_clean":"2024-06-09T22:20:57.597461+0000","last_became_active":"2024-06-09T22:16:53.339305+0000","last_became_peered":"2024-06-09T22:16:53.339305+0000","last_unstale":"2024-06-09T22:20:57.597461+0000","last_undegraded":"2024-06-09T22:20:57.597461+0000","last_fullsized":"2024-06-09T22:20:57.597461+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:08:25.800245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"100'8","reported_seq":245,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.704478+0000","last_change":"2024-06-09T22:16:40.044159+0000","last_active":"2024-06-09T22:20:57.704478+0000","last_peered":"2024-06-09T22:20:57.704478+0000","last_clean":"2024-06-09T22:20:57.704478+0000","last_became_active":"2024-06-09T22:16:40.043839+0000","last_became_peered":"2024-06-09T22:16:40.043839+0000","last_unstale":"2024-06-09T22:20:57.704478+0000","last_undegraded":"2024-06-09T22:20:57.704478+0000","last_fullsized":"2024-06-09T22:20:57.704478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:22:09.339219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1543,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":191,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694583+0000","last_change":"2024-06-09T22:16:43.202015+0000","last_active":"2024-06-09T22:20:57.694583+0000","last_peered":"2024-06-09T22:20:57.694583+0000","last_clean":"2024-06-09T22:20:57.694583+0000","last_became_active":"2024-06-09T22:16:43.201689+0000","last_became_peered":"2024-06-09T22:16:43.201689+0000","last_unstale":"2024-06-09T22:20:57.694583+0000","last_undegraded":"2024-06-09T22:20:57.694583+0000","last_fullsized":"2024-06-09T22:20:57.694583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:35:12.180399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597147+0000","last_change":"2024-06-09T22:18:26.698600+0000","last_active":"2024-06-09T22:20:57.597147+0000","last_peered":"2024-06-09T22:20:57.597147+0000","last_clean":"2024-06-09T22:20:57.597147+0000","last_became_active":"2024-06-09T22:18:26.698201+0000","last_became_peered":"2024-06-09T22:18:26.698201+0000","last_unstale":"2024-06-09T22:20:57.597147+0000","last_undegraded":"2024-06-09T22:20:57.597147+0000","last_fullsized":"2024-06-09T22:20:57.597147+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:55:13.513557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"97'7","reported_seq":57,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702844+0000","last_change":"2024-06-09T22:17:26.698204+0000","last_active":"2024-06-09T22:20:57.702844+0000","last_peered":"2024-06-09T22:20:57.702844+0000","last_clean":"2024-06-09T22:20:57.702844+0000","last_became_active":"2024-06-09T22:17:26.697716+0000","last_became_peered":"2024-06-09T22:17:26.697716+0000","last_unstale":"2024-06-09T22:20:57.702844+0000","last_undegraded":"2024-06-09T22:20:57.702844+0000","last_fullsized":"2024-06-09T22:20:57.702844+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:36:37.724459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703554+0000","last_change":"2024-06-09T22:16:43.198917+0000","last_active":"2024-06-09T22:20:57.703554+0000","last_peered":"2024-06-09T22:20:57.703554+0000","last_clean":"2024-06-09T22:20:57.703554+0000","last_became_active":"2024-06-09T22:16:43.198567+0000","last_became_peered":"2024-06-09T22:16:43.198567+0000","last_unstale":"2024-06-09T22:20:57.703554+0000","last_undegraded":"2024-06-09T22:20:57.703554+0000","last_fullsized":"2024-06-09T22:20:57.703554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:34.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710203+0000","last_change":"2024-06-09T22:16:53.342203+0000","last_active":"2024-06-09T22:20:57.710203+0000","last_peered":"2024-06-09T22:20:57.710203+0000","last_clean":"2024-06-09T22:20:57.710203+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:20:57.710203+0000","last_undegraded":"2024-06-09T22:20:57.710203+0000","last_fullsized":"2024-06-09T22:20:57.710203+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:41:41.491961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"97'3","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595514+0000","last_change":"2024-06-09T22:16:40.038684+0000","last_active":"2024-06-09T22:20:57.595514+0000","last_peered":"2024-06-09T22:20:57.595514+0000","last_clean":"2024-06-09T22:20:57.595514+0000","last_became_active":"2024-06-09T22:16:40.038386+0000","last_became_peered":"2024-06-09T22:16:40.038386+0000","last_unstale":"2024-06-09T22:20:57.595514+0000","last_undegraded":"2024-06-09T22:20:57.595514+0000","last_fullsized":"2024-06-09T22:20:57.595514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:59.188758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694679+0000","last_change":"2024-06-09T22:16:43.198837+0000","last_active":"2024-06-09T22:20:57.694679+0000","last_peered":"2024-06-09T22:20:57.694679+0000","last_clean":"2024-06-09T22:20:57.694679+0000","last_became_active":"2024-06-09T22:16:43.198515+0000","last_became_peered":"2024-06-09T22:16:43.198515+0000","last_unstale":"2024-06-09T22:20:57.694679+0000","last_undegraded":"2024-06-09T22:20:57.694679+0000","last_fullsized":"2024-06-09T22:20:57.694679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:36:13.863202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711076+0000","last_change":"2024-06-09T22:16:53.342688+0000","last_active":"2024-06-09T22:20:57.711076+0000","last_peered":"2024-06-09T22:20:57.711076+0000","last_clean":"2024-06-09T22:20:57.711076+0000","last_became_active":"2024-06-09T22:16:53.342314+0000","last_became_peered":"2024-06-09T22:16:53.342314+0000","last_unstale":"2024-06-09T22:20:57.711076+0000","last_undegraded":"2024-06-09T22:20:57.711076+0000","last_fullsized":"2024-06-09T22:20:57.711076+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:51:35.801587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"97'3","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703264+0000","last_change":"2024-06-09T22:16:40.041831+0000","last_active":"2024-06-09T22:20:57.703264+0000","last_peered":"2024-06-09T22:20:57.703264+0000","last_clean":"2024-06-09T22:20:57.703264+0000","last_became_active":"2024-06-09T22:16:40.041484+0000","last_became_peered":"2024-06-09T22:16:40.041484+0000","last_unstale":"2024-06-09T22:20:57.703264+0000","last_undegraded":"2024-06-09T22:20:57.703264+0000","last_fullsized":"2024-06-09T22:20:57.703264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:10:19.829357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692213+0000","last_change":"2024-06-09T22:16:43.210769+0000","last_active":"2024-06-09T22:20:57.692213+0000","last_peered":"2024-06-09T22:20:57.692213+0000","last_clean":"2024-06-09T22:20:57.692213+0000","last_became_active":"2024-06-09T22:16:43.210429+0000","last_became_peered":"2024-06-09T22:16:43.210429+0000","last_unstale":"2024-06-09T22:20:57.692213+0000","last_undegraded":"2024-06-09T22:20:57.692213+0000","last_fullsized":"2024-06-09T22:20:57.692213+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:12:28.788830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694708+0000","last_change":"2024-06-09T22:16:53.336536+0000","last_active":"2024-06-09T22:20:57.694708+0000","last_peered":"2024-06-09T22:20:57.694708+0000","last_clean":"2024-06-09T22:20:57.694708+0000","last_became_active":"2024-06-09T22:16:53.335178+0000","last_became_peered":"2024-06-09T22:16:53.335178+0000","last_unstale":"2024-06-09T22:20:57.694708+0000","last_undegraded":"2024-06-09T22:20:57.694708+0000","last_fullsized":"2024-06-09T22:20:57.694708+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:57:19.476483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'7","reported_seq":425,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.704124+0000","last_change":"2024-06-09T22:16:40.038484+0000","last_active":"2024-06-09T22:20:57.704124+0000","last_peered":"2024-06-09T22:20:57.704124+0000","last_clean":"2024-06-09T22:20:57.704124+0000","last_became_active":"2024-06-09T22:16:40.038084+0000","last_became_peered":"2024-06-09T22:16:40.038084+0000","last_unstale":"2024-06-09T22:20:57.704124+0000","last_undegraded":"2024-06-09T22:20:57.704124+0000","last_fullsized":"2024-06-09T22:20:57.704124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:19.733107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":374,"num_read_kb":374,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710525+0000","last_change":"2024-06-09T22:16:43.205137+0000","last_active":"2024-06-09T22:20:57.710525+0000","last_peered":"2024-06-09T22:20:57.710525+0000","last_clean":"2024-06-09T22:20:57.710525+0000","last_became_active":"2024-06-09T22:16:43.204499+0000","last_became_peered":"2024-06-09T22:16:43.204499+0000","last_unstale":"2024-06-09T22:20:57.710525+0000","last_undegraded":"2024-06-09T22:20:57.710525+0000","last_fullsized":"2024-06-09T22:20:57.710525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:15.718493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692194+0000","last_change":"2024-06-09T22:16:53.344260+0000","last_active":"2024-06-09T22:20:57.692194+0000","last_peered":"2024-06-09T22:20:57.692194+0000","last_clean":"2024-06-09T22:20:57.692194+0000","last_became_active":"2024-06-09T22:16:53.343840+0000","last_became_peered":"2024-06-09T22:16:53.343840+0000","last_unstale":"2024-06-09T22:20:57.692194+0000","last_undegraded":"2024-06-09T22:20:57.692194+0000","last_fullsized":"2024-06-09T22:20:57.692194+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:40:23.305509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"97'4","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596747+0000","last_change":"2024-06-09T22:16:40.042660+0000","last_active":"2024-06-09T22:20:57.596747+0000","last_peered":"2024-06-09T22:20:57.596747+0000","last_clean":"2024-06-09T22:20:57.596747+0000","last_became_active":"2024-06-09T22:16:40.041353+0000","last_became_peered":"2024-06-09T22:16:40.041353+0000","last_unstale":"2024-06-09T22:20:57.596747+0000","last_undegraded":"2024-06-09T22:20:57.596747+0000","last_fullsized":"2024-06-09T22:20:57.596747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:34:59.064827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711158+0000","last_change":"2024-06-09T22:16:43.205083+0000","last_active":"2024-06-09T22:20:57.711158+0000","last_peered":"2024-06-09T22:20:57.711158+0000","last_clean":"2024-06-09T22:20:57.711158+0000","last_became_active":"2024-06-09T22:16:43.204600+0000","last_became_peered":"2024-06-09T22:16:43.204600+0000","last_unstale":"2024-06-09T22:20:57.711158+0000","last_undegraded":"2024-06-09T22:20:57.711158+0000","last_fullsized":"2024-06-09T22:20:57.711158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:05:12.555465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703699+0000","last_change":"2024-06-09T22:16:53.336169+0000","last_active":"2024-06-09T22:20:57.703699+0000","last_peered":"2024-06-09T22:20:57.703699+0000","last_clean":"2024-06-09T22:20:57.703699+0000","last_became_active":"2024-06-09T22:16:53.335657+0000","last_became_peered":"2024-06-09T22:16:53.335657+0000","last_unstale":"2024-06-09T22:20:57.703699+0000","last_undegraded":"2024-06-09T22:20:57.703699+0000","last_fullsized":"2024-06-09T22:20:57.703699+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:53:21.282767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703313+0000","last_change":"2024-06-09T22:16:40.070568+0000","last_active":"2024-06-09T22:20:57.703313+0000","last_peered":"2024-06-09T22:20:57.703313+0000","last_clean":"2024-06-09T22:20:57.703313+0000","last_became_active":"2024-06-09T22:16:40.070205+0000","last_became_peered":"2024-06-09T22:16:40.070205+0000","last_unstale":"2024-06-09T22:20:57.703313+0000","last_undegraded":"2024-06-09T22:20:57.703313+0000","last_fullsized":"2024-06-09T22:20:57.703313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.449965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710336+0000","last_change":"2024-06-09T22:16:43.205219+0000","last_active":"2024-06-09T22:20:57.710336+0000","last_peered":"2024-06-09T22:20:57.710336+0000","last_clean":"2024-06-09T22:20:57.710336+0000","last_became_active":"2024-06-09T22:16:43.203467+0000","last_became_peered":"2024-06-09T22:16:43.203467+0000","last_unstale":"2024-06-09T22:20:57.710336+0000","last_undegraded":"2024-06-09T22:20:57.710336+0000","last_fullsized":"2024-06-09T22:20:57.710336+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:22.784034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692165+0000","last_change":"2024-06-09T22:16:53.338990+0000","last_active":"2024-06-09T22:20:57.692165+0000","last_peered":"2024-06-09T22:20:57.692165+0000","last_clean":"2024-06-09T22:20:57.692165+0000","last_became_active":"2024-06-09T22:16:53.338568+0000","last_became_peered":"2024-06-09T22:16:53.338568+0000","last_unstale":"2024-06-09T22:20:57.692165+0000","last_undegraded":"2024-06-09T22:20:57.692165+0000","last_fullsized":"2024-06-09T22:20:57.692165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:46:39.814043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"100'44","reported_seq":473,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.846818+0000","last_change":"2024-06-09T22:16:40.044404+0000","last_active":"2024-06-09T22:20:57.846818+0000","last_peered":"2024-06-09T22:20:57.846818+0000","last_clean":"2024-06-09T22:20:57.846818+0000","last_became_active":"2024-06-09T22:16:40.044044+0000","last_became_peered":"2024-06-09T22:16:40.044044+0000","last_unstale":"2024-06-09T22:20:57.846818+0000","last_undegraded":"2024-06-09T22:20:57.846818+0000","last_fullsized":"2024-06-09T22:20:57.846818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:50:27.038759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73538,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":385,"num_read_kb":260,"num_write":46,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694903+0000","last_change":"2024-06-09T22:16:43.206963+0000","last_active":"2024-06-09T22:20:57.694903+0000","last_peered":"2024-06-09T22:20:57.694903+0000","last_clean":"2024-06-09T22:20:57.694903+0000","last_became_active":"2024-06-09T22:16:43.206628+0000","last_became_peered":"2024-06-09T22:16:43.206628+0000","last_unstale":"2024-06-09T22:20:57.694903+0000","last_undegraded":"2024-06-09T22:20:57.694903+0000","last_fullsized":"2024-06-09T22:20:57.694903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:33:39.912236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703845+0000","last_change":"2024-06-09T22:16:53.339332+0000","last_active":"2024-06-09T22:20:57.703845+0000","last_peered":"2024-06-09T22:20:57.703845+0000","last_clean":"2024-06-09T22:20:57.703845+0000","last_became_active":"2024-06-09T22:16:53.338819+0000","last_became_peered":"2024-06-09T22:16:53.338819+0000","last_unstale":"2024-06-09T22:20:57.703845+0000","last_undegraded":"2024-06-09T22:20:57.703845+0000","last_fullsized":"2024-06-09T22:20:57.703845+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:57:41.142142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"97'9","reported_seq":33,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709996+0000","last_change":"2024-06-09T22:17:26.478256+0000","last_active":"2024-06-09T22:20:57.709996+0000","last_peered":"2024-06-09T22:20:57.709996+0000","last_clean":"2024-06-09T22:20:57.709996+0000","last_became_active":"2024-06-09T22:17:26.472264+0000","last_became_peered":"2024-06-09T22:17:26.472264+0000","last_unstale":"2024-06-09T22:20:57.709996+0000","last_undegraded":"2024-06-09T22:20:57.709996+0000","last_fullsized":"2024-06-09T22:20:57.709996+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:14:19.919841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"99'9","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595647+0000","last_change":"2024-06-09T22:16:43.204484+0000","last_active":"2024-06-09T22:20:57.595647+0000","last_peered":"2024-06-09T22:20:57.595647+0000","last_clean":"2024-06-09T22:20:57.595647+0000","last_became_active":"2024-06-09T22:16:43.204014+0000","last_became_peered":"2024-06-09T22:16:43.204014+0000","last_unstale":"2024-06-09T22:20:57.595647+0000","last_undegraded":"2024-06-09T22:20:57.595647+0000","last_fullsized":"2024-06-09T22:20:57.595647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:55:12.528231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710227+0000","last_change":"2024-06-09T22:17:26.475179+0000","last_active":"2024-06-09T22:20:57.710227+0000","last_peered":"2024-06-09T22:20:57.710227+0000","last_clean":"2024-06-09T22:20:57.710227+0000","last_became_active":"2024-06-09T22:17:26.474517+0000","last_became_peered":"2024-06-09T22:17:26.474517+0000","last_unstale":"2024-06-09T22:20:57.710227+0000","last_undegraded":"2024-06-09T22:20:57.710227+0000","last_fullsized":"2024-06-09T22:20:57.710227+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:26:02.387830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703306+0000","last_change":"2024-06-09T22:16:40.043066+0000","last_active":"2024-06-09T22:20:57.703306+0000","last_peered":"2024-06-09T22:20:57.703306+0000","last_clean":"2024-06-09T22:20:57.703306+0000","last_became_active":"2024-06-09T22:16:40.042619+0000","last_became_peered":"2024-06-09T22:16:40.042619+0000","last_unstale":"2024-06-09T22:20:57.703306+0000","last_undegraded":"2024-06-09T22:20:57.703306+0000","last_fullsized":"2024-06-09T22:20:57.703306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:29.689614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"99'15","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597173+0000","last_change":"2024-06-09T22:17:26.475566+0000","last_active":"2024-06-09T22:20:57.597173+0000","last_peered":"2024-06-09T22:20:57.597173+0000","last_clean":"2024-06-09T22:20:57.597173+0000","last_became_active":"2024-06-09T22:17:26.475037+0000","last_became_peered":"2024-06-09T22:17:26.475037+0000","last_unstale":"2024-06-09T22:20:57.597173+0000","last_undegraded":"2024-06-09T22:20:57.597173+0000","last_fullsized":"2024-06-09T22:20:57.597173+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:33.075144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694959+0000","last_change":"2024-06-09T22:17:26.475111+0000","last_active":"2024-06-09T22:20:57.694959+0000","last_peered":"2024-06-09T22:20:57.694959+0000","last_clean":"2024-06-09T22:20:57.694959+0000","last_became_active":"2024-06-09T22:17:26.474563+0000","last_became_peered":"2024-06-09T22:17:26.474563+0000","last_unstale":"2024-06-09T22:20:57.694959+0000","last_undegraded":"2024-06-09T22:20:57.694959+0000","last_fullsized":"2024-06-09T22:20:57.694959+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:09:08.458761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"100'9","reported_seq":246,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703821+0000","last_change":"2024-06-09T22:16:40.040538+0000","last_active":"2024-06-09T22:20:57.703821+0000","last_peered":"2024-06-09T22:20:57.703821+0000","last_clean":"2024-06-09T22:20:57.703821+0000","last_became_active":"2024-06-09T22:16:40.039896+0000","last_became_peered":"2024-06-09T22:16:40.039896+0000","last_unstale":"2024-06-09T22:20:57.703821+0000","last_undegraded":"2024-06-09T22:20:57.703821+0000","last_fullsized":"2024-06-09T22:20:57.703821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:00:33.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1817,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":6,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"99'6","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596912+0000","last_change":"2024-06-09T22:17:26.475568+0000","last_active":"2024-06-09T22:20:57.596912+0000","last_peered":"2024-06-09T22:20:57.596912+0000","last_clean":"2024-06-09T22:20:57.596912+0000","last_became_active":"2024-06-09T22:17:26.475163+0000","last_became_peered":"2024-06-09T22:17:26.475163+0000","last_unstale":"2024-06-09T22:20:57.596912+0000","last_undegraded":"2024-06-09T22:20:57.596912+0000","last_fullsized":"2024-06-09T22:20:57.596912+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:20.865164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595970+0000","last_change":"2024-06-09T22:16:53.338201+0000","last_active":"2024-06-09T22:20:57.595970+0000","last_peered":"2024-06-09T22:20:57.595970+0000","last_clean":"2024-06-09T22:20:57.595970+0000","last_became_active":"2024-06-09T22:16:53.337816+0000","last_became_peered":"2024-06-09T22:16:53.337816+0000","last_unstale":"2024-06-09T22:20:57.595970+0000","last_undegraded":"2024-06-09T22:20:57.595970+0000","last_fullsized":"2024-06-09T22:20:57.595970+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:35:51.861135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"97'17","reported_seq":65,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694467+0000","last_change":"2024-06-09T22:16:40.070770+0000","last_active":"2024-06-09T22:20:57.694467+0000","last_peered":"2024-06-09T22:20:57.694467+0000","last_clean":"2024-06-09T22:20:57.694467+0000","last_became_active":"2024-06-09T22:16:40.070398+0000","last_became_peered":"2024-06-09T22:16:40.070398+0000","last_unstale":"2024-06-09T22:20:57.694467+0000","last_undegraded":"2024-06-09T22:20:57.694467+0000","last_fullsized":"2024-06-09T22:20:57.694467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:04:58.273601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"99'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703349+0000","last_change":"2024-06-09T22:16:43.199246+0000","last_active":"2024-06-09T22:20:57.703349+0000","last_peered":"2024-06-09T22:20:57.703349+0000","last_clean":"2024-06-09T22:20:57.703349+0000","last_became_active":"2024-06-09T22:16:43.198006+0000","last_became_peered":"2024-06-09T22:16:43.198006+0000","last_unstale":"2024-06-09T22:20:57.703349+0000","last_undegraded":"2024-06-09T22:20:57.703349+0000","last_fullsized":"2024-06-09T22:20:57.703349+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:30:55.836628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703251+0000","last_change":"2024-06-09T22:16:53.342217+0000","last_active":"2024-06-09T22:20:57.703251+0000","last_peered":"2024-06-09T22:20:57.703251+0000","last_clean":"2024-06-09T22:20:57.703251+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:20:57.703251+0000","last_undegraded":"2024-06-09T22:20:57.703251+0000","last_fullsized":"2024-06-09T22:20:57.703251+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:25:28.635091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"100'4","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695066+0000","last_change":"2024-06-09T22:16:40.039841+0000","last_active":"2024-06-09T22:20:57.695066+0000","last_peered":"2024-06-09T22:20:57.695066+0000","last_clean":"2024-06-09T22:20:57.695066+0000","last_became_active":"2024-06-09T22:16:40.039416+0000","last_became_peered":"2024-06-09T22:16:40.039416+0000","last_unstale":"2024-06-09T22:20:57.695066+0000","last_undegraded":"2024-06-09T22:20:57.695066+0000","last_fullsized":"2024-06-09T22:20:57.695066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:27:36.213839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"99'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710082+0000","last_change":"2024-06-09T22:16:43.214363+0000","last_active":"2024-06-09T22:20:57.710082+0000","last_peered":"2024-06-09T22:20:57.710082+0000","last_clean":"2024-06-09T22:20:57.710082+0000","last_became_active":"2024-06-09T22:16:43.214015+0000","last_became_peered":"2024-06-09T22:16:43.214015+0000","last_unstale":"2024-06-09T22:20:57.710082+0000","last_undegraded":"2024-06-09T22:20:57.710082+0000","last_fullsized":"2024-06-09T22:20:57.710082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:13:06.924468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694673+0000","last_change":"2024-06-09T22:16:53.342151+0000","last_active":"2024-06-09T22:20:57.694673+0000","last_peered":"2024-06-09T22:20:57.694673+0000","last_clean":"2024-06-09T22:20:57.694673+0000","last_became_active":"2024-06-09T22:16:53.341743+0000","last_became_peered":"2024-06-09T22:16:53.341743+0000","last_unstale":"2024-06-09T22:20:57.694673+0000","last_undegraded":"2024-06-09T22:20:57.694673+0000","last_fullsized":"2024-06-09T22:20:57.694673+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:39:44.793301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"97'1","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692594+0000","last_change":"2024-06-09T22:17:26.472397+0000","last_active":"2024-06-09T22:20:57.692594+0000","last_peered":"2024-06-09T22:20:57.692594+0000","last_clean":"2024-06-09T22:20:57.692594+0000","last_became_active":"2024-06-09T22:17:26.471950+0000","last_became_peered":"2024-06-09T22:17:26.471950+0000","last_unstale":"2024-06-09T22:20:57.692594+0000","last_undegraded":"2024-06-09T22:20:57.692594+0000","last_fullsized":"2024-06-09T22:20:57.692594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:35:57.481510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"99'4","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703072+0000","last_change":"2024-06-09T22:16:43.214976+0000","last_active":"2024-06-09T22:20:57.703072+0000","last_peered":"2024-06-09T22:20:57.703072+0000","last_clean":"2024-06-09T22:20:57.703072+0000","last_became_active":"2024-06-09T22:16:43.214592+0000","last_became_peered":"2024-06-09T22:16:43.214592+0000","last_unstale":"2024-06-09T22:20:57.703072+0000","last_undegraded":"2024-06-09T22:20:57.703072+0000","last_fullsized":"2024-06-09T22:20:57.703072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:16:21.981144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595597+0000","last_change":"2024-06-09T22:16:53.342721+0000","last_active":"2024-06-09T22:20:57.595597+0000","last_peered":"2024-06-09T22:20:57.595597+0000","last_clean":"2024-06-09T22:20:57.595597+0000","last_became_active":"2024-06-09T22:16:53.342342+0000","last_became_peered":"2024-06-09T22:16:53.342342+0000","last_unstale":"2024-06-09T22:20:57.595597+0000","last_undegraded":"2024-06-09T22:20:57.595597+0000","last_fullsized":"2024-06-09T22:20:57.595597+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:22:07.469873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"97'5","reported_seq":241,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596811+0000","last_change":"2024-06-09T22:16:40.041307+0000","last_active":"2024-06-09T22:20:57.596811+0000","last_peered":"2024-06-09T22:20:57.596811+0000","last_clean":"2024-06-09T22:20:57.596811+0000","last_became_active":"2024-06-09T22:16:40.040819+0000","last_became_peered":"2024-06-09T22:16:40.040819+0000","last_unstale":"2024-06-09T22:20:57.596811+0000","last_undegraded":"2024-06-09T22:20:57.596811+0000","last_fullsized":"2024-06-09T22:20:57.596811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:14:29.658293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"99'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702925+0000","last_change":"2024-06-09T22:16:43.215719+0000","last_active":"2024-06-09T22:20:57.702925+0000","last_peered":"2024-06-09T22:20:57.702925+0000","last_clean":"2024-06-09T22:20:57.702925+0000","last_became_active":"2024-06-09T22:16:43.215356+0000","last_became_peered":"2024-06-09T22:16:43.215356+0000","last_unstale":"2024-06-09T22:20:57.702925+0000","last_undegraded":"2024-06-09T22:20:57.702925+0000","last_fullsized":"2024-06-09T22:20:57.702925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:15:07.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596001+0000","last_change":"2024-06-09T22:16:53.342948+0000","last_active":"2024-06-09T22:20:57.596001+0000","last_peered":"2024-06-09T22:20:57.596001+0000","last_clean":"2024-06-09T22:20:57.596001+0000","last_became_active":"2024-06-09T22:16:53.342352+0000","last_became_peered":"2024-06-09T22:16:53.342352+0000","last_unstale":"2024-06-09T22:20:57.596001+0000","last_undegraded":"2024-06-09T22:20:57.596001+0000","last_fullsized":"2024-06-09T22:20:57.596001+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:43:27.453789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703458+0000","last_change":"2024-06-09T22:16:43.204897+0000","last_active":"2024-06-09T22:20:57.703458+0000","last_peered":"2024-06-09T22:20:57.703458+0000","last_clean":"2024-06-09T22:20:57.703458+0000","last_became_active":"2024-06-09T22:16:43.204470+0000","last_became_peered":"2024-06-09T22:16:43.204470+0000","last_unstale":"2024-06-09T22:20:57.703458+0000","last_undegraded":"2024-06-09T22:20:57.703458+0000","last_fullsized":"2024-06-09T22:20:57.703458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:56.681392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"99'7","reported_seq":248,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:01.708116+0000","last_change":"2024-06-09T22:16:40.070079+0000","last_active":"2024-06-09T22:21:01.708116+0000","last_peered":"2024-06-09T22:21:01.708116+0000","last_clean":"2024-06-09T22:21:01.708116+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:21:01.708116+0000","last_undegraded":"2024-06-09T22:21:01.708116+0000","last_fullsized":"2024-06-09T22:21:01.708116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:00:45.132139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":193,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"97'1","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596821+0000","last_change":"2024-06-09T22:16:53.338891+0000","last_active":"2024-06-09T22:20:57.596821+0000","last_peered":"2024-06-09T22:20:57.596821+0000","last_clean":"2024-06-09T22:20:57.596821+0000","last_became_active":"2024-06-09T22:16:53.338526+0000","last_became_peered":"2024-06-09T22:16:53.338526+0000","last_unstale":"2024-06-09T22:20:57.596821+0000","last_undegraded":"2024-06-09T22:20:57.596821+0000","last_fullsized":"2024-06-09T22:20:57.596821+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:34:37.004407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695269+0000","last_change":"2024-06-09T22:16:43.199234+0000","last_active":"2024-06-09T22:20:57.695269+0000","last_peered":"2024-06-09T22:20:57.695269+0000","last_clean":"2024-06-09T22:20:57.695269+0000","last_became_active":"2024-06-09T22:16:43.192012+0000","last_became_peered":"2024-06-09T22:16:43.192012+0000","last_unstale":"2024-06-09T22:20:57.695269+0000","last_undegraded":"2024-06-09T22:20:57.695269+0000","last_fullsized":"2024-06-09T22:20:57.695269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:17.429236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"97'6","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703645+0000","last_change":"2024-06-09T22:16:40.041970+0000","last_active":"2024-06-09T22:20:57.703645+0000","last_peered":"2024-06-09T22:20:57.703645+0000","last_clean":"2024-06-09T22:20:57.703645+0000","last_became_active":"2024-06-09T22:16:40.040371+0000","last_became_peered":"2024-06-09T22:16:40.040371+0000","last_unstale":"2024-06-09T22:20:57.703645+0000","last_undegraded":"2024-06-09T22:20:57.703645+0000","last_fullsized":"2024-06-09T22:20:57.703645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:52:31.210423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":381,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596577+0000","last_change":"2024-06-09T22:15:15.367397+0000","last_active":"2024-06-09T22:20:57.596577+0000","last_peered":"2024-06-09T22:20:57.596577+0000","last_clean":"2024-06-09T22:20:57.596577+0000","last_became_active":"2024-06-09T22:15:15.366974+0000","last_became_peered":"2024-06-09T22:15:15.366974+0000","last_unstale":"2024-06-09T22:20:57.596577+0000","last_undegraded":"2024-06-09T22:20:57.596577+0000","last_fullsized":"2024-06-09T22:20:57.596577+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_clean_scrub_stamp":"2024-06-09T22:11:49.886056+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:24.018031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703964+0000","last_change":"2024-06-09T22:16:53.342249+0000","last_active":"2024-06-09T22:20:57.703964+0000","last_peered":"2024-06-09T22:20:57.703964+0000","last_clean":"2024-06-09T22:20:57.703964+0000","last_became_active":"2024-06-09T22:16:53.341779+0000","last_became_peered":"2024-06-09T22:16:53.341779+0000","last_unstale":"2024-06-09T22:20:57.703964+0000","last_undegraded":"2024-06-09T22:20:57.703964+0000","last_fullsized":"2024-06-09T22:20:57.703964+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:21:21.101578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"99'5","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595425+0000","last_change":"2024-06-09T22:16:43.200778+0000","last_active":"2024-06-09T22:20:57.595425+0000","last_peered":"2024-06-09T22:20:57.595425+0000","last_clean":"2024-06-09T22:20:57.595425+0000","last_became_active":"2024-06-09T22:16:43.200490+0000","last_became_peered":"2024-06-09T22:16:43.200490+0000","last_unstale":"2024-06-09T22:20:57.595425+0000","last_undegraded":"2024-06-09T22:20:57.595425+0000","last_fullsized":"2024-06-09T22:20:57.595425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:20:17.530204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"97'4","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596731+0000","last_change":"2024-06-09T22:16:40.041774+0000","last_active":"2024-06-09T22:20:57.596731+0000","last_peered":"2024-06-09T22:20:57.596731+0000","last_clean":"2024-06-09T22:20:57.596731+0000","last_became_active":"2024-06-09T22:16:40.041085+0000","last_became_peered":"2024-06-09T22:16:40.041085+0000","last_unstale":"2024-06-09T22:20:57.596731+0000","last_undegraded":"2024-06-09T22:20:57.596731+0000","last_fullsized":"2024-06-09T22:20:57.596731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:09:41.140083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710062+0000","last_change":"2024-06-09T22:16:53.336895+0000","last_active":"2024-06-09T22:20:57.710062+0000","last_peered":"2024-06-09T22:20:57.710062+0000","last_clean":"2024-06-09T22:20:57.710062+0000","last_became_active":"2024-06-09T22:16:53.336497+0000","last_became_peered":"2024-06-09T22:16:53.336497+0000","last_unstale":"2024-06-09T22:20:57.710062+0000","last_undegraded":"2024-06-09T22:20:57.710062+0000","last_fullsized":"2024-06-09T22:20:57.710062+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:29:52.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"97'4","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695101+0000","last_change":"2024-06-09T22:16:40.070909+0000","last_active":"2024-06-09T22:20:57.695101+0000","last_peered":"2024-06-09T22:20:57.695101+0000","last_clean":"2024-06-09T22:20:57.695101+0000","last_became_active":"2024-06-09T22:16:40.070556+0000","last_became_peered":"2024-06-09T22:16:40.070556+0000","last_unstale":"2024-06-09T22:20:57.695101+0000","last_undegraded":"2024-06-09T22:20:57.695101+0000","last_fullsized":"2024-06-09T22:20:57.695101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:59:17.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"99'9","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595793+0000","last_change":"2024-06-09T22:16:43.205951+0000","last_active":"2024-06-09T22:20:57.595793+0000","last_peered":"2024-06-09T22:20:57.595793+0000","last_clean":"2024-06-09T22:20:57.595793+0000","last_became_active":"2024-06-09T22:16:43.205594+0000","last_became_peered":"2024-06-09T22:16:43.205594+0000","last_unstale":"2024-06-09T22:20:57.595793+0000","last_undegraded":"2024-06-09T22:20:57.595793+0000","last_fullsized":"2024-06-09T22:20:57.595793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:17.956406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694724+0000","last_change":"2024-06-09T22:16:53.343292+0000","last_active":"2024-06-09T22:20:57.694724+0000","last_peered":"2024-06-09T22:20:57.694724+0000","last_clean":"2024-06-09T22:20:57.694724+0000","last_became_active":"2024-06-09T22:16:53.342900+0000","last_became_peered":"2024-06-09T22:16:53.342900+0000","last_unstale":"2024-06-09T22:20:57.694724+0000","last_undegraded":"2024-06-09T22:20:57.694724+0000","last_fullsized":"2024-06-09T22:20:57.694724+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:08:07.328250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"100'51","reported_seq":120,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.812147+0000","last_change":"2024-06-09T22:16:40.039417+0000","last_active":"2024-06-09T22:20:57.812147+0000","last_peered":"2024-06-09T22:20:57.812147+0000","last_clean":"2024-06-09T22:20:57.812147+0000","last_became_active":"2024-06-09T22:16:40.039099+0000","last_became_peered":"2024-06-09T22:16:40.039099+0000","last_unstale":"2024-06-09T22:20:57.812147+0000","last_undegraded":"2024-06-09T22:20:57.812147+0000","last_fullsized":"2024-06-09T22:20:57.812147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:22:59.284442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":90,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"99'5","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696131+0000","last_change":"2024-06-09T22:16:43.201285+0000","last_active":"2024-06-09T22:20:57.696131+0000","last_peered":"2024-06-09T22:20:57.696131+0000","last_clean":"2024-06-09T22:20:57.696131+0000","last_became_active":"2024-06-09T22:16:43.200993+0000","last_became_peered":"2024-06-09T22:16:43.200993+0000","last_unstale":"2024-06-09T22:20:57.696131+0000","last_undegraded":"2024-06-09T22:20:57.696131+0000","last_fullsized":"2024-06-09T22:20:57.696131+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:15:26.491588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596798+0000","last_change":"2024-06-09T22:16:53.334088+0000","last_active":"2024-06-09T22:20:57.596798+0000","last_peered":"2024-06-09T22:20:57.596798+0000","last_clean":"2024-06-09T22:20:57.596798+0000","last_became_active":"2024-06-09T22:16:53.333597+0000","last_became_peered":"2024-06-09T22:16:53.333597+0000","last_unstale":"2024-06-09T22:20:57.596798+0000","last_undegraded":"2024-06-09T22:20:57.596798+0000","last_fullsized":"2024-06-09T22:20:57.596798+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:50:22.041816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"97'7","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703700+0000","last_change":"2024-06-09T22:16:40.364982+0000","last_active":"2024-06-09T22:20:57.703700+0000","last_peered":"2024-06-09T22:20:57.703700+0000","last_clean":"2024-06-09T22:20:57.703700+0000","last_became_active":"2024-06-09T22:16:40.364585+0000","last_became_peered":"2024-06-09T22:16:40.364585+0000","last_unstale":"2024-06-09T22:20:57.703700+0000","last_undegraded":"2024-06-09T22:20:57.703700+0000","last_fullsized":"2024-06-09T22:20:57.703700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:28.777950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"99'7","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711181+0000","last_change":"2024-06-09T22:17:26.474491+0000","last_active":"2024-06-09T22:20:57.711181+0000","last_peered":"2024-06-09T22:20:57.711181+0000","last_clean":"2024-06-09T22:20:57.711181+0000","last_became_active":"2024-06-09T22:17:26.473975+0000","last_became_peered":"2024-06-09T22:17:26.473975+0000","last_unstale":"2024-06-09T22:20:57.711181+0000","last_undegraded":"2024-06-09T22:20:57.711181+0000","last_fullsized":"2024-06-09T22:20:57.711181+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:13:46.998595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703384+0000","last_change":"2024-06-09T22:16:53.342273+0000","last_active":"2024-06-09T22:20:57.703384+0000","last_peered":"2024-06-09T22:20:57.703384+0000","last_clean":"2024-06-09T22:20:57.703384+0000","last_became_active":"2024-06-09T22:16:53.341827+0000","last_became_peered":"2024-06-09T22:16:53.341827+0000","last_unstale":"2024-06-09T22:20:57.703384+0000","last_undegraded":"2024-06-09T22:20:57.703384+0000","last_fullsized":"2024-06-09T22:20:57.703384+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:23:26.099737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"97'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709982+0000","last_change":"2024-06-09T22:16:40.070498+0000","last_active":"2024-06-09T22:20:57.709982+0000","last_peered":"2024-06-09T22:20:57.709982+0000","last_clean":"2024-06-09T22:20:57.709982+0000","last_became_active":"2024-06-09T22:16:40.070144+0000","last_became_peered":"2024-06-09T22:16:40.070144+0000","last_unstale":"2024-06-09T22:20:57.709982+0000","last_undegraded":"2024-06-09T22:20:57.709982+0000","last_fullsized":"2024-06-09T22:20:57.709982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:52:22.214832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"99'2","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694467+0000","last_change":"2024-06-09T22:16:43.210448+0000","last_active":"2024-06-09T22:20:57.694467+0000","last_peered":"2024-06-09T22:20:57.694467+0000","last_clean":"2024-06-09T22:20:57.694467+0000","last_became_active":"2024-06-09T22:16:43.209980+0000","last_became_peered":"2024-06-09T22:16:43.209980+0000","last_unstale":"2024-06-09T22:20:57.694467+0000","last_undegraded":"2024-06-09T22:20:57.694467+0000","last_fullsized":"2024-06-09T22:20:57.694467+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:58:15.691207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692864+0000","last_change":"2024-06-09T22:16:53.340644+0000","last_active":"2024-06-09T22:20:57.692864+0000","last_peered":"2024-06-09T22:20:57.692864+0000","last_clean":"2024-06-09T22:20:57.692864+0000","last_became_active":"2024-06-09T22:16:53.339771+0000","last_became_peered":"2024-06-09T22:16:53.339771+0000","last_unstale":"2024-06-09T22:20:57.692864+0000","last_undegraded":"2024-06-09T22:20:57.692864+0000","last_fullsized":"2024-06-09T22:20:57.692864+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:45:23.281387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"97'7","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694859+0000","last_change":"2024-06-09T22:16:40.040461+0000","last_active":"2024-06-09T22:20:57.694859+0000","last_peered":"2024-06-09T22:20:57.694859+0000","last_clean":"2024-06-09T22:20:57.694859+0000","last_became_active":"2024-06-09T22:16:40.040168+0000","last_became_peered":"2024-06-09T22:16:40.040168+0000","last_unstale":"2024-06-09T22:20:57.694859+0000","last_undegraded":"2024-06-09T22:20:57.694859+0000","last_fullsized":"2024-06-09T22:20:57.694859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:28.936631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"99'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709897+0000","last_change":"2024-06-09T22:16:43.208779+0000","last_active":"2024-06-09T22:20:57.709897+0000","last_peered":"2024-06-09T22:20:57.709897+0000","last_clean":"2024-06-09T22:20:57.709897+0000","last_became_active":"2024-06-09T22:16:43.208136+0000","last_became_peered":"2024-06-09T22:16:43.208136+0000","last_unstale":"2024-06-09T22:20:57.709897+0000","last_undegraded":"2024-06-09T22:20:57.709897+0000","last_fullsized":"2024-06-09T22:20:57.709897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:22:39.721069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"97'1","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703384+0000","last_change":"2024-06-09T22:16:53.331241+0000","last_active":"2024-06-09T22:20:57.703384+0000","last_peered":"2024-06-09T22:20:57.703384+0000","last_clean":"2024-06-09T22:20:57.703384+0000","last_became_active":"2024-06-09T22:16:53.329589+0000","last_became_peered":"2024-06-09T22:16:53.329589+0000","last_unstale":"2024-06-09T22:20:57.703384+0000","last_undegraded":"2024-06-09T22:20:57.703384+0000","last_fullsized":"2024-06-09T22:20:57.703384+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:56:21.319711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"100'4","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596918+0000","last_change":"2024-06-09T22:16:40.042491+0000","last_active":"2024-06-09T22:20:57.596918+0000","last_peered":"2024-06-09T22:20:57.596918+0000","last_clean":"2024-06-09T22:20:57.596918+0000","last_became_active":"2024-06-09T22:16:40.041846+0000","last_became_peered":"2024-06-09T22:16:40.041846+0000","last_unstale":"2024-06-09T22:20:57.596918+0000","last_undegraded":"2024-06-09T22:20:57.596918+0000","last_fullsized":"2024-06-09T22:20:57.596918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:59.144003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695844+0000","last_change":"2024-06-09T22:16:43.209195+0000","last_active":"2024-06-09T22:20:57.695844+0000","last_peered":"2024-06-09T22:20:57.695844+0000","last_clean":"2024-06-09T22:20:57.695844+0000","last_became_active":"2024-06-09T22:16:43.208797+0000","last_became_peered":"2024-06-09T22:16:43.208797+0000","last_unstale":"2024-06-09T22:20:57.695844+0000","last_undegraded":"2024-06-09T22:20:57.695844+0000","last_fullsized":"2024-06-09T22:20:57.695844+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:05:31.689900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703290+0000","last_change":"2024-06-09T22:16:53.342173+0000","last_active":"2024-06-09T22:20:57.703290+0000","last_peered":"2024-06-09T22:20:57.703290+0000","last_clean":"2024-06-09T22:20:57.703290+0000","last_became_active":"2024-06-09T22:16:53.341712+0000","last_became_peered":"2024-06-09T22:16:53.341712+0000","last_unstale":"2024-06-09T22:20:57.703290+0000","last_undegraded":"2024-06-09T22:20:57.703290+0000","last_fullsized":"2024-06-09T22:20:57.703290+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:53:12.574764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"97'4","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694569+0000","last_change":"2024-06-09T22:16:40.044238+0000","last_active":"2024-06-09T22:20:57.694569+0000","last_peered":"2024-06-09T22:20:57.694569+0000","last_clean":"2024-06-09T22:20:57.694569+0000","last_became_active":"2024-06-09T22:16:40.043857+0000","last_became_peered":"2024-06-09T22:16:40.043857+0000","last_unstale":"2024-06-09T22:20:57.694569+0000","last_undegraded":"2024-06-09T22:20:57.694569+0000","last_fullsized":"2024-06-09T22:20:57.694569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:44:45.743307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"99'5","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596886+0000","last_change":"2024-06-09T22:16:43.202363+0000","last_active":"2024-06-09T22:20:57.596886+0000","last_peered":"2024-06-09T22:20:57.596886+0000","last_clean":"2024-06-09T22:20:57.596886+0000","last_became_active":"2024-06-09T22:16:43.202029+0000","last_became_peered":"2024-06-09T22:16:43.202029+0000","last_unstale":"2024-06-09T22:20:57.596886+0000","last_undegraded":"2024-06-09T22:20:57.596886+0000","last_fullsized":"2024-06-09T22:20:57.596886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:11:19.513055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710013+0000","last_change":"2024-06-09T22:16:53.333837+0000","last_active":"2024-06-09T22:20:57.710013+0000","last_peered":"2024-06-09T22:20:57.710013+0000","last_clean":"2024-06-09T22:20:57.710013+0000","last_became_active":"2024-06-09T22:16:53.333362+0000","last_became_peered":"2024-06-09T22:16:53.333362+0000","last_unstale":"2024-06-09T22:20:57.710013+0000","last_undegraded":"2024-06-09T22:20:57.710013+0000","last_fullsized":"2024-06-09T22:20:57.710013+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:47:04.463040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"99'7","reported_seq":243,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597416+0000","last_change":"2024-06-09T22:16:40.364834+0000","last_active":"2024-06-09T22:20:57.597416+0000","last_peered":"2024-06-09T22:20:57.597416+0000","last_clean":"2024-06-09T22:20:57.597416+0000","last_became_active":"2024-06-09T22:16:40.364433+0000","last_became_peered":"2024-06-09T22:16:40.364433+0000","last_unstale":"2024-06-09T22:20:57.597416+0000","last_undegraded":"2024-06-09T22:20:57.597416+0000","last_fullsized":"2024-06-09T22:20:57.597416+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:43:01.967911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":191,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597174+0000","last_change":"2024-06-09T22:16:43.201183+0000","last_active":"2024-06-09T22:20:57.597174+0000","last_peered":"2024-06-09T22:20:57.597174+0000","last_clean":"2024-06-09T22:20:57.597174+0000","last_became_active":"2024-06-09T22:16:43.200774+0000","last_became_peered":"2024-06-09T22:16:43.200774+0000","last_unstale":"2024-06-09T22:20:57.597174+0000","last_undegraded":"2024-06-09T22:20:57.597174+0000","last_fullsized":"2024-06-09T22:20:57.597174+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:07:55.772066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710583+0000","last_change":"2024-06-09T22:18:26.699190+0000","last_active":"2024-06-09T22:20:57.710583+0000","last_peered":"2024-06-09T22:20:57.710583+0000","last_clean":"2024-06-09T22:20:57.710583+0000","last_became_active":"2024-06-09T22:18:26.698614+0000","last_became_peered":"2024-06-09T22:18:26.698614+0000","last_unstale":"2024-06-09T22:20:57.710583+0000","last_undegraded":"2024-06-09T22:20:57.710583+0000","last_fullsized":"2024-06-09T22:20:57.710583+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:45.343605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,1,4],"acting":[0,10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"100'41","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.804410+0000","last_change":"2024-06-09T22:16:40.040582+0000","last_active":"2024-06-09T22:20:57.804410+0000","last_peered":"2024-06-09T22:20:57.804410+0000","last_clean":"2024-06-09T22:20:57.804410+0000","last_became_active":"2024-06-09T22:16:40.040152+0000","last_became_peered":"2024-06-09T22:16:40.040152+0000","last_unstale":"2024-06-09T22:20:57.804410+0000","last_undegraded":"2024-06-09T22:20:57.804410+0000","last_fullsized":"2024-06-09T22:20:57.804410+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:50:00.878096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596909+0000","last_change":"2024-06-09T22:16:43.206211+0000","last_active":"2024-06-09T22:20:57.596909+0000","last_peered":"2024-06-09T22:20:57.596909+0000","last_clean":"2024-06-09T22:20:57.596909+0000","last_became_active":"2024-06-09T22:16:43.205814+0000","last_became_peered":"2024-06-09T22:16:43.205814+0000","last_unstale":"2024-06-09T22:20:57.596909+0000","last_undegraded":"2024-06-09T22:20:57.596909+0000","last_fullsized":"2024-06-09T22:20:57.596909+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:20:19.058475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703043+0000","last_change":"2024-06-09T22:16:53.337695+0000","last_active":"2024-06-09T22:20:57.703043+0000","last_peered":"2024-06-09T22:20:57.703043+0000","last_clean":"2024-06-09T22:20:57.703043+0000","last_became_active":"2024-06-09T22:16:53.334872+0000","last_became_peered":"2024-06-09T22:16:53.334872+0000","last_unstale":"2024-06-09T22:20:57.703043+0000","last_undegraded":"2024-06-09T22:20:57.703043+0000","last_fullsized":"2024-06-09T22:20:57.703043+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:50:36.617736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596073+0000","last_change":"2024-06-09T22:16:40.036603+0000","last_active":"2024-06-09T22:20:57.596073+0000","last_peered":"2024-06-09T22:20:57.596073+0000","last_clean":"2024-06-09T22:20:57.596073+0000","last_became_active":"2024-06-09T22:16:40.036274+0000","last_became_peered":"2024-06-09T22:16:40.036274+0000","last_unstale":"2024-06-09T22:20:57.596073+0000","last_undegraded":"2024-06-09T22:20:57.596073+0000","last_fullsized":"2024-06-09T22:20:57.596073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:31.652897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"99'9","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596989+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:20:57.596989+0000","last_peered":"2024-06-09T22:20:57.596989+0000","last_clean":"2024-06-09T22:20:57.596989+0000","last_became_active":"2024-06-09T22:16:43.204197+0000","last_became_peered":"2024-06-09T22:16:43.204197+0000","last_unstale":"2024-06-09T22:20:57.596989+0000","last_undegraded":"2024-06-09T22:20:57.596989+0000","last_fullsized":"2024-06-09T22:20:57.596989+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:38:49.850120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710013+0000","last_change":"2024-06-09T22:16:43.208477+0000","last_active":"2024-06-09T22:20:57.710013+0000","last_peered":"2024-06-09T22:20:57.710013+0000","last_clean":"2024-06-09T22:20:57.710013+0000","last_became_active":"2024-06-09T22:16:43.206018+0000","last_became_peered":"2024-06-09T22:16:43.206018+0000","last_unstale":"2024-06-09T22:20:57.710013+0000","last_undegraded":"2024-06-09T22:20:57.710013+0000","last_fullsized":"2024-06-09T22:20:57.710013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:19:49.870825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"99'12","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597095+0000","last_change":"2024-06-09T22:16:43.206809+0000","last_active":"2024-06-09T22:20:57.597095+0000","last_peered":"2024-06-09T22:20:57.597095+0000","last_clean":"2024-06-09T22:20:57.597095+0000","last_became_active":"2024-06-09T22:16:43.206175+0000","last_became_peered":"2024-06-09T22:16:43.206175+0000","last_unstale":"2024-06-09T22:20:57.597095+0000","last_undegraded":"2024-06-09T22:20:57.597095+0000","last_fullsized":"2024-06-09T22:20:57.597095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:39:22.502266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596879+0000","last_change":"2024-06-09T22:16:43.202163+0000","last_active":"2024-06-09T22:20:57.596879+0000","last_peered":"2024-06-09T22:20:57.596879+0000","last_clean":"2024-06-09T22:20:57.596879+0000","last_became_active":"2024-06-09T22:16:43.201839+0000","last_became_peered":"2024-06-09T22:16:43.201839+0000","last_unstale":"2024-06-09T22:20:57.596879+0000","last_undegraded":"2024-06-09T22:20:57.596879+0000","last_fullsized":"2024-06-09T22:20:57.596879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:47:46.723289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710813+0000","last_change":"2024-06-09T22:16:43.202734+0000","last_active":"2024-06-09T22:20:57.710813+0000","last_peered":"2024-06-09T22:20:57.710813+0000","last_clean":"2024-06-09T22:20:57.710813+0000","last_became_active":"2024-06-09T22:16:43.201638+0000","last_became_peered":"2024-06-09T22:16:43.201638+0000","last_unstale":"2024-06-09T22:20:57.710813+0000","last_undegraded":"2024-06-09T22:20:57.710813+0000","last_fullsized":"2024-06-09T22:20:57.710813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:16.360985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"99'4","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595768+0000","last_change":"2024-06-09T22:16:43.199113+0000","last_active":"2024-06-09T22:20:57.595768+0000","last_peered":"2024-06-09T22:20:57.595768+0000","last_clean":"2024-06-09T22:20:57.595768+0000","last_became_active":"2024-06-09T22:16:43.198804+0000","last_became_peered":"2024-06-09T22:16:43.198804+0000","last_unstale":"2024-06-09T22:20:57.595768+0000","last_undegraded":"2024-06-09T22:20:57.595768+0000","last_fullsized":"2024-06-09T22:20:57.595768+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:18:08.999477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"99'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710339+0000","last_change":"2024-06-09T22:16:43.204445+0000","last_active":"2024-06-09T22:20:57.710339+0000","last_peered":"2024-06-09T22:20:57.710339+0000","last_clean":"2024-06-09T22:20:57.710339+0000","last_became_active":"2024-06-09T22:16:43.203081+0000","last_became_peered":"2024-06-09T22:16:43.203081+0000","last_unstale":"2024-06-09T22:20:57.710339+0000","last_undegraded":"2024-06-09T22:20:57.710339+0000","last_fullsized":"2024-06-09T22:20:57.710339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:39.303813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"99'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597592+0000","last_change":"2024-06-09T22:16:43.211402+0000","last_active":"2024-06-09T22:20:57.597592+0000","last_peered":"2024-06-09T22:20:57.597592+0000","last_clean":"2024-06-09T22:20:57.597592+0000","last_became_active":"2024-06-09T22:16:43.210871+0000","last_became_peered":"2024-06-09T22:16:43.210871+0000","last_unstale":"2024-06-09T22:20:57.597592+0000","last_undegraded":"2024-06-09T22:20:57.597592+0000","last_fullsized":"2024-06-09T22:20:57.597592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:07:21.467617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"99'2","reported_seq":42,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696237+0000","last_change":"2024-06-09T22:16:43.201159+0000","last_active":"2024-06-09T22:20:57.696237+0000","last_peered":"2024-06-09T22:20:57.696237+0000","last_clean":"2024-06-09T22:20:57.696237+0000","last_became_active":"2024-06-09T22:16:43.200850+0000","last_became_peered":"2024-06-09T22:16:43.200850+0000","last_unstale":"2024-06-09T22:20:57.696237+0000","last_undegraded":"2024-06-09T22:20:57.696237+0000","last_fullsized":"2024-06-09T22:20:57.696237+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:20.515309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"99'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703213+0000","last_change":"2024-06-09T22:16:43.196830+0000","last_active":"2024-06-09T22:20:57.703213+0000","last_peered":"2024-06-09T22:20:57.703213+0000","last_clean":"2024-06-09T22:20:57.703213+0000","last_became_active":"2024-06-09T22:16:43.195682+0000","last_became_peered":"2024-06-09T22:16:43.195682+0000","last_unstale":"2024-06-09T22:20:57.703213+0000","last_undegraded":"2024-06-09T22:20:57.703213+0000","last_fullsized":"2024-06-09T22:20:57.703213+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:34:18.770235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694921+0000","last_change":"2024-06-09T22:16:43.201959+0000","last_active":"2024-06-09T22:20:57.694921+0000","last_peered":"2024-06-09T22:20:57.694921+0000","last_clean":"2024-06-09T22:20:57.694921+0000","last_became_active":"2024-06-09T22:16:43.201669+0000","last_became_peered":"2024-06-09T22:16:43.201669+0000","last_unstale":"2024-06-09T22:20:57.694921+0000","last_undegraded":"2024-06-09T22:20:57.694921+0000","last_fullsized":"2024-06-09T22:20:57.694921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:48:57.348041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710198+0000","last_change":"2024-06-09T22:16:43.201823+0000","last_active":"2024-06-09T22:20:57.710198+0000","last_peered":"2024-06-09T22:20:57.710198+0000","last_clean":"2024-06-09T22:20:57.710198+0000","last_became_active":"2024-06-09T22:16:43.201434+0000","last_became_peered":"2024-06-09T22:16:43.201434+0000","last_unstale":"2024-06-09T22:20:57.710198+0000","last_undegraded":"2024-06-09T22:20:57.710198+0000","last_fullsized":"2024-06-09T22:20:57.710198+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:56:13.540895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"99'11","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595624+0000","last_change":"2024-06-09T22:16:43.197734+0000","last_active":"2024-06-09T22:20:57.595624+0000","last_peered":"2024-06-09T22:20:57.595624+0000","last_clean":"2024-06-09T22:20:57.595624+0000","last_became_active":"2024-06-09T22:16:43.197442+0000","last_became_peered":"2024-06-09T22:16:43.197442+0000","last_unstale":"2024-06-09T22:20:57.595624+0000","last_undegraded":"2024-06-09T22:20:57.595624+0000","last_fullsized":"2024-06-09T22:20:57.595624+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:35:40.720718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710339+0000","last_change":"2024-06-09T22:16:43.210375+0000","last_active":"2024-06-09T22:20:57.710339+0000","last_peered":"2024-06-09T22:20:57.710339+0000","last_clean":"2024-06-09T22:20:57.710339+0000","last_became_active":"2024-06-09T22:16:43.209647+0000","last_became_peered":"2024-06-09T22:16:43.209647+0000","last_unstale":"2024-06-09T22:20:57.710339+0000","last_undegraded":"2024-06-09T22:20:57.710339+0000","last_fullsized":"2024-06-09T22:20:57.710339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:11:57.593066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107643,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":607457,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2015,"num_read_kb":1129,"num_write":387,"num_write_kb":1481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":737280,"data_stored":1822761,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383747,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":580840,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645384,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32424,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712728,"statfs":{"total":95995035648,"available":95961833472,"internally_reserved":0,"allocated":1273856,"data_stored":1236852,"data_compressed":71140,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808909,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1167360,"data_stored":977483,"data_compressed":54405,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070546,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1159168,"data_stored":973765,"data_compressed":54405,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332183,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32340,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712812,"statfs":{"total":95995035648,"available":95961919488,"internally_reserved":0,"allocated":1187840,"data_stored":1178025,"data_compressed":71140,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495710,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1114112,"data_stored":576065,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36504,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708648,"statfs":{"total":95995035648,"available":95957655552,"internally_reserved":0,"allocated":1191936,"data_stored":650243,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708700,"statfs":{"total":95995035648,"available":95957708800,"internally_reserved":0,"allocated":1138688,"data_stored":577662,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":651847,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182507,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1171456,"data_stored":1100199,"data_compressed":60561,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920871,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32276,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712876,"statfs":{"total":95995035648,"available":95961985024,"internally_reserved":0,"allocated":1122304,"data_stored":580978,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757346,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36524,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708628,"statfs":{"total":95995035648,"available":95957635072,"internally_reserved":0,"allocated":1216512,"data_stored":1304758,"data_compressed":77296,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":75290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":76894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135341,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":208103,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-09T22:21:04.226 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-06-09T22:21:04.227 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-09T22:21:04.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: Updating smithi104:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-09T22:21:04.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: Updating smithi153:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-09T22:21:04.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: Updating smithi194:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-09T22:21:04.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:21:04.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:04.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:04.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:04 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.360+0000 7f92c31e7640 1 -- 172.21.15.104:0/2507682989 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92bc107980 msgr2=0x7f92bc107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:04.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.360+0000 7f92c31e7640 1 --2- 172.21.15.104:0/2507682989 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92bc107980 0x7f92bc107d60 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009930 tx=0x7f92b002a420 comp rx=0 tx=0).stop 2024-06-09T22:21:04.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.360+0000 7f92c31e7640 1 -- 172.21.15.104:0/2507682989 shutdown_connections 2024-06-09T22:21:04.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.360+0000 7f92c31e7640 1 --2- 172.21.15.104:0/2507682989 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92bc102330 0x7f92bc10e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.360+0000 7f92c31e7640 1 --2- 172.21.15.104:0/2507682989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc101970 0x7f92bc101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.360+0000 7f92c31e7640 1 --2- 172.21.15.104:0/2507682989 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92bc107980 0x7f92bc107d60 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.360+0000 7f92c31e7640 1 -- 172.21.15.104:0/2507682989 >> 172.21.15.104:0/2507682989 conn(0x7f92bc078d60 msgr2=0x7f92bc079170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:04.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.360+0000 7f92c31e7640 1 -- 172.21.15.104:0/2507682989 shutdown_connections 2024-06-09T22:21:04.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.360+0000 7f92c31e7640 1 -- 172.21.15.104:0/2507682989 wait complete. 2024-06-09T22:21:04.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92c31e7640 1 Processor -- start 2024-06-09T22:21:04.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92c31e7640 1 -- start start 2024-06-09T22:21:04.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92c31e7640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92bc101970 0x7f92bc1982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:04.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92c31e7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92bc102330 0x7f92bc198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:04.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92c31e7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc107980 0x7f92bc19cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:04.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92c31e7640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f92bc111d60 con 0x7f92bc102330 2024-06-09T22:21:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92c31e7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f92bc111be0 con 0x7f92bc107980 2024-06-09T22:21:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92c31e7640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f92bc111ee0 con 0x7f92bc101970 2024-06-09T22:21:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92bbfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92bc102330 0x7f92bc198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92bbfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92bc102330 0x7f92bc198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44456/0 (socket says 172.21.15.104:44456) 2024-06-09T22:21:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92bbfff640 1 -- 172.21.15.104:0/621667873 learned_addr learned my addr 172.21.15.104:0/621667873 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:04.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92c1784640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc107980 0x7f92bc19cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:04.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92c0f83640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92bc101970 0x7f92bc1982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:04.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92bbfff640 1 -- 172.21.15.104:0/621667873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92bc101970 msgr2=0x7f92bc1982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:04.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92bbfff640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92bc101970 0x7f92bc1982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92bbfff640 1 -- 172.21.15.104:0/621667873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc107980 msgr2=0x7f92bc19cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:04.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92bbfff640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc107980 0x7f92bc19cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.361+0000 7f92bbfff640 1 -- 172.21.15.104:0/621667873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92bc19d2b0 con 0x7f92bc102330 2024-06-09T22:21:04.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.362+0000 7f92bbfff640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92bc102330 0x7f92bc198800 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f92a800e9f0 tx=0x7f92a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:04.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.362+0000 7f92b9ffb640 1 -- 172.21.15.104:0/621667873 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a800cd80 con 0x7f92bc102330 2024-06-09T22:21:04.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.362+0000 7f92b9ffb640 1 -- 172.21.15.104:0/621667873 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a8010910 con 0x7f92bc102330 2024-06-09T22:21:04.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.362+0000 7f92c31e7640 1 -- 172.21.15.104:0/621667873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92bc19d5a0 con 0x7f92bc102330 2024-06-09T22:21:04.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.362+0000 7f92b9ffb640 1 -- 172.21.15.104:0/621667873 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a8018990 con 0x7f92bc102330 2024-06-09T22:21:04.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.362+0000 7f92c31e7640 1 -- 172.21.15.104:0/621667873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92bc1a4e30 con 0x7f92bc102330 2024-06-09T22:21:04.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.363+0000 7f92c31e7640 1 -- 172.21.15.104:0/621667873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92bc068c10 con 0x7f92bc102330 2024-06-09T22:21:04.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.364+0000 7f92b9ffb640 1 -- 172.21.15.104:0/621667873 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f92a8010ac0 con 0x7f92bc102330 2024-06-09T22:21:04.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.365+0000 7f92b9ffb640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f92ac0a2e20 0x7f92ac0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:04.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.365+0000 7f92c0f83640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f92ac0a2e20 0x7f92ac0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:04.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.365+0000 7f92b9ffb640 1 -- 172.21.15.104:0/621667873 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f92a80cfd60 con 0x7f92bc102330 2024-06-09T22:21:04.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.365+0000 7f92c0f83640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f92ac0a2e20 0x7f92ac0a52e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009900 tx=0x7f92b00058d0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:04.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.369+0000 7f92b9ffb640 1 -- 172.21.15.104:0/621667873 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f92a809bd70 con 0x7f92bc102330 2024-06-09T22:21:04.424 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: Updating smithi104:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-09T22:21:04.425 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: Updating smithi153:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-09T22:21:04.425 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: Updating smithi194:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-09T22:21:04.425 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.425 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.425 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.425 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.426 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.426 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.426 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.426 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:21:04.426 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:04.427 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:04.427 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:04 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:04.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.540+0000 7f92c31e7640 1 -- 172.21.15.104:0/621667873 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92bc061de0 con 0x7f92ac0a2e20 2024-06-09T22:21:04.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.549+0000 7f92b9ffb640 1 -- 172.21.15.104:0/621667873 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339859 (secure 0 0 0) 0x7f92bc061de0 con 0x7f92ac0a2e20 2024-06-09T22:21:04.552 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:04.553 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-06-09T22:21:04.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.553+0000 7f92c31e7640 1 -- 172.21.15.104:0/621667873 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f92ac0a2e20 msgr2=0x7f92ac0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:04.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.553+0000 7f92c31e7640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f92ac0a2e20 0x7f92ac0a52e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009900 tx=0x7f92b00058d0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.553+0000 7f92c31e7640 1 -- 172.21.15.104:0/621667873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92bc102330 msgr2=0x7f92bc198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:04.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.553+0000 7f92c31e7640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92bc102330 0x7f92bc198800 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f92a800e9f0 tx=0x7f92a800eeb0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.553+0000 7f92c31e7640 1 -- 172.21.15.104:0/621667873 shutdown_connections 2024-06-09T22:21:04.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.553+0000 7f92c31e7640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f92ac0a2e20 0x7f92ac0a52e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.553+0000 7f92c31e7640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc107980 0x7f92bc19cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.553+0000 7f92c31e7640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92bc102330 0x7f92bc198800 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.553+0000 7f92c31e7640 1 --2- 172.21.15.104:0/621667873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92bc101970 0x7f92bc1982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:04.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.553+0000 7f92c31e7640 1 -- 172.21.15.104:0/621667873 >> 172.21.15.104:0/621667873 conn(0x7f92bc078d60 msgr2=0x7f92bc104510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:04.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.553+0000 7f92c31e7640 1 -- 172.21.15.104:0/621667873 shutdown_connections 2024-06-09T22:21:04.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:04.553+0000 7f92c31e7640 1 -- 172.21.15.104:0/621667873 wait complete. 2024-06-09T22:21:04.563 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-06-09T22:21:03.609423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1198059,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2151,"num_read_kb":1248,"num_write":570,"num_write_kb":4421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":951,"ondisk_log_size":951,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383732,"kb_used_data":13740,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558092,"statfs":{"total":1151940427776,"available":1151547486208,"internally_reserved":0,"allocated":14069760,"data_stored":10388717,"data_compressed":600501,"data_compressed_allocated":815104,"data_compressed_original":4374528,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.989045"},"pg_stats":[{"pgid":"3.3f","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710445+0000","last_change":"2024-06-09T22:16:43.212998+0000","last_active":"2024-06-09T22:20:57.710445+0000","last_peered":"2024-06-09T22:20:57.710445+0000","last_clean":"2024-06-09T22:20:57.710445+0000","last_became_active":"2024-06-09T22:16:43.212614+0000","last_became_peered":"2024-06-09T22:16:43.212614+0000","last_unstale":"2024-06-09T22:20:57.710445+0000","last_undegraded":"2024-06-09T22:20:57.710445+0000","last_fullsized":"2024-06-09T22:20:57.710445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:22.728486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"99'4","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711160+0000","last_change":"2024-06-09T22:16:43.208746+0000","last_active":"2024-06-09T22:20:57.711160+0000","last_peered":"2024-06-09T22:20:57.711160+0000","last_clean":"2024-06-09T22:20:57.711160+0000","last_became_active":"2024-06-09T22:16:43.208458+0000","last_became_peered":"2024-06-09T22:16:43.208458+0000","last_unstale":"2024-06-09T22:20:57.711160+0000","last_undegraded":"2024-06-09T22:20:57.711160+0000","last_fullsized":"2024-06-09T22:20:57.711160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:24:35.350449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710878+0000","last_change":"2024-06-09T22:16:43.200770+0000","last_active":"2024-06-09T22:20:57.710878+0000","last_peered":"2024-06-09T22:20:57.710878+0000","last_clean":"2024-06-09T22:20:57.710878+0000","last_became_active":"2024-06-09T22:16:43.200449+0000","last_became_peered":"2024-06-09T22:16:43.200449+0000","last_unstale":"2024-06-09T22:20:57.710878+0000","last_undegraded":"2024-06-09T22:20:57.710878+0000","last_fullsized":"2024-06-09T22:20:57.710878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:51:09.017260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703458+0000","last_change":"2024-06-09T22:16:43.203844+0000","last_active":"2024-06-09T22:20:57.703458+0000","last_peered":"2024-06-09T22:20:57.703458+0000","last_clean":"2024-06-09T22:20:57.703458+0000","last_became_active":"2024-06-09T22:16:43.203431+0000","last_became_peered":"2024-06-09T22:16:43.203431+0000","last_unstale":"2024-06-09T22:20:57.703458+0000","last_undegraded":"2024-06-09T22:20:57.703458+0000","last_fullsized":"2024-06-09T22:20:57.703458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:51:36.913312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"99'4","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694538+0000","last_change":"2024-06-09T22:16:43.207597+0000","last_active":"2024-06-09T22:20:57.694538+0000","last_peered":"2024-06-09T22:20:57.694538+0000","last_clean":"2024-06-09T22:20:57.694538+0000","last_became_active":"2024-06-09T22:16:43.207266+0000","last_became_peered":"2024-06-09T22:16:43.207266+0000","last_unstale":"2024-06-09T22:20:57.694538+0000","last_undegraded":"2024-06-09T22:20:57.694538+0000","last_fullsized":"2024-06-09T22:20:57.694538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:47:05.558403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"99'6","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702988+0000","last_change":"2024-06-09T22:17:26.474121+0000","last_active":"2024-06-09T22:20:57.702988+0000","last_peered":"2024-06-09T22:20:57.702988+0000","last_clean":"2024-06-09T22:20:57.702988+0000","last_became_active":"2024-06-09T22:17:26.473619+0000","last_became_peered":"2024-06-09T22:17:26.473619+0000","last_unstale":"2024-06-09T22:20:57.702988+0000","last_undegraded":"2024-06-09T22:20:57.702988+0000","last_fullsized":"2024-06-09T22:20:57.702988+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:56:42.171235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"99'5","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596592+0000","last_change":"2024-06-09T22:16:43.208815+0000","last_active":"2024-06-09T22:20:57.596592+0000","last_peered":"2024-06-09T22:20:57.596592+0000","last_clean":"2024-06-09T22:20:57.596592+0000","last_became_active":"2024-06-09T22:16:43.208433+0000","last_became_peered":"2024-06-09T22:16:43.208433+0000","last_unstale":"2024-06-09T22:20:57.596592+0000","last_undegraded":"2024-06-09T22:20:57.596592+0000","last_fullsized":"2024-06-09T22:20:57.596592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:36:26.585037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694783+0000","last_change":"2024-06-09T22:16:43.210195+0000","last_active":"2024-06-09T22:20:57.694783+0000","last_peered":"2024-06-09T22:20:57.694783+0000","last_clean":"2024-06-09T22:20:57.694783+0000","last_became_active":"2024-06-09T22:16:43.209892+0000","last_became_peered":"2024-06-09T22:16:43.209892+0000","last_unstale":"2024-06-09T22:20:57.694783+0000","last_undegraded":"2024-06-09T22:20:57.694783+0000","last_fullsized":"2024-06-09T22:20:57.694783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:44.088212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710618+0000","last_change":"2024-06-09T22:16:43.202517+0000","last_active":"2024-06-09T22:20:57.710618+0000","last_peered":"2024-06-09T22:20:57.710618+0000","last_clean":"2024-06-09T22:20:57.710618+0000","last_became_active":"2024-06-09T22:16:43.201796+0000","last_became_peered":"2024-06-09T22:16:43.201796+0000","last_unstale":"2024-06-09T22:20:57.710618+0000","last_undegraded":"2024-06-09T22:20:57.710618+0000","last_fullsized":"2024-06-09T22:20:57.710618+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:30.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"99'16","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703409+0000","last_change":"2024-06-09T22:16:43.212732+0000","last_active":"2024-06-09T22:20:57.703409+0000","last_peered":"2024-06-09T22:20:57.703409+0000","last_clean":"2024-06-09T22:20:57.703409+0000","last_became_active":"2024-06-09T22:16:43.212048+0000","last_became_peered":"2024-06-09T22:16:43.212048+0000","last_unstale":"2024-06-09T22:20:57.703409+0000","last_undegraded":"2024-06-09T22:20:57.703409+0000","last_fullsized":"2024-06-09T22:20:57.703409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:47.383291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597025+0000","last_change":"2024-06-09T22:16:43.203778+0000","last_active":"2024-06-09T22:20:57.597025+0000","last_peered":"2024-06-09T22:20:57.597025+0000","last_clean":"2024-06-09T22:20:57.597025+0000","last_became_active":"2024-06-09T22:16:43.202954+0000","last_became_peered":"2024-06-09T22:16:43.202954+0000","last_unstale":"2024-06-09T22:20:57.597025+0000","last_undegraded":"2024-06-09T22:20:57.597025+0000","last_fullsized":"2024-06-09T22:20:57.597025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:56:53.708645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703526+0000","last_change":"2024-06-09T22:16:43.211425+0000","last_active":"2024-06-09T22:20:57.703526+0000","last_peered":"2024-06-09T22:20:57.703526+0000","last_clean":"2024-06-09T22:20:57.703526+0000","last_became_active":"2024-06-09T22:16:43.211001+0000","last_became_peered":"2024-06-09T22:16:43.211001+0000","last_unstale":"2024-06-09T22:20:57.703526+0000","last_undegraded":"2024-06-09T22:20:57.703526+0000","last_fullsized":"2024-06-09T22:20:57.703526+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:20:59.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"99'10","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595794+0000","last_change":"2024-06-09T22:16:43.208334+0000","last_active":"2024-06-09T22:20:57.595794+0000","last_peered":"2024-06-09T22:20:57.595794+0000","last_clean":"2024-06-09T22:20:57.595794+0000","last_became_active":"2024-06-09T22:16:43.207993+0000","last_became_peered":"2024-06-09T22:16:43.207993+0000","last_unstale":"2024-06-09T22:20:57.595794+0000","last_undegraded":"2024-06-09T22:20:57.595794+0000","last_fullsized":"2024-06-09T22:20:57.595794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:07:05.740748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710461+0000","last_change":"2024-06-09T22:16:43.207211+0000","last_active":"2024-06-09T22:20:57.710461+0000","last_peered":"2024-06-09T22:20:57.710461+0000","last_clean":"2024-06-09T22:20:57.710461+0000","last_became_active":"2024-06-09T22:16:43.206889+0000","last_became_peered":"2024-06-09T22:16:43.206889+0000","last_unstale":"2024-06-09T22:20:57.710461+0000","last_undegraded":"2024-06-09T22:20:57.710461+0000","last_fullsized":"2024-06-09T22:20:57.710461+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:25:47.461030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710404+0000","last_change":"2024-06-09T22:16:43.202202+0000","last_active":"2024-06-09T22:20:57.710404+0000","last_peered":"2024-06-09T22:20:57.710404+0000","last_clean":"2024-06-09T22:20:57.710404+0000","last_became_active":"2024-06-09T22:16:43.201834+0000","last_became_peered":"2024-06-09T22:16:43.201834+0000","last_unstale":"2024-06-09T22:20:57.710404+0000","last_undegraded":"2024-06-09T22:20:57.710404+0000","last_fullsized":"2024-06-09T22:20:57.710404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:31:51.296627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"99'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709927+0000","last_change":"2024-06-09T22:16:43.200637+0000","last_active":"2024-06-09T22:20:57.709927+0000","last_peered":"2024-06-09T22:20:57.709927+0000","last_clean":"2024-06-09T22:20:57.709927+0000","last_became_active":"2024-06-09T22:16:43.200268+0000","last_became_peered":"2024-06-09T22:16:43.200268+0000","last_unstale":"2024-06-09T22:20:57.709927+0000","last_undegraded":"2024-06-09T22:20:57.709927+0000","last_fullsized":"2024-06-09T22:20:57.709927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:25:59.881453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710229+0000","last_change":"2024-06-09T22:16:43.209212+0000","last_active":"2024-06-09T22:20:57.710229+0000","last_peered":"2024-06-09T22:20:57.710229+0000","last_clean":"2024-06-09T22:20:57.710229+0000","last_became_active":"2024-06-09T22:16:43.207789+0000","last_became_peered":"2024-06-09T22:16:43.207789+0000","last_unstale":"2024-06-09T22:20:57.710229+0000","last_undegraded":"2024-06-09T22:20:57.710229+0000","last_fullsized":"2024-06-09T22:20:57.710229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:17.316595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692756+0000","last_change":"2024-06-09T22:16:43.210935+0000","last_active":"2024-06-09T22:20:57.692756+0000","last_peered":"2024-06-09T22:20:57.692756+0000","last_clean":"2024-06-09T22:20:57.692756+0000","last_became_active":"2024-06-09T22:16:43.210631+0000","last_became_peered":"2024-06-09T22:16:43.210631+0000","last_unstale":"2024-06-09T22:20:57.692756+0000","last_undegraded":"2024-06-09T22:20:57.692756+0000","last_fullsized":"2024-06-09T22:20:57.692756+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:22:58.887313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"99'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597630+0000","last_change":"2024-06-09T22:16:43.198872+0000","last_active":"2024-06-09T22:20:57.597630+0000","last_peered":"2024-06-09T22:20:57.597630+0000","last_clean":"2024-06-09T22:20:57.597630+0000","last_became_active":"2024-06-09T22:16:43.198498+0000","last_became_peered":"2024-06-09T22:16:43.198498+0000","last_unstale":"2024-06-09T22:20:57.597630+0000","last_undegraded":"2024-06-09T22:20:57.597630+0000","last_fullsized":"2024-06-09T22:20:57.597630+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:25:28.853048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597223+0000","last_change":"2024-06-09T22:16:53.343656+0000","last_active":"2024-06-09T22:20:57.597223+0000","last_peered":"2024-06-09T22:20:57.597223+0000","last_clean":"2024-06-09T22:20:57.597223+0000","last_became_active":"2024-06-09T22:16:53.343281+0000","last_became_peered":"2024-06-09T22:16:53.343281+0000","last_unstale":"2024-06-09T22:20:57.597223+0000","last_undegraded":"2024-06-09T22:20:57.597223+0000","last_fullsized":"2024-06-09T22:20:57.597223+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:40.961842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"97'13","reported_seq":247,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.700687+0000","last_change":"2024-06-09T22:16:40.040145+0000","last_active":"2024-06-09T22:20:57.700687+0000","last_peered":"2024-06-09T22:20:57.700687+0000","last_clean":"2024-06-09T22:20:57.700687+0000","last_became_active":"2024-06-09T22:16:40.039787+0000","last_became_peered":"2024-06-09T22:16:40.039787+0000","last_unstale":"2024-06-09T22:20:57.700687+0000","last_undegraded":"2024-06-09T22:20:57.700687+0000","last_fullsized":"2024-06-09T22:20:57.700687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:52:32.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":188,"num_read_kb":188,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"99'5","reported_seq":23,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692731+0000","last_change":"2024-06-09T22:17:26.774053+0000","last_active":"2024-06-09T22:20:57.692731+0000","last_peered":"2024-06-09T22:20:57.692731+0000","last_clean":"2024-06-09T22:20:57.692731+0000","last_became_active":"2024-06-09T22:17:26.773786+0000","last_became_peered":"2024-06-09T22:17:26.773786+0000","last_unstale":"2024-06-09T22:20:57.692731+0000","last_undegraded":"2024-06-09T22:20:57.692731+0000","last_fullsized":"2024-06-09T22:20:57.692731+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:36.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703244+0000","last_change":"2024-06-09T22:16:53.332048+0000","last_active":"2024-06-09T22:20:57.703244+0000","last_peered":"2024-06-09T22:20:57.703244+0000","last_clean":"2024-06-09T22:20:57.703244+0000","last_became_active":"2024-06-09T22:16:53.330683+0000","last_became_peered":"2024-06-09T22:16:53.330683+0000","last_unstale":"2024-06-09T22:20:57.703244+0000","last_undegraded":"2024-06-09T22:20:57.703244+0000","last_fullsized":"2024-06-09T22:20:57.703244+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:32:23.219061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"97'2","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696082+0000","last_change":"2024-06-09T22:16:40.070193+0000","last_active":"2024-06-09T22:20:57.696082+0000","last_peered":"2024-06-09T22:20:57.696082+0000","last_clean":"2024-06-09T22:20:57.696082+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:20:57.696082+0000","last_undegraded":"2024-06-09T22:20:57.696082+0000","last_fullsized":"2024-06-09T22:20:57.696082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:19:19.787552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711015+0000","last_change":"2024-06-09T22:16:43.199814+0000","last_active":"2024-06-09T22:20:57.711015+0000","last_peered":"2024-06-09T22:20:57.711015+0000","last_clean":"2024-06-09T22:20:57.711015+0000","last_became_active":"2024-06-09T22:16:43.199487+0000","last_became_peered":"2024-06-09T22:16:43.199487+0000","last_unstale":"2024-06-09T22:20:57.711015+0000","last_undegraded":"2024-06-09T22:20:57.711015+0000","last_fullsized":"2024-06-09T22:20:57.711015+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:55.672050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703543+0000","last_change":"2024-06-09T22:16:53.336253+0000","last_active":"2024-06-09T22:20:57.703543+0000","last_peered":"2024-06-09T22:20:57.703543+0000","last_clean":"2024-06-09T22:20:57.703543+0000","last_became_active":"2024-06-09T22:16:53.335869+0000","last_became_peered":"2024-06-09T22:16:53.335869+0000","last_unstale":"2024-06-09T22:20:57.703543+0000","last_undegraded":"2024-06-09T22:20:57.703543+0000","last_fullsized":"2024-06-09T22:20:57.703543+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:28:12.894878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"97'12","reported_seq":60,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695155+0000","last_change":"2024-06-09T22:16:40.045365+0000","last_active":"2024-06-09T22:20:57.695155+0000","last_peered":"2024-06-09T22:20:57.695155+0000","last_clean":"2024-06-09T22:20:57.695155+0000","last_became_active":"2024-06-09T22:16:40.044989+0000","last_became_peered":"2024-06-09T22:16:40.044989+0000","last_unstale":"2024-06-09T22:20:57.695155+0000","last_undegraded":"2024-06-09T22:20:57.695155+0000","last_fullsized":"2024-06-09T22:20:57.695155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:01:05.850170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"99'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710115+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:20:57.710115+0000","last_peered":"2024-06-09T22:20:57.710115+0000","last_clean":"2024-06-09T22:20:57.710115+0000","last_became_active":"2024-06-09T22:16:43.203999+0000","last_became_peered":"2024-06-09T22:16:43.203999+0000","last_unstale":"2024-06-09T22:20:57.710115+0000","last_undegraded":"2024-06-09T22:20:57.710115+0000","last_fullsized":"2024-06-09T22:20:57.710115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:37:28.362655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703277+0000","last_change":"2024-06-09T22:16:53.333565+0000","last_active":"2024-06-09T22:20:57.703277+0000","last_peered":"2024-06-09T22:20:57.703277+0000","last_clean":"2024-06-09T22:20:57.703277+0000","last_became_active":"2024-06-09T22:16:53.332921+0000","last_became_peered":"2024-06-09T22:16:53.332921+0000","last_unstale":"2024-06-09T22:20:57.703277+0000","last_undegraded":"2024-06-09T22:20:57.703277+0000","last_fullsized":"2024-06-09T22:20:57.703277+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:32.085617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"97'5","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694916+0000","last_change":"2024-06-09T22:16:40.039766+0000","last_active":"2024-06-09T22:20:57.694916+0000","last_peered":"2024-06-09T22:20:57.694916+0000","last_clean":"2024-06-09T22:20:57.694916+0000","last_became_active":"2024-06-09T22:16:40.039329+0000","last_became_peered":"2024-06-09T22:16:40.039329+0000","last_unstale":"2024-06-09T22:20:57.694916+0000","last_undegraded":"2024-06-09T22:20:57.694916+0000","last_fullsized":"2024-06-09T22:20:57.694916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:53.673890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"99'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703385+0000","last_change":"2024-06-09T22:16:43.201200+0000","last_active":"2024-06-09T22:20:57.703385+0000","last_peered":"2024-06-09T22:20:57.703385+0000","last_clean":"2024-06-09T22:20:57.703385+0000","last_became_active":"2024-06-09T22:16:43.200830+0000","last_became_peered":"2024-06-09T22:16:43.200830+0000","last_unstale":"2024-06-09T22:20:57.703385+0000","last_undegraded":"2024-06-09T22:20:57.703385+0000","last_fullsized":"2024-06-09T22:20:57.703385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:38:52.093173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597461+0000","last_change":"2024-06-09T22:16:53.339811+0000","last_active":"2024-06-09T22:20:57.597461+0000","last_peered":"2024-06-09T22:20:57.597461+0000","last_clean":"2024-06-09T22:20:57.597461+0000","last_became_active":"2024-06-09T22:16:53.339305+0000","last_became_peered":"2024-06-09T22:16:53.339305+0000","last_unstale":"2024-06-09T22:20:57.597461+0000","last_undegraded":"2024-06-09T22:20:57.597461+0000","last_fullsized":"2024-06-09T22:20:57.597461+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:08:25.800245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"100'8","reported_seq":245,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.704478+0000","last_change":"2024-06-09T22:16:40.044159+0000","last_active":"2024-06-09T22:20:57.704478+0000","last_peered":"2024-06-09T22:20:57.704478+0000","last_clean":"2024-06-09T22:20:57.704478+0000","last_became_active":"2024-06-09T22:16:40.043839+0000","last_became_peered":"2024-06-09T22:16:40.043839+0000","last_unstale":"2024-06-09T22:20:57.704478+0000","last_undegraded":"2024-06-09T22:20:57.704478+0000","last_fullsized":"2024-06-09T22:20:57.704478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:22:09.339219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1543,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":191,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694583+0000","last_change":"2024-06-09T22:16:43.202015+0000","last_active":"2024-06-09T22:20:57.694583+0000","last_peered":"2024-06-09T22:20:57.694583+0000","last_clean":"2024-06-09T22:20:57.694583+0000","last_became_active":"2024-06-09T22:16:43.201689+0000","last_became_peered":"2024-06-09T22:16:43.201689+0000","last_unstale":"2024-06-09T22:20:57.694583+0000","last_undegraded":"2024-06-09T22:20:57.694583+0000","last_fullsized":"2024-06-09T22:20:57.694583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:35:12.180399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597147+0000","last_change":"2024-06-09T22:18:26.698600+0000","last_active":"2024-06-09T22:20:57.597147+0000","last_peered":"2024-06-09T22:20:57.597147+0000","last_clean":"2024-06-09T22:20:57.597147+0000","last_became_active":"2024-06-09T22:18:26.698201+0000","last_became_peered":"2024-06-09T22:18:26.698201+0000","last_unstale":"2024-06-09T22:20:57.597147+0000","last_undegraded":"2024-06-09T22:20:57.597147+0000","last_fullsized":"2024-06-09T22:20:57.597147+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:55:13.513557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"97'7","reported_seq":57,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702844+0000","last_change":"2024-06-09T22:17:26.698204+0000","last_active":"2024-06-09T22:20:57.702844+0000","last_peered":"2024-06-09T22:20:57.702844+0000","last_clean":"2024-06-09T22:20:57.702844+0000","last_became_active":"2024-06-09T22:17:26.697716+0000","last_became_peered":"2024-06-09T22:17:26.697716+0000","last_unstale":"2024-06-09T22:20:57.702844+0000","last_undegraded":"2024-06-09T22:20:57.702844+0000","last_fullsized":"2024-06-09T22:20:57.702844+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:36:37.724459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703554+0000","last_change":"2024-06-09T22:16:43.198917+0000","last_active":"2024-06-09T22:20:57.703554+0000","last_peered":"2024-06-09T22:20:57.703554+0000","last_clean":"2024-06-09T22:20:57.703554+0000","last_became_active":"2024-06-09T22:16:43.198567+0000","last_became_peered":"2024-06-09T22:16:43.198567+0000","last_unstale":"2024-06-09T22:20:57.703554+0000","last_undegraded":"2024-06-09T22:20:57.703554+0000","last_fullsized":"2024-06-09T22:20:57.703554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:34.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710203+0000","last_change":"2024-06-09T22:16:53.342203+0000","last_active":"2024-06-09T22:20:57.710203+0000","last_peered":"2024-06-09T22:20:57.710203+0000","last_clean":"2024-06-09T22:20:57.710203+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:20:57.710203+0000","last_undegraded":"2024-06-09T22:20:57.710203+0000","last_fullsized":"2024-06-09T22:20:57.710203+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:41:41.491961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"97'3","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595514+0000","last_change":"2024-06-09T22:16:40.038684+0000","last_active":"2024-06-09T22:20:57.595514+0000","last_peered":"2024-06-09T22:20:57.595514+0000","last_clean":"2024-06-09T22:20:57.595514+0000","last_became_active":"2024-06-09T22:16:40.038386+0000","last_became_peered":"2024-06-09T22:16:40.038386+0000","last_unstale":"2024-06-09T22:20:57.595514+0000","last_undegraded":"2024-06-09T22:20:57.595514+0000","last_fullsized":"2024-06-09T22:20:57.595514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:59.188758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694679+0000","last_change":"2024-06-09T22:16:43.198837+0000","last_active":"2024-06-09T22:20:57.694679+0000","last_peered":"2024-06-09T22:20:57.694679+0000","last_clean":"2024-06-09T22:20:57.694679+0000","last_became_active":"2024-06-09T22:16:43.198515+0000","last_became_peered":"2024-06-09T22:16:43.198515+0000","last_unstale":"2024-06-09T22:20:57.694679+0000","last_undegraded":"2024-06-09T22:20:57.694679+0000","last_fullsized":"2024-06-09T22:20:57.694679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:36:13.863202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711076+0000","last_change":"2024-06-09T22:16:53.342688+0000","last_active":"2024-06-09T22:20:57.711076+0000","last_peered":"2024-06-09T22:20:57.711076+0000","last_clean":"2024-06-09T22:20:57.711076+0000","last_became_active":"2024-06-09T22:16:53.342314+0000","last_became_peered":"2024-06-09T22:16:53.342314+0000","last_unstale":"2024-06-09T22:20:57.711076+0000","last_undegraded":"2024-06-09T22:20:57.711076+0000","last_fullsized":"2024-06-09T22:20:57.711076+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:51:35.801587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"97'3","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703264+0000","last_change":"2024-06-09T22:16:40.041831+0000","last_active":"2024-06-09T22:20:57.703264+0000","last_peered":"2024-06-09T22:20:57.703264+0000","last_clean":"2024-06-09T22:20:57.703264+0000","last_became_active":"2024-06-09T22:16:40.041484+0000","last_became_peered":"2024-06-09T22:16:40.041484+0000","last_unstale":"2024-06-09T22:20:57.703264+0000","last_undegraded":"2024-06-09T22:20:57.703264+0000","last_fullsized":"2024-06-09T22:20:57.703264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:10:19.829357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692213+0000","last_change":"2024-06-09T22:16:43.210769+0000","last_active":"2024-06-09T22:20:57.692213+0000","last_peered":"2024-06-09T22:20:57.692213+0000","last_clean":"2024-06-09T22:20:57.692213+0000","last_became_active":"2024-06-09T22:16:43.210429+0000","last_became_peered":"2024-06-09T22:16:43.210429+0000","last_unstale":"2024-06-09T22:20:57.692213+0000","last_undegraded":"2024-06-09T22:20:57.692213+0000","last_fullsized":"2024-06-09T22:20:57.692213+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:12:28.788830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694708+0000","last_change":"2024-06-09T22:16:53.336536+0000","last_active":"2024-06-09T22:20:57.694708+0000","last_peered":"2024-06-09T22:20:57.694708+0000","last_clean":"2024-06-09T22:20:57.694708+0000","last_became_active":"2024-06-09T22:16:53.335178+0000","last_became_peered":"2024-06-09T22:16:53.335178+0000","last_unstale":"2024-06-09T22:20:57.694708+0000","last_undegraded":"2024-06-09T22:20:57.694708+0000","last_fullsized":"2024-06-09T22:20:57.694708+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:57:19.476483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'7","reported_seq":425,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.704124+0000","last_change":"2024-06-09T22:16:40.038484+0000","last_active":"2024-06-09T22:20:57.704124+0000","last_peered":"2024-06-09T22:20:57.704124+0000","last_clean":"2024-06-09T22:20:57.704124+0000","last_became_active":"2024-06-09T22:16:40.038084+0000","last_became_peered":"2024-06-09T22:16:40.038084+0000","last_unstale":"2024-06-09T22:20:57.704124+0000","last_undegraded":"2024-06-09T22:20:57.704124+0000","last_fullsized":"2024-06-09T22:20:57.704124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:19.733107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":374,"num_read_kb":374,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710525+0000","last_change":"2024-06-09T22:16:43.205137+0000","last_active":"2024-06-09T22:20:57.710525+0000","last_peered":"2024-06-09T22:20:57.710525+0000","last_clean":"2024-06-09T22:20:57.710525+0000","last_became_active":"2024-06-09T22:16:43.204499+0000","last_became_peered":"2024-06-09T22:16:43.204499+0000","last_unstale":"2024-06-09T22:20:57.710525+0000","last_undegraded":"2024-06-09T22:20:57.710525+0000","last_fullsized":"2024-06-09T22:20:57.710525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:15.718493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692194+0000","last_change":"2024-06-09T22:16:53.344260+0000","last_active":"2024-06-09T22:20:57.692194+0000","last_peered":"2024-06-09T22:20:57.692194+0000","last_clean":"2024-06-09T22:20:57.692194+0000","last_became_active":"2024-06-09T22:16:53.343840+0000","last_became_peered":"2024-06-09T22:16:53.343840+0000","last_unstale":"2024-06-09T22:20:57.692194+0000","last_undegraded":"2024-06-09T22:20:57.692194+0000","last_fullsized":"2024-06-09T22:20:57.692194+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:40:23.305509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"97'4","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596747+0000","last_change":"2024-06-09T22:16:40.042660+0000","last_active":"2024-06-09T22:20:57.596747+0000","last_peered":"2024-06-09T22:20:57.596747+0000","last_clean":"2024-06-09T22:20:57.596747+0000","last_became_active":"2024-06-09T22:16:40.041353+0000","last_became_peered":"2024-06-09T22:16:40.041353+0000","last_unstale":"2024-06-09T22:20:57.596747+0000","last_undegraded":"2024-06-09T22:20:57.596747+0000","last_fullsized":"2024-06-09T22:20:57.596747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:34:59.064827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711158+0000","last_change":"2024-06-09T22:16:43.205083+0000","last_active":"2024-06-09T22:20:57.711158+0000","last_peered":"2024-06-09T22:20:57.711158+0000","last_clean":"2024-06-09T22:20:57.711158+0000","last_became_active":"2024-06-09T22:16:43.204600+0000","last_became_peered":"2024-06-09T22:16:43.204600+0000","last_unstale":"2024-06-09T22:20:57.711158+0000","last_undegraded":"2024-06-09T22:20:57.711158+0000","last_fullsized":"2024-06-09T22:20:57.711158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:05:12.555465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703699+0000","last_change":"2024-06-09T22:16:53.336169+0000","last_active":"2024-06-09T22:20:57.703699+0000","last_peered":"2024-06-09T22:20:57.703699+0000","last_clean":"2024-06-09T22:20:57.703699+0000","last_became_active":"2024-06-09T22:16:53.335657+0000","last_became_peered":"2024-06-09T22:16:53.335657+0000","last_unstale":"2024-06-09T22:20:57.703699+0000","last_undegraded":"2024-06-09T22:20:57.703699+0000","last_fullsized":"2024-06-09T22:20:57.703699+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:53:21.282767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703313+0000","last_change":"2024-06-09T22:16:40.070568+0000","last_active":"2024-06-09T22:20:57.703313+0000","last_peered":"2024-06-09T22:20:57.703313+0000","last_clean":"2024-06-09T22:20:57.703313+0000","last_became_active":"2024-06-09T22:16:40.070205+0000","last_became_peered":"2024-06-09T22:16:40.070205+0000","last_unstale":"2024-06-09T22:20:57.703313+0000","last_undegraded":"2024-06-09T22:20:57.703313+0000","last_fullsized":"2024-06-09T22:20:57.703313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.449965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710336+0000","last_change":"2024-06-09T22:16:43.205219+0000","last_active":"2024-06-09T22:20:57.710336+0000","last_peered":"2024-06-09T22:20:57.710336+0000","last_clean":"2024-06-09T22:20:57.710336+0000","last_became_active":"2024-06-09T22:16:43.203467+0000","last_became_peered":"2024-06-09T22:16:43.203467+0000","last_unstale":"2024-06-09T22:20:57.710336+0000","last_undegraded":"2024-06-09T22:20:57.710336+0000","last_fullsized":"2024-06-09T22:20:57.710336+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:22.784034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692165+0000","last_change":"2024-06-09T22:16:53.338990+0000","last_active":"2024-06-09T22:20:57.692165+0000","last_peered":"2024-06-09T22:20:57.692165+0000","last_clean":"2024-06-09T22:20:57.692165+0000","last_became_active":"2024-06-09T22:16:53.338568+0000","last_became_peered":"2024-06-09T22:16:53.338568+0000","last_unstale":"2024-06-09T22:20:57.692165+0000","last_undegraded":"2024-06-09T22:20:57.692165+0000","last_fullsized":"2024-06-09T22:20:57.692165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:46:39.814043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"100'44","reported_seq":473,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.846818+0000","last_change":"2024-06-09T22:16:40.044404+0000","last_active":"2024-06-09T22:20:57.846818+0000","last_peered":"2024-06-09T22:20:57.846818+0000","last_clean":"2024-06-09T22:20:57.846818+0000","last_became_active":"2024-06-09T22:16:40.044044+0000","last_became_peered":"2024-06-09T22:16:40.044044+0000","last_unstale":"2024-06-09T22:20:57.846818+0000","last_undegraded":"2024-06-09T22:20:57.846818+0000","last_fullsized":"2024-06-09T22:20:57.846818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:50:27.038759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73538,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":385,"num_read_kb":260,"num_write":46,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694903+0000","last_change":"2024-06-09T22:16:43.206963+0000","last_active":"2024-06-09T22:20:57.694903+0000","last_peered":"2024-06-09T22:20:57.694903+0000","last_clean":"2024-06-09T22:20:57.694903+0000","last_became_active":"2024-06-09T22:16:43.206628+0000","last_became_peered":"2024-06-09T22:16:43.206628+0000","last_unstale":"2024-06-09T22:20:57.694903+0000","last_undegraded":"2024-06-09T22:20:57.694903+0000","last_fullsized":"2024-06-09T22:20:57.694903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:33:39.912236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703845+0000","last_change":"2024-06-09T22:16:53.339332+0000","last_active":"2024-06-09T22:20:57.703845+0000","last_peered":"2024-06-09T22:20:57.703845+0000","last_clean":"2024-06-09T22:20:57.703845+0000","last_became_active":"2024-06-09T22:16:53.338819+0000","last_became_peered":"2024-06-09T22:16:53.338819+0000","last_unstale":"2024-06-09T22:20:57.703845+0000","last_undegraded":"2024-06-09T22:20:57.703845+0000","last_fullsized":"2024-06-09T22:20:57.703845+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:57:41.142142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"97'9","reported_seq":33,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709996+0000","last_change":"2024-06-09T22:17:26.478256+0000","last_active":"2024-06-09T22:20:57.709996+0000","last_peered":"2024-06-09T22:20:57.709996+0000","last_clean":"2024-06-09T22:20:57.709996+0000","last_became_active":"2024-06-09T22:17:26.472264+0000","last_became_peered":"2024-06-09T22:17:26.472264+0000","last_unstale":"2024-06-09T22:20:57.709996+0000","last_undegraded":"2024-06-09T22:20:57.709996+0000","last_fullsized":"2024-06-09T22:20:57.709996+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:14:19.919841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"99'9","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595647+0000","last_change":"2024-06-09T22:16:43.204484+0000","last_active":"2024-06-09T22:20:57.595647+0000","last_peered":"2024-06-09T22:20:57.595647+0000","last_clean":"2024-06-09T22:20:57.595647+0000","last_became_active":"2024-06-09T22:16:43.204014+0000","last_became_peered":"2024-06-09T22:16:43.204014+0000","last_unstale":"2024-06-09T22:20:57.595647+0000","last_undegraded":"2024-06-09T22:20:57.595647+0000","last_fullsized":"2024-06-09T22:20:57.595647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:55:12.528231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710227+0000","last_change":"2024-06-09T22:17:26.475179+0000","last_active":"2024-06-09T22:20:57.710227+0000","last_peered":"2024-06-09T22:20:57.710227+0000","last_clean":"2024-06-09T22:20:57.710227+0000","last_became_active":"2024-06-09T22:17:26.474517+0000","last_became_peered":"2024-06-09T22:17:26.474517+0000","last_unstale":"2024-06-09T22:20:57.710227+0000","last_undegraded":"2024-06-09T22:20:57.710227+0000","last_fullsized":"2024-06-09T22:20:57.710227+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:26:02.387830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703306+0000","last_change":"2024-06-09T22:16:40.043066+0000","last_active":"2024-06-09T22:20:57.703306+0000","last_peered":"2024-06-09T22:20:57.703306+0000","last_clean":"2024-06-09T22:20:57.703306+0000","last_became_active":"2024-06-09T22:16:40.042619+0000","last_became_peered":"2024-06-09T22:16:40.042619+0000","last_unstale":"2024-06-09T22:20:57.703306+0000","last_undegraded":"2024-06-09T22:20:57.703306+0000","last_fullsized":"2024-06-09T22:20:57.703306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:29.689614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"99'15","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597173+0000","last_change":"2024-06-09T22:17:26.475566+0000","last_active":"2024-06-09T22:20:57.597173+0000","last_peered":"2024-06-09T22:20:57.597173+0000","last_clean":"2024-06-09T22:20:57.597173+0000","last_became_active":"2024-06-09T22:17:26.475037+0000","last_became_peered":"2024-06-09T22:17:26.475037+0000","last_unstale":"2024-06-09T22:20:57.597173+0000","last_undegraded":"2024-06-09T22:20:57.597173+0000","last_fullsized":"2024-06-09T22:20:57.597173+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:33.075144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694959+0000","last_change":"2024-06-09T22:17:26.475111+0000","last_active":"2024-06-09T22:20:57.694959+0000","last_peered":"2024-06-09T22:20:57.694959+0000","last_clean":"2024-06-09T22:20:57.694959+0000","last_became_active":"2024-06-09T22:17:26.474563+0000","last_became_peered":"2024-06-09T22:17:26.474563+0000","last_unstale":"2024-06-09T22:20:57.694959+0000","last_undegraded":"2024-06-09T22:20:57.694959+0000","last_fullsized":"2024-06-09T22:20:57.694959+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:09:08.458761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"100'9","reported_seq":246,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703821+0000","last_change":"2024-06-09T22:16:40.040538+0000","last_active":"2024-06-09T22:20:57.703821+0000","last_peered":"2024-06-09T22:20:57.703821+0000","last_clean":"2024-06-09T22:20:57.703821+0000","last_became_active":"2024-06-09T22:16:40.039896+0000","last_became_peered":"2024-06-09T22:16:40.039896+0000","last_unstale":"2024-06-09T22:20:57.703821+0000","last_undegraded":"2024-06-09T22:20:57.703821+0000","last_fullsized":"2024-06-09T22:20:57.703821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:00:33.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1817,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":6,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"99'6","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596912+0000","last_change":"2024-06-09T22:17:26.475568+0000","last_active":"2024-06-09T22:20:57.596912+0000","last_peered":"2024-06-09T22:20:57.596912+0000","last_clean":"2024-06-09T22:20:57.596912+0000","last_became_active":"2024-06-09T22:17:26.475163+0000","last_became_peered":"2024-06-09T22:17:26.475163+0000","last_unstale":"2024-06-09T22:20:57.596912+0000","last_undegraded":"2024-06-09T22:20:57.596912+0000","last_fullsized":"2024-06-09T22:20:57.596912+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:20.865164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595970+0000","last_change":"2024-06-09T22:16:53.338201+0000","last_active":"2024-06-09T22:20:57.595970+0000","last_peered":"2024-06-09T22:20:57.595970+0000","last_clean":"2024-06-09T22:20:57.595970+0000","last_became_active":"2024-06-09T22:16:53.337816+0000","last_became_peered":"2024-06-09T22:16:53.337816+0000","last_unstale":"2024-06-09T22:20:57.595970+0000","last_undegraded":"2024-06-09T22:20:57.595970+0000","last_fullsized":"2024-06-09T22:20:57.595970+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:35:51.861135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"97'17","reported_seq":65,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694467+0000","last_change":"2024-06-09T22:16:40.070770+0000","last_active":"2024-06-09T22:20:57.694467+0000","last_peered":"2024-06-09T22:20:57.694467+0000","last_clean":"2024-06-09T22:20:57.694467+0000","last_became_active":"2024-06-09T22:16:40.070398+0000","last_became_peered":"2024-06-09T22:16:40.070398+0000","last_unstale":"2024-06-09T22:20:57.694467+0000","last_undegraded":"2024-06-09T22:20:57.694467+0000","last_fullsized":"2024-06-09T22:20:57.694467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:04:58.273601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"99'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703349+0000","last_change":"2024-06-09T22:16:43.199246+0000","last_active":"2024-06-09T22:20:57.703349+0000","last_peered":"2024-06-09T22:20:57.703349+0000","last_clean":"2024-06-09T22:20:57.703349+0000","last_became_active":"2024-06-09T22:16:43.198006+0000","last_became_peered":"2024-06-09T22:16:43.198006+0000","last_unstale":"2024-06-09T22:20:57.703349+0000","last_undegraded":"2024-06-09T22:20:57.703349+0000","last_fullsized":"2024-06-09T22:20:57.703349+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:30:55.836628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703251+0000","last_change":"2024-06-09T22:16:53.342217+0000","last_active":"2024-06-09T22:20:57.703251+0000","last_peered":"2024-06-09T22:20:57.703251+0000","last_clean":"2024-06-09T22:20:57.703251+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:20:57.703251+0000","last_undegraded":"2024-06-09T22:20:57.703251+0000","last_fullsized":"2024-06-09T22:20:57.703251+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:25:28.635091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"100'4","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695066+0000","last_change":"2024-06-09T22:16:40.039841+0000","last_active":"2024-06-09T22:20:57.695066+0000","last_peered":"2024-06-09T22:20:57.695066+0000","last_clean":"2024-06-09T22:20:57.695066+0000","last_became_active":"2024-06-09T22:16:40.039416+0000","last_became_peered":"2024-06-09T22:16:40.039416+0000","last_unstale":"2024-06-09T22:20:57.695066+0000","last_undegraded":"2024-06-09T22:20:57.695066+0000","last_fullsized":"2024-06-09T22:20:57.695066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:27:36.213839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"99'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710082+0000","last_change":"2024-06-09T22:16:43.214363+0000","last_active":"2024-06-09T22:20:57.710082+0000","last_peered":"2024-06-09T22:20:57.710082+0000","last_clean":"2024-06-09T22:20:57.710082+0000","last_became_active":"2024-06-09T22:16:43.214015+0000","last_became_peered":"2024-06-09T22:16:43.214015+0000","last_unstale":"2024-06-09T22:20:57.710082+0000","last_undegraded":"2024-06-09T22:20:57.710082+0000","last_fullsized":"2024-06-09T22:20:57.710082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:13:06.924468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694673+0000","last_change":"2024-06-09T22:16:53.342151+0000","last_active":"2024-06-09T22:20:57.694673+0000","last_peered":"2024-06-09T22:20:57.694673+0000","last_clean":"2024-06-09T22:20:57.694673+0000","last_became_active":"2024-06-09T22:16:53.341743+0000","last_became_peered":"2024-06-09T22:16:53.341743+0000","last_unstale":"2024-06-09T22:20:57.694673+0000","last_undegraded":"2024-06-09T22:20:57.694673+0000","last_fullsized":"2024-06-09T22:20:57.694673+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:39:44.793301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"97'1","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692594+0000","last_change":"2024-06-09T22:17:26.472397+0000","last_active":"2024-06-09T22:20:57.692594+0000","last_peered":"2024-06-09T22:20:57.692594+0000","last_clean":"2024-06-09T22:20:57.692594+0000","last_became_active":"2024-06-09T22:17:26.471950+0000","last_became_peered":"2024-06-09T22:17:26.471950+0000","last_unstale":"2024-06-09T22:20:57.692594+0000","last_undegraded":"2024-06-09T22:20:57.692594+0000","last_fullsized":"2024-06-09T22:20:57.692594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:35:57.481510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"99'4","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703072+0000","last_change":"2024-06-09T22:16:43.214976+0000","last_active":"2024-06-09T22:20:57.703072+0000","last_peered":"2024-06-09T22:20:57.703072+0000","last_clean":"2024-06-09T22:20:57.703072+0000","last_became_active":"2024-06-09T22:16:43.214592+0000","last_became_peered":"2024-06-09T22:16:43.214592+0000","last_unstale":"2024-06-09T22:20:57.703072+0000","last_undegraded":"2024-06-09T22:20:57.703072+0000","last_fullsized":"2024-06-09T22:20:57.703072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:16:21.981144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595597+0000","last_change":"2024-06-09T22:16:53.342721+0000","last_active":"2024-06-09T22:20:57.595597+0000","last_peered":"2024-06-09T22:20:57.595597+0000","last_clean":"2024-06-09T22:20:57.595597+0000","last_became_active":"2024-06-09T22:16:53.342342+0000","last_became_peered":"2024-06-09T22:16:53.342342+0000","last_unstale":"2024-06-09T22:20:57.595597+0000","last_undegraded":"2024-06-09T22:20:57.595597+0000","last_fullsized":"2024-06-09T22:20:57.595597+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:22:07.469873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"97'5","reported_seq":241,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596811+0000","last_change":"2024-06-09T22:16:40.041307+0000","last_active":"2024-06-09T22:20:57.596811+0000","last_peered":"2024-06-09T22:20:57.596811+0000","last_clean":"2024-06-09T22:20:57.596811+0000","last_became_active":"2024-06-09T22:16:40.040819+0000","last_became_peered":"2024-06-09T22:16:40.040819+0000","last_unstale":"2024-06-09T22:20:57.596811+0000","last_undegraded":"2024-06-09T22:20:57.596811+0000","last_fullsized":"2024-06-09T22:20:57.596811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:14:29.658293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"99'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702925+0000","last_change":"2024-06-09T22:16:43.215719+0000","last_active":"2024-06-09T22:20:57.702925+0000","last_peered":"2024-06-09T22:20:57.702925+0000","last_clean":"2024-06-09T22:20:57.702925+0000","last_became_active":"2024-06-09T22:16:43.215356+0000","last_became_peered":"2024-06-09T22:16:43.215356+0000","last_unstale":"2024-06-09T22:20:57.702925+0000","last_undegraded":"2024-06-09T22:20:57.702925+0000","last_fullsized":"2024-06-09T22:20:57.702925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:15:07.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596001+0000","last_change":"2024-06-09T22:16:53.342948+0000","last_active":"2024-06-09T22:20:57.596001+0000","last_peered":"2024-06-09T22:20:57.596001+0000","last_clean":"2024-06-09T22:20:57.596001+0000","last_became_active":"2024-06-09T22:16:53.342352+0000","last_became_peered":"2024-06-09T22:16:53.342352+0000","last_unstale":"2024-06-09T22:20:57.596001+0000","last_undegraded":"2024-06-09T22:20:57.596001+0000","last_fullsized":"2024-06-09T22:20:57.596001+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:43:27.453789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703458+0000","last_change":"2024-06-09T22:16:43.204897+0000","last_active":"2024-06-09T22:20:57.703458+0000","last_peered":"2024-06-09T22:20:57.703458+0000","last_clean":"2024-06-09T22:20:57.703458+0000","last_became_active":"2024-06-09T22:16:43.204470+0000","last_became_peered":"2024-06-09T22:16:43.204470+0000","last_unstale":"2024-06-09T22:20:57.703458+0000","last_undegraded":"2024-06-09T22:20:57.703458+0000","last_fullsized":"2024-06-09T22:20:57.703458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:56.681392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"99'7","reported_seq":248,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:01.708116+0000","last_change":"2024-06-09T22:16:40.070079+0000","last_active":"2024-06-09T22:21:01.708116+0000","last_peered":"2024-06-09T22:21:01.708116+0000","last_clean":"2024-06-09T22:21:01.708116+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:21:01.708116+0000","last_undegraded":"2024-06-09T22:21:01.708116+0000","last_fullsized":"2024-06-09T22:21:01.708116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:00:45.132139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":193,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"97'1","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596821+0000","last_change":"2024-06-09T22:16:53.338891+0000","last_active":"2024-06-09T22:20:57.596821+0000","last_peered":"2024-06-09T22:20:57.596821+0000","last_clean":"2024-06-09T22:20:57.596821+0000","last_became_active":"2024-06-09T22:16:53.338526+0000","last_became_peered":"2024-06-09T22:16:53.338526+0000","last_unstale":"2024-06-09T22:20:57.596821+0000","last_undegraded":"2024-06-09T22:20:57.596821+0000","last_fullsized":"2024-06-09T22:20:57.596821+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:34:37.004407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695269+0000","last_change":"2024-06-09T22:16:43.199234+0000","last_active":"2024-06-09T22:20:57.695269+0000","last_peered":"2024-06-09T22:20:57.695269+0000","last_clean":"2024-06-09T22:20:57.695269+0000","last_became_active":"2024-06-09T22:16:43.192012+0000","last_became_peered":"2024-06-09T22:16:43.192012+0000","last_unstale":"2024-06-09T22:20:57.695269+0000","last_undegraded":"2024-06-09T22:20:57.695269+0000","last_fullsized":"2024-06-09T22:20:57.695269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:17.429236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"97'6","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703645+0000","last_change":"2024-06-09T22:16:40.041970+0000","last_active":"2024-06-09T22:20:57.703645+0000","last_peered":"2024-06-09T22:20:57.703645+0000","last_clean":"2024-06-09T22:20:57.703645+0000","last_became_active":"2024-06-09T22:16:40.040371+0000","last_became_peered":"2024-06-09T22:16:40.040371+0000","last_unstale":"2024-06-09T22:20:57.703645+0000","last_undegraded":"2024-06-09T22:20:57.703645+0000","last_fullsized":"2024-06-09T22:20:57.703645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:52:31.210423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":381,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596577+0000","last_change":"2024-06-09T22:15:15.367397+0000","last_active":"2024-06-09T22:20:57.596577+0000","last_peered":"2024-06-09T22:20:57.596577+0000","last_clean":"2024-06-09T22:20:57.596577+0000","last_became_active":"2024-06-09T22:15:15.366974+0000","last_became_peered":"2024-06-09T22:15:15.366974+0000","last_unstale":"2024-06-09T22:20:57.596577+0000","last_undegraded":"2024-06-09T22:20:57.596577+0000","last_fullsized":"2024-06-09T22:20:57.596577+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_clean_scrub_stamp":"2024-06-09T22:11:49.886056+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:24.018031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703964+0000","last_change":"2024-06-09T22:16:53.342249+0000","last_active":"2024-06-09T22:20:57.703964+0000","last_peered":"2024-06-09T22:20:57.703964+0000","last_clean":"2024-06-09T22:20:57.703964+0000","last_became_active":"2024-06-09T22:16:53.341779+0000","last_became_peered":"2024-06-09T22:16:53.341779+0000","last_unstale":"2024-06-09T22:20:57.703964+0000","last_undegraded":"2024-06-09T22:20:57.703964+0000","last_fullsized":"2024-06-09T22:20:57.703964+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:21:21.101578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"99'5","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595425+0000","last_change":"2024-06-09T22:16:43.200778+0000","last_active":"2024-06-09T22:20:57.595425+0000","last_peered":"2024-06-09T22:20:57.595425+0000","last_clean":"2024-06-09T22:20:57.595425+0000","last_became_active":"2024-06-09T22:16:43.200490+0000","last_became_peered":"2024-06-09T22:16:43.200490+0000","last_unstale":"2024-06-09T22:20:57.595425+0000","last_undegraded":"2024-06-09T22:20:57.595425+0000","last_fullsized":"2024-06-09T22:20:57.595425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:20:17.530204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"97'4","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596731+0000","last_change":"2024-06-09T22:16:40.041774+0000","last_active":"2024-06-09T22:20:57.596731+0000","last_peered":"2024-06-09T22:20:57.596731+0000","last_clean":"2024-06-09T22:20:57.596731+0000","last_became_active":"2024-06-09T22:16:40.041085+0000","last_became_peered":"2024-06-09T22:16:40.041085+0000","last_unstale":"2024-06-09T22:20:57.596731+0000","last_undegraded":"2024-06-09T22:20:57.596731+0000","last_fullsized":"2024-06-09T22:20:57.596731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:09:41.140083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710062+0000","last_change":"2024-06-09T22:16:53.336895+0000","last_active":"2024-06-09T22:20:57.710062+0000","last_peered":"2024-06-09T22:20:57.710062+0000","last_clean":"2024-06-09T22:20:57.710062+0000","last_became_active":"2024-06-09T22:16:53.336497+0000","last_became_peered":"2024-06-09T22:16:53.336497+0000","last_unstale":"2024-06-09T22:20:57.710062+0000","last_undegraded":"2024-06-09T22:20:57.710062+0000","last_fullsized":"2024-06-09T22:20:57.710062+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:29:52.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"97'4","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695101+0000","last_change":"2024-06-09T22:16:40.070909+0000","last_active":"2024-06-09T22:20:57.695101+0000","last_peered":"2024-06-09T22:20:57.695101+0000","last_clean":"2024-06-09T22:20:57.695101+0000","last_became_active":"2024-06-09T22:16:40.070556+0000","last_became_peered":"2024-06-09T22:16:40.070556+0000","last_unstale":"2024-06-09T22:20:57.695101+0000","last_undegraded":"2024-06-09T22:20:57.695101+0000","last_fullsized":"2024-06-09T22:20:57.695101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:59:17.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"99'9","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595793+0000","last_change":"2024-06-09T22:16:43.205951+0000","last_active":"2024-06-09T22:20:57.595793+0000","last_peered":"2024-06-09T22:20:57.595793+0000","last_clean":"2024-06-09T22:20:57.595793+0000","last_became_active":"2024-06-09T22:16:43.205594+0000","last_became_peered":"2024-06-09T22:16:43.205594+0000","last_unstale":"2024-06-09T22:20:57.595793+0000","last_undegraded":"2024-06-09T22:20:57.595793+0000","last_fullsized":"2024-06-09T22:20:57.595793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:17.956406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694724+0000","last_change":"2024-06-09T22:16:53.343292+0000","last_active":"2024-06-09T22:20:57.694724+0000","last_peered":"2024-06-09T22:20:57.694724+0000","last_clean":"2024-06-09T22:20:57.694724+0000","last_became_active":"2024-06-09T22:16:53.342900+0000","last_became_peered":"2024-06-09T22:16:53.342900+0000","last_unstale":"2024-06-09T22:20:57.694724+0000","last_undegraded":"2024-06-09T22:20:57.694724+0000","last_fullsized":"2024-06-09T22:20:57.694724+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:08:07.328250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"100'51","reported_seq":120,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.812147+0000","last_change":"2024-06-09T22:16:40.039417+0000","last_active":"2024-06-09T22:20:57.812147+0000","last_peered":"2024-06-09T22:20:57.812147+0000","last_clean":"2024-06-09T22:20:57.812147+0000","last_became_active":"2024-06-09T22:16:40.039099+0000","last_became_peered":"2024-06-09T22:16:40.039099+0000","last_unstale":"2024-06-09T22:20:57.812147+0000","last_undegraded":"2024-06-09T22:20:57.812147+0000","last_fullsized":"2024-06-09T22:20:57.812147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:22:59.284442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":90,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"99'5","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696131+0000","last_change":"2024-06-09T22:16:43.201285+0000","last_active":"2024-06-09T22:20:57.696131+0000","last_peered":"2024-06-09T22:20:57.696131+0000","last_clean":"2024-06-09T22:20:57.696131+0000","last_became_active":"2024-06-09T22:16:43.200993+0000","last_became_peered":"2024-06-09T22:16:43.200993+0000","last_unstale":"2024-06-09T22:20:57.696131+0000","last_undegraded":"2024-06-09T22:20:57.696131+0000","last_fullsized":"2024-06-09T22:20:57.696131+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:15:26.491588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596798+0000","last_change":"2024-06-09T22:16:53.334088+0000","last_active":"2024-06-09T22:20:57.596798+0000","last_peered":"2024-06-09T22:20:57.596798+0000","last_clean":"2024-06-09T22:20:57.596798+0000","last_became_active":"2024-06-09T22:16:53.333597+0000","last_became_peered":"2024-06-09T22:16:53.333597+0000","last_unstale":"2024-06-09T22:20:57.596798+0000","last_undegraded":"2024-06-09T22:20:57.596798+0000","last_fullsized":"2024-06-09T22:20:57.596798+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:50:22.041816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"97'7","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703700+0000","last_change":"2024-06-09T22:16:40.364982+0000","last_active":"2024-06-09T22:20:57.703700+0000","last_peered":"2024-06-09T22:20:57.703700+0000","last_clean":"2024-06-09T22:20:57.703700+0000","last_became_active":"2024-06-09T22:16:40.364585+0000","last_became_peered":"2024-06-09T22:16:40.364585+0000","last_unstale":"2024-06-09T22:20:57.703700+0000","last_undegraded":"2024-06-09T22:20:57.703700+0000","last_fullsized":"2024-06-09T22:20:57.703700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:28.777950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"99'7","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711181+0000","last_change":"2024-06-09T22:17:26.474491+0000","last_active":"2024-06-09T22:20:57.711181+0000","last_peered":"2024-06-09T22:20:57.711181+0000","last_clean":"2024-06-09T22:20:57.711181+0000","last_became_active":"2024-06-09T22:17:26.473975+0000","last_became_peered":"2024-06-09T22:17:26.473975+0000","last_unstale":"2024-06-09T22:20:57.711181+0000","last_undegraded":"2024-06-09T22:20:57.711181+0000","last_fullsized":"2024-06-09T22:20:57.711181+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:13:46.998595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703384+0000","last_change":"2024-06-09T22:16:53.342273+0000","last_active":"2024-06-09T22:20:57.703384+0000","last_peered":"2024-06-09T22:20:57.703384+0000","last_clean":"2024-06-09T22:20:57.703384+0000","last_became_active":"2024-06-09T22:16:53.341827+0000","last_became_peered":"2024-06-09T22:16:53.341827+0000","last_unstale":"2024-06-09T22:20:57.703384+0000","last_undegraded":"2024-06-09T22:20:57.703384+0000","last_fullsized":"2024-06-09T22:20:57.703384+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:23:26.099737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"97'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709982+0000","last_change":"2024-06-09T22:16:40.070498+0000","last_active":"2024-06-09T22:20:57.709982+0000","last_peered":"2024-06-09T22:20:57.709982+0000","last_clean":"2024-06-09T22:20:57.709982+0000","last_became_active":"2024-06-09T22:16:40.070144+0000","last_became_peered":"2024-06-09T22:16:40.070144+0000","last_unstale":"2024-06-09T22:20:57.709982+0000","last_undegraded":"2024-06-09T22:20:57.709982+0000","last_fullsized":"2024-06-09T22:20:57.709982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:52:22.214832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"99'2","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694467+0000","last_change":"2024-06-09T22:16:43.210448+0000","last_active":"2024-06-09T22:20:57.694467+0000","last_peered":"2024-06-09T22:20:57.694467+0000","last_clean":"2024-06-09T22:20:57.694467+0000","last_became_active":"2024-06-09T22:16:43.209980+0000","last_became_peered":"2024-06-09T22:16:43.209980+0000","last_unstale":"2024-06-09T22:20:57.694467+0000","last_undegraded":"2024-06-09T22:20:57.694467+0000","last_fullsized":"2024-06-09T22:20:57.694467+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:58:15.691207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692864+0000","last_change":"2024-06-09T22:16:53.340644+0000","last_active":"2024-06-09T22:20:57.692864+0000","last_peered":"2024-06-09T22:20:57.692864+0000","last_clean":"2024-06-09T22:20:57.692864+0000","last_became_active":"2024-06-09T22:16:53.339771+0000","last_became_peered":"2024-06-09T22:16:53.339771+0000","last_unstale":"2024-06-09T22:20:57.692864+0000","last_undegraded":"2024-06-09T22:20:57.692864+0000","last_fullsized":"2024-06-09T22:20:57.692864+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:45:23.281387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"97'7","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694859+0000","last_change":"2024-06-09T22:16:40.040461+0000","last_active":"2024-06-09T22:20:57.694859+0000","last_peered":"2024-06-09T22:20:57.694859+0000","last_clean":"2024-06-09T22:20:57.694859+0000","last_became_active":"2024-06-09T22:16:40.040168+0000","last_became_peered":"2024-06-09T22:16:40.040168+0000","last_unstale":"2024-06-09T22:20:57.694859+0000","last_undegraded":"2024-06-09T22:20:57.694859+0000","last_fullsized":"2024-06-09T22:20:57.694859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:28.936631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"99'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709897+0000","last_change":"2024-06-09T22:16:43.208779+0000","last_active":"2024-06-09T22:20:57.709897+0000","last_peered":"2024-06-09T22:20:57.709897+0000","last_clean":"2024-06-09T22:20:57.709897+0000","last_became_active":"2024-06-09T22:16:43.208136+0000","last_became_peered":"2024-06-09T22:16:43.208136+0000","last_unstale":"2024-06-09T22:20:57.709897+0000","last_undegraded":"2024-06-09T22:20:57.709897+0000","last_fullsized":"2024-06-09T22:20:57.709897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:22:39.721069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"97'1","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703384+0000","last_change":"2024-06-09T22:16:53.331241+0000","last_active":"2024-06-09T22:20:57.703384+0000","last_peered":"2024-06-09T22:20:57.703384+0000","last_clean":"2024-06-09T22:20:57.703384+0000","last_became_active":"2024-06-09T22:16:53.329589+0000","last_became_peered":"2024-06-09T22:16:53.329589+0000","last_unstale":"2024-06-09T22:20:57.703384+0000","last_undegraded":"2024-06-09T22:20:57.703384+0000","last_fullsized":"2024-06-09T22:20:57.703384+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:56:21.319711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"100'4","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596918+0000","last_change":"2024-06-09T22:16:40.042491+0000","last_active":"2024-06-09T22:20:57.596918+0000","last_peered":"2024-06-09T22:20:57.596918+0000","last_clean":"2024-06-09T22:20:57.596918+0000","last_became_active":"2024-06-09T22:16:40.041846+0000","last_became_peered":"2024-06-09T22:16:40.041846+0000","last_unstale":"2024-06-09T22:20:57.596918+0000","last_undegraded":"2024-06-09T22:20:57.596918+0000","last_fullsized":"2024-06-09T22:20:57.596918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:59.144003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695844+0000","last_change":"2024-06-09T22:16:43.209195+0000","last_active":"2024-06-09T22:20:57.695844+0000","last_peered":"2024-06-09T22:20:57.695844+0000","last_clean":"2024-06-09T22:20:57.695844+0000","last_became_active":"2024-06-09T22:16:43.208797+0000","last_became_peered":"2024-06-09T22:16:43.208797+0000","last_unstale":"2024-06-09T22:20:57.695844+0000","last_undegraded":"2024-06-09T22:20:57.695844+0000","last_fullsized":"2024-06-09T22:20:57.695844+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:05:31.689900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703290+0000","last_change":"2024-06-09T22:16:53.342173+0000","last_active":"2024-06-09T22:20:57.703290+0000","last_peered":"2024-06-09T22:20:57.703290+0000","last_clean":"2024-06-09T22:20:57.703290+0000","last_became_active":"2024-06-09T22:16:53.341712+0000","last_became_peered":"2024-06-09T22:16:53.341712+0000","last_unstale":"2024-06-09T22:20:57.703290+0000","last_undegraded":"2024-06-09T22:20:57.703290+0000","last_fullsized":"2024-06-09T22:20:57.703290+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:53:12.574764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"97'4","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694569+0000","last_change":"2024-06-09T22:16:40.044238+0000","last_active":"2024-06-09T22:20:57.694569+0000","last_peered":"2024-06-09T22:20:57.694569+0000","last_clean":"2024-06-09T22:20:57.694569+0000","last_became_active":"2024-06-09T22:16:40.043857+0000","last_became_peered":"2024-06-09T22:16:40.043857+0000","last_unstale":"2024-06-09T22:20:57.694569+0000","last_undegraded":"2024-06-09T22:20:57.694569+0000","last_fullsized":"2024-06-09T22:20:57.694569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:44:45.743307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"99'5","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596886+0000","last_change":"2024-06-09T22:16:43.202363+0000","last_active":"2024-06-09T22:20:57.596886+0000","last_peered":"2024-06-09T22:20:57.596886+0000","last_clean":"2024-06-09T22:20:57.596886+0000","last_became_active":"2024-06-09T22:16:43.202029+0000","last_became_peered":"2024-06-09T22:16:43.202029+0000","last_unstale":"2024-06-09T22:20:57.596886+0000","last_undegraded":"2024-06-09T22:20:57.596886+0000","last_fullsized":"2024-06-09T22:20:57.596886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:11:19.513055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710013+0000","last_change":"2024-06-09T22:16:53.333837+0000","last_active":"2024-06-09T22:20:57.710013+0000","last_peered":"2024-06-09T22:20:57.710013+0000","last_clean":"2024-06-09T22:20:57.710013+0000","last_became_active":"2024-06-09T22:16:53.333362+0000","last_became_peered":"2024-06-09T22:16:53.333362+0000","last_unstale":"2024-06-09T22:20:57.710013+0000","last_undegraded":"2024-06-09T22:20:57.710013+0000","last_fullsized":"2024-06-09T22:20:57.710013+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:47:04.463040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"99'7","reported_seq":243,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597416+0000","last_change":"2024-06-09T22:16:40.364834+0000","last_active":"2024-06-09T22:20:57.597416+0000","last_peered":"2024-06-09T22:20:57.597416+0000","last_clean":"2024-06-09T22:20:57.597416+0000","last_became_active":"2024-06-09T22:16:40.364433+0000","last_became_peered":"2024-06-09T22:16:40.364433+0000","last_unstale":"2024-06-09T22:20:57.597416+0000","last_undegraded":"2024-06-09T22:20:57.597416+0000","last_fullsized":"2024-06-09T22:20:57.597416+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:43:01.967911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":191,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597174+0000","last_change":"2024-06-09T22:16:43.201183+0000","last_active":"2024-06-09T22:20:57.597174+0000","last_peered":"2024-06-09T22:20:57.597174+0000","last_clean":"2024-06-09T22:20:57.597174+0000","last_became_active":"2024-06-09T22:16:43.200774+0000","last_became_peered":"2024-06-09T22:16:43.200774+0000","last_unstale":"2024-06-09T22:20:57.597174+0000","last_undegraded":"2024-06-09T22:20:57.597174+0000","last_fullsized":"2024-06-09T22:20:57.597174+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:07:55.772066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710583+0000","last_change":"2024-06-09T22:18:26.699190+0000","last_active":"2024-06-09T22:20:57.710583+0000","last_peered":"2024-06-09T22:20:57.710583+0000","last_clean":"2024-06-09T22:20:57.710583+0000","last_became_active":"2024-06-09T22:18:26.698614+0000","last_became_peered":"2024-06-09T22:18:26.698614+0000","last_unstale":"2024-06-09T22:20:57.710583+0000","last_undegraded":"2024-06-09T22:20:57.710583+0000","last_fullsized":"2024-06-09T22:20:57.710583+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:45.343605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,1,4],"acting":[0,10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"100'41","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.804410+0000","last_change":"2024-06-09T22:16:40.040582+0000","last_active":"2024-06-09T22:20:57.804410+0000","last_peered":"2024-06-09T22:20:57.804410+0000","last_clean":"2024-06-09T22:20:57.804410+0000","last_became_active":"2024-06-09T22:16:40.040152+0000","last_became_peered":"2024-06-09T22:16:40.040152+0000","last_unstale":"2024-06-09T22:20:57.804410+0000","last_undegraded":"2024-06-09T22:20:57.804410+0000","last_fullsized":"2024-06-09T22:20:57.804410+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:50:00.878096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596909+0000","last_change":"2024-06-09T22:16:43.206211+0000","last_active":"2024-06-09T22:20:57.596909+0000","last_peered":"2024-06-09T22:20:57.596909+0000","last_clean":"2024-06-09T22:20:57.596909+0000","last_became_active":"2024-06-09T22:16:43.205814+0000","last_became_peered":"2024-06-09T22:16:43.205814+0000","last_unstale":"2024-06-09T22:20:57.596909+0000","last_undegraded":"2024-06-09T22:20:57.596909+0000","last_fullsized":"2024-06-09T22:20:57.596909+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:20:19.058475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703043+0000","last_change":"2024-06-09T22:16:53.337695+0000","last_active":"2024-06-09T22:20:57.703043+0000","last_peered":"2024-06-09T22:20:57.703043+0000","last_clean":"2024-06-09T22:20:57.703043+0000","last_became_active":"2024-06-09T22:16:53.334872+0000","last_became_peered":"2024-06-09T22:16:53.334872+0000","last_unstale":"2024-06-09T22:20:57.703043+0000","last_undegraded":"2024-06-09T22:20:57.703043+0000","last_fullsized":"2024-06-09T22:20:57.703043+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:50:36.617736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596073+0000","last_change":"2024-06-09T22:16:40.036603+0000","last_active":"2024-06-09T22:20:57.596073+0000","last_peered":"2024-06-09T22:20:57.596073+0000","last_clean":"2024-06-09T22:20:57.596073+0000","last_became_active":"2024-06-09T22:16:40.036274+0000","last_became_peered":"2024-06-09T22:16:40.036274+0000","last_unstale":"2024-06-09T22:20:57.596073+0000","last_undegraded":"2024-06-09T22:20:57.596073+0000","last_fullsized":"2024-06-09T22:20:57.596073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:31.652897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"99'9","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596989+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:20:57.596989+0000","last_peered":"2024-06-09T22:20:57.596989+0000","last_clean":"2024-06-09T22:20:57.596989+0000","last_became_active":"2024-06-09T22:16:43.204197+0000","last_became_peered":"2024-06-09T22:16:43.204197+0000","last_unstale":"2024-06-09T22:20:57.596989+0000","last_undegraded":"2024-06-09T22:20:57.596989+0000","last_fullsized":"2024-06-09T22:20:57.596989+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:38:49.850120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710013+0000","last_change":"2024-06-09T22:16:43.208477+0000","last_active":"2024-06-09T22:20:57.710013+0000","last_peered":"2024-06-09T22:20:57.710013+0000","last_clean":"2024-06-09T22:20:57.710013+0000","last_became_active":"2024-06-09T22:16:43.206018+0000","last_became_peered":"2024-06-09T22:16:43.206018+0000","last_unstale":"2024-06-09T22:20:57.710013+0000","last_undegraded":"2024-06-09T22:20:57.710013+0000","last_fullsized":"2024-06-09T22:20:57.710013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:19:49.870825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"99'12","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597095+0000","last_change":"2024-06-09T22:16:43.206809+0000","last_active":"2024-06-09T22:20:57.597095+0000","last_peered":"2024-06-09T22:20:57.597095+0000","last_clean":"2024-06-09T22:20:57.597095+0000","last_became_active":"2024-06-09T22:16:43.206175+0000","last_became_peered":"2024-06-09T22:16:43.206175+0000","last_unstale":"2024-06-09T22:20:57.597095+0000","last_undegraded":"2024-06-09T22:20:57.597095+0000","last_fullsized":"2024-06-09T22:20:57.597095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:39:22.502266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596879+0000","last_change":"2024-06-09T22:16:43.202163+0000","last_active":"2024-06-09T22:20:57.596879+0000","last_peered":"2024-06-09T22:20:57.596879+0000","last_clean":"2024-06-09T22:20:57.596879+0000","last_became_active":"2024-06-09T22:16:43.201839+0000","last_became_peered":"2024-06-09T22:16:43.201839+0000","last_unstale":"2024-06-09T22:20:57.596879+0000","last_undegraded":"2024-06-09T22:20:57.596879+0000","last_fullsized":"2024-06-09T22:20:57.596879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:47:46.723289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710813+0000","last_change":"2024-06-09T22:16:43.202734+0000","last_active":"2024-06-09T22:20:57.710813+0000","last_peered":"2024-06-09T22:20:57.710813+0000","last_clean":"2024-06-09T22:20:57.710813+0000","last_became_active":"2024-06-09T22:16:43.201638+0000","last_became_peered":"2024-06-09T22:16:43.201638+0000","last_unstale":"2024-06-09T22:20:57.710813+0000","last_undegraded":"2024-06-09T22:20:57.710813+0000","last_fullsized":"2024-06-09T22:20:57.710813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:16.360985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"99'4","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595768+0000","last_change":"2024-06-09T22:16:43.199113+0000","last_active":"2024-06-09T22:20:57.595768+0000","last_peered":"2024-06-09T22:20:57.595768+0000","last_clean":"2024-06-09T22:20:57.595768+0000","last_became_active":"2024-06-09T22:16:43.198804+0000","last_became_peered":"2024-06-09T22:16:43.198804+0000","last_unstale":"2024-06-09T22:20:57.595768+0000","last_undegraded":"2024-06-09T22:20:57.595768+0000","last_fullsized":"2024-06-09T22:20:57.595768+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:18:08.999477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"99'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710339+0000","last_change":"2024-06-09T22:16:43.204445+0000","last_active":"2024-06-09T22:20:57.710339+0000","last_peered":"2024-06-09T22:20:57.710339+0000","last_clean":"2024-06-09T22:20:57.710339+0000","last_became_active":"2024-06-09T22:16:43.203081+0000","last_became_peered":"2024-06-09T22:16:43.203081+0000","last_unstale":"2024-06-09T22:20:57.710339+0000","last_undegraded":"2024-06-09T22:20:57.710339+0000","last_fullsized":"2024-06-09T22:20:57.710339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:39.303813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"99'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597592+0000","last_change":"2024-06-09T22:16:43.211402+0000","last_active":"2024-06-09T22:20:57.597592+0000","last_peered":"2024-06-09T22:20:57.597592+0000","last_clean":"2024-06-09T22:20:57.597592+0000","last_became_active":"2024-06-09T22:16:43.210871+0000","last_became_peered":"2024-06-09T22:16:43.210871+0000","last_unstale":"2024-06-09T22:20:57.597592+0000","last_undegraded":"2024-06-09T22:20:57.597592+0000","last_fullsized":"2024-06-09T22:20:57.597592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:07:21.467617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"99'2","reported_seq":42,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696237+0000","last_change":"2024-06-09T22:16:43.201159+0000","last_active":"2024-06-09T22:20:57.696237+0000","last_peered":"2024-06-09T22:20:57.696237+0000","last_clean":"2024-06-09T22:20:57.696237+0000","last_became_active":"2024-06-09T22:16:43.200850+0000","last_became_peered":"2024-06-09T22:16:43.200850+0000","last_unstale":"2024-06-09T22:20:57.696237+0000","last_undegraded":"2024-06-09T22:20:57.696237+0000","last_fullsized":"2024-06-09T22:20:57.696237+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:20.515309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"99'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703213+0000","last_change":"2024-06-09T22:16:43.196830+0000","last_active":"2024-06-09T22:20:57.703213+0000","last_peered":"2024-06-09T22:20:57.703213+0000","last_clean":"2024-06-09T22:20:57.703213+0000","last_became_active":"2024-06-09T22:16:43.195682+0000","last_became_peered":"2024-06-09T22:16:43.195682+0000","last_unstale":"2024-06-09T22:20:57.703213+0000","last_undegraded":"2024-06-09T22:20:57.703213+0000","last_fullsized":"2024-06-09T22:20:57.703213+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:34:18.770235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694921+0000","last_change":"2024-06-09T22:16:43.201959+0000","last_active":"2024-06-09T22:20:57.694921+0000","last_peered":"2024-06-09T22:20:57.694921+0000","last_clean":"2024-06-09T22:20:57.694921+0000","last_became_active":"2024-06-09T22:16:43.201669+0000","last_became_peered":"2024-06-09T22:16:43.201669+0000","last_unstale":"2024-06-09T22:20:57.694921+0000","last_undegraded":"2024-06-09T22:20:57.694921+0000","last_fullsized":"2024-06-09T22:20:57.694921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:48:57.348041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710198+0000","last_change":"2024-06-09T22:16:43.201823+0000","last_active":"2024-06-09T22:20:57.710198+0000","last_peered":"2024-06-09T22:20:57.710198+0000","last_clean":"2024-06-09T22:20:57.710198+0000","last_became_active":"2024-06-09T22:16:43.201434+0000","last_became_peered":"2024-06-09T22:16:43.201434+0000","last_unstale":"2024-06-09T22:20:57.710198+0000","last_undegraded":"2024-06-09T22:20:57.710198+0000","last_fullsized":"2024-06-09T22:20:57.710198+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:56:13.540895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"99'11","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595624+0000","last_change":"2024-06-09T22:16:43.197734+0000","last_active":"2024-06-09T22:20:57.595624+0000","last_peered":"2024-06-09T22:20:57.595624+0000","last_clean":"2024-06-09T22:20:57.595624+0000","last_became_active":"2024-06-09T22:16:43.197442+0000","last_became_peered":"2024-06-09T22:16:43.197442+0000","last_unstale":"2024-06-09T22:20:57.595624+0000","last_undegraded":"2024-06-09T22:20:57.595624+0000","last_fullsized":"2024-06-09T22:20:57.595624+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:35:40.720718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710339+0000","last_change":"2024-06-09T22:16:43.210375+0000","last_active":"2024-06-09T22:20:57.710339+0000","last_peered":"2024-06-09T22:20:57.710339+0000","last_clean":"2024-06-09T22:20:57.710339+0000","last_became_active":"2024-06-09T22:16:43.209647+0000","last_became_peered":"2024-06-09T22:16:43.209647+0000","last_unstale":"2024-06-09T22:20:57.710339+0000","last_undegraded":"2024-06-09T22:20:57.710339+0000","last_fullsized":"2024-06-09T22:20:57.710339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:11:57.593066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107643,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":607457,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2015,"num_read_kb":1129,"num_write":387,"num_write_kb":1481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":737280,"data_stored":1822761,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383747,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":580840,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645384,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32424,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712728,"statfs":{"total":95995035648,"available":95961833472,"internally_reserved":0,"allocated":1273856,"data_stored":1236852,"data_compressed":71140,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808909,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1167360,"data_stored":977483,"data_compressed":54405,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070546,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1159168,"data_stored":973765,"data_compressed":54405,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332183,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32340,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712812,"statfs":{"total":95995035648,"available":95961919488,"internally_reserved":0,"allocated":1187840,"data_stored":1178025,"data_compressed":71140,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495710,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1114112,"data_stored":576065,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36504,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708648,"statfs":{"total":95995035648,"available":95957655552,"internally_reserved":0,"allocated":1191936,"data_stored":650243,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708700,"statfs":{"total":95995035648,"available":95957708800,"internally_reserved":0,"allocated":1138688,"data_stored":577662,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":651847,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182507,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1171456,"data_stored":1100199,"data_compressed":60561,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920871,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32276,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712876,"statfs":{"total":95995035648,"available":95961985024,"internally_reserved":0,"allocated":1122304,"data_stored":580978,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757346,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36524,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708628,"statfs":{"total":95995035648,"available":95957635072,"internally_reserved":0,"allocated":1216512,"data_stored":1304758,"data_compressed":77296,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":75290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":76894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135341,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":208103,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-09T22:21:04.568 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:21:04.841 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:05.505 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:05 smithi104 ceph-mon[32389]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-06-09T22:21:05.506 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:05 smithi104 ceph-mon[32389]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:05.506 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:05 smithi104 ceph-mon[32389]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:05.506 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:05 smithi104 ceph-mon[32389]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:05.557 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:05 smithi194 ceph-mon[36856]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-06-09T22:21:05.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:05 smithi194 ceph-mon[36856]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:05.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:05 smithi194 ceph-mon[36856]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:05.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:05 smithi194 ceph-mon[36856]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:05.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:05 smithi153 ceph-mon[36599]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-06-09T22:21:05.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:05 smithi153 ceph-mon[36599]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:05.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:05 smithi153 ceph-mon[36599]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:05.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:05 smithi153 ceph-mon[36599]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:06.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.594+0000 7f12ac702700 1 -- 172.21.15.104:0/3506895557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a4101d80 msgr2=0x7f12a410e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:06.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.594+0000 7f12ac702700 1 --2- 172.21.15.104:0/3506895557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a4101d80 0x7f12a410e240 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f129c00b0a0 tx=0x7f129c00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:21:06.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.594+0000 7f12ac702700 1 -- 172.21.15.104:0/3506895557 shutdown_connections 2024-06-09T22:21:06.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.594+0000 7f12ac702700 1 --2- 172.21.15.104:0/3506895557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a4101d80 0x7f12a410e240 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:06.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.594+0000 7f12ac702700 1 --2- 172.21.15.104:0/3506895557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a4107440 0x7f12a4107820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:06.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.594+0000 7f12ac702700 1 --2- 172.21.15.104:0/3506895557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12a41013c0 0x7f12a4101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:06.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.594+0000 7f12ac702700 1 -- 172.21.15.104:0/3506895557 >> 172.21.15.104:0/3506895557 conn(0x7f12a4077400 msgr2=0x7f12a4077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:06.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.594+0000 7f12ac702700 1 -- 172.21.15.104:0/3506895557 shutdown_connections 2024-06-09T22:21:06.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.594+0000 7f12ac702700 1 -- 172.21.15.104:0/3506895557 wait complete. 2024-06-09T22:21:06.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.594+0000 7f12ac702700 1 Processor -- start 2024-06-09T22:21:06.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.594+0000 7f12ac702700 1 -- start start 2024-06-09T22:21:06.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12ac702700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a41013c0 0x7f12a419e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:06.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12ac702700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a4101d80 0x7f12a419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:06.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12ac702700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12a4107440 0x7f12a4198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:06.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12ac702700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4111680 con 0x7f12a4101d80 2024-06-09T22:21:06.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12ac702700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4111500 con 0x7f12a41013c0 2024-06-09T22:21:06.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12ac702700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4111800 con 0x7f12a4107440 2024-06-09T22:21:06.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12a9c9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a4101d80 0x7f12a419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:06.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12a9c9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a4101d80 0x7f12a419ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37714/0 (socket says 172.21.15.104:37714) 2024-06-09T22:21:06.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12a9c9d700 1 -- 172.21.15.104:0/4261680325 learned_addr learned my addr 172.21.15.104:0/4261680325 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:06.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12aac9f700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12a4107440 0x7f12a4198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:06.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12aa49e700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a41013c0 0x7f12a419e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:06.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12a9c9d700 1 -- 172.21.15.104:0/4261680325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12a4107440 msgr2=0x7f12a4198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:06.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12a9c9d700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12a4107440 0x7f12a4198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:06.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12a9c9d700 1 -- 172.21.15.104:0/4261680325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a41013c0 msgr2=0x7f12a419e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:06.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12a9c9d700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a41013c0 0x7f12a419e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:06.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12a9c9d700 1 -- 172.21.15.104:0/4261680325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a4199170 con 0x7f12a4101d80 2024-06-09T22:21:06.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f12a9c9d700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a4101d80 0x7f12a419ed40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f129400c970 tx=0x7f129400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:06.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.595+0000 7f129b7fe700 1 -- 172.21.15.104:0/4261680325 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12940078d0 con 0x7f12a4101d80 2024-06-09T22:21:06.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.596+0000 7f12ac702700 1 -- 172.21.15.104:0/4261680325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a4199460 con 0x7f12a4101d80 2024-06-09T22:21:06.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.596+0000 7f129b7fe700 1 -- 172.21.15.104:0/4261680325 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1294018480 con 0x7f12a4101d80 2024-06-09T22:21:06.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.596+0000 7f129b7fe700 1 -- 172.21.15.104:0/4261680325 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129400f6b0 con 0x7f12a4101d80 2024-06-09T22:21:06.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.596+0000 7f12ac702700 1 -- 172.21.15.104:0/4261680325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a41a56d0 con 0x7f12a4101d80 2024-06-09T22:21:06.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.596+0000 7f12ac702700 1 -- 172.21.15.104:0/4261680325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12a4102ab0 con 0x7f12a4101d80 2024-06-09T22:21:06.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.598+0000 7f129b7fe700 1 -- 172.21.15.104:0/4261680325 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f1294018630 con 0x7f12a4101d80 2024-06-09T22:21:06.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.598+0000 7f129b7fe700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12900a2eb0 0x7f12900a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:06.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.599+0000 7f12aa49e700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12900a2eb0 0x7f12900a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:06.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.599+0000 7f129b7fe700 1 -- 172.21.15.104:0/4261680325 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f12940cfe70 con 0x7f12a4101d80 2024-06-09T22:21:06.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.599+0000 7f12aa49e700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12900a2eb0 0x7f12900a5370 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f12a4101330 tx=0x7f12a000b410 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:06.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.602+0000 7f129b7fe700 1 -- 172.21.15.104:0/4261680325 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f129409c5b0 con 0x7f12a4101d80 2024-06-09T22:21:06.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.843+0000 7f12ac702700 1 -- 172.21.15.104:0/4261680325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12a4069180 con 0x7f12a4101d80 2024-06-09T22:21:06.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.843+0000 7f129b7fe700 1 -- 172.21.15.104:0/4261680325 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11953 (secure 0 0 0) 0x7f12a4069180 con 0x7f12a4101d80 2024-06-09T22:21:06.842 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:06.842 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:20:55.680750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24460,24448,24442,24403,24419]},"id":1}]} 2024-06-09T22:21:06.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.845+0000 7f12ac702700 1 -- 172.21.15.104:0/4261680325 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12900a2eb0 msgr2=0x7f12900a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:06.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.845+0000 7f12ac702700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12900a2eb0 0x7f12900a5370 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f12a4101330 tx=0x7f12a000b410 comp rx=0 tx=0).stop 2024-06-09T22:21:06.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.845+0000 7f12ac702700 1 -- 172.21.15.104:0/4261680325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a4101d80 msgr2=0x7f12a419ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:06.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.845+0000 7f12ac702700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a4101d80 0x7f12a419ed40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f129400c970 tx=0x7f129400cc80 comp rx=0 tx=0).stop 2024-06-09T22:21:06.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.846+0000 7f12ac702700 1 -- 172.21.15.104:0/4261680325 shutdown_connections 2024-06-09T22:21:06.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.846+0000 7f12ac702700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12900a2eb0 0x7f12900a5370 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:06.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.846+0000 7f12ac702700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12a4107440 0x7f12a4198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:06.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.846+0000 7f12ac702700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a41013c0 0x7f12a419e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:06.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.846+0000 7f12ac702700 1 --2- 172.21.15.104:0/4261680325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12a4101d80 0x7f12a419ed40 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:06.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.846+0000 7f12ac702700 1 -- 172.21.15.104:0/4261680325 >> 172.21.15.104:0/4261680325 conn(0x7f12a4077400 msgr2=0x7f12a406ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:06.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.846+0000 7f12ac702700 1 -- 172.21.15.104:0/4261680325 shutdown_connections 2024-06-09T22:21:06.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:06.846+0000 7f12ac702700 1 -- 172.21.15.104:0/4261680325 wait complete. 2024-06-09T22:21:06.846 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 29 2024-06-09T22:21:07.505 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:07 smithi104 ceph-mon[32389]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.4 KiB/s wr, 26 op/s 2024-06-09T22:21:07.506 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:07 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/4261680325' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:07.557 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:21:07.557 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:07 smithi194 ceph-mon[36856]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.4 KiB/s wr, 26 op/s 2024-06-09T22:21:07.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:07 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/4261680325' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:07.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:07 smithi153 ceph-mon[36599]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.4 KiB/s wr, 26 op/s 2024-06-09T22:21:07.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:07 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/4261680325' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:07.799 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:09.505 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:09 smithi104 ceph-mon[32389]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.1 KiB/s wr, 20 op/s 2024-06-09T22:21:09.557 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:09 smithi194 ceph-mon[36856]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.1 KiB/s wr, 20 op/s 2024-06-09T22:21:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.677+0000 7f41c9ef2700 1 -- 172.21.15.104:0/1483702048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41c406b5e0 msgr2=0x7f41c410f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.677+0000 7f41c9ef2700 1 --2- 172.21.15.104:0/1483702048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41c406b5e0 0x7f41c410f320 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00b0a0 tx=0x7f41bc00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:21:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.677+0000 7f41c9ef2700 1 -- 172.21.15.104:0/1483702048 shutdown_connections 2024-06-09T22:21:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.677+0000 7f41c9ef2700 1 --2- 172.21.15.104:0/1483702048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41c406b5e0 0x7f41c410f320 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.677+0000 7f41c9ef2700 1 --2- 172.21.15.104:0/1483702048 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41c4108500 0x7f41c41088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.677+0000 7f41c9ef2700 1 --2- 172.21.15.104:0/1483702048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41c406ac20 0x7f41c406b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.677+0000 7f41c9ef2700 1 -- 172.21.15.104:0/1483702048 >> 172.21.15.104:0/1483702048 conn(0x7f41c4078340 msgr2=0x7f41c4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.677+0000 7f41c9ef2700 1 -- 172.21.15.104:0/1483702048 shutdown_connections 2024-06-09T22:21:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.677+0000 7f41c9ef2700 1 -- 172.21.15.104:0/1483702048 wait complete. 2024-06-09T22:21:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41c9ef2700 1 Processor -- start 2024-06-09T22:21:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41c9ef2700 1 -- start start 2024-06-09T22:21:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41c9ef2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41c406ac20 0x7f41c419e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41c9ef2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41c406b5e0 0x7f41c419eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41c9ef2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41c4108500 0x7f41c4198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41c9ef2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f41c41127e0 con 0x7f41c406b5e0 2024-06-09T22:21:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41c9ef2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4112660 con 0x7f41c406ac20 2024-06-09T22:21:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41c9ef2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4112960 con 0x7f41c4108500 2024-06-09T22:21:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41baffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41c406b5e0 0x7f41c419eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41baffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41c406b5e0 0x7f41c419eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37738/0 (socket says 172.21.15.104:37738) 2024-06-09T22:21:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41baffd700 1 -- 172.21.15.104:0/2668128281 learned_addr learned my addr 172.21.15.104:0/2668128281 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41c37fe700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41c406ac20 0x7f41c419e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.678+0000 7f41c3fff700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41c4108500 0x7f41c4198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.679+0000 7f41baffd700 1 -- 172.21.15.104:0/2668128281 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41c4108500 msgr2=0x7f41c4198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.679+0000 7f41baffd700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41c4108500 0x7f41c4198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.679+0000 7f41baffd700 1 -- 172.21.15.104:0/2668128281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41c406ac20 msgr2=0x7f41c419e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.679+0000 7f41baffd700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41c406ac20 0x7f41c419e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.679+0000 7f41baffd700 1 -- 172.21.15.104:0/2668128281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c4199180 con 0x7f41c406b5e0 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.679+0000 7f41c3fff700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41c4108500 0x7f41c4198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.679+0000 7f41baffd700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41c406b5e0 0x7f41c419eda0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00b530 tx=0x7f41ac00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.679+0000 7f41c17fa700 1 -- 172.21.15.104:0/2668128281 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac015020 con 0x7f41c406b5e0 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.679+0000 7f41c9ef2700 1 -- 172.21.15.104:0/2668128281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c4199470 con 0x7f41c406b5e0 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.679+0000 7f41c17fa700 1 -- 172.21.15.104:0/2668128281 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41ac011d10 con 0x7f41c406b5e0 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.679+0000 7f41c17fa700 1 -- 172.21.15.104:0/2668128281 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac010e50 con 0x7f41c406b5e0 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.679+0000 7f41c9ef2700 1 -- 172.21.15.104:0/2668128281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c41a5730 con 0x7f41c406b5e0 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.680+0000 7f41c9ef2700 1 -- 172.21.15.104:0/2668128281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41c4069180 con 0x7f41c406b5e0 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.681+0000 7f41c17fa700 1 -- 172.21.15.104:0/2668128281 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f41ac018020 con 0x7f41c406b5e0 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.682+0000 7f41c17fa700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f41b00a3050 0x7f41b00a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.682+0000 7f41c37fe700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f41b00a3050 0x7f41b00a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:09.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.682+0000 7f41c17fa700 1 -- 172.21.15.104:0/2668128281 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f41ac0cf650 con 0x7f41c406b5e0 2024-06-09T22:21:09.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.682+0000 7f41c37fe700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f41b00a3050 0x7f41b00a5510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f41b4005ea0 tx=0x7f41b4005e30 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:09.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.685+0000 7f41c17fa700 1 -- 172.21.15.104:0/2668128281 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f41ac01f020 con 0x7f41c406b5e0 2024-06-09T22:21:09.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:09 smithi153 ceph-mon[36599]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.1 KiB/s wr, 20 op/s 2024-06-09T22:21:09.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.881+0000 7f41c9ef2700 1 -- 172.21.15.104:0/2668128281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41c4199f50 con 0x7f41c406b5e0 2024-06-09T22:21:09.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.883+0000 7f41c17fa700 1 -- 172.21.15.104:0/2668128281 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+25144 (secure 0 0 0) 0x7f41ac09bdc0 con 0x7f41c406b5e0 2024-06-09T22:21:09.883 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:09.883 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":100,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:20:57.590817+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:0/2389242747":"2024-06-10T22:20:57.590784+0000","172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6801/1906048760":"2024-06-10T22:20:57.590784+0000","172.21.15.104:6800/1906048760":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/517954187":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/167857460":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/357628765":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:21:09.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.885+0000 7f41c9ef2700 1 -- 172.21.15.104:0/2668128281 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f41b00a3050 msgr2=0x7f41b00a5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:09.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.885+0000 7f41c9ef2700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f41b00a3050 0x7f41b00a5510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f41b4005ea0 tx=0x7f41b4005e30 comp rx=0 tx=0).stop 2024-06-09T22:21:09.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.885+0000 7f41c9ef2700 1 -- 172.21.15.104:0/2668128281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41c406b5e0 msgr2=0x7f41c419eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:09.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.885+0000 7f41c9ef2700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41c406b5e0 0x7f41c419eda0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00b530 tx=0x7f41ac00b8f0 comp rx=0 tx=0).stop 2024-06-09T22:21:09.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.885+0000 7f41c9ef2700 1 -- 172.21.15.104:0/2668128281 shutdown_connections 2024-06-09T22:21:09.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.885+0000 7f41c9ef2700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f41b00a3050 0x7f41b00a5510 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:09.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.885+0000 7f41c9ef2700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41c4108500 0x7f41c4198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:09.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.885+0000 7f41c9ef2700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41c406ac20 0x7f41c419e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:09.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.885+0000 7f41c9ef2700 1 --2- 172.21.15.104:0/2668128281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41c406b5e0 0x7f41c419eda0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:09.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.885+0000 7f41c9ef2700 1 -- 172.21.15.104:0/2668128281 >> 172.21.15.104:0/2668128281 conn(0x7f41c4078340 msgr2=0x7f41c40ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:09.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.885+0000 7f41c9ef2700 1 -- 172.21.15.104:0/2668128281 shutdown_connections 2024-06-09T22:21:09.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:09.885+0000 7f41c9ef2700 1 -- 172.21.15.104:0/2668128281 wait complete. 2024-06-09T22:21:10.505 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:10 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2668128281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:21:10.557 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:10 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2668128281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:21:10.692 DEBUG:tasks.fwd_scrub:Yielding 2024-06-09T22:21:10.692 INFO:teuthology.run_tasks:Running task exec... 2024-06-09T22:21:10.702 INFO:teuthology.task.exec:Executing custom commands... 2024-06-09T22:21:10.702 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi104.front.sepia.ceph.com 2024-06-09T22:21:10.702 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-06-09T22:21:10.702 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-06-09T22:21:10.703 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-09T22:21:10.703 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-09T22:21:10.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:10 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2668128281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:21:10.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.813+0000 7f4612f14640 1 -- 172.21.15.104:0/1679833911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f460c1079d0 msgr2=0x7f460c107db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:10.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.813+0000 7f4612f14640 1 --2- 172.21.15.104:0/1679833911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f460c1079d0 0x7f460c107db0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4600009930 tx=0x7f460002a420 comp rx=0 tx=0).stop 2024-06-09T22:21:10.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.813+0000 7f4612f14640 1 -- 172.21.15.104:0/1679833911 shutdown_connections 2024-06-09T22:21:10.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.813+0000 7f4612f14640 1 --2- 172.21.15.104:0/1679833911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f460c102380 0x7f460c10e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:10.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.813+0000 7f4612f14640 1 --2- 172.21.15.104:0/1679833911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f460c1019c0 0x7f460c101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:10.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.813+0000 7f4612f14640 1 --2- 172.21.15.104:0/1679833911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f460c1079d0 0x7f460c107db0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:10.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.813+0000 7f4612f14640 1 -- 172.21.15.104:0/1679833911 >> 172.21.15.104:0/1679833911 conn(0x7f460c0fd450 msgr2=0x7f460c0ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:10.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.813+0000 7f4612f14640 1 -- 172.21.15.104:0/1679833911 shutdown_connections 2024-06-09T22:21:10.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.813+0000 7f4612f14640 1 -- 172.21.15.104:0/1679833911 wait complete. 2024-06-09T22:21:10.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f4612f14640 1 Processor -- start 2024-06-09T22:21:10.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f4612f14640 1 -- start start 2024-06-09T22:21:10.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f4612f14640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f460c1019c0 0x7f460c073ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:10.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f4612f14640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f460c102380 0x7f460c0741e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:10.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f4612f14640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f460c1079d0 0x7f460c074720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:10.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f4612f14640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f460c111e00 con 0x7f460c1079d0 2024-06-09T22:21:10.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f4612f14640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f460c111c80 con 0x7f460c102380 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f4612f14640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f460c111f80 con 0x7f460c1019c0 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f46114b1640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f460c1079d0 0x7f460c074720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f46114b1640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f460c1079d0 0x7f460c074720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37760/0 (socket says 172.21.15.104:37760) 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f46114b1640 1 -- 172.21.15.104:0/4293344982 learned_addr learned my addr 172.21.15.104:0/4293344982 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f4610cb0640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f460c1019c0 0x7f460c073ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f460bfff640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f460c102380 0x7f460c0741e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f46114b1640 1 -- 172.21.15.104:0/4293344982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f460c1019c0 msgr2=0x7f460c073ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f46114b1640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f460c1019c0 0x7f460c073ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f46114b1640 1 -- 172.21.15.104:0/4293344982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f460c102380 msgr2=0x7f460c0741e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f46114b1640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f460c102380 0x7f460c0741e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f46114b1640 1 -- 172.21.15.104:0/4293344982 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f460c1a66c0 con 0x7f460c1079d0 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.814+0000 7f46114b1640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f460c1079d0 0x7f460c074720 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f460400cce0 tx=0x7f460400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.815+0000 7f4609ffb640 1 -- 172.21.15.104:0/4293344982 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460400f450 con 0x7f460c1079d0 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.815+0000 7f4609ffb640 1 -- 172.21.15.104:0/4293344982 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f460400fad0 con 0x7f460c1079d0 2024-06-09T22:21:10.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.815+0000 7f4612f14640 1 -- 172.21.15.104:0/4293344982 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f460c1a69b0 con 0x7f460c1079d0 2024-06-09T22:21:10.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.815+0000 7f4609ffb640 1 -- 172.21.15.104:0/4293344982 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4604018c10 con 0x7f460c1079d0 2024-06-09T22:21:10.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.815+0000 7f4612f14640 1 -- 172.21.15.104:0/4293344982 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f460c1a6f00 con 0x7f460c1079d0 2024-06-09T22:21:10.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.816+0000 7f4612f14640 1 -- 172.21.15.104:0/4293344982 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f460c04f760 con 0x7f460c1079d0 2024-06-09T22:21:10.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.817+0000 7f4609ffb640 1 -- 172.21.15.104:0/4293344982 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f460400f5f0 con 0x7f460c1079d0 2024-06-09T22:21:10.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.817+0000 7f4609ffb640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45fc0a2f40 0x7f45fc0a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:10.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.818+0000 7f4609ffb640 1 -- 172.21.15.104:0/4293344982 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f46040d0010 con 0x7f460c1079d0 2024-06-09T22:21:10.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.818+0000 7f4610cb0640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45fc0a2f40 0x7f45fc0a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:10.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.818+0000 7f4610cb0640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45fc0a2f40 0x7f45fc0a5400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4600006010 tx=0x7f4600000bc0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:10.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:10.821+0000 7f4609ffb640 1 -- 172.21.15.104:0/4293344982 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4604014030 con 0x7f460c1079d0 2024-06-09T22:21:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:11.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.013+0000 7f4612f14640 1 -- 172.21.15.104:0/4293344982 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f460c0751d0 con 0x7f460c1079d0 2024-06-09T22:21:11.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.017+0000 7f4609ffb640 1 -- 172.21.15.104:0/4293344982 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v31)=0 v31) ==== 129+0+0 (secure 0 0 0) 0x7f460c0751d0 con 0x7f460c1079d0 2024-06-09T22:21:11.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.019+0000 7f4612f14640 1 -- 172.21.15.104:0/4293344982 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45fc0a2f40 msgr2=0x7f45fc0a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.019+0000 7f4612f14640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45fc0a2f40 0x7f45fc0a5400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4600006010 tx=0x7f4600000bc0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.019+0000 7f4612f14640 1 -- 172.21.15.104:0/4293344982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f460c1079d0 msgr2=0x7f460c074720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.019+0000 7f4612f14640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f460c1079d0 0x7f460c074720 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f460400cce0 tx=0x7f460400a620 comp rx=0 tx=0).stop 2024-06-09T22:21:11.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.019+0000 7f4612f14640 1 -- 172.21.15.104:0/4293344982 shutdown_connections 2024-06-09T22:21:11.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.019+0000 7f4612f14640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45fc0a2f40 0x7f45fc0a5400 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.019+0000 7f4612f14640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f460c1079d0 0x7f460c074720 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.019+0000 7f4612f14640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f460c102380 0x7f460c0741e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.019+0000 7f4612f14640 1 --2- 172.21.15.104:0/4293344982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f460c1019c0 0x7f460c073ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.019+0000 7f4612f14640 1 -- 172.21.15.104:0/4293344982 >> 172.21.15.104:0/4293344982 conn(0x7f460c0fd450 msgr2=0x7f460c0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:11.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.019+0000 7f4612f14640 1 -- 172.21.15.104:0/4293344982 shutdown_connections 2024-06-09T22:21:11.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.019+0000 7f4612f14640 1 -- 172.21.15.104:0/4293344982 wait complete. 2024-06-09T22:21:11.029 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-06-09T22:21:11.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.138+0000 7f4e7e631640 1 -- 172.21.15.104:0/2816542476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e781008d0 msgr2=0x7f4e78100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.138+0000 7f4e7e631640 1 --2- 172.21.15.104:0/2816542476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e781008d0 0x7f4e78100d50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009930 tx=0x7f4e6c02a420 comp rx=0 tx=0).stop 2024-06-09T22:21:11.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.138+0000 7f4e7e631640 1 -- 172.21.15.104:0/2816542476 shutdown_connections 2024-06-09T22:21:11.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.138+0000 7f4e7e631640 1 --2- 172.21.15.104:0/2816542476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e78101290 0x7f4e7810d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.138+0000 7f4e7e631640 1 --2- 172.21.15.104:0/2816542476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e781008d0 0x7f4e78100d50 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.138+0000 7f4e7e631640 1 --2- 172.21.15.104:0/2816542476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e78106890 0x7f4e78106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.138+0000 7f4e7e631640 1 -- 172.21.15.104:0/2816542476 >> 172.21.15.104:0/2816542476 conn(0x7f4e78077a20 msgr2=0x7f4e78077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:11.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.138+0000 7f4e7e631640 1 -- 172.21.15.104:0/2816542476 shutdown_connections 2024-06-09T22:21:11.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.138+0000 7f4e7e631640 1 -- 172.21.15.104:0/2816542476 wait complete. 2024-06-09T22:21:11.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7e631640 1 Processor -- start 2024-06-09T22:21:11.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7e631640 1 -- start start 2024-06-09T22:21:11.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7e631640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e781008d0 0x7f4e781981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:11.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7e631640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e78101290 0x7f4e78198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:11.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7e631640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e78106890 0x7f4e7819cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:11.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7e631640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4e78110ca0 con 0x7f4e78106890 2024-06-09T22:21:11.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7e631640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e78110b20 con 0x7f4e78101290 2024-06-09T22:21:11.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7e631640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e78110e20 con 0x7f4e781008d0 2024-06-09T22:21:11.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7cbce640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e78106890 0x7f4e7819cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:11.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e77fff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e781008d0 0x7f4e781981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:11.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7cbce640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e78106890 0x7f4e7819cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37772/0 (socket says 172.21.15.104:37772) 2024-06-09T22:21:11.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7cbce640 1 -- 172.21.15.104:0/2149722122 learned_addr learned my addr 172.21.15.104:0/2149722122 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:11.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e777fe640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e78101290 0x7f4e78198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:11.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7cbce640 1 -- 172.21.15.104:0/2149722122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e781008d0 msgr2=0x7f4e781981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7cbce640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e781008d0 0x7f4e781981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7cbce640 1 -- 172.21.15.104:0/2149722122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e78101290 msgr2=0x7f4e78198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7cbce640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e78101290 0x7f4e78198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e7cbce640 1 -- 172.21.15.104:0/2149722122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7819d1e0 con 0x7f4e78106890 2024-06-09T22:21:11.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.139+0000 7f4e77fff640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e781008d0 0x7f4e781981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:21:11.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.140+0000 7f4e7cbce640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e78106890 0x7f4e7819cab0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000cce0 tx=0x7f4e7000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:11.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.140+0000 7f4e757fa640 1 -- 172.21.15.104:0/2149722122 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000f450 con 0x7f4e78106890 2024-06-09T22:21:11.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.140+0000 7f4e757fa640 1 -- 172.21.15.104:0/2149722122 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e7000fad0 con 0x7f4e78106890 2024-06-09T22:21:11.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.140+0000 7f4e7e631640 1 -- 172.21.15.104:0/2149722122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e7819d4d0 con 0x7f4e78106890 2024-06-09T22:21:11.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.140+0000 7f4e757fa640 1 -- 172.21.15.104:0/2149722122 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70018c10 con 0x7f4e78106890 2024-06-09T22:21:11.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.140+0000 7f4e7e631640 1 -- 172.21.15.104:0/2149722122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e781a4d60 con 0x7f4e78106890 2024-06-09T22:21:11.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.141+0000 7f4e7e631640 1 -- 172.21.15.104:0/2149722122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e7804f760 con 0x7f4e78106890 2024-06-09T22:21:11.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.142+0000 7f4e757fa640 1 -- 172.21.15.104:0/2149722122 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f4e7000f5f0 con 0x7f4e78106890 2024-06-09T22:21:11.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.143+0000 7f4e757fa640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4e600a2ec0 0x7f4e600a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:11.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.143+0000 7f4e757fa640 1 -- 172.21.15.104:0/2149722122 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f4e700d0080 con 0x7f4e78106890 2024-06-09T22:21:11.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.143+0000 7f4e77fff640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4e600a2ec0 0x7f4e600a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:11.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.143+0000 7f4e77fff640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4e600a2ec0 0x7f4e600a5380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009f40 tx=0x7f4e68005f90 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:11.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.148+0000 7f4e757fa640 1 -- 172.21.15.104:0/2149722122 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4e70014030 con 0x7f4e78106890 2024-06-09T22:21:11.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.333+0000 7f4e7e631640 1 -- 172.21.15.104:0/2149722122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f4e7802d2e0 con 0x7f4e78106890 2024-06-09T22:21:11.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.339+0000 7f4e757fa640 1 -- 172.21.15.104:0/2149722122 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v32)=0 v32) ==== 138+0+0 (secure 0 0 0) 0x7f4e7009c6e0 con 0x7f4e78106890 2024-06-09T22:21:11.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.342+0000 7f4e7e631640 1 -- 172.21.15.104:0/2149722122 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4e600a2ec0 msgr2=0x7f4e600a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.342+0000 7f4e7e631640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4e600a2ec0 0x7f4e600a5380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009f40 tx=0x7f4e68005f90 comp rx=0 tx=0).stop 2024-06-09T22:21:11.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.342+0000 7f4e7e631640 1 -- 172.21.15.104:0/2149722122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e78106890 msgr2=0x7f4e7819cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.342+0000 7f4e7e631640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e78106890 0x7f4e7819cab0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000cce0 tx=0x7f4e7000a620 comp rx=0 tx=0).stop 2024-06-09T22:21:11.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.342+0000 7f4e7e631640 1 -- 172.21.15.104:0/2149722122 shutdown_connections 2024-06-09T22:21:11.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.342+0000 7f4e7e631640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4e600a2ec0 0x7f4e600a5380 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.342+0000 7f4e7e631640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e78106890 0x7f4e7819cab0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.342+0000 7f4e7e631640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e78101290 0x7f4e78198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.342+0000 7f4e7e631640 1 --2- 172.21.15.104:0/2149722122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e781008d0 0x7f4e781981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.342+0000 7f4e7e631640 1 -- 172.21.15.104:0/2149722122 >> 172.21.15.104:0/2149722122 conn(0x7f4e78077a20 msgr2=0x7f4e78069290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:11.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.342+0000 7f4e7e631640 1 -- 172.21.15.104:0/2149722122 shutdown_connections 2024-06-09T22:21:11.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.342+0000 7f4e7e631640 1 -- 172.21.15.104:0/2149722122 wait complete. 2024-06-09T22:21:11.351 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:11 smithi104 ceph-mon[32389]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 931 B/s wr, 21 op/s 2024-06-09T22:21:11.351 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:11 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/4293344982' entity='client.admin' 2024-06-09T22:21:11.351 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:11 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:21:11.351 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:11 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:11.351 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:11 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:11.352 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:11 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:11.352 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-06-09T22:21:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.463+0000 7f09448ba640 1 -- 172.21.15.104:0/3642560456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09401079b0 msgr2=0x7f0940107d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.463+0000 7f09448ba640 1 --2- 172.21.15.104:0/3642560456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09401079b0 0x7f0940107d90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0930009930 tx=0x7f093002a420 comp rx=0 tx=0).stop 2024-06-09T22:21:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.463+0000 7f09448ba640 1 -- 172.21.15.104:0/3642560456 shutdown_connections 2024-06-09T22:21:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.463+0000 7f09448ba640 1 --2- 172.21.15.104:0/3642560456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0940102360 0x7f094010e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.463+0000 7f09448ba640 1 --2- 172.21.15.104:0/3642560456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09401019a0 0x7f0940101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.463+0000 7f09448ba640 1 --2- 172.21.15.104:0/3642560456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09401079b0 0x7f0940107d90 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.463+0000 7f09448ba640 1 -- 172.21.15.104:0/3642560456 >> 172.21.15.104:0/3642560456 conn(0x7f09400fd490 msgr2=0x7f09400ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.463+0000 7f09448ba640 1 -- 172.21.15.104:0/3642560456 shutdown_connections 2024-06-09T22:21:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.463+0000 7f09448ba640 1 -- 172.21.15.104:0/3642560456 wait complete. 2024-06-09T22:21:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f09448ba640 1 Processor -- start 2024-06-09T22:21:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f09448ba640 1 -- start start 2024-06-09T22:21:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f09448ba640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09401019a0 0x7f0940198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f09448ba640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0940102360 0x7f09401987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f09448ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09401079b0 0x7f094019caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f09448ba640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0940111dc0 con 0x7f09401019a0 2024-06-09T22:21:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f09448ba640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0940111c40 con 0x7f09401079b0 2024-06-09T22:21:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f09448ba640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0940111f40 con 0x7f0940102360 2024-06-09T22:21:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f093e59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09401019a0 0x7f0940198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f093e59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09401019a0 0x7f0940198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37784/0 (socket says 172.21.15.104:37784) 2024-06-09T22:21:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f093e59c640 1 -- 172.21.15.104:0/1117895636 learned_addr learned my addr 172.21.15.104:0/1117895636 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f093dd9b640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0940102360 0x7f09401987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.464+0000 7f093ed9d640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09401079b0 0x7f094019caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.465+0000 7f093e59c640 1 -- 172.21.15.104:0/1117895636 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0940102360 msgr2=0x7f09401987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.465+0000 7f093e59c640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0940102360 0x7f09401987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.465+0000 7f093e59c640 1 -- 172.21.15.104:0/1117895636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09401079b0 msgr2=0x7f094019caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.465+0000 7f093e59c640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09401079b0 0x7f094019caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.465+0000 7f093e59c640 1 -- 172.21.15.104:0/1117895636 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094019d220 con 0x7f09401019a0 2024-06-09T22:21:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.465+0000 7f093dd9b640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0940102360 0x7f09401987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.465+0000 7f093e59c640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09401019a0 0x7f0940198280 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0930009230 tx=0x7f0930004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:11.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.465+0000 7f092f7fe640 1 -- 172.21.15.104:0/1117895636 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093003f030 con 0x7f09401019a0 2024-06-09T22:21:11.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.465+0000 7f092f7fe640 1 -- 172.21.15.104:0/1117895636 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0930003ad0 con 0x7f09401019a0 2024-06-09T22:21:11.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.465+0000 7f09448ba640 1 -- 172.21.15.104:0/1117895636 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f094019d4b0 con 0x7f09401019a0 2024-06-09T22:21:11.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.465+0000 7f092f7fe640 1 -- 172.21.15.104:0/1117895636 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930035070 con 0x7f09401019a0 2024-06-09T22:21:11.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.465+0000 7f09448ba640 1 -- 172.21.15.104:0/1117895636 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09401a4da0 con 0x7f09401019a0 2024-06-09T22:21:11.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.466+0000 7f09448ba640 1 -- 172.21.15.104:0/1117895636 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f094004f760 con 0x7f09401019a0 2024-06-09T22:21:11.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.467+0000 7f092f7fe640 1 -- 172.21.15.104:0/1117895636 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f0930042030 con 0x7f09401019a0 2024-06-09T22:21:11.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.467+0000 7f092f7fe640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f09280a2ec0 0x7f09280a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:11.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.468+0000 7f092f7fe640 1 -- 172.21.15.104:0/1117895636 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f09300f0090 con 0x7f09401019a0 2024-06-09T22:21:11.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.468+0000 7f093dd9b640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f09280a2ec0 0x7f09280a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:11.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.468+0000 7f093dd9b640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f09280a2ec0 0x7f09280a5380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0940106fa0 tx=0x7f0934005f90 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:11.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.472+0000 7f092f7fe640 1 -- 172.21.15.104:0/1117895636 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f09300bc7c0 con 0x7f09401019a0 2024-06-09T22:21:11.557 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:11 smithi194 ceph-mon[36856]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 931 B/s wr, 21 op/s 2024-06-09T22:21:11.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:11 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/4293344982' entity='client.admin' 2024-06-09T22:21:11.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:11 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:21:11.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:11 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:11.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:11 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:11.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:11 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:11.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.653+0000 7f09448ba640 1 -- 172.21.15.104:0/1117895636 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f0940109f10 con 0x7f09401019a0 2024-06-09T22:21:11.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.658+0000 7f092f7fe640 1 -- 172.21.15.104:0/1117895636 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v33)=0 v33) ==== 133+0+0 (secure 0 0 0) 0x7f09300bc0c0 con 0x7f09401019a0 2024-06-09T22:21:11.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.661+0000 7f09448ba640 1 -- 172.21.15.104:0/1117895636 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f09280a2ec0 msgr2=0x7f09280a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.661+0000 7f09448ba640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f09280a2ec0 0x7f09280a5380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0940106fa0 tx=0x7f0934005f90 comp rx=0 tx=0).stop 2024-06-09T22:21:11.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.661+0000 7f09448ba640 1 -- 172.21.15.104:0/1117895636 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09401019a0 msgr2=0x7f0940198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.661+0000 7f09448ba640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09401019a0 0x7f0940198280 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0930009230 tx=0x7f0930004270 comp rx=0 tx=0).stop 2024-06-09T22:21:11.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.661+0000 7f09448ba640 1 -- 172.21.15.104:0/1117895636 shutdown_connections 2024-06-09T22:21:11.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.661+0000 7f09448ba640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f09280a2ec0 0x7f09280a5380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.661+0000 7f09448ba640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09401079b0 0x7f094019caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.661+0000 7f09448ba640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0940102360 0x7f09401987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.661+0000 7f09448ba640 1 --2- 172.21.15.104:0/1117895636 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09401019a0 0x7f0940198280 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.661+0000 7f09448ba640 1 -- 172.21.15.104:0/1117895636 >> 172.21.15.104:0/1117895636 conn(0x7f09400fd490 msgr2=0x7f09400fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:11.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.661+0000 7f09448ba640 1 -- 172.21.15.104:0/1117895636 shutdown_connections 2024-06-09T22:21:11.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.661+0000 7f09448ba640 1 -- 172.21.15.104:0/1117895636 wait complete. 2024-06-09T22:21:11.672 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-06-09T22:21:11.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:11 smithi153 ceph-mon[36599]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 931 B/s wr, 21 op/s 2024-06-09T22:21:11.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:11 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/4293344982' entity='client.admin' 2024-06-09T22:21:11.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:11 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:21:11.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:11 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:11.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:11 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:11.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:11 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 -- 172.21.15.104:0/2517700581 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5430101290 msgr2=0x7f543010d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 --2- 172.21.15.104:0/2517700581 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5430101290 0x7f543010d750 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5428009cd0 tx=0x7f542802a8e0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 -- 172.21.15.104:0/2517700581 shutdown_connections 2024-06-09T22:21:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 --2- 172.21.15.104:0/2517700581 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5430101290 0x7f543010d750 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 --2- 172.21.15.104:0/2517700581 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54301008d0 0x7f5430100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 --2- 172.21.15.104:0/2517700581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5430106890 0x7f5430106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 -- 172.21.15.104:0/2517700581 >> 172.21.15.104:0/2517700581 conn(0x7f5430077a20 msgr2=0x7f5430077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 -- 172.21.15.104:0/2517700581 shutdown_connections 2024-06-09T22:21:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 -- 172.21.15.104:0/2517700581 wait complete. 2024-06-09T22:21:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 Processor -- start 2024-06-09T22:21:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 -- start start 2024-06-09T22:21:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54301008d0 0x7f54301981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5430101290 0x7f5430198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5430106890 0x7f543019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5430111da0 con 0x7f54301008d0 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5430111c20 con 0x7f5430101290 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.779+0000 7f5436eb3640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5430111f20 con 0x7f5430106890 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f5434c4f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54301008d0 0x7f54301981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f5434c4f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54301008d0 0x7f54301981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37806/0 (socket says 172.21.15.104:37806) 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f5434c4f640 1 -- 172.21.15.104:0/2450438040 learned_addr learned my addr 172.21.15.104:0/2450438040 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f542ffff640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5430101290 0x7f5430198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f5435450640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5430106890 0x7f543019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f5434c4f640 1 -- 172.21.15.104:0/2450438040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5430106890 msgr2=0x7f543019ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f5434c4f640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5430106890 0x7f543019ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f5434c4f640 1 -- 172.21.15.104:0/2450438040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5430101290 msgr2=0x7f5430198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f5434c4f640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5430101290 0x7f5430198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:11.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f5434c4f640 1 -- 172.21.15.104:0/2450438040 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543019d160 con 0x7f54301008d0 2024-06-09T22:21:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f5434c4f640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54301008d0 0x7f54301981c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f542400d950 tx=0x7f542400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f542dffb640 1 -- 172.21.15.104:0/2450438040 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424017030 con 0x7f54301008d0 2024-06-09T22:21:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f542dffb640 1 -- 172.21.15.104:0/2450438040 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5424004020 con 0x7f54301008d0 2024-06-09T22:21:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f5436eb3640 1 -- 172.21.15.104:0/2450438040 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f543019d450 con 0x7f54301008d0 2024-06-09T22:21:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f542dffb640 1 -- 172.21.15.104:0/2450438040 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54240047a0 con 0x7f54301008d0 2024-06-09T22:21:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.780+0000 7f5436eb3640 1 -- 172.21.15.104:0/2450438040 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54301a4ce0 con 0x7f54301008d0 2024-06-09T22:21:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.781+0000 7f5436eb3640 1 -- 172.21.15.104:0/2450438040 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f543004f760 con 0x7f54301008d0 2024-06-09T22:21:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.782+0000 7f542dffb640 1 -- 172.21.15.104:0/2450438040 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f54240041d0 con 0x7f54301008d0 2024-06-09T22:21:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.783+0000 7f542dffb640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f54200a2ec0 0x7f54200a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.783+0000 7f542ffff640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f54200a2ec0 0x7f54200a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.783+0000 7f542dffb640 1 -- 172.21.15.104:0/2450438040 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f54240cf3c0 con 0x7f54301008d0 2024-06-09T22:21:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.784+0000 7f542ffff640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f54200a2ec0 0x7f54200a5380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5430077f40 tx=0x7f541c00b390 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:11.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.787+0000 7f542dffb640 1 -- 172.21.15.104:0/2450438040 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f542409ba20 con 0x7f54301008d0 2024-06-09T22:21:11.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:11.990+0000 7f5436eb3640 1 -- 172.21.15.104:0/2450438040 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f5430103a60 con 0x7f54200a2ec0 2024-06-09T22:21:12.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.057+0000 7f542dffb640 1 -- 172.21.15.104:0/2450438040 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5430103a60 con 0x7f54200a2ec0 2024-06-09T22:21:12.056 INFO:teuthology.orchestra.run.smithi104.stdout:Schedule set for path / 2024-06-09T22:21:12.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.059+0000 7f5436eb3640 1 -- 172.21.15.104:0/2450438040 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f54200a2ec0 msgr2=0x7f54200a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.059+0000 7f5436eb3640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f54200a2ec0 0x7f54200a5380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5430077f40 tx=0x7f541c00b390 comp rx=0 tx=0).stop 2024-06-09T22:21:12.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.059+0000 7f5436eb3640 1 -- 172.21.15.104:0/2450438040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54301008d0 msgr2=0x7f54301981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.059+0000 7f5436eb3640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54301008d0 0x7f54301981c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f542400d950 tx=0x7f542400de10 comp rx=0 tx=0).stop 2024-06-09T22:21:12.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.059+0000 7f5436eb3640 1 -- 172.21.15.104:0/2450438040 shutdown_connections 2024-06-09T22:21:12.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.059+0000 7f5436eb3640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f54200a2ec0 0x7f54200a5380 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.059+0000 7f5436eb3640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5430106890 0x7f543019ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.059+0000 7f5436eb3640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5430101290 0x7f5430198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.059+0000 7f5436eb3640 1 --2- 172.21.15.104:0/2450438040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54301008d0 0x7f54301981c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.059+0000 7f5436eb3640 1 -- 172.21.15.104:0/2450438040 >> 172.21.15.104:0/2450438040 conn(0x7f5430077a20 msgr2=0x7f5430103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:12.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.059+0000 7f5436eb3640 1 -- 172.21.15.104:0/2450438040 shutdown_connections 2024-06-09T22:21:12.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.059+0000 7f5436eb3640 1 -- 172.21.15.104:0/2450438040 wait complete. 2024-06-09T22:21:12.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-06-09T22:21:12.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.175+0000 7ff8e2d12640 1 -- 172.21.15.104:0/1998924238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8dc1008d0 msgr2=0x7ff8dc100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.175+0000 7ff8e2d12640 1 --2- 172.21.15.104:0/1998924238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8dc1008d0 0x7ff8dc100d50 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8009930 tx=0x7ff8c802a420 comp rx=0 tx=0).stop 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.175+0000 7ff8e2d12640 1 -- 172.21.15.104:0/1998924238 shutdown_connections 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.175+0000 7ff8e2d12640 1 --2- 172.21.15.104:0/1998924238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc101290 0x7ff8dc10d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.175+0000 7ff8e2d12640 1 --2- 172.21.15.104:0/1998924238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8dc1008d0 0x7ff8dc100d50 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.175+0000 7ff8e2d12640 1 --2- 172.21.15.104:0/1998924238 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8dc106890 0x7ff8dc106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.175+0000 7ff8e2d12640 1 -- 172.21.15.104:0/1998924238 >> 172.21.15.104:0/1998924238 conn(0x7ff8dc077a20 msgr2=0x7ff8dc077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.175+0000 7ff8e2d12640 1 -- 172.21.15.104:0/1998924238 shutdown_connections 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.175+0000 7ff8e2d12640 1 -- 172.21.15.104:0/1998924238 wait complete. 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e2d12640 1 Processor -- start 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e2d12640 1 -- start start 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e2d12640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8dc1008d0 0x7ff8dc1971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e2d12640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8dc101290 0x7ff8dc197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e2d12640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc106890 0x7ff8dc19ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e2d12640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc110ca0 con 0x7ff8dc1008d0 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e2d12640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc110b20 con 0x7ff8dc106890 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e2d12640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc110e20 con 0x7ff8dc101290 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e0aae640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8dc1008d0 0x7ff8dc1971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e0aae640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8dc1008d0 0x7ff8dc1971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37824/0 (socket says 172.21.15.104:37824) 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e0aae640 1 -- 172.21.15.104:0/781296102 learned_addr learned my addr 172.21.15.104:0/781296102 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e12af640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc106890 0x7ff8dc19ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8dbfff640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8dc101290 0x7ff8dc197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e0aae640 1 -- 172.21.15.104:0/781296102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8dc101290 msgr2=0x7ff8dc197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e0aae640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8dc101290 0x7ff8dc197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e0aae640 1 -- 172.21.15.104:0/781296102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc106890 msgr2=0x7ff8dc19ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e0aae640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc106890 0x7ff8dc19ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e0aae640 1 -- 172.21.15.104:0/781296102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc19c1b0 con 0x7ff8dc1008d0 2024-06-09T22:21:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8dbfff640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8dc101290 0x7ff8dc197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.176+0000 7ff8e12af640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc106890 0x7ff8dc19ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.177+0000 7ff8e0aae640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8dc1008d0 0x7ff8dc1971c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000d770 tx=0x7ff8d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.177+0000 7ff8d9ffb640 1 -- 172.21.15.104:0/781296102 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0017070 con 0x7ff8dc1008d0 2024-06-09T22:21:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.177+0000 7ff8e2d12640 1 -- 172.21.15.104:0/781296102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8dc19c4a0 con 0x7ff8dc1008d0 2024-06-09T22:21:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.177+0000 7ff8d9ffb640 1 -- 172.21.15.104:0/781296102 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8d0005500 con 0x7ff8dc1008d0 2024-06-09T22:21:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.177+0000 7ff8d9ffb640 1 -- 172.21.15.104:0/781296102 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0004e00 con 0x7ff8dc1008d0 2024-06-09T22:21:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.177+0000 7ff8e2d12640 1 -- 172.21.15.104:0/781296102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8dc1a3d30 con 0x7ff8dc1008d0 2024-06-09T22:21:12.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.178+0000 7ff8e2d12640 1 -- 172.21.15.104:0/781296102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8dc04f760 con 0x7ff8dc1008d0 2024-06-09T22:21:12.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.179+0000 7ff8d9ffb640 1 -- 172.21.15.104:0/781296102 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7ff8d0004fa0 con 0x7ff8dc1008d0 2024-06-09T22:21:12.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.180+0000 7ff8d9ffb640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff8cc0a2ec0 0x7ff8cc0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.180+0000 7ff8dbfff640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff8cc0a2ec0 0x7ff8cc0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.180+0000 7ff8d9ffb640 1 -- 172.21.15.104:0/781296102 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff8d00cf450 con 0x7ff8dc1008d0 2024-06-09T22:21:12.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.180+0000 7ff8dbfff640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff8cc0a2ec0 0x7ff8cc0a5380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8000c00 tx=0x7ff8c8032c70 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:12.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.189+0000 7ff8d9ffb640 1 -- 172.21.15.104:0/781296102 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff8d009bb50 con 0x7ff8dc1008d0 2024-06-09T22:21:12.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.398+0000 7ff8e2d12640 1 -- 172.21.15.104:0/781296102 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7ff8dc108e50 con 0x7ff8cc0a2ec0 2024-06-09T22:21:12.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.458+0000 7ff8d9ffb640 1 -- 172.21.15.104:0/781296102 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7ff8dc108e50 con 0x7ff8cc0a2ec0 2024-06-09T22:21:12.457 INFO:teuthology.orchestra.run.smithi104.stdout:Retention added to path / 2024-06-09T22:21:12.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.460+0000 7ff8e2d12640 1 -- 172.21.15.104:0/781296102 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff8cc0a2ec0 msgr2=0x7ff8cc0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.460+0000 7ff8e2d12640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff8cc0a2ec0 0x7ff8cc0a5380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8000c00 tx=0x7ff8c8032c70 comp rx=0 tx=0).stop 2024-06-09T22:21:12.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.460+0000 7ff8e2d12640 1 -- 172.21.15.104:0/781296102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8dc1008d0 msgr2=0x7ff8dc1971c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.460+0000 7ff8e2d12640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8dc1008d0 0x7ff8dc1971c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000d770 tx=0x7ff8d000dc30 comp rx=0 tx=0).stop 2024-06-09T22:21:12.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.460+0000 7ff8e2d12640 1 -- 172.21.15.104:0/781296102 shutdown_connections 2024-06-09T22:21:12.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.460+0000 7ff8e2d12640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff8cc0a2ec0 0x7ff8cc0a5380 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.460+0000 7ff8e2d12640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc106890 0x7ff8dc19ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.460+0000 7ff8e2d12640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8dc101290 0x7ff8dc197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.460+0000 7ff8e2d12640 1 --2- 172.21.15.104:0/781296102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8dc1008d0 0x7ff8dc1971c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.460+0000 7ff8e2d12640 1 -- 172.21.15.104:0/781296102 >> 172.21.15.104:0/781296102 conn(0x7ff8dc077a20 msgr2=0x7ff8dc103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:12.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.460+0000 7ff8e2d12640 1 -- 172.21.15.104:0/781296102 shutdown_connections 2024-06-09T22:21:12.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.460+0000 7ff8e2d12640 1 -- 172.21.15.104:0/781296102 wait complete. 2024-06-09T22:21:12.470 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-06-09T22:21:12.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.577+0000 7f4ea115a640 1 -- 172.21.15.104:0/2454664925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c106890 msgr2=0x7f4e9c106c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.578+0000 7f4ea115a640 1 --2- 172.21.15.104:0/2454664925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c106890 0x7f4e9c106c70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c009930 tx=0x7f4e8c02a420 comp rx=0 tx=0).stop 2024-06-09T22:21:12.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.578+0000 7f4ea115a640 1 -- 172.21.15.104:0/2454664925 shutdown_connections 2024-06-09T22:21:12.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.578+0000 7f4ea115a640 1 --2- 172.21.15.104:0/2454664925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e9c101290 0x7f4e9c10d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.578+0000 7f4ea115a640 1 --2- 172.21.15.104:0/2454664925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c1008d0 0x7f4e9c100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.578+0000 7f4ea115a640 1 --2- 172.21.15.104:0/2454664925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c106890 0x7f4e9c106c70 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.578+0000 7f4ea115a640 1 -- 172.21.15.104:0/2454664925 >> 172.21.15.104:0/2454664925 conn(0x7f4e9c077a20 msgr2=0x7f4e9c077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:12.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.578+0000 7f4ea115a640 1 -- 172.21.15.104:0/2454664925 shutdown_connections 2024-06-09T22:21:12.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.578+0000 7f4ea115a640 1 -- 172.21.15.104:0/2454664925 wait complete. 2024-06-09T22:21:12.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.578+0000 7f4ea115a640 1 Processor -- start 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4ea115a640 1 -- start start 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4ea115a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c1008d0 0x7f4e9c1970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4ea115a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e9c101290 0x7f4e9c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4ea115a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c106890 0x7f4e9c19b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4ea115a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c110ca0 con 0x7f4e9c106890 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4ea115a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c110b20 con 0x7f4e9c1008d0 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4ea115a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c110e20 con 0x7f4e9c101290 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e9b59e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c106890 0x7f4e9c19b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e9b59e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c106890 0x7f4e9c19b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37836/0 (socket says 172.21.15.104:37836) 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e9b59e640 1 -- 172.21.15.104:0/4202620268 learned_addr learned my addr 172.21.15.104:0/4202620268 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e9ad9d640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c1008d0 0x7f4e9c1970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e9259c640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e9c101290 0x7f4e9c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e9b59e640 1 -- 172.21.15.104:0/4202620268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e9c101290 msgr2=0x7f4e9c1975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e9b59e640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e9c101290 0x7f4e9c1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e9b59e640 1 -- 172.21.15.104:0/4202620268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c1008d0 msgr2=0x7f4e9c1970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e9b59e640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c1008d0 0x7f4e9c1970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e9b59e640 1 -- 172.21.15.104:0/4202620268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9c19c040 con 0x7f4e9c106890 2024-06-09T22:21:12.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e9b59e640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c106890 0x7f4e9c19b910 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400eb10 tx=0x7f4e9400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:12.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e98d99640 1 -- 172.21.15.104:0/4202620268 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e94004b20 con 0x7f4e9c106890 2024-06-09T22:21:12.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e98d99640 1 -- 172.21.15.104:0/4202620268 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e94004cc0 con 0x7f4e9c106890 2024-06-09T22:21:12.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4ea115a640 1 -- 172.21.15.104:0/4202620268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e9c19c330 con 0x7f4e9c106890 2024-06-09T22:21:12.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.579+0000 7f4e98d99640 1 -- 172.21.15.104:0/4202620268 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e94010760 con 0x7f4e9c106890 2024-06-09T22:21:12.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.580+0000 7f4ea115a640 1 -- 172.21.15.104:0/4202620268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e9c1a3bc0 con 0x7f4e9c106890 2024-06-09T22:21:12.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.580+0000 7f4ea115a640 1 -- 172.21.15.104:0/4202620268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e9c04f760 con 0x7f4e9c106890 2024-06-09T22:21:12.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.582+0000 7f4e98d99640 1 -- 172.21.15.104:0/4202620268 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f4e94005020 con 0x7f4e9c106890 2024-06-09T22:21:12.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.582+0000 7f4e98d99640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4e880a2ec0 0x7f4e880a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.582+0000 7f4e9ad9d640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4e880a2ec0 0x7f4e880a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.582+0000 7f4e98d99640 1 -- 172.21.15.104:0/4202620268 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f4e940d0210 con 0x7f4e9c106890 2024-06-09T22:21:12.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.583+0000 7f4e9ad9d640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4e880a2ec0 0x7f4e880a5380 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c000c00 tx=0x7f4e8c005a20 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:12.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.587+0000 7f4e98d99640 1 -- 172.21.15.104:0/4202620268 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4e9409c870 con 0x7f4e9c106890 2024-06-09T22:21:12.651 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:12 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2149722122' entity='client.admin' 2024-06-09T22:21:12.652 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:12 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:21:12.652 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:12 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:12.652 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:12 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:12.652 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:12 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:12.652 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:12 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1117895636' entity='client.admin' 2024-06-09T22:21:12.652 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:12 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:21:12.652 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:12 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:12.652 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:12 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:12.652 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:12 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:12.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:12 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2149722122' entity='client.admin' 2024-06-09T22:21:12.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:12 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:21:12.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:12 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:12.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:12 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:12.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:12 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:12.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:12 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1117895636' entity='client.admin' 2024-06-09T22:21:12.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:12 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:21:12.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:12 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:12.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:12 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:12.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:12 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:12.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.762+0000 7efca56cc700 1 -- 172.21.15.104:0/1187452993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efca0102530 msgr2=0x7efca01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.762+0000 7efca56cc700 1 --2- 172.21.15.104:0/1187452993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efca0102530 0x7efca01029b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7efc90009920 tx=0x7efc90009c30 comp rx=0 tx=0).stop 2024-06-09T22:21:12.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.762+0000 7efca56cc700 1 -- 172.21.15.104:0/1187452993 shutdown_connections 2024-06-09T22:21:12.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.762+0000 7efca56cc700 1 --2- 172.21.15.104:0/1187452993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca0102ef0 0x7efca010f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.762+0000 7efca56cc700 1 --2- 172.21.15.104:0/1187452993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca01085d0 0x7efca01089b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.762+0000 7efca56cc700 1 --2- 172.21.15.104:0/1187452993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efca0102530 0x7efca01029b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.762+0000 7efca56cc700 1 -- 172.21.15.104:0/1187452993 >> 172.21.15.104:0/1187452993 conn(0x7efca0079640 msgr2=0x7efca0079a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:12.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.762+0000 7efca56cc700 1 -- 172.21.15.104:0/1187452993 shutdown_connections 2024-06-09T22:21:12.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.762+0000 7efca56cc700 1 -- 172.21.15.104:0/1187452993 wait complete. 2024-06-09T22:21:12.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efca56cc700 1 Processor -- start 2024-06-09T22:21:12.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efca56cc700 1 -- start start 2024-06-09T22:21:12.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efca56cc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca0102530 0x7efca019e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efca56cc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca0102ef0 0x7efca019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efca56cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efca01085d0 0x7efca0198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efca56cc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efca0112810 con 0x7efca01085d0 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efca56cc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efca0112690 con 0x7efca0102530 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efca56cc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efca0112990 con 0x7efca0102ef0 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efc9f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efca01085d0 0x7efca0198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efc9f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efca01085d0 0x7efca0198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37862/0 (socket says 172.21.15.104:37862) 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efc9f7fe700 1 -- 172.21.15.104:0/1838269401 learned_addr learned my addr 172.21.15.104:0/1838269401 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efc9effd700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca0102530 0x7efca019e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca0102ef0 0x7efca019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efc9f7fe700 1 -- 172.21.15.104:0/1838269401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca0102ef0 msgr2=0x7efca019ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efc9f7fe700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca0102ef0 0x7efca019ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efc9f7fe700 1 -- 172.21.15.104:0/1838269401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca0102530 msgr2=0x7efca019e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efc9f7fe700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca0102530 0x7efca019e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efc9f7fe700 1 -- 172.21.15.104:0/1838269401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca0199250 con 0x7efca01085d0 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.763+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca0102ef0 0x7efca019ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.764+0000 7efc9f7fe700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efca01085d0 0x7efca0198af0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7efc9800ef30 tx=0x7efc9800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.764+0000 7efc87fff700 1 -- 172.21.15.104:0/1838269401 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc980092e0 con 0x7efca01085d0 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.764+0000 7efca56cc700 1 -- 172.21.15.104:0/1838269401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca01994e0 con 0x7efca01085d0 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.764+0000 7efc87fff700 1 -- 172.21.15.104:0/1838269401 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc98004bb0 con 0x7efca01085d0 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.764+0000 7efc87fff700 1 -- 172.21.15.104:0/1838269401 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc98004280 con 0x7efca01085d0 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.764+0000 7efca56cc700 1 -- 172.21.15.104:0/1838269401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca01a57b0 con 0x7efca01085d0 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.766+0000 7efc87fff700 1 -- 172.21.15.104:0/1838269401 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7efc9801b030 con 0x7efca01085d0 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.767+0000 7efc87fff700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efc880a2eb0 0x7efc880a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.767+0000 7efc9effd700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efc880a2eb0 0x7efc880a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.767+0000 7efc87fff700 1 -- 172.21.15.104:0/1838269401 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7efc980cf650 con 0x7efca01085d0 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efc9effd700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efc880a2eb0 0x7efc880a5370 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efca0108010 tx=0x7efc94009310 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efcaff4fb80 1 Processor -- start 2024-06-09T22:21:12.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efcaff4fb80 1 -- start start 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efcaff4fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c66ffe0 0x56018c6703c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efcaff4fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c670900 0x56018c66a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efcaff4fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66a820 0x56018c66ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efcaff4fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56018c55cfe0 con 0x56018c670900 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efcaff4fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56018c5bb8d0 con 0x56018c66ffe0 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efcaff4fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56018c5234e0 con 0x56018c66a820 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efc9e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c670900 0x56018c66a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efc9e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c670900 0x56018c66a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37872/0 (socket says 172.21.15.104:37872) 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efc9e7fc700 1 -- 172.21.15.104:0/686046519 learned_addr learned my addr 172.21.15.104:0/686046519 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.768+0000 7efc9f7fe700 1 --2- 172.21.15.104:0/686046519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66a820 0x56018c66ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.769+0000 7efc9effd700 1 --2- 172.21.15.104:0/686046519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c66ffe0 0x56018c6703c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.769+0000 7efc9e7fc700 1 -- 172.21.15.104:0/686046519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66a820 msgr2=0x56018c66ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.769+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/686046519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66a820 0x56018c66ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.769+0000 7efc9e7fc700 1 -- 172.21.15.104:0/686046519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c66ffe0 msgr2=0x56018c6703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.769+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/686046519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c66ffe0 0x56018c6703c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.769+0000 7efc9e7fc700 1 -- 172.21.15.104:0/686046519 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56018c50a950 con 0x56018c670900 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.769+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/686046519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c670900 0x56018c66a2e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7efc90009590 tx=0x7efc900095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72f0d3de7644d32 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.769+0000 7efc857fa700 1 -- 172.21.15.104:0/686046519 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90027070 con 0x56018c670900 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.769+0000 7efc857fa700 1 -- 172.21.15.104:0/686046519 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc90005040 con 0x56018c670900 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.769+0000 7efc857fa700 1 -- 172.21.15.104:0/686046519 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90004b60 con 0x56018c670900 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.770+0000 7efcaff4fb80 1 -- 172.21.15.104:0/686046519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c670900 msgr2=0x56018c66a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.770+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/686046519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c670900 0x56018c66a2e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7efc90009590 tx=0x7efc900095c0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.770+0000 7efcaff4fb80 1 -- 172.21.15.104:0/686046519 shutdown_connections 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.770+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/686046519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66a820 0x56018c66ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.770+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/686046519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c66ffe0 0x56018c6703c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.770+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/686046519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c670900 0x56018c66a2e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.770+0000 7efcaff4fb80 1 -- 172.21.15.104:0/686046519 >> 172.21.15.104:0/686046519 conn(0x56018c597100 msgr2=0x56018c6652a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:12.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.770+0000 7efcaff4fb80 1 -- 172.21.15.104:0/686046519 shutdown_connections 2024-06-09T22:21:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.770+0000 7efcaff4fb80 1 -- 172.21.15.104:0/686046519 wait complete. 2024-06-09T22:21:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.770+0000 7efcaff4fb80 1 -- start start 2024-06-09T22:21:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.770+0000 7efcaff4fb80 1 Processor -- start 2024-06-09T22:21:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efcaff4fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 0x56018c6766d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efcaff4fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66ffe0 0x56018c6739d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efcaff4fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 0x56018c673f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efcaff4fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56018c55c040 con 0x56018c66a820 2024-06-09T22:21:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efcaff4fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56018c5b7890 con 0x56018c670900 2024-06-09T22:21:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efcaff4fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56018c677280 con 0x56018c66ffe0 2024-06-09T22:21:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efc9effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 0x56018c6766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efcaff4fb80 10 client.0 _ll_register_callbacks cb 0x56018c551630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efc9effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 0x56018c6766d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37878/0 (socket says 172.21.15.104:37878) 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efc9effd700 1 -- 172.21.15.104:0/969162536 learned_addr learned my addr 172.21.15.104:0/969162536 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66ffe0 0x56018c6739d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efc9f7fe700 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 0x56018c673f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efc9effd700 1 -- 172.21.15.104:0/969162536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66ffe0 msgr2=0x56018c6739d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efc9effd700 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66ffe0 0x56018c6739d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efc9effd700 1 -- 172.21.15.104:0/969162536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 msgr2=0x56018c673f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efc9effd700 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 0x56018c673f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efc9effd700 1 -- 172.21.15.104:0/969162536 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56018c5817d0 con 0x56018c66a820 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efc9effd700 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 0x56018c6766d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7efc9401ed00 tx=0x7efc940047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34df1c3df3f29d74 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.771+0000 7efc82ff5700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efc82ff5700 1 -- 172.21.15.104:0/969162536 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9401e250 con 0x56018c66a820 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efc82ff5700 1 -- 172.21.15.104:0/969162536 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7efc9401d410 con 0x56018c66a820 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efcaff4fb80 1 -- 172.21.15.104:0/969162536 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc90005040 con 0x56018c66a820 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efc82ff5700 1 -- 172.21.15.104:0/969162536 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efc9401d610 con 0x56018c66a820 2024-06-09T22:21:12.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efcaff4fb80 10 client.15462 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efcaff4fb80 1 -- 172.21.15.104:0/969162536 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56018c681a40 con 0x56018c66a820 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efc82ff5700 1 -- 172.21.15.104:0/969162536 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc90005040 con 0x56018c66a820 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efc82ff5700 1 -- 172.21.15.104:0/969162536 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efc94025070 con 0x56018c66a820 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efcaff4fb80 10 client.15462 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efcaff4fb80 10 client.15462 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efcaff4fb80 20 client.15462 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56018c684080 0x56018c6864b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efcaff4fb80 4 client.15462 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efcaff4fb80 1 -- 172.21.15.104:0/969162536 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56018c5c8dd0 con 0x56018c684080 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.772+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56018c684080 0x56018c6864b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.773+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56018c684080 0x56018c6864b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d3005afc2cb76f0 server_cookie=e64971a6a666cbfb in_seq=0 out_seq=0 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.773+0000 7efc82ff5700 10 client.15462 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efc82ff5700 1 -- 172.21.15.104:0/969162536 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56018c5c8dd0 con 0x56018c684080 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efc82ff5700 10 client.15462 handle_command_reply: tid=0 2024-06-09T22:21:12.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efcaff4fb80 1 client.15462 shutdown 2024-06-09T22:21:12.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efcaff4fb80 1 -- 172.21.15.104:0/969162536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 msgr2=0x56018c6766d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 0x56018c6766d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7efc9401ed00 tx=0x7efc940047c0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efcaff4fb80 1 -- 172.21.15.104:0/969162536 shutdown_connections 2024-06-09T22:21:12.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56018c684080 0x56018c6864b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 0x56018c6766d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 0x56018c673f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/969162536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66ffe0 0x56018c6739d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efcaff4fb80 1 -- 172.21.15.104:0/969162536 >> 172.21.15.104:0/969162536 conn(0x56018c597100 msgr2=0x56018c664780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:12.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efc82ff5700 0 client.15462 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:12.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efcaff4fb80 1 -- 172.21.15.104:0/969162536 shutdown_connections 2024-06-09T22:21:12.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efcaff4fb80 1 -- 172.21.15.104:0/969162536 wait complete. 2024-06-09T22:21:12.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.774+0000 7efcaff4fb80 20 client.15462 trim_cache size 0 max 16384 2024-06-09T22:21:12.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:12 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2149722122' entity='client.admin' 2024-06-09T22:21:12.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:12 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:21:12.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:12 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:12.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:12 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:12.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:12 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:12.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:12 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1117895636' entity='client.admin' 2024-06-09T22:21:12.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:12 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:21:12.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:12 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:21:12.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:12 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:21:12.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:12 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:21:12.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efcaff4fb80 1 Processor -- start 2024-06-09T22:21:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efcaff4fb80 1 -- start start 2024-06-09T22:21:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efcaff4fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66a820 0x56018c671610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efcaff4fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66ffe0 0x56018c675910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efcaff4fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 0x56018c664f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efcaff4fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efca01ae5b0 con 0x56018c66ffe0 2024-06-09T22:21:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efcaff4fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efca01b00f0 con 0x56018c670900 2024-06-09T22:21:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efcaff4fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efca01aec80 con 0x56018c66a820 2024-06-09T22:21:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efc9e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66ffe0 0x56018c675910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efc9e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66ffe0 0x56018c675910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37884/0 (socket says 172.21.15.104:37884) 2024-06-09T22:21:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efc9e7fc700 1 -- 172.21.15.104:0/3259944969 learned_addr learned my addr 172.21.15.104:0/3259944969 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efc9effd700 1 --2- 172.21.15.104:0/3259944969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66a820 0x56018c671610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efc9e7fc700 1 -- 172.21.15.104:0/3259944969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66a820 msgr2=0x56018c671610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/3259944969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66a820 0x56018c671610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efc9e7fc700 1 -- 172.21.15.104:0/3259944969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 msgr2=0x56018c664f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/3259944969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 0x56018c664f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efc9e7fc700 1 -- 172.21.15.104:0/3259944969 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56018c66a620 con 0x56018c66ffe0 2024-06-09T22:21:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.811+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/3259944969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66ffe0 0x56018c675910 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7efc90009c70 tx=0x7efc900046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde8f4569a61f624 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.812+0000 7efc84ff9700 1 -- 172.21.15.104:0/3259944969 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90031070 con 0x56018c66ffe0 2024-06-09T22:21:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.812+0000 7efc84ff9700 1 -- 172.21.15.104:0/3259944969 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc90005520 con 0x56018c66ffe0 2024-06-09T22:21:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.812+0000 7efc84ff9700 1 -- 172.21.15.104:0/3259944969 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90027040 con 0x56018c66ffe0 2024-06-09T22:21:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.812+0000 7f4ea115a640 1 -- 172.21.15.104:0/4202620268 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f4e9c061de0 con 0x7f4e880a2ec0 2024-06-09T22:21:12.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.812+0000 7efcaff4fb80 1 -- 172.21.15.104:0/3259944969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66ffe0 msgr2=0x56018c675910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.812+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/3259944969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66ffe0 0x56018c675910 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7efc90009c70 tx=0x7efc900046f0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.812+0000 7efcaff4fb80 1 -- 172.21.15.104:0/3259944969 shutdown_connections 2024-06-09T22:21:12.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.812+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/3259944969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 0x56018c664f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.812+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/3259944969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66a820 0x56018c671610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.812+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/3259944969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66ffe0 0x56018c675910 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.812+0000 7efcaff4fb80 1 -- 172.21.15.104:0/3259944969 >> 172.21.15.104:0/3259944969 conn(0x56018c597100 msgr2=0x56018c675200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:12.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.812+0000 7efcaff4fb80 1 -- 172.21.15.104:0/3259944969 shutdown_connections 2024-06-09T22:21:12.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efcaff4fb80 1 -- 172.21.15.104:0/3259944969 wait complete. 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efcaff4fb80 1 -- start start 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efcaff4fb80 1 Processor -- start 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efcaff4fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 0x56018c67c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efcaff4fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66ffe0 0x56018c67ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efcaff4fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 0x56018c679510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efcaff4fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efca01afa20 con 0x56018c66a820 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efcaff4fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efca01af350 con 0x56018c670900 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efcaff4fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efca01ac530 con 0x56018c66ffe0 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efcaff4fb80 10 client.0 _ll_register_callbacks cb 0x56018c66b1b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efc9effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 0x56018c67c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efc9effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 0x56018c67c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37886/0 (socket says 172.21.15.104:37886) 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.813+0000 7efc9effd700 1 -- 172.21.15.104:0/4127801290 learned_addr learned my addr 172.21.15.104:0/4127801290 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66ffe0 0x56018c67ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc9f7fe700 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 0x56018c679510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc9effd700 1 -- 172.21.15.104:0/4127801290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66ffe0 msgr2=0x56018c67ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc9effd700 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66ffe0 0x56018c67ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc9effd700 1 -- 172.21.15.104:0/4127801290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 msgr2=0x56018c679510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc9effd700 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 0x56018c679510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc9effd700 1 -- 172.21.15.104:0/4127801290 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56018c679b00 con 0x56018c66a820 2024-06-09T22:21:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66ffe0 0x56018c67ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:21:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc9f7fe700 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 0x56018c679510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc9effd700 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 0x56018c67c540 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7efc94011c00 tx=0x7efc94027770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65695a36b99fa785 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc82ff5700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc82ff5700 1 -- 172.21.15.104:0/4127801290 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9400ea20 con 0x56018c66a820 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc82ff5700 1 -- 172.21.15.104:0/4127801290 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7efc9401d410 con 0x56018c66a820 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efcaff4fb80 1 -- 172.21.15.104:0/4127801290 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc90005520 con 0x56018c66a820 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efc82ff5700 1 -- 172.21.15.104:0/4127801290 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efc94043050 con 0x56018c66a820 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efcaff4fb80 10 client.15468 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.814+0000 7efcaff4fb80 1 -- 172.21.15.104:0/4127801290 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56018c6f3a70 con 0x56018c66a820 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.815+0000 7efc82ff5700 1 -- 172.21.15.104:0/4127801290 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc90005520 con 0x56018c66a820 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.815+0000 7efc82ff5700 1 -- 172.21.15.104:0/4127801290 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efc94025050 con 0x56018c66a820 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.815+0000 7efcaff4fb80 10 client.15468 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.815+0000 7efcaff4fb80 10 client.15468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.815+0000 7efcaff4fb80 20 client.15468 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.815+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56018c6f41d0 0x56018c6f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.815+0000 7efcaff4fb80 4 client.15468 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-09T22:21:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.815+0000 7efcaff4fb80 1 -- 172.21.15.104:0/4127801290 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56018c5c3350 con 0x56018c6f41d0 2024-06-09T22:21:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.815+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56018c6f41d0 0x56018c6f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.816+0000 7efc82ff5700 10 client.15468 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.816+0000 7efc9e7fc700 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56018c6f41d0 0x56018c6f65f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b307fe3ad790ea3 server_cookie=7c96740480dd9ed7 in_seq=0 out_seq=0 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efc82ff5700 1 -- 172.21.15.104:0/4127801290 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56018c5c3350 con 0x56018c6f41d0 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efc82ff5700 10 client.15468 handle_command_reply: tid=0 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efcaff4fb80 1 client.15468 shutdown 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efcaff4fb80 1 -- 172.21.15.104:0/4127801290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 msgr2=0x56018c67c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 0x56018c67c540 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7efc94011c00 tx=0x7efc94027770 comp rx=0 tx=0).stop 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efcaff4fb80 1 -- 172.21.15.104:0/4127801290 shutdown_connections 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56018c6f41d0 0x56018c6f65f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56018c66a820 0x56018c67c540 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56018c670900 0x56018c679510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efc82ff5700 0 client.15468 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efcaff4fb80 1 --2- 172.21.15.104:0/4127801290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56018c66ffe0 0x56018c67ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efcaff4fb80 1 -- 172.21.15.104:0/4127801290 >> 172.21.15.104:0/4127801290 conn(0x56018c597100 msgr2=0x56018c669b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efcaff4fb80 1 -- 172.21.15.104:0/4127801290 shutdown_connections 2024-06-09T22:21:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efcaff4fb80 1 -- 172.21.15.104:0/4127801290 wait complete. 2024-06-09T22:21:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.817+0000 7efcaff4fb80 20 client.15468 trim_cache size 0 max 16384 2024-06-09T22:21:12.821 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:21:12.821 INFO:teuthology.orchestra.run.smithi104.stdout: "return_code": 0, 2024-06-09T22:21:12.821 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_tag": "1c8f834a-0287-4773-a482-c93999ce0077", 2024-06-09T22:21:12.821 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": "asynchronous" 2024-06-09T22:21:12.821 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:21:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.818+0000 7efca56cc700 1 -- 172.21.15.104:0/1838269401 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efc880a2eb0 msgr2=0x7efc880a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.818+0000 7efca56cc700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efc880a2eb0 0x7efc880a5370 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efca0108010 tx=0x7efc94009310 comp rx=0 tx=0).stop 2024-06-09T22:21:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.818+0000 7efca56cc700 1 -- 172.21.15.104:0/1838269401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efca01085d0 msgr2=0x7efca0198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.818+0000 7efca56cc700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efca01085d0 0x7efca0198af0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7efc9800ef30 tx=0x7efc9800c3a0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.818+0000 7efca56cc700 1 -- 172.21.15.104:0/1838269401 shutdown_connections 2024-06-09T22:21:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.818+0000 7efca56cc700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efc880a2eb0 0x7efc880a5370 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.818+0000 7efca56cc700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efca01085d0 0x7efca0198af0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.818+0000 7efca56cc700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca0102530 0x7efca019e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.818+0000 7efca56cc700 1 --2- 172.21.15.104:0/1838269401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca0102ef0 0x7efca019ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.818+0000 7efca56cc700 1 -- 172.21.15.104:0/1838269401 >> 172.21.15.104:0/1838269401 conn(0x7efca0079640 msgr2=0x7efca0101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:12.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.818+0000 7efca56cc700 1 -- 172.21.15.104:0/1838269401 shutdown_connections 2024-06-09T22:21:12.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.818+0000 7efca56cc700 1 -- 172.21.15.104:0/1838269401 wait complete. 2024-06-09T22:21:12.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.821+0000 7f4e98d99640 1 -- 172.21.15.104:0/4202620268 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f4e9c061de0 con 0x7f4e880a2ec0 2024-06-09T22:21:12.823 INFO:teuthology.orchestra.run.smithi104.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-06-09T00:00:00", "created": "2024-06-09T22:21:11", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-06-09T22:21:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.823+0000 7f4ea115a640 1 -- 172.21.15.104:0/4202620268 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4e880a2ec0 msgr2=0x7f4e880a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.823+0000 7f4ea115a640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4e880a2ec0 0x7f4e880a5380 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c000c00 tx=0x7f4e8c005a20 comp rx=0 tx=0).stop 2024-06-09T22:21:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.823+0000 7f4ea115a640 1 -- 172.21.15.104:0/4202620268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c106890 msgr2=0x7f4e9c19b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.823+0000 7f4ea115a640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c106890 0x7f4e9c19b910 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400eb10 tx=0x7f4e9400c2d0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.823+0000 7f4ea115a640 1 -- 172.21.15.104:0/4202620268 shutdown_connections 2024-06-09T22:21:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.823+0000 7f4ea115a640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4e880a2ec0 0x7f4e880a5380 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.823+0000 7f4ea115a640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c106890 0x7f4e9c19b910 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.823+0000 7f4ea115a640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e9c101290 0x7f4e9c1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.823+0000 7f4ea115a640 1 --2- 172.21.15.104:0/4202620268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c1008d0 0x7f4e9c1970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.823+0000 7f4ea115a640 1 -- 172.21.15.104:0/4202620268 >> 172.21.15.104:0/4202620268 conn(0x7f4e9c077a20 msgr2=0x7f4e9c1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:12.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.823+0000 7f4ea115a640 1 -- 172.21.15.104:0/4202620268 shutdown_connections 2024-06-09T22:21:12.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.823+0000 7f4ea115a640 1 -- 172.21.15.104:0/4202620268 wait complete. 2024-06-09T22:21:12.834 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-06-09T22:21:12.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 -- 172.21.15.104:0/3876993429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3bc1008d0 msgr2=0x7ff3bc100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 --2- 172.21.15.104:0/3876993429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3bc1008d0 0x7ff3bc100d50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009930 tx=0x7ff3ac02a420 comp rx=0 tx=0).stop 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 -- 172.21.15.104:0/3876993429 shutdown_connections 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 --2- 172.21.15.104:0/3876993429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3bc101290 0x7ff3bc10d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 --2- 172.21.15.104:0/3876993429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3bc1008d0 0x7ff3bc100d50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 --2- 172.21.15.104:0/3876993429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc106890 0x7ff3bc106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 -- 172.21.15.104:0/3876993429 >> 172.21.15.104:0/3876993429 conn(0x7ff3bc077a20 msgr2=0x7ff3bc077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 -- 172.21.15.104:0/3876993429 shutdown_connections 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 -- 172.21.15.104:0/3876993429 wait complete. 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 Processor -- start 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 -- start start 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3bc1008d0 0x7ff3bc199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc101290 0x7ff3bc199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3bc106890 0x7ff3bc1a4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc110ca0 con 0x7ff3bc1008d0 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc110b20 con 0x7ff3bc101290 2024-06-09T22:21:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.939+0000 7ff3c2d16640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc110e20 con 0x7ff3bc106890 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3c0ab2640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3bc1008d0 0x7ff3bc199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3c0ab2640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3bc1008d0 0x7ff3bc199000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37914/0 (socket says 172.21.15.104:37914) 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3c0ab2640 1 -- 172.21.15.104:0/1297246029 learned_addr learned my addr 172.21.15.104:0/1297246029 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3c12b3640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3bc106890 0x7ff3bc1a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3b3fff640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc101290 0x7ff3bc199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3c0ab2640 1 -- 172.21.15.104:0/1297246029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3bc106890 msgr2=0x7ff3bc1a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3c0ab2640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3bc106890 0x7ff3bc1a4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3c0ab2640 1 -- 172.21.15.104:0/1297246029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc101290 msgr2=0x7ff3bc199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3c0ab2640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc101290 0x7ff3bc199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3c0ab2640 1 -- 172.21.15.104:0/1297246029 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3bc1a45c0 con 0x7ff3bc1008d0 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3b3fff640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc101290 0x7ff3bc199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3c0ab2640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3bc1008d0 0x7ff3bc199000 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800d950 tx=0x7ff3b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3b1ffb640 1 -- 172.21.15.104:0/1297246029 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8017070 con 0x7ff3bc1008d0 2024-06-09T22:21:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3c2d16640 1 -- 172.21.15.104:0/1297246029 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3bc1a4790 con 0x7ff3bc1008d0 2024-06-09T22:21:12.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3b1ffb640 1 -- 172.21.15.104:0/1297246029 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3b8009c30 con 0x7ff3bc1008d0 2024-06-09T22:21:12.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3b1ffb640 1 -- 172.21.15.104:0/1297246029 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8020060 con 0x7ff3bc1008d0 2024-06-09T22:21:12.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.940+0000 7ff3c2d16640 1 -- 172.21.15.104:0/1297246029 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3bc1a4d00 con 0x7ff3bc1008d0 2024-06-09T22:21:12.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.941+0000 7ff3c2d16640 1 -- 172.21.15.104:0/1297246029 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3bc04f760 con 0x7ff3bc1008d0 2024-06-09T22:21:12.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.942+0000 7ff3b1ffb640 1 -- 172.21.15.104:0/1297246029 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7ff3b80109d0 con 0x7ff3bc1008d0 2024-06-09T22:21:12.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.943+0000 7ff3b1ffb640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff3a40a2ec0 0x7ff3a40a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:12.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.943+0000 7ff3b3fff640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff3a40a2ec0 0x7ff3a40a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:12.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.943+0000 7ff3b1ffb640 1 -- 172.21.15.104:0/1297246029 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff3b80cf7f0 con 0x7ff3bc1008d0 2024-06-09T22:21:12.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.943+0000 7ff3b3fff640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff3a40a2ec0 0x7ff3a40a5380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac000c00 tx=0x7ff3ac032c70 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:12.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:12.947+0000 7ff3b1ffb640 1 -- 172.21.15.104:0/1297246029 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff3b8009de0 con 0x7ff3bc1008d0 2024-06-09T22:21:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.152+0000 7ff3c2d16640 1 -- 172.21.15.104:0/1297246029 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7ff3bc061de0 con 0x7ff3a40a2ec0 2024-06-09T22:21:13.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.159+0000 7ff3b1ffb640 1 -- 172.21.15.104:0/1297246029 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7ff3bc061de0 con 0x7ff3a40a2ec0 2024-06-09T22:21:13.158 INFO:teuthology.orchestra.run.smithi104.stdout:/ 1m 6m3h 2024-06-09T22:21:13.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.161+0000 7ff3c2d16640 1 -- 172.21.15.104:0/1297246029 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff3a40a2ec0 msgr2=0x7ff3a40a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:13.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.161+0000 7ff3c2d16640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff3a40a2ec0 0x7ff3a40a5380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac000c00 tx=0x7ff3ac032c70 comp rx=0 tx=0).stop 2024-06-09T22:21:13.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.161+0000 7ff3c2d16640 1 -- 172.21.15.104:0/1297246029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3bc1008d0 msgr2=0x7ff3bc199000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.161+0000 7ff3c2d16640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3bc1008d0 0x7ff3bc199000 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800d950 tx=0x7ff3b800de10 comp rx=0 tx=0).stop 2024-06-09T22:21:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.161+0000 7ff3c2d16640 1 -- 172.21.15.104:0/1297246029 shutdown_connections 2024-06-09T22:21:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.161+0000 7ff3c2d16640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff3a40a2ec0 0x7ff3a40a5380 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.161+0000 7ff3c2d16640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3bc106890 0x7ff3bc1a4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.161+0000 7ff3c2d16640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc101290 0x7ff3bc199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.161+0000 7ff3c2d16640 1 --2- 172.21.15.104:0/1297246029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3bc1008d0 0x7ff3bc199000 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.161+0000 7ff3c2d16640 1 -- 172.21.15.104:0/1297246029 >> 172.21.15.104:0/1297246029 conn(0x7ff3bc077a20 msgr2=0x7ff3bc1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.161+0000 7ff3c2d16640 1 -- 172.21.15.104:0/1297246029 shutdown_connections 2024-06-09T22:21:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:13.161+0000 7ff3c2d16640 1 -- 172.21.15.104:0/1297246029 wait complete. 2024-06-09T22:21:13.171 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-06-09T22:21:13.197 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-06-09T22:21:13.209 INFO:teuthology.run_tasks:Running task background_exec... 2024-06-09T22:21:13.219 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi104.front.sepia.ceph.com 2024-06-09T22:21:13.219 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-06-09T22:21:13.240 INFO:teuthology.run_tasks:Running task quiescer... 2024-06-09T22:21:13.252 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-09T22:21:13.252 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-09T22:21:13.505 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:13 smithi104 ceph-mon[32389]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 853 B/s wr, 19 op/s 2024-06-09T22:21:13.505 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:13 smithi104 ceph-mon[32389]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:13.506 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:13 smithi104 ceph-mon[32389]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:13.566 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:21:13.623 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:13.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:13 smithi153 ceph-mon[36599]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 853 B/s wr, 19 op/s 2024-06-09T22:21:13.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:13 smithi153 ceph-mon[36599]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:13.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:13 smithi153 ceph-mon[36599]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:13.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:13 smithi194 ceph-mon[36856]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 853 B/s wr, 19 op/s 2024-06-09T22:21:13.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:13 smithi194 ceph-mon[36856]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:13.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:13 smithi194 ceph-mon[36856]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:14.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:14 smithi153 ceph-mon[36599]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:14.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:14 smithi153 ceph-mon[36599]: scrub queued for path: / 2024-06-09T22:21:14.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:14 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:21:14.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:14 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:21:14.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:14 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:21:14.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:14 smithi153 ceph-mon[36599]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:14.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:14 smithi104 ceph-mon[32389]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:14.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:14 smithi104 ceph-mon[32389]: scrub queued for path: / 2024-06-09T22:21:14.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:14 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:21:14.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:14 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:21:14.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:14 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:21:14.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:14 smithi104 ceph-mon[32389]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:14.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:14 smithi194 ceph-mon[36856]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:14.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:14 smithi194 ceph-mon[36856]: scrub queued for path: / 2024-06-09T22:21:14.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:14 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:21:14.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:14 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:21:14.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:14 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:21:14.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:14 smithi194 ceph-mon[36856]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:21:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:15.689 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:15 smithi104 ceph-mon[32389]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 40 KiB/s wr, 25 op/s 2024-06-09T22:21:15.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:15 smithi153 ceph-mon[36599]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 40 KiB/s wr, 25 op/s 2024-06-09T22:21:15.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:15 smithi194 ceph-mon[36856]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 40 KiB/s wr, 25 op/s 2024-06-09T22:21:16.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.102+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/784818057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb0101620 msgr2=0x7fbcb0101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:16.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.102+0000 7fbcb5e4e700 1 --2- 172.21.15.104:0/784818057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb0101620 0x7fbcb0101a00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbca00098f0 tx=0x7fbca0009c00 comp rx=0 tx=0).stop 2024-06-09T22:21:16.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.102+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/784818057 shutdown_connections 2024-06-09T22:21:16.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.102+0000 7fbcb5e4e700 1 --2- 172.21.15.104:0/784818057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcb01066b0 0x7fbcb010e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:16.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.102+0000 7fbcb5e4e700 1 --2- 172.21.15.104:0/784818057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb0101620 0x7fbcb0101a00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:16.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.102+0000 7fbcb5e4e700 1 --2- 172.21.15.104:0/784818057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb0101fd0 0x7fbcb0105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:16.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.102+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/784818057 >> 172.21.15.104:0/784818057 conn(0x7fbcb0078340 msgr2=0x7fbcb0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:16.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.102+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/784818057 shutdown_connections 2024-06-09T22:21:16.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.102+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/784818057 wait complete. 2024-06-09T22:21:16.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.102+0000 7fbcb5e4e700 1 Processor -- start 2024-06-09T22:21:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcb5e4e700 1 -- start start 2024-06-09T22:21:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcb5e4e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcb0101620 0x7fbcb0199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcb5e4e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb0101fd0 0x7fbcb019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcb5e4e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb01066b0 0x7fbcb01a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcb5e4e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0111710 con 0x7fbcb01066b0 2024-06-09T22:21:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcb5e4e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0111590 con 0x7fbcb0101fd0 2024-06-09T22:21:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcb5e4e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0111890 con 0x7fbcb0101620 2024-06-09T22:21:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcaffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb01066b0 0x7fbcb01a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcaffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb01066b0 0x7fbcb01a4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55428/0 (socket says 172.21.15.104:55428) 2024-06-09T22:21:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcaffff700 1 -- 172.21.15.104:0/146801189 learned_addr learned my addr 172.21.15.104:0/146801189 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcaf7fe700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcb0101620 0x7fbcb0199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcaeffd700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb0101fd0 0x7fbcb019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcaffff700 1 -- 172.21.15.104:0/146801189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcb0101620 msgr2=0x7fbcb0199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcaffff700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcb0101620 0x7fbcb0199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcaffff700 1 -- 172.21.15.104:0/146801189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb0101fd0 msgr2=0x7fbcb019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcaffff700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb0101fd0 0x7fbcb019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcaffff700 1 -- 172.21.15.104:0/146801189 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb01a51b0 con 0x7fbcb01066b0 2024-06-09T22:21:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcaf7fe700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcb0101620 0x7fbcb0199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:21:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.103+0000 7fbcaffff700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb01066b0 0x7fbcb01a4c70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fbca800b970 tx=0x7fbca800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.104+0000 7fbcacff9700 1 -- 172.21.15.104:0/146801189 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8019070 con 0x7fbcb01066b0 2024-06-09T22:21:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.104+0000 7fbcacff9700 1 -- 172.21.15.104:0/146801189 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbca800ce10 con 0x7fbcb01066b0 2024-06-09T22:21:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.104+0000 7fbcacff9700 1 -- 172.21.15.104:0/146801189 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8007370 con 0x7fbcb01066b0 2024-06-09T22:21:16.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.104+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/146801189 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcb01a53e0 con 0x7fbcb01066b0 2024-06-09T22:21:16.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.104+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/146801189 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcb01a58a0 con 0x7fbcb01066b0 2024-06-09T22:21:16.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.105+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/146801189 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcb0069180 con 0x7fbcb01066b0 2024-06-09T22:21:16.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.106+0000 7fbcacff9700 1 -- 172.21.15.104:0/146801189 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7fbca801c070 con 0x7fbcb01066b0 2024-06-09T22:21:16.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.106+0000 7fbcacff9700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbc980a2f30 0x7fbc980a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:16.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.107+0000 7fbcaf7fe700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbc980a2f30 0x7fbc980a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:16.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.107+0000 7fbcacff9700 1 -- 172.21.15.104:0/146801189 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fbca80ceca0 con 0x7fbcb01066b0 2024-06-09T22:21:16.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.107+0000 7fbcaf7fe700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbc980a2f30 0x7fbc980a53f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbca000b2e0 tx=0x7fbca0019040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:16.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.111+0000 7fbcacff9700 1 -- 172.21.15.104:0/146801189 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbca809b410 con 0x7fbcb01066b0 2024-06-09T22:21:16.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.309+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/146801189 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbcb00628d0 con 0x7fbc980a2f30 2024-06-09T22:21:16.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.319+0000 7fbcacff9700 1 -- 172.21.15.104:0/146801189 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339896 (secure 0 0 0) 0x7fbcb00628d0 con 0x7fbc980a2f30 2024-06-09T22:21:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:16.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.323+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/146801189 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbc980a2f30 msgr2=0x7fbc980a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.323+0000 7fbcb5e4e700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbc980a2f30 0x7fbc980a53f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbca000b2e0 tx=0x7fbca0019040 comp rx=0 tx=0).stop 2024-06-09T22:21:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.324+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/146801189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb01066b0 msgr2=0x7fbcb01a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.324+0000 7fbcb5e4e700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb01066b0 0x7fbcb01a4c70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fbca800b970 tx=0x7fbca800bc80 comp rx=0 tx=0).stop 2024-06-09T22:21:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.324+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/146801189 shutdown_connections 2024-06-09T22:21:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.324+0000 7fbcb5e4e700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbc980a2f30 0x7fbc980a53f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.324+0000 7fbcb5e4e700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcb01066b0 0x7fbcb01a4c70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.324+0000 7fbcb5e4e700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcb0101620 0x7fbcb0199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.324+0000 7fbcb5e4e700 1 --2- 172.21.15.104:0/146801189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb0101fd0 0x7fbcb019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.324+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/146801189 >> 172.21.15.104:0/146801189 conn(0x7fbcb0078340 msgr2=0x7fbcb00ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.324+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/146801189 shutdown_connections 2024-06-09T22:21:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:16.324+0000 7fbcb5e4e700 1 -- 172.21.15.104:0/146801189 wait complete. 2024-06-09T22:21:16.328 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-06-09T22:21:17.469 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-06-09T22:21:15.612241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1399909,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2658,"num_read_kb":1774,"num_write":727,"num_write_kb":5348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1051,"ondisk_log_size":1051,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383836,"kb_used_data":13844,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557988,"statfs":{"total":1151940427776,"available":1151547379712,"internally_reserved":0,"allocated":14176256,"data_stored":11131229,"data_compressed":634224,"data_compressed_allocated":860160,"data_compressed_original":5050368,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":201850,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":507,"num_read_kb":526,"num_write":157,"num_write_kb":927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002817"},"pg_stats":[{"pgid":"3.3f","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710445+0000","last_change":"2024-06-09T22:16:43.212998+0000","last_active":"2024-06-09T22:20:57.710445+0000","last_peered":"2024-06-09T22:20:57.710445+0000","last_clean":"2024-06-09T22:20:57.710445+0000","last_became_active":"2024-06-09T22:16:43.212614+0000","last_became_peered":"2024-06-09T22:16:43.212614+0000","last_unstale":"2024-06-09T22:20:57.710445+0000","last_undegraded":"2024-06-09T22:20:57.710445+0000","last_fullsized":"2024-06-09T22:20:57.710445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:22.728486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"99'4","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711160+0000","last_change":"2024-06-09T22:16:43.208746+0000","last_active":"2024-06-09T22:20:57.711160+0000","last_peered":"2024-06-09T22:20:57.711160+0000","last_clean":"2024-06-09T22:20:57.711160+0000","last_became_active":"2024-06-09T22:16:43.208458+0000","last_became_peered":"2024-06-09T22:16:43.208458+0000","last_unstale":"2024-06-09T22:20:57.711160+0000","last_undegraded":"2024-06-09T22:20:57.711160+0000","last_fullsized":"2024-06-09T22:20:57.711160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:24:35.350449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710878+0000","last_change":"2024-06-09T22:16:43.200770+0000","last_active":"2024-06-09T22:20:57.710878+0000","last_peered":"2024-06-09T22:20:57.710878+0000","last_clean":"2024-06-09T22:20:57.710878+0000","last_became_active":"2024-06-09T22:16:43.200449+0000","last_became_peered":"2024-06-09T22:16:43.200449+0000","last_unstale":"2024-06-09T22:20:57.710878+0000","last_undegraded":"2024-06-09T22:20:57.710878+0000","last_fullsized":"2024-06-09T22:20:57.710878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:51:09.017260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703458+0000","last_change":"2024-06-09T22:16:43.203844+0000","last_active":"2024-06-09T22:20:57.703458+0000","last_peered":"2024-06-09T22:20:57.703458+0000","last_clean":"2024-06-09T22:20:57.703458+0000","last_became_active":"2024-06-09T22:16:43.203431+0000","last_became_peered":"2024-06-09T22:16:43.203431+0000","last_unstale":"2024-06-09T22:20:57.703458+0000","last_undegraded":"2024-06-09T22:20:57.703458+0000","last_fullsized":"2024-06-09T22:20:57.703458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:51:36.913312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"99'4","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694538+0000","last_change":"2024-06-09T22:16:43.207597+0000","last_active":"2024-06-09T22:20:57.694538+0000","last_peered":"2024-06-09T22:20:57.694538+0000","last_clean":"2024-06-09T22:20:57.694538+0000","last_became_active":"2024-06-09T22:16:43.207266+0000","last_became_peered":"2024-06-09T22:16:43.207266+0000","last_unstale":"2024-06-09T22:20:57.694538+0000","last_undegraded":"2024-06-09T22:20:57.694538+0000","last_fullsized":"2024-06-09T22:20:57.694538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:47:05.558403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"99'6","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702988+0000","last_change":"2024-06-09T22:17:26.474121+0000","last_active":"2024-06-09T22:20:57.702988+0000","last_peered":"2024-06-09T22:20:57.702988+0000","last_clean":"2024-06-09T22:20:57.702988+0000","last_became_active":"2024-06-09T22:17:26.473619+0000","last_became_peered":"2024-06-09T22:17:26.473619+0000","last_unstale":"2024-06-09T22:20:57.702988+0000","last_undegraded":"2024-06-09T22:20:57.702988+0000","last_fullsized":"2024-06-09T22:20:57.702988+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:56:42.171235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"99'5","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596592+0000","last_change":"2024-06-09T22:16:43.208815+0000","last_active":"2024-06-09T22:20:57.596592+0000","last_peered":"2024-06-09T22:20:57.596592+0000","last_clean":"2024-06-09T22:20:57.596592+0000","last_became_active":"2024-06-09T22:16:43.208433+0000","last_became_peered":"2024-06-09T22:16:43.208433+0000","last_unstale":"2024-06-09T22:20:57.596592+0000","last_undegraded":"2024-06-09T22:20:57.596592+0000","last_fullsized":"2024-06-09T22:20:57.596592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:36:26.585037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694783+0000","last_change":"2024-06-09T22:16:43.210195+0000","last_active":"2024-06-09T22:20:57.694783+0000","last_peered":"2024-06-09T22:20:57.694783+0000","last_clean":"2024-06-09T22:20:57.694783+0000","last_became_active":"2024-06-09T22:16:43.209892+0000","last_became_peered":"2024-06-09T22:16:43.209892+0000","last_unstale":"2024-06-09T22:20:57.694783+0000","last_undegraded":"2024-06-09T22:20:57.694783+0000","last_fullsized":"2024-06-09T22:20:57.694783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:44.088212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710618+0000","last_change":"2024-06-09T22:16:43.202517+0000","last_active":"2024-06-09T22:20:57.710618+0000","last_peered":"2024-06-09T22:20:57.710618+0000","last_clean":"2024-06-09T22:20:57.710618+0000","last_became_active":"2024-06-09T22:16:43.201796+0000","last_became_peered":"2024-06-09T22:16:43.201796+0000","last_unstale":"2024-06-09T22:20:57.710618+0000","last_undegraded":"2024-06-09T22:20:57.710618+0000","last_fullsized":"2024-06-09T22:20:57.710618+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:30.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"99'16","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703409+0000","last_change":"2024-06-09T22:16:43.212732+0000","last_active":"2024-06-09T22:20:57.703409+0000","last_peered":"2024-06-09T22:20:57.703409+0000","last_clean":"2024-06-09T22:20:57.703409+0000","last_became_active":"2024-06-09T22:16:43.212048+0000","last_became_peered":"2024-06-09T22:16:43.212048+0000","last_unstale":"2024-06-09T22:20:57.703409+0000","last_undegraded":"2024-06-09T22:20:57.703409+0000","last_fullsized":"2024-06-09T22:20:57.703409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:47.383291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597025+0000","last_change":"2024-06-09T22:16:43.203778+0000","last_active":"2024-06-09T22:20:57.597025+0000","last_peered":"2024-06-09T22:20:57.597025+0000","last_clean":"2024-06-09T22:20:57.597025+0000","last_became_active":"2024-06-09T22:16:43.202954+0000","last_became_peered":"2024-06-09T22:16:43.202954+0000","last_unstale":"2024-06-09T22:20:57.597025+0000","last_undegraded":"2024-06-09T22:20:57.597025+0000","last_fullsized":"2024-06-09T22:20:57.597025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:56:53.708645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703526+0000","last_change":"2024-06-09T22:16:43.211425+0000","last_active":"2024-06-09T22:20:57.703526+0000","last_peered":"2024-06-09T22:20:57.703526+0000","last_clean":"2024-06-09T22:20:57.703526+0000","last_became_active":"2024-06-09T22:16:43.211001+0000","last_became_peered":"2024-06-09T22:16:43.211001+0000","last_unstale":"2024-06-09T22:20:57.703526+0000","last_undegraded":"2024-06-09T22:20:57.703526+0000","last_fullsized":"2024-06-09T22:20:57.703526+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:20:59.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"99'10","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595794+0000","last_change":"2024-06-09T22:16:43.208334+0000","last_active":"2024-06-09T22:20:57.595794+0000","last_peered":"2024-06-09T22:20:57.595794+0000","last_clean":"2024-06-09T22:20:57.595794+0000","last_became_active":"2024-06-09T22:16:43.207993+0000","last_became_peered":"2024-06-09T22:16:43.207993+0000","last_unstale":"2024-06-09T22:20:57.595794+0000","last_undegraded":"2024-06-09T22:20:57.595794+0000","last_fullsized":"2024-06-09T22:20:57.595794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:07:05.740748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710461+0000","last_change":"2024-06-09T22:16:43.207211+0000","last_active":"2024-06-09T22:20:57.710461+0000","last_peered":"2024-06-09T22:20:57.710461+0000","last_clean":"2024-06-09T22:20:57.710461+0000","last_became_active":"2024-06-09T22:16:43.206889+0000","last_became_peered":"2024-06-09T22:16:43.206889+0000","last_unstale":"2024-06-09T22:20:57.710461+0000","last_undegraded":"2024-06-09T22:20:57.710461+0000","last_fullsized":"2024-06-09T22:20:57.710461+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:25:47.461030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710404+0000","last_change":"2024-06-09T22:16:43.202202+0000","last_active":"2024-06-09T22:20:57.710404+0000","last_peered":"2024-06-09T22:20:57.710404+0000","last_clean":"2024-06-09T22:20:57.710404+0000","last_became_active":"2024-06-09T22:16:43.201834+0000","last_became_peered":"2024-06-09T22:16:43.201834+0000","last_unstale":"2024-06-09T22:20:57.710404+0000","last_undegraded":"2024-06-09T22:20:57.710404+0000","last_fullsized":"2024-06-09T22:20:57.710404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:31:51.296627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"99'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709927+0000","last_change":"2024-06-09T22:16:43.200637+0000","last_active":"2024-06-09T22:20:57.709927+0000","last_peered":"2024-06-09T22:20:57.709927+0000","last_clean":"2024-06-09T22:20:57.709927+0000","last_became_active":"2024-06-09T22:16:43.200268+0000","last_became_peered":"2024-06-09T22:16:43.200268+0000","last_unstale":"2024-06-09T22:20:57.709927+0000","last_undegraded":"2024-06-09T22:20:57.709927+0000","last_fullsized":"2024-06-09T22:20:57.709927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:25:59.881453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710229+0000","last_change":"2024-06-09T22:16:43.209212+0000","last_active":"2024-06-09T22:20:57.710229+0000","last_peered":"2024-06-09T22:20:57.710229+0000","last_clean":"2024-06-09T22:20:57.710229+0000","last_became_active":"2024-06-09T22:16:43.207789+0000","last_became_peered":"2024-06-09T22:16:43.207789+0000","last_unstale":"2024-06-09T22:20:57.710229+0000","last_undegraded":"2024-06-09T22:20:57.710229+0000","last_fullsized":"2024-06-09T22:20:57.710229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:17.316595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692756+0000","last_change":"2024-06-09T22:16:43.210935+0000","last_active":"2024-06-09T22:20:57.692756+0000","last_peered":"2024-06-09T22:20:57.692756+0000","last_clean":"2024-06-09T22:20:57.692756+0000","last_became_active":"2024-06-09T22:16:43.210631+0000","last_became_peered":"2024-06-09T22:16:43.210631+0000","last_unstale":"2024-06-09T22:20:57.692756+0000","last_undegraded":"2024-06-09T22:20:57.692756+0000","last_fullsized":"2024-06-09T22:20:57.692756+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:22:58.887313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"99'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597630+0000","last_change":"2024-06-09T22:16:43.198872+0000","last_active":"2024-06-09T22:20:57.597630+0000","last_peered":"2024-06-09T22:20:57.597630+0000","last_clean":"2024-06-09T22:20:57.597630+0000","last_became_active":"2024-06-09T22:16:43.198498+0000","last_became_peered":"2024-06-09T22:16:43.198498+0000","last_unstale":"2024-06-09T22:20:57.597630+0000","last_undegraded":"2024-06-09T22:20:57.597630+0000","last_fullsized":"2024-06-09T22:20:57.597630+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:25:28.853048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597223+0000","last_change":"2024-06-09T22:16:53.343656+0000","last_active":"2024-06-09T22:20:57.597223+0000","last_peered":"2024-06-09T22:20:57.597223+0000","last_clean":"2024-06-09T22:20:57.597223+0000","last_became_active":"2024-06-09T22:16:53.343281+0000","last_became_peered":"2024-06-09T22:16:53.343281+0000","last_unstale":"2024-06-09T22:20:57.597223+0000","last_undegraded":"2024-06-09T22:20:57.597223+0000","last_fullsized":"2024-06-09T22:20:57.597223+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:40.961842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"100'14","reported_seq":264,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:12.719872+0000","last_change":"2024-06-09T22:16:40.040145+0000","last_active":"2024-06-09T22:21:12.719872+0000","last_peered":"2024-06-09T22:21:12.719872+0000","last_clean":"2024-06-09T22:21:12.719872+0000","last_became_active":"2024-06-09T22:16:40.039787+0000","last_became_peered":"2024-06-09T22:16:40.039787+0000","last_unstale":"2024-06-09T22:21:12.719872+0000","last_undegraded":"2024-06-09T22:21:12.719872+0000","last_fullsized":"2024-06-09T22:21:12.719872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:52:32.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":204,"num_read_kb":204,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"99'5","reported_seq":23,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692731+0000","last_change":"2024-06-09T22:17:26.774053+0000","last_active":"2024-06-09T22:20:57.692731+0000","last_peered":"2024-06-09T22:20:57.692731+0000","last_clean":"2024-06-09T22:20:57.692731+0000","last_became_active":"2024-06-09T22:17:26.773786+0000","last_became_peered":"2024-06-09T22:17:26.773786+0000","last_unstale":"2024-06-09T22:20:57.692731+0000","last_undegraded":"2024-06-09T22:20:57.692731+0000","last_fullsized":"2024-06-09T22:20:57.692731+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:36.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703244+0000","last_change":"2024-06-09T22:16:53.332048+0000","last_active":"2024-06-09T22:20:57.703244+0000","last_peered":"2024-06-09T22:20:57.703244+0000","last_clean":"2024-06-09T22:20:57.703244+0000","last_became_active":"2024-06-09T22:16:53.330683+0000","last_became_peered":"2024-06-09T22:16:53.330683+0000","last_unstale":"2024-06-09T22:20:57.703244+0000","last_undegraded":"2024-06-09T22:20:57.703244+0000","last_fullsized":"2024-06-09T22:20:57.703244+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:32:23.219061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"97'2","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696082+0000","last_change":"2024-06-09T22:16:40.070193+0000","last_active":"2024-06-09T22:20:57.696082+0000","last_peered":"2024-06-09T22:20:57.696082+0000","last_clean":"2024-06-09T22:20:57.696082+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:20:57.696082+0000","last_undegraded":"2024-06-09T22:20:57.696082+0000","last_fullsized":"2024-06-09T22:20:57.696082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:19:19.787552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711015+0000","last_change":"2024-06-09T22:16:43.199814+0000","last_active":"2024-06-09T22:20:57.711015+0000","last_peered":"2024-06-09T22:20:57.711015+0000","last_clean":"2024-06-09T22:20:57.711015+0000","last_became_active":"2024-06-09T22:16:43.199487+0000","last_became_peered":"2024-06-09T22:16:43.199487+0000","last_unstale":"2024-06-09T22:20:57.711015+0000","last_undegraded":"2024-06-09T22:20:57.711015+0000","last_fullsized":"2024-06-09T22:20:57.711015+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:55.672050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703543+0000","last_change":"2024-06-09T22:16:53.336253+0000","last_active":"2024-06-09T22:20:57.703543+0000","last_peered":"2024-06-09T22:20:57.703543+0000","last_clean":"2024-06-09T22:20:57.703543+0000","last_became_active":"2024-06-09T22:16:53.335869+0000","last_became_peered":"2024-06-09T22:16:53.335869+0000","last_unstale":"2024-06-09T22:20:57.703543+0000","last_undegraded":"2024-06-09T22:20:57.703543+0000","last_fullsized":"2024-06-09T22:20:57.703543+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:28:12.894878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"97'12","reported_seq":60,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695155+0000","last_change":"2024-06-09T22:16:40.045365+0000","last_active":"2024-06-09T22:20:57.695155+0000","last_peered":"2024-06-09T22:20:57.695155+0000","last_clean":"2024-06-09T22:20:57.695155+0000","last_became_active":"2024-06-09T22:16:40.044989+0000","last_became_peered":"2024-06-09T22:16:40.044989+0000","last_unstale":"2024-06-09T22:20:57.695155+0000","last_undegraded":"2024-06-09T22:20:57.695155+0000","last_fullsized":"2024-06-09T22:20:57.695155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:01:05.850170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"99'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710115+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:20:57.710115+0000","last_peered":"2024-06-09T22:20:57.710115+0000","last_clean":"2024-06-09T22:20:57.710115+0000","last_became_active":"2024-06-09T22:16:43.203999+0000","last_became_peered":"2024-06-09T22:16:43.203999+0000","last_unstale":"2024-06-09T22:20:57.710115+0000","last_undegraded":"2024-06-09T22:20:57.710115+0000","last_fullsized":"2024-06-09T22:20:57.710115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:37:28.362655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703277+0000","last_change":"2024-06-09T22:16:53.333565+0000","last_active":"2024-06-09T22:20:57.703277+0000","last_peered":"2024-06-09T22:20:57.703277+0000","last_clean":"2024-06-09T22:20:57.703277+0000","last_became_active":"2024-06-09T22:16:53.332921+0000","last_became_peered":"2024-06-09T22:16:53.332921+0000","last_unstale":"2024-06-09T22:20:57.703277+0000","last_undegraded":"2024-06-09T22:20:57.703277+0000","last_fullsized":"2024-06-09T22:20:57.703277+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:32.085617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"97'5","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694916+0000","last_change":"2024-06-09T22:16:40.039766+0000","last_active":"2024-06-09T22:20:57.694916+0000","last_peered":"2024-06-09T22:20:57.694916+0000","last_clean":"2024-06-09T22:20:57.694916+0000","last_became_active":"2024-06-09T22:16:40.039329+0000","last_became_peered":"2024-06-09T22:16:40.039329+0000","last_unstale":"2024-06-09T22:20:57.694916+0000","last_undegraded":"2024-06-09T22:20:57.694916+0000","last_fullsized":"2024-06-09T22:20:57.694916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:53.673890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"99'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703385+0000","last_change":"2024-06-09T22:16:43.201200+0000","last_active":"2024-06-09T22:20:57.703385+0000","last_peered":"2024-06-09T22:20:57.703385+0000","last_clean":"2024-06-09T22:20:57.703385+0000","last_became_active":"2024-06-09T22:16:43.200830+0000","last_became_peered":"2024-06-09T22:16:43.200830+0000","last_unstale":"2024-06-09T22:20:57.703385+0000","last_undegraded":"2024-06-09T22:20:57.703385+0000","last_fullsized":"2024-06-09T22:20:57.703385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:38:52.093173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597461+0000","last_change":"2024-06-09T22:16:53.339811+0000","last_active":"2024-06-09T22:20:57.597461+0000","last_peered":"2024-06-09T22:20:57.597461+0000","last_clean":"2024-06-09T22:20:57.597461+0000","last_became_active":"2024-06-09T22:16:53.339305+0000","last_became_peered":"2024-06-09T22:16:53.339305+0000","last_unstale":"2024-06-09T22:20:57.597461+0000","last_undegraded":"2024-06-09T22:20:57.597461+0000","last_fullsized":"2024-06-09T22:20:57.597461+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:08:25.800245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"100'9","reported_seq":262,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:11.721311+0000","last_change":"2024-06-09T22:16:40.044159+0000","last_active":"2024-06-09T22:21:11.721311+0000","last_peered":"2024-06-09T22:21:11.721311+0000","last_clean":"2024-06-09T22:21:11.721311+0000","last_became_active":"2024-06-09T22:16:40.043839+0000","last_became_peered":"2024-06-09T22:16:40.043839+0000","last_unstale":"2024-06-09T22:21:11.721311+0000","last_undegraded":"2024-06-09T22:21:11.721311+0000","last_fullsized":"2024-06-09T22:21:11.721311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:22:09.339219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":207,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694583+0000","last_change":"2024-06-09T22:16:43.202015+0000","last_active":"2024-06-09T22:20:57.694583+0000","last_peered":"2024-06-09T22:20:57.694583+0000","last_clean":"2024-06-09T22:20:57.694583+0000","last_became_active":"2024-06-09T22:16:43.201689+0000","last_became_peered":"2024-06-09T22:16:43.201689+0000","last_unstale":"2024-06-09T22:20:57.694583+0000","last_undegraded":"2024-06-09T22:20:57.694583+0000","last_fullsized":"2024-06-09T22:20:57.694583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:35:12.180399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597147+0000","last_change":"2024-06-09T22:18:26.698600+0000","last_active":"2024-06-09T22:20:57.597147+0000","last_peered":"2024-06-09T22:20:57.597147+0000","last_clean":"2024-06-09T22:20:57.597147+0000","last_became_active":"2024-06-09T22:18:26.698201+0000","last_became_peered":"2024-06-09T22:18:26.698201+0000","last_unstale":"2024-06-09T22:20:57.597147+0000","last_undegraded":"2024-06-09T22:20:57.597147+0000","last_fullsized":"2024-06-09T22:20:57.597147+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:55:13.513557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"97'7","reported_seq":57,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702844+0000","last_change":"2024-06-09T22:17:26.698204+0000","last_active":"2024-06-09T22:20:57.702844+0000","last_peered":"2024-06-09T22:20:57.702844+0000","last_clean":"2024-06-09T22:20:57.702844+0000","last_became_active":"2024-06-09T22:17:26.697716+0000","last_became_peered":"2024-06-09T22:17:26.697716+0000","last_unstale":"2024-06-09T22:20:57.702844+0000","last_undegraded":"2024-06-09T22:20:57.702844+0000","last_fullsized":"2024-06-09T22:20:57.702844+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:36:37.724459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703554+0000","last_change":"2024-06-09T22:16:43.198917+0000","last_active":"2024-06-09T22:20:57.703554+0000","last_peered":"2024-06-09T22:20:57.703554+0000","last_clean":"2024-06-09T22:20:57.703554+0000","last_became_active":"2024-06-09T22:16:43.198567+0000","last_became_peered":"2024-06-09T22:16:43.198567+0000","last_unstale":"2024-06-09T22:20:57.703554+0000","last_undegraded":"2024-06-09T22:20:57.703554+0000","last_fullsized":"2024-06-09T22:20:57.703554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:34.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710233+0000","last_change":"2024-06-09T22:16:53.342203+0000","last_active":"2024-06-09T22:20:57.710233+0000","last_peered":"2024-06-09T22:20:57.710233+0000","last_clean":"2024-06-09T22:20:57.710233+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:20:57.710233+0000","last_undegraded":"2024-06-09T22:20:57.710233+0000","last_fullsized":"2024-06-09T22:20:57.710233+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:41:41.491961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"97'3","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595514+0000","last_change":"2024-06-09T22:16:40.038684+0000","last_active":"2024-06-09T22:20:57.595514+0000","last_peered":"2024-06-09T22:20:57.595514+0000","last_clean":"2024-06-09T22:20:57.595514+0000","last_became_active":"2024-06-09T22:16:40.038386+0000","last_became_peered":"2024-06-09T22:16:40.038386+0000","last_unstale":"2024-06-09T22:20:57.595514+0000","last_undegraded":"2024-06-09T22:20:57.595514+0000","last_fullsized":"2024-06-09T22:20:57.595514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:59.188758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694679+0000","last_change":"2024-06-09T22:16:43.198837+0000","last_active":"2024-06-09T22:20:57.694679+0000","last_peered":"2024-06-09T22:20:57.694679+0000","last_clean":"2024-06-09T22:20:57.694679+0000","last_became_active":"2024-06-09T22:16:43.198515+0000","last_became_peered":"2024-06-09T22:16:43.198515+0000","last_unstale":"2024-06-09T22:20:57.694679+0000","last_undegraded":"2024-06-09T22:20:57.694679+0000","last_fullsized":"2024-06-09T22:20:57.694679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:36:13.863202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711076+0000","last_change":"2024-06-09T22:16:53.342688+0000","last_active":"2024-06-09T22:20:57.711076+0000","last_peered":"2024-06-09T22:20:57.711076+0000","last_clean":"2024-06-09T22:20:57.711076+0000","last_became_active":"2024-06-09T22:16:53.342314+0000","last_became_peered":"2024-06-09T22:16:53.342314+0000","last_unstale":"2024-06-09T22:20:57.711076+0000","last_undegraded":"2024-06-09T22:20:57.711076+0000","last_fullsized":"2024-06-09T22:20:57.711076+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:51:35.801587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"97'3","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703264+0000","last_change":"2024-06-09T22:16:40.041831+0000","last_active":"2024-06-09T22:20:57.703264+0000","last_peered":"2024-06-09T22:20:57.703264+0000","last_clean":"2024-06-09T22:20:57.703264+0000","last_became_active":"2024-06-09T22:16:40.041484+0000","last_became_peered":"2024-06-09T22:16:40.041484+0000","last_unstale":"2024-06-09T22:20:57.703264+0000","last_undegraded":"2024-06-09T22:20:57.703264+0000","last_fullsized":"2024-06-09T22:20:57.703264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:10:19.829357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692213+0000","last_change":"2024-06-09T22:16:43.210769+0000","last_active":"2024-06-09T22:20:57.692213+0000","last_peered":"2024-06-09T22:20:57.692213+0000","last_clean":"2024-06-09T22:20:57.692213+0000","last_became_active":"2024-06-09T22:16:43.210429+0000","last_became_peered":"2024-06-09T22:16:43.210429+0000","last_unstale":"2024-06-09T22:20:57.692213+0000","last_undegraded":"2024-06-09T22:20:57.692213+0000","last_fullsized":"2024-06-09T22:20:57.692213+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:12:28.788830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694708+0000","last_change":"2024-06-09T22:16:53.336536+0000","last_active":"2024-06-09T22:20:57.694708+0000","last_peered":"2024-06-09T22:20:57.694708+0000","last_clean":"2024-06-09T22:20:57.694708+0000","last_became_active":"2024-06-09T22:16:53.335178+0000","last_became_peered":"2024-06-09T22:16:53.335178+0000","last_unstale":"2024-06-09T22:20:57.694708+0000","last_undegraded":"2024-06-09T22:20:57.694708+0000","last_fullsized":"2024-06-09T22:20:57.694708+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:57:19.476483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'8","reported_seq":456,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:12.721930+0000","last_change":"2024-06-09T22:16:40.038484+0000","last_active":"2024-06-09T22:21:12.721930+0000","last_peered":"2024-06-09T22:21:12.721930+0000","last_clean":"2024-06-09T22:21:12.721930+0000","last_became_active":"2024-06-09T22:16:40.038084+0000","last_became_peered":"2024-06-09T22:16:40.038084+0000","last_unstale":"2024-06-09T22:21:12.721930+0000","last_undegraded":"2024-06-09T22:21:12.721930+0000","last_fullsized":"2024-06-09T22:21:12.721930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:19.733107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":404,"num_read_kb":404,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710525+0000","last_change":"2024-06-09T22:16:43.205137+0000","last_active":"2024-06-09T22:20:57.710525+0000","last_peered":"2024-06-09T22:20:57.710525+0000","last_clean":"2024-06-09T22:20:57.710525+0000","last_became_active":"2024-06-09T22:16:43.204499+0000","last_became_peered":"2024-06-09T22:16:43.204499+0000","last_unstale":"2024-06-09T22:20:57.710525+0000","last_undegraded":"2024-06-09T22:20:57.710525+0000","last_fullsized":"2024-06-09T22:20:57.710525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:15.718493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692194+0000","last_change":"2024-06-09T22:16:53.344260+0000","last_active":"2024-06-09T22:20:57.692194+0000","last_peered":"2024-06-09T22:20:57.692194+0000","last_clean":"2024-06-09T22:20:57.692194+0000","last_became_active":"2024-06-09T22:16:53.343840+0000","last_became_peered":"2024-06-09T22:16:53.343840+0000","last_unstale":"2024-06-09T22:20:57.692194+0000","last_undegraded":"2024-06-09T22:20:57.692194+0000","last_fullsized":"2024-06-09T22:20:57.692194+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:40:23.305509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"97'4","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596747+0000","last_change":"2024-06-09T22:16:40.042660+0000","last_active":"2024-06-09T22:20:57.596747+0000","last_peered":"2024-06-09T22:20:57.596747+0000","last_clean":"2024-06-09T22:20:57.596747+0000","last_became_active":"2024-06-09T22:16:40.041353+0000","last_became_peered":"2024-06-09T22:16:40.041353+0000","last_unstale":"2024-06-09T22:20:57.596747+0000","last_undegraded":"2024-06-09T22:20:57.596747+0000","last_fullsized":"2024-06-09T22:20:57.596747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:34:59.064827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711158+0000","last_change":"2024-06-09T22:16:43.205083+0000","last_active":"2024-06-09T22:20:57.711158+0000","last_peered":"2024-06-09T22:20:57.711158+0000","last_clean":"2024-06-09T22:20:57.711158+0000","last_became_active":"2024-06-09T22:16:43.204600+0000","last_became_peered":"2024-06-09T22:16:43.204600+0000","last_unstale":"2024-06-09T22:20:57.711158+0000","last_undegraded":"2024-06-09T22:20:57.711158+0000","last_fullsized":"2024-06-09T22:20:57.711158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:05:12.555465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703699+0000","last_change":"2024-06-09T22:16:53.336169+0000","last_active":"2024-06-09T22:20:57.703699+0000","last_peered":"2024-06-09T22:20:57.703699+0000","last_clean":"2024-06-09T22:20:57.703699+0000","last_became_active":"2024-06-09T22:16:53.335657+0000","last_became_peered":"2024-06-09T22:16:53.335657+0000","last_unstale":"2024-06-09T22:20:57.703699+0000","last_undegraded":"2024-06-09T22:20:57.703699+0000","last_fullsized":"2024-06-09T22:20:57.703699+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:53:21.282767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703313+0000","last_change":"2024-06-09T22:16:40.070568+0000","last_active":"2024-06-09T22:20:57.703313+0000","last_peered":"2024-06-09T22:20:57.703313+0000","last_clean":"2024-06-09T22:20:57.703313+0000","last_became_active":"2024-06-09T22:16:40.070205+0000","last_became_peered":"2024-06-09T22:16:40.070205+0000","last_unstale":"2024-06-09T22:20:57.703313+0000","last_undegraded":"2024-06-09T22:20:57.703313+0000","last_fullsized":"2024-06-09T22:20:57.703313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.449965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710336+0000","last_change":"2024-06-09T22:16:43.205219+0000","last_active":"2024-06-09T22:20:57.710336+0000","last_peered":"2024-06-09T22:20:57.710336+0000","last_clean":"2024-06-09T22:20:57.710336+0000","last_became_active":"2024-06-09T22:16:43.203467+0000","last_became_peered":"2024-06-09T22:16:43.203467+0000","last_unstale":"2024-06-09T22:20:57.710336+0000","last_undegraded":"2024-06-09T22:20:57.710336+0000","last_fullsized":"2024-06-09T22:20:57.710336+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:22.784034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692165+0000","last_change":"2024-06-09T22:16:53.338990+0000","last_active":"2024-06-09T22:20:57.692165+0000","last_peered":"2024-06-09T22:20:57.692165+0000","last_clean":"2024-06-09T22:20:57.692165+0000","last_became_active":"2024-06-09T22:16:53.338568+0000","last_became_peered":"2024-06-09T22:16:53.338568+0000","last_unstale":"2024-06-09T22:20:57.692165+0000","last_undegraded":"2024-06-09T22:20:57.692165+0000","last_fullsized":"2024-06-09T22:20:57.692165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:46:39.814043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"100'48","reported_seq":510,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:12.822797+0000","last_change":"2024-06-09T22:16:40.044404+0000","last_active":"2024-06-09T22:21:12.822797+0000","last_peered":"2024-06-09T22:21:12.822797+0000","last_clean":"2024-06-09T22:21:12.822797+0000","last_became_active":"2024-06-09T22:16:40.044044+0000","last_became_peered":"2024-06-09T22:16:40.044044+0000","last_unstale":"2024-06-09T22:21:12.822797+0000","last_undegraded":"2024-06-09T22:21:12.822797+0000","last_fullsized":"2024-06-09T22:21:12.822797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:50:27.038759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77586,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":418,"num_read_kb":285,"num_write":50,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694903+0000","last_change":"2024-06-09T22:16:43.206963+0000","last_active":"2024-06-09T22:20:57.694903+0000","last_peered":"2024-06-09T22:20:57.694903+0000","last_clean":"2024-06-09T22:20:57.694903+0000","last_became_active":"2024-06-09T22:16:43.206628+0000","last_became_peered":"2024-06-09T22:16:43.206628+0000","last_unstale":"2024-06-09T22:20:57.694903+0000","last_undegraded":"2024-06-09T22:20:57.694903+0000","last_fullsized":"2024-06-09T22:20:57.694903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:33:39.912236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703845+0000","last_change":"2024-06-09T22:16:53.339332+0000","last_active":"2024-06-09T22:20:57.703845+0000","last_peered":"2024-06-09T22:20:57.703845+0000","last_clean":"2024-06-09T22:20:57.703845+0000","last_became_active":"2024-06-09T22:16:53.338819+0000","last_became_peered":"2024-06-09T22:16:53.338819+0000","last_unstale":"2024-06-09T22:20:57.703845+0000","last_undegraded":"2024-06-09T22:20:57.703845+0000","last_fullsized":"2024-06-09T22:20:57.703845+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:57:41.142142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"97'9","reported_seq":33,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709996+0000","last_change":"2024-06-09T22:17:26.478256+0000","last_active":"2024-06-09T22:20:57.709996+0000","last_peered":"2024-06-09T22:20:57.709996+0000","last_clean":"2024-06-09T22:20:57.709996+0000","last_became_active":"2024-06-09T22:17:26.472264+0000","last_became_peered":"2024-06-09T22:17:26.472264+0000","last_unstale":"2024-06-09T22:20:57.709996+0000","last_undegraded":"2024-06-09T22:20:57.709996+0000","last_fullsized":"2024-06-09T22:20:57.709996+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:14:19.919841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"99'9","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595647+0000","last_change":"2024-06-09T22:16:43.204484+0000","last_active":"2024-06-09T22:20:57.595647+0000","last_peered":"2024-06-09T22:20:57.595647+0000","last_clean":"2024-06-09T22:20:57.595647+0000","last_became_active":"2024-06-09T22:16:43.204014+0000","last_became_peered":"2024-06-09T22:16:43.204014+0000","last_unstale":"2024-06-09T22:20:57.595647+0000","last_undegraded":"2024-06-09T22:20:57.595647+0000","last_fullsized":"2024-06-09T22:20:57.595647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:55:12.528231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710227+0000","last_change":"2024-06-09T22:17:26.475179+0000","last_active":"2024-06-09T22:20:57.710227+0000","last_peered":"2024-06-09T22:20:57.710227+0000","last_clean":"2024-06-09T22:20:57.710227+0000","last_became_active":"2024-06-09T22:17:26.474517+0000","last_became_peered":"2024-06-09T22:17:26.474517+0000","last_unstale":"2024-06-09T22:20:57.710227+0000","last_undegraded":"2024-06-09T22:20:57.710227+0000","last_fullsized":"2024-06-09T22:20:57.710227+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:26:02.387830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703306+0000","last_change":"2024-06-09T22:16:40.043066+0000","last_active":"2024-06-09T22:20:57.703306+0000","last_peered":"2024-06-09T22:20:57.703306+0000","last_clean":"2024-06-09T22:20:57.703306+0000","last_became_active":"2024-06-09T22:16:40.042619+0000","last_became_peered":"2024-06-09T22:16:40.042619+0000","last_unstale":"2024-06-09T22:20:57.703306+0000","last_undegraded":"2024-06-09T22:20:57.703306+0000","last_fullsized":"2024-06-09T22:20:57.703306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:29.689614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"99'15","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597173+0000","last_change":"2024-06-09T22:17:26.475566+0000","last_active":"2024-06-09T22:20:57.597173+0000","last_peered":"2024-06-09T22:20:57.597173+0000","last_clean":"2024-06-09T22:20:57.597173+0000","last_became_active":"2024-06-09T22:17:26.475037+0000","last_became_peered":"2024-06-09T22:17:26.475037+0000","last_unstale":"2024-06-09T22:20:57.597173+0000","last_undegraded":"2024-06-09T22:20:57.597173+0000","last_fullsized":"2024-06-09T22:20:57.597173+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:33.075144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694959+0000","last_change":"2024-06-09T22:17:26.475111+0000","last_active":"2024-06-09T22:20:57.694959+0000","last_peered":"2024-06-09T22:20:57.694959+0000","last_clean":"2024-06-09T22:20:57.694959+0000","last_became_active":"2024-06-09T22:17:26.474563+0000","last_became_peered":"2024-06-09T22:17:26.474563+0000","last_unstale":"2024-06-09T22:20:57.694959+0000","last_undegraded":"2024-06-09T22:20:57.694959+0000","last_fullsized":"2024-06-09T22:20:57.694959+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:09:08.458761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"100'9","reported_seq":262,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:11.720643+0000","last_change":"2024-06-09T22:16:40.040538+0000","last_active":"2024-06-09T22:21:11.720643+0000","last_peered":"2024-06-09T22:21:11.720643+0000","last_clean":"2024-06-09T22:21:11.720643+0000","last_became_active":"2024-06-09T22:16:40.039896+0000","last_became_peered":"2024-06-09T22:16:40.039896+0000","last_unstale":"2024-06-09T22:21:11.720643+0000","last_undegraded":"2024-06-09T22:21:11.720643+0000","last_fullsized":"2024-06-09T22:21:11.720643+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:00:33.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1817,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":209,"num_read_kb":7,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"99'6","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596912+0000","last_change":"2024-06-09T22:17:26.475568+0000","last_active":"2024-06-09T22:20:57.596912+0000","last_peered":"2024-06-09T22:20:57.596912+0000","last_clean":"2024-06-09T22:20:57.596912+0000","last_became_active":"2024-06-09T22:17:26.475163+0000","last_became_peered":"2024-06-09T22:17:26.475163+0000","last_unstale":"2024-06-09T22:20:57.596912+0000","last_undegraded":"2024-06-09T22:20:57.596912+0000","last_fullsized":"2024-06-09T22:20:57.596912+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:20.865164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595970+0000","last_change":"2024-06-09T22:16:53.338201+0000","last_active":"2024-06-09T22:20:57.595970+0000","last_peered":"2024-06-09T22:20:57.595970+0000","last_clean":"2024-06-09T22:20:57.595970+0000","last_became_active":"2024-06-09T22:16:53.337816+0000","last_became_peered":"2024-06-09T22:16:53.337816+0000","last_unstale":"2024-06-09T22:20:57.595970+0000","last_undegraded":"2024-06-09T22:20:57.595970+0000","last_fullsized":"2024-06-09T22:20:57.595970+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:35:51.861135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"97'17","reported_seq":65,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694467+0000","last_change":"2024-06-09T22:16:40.070770+0000","last_active":"2024-06-09T22:20:57.694467+0000","last_peered":"2024-06-09T22:20:57.694467+0000","last_clean":"2024-06-09T22:20:57.694467+0000","last_became_active":"2024-06-09T22:16:40.070398+0000","last_became_peered":"2024-06-09T22:16:40.070398+0000","last_unstale":"2024-06-09T22:20:57.694467+0000","last_undegraded":"2024-06-09T22:20:57.694467+0000","last_fullsized":"2024-06-09T22:20:57.694467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:04:58.273601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"99'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703349+0000","last_change":"2024-06-09T22:16:43.199246+0000","last_active":"2024-06-09T22:20:57.703349+0000","last_peered":"2024-06-09T22:20:57.703349+0000","last_clean":"2024-06-09T22:20:57.703349+0000","last_became_active":"2024-06-09T22:16:43.198006+0000","last_became_peered":"2024-06-09T22:16:43.198006+0000","last_unstale":"2024-06-09T22:20:57.703349+0000","last_undegraded":"2024-06-09T22:20:57.703349+0000","last_fullsized":"2024-06-09T22:20:57.703349+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:30:55.836628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703251+0000","last_change":"2024-06-09T22:16:53.342217+0000","last_active":"2024-06-09T22:20:57.703251+0000","last_peered":"2024-06-09T22:20:57.703251+0000","last_clean":"2024-06-09T22:20:57.703251+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:20:57.703251+0000","last_undegraded":"2024-06-09T22:20:57.703251+0000","last_fullsized":"2024-06-09T22:20:57.703251+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:25:28.635091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"100'4","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:58.449594+0000","last_change":"2024-06-09T22:16:40.039841+0000","last_active":"2024-06-09T22:20:58.449594+0000","last_peered":"2024-06-09T22:20:58.449594+0000","last_clean":"2024-06-09T22:20:58.449594+0000","last_became_active":"2024-06-09T22:16:40.039416+0000","last_became_peered":"2024-06-09T22:16:40.039416+0000","last_unstale":"2024-06-09T22:20:58.449594+0000","last_undegraded":"2024-06-09T22:20:58.449594+0000","last_fullsized":"2024-06-09T22:20:58.449594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:27:36.213839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"99'5","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710159+0000","last_change":"2024-06-09T22:16:43.214363+0000","last_active":"2024-06-09T22:20:57.710159+0000","last_peered":"2024-06-09T22:20:57.710159+0000","last_clean":"2024-06-09T22:20:57.710159+0000","last_became_active":"2024-06-09T22:16:43.214015+0000","last_became_peered":"2024-06-09T22:16:43.214015+0000","last_unstale":"2024-06-09T22:20:57.710159+0000","last_undegraded":"2024-06-09T22:20:57.710159+0000","last_fullsized":"2024-06-09T22:20:57.710159+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:13:06.924468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694710+0000","last_change":"2024-06-09T22:16:53.342151+0000","last_active":"2024-06-09T22:20:57.694710+0000","last_peered":"2024-06-09T22:20:57.694710+0000","last_clean":"2024-06-09T22:20:57.694710+0000","last_became_active":"2024-06-09T22:16:53.341743+0000","last_became_peered":"2024-06-09T22:16:53.341743+0000","last_unstale":"2024-06-09T22:20:57.694710+0000","last_undegraded":"2024-06-09T22:20:57.694710+0000","last_fullsized":"2024-06-09T22:20:57.694710+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:39:44.793301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"97'1","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692594+0000","last_change":"2024-06-09T22:17:26.472397+0000","last_active":"2024-06-09T22:20:57.692594+0000","last_peered":"2024-06-09T22:20:57.692594+0000","last_clean":"2024-06-09T22:20:57.692594+0000","last_became_active":"2024-06-09T22:17:26.471950+0000","last_became_peered":"2024-06-09T22:17:26.471950+0000","last_unstale":"2024-06-09T22:20:57.692594+0000","last_undegraded":"2024-06-09T22:20:57.692594+0000","last_fullsized":"2024-06-09T22:20:57.692594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:35:57.481510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"99'4","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703072+0000","last_change":"2024-06-09T22:16:43.214976+0000","last_active":"2024-06-09T22:20:57.703072+0000","last_peered":"2024-06-09T22:20:57.703072+0000","last_clean":"2024-06-09T22:20:57.703072+0000","last_became_active":"2024-06-09T22:16:43.214592+0000","last_became_peered":"2024-06-09T22:16:43.214592+0000","last_unstale":"2024-06-09T22:20:57.703072+0000","last_undegraded":"2024-06-09T22:20:57.703072+0000","last_fullsized":"2024-06-09T22:20:57.703072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:16:21.981144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595645+0000","last_change":"2024-06-09T22:16:53.342721+0000","last_active":"2024-06-09T22:20:57.595645+0000","last_peered":"2024-06-09T22:20:57.595645+0000","last_clean":"2024-06-09T22:20:57.595645+0000","last_became_active":"2024-06-09T22:16:53.342342+0000","last_became_peered":"2024-06-09T22:16:53.342342+0000","last_unstale":"2024-06-09T22:20:57.595645+0000","last_undegraded":"2024-06-09T22:20:57.595645+0000","last_fullsized":"2024-06-09T22:20:57.595645+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:22:07.469873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"100'6","reported_seq":258,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:11.719112+0000","last_change":"2024-06-09T22:16:40.041307+0000","last_active":"2024-06-09T22:21:11.719112+0000","last_peered":"2024-06-09T22:21:11.719112+0000","last_clean":"2024-06-09T22:21:11.719112+0000","last_became_active":"2024-06-09T22:16:40.040819+0000","last_became_peered":"2024-06-09T22:16:40.040819+0000","last_unstale":"2024-06-09T22:21:11.719112+0000","last_undegraded":"2024-06-09T22:21:11.719112+0000","last_fullsized":"2024-06-09T22:21:11.719112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:14:29.658293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":978,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":204,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"99'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702925+0000","last_change":"2024-06-09T22:16:43.215719+0000","last_active":"2024-06-09T22:20:57.702925+0000","last_peered":"2024-06-09T22:20:57.702925+0000","last_clean":"2024-06-09T22:20:57.702925+0000","last_became_active":"2024-06-09T22:16:43.215356+0000","last_became_peered":"2024-06-09T22:16:43.215356+0000","last_unstale":"2024-06-09T22:20:57.702925+0000","last_undegraded":"2024-06-09T22:20:57.702925+0000","last_fullsized":"2024-06-09T22:20:57.702925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:15:07.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596028+0000","last_change":"2024-06-09T22:16:53.342948+0000","last_active":"2024-06-09T22:20:57.596028+0000","last_peered":"2024-06-09T22:20:57.596028+0000","last_clean":"2024-06-09T22:20:57.596028+0000","last_became_active":"2024-06-09T22:16:53.342352+0000","last_became_peered":"2024-06-09T22:16:53.342352+0000","last_unstale":"2024-06-09T22:20:57.596028+0000","last_undegraded":"2024-06-09T22:20:57.596028+0000","last_fullsized":"2024-06-09T22:20:57.596028+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:43:27.453789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703458+0000","last_change":"2024-06-09T22:16:43.204897+0000","last_active":"2024-06-09T22:20:57.703458+0000","last_peered":"2024-06-09T22:20:57.703458+0000","last_clean":"2024-06-09T22:20:57.703458+0000","last_became_active":"2024-06-09T22:16:43.204470+0000","last_became_peered":"2024-06-09T22:16:43.204470+0000","last_unstale":"2024-06-09T22:20:57.703458+0000","last_undegraded":"2024-06-09T22:20:57.703458+0000","last_fullsized":"2024-06-09T22:20:57.703458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:56.681392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"99'7","reported_seq":258,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:11.720024+0000","last_change":"2024-06-09T22:16:40.070079+0000","last_active":"2024-06-09T22:21:11.720024+0000","last_peered":"2024-06-09T22:21:11.720024+0000","last_clean":"2024-06-09T22:21:11.720024+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:21:11.720024+0000","last_undegraded":"2024-06-09T22:21:11.720024+0000","last_fullsized":"2024-06-09T22:21:11.720024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:00:45.132139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":203,"num_read_kb":203,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"97'1","reported_seq":33,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596870+0000","last_change":"2024-06-09T22:16:53.338891+0000","last_active":"2024-06-09T22:20:57.596870+0000","last_peered":"2024-06-09T22:20:57.596870+0000","last_clean":"2024-06-09T22:20:57.596870+0000","last_became_active":"2024-06-09T22:16:53.338526+0000","last_became_peered":"2024-06-09T22:16:53.338526+0000","last_unstale":"2024-06-09T22:20:57.596870+0000","last_undegraded":"2024-06-09T22:20:57.596870+0000","last_fullsized":"2024-06-09T22:20:57.596870+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:34:37.004407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695269+0000","last_change":"2024-06-09T22:16:43.199234+0000","last_active":"2024-06-09T22:20:57.695269+0000","last_peered":"2024-06-09T22:20:57.695269+0000","last_clean":"2024-06-09T22:20:57.695269+0000","last_became_active":"2024-06-09T22:16:43.192012+0000","last_became_peered":"2024-06-09T22:16:43.192012+0000","last_unstale":"2024-06-09T22:20:57.695269+0000","last_undegraded":"2024-06-09T22:20:57.695269+0000","last_fullsized":"2024-06-09T22:20:57.695269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:17.429236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"97'6","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703645+0000","last_change":"2024-06-09T22:16:40.041970+0000","last_active":"2024-06-09T22:20:57.703645+0000","last_peered":"2024-06-09T22:20:57.703645+0000","last_clean":"2024-06-09T22:20:57.703645+0000","last_became_active":"2024-06-09T22:16:40.040371+0000","last_became_peered":"2024-06-09T22:16:40.040371+0000","last_unstale":"2024-06-09T22:20:57.703645+0000","last_undegraded":"2024-06-09T22:20:57.703645+0000","last_fullsized":"2024-06-09T22:20:57.703645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:52:31.210423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"100'148","reported_seq":425,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:59.707074+0000","last_change":"2024-06-09T22:15:15.367397+0000","last_active":"2024-06-09T22:20:59.707074+0000","last_peered":"2024-06-09T22:20:59.707074+0000","last_clean":"2024-06-09T22:20:59.707074+0000","last_became_active":"2024-06-09T22:15:15.366974+0000","last_became_peered":"2024-06-09T22:15:15.366974+0000","last_unstale":"2024-06-09T22:20:59.707074+0000","last_undegraded":"2024-06-09T22:20:59.707074+0000","last_fullsized":"2024-06-09T22:20:59.707074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_clean_scrub_stamp":"2024-06-09T22:11:49.886056+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:24.018031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.704010+0000","last_change":"2024-06-09T22:16:53.342249+0000","last_active":"2024-06-09T22:20:57.704010+0000","last_peered":"2024-06-09T22:20:57.704010+0000","last_clean":"2024-06-09T22:20:57.704010+0000","last_became_active":"2024-06-09T22:16:53.341779+0000","last_became_peered":"2024-06-09T22:16:53.341779+0000","last_unstale":"2024-06-09T22:20:57.704010+0000","last_undegraded":"2024-06-09T22:20:57.704010+0000","last_fullsized":"2024-06-09T22:20:57.704010+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:21:21.101578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"99'5","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595425+0000","last_change":"2024-06-09T22:16:43.200778+0000","last_active":"2024-06-09T22:20:57.595425+0000","last_peered":"2024-06-09T22:20:57.595425+0000","last_clean":"2024-06-09T22:20:57.595425+0000","last_became_active":"2024-06-09T22:16:43.200490+0000","last_became_peered":"2024-06-09T22:16:43.200490+0000","last_unstale":"2024-06-09T22:20:57.595425+0000","last_undegraded":"2024-06-09T22:20:57.595425+0000","last_fullsized":"2024-06-09T22:20:57.595425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:20:17.530204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"97'4","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596731+0000","last_change":"2024-06-09T22:16:40.041774+0000","last_active":"2024-06-09T22:20:57.596731+0000","last_peered":"2024-06-09T22:20:57.596731+0000","last_clean":"2024-06-09T22:20:57.596731+0000","last_became_active":"2024-06-09T22:16:40.041085+0000","last_became_peered":"2024-06-09T22:16:40.041085+0000","last_unstale":"2024-06-09T22:20:57.596731+0000","last_undegraded":"2024-06-09T22:20:57.596731+0000","last_fullsized":"2024-06-09T22:20:57.596731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:09:41.140083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710137+0000","last_change":"2024-06-09T22:16:53.336895+0000","last_active":"2024-06-09T22:20:57.710137+0000","last_peered":"2024-06-09T22:20:57.710137+0000","last_clean":"2024-06-09T22:20:57.710137+0000","last_became_active":"2024-06-09T22:16:53.336497+0000","last_became_peered":"2024-06-09T22:16:53.336497+0000","last_unstale":"2024-06-09T22:20:57.710137+0000","last_undegraded":"2024-06-09T22:20:57.710137+0000","last_fullsized":"2024-06-09T22:20:57.710137+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:29:52.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"97'4","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695101+0000","last_change":"2024-06-09T22:16:40.070909+0000","last_active":"2024-06-09T22:20:57.695101+0000","last_peered":"2024-06-09T22:20:57.695101+0000","last_clean":"2024-06-09T22:20:57.695101+0000","last_became_active":"2024-06-09T22:16:40.070556+0000","last_became_peered":"2024-06-09T22:16:40.070556+0000","last_unstale":"2024-06-09T22:20:57.695101+0000","last_undegraded":"2024-06-09T22:20:57.695101+0000","last_fullsized":"2024-06-09T22:20:57.695101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:59:17.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"99'9","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595793+0000","last_change":"2024-06-09T22:16:43.205951+0000","last_active":"2024-06-09T22:20:57.595793+0000","last_peered":"2024-06-09T22:20:57.595793+0000","last_clean":"2024-06-09T22:20:57.595793+0000","last_became_active":"2024-06-09T22:16:43.205594+0000","last_became_peered":"2024-06-09T22:16:43.205594+0000","last_unstale":"2024-06-09T22:20:57.595793+0000","last_undegraded":"2024-06-09T22:20:57.595793+0000","last_fullsized":"2024-06-09T22:20:57.595793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:17.956406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694724+0000","last_change":"2024-06-09T22:16:53.343292+0000","last_active":"2024-06-09T22:20:57.694724+0000","last_peered":"2024-06-09T22:20:57.694724+0000","last_clean":"2024-06-09T22:20:57.694724+0000","last_became_active":"2024-06-09T22:16:53.342900+0000","last_became_peered":"2024-06-09T22:16:53.342900+0000","last_unstale":"2024-06-09T22:20:57.694724+0000","last_undegraded":"2024-06-09T22:20:57.694724+0000","last_fullsized":"2024-06-09T22:20:57.694724+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:08:07.328250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"100'105","reported_seq":243,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:13.158591+0000","last_change":"2024-06-09T22:16:40.039417+0000","last_active":"2024-06-09T22:21:13.158591+0000","last_peered":"2024-06-09T22:21:13.158591+0000","last_clean":"2024-06-09T22:21:13.158591+0000","last_became_active":"2024-06-09T22:16:40.039099+0000","last_became_peered":"2024-06-09T22:16:40.039099+0000","last_unstale":"2024-06-09T22:21:13.158591+0000","last_undegraded":"2024-06-09T22:21:13.158591+0000","last_fullsized":"2024-06-09T22:21:13.158591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:22:59.284442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":191,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"99'5","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696372+0000","last_change":"2024-06-09T22:16:43.201285+0000","last_active":"2024-06-09T22:20:57.696372+0000","last_peered":"2024-06-09T22:20:57.696372+0000","last_clean":"2024-06-09T22:20:57.696372+0000","last_became_active":"2024-06-09T22:16:43.200993+0000","last_became_peered":"2024-06-09T22:16:43.200993+0000","last_unstale":"2024-06-09T22:20:57.696372+0000","last_undegraded":"2024-06-09T22:20:57.696372+0000","last_fullsized":"2024-06-09T22:20:57.696372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:15:26.491588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596798+0000","last_change":"2024-06-09T22:16:53.334088+0000","last_active":"2024-06-09T22:20:57.596798+0000","last_peered":"2024-06-09T22:20:57.596798+0000","last_clean":"2024-06-09T22:20:57.596798+0000","last_became_active":"2024-06-09T22:16:53.333597+0000","last_became_peered":"2024-06-09T22:16:53.333597+0000","last_unstale":"2024-06-09T22:20:57.596798+0000","last_undegraded":"2024-06-09T22:20:57.596798+0000","last_fullsized":"2024-06-09T22:20:57.596798+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:50:22.041816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"97'7","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703700+0000","last_change":"2024-06-09T22:16:40.364982+0000","last_active":"2024-06-09T22:20:57.703700+0000","last_peered":"2024-06-09T22:20:57.703700+0000","last_clean":"2024-06-09T22:20:57.703700+0000","last_became_active":"2024-06-09T22:16:40.364585+0000","last_became_peered":"2024-06-09T22:16:40.364585+0000","last_unstale":"2024-06-09T22:20:57.703700+0000","last_undegraded":"2024-06-09T22:20:57.703700+0000","last_fullsized":"2024-06-09T22:20:57.703700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:28.777950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"99'7","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711181+0000","last_change":"2024-06-09T22:17:26.474491+0000","last_active":"2024-06-09T22:20:57.711181+0000","last_peered":"2024-06-09T22:20:57.711181+0000","last_clean":"2024-06-09T22:20:57.711181+0000","last_became_active":"2024-06-09T22:17:26.473975+0000","last_became_peered":"2024-06-09T22:17:26.473975+0000","last_unstale":"2024-06-09T22:20:57.711181+0000","last_undegraded":"2024-06-09T22:20:57.711181+0000","last_fullsized":"2024-06-09T22:20:57.711181+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:13:46.998595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703384+0000","last_change":"2024-06-09T22:16:53.342273+0000","last_active":"2024-06-09T22:20:57.703384+0000","last_peered":"2024-06-09T22:20:57.703384+0000","last_clean":"2024-06-09T22:20:57.703384+0000","last_became_active":"2024-06-09T22:16:53.341827+0000","last_became_peered":"2024-06-09T22:16:53.341827+0000","last_unstale":"2024-06-09T22:20:57.703384+0000","last_undegraded":"2024-06-09T22:20:57.703384+0000","last_fullsized":"2024-06-09T22:20:57.703384+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:23:26.099737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"97'2","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710040+0000","last_change":"2024-06-09T22:16:40.070498+0000","last_active":"2024-06-09T22:20:57.710040+0000","last_peered":"2024-06-09T22:20:57.710040+0000","last_clean":"2024-06-09T22:20:57.710040+0000","last_became_active":"2024-06-09T22:16:40.070144+0000","last_became_peered":"2024-06-09T22:16:40.070144+0000","last_unstale":"2024-06-09T22:20:57.710040+0000","last_undegraded":"2024-06-09T22:20:57.710040+0000","last_fullsized":"2024-06-09T22:20:57.710040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:52:22.214832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"99'2","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694528+0000","last_change":"2024-06-09T22:16:43.210448+0000","last_active":"2024-06-09T22:20:57.694528+0000","last_peered":"2024-06-09T22:20:57.694528+0000","last_clean":"2024-06-09T22:20:57.694528+0000","last_became_active":"2024-06-09T22:16:43.209980+0000","last_became_peered":"2024-06-09T22:16:43.209980+0000","last_unstale":"2024-06-09T22:20:57.694528+0000","last_undegraded":"2024-06-09T22:20:57.694528+0000","last_fullsized":"2024-06-09T22:20:57.694528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:58:15.691207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692864+0000","last_change":"2024-06-09T22:16:53.340644+0000","last_active":"2024-06-09T22:20:57.692864+0000","last_peered":"2024-06-09T22:20:57.692864+0000","last_clean":"2024-06-09T22:20:57.692864+0000","last_became_active":"2024-06-09T22:16:53.339771+0000","last_became_peered":"2024-06-09T22:16:53.339771+0000","last_unstale":"2024-06-09T22:20:57.692864+0000","last_undegraded":"2024-06-09T22:20:57.692864+0000","last_fullsized":"2024-06-09T22:20:57.692864+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:45:23.281387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"97'7","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694859+0000","last_change":"2024-06-09T22:16:40.040461+0000","last_active":"2024-06-09T22:20:57.694859+0000","last_peered":"2024-06-09T22:20:57.694859+0000","last_clean":"2024-06-09T22:20:57.694859+0000","last_became_active":"2024-06-09T22:16:40.040168+0000","last_became_peered":"2024-06-09T22:16:40.040168+0000","last_unstale":"2024-06-09T22:20:57.694859+0000","last_undegraded":"2024-06-09T22:20:57.694859+0000","last_fullsized":"2024-06-09T22:20:57.694859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:28.936631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709959+0000","last_change":"2024-06-09T22:16:43.208779+0000","last_active":"2024-06-09T22:20:57.709959+0000","last_peered":"2024-06-09T22:20:57.709959+0000","last_clean":"2024-06-09T22:20:57.709959+0000","last_became_active":"2024-06-09T22:16:43.208136+0000","last_became_peered":"2024-06-09T22:16:43.208136+0000","last_unstale":"2024-06-09T22:20:57.709959+0000","last_undegraded":"2024-06-09T22:20:57.709959+0000","last_fullsized":"2024-06-09T22:20:57.709959+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:22:39.721069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"97'1","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703384+0000","last_change":"2024-06-09T22:16:53.331241+0000","last_active":"2024-06-09T22:20:57.703384+0000","last_peered":"2024-06-09T22:20:57.703384+0000","last_clean":"2024-06-09T22:20:57.703384+0000","last_became_active":"2024-06-09T22:16:53.329589+0000","last_became_peered":"2024-06-09T22:16:53.329589+0000","last_unstale":"2024-06-09T22:20:57.703384+0000","last_undegraded":"2024-06-09T22:20:57.703384+0000","last_fullsized":"2024-06-09T22:20:57.703384+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:56:21.319711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"100'4","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:58.422275+0000","last_change":"2024-06-09T22:16:40.042491+0000","last_active":"2024-06-09T22:20:58.422275+0000","last_peered":"2024-06-09T22:20:58.422275+0000","last_clean":"2024-06-09T22:20:58.422275+0000","last_became_active":"2024-06-09T22:16:40.041846+0000","last_became_peered":"2024-06-09T22:16:40.041846+0000","last_unstale":"2024-06-09T22:20:58.422275+0000","last_undegraded":"2024-06-09T22:20:58.422275+0000","last_fullsized":"2024-06-09T22:20:58.422275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:59.144003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695873+0000","last_change":"2024-06-09T22:16:43.209195+0000","last_active":"2024-06-09T22:20:57.695873+0000","last_peered":"2024-06-09T22:20:57.695873+0000","last_clean":"2024-06-09T22:20:57.695873+0000","last_became_active":"2024-06-09T22:16:43.208797+0000","last_became_peered":"2024-06-09T22:16:43.208797+0000","last_unstale":"2024-06-09T22:20:57.695873+0000","last_undegraded":"2024-06-09T22:20:57.695873+0000","last_fullsized":"2024-06-09T22:20:57.695873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:05:31.689900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703290+0000","last_change":"2024-06-09T22:16:53.342173+0000","last_active":"2024-06-09T22:20:57.703290+0000","last_peered":"2024-06-09T22:20:57.703290+0000","last_clean":"2024-06-09T22:20:57.703290+0000","last_became_active":"2024-06-09T22:16:53.341712+0000","last_became_peered":"2024-06-09T22:16:53.341712+0000","last_unstale":"2024-06-09T22:20:57.703290+0000","last_undegraded":"2024-06-09T22:20:57.703290+0000","last_fullsized":"2024-06-09T22:20:57.703290+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:53:12.574764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"97'4","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694627+0000","last_change":"2024-06-09T22:16:40.044238+0000","last_active":"2024-06-09T22:20:57.694627+0000","last_peered":"2024-06-09T22:20:57.694627+0000","last_clean":"2024-06-09T22:20:57.694627+0000","last_became_active":"2024-06-09T22:16:40.043857+0000","last_became_peered":"2024-06-09T22:16:40.043857+0000","last_unstale":"2024-06-09T22:20:57.694627+0000","last_undegraded":"2024-06-09T22:20:57.694627+0000","last_fullsized":"2024-06-09T22:20:57.694627+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:44:45.743307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"99'5","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597096+0000","last_change":"2024-06-09T22:16:43.202363+0000","last_active":"2024-06-09T22:20:57.597096+0000","last_peered":"2024-06-09T22:20:57.597096+0000","last_clean":"2024-06-09T22:20:57.597096+0000","last_became_active":"2024-06-09T22:16:43.202029+0000","last_became_peered":"2024-06-09T22:16:43.202029+0000","last_unstale":"2024-06-09T22:20:57.597096+0000","last_undegraded":"2024-06-09T22:20:57.597096+0000","last_fullsized":"2024-06-09T22:20:57.597096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:11:19.513055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710013+0000","last_change":"2024-06-09T22:16:53.333837+0000","last_active":"2024-06-09T22:20:57.710013+0000","last_peered":"2024-06-09T22:20:57.710013+0000","last_clean":"2024-06-09T22:20:57.710013+0000","last_became_active":"2024-06-09T22:16:53.333362+0000","last_became_peered":"2024-06-09T22:16:53.333362+0000","last_unstale":"2024-06-09T22:20:57.710013+0000","last_undegraded":"2024-06-09T22:20:57.710013+0000","last_fullsized":"2024-06-09T22:20:57.710013+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:47:04.463040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"100'8","reported_seq":261,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:12.722444+0000","last_change":"2024-06-09T22:16:40.364834+0000","last_active":"2024-06-09T22:21:12.722444+0000","last_peered":"2024-06-09T22:21:12.722444+0000","last_clean":"2024-06-09T22:21:12.722444+0000","last_became_active":"2024-06-09T22:16:40.364433+0000","last_became_peered":"2024-06-09T22:16:40.364433+0000","last_unstale":"2024-06-09T22:21:12.722444+0000","last_undegraded":"2024-06-09T22:21:12.722444+0000","last_fullsized":"2024-06-09T22:21:12.722444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:43:01.967911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2091,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":207,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"99'7","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597232+0000","last_change":"2024-06-09T22:16:43.201183+0000","last_active":"2024-06-09T22:20:57.597232+0000","last_peered":"2024-06-09T22:20:57.597232+0000","last_clean":"2024-06-09T22:20:57.597232+0000","last_became_active":"2024-06-09T22:16:43.200774+0000","last_became_peered":"2024-06-09T22:16:43.200774+0000","last_unstale":"2024-06-09T22:20:57.597232+0000","last_undegraded":"2024-06-09T22:20:57.597232+0000","last_fullsized":"2024-06-09T22:20:57.597232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:07:55.772066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710583+0000","last_change":"2024-06-09T22:18:26.699190+0000","last_active":"2024-06-09T22:20:57.710583+0000","last_peered":"2024-06-09T22:20:57.710583+0000","last_clean":"2024-06-09T22:20:57.710583+0000","last_became_active":"2024-06-09T22:18:26.698614+0000","last_became_peered":"2024-06-09T22:18:26.698614+0000","last_unstale":"2024-06-09T22:20:57.710583+0000","last_undegraded":"2024-06-09T22:20:57.710583+0000","last_fullsized":"2024-06-09T22:20:57.710583+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:45.343605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,1,4],"acting":[0,10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"100'67","reported_seq":145,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:12.818366+0000","last_change":"2024-06-09T22:16:40.040582+0000","last_active":"2024-06-09T22:21:12.818366+0000","last_peered":"2024-06-09T22:21:12.818366+0000","last_clean":"2024-06-09T22:21:12.818366+0000","last_became_active":"2024-06-09T22:16:40.040152+0000","last_became_peered":"2024-06-09T22:16:40.040152+0000","last_unstale":"2024-06-09T22:21:12.818366+0000","last_undegraded":"2024-06-09T22:21:12.818366+0000","last_fullsized":"2024-06-09T22:21:12.818366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:50:00.878096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"99'6","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596961+0000","last_change":"2024-06-09T22:16:43.206211+0000","last_active":"2024-06-09T22:20:57.596961+0000","last_peered":"2024-06-09T22:20:57.596961+0000","last_clean":"2024-06-09T22:20:57.596961+0000","last_became_active":"2024-06-09T22:16:43.205814+0000","last_became_peered":"2024-06-09T22:16:43.205814+0000","last_unstale":"2024-06-09T22:20:57.596961+0000","last_undegraded":"2024-06-09T22:20:57.596961+0000","last_fullsized":"2024-06-09T22:20:57.596961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:20:19.058475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703101+0000","last_change":"2024-06-09T22:16:53.337695+0000","last_active":"2024-06-09T22:20:57.703101+0000","last_peered":"2024-06-09T22:20:57.703101+0000","last_clean":"2024-06-09T22:20:57.703101+0000","last_became_active":"2024-06-09T22:16:53.334872+0000","last_became_peered":"2024-06-09T22:16:53.334872+0000","last_unstale":"2024-06-09T22:20:57.703101+0000","last_undegraded":"2024-06-09T22:20:57.703101+0000","last_fullsized":"2024-06-09T22:20:57.703101+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:50:36.617736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"97'5","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596104+0000","last_change":"2024-06-09T22:16:40.036603+0000","last_active":"2024-06-09T22:20:57.596104+0000","last_peered":"2024-06-09T22:20:57.596104+0000","last_clean":"2024-06-09T22:20:57.596104+0000","last_became_active":"2024-06-09T22:16:40.036274+0000","last_became_peered":"2024-06-09T22:16:40.036274+0000","last_unstale":"2024-06-09T22:20:57.596104+0000","last_undegraded":"2024-06-09T22:20:57.596104+0000","last_fullsized":"2024-06-09T22:20:57.596104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:31.652897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"99'9","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596989+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:20:57.596989+0000","last_peered":"2024-06-09T22:20:57.596989+0000","last_clean":"2024-06-09T22:20:57.596989+0000","last_became_active":"2024-06-09T22:16:43.204197+0000","last_became_peered":"2024-06-09T22:16:43.204197+0000","last_unstale":"2024-06-09T22:20:57.596989+0000","last_undegraded":"2024-06-09T22:20:57.596989+0000","last_fullsized":"2024-06-09T22:20:57.596989+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:38:49.850120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710013+0000","last_change":"2024-06-09T22:16:43.208477+0000","last_active":"2024-06-09T22:20:57.710013+0000","last_peered":"2024-06-09T22:20:57.710013+0000","last_clean":"2024-06-09T22:20:57.710013+0000","last_became_active":"2024-06-09T22:16:43.206018+0000","last_became_peered":"2024-06-09T22:16:43.206018+0000","last_unstale":"2024-06-09T22:20:57.710013+0000","last_undegraded":"2024-06-09T22:20:57.710013+0000","last_fullsized":"2024-06-09T22:20:57.710013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:19:49.870825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"99'12","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597095+0000","last_change":"2024-06-09T22:16:43.206809+0000","last_active":"2024-06-09T22:20:57.597095+0000","last_peered":"2024-06-09T22:20:57.597095+0000","last_clean":"2024-06-09T22:20:57.597095+0000","last_became_active":"2024-06-09T22:16:43.206175+0000","last_became_peered":"2024-06-09T22:16:43.206175+0000","last_unstale":"2024-06-09T22:20:57.597095+0000","last_undegraded":"2024-06-09T22:20:57.597095+0000","last_fullsized":"2024-06-09T22:20:57.597095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:39:22.502266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596879+0000","last_change":"2024-06-09T22:16:43.202163+0000","last_active":"2024-06-09T22:20:57.596879+0000","last_peered":"2024-06-09T22:20:57.596879+0000","last_clean":"2024-06-09T22:20:57.596879+0000","last_became_active":"2024-06-09T22:16:43.201839+0000","last_became_peered":"2024-06-09T22:16:43.201839+0000","last_unstale":"2024-06-09T22:20:57.596879+0000","last_undegraded":"2024-06-09T22:20:57.596879+0000","last_fullsized":"2024-06-09T22:20:57.596879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:47:46.723289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710813+0000","last_change":"2024-06-09T22:16:43.202734+0000","last_active":"2024-06-09T22:20:57.710813+0000","last_peered":"2024-06-09T22:20:57.710813+0000","last_clean":"2024-06-09T22:20:57.710813+0000","last_became_active":"2024-06-09T22:16:43.201638+0000","last_became_peered":"2024-06-09T22:16:43.201638+0000","last_unstale":"2024-06-09T22:20:57.710813+0000","last_undegraded":"2024-06-09T22:20:57.710813+0000","last_fullsized":"2024-06-09T22:20:57.710813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:16.360985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"99'4","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595768+0000","last_change":"2024-06-09T22:16:43.199113+0000","last_active":"2024-06-09T22:20:57.595768+0000","last_peered":"2024-06-09T22:20:57.595768+0000","last_clean":"2024-06-09T22:20:57.595768+0000","last_became_active":"2024-06-09T22:16:43.198804+0000","last_became_peered":"2024-06-09T22:16:43.198804+0000","last_unstale":"2024-06-09T22:20:57.595768+0000","last_undegraded":"2024-06-09T22:20:57.595768+0000","last_fullsized":"2024-06-09T22:20:57.595768+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:18:08.999477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"99'6","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710407+0000","last_change":"2024-06-09T22:16:43.204445+0000","last_active":"2024-06-09T22:20:57.710407+0000","last_peered":"2024-06-09T22:20:57.710407+0000","last_clean":"2024-06-09T22:20:57.710407+0000","last_became_active":"2024-06-09T22:16:43.203081+0000","last_became_peered":"2024-06-09T22:16:43.203081+0000","last_unstale":"2024-06-09T22:20:57.710407+0000","last_undegraded":"2024-06-09T22:20:57.710407+0000","last_fullsized":"2024-06-09T22:20:57.710407+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:39.303813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"99'15","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597639+0000","last_change":"2024-06-09T22:16:43.211402+0000","last_active":"2024-06-09T22:20:57.597639+0000","last_peered":"2024-06-09T22:20:57.597639+0000","last_clean":"2024-06-09T22:20:57.597639+0000","last_became_active":"2024-06-09T22:16:43.210871+0000","last_became_peered":"2024-06-09T22:16:43.210871+0000","last_unstale":"2024-06-09T22:20:57.597639+0000","last_undegraded":"2024-06-09T22:20:57.597639+0000","last_fullsized":"2024-06-09T22:20:57.597639+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:07:21.467617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"99'2","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696286+0000","last_change":"2024-06-09T22:16:43.201159+0000","last_active":"2024-06-09T22:20:57.696286+0000","last_peered":"2024-06-09T22:20:57.696286+0000","last_clean":"2024-06-09T22:20:57.696286+0000","last_became_active":"2024-06-09T22:16:43.200850+0000","last_became_peered":"2024-06-09T22:16:43.200850+0000","last_unstale":"2024-06-09T22:20:57.696286+0000","last_undegraded":"2024-06-09T22:20:57.696286+0000","last_fullsized":"2024-06-09T22:20:57.696286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:20.515309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"99'6","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703351+0000","last_change":"2024-06-09T22:16:43.196830+0000","last_active":"2024-06-09T22:20:57.703351+0000","last_peered":"2024-06-09T22:20:57.703351+0000","last_clean":"2024-06-09T22:20:57.703351+0000","last_became_active":"2024-06-09T22:16:43.195682+0000","last_became_peered":"2024-06-09T22:16:43.195682+0000","last_unstale":"2024-06-09T22:20:57.703351+0000","last_undegraded":"2024-06-09T22:20:57.703351+0000","last_fullsized":"2024-06-09T22:20:57.703351+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:34:18.770235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694976+0000","last_change":"2024-06-09T22:16:43.201959+0000","last_active":"2024-06-09T22:20:57.694976+0000","last_peered":"2024-06-09T22:20:57.694976+0000","last_clean":"2024-06-09T22:20:57.694976+0000","last_became_active":"2024-06-09T22:16:43.201669+0000","last_became_peered":"2024-06-09T22:16:43.201669+0000","last_unstale":"2024-06-09T22:20:57.694976+0000","last_undegraded":"2024-06-09T22:20:57.694976+0000","last_fullsized":"2024-06-09T22:20:57.694976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:48:57.348041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"99'10","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710243+0000","last_change":"2024-06-09T22:16:43.201823+0000","last_active":"2024-06-09T22:20:57.710243+0000","last_peered":"2024-06-09T22:20:57.710243+0000","last_clean":"2024-06-09T22:20:57.710243+0000","last_became_active":"2024-06-09T22:16:43.201434+0000","last_became_peered":"2024-06-09T22:16:43.201434+0000","last_unstale":"2024-06-09T22:20:57.710243+0000","last_undegraded":"2024-06-09T22:20:57.710243+0000","last_fullsized":"2024-06-09T22:20:57.710243+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:56:13.540895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"99'11","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595673+0000","last_change":"2024-06-09T22:16:43.197734+0000","last_active":"2024-06-09T22:20:57.595673+0000","last_peered":"2024-06-09T22:20:57.595673+0000","last_clean":"2024-06-09T22:20:57.595673+0000","last_became_active":"2024-06-09T22:16:43.197442+0000","last_became_peered":"2024-06-09T22:16:43.197442+0000","last_unstale":"2024-06-09T22:20:57.595673+0000","last_undegraded":"2024-06-09T22:20:57.595673+0000","last_fullsized":"2024-06-09T22:20:57.595673+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:35:40.720718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710386+0000","last_change":"2024-06-09T22:16:43.210375+0000","last_active":"2024-06-09T22:20:57.710386+0000","last_peered":"2024-06-09T22:20:57.710386+0000","last_clean":"2024-06-09T22:20:57.710386+0000","last_became_active":"2024-06-09T22:16:43.209647+0000","last_became_peered":"2024-06-09T22:16:43.209647+0000","last_unstale":"2024-06-09T22:20:57.710386+0000","last_undegraded":"2024-06-09T22:20:57.710386+0000","last_fullsized":"2024-06-09T22:20:57.710386+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:11:57.593066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107643,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":809307,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2421,"num_read_kb":1444,"num_write":524,"num_write_kb":2408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":823296,"data_stored":2427921,"data_compressed":103473,"data_compressed_allocated":135168,"data_compressed_original":2101248,"omap_allocated":0,"internal_metadata":0},"log_size":401,"ondisk_log_size":401,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383750,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":581230,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645387,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32476,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712676,"statfs":{"total":95995035648,"available":95961780224,"internally_reserved":0,"allocated":1323008,"data_stored":1574104,"data_compressed":86342,"data_compressed_allocated":114688,"data_compressed_original":995328,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808912,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":977483,"data_compressed":54548,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070549,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1159168,"data_stored":973765,"data_compressed":54548,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332186,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32336,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712816,"statfs":{"total":95995035648,"available":95961923584,"internally_reserved":0,"allocated":1187840,"data_stored":1179585,"data_compressed":71140,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495712,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1114112,"data_stored":576065,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36504,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708648,"statfs":{"total":95995035648,"available":95957655552,"internally_reserved":0,"allocated":1196032,"data_stored":654681,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738493,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708700,"statfs":{"total":95995035648,"available":95957708800,"internally_reserved":0,"allocated":1138688,"data_stored":578052,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411443,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1208320,"data_stored":656675,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182510,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28192,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716960,"statfs":{"total":95995035648,"available":95966167040,"internally_reserved":0,"allocated":1196032,"data_stored":1296831,"data_compressed":69750,"data_compressed_allocated":94208,"data_compressed_original":798720,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920874,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32276,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712876,"statfs":{"total":95995035648,"available":95961985024,"internally_reserved":0,"allocated":1122304,"data_stored":580978,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757349,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36556,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708596,"statfs":{"total":95995035648,"available":95957602304,"internally_reserved":0,"allocated":1241088,"data_stored":1501780,"data_compressed":86342,"data_compressed_allocated":114688,"data_compressed_original":995328,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":79728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":81722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":721878,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":332363,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":408783,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-09T22:21:17.474 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-09T22:21:17.681 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:17 smithi104 ceph-mon[32389]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-06-09T22:21:17.681 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:17 smithi104 ceph-mon[32389]: from='client.15480 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:17.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:17 smithi153 ceph-mon[36599]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-06-09T22:21:17.813 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:17 smithi153 ceph-mon[36599]: from='client.15480 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:17.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:17 smithi194 ceph-mon[36856]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-06-09T22:21:17.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:17 smithi194 ceph-mon[36856]: from='client.15480 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.935+0000 7fca33b31700 1 -- 172.21.15.104:0/3993660200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca2c0746c0 msgr2=0x7fca2c074aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.935+0000 7fca33b31700 1 --2- 172.21.15.104:0/3993660200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca2c0746c0 0x7fca2c074aa0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fca28009990 tx=0x7fca28009ca0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.935+0000 7fca33b31700 1 -- 172.21.15.104:0/3993660200 shutdown_connections 2024-06-09T22:21:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.935+0000 7fca33b31700 1 --2- 172.21.15.104:0/3993660200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca2c0746c0 0x7fca2c074aa0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.935+0000 7fca33b31700 1 --2- 172.21.15.104:0/3993660200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca2c113660 0x7fca2c115a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.935+0000 7fca33b31700 1 --2- 172.21.15.104:0/3993660200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca2c074fe0 0x7fca2c113120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.935+0000 7fca33b31700 1 -- 172.21.15.104:0/3993660200 >> 172.21.15.104:0/3993660200 conn(0x7fca2c0795f0 msgr2=0x7fca2c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.935+0000 7fca33b31700 1 -- 172.21.15.104:0/3993660200 shutdown_connections 2024-06-09T22:21:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.935+0000 7fca33b31700 1 -- 172.21.15.104:0/3993660200 wait complete. 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.935+0000 7fca33b31700 1 Processor -- start 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.935+0000 7fca33b31700 1 -- start start 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca33b31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca2c0746c0 0x7fca2c19f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca33b31700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca2c074fe0 0x7fca2c19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca33b31700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca2c113660 0x7fca2c1a3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca33b31700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c118e10 con 0x7fca2c0746c0 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca33b31700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c118c90 con 0x7fca2c113660 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca33b31700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c118f90 con 0x7fca2c074fe0 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca318cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca2c0746c0 0x7fca2c19f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca318cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca2c0746c0 0x7fca2c19f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55444/0 (socket says 172.21.15.104:55444) 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca318cd700 1 -- 172.21.15.104:0/195886959 learned_addr learned my addr 172.21.15.104:0/195886959 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca310cc700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca2c074fe0 0x7fca2c19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca320ce700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca2c113660 0x7fca2c1a3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca318cd700 1 -- 172.21.15.104:0/195886959 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca2c074fe0 msgr2=0x7fca2c19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:17.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca318cd700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca2c074fe0 0x7fca2c19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca318cd700 1 -- 172.21.15.104:0/195886959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca2c113660 msgr2=0x7fca2c1a3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:17.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca318cd700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca2c113660 0x7fca2c1a3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca318cd700 1 -- 172.21.15.104:0/195886959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2c1a4330 con 0x7fca2c0746c0 2024-06-09T22:21:17.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca318cd700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca2c0746c0 0x7fca2c19f320 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fca280038b0 tx=0x7fca28009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:17.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca22ffd700 1 -- 172.21.15.104:0/195886959 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28027070 con 0x7fca2c0746c0 2024-06-09T22:21:17.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca33b31700 1 -- 172.21.15.104:0/195886959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca2c1a45c0 con 0x7fca2c0746c0 2024-06-09T22:21:17.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca22ffd700 1 -- 172.21.15.104:0/195886959 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca28004050 con 0x7fca2c0746c0 2024-06-09T22:21:17.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.936+0000 7fca22ffd700 1 -- 172.21.15.104:0/195886959 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28022cd0 con 0x7fca2c0746c0 2024-06-09T22:21:17.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.937+0000 7fca33b31700 1 -- 172.21.15.104:0/195886959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca2c1abea0 con 0x7fca2c0746c0 2024-06-09T22:21:17.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.939+0000 7fca22ffd700 1 -- 172.21.15.104:0/195886959 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7fca2802a070 con 0x7fca2c0746c0 2024-06-09T22:21:17.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.940+0000 7fca22ffd700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fca180a2eb0 0x7fca180a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.940+0000 7fca310cc700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fca180a2eb0 0x7fca180a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.940+0000 7fca22ffd700 1 -- 172.21.15.104:0/195886959 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fca280d0870 con 0x7fca2c0746c0 2024-06-09T22:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.940+0000 7fca3e3b4b80 1 Processor -- start 2024-06-09T22:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca310cc700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fca180a2eb0 0x7fca180a5370 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fca2c10ff30 tx=0x7fca1c009190 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca3e3b4b80 1 -- start start 2024-06-09T22:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca3e3b4b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b33f60 0x55ef59b34340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca3e3b4b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b34880 0x55ef59b2e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca3e3b4b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b2e7a0 0x55ef59b2ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca3e3b4b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55ef59a20f60 con 0x55ef59b34880 2024-06-09T22:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca3e3b4b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ef59a7f850 con 0x55ef59b33f60 2024-06-09T22:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca3e3b4b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ef599e7460 con 0x55ef59b2e7a0 2024-06-09T22:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca310cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b34880 0x55ef59b2e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca310cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b34880 0x55ef59b2e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55446/0 (socket says 172.21.15.104:55446) 2024-06-09T22:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca310cc700 1 -- 172.21.15.104:0/1243646329 learned_addr learned my addr 172.21.15.104:0/1243646329 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca320ce700 1 --2- 172.21.15.104:0/1243646329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b2e7a0 0x55ef59b2ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca318cd700 1 --2- 172.21.15.104:0/1243646329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b33f60 0x55ef59b34340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca310cc700 1 -- 172.21.15.104:0/1243646329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b2e7a0 msgr2=0x55ef59b2ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca310cc700 1 --2- 172.21.15.104:0/1243646329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b2e7a0 0x55ef59b2ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca310cc700 1 -- 172.21.15.104:0/1243646329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b33f60 msgr2=0x55ef59b34340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca310cc700 1 --2- 172.21.15.104:0/1243646329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b33f60 0x55ef59b34340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca310cc700 1 -- 172.21.15.104:0/1243646329 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef599ce8d0 con 0x55ef59b34880 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.941+0000 7fca318cd700 1 --2- 172.21.15.104:0/1243646329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b33f60 0x55ef59b34340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.942+0000 7fca310cc700 1 --2- 172.21.15.104:0/1243646329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b34880 0x55ef59b2e260 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009fd0 tx=0x7fca1c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2adeaed757d78011 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.942+0000 7fca17fff700 1 -- 172.21.15.104:0/1243646329 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c01b040 con 0x55ef59b34880 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.942+0000 7fca17fff700 1 -- 172.21.15.104:0/1243646329 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca1c024070 con 0x55ef59b34880 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.942+0000 7fca17fff700 1 -- 172.21.15.104:0/1243646329 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c013740 con 0x55ef59b34880 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.943+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/1243646329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b34880 msgr2=0x55ef59b2e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.943+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/1243646329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b34880 0x55ef59b2e260 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009fd0 tx=0x7fca1c011000 comp rx=0 tx=0).stop 2024-06-09T22:21:17.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.943+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/1243646329 shutdown_connections 2024-06-09T22:21:17.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.943+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/1243646329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b2e7a0 0x55ef59b2ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.943+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/1243646329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b33f60 0x55ef59b34340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.943+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/1243646329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b34880 0x55ef59b2e260 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.943+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/1243646329 >> 172.21.15.104:0/1243646329 conn(0x55ef59a5b080 msgr2=0x55ef59b29220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:17.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.943+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/1243646329 shutdown_connections 2024-06-09T22:21:17.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.943+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/1243646329 wait complete. 2024-06-09T22:21:17.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.943+0000 7fca3e3b4b80 1 -- start start 2024-06-09T22:21:17.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.943+0000 7fca3e3b4b80 1 Processor -- start 2024-06-09T22:21:17.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca3e3b4b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b2e7a0 0x55ef59b3a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca3e3b4b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b33f60 0x55ef59b378d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca3e3b4b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b34880 0x55ef59b37e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca3e3b4b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55ef59a1ffc0 con 0x55ef59b2e7a0 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca3e3b4b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ef59a7b810 con 0x55ef59b34880 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca3e3b4b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ef59b3b260 con 0x55ef59b33f60 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca3e3b4b80 10 client.0 _ll_register_callbacks cb 0x55ef59a155b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca318cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b2e7a0 0x55ef59b3a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca318cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b2e7a0 0x55ef59b3a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55456/0 (socket says 172.21.15.104:55456) 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca318cd700 1 -- 172.21.15.104:0/3360086222 learned_addr learned my addr 172.21.15.104:0/3360086222 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca310cc700 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b33f60 0x55ef59b378d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca320ce700 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b34880 0x55ef59b37e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca318cd700 1 -- 172.21.15.104:0/3360086222 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b33f60 msgr2=0x55ef59b378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca318cd700 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b33f60 0x55ef59b378d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca318cd700 1 -- 172.21.15.104:0/3360086222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b34880 msgr2=0x55ef59b37e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca318cd700 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b34880 0x55ef59b37e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca318cd700 1 -- 172.21.15.104:0/3360086222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef59a45750 con 0x55ef59b2e7a0 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.944+0000 7fca310cc700 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b33f60 0x55ef59b378d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:21:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.945+0000 7fca318cd700 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b2e7a0 0x55ef59b3a5d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fca2809c7f0 tx=0x7fca28022730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd5b15041818d459 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:17.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.945+0000 7fca157fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:17.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.945+0000 7fca157fa700 1 -- 172.21.15.104:0/3360086222 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2809c1c0 con 0x55ef59b2e7a0 2024-06-09T22:21:17.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.945+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3360086222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca1c024070 con 0x55ef59b2e7a0 2024-06-09T22:21:17.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.945+0000 7fca157fa700 1 -- 172.21.15.104:0/3360086222 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fca280094b0 con 0x55ef59b2e7a0 2024-06-09T22:21:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.945+0000 7fca157fa700 1 -- 172.21.15.104:0/3360086222 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fca2809cf60 con 0x55ef59b2e7a0 2024-06-09T22:21:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.945+0000 7fca3e3b4b80 10 client.15492 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.945+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3360086222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ef59b45990 con 0x55ef59b2e7a0 2024-06-09T22:21:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.945+0000 7fca157fa700 1 -- 172.21.15.104:0/3360086222 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca1c024070 con 0x55ef59b2e7a0 2024-06-09T22:21:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.945+0000 7fca157fa700 1 -- 172.21.15.104:0/3360086222 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fca280cddb0 con 0x55ef59b2e7a0 2024-06-09T22:21:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.945+0000 7fca3e3b4b80 10 client.15492 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.945+0000 7fca3e3b4b80 10 client.15492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:21:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.946+0000 7fca3e3b4b80 20 client.15492 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.946+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ef59b47e60 0x55ef59b4a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.946+0000 7fca3e3b4b80 4 client.15492 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.946+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3360086222 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef59a8cd50 con 0x55ef59b47e60 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.946+0000 7fca310cc700 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ef59b47e60 0x55ef59b4a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.946+0000 7fca157fa700 10 client.15492 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.946+0000 7fca310cc700 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ef59b47e60 0x55ef59b4a290 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f52f818a2f0f46c server_cookie=a0a1ac4729e6d37b in_seq=0 out_seq=0 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca157fa700 1 -- 172.21.15.104:0/3360086222 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55ef59a8cd50 con 0x55ef59b47e60 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca157fa700 10 client.15492 handle_command_reply: tid=0 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca3e3b4b80 1 client.15492 shutdown 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3360086222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b2e7a0 msgr2=0x55ef59b3a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b2e7a0 0x55ef59b3a5d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fca2809c7f0 tx=0x7fca28022730 comp rx=0 tx=0).stop 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3360086222 shutdown_connections 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ef59b47e60 0x55ef59b4a290 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b2e7a0 0x55ef59b3a5d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca157fa700 0 client.15492 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b34880 0x55ef59b37e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/3360086222 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b33f60 0x55ef59b378d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3360086222 >> 172.21.15.104:0/3360086222 conn(0x55ef59a5b080 msgr2=0x55ef59b28700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3360086222 shutdown_connections 2024-06-09T22:21:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3360086222 wait complete. 2024-06-09T22:21:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.947+0000 7fca3e3b4b80 20 client.15492 trim_cache size 0 max 16384 2024-06-09T22:21:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.968+0000 7fca3e3b4b80 1 Processor -- start 2024-06-09T22:21:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.968+0000 7fca3e3b4b80 1 -- start start 2024-06-09T22:21:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca3e3b4b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b2e7a0 0x55ef59b2e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca3e3b4b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 0x55ef59b3a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca3e3b4b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b34880 0x55ef59b28e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca3e3b4b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c1b4b50 con 0x55ef59b33f60 2024-06-09T22:21:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca3e3b4b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c1b6690 con 0x55ef59b2e7a0 2024-06-09T22:21:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca3e3b4b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c1b5220 con 0x55ef59b34880 2024-06-09T22:21:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca310cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 0x55ef59b3a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca310cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 0x55ef59b3a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55458/0 (socket says 172.21.15.104:55458) 2024-06-09T22:21:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca310cc700 1 -- 172.21.15.104:0/189332236 learned_addr learned my addr 172.21.15.104:0/189332236 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca318cd700 1 --2- 172.21.15.104:0/189332236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b2e7a0 0x55ef59b2e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca320ce700 1 --2- 172.21.15.104:0/189332236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b34880 0x55ef59b28e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca310cc700 1 -- 172.21.15.104:0/189332236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b34880 msgr2=0x55ef59b28e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca310cc700 1 --2- 172.21.15.104:0/189332236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b34880 0x55ef59b28e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca310cc700 1 -- 172.21.15.104:0/189332236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b2e7a0 msgr2=0x55ef59b2e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca310cc700 1 --2- 172.21.15.104:0/189332236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b2e7a0 0x55ef59b2e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca310cc700 1 -- 172.21.15.104:0/189332236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef59b34680 con 0x55ef59b33f60 2024-06-09T22:21:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.969+0000 7fca318cd700 1 --2- 172.21.15.104:0/189332236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b2e7a0 0x55ef59b2e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:21:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.970+0000 7fca310cc700 1 --2- 172.21.15.104:0/189332236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 0x55ef59b3a110 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00e9d0 tx=0x7fca1c015e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fefafb98234f9b44 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.970+0000 7fca177fe700 1 -- 172.21.15.104:0/189332236 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c035020 con 0x55ef59b33f60 2024-06-09T22:21:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.970+0000 7fca177fe700 1 -- 172.21.15.104:0/189332236 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca1c024040 con 0x55ef59b33f60 2024-06-09T22:21:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.970+0000 7fca177fe700 1 -- 172.21.15.104:0/189332236 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c031c20 con 0x55ef59b33f60 2024-06-09T22:21:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.971+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/189332236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 msgr2=0x55ef59b3a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.971+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/189332236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 0x55ef59b3a110 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00e9d0 tx=0x7fca1c015e40 comp rx=0 tx=0).stop 2024-06-09T22:21:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.971+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/189332236 shutdown_connections 2024-06-09T22:21:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.971+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/189332236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b34880 0x55ef59b28e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.971+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/189332236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b2e7a0 0x55ef59b2e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.971+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/189332236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 0x55ef59b3a110 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.971+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/189332236 >> 172.21.15.104:0/189332236 conn(0x55ef59a5b080 msgr2=0x55ef59b332d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.971+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/189332236 shutdown_connections 2024-06-09T22:21:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.971+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/189332236 wait complete. 2024-06-09T22:21:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.971+0000 7fca3e3b4b80 1 -- start start 2024-06-09T22:21:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.971+0000 7fca3e3b4b80 1 Processor -- start 2024-06-09T22:21:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca3e3b4b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b2e7a0 0x55ef59b40e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca3e3b4b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 0x55ef59b4f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca3e3b4b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b34880 0x55ef59b4fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca3e3b4b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c1b5fc0 con 0x55ef59b33f60 2024-06-09T22:21:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca3e3b4b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c1b58f0 con 0x55ef59b2e7a0 2024-06-09T22:21:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca3e3b4b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c1b2ad0 con 0x55ef59b34880 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca3e3b4b80 10 client.0 _ll_register_callbacks cb 0x55ef59a155b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca310cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 0x55ef59b4f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca310cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 0x55ef59b4f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55464/0 (socket says 172.21.15.104:55464) 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca310cc700 1 -- 172.21.15.104:0/3497462414 learned_addr learned my addr 172.21.15.104:0/3497462414 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca318cd700 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b2e7a0 0x55ef59b40e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca320ce700 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b34880 0x55ef59b4fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca310cc700 1 -- 172.21.15.104:0/3497462414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b34880 msgr2=0x55ef59b4fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca310cc700 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b34880 0x55ef59b4fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca310cc700 1 -- 172.21.15.104:0/3497462414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b2e7a0 msgr2=0x55ef59b40e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca310cc700 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b2e7a0 0x55ef59b40e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca310cc700 1 -- 172.21.15.104:0/3497462414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef59bb1b50 con 0x55ef59b33f60 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.972+0000 7fca318cd700 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b2e7a0 0x55ef59b40e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:21:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.973+0000 7fca310cc700 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 0x55ef59b4f720 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0048a0 tx=0x7fca1c009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254d16c03aed7d0f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.973+0000 7fca157fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.973+0000 7fca157fa700 1 -- 172.21.15.104:0/3497462414 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c03e020 con 0x55ef59b33f60 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.973+0000 7fca157fa700 1 -- 172.21.15.104:0/3497462414 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fca1c031360 con 0x55ef59b33f60 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.973+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3497462414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca1c024040 con 0x55ef59b33f60 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.973+0000 7fca157fa700 1 -- 172.21.15.104:0/3497462414 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fca1c035040 con 0x55ef59b33f60 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.973+0000 7fca3e3b4b80 10 client.15498 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.973+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3497462414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ef59bb2d40 con 0x55ef59b33f60 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.974+0000 7fca157fa700 1 -- 172.21.15.104:0/3497462414 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca1c024040 con 0x55ef59b33f60 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.974+0000 7fca157fa700 1 -- 172.21.15.104:0/3497462414 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fca1c01e030 con 0x55ef59b33f60 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.974+0000 7fca3e3b4b80 10 client.15498 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.974+0000 7fca3e3b4b80 10 client.15498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.974+0000 7fca3e3b4b80 20 client.15498 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.974+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ef59bb3080 0x55ef59bb54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.974+0000 7fca3e3b4b80 4 client.15498 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:21:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.974+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3497462414 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ef59a872d0 con 0x55ef59bb3080 2024-06-09T22:21:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.974+0000 7fca318cd700 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ef59bb3080 0x55ef59bb54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca157fa700 10 client.15498 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca318cd700 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ef59bb3080 0x55ef59bb54a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37c34da30f2f969d server_cookie=8bca30d55f71b926 in_seq=0 out_seq=0 2024-06-09T22:21:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca157fa700 1 -- 172.21.15.104:0/3497462414 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ef59a872d0 con 0x55ef59bb3080 2024-06-09T22:21:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca157fa700 10 client.15498 handle_command_reply: tid=0 2024-06-09T22:21:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca3e3b4b80 1 client.15498 shutdown 2024-06-09T22:21:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3497462414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 msgr2=0x55ef59b4f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 0x55ef59b4f720 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0048a0 tx=0x7fca1c009680 comp rx=0 tx=0).stop 2024-06-09T22:21:17.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3497462414 shutdown_connections 2024-06-09T22:21:17.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ef59bb3080 0x55ef59bb54a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca157fa700 0 client.15498 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:17.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef59b34880 0x55ef59b4fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef59b2e7a0 0x55ef59b40e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca3e3b4b80 1 --2- 172.21.15.104:0/3497462414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef59b33f60 0x55ef59b4f720 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.975+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3497462414 >> 172.21.15.104:0/3497462414 conn(0x55ef59a5b080 msgr2=0x55ef59b370c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:17.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3497462414 shutdown_connections 2024-06-09T22:21:17.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca3e3b4b80 1 -- 172.21.15.104:0/3497462414 wait complete. 2024-06-09T22:21:17.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca3e3b4b80 20 client.15498 trim_cache size 0 max 16384 2024-06-09T22:21:17.980 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:21:17.980 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-09T22:21:17.980 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": { 2024-06-09T22:21:17.980 INFO:teuthology.orchestra.run.smithi104.stdout: "1c8f834a-0287-4773-a482-c93999ce0077": { 2024-06-09T22:21:17.980 INFO:teuthology.orchestra.run.smithi104.stdout: "path": "/", 2024-06-09T22:21:17.980 INFO:teuthology.orchestra.run.smithi104.stdout: "tag": "1c8f834a-0287-4773-a482-c93999ce0077", 2024-06-09T22:21:17.980 INFO:teuthology.orchestra.run.smithi104.stdout: "options": "recursive,force" 2024-06-09T22:21:17.980 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:21:17.980 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:21:17.980 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:21:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca33b31700 1 -- 172.21.15.104:0/195886959 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fca180a2eb0 msgr2=0x7fca180a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca33b31700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fca180a2eb0 0x7fca180a5370 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fca2c10ff30 tx=0x7fca1c009190 comp rx=0 tx=0).stop 2024-06-09T22:21:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca33b31700 1 -- 172.21.15.104:0/195886959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca2c0746c0 msgr2=0x7fca2c19f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca33b31700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca2c0746c0 0x7fca2c19f320 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fca280038b0 tx=0x7fca28009f50 comp rx=0 tx=0).stop 2024-06-09T22:21:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca33b31700 1 -- 172.21.15.104:0/195886959 shutdown_connections 2024-06-09T22:21:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca33b31700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fca180a2eb0 0x7fca180a5370 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca33b31700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca2c0746c0 0x7fca2c19f320 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca33b31700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca2c113660 0x7fca2c1a3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca33b31700 1 --2- 172.21.15.104:0/195886959 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca2c074fe0 0x7fca2c19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca33b31700 1 -- 172.21.15.104:0/195886959 >> 172.21.15.104:0/195886959 conn(0x7fca2c0795f0 msgr2=0x7fca2c112490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.976+0000 7fca33b31700 1 -- 172.21.15.104:0/195886959 shutdown_connections 2024-06-09T22:21:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:17.977+0000 7fca33b31700 1 -- 172.21.15.104:0/195886959 wait complete. 2024-06-09T22:21:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:18.813 INFO:tasks.cephfs.filesystem:scrub status for tag:1c8f834a-0287-4773-a482-c93999ce0077 - {'path': '/', 'tag': '1c8f834a-0287-4773-a482-c93999ce0077', 'options': 'recursive,force'} 2024-06-09T22:21:19.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:19 smithi153 ceph-mon[36599]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-06-09T22:21:19.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:19 smithi104 ceph-mon[32389]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-06-09T22:21:19.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:19 smithi194 ceph-mon[36856]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-06-09T22:21:20.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.595+0000 7fa5dbce6700 1 -- 172.21.15.104:0/2812266864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d4102550 msgr2=0x7fa5d410e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:20.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.595+0000 7fa5dbce6700 1 --2- 172.21.15.104:0/2812266864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d4102550 0x7fa5d410e210 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00b0a0 tx=0x7fa5cc00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:21:20.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.595+0000 7fa5dbce6700 1 -- 172.21.15.104:0/2812266864 shutdown_connections 2024-06-09T22:21:20.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.595+0000 7fa5dbce6700 1 --2- 172.21.15.104:0/2812266864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d4102550 0x7fa5d410e210 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.595+0000 7fa5dbce6700 1 --2- 172.21.15.104:0/2812266864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4106370 0x7fa5d4106750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.595+0000 7fa5dbce6700 1 --2- 172.21.15.104:0/2812266864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5d4101b90 0x7fa5d4102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.595+0000 7fa5dbce6700 1 -- 172.21.15.104:0/2812266864 >> 172.21.15.104:0/2812266864 conn(0x7fa5d4077a00 msgr2=0x7fa5d40fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.595+0000 7fa5dbce6700 1 -- 172.21.15.104:0/2812266864 shutdown_connections 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.596+0000 7fa5dbce6700 1 -- 172.21.15.104:0/2812266864 wait complete. 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.596+0000 7fa5dbce6700 1 Processor -- start 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.596+0000 7fa5dbce6700 1 -- start start 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.596+0000 7fa5dbce6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4101b90 0x7fa5d419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.596+0000 7fa5dbce6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d4102550 0x7fa5d419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.596+0000 7fa5dbce6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5d4106370 0x7fa5d41988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.596+0000 7fa5dbce6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d4111640 con 0x7fa5d4102550 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.596+0000 7fa5dbce6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d41114c0 con 0x7fa5d4101b90 2024-06-09T22:21:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.596+0000 7fa5dbce6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d41117c0 con 0x7fa5d4106370 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.596+0000 7fa5d9281700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d4102550 0x7fa5d419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5d9281700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d4102550 0x7fa5d419ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55486/0 (socket says 172.21.15.104:55486) 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5d9281700 1 -- 172.21.15.104:0/1249617387 learned_addr learned my addr 172.21.15.104:0/1249617387 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5d9a82700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4101b90 0x7fa5d419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5da283700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5d4106370 0x7fa5d41988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5d9281700 1 -- 172.21.15.104:0/1249617387 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5d4106370 msgr2=0x7fa5d41988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5d9281700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5d4106370 0x7fa5d41988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5d9281700 1 -- 172.21.15.104:0/1249617387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4101b90 msgr2=0x7fa5d419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5d9281700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4101b90 0x7fa5d419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5d9281700 1 -- 172.21.15.104:0/1249617387 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d4199030 con 0x7fa5d4102550 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5d9281700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d4102550 0x7fa5d419ec50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400c920 tx=0x7fa5c400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5caffd700 1 -- 172.21.15.104:0/1249617387 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4004a30 con 0x7fa5d4102550 2024-06-09T22:21:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5caffd700 1 -- 172.21.15.104:0/1249617387 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5c4005500 con 0x7fa5d4102550 2024-06-09T22:21:20.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5dbce6700 1 -- 172.21.15.104:0/1249617387 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d4199320 con 0x7fa5d4102550 2024-06-09T22:21:20.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5caffd700 1 -- 172.21.15.104:0/1249617387 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4007d50 con 0x7fa5d4102550 2024-06-09T22:21:20.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.597+0000 7fa5dbce6700 1 -- 172.21.15.104:0/1249617387 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d41a55e0 con 0x7fa5d4102550 2024-06-09T22:21:20.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.598+0000 7fa5dbce6700 1 -- 172.21.15.104:0/1249617387 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d4069180 con 0x7fa5d4102550 2024-06-09T22:21:20.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.599+0000 7fa5caffd700 1 -- 172.21.15.104:0/1249617387 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7fa5c4004d60 con 0x7fa5d4102550 2024-06-09T22:21:20.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.600+0000 7fa5caffd700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa5c00a2eb0 0x7fa5c00a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:20.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.600+0000 7fa5d9a82700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa5c00a2eb0 0x7fa5c00a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:20.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.601+0000 7fa5caffd700 1 -- 172.21.15.104:0/1249617387 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fa5c40cf990 con 0x7fa5d4102550 2024-06-09T22:21:20.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.601+0000 7fa5d9a82700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa5c00a2eb0 0x7fa5c00a5370 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4199a10 tx=0x7fa5d000b410 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:20.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.604+0000 7fa5caffd700 1 -- 172.21.15.104:0/1249617387 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa5c4014030 con 0x7fa5d4102550 2024-06-09T22:21:20.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.796+0000 7fa5dbce6700 1 -- 172.21.15.104:0/1249617387 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa5d4100a80 con 0x7fa5c00a2eb0 2024-06-09T22:21:20.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.805+0000 7fa5caffd700 1 -- 172.21.15.104:0/1249617387 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339896 (secure 0 0 0) 0x7fa5d4100a80 con 0x7fa5c00a2eb0 2024-06-09T22:21:20.809 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:20.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.809+0000 7fa5dbce6700 1 -- 172.21.15.104:0/1249617387 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa5c00a2eb0 msgr2=0x7fa5c00a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:20.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.809+0000 7fa5dbce6700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa5c00a2eb0 0x7fa5c00a5370 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4199a10 tx=0x7fa5d000b410 comp rx=0 tx=0).stop 2024-06-09T22:21:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.809+0000 7fa5dbce6700 1 -- 172.21.15.104:0/1249617387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d4102550 msgr2=0x7fa5d419ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.809+0000 7fa5dbce6700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d4102550 0x7fa5d419ec50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400c920 tx=0x7fa5c400cc30 comp rx=0 tx=0).stop 2024-06-09T22:21:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.809+0000 7fa5dbce6700 1 -- 172.21.15.104:0/1249617387 shutdown_connections 2024-06-09T22:21:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.809+0000 7fa5dbce6700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa5c00a2eb0 0x7fa5c00a5370 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.809+0000 7fa5dbce6700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5d4106370 0x7fa5d41988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.809+0000 7fa5dbce6700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4101b90 0x7fa5d419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.809+0000 7fa5dbce6700 1 --2- 172.21.15.104:0/1249617387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d4102550 0x7fa5d419ec50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.809+0000 7fa5dbce6700 1 -- 172.21.15.104:0/1249617387 >> 172.21.15.104:0/1249617387 conn(0x7fa5d4077a00 msgr2=0x7fa5d40fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.809+0000 7fa5dbce6700 1 -- 172.21.15.104:0/1249617387 shutdown_connections 2024-06-09T22:21:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:20.809+0000 7fa5dbce6700 1 -- 172.21.15.104:0/1249617387 wait complete. 2024-06-09T22:21:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-06-09T22:21:21.652 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-06-09T22:21:19.613102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1399909,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2709,"num_read_kb":1801,"num_write":727,"num_write_kb":5348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1051,"ondisk_log_size":1051,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383836,"kb_used_data":13844,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557988,"statfs":{"total":1151940427776,"available":1151547379712,"internally_reserved":0,"allocated":14176256,"data_stored":11131229,"data_compressed":634224,"data_compressed_allocated":860160,"data_compressed_original":5050368,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":197770,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":406,"num_read_kb":309,"num_write":128,"num_write_kb":917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002671"},"pg_stats":[{"pgid":"3.3f","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710445+0000","last_change":"2024-06-09T22:16:43.212998+0000","last_active":"2024-06-09T22:20:57.710445+0000","last_peered":"2024-06-09T22:20:57.710445+0000","last_clean":"2024-06-09T22:20:57.710445+0000","last_became_active":"2024-06-09T22:16:43.212614+0000","last_became_peered":"2024-06-09T22:16:43.212614+0000","last_unstale":"2024-06-09T22:20:57.710445+0000","last_undegraded":"2024-06-09T22:20:57.710445+0000","last_fullsized":"2024-06-09T22:20:57.710445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:22.728486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"99'4","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711160+0000","last_change":"2024-06-09T22:16:43.208746+0000","last_active":"2024-06-09T22:20:57.711160+0000","last_peered":"2024-06-09T22:20:57.711160+0000","last_clean":"2024-06-09T22:20:57.711160+0000","last_became_active":"2024-06-09T22:16:43.208458+0000","last_became_peered":"2024-06-09T22:16:43.208458+0000","last_unstale":"2024-06-09T22:20:57.711160+0000","last_undegraded":"2024-06-09T22:20:57.711160+0000","last_fullsized":"2024-06-09T22:20:57.711160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:24:35.350449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710878+0000","last_change":"2024-06-09T22:16:43.200770+0000","last_active":"2024-06-09T22:20:57.710878+0000","last_peered":"2024-06-09T22:20:57.710878+0000","last_clean":"2024-06-09T22:20:57.710878+0000","last_became_active":"2024-06-09T22:16:43.200449+0000","last_became_peered":"2024-06-09T22:16:43.200449+0000","last_unstale":"2024-06-09T22:20:57.710878+0000","last_undegraded":"2024-06-09T22:20:57.710878+0000","last_fullsized":"2024-06-09T22:20:57.710878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:51:09.017260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703458+0000","last_change":"2024-06-09T22:16:43.203844+0000","last_active":"2024-06-09T22:20:57.703458+0000","last_peered":"2024-06-09T22:20:57.703458+0000","last_clean":"2024-06-09T22:20:57.703458+0000","last_became_active":"2024-06-09T22:16:43.203431+0000","last_became_peered":"2024-06-09T22:16:43.203431+0000","last_unstale":"2024-06-09T22:20:57.703458+0000","last_undegraded":"2024-06-09T22:20:57.703458+0000","last_fullsized":"2024-06-09T22:20:57.703458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:51:36.913312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"99'4","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694538+0000","last_change":"2024-06-09T22:16:43.207597+0000","last_active":"2024-06-09T22:20:57.694538+0000","last_peered":"2024-06-09T22:20:57.694538+0000","last_clean":"2024-06-09T22:20:57.694538+0000","last_became_active":"2024-06-09T22:16:43.207266+0000","last_became_peered":"2024-06-09T22:16:43.207266+0000","last_unstale":"2024-06-09T22:20:57.694538+0000","last_undegraded":"2024-06-09T22:20:57.694538+0000","last_fullsized":"2024-06-09T22:20:57.694538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:47:05.558403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"99'6","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702988+0000","last_change":"2024-06-09T22:17:26.474121+0000","last_active":"2024-06-09T22:20:57.702988+0000","last_peered":"2024-06-09T22:20:57.702988+0000","last_clean":"2024-06-09T22:20:57.702988+0000","last_became_active":"2024-06-09T22:17:26.473619+0000","last_became_peered":"2024-06-09T22:17:26.473619+0000","last_unstale":"2024-06-09T22:20:57.702988+0000","last_undegraded":"2024-06-09T22:20:57.702988+0000","last_fullsized":"2024-06-09T22:20:57.702988+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:56:42.171235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"99'5","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596592+0000","last_change":"2024-06-09T22:16:43.208815+0000","last_active":"2024-06-09T22:20:57.596592+0000","last_peered":"2024-06-09T22:20:57.596592+0000","last_clean":"2024-06-09T22:20:57.596592+0000","last_became_active":"2024-06-09T22:16:43.208433+0000","last_became_peered":"2024-06-09T22:16:43.208433+0000","last_unstale":"2024-06-09T22:20:57.596592+0000","last_undegraded":"2024-06-09T22:20:57.596592+0000","last_fullsized":"2024-06-09T22:20:57.596592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:36:26.585037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694783+0000","last_change":"2024-06-09T22:16:43.210195+0000","last_active":"2024-06-09T22:20:57.694783+0000","last_peered":"2024-06-09T22:20:57.694783+0000","last_clean":"2024-06-09T22:20:57.694783+0000","last_became_active":"2024-06-09T22:16:43.209892+0000","last_became_peered":"2024-06-09T22:16:43.209892+0000","last_unstale":"2024-06-09T22:20:57.694783+0000","last_undegraded":"2024-06-09T22:20:57.694783+0000","last_fullsized":"2024-06-09T22:20:57.694783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:44.088212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710618+0000","last_change":"2024-06-09T22:16:43.202517+0000","last_active":"2024-06-09T22:20:57.710618+0000","last_peered":"2024-06-09T22:20:57.710618+0000","last_clean":"2024-06-09T22:20:57.710618+0000","last_became_active":"2024-06-09T22:16:43.201796+0000","last_became_peered":"2024-06-09T22:16:43.201796+0000","last_unstale":"2024-06-09T22:20:57.710618+0000","last_undegraded":"2024-06-09T22:20:57.710618+0000","last_fullsized":"2024-06-09T22:20:57.710618+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:30.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"99'16","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703409+0000","last_change":"2024-06-09T22:16:43.212732+0000","last_active":"2024-06-09T22:20:57.703409+0000","last_peered":"2024-06-09T22:20:57.703409+0000","last_clean":"2024-06-09T22:20:57.703409+0000","last_became_active":"2024-06-09T22:16:43.212048+0000","last_became_peered":"2024-06-09T22:16:43.212048+0000","last_unstale":"2024-06-09T22:20:57.703409+0000","last_undegraded":"2024-06-09T22:20:57.703409+0000","last_fullsized":"2024-06-09T22:20:57.703409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:47.383291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597025+0000","last_change":"2024-06-09T22:16:43.203778+0000","last_active":"2024-06-09T22:20:57.597025+0000","last_peered":"2024-06-09T22:20:57.597025+0000","last_clean":"2024-06-09T22:20:57.597025+0000","last_became_active":"2024-06-09T22:16:43.202954+0000","last_became_peered":"2024-06-09T22:16:43.202954+0000","last_unstale":"2024-06-09T22:20:57.597025+0000","last_undegraded":"2024-06-09T22:20:57.597025+0000","last_fullsized":"2024-06-09T22:20:57.597025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:56:53.708645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703526+0000","last_change":"2024-06-09T22:16:43.211425+0000","last_active":"2024-06-09T22:20:57.703526+0000","last_peered":"2024-06-09T22:20:57.703526+0000","last_clean":"2024-06-09T22:20:57.703526+0000","last_became_active":"2024-06-09T22:16:43.211001+0000","last_became_peered":"2024-06-09T22:16:43.211001+0000","last_unstale":"2024-06-09T22:20:57.703526+0000","last_undegraded":"2024-06-09T22:20:57.703526+0000","last_fullsized":"2024-06-09T22:20:57.703526+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:20:59.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"99'10","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595794+0000","last_change":"2024-06-09T22:16:43.208334+0000","last_active":"2024-06-09T22:20:57.595794+0000","last_peered":"2024-06-09T22:20:57.595794+0000","last_clean":"2024-06-09T22:20:57.595794+0000","last_became_active":"2024-06-09T22:16:43.207993+0000","last_became_peered":"2024-06-09T22:16:43.207993+0000","last_unstale":"2024-06-09T22:20:57.595794+0000","last_undegraded":"2024-06-09T22:20:57.595794+0000","last_fullsized":"2024-06-09T22:20:57.595794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:07:05.740748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710461+0000","last_change":"2024-06-09T22:16:43.207211+0000","last_active":"2024-06-09T22:20:57.710461+0000","last_peered":"2024-06-09T22:20:57.710461+0000","last_clean":"2024-06-09T22:20:57.710461+0000","last_became_active":"2024-06-09T22:16:43.206889+0000","last_became_peered":"2024-06-09T22:16:43.206889+0000","last_unstale":"2024-06-09T22:20:57.710461+0000","last_undegraded":"2024-06-09T22:20:57.710461+0000","last_fullsized":"2024-06-09T22:20:57.710461+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:25:47.461030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710404+0000","last_change":"2024-06-09T22:16:43.202202+0000","last_active":"2024-06-09T22:20:57.710404+0000","last_peered":"2024-06-09T22:20:57.710404+0000","last_clean":"2024-06-09T22:20:57.710404+0000","last_became_active":"2024-06-09T22:16:43.201834+0000","last_became_peered":"2024-06-09T22:16:43.201834+0000","last_unstale":"2024-06-09T22:20:57.710404+0000","last_undegraded":"2024-06-09T22:20:57.710404+0000","last_fullsized":"2024-06-09T22:20:57.710404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:31:51.296627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"99'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709927+0000","last_change":"2024-06-09T22:16:43.200637+0000","last_active":"2024-06-09T22:20:57.709927+0000","last_peered":"2024-06-09T22:20:57.709927+0000","last_clean":"2024-06-09T22:20:57.709927+0000","last_became_active":"2024-06-09T22:16:43.200268+0000","last_became_peered":"2024-06-09T22:16:43.200268+0000","last_unstale":"2024-06-09T22:20:57.709927+0000","last_undegraded":"2024-06-09T22:20:57.709927+0000","last_fullsized":"2024-06-09T22:20:57.709927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:25:59.881453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710229+0000","last_change":"2024-06-09T22:16:43.209212+0000","last_active":"2024-06-09T22:20:57.710229+0000","last_peered":"2024-06-09T22:20:57.710229+0000","last_clean":"2024-06-09T22:20:57.710229+0000","last_became_active":"2024-06-09T22:16:43.207789+0000","last_became_peered":"2024-06-09T22:16:43.207789+0000","last_unstale":"2024-06-09T22:20:57.710229+0000","last_undegraded":"2024-06-09T22:20:57.710229+0000","last_fullsized":"2024-06-09T22:20:57.710229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:17.316595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692756+0000","last_change":"2024-06-09T22:16:43.210935+0000","last_active":"2024-06-09T22:20:57.692756+0000","last_peered":"2024-06-09T22:20:57.692756+0000","last_clean":"2024-06-09T22:20:57.692756+0000","last_became_active":"2024-06-09T22:16:43.210631+0000","last_became_peered":"2024-06-09T22:16:43.210631+0000","last_unstale":"2024-06-09T22:20:57.692756+0000","last_undegraded":"2024-06-09T22:20:57.692756+0000","last_fullsized":"2024-06-09T22:20:57.692756+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:22:58.887313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597675+0000","last_change":"2024-06-09T22:16:43.198872+0000","last_active":"2024-06-09T22:20:57.597675+0000","last_peered":"2024-06-09T22:20:57.597675+0000","last_clean":"2024-06-09T22:20:57.597675+0000","last_became_active":"2024-06-09T22:16:43.198498+0000","last_became_peered":"2024-06-09T22:16:43.198498+0000","last_unstale":"2024-06-09T22:20:57.597675+0000","last_undegraded":"2024-06-09T22:20:57.597675+0000","last_fullsized":"2024-06-09T22:20:57.597675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:25:28.853048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597223+0000","last_change":"2024-06-09T22:16:53.343656+0000","last_active":"2024-06-09T22:20:57.597223+0000","last_peered":"2024-06-09T22:20:57.597223+0000","last_clean":"2024-06-09T22:20:57.597223+0000","last_became_active":"2024-06-09T22:16:53.343281+0000","last_became_peered":"2024-06-09T22:16:53.343281+0000","last_unstale":"2024-06-09T22:20:57.597223+0000","last_undegraded":"2024-06-09T22:20:57.597223+0000","last_fullsized":"2024-06-09T22:20:57.597223+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:40.961842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"100'14","reported_seq":269,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:16.725776+0000","last_change":"2024-06-09T22:16:40.040145+0000","last_active":"2024-06-09T22:21:16.725776+0000","last_peered":"2024-06-09T22:21:16.725776+0000","last_clean":"2024-06-09T22:21:16.725776+0000","last_became_active":"2024-06-09T22:16:40.039787+0000","last_became_peered":"2024-06-09T22:16:40.039787+0000","last_unstale":"2024-06-09T22:21:16.725776+0000","last_undegraded":"2024-06-09T22:21:16.725776+0000","last_fullsized":"2024-06-09T22:21:16.725776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:52:32.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":209,"num_read_kb":209,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"99'5","reported_seq":23,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692731+0000","last_change":"2024-06-09T22:17:26.774053+0000","last_active":"2024-06-09T22:20:57.692731+0000","last_peered":"2024-06-09T22:20:57.692731+0000","last_clean":"2024-06-09T22:20:57.692731+0000","last_became_active":"2024-06-09T22:17:26.773786+0000","last_became_peered":"2024-06-09T22:17:26.773786+0000","last_unstale":"2024-06-09T22:20:57.692731+0000","last_undegraded":"2024-06-09T22:20:57.692731+0000","last_fullsized":"2024-06-09T22:20:57.692731+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:36.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703244+0000","last_change":"2024-06-09T22:16:53.332048+0000","last_active":"2024-06-09T22:20:57.703244+0000","last_peered":"2024-06-09T22:20:57.703244+0000","last_clean":"2024-06-09T22:20:57.703244+0000","last_became_active":"2024-06-09T22:16:53.330683+0000","last_became_peered":"2024-06-09T22:16:53.330683+0000","last_unstale":"2024-06-09T22:20:57.703244+0000","last_undegraded":"2024-06-09T22:20:57.703244+0000","last_fullsized":"2024-06-09T22:20:57.703244+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:32:23.219061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"97'2","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696082+0000","last_change":"2024-06-09T22:16:40.070193+0000","last_active":"2024-06-09T22:20:57.696082+0000","last_peered":"2024-06-09T22:20:57.696082+0000","last_clean":"2024-06-09T22:20:57.696082+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:20:57.696082+0000","last_undegraded":"2024-06-09T22:20:57.696082+0000","last_fullsized":"2024-06-09T22:20:57.696082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:19:19.787552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711015+0000","last_change":"2024-06-09T22:16:43.199814+0000","last_active":"2024-06-09T22:20:57.711015+0000","last_peered":"2024-06-09T22:20:57.711015+0000","last_clean":"2024-06-09T22:20:57.711015+0000","last_became_active":"2024-06-09T22:16:43.199487+0000","last_became_peered":"2024-06-09T22:16:43.199487+0000","last_unstale":"2024-06-09T22:20:57.711015+0000","last_undegraded":"2024-06-09T22:20:57.711015+0000","last_fullsized":"2024-06-09T22:20:57.711015+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:55.672050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703543+0000","last_change":"2024-06-09T22:16:53.336253+0000","last_active":"2024-06-09T22:20:57.703543+0000","last_peered":"2024-06-09T22:20:57.703543+0000","last_clean":"2024-06-09T22:20:57.703543+0000","last_became_active":"2024-06-09T22:16:53.335869+0000","last_became_peered":"2024-06-09T22:16:53.335869+0000","last_unstale":"2024-06-09T22:20:57.703543+0000","last_undegraded":"2024-06-09T22:20:57.703543+0000","last_fullsized":"2024-06-09T22:20:57.703543+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:28:12.894878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"97'12","reported_seq":60,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695155+0000","last_change":"2024-06-09T22:16:40.045365+0000","last_active":"2024-06-09T22:20:57.695155+0000","last_peered":"2024-06-09T22:20:57.695155+0000","last_clean":"2024-06-09T22:20:57.695155+0000","last_became_active":"2024-06-09T22:16:40.044989+0000","last_became_peered":"2024-06-09T22:16:40.044989+0000","last_unstale":"2024-06-09T22:20:57.695155+0000","last_undegraded":"2024-06-09T22:20:57.695155+0000","last_fullsized":"2024-06-09T22:20:57.695155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:01:05.850170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"99'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710115+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:20:57.710115+0000","last_peered":"2024-06-09T22:20:57.710115+0000","last_clean":"2024-06-09T22:20:57.710115+0000","last_became_active":"2024-06-09T22:16:43.203999+0000","last_became_peered":"2024-06-09T22:16:43.203999+0000","last_unstale":"2024-06-09T22:20:57.710115+0000","last_undegraded":"2024-06-09T22:20:57.710115+0000","last_fullsized":"2024-06-09T22:20:57.710115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:37:28.362655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703277+0000","last_change":"2024-06-09T22:16:53.333565+0000","last_active":"2024-06-09T22:20:57.703277+0000","last_peered":"2024-06-09T22:20:57.703277+0000","last_clean":"2024-06-09T22:20:57.703277+0000","last_became_active":"2024-06-09T22:16:53.332921+0000","last_became_peered":"2024-06-09T22:16:53.332921+0000","last_unstale":"2024-06-09T22:20:57.703277+0000","last_undegraded":"2024-06-09T22:20:57.703277+0000","last_fullsized":"2024-06-09T22:20:57.703277+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:32.085617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"97'5","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694916+0000","last_change":"2024-06-09T22:16:40.039766+0000","last_active":"2024-06-09T22:20:57.694916+0000","last_peered":"2024-06-09T22:20:57.694916+0000","last_clean":"2024-06-09T22:20:57.694916+0000","last_became_active":"2024-06-09T22:16:40.039329+0000","last_became_peered":"2024-06-09T22:16:40.039329+0000","last_unstale":"2024-06-09T22:20:57.694916+0000","last_undegraded":"2024-06-09T22:20:57.694916+0000","last_fullsized":"2024-06-09T22:20:57.694916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:53.673890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"99'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703385+0000","last_change":"2024-06-09T22:16:43.201200+0000","last_active":"2024-06-09T22:20:57.703385+0000","last_peered":"2024-06-09T22:20:57.703385+0000","last_clean":"2024-06-09T22:20:57.703385+0000","last_became_active":"2024-06-09T22:16:43.200830+0000","last_became_peered":"2024-06-09T22:16:43.200830+0000","last_unstale":"2024-06-09T22:20:57.703385+0000","last_undegraded":"2024-06-09T22:20:57.703385+0000","last_fullsized":"2024-06-09T22:20:57.703385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:38:52.093173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597461+0000","last_change":"2024-06-09T22:16:53.339811+0000","last_active":"2024-06-09T22:20:57.597461+0000","last_peered":"2024-06-09T22:20:57.597461+0000","last_clean":"2024-06-09T22:20:57.597461+0000","last_became_active":"2024-06-09T22:16:53.339305+0000","last_became_peered":"2024-06-09T22:16:53.339305+0000","last_unstale":"2024-06-09T22:20:57.597461+0000","last_undegraded":"2024-06-09T22:20:57.597461+0000","last_fullsized":"2024-06-09T22:20:57.597461+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:08:25.800245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"100'9","reported_seq":267,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:16.727356+0000","last_change":"2024-06-09T22:16:40.044159+0000","last_active":"2024-06-09T22:21:16.727356+0000","last_peered":"2024-06-09T22:21:16.727356+0000","last_clean":"2024-06-09T22:21:16.727356+0000","last_became_active":"2024-06-09T22:16:40.043839+0000","last_became_peered":"2024-06-09T22:16:40.043839+0000","last_unstale":"2024-06-09T22:21:16.727356+0000","last_undegraded":"2024-06-09T22:21:16.727356+0000","last_fullsized":"2024-06-09T22:21:16.727356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:22:09.339219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694583+0000","last_change":"2024-06-09T22:16:43.202015+0000","last_active":"2024-06-09T22:20:57.694583+0000","last_peered":"2024-06-09T22:20:57.694583+0000","last_clean":"2024-06-09T22:20:57.694583+0000","last_became_active":"2024-06-09T22:16:43.201689+0000","last_became_peered":"2024-06-09T22:16:43.201689+0000","last_unstale":"2024-06-09T22:20:57.694583+0000","last_undegraded":"2024-06-09T22:20:57.694583+0000","last_fullsized":"2024-06-09T22:20:57.694583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:35:12.180399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597147+0000","last_change":"2024-06-09T22:18:26.698600+0000","last_active":"2024-06-09T22:20:57.597147+0000","last_peered":"2024-06-09T22:20:57.597147+0000","last_clean":"2024-06-09T22:20:57.597147+0000","last_became_active":"2024-06-09T22:18:26.698201+0000","last_became_peered":"2024-06-09T22:18:26.698201+0000","last_unstale":"2024-06-09T22:20:57.597147+0000","last_undegraded":"2024-06-09T22:20:57.597147+0000","last_fullsized":"2024-06-09T22:20:57.597147+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:55:13.513557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"97'7","reported_seq":57,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702844+0000","last_change":"2024-06-09T22:17:26.698204+0000","last_active":"2024-06-09T22:20:57.702844+0000","last_peered":"2024-06-09T22:20:57.702844+0000","last_clean":"2024-06-09T22:20:57.702844+0000","last_became_active":"2024-06-09T22:17:26.697716+0000","last_became_peered":"2024-06-09T22:17:26.697716+0000","last_unstale":"2024-06-09T22:20:57.702844+0000","last_undegraded":"2024-06-09T22:20:57.702844+0000","last_fullsized":"2024-06-09T22:20:57.702844+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:36:37.724459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703554+0000","last_change":"2024-06-09T22:16:43.198917+0000","last_active":"2024-06-09T22:20:57.703554+0000","last_peered":"2024-06-09T22:20:57.703554+0000","last_clean":"2024-06-09T22:20:57.703554+0000","last_became_active":"2024-06-09T22:16:43.198567+0000","last_became_peered":"2024-06-09T22:16:43.198567+0000","last_unstale":"2024-06-09T22:20:57.703554+0000","last_undegraded":"2024-06-09T22:20:57.703554+0000","last_fullsized":"2024-06-09T22:20:57.703554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:34.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710233+0000","last_change":"2024-06-09T22:16:53.342203+0000","last_active":"2024-06-09T22:20:57.710233+0000","last_peered":"2024-06-09T22:20:57.710233+0000","last_clean":"2024-06-09T22:20:57.710233+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:20:57.710233+0000","last_undegraded":"2024-06-09T22:20:57.710233+0000","last_fullsized":"2024-06-09T22:20:57.710233+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:41:41.491961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"97'3","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595514+0000","last_change":"2024-06-09T22:16:40.038684+0000","last_active":"2024-06-09T22:20:57.595514+0000","last_peered":"2024-06-09T22:20:57.595514+0000","last_clean":"2024-06-09T22:20:57.595514+0000","last_became_active":"2024-06-09T22:16:40.038386+0000","last_became_peered":"2024-06-09T22:16:40.038386+0000","last_unstale":"2024-06-09T22:20:57.595514+0000","last_undegraded":"2024-06-09T22:20:57.595514+0000","last_fullsized":"2024-06-09T22:20:57.595514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:59.188758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694679+0000","last_change":"2024-06-09T22:16:43.198837+0000","last_active":"2024-06-09T22:20:57.694679+0000","last_peered":"2024-06-09T22:20:57.694679+0000","last_clean":"2024-06-09T22:20:57.694679+0000","last_became_active":"2024-06-09T22:16:43.198515+0000","last_became_peered":"2024-06-09T22:16:43.198515+0000","last_unstale":"2024-06-09T22:20:57.694679+0000","last_undegraded":"2024-06-09T22:20:57.694679+0000","last_fullsized":"2024-06-09T22:20:57.694679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:36:13.863202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711076+0000","last_change":"2024-06-09T22:16:53.342688+0000","last_active":"2024-06-09T22:20:57.711076+0000","last_peered":"2024-06-09T22:20:57.711076+0000","last_clean":"2024-06-09T22:20:57.711076+0000","last_became_active":"2024-06-09T22:16:53.342314+0000","last_became_peered":"2024-06-09T22:16:53.342314+0000","last_unstale":"2024-06-09T22:20:57.711076+0000","last_undegraded":"2024-06-09T22:20:57.711076+0000","last_fullsized":"2024-06-09T22:20:57.711076+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:51:35.801587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"97'3","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703264+0000","last_change":"2024-06-09T22:16:40.041831+0000","last_active":"2024-06-09T22:20:57.703264+0000","last_peered":"2024-06-09T22:20:57.703264+0000","last_clean":"2024-06-09T22:20:57.703264+0000","last_became_active":"2024-06-09T22:16:40.041484+0000","last_became_peered":"2024-06-09T22:16:40.041484+0000","last_unstale":"2024-06-09T22:20:57.703264+0000","last_undegraded":"2024-06-09T22:20:57.703264+0000","last_fullsized":"2024-06-09T22:20:57.703264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:10:19.829357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692213+0000","last_change":"2024-06-09T22:16:43.210769+0000","last_active":"2024-06-09T22:20:57.692213+0000","last_peered":"2024-06-09T22:20:57.692213+0000","last_clean":"2024-06-09T22:20:57.692213+0000","last_became_active":"2024-06-09T22:16:43.210429+0000","last_became_peered":"2024-06-09T22:16:43.210429+0000","last_unstale":"2024-06-09T22:20:57.692213+0000","last_undegraded":"2024-06-09T22:20:57.692213+0000","last_fullsized":"2024-06-09T22:20:57.692213+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:12:28.788830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694708+0000","last_change":"2024-06-09T22:16:53.336536+0000","last_active":"2024-06-09T22:20:57.694708+0000","last_peered":"2024-06-09T22:20:57.694708+0000","last_clean":"2024-06-09T22:20:57.694708+0000","last_became_active":"2024-06-09T22:16:53.335178+0000","last_became_peered":"2024-06-09T22:16:53.335178+0000","last_unstale":"2024-06-09T22:20:57.694708+0000","last_undegraded":"2024-06-09T22:20:57.694708+0000","last_fullsized":"2024-06-09T22:20:57.694708+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:57:19.476483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'8","reported_seq":466,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:17.728156+0000","last_change":"2024-06-09T22:16:40.038484+0000","last_active":"2024-06-09T22:21:17.728156+0000","last_peered":"2024-06-09T22:21:17.728156+0000","last_clean":"2024-06-09T22:21:17.728156+0000","last_became_active":"2024-06-09T22:16:40.038084+0000","last_became_peered":"2024-06-09T22:16:40.038084+0000","last_unstale":"2024-06-09T22:21:17.728156+0000","last_undegraded":"2024-06-09T22:21:17.728156+0000","last_fullsized":"2024-06-09T22:21:17.728156+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:19.733107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":414,"num_read_kb":414,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710525+0000","last_change":"2024-06-09T22:16:43.205137+0000","last_active":"2024-06-09T22:20:57.710525+0000","last_peered":"2024-06-09T22:20:57.710525+0000","last_clean":"2024-06-09T22:20:57.710525+0000","last_became_active":"2024-06-09T22:16:43.204499+0000","last_became_peered":"2024-06-09T22:16:43.204499+0000","last_unstale":"2024-06-09T22:20:57.710525+0000","last_undegraded":"2024-06-09T22:20:57.710525+0000","last_fullsized":"2024-06-09T22:20:57.710525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:15.718493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692194+0000","last_change":"2024-06-09T22:16:53.344260+0000","last_active":"2024-06-09T22:20:57.692194+0000","last_peered":"2024-06-09T22:20:57.692194+0000","last_clean":"2024-06-09T22:20:57.692194+0000","last_became_active":"2024-06-09T22:16:53.343840+0000","last_became_peered":"2024-06-09T22:16:53.343840+0000","last_unstale":"2024-06-09T22:20:57.692194+0000","last_undegraded":"2024-06-09T22:20:57.692194+0000","last_fullsized":"2024-06-09T22:20:57.692194+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:40:23.305509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"97'4","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596747+0000","last_change":"2024-06-09T22:16:40.042660+0000","last_active":"2024-06-09T22:20:57.596747+0000","last_peered":"2024-06-09T22:20:57.596747+0000","last_clean":"2024-06-09T22:20:57.596747+0000","last_became_active":"2024-06-09T22:16:40.041353+0000","last_became_peered":"2024-06-09T22:16:40.041353+0000","last_unstale":"2024-06-09T22:20:57.596747+0000","last_undegraded":"2024-06-09T22:20:57.596747+0000","last_fullsized":"2024-06-09T22:20:57.596747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:34:59.064827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711158+0000","last_change":"2024-06-09T22:16:43.205083+0000","last_active":"2024-06-09T22:20:57.711158+0000","last_peered":"2024-06-09T22:20:57.711158+0000","last_clean":"2024-06-09T22:20:57.711158+0000","last_became_active":"2024-06-09T22:16:43.204600+0000","last_became_peered":"2024-06-09T22:16:43.204600+0000","last_unstale":"2024-06-09T22:20:57.711158+0000","last_undegraded":"2024-06-09T22:20:57.711158+0000","last_fullsized":"2024-06-09T22:20:57.711158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:05:12.555465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703699+0000","last_change":"2024-06-09T22:16:53.336169+0000","last_active":"2024-06-09T22:20:57.703699+0000","last_peered":"2024-06-09T22:20:57.703699+0000","last_clean":"2024-06-09T22:20:57.703699+0000","last_became_active":"2024-06-09T22:16:53.335657+0000","last_became_peered":"2024-06-09T22:16:53.335657+0000","last_unstale":"2024-06-09T22:20:57.703699+0000","last_undegraded":"2024-06-09T22:20:57.703699+0000","last_fullsized":"2024-06-09T22:20:57.703699+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:53:21.282767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703313+0000","last_change":"2024-06-09T22:16:40.070568+0000","last_active":"2024-06-09T22:20:57.703313+0000","last_peered":"2024-06-09T22:20:57.703313+0000","last_clean":"2024-06-09T22:20:57.703313+0000","last_became_active":"2024-06-09T22:16:40.070205+0000","last_became_peered":"2024-06-09T22:16:40.070205+0000","last_unstale":"2024-06-09T22:20:57.703313+0000","last_undegraded":"2024-06-09T22:20:57.703313+0000","last_fullsized":"2024-06-09T22:20:57.703313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.449965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710336+0000","last_change":"2024-06-09T22:16:43.205219+0000","last_active":"2024-06-09T22:20:57.710336+0000","last_peered":"2024-06-09T22:20:57.710336+0000","last_clean":"2024-06-09T22:20:57.710336+0000","last_became_active":"2024-06-09T22:16:43.203467+0000","last_became_peered":"2024-06-09T22:16:43.203467+0000","last_unstale":"2024-06-09T22:20:57.710336+0000","last_undegraded":"2024-06-09T22:20:57.710336+0000","last_fullsized":"2024-06-09T22:20:57.710336+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:22.784034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692165+0000","last_change":"2024-06-09T22:16:53.338990+0000","last_active":"2024-06-09T22:20:57.692165+0000","last_peered":"2024-06-09T22:20:57.692165+0000","last_clean":"2024-06-09T22:20:57.692165+0000","last_became_active":"2024-06-09T22:16:53.338568+0000","last_became_peered":"2024-06-09T22:16:53.338568+0000","last_unstale":"2024-06-09T22:20:57.692165+0000","last_undegraded":"2024-06-09T22:20:57.692165+0000","last_fullsized":"2024-06-09T22:20:57.692165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:46:39.814043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"100'48","reported_seq":521,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:17.728316+0000","last_change":"2024-06-09T22:16:40.044404+0000","last_active":"2024-06-09T22:21:17.728316+0000","last_peered":"2024-06-09T22:21:17.728316+0000","last_clean":"2024-06-09T22:21:17.728316+0000","last_became_active":"2024-06-09T22:16:40.044044+0000","last_became_peered":"2024-06-09T22:16:40.044044+0000","last_unstale":"2024-06-09T22:21:17.728316+0000","last_undegraded":"2024-06-09T22:21:17.728316+0000","last_fullsized":"2024-06-09T22:21:17.728316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:50:27.038759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77586,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":429,"num_read_kb":292,"num_write":50,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694903+0000","last_change":"2024-06-09T22:16:43.206963+0000","last_active":"2024-06-09T22:20:57.694903+0000","last_peered":"2024-06-09T22:20:57.694903+0000","last_clean":"2024-06-09T22:20:57.694903+0000","last_became_active":"2024-06-09T22:16:43.206628+0000","last_became_peered":"2024-06-09T22:16:43.206628+0000","last_unstale":"2024-06-09T22:20:57.694903+0000","last_undegraded":"2024-06-09T22:20:57.694903+0000","last_fullsized":"2024-06-09T22:20:57.694903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:33:39.912236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703845+0000","last_change":"2024-06-09T22:16:53.339332+0000","last_active":"2024-06-09T22:20:57.703845+0000","last_peered":"2024-06-09T22:20:57.703845+0000","last_clean":"2024-06-09T22:20:57.703845+0000","last_became_active":"2024-06-09T22:16:53.338819+0000","last_became_peered":"2024-06-09T22:16:53.338819+0000","last_unstale":"2024-06-09T22:20:57.703845+0000","last_undegraded":"2024-06-09T22:20:57.703845+0000","last_fullsized":"2024-06-09T22:20:57.703845+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:57:41.142142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"97'9","reported_seq":33,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709996+0000","last_change":"2024-06-09T22:17:26.478256+0000","last_active":"2024-06-09T22:20:57.709996+0000","last_peered":"2024-06-09T22:20:57.709996+0000","last_clean":"2024-06-09T22:20:57.709996+0000","last_became_active":"2024-06-09T22:17:26.472264+0000","last_became_peered":"2024-06-09T22:17:26.472264+0000","last_unstale":"2024-06-09T22:20:57.709996+0000","last_undegraded":"2024-06-09T22:20:57.709996+0000","last_fullsized":"2024-06-09T22:20:57.709996+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:14:19.919841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"99'9","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595647+0000","last_change":"2024-06-09T22:16:43.204484+0000","last_active":"2024-06-09T22:20:57.595647+0000","last_peered":"2024-06-09T22:20:57.595647+0000","last_clean":"2024-06-09T22:20:57.595647+0000","last_became_active":"2024-06-09T22:16:43.204014+0000","last_became_peered":"2024-06-09T22:16:43.204014+0000","last_unstale":"2024-06-09T22:20:57.595647+0000","last_undegraded":"2024-06-09T22:20:57.595647+0000","last_fullsized":"2024-06-09T22:20:57.595647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:55:12.528231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710227+0000","last_change":"2024-06-09T22:17:26.475179+0000","last_active":"2024-06-09T22:20:57.710227+0000","last_peered":"2024-06-09T22:20:57.710227+0000","last_clean":"2024-06-09T22:20:57.710227+0000","last_became_active":"2024-06-09T22:17:26.474517+0000","last_became_peered":"2024-06-09T22:17:26.474517+0000","last_unstale":"2024-06-09T22:20:57.710227+0000","last_undegraded":"2024-06-09T22:20:57.710227+0000","last_fullsized":"2024-06-09T22:20:57.710227+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:26:02.387830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"97'5","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703424+0000","last_change":"2024-06-09T22:16:40.043066+0000","last_active":"2024-06-09T22:20:57.703424+0000","last_peered":"2024-06-09T22:20:57.703424+0000","last_clean":"2024-06-09T22:20:57.703424+0000","last_became_active":"2024-06-09T22:16:40.042619+0000","last_became_peered":"2024-06-09T22:16:40.042619+0000","last_unstale":"2024-06-09T22:20:57.703424+0000","last_undegraded":"2024-06-09T22:20:57.703424+0000","last_fullsized":"2024-06-09T22:20:57.703424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:29.689614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"99'15","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597173+0000","last_change":"2024-06-09T22:17:26.475566+0000","last_active":"2024-06-09T22:20:57.597173+0000","last_peered":"2024-06-09T22:20:57.597173+0000","last_clean":"2024-06-09T22:20:57.597173+0000","last_became_active":"2024-06-09T22:17:26.475037+0000","last_became_peered":"2024-06-09T22:17:26.475037+0000","last_unstale":"2024-06-09T22:20:57.597173+0000","last_undegraded":"2024-06-09T22:20:57.597173+0000","last_fullsized":"2024-06-09T22:20:57.597173+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:33.075144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694959+0000","last_change":"2024-06-09T22:17:26.475111+0000","last_active":"2024-06-09T22:20:57.694959+0000","last_peered":"2024-06-09T22:20:57.694959+0000","last_clean":"2024-06-09T22:20:57.694959+0000","last_became_active":"2024-06-09T22:17:26.474563+0000","last_became_peered":"2024-06-09T22:17:26.474563+0000","last_unstale":"2024-06-09T22:20:57.694959+0000","last_undegraded":"2024-06-09T22:20:57.694959+0000","last_fullsized":"2024-06-09T22:20:57.694959+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:09:08.458761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"100'9","reported_seq":268,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:17.728278+0000","last_change":"2024-06-09T22:16:40.040538+0000","last_active":"2024-06-09T22:21:17.728278+0000","last_peered":"2024-06-09T22:21:17.728278+0000","last_clean":"2024-06-09T22:21:17.728278+0000","last_became_active":"2024-06-09T22:16:40.039896+0000","last_became_peered":"2024-06-09T22:16:40.039896+0000","last_unstale":"2024-06-09T22:21:17.728278+0000","last_undegraded":"2024-06-09T22:21:17.728278+0000","last_fullsized":"2024-06-09T22:21:17.728278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:00:33.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1817,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":214,"num_read_kb":7,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"99'6","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596912+0000","last_change":"2024-06-09T22:17:26.475568+0000","last_active":"2024-06-09T22:20:57.596912+0000","last_peered":"2024-06-09T22:20:57.596912+0000","last_clean":"2024-06-09T22:20:57.596912+0000","last_became_active":"2024-06-09T22:17:26.475163+0000","last_became_peered":"2024-06-09T22:17:26.475163+0000","last_unstale":"2024-06-09T22:20:57.596912+0000","last_undegraded":"2024-06-09T22:20:57.596912+0000","last_fullsized":"2024-06-09T22:20:57.596912+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:20.865164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595970+0000","last_change":"2024-06-09T22:16:53.338201+0000","last_active":"2024-06-09T22:20:57.595970+0000","last_peered":"2024-06-09T22:20:57.595970+0000","last_clean":"2024-06-09T22:20:57.595970+0000","last_became_active":"2024-06-09T22:16:53.337816+0000","last_became_peered":"2024-06-09T22:16:53.337816+0000","last_unstale":"2024-06-09T22:20:57.595970+0000","last_undegraded":"2024-06-09T22:20:57.595970+0000","last_fullsized":"2024-06-09T22:20:57.595970+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:35:51.861135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"97'17","reported_seq":66,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694533+0000","last_change":"2024-06-09T22:16:40.070770+0000","last_active":"2024-06-09T22:20:57.694533+0000","last_peered":"2024-06-09T22:20:57.694533+0000","last_clean":"2024-06-09T22:20:57.694533+0000","last_became_active":"2024-06-09T22:16:40.070398+0000","last_became_peered":"2024-06-09T22:16:40.070398+0000","last_unstale":"2024-06-09T22:20:57.694533+0000","last_undegraded":"2024-06-09T22:20:57.694533+0000","last_fullsized":"2024-06-09T22:20:57.694533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:04:58.273601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"99'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703349+0000","last_change":"2024-06-09T22:16:43.199246+0000","last_active":"2024-06-09T22:20:57.703349+0000","last_peered":"2024-06-09T22:20:57.703349+0000","last_clean":"2024-06-09T22:20:57.703349+0000","last_became_active":"2024-06-09T22:16:43.198006+0000","last_became_peered":"2024-06-09T22:16:43.198006+0000","last_unstale":"2024-06-09T22:20:57.703349+0000","last_undegraded":"2024-06-09T22:20:57.703349+0000","last_fullsized":"2024-06-09T22:20:57.703349+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:30:55.836628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703304+0000","last_change":"2024-06-09T22:16:53.342217+0000","last_active":"2024-06-09T22:20:57.703304+0000","last_peered":"2024-06-09T22:20:57.703304+0000","last_clean":"2024-06-09T22:20:57.703304+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:20:57.703304+0000","last_undegraded":"2024-06-09T22:20:57.703304+0000","last_fullsized":"2024-06-09T22:20:57.703304+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:25:28.635091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"100'4","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:58.449594+0000","last_change":"2024-06-09T22:16:40.039841+0000","last_active":"2024-06-09T22:20:58.449594+0000","last_peered":"2024-06-09T22:20:58.449594+0000","last_clean":"2024-06-09T22:20:58.449594+0000","last_became_active":"2024-06-09T22:16:40.039416+0000","last_became_peered":"2024-06-09T22:16:40.039416+0000","last_unstale":"2024-06-09T22:20:58.449594+0000","last_undegraded":"2024-06-09T22:20:58.449594+0000","last_fullsized":"2024-06-09T22:20:58.449594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:27:36.213839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"99'5","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710159+0000","last_change":"2024-06-09T22:16:43.214363+0000","last_active":"2024-06-09T22:20:57.710159+0000","last_peered":"2024-06-09T22:20:57.710159+0000","last_clean":"2024-06-09T22:20:57.710159+0000","last_became_active":"2024-06-09T22:16:43.214015+0000","last_became_peered":"2024-06-09T22:16:43.214015+0000","last_unstale":"2024-06-09T22:20:57.710159+0000","last_undegraded":"2024-06-09T22:20:57.710159+0000","last_fullsized":"2024-06-09T22:20:57.710159+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:13:06.924468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694710+0000","last_change":"2024-06-09T22:16:53.342151+0000","last_active":"2024-06-09T22:20:57.694710+0000","last_peered":"2024-06-09T22:20:57.694710+0000","last_clean":"2024-06-09T22:20:57.694710+0000","last_became_active":"2024-06-09T22:16:53.341743+0000","last_became_peered":"2024-06-09T22:16:53.341743+0000","last_unstale":"2024-06-09T22:20:57.694710+0000","last_undegraded":"2024-06-09T22:20:57.694710+0000","last_fullsized":"2024-06-09T22:20:57.694710+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:39:44.793301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"97'1","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692594+0000","last_change":"2024-06-09T22:17:26.472397+0000","last_active":"2024-06-09T22:20:57.692594+0000","last_peered":"2024-06-09T22:20:57.692594+0000","last_clean":"2024-06-09T22:20:57.692594+0000","last_became_active":"2024-06-09T22:17:26.471950+0000","last_became_peered":"2024-06-09T22:17:26.471950+0000","last_unstale":"2024-06-09T22:20:57.692594+0000","last_undegraded":"2024-06-09T22:20:57.692594+0000","last_fullsized":"2024-06-09T22:20:57.692594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:35:57.481510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"99'4","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703072+0000","last_change":"2024-06-09T22:16:43.214976+0000","last_active":"2024-06-09T22:20:57.703072+0000","last_peered":"2024-06-09T22:20:57.703072+0000","last_clean":"2024-06-09T22:20:57.703072+0000","last_became_active":"2024-06-09T22:16:43.214592+0000","last_became_peered":"2024-06-09T22:16:43.214592+0000","last_unstale":"2024-06-09T22:20:57.703072+0000","last_undegraded":"2024-06-09T22:20:57.703072+0000","last_fullsized":"2024-06-09T22:20:57.703072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:16:21.981144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595645+0000","last_change":"2024-06-09T22:16:53.342721+0000","last_active":"2024-06-09T22:20:57.595645+0000","last_peered":"2024-06-09T22:20:57.595645+0000","last_clean":"2024-06-09T22:20:57.595645+0000","last_became_active":"2024-06-09T22:16:53.342342+0000","last_became_peered":"2024-06-09T22:16:53.342342+0000","last_unstale":"2024-06-09T22:20:57.595645+0000","last_undegraded":"2024-06-09T22:20:57.595645+0000","last_fullsized":"2024-06-09T22:20:57.595645+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:22:07.469873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"100'6","reported_seq":263,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:16.726441+0000","last_change":"2024-06-09T22:16:40.041307+0000","last_active":"2024-06-09T22:21:16.726441+0000","last_peered":"2024-06-09T22:21:16.726441+0000","last_clean":"2024-06-09T22:21:16.726441+0000","last_became_active":"2024-06-09T22:16:40.040819+0000","last_became_peered":"2024-06-09T22:16:40.040819+0000","last_unstale":"2024-06-09T22:21:16.726441+0000","last_undegraded":"2024-06-09T22:21:16.726441+0000","last_fullsized":"2024-06-09T22:21:16.726441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:14:29.658293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":978,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":209,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"99'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702925+0000","last_change":"2024-06-09T22:16:43.215719+0000","last_active":"2024-06-09T22:20:57.702925+0000","last_peered":"2024-06-09T22:20:57.702925+0000","last_clean":"2024-06-09T22:20:57.702925+0000","last_became_active":"2024-06-09T22:16:43.215356+0000","last_became_peered":"2024-06-09T22:16:43.215356+0000","last_unstale":"2024-06-09T22:20:57.702925+0000","last_undegraded":"2024-06-09T22:20:57.702925+0000","last_fullsized":"2024-06-09T22:20:57.702925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:15:07.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596028+0000","last_change":"2024-06-09T22:16:53.342948+0000","last_active":"2024-06-09T22:20:57.596028+0000","last_peered":"2024-06-09T22:20:57.596028+0000","last_clean":"2024-06-09T22:20:57.596028+0000","last_became_active":"2024-06-09T22:16:53.342352+0000","last_became_peered":"2024-06-09T22:16:53.342352+0000","last_unstale":"2024-06-09T22:20:57.596028+0000","last_undegraded":"2024-06-09T22:20:57.596028+0000","last_fullsized":"2024-06-09T22:20:57.596028+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:43:27.453789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703458+0000","last_change":"2024-06-09T22:16:43.204897+0000","last_active":"2024-06-09T22:20:57.703458+0000","last_peered":"2024-06-09T22:20:57.703458+0000","last_clean":"2024-06-09T22:20:57.703458+0000","last_became_active":"2024-06-09T22:16:43.204470+0000","last_became_peered":"2024-06-09T22:16:43.204470+0000","last_unstale":"2024-06-09T22:20:57.703458+0000","last_undegraded":"2024-06-09T22:20:57.703458+0000","last_fullsized":"2024-06-09T22:20:57.703458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:56.681392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"99'7","reported_seq":263,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:16.725979+0000","last_change":"2024-06-09T22:16:40.070079+0000","last_active":"2024-06-09T22:21:16.725979+0000","last_peered":"2024-06-09T22:21:16.725979+0000","last_clean":"2024-06-09T22:21:16.725979+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:21:16.725979+0000","last_undegraded":"2024-06-09T22:21:16.725979+0000","last_fullsized":"2024-06-09T22:21:16.725979+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:00:45.132139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":208,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"97'1","reported_seq":33,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596870+0000","last_change":"2024-06-09T22:16:53.338891+0000","last_active":"2024-06-09T22:20:57.596870+0000","last_peered":"2024-06-09T22:20:57.596870+0000","last_clean":"2024-06-09T22:20:57.596870+0000","last_became_active":"2024-06-09T22:16:53.338526+0000","last_became_peered":"2024-06-09T22:16:53.338526+0000","last_unstale":"2024-06-09T22:20:57.596870+0000","last_undegraded":"2024-06-09T22:20:57.596870+0000","last_fullsized":"2024-06-09T22:20:57.596870+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:34:37.004407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695269+0000","last_change":"2024-06-09T22:16:43.199234+0000","last_active":"2024-06-09T22:20:57.695269+0000","last_peered":"2024-06-09T22:20:57.695269+0000","last_clean":"2024-06-09T22:20:57.695269+0000","last_became_active":"2024-06-09T22:16:43.192012+0000","last_became_peered":"2024-06-09T22:16:43.192012+0000","last_unstale":"2024-06-09T22:20:57.695269+0000","last_undegraded":"2024-06-09T22:20:57.695269+0000","last_fullsized":"2024-06-09T22:20:57.695269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:17.429236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"97'6","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703645+0000","last_change":"2024-06-09T22:16:40.041970+0000","last_active":"2024-06-09T22:20:57.703645+0000","last_peered":"2024-06-09T22:20:57.703645+0000","last_clean":"2024-06-09T22:20:57.703645+0000","last_became_active":"2024-06-09T22:16:40.040371+0000","last_became_peered":"2024-06-09T22:16:40.040371+0000","last_unstale":"2024-06-09T22:20:57.703645+0000","last_undegraded":"2024-06-09T22:20:57.703645+0000","last_fullsized":"2024-06-09T22:20:57.703645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:52:31.210423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"100'148","reported_seq":425,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:59.707074+0000","last_change":"2024-06-09T22:15:15.367397+0000","last_active":"2024-06-09T22:20:59.707074+0000","last_peered":"2024-06-09T22:20:59.707074+0000","last_clean":"2024-06-09T22:20:59.707074+0000","last_became_active":"2024-06-09T22:15:15.366974+0000","last_became_peered":"2024-06-09T22:15:15.366974+0000","last_unstale":"2024-06-09T22:20:59.707074+0000","last_undegraded":"2024-06-09T22:20:59.707074+0000","last_fullsized":"2024-06-09T22:20:59.707074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_clean_scrub_stamp":"2024-06-09T22:11:49.886056+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:24.018031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.704010+0000","last_change":"2024-06-09T22:16:53.342249+0000","last_active":"2024-06-09T22:20:57.704010+0000","last_peered":"2024-06-09T22:20:57.704010+0000","last_clean":"2024-06-09T22:20:57.704010+0000","last_became_active":"2024-06-09T22:16:53.341779+0000","last_became_peered":"2024-06-09T22:16:53.341779+0000","last_unstale":"2024-06-09T22:20:57.704010+0000","last_undegraded":"2024-06-09T22:20:57.704010+0000","last_fullsized":"2024-06-09T22:20:57.704010+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:21:21.101578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"99'5","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595425+0000","last_change":"2024-06-09T22:16:43.200778+0000","last_active":"2024-06-09T22:20:57.595425+0000","last_peered":"2024-06-09T22:20:57.595425+0000","last_clean":"2024-06-09T22:20:57.595425+0000","last_became_active":"2024-06-09T22:16:43.200490+0000","last_became_peered":"2024-06-09T22:16:43.200490+0000","last_unstale":"2024-06-09T22:20:57.595425+0000","last_undegraded":"2024-06-09T22:20:57.595425+0000","last_fullsized":"2024-06-09T22:20:57.595425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:20:17.530204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"97'4","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596731+0000","last_change":"2024-06-09T22:16:40.041774+0000","last_active":"2024-06-09T22:20:57.596731+0000","last_peered":"2024-06-09T22:20:57.596731+0000","last_clean":"2024-06-09T22:20:57.596731+0000","last_became_active":"2024-06-09T22:16:40.041085+0000","last_became_peered":"2024-06-09T22:16:40.041085+0000","last_unstale":"2024-06-09T22:20:57.596731+0000","last_undegraded":"2024-06-09T22:20:57.596731+0000","last_fullsized":"2024-06-09T22:20:57.596731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:09:41.140083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710137+0000","last_change":"2024-06-09T22:16:53.336895+0000","last_active":"2024-06-09T22:20:57.710137+0000","last_peered":"2024-06-09T22:20:57.710137+0000","last_clean":"2024-06-09T22:20:57.710137+0000","last_became_active":"2024-06-09T22:16:53.336497+0000","last_became_peered":"2024-06-09T22:16:53.336497+0000","last_unstale":"2024-06-09T22:20:57.710137+0000","last_undegraded":"2024-06-09T22:20:57.710137+0000","last_fullsized":"2024-06-09T22:20:57.710137+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:29:52.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"97'4","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695101+0000","last_change":"2024-06-09T22:16:40.070909+0000","last_active":"2024-06-09T22:20:57.695101+0000","last_peered":"2024-06-09T22:20:57.695101+0000","last_clean":"2024-06-09T22:20:57.695101+0000","last_became_active":"2024-06-09T22:16:40.070556+0000","last_became_peered":"2024-06-09T22:16:40.070556+0000","last_unstale":"2024-06-09T22:20:57.695101+0000","last_undegraded":"2024-06-09T22:20:57.695101+0000","last_fullsized":"2024-06-09T22:20:57.695101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:59:17.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"99'9","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595793+0000","last_change":"2024-06-09T22:16:43.205951+0000","last_active":"2024-06-09T22:20:57.595793+0000","last_peered":"2024-06-09T22:20:57.595793+0000","last_clean":"2024-06-09T22:20:57.595793+0000","last_became_active":"2024-06-09T22:16:43.205594+0000","last_became_peered":"2024-06-09T22:16:43.205594+0000","last_unstale":"2024-06-09T22:20:57.595793+0000","last_undegraded":"2024-06-09T22:20:57.595793+0000","last_fullsized":"2024-06-09T22:20:57.595793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:17.956406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694724+0000","last_change":"2024-06-09T22:16:53.343292+0000","last_active":"2024-06-09T22:20:57.694724+0000","last_peered":"2024-06-09T22:20:57.694724+0000","last_clean":"2024-06-09T22:20:57.694724+0000","last_became_active":"2024-06-09T22:16:53.342900+0000","last_became_peered":"2024-06-09T22:16:53.342900+0000","last_unstale":"2024-06-09T22:20:57.694724+0000","last_undegraded":"2024-06-09T22:20:57.694724+0000","last_fullsized":"2024-06-09T22:20:57.694724+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:08:07.328250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"100'105","reported_seq":243,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:13.158591+0000","last_change":"2024-06-09T22:16:40.039417+0000","last_active":"2024-06-09T22:21:13.158591+0000","last_peered":"2024-06-09T22:21:13.158591+0000","last_clean":"2024-06-09T22:21:13.158591+0000","last_became_active":"2024-06-09T22:16:40.039099+0000","last_became_peered":"2024-06-09T22:16:40.039099+0000","last_unstale":"2024-06-09T22:21:13.158591+0000","last_undegraded":"2024-06-09T22:21:13.158591+0000","last_fullsized":"2024-06-09T22:21:13.158591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:22:59.284442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":191,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"99'5","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696372+0000","last_change":"2024-06-09T22:16:43.201285+0000","last_active":"2024-06-09T22:20:57.696372+0000","last_peered":"2024-06-09T22:20:57.696372+0000","last_clean":"2024-06-09T22:20:57.696372+0000","last_became_active":"2024-06-09T22:16:43.200993+0000","last_became_peered":"2024-06-09T22:16:43.200993+0000","last_unstale":"2024-06-09T22:20:57.696372+0000","last_undegraded":"2024-06-09T22:20:57.696372+0000","last_fullsized":"2024-06-09T22:20:57.696372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:15:26.491588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596798+0000","last_change":"2024-06-09T22:16:53.334088+0000","last_active":"2024-06-09T22:20:57.596798+0000","last_peered":"2024-06-09T22:20:57.596798+0000","last_clean":"2024-06-09T22:20:57.596798+0000","last_became_active":"2024-06-09T22:16:53.333597+0000","last_became_peered":"2024-06-09T22:16:53.333597+0000","last_unstale":"2024-06-09T22:20:57.596798+0000","last_undegraded":"2024-06-09T22:20:57.596798+0000","last_fullsized":"2024-06-09T22:20:57.596798+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:50:22.041816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"97'7","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703700+0000","last_change":"2024-06-09T22:16:40.364982+0000","last_active":"2024-06-09T22:20:57.703700+0000","last_peered":"2024-06-09T22:20:57.703700+0000","last_clean":"2024-06-09T22:20:57.703700+0000","last_became_active":"2024-06-09T22:16:40.364585+0000","last_became_peered":"2024-06-09T22:16:40.364585+0000","last_unstale":"2024-06-09T22:20:57.703700+0000","last_undegraded":"2024-06-09T22:20:57.703700+0000","last_fullsized":"2024-06-09T22:20:57.703700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:28.777950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"99'7","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711181+0000","last_change":"2024-06-09T22:17:26.474491+0000","last_active":"2024-06-09T22:20:57.711181+0000","last_peered":"2024-06-09T22:20:57.711181+0000","last_clean":"2024-06-09T22:20:57.711181+0000","last_became_active":"2024-06-09T22:17:26.473975+0000","last_became_peered":"2024-06-09T22:17:26.473975+0000","last_unstale":"2024-06-09T22:20:57.711181+0000","last_undegraded":"2024-06-09T22:20:57.711181+0000","last_fullsized":"2024-06-09T22:20:57.711181+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:13:46.998595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703384+0000","last_change":"2024-06-09T22:16:53.342273+0000","last_active":"2024-06-09T22:20:57.703384+0000","last_peered":"2024-06-09T22:20:57.703384+0000","last_clean":"2024-06-09T22:20:57.703384+0000","last_became_active":"2024-06-09T22:16:53.341827+0000","last_became_peered":"2024-06-09T22:16:53.341827+0000","last_unstale":"2024-06-09T22:20:57.703384+0000","last_undegraded":"2024-06-09T22:20:57.703384+0000","last_fullsized":"2024-06-09T22:20:57.703384+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:23:26.099737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"97'2","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710040+0000","last_change":"2024-06-09T22:16:40.070498+0000","last_active":"2024-06-09T22:20:57.710040+0000","last_peered":"2024-06-09T22:20:57.710040+0000","last_clean":"2024-06-09T22:20:57.710040+0000","last_became_active":"2024-06-09T22:16:40.070144+0000","last_became_peered":"2024-06-09T22:16:40.070144+0000","last_unstale":"2024-06-09T22:20:57.710040+0000","last_undegraded":"2024-06-09T22:20:57.710040+0000","last_fullsized":"2024-06-09T22:20:57.710040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:52:22.214832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"99'2","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694528+0000","last_change":"2024-06-09T22:16:43.210448+0000","last_active":"2024-06-09T22:20:57.694528+0000","last_peered":"2024-06-09T22:20:57.694528+0000","last_clean":"2024-06-09T22:20:57.694528+0000","last_became_active":"2024-06-09T22:16:43.209980+0000","last_became_peered":"2024-06-09T22:16:43.209980+0000","last_unstale":"2024-06-09T22:20:57.694528+0000","last_undegraded":"2024-06-09T22:20:57.694528+0000","last_fullsized":"2024-06-09T22:20:57.694528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:58:15.691207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692864+0000","last_change":"2024-06-09T22:16:53.340644+0000","last_active":"2024-06-09T22:20:57.692864+0000","last_peered":"2024-06-09T22:20:57.692864+0000","last_clean":"2024-06-09T22:20:57.692864+0000","last_became_active":"2024-06-09T22:16:53.339771+0000","last_became_peered":"2024-06-09T22:16:53.339771+0000","last_unstale":"2024-06-09T22:20:57.692864+0000","last_undegraded":"2024-06-09T22:20:57.692864+0000","last_fullsized":"2024-06-09T22:20:57.692864+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:45:23.281387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"97'7","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694859+0000","last_change":"2024-06-09T22:16:40.040461+0000","last_active":"2024-06-09T22:20:57.694859+0000","last_peered":"2024-06-09T22:20:57.694859+0000","last_clean":"2024-06-09T22:20:57.694859+0000","last_became_active":"2024-06-09T22:16:40.040168+0000","last_became_peered":"2024-06-09T22:16:40.040168+0000","last_unstale":"2024-06-09T22:20:57.694859+0000","last_undegraded":"2024-06-09T22:20:57.694859+0000","last_fullsized":"2024-06-09T22:20:57.694859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:28.936631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709959+0000","last_change":"2024-06-09T22:16:43.208779+0000","last_active":"2024-06-09T22:20:57.709959+0000","last_peered":"2024-06-09T22:20:57.709959+0000","last_clean":"2024-06-09T22:20:57.709959+0000","last_became_active":"2024-06-09T22:16:43.208136+0000","last_became_peered":"2024-06-09T22:16:43.208136+0000","last_unstale":"2024-06-09T22:20:57.709959+0000","last_undegraded":"2024-06-09T22:20:57.709959+0000","last_fullsized":"2024-06-09T22:20:57.709959+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:22:39.721069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"97'1","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703384+0000","last_change":"2024-06-09T22:16:53.331241+0000","last_active":"2024-06-09T22:20:57.703384+0000","last_peered":"2024-06-09T22:20:57.703384+0000","last_clean":"2024-06-09T22:20:57.703384+0000","last_became_active":"2024-06-09T22:16:53.329589+0000","last_became_peered":"2024-06-09T22:16:53.329589+0000","last_unstale":"2024-06-09T22:20:57.703384+0000","last_undegraded":"2024-06-09T22:20:57.703384+0000","last_fullsized":"2024-06-09T22:20:57.703384+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:56:21.319711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"100'4","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:58.422275+0000","last_change":"2024-06-09T22:16:40.042491+0000","last_active":"2024-06-09T22:20:58.422275+0000","last_peered":"2024-06-09T22:20:58.422275+0000","last_clean":"2024-06-09T22:20:58.422275+0000","last_became_active":"2024-06-09T22:16:40.041846+0000","last_became_peered":"2024-06-09T22:16:40.041846+0000","last_unstale":"2024-06-09T22:20:58.422275+0000","last_undegraded":"2024-06-09T22:20:58.422275+0000","last_fullsized":"2024-06-09T22:20:58.422275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:59.144003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695873+0000","last_change":"2024-06-09T22:16:43.209195+0000","last_active":"2024-06-09T22:20:57.695873+0000","last_peered":"2024-06-09T22:20:57.695873+0000","last_clean":"2024-06-09T22:20:57.695873+0000","last_became_active":"2024-06-09T22:16:43.208797+0000","last_became_peered":"2024-06-09T22:16:43.208797+0000","last_unstale":"2024-06-09T22:20:57.695873+0000","last_undegraded":"2024-06-09T22:20:57.695873+0000","last_fullsized":"2024-06-09T22:20:57.695873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:05:31.689900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703290+0000","last_change":"2024-06-09T22:16:53.342173+0000","last_active":"2024-06-09T22:20:57.703290+0000","last_peered":"2024-06-09T22:20:57.703290+0000","last_clean":"2024-06-09T22:20:57.703290+0000","last_became_active":"2024-06-09T22:16:53.341712+0000","last_became_peered":"2024-06-09T22:16:53.341712+0000","last_unstale":"2024-06-09T22:20:57.703290+0000","last_undegraded":"2024-06-09T22:20:57.703290+0000","last_fullsized":"2024-06-09T22:20:57.703290+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:53:12.574764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"97'4","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694627+0000","last_change":"2024-06-09T22:16:40.044238+0000","last_active":"2024-06-09T22:20:57.694627+0000","last_peered":"2024-06-09T22:20:57.694627+0000","last_clean":"2024-06-09T22:20:57.694627+0000","last_became_active":"2024-06-09T22:16:40.043857+0000","last_became_peered":"2024-06-09T22:16:40.043857+0000","last_unstale":"2024-06-09T22:20:57.694627+0000","last_undegraded":"2024-06-09T22:20:57.694627+0000","last_fullsized":"2024-06-09T22:20:57.694627+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:44:45.743307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"99'5","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597096+0000","last_change":"2024-06-09T22:16:43.202363+0000","last_active":"2024-06-09T22:20:57.597096+0000","last_peered":"2024-06-09T22:20:57.597096+0000","last_clean":"2024-06-09T22:20:57.597096+0000","last_became_active":"2024-06-09T22:16:43.202029+0000","last_became_peered":"2024-06-09T22:16:43.202029+0000","last_unstale":"2024-06-09T22:20:57.597096+0000","last_undegraded":"2024-06-09T22:20:57.597096+0000","last_fullsized":"2024-06-09T22:20:57.597096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:11:19.513055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710066+0000","last_change":"2024-06-09T22:16:53.333837+0000","last_active":"2024-06-09T22:20:57.710066+0000","last_peered":"2024-06-09T22:20:57.710066+0000","last_clean":"2024-06-09T22:20:57.710066+0000","last_became_active":"2024-06-09T22:16:53.333362+0000","last_became_peered":"2024-06-09T22:16:53.333362+0000","last_unstale":"2024-06-09T22:20:57.710066+0000","last_undegraded":"2024-06-09T22:20:57.710066+0000","last_fullsized":"2024-06-09T22:20:57.710066+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:47:04.463040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"100'8","reported_seq":266,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:16.727233+0000","last_change":"2024-06-09T22:16:40.364834+0000","last_active":"2024-06-09T22:21:16.727233+0000","last_peered":"2024-06-09T22:21:16.727233+0000","last_clean":"2024-06-09T22:21:16.727233+0000","last_became_active":"2024-06-09T22:16:40.364433+0000","last_became_peered":"2024-06-09T22:16:40.364433+0000","last_unstale":"2024-06-09T22:21:16.727233+0000","last_undegraded":"2024-06-09T22:21:16.727233+0000","last_fullsized":"2024-06-09T22:21:16.727233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:43:01.967911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2091,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":212,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"99'7","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597232+0000","last_change":"2024-06-09T22:16:43.201183+0000","last_active":"2024-06-09T22:20:57.597232+0000","last_peered":"2024-06-09T22:20:57.597232+0000","last_clean":"2024-06-09T22:20:57.597232+0000","last_became_active":"2024-06-09T22:16:43.200774+0000","last_became_peered":"2024-06-09T22:16:43.200774+0000","last_unstale":"2024-06-09T22:20:57.597232+0000","last_undegraded":"2024-06-09T22:20:57.597232+0000","last_fullsized":"2024-06-09T22:20:57.597232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:07:55.772066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710583+0000","last_change":"2024-06-09T22:18:26.699190+0000","last_active":"2024-06-09T22:20:57.710583+0000","last_peered":"2024-06-09T22:20:57.710583+0000","last_clean":"2024-06-09T22:20:57.710583+0000","last_became_active":"2024-06-09T22:18:26.698614+0000","last_became_peered":"2024-06-09T22:18:26.698614+0000","last_unstale":"2024-06-09T22:20:57.710583+0000","last_undegraded":"2024-06-09T22:20:57.710583+0000","last_fullsized":"2024-06-09T22:20:57.710583+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:45.343605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,1,4],"acting":[0,10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"100'67","reported_seq":145,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:12.818366+0000","last_change":"2024-06-09T22:16:40.040582+0000","last_active":"2024-06-09T22:21:12.818366+0000","last_peered":"2024-06-09T22:21:12.818366+0000","last_clean":"2024-06-09T22:21:12.818366+0000","last_became_active":"2024-06-09T22:16:40.040152+0000","last_became_peered":"2024-06-09T22:16:40.040152+0000","last_unstale":"2024-06-09T22:21:12.818366+0000","last_undegraded":"2024-06-09T22:21:12.818366+0000","last_fullsized":"2024-06-09T22:21:12.818366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:50:00.878096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"99'6","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596961+0000","last_change":"2024-06-09T22:16:43.206211+0000","last_active":"2024-06-09T22:20:57.596961+0000","last_peered":"2024-06-09T22:20:57.596961+0000","last_clean":"2024-06-09T22:20:57.596961+0000","last_became_active":"2024-06-09T22:16:43.205814+0000","last_became_peered":"2024-06-09T22:16:43.205814+0000","last_unstale":"2024-06-09T22:20:57.596961+0000","last_undegraded":"2024-06-09T22:20:57.596961+0000","last_fullsized":"2024-06-09T22:20:57.596961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:20:19.058475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703101+0000","last_change":"2024-06-09T22:16:53.337695+0000","last_active":"2024-06-09T22:20:57.703101+0000","last_peered":"2024-06-09T22:20:57.703101+0000","last_clean":"2024-06-09T22:20:57.703101+0000","last_became_active":"2024-06-09T22:16:53.334872+0000","last_became_peered":"2024-06-09T22:16:53.334872+0000","last_unstale":"2024-06-09T22:20:57.703101+0000","last_undegraded":"2024-06-09T22:20:57.703101+0000","last_fullsized":"2024-06-09T22:20:57.703101+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:50:36.617736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"97'5","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596104+0000","last_change":"2024-06-09T22:16:40.036603+0000","last_active":"2024-06-09T22:20:57.596104+0000","last_peered":"2024-06-09T22:20:57.596104+0000","last_clean":"2024-06-09T22:20:57.596104+0000","last_became_active":"2024-06-09T22:16:40.036274+0000","last_became_peered":"2024-06-09T22:16:40.036274+0000","last_unstale":"2024-06-09T22:20:57.596104+0000","last_undegraded":"2024-06-09T22:20:57.596104+0000","last_fullsized":"2024-06-09T22:20:57.596104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:31.652897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"99'9","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597041+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:20:57.597041+0000","last_peered":"2024-06-09T22:20:57.597041+0000","last_clean":"2024-06-09T22:20:57.597041+0000","last_became_active":"2024-06-09T22:16:43.204197+0000","last_became_peered":"2024-06-09T22:16:43.204197+0000","last_unstale":"2024-06-09T22:20:57.597041+0000","last_undegraded":"2024-06-09T22:20:57.597041+0000","last_fullsized":"2024-06-09T22:20:57.597041+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:38:49.850120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710013+0000","last_change":"2024-06-09T22:16:43.208477+0000","last_active":"2024-06-09T22:20:57.710013+0000","last_peered":"2024-06-09T22:20:57.710013+0000","last_clean":"2024-06-09T22:20:57.710013+0000","last_became_active":"2024-06-09T22:16:43.206018+0000","last_became_peered":"2024-06-09T22:16:43.206018+0000","last_unstale":"2024-06-09T22:20:57.710013+0000","last_undegraded":"2024-06-09T22:20:57.710013+0000","last_fullsized":"2024-06-09T22:20:57.710013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:19:49.870825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"99'12","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597095+0000","last_change":"2024-06-09T22:16:43.206809+0000","last_active":"2024-06-09T22:20:57.597095+0000","last_peered":"2024-06-09T22:20:57.597095+0000","last_clean":"2024-06-09T22:20:57.597095+0000","last_became_active":"2024-06-09T22:16:43.206175+0000","last_became_peered":"2024-06-09T22:16:43.206175+0000","last_unstale":"2024-06-09T22:20:57.597095+0000","last_undegraded":"2024-06-09T22:20:57.597095+0000","last_fullsized":"2024-06-09T22:20:57.597095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:39:22.502266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596879+0000","last_change":"2024-06-09T22:16:43.202163+0000","last_active":"2024-06-09T22:20:57.596879+0000","last_peered":"2024-06-09T22:20:57.596879+0000","last_clean":"2024-06-09T22:20:57.596879+0000","last_became_active":"2024-06-09T22:16:43.201839+0000","last_became_peered":"2024-06-09T22:16:43.201839+0000","last_unstale":"2024-06-09T22:20:57.596879+0000","last_undegraded":"2024-06-09T22:20:57.596879+0000","last_fullsized":"2024-06-09T22:20:57.596879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:47:46.723289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710813+0000","last_change":"2024-06-09T22:16:43.202734+0000","last_active":"2024-06-09T22:20:57.710813+0000","last_peered":"2024-06-09T22:20:57.710813+0000","last_clean":"2024-06-09T22:20:57.710813+0000","last_became_active":"2024-06-09T22:16:43.201638+0000","last_became_peered":"2024-06-09T22:16:43.201638+0000","last_unstale":"2024-06-09T22:20:57.710813+0000","last_undegraded":"2024-06-09T22:20:57.710813+0000","last_fullsized":"2024-06-09T22:20:57.710813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:16.360985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"99'4","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595768+0000","last_change":"2024-06-09T22:16:43.199113+0000","last_active":"2024-06-09T22:20:57.595768+0000","last_peered":"2024-06-09T22:20:57.595768+0000","last_clean":"2024-06-09T22:20:57.595768+0000","last_became_active":"2024-06-09T22:16:43.198804+0000","last_became_peered":"2024-06-09T22:16:43.198804+0000","last_unstale":"2024-06-09T22:20:57.595768+0000","last_undegraded":"2024-06-09T22:20:57.595768+0000","last_fullsized":"2024-06-09T22:20:57.595768+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:18:08.999477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"99'6","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710407+0000","last_change":"2024-06-09T22:16:43.204445+0000","last_active":"2024-06-09T22:20:57.710407+0000","last_peered":"2024-06-09T22:20:57.710407+0000","last_clean":"2024-06-09T22:20:57.710407+0000","last_became_active":"2024-06-09T22:16:43.203081+0000","last_became_peered":"2024-06-09T22:16:43.203081+0000","last_unstale":"2024-06-09T22:20:57.710407+0000","last_undegraded":"2024-06-09T22:20:57.710407+0000","last_fullsized":"2024-06-09T22:20:57.710407+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:39.303813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"99'15","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597639+0000","last_change":"2024-06-09T22:16:43.211402+0000","last_active":"2024-06-09T22:20:57.597639+0000","last_peered":"2024-06-09T22:20:57.597639+0000","last_clean":"2024-06-09T22:20:57.597639+0000","last_became_active":"2024-06-09T22:16:43.210871+0000","last_became_peered":"2024-06-09T22:16:43.210871+0000","last_unstale":"2024-06-09T22:20:57.597639+0000","last_undegraded":"2024-06-09T22:20:57.597639+0000","last_fullsized":"2024-06-09T22:20:57.597639+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:07:21.467617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"99'2","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696286+0000","last_change":"2024-06-09T22:16:43.201159+0000","last_active":"2024-06-09T22:20:57.696286+0000","last_peered":"2024-06-09T22:20:57.696286+0000","last_clean":"2024-06-09T22:20:57.696286+0000","last_became_active":"2024-06-09T22:16:43.200850+0000","last_became_peered":"2024-06-09T22:16:43.200850+0000","last_unstale":"2024-06-09T22:20:57.696286+0000","last_undegraded":"2024-06-09T22:20:57.696286+0000","last_fullsized":"2024-06-09T22:20:57.696286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:20.515309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"99'6","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703351+0000","last_change":"2024-06-09T22:16:43.196830+0000","last_active":"2024-06-09T22:20:57.703351+0000","last_peered":"2024-06-09T22:20:57.703351+0000","last_clean":"2024-06-09T22:20:57.703351+0000","last_became_active":"2024-06-09T22:16:43.195682+0000","last_became_peered":"2024-06-09T22:16:43.195682+0000","last_unstale":"2024-06-09T22:20:57.703351+0000","last_undegraded":"2024-06-09T22:20:57.703351+0000","last_fullsized":"2024-06-09T22:20:57.703351+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:34:18.770235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694976+0000","last_change":"2024-06-09T22:16:43.201959+0000","last_active":"2024-06-09T22:20:57.694976+0000","last_peered":"2024-06-09T22:20:57.694976+0000","last_clean":"2024-06-09T22:20:57.694976+0000","last_became_active":"2024-06-09T22:16:43.201669+0000","last_became_peered":"2024-06-09T22:16:43.201669+0000","last_unstale":"2024-06-09T22:20:57.694976+0000","last_undegraded":"2024-06-09T22:20:57.694976+0000","last_fullsized":"2024-06-09T22:20:57.694976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:48:57.348041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"99'10","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710243+0000","last_change":"2024-06-09T22:16:43.201823+0000","last_active":"2024-06-09T22:20:57.710243+0000","last_peered":"2024-06-09T22:20:57.710243+0000","last_clean":"2024-06-09T22:20:57.710243+0000","last_became_active":"2024-06-09T22:16:43.201434+0000","last_became_peered":"2024-06-09T22:16:43.201434+0000","last_unstale":"2024-06-09T22:20:57.710243+0000","last_undegraded":"2024-06-09T22:20:57.710243+0000","last_fullsized":"2024-06-09T22:20:57.710243+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:56:13.540895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"99'11","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595673+0000","last_change":"2024-06-09T22:16:43.197734+0000","last_active":"2024-06-09T22:20:57.595673+0000","last_peered":"2024-06-09T22:20:57.595673+0000","last_clean":"2024-06-09T22:20:57.595673+0000","last_became_active":"2024-06-09T22:16:43.197442+0000","last_became_peered":"2024-06-09T22:16:43.197442+0000","last_unstale":"2024-06-09T22:20:57.595673+0000","last_undegraded":"2024-06-09T22:20:57.595673+0000","last_fullsized":"2024-06-09T22:20:57.595673+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:35:40.720718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710386+0000","last_change":"2024-06-09T22:16:43.210375+0000","last_active":"2024-06-09T22:20:57.710386+0000","last_peered":"2024-06-09T22:20:57.710386+0000","last_clean":"2024-06-09T22:20:57.710386+0000","last_became_active":"2024-06-09T22:16:43.209647+0000","last_became_peered":"2024-06-09T22:16:43.209647+0000","last_unstale":"2024-06-09T22:20:57.710386+0000","last_undegraded":"2024-06-09T22:20:57.710386+0000","last_fullsized":"2024-06-09T22:20:57.710386+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:11:57.593066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107643,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":809307,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2472,"num_read_kb":1471,"num_write":524,"num_write_kb":2408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":823296,"data_stored":2427921,"data_compressed":103473,"data_compressed_allocated":135168,"data_compressed_original":2101248,"omap_allocated":0,"internal_metadata":0},"log_size":401,"ondisk_log_size":401,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383751,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":581230,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645388,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32476,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712676,"statfs":{"total":95995035648,"available":95961780224,"internally_reserved":0,"allocated":1323008,"data_stored":1574104,"data_compressed":86342,"data_compressed_allocated":114688,"data_compressed_original":995328,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808913,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":977483,"data_compressed":54548,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070550,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1159168,"data_stored":973765,"data_compressed":54548,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332187,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32336,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712816,"statfs":{"total":95995035648,"available":95961923584,"internally_reserved":0,"allocated":1187840,"data_stored":1179585,"data_compressed":71140,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495713,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1114112,"data_stored":576065,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36504,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708648,"statfs":{"total":95995035648,"available":95957655552,"internally_reserved":0,"allocated":1196032,"data_stored":654681,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708700,"statfs":{"total":95995035648,"available":95957708800,"internally_reserved":0,"allocated":1138688,"data_stored":578052,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411444,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1208320,"data_stored":656675,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182511,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28192,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716960,"statfs":{"total":95995035648,"available":95966167040,"internally_reserved":0,"allocated":1196032,"data_stored":1296831,"data_compressed":69750,"data_compressed_allocated":94208,"data_compressed_original":798720,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920875,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32276,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712876,"statfs":{"total":95995035648,"available":95961985024,"internally_reserved":0,"allocated":1122304,"data_stored":580978,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757350,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36556,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708596,"statfs":{"total":95995035648,"available":95957602304,"internally_reserved":0,"allocated":1241088,"data_stored":1501780,"data_compressed":86342,"data_compressed_allocated":114688,"data_compressed_original":995328,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":79728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":81722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":721878,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":332363,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":408783,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-09T22:21:21.657 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-09T22:21:21.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-09T22:21:21.682 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:21 smithi104 ceph-mon[32389]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-06-09T22:21:21.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:21 smithi153 ceph-mon[36599]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-06-09T22:21:21.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:21 smithi194 ceph-mon[36856]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-06-09T22:21:21.899 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:22.730 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:22 smithi104 ceph-mon[32389]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:22.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:22 smithi153 ceph-mon[36599]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:22.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:22 smithi194 ceph-mon[36856]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:23.718 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:23 smithi104 ceph-mon[32389]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-06-09T22:21:23.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.738+0000 7fa1568f2700 1 -- 172.21.15.104:0/2459232997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa150101af0 msgr2=0x7fa150101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:23.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.738+0000 7fa1568f2700 1 --2- 172.21.15.104:0/2459232997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa150101af0 0x7fa150101f70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa138009990 tx=0x7fa138009ca0 comp rx=0 tx=0).stop 2024-06-09T22:21:23.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.738+0000 7fa1568f2700 1 -- 172.21.15.104:0/2459232997 shutdown_connections 2024-06-09T22:21:23.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.738+0000 7fa1568f2700 1 --2- 172.21.15.104:0/2459232997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1501024b0 0x7fa15010e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:23.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.738+0000 7fa1568f2700 1 --2- 172.21.15.104:0/2459232997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1501063c0 0x7fa1501067a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:23.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.738+0000 7fa1568f2700 1 --2- 172.21.15.104:0/2459232997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa150101af0 0x7fa150101f70 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:23.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.738+0000 7fa1568f2700 1 -- 172.21.15.104:0/2459232997 >> 172.21.15.104:0/2459232997 conn(0x7fa150077990 msgr2=0x7fa1500fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:23.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.738+0000 7fa1568f2700 1 -- 172.21.15.104:0/2459232997 shutdown_connections 2024-06-09T22:21:23.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.738+0000 7fa1568f2700 1 -- 172.21.15.104:0/2459232997 wait complete. 2024-06-09T22:21:23.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.739+0000 7fa1568f2700 1 Processor -- start 2024-06-09T22:21:23.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.739+0000 7fa1568f2700 1 -- start start 2024-06-09T22:21:23.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.739+0000 7fa1568f2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa150101af0 0x7fa15019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:23.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.739+0000 7fa1568f2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1501024b0 0x7fa15019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:23.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.739+0000 7fa1568f2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1501063c0 0x7fa150198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:23.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.739+0000 7fa1568f2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa1501116b0 con 0x7fa150101af0 2024-06-09T22:21:23.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.739+0000 7fa1568f2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa150111530 con 0x7fa1501063c0 2024-06-09T22:21:23.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.739+0000 7fa1568f2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa150111830 con 0x7fa1501024b0 2024-06-09T22:21:23.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.739+0000 7fa14ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa150101af0 0x7fa15019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:23.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.739+0000 7fa14ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa150101af0 0x7fa15019e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55500/0 (socket says 172.21.15.104:55500) 2024-06-09T22:21:23.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.739+0000 7fa14ffff700 1 -- 172.21.15.104:0/1587384784 learned_addr learned my addr 172.21.15.104:0/1587384784 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:23.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.739+0000 7fa147fff700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1501024b0 0x7fa15019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:23.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.740+0000 7fa154e8f700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1501063c0 0x7fa150198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:23.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.740+0000 7fa14ffff700 1 -- 172.21.15.104:0/1587384784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1501024b0 msgr2=0x7fa15019ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:23.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.740+0000 7fa14ffff700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1501024b0 0x7fa15019ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:23.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.740+0000 7fa14ffff700 1 -- 172.21.15.104:0/1587384784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1501063c0 msgr2=0x7fa150198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:23.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.740+0000 7fa14ffff700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1501063c0 0x7fa150198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:23.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.740+0000 7fa14ffff700 1 -- 172.21.15.104:0/1587384784 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa150199190 con 0x7fa150101af0 2024-06-09T22:21:23.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.740+0000 7fa14ffff700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa150101af0 0x7fa15019e820 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa14000e9a0 tx=0x7fa14000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:23.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.740+0000 7fa14dffb700 1 -- 172.21.15.104:0/1587384784 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14000cbd0 con 0x7fa150101af0 2024-06-09T22:21:23.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.740+0000 7fa1568f2700 1 -- 172.21.15.104:0/1587384784 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa150199480 con 0x7fa150101af0 2024-06-09T22:21:23.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.740+0000 7fa14dffb700 1 -- 172.21.15.104:0/1587384784 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa14000cd70 con 0x7fa150101af0 2024-06-09T22:21:23.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.740+0000 7fa14dffb700 1 -- 172.21.15.104:0/1587384784 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa140018750 con 0x7fa150101af0 2024-06-09T22:21:23.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.740+0000 7fa1568f2700 1 -- 172.21.15.104:0/1587384784 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1501a56f0 con 0x7fa150101af0 2024-06-09T22:21:23.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.741+0000 7fa1568f2700 1 -- 172.21.15.104:0/1587384784 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa150069180 con 0x7fa150101af0 2024-06-09T22:21:23.743 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:23 smithi153 ceph-mon[36599]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-06-09T22:21:23.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.743+0000 7fa14dffb700 1 -- 172.21.15.104:0/1587384784 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7fa140010430 con 0x7fa150101af0 2024-06-09T22:21:23.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.744+0000 7fa14dffb700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa13c0a2eb0 0x7fa13c0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:23.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.744+0000 7fa147fff700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa13c0a2eb0 0x7fa13c0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:23.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.744+0000 7fa14dffb700 1 -- 172.21.15.104:0/1587384784 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fa1400d01f0 con 0x7fa150101af0 2024-06-09T22:21:23.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.744+0000 7fa147fff700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa13c0a2eb0 0x7fa13c0a5370 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa13800b2e0 tx=0x7fa138005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:23.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.747+0000 7fa14dffb700 1 -- 172.21.15.104:0/1587384784 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa14009c850 con 0x7fa150101af0 2024-06-09T22:21:23.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:23 smithi194 ceph-mon[36856]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-06-09T22:21:23.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.939+0000 7fa1568f2700 1 -- 172.21.15.104:0/1587384784 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa1500628d0 con 0x7fa13c0a2eb0 2024-06-09T22:21:23.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.948+0000 7fa14dffb700 1 -- 172.21.15.104:0/1587384784 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339896 (secure 0 0 0) 0x7fa1500628d0 con 0x7fa13c0a2eb0 2024-06-09T22:21:23.951 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:23.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.952+0000 7fa1568f2700 1 -- 172.21.15.104:0/1587384784 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa13c0a2eb0 msgr2=0x7fa13c0a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:23.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.952+0000 7fa1568f2700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa13c0a2eb0 0x7fa13c0a5370 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa13800b2e0 tx=0x7fa138005960 comp rx=0 tx=0).stop 2024-06-09T22:21:23.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.952+0000 7fa1568f2700 1 -- 172.21.15.104:0/1587384784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa150101af0 msgr2=0x7fa15019e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:23.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.952+0000 7fa1568f2700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa150101af0 0x7fa15019e820 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa14000e9a0 tx=0x7fa14000ecb0 comp rx=0 tx=0).stop 2024-06-09T22:21:23.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.952+0000 7fa1568f2700 1 -- 172.21.15.104:0/1587384784 shutdown_connections 2024-06-09T22:21:23.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.952+0000 7fa1568f2700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa13c0a2eb0 0x7fa13c0a5370 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:23.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.952+0000 7fa1568f2700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa150101af0 0x7fa15019e820 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:23.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.952+0000 7fa1568f2700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1501063c0 0x7fa150198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:23.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.952+0000 7fa1568f2700 1 --2- 172.21.15.104:0/1587384784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1501024b0 0x7fa15019ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:23.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.952+0000 7fa1568f2700 1 -- 172.21.15.104:0/1587384784 >> 172.21.15.104:0/1587384784 conn(0x7fa150077990 msgr2=0x7fa150100c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:23.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.952+0000 7fa1568f2700 1 -- 172.21.15.104:0/1587384784 shutdown_connections 2024-06-09T22:21:23.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:23.952+0000 7fa1568f2700 1 -- 172.21.15.104:0/1587384784 wait complete. 2024-06-09T22:21:23.955 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-06-09T22:21:24.862 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-06-09T22:21:23.614216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1399909,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2729,"num_read_kb":1806,"num_write":727,"num_write_kb":5348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1051,"ondisk_log_size":1051,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383836,"kb_used_data":13844,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557988,"statfs":{"total":1151940427776,"available":1151547379712,"internally_reserved":0,"allocated":14176256,"data_stored":11131229,"data_compressed":634224,"data_compressed_allocated":860160,"data_compressed_original":5050368,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":197770,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":376,"num_read_kb":289,"num_write":128,"num_write_kb":917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002922"},"pg_stats":[{"pgid":"3.3f","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710445+0000","last_change":"2024-06-09T22:16:43.212998+0000","last_active":"2024-06-09T22:20:57.710445+0000","last_peered":"2024-06-09T22:20:57.710445+0000","last_clean":"2024-06-09T22:20:57.710445+0000","last_became_active":"2024-06-09T22:16:43.212614+0000","last_became_peered":"2024-06-09T22:16:43.212614+0000","last_unstale":"2024-06-09T22:20:57.710445+0000","last_undegraded":"2024-06-09T22:20:57.710445+0000","last_fullsized":"2024-06-09T22:20:57.710445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:22.728486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"99'4","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711160+0000","last_change":"2024-06-09T22:16:43.208746+0000","last_active":"2024-06-09T22:20:57.711160+0000","last_peered":"2024-06-09T22:20:57.711160+0000","last_clean":"2024-06-09T22:20:57.711160+0000","last_became_active":"2024-06-09T22:16:43.208458+0000","last_became_peered":"2024-06-09T22:16:43.208458+0000","last_unstale":"2024-06-09T22:20:57.711160+0000","last_undegraded":"2024-06-09T22:20:57.711160+0000","last_fullsized":"2024-06-09T22:20:57.711160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:24:35.350449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710878+0000","last_change":"2024-06-09T22:16:43.200770+0000","last_active":"2024-06-09T22:20:57.710878+0000","last_peered":"2024-06-09T22:20:57.710878+0000","last_clean":"2024-06-09T22:20:57.710878+0000","last_became_active":"2024-06-09T22:16:43.200449+0000","last_became_peered":"2024-06-09T22:16:43.200449+0000","last_unstale":"2024-06-09T22:20:57.710878+0000","last_undegraded":"2024-06-09T22:20:57.710878+0000","last_fullsized":"2024-06-09T22:20:57.710878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:51:09.017260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703458+0000","last_change":"2024-06-09T22:16:43.203844+0000","last_active":"2024-06-09T22:20:57.703458+0000","last_peered":"2024-06-09T22:20:57.703458+0000","last_clean":"2024-06-09T22:20:57.703458+0000","last_became_active":"2024-06-09T22:16:43.203431+0000","last_became_peered":"2024-06-09T22:16:43.203431+0000","last_unstale":"2024-06-09T22:20:57.703458+0000","last_undegraded":"2024-06-09T22:20:57.703458+0000","last_fullsized":"2024-06-09T22:20:57.703458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:51:36.913312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"99'4","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694538+0000","last_change":"2024-06-09T22:16:43.207597+0000","last_active":"2024-06-09T22:20:57.694538+0000","last_peered":"2024-06-09T22:20:57.694538+0000","last_clean":"2024-06-09T22:20:57.694538+0000","last_became_active":"2024-06-09T22:16:43.207266+0000","last_became_peered":"2024-06-09T22:16:43.207266+0000","last_unstale":"2024-06-09T22:20:57.694538+0000","last_undegraded":"2024-06-09T22:20:57.694538+0000","last_fullsized":"2024-06-09T22:20:57.694538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:47:05.558403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"99'6","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702988+0000","last_change":"2024-06-09T22:17:26.474121+0000","last_active":"2024-06-09T22:20:57.702988+0000","last_peered":"2024-06-09T22:20:57.702988+0000","last_clean":"2024-06-09T22:20:57.702988+0000","last_became_active":"2024-06-09T22:17:26.473619+0000","last_became_peered":"2024-06-09T22:17:26.473619+0000","last_unstale":"2024-06-09T22:20:57.702988+0000","last_undegraded":"2024-06-09T22:20:57.702988+0000","last_fullsized":"2024-06-09T22:20:57.702988+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:56:42.171235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"99'5","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596592+0000","last_change":"2024-06-09T22:16:43.208815+0000","last_active":"2024-06-09T22:20:57.596592+0000","last_peered":"2024-06-09T22:20:57.596592+0000","last_clean":"2024-06-09T22:20:57.596592+0000","last_became_active":"2024-06-09T22:16:43.208433+0000","last_became_peered":"2024-06-09T22:16:43.208433+0000","last_unstale":"2024-06-09T22:20:57.596592+0000","last_undegraded":"2024-06-09T22:20:57.596592+0000","last_fullsized":"2024-06-09T22:20:57.596592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:36:26.585037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694783+0000","last_change":"2024-06-09T22:16:43.210195+0000","last_active":"2024-06-09T22:20:57.694783+0000","last_peered":"2024-06-09T22:20:57.694783+0000","last_clean":"2024-06-09T22:20:57.694783+0000","last_became_active":"2024-06-09T22:16:43.209892+0000","last_became_peered":"2024-06-09T22:16:43.209892+0000","last_unstale":"2024-06-09T22:20:57.694783+0000","last_undegraded":"2024-06-09T22:20:57.694783+0000","last_fullsized":"2024-06-09T22:20:57.694783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:44.088212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710618+0000","last_change":"2024-06-09T22:16:43.202517+0000","last_active":"2024-06-09T22:20:57.710618+0000","last_peered":"2024-06-09T22:20:57.710618+0000","last_clean":"2024-06-09T22:20:57.710618+0000","last_became_active":"2024-06-09T22:16:43.201796+0000","last_became_peered":"2024-06-09T22:16:43.201796+0000","last_unstale":"2024-06-09T22:20:57.710618+0000","last_undegraded":"2024-06-09T22:20:57.710618+0000","last_fullsized":"2024-06-09T22:20:57.710618+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:30.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"99'16","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703409+0000","last_change":"2024-06-09T22:16:43.212732+0000","last_active":"2024-06-09T22:20:57.703409+0000","last_peered":"2024-06-09T22:20:57.703409+0000","last_clean":"2024-06-09T22:20:57.703409+0000","last_became_active":"2024-06-09T22:16:43.212048+0000","last_became_peered":"2024-06-09T22:16:43.212048+0000","last_unstale":"2024-06-09T22:20:57.703409+0000","last_undegraded":"2024-06-09T22:20:57.703409+0000","last_fullsized":"2024-06-09T22:20:57.703409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:47.383291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597025+0000","last_change":"2024-06-09T22:16:43.203778+0000","last_active":"2024-06-09T22:20:57.597025+0000","last_peered":"2024-06-09T22:20:57.597025+0000","last_clean":"2024-06-09T22:20:57.597025+0000","last_became_active":"2024-06-09T22:16:43.202954+0000","last_became_peered":"2024-06-09T22:16:43.202954+0000","last_unstale":"2024-06-09T22:20:57.597025+0000","last_undegraded":"2024-06-09T22:20:57.597025+0000","last_fullsized":"2024-06-09T22:20:57.597025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:56:53.708645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703526+0000","last_change":"2024-06-09T22:16:43.211425+0000","last_active":"2024-06-09T22:20:57.703526+0000","last_peered":"2024-06-09T22:20:57.703526+0000","last_clean":"2024-06-09T22:20:57.703526+0000","last_became_active":"2024-06-09T22:16:43.211001+0000","last_became_peered":"2024-06-09T22:16:43.211001+0000","last_unstale":"2024-06-09T22:20:57.703526+0000","last_undegraded":"2024-06-09T22:20:57.703526+0000","last_fullsized":"2024-06-09T22:20:57.703526+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:20:59.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"99'10","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595794+0000","last_change":"2024-06-09T22:16:43.208334+0000","last_active":"2024-06-09T22:20:57.595794+0000","last_peered":"2024-06-09T22:20:57.595794+0000","last_clean":"2024-06-09T22:20:57.595794+0000","last_became_active":"2024-06-09T22:16:43.207993+0000","last_became_peered":"2024-06-09T22:16:43.207993+0000","last_unstale":"2024-06-09T22:20:57.595794+0000","last_undegraded":"2024-06-09T22:20:57.595794+0000","last_fullsized":"2024-06-09T22:20:57.595794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:07:05.740748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710461+0000","last_change":"2024-06-09T22:16:43.207211+0000","last_active":"2024-06-09T22:20:57.710461+0000","last_peered":"2024-06-09T22:20:57.710461+0000","last_clean":"2024-06-09T22:20:57.710461+0000","last_became_active":"2024-06-09T22:16:43.206889+0000","last_became_peered":"2024-06-09T22:16:43.206889+0000","last_unstale":"2024-06-09T22:20:57.710461+0000","last_undegraded":"2024-06-09T22:20:57.710461+0000","last_fullsized":"2024-06-09T22:20:57.710461+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:25:47.461030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710404+0000","last_change":"2024-06-09T22:16:43.202202+0000","last_active":"2024-06-09T22:20:57.710404+0000","last_peered":"2024-06-09T22:20:57.710404+0000","last_clean":"2024-06-09T22:20:57.710404+0000","last_became_active":"2024-06-09T22:16:43.201834+0000","last_became_peered":"2024-06-09T22:16:43.201834+0000","last_unstale":"2024-06-09T22:20:57.710404+0000","last_undegraded":"2024-06-09T22:20:57.710404+0000","last_fullsized":"2024-06-09T22:20:57.710404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:31:51.296627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"99'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709927+0000","last_change":"2024-06-09T22:16:43.200637+0000","last_active":"2024-06-09T22:20:57.709927+0000","last_peered":"2024-06-09T22:20:57.709927+0000","last_clean":"2024-06-09T22:20:57.709927+0000","last_became_active":"2024-06-09T22:16:43.200268+0000","last_became_peered":"2024-06-09T22:16:43.200268+0000","last_unstale":"2024-06-09T22:20:57.709927+0000","last_undegraded":"2024-06-09T22:20:57.709927+0000","last_fullsized":"2024-06-09T22:20:57.709927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:25:59.881453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710229+0000","last_change":"2024-06-09T22:16:43.209212+0000","last_active":"2024-06-09T22:20:57.710229+0000","last_peered":"2024-06-09T22:20:57.710229+0000","last_clean":"2024-06-09T22:20:57.710229+0000","last_became_active":"2024-06-09T22:16:43.207789+0000","last_became_peered":"2024-06-09T22:16:43.207789+0000","last_unstale":"2024-06-09T22:20:57.710229+0000","last_undegraded":"2024-06-09T22:20:57.710229+0000","last_fullsized":"2024-06-09T22:20:57.710229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:17.316595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692756+0000","last_change":"2024-06-09T22:16:43.210935+0000","last_active":"2024-06-09T22:20:57.692756+0000","last_peered":"2024-06-09T22:20:57.692756+0000","last_clean":"2024-06-09T22:20:57.692756+0000","last_became_active":"2024-06-09T22:16:43.210631+0000","last_became_peered":"2024-06-09T22:16:43.210631+0000","last_unstale":"2024-06-09T22:20:57.692756+0000","last_undegraded":"2024-06-09T22:20:57.692756+0000","last_fullsized":"2024-06-09T22:20:57.692756+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:22:58.887313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"99'7","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597675+0000","last_change":"2024-06-09T22:16:43.198872+0000","last_active":"2024-06-09T22:20:57.597675+0000","last_peered":"2024-06-09T22:20:57.597675+0000","last_clean":"2024-06-09T22:20:57.597675+0000","last_became_active":"2024-06-09T22:16:43.198498+0000","last_became_peered":"2024-06-09T22:16:43.198498+0000","last_unstale":"2024-06-09T22:20:57.597675+0000","last_undegraded":"2024-06-09T22:20:57.597675+0000","last_fullsized":"2024-06-09T22:20:57.597675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:25:28.853048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597223+0000","last_change":"2024-06-09T22:16:53.343656+0000","last_active":"2024-06-09T22:20:57.597223+0000","last_peered":"2024-06-09T22:20:57.597223+0000","last_clean":"2024-06-09T22:20:57.597223+0000","last_became_active":"2024-06-09T22:16:53.343281+0000","last_became_peered":"2024-06-09T22:16:53.343281+0000","last_unstale":"2024-06-09T22:20:57.597223+0000","last_undegraded":"2024-06-09T22:20:57.597223+0000","last_fullsized":"2024-06-09T22:20:57.597223+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:40.961842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"100'14","reported_seq":269,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:16.725776+0000","last_change":"2024-06-09T22:16:40.040145+0000","last_active":"2024-06-09T22:21:16.725776+0000","last_peered":"2024-06-09T22:21:16.725776+0000","last_clean":"2024-06-09T22:21:16.725776+0000","last_became_active":"2024-06-09T22:16:40.039787+0000","last_became_peered":"2024-06-09T22:16:40.039787+0000","last_unstale":"2024-06-09T22:21:16.725776+0000","last_undegraded":"2024-06-09T22:21:16.725776+0000","last_fullsized":"2024-06-09T22:21:16.725776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:52:32.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":209,"num_read_kb":209,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"99'5","reported_seq":23,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692731+0000","last_change":"2024-06-09T22:17:26.774053+0000","last_active":"2024-06-09T22:20:57.692731+0000","last_peered":"2024-06-09T22:20:57.692731+0000","last_clean":"2024-06-09T22:20:57.692731+0000","last_became_active":"2024-06-09T22:17:26.773786+0000","last_became_peered":"2024-06-09T22:17:26.773786+0000","last_unstale":"2024-06-09T22:20:57.692731+0000","last_undegraded":"2024-06-09T22:20:57.692731+0000","last_fullsized":"2024-06-09T22:20:57.692731+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:36.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703244+0000","last_change":"2024-06-09T22:16:53.332048+0000","last_active":"2024-06-09T22:20:57.703244+0000","last_peered":"2024-06-09T22:20:57.703244+0000","last_clean":"2024-06-09T22:20:57.703244+0000","last_became_active":"2024-06-09T22:16:53.330683+0000","last_became_peered":"2024-06-09T22:16:53.330683+0000","last_unstale":"2024-06-09T22:20:57.703244+0000","last_undegraded":"2024-06-09T22:20:57.703244+0000","last_fullsized":"2024-06-09T22:20:57.703244+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:32:23.219061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"97'2","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696082+0000","last_change":"2024-06-09T22:16:40.070193+0000","last_active":"2024-06-09T22:20:57.696082+0000","last_peered":"2024-06-09T22:20:57.696082+0000","last_clean":"2024-06-09T22:20:57.696082+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:20:57.696082+0000","last_undegraded":"2024-06-09T22:20:57.696082+0000","last_fullsized":"2024-06-09T22:20:57.696082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:19:19.787552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711015+0000","last_change":"2024-06-09T22:16:43.199814+0000","last_active":"2024-06-09T22:20:57.711015+0000","last_peered":"2024-06-09T22:20:57.711015+0000","last_clean":"2024-06-09T22:20:57.711015+0000","last_became_active":"2024-06-09T22:16:43.199487+0000","last_became_peered":"2024-06-09T22:16:43.199487+0000","last_unstale":"2024-06-09T22:20:57.711015+0000","last_undegraded":"2024-06-09T22:20:57.711015+0000","last_fullsized":"2024-06-09T22:20:57.711015+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:55.672050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703543+0000","last_change":"2024-06-09T22:16:53.336253+0000","last_active":"2024-06-09T22:20:57.703543+0000","last_peered":"2024-06-09T22:20:57.703543+0000","last_clean":"2024-06-09T22:20:57.703543+0000","last_became_active":"2024-06-09T22:16:53.335869+0000","last_became_peered":"2024-06-09T22:16:53.335869+0000","last_unstale":"2024-06-09T22:20:57.703543+0000","last_undegraded":"2024-06-09T22:20:57.703543+0000","last_fullsized":"2024-06-09T22:20:57.703543+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:28:12.894878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"97'12","reported_seq":60,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695155+0000","last_change":"2024-06-09T22:16:40.045365+0000","last_active":"2024-06-09T22:20:57.695155+0000","last_peered":"2024-06-09T22:20:57.695155+0000","last_clean":"2024-06-09T22:20:57.695155+0000","last_became_active":"2024-06-09T22:16:40.044989+0000","last_became_peered":"2024-06-09T22:16:40.044989+0000","last_unstale":"2024-06-09T22:20:57.695155+0000","last_undegraded":"2024-06-09T22:20:57.695155+0000","last_fullsized":"2024-06-09T22:20:57.695155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:01:05.850170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"99'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710115+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:20:57.710115+0000","last_peered":"2024-06-09T22:20:57.710115+0000","last_clean":"2024-06-09T22:20:57.710115+0000","last_became_active":"2024-06-09T22:16:43.203999+0000","last_became_peered":"2024-06-09T22:16:43.203999+0000","last_unstale":"2024-06-09T22:20:57.710115+0000","last_undegraded":"2024-06-09T22:20:57.710115+0000","last_fullsized":"2024-06-09T22:20:57.710115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:37:28.362655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703277+0000","last_change":"2024-06-09T22:16:53.333565+0000","last_active":"2024-06-09T22:20:57.703277+0000","last_peered":"2024-06-09T22:20:57.703277+0000","last_clean":"2024-06-09T22:20:57.703277+0000","last_became_active":"2024-06-09T22:16:53.332921+0000","last_became_peered":"2024-06-09T22:16:53.332921+0000","last_unstale":"2024-06-09T22:20:57.703277+0000","last_undegraded":"2024-06-09T22:20:57.703277+0000","last_fullsized":"2024-06-09T22:20:57.703277+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:32.085617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"97'5","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694916+0000","last_change":"2024-06-09T22:16:40.039766+0000","last_active":"2024-06-09T22:20:57.694916+0000","last_peered":"2024-06-09T22:20:57.694916+0000","last_clean":"2024-06-09T22:20:57.694916+0000","last_became_active":"2024-06-09T22:16:40.039329+0000","last_became_peered":"2024-06-09T22:16:40.039329+0000","last_unstale":"2024-06-09T22:20:57.694916+0000","last_undegraded":"2024-06-09T22:20:57.694916+0000","last_fullsized":"2024-06-09T22:20:57.694916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:53.673890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"99'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703385+0000","last_change":"2024-06-09T22:16:43.201200+0000","last_active":"2024-06-09T22:20:57.703385+0000","last_peered":"2024-06-09T22:20:57.703385+0000","last_clean":"2024-06-09T22:20:57.703385+0000","last_became_active":"2024-06-09T22:16:43.200830+0000","last_became_peered":"2024-06-09T22:16:43.200830+0000","last_unstale":"2024-06-09T22:20:57.703385+0000","last_undegraded":"2024-06-09T22:20:57.703385+0000","last_fullsized":"2024-06-09T22:20:57.703385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:38:52.093173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597461+0000","last_change":"2024-06-09T22:16:53.339811+0000","last_active":"2024-06-09T22:20:57.597461+0000","last_peered":"2024-06-09T22:20:57.597461+0000","last_clean":"2024-06-09T22:20:57.597461+0000","last_became_active":"2024-06-09T22:16:53.339305+0000","last_became_peered":"2024-06-09T22:16:53.339305+0000","last_unstale":"2024-06-09T22:20:57.597461+0000","last_undegraded":"2024-06-09T22:20:57.597461+0000","last_fullsized":"2024-06-09T22:20:57.597461+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:08:25.800245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"100'9","reported_seq":272,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:21.735279+0000","last_change":"2024-06-09T22:16:40.044159+0000","last_active":"2024-06-09T22:21:21.735279+0000","last_peered":"2024-06-09T22:21:21.735279+0000","last_clean":"2024-06-09T22:21:21.735279+0000","last_became_active":"2024-06-09T22:16:40.043839+0000","last_became_peered":"2024-06-09T22:16:40.043839+0000","last_unstale":"2024-06-09T22:21:21.735279+0000","last_undegraded":"2024-06-09T22:21:21.735279+0000","last_fullsized":"2024-06-09T22:21:21.735279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:22:09.339219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":217,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694583+0000","last_change":"2024-06-09T22:16:43.202015+0000","last_active":"2024-06-09T22:20:57.694583+0000","last_peered":"2024-06-09T22:20:57.694583+0000","last_clean":"2024-06-09T22:20:57.694583+0000","last_became_active":"2024-06-09T22:16:43.201689+0000","last_became_peered":"2024-06-09T22:16:43.201689+0000","last_unstale":"2024-06-09T22:20:57.694583+0000","last_undegraded":"2024-06-09T22:20:57.694583+0000","last_fullsized":"2024-06-09T22:20:57.694583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:35:12.180399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597147+0000","last_change":"2024-06-09T22:18:26.698600+0000","last_active":"2024-06-09T22:20:57.597147+0000","last_peered":"2024-06-09T22:20:57.597147+0000","last_clean":"2024-06-09T22:20:57.597147+0000","last_became_active":"2024-06-09T22:18:26.698201+0000","last_became_peered":"2024-06-09T22:18:26.698201+0000","last_unstale":"2024-06-09T22:20:57.597147+0000","last_undegraded":"2024-06-09T22:20:57.597147+0000","last_fullsized":"2024-06-09T22:20:57.597147+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:55:13.513557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,8],"acting":[2,6,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"97'7","reported_seq":57,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702844+0000","last_change":"2024-06-09T22:17:26.698204+0000","last_active":"2024-06-09T22:20:57.702844+0000","last_peered":"2024-06-09T22:20:57.702844+0000","last_clean":"2024-06-09T22:20:57.702844+0000","last_became_active":"2024-06-09T22:17:26.697716+0000","last_became_peered":"2024-06-09T22:17:26.697716+0000","last_unstale":"2024-06-09T22:20:57.702844+0000","last_undegraded":"2024-06-09T22:20:57.702844+0000","last_fullsized":"2024-06-09T22:20:57.702844+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:36:37.724459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"99'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703554+0000","last_change":"2024-06-09T22:16:43.198917+0000","last_active":"2024-06-09T22:20:57.703554+0000","last_peered":"2024-06-09T22:20:57.703554+0000","last_clean":"2024-06-09T22:20:57.703554+0000","last_became_active":"2024-06-09T22:16:43.198567+0000","last_became_peered":"2024-06-09T22:16:43.198567+0000","last_unstale":"2024-06-09T22:20:57.703554+0000","last_undegraded":"2024-06-09T22:20:57.703554+0000","last_fullsized":"2024-06-09T22:20:57.703554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:34.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710233+0000","last_change":"2024-06-09T22:16:53.342203+0000","last_active":"2024-06-09T22:20:57.710233+0000","last_peered":"2024-06-09T22:20:57.710233+0000","last_clean":"2024-06-09T22:20:57.710233+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:20:57.710233+0000","last_undegraded":"2024-06-09T22:20:57.710233+0000","last_fullsized":"2024-06-09T22:20:57.710233+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:41:41.491961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"97'3","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595514+0000","last_change":"2024-06-09T22:16:40.038684+0000","last_active":"2024-06-09T22:20:57.595514+0000","last_peered":"2024-06-09T22:20:57.595514+0000","last_clean":"2024-06-09T22:20:57.595514+0000","last_became_active":"2024-06-09T22:16:40.038386+0000","last_became_peered":"2024-06-09T22:16:40.038386+0000","last_unstale":"2024-06-09T22:20:57.595514+0000","last_undegraded":"2024-06-09T22:20:57.595514+0000","last_fullsized":"2024-06-09T22:20:57.595514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:59.188758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694679+0000","last_change":"2024-06-09T22:16:43.198837+0000","last_active":"2024-06-09T22:20:57.694679+0000","last_peered":"2024-06-09T22:20:57.694679+0000","last_clean":"2024-06-09T22:20:57.694679+0000","last_became_active":"2024-06-09T22:16:43.198515+0000","last_became_peered":"2024-06-09T22:16:43.198515+0000","last_unstale":"2024-06-09T22:20:57.694679+0000","last_undegraded":"2024-06-09T22:20:57.694679+0000","last_fullsized":"2024-06-09T22:20:57.694679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:36:13.863202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711076+0000","last_change":"2024-06-09T22:16:53.342688+0000","last_active":"2024-06-09T22:20:57.711076+0000","last_peered":"2024-06-09T22:20:57.711076+0000","last_clean":"2024-06-09T22:20:57.711076+0000","last_became_active":"2024-06-09T22:16:53.342314+0000","last_became_peered":"2024-06-09T22:16:53.342314+0000","last_unstale":"2024-06-09T22:20:57.711076+0000","last_undegraded":"2024-06-09T22:20:57.711076+0000","last_fullsized":"2024-06-09T22:20:57.711076+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:51:35.801587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"97'3","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703264+0000","last_change":"2024-06-09T22:16:40.041831+0000","last_active":"2024-06-09T22:20:57.703264+0000","last_peered":"2024-06-09T22:20:57.703264+0000","last_clean":"2024-06-09T22:20:57.703264+0000","last_became_active":"2024-06-09T22:16:40.041484+0000","last_became_peered":"2024-06-09T22:16:40.041484+0000","last_unstale":"2024-06-09T22:20:57.703264+0000","last_undegraded":"2024-06-09T22:20:57.703264+0000","last_fullsized":"2024-06-09T22:20:57.703264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:10:19.829357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692213+0000","last_change":"2024-06-09T22:16:43.210769+0000","last_active":"2024-06-09T22:20:57.692213+0000","last_peered":"2024-06-09T22:20:57.692213+0000","last_clean":"2024-06-09T22:20:57.692213+0000","last_became_active":"2024-06-09T22:16:43.210429+0000","last_became_peered":"2024-06-09T22:16:43.210429+0000","last_unstale":"2024-06-09T22:20:57.692213+0000","last_undegraded":"2024-06-09T22:20:57.692213+0000","last_fullsized":"2024-06-09T22:20:57.692213+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:12:28.788830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694708+0000","last_change":"2024-06-09T22:16:53.336536+0000","last_active":"2024-06-09T22:20:57.694708+0000","last_peered":"2024-06-09T22:20:57.694708+0000","last_clean":"2024-06-09T22:20:57.694708+0000","last_became_active":"2024-06-09T22:16:53.335178+0000","last_became_peered":"2024-06-09T22:16:53.335178+0000","last_unstale":"2024-06-09T22:20:57.694708+0000","last_undegraded":"2024-06-09T22:20:57.694708+0000","last_fullsized":"2024-06-09T22:20:57.694708+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:57:19.476483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'8","reported_seq":466,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:17.728156+0000","last_change":"2024-06-09T22:16:40.038484+0000","last_active":"2024-06-09T22:21:17.728156+0000","last_peered":"2024-06-09T22:21:17.728156+0000","last_clean":"2024-06-09T22:21:17.728156+0000","last_became_active":"2024-06-09T22:16:40.038084+0000","last_became_peered":"2024-06-09T22:16:40.038084+0000","last_unstale":"2024-06-09T22:21:17.728156+0000","last_undegraded":"2024-06-09T22:21:17.728156+0000","last_fullsized":"2024-06-09T22:21:17.728156+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:19.733107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":414,"num_read_kb":414,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"99'10","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710525+0000","last_change":"2024-06-09T22:16:43.205137+0000","last_active":"2024-06-09T22:20:57.710525+0000","last_peered":"2024-06-09T22:20:57.710525+0000","last_clean":"2024-06-09T22:20:57.710525+0000","last_became_active":"2024-06-09T22:16:43.204499+0000","last_became_peered":"2024-06-09T22:16:43.204499+0000","last_unstale":"2024-06-09T22:20:57.710525+0000","last_undegraded":"2024-06-09T22:20:57.710525+0000","last_fullsized":"2024-06-09T22:20:57.710525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:15.718493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692194+0000","last_change":"2024-06-09T22:16:53.344260+0000","last_active":"2024-06-09T22:20:57.692194+0000","last_peered":"2024-06-09T22:20:57.692194+0000","last_clean":"2024-06-09T22:20:57.692194+0000","last_became_active":"2024-06-09T22:16:53.343840+0000","last_became_peered":"2024-06-09T22:16:53.343840+0000","last_unstale":"2024-06-09T22:20:57.692194+0000","last_undegraded":"2024-06-09T22:20:57.692194+0000","last_fullsized":"2024-06-09T22:20:57.692194+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:40:23.305509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"97'4","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596747+0000","last_change":"2024-06-09T22:16:40.042660+0000","last_active":"2024-06-09T22:20:57.596747+0000","last_peered":"2024-06-09T22:20:57.596747+0000","last_clean":"2024-06-09T22:20:57.596747+0000","last_became_active":"2024-06-09T22:16:40.041353+0000","last_became_peered":"2024-06-09T22:16:40.041353+0000","last_unstale":"2024-06-09T22:20:57.596747+0000","last_undegraded":"2024-06-09T22:20:57.596747+0000","last_fullsized":"2024-06-09T22:20:57.596747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:34:59.064827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711158+0000","last_change":"2024-06-09T22:16:43.205083+0000","last_active":"2024-06-09T22:20:57.711158+0000","last_peered":"2024-06-09T22:20:57.711158+0000","last_clean":"2024-06-09T22:20:57.711158+0000","last_became_active":"2024-06-09T22:16:43.204600+0000","last_became_peered":"2024-06-09T22:16:43.204600+0000","last_unstale":"2024-06-09T22:20:57.711158+0000","last_undegraded":"2024-06-09T22:20:57.711158+0000","last_fullsized":"2024-06-09T22:20:57.711158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:05:12.555465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703699+0000","last_change":"2024-06-09T22:16:53.336169+0000","last_active":"2024-06-09T22:20:57.703699+0000","last_peered":"2024-06-09T22:20:57.703699+0000","last_clean":"2024-06-09T22:20:57.703699+0000","last_became_active":"2024-06-09T22:16:53.335657+0000","last_became_peered":"2024-06-09T22:16:53.335657+0000","last_unstale":"2024-06-09T22:20:57.703699+0000","last_undegraded":"2024-06-09T22:20:57.703699+0000","last_fullsized":"2024-06-09T22:20:57.703699+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:53:21.282767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"97'5","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703313+0000","last_change":"2024-06-09T22:16:40.070568+0000","last_active":"2024-06-09T22:20:57.703313+0000","last_peered":"2024-06-09T22:20:57.703313+0000","last_clean":"2024-06-09T22:20:57.703313+0000","last_became_active":"2024-06-09T22:16:40.070205+0000","last_became_peered":"2024-06-09T22:16:40.070205+0000","last_unstale":"2024-06-09T22:20:57.703313+0000","last_undegraded":"2024-06-09T22:20:57.703313+0000","last_fullsized":"2024-06-09T22:20:57.703313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.449965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"99'11","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710336+0000","last_change":"2024-06-09T22:16:43.205219+0000","last_active":"2024-06-09T22:20:57.710336+0000","last_peered":"2024-06-09T22:20:57.710336+0000","last_clean":"2024-06-09T22:20:57.710336+0000","last_became_active":"2024-06-09T22:16:43.203467+0000","last_became_peered":"2024-06-09T22:16:43.203467+0000","last_unstale":"2024-06-09T22:20:57.710336+0000","last_undegraded":"2024-06-09T22:20:57.710336+0000","last_fullsized":"2024-06-09T22:20:57.710336+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:22.784034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692165+0000","last_change":"2024-06-09T22:16:53.338990+0000","last_active":"2024-06-09T22:20:57.692165+0000","last_peered":"2024-06-09T22:20:57.692165+0000","last_clean":"2024-06-09T22:20:57.692165+0000","last_became_active":"2024-06-09T22:16:53.338568+0000","last_became_peered":"2024-06-09T22:16:53.338568+0000","last_unstale":"2024-06-09T22:20:57.692165+0000","last_undegraded":"2024-06-09T22:20:57.692165+0000","last_fullsized":"2024-06-09T22:20:57.692165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:46:39.814043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"100'48","reported_seq":521,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:17.728316+0000","last_change":"2024-06-09T22:16:40.044404+0000","last_active":"2024-06-09T22:21:17.728316+0000","last_peered":"2024-06-09T22:21:17.728316+0000","last_clean":"2024-06-09T22:21:17.728316+0000","last_became_active":"2024-06-09T22:16:40.044044+0000","last_became_peered":"2024-06-09T22:16:40.044044+0000","last_unstale":"2024-06-09T22:21:17.728316+0000","last_undegraded":"2024-06-09T22:21:17.728316+0000","last_fullsized":"2024-06-09T22:21:17.728316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:50:27.038759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77586,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":429,"num_read_kb":292,"num_write":50,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"99'6","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694903+0000","last_change":"2024-06-09T22:16:43.206963+0000","last_active":"2024-06-09T22:20:57.694903+0000","last_peered":"2024-06-09T22:20:57.694903+0000","last_clean":"2024-06-09T22:20:57.694903+0000","last_became_active":"2024-06-09T22:16:43.206628+0000","last_became_peered":"2024-06-09T22:16:43.206628+0000","last_unstale":"2024-06-09T22:20:57.694903+0000","last_undegraded":"2024-06-09T22:20:57.694903+0000","last_fullsized":"2024-06-09T22:20:57.694903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:33:39.912236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703845+0000","last_change":"2024-06-09T22:16:53.339332+0000","last_active":"2024-06-09T22:20:57.703845+0000","last_peered":"2024-06-09T22:20:57.703845+0000","last_clean":"2024-06-09T22:20:57.703845+0000","last_became_active":"2024-06-09T22:16:53.338819+0000","last_became_peered":"2024-06-09T22:16:53.338819+0000","last_unstale":"2024-06-09T22:20:57.703845+0000","last_undegraded":"2024-06-09T22:20:57.703845+0000","last_fullsized":"2024-06-09T22:20:57.703845+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:57:41.142142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"97'9","reported_seq":33,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709996+0000","last_change":"2024-06-09T22:17:26.478256+0000","last_active":"2024-06-09T22:20:57.709996+0000","last_peered":"2024-06-09T22:20:57.709996+0000","last_clean":"2024-06-09T22:20:57.709996+0000","last_became_active":"2024-06-09T22:17:26.472264+0000","last_became_peered":"2024-06-09T22:17:26.472264+0000","last_unstale":"2024-06-09T22:20:57.709996+0000","last_undegraded":"2024-06-09T22:20:57.709996+0000","last_fullsized":"2024-06-09T22:20:57.709996+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:14:19.919841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"99'9","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595697+0000","last_change":"2024-06-09T22:16:43.204484+0000","last_active":"2024-06-09T22:20:57.595697+0000","last_peered":"2024-06-09T22:20:57.595697+0000","last_clean":"2024-06-09T22:20:57.595697+0000","last_became_active":"2024-06-09T22:16:43.204014+0000","last_became_peered":"2024-06-09T22:16:43.204014+0000","last_unstale":"2024-06-09T22:20:57.595697+0000","last_undegraded":"2024-06-09T22:20:57.595697+0000","last_fullsized":"2024-06-09T22:20:57.595697+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:55:12.528231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710227+0000","last_change":"2024-06-09T22:17:26.475179+0000","last_active":"2024-06-09T22:20:57.710227+0000","last_peered":"2024-06-09T22:20:57.710227+0000","last_clean":"2024-06-09T22:20:57.710227+0000","last_became_active":"2024-06-09T22:17:26.474517+0000","last_became_peered":"2024-06-09T22:17:26.474517+0000","last_unstale":"2024-06-09T22:20:57.710227+0000","last_undegraded":"2024-06-09T22:20:57.710227+0000","last_fullsized":"2024-06-09T22:20:57.710227+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:26:02.387830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"97'5","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703424+0000","last_change":"2024-06-09T22:16:40.043066+0000","last_active":"2024-06-09T22:20:57.703424+0000","last_peered":"2024-06-09T22:20:57.703424+0000","last_clean":"2024-06-09T22:20:57.703424+0000","last_became_active":"2024-06-09T22:16:40.042619+0000","last_became_peered":"2024-06-09T22:16:40.042619+0000","last_unstale":"2024-06-09T22:20:57.703424+0000","last_undegraded":"2024-06-09T22:20:57.703424+0000","last_fullsized":"2024-06-09T22:20:57.703424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:29.689614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"99'15","reported_seq":63,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597237+0000","last_change":"2024-06-09T22:17:26.475566+0000","last_active":"2024-06-09T22:20:57.597237+0000","last_peered":"2024-06-09T22:20:57.597237+0000","last_clean":"2024-06-09T22:20:57.597237+0000","last_became_active":"2024-06-09T22:17:26.475037+0000","last_became_peered":"2024-06-09T22:17:26.475037+0000","last_unstale":"2024-06-09T22:20:57.597237+0000","last_undegraded":"2024-06-09T22:20:57.597237+0000","last_fullsized":"2024-06-09T22:20:57.597237+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:33.075144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694959+0000","last_change":"2024-06-09T22:17:26.475111+0000","last_active":"2024-06-09T22:20:57.694959+0000","last_peered":"2024-06-09T22:20:57.694959+0000","last_clean":"2024-06-09T22:20:57.694959+0000","last_became_active":"2024-06-09T22:17:26.474563+0000","last_became_peered":"2024-06-09T22:17:26.474563+0000","last_unstale":"2024-06-09T22:20:57.694959+0000","last_undegraded":"2024-06-09T22:20:57.694959+0000","last_fullsized":"2024-06-09T22:20:57.694959+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:09:08.458761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"100'9","reported_seq":273,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:21.739753+0000","last_change":"2024-06-09T22:16:40.040538+0000","last_active":"2024-06-09T22:21:21.739753+0000","last_peered":"2024-06-09T22:21:21.739753+0000","last_clean":"2024-06-09T22:21:21.739753+0000","last_became_active":"2024-06-09T22:16:40.039896+0000","last_became_peered":"2024-06-09T22:16:40.039896+0000","last_unstale":"2024-06-09T22:21:21.739753+0000","last_undegraded":"2024-06-09T22:21:21.739753+0000","last_fullsized":"2024-06-09T22:21:21.739753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:00:33.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1817,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":219,"num_read_kb":7,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"99'6","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596968+0000","last_change":"2024-06-09T22:17:26.475568+0000","last_active":"2024-06-09T22:20:57.596968+0000","last_peered":"2024-06-09T22:20:57.596968+0000","last_clean":"2024-06-09T22:20:57.596968+0000","last_became_active":"2024-06-09T22:17:26.475163+0000","last_became_peered":"2024-06-09T22:17:26.475163+0000","last_unstale":"2024-06-09T22:20:57.596968+0000","last_undegraded":"2024-06-09T22:20:57.596968+0000","last_fullsized":"2024-06-09T22:20:57.596968+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:54:20.865164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595970+0000","last_change":"2024-06-09T22:16:53.338201+0000","last_active":"2024-06-09T22:20:57.595970+0000","last_peered":"2024-06-09T22:20:57.595970+0000","last_clean":"2024-06-09T22:20:57.595970+0000","last_became_active":"2024-06-09T22:16:53.337816+0000","last_became_peered":"2024-06-09T22:16:53.337816+0000","last_unstale":"2024-06-09T22:20:57.595970+0000","last_undegraded":"2024-06-09T22:20:57.595970+0000","last_fullsized":"2024-06-09T22:20:57.595970+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:35:51.861135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"97'17","reported_seq":66,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694533+0000","last_change":"2024-06-09T22:16:40.070770+0000","last_active":"2024-06-09T22:20:57.694533+0000","last_peered":"2024-06-09T22:20:57.694533+0000","last_clean":"2024-06-09T22:20:57.694533+0000","last_became_active":"2024-06-09T22:16:40.070398+0000","last_became_peered":"2024-06-09T22:16:40.070398+0000","last_unstale":"2024-06-09T22:20:57.694533+0000","last_undegraded":"2024-06-09T22:20:57.694533+0000","last_fullsized":"2024-06-09T22:20:57.694533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:04:58.273601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"99'6","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703445+0000","last_change":"2024-06-09T22:16:43.199246+0000","last_active":"2024-06-09T22:20:57.703445+0000","last_peered":"2024-06-09T22:20:57.703445+0000","last_clean":"2024-06-09T22:20:57.703445+0000","last_became_active":"2024-06-09T22:16:43.198006+0000","last_became_peered":"2024-06-09T22:16:43.198006+0000","last_unstale":"2024-06-09T22:20:57.703445+0000","last_undegraded":"2024-06-09T22:20:57.703445+0000","last_fullsized":"2024-06-09T22:20:57.703445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:30:55.836628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703304+0000","last_change":"2024-06-09T22:16:53.342217+0000","last_active":"2024-06-09T22:20:57.703304+0000","last_peered":"2024-06-09T22:20:57.703304+0000","last_clean":"2024-06-09T22:20:57.703304+0000","last_became_active":"2024-06-09T22:16:53.341807+0000","last_became_peered":"2024-06-09T22:16:53.341807+0000","last_unstale":"2024-06-09T22:20:57.703304+0000","last_undegraded":"2024-06-09T22:20:57.703304+0000","last_fullsized":"2024-06-09T22:20:57.703304+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:25:28.635091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"100'4","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:58.449594+0000","last_change":"2024-06-09T22:16:40.039841+0000","last_active":"2024-06-09T22:20:58.449594+0000","last_peered":"2024-06-09T22:20:58.449594+0000","last_clean":"2024-06-09T22:20:58.449594+0000","last_became_active":"2024-06-09T22:16:40.039416+0000","last_became_peered":"2024-06-09T22:16:40.039416+0000","last_unstale":"2024-06-09T22:20:58.449594+0000","last_undegraded":"2024-06-09T22:20:58.449594+0000","last_fullsized":"2024-06-09T22:20:58.449594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:27:36.213839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"99'5","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710159+0000","last_change":"2024-06-09T22:16:43.214363+0000","last_active":"2024-06-09T22:20:57.710159+0000","last_peered":"2024-06-09T22:20:57.710159+0000","last_clean":"2024-06-09T22:20:57.710159+0000","last_became_active":"2024-06-09T22:16:43.214015+0000","last_became_peered":"2024-06-09T22:16:43.214015+0000","last_unstale":"2024-06-09T22:20:57.710159+0000","last_undegraded":"2024-06-09T22:20:57.710159+0000","last_fullsized":"2024-06-09T22:20:57.710159+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:13:06.924468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694710+0000","last_change":"2024-06-09T22:16:53.342151+0000","last_active":"2024-06-09T22:20:57.694710+0000","last_peered":"2024-06-09T22:20:57.694710+0000","last_clean":"2024-06-09T22:20:57.694710+0000","last_became_active":"2024-06-09T22:16:53.341743+0000","last_became_peered":"2024-06-09T22:16:53.341743+0000","last_unstale":"2024-06-09T22:20:57.694710+0000","last_undegraded":"2024-06-09T22:20:57.694710+0000","last_fullsized":"2024-06-09T22:20:57.694710+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:39:44.793301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"97'1","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692594+0000","last_change":"2024-06-09T22:17:26.472397+0000","last_active":"2024-06-09T22:20:57.692594+0000","last_peered":"2024-06-09T22:20:57.692594+0000","last_clean":"2024-06-09T22:20:57.692594+0000","last_became_active":"2024-06-09T22:17:26.471950+0000","last_became_peered":"2024-06-09T22:17:26.471950+0000","last_unstale":"2024-06-09T22:20:57.692594+0000","last_undegraded":"2024-06-09T22:20:57.692594+0000","last_fullsized":"2024-06-09T22:20:57.692594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:35:57.481510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"99'4","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703072+0000","last_change":"2024-06-09T22:16:43.214976+0000","last_active":"2024-06-09T22:20:57.703072+0000","last_peered":"2024-06-09T22:20:57.703072+0000","last_clean":"2024-06-09T22:20:57.703072+0000","last_became_active":"2024-06-09T22:16:43.214592+0000","last_became_peered":"2024-06-09T22:16:43.214592+0000","last_unstale":"2024-06-09T22:20:57.703072+0000","last_undegraded":"2024-06-09T22:20:57.703072+0000","last_fullsized":"2024-06-09T22:20:57.703072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:16:21.981144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595645+0000","last_change":"2024-06-09T22:16:53.342721+0000","last_active":"2024-06-09T22:20:57.595645+0000","last_peered":"2024-06-09T22:20:57.595645+0000","last_clean":"2024-06-09T22:20:57.595645+0000","last_became_active":"2024-06-09T22:16:53.342342+0000","last_became_peered":"2024-06-09T22:16:53.342342+0000","last_unstale":"2024-06-09T22:20:57.595645+0000","last_undegraded":"2024-06-09T22:20:57.595645+0000","last_fullsized":"2024-06-09T22:20:57.595645+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:22:07.469873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"100'6","reported_seq":268,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:21.757932+0000","last_change":"2024-06-09T22:16:40.041307+0000","last_active":"2024-06-09T22:21:21.757932+0000","last_peered":"2024-06-09T22:21:21.757932+0000","last_clean":"2024-06-09T22:21:21.757932+0000","last_became_active":"2024-06-09T22:16:40.040819+0000","last_became_peered":"2024-06-09T22:16:40.040819+0000","last_unstale":"2024-06-09T22:21:21.757932+0000","last_undegraded":"2024-06-09T22:21:21.757932+0000","last_fullsized":"2024-06-09T22:21:21.757932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:14:29.658293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":978,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":214,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"99'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.702925+0000","last_change":"2024-06-09T22:16:43.215719+0000","last_active":"2024-06-09T22:20:57.702925+0000","last_peered":"2024-06-09T22:20:57.702925+0000","last_clean":"2024-06-09T22:20:57.702925+0000","last_became_active":"2024-06-09T22:16:43.215356+0000","last_became_peered":"2024-06-09T22:16:43.215356+0000","last_unstale":"2024-06-09T22:20:57.702925+0000","last_undegraded":"2024-06-09T22:20:57.702925+0000","last_fullsized":"2024-06-09T22:20:57.702925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:15:07.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596028+0000","last_change":"2024-06-09T22:16:53.342948+0000","last_active":"2024-06-09T22:20:57.596028+0000","last_peered":"2024-06-09T22:20:57.596028+0000","last_clean":"2024-06-09T22:20:57.596028+0000","last_became_active":"2024-06-09T22:16:53.342352+0000","last_became_peered":"2024-06-09T22:16:53.342352+0000","last_unstale":"2024-06-09T22:20:57.596028+0000","last_undegraded":"2024-06-09T22:20:57.596028+0000","last_fullsized":"2024-06-09T22:20:57.596028+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:43:27.453789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703458+0000","last_change":"2024-06-09T22:16:43.204897+0000","last_active":"2024-06-09T22:20:57.703458+0000","last_peered":"2024-06-09T22:20:57.703458+0000","last_clean":"2024-06-09T22:20:57.703458+0000","last_became_active":"2024-06-09T22:16:43.204470+0000","last_became_peered":"2024-06-09T22:16:43.204470+0000","last_unstale":"2024-06-09T22:20:57.703458+0000","last_undegraded":"2024-06-09T22:20:57.703458+0000","last_fullsized":"2024-06-09T22:20:57.703458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:56.681392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"99'7","reported_seq":268,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:21.739264+0000","last_change":"2024-06-09T22:16:40.070079+0000","last_active":"2024-06-09T22:21:21.739264+0000","last_peered":"2024-06-09T22:21:21.739264+0000","last_clean":"2024-06-09T22:21:21.739264+0000","last_became_active":"2024-06-09T22:16:40.069709+0000","last_became_peered":"2024-06-09T22:16:40.069709+0000","last_unstale":"2024-06-09T22:21:21.739264+0000","last_undegraded":"2024-06-09T22:21:21.739264+0000","last_fullsized":"2024-06-09T22:21:21.739264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:00:45.132139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":213,"num_read_kb":213,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"97'1","reported_seq":33,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596870+0000","last_change":"2024-06-09T22:16:53.338891+0000","last_active":"2024-06-09T22:20:57.596870+0000","last_peered":"2024-06-09T22:20:57.596870+0000","last_clean":"2024-06-09T22:20:57.596870+0000","last_became_active":"2024-06-09T22:16:53.338526+0000","last_became_peered":"2024-06-09T22:16:53.338526+0000","last_unstale":"2024-06-09T22:20:57.596870+0000","last_undegraded":"2024-06-09T22:20:57.596870+0000","last_fullsized":"2024-06-09T22:20:57.596870+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:34:37.004407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695269+0000","last_change":"2024-06-09T22:16:43.199234+0000","last_active":"2024-06-09T22:20:57.695269+0000","last_peered":"2024-06-09T22:20:57.695269+0000","last_clean":"2024-06-09T22:20:57.695269+0000","last_became_active":"2024-06-09T22:16:43.192012+0000","last_became_peered":"2024-06-09T22:16:43.192012+0000","last_unstale":"2024-06-09T22:20:57.695269+0000","last_undegraded":"2024-06-09T22:20:57.695269+0000","last_fullsized":"2024-06-09T22:20:57.695269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:17.429236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"97'6","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703645+0000","last_change":"2024-06-09T22:16:40.041970+0000","last_active":"2024-06-09T22:20:57.703645+0000","last_peered":"2024-06-09T22:20:57.703645+0000","last_clean":"2024-06-09T22:20:57.703645+0000","last_became_active":"2024-06-09T22:16:40.040371+0000","last_became_peered":"2024-06-09T22:16:40.040371+0000","last_unstale":"2024-06-09T22:20:57.703645+0000","last_undegraded":"2024-06-09T22:20:57.703645+0000","last_fullsized":"2024-06-09T22:20:57.703645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:52:31.210423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"100'148","reported_seq":425,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:59.707074+0000","last_change":"2024-06-09T22:15:15.367397+0000","last_active":"2024-06-09T22:20:59.707074+0000","last_peered":"2024-06-09T22:20:59.707074+0000","last_clean":"2024-06-09T22:20:59.707074+0000","last_became_active":"2024-06-09T22:15:15.366974+0000","last_became_peered":"2024-06-09T22:15:15.366974+0000","last_unstale":"2024-06-09T22:20:59.707074+0000","last_undegraded":"2024-06-09T22:20:59.707074+0000","last_fullsized":"2024-06-09T22:20:59.707074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:11:49.886056+0000","last_clean_scrub_stamp":"2024-06-09T22:11:49.886056+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:44:24.018031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.704010+0000","last_change":"2024-06-09T22:16:53.342249+0000","last_active":"2024-06-09T22:20:57.704010+0000","last_peered":"2024-06-09T22:20:57.704010+0000","last_clean":"2024-06-09T22:20:57.704010+0000","last_became_active":"2024-06-09T22:16:53.341779+0000","last_became_peered":"2024-06-09T22:16:53.341779+0000","last_unstale":"2024-06-09T22:20:57.704010+0000","last_undegraded":"2024-06-09T22:20:57.704010+0000","last_fullsized":"2024-06-09T22:20:57.704010+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:21:21.101578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"99'5","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595425+0000","last_change":"2024-06-09T22:16:43.200778+0000","last_active":"2024-06-09T22:20:57.595425+0000","last_peered":"2024-06-09T22:20:57.595425+0000","last_clean":"2024-06-09T22:20:57.595425+0000","last_became_active":"2024-06-09T22:16:43.200490+0000","last_became_peered":"2024-06-09T22:16:43.200490+0000","last_unstale":"2024-06-09T22:20:57.595425+0000","last_undegraded":"2024-06-09T22:20:57.595425+0000","last_fullsized":"2024-06-09T22:20:57.595425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:20:17.530204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"97'4","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596731+0000","last_change":"2024-06-09T22:16:40.041774+0000","last_active":"2024-06-09T22:20:57.596731+0000","last_peered":"2024-06-09T22:20:57.596731+0000","last_clean":"2024-06-09T22:20:57.596731+0000","last_became_active":"2024-06-09T22:16:40.041085+0000","last_became_peered":"2024-06-09T22:16:40.041085+0000","last_unstale":"2024-06-09T22:20:57.596731+0000","last_undegraded":"2024-06-09T22:20:57.596731+0000","last_fullsized":"2024-06-09T22:20:57.596731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:09:41.140083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710137+0000","last_change":"2024-06-09T22:16:53.336895+0000","last_active":"2024-06-09T22:20:57.710137+0000","last_peered":"2024-06-09T22:20:57.710137+0000","last_clean":"2024-06-09T22:20:57.710137+0000","last_became_active":"2024-06-09T22:16:53.336497+0000","last_became_peered":"2024-06-09T22:16:53.336497+0000","last_unstale":"2024-06-09T22:20:57.710137+0000","last_undegraded":"2024-06-09T22:20:57.710137+0000","last_fullsized":"2024-06-09T22:20:57.710137+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:29:52.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"97'4","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695101+0000","last_change":"2024-06-09T22:16:40.070909+0000","last_active":"2024-06-09T22:20:57.695101+0000","last_peered":"2024-06-09T22:20:57.695101+0000","last_clean":"2024-06-09T22:20:57.695101+0000","last_became_active":"2024-06-09T22:16:40.070556+0000","last_became_peered":"2024-06-09T22:16:40.070556+0000","last_unstale":"2024-06-09T22:20:57.695101+0000","last_undegraded":"2024-06-09T22:20:57.695101+0000","last_fullsized":"2024-06-09T22:20:57.695101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:59:17.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"99'9","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595793+0000","last_change":"2024-06-09T22:16:43.205951+0000","last_active":"2024-06-09T22:20:57.595793+0000","last_peered":"2024-06-09T22:20:57.595793+0000","last_clean":"2024-06-09T22:20:57.595793+0000","last_became_active":"2024-06-09T22:16:43.205594+0000","last_became_peered":"2024-06-09T22:16:43.205594+0000","last_unstale":"2024-06-09T22:20:57.595793+0000","last_undegraded":"2024-06-09T22:20:57.595793+0000","last_fullsized":"2024-06-09T22:20:57.595793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:17.956406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694724+0000","last_change":"2024-06-09T22:16:53.343292+0000","last_active":"2024-06-09T22:20:57.694724+0000","last_peered":"2024-06-09T22:20:57.694724+0000","last_clean":"2024-06-09T22:20:57.694724+0000","last_became_active":"2024-06-09T22:16:53.342900+0000","last_became_peered":"2024-06-09T22:16:53.342900+0000","last_unstale":"2024-06-09T22:20:57.694724+0000","last_undegraded":"2024-06-09T22:20:57.694724+0000","last_fullsized":"2024-06-09T22:20:57.694724+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:08:07.328250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"100'105","reported_seq":243,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:13.158591+0000","last_change":"2024-06-09T22:16:40.039417+0000","last_active":"2024-06-09T22:21:13.158591+0000","last_peered":"2024-06-09T22:21:13.158591+0000","last_clean":"2024-06-09T22:21:13.158591+0000","last_became_active":"2024-06-09T22:16:40.039099+0000","last_became_peered":"2024-06-09T22:16:40.039099+0000","last_unstale":"2024-06-09T22:21:13.158591+0000","last_undegraded":"2024-06-09T22:21:13.158591+0000","last_fullsized":"2024-06-09T22:21:13.158591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:22:59.284442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":191,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"99'5","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696372+0000","last_change":"2024-06-09T22:16:43.201285+0000","last_active":"2024-06-09T22:20:57.696372+0000","last_peered":"2024-06-09T22:20:57.696372+0000","last_clean":"2024-06-09T22:20:57.696372+0000","last_became_active":"2024-06-09T22:16:43.200993+0000","last_became_peered":"2024-06-09T22:16:43.200993+0000","last_unstale":"2024-06-09T22:20:57.696372+0000","last_undegraded":"2024-06-09T22:20:57.696372+0000","last_fullsized":"2024-06-09T22:20:57.696372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:15:26.491588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596798+0000","last_change":"2024-06-09T22:16:53.334088+0000","last_active":"2024-06-09T22:20:57.596798+0000","last_peered":"2024-06-09T22:20:57.596798+0000","last_clean":"2024-06-09T22:20:57.596798+0000","last_became_active":"2024-06-09T22:16:53.333597+0000","last_became_peered":"2024-06-09T22:16:53.333597+0000","last_unstale":"2024-06-09T22:20:57.596798+0000","last_undegraded":"2024-06-09T22:20:57.596798+0000","last_fullsized":"2024-06-09T22:20:57.596798+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:50:22.041816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"97'7","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703700+0000","last_change":"2024-06-09T22:16:40.364982+0000","last_active":"2024-06-09T22:20:57.703700+0000","last_peered":"2024-06-09T22:20:57.703700+0000","last_clean":"2024-06-09T22:20:57.703700+0000","last_became_active":"2024-06-09T22:16:40.364585+0000","last_became_peered":"2024-06-09T22:16:40.364585+0000","last_unstale":"2024-06-09T22:20:57.703700+0000","last_undegraded":"2024-06-09T22:20:57.703700+0000","last_fullsized":"2024-06-09T22:20:57.703700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:59:28.777950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"99'7","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.711181+0000","last_change":"2024-06-09T22:17:26.474491+0000","last_active":"2024-06-09T22:20:57.711181+0000","last_peered":"2024-06-09T22:20:57.711181+0000","last_clean":"2024-06-09T22:20:57.711181+0000","last_became_active":"2024-06-09T22:17:26.473975+0000","last_became_peered":"2024-06-09T22:17:26.473975+0000","last_unstale":"2024-06-09T22:20:57.711181+0000","last_undegraded":"2024-06-09T22:20:57.711181+0000","last_fullsized":"2024-06-09T22:20:57.711181+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:13:46.998595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703384+0000","last_change":"2024-06-09T22:16:53.342273+0000","last_active":"2024-06-09T22:20:57.703384+0000","last_peered":"2024-06-09T22:20:57.703384+0000","last_clean":"2024-06-09T22:20:57.703384+0000","last_became_active":"2024-06-09T22:16:53.341827+0000","last_became_peered":"2024-06-09T22:16:53.341827+0000","last_unstale":"2024-06-09T22:20:57.703384+0000","last_undegraded":"2024-06-09T22:20:57.703384+0000","last_fullsized":"2024-06-09T22:20:57.703384+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:23:26.099737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"97'2","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710040+0000","last_change":"2024-06-09T22:16:40.070498+0000","last_active":"2024-06-09T22:20:57.710040+0000","last_peered":"2024-06-09T22:20:57.710040+0000","last_clean":"2024-06-09T22:20:57.710040+0000","last_became_active":"2024-06-09T22:16:40.070144+0000","last_became_peered":"2024-06-09T22:16:40.070144+0000","last_unstale":"2024-06-09T22:20:57.710040+0000","last_undegraded":"2024-06-09T22:20:57.710040+0000","last_fullsized":"2024-06-09T22:20:57.710040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:52:22.214832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"99'2","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694528+0000","last_change":"2024-06-09T22:16:43.210448+0000","last_active":"2024-06-09T22:20:57.694528+0000","last_peered":"2024-06-09T22:20:57.694528+0000","last_clean":"2024-06-09T22:20:57.694528+0000","last_became_active":"2024-06-09T22:16:43.209980+0000","last_became_peered":"2024-06-09T22:16:43.209980+0000","last_unstale":"2024-06-09T22:20:57.694528+0000","last_undegraded":"2024-06-09T22:20:57.694528+0000","last_fullsized":"2024-06-09T22:20:57.694528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:58:15.691207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.692864+0000","last_change":"2024-06-09T22:16:53.340644+0000","last_active":"2024-06-09T22:20:57.692864+0000","last_peered":"2024-06-09T22:20:57.692864+0000","last_clean":"2024-06-09T22:20:57.692864+0000","last_became_active":"2024-06-09T22:16:53.339771+0000","last_became_peered":"2024-06-09T22:16:53.339771+0000","last_unstale":"2024-06-09T22:20:57.692864+0000","last_undegraded":"2024-06-09T22:20:57.692864+0000","last_fullsized":"2024-06-09T22:20:57.692864+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:45:23.281387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"97'7","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694859+0000","last_change":"2024-06-09T22:16:40.040461+0000","last_active":"2024-06-09T22:20:57.694859+0000","last_peered":"2024-06-09T22:20:57.694859+0000","last_clean":"2024-06-09T22:20:57.694859+0000","last_became_active":"2024-06-09T22:16:40.040168+0000","last_became_peered":"2024-06-09T22:16:40.040168+0000","last_unstale":"2024-06-09T22:20:57.694859+0000","last_undegraded":"2024-06-09T22:20:57.694859+0000","last_fullsized":"2024-06-09T22:20:57.694859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:28.936631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.709959+0000","last_change":"2024-06-09T22:16:43.208779+0000","last_active":"2024-06-09T22:20:57.709959+0000","last_peered":"2024-06-09T22:20:57.709959+0000","last_clean":"2024-06-09T22:20:57.709959+0000","last_became_active":"2024-06-09T22:16:43.208136+0000","last_became_peered":"2024-06-09T22:16:43.208136+0000","last_unstale":"2024-06-09T22:20:57.709959+0000","last_undegraded":"2024-06-09T22:20:57.709959+0000","last_fullsized":"2024-06-09T22:20:57.709959+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:22:39.721069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"97'1","reported_seq":33,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703449+0000","last_change":"2024-06-09T22:16:53.331241+0000","last_active":"2024-06-09T22:20:57.703449+0000","last_peered":"2024-06-09T22:20:57.703449+0000","last_clean":"2024-06-09T22:20:57.703449+0000","last_became_active":"2024-06-09T22:16:53.329589+0000","last_became_peered":"2024-06-09T22:16:53.329589+0000","last_unstale":"2024-06-09T22:20:57.703449+0000","last_undegraded":"2024-06-09T22:20:57.703449+0000","last_fullsized":"2024-06-09T22:20:57.703449+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:56:21.319711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"100'4","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:58.422275+0000","last_change":"2024-06-09T22:16:40.042491+0000","last_active":"2024-06-09T22:20:58.422275+0000","last_peered":"2024-06-09T22:20:58.422275+0000","last_clean":"2024-06-09T22:20:58.422275+0000","last_became_active":"2024-06-09T22:16:40.041846+0000","last_became_peered":"2024-06-09T22:16:40.041846+0000","last_unstale":"2024-06-09T22:20:58.422275+0000","last_undegraded":"2024-06-09T22:20:58.422275+0000","last_fullsized":"2024-06-09T22:20:58.422275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:59.144003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.695873+0000","last_change":"2024-06-09T22:16:43.209195+0000","last_active":"2024-06-09T22:20:57.695873+0000","last_peered":"2024-06-09T22:20:57.695873+0000","last_clean":"2024-06-09T22:20:57.695873+0000","last_became_active":"2024-06-09T22:16:43.208797+0000","last_became_peered":"2024-06-09T22:16:43.208797+0000","last_unstale":"2024-06-09T22:20:57.695873+0000","last_undegraded":"2024-06-09T22:20:57.695873+0000","last_fullsized":"2024-06-09T22:20:57.695873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:05:31.689900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703334+0000","last_change":"2024-06-09T22:16:53.342173+0000","last_active":"2024-06-09T22:20:57.703334+0000","last_peered":"2024-06-09T22:20:57.703334+0000","last_clean":"2024-06-09T22:20:57.703334+0000","last_became_active":"2024-06-09T22:16:53.341712+0000","last_became_peered":"2024-06-09T22:16:53.341712+0000","last_unstale":"2024-06-09T22:20:57.703334+0000","last_undegraded":"2024-06-09T22:20:57.703334+0000","last_fullsized":"2024-06-09T22:20:57.703334+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:53:12.574764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"97'4","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694627+0000","last_change":"2024-06-09T22:16:40.044238+0000","last_active":"2024-06-09T22:20:57.694627+0000","last_peered":"2024-06-09T22:20:57.694627+0000","last_clean":"2024-06-09T22:20:57.694627+0000","last_became_active":"2024-06-09T22:16:40.043857+0000","last_became_peered":"2024-06-09T22:16:40.043857+0000","last_unstale":"2024-06-09T22:20:57.694627+0000","last_undegraded":"2024-06-09T22:20:57.694627+0000","last_fullsized":"2024-06-09T22:20:57.694627+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:44:45.743307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"99'5","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597096+0000","last_change":"2024-06-09T22:16:43.202363+0000","last_active":"2024-06-09T22:20:57.597096+0000","last_peered":"2024-06-09T22:20:57.597096+0000","last_clean":"2024-06-09T22:20:57.597096+0000","last_became_active":"2024-06-09T22:16:43.202029+0000","last_became_peered":"2024-06-09T22:16:43.202029+0000","last_unstale":"2024-06-09T22:20:57.597096+0000","last_undegraded":"2024-06-09T22:20:57.597096+0000","last_fullsized":"2024-06-09T22:20:57.597096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:11:19.513055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710066+0000","last_change":"2024-06-09T22:16:53.333837+0000","last_active":"2024-06-09T22:20:57.710066+0000","last_peered":"2024-06-09T22:20:57.710066+0000","last_clean":"2024-06-09T22:20:57.710066+0000","last_became_active":"2024-06-09T22:16:53.333362+0000","last_became_peered":"2024-06-09T22:16:53.333362+0000","last_unstale":"2024-06-09T22:20:57.710066+0000","last_undegraded":"2024-06-09T22:20:57.710066+0000","last_fullsized":"2024-06-09T22:20:57.710066+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:47:04.463040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"100'8","reported_seq":266,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:16.727233+0000","last_change":"2024-06-09T22:16:40.364834+0000","last_active":"2024-06-09T22:21:16.727233+0000","last_peered":"2024-06-09T22:21:16.727233+0000","last_clean":"2024-06-09T22:21:16.727233+0000","last_became_active":"2024-06-09T22:16:40.364433+0000","last_became_peered":"2024-06-09T22:16:40.364433+0000","last_unstale":"2024-06-09T22:21:16.727233+0000","last_undegraded":"2024-06-09T22:21:16.727233+0000","last_fullsized":"2024-06-09T22:21:16.727233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:43:01.967911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2091,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":212,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"99'7","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597232+0000","last_change":"2024-06-09T22:16:43.201183+0000","last_active":"2024-06-09T22:20:57.597232+0000","last_peered":"2024-06-09T22:20:57.597232+0000","last_clean":"2024-06-09T22:20:57.597232+0000","last_became_active":"2024-06-09T22:16:43.200774+0000","last_became_peered":"2024-06-09T22:16:43.200774+0000","last_unstale":"2024-06-09T22:20:57.597232+0000","last_undegraded":"2024-06-09T22:20:57.597232+0000","last_fullsized":"2024-06-09T22:20:57.597232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:07:55.772066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710583+0000","last_change":"2024-06-09T22:18:26.699190+0000","last_active":"2024-06-09T22:20:57.710583+0000","last_peered":"2024-06-09T22:20:57.710583+0000","last_clean":"2024-06-09T22:20:57.710583+0000","last_became_active":"2024-06-09T22:18:26.698614+0000","last_became_peered":"2024-06-09T22:18:26.698614+0000","last_unstale":"2024-06-09T22:20:57.710583+0000","last_undegraded":"2024-06-09T22:20:57.710583+0000","last_fullsized":"2024-06-09T22:20:57.710583+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:03:45.343605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,1,4],"acting":[0,10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"100'67","reported_seq":145,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:21:12.818366+0000","last_change":"2024-06-09T22:16:40.040582+0000","last_active":"2024-06-09T22:21:12.818366+0000","last_peered":"2024-06-09T22:21:12.818366+0000","last_clean":"2024-06-09T22:21:12.818366+0000","last_became_active":"2024-06-09T22:16:40.040152+0000","last_became_peered":"2024-06-09T22:16:40.040152+0000","last_unstale":"2024-06-09T22:21:12.818366+0000","last_undegraded":"2024-06-09T22:21:12.818366+0000","last_fullsized":"2024-06-09T22:21:12.818366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:50:00.878096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"99'6","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596961+0000","last_change":"2024-06-09T22:16:43.206211+0000","last_active":"2024-06-09T22:20:57.596961+0000","last_peered":"2024-06-09T22:20:57.596961+0000","last_clean":"2024-06-09T22:20:57.596961+0000","last_became_active":"2024-06-09T22:16:43.205814+0000","last_became_peered":"2024-06-09T22:16:43.205814+0000","last_unstale":"2024-06-09T22:20:57.596961+0000","last_undegraded":"2024-06-09T22:20:57.596961+0000","last_fullsized":"2024-06-09T22:20:57.596961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:20:19.058475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703101+0000","last_change":"2024-06-09T22:16:53.337695+0000","last_active":"2024-06-09T22:20:57.703101+0000","last_peered":"2024-06-09T22:20:57.703101+0000","last_clean":"2024-06-09T22:20:57.703101+0000","last_became_active":"2024-06-09T22:16:53.334872+0000","last_became_peered":"2024-06-09T22:16:53.334872+0000","last_unstale":"2024-06-09T22:20:57.703101+0000","last_undegraded":"2024-06-09T22:20:57.703101+0000","last_fullsized":"2024-06-09T22:20:57.703101+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:52.314914+0000","last_clean_scrub_stamp":"2024-06-09T22:16:52.314914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:50:36.617736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"97'5","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596104+0000","last_change":"2024-06-09T22:16:40.036603+0000","last_active":"2024-06-09T22:20:57.596104+0000","last_peered":"2024-06-09T22:20:57.596104+0000","last_clean":"2024-06-09T22:20:57.596104+0000","last_became_active":"2024-06-09T22:16:40.036274+0000","last_became_peered":"2024-06-09T22:16:40.036274+0000","last_unstale":"2024-06-09T22:20:57.596104+0000","last_undegraded":"2024-06-09T22:20:57.596104+0000","last_fullsized":"2024-06-09T22:20:57.596104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:39.026221+0000","last_clean_scrub_stamp":"2024-06-09T22:16:39.026221+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:42:31.652897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"99'9","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597041+0000","last_change":"2024-06-09T22:16:43.204544+0000","last_active":"2024-06-09T22:20:57.597041+0000","last_peered":"2024-06-09T22:20:57.597041+0000","last_clean":"2024-06-09T22:20:57.597041+0000","last_became_active":"2024-06-09T22:16:43.204197+0000","last_became_peered":"2024-06-09T22:16:43.204197+0000","last_unstale":"2024-06-09T22:20:57.597041+0000","last_undegraded":"2024-06-09T22:20:57.597041+0000","last_fullsized":"2024-06-09T22:20:57.597041+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:38:49.850120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"99'9","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710013+0000","last_change":"2024-06-09T22:16:43.208477+0000","last_active":"2024-06-09T22:20:57.710013+0000","last_peered":"2024-06-09T22:20:57.710013+0000","last_clean":"2024-06-09T22:20:57.710013+0000","last_became_active":"2024-06-09T22:16:43.206018+0000","last_became_peered":"2024-06-09T22:16:43.206018+0000","last_unstale":"2024-06-09T22:20:57.710013+0000","last_undegraded":"2024-06-09T22:20:57.710013+0000","last_fullsized":"2024-06-09T22:20:57.710013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:19:49.870825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"99'12","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597095+0000","last_change":"2024-06-09T22:16:43.206809+0000","last_active":"2024-06-09T22:20:57.597095+0000","last_peered":"2024-06-09T22:20:57.597095+0000","last_clean":"2024-06-09T22:20:57.597095+0000","last_became_active":"2024-06-09T22:16:43.206175+0000","last_became_peered":"2024-06-09T22:16:43.206175+0000","last_unstale":"2024-06-09T22:20:57.597095+0000","last_undegraded":"2024-06-09T22:20:57.597095+0000","last_fullsized":"2024-06-09T22:20:57.597095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:39:22.502266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.596879+0000","last_change":"2024-06-09T22:16:43.202163+0000","last_active":"2024-06-09T22:20:57.596879+0000","last_peered":"2024-06-09T22:20:57.596879+0000","last_clean":"2024-06-09T22:20:57.596879+0000","last_became_active":"2024-06-09T22:16:43.201839+0000","last_became_peered":"2024-06-09T22:16:43.201839+0000","last_unstale":"2024-06-09T22:20:57.596879+0000","last_undegraded":"2024-06-09T22:20:57.596879+0000","last_fullsized":"2024-06-09T22:20:57.596879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:47:46.723289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"99'7","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710813+0000","last_change":"2024-06-09T22:16:43.202734+0000","last_active":"2024-06-09T22:20:57.710813+0000","last_peered":"2024-06-09T22:20:57.710813+0000","last_clean":"2024-06-09T22:20:57.710813+0000","last_became_active":"2024-06-09T22:16:43.201638+0000","last_became_peered":"2024-06-09T22:16:43.201638+0000","last_unstale":"2024-06-09T22:20:57.710813+0000","last_undegraded":"2024-06-09T22:20:57.710813+0000","last_fullsized":"2024-06-09T22:20:57.710813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:39:16.360985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"99'4","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595768+0000","last_change":"2024-06-09T22:16:43.199113+0000","last_active":"2024-06-09T22:20:57.595768+0000","last_peered":"2024-06-09T22:20:57.595768+0000","last_clean":"2024-06-09T22:20:57.595768+0000","last_became_active":"2024-06-09T22:16:43.198804+0000","last_became_peered":"2024-06-09T22:16:43.198804+0000","last_unstale":"2024-06-09T22:20:57.595768+0000","last_undegraded":"2024-06-09T22:20:57.595768+0000","last_fullsized":"2024-06-09T22:20:57.595768+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T22:18:08.999477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"99'6","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710407+0000","last_change":"2024-06-09T22:16:43.204445+0000","last_active":"2024-06-09T22:20:57.710407+0000","last_peered":"2024-06-09T22:20:57.710407+0000","last_clean":"2024-06-09T22:20:57.710407+0000","last_became_active":"2024-06-09T22:16:43.203081+0000","last_became_peered":"2024-06-09T22:16:43.203081+0000","last_unstale":"2024-06-09T22:20:57.710407+0000","last_undegraded":"2024-06-09T22:20:57.710407+0000","last_fullsized":"2024-06-09T22:20:57.710407+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:39.303813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"99'15","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.597639+0000","last_change":"2024-06-09T22:16:43.211402+0000","last_active":"2024-06-09T22:20:57.597639+0000","last_peered":"2024-06-09T22:20:57.597639+0000","last_clean":"2024-06-09T22:20:57.597639+0000","last_became_active":"2024-06-09T22:16:43.210871+0000","last_became_peered":"2024-06-09T22:16:43.210871+0000","last_unstale":"2024-06-09T22:20:57.597639+0000","last_undegraded":"2024-06-09T22:20:57.597639+0000","last_fullsized":"2024-06-09T22:20:57.597639+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:07:21.467617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"99'2","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.696286+0000","last_change":"2024-06-09T22:16:43.201159+0000","last_active":"2024-06-09T22:20:57.696286+0000","last_peered":"2024-06-09T22:20:57.696286+0000","last_clean":"2024-06-09T22:20:57.696286+0000","last_became_active":"2024-06-09T22:16:43.200850+0000","last_became_peered":"2024-06-09T22:16:43.200850+0000","last_unstale":"2024-06-09T22:20:57.696286+0000","last_undegraded":"2024-06-09T22:20:57.696286+0000","last_fullsized":"2024-06-09T22:20:57.696286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:20.515309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"99'6","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.703351+0000","last_change":"2024-06-09T22:16:43.196830+0000","last_active":"2024-06-09T22:20:57.703351+0000","last_peered":"2024-06-09T22:20:57.703351+0000","last_clean":"2024-06-09T22:20:57.703351+0000","last_became_active":"2024-06-09T22:16:43.195682+0000","last_became_peered":"2024-06-09T22:16:43.195682+0000","last_unstale":"2024-06-09T22:20:57.703351+0000","last_undegraded":"2024-06-09T22:20:57.703351+0000","last_fullsized":"2024-06-09T22:20:57.703351+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:34:18.770235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"99'8","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.694976+0000","last_change":"2024-06-09T22:16:43.201959+0000","last_active":"2024-06-09T22:20:57.694976+0000","last_peered":"2024-06-09T22:20:57.694976+0000","last_clean":"2024-06-09T22:20:57.694976+0000","last_became_active":"2024-06-09T22:16:43.201669+0000","last_became_peered":"2024-06-09T22:16:43.201669+0000","last_unstale":"2024-06-09T22:20:57.694976+0000","last_undegraded":"2024-06-09T22:20:57.694976+0000","last_fullsized":"2024-06-09T22:20:57.694976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:48:57.348041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"99'10","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710243+0000","last_change":"2024-06-09T22:16:43.201823+0000","last_active":"2024-06-09T22:20:57.710243+0000","last_peered":"2024-06-09T22:20:57.710243+0000","last_clean":"2024-06-09T22:20:57.710243+0000","last_became_active":"2024-06-09T22:16:43.201434+0000","last_became_peered":"2024-06-09T22:16:43.201434+0000","last_unstale":"2024-06-09T22:20:57.710243+0000","last_undegraded":"2024-06-09T22:20:57.710243+0000","last_fullsized":"2024-06-09T22:20:57.710243+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:56:13.540895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"99'11","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.595673+0000","last_change":"2024-06-09T22:16:43.197734+0000","last_active":"2024-06-09T22:20:57.595673+0000","last_peered":"2024-06-09T22:20:57.595673+0000","last_clean":"2024-06-09T22:20:57.595673+0000","last_became_active":"2024-06-09T22:16:43.197442+0000","last_became_peered":"2024-06-09T22:16:43.197442+0000","last_unstale":"2024-06-09T22:20:57.595673+0000","last_undegraded":"2024-06-09T22:20:57.595673+0000","last_fullsized":"2024-06-09T22:20:57.595673+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-10T23:35:40.720718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"99'8","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-09T22:20:57.710386+0000","last_change":"2024-06-09T22:16:43.210375+0000","last_active":"2024-06-09T22:20:57.710386+0000","last_peered":"2024-06-09T22:20:57.710386+0000","last_clean":"2024-06-09T22:20:57.710386+0000","last_became_active":"2024-06-09T22:16:43.209647+0000","last_became_peered":"2024-06-09T22:16:43.209647+0000","last_unstale":"2024-06-09T22:20:57.710386+0000","last_undegraded":"2024-06-09T22:20:57.710386+0000","last_fullsized":"2024-06-09T22:20:57.710386+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-09T22:16:42.179147+0000","last_clean_scrub_stamp":"2024-06-09T22:16:42.179147+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:11:57.593066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107643,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":809307,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2492,"num_read_kb":1476,"num_write":524,"num_write_kb":2408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":823296,"data_stored":2427921,"data_compressed":103473,"data_compressed_allocated":135168,"data_compressed_original":2101248,"omap_allocated":0,"internal_metadata":0},"log_size":401,"ondisk_log_size":401,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383751,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":581230,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645388,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32476,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712676,"statfs":{"total":95995035648,"available":95961780224,"internally_reserved":0,"allocated":1323008,"data_stored":1574104,"data_compressed":86342,"data_compressed_allocated":114688,"data_compressed_original":995328,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808913,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":977483,"data_compressed":54548,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070550,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1159168,"data_stored":973765,"data_compressed":54548,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332188,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32336,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712816,"statfs":{"total":95995035648,"available":95961923584,"internally_reserved":0,"allocated":1187840,"data_stored":1179585,"data_compressed":71140,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495714,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1114112,"data_stored":576065,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36504,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708648,"statfs":{"total":95995035648,"available":95957655552,"internally_reserved":0,"allocated":1196032,"data_stored":654681,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708700,"statfs":{"total":95995035648,"available":95957708800,"internally_reserved":0,"allocated":1138688,"data_stored":578052,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411444,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1208320,"data_stored":656675,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182511,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28192,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716960,"statfs":{"total":95995035648,"available":95966167040,"internally_reserved":0,"allocated":1196032,"data_stored":1296831,"data_compressed":69750,"data_compressed_allocated":94208,"data_compressed_original":798720,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920876,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32276,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712876,"statfs":{"total":95995035648,"available":95961985024,"internally_reserved":0,"allocated":1122304,"data_stored":580978,"data_compressed":35259,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757351,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36556,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708596,"statfs":{"total":95995035648,"available":95957602304,"internally_reserved":0,"allocated":1241088,"data_stored":1501780,"data_compressed":86342,"data_compressed_allocated":114688,"data_compressed_original":995328,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35881,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":79728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":81722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":721878,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":332363,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":408783,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-09T22:21:24.867 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:21:25.111 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:25.484 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:25 smithi104 ceph-mon[32389]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 KiB/s wr, 41 op/s 2024-06-09T22:21:25.484 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:25 smithi104 ceph-mon[32389]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:25.484 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:25 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:21:25.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:25 smithi153 ceph-mon[36599]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 KiB/s wr, 41 op/s 2024-06-09T22:21:25.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:25 smithi153 ceph-mon[36599]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:25.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:25 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:21:25.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:25 smithi194 ceph-mon[36856]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 KiB/s wr, 41 op/s 2024-06-09T22:21:25.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:25 smithi194 ceph-mon[36856]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-09T22:21:25.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:25 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:21:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 -- 172.21.15.104:0/3440216021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f0c106890 msgr2=0x7f8f0c106c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 --2- 172.21.15.104:0/3440216021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f0c106890 0x7f8f0c106c70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8efc009930 tx=0x7f8efc02a420 comp rx=0 tx=0).stop 2024-06-09T22:21:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 -- 172.21.15.104:0/3440216021 shutdown_connections 2024-06-09T22:21:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 --2- 172.21.15.104:0/3440216021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f0c101290 0x7f8f0c10d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 --2- 172.21.15.104:0/3440216021 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f0c1008d0 0x7f8f0c100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 --2- 172.21.15.104:0/3440216021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f0c106890 0x7f8f0c106c70 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 -- 172.21.15.104:0/3440216021 >> 172.21.15.104:0/3440216021 conn(0x7f8f0c077a20 msgr2=0x7f8f0c077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 -- 172.21.15.104:0/3440216021 shutdown_connections 2024-06-09T22:21:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 -- 172.21.15.104:0/3440216021 wait complete. 2024-06-09T22:21:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 Processor -- start 2024-06-09T22:21:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 -- start start 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f0c1008d0 0x7f8f0c199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f0c101290 0x7f8f0c1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f0c106890 0x7f8f0c1a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c110ca0 con 0x7f8f0c101290 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c110b20 con 0x7f8f0c106890 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.381+0000 7f8f1208c640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c110e20 con 0x7f8f0c1008d0 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f03fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f0c101290 0x7f8f0c1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f03fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f0c101290 0x7f8f0c1995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37800/0 (socket says 172.21.15.104:37800) 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f03fff640 1 -- 172.21.15.104:0/2470709597 learned_addr learned my addr 172.21.15.104:0/2470709597 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f0bfff640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f0c106890 0x7f8f0c1a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f0b7fe640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f0c1008d0 0x7f8f0c199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f03fff640 1 -- 172.21.15.104:0/2470709597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f0c1008d0 msgr2=0x7f8f0c199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f03fff640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f0c1008d0 0x7f8f0c199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f03fff640 1 -- 172.21.15.104:0/2470709597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f0c106890 msgr2=0x7f8f0c1a40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:26.384 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f03fff640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f0c106890 0x7f8f0c1a40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.384 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f03fff640 1 -- 172.21.15.104:0/2470709597 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0c1a4620 con 0x7f8f0c101290 2024-06-09T22:21:26.384 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f03fff640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f0c101290 0x7f8f0c1995a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8ef400e9f0 tx=0x7f8ef400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:26.384 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f097fa640 1 -- 172.21.15.104:0/2470709597 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef400cd80 con 0x7f8f0c101290 2024-06-09T22:21:26.384 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f1208c640 1 -- 172.21.15.104:0/2470709597 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0c1a47f0 con 0x7f8f0c101290 2024-06-09T22:21:26.384 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f097fa640 1 -- 172.21.15.104:0/2470709597 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ef4005500 con 0x7f8f0c101290 2024-06-09T22:21:26.384 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.382+0000 7f8f097fa640 1 -- 172.21.15.104:0/2470709597 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4010690 con 0x7f8f0c101290 2024-06-09T22:21:26.384 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.383+0000 7f8f1208c640 1 -- 172.21.15.104:0/2470709597 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f0c1a4cc0 con 0x7f8f0c101290 2024-06-09T22:21:26.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.385+0000 7f8f14d9a740 1 Processor -- start 2024-06-09T22:21:26.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.385+0000 7f8f14d9a740 1 -- start start 2024-06-09T22:21:26.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.385+0000 7f8f097fa640 1 -- 172.21.15.104:0/2470709597 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f8ef40046c0 con 0x7f8f0c101290 2024-06-09T22:21:26.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.385+0000 7f8f14d9a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867050 0x565473867430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.385+0000 7f8f14d9a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867970 0x565473876220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.385+0000 7f8f14d9a740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 0x565473878b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.385+0000 7f8f14d9a740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5654737cb4f0 con 0x565473876760 2024-06-09T22:21:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.385+0000 7f8f14d9a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x565473769870 con 0x565473867970 2024-06-09T22:21:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.385+0000 7f8f14d9a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5654737c74b0 con 0x565473867050 2024-06-09T22:21:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.385+0000 7f8f03fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 0x565473878b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f03fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 0x565473878b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37812/0 (socket says 172.21.15.104:37812) 2024-06-09T22:21:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f03fff640 1 -- 172.21.15.104:0/767437883 learned_addr learned my addr 172.21.15.104:0/767437883 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.385+0000 7f8f0bfff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867050 0x565473867430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f0b7fe640 1 --2- 172.21.15.104:0/767437883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867970 0x565473876220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f03fff640 1 -- 172.21.15.104:0/767437883 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867050 msgr2=0x565473867430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f03fff640 1 --2- 172.21.15.104:0/767437883 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867050 0x565473867430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f03fff640 1 -- 172.21.15.104:0/767437883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867970 msgr2=0x565473876220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f03fff640 1 --2- 172.21.15.104:0/767437883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867970 0x565473876220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f03fff640 1 -- 172.21.15.104:0/767437883 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654737859b0 con 0x565473876760 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f03fff640 1 --2- 172.21.15.104:0/767437883 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 0x565473878b50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40cd110 tx=0x7f8ef40d3f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722ab5fddb7b52c6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f01ffb640 1 -- 172.21.15.104:0/767437883 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef40e3070 con 0x565473876760 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f097fa640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8ef80a2ec0 0x7f8ef80a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f01ffb640 1 -- 172.21.15.104:0/767437883 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ef40d8c80 con 0x565473876760 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f01ffb640 1 -- 172.21.15.104:0/767437883 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef40dd5c0 con 0x565473876760 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f0bfff640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8ef80a2ec0 0x7f8ef80a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.386+0000 7f8f097fa640 1 -- 172.21.15.104:0/2470709597 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8ef40cff40 con 0x7f8f0c101290 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.387+0000 7f8f14d9a740 1 -- 172.21.15.104:0/767437883 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 msgr2=0x565473878b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.387+0000 7f8f0bfff640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8ef80a2ec0 0x7f8ef80a5380 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8f04005fd0 tx=0x7f8f040096a0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.387+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/767437883 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 0x565473878b50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40cd110 tx=0x7f8ef40d3f90 comp rx=0 tx=0).stop 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.387+0000 7f8f14d9a740 1 -- 172.21.15.104:0/767437883 shutdown_connections 2024-06-09T22:21:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.387+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/767437883 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 0x565473878b50 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.387+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/767437883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867970 0x565473876220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.387+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/767437883 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867050 0x565473867430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.387+0000 7f8f14d9a740 1 -- 172.21.15.104:0/767437883 >> 172.21.15.104:0/767437883 conn(0x565473863680 msgr2=0x565473863a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.387+0000 7f8f14d9a740 1 -- 172.21.15.104:0/767437883 shutdown_connections 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.387+0000 7f8f14d9a740 1 -- 172.21.15.104:0/767437883 wait complete. 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.387+0000 7f8f14d9a740 1 -- start start 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f14d9a740 1 Processor -- start 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f14d9a740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 0x565473873730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f14d9a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867970 0x565473873c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f14d9a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473876760 0x5654738741b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f14d9a740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x565473879390 con 0x565473867050 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f14d9a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x565473754ae0 con 0x565473867970 2024-06-09T22:21:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f14d9a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x565473879510 con 0x565473876760 2024-06-09T22:21:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f14d9a740 10 client.0 _ll_register_callbacks cb 0x56547378d9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f0b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 0x565473873730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f0b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 0x565473873730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37828/0 (socket says 172.21.15.104:37828) 2024-06-09T22:21:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f0b7fe640 1 -- 172.21.15.104:0/200325640 learned_addr learned my addr 172.21.15.104:0/200325640 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f03fff640 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867970 0x565473873c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f0bfff640 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473876760 0x5654738741b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f0b7fe640 1 -- 172.21.15.104:0/200325640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473876760 msgr2=0x5654738741b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f0b7fe640 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473876760 0x5654738741b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f0b7fe640 1 -- 172.21.15.104:0/200325640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867970 msgr2=0x565473873c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f0b7fe640 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867970 0x565473873c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.388+0000 7f8f0b7fe640 1 -- 172.21.15.104:0/200325640 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654734df210 con 0x565473867050 2024-06-09T22:21:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8f0b7fe640 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 0x565473873730 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8efc009930 tx=0x7f8efc0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27371f548af93fc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8eeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8eeaffd640 1 -- 172.21.15.104:0/200325640 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc035070 con 0x565473867050 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8eeaffd640 1 -- 172.21.15.104:0/200325640 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8efc004040 con 0x565473867050 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8f14d9a740 1 -- 172.21.15.104:0/200325640 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ef40d8c80 con 0x565473867050 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8eeaffd640 1 -- 172.21.15.104:0/200325640 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8efc004240 con 0x565473867050 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8f14d9a740 10 client.15522 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8f14d9a740 1 -- 172.21.15.104:0/200325640 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5654735ab620 con 0x565473867050 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8eeaffd640 1 -- 172.21.15.104:0/200325640 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ef40d8c80 con 0x565473867050 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8eeaffd640 1 -- 172.21.15.104:0/200325640 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8efc004a30 con 0x565473867050 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8f14d9a740 10 client.15522 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8f14d9a740 10 client.15522 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8f14d9a740 20 client.15522 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5654738869b0 0x565473888e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8f14d9a740 4 client.15522 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.389+0000 7f8f14d9a740 1 -- 172.21.15.104:0/200325640 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654737def10 con 0x5654738869b0 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.390+0000 7f8f03fff640 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5654738869b0 0x565473888e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.390+0000 7f8eeaffd640 10 client.15522 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.390+0000 7f8f03fff640 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5654738869b0 0x565473888e00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=991bf82412128f00 server_cookie=8bf470668a2f4c3b in_seq=0 out_seq=0 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8eeaffd640 1 -- 172.21.15.104:0/200325640 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5654737def10 con 0x5654738869b0 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8eeaffd640 10 client.15522 handle_command_reply: tid=0 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8f14d9a740 1 client.15522 shutdown 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8f14d9a740 1 -- 172.21.15.104:0/200325640 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 msgr2=0x565473873730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 0x565473873730 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8efc009930 tx=0x7f8efc0317e0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8f14d9a740 1 -- 172.21.15.104:0/200325640 shutdown_connections 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5654738869b0 0x565473888e00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473876760 0x5654738741b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8eeaffd640 0 client.15522 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867970 0x565473873c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/200325640 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 0x565473873730 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8f14d9a740 1 -- 172.21.15.104:0/200325640 >> 172.21.15.104:0/200325640 conn(0x565473863680 msgr2=0x5654738694f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8f14d9a740 1 -- 172.21.15.104:0/200325640 shutdown_connections 2024-06-09T22:21:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8f14d9a740 1 -- 172.21.15.104:0/200325640 wait complete. 2024-06-09T22:21:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.391+0000 7f8f14d9a740 20 client.15522 trim_cache size 0 max 16384 2024-06-09T22:21:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.415+0000 7f8f14d9a740 1 Processor -- start 2024-06-09T22:21:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.415+0000 7f8f14d9a740 1 -- start start 2024-06-09T22:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f14d9a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867050 0x565473882da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f14d9a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867970 0x56547388e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f14d9a740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 0x56547388ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f14d9a740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c1acb10 con 0x565473876760 2024-06-09T22:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f14d9a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c1ae650 con 0x565473867050 2024-06-09T22:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f14d9a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c1ad1e0 con 0x565473867970 2024-06-09T22:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f0bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 0x56547388ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f0bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 0x56547388ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37838/0 (socket says 172.21.15.104:37838) 2024-06-09T22:21:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f0bfff640 1 -- 172.21.15.104:0/1026394901 learned_addr learned my addr 172.21.15.104:0/1026394901 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f03fff640 1 --2- 172.21.15.104:0/1026394901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867970 0x56547388e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f0b7fe640 1 --2- 172.21.15.104:0/1026394901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867050 0x565473882da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f0bfff640 1 -- 172.21.15.104:0/1026394901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867970 msgr2=0x56547388e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f0bfff640 1 --2- 172.21.15.104:0/1026394901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867970 0x56547388e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f0bfff640 1 -- 172.21.15.104:0/1026394901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867050 msgr2=0x565473882da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f0bfff640 1 --2- 172.21.15.104:0/1026394901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867050 0x565473882da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f0bfff640 1 -- 172.21.15.104:0/1026394901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654738677a0 con 0x565473876760 2024-06-09T22:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.416+0000 7f8f03fff640 1 --2- 172.21.15.104:0/1026394901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867970 0x56547388e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:21:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.417+0000 7f8f0bfff640 1 --2- 172.21.15.104:0/1026394901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 0x56547388ec50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8f040109a0 tx=0x7f8f04016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866887e757c4d32f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:26.418 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.417+0000 7f8f017fa640 1 -- 172.21.15.104:0/1026394901 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f04003970 con 0x565473876760 2024-06-09T22:21:26.418 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.417+0000 7f8f017fa640 1 -- 172.21.15.104:0/1026394901 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f0401f930 con 0x565473876760 2024-06-09T22:21:26.418 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.417+0000 7f8f017fa640 1 -- 172.21.15.104:0/1026394901 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f04004510 con 0x565473876760 2024-06-09T22:21:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.417+0000 7f8f14d9a740 1 -- 172.21.15.104:0/1026394901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 msgr2=0x56547388ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.417+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/1026394901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 0x56547388ec50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8f040109a0 tx=0x7f8f04016000 comp rx=0 tx=0).stop 2024-06-09T22:21:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.417+0000 7f8f14d9a740 1 -- 172.21.15.104:0/1026394901 shutdown_connections 2024-06-09T22:21:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.417+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/1026394901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473876760 0x56547388ec50 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.417+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/1026394901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867970 0x56547388e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.417+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/1026394901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473867050 0x565473882da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f14d9a740 1 -- 172.21.15.104:0/1026394901 >> 172.21.15.104:0/1026394901 conn(0x565473863680 msgr2=0x565473872000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f14d9a740 1 -- 172.21.15.104:0/1026394901 shutdown_connections 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f14d9a740 1 -- 172.21.15.104:0/1026394901 wait complete. 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f14d9a740 1 -- start start 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f14d9a740 1 Processor -- start 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f14d9a740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 0x565473879f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f14d9a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867970 0x56547387a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f14d9a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473876760 0x5654738e96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f14d9a740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c1adf80 con 0x565473867050 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f14d9a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c1ad8b0 con 0x565473876760 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f14d9a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c1aaad0 con 0x565473867970 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f14d9a740 10 client.0 _ll_register_callbacks cb 0x7f8ef4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f0b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 0x565473879f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.418+0000 7f8f0b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 0x565473879f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37842/0 (socket says 172.21.15.104:37842) 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f0b7fe640 1 -- 172.21.15.104:0/2343801508 learned_addr learned my addr 172.21.15.104:0/2343801508 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f0bfff640 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473876760 0x5654738e96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f03fff640 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867970 0x56547387a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f0b7fe640 1 -- 172.21.15.104:0/2343801508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867970 msgr2=0x56547387a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f0b7fe640 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867970 0x56547387a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f0b7fe640 1 -- 172.21.15.104:0/2343801508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473876760 msgr2=0x5654738e96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f0b7fe640 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473876760 0x5654738e96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f0b7fe640 1 -- 172.21.15.104:0/2343801508 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654738e9ce0 con 0x565473867050 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f0bfff640 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473876760 0x5654738e96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f0b7fe640 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 0x565473879f50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8efc004e80 tx=0x7f8efc004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd59e5be5ccb013 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8eeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8eeaffd640 1 -- 172.21.15.104:0/2343801508 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc035050 con 0x565473867050 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8eeaffd640 1 -- 172.21.15.104:0/2343801508 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8efc0048c0 con 0x565473867050 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f14d9a740 1 -- 172.21.15.104:0/2343801508 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f0401f930 con 0x565473867050 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8eeaffd640 1 -- 172.21.15.104:0/2343801508 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8efc05e050 con 0x565473867050 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f14d9a740 10 client.15528 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8f14d9a740 1 -- 172.21.15.104:0/2343801508 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5654738f10a0 con 0x565473867050 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.419+0000 7f8eeaffd640 1 -- 172.21.15.104:0/2343801508 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f0401f930 con 0x565473867050 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.420+0000 7f8eeaffd640 1 -- 172.21.15.104:0/2343801508 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8efc059030 con 0x565473867050 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.420+0000 7f8f14d9a740 10 client.15528 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.420+0000 7f8f14d9a740 10 client.15528 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.420+0000 7f8f14d9a740 20 client.15528 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.420+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5654738f18d0 0x5654738f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.420+0000 7f8f14d9a740 4 client.15528 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.420+0000 7f8f14d9a740 1 -- 172.21.15.104:0/2343801508 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5654737d8c20 con 0x5654738f18d0 2024-06-09T22:21:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.420+0000 7f8f03fff640 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5654738f18d0 0x5654738f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.421+0000 7f8eeaffd640 10 client.15528 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.421+0000 7f8f03fff640 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5654738f18d0 0x5654738f3cf0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbfdd69a7c34cc0 server_cookie=6340aa9dc8019814 in_seq=0 out_seq=0 2024-06-09T22:21:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8eeaffd640 1 -- 172.21.15.104:0/2343801508 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5654737d8c20 con 0x5654738f18d0 2024-06-09T22:21:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8eeaffd640 10 client.15528 handle_command_reply: tid=0 2024-06-09T22:21:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8f14d9a740 1 client.15528 shutdown 2024-06-09T22:21:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8f14d9a740 1 -- 172.21.15.104:0/2343801508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 msgr2=0x565473879f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 0x565473879f50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8efc004e80 tx=0x7f8efc004610 comp rx=0 tx=0).stop 2024-06-09T22:21:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8f14d9a740 1 -- 172.21.15.104:0/2343801508 shutdown_connections 2024-06-09T22:21:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5654738f18d0 0x5654738f3cf0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565473876760 0x5654738e96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x565473867970 0x56547387a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8f14d9a740 1 --2- 172.21.15.104:0/2343801508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x565473867050 0x565473879f50 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8f14d9a740 1 -- 172.21.15.104:0/2343801508 >> 172.21.15.104:0/2343801508 conn(0x565473863680 msgr2=0x565473861f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8eeaffd640 0 client.15528 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8f14d9a740 1 -- 172.21.15.104:0/2343801508 shutdown_connections 2024-06-09T22:21:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8f14d9a740 1 -- 172.21.15.104:0/2343801508 wait complete. 2024-06-09T22:21:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.433+0000 7f8f14d9a740 20 client.15528 trim_cache size 0 max 16384 2024-06-09T22:21:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:21:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:21:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:21:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.434+0000 7f8f1208c640 1 -- 172.21.15.104:0/2470709597 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8ef80a2ec0 msgr2=0x7f8ef80a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.434+0000 7f8f1208c640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8ef80a2ec0 0x7f8ef80a5380 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8f04005fd0 tx=0x7f8f040096a0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.434+0000 7f8f1208c640 1 -- 172.21.15.104:0/2470709597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f0c101290 msgr2=0x7f8f0c1995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.434+0000 7f8f1208c640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f0c101290 0x7f8f0c1995a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8ef400e9f0 tx=0x7f8ef400edf0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.434+0000 7f8f1208c640 1 -- 172.21.15.104:0/2470709597 shutdown_connections 2024-06-09T22:21:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.434+0000 7f8f1208c640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8ef80a2ec0 0x7f8ef80a5380 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.434+0000 7f8f1208c640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f0c106890 0x7f8f0c1a40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.434+0000 7f8f1208c640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f0c101290 0x7f8f0c1995a0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.434+0000 7f8f1208c640 1 --2- 172.21.15.104:0/2470709597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f0c1008d0 0x7f8f0c199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.434+0000 7f8f1208c640 1 -- 172.21.15.104:0/2470709597 >> 172.21.15.104:0/2470709597 conn(0x7f8f0c077a20 msgr2=0x7f8f0c1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.434+0000 7f8f1208c640 1 -- 172.21.15.104:0/2470709597 shutdown_connections 2024-06-09T22:21:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:26.434+0000 7f8f1208c640 1 -- 172.21.15.104:0/2470709597 wait complete. 2024-06-09T22:21:26.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.973+0000 7f2c40f44700 1 -- 172.21.15.104:0/2160847389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c3c06b810 msgr2=0x7f2c3c1070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:26.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.973+0000 7f2c40f44700 1 --2- 172.21.15.104:0/2160847389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c3c06b810 0x7f2c3c1070d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c0099c0 tx=0x7f2c2c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.973+0000 7f2c40f44700 1 -- 172.21.15.104:0/2160847389 shutdown_connections 2024-06-09T22:21:26.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.973+0000 7f2c40f44700 1 --2- 172.21.15.104:0/2160847389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3c1077b0 0x7f2c3c10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.973+0000 7f2c40f44700 1 --2- 172.21.15.104:0/2160847389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c3c06ae60 0x7f2c3c06b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.973+0000 7f2c40f44700 1 --2- 172.21.15.104:0/2160847389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c3c06b810 0x7f2c3c1070d0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.973+0000 7f2c40f44700 1 -- 172.21.15.104:0/2160847389 >> 172.21.15.104:0/2160847389 conn(0x7f2c3c0773e0 msgr2=0x7f2c3c0777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:26.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.973+0000 7f2c40f44700 1 -- 172.21.15.104:0/2160847389 shutdown_connections 2024-06-09T22:21:26.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.973+0000 7f2c40f44700 1 -- 172.21.15.104:0/2160847389 wait complete. 2024-06-09T22:21:26.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.974+0000 7f2c40f44700 1 Processor -- start 2024-06-09T22:21:26.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.974+0000 7f2c40f44700 1 -- start start 2024-06-09T22:21:26.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.974+0000 7f2c40f44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c3c06ae60 0x7f2c3c1014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.974+0000 7f2c40f44700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c3c06b810 0x7f2c3c101a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.974+0000 7f2c40f44700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3c1077b0 0x7f2c3c101f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.974+0000 7f2c40f44700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c112830 con 0x7f2c3c06ae60 2024-06-09T22:21:26.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.974+0000 7f2c40f44700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c1126b0 con 0x7f2c3c1077b0 2024-06-09T22:21:26.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.974+0000 7f2c40f44700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c1129b0 con 0x7f2c3c06b810 2024-06-09T22:21:26.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c3a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c3c06ae60 0x7f2c3c1014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c3a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c3c06ae60 0x7f2c3c1014f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37868/0 (socket says 172.21.15.104:37868) 2024-06-09T22:21:26.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c3a59c700 1 -- 172.21.15.104:0/2505670 learned_addr learned my addr 172.21.15.104:0/2505670 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:26.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c39d9b700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c3c06b810 0x7f2c3c101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c3ad9d700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3c1077b0 0x7f2c3c101f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c3a59c700 1 -- 172.21.15.104:0/2505670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c3c06b810 msgr2=0x7f2c3c101a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c3a59c700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c3c06b810 0x7f2c3c101a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c3a59c700 1 -- 172.21.15.104:0/2505670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3c1077b0 msgr2=0x7f2c3c101f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c3a59c700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3c1077b0 0x7f2c3c101f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c3a59c700 1 -- 172.21.15.104:0/2505670 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c3c1a7250 con 0x7f2c3c06ae60 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c39d9b700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c3c06b810 0x7f2c3c101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c3a59c700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c3c06ae60 0x7f2c3c1014f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000b530 tx=0x7f2c3000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c2b7fe700 1 -- 172.21.15.104:0/2505670 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c30015020 con 0x7f2c3c06ae60 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c2b7fe700 1 -- 172.21.15.104:0/2505670 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c30011d10 con 0x7f2c3c06ae60 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c40f44700 1 -- 172.21.15.104:0/2505670 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c3c1a7540 con 0x7f2c3c06ae60 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c2b7fe700 1 -- 172.21.15.104:0/2505670 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c30010e50 con 0x7f2c3c06ae60 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.975+0000 7f2c40f44700 1 -- 172.21.15.104:0/2505670 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c3c1a7a00 con 0x7f2c3c06ae60 2024-06-09T22:21:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.976+0000 7f2c40f44700 1 -- 172.21.15.104:0/2505670 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c3c069180 con 0x7f2c3c06ae60 2024-06-09T22:21:26.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.977+0000 7f2c2b7fe700 1 -- 172.21.15.104:0/2505670 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f2c30018020 con 0x7f2c3c06ae60 2024-06-09T22:21:26.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.978+0000 7f2c2b7fe700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2c240a3050 0x7f2c240a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:26.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.978+0000 7f2c2b7fe700 1 -- 172.21.15.104:0/2505670 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f2c300cf6a0 con 0x7f2c3c06ae60 2024-06-09T22:21:26.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.978+0000 7f2c39d9b700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2c240a3050 0x7f2c240a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:26.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.978+0000 7f2c39d9b700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2c240a3050 0x7f2c240a5510 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c000c00 tx=0x7f2c2c005fd0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:26.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:26.982+0000 7f2c2b7fe700 1 -- 172.21.15.104:0/2505670 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2c3001f020 con 0x7f2c3c06ae60 2024-06-09T22:21:27.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.222+0000 7f2c40f44700 1 -- 172.21.15.104:0/2505670 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c3c102a20 con 0x7f2c3c06ae60 2024-06-09T22:21:27.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.223+0000 7f2c2b7fe700 1 -- 172.21.15.104:0/2505670 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11953 (secure 0 0 0) 0x7f2c3009be10 con 0x7f2c3c06ae60 2024-06-09T22:21:27.222 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:27.222 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:20:55.680750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24460,24448,24442,24403,24419]},"id":1}]} 2024-06-09T22:21:27.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.225+0000 7f2c40f44700 1 -- 172.21.15.104:0/2505670 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2c240a3050 msgr2=0x7f2c240a5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:27.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.225+0000 7f2c40f44700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2c240a3050 0x7f2c240a5510 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c000c00 tx=0x7f2c2c005fd0 comp rx=0 tx=0).stop 2024-06-09T22:21:27.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.225+0000 7f2c40f44700 1 -- 172.21.15.104:0/2505670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c3c06ae60 msgr2=0x7f2c3c1014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:27.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.225+0000 7f2c40f44700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c3c06ae60 0x7f2c3c1014f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000b530 tx=0x7f2c3000b8f0 comp rx=0 tx=0).stop 2024-06-09T22:21:27.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.225+0000 7f2c40f44700 1 -- 172.21.15.104:0/2505670 shutdown_connections 2024-06-09T22:21:27.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.225+0000 7f2c40f44700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2c240a3050 0x7f2c240a5510 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:27.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.225+0000 7f2c40f44700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c3c06ae60 0x7f2c3c1014f0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:27.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.225+0000 7f2c40f44700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3c1077b0 0x7f2c3c101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:27.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.225+0000 7f2c40f44700 1 --2- 172.21.15.104:0/2505670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c3c06b810 0x7f2c3c101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:27.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.225+0000 7f2c40f44700 1 -- 172.21.15.104:0/2505670 >> 172.21.15.104:0/2505670 conn(0x7f2c3c0773e0 msgr2=0x7f2c3c106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:27.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.225+0000 7f2c40f44700 1 -- 172.21.15.104:0/2505670 shutdown_connections 2024-06-09T22:21:27.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:27.225+0000 7f2c40f44700 1 -- 172.21.15.104:0/2505670 wait complete. 2024-06-09T22:21:27.225 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 29 2024-06-09T22:21:27.506 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:27 smithi104 ceph-mon[32389]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 38 KiB/s wr, 38 op/s 2024-06-09T22:21:27.506 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:27 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2505670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:27.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:27 smithi153 ceph-mon[36599]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 38 KiB/s wr, 38 op/s 2024-06-09T22:21:27.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:27 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2505670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:27.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:27 smithi194 ceph-mon[36856]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 38 KiB/s wr, 38 op/s 2024-06-09T22:21:27.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:27 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2505670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:28.015 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:21:28.260 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:29.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:29 smithi153 ceph-mon[36599]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-06-09T22:21:29.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:29 smithi104 ceph-mon[32389]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-06-09T22:21:29.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:29 smithi194 ceph-mon[36856]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-06-09T22:21:30.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.075+0000 7f1f6c588700 1 -- 172.21.15.104:0/871918742 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f641030f0 msgr2=0x7f1f64107110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:30.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.075+0000 7f1f6c588700 1 --2- 172.21.15.104:0/871918742 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f641030f0 0x7f1f64107110 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1f600099c0 tx=0x7f1f60009cd0 comp rx=0 tx=0).stop 2024-06-09T22:21:30.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.075+0000 7f1f6c588700 1 -- 172.21.15.104:0/871918742 shutdown_connections 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.075+0000 7f1f6c588700 1 --2- 172.21.15.104:0/871918742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f64107840 0x7f1f6410f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.075+0000 7f1f6c588700 1 --2- 172.21.15.104:0/871918742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f64102740 0x7f1f64102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.075+0000 7f1f6c588700 1 --2- 172.21.15.104:0/871918742 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f641030f0 0x7f1f64107110 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.075+0000 7f1f6c588700 1 -- 172.21.15.104:0/871918742 >> 172.21.15.104:0/871918742 conn(0x7f1f640795f0 msgr2=0x7f1f64079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.076+0000 7f1f6c588700 1 -- 172.21.15.104:0/871918742 shutdown_connections 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.076+0000 7f1f6c588700 1 -- 172.21.15.104:0/871918742 wait complete. 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.076+0000 7f1f6c588700 1 Processor -- start 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.076+0000 7f1f6c588700 1 -- start start 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.076+0000 7f1f6c588700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f64102740 0x7f1f6419e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.076+0000 7f1f6c588700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f641030f0 0x7f1f6419ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.076+0000 7f1f6c588700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f64107840 0x7f1f64198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.076+0000 7f1f6c588700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1f641127d0 con 0x7f1f64102740 2024-06-09T22:21:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6c588700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1f64112650 con 0x7f1f64107840 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6c588700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1f64112950 con 0x7f1f641030f0 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6a324700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f64102740 0x7f1f6419e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6a324700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f64102740 0x7f1f6419e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37882/0 (socket says 172.21.15.104:37882) 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6a324700 1 -- 172.21.15.104:0/486685289 learned_addr learned my addr 172.21.15.104:0/486685289 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f69b23700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f641030f0 0x7f1f6419ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6a324700 1 -- 172.21.15.104:0/486685289 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f641030f0 msgr2=0x7f1f6419ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6a324700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f641030f0 0x7f1f6419ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6a324700 1 -- 172.21.15.104:0/486685289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f64107840 msgr2=0x7f1f64198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6ab25700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f64107840 0x7f1f64198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6a324700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f64107840 0x7f1f64198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6a324700 1 -- 172.21.15.104:0/486685289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f641990f0 con 0x7f1f64102740 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6ab25700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f64107840 0x7f1f64198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6a324700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f64102740 0x7f1f6419e7d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400c9b0 tx=0x7f1f5400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f5b7fe700 1 -- 172.21.15.104:0/486685289 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f54007b00 con 0x7f1f64102740 2024-06-09T22:21:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6c588700 1 -- 172.21.15.104:0/486685289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f641993e0 con 0x7f1f64102740 2024-06-09T22:21:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f5b7fe700 1 -- 172.21.15.104:0/486685289 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f54007ca0 con 0x7f1f64102740 2024-06-09T22:21:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f5b7fe700 1 -- 172.21.15.104:0/486685289 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f54018780 con 0x7f1f64102740 2024-06-09T22:21:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.077+0000 7f1f6c588700 1 -- 172.21.15.104:0/486685289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f641a56a0 con 0x7f1f64102740 2024-06-09T22:21:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.078+0000 7f1f6c588700 1 -- 172.21.15.104:0/486685289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f64069180 con 0x7f1f64102740 2024-06-09T22:21:30.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.079+0000 7f1f5b7fe700 1 -- 172.21.15.104:0/486685289 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f1f5400f450 con 0x7f1f64102740 2024-06-09T22:21:30.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.080+0000 7f1f5b7fe700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1f500a2eb0 0x7f1f500a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:30.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.080+0000 7f1f69b23700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1f500a2eb0 0x7f1f500a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:30.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.080+0000 7f1f5b7fe700 1 -- 172.21.15.104:0/486685289 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1f540cfae0 con 0x7f1f64102740 2024-06-09T22:21:30.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.081+0000 7f1f69b23700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1f500a2eb0 0x7f1f500a5370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1f64108bb0 tx=0x7f1f60005fd0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:30.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.085+0000 7f1f5b7fe700 1 -- 172.21.15.104:0/486685289 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1f54014030 con 0x7f1f64102740 2024-06-09T22:21:30.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.331+0000 7f1f6c588700 1 -- 172.21.15.104:0/486685289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f64199ec0 con 0x7f1f64102740 2024-06-09T22:21:30.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.332+0000 7f1f5b7fe700 1 -- 172.21.15.104:0/486685289 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11953 (secure 0 0 0) 0x7f1f5409c140 con 0x7f1f64102740 2024-06-09T22:21:30.331 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:30.331 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:20:55.680750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24460,24448,24442,24403,24419]},"id":1}]} 2024-06-09T22:21:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.334+0000 7f1f6c588700 1 -- 172.21.15.104:0/486685289 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1f500a2eb0 msgr2=0x7f1f500a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.334+0000 7f1f6c588700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1f500a2eb0 0x7f1f500a5370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1f64108bb0 tx=0x7f1f60005fd0 comp rx=0 tx=0).stop 2024-06-09T22:21:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.334+0000 7f1f6c588700 1 -- 172.21.15.104:0/486685289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f64102740 msgr2=0x7f1f6419e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.334+0000 7f1f6c588700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f64102740 0x7f1f6419e7d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400c9b0 tx=0x7f1f5400cd70 comp rx=0 tx=0).stop 2024-06-09T22:21:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.334+0000 7f1f6c588700 1 -- 172.21.15.104:0/486685289 shutdown_connections 2024-06-09T22:21:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.334+0000 7f1f6c588700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1f500a2eb0 0x7f1f500a5370 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.334+0000 7f1f6c588700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f64102740 0x7f1f6419e7d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.334+0000 7f1f6c588700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f64107840 0x7f1f64198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.334+0000 7f1f6c588700 1 --2- 172.21.15.104:0/486685289 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f641030f0 0x7f1f6419ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.334+0000 7f1f6c588700 1 -- 172.21.15.104:0/486685289 >> 172.21.15.104:0/486685289 conn(0x7f1f640795f0 msgr2=0x7f1f64100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.334+0000 7f1f6c588700 1 -- 172.21.15.104:0/486685289 shutdown_connections 2024-06-09T22:21:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:30.334+0000 7f1f6c588700 1 -- 172.21.15.104:0/486685289 wait complete. 2024-06-09T22:21:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 29 2024-06-09T22:21:30.692 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:30 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/486685289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:30.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:30 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/486685289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:30.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:30 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/486685289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:31.158 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:21:31.403 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:31.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:31 smithi153 ceph-mon[36599]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.4 KiB/s wr, 16 op/s 2024-06-09T22:21:31.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:31 smithi104 ceph-mon[32389]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.4 KiB/s wr, 16 op/s 2024-06-09T22:21:31.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:31 smithi194 ceph-mon[36856]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.4 KiB/s wr, 16 op/s 2024-06-09T22:21:33.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.293+0000 7ffb117d5700 1 -- 172.21.15.104:0/321400646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c107840 msgr2=0x7ffb0c10f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:33.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.293+0000 7ffb117d5700 1 --2- 172.21.15.104:0/321400646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c107840 0x7ffb0c10f3c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400b0d0 tx=0x7ffb0400b3e0 comp rx=0 tx=0).stop 2024-06-09T22:21:33.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.293+0000 7ffb117d5700 1 -- 172.21.15.104:0/321400646 shutdown_connections 2024-06-09T22:21:33.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.293+0000 7ffb117d5700 1 --2- 172.21.15.104:0/321400646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c107840 0x7ffb0c10f3c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:33.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.293+0000 7ffb117d5700 1 --2- 172.21.15.104:0/321400646 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb0c102740 0x7ffb0c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:33.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.293+0000 7ffb117d5700 1 --2- 172.21.15.104:0/321400646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0c1030f0 0x7ffb0c107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:33.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.293+0000 7ffb117d5700 1 -- 172.21.15.104:0/321400646 >> 172.21.15.104:0/321400646 conn(0x7ffb0c0795f0 msgr2=0x7ffb0c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:33.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.293+0000 7ffb117d5700 1 -- 172.21.15.104:0/321400646 shutdown_connections 2024-06-09T22:21:33.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.293+0000 7ffb117d5700 1 -- 172.21.15.104:0/321400646 wait complete. 2024-06-09T22:21:33.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.294+0000 7ffb117d5700 1 Processor -- start 2024-06-09T22:21:33.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.294+0000 7ffb117d5700 1 -- start start 2024-06-09T22:21:33.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.294+0000 7ffb117d5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0c102740 0x7ffb0c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:33.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.294+0000 7ffb117d5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c1030f0 0x7ffb0c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:33.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.294+0000 7ffb117d5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb0c107840 0x7ffb0c1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:33.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.294+0000 7ffb117d5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c112780 con 0x7ffb0c1030f0 2024-06-09T22:21:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.294+0000 7ffb117d5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c112600 con 0x7ffb0c102740 2024-06-09T22:21:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.294+0000 7ffb117d5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c112900 con 0x7ffb0c107840 2024-06-09T22:21:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb0a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c1030f0 0x7ffb0c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb0a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c1030f0 0x7ffb0c19ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37908/0 (socket says 172.21.15.104:37908) 2024-06-09T22:21:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb0a7fc700 1 -- 172.21.15.104:0/3734773254 learned_addr learned my addr 172.21.15.104:0/3734773254 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb0affd700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0c102740 0x7ffb0c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb0b7fe700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb0c107840 0x7ffb0c1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb0a7fc700 1 -- 172.21.15.104:0/3734773254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb0c107840 msgr2=0x7ffb0c1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb0a7fc700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb0c107840 0x7ffb0c1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb0a7fc700 1 -- 172.21.15.104:0/3734773254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0c102740 msgr2=0x7ffb0c19e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb0a7fc700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0c102740 0x7ffb0c19e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb0a7fc700 1 -- 172.21.15.104:0/3734773254 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb0c199020 con 0x7ffb0c1030f0 2024-06-09T22:21:33.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb0a7fc700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c1030f0 0x7ffb0c19ec40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00c9b0 tx=0x7ffafc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:33.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb03fff700 1 -- 172.21.15.104:0/3734773254 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc007b00 con 0x7ffb0c1030f0 2024-06-09T22:21:33.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb117d5700 1 -- 172.21.15.104:0/3734773254 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb0c199310 con 0x7ffb0c1030f0 2024-06-09T22:21:33.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb03fff700 1 -- 172.21.15.104:0/3734773254 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffafc007ca0 con 0x7ffb0c1030f0 2024-06-09T22:21:33.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb03fff700 1 -- 172.21.15.104:0/3734773254 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc018730 con 0x7ffb0c1030f0 2024-06-09T22:21:33.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.295+0000 7ffb117d5700 1 -- 172.21.15.104:0/3734773254 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb0c1a55d0 con 0x7ffb0c1030f0 2024-06-09T22:21:33.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.296+0000 7ffb117d5700 1 -- 172.21.15.104:0/3734773254 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb0c10aa10 con 0x7ffb0c1030f0 2024-06-09T22:21:33.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.298+0000 7ffb03fff700 1 -- 172.21.15.104:0/3734773254 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7ffafc0188d0 con 0x7ffb0c1030f0 2024-06-09T22:21:33.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.298+0000 7ffb03fff700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffaf80a2eb0 0x7ffaf80a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:33.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.298+0000 7ffb03fff700 1 -- 172.21.15.104:0/3734773254 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ffafc0cfcd0 con 0x7ffb0c1030f0 2024-06-09T22:21:33.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.298+0000 7ffb0affd700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffaf80a2eb0 0x7ffaf80a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:33.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.299+0000 7ffb0affd700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffaf80a2eb0 0x7ffaf80a5370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c199a00 tx=0x7ffaf400b3c0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:33.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.302+0000 7ffb03fff700 1 -- 172.21.15.104:0/3734773254 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffafc014030 con 0x7ffb0c1030f0 2024-06-09T22:21:33.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.501+0000 7ffb117d5700 1 -- 172.21.15.104:0/3734773254 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffb0c10a3f0 con 0x7ffb0c1030f0 2024-06-09T22:21:33.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.502+0000 7ffb03fff700 1 -- 172.21.15.104:0/3734773254 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+25144 (secure 0 0 0) 0x7ffafc09c3c0 con 0x7ffb0c1030f0 2024-06-09T22:21:33.502 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:33.502 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":100,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:20:57.590817+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:0/2389242747":"2024-06-10T22:20:57.590784+0000","172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6801/1906048760":"2024-06-10T22:20:57.590784+0000","172.21.15.104:6800/1906048760":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/517954187":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/167857460":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/357628765":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:21:33.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.505+0000 7ffb117d5700 1 -- 172.21.15.104:0/3734773254 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffaf80a2eb0 msgr2=0x7ffaf80a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:33.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.505+0000 7ffb117d5700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffaf80a2eb0 0x7ffaf80a5370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c199a00 tx=0x7ffaf400b3c0 comp rx=0 tx=0).stop 2024-06-09T22:21:33.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.505+0000 7ffb117d5700 1 -- 172.21.15.104:0/3734773254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c1030f0 msgr2=0x7ffb0c19ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:33.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.505+0000 7ffb117d5700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c1030f0 0x7ffb0c19ec40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00c9b0 tx=0x7ffafc00ccc0 comp rx=0 tx=0).stop 2024-06-09T22:21:33.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.505+0000 7ffb117d5700 1 -- 172.21.15.104:0/3734773254 shutdown_connections 2024-06-09T22:21:33.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.505+0000 7ffb117d5700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffaf80a2eb0 0x7ffaf80a5370 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:33.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.505+0000 7ffb117d5700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb0c107840 0x7ffb0c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:33.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.505+0000 7ffb117d5700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0c102740 0x7ffb0c19e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:33.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.505+0000 7ffb117d5700 1 --2- 172.21.15.104:0/3734773254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c1030f0 0x7ffb0c19ec40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:33.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.505+0000 7ffb117d5700 1 -- 172.21.15.104:0/3734773254 >> 172.21.15.104:0/3734773254 conn(0x7ffb0c0795f0 msgr2=0x7ffb0c1004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:33.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.505+0000 7ffb117d5700 1 -- 172.21.15.104:0/3734773254 shutdown_connections 2024-06-09T22:21:33.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:33.505+0000 7ffb117d5700 1 -- 172.21.15.104:0/3734773254 wait complete. 2024-06-09T22:21:33.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:33 smithi153 ceph-mon[36599]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-06-09T22:21:33.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:33 smithi104 ceph-mon[32389]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-06-09T22:21:33.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:33 smithi194 ceph-mon[36856]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-06-09T22:21:34.317 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-06-09T22:21:34.561 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:34.682 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:34 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3734773254' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:21:34.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:34 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3734773254' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:21:34.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:34 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3734773254' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:21:35.724 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:35 smithi104 ceph-mon[32389]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.4 KiB/s wr, 14 op/s 2024-06-09T22:21:35.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:35 smithi153 ceph-mon[36599]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.4 KiB/s wr, 14 op/s 2024-06-09T22:21:35.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:35 smithi194 ceph-mon[36856]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.4 KiB/s wr, 14 op/s 2024-06-09T22:21:37.325 INFO:teuthology.orchestra.run.smithi104.stdout:{"run_dir":"/var/run/ceph"} 2024-06-09T22:21:37.326 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-06-09T22:21:37.326 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 178 2024-06-09T22:21:37.326 DEBUG:tasks.quiescer:Yielding 2024-06-09T22:21:37.326 INFO:teuthology.run_tasks:Running task workunit... 2024-06-09T22:21:37.340 INFO:tasks.workunit:Pulling workunits from ref 7b7a5d375cf4e4afab96b9d4d229b699ae8304bb 2024-06-09T22:21:37.341 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-06-09T22:21:37.341 INFO:tasks.workunit:timeout=3h 2024-06-09T22:21:37.341 INFO:tasks.workunit:cleanup=True 2024-06-09T22:21:37.341 DEBUG:teuthology.orchestra.run.smithi104:> stat -- /home/ubuntu/cephtest/mnt.0 2024-06-09T22:21:37.342 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:21:37.358 INFO:teuthology.orchestra.run.smithi104.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-06-09T22:21:37.359 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-06-09T22:21:37.359 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6bh/107d Inode: 1099511628283 Links: 2 2024-06-09T22:21:37.359 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T22:21:37.359 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-06-09T22:21:37.359 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-06-09 22:18:05.381862828 +0000 2024-06-09T22:21:37.359 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-06-09 22:18:05.381862828 +0000 2024-06-09T22:21:37.359 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-06-09 22:20:54.989610388 +0000 2024-06-09T22:21:37.359 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-06-09 22:18:05.381862828 +0000 2024-06-09T22:21:37.359 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-06-09T22:21:37.359 DEBUG:teuthology.orchestra.run.smithi104:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-06-09T22:21:37.433 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7b7a5d375cf4e4afab96b9d4d229b699ae8304bb 2024-06-09T22:21:37.490 INFO:tasks.workunit.client.0.smithi104.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-06-09T22:21:37.595 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:37.721 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:37 smithi104 ceph-mon[32389]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.4 KiB/s wr, 15 op/s 2024-06-09T22:21:37.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:37 smithi153 ceph-mon[36599]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.4 KiB/s wr, 15 op/s 2024-06-09T22:21:37.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:37 smithi194 ceph-mon[36856]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.4 KiB/s wr, 15 op/s 2024-06-09T22:21:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.329+0000 7fd1685ac700 1 -- 172.21.15.104:0/3476083855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd160107550 msgr2=0x7fd160107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.329+0000 7fd1685ac700 1 --2- 172.21.15.104:0/3476083855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd160107550 0x7fd160107930 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009990 tx=0x7fd15c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.329+0000 7fd1685ac700 1 -- 172.21.15.104:0/3476083855 shutdown_connections 2024-06-09T22:21:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.329+0000 7fd1685ac700 1 --2- 172.21.15.104:0/3476083855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd160107550 0x7fd160107930 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.329+0000 7fd1685ac700 1 --2- 172.21.15.104:0/3476083855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1601017f0 0x7fd16010f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.329+0000 7fd1685ac700 1 --2- 172.21.15.104:0/3476083855 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd160100e30 0x7fd1601012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.329+0000 7fd1685ac700 1 -- 172.21.15.104:0/3476083855 >> 172.21.15.104:0/3476083855 conn(0x7fd1600766b0 msgr2=0x7fd160076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:39.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.329+0000 7fd1685ac700 1 -- 172.21.15.104:0/3476083855 shutdown_connections 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.329+0000 7fd1685ac700 1 -- 172.21.15.104:0/3476083855 wait complete. 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.329+0000 7fd1685ac700 1 Processor -- start 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.329+0000 7fd1685ac700 1 -- start start 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd1685ac700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd160100e30 0x7fd16019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd1685ac700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1601017f0 0x7fd16019ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd1685ac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd160107550 0x7fd160198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd1685ac700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd160112790 con 0x7fd1601017f0 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd1685ac700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd160112610 con 0x7fd160107550 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd1685ac700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd160112910 con 0x7fd160100e30 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd165b47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1601017f0 0x7fd16019ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd165b47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1601017f0 0x7fd16019ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53518/0 (socket says 172.21.15.104:53518) 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd165b47700 1 -- 172.21.15.104:0/1981203183 learned_addr learned my addr 172.21.15.104:0/1981203183 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:39.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd166348700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd160100e30 0x7fd16019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd166b49700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd160107550 0x7fd160198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd165b47700 1 -- 172.21.15.104:0/1981203183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd160100e30 msgr2=0x7fd16019e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd165b47700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd160100e30 0x7fd16019e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd165b47700 1 -- 172.21.15.104:0/1981203183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd160107550 msgr2=0x7fd160198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd165b47700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd160107550 0x7fd160198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd165b47700 1 -- 172.21.15.104:0/1981203183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1601990d0 con 0x7fd1601017f0 2024-06-09T22:21:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd165b47700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1601017f0 0x7fd16019ecf0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd15000c9b0 tx=0x7fd15000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd1577fe700 1 -- 172.21.15.104:0/1981203183 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150007b00 con 0x7fd1601017f0 2024-06-09T22:21:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd1577fe700 1 -- 172.21.15.104:0/1981203183 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd150007ca0 con 0x7fd1601017f0 2024-06-09T22:21:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.330+0000 7fd1685ac700 1 -- 172.21.15.104:0/1981203183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1601993c0 con 0x7fd1601017f0 2024-06-09T22:21:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.331+0000 7fd1577fe700 1 -- 172.21.15.104:0/1981203183 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150018730 con 0x7fd1601017f0 2024-06-09T22:21:39.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.331+0000 7fd1685ac700 1 -- 172.21.15.104:0/1981203183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1601a5680 con 0x7fd1601017f0 2024-06-09T22:21:39.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.332+0000 7fd1685ac700 1 -- 172.21.15.104:0/1981203183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd160069180 con 0x7fd1601017f0 2024-06-09T22:21:39.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.333+0000 7fd1577fe700 1 -- 172.21.15.104:0/1981203183 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7fd1500188d0 con 0x7fd1601017f0 2024-06-09T22:21:39.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.333+0000 7fd1577fe700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd14c0a2f30 0x7fd14c0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.333+0000 7fd166348700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd14c0a2f30 0x7fd14c0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.333+0000 7fd1577fe700 1 -- 172.21.15.104:0/1981203183 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fd1500cfa00 con 0x7fd1601017f0 2024-06-09T22:21:39.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.334+0000 7fd166348700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd14c0a2f30 0x7fd14c0a53f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd15c006010 tx=0x7fd15c005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:39.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.338+0000 7fd1577fe700 1 -- 172.21.15.104:0/1981203183 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd150014030 con 0x7fd1601017f0 2024-06-09T22:21:39.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.524+0000 7fe43195a640 1 -- 172.21.15.104:0/4019234822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe42c1008c0 msgr2=0x7fe42c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.524+0000 7fe43195a640 1 --2- 172.21.15.104:0/4019234822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe42c1008c0 0x7fe42c100d40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe414009930 tx=0x7fe41402a420 comp rx=0 tx=0).stop 2024-06-09T22:21:39.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.524+0000 7fe43195a640 1 -- 172.21.15.104:0/4019234822 shutdown_connections 2024-06-09T22:21:39.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.524+0000 7fe43195a640 1 --2- 172.21.15.104:0/4019234822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe42c101280 0x7fe42c10d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.524+0000 7fe43195a640 1 --2- 172.21.15.104:0/4019234822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe42c1008c0 0x7fe42c100d40 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.524+0000 7fe43195a640 1 --2- 172.21.15.104:0/4019234822 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe42c1068d0 0x7fe42c106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.524+0000 7fe43195a640 1 -- 172.21.15.104:0/4019234822 >> 172.21.15.104:0/4019234822 conn(0x7fe42c076c20 msgr2=0x7fe42c077030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.524+0000 7fe43195a640 1 -- 172.21.15.104:0/4019234822 shutdown_connections 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.524+0000 7fe43195a640 1 -- 172.21.15.104:0/4019234822 wait complete. 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe43195a640 1 Processor -- start 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe43195a640 1 -- start start 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe43195a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe42c1008c0 0x7fe42c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe43195a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe42c101280 0x7fe42c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe43195a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe42c1068d0 0x7fe42c19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe43195a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c110ce0 con 0x7fe42c101280 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe43195a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c110b60 con 0x7fe42c1008c0 2024-06-09T22:21:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe43195a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c110e60 con 0x7fe42c1068d0 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe423fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe42c101280 0x7fe42c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe423fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe42c101280 0x7fe42c198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53534/0 (socket says 172.21.15.104:53534) 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe423fff640 1 -- 172.21.15.104:0/2836240057 learned_addr learned my addr 172.21.15.104:0/2836240057 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe42b7fe640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe42c1068d0 0x7fe42c19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.525+0000 7fe42affd640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe42c1008c0 0x7fe42c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.526+0000 7fe423fff640 1 -- 172.21.15.104:0/2836240057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe42c1068d0 msgr2=0x7fe42c19caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.526+0000 7fe423fff640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe42c1068d0 0x7fe42c19caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.526+0000 7fe423fff640 1 -- 172.21.15.104:0/2836240057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe42c1008c0 msgr2=0x7fe42c198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.526+0000 7fe423fff640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe42c1008c0 0x7fe42c198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.526+0000 7fe423fff640 1 -- 172.21.15.104:0/2836240057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42c19d1d0 con 0x7fe42c101280 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.526+0000 7fe423fff640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe42c101280 0x7fe42c198770 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe414009230 tx=0x7fe414004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.526+0000 7fe430958640 1 -- 172.21.15.104:0/2836240057 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41403f030 con 0x7fe42c101280 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.526+0000 7fe43195a640 1 -- 172.21.15.104:0/2836240057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe42c19d460 con 0x7fe42c101280 2024-06-09T22:21:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.526+0000 7fe430958640 1 -- 172.21.15.104:0/2836240057 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe414003ad0 con 0x7fe42c101280 2024-06-09T22:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.526+0000 7fe430958640 1 -- 172.21.15.104:0/2836240057 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414035070 con 0x7fe42c101280 2024-06-09T22:21:39.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.526+0000 7fe43195a640 1 -- 172.21.15.104:0/2836240057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe42c1a4d50 con 0x7fe42c101280 2024-06-09T22:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe430958640 1 -- 172.21.15.104:0/2836240057 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7fe414042030 con 0x7fe42c101280 2024-06-09T22:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe434669740 1 Processor -- start 2024-06-09T22:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe434669740 1 -- start start 2024-06-09T22:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe434669740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3170 0x555e96eb3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe434669740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3a90 0x555e96ebde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe434669740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 0x555e96ec0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe434669740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x555e96e174f0 con 0x555e96eb3170 2024-06-09T22:21:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe434669740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555e96db5870 con 0x555e96eb3a90 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe434669740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555e96e134b0 con 0x555e96ebe350 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe42b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3170 0x555e96eb3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe42b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3170 0x555e96eb3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53536/0 (socket says 172.21.15.104:53536) 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe42b7fe640 1 -- 172.21.15.104:0/489723452 learned_addr learned my addr 172.21.15.104:0/489723452 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe430958640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe4180a2ec0 0x7fe4180a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe423fff640 1 --2- 172.21.15.104:0/489723452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 0x555e96ec0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe42affd640 1 --2- 172.21.15.104:0/489723452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3a90 0x555e96ebde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe42b7fe640 1 -- 172.21.15.104:0/489723452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 msgr2=0x555e96ec0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe42b7fe640 1 --2- 172.21.15.104:0/489723452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 0x555e96ec0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe42b7fe640 1 -- 172.21.15.104:0/489723452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3a90 msgr2=0x555e96ebde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe42b7fe640 1 --2- 172.21.15.104:0/489723452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3a90 0x555e96ebde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe42b7fe640 1 -- 172.21.15.104:0/489723452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e96dd19b0 con 0x555e96eb3170 2024-06-09T22:21:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.529+0000 7fe42b7fe640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe4180a2ec0 0x7fe4180a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.530+0000 7fe430958640 1 -- 172.21.15.104:0/2836240057 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fe4140efa50 con 0x7fe42c101280 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.530+0000 7fe42b7fe640 1 --2- 172.21.15.104:0/489723452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3170 0x555e96eb3550 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe42400f440 tx=0x7fe42400f900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52dd2e60a2ac24fd server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.530+0000 7fe421ffb640 1 -- 172.21.15.104:0/489723452 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe424019870 con 0x555e96eb3170 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.530+0000 7fe421ffb640 1 -- 172.21.15.104:0/489723452 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42401a410 con 0x555e96eb3170 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.530+0000 7fe421ffb640 1 -- 172.21.15.104:0/489723452 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42400d560 con 0x555e96eb3170 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 -- 172.21.15.104:0/489723452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3170 msgr2=0x555e96eb3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe42b7fe640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe4180a2ec0 0x7fe4180a5380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe424020000 tx=0x7fe42401d000 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 --2- 172.21.15.104:0/489723452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3170 0x555e96eb3550 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe42400f440 tx=0x7fe42400f900 comp rx=0 tx=0).stop 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 -- 172.21.15.104:0/489723452 shutdown_connections 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 --2- 172.21.15.104:0/489723452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 0x555e96ec0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 --2- 172.21.15.104:0/489723452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3a90 0x555e96ebde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 --2- 172.21.15.104:0/489723452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3170 0x555e96eb3550 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 -- 172.21.15.104:0/489723452 >> 172.21.15.104:0/489723452 conn(0x555e96eaf680 msgr2=0x555e96eafa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 -- 172.21.15.104:0/489723452 shutdown_connections 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 -- 172.21.15.104:0/489723452 wait complete. 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 -- start start 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 Processor -- start 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96eb3170 0x555e96ebb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ebb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96ebe350 0x555e96ebbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x555e96ec0f80 con 0x555e96eb3a90 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555e96da0ae0 con 0x555e96ebe350 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.531+0000 7fe434669740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555e96ec1100 con 0x555e96eb3170 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe434669740 10 client.0 _ll_register_callbacks cb 0x555e96dd99b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe423fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ebb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe423fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ebb7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53542/0 (socket says 172.21.15.104:53542) 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe423fff640 1 -- 172.21.15.104:0/2054921272 learned_addr learned my addr 172.21.15.104:0/2054921272 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe42affd640 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96eb3170 0x555e96ebb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe42b7fe640 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96ebe350 0x555e96ebbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe423fff640 1 -- 172.21.15.104:0/2054921272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96eb3170 msgr2=0x555e96ebb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe423fff640 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96eb3170 0x555e96ebb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe423fff640 1 -- 172.21.15.104:0/2054921272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96ebe350 msgr2=0x555e96ebbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe423fff640 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96ebe350 0x555e96ebbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe423fff640 1 -- 172.21.15.104:0/2054921272 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e96b2b210 con 0x555e96eb3a90 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe42affd640 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96eb3170 0x555e96ebb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe423fff640 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ebb7f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fe414009900 tx=0x7fe41402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42306098e36b4012 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe40affd640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe40affd640 1 -- 172.21.15.104:0/2054921272 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414049070 con 0x555e96eb3a90 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe40affd640 1 -- 172.21.15.104:0/2054921272 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fe4140b81a0 con 0x555e96eb3a90 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe434669740 1 -- 172.21.15.104:0/2054921272 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe42401a410 con 0x555e96eb3a90 2024-06-09T22:21:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe40affd640 1 -- 172.21.15.104:0/2054921272 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe4140bbd90 con 0x555e96eb3a90 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe434669740 10 client.15564 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.532+0000 7fe434669740 1 -- 172.21.15.104:0/2054921272 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555e96bf7620 con 0x555e96eb3a90 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.533+0000 7fe40affd640 1 -- 172.21.15.104:0/2054921272 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42401a410 con 0x555e96eb3a90 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.533+0000 7fe40affd640 1 -- 172.21.15.104:0/2054921272 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe41404c060 con 0x555e96eb3a90 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.533+0000 7fe434669740 10 client.15564 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.533+0000 7fe434669740 10 client.15564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.533+0000 7fe434669740 20 client.15564 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.533+0000 7fe434669740 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555e96ece3e0 0x555e96ed08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.533+0000 7fe434669740 4 client.15564 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.533+0000 7fe434669740 1 -- 172.21.15.104:0/2054921272 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e96e2af10 con 0x555e96ece3e0 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.533+0000 7fe42affd640 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555e96ece3e0 0x555e96ed08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.534+0000 7fe40affd640 10 client.15564 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.534+0000 7fe42affd640 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555e96ece3e0 0x555e96ed08a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bca06da51e7e3104 server_cookie=17d30e12e3362495 in_seq=0 out_seq=0 2024-06-09T22:21:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.534+0000 7fe40affd640 1 -- 172.21.15.104:0/2054921272 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x555e96e2af10 con 0x555e96ece3e0 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.534+0000 7fe40affd640 10 client.15564 handle_command_reply: tid=0 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.534+0000 7fe434669740 1 client.15564 shutdown 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.535+0000 7fe434669740 1 -- 172.21.15.104:0/2054921272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 msgr2=0x555e96ebb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.535+0000 7fe434669740 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ebb7f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fe414009900 tx=0x7fe41402a420 comp rx=0 tx=0).stop 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.535+0000 7fe434669740 1 -- 172.21.15.104:0/2054921272 shutdown_connections 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.535+0000 7fe434669740 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555e96ece3e0 0x555e96ed08a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.535+0000 7fe434669740 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96ebe350 0x555e96ebbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.535+0000 7fe434669740 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ebb7f0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.535+0000 7fe434669740 1 --2- 172.21.15.104:0/2054921272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96eb3170 0x555e96ebb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.535+0000 7fe40affd640 0 client.15564 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.535+0000 7fe434669740 1 -- 172.21.15.104:0/2054921272 >> 172.21.15.104:0/2054921272 conn(0x555e96eaf680 msgr2=0x555e96eb5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.535+0000 7fe434669740 1 -- 172.21.15.104:0/2054921272 shutdown_connections 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.535+0000 7fe434669740 1 -- 172.21.15.104:0/2054921272 wait complete. 2024-06-09T22:21:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.535+0000 7fe434669740 20 client.15564 trim_cache size 0 max 16384 2024-06-09T22:21:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe434669740 1 Processor -- start 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe434669740 1 -- start start 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe434669740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3170 0x555e96ed61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe434669740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ed66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe434669740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 0x555e96ebcae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe434669740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c1aecc0 con 0x555e96eb3a90 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe434669740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c1b0800 con 0x555e96eb3170 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe434669740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c1af390 con 0x555e96ebe350 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe423fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ed66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe423fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ed66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53556/0 (socket says 172.21.15.104:53556) 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe423fff640 1 -- 172.21.15.104:0/3180225037 learned_addr learned my addr 172.21.15.104:0/3180225037 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe42affd640 1 --2- 172.21.15.104:0/3180225037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3170 0x555e96ed61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe423fff640 1 -- 172.21.15.104:0/3180225037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 msgr2=0x555e96ebcae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe423fff640 1 --2- 172.21.15.104:0/3180225037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 0x555e96ebcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe423fff640 1 -- 172.21.15.104:0/3180225037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3170 msgr2=0x555e96ed61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe423fff640 1 --2- 172.21.15.104:0/3180225037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3170 0x555e96ed61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe423fff640 1 -- 172.21.15.104:0/3180225037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e96ed0f20 con 0x555e96eb3a90 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe42affd640 1 --2- 172.21.15.104:0/3180225037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3170 0x555e96ed61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe423fff640 1 --2- 172.21.15.104:0/3180225037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ed66f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe4140b8500 tx=0x7fe4140045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed141693cf8b5eb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe4217fa640 1 -- 172.21.15.104:0/3180225037 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414049070 con 0x555e96eb3a90 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe4217fa640 1 -- 172.21.15.104:0/3180225037 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4140c46f0 con 0x555e96eb3a90 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.550+0000 7fe4217fa640 1 -- 172.21.15.104:0/3180225037 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4140d3070 con 0x555e96eb3a90 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.551+0000 7fe434669740 1 -- 172.21.15.104:0/3180225037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 msgr2=0x555e96ed66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.551+0000 7fe434669740 1 --2- 172.21.15.104:0/3180225037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ed66f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe4140b8500 tx=0x7fe4140045f0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.551+0000 7fe434669740 1 -- 172.21.15.104:0/3180225037 shutdown_connections 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.551+0000 7fe434669740 1 --2- 172.21.15.104:0/3180225037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 0x555e96ebcae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.551+0000 7fe434669740 1 --2- 172.21.15.104:0/3180225037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ed66f0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.551+0000 7fe434669740 1 --2- 172.21.15.104:0/3180225037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3170 0x555e96ed61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.551+0000 7fe434669740 1 -- 172.21.15.104:0/3180225037 >> 172.21.15.104:0/3180225037 conn(0x555e96eaf680 msgr2=0x555e96eb6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.551+0000 7fe434669740 1 -- 172.21.15.104:0/3180225037 shutdown_connections 2024-06-09T22:21:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.551+0000 7fe434669740 1 -- 172.21.15.104:0/3180225037 wait complete. 2024-06-09T22:21:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe434669740 1 -- start start 2024-06-09T22:21:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe434669740 1 Processor -- start 2024-06-09T22:21:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe434669740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3170 0x555e96ec6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe434669740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ec1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe434669740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 0x555e96ec20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe434669740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c1b0130 con 0x555e96eb3a90 2024-06-09T22:21:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe434669740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c1afa60 con 0x555e96eb3170 2024-06-09T22:21:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe434669740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c1acc80 con 0x555e96ebe350 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe434669740 10 client.0 _ll_register_callbacks cb 0x7fe414005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe423fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ec1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe423fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ec1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53568/0 (socket says 172.21.15.104:53568) 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe423fff640 1 -- 172.21.15.104:0/861176673 learned_addr learned my addr 172.21.15.104:0/861176673 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe42b7fe640 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 0x555e96ec20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe42affd640 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3170 0x555e96ec6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe423fff640 1 -- 172.21.15.104:0/861176673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 msgr2=0x555e96ec20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe423fff640 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 0x555e96ec20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe423fff640 1 -- 172.21.15.104:0/861176673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3170 msgr2=0x555e96ec6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe423fff640 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3170 0x555e96ec6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.552+0000 7fe423fff640 1 -- 172.21.15.104:0/861176673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e96ec2660 con 0x555e96eb3a90 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe423fff640 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ec1bb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe41403d040 tx=0x7fe4140bffb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf771489e7f89e1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe40affd640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe40affd640 1 -- 172.21.15.104:0/861176673 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414049050 con 0x555e96eb3a90 2024-06-09T22:21:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe40affd640 1 -- 172.21.15.104:0/861176673 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fe414031dd0 con 0x555e96eb3a90 2024-06-09T22:21:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe434669740 1 -- 172.21.15.104:0/861176673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4140c46f0 con 0x555e96eb3a90 2024-06-09T22:21:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe40affd640 1 -- 172.21.15.104:0/861176673 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe4140d3070 con 0x555e96eb3a90 2024-06-09T22:21:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe434669740 10 client.15570 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe434669740 1 -- 172.21.15.104:0/861176673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555e96ec2660 con 0x555e96eb3a90 2024-06-09T22:21:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe40affd640 1 -- 172.21.15.104:0/861176673 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4140c46f0 con 0x555e96eb3a90 2024-06-09T22:21:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe40affd640 1 -- 172.21.15.104:0/861176673 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe4140b7880 con 0x555e96eb3a90 2024-06-09T22:21:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe434669740 10 client.15570 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe434669740 10 client.15570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:21:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe434669740 20 client.15570 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe434669740 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555e96f39210 0x555e96f3b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe434669740 4 client.15570 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:21:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.553+0000 7fe434669740 1 -- 172.21.15.104:0/861176673 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555e96e24c20 con 0x555e96f39210 2024-06-09T22:21:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.554+0000 7fe42affd640 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555e96f39210 0x555e96f3b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.554+0000 7fe40affd640 10 client.15570 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.554+0000 7fe42affd640 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555e96f39210 0x555e96f3b630 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=305c5500ba30888c server_cookie=68631eee657cafc3 in_seq=0 out_seq=0 2024-06-09T22:21:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe40affd640 1 -- 172.21.15.104:0/861176673 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x555e96e24c20 con 0x555e96f39210 2024-06-09T22:21:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe40affd640 10 client.15570 handle_command_reply: tid=0 2024-06-09T22:21:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe434669740 1 client.15570 shutdown 2024-06-09T22:21:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe434669740 1 -- 172.21.15.104:0/861176673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 msgr2=0x555e96ec1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe434669740 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ec1bb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe41403d040 tx=0x7fe4140bffb0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe434669740 1 -- 172.21.15.104:0/861176673 shutdown_connections 2024-06-09T22:21:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe434669740 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555e96f39210 0x555e96f3b630 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe434669740 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555e96ebe350 0x555e96ec20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe40affd640 0 client.15570 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe434669740 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555e96eb3a90 0x555e96ec1bb0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe434669740 1 --2- 172.21.15.104:0/861176673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e96eb3170 0x555e96ec6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe434669740 1 -- 172.21.15.104:0/861176673 >> 172.21.15.104:0/861176673 conn(0x555e96eaf680 msgr2=0x555e96eb4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe434669740 1 -- 172.21.15.104:0/861176673 shutdown_connections 2024-06-09T22:21:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.562+0000 7fe434669740 1 -- 172.21.15.104:0/861176673 wait complete. 2024-06-09T22:21:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe434669740 20 client.15570 trim_cache size 0 max 16384 2024-06-09T22:21:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:21:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:21:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:21:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe43195a640 1 -- 172.21.15.104:0/2836240057 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe4180a2ec0 msgr2=0x7fe4180a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe43195a640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe4180a2ec0 0x7fe4180a5380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe424020000 tx=0x7fe42401d000 comp rx=0 tx=0).stop 2024-06-09T22:21:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe43195a640 1 -- 172.21.15.104:0/2836240057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe42c101280 msgr2=0x7fe42c198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe43195a640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe42c101280 0x7fe42c198770 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe414009230 tx=0x7fe414004270 comp rx=0 tx=0).stop 2024-06-09T22:21:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe43195a640 1 -- 172.21.15.104:0/2836240057 shutdown_connections 2024-06-09T22:21:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe43195a640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe4180a2ec0 0x7fe4180a5380 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe43195a640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe42c1068d0 0x7fe42c19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe43195a640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe42c101280 0x7fe42c198770 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe43195a640 1 --2- 172.21.15.104:0/2836240057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe42c1008c0 0x7fe42c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe43195a640 1 -- 172.21.15.104:0/2836240057 >> 172.21.15.104:0/2836240057 conn(0x7fe42c076c20 msgr2=0x7fe42c06a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe43195a640 1 -- 172.21.15.104:0/2836240057 shutdown_connections 2024-06-09T22:21:39.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:39.563+0000 7fe43195a640 1 -- 172.21.15.104:0/2836240057 wait complete. 2024-06-09T22:21:39.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.598+0000 7fd1685ac700 1 -- 172.21.15.104:0/1981203183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1601a5940 con 0x7fd1601017f0 2024-06-09T22:21:39.598 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:39 smithi104 ceph-mon[32389]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-06-09T22:21:39.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.599+0000 7fd1577fe700 1 -- 172.21.15.104:0/1981203183 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11953 (secure 0 0 0) 0x7fd15009c060 con 0x7fd1601017f0 2024-06-09T22:21:39.598 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:39.598 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:20:55.680750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24460,24448,24442,24403,24419]},"id":1}]} 2024-06-09T22:21:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.601+0000 7fd1685ac700 1 -- 172.21.15.104:0/1981203183 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd14c0a2f30 msgr2=0x7fd14c0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.601+0000 7fd1685ac700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd14c0a2f30 0x7fd14c0a53f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd15c006010 tx=0x7fd15c005960 comp rx=0 tx=0).stop 2024-06-09T22:21:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.601+0000 7fd1685ac700 1 -- 172.21.15.104:0/1981203183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1601017f0 msgr2=0x7fd16019ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.601+0000 7fd1685ac700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1601017f0 0x7fd16019ecf0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd15000c9b0 tx=0x7fd15000ccc0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.602+0000 7fd1685ac700 1 -- 172.21.15.104:0/1981203183 shutdown_connections 2024-06-09T22:21:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.602+0000 7fd1685ac700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd14c0a2f30 0x7fd14c0a53f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.602+0000 7fd1685ac700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd160107550 0x7fd160198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.602+0000 7fd1685ac700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd160100e30 0x7fd16019e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.602+0000 7fd1685ac700 1 --2- 172.21.15.104:0/1981203183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1601017f0 0x7fd16019ecf0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.602+0000 7fd1685ac700 1 -- 172.21.15.104:0/1981203183 >> 172.21.15.104:0/1981203183 conn(0x7fd1600766b0 msgr2=0x7fd1600ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.602+0000 7fd1685ac700 1 -- 172.21.15.104:0/1981203183 shutdown_connections 2024-06-09T22:21:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:39.602+0000 7fd1685ac700 1 -- 172.21.15.104:0/1981203183 wait complete. 2024-06-09T22:21:39.602 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 29 2024-06-09T22:21:39.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:39 smithi153 ceph-mon[36599]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-06-09T22:21:39.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:39 smithi194 ceph-mon[36856]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-06-09T22:21:40.503 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-06-09T22:16:45.150445+0000', 'modified': '2024-06-09T22:20:55.680750+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24460, 'mds_1': 24448, 'mds_2': 24442, 'mds_3': 24419, 'mds_4': 24403}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14571': {'gid': 14571, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6839/647544853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6838', 'nonce': 647544853}, {'type': 'v1', 'addr': '172.21.15.104:6839', 'nonce': 647544853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14583': {'gid': 14583, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6841/3830332242', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6840', 'nonce': 3830332242}, {'type': 'v1', 'addr': '172.21.15.104:6841', 'nonce': 3830332242}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24362': {'gid': 24362, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6833/3968887191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6832', 'nonce': 3968887191}, {'type': 'v1', 'addr': '172.21.15.153:6833', 'nonce': 3968887191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24383': {'gid': 24383, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6835/3075465176', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 3075465176}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 3075465176}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24398': {'gid': 24398, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6837/3294454895', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 3294454895}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 3294454895}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24403': {'gid': 24403, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.194:6833/3284873174', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6832', 'nonce': 3284873174}, {'type': 'v1', 'addr': '172.21.15.194:6833', 'nonce': 3284873174}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24419': {'gid': 24419, 'name': 'b', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.153:6839/779682823', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 779682823}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 779682823}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 26, 'addr': '172.21.15.194:6835/2995819396', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6834', 'nonce': 2995819396}, {'type': 'v1', 'addr': '172.21.15.194:6835', 'nonce': 2995819396}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24448': {'gid': 24448, 'name': 'c', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 25, 'addr': '172.21.15.194:6837/464494707', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6836', 'nonce': 464494707}, {'type': 'v1', 'addr': '172.21.15.194:6837', 'nonce': 464494707}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.194:6839/2028005544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6838', 'nonce': 2028005544}, {'type': 'v1', 'addr': '172.21.15.194:6839', 'nonce': 2028005544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24460, 'qdb_cluster': [24460, 24448, 24442, 24403, 24419]} 2024-06-09T22:21:40.504 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-06-09T22:21:40.504 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.l status 2024-06-09T22:21:40.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:40 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1981203183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:40.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:40 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1981203183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:40.786 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:40.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:40 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1981203183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:41.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:41 smithi153 ceph-mon[36599]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.6 KiB/s wr, 17 op/s 2024-06-09T22:21:41.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:41 smithi104 ceph-mon[32389]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.6 KiB/s wr, 17 op/s 2024-06-09T22:21:41.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:41 smithi194 ceph-mon[36856]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.6 KiB/s wr, 17 op/s 2024-06-09T22:21:42.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.544+0000 7fb41cd3b700 1 -- 172.21.15.104:0/1850644954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb418101a60 msgr2=0x7fb418105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:42.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.544+0000 7fb41cd3b700 1 --2- 172.21.15.104:0/1850644954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb418101a60 0x7fb418105820 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009990 tx=0x7fb40c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.544+0000 7fb41cd3b700 1 -- 172.21.15.104:0/1850644954 shutdown_connections 2024-06-09T22:21:42.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.544+0000 7fb41cd3b700 1 --2- 172.21.15.104:0/1850644954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb418105f50 0x7fb41810da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.544+0000 7fb41cd3b700 1 --2- 172.21.15.104:0/1850644954 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb4181010b0 0x7fb418101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.544+0000 7fb41cd3b700 1 --2- 172.21.15.104:0/1850644954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb418101a60 0x7fb418105820 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.544+0000 7fb41cd3b700 1 -- 172.21.15.104:0/1850644954 >> 172.21.15.104:0/1850644954 conn(0x7fb4180766b0 msgr2=0x7fb418076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:42.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.544+0000 7fb41cd3b700 1 -- 172.21.15.104:0/1850644954 shutdown_connections 2024-06-09T22:21:42.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.544+0000 7fb41cd3b700 1 -- 172.21.15.104:0/1850644954 wait complete. 2024-06-09T22:21:42.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.544+0000 7fb41cd3b700 1 Processor -- start 2024-06-09T22:21:42.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.544+0000 7fb41cd3b700 1 -- start start 2024-06-09T22:21:42.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41cd3b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4181010b0 0x7fb41819e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41cd3b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb418101a60 0x7fb41819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41cd3b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb418105f50 0x7fb4181989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41cd3b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb418112870 con 0x7fb4181010b0 2024-06-09T22:21:42.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41cd3b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb4181126f0 con 0x7fb418101a60 2024-06-09T22:21:42.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41cd3b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb4181129f0 con 0x7fb418105f50 2024-06-09T22:21:42.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4181010b0 0x7fb41819e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4181010b0 0x7fb41819e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53586/0 (socket says 172.21.15.104:53586) 2024-06-09T22:21:42.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41659c700 1 -- 172.21.15.104:0/2533449224 learned_addr learned my addr 172.21.15.104:0/2533449224 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:42.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb416d9d700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb418105f50 0x7fb4181989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb415d9b700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb418101a60 0x7fb41819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41659c700 1 -- 172.21.15.104:0/2533449224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb418105f50 msgr2=0x7fb4181989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:42.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41659c700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb418105f50 0x7fb4181989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41659c700 1 -- 172.21.15.104:0/2533449224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb418101a60 msgr2=0x7fb41819ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:42.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41659c700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb418101a60 0x7fb41819ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41659c700 1 -- 172.21.15.104:0/2533449224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb418199110 con 0x7fb4181010b0 2024-06-09T22:21:42.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb415d9b700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb418101a60 0x7fb41819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:42.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb41659c700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4181010b0 0x7fb41819e7f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb40800d980 tx=0x7fb40800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:42.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.545+0000 7fb4077fe700 1 -- 172.21.15.104:0/2533449224 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb408017070 con 0x7fb4181010b0 2024-06-09T22:21:42.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.546+0000 7fb41cd3b700 1 -- 172.21.15.104:0/2533449224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb418199400 con 0x7fb4181010b0 2024-06-09T22:21:42.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.546+0000 7fb4077fe700 1 -- 172.21.15.104:0/2533449224 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb408004d10 con 0x7fb4181010b0 2024-06-09T22:21:42.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.546+0000 7fb4077fe700 1 -- 172.21.15.104:0/2533449224 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4080055c0 con 0x7fb4181010b0 2024-06-09T22:21:42.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.546+0000 7fb41cd3b700 1 -- 172.21.15.104:0/2533449224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4181a56c0 con 0x7fb4181010b0 2024-06-09T22:21:42.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.548+0000 7fb4077fe700 1 -- 172.21.15.104:0/2533449224 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7fb408004750 con 0x7fb4181010b0 2024-06-09T22:21:42.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.548+0000 7fb4077fe700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb4000a2eb0 0x7fb4000a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.548+0000 7fb415d9b700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb4000a2eb0 0x7fb4000a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.549+0000 7fb4077fe700 1 -- 172.21.15.104:0/2533449224 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb4080cfa20 con 0x7fb4181010b0 2024-06-09T22:21:42.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.549+0000 7fb415d9b700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb4000a2eb0 0x7fb4000a5370 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00b5c0 tx=0x7fb40c005fb0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:42.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.549+0000 7fb4275beb80 1 Processor -- start 2024-06-09T22:21:42.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.549+0000 7fb4275beb80 1 -- start start 2024-06-09T22:21:42.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb4275beb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a12766d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb4275beb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12766dd10 0x55a12767a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb4275beb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 0x55a12767adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb4275beb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a12755ff10 con 0x55a12767a950 2024-06-09T22:21:42.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb4275beb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a1275be800 con 0x55a12766d3f0 2024-06-09T22:21:42.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb4275beb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a127526410 con 0x55a12766dd10 2024-06-09T22:21:42.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb416d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 0x55a12767adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb416d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 0x55a12767adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53602/0 (socket says 172.21.15.104:53602) 2024-06-09T22:21:42.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb416d9d700 1 -- 172.21.15.104:0/2397582259 learned_addr learned my addr 172.21.15.104:0/2397582259 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:42.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb41659c700 1 --2- 172.21.15.104:0/2397582259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a12766d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb415d9b700 1 --2- 172.21.15.104:0/2397582259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12766dd10 0x55a12767a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb416d9d700 1 -- 172.21.15.104:0/2397582259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12766dd10 msgr2=0x55a12767a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:42.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb416d9d700 1 --2- 172.21.15.104:0/2397582259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12766dd10 0x55a12767a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb416d9d700 1 -- 172.21.15.104:0/2397582259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 msgr2=0x55a12766d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:42.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb416d9d700 1 --2- 172.21.15.104:0/2397582259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a12766d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.550+0000 7fb416d9d700 1 -- 172.21.15.104:0/2397582259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a12750d880 con 0x55a12767a950 2024-06-09T22:21:42.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.551+0000 7fb416d9d700 1 --2- 172.21.15.104:0/2397582259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 0x55a12767adb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb41000e190 tx=0x7fb41000e4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bc739b21a47406 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:42.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.551+0000 7fb404ff9700 1 -- 172.21.15.104:0/2397582259 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41000ecd0 con 0x55a12767a950 2024-06-09T22:21:42.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.551+0000 7fb404ff9700 1 -- 172.21.15.104:0/2397582259 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb41000ee70 con 0x55a12767a950 2024-06-09T22:21:42.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.551+0000 7fb404ff9700 1 -- 172.21.15.104:0/2397582259 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410005020 con 0x55a12767a950 2024-06-09T22:21:42.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 -- 172.21.15.104:0/2397582259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 msgr2=0x55a12767adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:42.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 --2- 172.21.15.104:0/2397582259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 0x55a12767adb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb41000e190 tx=0x7fb41000e4a0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 -- 172.21.15.104:0/2397582259 shutdown_connections 2024-06-09T22:21:42.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 --2- 172.21.15.104:0/2397582259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 0x55a12767adb0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 --2- 172.21.15.104:0/2397582259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a12766d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 --2- 172.21.15.104:0/2397582259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12766dd10 0x55a12767a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 -- 172.21.15.104:0/2397582259 >> 172.21.15.104:0/2397582259 conn(0x55a12759a030 msgr2=0x55a127668420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 -- 172.21.15.104:0/2397582259 shutdown_connections 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 -- 172.21.15.104:0/2397582259 wait complete. 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 -- start start 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 Processor -- start 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127681ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 0x55a12767f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12767a950 0x55a12767f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a12755ef70 con 0x55a12766dd10 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.552+0000 7fb4275beb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a1275ba7c0 con 0x55a12766d3f0 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb4275beb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a127682a40 con 0x55a12767a950 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb4275beb80 10 client.0 _ll_register_callbacks cb 0x55a127554560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb415d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 0x55a12767f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb415d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 0x55a12767f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53608/0 (socket says 172.21.15.104:53608) 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb415d9b700 1 -- 172.21.15.104:0/3794944349 learned_addr learned my addr 172.21.15.104:0/3794944349 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb41659c700 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127681ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb416d9d700 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12767a950 0x55a12767f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb415d9b700 1 -- 172.21.15.104:0/3794944349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12767a950 msgr2=0x55a12767f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb415d9b700 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12767a950 0x55a12767f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb415d9b700 1 -- 172.21.15.104:0/3794944349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 msgr2=0x55a127681ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb415d9b700 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127681ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb415d9b700 1 -- 172.21.15.104:0/3794944349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a127584700 con 0x55a12766dd10 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb415d9b700 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 0x55a12767f1e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009990 tx=0x7fb40c019050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fc68fcdcb12ad4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb3fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb3fdffb700 1 -- 172.21.15.104:0/3794944349 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c003d70 con 0x55a12766dd10 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb3fdffb700 1 -- 172.21.15.104:0/3794944349 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb40c0050d0 con 0x55a12766dd10 2024-06-09T22:21:42.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.553+0000 7fb4275beb80 1 -- 172.21.15.104:0/3794944349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb41000ee70 con 0x55a12766dd10 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.554+0000 7fb3fdffb700 1 -- 172.21.15.104:0/3794944349 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb40c03b050 con 0x55a12766dd10 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.554+0000 7fb4275beb80 10 client.15582 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.554+0000 7fb4275beb80 1 -- 172.21.15.104:0/3794944349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a12768d240 con 0x55a12766dd10 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.554+0000 7fb3fdffb700 1 -- 172.21.15.104:0/3794944349 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb41000ee70 con 0x55a12766dd10 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.554+0000 7fb3fdffb700 1 -- 172.21.15.104:0/3794944349 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb40c030410 con 0x55a12766dd10 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.554+0000 7fb4275beb80 10 client.15582 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.554+0000 7fb4275beb80 10 client.15582 resolve_mds: resolved name 'l' to daemon mds.l 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.554+0000 7fb4275beb80 20 client.15582 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.554+0000 7fb4275beb80 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55a12768f8d0 0x55a127690d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.554+0000 7fb4275beb80 4 client.15582 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.554+0000 7fb4275beb80 1 -- 172.21.15.104:0/3794944349 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1275cbd00 con 0x55a12768f8d0 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.555+0000 7fb41659c700 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55a12768f8d0 0x55a127690d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.555+0000 7fb41659c700 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55a12768f8d0 0x55a127690d40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d70d8e3219b2a0e server_cookie=28bdbd8eded9a56d in_seq=0 out_seq=0 2024-06-09T22:21:42.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.555+0000 7fb3fdffb700 10 client.15582 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:21:42.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb3fdffb700 1 -- 172.21.15.104:0/3794944349 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a1275cbd00 con 0x55a12768f8d0 2024-06-09T22:21:42.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb3fdffb700 10 client.15582 handle_command_reply: tid=0 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb4275beb80 1 client.15582 shutdown 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb4275beb80 1 -- 172.21.15.104:0/3794944349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 msgr2=0x55a12767f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb4275beb80 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 0x55a12767f1e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009990 tx=0x7fb40c019050 comp rx=0 tx=0).stop 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb4275beb80 1 -- 172.21.15.104:0/3794944349 shutdown_connections 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb4275beb80 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55a12768f8d0 0x55a127690d40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb4275beb80 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12767a950 0x55a12767f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb4275beb80 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127681ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb4275beb80 1 --2- 172.21.15.104:0/3794944349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 0x55a12767f1e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb3fdffb700 0 client.15582 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb4275beb80 1 -- 172.21.15.104:0/3794944349 >> 172.21.15.104:0/3794944349 conn(0x55a12759a030 msgr2=0x55a12767bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb4275beb80 1 -- 172.21.15.104:0/3794944349 shutdown_connections 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb4275beb80 1 -- 172.21.15.104:0/3794944349 wait complete. 2024-06-09T22:21:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.556+0000 7fb4275beb80 20 client.15582 trim_cache size 0 max 16384 2024-06-09T22:21:42.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.577+0000 7fb4275beb80 1 Processor -- start 2024-06-09T22:21:42.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.577+0000 7fb4275beb80 1 -- start start 2024-06-09T22:21:42.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.577+0000 7fb4275beb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127681a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.577+0000 7fb4275beb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12766dd10 0x55a12767c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.577+0000 7fb4275beb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 0x55a12767c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.577+0000 7fb4275beb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a12768f080 con 0x55a12767a950 2024-06-09T22:21:42.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.577+0000 7fb4275beb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a12768f750 con 0x55a12766d3f0 2024-06-09T22:21:42.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.577+0000 7fb4275beb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a12768fe20 con 0x55a12766dd10 2024-06-09T22:21:42.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb416d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 0x55a12767c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb416d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 0x55a12767c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53618/0 (socket says 172.21.15.104:53618) 2024-06-09T22:21:42.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb416d9d700 1 -- 172.21.15.104:0/583148453 learned_addr learned my addr 172.21.15.104:0/583148453 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:42.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb415d9b700 1 --2- 172.21.15.104:0/583148453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12766dd10 0x55a12767c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb41659c700 1 --2- 172.21.15.104:0/583148453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127681a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb416d9d700 1 -- 172.21.15.104:0/583148453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12766dd10 msgr2=0x55a12767c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:42.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb416d9d700 1 --2- 172.21.15.104:0/583148453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12766dd10 0x55a12767c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb416d9d700 1 -- 172.21.15.104:0/583148453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 msgr2=0x55a127681a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:42.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb416d9d700 1 --2- 172.21.15.104:0/583148453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127681a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb416d9d700 1 -- 172.21.15.104:0/583148453 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a12766e400 con 0x55a12767a950 2024-06-09T22:21:42.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb41659c700 1 --2- 172.21.15.104:0/583148453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127681a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:42.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb416d9d700 1 --2- 172.21.15.104:0/583148453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 0x55a12767c9e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb410007a90 tx=0x7fb410007d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57bcd89dff70d844 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:42.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb3fffff700 1 -- 172.21.15.104:0/583148453 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410022070 con 0x55a12767a950 2024-06-09T22:21:42.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb3fffff700 1 -- 172.21.15.104:0/583148453 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb410007720 con 0x55a12767a950 2024-06-09T22:21:42.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.578+0000 7fb3fffff700 1 -- 172.21.15.104:0/583148453 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4100050c0 con 0x55a12767a950 2024-06-09T22:21:42.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.579+0000 7fb4275beb80 1 -- 172.21.15.104:0/583148453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 msgr2=0x55a12767c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.579+0000 7fb4275beb80 1 --2- 172.21.15.104:0/583148453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 0x55a12767c9e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb410007a90 tx=0x7fb410007d00 comp rx=0 tx=0).stop 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.579+0000 7fb4275beb80 1 -- 172.21.15.104:0/583148453 shutdown_connections 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.579+0000 7fb4275beb80 1 --2- 172.21.15.104:0/583148453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12767a950 0x55a12767c9e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.579+0000 7fb4275beb80 1 --2- 172.21.15.104:0/583148453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127681a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.579+0000 7fb4275beb80 1 --2- 172.21.15.104:0/583148453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12766dd10 0x55a12767c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.579+0000 7fb4275beb80 1 -- 172.21.15.104:0/583148453 >> 172.21.15.104:0/583148453 conn(0x55a12759a030 msgr2=0x55a12767daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.579+0000 7fb4275beb80 1 -- 172.21.15.104:0/583148453 shutdown_connections 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.579+0000 7fb4275beb80 1 -- 172.21.15.104:0/583148453 wait complete. 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb4275beb80 1 -- start start 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb4275beb80 1 Processor -- start 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb4275beb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127694040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb4275beb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 0x55a127694580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb4275beb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12767a950 0x55a127694ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb4275beb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a127690bc0 con 0x55a12766dd10 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb4275beb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a1276904f0 con 0x55a12766d3f0 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb4275beb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a127691290 con 0x55a12767a950 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb4275beb80 10 client.0 _ll_register_callbacks cb 0x55a127554560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb415d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 0x55a127694580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb415d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 0x55a127694580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53626/0 (socket says 172.21.15.104:53626) 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb415d9b700 1 -- 172.21.15.104:0/2492255647 learned_addr learned my addr 172.21.15.104:0/2492255647 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb41659c700 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127694040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb416d9d700 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12767a950 0x55a127694ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb415d9b700 1 -- 172.21.15.104:0/2492255647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12767a950 msgr2=0x55a127694ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb415d9b700 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12767a950 0x55a127694ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb415d9b700 1 -- 172.21.15.104:0/2492255647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 msgr2=0x55a127694040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb415d9b700 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127694040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.580+0000 7fb415d9b700 1 -- 172.21.15.104:0/2492255647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1276989c0 con 0x55a12766dd10 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb415d9b700 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 0x55a127694580 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x55a127669db0 tx=0x7fb40c021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a88c823c9f5ba48 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb3fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb3fdffb700 1 -- 172.21.15.104:0/2492255647 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c045070 con 0x55a12766dd10 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb3fdffb700 1 -- 172.21.15.104:0/2492255647 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb40c003b70 con 0x55a12766dd10 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb4275beb80 1 -- 172.21.15.104:0/2492255647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb410007720 con 0x55a12766dd10 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb3fdffb700 1 -- 172.21.15.104:0/2492255647 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb40c030610 con 0x55a12766dd10 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb4275beb80 10 client.15588 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb4275beb80 1 -- 172.21.15.104:0/2492255647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a1276fb570 con 0x55a12766dd10 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb3fdffb700 1 -- 172.21.15.104:0/2492255647 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb410007720 con 0x55a12766dd10 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb3fdffb700 1 -- 172.21.15.104:0/2492255647 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb40c00fab0 con 0x55a12766dd10 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb4275beb80 10 client.15588 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb4275beb80 10 client.15588 resolve_mds: resolved name 'l' to daemon mds.l 2024-06-09T22:21:42.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb4275beb80 20 client.15588 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb4275beb80 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55a127700140 0x55a1276fee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb4275beb80 4 client.15588 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.581+0000 7fb4275beb80 1 -- 172.21.15.104:0/2492255647 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "status"}) -- 0x55a1275c6280 con 0x55a127700140 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb41659c700 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55a127700140 0x55a1276fee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb41659c700 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55a127700140 0x55a1276fee50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7388fe207fa8663 server_cookie=64eb3382075a1957 in_seq=0 out_seq=0 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb3fdffb700 10 client.15588 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb3fdffb700 1 -- 172.21.15.104:0/2492255647 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55a1275c6280 con 0x55a127700140 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb3fdffb700 10 client.15588 handle_command_reply: tid=0 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb4275beb80 1 client.15588 shutdown 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb4275beb80 1 -- 172.21.15.104:0/2492255647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 msgr2=0x55a127694580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb4275beb80 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 0x55a127694580 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x55a127669db0 tx=0x7fb40c021000 comp rx=0 tx=0).stop 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb4275beb80 1 -- 172.21.15.104:0/2492255647 shutdown_connections 2024-06-09T22:21:42.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb4275beb80 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55a127700140 0x55a1276fee50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb4275beb80 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a12767a950 0x55a127694ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb4275beb80 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a12766d3f0 0x55a127694040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb4275beb80 1 --2- 172.21.15.104:0/2492255647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a12766dd10 0x55a127694580 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb4275beb80 1 -- 172.21.15.104:0/2492255647 >> 172.21.15.104:0/2492255647 conn(0x55a12759a030 msgr2=0x55a12766c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:42.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb3fdffb700 0 client.15588 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:21:42.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb4275beb80 1 -- 172.21.15.104:0/2492255647 shutdown_connections 2024-06-09T22:21:42.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.582+0000 7fb4275beb80 1 -- 172.21.15.104:0/2492255647 wait complete. 2024-06-09T22:21:42.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb4275beb80 20 client.15588 trim_cache size 0 max 16384 2024-06-09T22:21:42.587 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:21:42.587 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:21:42.587 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 4, 2024-06-09T22:21:42.588 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24403, 2024-06-09T22:21:42.588 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:21:42.588 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:21:42.588 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:21:42.588 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 232.146090856, 2024-06-09T22:21:42.588 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 29, 2024-06-09T22:21:42.588 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 100, 2024-06-09T22:21:42.588 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 99, 2024-06-09T22:21:42.588 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 348.68601335300002 2024-06-09T22:21:42.588 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:21:42.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb41cd3b700 1 -- 172.21.15.104:0/2533449224 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb4000a2eb0 msgr2=0x7fb4000a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:42.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb41cd3b700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb4000a2eb0 0x7fb4000a5370 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00b5c0 tx=0x7fb40c005fb0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb41cd3b700 1 -- 172.21.15.104:0/2533449224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4181010b0 msgr2=0x7fb41819e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:42.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb41cd3b700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4181010b0 0x7fb41819e7f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb40800d980 tx=0x7fb40800dd40 comp rx=0 tx=0).stop 2024-06-09T22:21:42.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb41cd3b700 1 -- 172.21.15.104:0/2533449224 shutdown_connections 2024-06-09T22:21:42.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb41cd3b700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb4000a2eb0 0x7fb4000a5370 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb41cd3b700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4181010b0 0x7fb41819e7f0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb41cd3b700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb418105f50 0x7fb4181989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb41cd3b700 1 --2- 172.21.15.104:0/2533449224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb418101a60 0x7fb41819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:42.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb41cd3b700 1 -- 172.21.15.104:0/2533449224 >> 172.21.15.104:0/2533449224 conn(0x7fb4180766b0 msgr2=0x7fb4180fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:42.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb41cd3b700 1 -- 172.21.15.104:0/2533449224 shutdown_connections 2024-06-09T22:21:42.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:42.583+0000 7fb41cd3b700 1 -- 172.21.15.104:0/2533449224 wait complete. 2024-06-09T22:21:43.454 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-06-09T22:21:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:43.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:43 smithi153 ceph-mon[36599]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-06-09T22:21:43.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:43 smithi104 ceph-mon[32389]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-06-09T22:21:43.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:43 smithi194 ceph-mon[36856]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-06-09T22:21:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.397+0000 7f26ff654700 1 -- 172.21.15.104:0/1935356390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26f8076470 msgr2=0x7f26f80746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.397+0000 7f26ff654700 1 --2- 172.21.15.104:0/1935356390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26f8076470 0x7f26f80746c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009990 tx=0x7f26f0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.397+0000 7f26ff654700 1 -- 172.21.15.104:0/1935356390 shutdown_connections 2024-06-09T22:21:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.397+0000 7f26ff654700 1 --2- 172.21.15.104:0/1935356390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26f8076470 0x7f26f80746c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.397+0000 7f26ff654700 1 --2- 172.21.15.104:0/1935356390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26f8113190 0x7f26f8115580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.397+0000 7f26ff654700 1 --2- 172.21.15.104:0/1935356390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26f8074c00 0x7f26f8075080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.397+0000 7f26ff654700 1 -- 172.21.15.104:0/1935356390 >> 172.21.15.104:0/1935356390 conn(0x7f26f8079650 msgr2=0x7f26f8079a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.397+0000 7f26ff654700 1 -- 172.21.15.104:0/1935356390 shutdown_connections 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.397+0000 7f26ff654700 1 -- 172.21.15.104:0/1935356390 wait complete. 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.397+0000 7f26ff654700 1 Processor -- start 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.397+0000 7f26ff654700 1 -- start start 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26ff654700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26f8074c00 0x7f26f819f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26ff654700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26f8076470 0x7f26f819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26ff654700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26f8113190 0x7f26f81a3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26ff654700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f26f8118de0 con 0x7f26f8113190 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26ff654700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f26f8118c60 con 0x7f26f8076470 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26ff654700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f26f8118f60 con 0x7f26f8074c00 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26fd3f0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26f8074c00 0x7f26f819f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26fd3f0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26f8074c00 0x7f26f819f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:46330/0 (socket says 172.21.15.104:46330) 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26fd3f0700 1 -- 172.21.15.104:0/775994119 learned_addr learned my addr 172.21.15.104:0/775994119 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26fdbf1700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26f8113190 0x7f26f81a3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26fd3f0700 1 -- 172.21.15.104:0/775994119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26f8076470 msgr2=0x7f26f819f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26fcbef700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26f8076470 0x7f26f819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26f8076470 0x7f26f819f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26fd3f0700 1 -- 172.21.15.104:0/775994119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26f8113190 msgr2=0x7f26f81a3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26f8113190 0x7f26f81a3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26fd3f0700 1 -- 172.21.15.104:0/775994119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f81a4320 con 0x7f26f8074c00 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26fcbef700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26f8076470 0x7f26f819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.398+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26f8074c00 0x7f26f819f310 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f26f0000c00 tx=0x7f26f000bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.399+0000 7f26f67fc700 1 -- 172.21.15.104:0/775994119 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f0027070 con 0x7f26f8074c00 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.399+0000 7f26ff654700 1 -- 172.21.15.104:0/775994119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f81a45b0 con 0x7f26f8074c00 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.399+0000 7f26ff654700 1 -- 172.21.15.104:0/775994119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f81abe90 con 0x7f26f8074c00 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.399+0000 7f26f67fc700 1 -- 172.21.15.104:0/775994119 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26f0003d70 con 0x7f26f8074c00 2024-06-09T22:21:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.399+0000 7f26f67fc700 1 -- 172.21.15.104:0/775994119 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f0022e60 con 0x7f26f8074c00 2024-06-09T22:21:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.401+0000 7f26f67fc700 1 -- 172.21.15.104:0/775994119 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f26f002a070 con 0x7f26f8074c00 2024-06-09T22:21:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.402+0000 7f26f67fc700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f26e00a2eb0 0x7f26e00a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.402+0000 7f26f67fc700 1 -- 172.21.15.104:0/775994119 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f26f00d08c0 con 0x7f26f8074c00 2024-06-09T22:21:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.402+0000 7f26fcbef700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f26e00a2eb0 0x7f26e00a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f2709ed7b80 1 Processor -- start 2024-06-09T22:21:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f26fcbef700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f26e00a2eb0 0x7f26e00a5370 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f26f8079ec0 tx=0x7f26ec009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f2709ed7b80 1 -- start start 2024-06-09T22:21:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f2709ed7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd6f10 0x557865cd72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f2709ed7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865cd1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f2709ed7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cd1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f2709ed7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557865bc3f10 con 0x557865cd7830 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f2709ed7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557865c22800 con 0x557865cd6f10 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f2709ed7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557865b8a410 con 0x557865cd1750 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f26fcbef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865cd1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f26fcbef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865cd1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53654/0 (socket says 172.21.15.104:53654) 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f26fcbef700 1 -- 172.21.15.104:0/3288887682 learned_addr learned my addr 172.21.15.104:0/3288887682 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.403+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/3288887682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd6f10 0x557865cd72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.404+0000 7f26fdbf1700 1 --2- 172.21.15.104:0/3288887682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cd1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.404+0000 7f26fcbef700 1 -- 172.21.15.104:0/3288887682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 msgr2=0x557865cd1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.404+0000 7f26fcbef700 1 --2- 172.21.15.104:0/3288887682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cd1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.404+0000 7f26fcbef700 1 -- 172.21.15.104:0/3288887682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd6f10 msgr2=0x557865cd72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.404+0000 7f26fcbef700 1 --2- 172.21.15.104:0/3288887682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd6f10 0x557865cd72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.404+0000 7f26fcbef700 1 -- 172.21.15.104:0/3288887682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557865b71880 con 0x557865cd7830 2024-06-09T22:21:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.404+0000 7f26fcbef700 1 --2- 172.21.15.104:0/3288887682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865cd1210 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f26ec010ce0 tx=0x7f26ec013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e544963a64523 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.404+0000 7f26df7fe700 1 -- 172.21.15.104:0/3288887682 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec023070 con 0x557865cd7830 2024-06-09T22:21:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.404+0000 7f26df7fe700 1 -- 172.21.15.104:0/3288887682 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26ec01f450 con 0x557865cd7830 2024-06-09T22:21:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.404+0000 7f26df7fe700 1 -- 172.21.15.104:0/3288887682 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec01e610 con 0x557865cd7830 2024-06-09T22:21:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.405+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288887682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 msgr2=0x557865cd1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.405+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/3288887682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865cd1210 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f26ec010ce0 tx=0x7f26ec013000 comp rx=0 tx=0).stop 2024-06-09T22:21:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.405+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288887682 shutdown_connections 2024-06-09T22:21:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.405+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/3288887682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cd1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.405+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/3288887682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd6f10 0x557865cd72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.405+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/3288887682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865cd1210 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.405+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288887682 >> 172.21.15.104:0/3288887682 conn(0x557865bfe030 msgr2=0x557865ccc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.405+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288887682 shutdown_connections 2024-06-09T22:21:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.405+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288887682 wait complete. 2024-06-09T22:21:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f2709ed7b80 1 -- start start 2024-06-09T22:21:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f2709ed7b80 1 Processor -- start 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f2709ed7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cdd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f2709ed7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd6f10 0x557865cda940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f2709ed7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd7830 0x557865cdae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f2709ed7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557865bc2f70 con 0x557865cd6f10 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f2709ed7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557865c1e7c0 con 0x557865cd7830 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f2709ed7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557865cde210 con 0x557865cd1750 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f2709ed7b80 10 client.0 _ll_register_callbacks cb 0x557865bb8560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f26fcbef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd6f10 0x557865cda940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f26fcbef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd6f10 0x557865cda940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53664/0 (socket says 172.21.15.104:53664) 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f26fcbef700 1 -- 172.21.15.104:0/3288228740 learned_addr learned my addr 172.21.15.104:0/3288228740 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.406+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cdd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f26fdbf1700 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd7830 0x557865cdae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f26fd3f0700 1 -- 172.21.15.104:0/3288228740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd7830 msgr2=0x557865cdae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd7830 0x557865cdae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f26fd3f0700 1 -- 172.21.15.104:0/3288228740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd6f10 msgr2=0x557865cda940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd6f10 0x557865cda940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f26fd3f0700 1 -- 172.21.15.104:0/3288228740 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557865be8700 con 0x557865cd1750 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f26fcbef700 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd6f10 0x557865cda940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cdd5f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f26f00cdb20 tx=0x7f26f009c290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deccf2a1476a3c36 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f26dcff9700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f26dcff9700 1 -- 172.21.15.104:0/3288228740 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f00a0c60 con 0x557865cd1750 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288228740 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26ec01f450 con 0x557865cd1750 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f26dcff9700 1 -- 172.21.15.104:0/3288228740 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f26f00227c0 con 0x557865cd1750 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f26dcff9700 1 -- 172.21.15.104:0/3288228740 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f26f00a5490 con 0x557865cd1750 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f2709ed7b80 10 client.25186 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.407+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288228740 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557865ce8a20 con 0x557865cd1750 2024-06-09T22:21:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.408+0000 7f26dcff9700 1 -- 172.21.15.104:0/3288228740 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26ec01f450 con 0x557865cd1750 2024-06-09T22:21:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.408+0000 7f26dcff9700 1 -- 172.21.15.104:0/3288228740 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f26f00a5000 con 0x557865cd1750 2024-06-09T22:21:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.408+0000 7f2709ed7b80 10 client.25186 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.408+0000 7f2709ed7b80 10 client.25186 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-09T22:21:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.408+0000 7f2709ed7b80 20 client.25186 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.408+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x557865ceb090 0x557865cec500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.408+0000 7f2709ed7b80 4 client.25186 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:21:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.408+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288228740 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557865c2fd00 con 0x557865ceb090 2024-06-09T22:21:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.408+0000 7f26fcbef700 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x557865ceb090 0x557865cec500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.409+0000 7f26fcbef700 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x557865ceb090 0x557865cec500 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d249025fa554f64b server_cookie=d39d13438c468590 in_seq=0 out_seq=0 2024-06-09T22:21:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.409+0000 7f26dcff9700 10 client.25186 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:21:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f26dcff9700 1 -- 172.21.15.104:0/3288228740 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x557865c2fd00 con 0x557865ceb090 2024-06-09T22:21:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f26dcff9700 10 client.25186 handle_command_reply: tid=0 2024-06-09T22:21:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f2709ed7b80 1 client.25186 shutdown 2024-06-09T22:21:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288228740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 msgr2=0x557865cdd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cdd5f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f26f00cdb20 tx=0x7f26f009c290 comp rx=0 tx=0).stop 2024-06-09T22:21:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288228740 shutdown_connections 2024-06-09T22:21:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd7830 0x557865cdae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cdd5f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x557865ceb090 0x557865cec500 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/3288228740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd6f10 0x557865cda940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288228740 >> 172.21.15.104:0/3288228740 conn(0x557865bfe030 msgr2=0x557865ccb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f26dcff9700 0 client.25186 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:21:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288228740 shutdown_connections 2024-06-09T22:21:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f2709ed7b80 1 -- 172.21.15.104:0/3288228740 wait complete. 2024-06-09T22:21:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.410+0000 7f2709ed7b80 20 client.25186 trim_cache size 0 max 16384 2024-06-09T22:21:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.431+0000 7f2709ed7b80 1 Processor -- start 2024-06-09T22:21:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.431+0000 7f2709ed7b80 1 -- start start 2024-06-09T22:21:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.431+0000 7f2709ed7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd1750 0x557865cd1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.431+0000 7f2709ed7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd6f10 0x557865cdaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.431+0000 7f2709ed7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865ccbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.431+0000 7f2709ed7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557865cea840 con 0x557865cd7830 2024-06-09T22:21:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.431+0000 7f2709ed7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557865ceaf10 con 0x557865cd1750 2024-06-09T22:21:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.431+0000 7f2709ed7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557865ceb5e0 con 0x557865cd6f10 2024-06-09T22:21:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.431+0000 7f26fdbf1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865ccbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.431+0000 7f26fdbf1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865ccbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53680/0 (socket says 172.21.15.104:53680) 2024-06-09T22:21:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.431+0000 7f26fdbf1700 1 -- 172.21.15.104:0/2731281752 learned_addr learned my addr 172.21.15.104:0/2731281752 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.432+0000 7f26fcbef700 1 --2- 172.21.15.104:0/2731281752 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd6f10 0x557865cdaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.432+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/2731281752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd1750 0x557865cd1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.432+0000 7f26fdbf1700 1 -- 172.21.15.104:0/2731281752 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd6f10 msgr2=0x557865cdaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.432+0000 7f26fdbf1700 1 --2- 172.21.15.104:0/2731281752 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd6f10 0x557865cdaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.432+0000 7f26fdbf1700 1 -- 172.21.15.104:0/2731281752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd1750 msgr2=0x557865cd1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.432+0000 7f26fdbf1700 1 --2- 172.21.15.104:0/2731281752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd1750 0x557865cd1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.432+0000 7f26fdbf1700 1 -- 172.21.15.104:0/2731281752 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557865cd8eb0 con 0x557865cd7830 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.432+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/2731281752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd1750 0x557865cd1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.432+0000 7f26fdbf1700 1 --2- 172.21.15.104:0/2731281752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865ccbe30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f26e8003e60 tx=0x7f26e8009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a9967c0b0958e7 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.432+0000 7f26deffd700 1 -- 172.21.15.104:0/2731281752 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e801a020 con 0x557865cd7830 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.432+0000 7f26deffd700 1 -- 172.21.15.104:0/2731281752 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26e80046b0 con 0x557865cd7830 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.432+0000 7f26deffd700 1 -- 172.21.15.104:0/2731281752 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e8016800 con 0x557865cd7830 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.433+0000 7f2709ed7b80 1 -- 172.21.15.104:0/2731281752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 msgr2=0x557865ccbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:45.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.433+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/2731281752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865ccbe30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f26e8003e60 tx=0x7f26e8009c20 comp rx=0 tx=0).stop 2024-06-09T22:21:45.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.433+0000 7f2709ed7b80 1 -- 172.21.15.104:0/2731281752 shutdown_connections 2024-06-09T22:21:45.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.433+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/2731281752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865ccbe30 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.433+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/2731281752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd1750 0x557865cd1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.433+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/2731281752 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd6f10 0x557865cdaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.433+0000 7f2709ed7b80 1 -- 172.21.15.104:0/2731281752 >> 172.21.15.104:0/2731281752 conn(0x557865bfe030 msgr2=0x557865cd9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:45.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.433+0000 7f2709ed7b80 1 -- 172.21.15.104:0/2731281752 shutdown_connections 2024-06-09T22:21:45.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.433+0000 7f2709ed7b80 1 -- 172.21.15.104:0/2731281752 wait complete. 2024-06-09T22:21:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.433+0000 7f2709ed7b80 1 -- start start 2024-06-09T22:21:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.433+0000 7f2709ed7b80 1 Processor -- start 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f2709ed7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cf5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f2709ed7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd6f10 0x557865ccb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f2709ed7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865ccb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f2709ed7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557865cec380 con 0x557865cd7830 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f2709ed7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557865cebcb0 con 0x557865cd6f10 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f2709ed7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557865ceca50 con 0x557865cd1750 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f2709ed7b80 10 client.0 _ll_register_callbacks cb 0x557865bb8560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26fdbf1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865ccb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26fdbf1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865ccb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53690/0 (socket says 172.21.15.104:53690) 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26fdbf1700 1 -- 172.21.15.104:0/1163452997 learned_addr learned my addr 172.21.15.104:0/1163452997 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cf5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26fcbef700 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd6f10 0x557865ccb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26fdbf1700 1 -- 172.21.15.104:0/1163452997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 msgr2=0x557865cf5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26fdbf1700 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cf5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26fdbf1700 1 -- 172.21.15.104:0/1163452997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd6f10 msgr2=0x557865ccb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26fdbf1700 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd6f10 0x557865ccb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26fdbf1700 1 -- 172.21.15.104:0/1163452997 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557865ccbc40 con 0x557865cd7830 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26fdbf1700 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865ccb700 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f26e8007f20 tx=0x7f26e8007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f350c330a94f796 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26dcff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26dcff9700 1 -- 172.21.15.104:0/1163452997 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e801a030 con 0x557865cd7830 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f26dcff9700 1 -- 172.21.15.104:0/1163452997 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f26e8004860 con 0x557865cd7830 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.434+0000 7f2709ed7b80 1 -- 172.21.15.104:0/1163452997 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26e80046b0 con 0x557865cd7830 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.435+0000 7f26dcff9700 1 -- 172.21.15.104:0/1163452997 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f26e8032050 con 0x557865cd7830 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.435+0000 7f2709ed7b80 10 client.15606 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.435+0000 7f2709ed7b80 1 -- 172.21.15.104:0/1163452997 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557865cf5760 con 0x557865cd7830 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.435+0000 7f26dcff9700 1 -- 172.21.15.104:0/1163452997 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26e80046b0 con 0x557865cd7830 2024-06-09T22:21:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.435+0000 7f26dcff9700 1 -- 172.21.15.104:0/1163452997 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f26e80297b0 con 0x557865cd7830 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.435+0000 7f2709ed7b80 10 client.15606 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.435+0000 7f2709ed7b80 10 client.15606 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.435+0000 7f2709ed7b80 20 client.15606 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.435+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x557865d5af40 0x557865d59c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.435+0000 7f2709ed7b80 4 client.15606 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.435+0000 7f2709ed7b80 1 -- 172.21.15.104:0/1163452997 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "status"}) -- 0x557865c2a280 con 0x557865d5af40 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.435+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x557865d5af40 0x557865d59c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f26fd3f0700 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x557865d5af40 0x557865d59c50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2d6f1e93fec544af server_cookie=41d3053d02af04c0 in_seq=0 out_seq=0 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f26dcff9700 10 client.15606 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f26dcff9700 1 -- 172.21.15.104:0/1163452997 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x557865c2a280 con 0x557865d5af40 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f26dcff9700 10 client.15606 handle_command_reply: tid=0 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f2709ed7b80 1 client.15606 shutdown 2024-06-09T22:21:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f2709ed7b80 1 -- 172.21.15.104:0/1163452997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 msgr2=0x557865ccb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865ccb700 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f26e8007f20 tx=0x7f26e8007c90 comp rx=0 tx=0).stop 2024-06-09T22:21:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f2709ed7b80 1 -- 172.21.15.104:0/1163452997 shutdown_connections 2024-06-09T22:21:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x557865d5af40 0x557865d59c50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557865cd7830 0x557865ccb700 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557865cd1750 0x557865cf5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f2709ed7b80 1 --2- 172.21.15.104:0/1163452997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557865cd6f10 0x557865ccb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f2709ed7b80 1 -- 172.21.15.104:0/1163452997 >> 172.21.15.104:0/1163452997 conn(0x557865bfe030 msgr2=0x557865cd7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f26dcff9700 0 client.15606 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:21:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f2709ed7b80 1 -- 172.21.15.104:0/1163452997 shutdown_connections 2024-06-09T22:21:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f2709ed7b80 1 -- 172.21.15.104:0/1163452997 wait complete. 2024-06-09T22:21:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.436+0000 7f2709ed7b80 20 client.15606 trim_cache size 0 max 16384 2024-06-09T22:21:45.440 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:21:45.441 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:21:45.441 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 3, 2024-06-09T22:21:45.441 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24419, 2024-06-09T22:21:45.441 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:21:45.441 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:21:45.441 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:21:45.441 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 236.00428471000001, 2024-06-09T22:21:45.441 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 29, 2024-06-09T22:21:45.441 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 100, 2024-06-09T22:21:45.441 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 99, 2024-06-09T22:21:45.441 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 315.42874121699998 2024-06-09T22:21:45.441 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:21:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.437+0000 7f26ff654700 1 -- 172.21.15.104:0/775994119 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f26e00a2eb0 msgr2=0x7f26e00a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.437+0000 7f26ff654700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f26e00a2eb0 0x7f26e00a5370 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f26f8079ec0 tx=0x7f26ec009260 comp rx=0 tx=0).stop 2024-06-09T22:21:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.437+0000 7f26ff654700 1 -- 172.21.15.104:0/775994119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26f8074c00 msgr2=0x7f26f819f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.437+0000 7f26ff654700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26f8074c00 0x7f26f819f310 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f26f0000c00 tx=0x7f26f000bab0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.437+0000 7f26ff654700 1 -- 172.21.15.104:0/775994119 shutdown_connections 2024-06-09T22:21:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.437+0000 7f26ff654700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f26e00a2eb0 0x7f26e00a5370 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.437+0000 7f26ff654700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26f8113190 0x7f26f81a3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.437+0000 7f26ff654700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26f8074c00 0x7f26f819f310 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.437+0000 7f26ff654700 1 --2- 172.21.15.104:0/775994119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26f8076470 0x7f26f819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.437+0000 7f26ff654700 1 -- 172.21.15.104:0/775994119 >> 172.21.15.104:0/775994119 conn(0x7f26f8079650 msgr2=0x7f26f8111ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.437+0000 7f26ff654700 1 -- 172.21.15.104:0/775994119 shutdown_connections 2024-06-09T22:21:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:45.437+0000 7f26ff654700 1 -- 172.21.15.104:0/775994119 wait complete. 2024-06-09T22:21:45.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:45 smithi153 ceph-mon[36599]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-06-09T22:21:45.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:45 smithi104 ceph-mon[32389]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-06-09T22:21:45.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:45 smithi194 ceph-mon[36856]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-06-09T22:21:46.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.f status 2024-06-09T22:21:46.517 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:47.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:47 smithi153 ceph-mon[36599]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-06-09T22:21:47.755 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:47 smithi104 ceph-mon[32389]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-06-09T22:21:47.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:47 smithi194 ceph-mon[36856]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-06-09T22:21:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.384+0000 7f84bb8be700 1 -- 172.21.15.104:0/1983937654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84b4101d30 msgr2=0x7f84b410e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.384+0000 7f84bb8be700 1 --2- 172.21.15.104:0/1983937654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84b4101d30 0x7f84b410e210 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00b0a0 tx=0x7f84ac00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.384+0000 7f84bb8be700 1 -- 172.21.15.104:0/1983937654 shutdown_connections 2024-06-09T22:21:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.384+0000 7f84bb8be700 1 --2- 172.21.15.104:0/1983937654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84b4101d30 0x7f84b410e210 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.384+0000 7f84bb8be700 1 --2- 172.21.15.104:0/1983937654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84b41073d0 0x7f84b41077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.384+0000 7f84bb8be700 1 --2- 172.21.15.104:0/1983937654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84b4101390 0x7f84b41017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.384+0000 7f84bb8be700 1 -- 172.21.15.104:0/1983937654 >> 172.21.15.104:0/1983937654 conn(0x7f84b40782c0 msgr2=0x7f84b40786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.384+0000 7f84bb8be700 1 -- 172.21.15.104:0/1983937654 shutdown_connections 2024-06-09T22:21:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.384+0000 7f84bb8be700 1 -- 172.21.15.104:0/1983937654 wait complete. 2024-06-09T22:21:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.385+0000 7f84bb8be700 1 Processor -- start 2024-06-09T22:21:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.385+0000 7f84bb8be700 1 -- start start 2024-06-09T22:21:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84bb8be700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84b4101390 0x7f84b419e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84bb8be700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84b4101d30 0x7f84b419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84bb8be700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84b41073d0 0x7f84b41989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84bb8be700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f84b4111650 con 0x7f84b4101390 2024-06-09T22:21:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84bb8be700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84b41114d0 con 0x7f84b4101d30 2024-06-09T22:21:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84bb8be700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f84b41117d0 con 0x7f84b41073d0 2024-06-09T22:21:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84b965a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84b4101390 0x7f84b419e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84b965a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84b4101390 0x7f84b419e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44210/0 (socket says 172.21.15.104:44210) 2024-06-09T22:21:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84b965a700 1 -- 172.21.15.104:0/1571735598 learned_addr learned my addr 172.21.15.104:0/1571735598 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84b8e59700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84b4101d30 0x7f84b419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84b9e5b700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84b41073d0 0x7f84b41989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84b965a700 1 -- 172.21.15.104:0/1571735598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84b41073d0 msgr2=0x7f84b41989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84b965a700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84b41073d0 0x7f84b41989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84b965a700 1 -- 172.21.15.104:0/1571735598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84b4101d30 msgr2=0x7f84b419ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84b965a700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84b4101d30 0x7f84b419ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84b965a700 1 -- 172.21.15.104:0/1571735598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84b4199110 con 0x7f84b4101390 2024-06-09T22:21:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84b965a700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84b4101390 0x7f84b419e7f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f84a400d980 tx=0x7f84a400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84aa7fc700 1 -- 172.21.15.104:0/1571735598 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a4016070 con 0x7f84b4101390 2024-06-09T22:21:48.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84aa7fc700 1 -- 172.21.15.104:0/1571735598 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84a4004500 con 0x7f84b4101390 2024-06-09T22:21:48.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84aa7fc700 1 -- 172.21.15.104:0/1571735598 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a4004d00 con 0x7f84b4101390 2024-06-09T22:21:48.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84bb8be700 1 -- 172.21.15.104:0/1571735598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84b4199400 con 0x7f84b4101390 2024-06-09T22:21:48.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.386+0000 7f84bb8be700 1 -- 172.21.15.104:0/1571735598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84b41a56c0 con 0x7f84b4101390 2024-06-09T22:21:48.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.389+0000 7f84aa7fc700 1 -- 172.21.15.104:0/1571735598 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f84a400bbf0 con 0x7f84b4101390 2024-06-09T22:21:48.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.390+0000 7f84aa7fc700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84a00a2eb0 0x7f84a00a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.390+0000 7f84b8e59700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84a00a2eb0 0x7f84a00a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.390+0000 7f84aa7fc700 1 -- 172.21.15.104:0/1571735598 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f84a40cf850 con 0x7f84b4101390 2024-06-09T22:21:48.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.390+0000 7f84b8e59700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84a00a2eb0 0x7f84a00a5370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f84b0005950 tx=0x7f84b00058e0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:48.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84c6141b80 1 Processor -- start 2024-06-09T22:21:48.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84c6141b80 1 -- start start 2024-06-09T22:21:48.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84c6141b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db855f0 0x557d3db859d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84c6141b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db85f10 0x557d3db7f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84c6141b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 0x557d3db802f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84c6141b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557d3dad17c0 con 0x557d3db7fed0 2024-06-09T22:21:48.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84c6141b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557d3da796d0 con 0x557d3db855f0 2024-06-09T22:21:48.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84c6141b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557d3dacd780 con 0x557d3db85f10 2024-06-09T22:21:48.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84b9e5b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 0x557d3db802f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84b9e5b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 0x557d3db802f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44212/0 (socket says 172.21.15.104:44212) 2024-06-09T22:21:48.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84b9e5b700 1 -- 172.21.15.104:0/3044988690 learned_addr learned my addr 172.21.15.104:0/3044988690 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:48.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84b965a700 1 --2- 172.21.15.104:0/3044988690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db855f0 0x557d3db859d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84b8e59700 1 --2- 172.21.15.104:0/3044988690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db85f10 0x557d3db7f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84b9e5b700 1 -- 172.21.15.104:0/3044988690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db85f10 msgr2=0x557d3db7f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:48.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84b9e5b700 1 --2- 172.21.15.104:0/3044988690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db85f10 0x557d3db7f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84b9e5b700 1 -- 172.21.15.104:0/3044988690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db855f0 msgr2=0x557d3db859d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:48.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84b9e5b700 1 --2- 172.21.15.104:0/3044988690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db855f0 0x557d3db859d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.391+0000 7f84b9e5b700 1 -- 172.21.15.104:0/3044988690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d3da6ff00 con 0x557d3db7fed0 2024-06-09T22:21:48.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.392+0000 7f84b9e5b700 1 --2- 172.21.15.104:0/3044988690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 0x557d3db802f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f84ac006070 tx=0x7f84ac009e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fee4889ea005ac4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:48.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.392+0000 7f849f7fe700 1 -- 172.21.15.104:0/3044988690 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac004a50 con 0x557d3db7fed0 2024-06-09T22:21:48.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.392+0000 7f849f7fe700 1 -- 172.21.15.104:0/3044988690 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84ac004bf0 con 0x557d3db7fed0 2024-06-09T22:21:48.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.392+0000 7f849f7fe700 1 -- 172.21.15.104:0/3044988690 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac0052a0 con 0x557d3db7fed0 2024-06-09T22:21:48.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.392+0000 7f84c6141b80 1 -- 172.21.15.104:0/3044988690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 msgr2=0x557d3db802f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:48.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.392+0000 7f84c6141b80 1 --2- 172.21.15.104:0/3044988690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 0x557d3db802f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f84ac006070 tx=0x7f84ac009e10 comp rx=0 tx=0).stop 2024-06-09T22:21:48.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.393+0000 7f84c6141b80 1 -- 172.21.15.104:0/3044988690 shutdown_connections 2024-06-09T22:21:48.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.393+0000 7f84c6141b80 1 --2- 172.21.15.104:0/3044988690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 0x557d3db802f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.393+0000 7f84c6141b80 1 --2- 172.21.15.104:0/3044988690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db855f0 0x557d3db859d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.393+0000 7f84c6141b80 1 --2- 172.21.15.104:0/3044988690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db85f10 0x557d3db7f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.393+0000 7f84c6141b80 1 -- 172.21.15.104:0/3044988690 >> 172.21.15.104:0/3044988690 conn(0x557d3daaac70 msgr2=0x557d3db7a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:48.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.393+0000 7f84c6141b80 1 -- 172.21.15.104:0/3044988690 shutdown_connections 2024-06-09T22:21:48.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.393+0000 7f84c6141b80 1 -- 172.21.15.104:0/3044988690 wait complete. 2024-06-09T22:21:48.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.393+0000 7f84c6141b80 1 -- start start 2024-06-09T22:21:48.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.393+0000 7f84c6141b80 1 Processor -- start 2024-06-09T22:21:48.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.393+0000 7f84c6141b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 0x557d3db8bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.393+0000 7f84c6141b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db855f0 0x557d3db88f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.393+0000 7f84c6141b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db85f10 0x557d3db894a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84c6141b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557d3db8c920 con 0x557d3db7fed0 2024-06-09T22:21:48.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84c6141b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557d3da71ba0 con 0x557d3db855f0 2024-06-09T22:21:48.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84c6141b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557d3db8caa0 con 0x557d3db85f10 2024-06-09T22:21:48.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84c6141b80 10 client.0 _ll_register_callbacks cb 0x557d3da6b8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:48.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84b965a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 0x557d3db8bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84b965a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 0x557d3db8bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44218/0 (socket says 172.21.15.104:44218) 2024-06-09T22:21:48.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84b965a700 1 -- 172.21.15.104:0/3503896445 learned_addr learned my addr 172.21.15.104:0/3503896445 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:48.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84b9e5b700 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db85f10 0x557d3db894a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84b8e59700 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db855f0 0x557d3db88f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84b965a700 1 -- 172.21.15.104:0/3503896445 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db85f10 msgr2=0x557d3db894a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:48.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84b965a700 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db85f10 0x557d3db894a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84b965a700 1 -- 172.21.15.104:0/3503896445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db855f0 msgr2=0x557d3db88f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:48.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84b965a700 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db855f0 0x557d3db88f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84b965a700 1 -- 172.21.15.104:0/3503896445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d3da20880 con 0x557d3db7fed0 2024-06-09T22:21:48.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84b8e59700 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db855f0 0x557d3db88f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:48.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84b965a700 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 0x557d3db8bc60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f84a400b540 tx=0x7f84a409b210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0238a9cfa4c62b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:48.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f849cff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:48.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f849cff9700 1 -- 172.21.15.104:0/3503896445 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a40b4060 con 0x557d3db7fed0 2024-06-09T22:21:48.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f849cff9700 1 -- 172.21.15.104:0/3503896445 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f84a409bc10 con 0x557d3db7fed0 2024-06-09T22:21:48.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84c6141b80 1 -- 172.21.15.104:0/3503896445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84ac004bf0 con 0x557d3db7fed0 2024-06-09T22:21:48.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f849cff9700 1 -- 172.21.15.104:0/3503896445 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f84a409fb60 con 0x557d3db7fed0 2024-06-09T22:21:48.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84c6141b80 10 client.15618 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:48.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.394+0000 7f84c6141b80 1 -- 172.21.15.104:0/3503896445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557d3db97280 con 0x557d3db7fed0 2024-06-09T22:21:48.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.395+0000 7f849cff9700 1 -- 172.21.15.104:0/3503896445 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84ac004bf0 con 0x557d3db7fed0 2024-06-09T22:21:48.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.395+0000 7f849cff9700 1 -- 172.21.15.104:0/3503896445 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f84a40ccd90 con 0x557d3db7fed0 2024-06-09T22:21:48.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.395+0000 7f84c6141b80 10 client.15618 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:48.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.395+0000 7f84c6141b80 10 client.15618 resolve_mds: resolved name 'f' to daemon mds.f 2024-06-09T22:21:48.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.395+0000 7f84c6141b80 20 client.15618 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:48.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.395+0000 7f84c6141b80 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x557d3db99880 0x557d3db9acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.395+0000 7f84c6141b80 4 client.15618 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:21:48.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.395+0000 7f84c6141b80 1 -- 172.21.15.104:0/3503896445 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d3dadecc0 con 0x557d3db99880 2024-06-09T22:21:48.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.395+0000 7f84b8e59700 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x557d3db99880 0x557d3db9acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.395+0000 7f849cff9700 10 client.15618 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:21:48.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.395+0000 7f84b8e59700 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x557d3db99880 0x557d3db9acf0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dfe29514a9cc4d36 server_cookie=a08e0b1718c13c1 in_seq=0 out_seq=0 2024-06-09T22:21:48.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.396+0000 7f849cff9700 1 -- 172.21.15.104:0/3503896445 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x557d3dadecc0 con 0x557d3db99880 2024-06-09T22:21:48.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.396+0000 7f849cff9700 10 client.15618 handle_command_reply: tid=0 2024-06-09T22:21:48.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.396+0000 7f84c6141b80 1 client.15618 shutdown 2024-06-09T22:21:48.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.396+0000 7f84c6141b80 1 -- 172.21.15.104:0/3503896445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 msgr2=0x557d3db8bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:48.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.396+0000 7f84c6141b80 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 0x557d3db8bc60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f84a400b540 tx=0x7f84a409b210 comp rx=0 tx=0).stop 2024-06-09T22:21:48.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.396+0000 7f84c6141b80 1 -- 172.21.15.104:0/3503896445 shutdown_connections 2024-06-09T22:21:48.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.396+0000 7f84c6141b80 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x557d3db99880 0x557d3db9acf0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.396+0000 7f84c6141b80 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db7fed0 0x557d3db8bc60 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.396+0000 7f84c6141b80 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db85f10 0x557d3db894a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.396+0000 7f84c6141b80 1 --2- 172.21.15.104:0/3503896445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db855f0 0x557d3db88f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.396+0000 7f84c6141b80 1 -- 172.21.15.104:0/3503896445 >> 172.21.15.104:0/3503896445 conn(0x557d3daaac70 msgr2=0x557d3db7e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:48.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.396+0000 7f849cff9700 0 client.15618 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:21:48.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.397+0000 7f84c6141b80 1 -- 172.21.15.104:0/3503896445 shutdown_connections 2024-06-09T22:21:48.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.397+0000 7f84c6141b80 1 -- 172.21.15.104:0/3503896445 wait complete. 2024-06-09T22:21:48.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.397+0000 7f84c6141b80 20 client.15618 trim_cache size 0 max 16384 2024-06-09T22:21:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.418+0000 7f84c6141b80 1 Processor -- start 2024-06-09T22:21:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.418+0000 7f84c6141b80 1 -- start start 2024-06-09T22:21:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.418+0000 7f84c6141b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db7fed0 0x557d3db7f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.418+0000 7f84c6141b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db855f0 0x557d3db73db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.418+0000 7f84c6141b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db85f10 0x557d3db742f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.418+0000 7f84c6141b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557d3db99030 con 0x557d3db855f0 2024-06-09T22:21:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.418+0000 7f84c6141b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557d3db99700 con 0x557d3db85f10 2024-06-09T22:21:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.418+0000 7f84c6141b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557d3db99dd0 con 0x557d3db7fed0 2024-06-09T22:21:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.418+0000 7f84b8e59700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db855f0 0x557d3db73db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.418+0000 7f84b8e59700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db855f0 0x557d3db73db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44236/0 (socket says 172.21.15.104:44236) 2024-06-09T22:21:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.418+0000 7f84b8e59700 1 -- 172.21.15.104:0/1612858746 learned_addr learned my addr 172.21.15.104:0/1612858746 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.419+0000 7f84b965a700 1 --2- 172.21.15.104:0/1612858746 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db7fed0 0x557d3db7f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.419+0000 7f84b8e59700 1 -- 172.21.15.104:0/1612858746 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db7fed0 msgr2=0x557d3db7f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.419+0000 7f84b8e59700 1 --2- 172.21.15.104:0/1612858746 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db7fed0 0x557d3db7f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.419+0000 7f84b8e59700 1 -- 172.21.15.104:0/1612858746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db85f10 msgr2=0x557d3db742f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.419+0000 7f84b8e59700 1 --2- 172.21.15.104:0/1612858746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db85f10 0x557d3db742f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.419+0000 7f84b8e59700 1 -- 172.21.15.104:0/1612858746 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d3db7b4b0 con 0x557d3db855f0 2024-06-09T22:21:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.419+0000 7f84b8e59700 1 --2- 172.21.15.104:0/1612858746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db855f0 0x557d3db73db0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f84b4078830 tx=0x7f84b0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4cb2cee1adfa58 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.419+0000 7f849effd700 1 -- 172.21.15.104:0/1612858746 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b0004020 con 0x557d3db855f0 2024-06-09T22:21:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.419+0000 7f849effd700 1 -- 172.21.15.104:0/1612858746 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84b0017040 con 0x557d3db855f0 2024-06-09T22:21:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.419+0000 7f849effd700 1 -- 172.21.15.104:0/1612858746 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b0004a50 con 0x557d3db855f0 2024-06-09T22:21:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.420+0000 7f84c6141b80 1 -- 172.21.15.104:0/1612858746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db855f0 msgr2=0x557d3db73db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:48.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.420+0000 7f84c6141b80 1 --2- 172.21.15.104:0/1612858746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db855f0 0x557d3db73db0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f84b4078830 tx=0x7f84b0013000 comp rx=0 tx=0).stop 2024-06-09T22:21:48.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.420+0000 7f84c6141b80 1 -- 172.21.15.104:0/1612858746 shutdown_connections 2024-06-09T22:21:48.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.420+0000 7f84c6141b80 1 --2- 172.21.15.104:0/1612858746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db85f10 0x557d3db742f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.420+0000 7f84c6141b80 1 --2- 172.21.15.104:0/1612858746 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db7fed0 0x557d3db7f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.420+0000 7f84c6141b80 1 --2- 172.21.15.104:0/1612858746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db855f0 0x557d3db73db0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.420+0000 7f84c6141b80 1 -- 172.21.15.104:0/1612858746 >> 172.21.15.104:0/1612858746 conn(0x557d3daaac70 msgr2=0x557d3db87870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:48.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.420+0000 7f84c6141b80 1 -- 172.21.15.104:0/1612858746 shutdown_connections 2024-06-09T22:21:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.420+0000 7f84c6141b80 1 -- 172.21.15.104:0/1612858746 wait complete. 2024-06-09T22:21:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.420+0000 7f84c6141b80 1 -- start start 2024-06-09T22:21:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.420+0000 7f84c6141b80 1 Processor -- start 2024-06-09T22:21:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84c6141b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db7fed0 0x557d3dba4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84c6141b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db855f0 0x557d3dba46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84c6141b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db85f10 0x557d3db9f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84c6141b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557d3db9ab70 con 0x557d3db85f10 2024-06-09T22:21:48.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84c6141b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557d3db9a4a0 con 0x557d3db7fed0 2024-06-09T22:21:48.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84c6141b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557d3db9b240 con 0x557d3db855f0 2024-06-09T22:21:48.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84c6141b80 10 client.0 _ll_register_callbacks cb 0x557d3da6b8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:48.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84b9e5b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db85f10 0x557d3db9f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84b9e5b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db85f10 0x557d3db9f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44238/0 (socket says 172.21.15.104:44238) 2024-06-09T22:21:48.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84b9e5b700 1 -- 172.21.15.104:0/294570344 learned_addr learned my addr 172.21.15.104:0/294570344 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:48.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84b8e59700 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db855f0 0x557d3dba46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84b965a700 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db7fed0 0x557d3dba4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84b9e5b700 1 -- 172.21.15.104:0/294570344 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db855f0 msgr2=0x557d3dba46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:48.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84b9e5b700 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db855f0 0x557d3dba46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84b9e5b700 1 -- 172.21.15.104:0/294570344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db7fed0 msgr2=0x557d3dba4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:48.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84b9e5b700 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db7fed0 0x557d3dba4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84b9e5b700 1 -- 172.21.15.104:0/294570344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d3dba4cf0 con 0x557d3db85f10 2024-06-09T22:21:48.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84b9e5b700 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db85f10 0x557d3db9f120 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f84ac000f80 tx=0x7f84ac004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b65af0a540b468 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:48.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f849cff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:48.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f849cff9700 1 -- 172.21.15.104:0/294570344 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac0076a0 con 0x557d3db85f10 2024-06-09T22:21:48.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f84c6141b80 1 -- 172.21.15.104:0/294570344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84b0017040 con 0x557d3db85f10 2024-06-09T22:21:48.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.421+0000 7f849cff9700 1 -- 172.21.15.104:0/294570344 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f84ac007840 con 0x557d3db85f10 2024-06-09T22:21:48.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.422+0000 7f849cff9700 1 -- 172.21.15.104:0/294570344 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f84ac007a40 con 0x557d3db85f10 2024-06-09T22:21:48.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.422+0000 7f84c6141b80 10 client.15624 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:48.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.422+0000 7f84c6141b80 1 -- 172.21.15.104:0/294570344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557d3dc056a0 con 0x557d3db85f10 2024-06-09T22:21:48.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.422+0000 7f849cff9700 1 -- 172.21.15.104:0/294570344 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84b0017040 con 0x557d3db85f10 2024-06-09T22:21:48.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.422+0000 7f849cff9700 1 -- 172.21.15.104:0/294570344 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f84ac011070 con 0x557d3db85f10 2024-06-09T22:21:48.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.422+0000 7f84c6141b80 10 client.15624 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:48.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.422+0000 7f84c6141b80 10 client.15624 resolve_mds: resolved name 'f' to daemon mds.f 2024-06-09T22:21:48.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.422+0000 7f84c6141b80 20 client.15624 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:48.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.422+0000 7f84c6141b80 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x557d3dc09cf0 0x557d3dc08a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:48.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.422+0000 7f84c6141b80 4 client.15624 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:21:48.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.422+0000 7f84c6141b80 1 -- 172.21.15.104:0/294570344 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "status"}) -- 0x557d3dad9240 con 0x557d3dc09cf0 2024-06-09T22:21:48.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.422+0000 7f84b965a700 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x557d3dc09cf0 0x557d3dc08a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:48.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84b965a700 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x557d3dc09cf0 0x557d3dc08a00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4e9c438cafbdc358 server_cookie=df511d9e8f9e6285 in_seq=0 out_seq=0 2024-06-09T22:21:48.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f849cff9700 10 client.15624 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:21:48.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f849cff9700 1 -- 172.21.15.104:0/294570344 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x557d3dad9240 con 0x557d3dc09cf0 2024-06-09T22:21:48.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f849cff9700 10 client.15624 handle_command_reply: tid=0 2024-06-09T22:21:48.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84c6141b80 1 client.15624 shutdown 2024-06-09T22:21:48.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84c6141b80 1 -- 172.21.15.104:0/294570344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db85f10 msgr2=0x557d3db9f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:48.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84c6141b80 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db85f10 0x557d3db9f120 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f84ac000f80 tx=0x7f84ac004c80 comp rx=0 tx=0).stop 2024-06-09T22:21:48.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84c6141b80 1 -- 172.21.15.104:0/294570344 shutdown_connections 2024-06-09T22:21:48.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84c6141b80 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x557d3dc09cf0 0x557d3dc08a00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84c6141b80 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557d3db85f10 0x557d3db9f120 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84c6141b80 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557d3db7fed0 0x557d3dba4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f849cff9700 0 client.15624 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:21:48.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84c6141b80 1 --2- 172.21.15.104:0/294570344 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d3db855f0 0x557d3dba46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84c6141b80 1 -- 172.21.15.104:0/294570344 >> 172.21.15.104:0/294570344 conn(0x557d3daaac70 msgr2=0x557d3db862f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:48.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84c6141b80 1 -- 172.21.15.104:0/294570344 shutdown_connections 2024-06-09T22:21:48.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84c6141b80 1 -- 172.21.15.104:0/294570344 wait complete. 2024-06-09T22:21:48.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.423+0000 7f84c6141b80 20 client.15624 trim_cache size 0 max 16384 2024-06-09T22:21:48.449 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:21:48.449 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:21:48.449 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 2, 2024-06-09T22:21:48.449 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24442, 2024-06-09T22:21:48.449 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:21:48.450 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:21:48.450 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:21:48.450 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 240.00397810999999, 2024-06-09T22:21:48.450 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 29, 2024-06-09T22:21:48.450 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 100, 2024-06-09T22:21:48.510 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 99, 2024-06-09T22:21:48.511 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 337.329192261 2024-06-09T22:21:48.511 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:21:48.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.424+0000 7f84bb8be700 1 -- 172.21.15.104:0/1571735598 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84a00a2eb0 msgr2=0x7f84a00a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:48.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.424+0000 7f84bb8be700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84a00a2eb0 0x7f84a00a5370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f84b0005950 tx=0x7f84b00058e0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.424+0000 7f84bb8be700 1 -- 172.21.15.104:0/1571735598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84b4101390 msgr2=0x7f84b419e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:48.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.424+0000 7f84bb8be700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84b4101390 0x7f84b419e7f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f84a400d980 tx=0x7f84a400dd40 comp rx=0 tx=0).stop 2024-06-09T22:21:48.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.424+0000 7f84bb8be700 1 -- 172.21.15.104:0/1571735598 shutdown_connections 2024-06-09T22:21:48.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.424+0000 7f84bb8be700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84a00a2eb0 0x7f84a00a5370 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.424+0000 7f84bb8be700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84b4101390 0x7f84b419e7f0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.424+0000 7f84bb8be700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84b41073d0 0x7f84b41989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.424+0000 7f84bb8be700 1 --2- 172.21.15.104:0/1571735598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84b4101d30 0x7f84b419ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:48.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.424+0000 7f84bb8be700 1 -- 172.21.15.104:0/1571735598 >> 172.21.15.104:0/1571735598 conn(0x7f84b40782c0 msgr2=0x7f84b4100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:48.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.424+0000 7f84bb8be700 1 -- 172.21.15.104:0/1571735598 shutdown_connections 2024-06-09T22:21:48.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:48.424+0000 7f84bb8be700 1 -- 172.21.15.104:0/1571735598 wait complete. 2024-06-09T22:21:48.814 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:21:49.272 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.c status 2024-06-09T22:21:49.393 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:49.505 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:49 smithi104 ceph-mon[32389]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-06-09T22:21:49.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:49 smithi153 ceph-mon[36599]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-06-09T22:21:49.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:49 smithi194 ceph-mon[36856]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-06-09T22:21:50.146 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:51.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.309+0000 7f6da1db9700 1 -- 172.21.15.104:0/552992161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d9c108500 msgr2=0x7f6d9c1088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:51.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.309+0000 7f6da1db9700 1 --2- 172.21.15.104:0/552992161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d9c108500 0x7f6d9c1088e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009990 tx=0x7f6d8c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.309+0000 7f6da1db9700 1 -- 172.21.15.104:0/552992161 shutdown_connections 2024-06-09T22:21:51.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.309+0000 7f6da1db9700 1 --2- 172.21.15.104:0/552992161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d9c108500 0x7f6d9c1088e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.309+0000 7f6da1db9700 1 --2- 172.21.15.104:0/552992161 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d9c06ab30 0x7f6d9c10f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.309+0000 7f6da1db9700 1 --2- 172.21.15.104:0/552992161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c06a170 0x7f6d9c06a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.309+0000 7f6da1db9700 1 -- 172.21.15.104:0/552992161 >> 172.21.15.104:0/552992161 conn(0x7f6d9c078340 msgr2=0x7f6d9c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:51.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.309+0000 7f6da1db9700 1 -- 172.21.15.104:0/552992161 shutdown_connections 2024-06-09T22:21:51.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.309+0000 7f6da1db9700 1 -- 172.21.15.104:0/552992161 wait complete. 2024-06-09T22:21:51.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6da1db9700 1 Processor -- start 2024-06-09T22:21:51.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6da1db9700 1 -- start start 2024-06-09T22:21:51.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6da1db9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c06a170 0x7f6d9c101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6da1db9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d9c06ab30 0x7f6d9c1019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6da1db9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d9c108500 0x7f6d9c101ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6da1db9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c112760 con 0x7f6d9c108500 2024-06-09T22:21:51.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6da1db9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c1125e0 con 0x7f6d9c06a170 2024-06-09T22:21:51.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6da1db9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c1128e0 con 0x7f6d9c06ab30 2024-06-09T22:21:51.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6d9bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d9c108500 0x7f6d9c101ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6d9bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d9c108500 0x7f6d9c101ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44258/0 (socket says 172.21.15.104:44258) 2024-06-09T22:21:51.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6d9bfff700 1 -- 172.21.15.104:0/564185446 learned_addr learned my addr 172.21.15.104:0/564185446 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:51.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6d9affd700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d9c06ab30 0x7f6d9c1019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.310+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c06a170 0x7f6d9c101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.311+0000 7f6d9bfff700 1 -- 172.21.15.104:0/564185446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d9c06ab30 msgr2=0x7f6d9c1019a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:51.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.311+0000 7f6d9bfff700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d9c06ab30 0x7f6d9c1019a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.311+0000 7f6d9bfff700 1 -- 172.21.15.104:0/564185446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c06a170 msgr2=0x7f6d9c101460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:51.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.311+0000 7f6d9bfff700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c06a170 0x7f6d9c101460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.311+0000 7f6d9bfff700 1 -- 172.21.15.104:0/564185446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d9c1a70a0 con 0x7f6d9c108500 2024-06-09T22:21:51.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.311+0000 7f6d9bfff700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d9c108500 0x7f6d9c101ee0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6d94007d70 tx=0x7f6d9400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:51.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.311+0000 7f6d98ff9700 1 -- 172.21.15.104:0/564185446 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94016020 con 0x7f6d9c108500 2024-06-09T22:21:51.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.311+0000 7f6d98ff9700 1 -- 172.21.15.104:0/564185446 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d9400ac90 con 0x7f6d9c108500 2024-06-09T22:21:51.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.311+0000 7f6d98ff9700 1 -- 172.21.15.104:0/564185446 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d940115c0 con 0x7f6d9c108500 2024-06-09T22:21:51.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.311+0000 7f6da1db9700 1 -- 172.21.15.104:0/564185446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d9c1a7390 con 0x7f6d9c108500 2024-06-09T22:21:51.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.311+0000 7f6da1db9700 1 -- 172.21.15.104:0/564185446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d9c1a7880 con 0x7f6d9c108500 2024-06-09T22:21:51.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.313+0000 7f6d98ff9700 1 -- 172.21.15.104:0/564185446 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f6d94019020 con 0x7f6d9c108500 2024-06-09T22:21:51.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.314+0000 7f6d98ff9700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6d840a2eb0 0x7f6d840a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.314+0000 7f6d98ff9700 1 -- 172.21.15.104:0/564185446 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6d940cf870 con 0x7f6d9c108500 2024-06-09T22:21:51.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.314+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6d840a2eb0 0x7f6d840a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.315+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6d840a2eb0 0x7f6d840a5370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c00b5c0 tx=0x7f6d8c005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:51.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.315+0000 7f6dac63cb80 1 Processor -- start 2024-06-09T22:21:51.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.315+0000 7f6dac63cb80 1 -- start start 2024-06-09T22:21:51.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.315+0000 7f6dac63cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b6820df60 0x555b6820e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.315+0000 7f6dac63cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b68208260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.315+0000 7f6dac63cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b682087a0 0x555b68208bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.315+0000 7f6dac63cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x555b680faf60 con 0x555b682087a0 2024-06-09T22:21:51.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.315+0000 7f6dac63cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555b68159850 con 0x555b6820df60 2024-06-09T22:21:51.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.315+0000 7f6dac63cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555b680c1460 con 0x555b6820e880 2024-06-09T22:21:51.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.316+0000 7f6d9affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b68208260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.316+0000 7f6d9affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b68208260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:46424/0 (socket says 172.21.15.104:46424) 2024-06-09T22:21:51.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.316+0000 7f6d9affd700 1 -- 172.21.15.104:0/3599918872 learned_addr learned my addr 172.21.15.104:0/3599918872 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:51.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.316+0000 7f6d9bfff700 1 --2- 172.21.15.104:0/3599918872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b682087a0 0x555b68208bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.316+0000 7f6d9affd700 1 -- 172.21.15.104:0/3599918872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b6820df60 msgr2=0x555b6820e340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-09T22:21:51.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.316+0000 7f6d9affd700 1 --2- 172.21.15.104:0/3599918872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b6820df60 0x555b6820e340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.316+0000 7f6d9affd700 1 -- 172.21.15.104:0/3599918872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b682087a0 msgr2=0x555b68208bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:51.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.316+0000 7f6d9affd700 1 --2- 172.21.15.104:0/3599918872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b682087a0 0x555b68208bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.316+0000 7f6d9affd700 1 -- 172.21.15.104:0/3599918872 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b680a88d0 con 0x555b6820e880 2024-06-09T22:21:51.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.316+0000 7f6d9affd700 1 --2- 172.21.15.104:0/3599918872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b68208260 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6d90007c60 tx=0x7f6d90007f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91c1cc7e4dac391f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:51.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.316+0000 7f6d81ffb700 1 -- 172.21.15.104:0/3599918872 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d90010ca0 con 0x555b6820e880 2024-06-09T22:21:51.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.317+0000 7f6d81ffb700 1 -- 172.21.15.104:0/3599918872 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d90015070 con 0x555b6820e880 2024-06-09T22:21:51.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.317+0000 7f6d81ffb700 1 -- 172.21.15.104:0/3599918872 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d90003df0 con 0x555b6820e880 2024-06-09T22:21:51.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.317+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3599918872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 msgr2=0x555b68208260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:51.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.317+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3599918872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b68208260 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6d90007c60 tx=0x7f6d90007f70 comp rx=0 tx=0).stop 2024-06-09T22:21:51.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.317+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3599918872 shutdown_connections 2024-06-09T22:21:51.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.317+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3599918872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b682087a0 0x555b68208bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.317+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3599918872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b6820df60 0x555b6820e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.317+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3599918872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b68208260 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.317+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3599918872 >> 172.21.15.104:0/3599918872 conn(0x555b68135080 msgr2=0x555b68203220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:51.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.317+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3599918872 shutdown_connections 2024-06-09T22:21:51.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.317+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3599918872 wait complete. 2024-06-09T22:21:51.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.318+0000 7f6dac63cb80 1 -- start start 2024-06-09T22:21:51.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.318+0000 7f6dac63cb80 1 Processor -- start 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.318+0000 7f6dac63cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b682087a0 0x555b682146d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.318+0000 7f6dac63cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820df60 0x555b682119d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.318+0000 7f6dac63cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820e880 0x555b68211f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.318+0000 7f6dac63cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x555b680f9fc0 con 0x555b6820e880 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.318+0000 7f6dac63cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555b68155810 con 0x555b682087a0 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.318+0000 7f6dac63cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555b68215260 con 0x555b6820df60 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.318+0000 7f6dac63cb80 10 client.0 _ll_register_callbacks cb 0x555b680ef5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.318+0000 7f6d9bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820e880 0x555b68211f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.318+0000 7f6d9bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820e880 0x555b68211f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44278/0 (socket says 172.21.15.104:44278) 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.318+0000 7f6d9bfff700 1 -- 172.21.15.104:0/3907262206 learned_addr learned my addr 172.21.15.104:0/3907262206 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b682087a0 0x555b682146d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6d9affd700 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820df60 0x555b682119d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6d9b7fe700 1 -- 172.21.15.104:0/3907262206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820df60 msgr2=0x555b682119d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820df60 0x555b682119d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6d9b7fe700 1 -- 172.21.15.104:0/3907262206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820e880 msgr2=0x555b68211f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820e880 0x555b68211f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6d9b7fe700 1 -- 172.21.15.104:0/3907262206 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b6811f750 con 0x555b682087a0 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6d9bfff700 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820e880 0x555b68211f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b682087a0 0x555b682146d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c0059d0 tx=0x7f6d8c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c20fc9e82cf866a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6d7affd700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6d7affd700 1 -- 172.21.15.104:0/3907262206 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c024410 con 0x555b682087a0 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3907262206 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d90015070 con 0x555b682087a0 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.319+0000 7f6d7affd700 1 -- 172.21.15.104:0/3907262206 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6d8c0245b0 con 0x555b682087a0 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.320+0000 7f6d7affd700 1 -- 172.21.15.104:0/3907262206 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6d8c0247b0 con 0x555b682087a0 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.320+0000 7f6dac63cb80 10 client.25136 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.320+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3907262206 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555b6821fb20 con 0x555b682087a0 2024-06-09T22:21:51.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.320+0000 7f6d7affd700 1 -- 172.21.15.104:0/3907262206 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d90015070 con 0x555b682087a0 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.320+0000 7f6d7affd700 1 -- 172.21.15.104:0/3907262206 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6d8c02f9e0 con 0x555b682087a0 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.320+0000 7f6dac63cb80 10 client.25136 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.320+0000 7f6dac63cb80 10 client.25136 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.320+0000 7f6dac63cb80 20 client.25136 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.320+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555b68222150 0x555b68224560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.320+0000 7f6dac63cb80 4 client.25136 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.320+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3907262206 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b68166d50 con 0x555b68222150 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.320+0000 7f6d9affd700 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555b68222150 0x555b68224560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.321+0000 7f6d7affd700 10 client.25136 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.321+0000 7f6d9affd700 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555b68222150 0x555b68224560 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a84ac836ea174f4 server_cookie=c5323165afcc929a in_seq=0 out_seq=0 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6d7affd700 1 -- 172.21.15.104:0/3907262206 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x555b68166d50 con 0x555b68222150 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6d7affd700 10 client.25136 handle_command_reply: tid=0 2024-06-09T22:21:51.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6dac63cb80 1 client.25136 shutdown 2024-06-09T22:21:51.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3907262206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b682087a0 msgr2=0x555b682146d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:51.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b682087a0 0x555b682146d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c0059d0 tx=0x7f6d8c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3907262206 shutdown_connections 2024-06-09T22:21:51.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555b68222150 0x555b68224560 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820e880 0x555b68211f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6d7affd700 0 client.25136 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:51.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b682087a0 0x555b682146d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3907262206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820df60 0x555b682119d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3907262206 >> 172.21.15.104:0/3907262206 conn(0x555b68135080 msgr2=0x555b68202700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:51.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3907262206 shutdown_connections 2024-06-09T22:21:51.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3907262206 wait complete. 2024-06-09T22:21:51.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.322+0000 7f6dac63cb80 20 client.25136 trim_cache size 0 max 16384 2024-06-09T22:21:51.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.343+0000 7f6dac63cb80 1 Processor -- start 2024-06-09T22:21:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.343+0000 7f6dac63cb80 1 -- start start 2024-06-09T22:21:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.343+0000 7f6dac63cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b682087a0 0x555b6821e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.343+0000 7f6dac63cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b6820df60 0x555b68210ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.343+0000 7f6dac63cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b68214210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.343+0000 7f6dac63cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c1b05e0 con 0x555b682087a0 2024-06-09T22:21:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.343+0000 7f6dac63cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c1b2120 con 0x555b6820df60 2024-06-09T22:21:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.343+0000 7f6dac63cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c1b0cb0 con 0x555b6820e880 2024-06-09T22:21:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.343+0000 7f6d9b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b682087a0 0x555b6821e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d9b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b682087a0 0x555b6821e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44282/0 (socket says 172.21.15.104:44282) 2024-06-09T22:21:51.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d9b7fe700 1 -- 172.21.15.104:0/3813649721 learned_addr learned my addr 172.21.15.104:0/3813649721 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d9affd700 1 --2- 172.21.15.104:0/3813649721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b6820df60 0x555b68210ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d9b7fe700 1 -- 172.21.15.104:0/3813649721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 msgr2=0x555b68214210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/3813649721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b68214210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d9b7fe700 1 -- 172.21.15.104:0/3813649721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b6820df60 msgr2=0x555b68210ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/3813649721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b6820df60 0x555b68210ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d9b7fe700 1 -- 172.21.15.104:0/3813649721 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b682085a0 con 0x555b682087a0 2024-06-09T22:21:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d9affd700 1 --2- 172.21.15.104:0/3813649721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b6820df60 0x555b68210ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:21:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/3813649721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b682087a0 0x555b6821e3b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c0094e0 tx=0x7f6d8c0038c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824d367c2d35850b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d817fa700 1 -- 172.21.15.104:0/3813649721 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c0243e0 con 0x555b682087a0 2024-06-09T22:21:51.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d817fa700 1 -- 172.21.15.104:0/3813649721 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d8c024e30 con 0x555b682087a0 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.344+0000 7f6d817fa700 1 -- 172.21.15.104:0/3813649721 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c0176a0 con 0x555b682087a0 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.345+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3813649721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b682087a0 msgr2=0x555b6821e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.345+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3813649721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b682087a0 0x555b6821e3b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c0094e0 tx=0x7f6d8c0038c0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.345+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3813649721 shutdown_connections 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.345+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3813649721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b682087a0 0x555b6821e3b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.345+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3813649721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b68214210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.345+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/3813649721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b6820df60 0x555b68210ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.345+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3813649721 >> 172.21.15.104:0/3813649721 conn(0x555b68135080 msgr2=0x555b6820d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.345+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3813649721 shutdown_connections 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.345+0000 7f6dac63cb80 1 -- 172.21.15.104:0/3813649721 wait complete. 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.345+0000 7f6dac63cb80 1 -- start start 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6dac63cb80 1 Processor -- start 2024-06-09T22:21:51.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6dac63cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b682087a0 0x555b68229a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6dac63cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820df60 0x555b68229fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6dac63cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b6822a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6dac63cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c1b1a50 con 0x555b6820df60 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6dac63cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c1b1380 con 0x555b682087a0 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6dac63cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c1ae5b0 con 0x555b6820e880 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6dac63cb80 10 client.0 _ll_register_callbacks cb 0x555b680ef5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6d9affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820df60 0x555b68229fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6d9affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820df60 0x555b68229fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44288/0 (socket says 172.21.15.104:44288) 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6d9affd700 1 -- 172.21.15.104:0/1854448834 learned_addr learned my addr 172.21.15.104:0/1854448834 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b682087a0 0x555b68229a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6d9bfff700 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b6822a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6d9affd700 1 -- 172.21.15.104:0/1854448834 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 msgr2=0x555b6822a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6d9affd700 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b6822a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6d9affd700 1 -- 172.21.15.104:0/1854448834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b682087a0 msgr2=0x555b68229a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:51.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6d9affd700 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b682087a0 0x555b68229a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6d9affd700 1 -- 172.21.15.104:0/1854448834 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b6828be30 con 0x555b6820df60 2024-06-09T22:21:51.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.346+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b682087a0 0x555b68229a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:21:51.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.347+0000 7f6d9affd700 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820df60 0x555b68229fd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6d90007bb0 tx=0x7f6d90003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff4c50e1d97d489 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:51.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.347+0000 7f6d7affd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:51.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.347+0000 7f6d7affd700 1 -- 172.21.15.104:0/1854448834 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d90003ba0 con 0x555b6820df60 2024-06-09T22:21:51.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.347+0000 7f6d7affd700 1 -- 172.21.15.104:0/1854448834 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6d90003d40 con 0x555b6820df60 2024-06-09T22:21:51.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.347+0000 7f6dac63cb80 1 -- 172.21.15.104:0/1854448834 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d8c024e30 con 0x555b6820df60 2024-06-09T22:21:51.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.347+0000 7f6d7affd700 1 -- 172.21.15.104:0/1854448834 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6d90010e70 con 0x555b6820df60 2024-06-09T22:21:51.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.347+0000 7f6dac63cb80 10 client.15642 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:51.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.347+0000 7f6dac63cb80 1 -- 172.21.15.104:0/1854448834 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555b6828d100 con 0x555b6820df60 2024-06-09T22:21:51.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.347+0000 7f6d7affd700 1 -- 172.21.15.104:0/1854448834 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d8c024e30 con 0x555b6820df60 2024-06-09T22:21:51.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.347+0000 7f6d7affd700 1 -- 172.21.15.104:0/1854448834 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6d90026a50 con 0x555b6820df60 2024-06-09T22:21:51.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.348+0000 7f6dac63cb80 10 client.15642 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:51.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.348+0000 7f6dac63cb80 10 client.15642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:21:51.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.348+0000 7f6dac63cb80 20 client.15642 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:51.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.348+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555b6828d530 0x555b6828f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:51.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.348+0000 7f6dac63cb80 4 client.15642 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:21:51.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.348+0000 7f6dac63cb80 1 -- 172.21.15.104:0/1854448834 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555b681612d0 con 0x555b6828d530 2024-06-09T22:21:51.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.348+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555b6828d530 0x555b6828f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:51.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.348+0000 7f6d7affd700 10 client.15642 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:51.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.348+0000 7f6d9b7fe700 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555b6828d530 0x555b6828f920 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e24004d6a8fee3b7 server_cookie=78e18e605d104da2 in_seq=0 out_seq=0 2024-06-09T22:21:51.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.348+0000 7f6d7affd700 1 -- 172.21.15.104:0/1854448834 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555b681612d0 con 0x555b6828d530 2024-06-09T22:21:51.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.348+0000 7f6d7affd700 10 client.15642 handle_command_reply: tid=0 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.348+0000 7f6dac63cb80 1 client.15642 shutdown 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6dac63cb80 1 -- 172.21.15.104:0/1854448834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820df60 msgr2=0x555b68229fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820df60 0x555b68229fd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6d90007bb0 tx=0x7f6d90003760 comp rx=0 tx=0).stop 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6dac63cb80 1 -- 172.21.15.104:0/1854448834 shutdown_connections 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555b6828d530 0x555b6828f920 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555b6820e880 0x555b6822a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b682087a0 0x555b68229a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6d7affd700 0 client.15642 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6dac63cb80 1 --2- 172.21.15.104:0/1854448834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555b6820df60 0x555b68229fd0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6dac63cb80 1 -- 172.21.15.104:0/1854448834 >> 172.21.15.104:0/1854448834 conn(0x555b68135080 msgr2=0x555b68211250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6dac63cb80 1 -- 172.21.15.104:0/1854448834 shutdown_connections 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6dac63cb80 1 -- 172.21.15.104:0/1854448834 wait complete. 2024-06-09T22:21:51.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6dac63cb80 20 client.15642 trim_cache size 0 max 16384 2024-06-09T22:21:51.354 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:21:51.354 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "no active scrubs running", 2024-06-09T22:21:51.354 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": {} 2024-06-09T22:21:51.354 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:21:51.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6da1db9700 1 -- 172.21.15.104:0/564185446 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6d840a2eb0 msgr2=0x7f6d840a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:51.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6da1db9700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6d840a2eb0 0x7f6d840a5370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c00b5c0 tx=0x7f6d8c005960 comp rx=0 tx=0).stop 2024-06-09T22:21:51.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6da1db9700 1 -- 172.21.15.104:0/564185446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d9c108500 msgr2=0x7f6d9c101ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:51.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.349+0000 7f6da1db9700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d9c108500 0x7f6d9c101ee0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6d94007d70 tx=0x7f6d9400a340 comp rx=0 tx=0).stop 2024-06-09T22:21:51.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.350+0000 7f6da1db9700 1 -- 172.21.15.104:0/564185446 shutdown_connections 2024-06-09T22:21:51.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.350+0000 7f6da1db9700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6d840a2eb0 0x7f6d840a5370 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.350+0000 7f6da1db9700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d9c108500 0x7f6d9c101ee0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.350+0000 7f6da1db9700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c06a170 0x7f6d9c101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.350+0000 7f6da1db9700 1 --2- 172.21.15.104:0/564185446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d9c06ab30 0x7f6d9c1019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:51.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.350+0000 7f6da1db9700 1 -- 172.21.15.104:0/564185446 >> 172.21.15.104:0/564185446 conn(0x7f6d9c078340 msgr2=0x7f6d9c0ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:51.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.350+0000 7f6da1db9700 1 -- 172.21.15.104:0/564185446 shutdown_connections 2024-06-09T22:21:51.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:51.350+0000 7f6da1db9700 1 -- 172.21.15.104:0/564185446 wait complete. 2024-06-09T22:21:51.579 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:51 smithi104 ceph-mon[32389]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.1 KiB/s wr, 15 op/s 2024-06-09T22:21:51.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:51 smithi153 ceph-mon[36599]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.1 KiB/s wr, 15 op/s 2024-06-09T22:21:51.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:51 smithi194 ceph-mon[36856]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.1 KiB/s wr, 15 op/s 2024-06-09T22:21:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.658+0000 7fea0e89e640 1 -- 172.21.15.104:0/538035174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea081019d0 msgr2=0x7fea0810d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.658+0000 7fea0e89e640 1 --2- 172.21.15.104:0/538035174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea081019d0 0x7fea0810d700 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fea00009ca0 tx=0x7fea0002a8b0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.658+0000 7fea0e89e640 1 -- 172.21.15.104:0/538035174 shutdown_connections 2024-06-09T22:21:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.658+0000 7fea0e89e640 1 --2- 172.21.15.104:0/538035174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea081019d0 0x7fea0810d700 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.658+0000 7fea0e89e640 1 --2- 172.21.15.104:0/538035174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea08101030 0x7fea08101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.658+0000 7fea0e89e640 1 --2- 172.21.15.104:0/538035174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea08105890 0x7fea08105c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.658+0000 7fea0e89e640 1 -- 172.21.15.104:0/538035174 >> 172.21.15.104:0/538035174 conn(0x7fea08076dc0 msgr2=0x7fea080fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.658+0000 7fea0e89e640 1 -- 172.21.15.104:0/538035174 shutdown_connections 2024-06-09T22:21:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.659+0000 7fea0e89e640 1 -- 172.21.15.104:0/538035174 wait complete. 2024-06-09T22:21:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.659+0000 7fea0e89e640 1 Processor -- start 2024-06-09T22:21:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.659+0000 7fea0e89e640 1 -- start start 2024-06-09T22:21:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.659+0000 7fea0e89e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea08101030 0x7fea08198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.659+0000 7fea0e89e640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea081019d0 0x7fea08198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.659+0000 7fea0e89e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea08105890 0x7fea0819cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.659+0000 7fea0e89e640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fea08110d90 con 0x7fea08101030 2024-06-09T22:21:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.659+0000 7fea0e89e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea08110c10 con 0x7fea08105890 2024-06-09T22:21:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.659+0000 7fea0e89e640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fea08110f10 con 0x7fea081019d0 2024-06-09T22:21:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.659+0000 7fea07fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea08101030 0x7fea08198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.659+0000 7fea07fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea08101030 0x7fea08198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44304/0 (socket says 172.21.15.104:44304) 2024-06-09T22:21:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.659+0000 7fea07fff640 1 -- 172.21.15.104:0/910653884 learned_addr learned my addr 172.21.15.104:0/910653884 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fe9fffff640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea081019d0 0x7fea08198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fea0ce3b640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea08105890 0x7fea0819cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fea07fff640 1 -- 172.21.15.104:0/910653884 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea081019d0 msgr2=0x7fea08198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fea07fff640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea081019d0 0x7fea08198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fea07fff640 1 -- 172.21.15.104:0/910653884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea08105890 msgr2=0x7fea0819cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fea07fff640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea08105890 0x7fea0819cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fea07fff640 1 -- 172.21.15.104:0/910653884 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea0819d1f0 con 0x7fea08101030 2024-06-09T22:21:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fea07fff640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea08101030 0x7fea08198200 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe9f800d950 tx=0x7fe9f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:52.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fea05ffb640 1 -- 172.21.15.104:0/910653884 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8017070 con 0x7fea08101030 2024-06-09T22:21:52.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fea05ffb640 1 -- 172.21.15.104:0/910653884 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9f8009c30 con 0x7fea08101030 2024-06-09T22:21:52.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fea0e89e640 1 -- 172.21.15.104:0/910653884 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea0819d4e0 con 0x7fea08101030 2024-06-09T22:21:52.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fea05ffb640 1 -- 172.21.15.104:0/910653884 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f800f6b0 con 0x7fea08101030 2024-06-09T22:21:52.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.660+0000 7fea0e89e640 1 -- 172.21.15.104:0/910653884 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea081a4d70 con 0x7fea08101030 2024-06-09T22:21:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea05ffb640 1 -- 172.21.15.104:0/910653884 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7fe9f80109d0 con 0x7fea08101030 2024-06-09T22:21:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea115ae740 1 Processor -- start 2024-06-09T22:21:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea115ae740 1 -- start start 2024-06-09T22:21:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea115ae740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301b170 0x557a6301b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea115ae740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301ba90 0x557a63025e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea115ae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a63026350 0x557a63028740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea115ae740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557a62f7f4f0 con 0x557a6301ba90 2024-06-09T22:21:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea115ae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557a62f1d870 con 0x557a63026350 2024-06-09T22:21:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea115ae740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557a62f7b4b0 con 0x557a6301b170 2024-06-09T22:21:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea07fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301ba90 0x557a63025e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea05ffb640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe9f40a2ec0 0x7fe9f40a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea07fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301ba90 0x557a63025e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44310/0 (socket says 172.21.15.104:44310) 2024-06-09T22:21:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea07fff640 1 -- 172.21.15.104:0/2742250110 learned_addr learned my addr 172.21.15.104:0/2742250110 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.663+0000 7fea0ce3b640 1 --2- 172.21.15.104:0/2742250110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301b170 0x557a6301b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fe9fffff640 1 --2- 172.21.15.104:0/2742250110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a63026350 0x557a63028740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fea0ce3b640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe9f40a2ec0 0x7fe9f40a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fea05ffb640 1 -- 172.21.15.104:0/910653884 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fe9f80cf6c0 con 0x7fea08101030 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fea07fff640 1 -- 172.21.15.104:0/2742250110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301b170 msgr2=0x557a6301b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fea07fff640 1 --2- 172.21.15.104:0/2742250110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301b170 0x557a6301b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fea07fff640 1 -- 172.21.15.104:0/2742250110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a63026350 msgr2=0x557a63028740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fea07fff640 1 --2- 172.21.15.104:0/2742250110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a63026350 0x557a63028740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fea07fff640 1 -- 172.21.15.104:0/2742250110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a62f399b0 con 0x557a6301ba90 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fea07fff640 1 --2- 172.21.15.104:0/2742250110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301ba90 0x557a63025e10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe9f809f510 tx=0x7fe9f800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aad25bc905d1d79 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fea0ce3b640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe9f40a2ec0 0x7fe9f40a5380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fea00009430 tx=0x7fea000095c0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fe9fe7fc640 1 -- 172.21.15.104:0/2742250110 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8020070 con 0x557a6301ba90 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fe9fe7fc640 1 -- 172.21.15.104:0/2742250110 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9f809f9d0 con 0x557a6301ba90 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.664+0000 7fe9fe7fc640 1 -- 172.21.15.104:0/2742250110 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f809f540 con 0x557a6301ba90 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.665+0000 7fea115ae740 1 -- 172.21.15.104:0/2742250110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301ba90 msgr2=0x557a63025e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.665+0000 7fea115ae740 1 --2- 172.21.15.104:0/2742250110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301ba90 0x557a63025e10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe9f809f510 tx=0x7fe9f800bf60 comp rx=0 tx=0).stop 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.665+0000 7fea115ae740 1 -- 172.21.15.104:0/2742250110 shutdown_connections 2024-06-09T22:21:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.665+0000 7fea115ae740 1 --2- 172.21.15.104:0/2742250110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a63026350 0x557a63028740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.665+0000 7fea115ae740 1 --2- 172.21.15.104:0/2742250110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301ba90 0x557a63025e10 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.665+0000 7fea115ae740 1 --2- 172.21.15.104:0/2742250110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301b170 0x557a6301b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.665+0000 7fea115ae740 1 -- 172.21.15.104:0/2742250110 >> 172.21.15.104:0/2742250110 conn(0x557a63017680 msgr2=0x557a63017a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.665+0000 7fea115ae740 1 -- 172.21.15.104:0/2742250110 shutdown_connections 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.665+0000 7fea115ae740 1 -- 172.21.15.104:0/2742250110 wait complete. 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.665+0000 7fea115ae740 1 -- start start 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea115ae740 1 Processor -- start 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea115ae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a6301b170 0x557a63023360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea115ae740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a630238a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea115ae740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 0x557a63023de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea115ae740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557a6302d2d0 con 0x557a63026350 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea115ae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557a62f08ae0 con 0x557a6301b170 2024-06-09T22:21:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea115ae740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557a6302d450 con 0x557a6301ba90 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea115ae740 10 client.0 _ll_register_callbacks cb 0x557a62f419b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea0ce3b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 0x557a63023de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea0ce3b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 0x557a63023de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44318/0 (socket says 172.21.15.104:44318) 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea0ce3b640 1 -- 172.21.15.104:0/847621825 learned_addr learned my addr 172.21.15.104:0/847621825 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fe9fffff640 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a630238a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea07fff640 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a6301b170 0x557a63023360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea0ce3b640 1 -- 172.21.15.104:0/847621825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 msgr2=0x557a630238a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea0ce3b640 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a630238a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea0ce3b640 1 -- 172.21.15.104:0/847621825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a6301b170 msgr2=0x557a63023360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea0ce3b640 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a6301b170 0x557a63023360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea0ce3b640 1 -- 172.21.15.104:0/847621825 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a62c93210 con 0x557a63026350 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.666+0000 7fea07fff640 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a6301b170 0x557a63023360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:21:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fea0ce3b640 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 0x557a63023de0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fea00030a70 tx=0x7fea0002a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ab4df64a6d1b7e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fe9e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fe9e77fe640 1 -- 172.21.15.104:0/847621825 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00051070 con 0x557a63026350 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fea115ae740 1 -- 172.21.15.104:0/847621825 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557a62f92f10 con 0x557a63026350 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fe9e77fe640 1 -- 172.21.15.104:0/847621825 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fea00004250 con 0x557a63026350 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fe9e77fe640 1 -- 172.21.15.104:0/847621825 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fea00004450 con 0x557a63026350 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fea115ae740 10 client.15654 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fea115ae740 1 -- 172.21.15.104:0/847621825 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557a62d5f620 con 0x557a63026350 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fe9e77fe640 1 -- 172.21.15.104:0/847621825 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557a62f92f10 con 0x557a63026350 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fe9e77fe640 1 -- 172.21.15.104:0/847621825 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fea00031440 con 0x557a63026350 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fea115ae740 10 client.15654 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fea115ae740 10 client.15654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fea115ae740 20 client.15654 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fea115ae740 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557a6303a900 0x557a6303cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fea115ae740 4 client.15654 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.667+0000 7fea115ae740 1 -- 172.21.15.104:0/847621825 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a62f8cc20 con 0x557a6303a900 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.668+0000 7fea07fff640 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557a6303a900 0x557a6303cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.668+0000 7fe9e77fe640 10 client.15654 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.668+0000 7fea07fff640 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557a6303a900 0x557a6303cd10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88ecdeb3ca709261 server_cookie=c6f7d665565d04c6 in_seq=0 out_seq=0 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fe9e77fe640 1 -- 172.21.15.104:0/847621825 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x557a62f8cc20 con 0x557a6303a900 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fe9e77fe640 10 client.15654 handle_command_reply: tid=0 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fea115ae740 1 client.15654 shutdown 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fea115ae740 1 -- 172.21.15.104:0/847621825 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 msgr2=0x557a63023de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fea115ae740 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 0x557a63023de0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fea00030a70 tx=0x7fea0002a8b0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fea115ae740 1 -- 172.21.15.104:0/847621825 shutdown_connections 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fea115ae740 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557a6303a900 0x557a6303cd10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fea115ae740 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 0x557a63023de0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fea115ae740 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a630238a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fe9e77fe640 0 client.15654 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fea115ae740 1 --2- 172.21.15.104:0/847621825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a6301b170 0x557a63023360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fea115ae740 1 -- 172.21.15.104:0/847621825 >> 172.21.15.104:0/847621825 conn(0x557a63017680 msgr2=0x557a6301d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fea115ae740 1 -- 172.21.15.104:0/847621825 shutdown_connections 2024-06-09T22:21:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fea115ae740 1 -- 172.21.15.104:0/847621825 wait complete. 2024-06-09T22:21:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.669+0000 7fea115ae740 20 client.15654 trim_cache size 0 max 16384 2024-06-09T22:21:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.684+0000 7fea115ae740 1 Processor -- start 2024-06-09T22:21:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.684+0000 7fea115ae740 1 -- start start 2024-06-09T22:21:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea115ae740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301b170 0x557a63016f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea115ae740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a6301db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea115ae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a63026350 0x557a63042620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea115ae740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fea081acb80 con 0x557a6301b170 2024-06-09T22:21:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea115ae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea081ae6c0 con 0x557a63026350 2024-06-09T22:21:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea115ae740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fea081ad250 con 0x557a6301ba90 2024-06-09T22:21:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea07fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301b170 0x557a63016f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea07fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301b170 0x557a63016f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44330/0 (socket says 172.21.15.104:44330) 2024-06-09T22:21:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea07fff640 1 -- 172.21.15.104:0/764070134 learned_addr learned my addr 172.21.15.104:0/764070134 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fe9fffff640 1 --2- 172.21.15.104:0/764070134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a6301db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea07fff640 1 -- 172.21.15.104:0/764070134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 msgr2=0x557a6301db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea07fff640 1 --2- 172.21.15.104:0/764070134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a6301db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea07fff640 1 -- 172.21.15.104:0/764070134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a63026350 msgr2=0x557a63042620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea0ce3b640 1 --2- 172.21.15.104:0/764070134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a63026350 0x557a63042620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea07fff640 1 --2- 172.21.15.104:0/764070134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a63026350 0x557a63042620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea07fff640 1 -- 172.21.15.104:0/764070134 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a63011b70 con 0x557a6301b170 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fe9fffff640 1 --2- 172.21.15.104:0/764070134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a6301db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fea07fff640 1 --2- 172.21.15.104:0/764070134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301b170 0x557a63016f60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe9f80058c0 tx=0x7fe9f8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88fafacf86e70688 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.685+0000 7fe9fdffb640 1 -- 172.21.15.104:0/764070134 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8020040 con 0x557a6301b170 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.686+0000 7fe9fdffb640 1 -- 172.21.15.104:0/764070134 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9f80cc860 con 0x557a6301b170 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.686+0000 7fe9fdffb640 1 -- 172.21.15.104:0/764070134 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f80cce60 con 0x557a6301b170 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.686+0000 7fea115ae740 1 -- 172.21.15.104:0/764070134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301b170 msgr2=0x557a63016f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.686+0000 7fea115ae740 1 --2- 172.21.15.104:0/764070134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301b170 0x557a63016f60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe9f80058c0 tx=0x7fe9f8000bc0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.686+0000 7fea115ae740 1 -- 172.21.15.104:0/764070134 shutdown_connections 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.686+0000 7fea115ae740 1 --2- 172.21.15.104:0/764070134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a63026350 0x557a63042620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.686+0000 7fea115ae740 1 --2- 172.21.15.104:0/764070134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a6301db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.686+0000 7fea115ae740 1 --2- 172.21.15.104:0/764070134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a6301b170 0x557a63016f60 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.686+0000 7fea115ae740 1 -- 172.21.15.104:0/764070134 >> 172.21.15.104:0/764070134 conn(0x557a63017680 msgr2=0x557a63025420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.686+0000 7fea115ae740 1 -- 172.21.15.104:0/764070134 shutdown_connections 2024-06-09T22:21:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.686+0000 7fea115ae740 1 -- 172.21.15.104:0/764070134 wait complete. 2024-06-09T22:21:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea115ae740 1 -- start start 2024-06-09T22:21:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea115ae740 1 Processor -- start 2024-06-09T22:21:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea115ae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a6301b170 0x557a63032f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea115ae740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a63020240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea115ae740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 0x557a63020960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea115ae740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fea081adff0 con 0x557a63026350 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea115ae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea081ad920 con 0x557a6301b170 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea115ae740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fea081aab40 con 0x557a6301ba90 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea115ae740 10 client.0 _ll_register_callbacks cb 0x7fea000098f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea0ce3b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 0x557a63020960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea0ce3b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 0x557a63020960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44344/0 (socket says 172.21.15.104:44344) 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea0ce3b640 1 -- 172.21.15.104:0/578664311 learned_addr learned my addr 172.21.15.104:0/578664311 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fea07fff640 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a6301b170 0x557a63032f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.687+0000 7fe9fffff640 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a63020240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea0ce3b640 1 -- 172.21.15.104:0/578664311 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 msgr2=0x557a63020240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea0ce3b640 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a63020240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea0ce3b640 1 -- 172.21.15.104:0/578664311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a6301b170 msgr2=0x557a63032f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea0ce3b640 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a6301b170 0x557a63032f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea0ce3b640 1 -- 172.21.15.104:0/578664311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a630a45f0 con 0x557a63026350 2024-06-09T22:21:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea0ce3b640 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 0x557a63020960 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fea00003e00 tx=0x7fea00005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb932ae1e77fcf1b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fe9e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fe9e77fe640 1 -- 172.21.15.104:0/578664311 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00066060 con 0x557a63026350 2024-06-09T22:21:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fe9e77fe640 1 -- 172.21.15.104:0/578664311 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fea00004250 con 0x557a63026350 2024-06-09T22:21:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea115ae740 1 -- 172.21.15.104:0/578664311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9f80cc860 con 0x557a63026350 2024-06-09T22:21:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fe9e77fe640 1 -- 172.21.15.104:0/578664311 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fea0006d050 con 0x557a63026350 2024-06-09T22:21:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea115ae740 10 client.15660 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea115ae740 1 -- 172.21.15.104:0/578664311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557a630a56f0 con 0x557a63026350 2024-06-09T22:21:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fe9e77fe640 1 -- 172.21.15.104:0/578664311 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9f80cc860 con 0x557a63026350 2024-06-09T22:21:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fe9e77fe640 1 -- 172.21.15.104:0/578664311 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fea000569b0 con 0x557a63026350 2024-06-09T22:21:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea115ae740 10 client.15660 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea115ae740 10 client.15660 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:21:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea115ae740 20 client.15660 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea115ae740 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557a630a5e10 0x557a630a8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea115ae740 4 client.15660 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:21:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.688+0000 7fea115ae740 1 -- 172.21.15.104:0/578664311 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a62f86f70 con 0x557a630a5e10 2024-06-09T22:21:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.689+0000 7fea07fff640 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557a630a5e10 0x557a630a8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.689+0000 7fe9e77fe640 10 client.15660 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.689+0000 7fea07fff640 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557a630a5e10 0x557a630a8230 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d993048f9fd8714b server_cookie=c8415ba754ce512c in_seq=0 out_seq=0 2024-06-09T22:21:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.693+0000 7fe9e77fe640 1 -- 172.21.15.104:0/578664311 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x557a62f86f70 con 0x557a630a5e10 2024-06-09T22:21:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.693+0000 7fe9e77fe640 10 client.15660 handle_command_reply: tid=0 2024-06-09T22:21:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.693+0000 7fea115ae740 1 client.15660 shutdown 2024-06-09T22:21:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.693+0000 7fea115ae740 1 -- 172.21.15.104:0/578664311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 msgr2=0x557a63020960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.693+0000 7fea115ae740 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 0x557a63020960 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fea00003e00 tx=0x7fea00005df0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.693+0000 7fea115ae740 1 -- 172.21.15.104:0/578664311 shutdown_connections 2024-06-09T22:21:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.693+0000 7fea115ae740 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557a630a5e10 0x557a630a8230 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.693+0000 7fea115ae740 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557a63026350 0x557a63020960 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.693+0000 7fea115ae740 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a6301ba90 0x557a63020240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.693+0000 7fea115ae740 1 --2- 172.21.15.104:0/578664311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a6301b170 0x557a63032f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.693+0000 7fe9e77fe640 0 client.15660 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:21:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.693+0000 7fea115ae740 1 -- 172.21.15.104:0/578664311 >> 172.21.15.104:0/578664311 conn(0x557a63017680 msgr2=0x557a63024b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea115ae740 1 -- 172.21.15.104:0/578664311 shutdown_connections 2024-06-09T22:21:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea115ae740 1 -- 172.21.15.104:0/578664311 wait complete. 2024-06-09T22:21:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea115ae740 20 client.15660 trim_cache size 0 max 16384 2024-06-09T22:21:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:21:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:21:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:21:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea0e89e640 1 -- 172.21.15.104:0/910653884 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe9f40a2ec0 msgr2=0x7fe9f40a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea0e89e640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe9f40a2ec0 0x7fe9f40a5380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fea00009430 tx=0x7fea000095c0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea0e89e640 1 -- 172.21.15.104:0/910653884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea08101030 msgr2=0x7fea08198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea0e89e640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea08101030 0x7fea08198200 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe9f800d950 tx=0x7fe9f800de10 comp rx=0 tx=0).stop 2024-06-09T22:21:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea0e89e640 1 -- 172.21.15.104:0/910653884 shutdown_connections 2024-06-09T22:21:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea0e89e640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe9f40a2ec0 0x7fe9f40a5380 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea0e89e640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea08105890 0x7fea0819cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea0e89e640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea081019d0 0x7fea08198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea0e89e640 1 --2- 172.21.15.104:0/910653884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea08101030 0x7fea08198200 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea0e89e640 1 -- 172.21.15.104:0/910653884 >> 172.21.15.104:0/910653884 conn(0x7fea08076dc0 msgr2=0x7fea080fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea0e89e640 1 -- 172.21.15.104:0/910653884 shutdown_connections 2024-06-09T22:21:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:21:52.694+0000 7fea0e89e640 1 -- 172.21.15.104:0/910653884 wait complete. 2024-06-09T22:21:53.379 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-09T22:21:53.379 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:21:53.654 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:53 smithi104 ceph-mon[32389]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-06-09T22:21:53.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:53 smithi153 ceph-mon[36599]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-06-09T22:21:53.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:53 smithi194 ceph-mon[36856]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-06-09T22:21:53.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.920+0000 7faa9e49d700 1 -- 172.21.15.104:0/1270681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa98102ea0 msgr2=0x7faa9810f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:53.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.920+0000 7faa9e49d700 1 --2- 172.21.15.104:0/1270681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa98102ea0 0x7faa9810f380 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7faa9000b0a0 tx=0x7faa9000b3b0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.920+0000 7faa9e49d700 1 -- 172.21.15.104:0/1270681 shutdown_connections 2024-06-09T22:21:53.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.920+0000 7faa9e49d700 1 --2- 172.21.15.104:0/1270681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa98102ea0 0x7faa9810f380 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.920+0000 7faa9e49d700 1 --2- 172.21.15.104:0/1270681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa981085f0 0x7faa981089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.920+0000 7faa9e49d700 1 --2- 172.21.15.104:0/1270681 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa981024e0 0x7faa98102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.920+0000 7faa9e49d700 1 -- 172.21.15.104:0/1270681 >> 172.21.15.104:0/1270681 conn(0x7faa98079640 msgr2=0x7faa98079a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:53.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.920+0000 7faa9e49d700 1 -- 172.21.15.104:0/1270681 shutdown_connections 2024-06-09T22:21:53.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.921+0000 7faa9e49d700 1 -- 172.21.15.104:0/1270681 wait complete. 2024-06-09T22:21:53.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.922+0000 7faa9e49d700 1 Processor -- start 2024-06-09T22:21:53.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.922+0000 7faa9e49d700 1 -- start start 2024-06-09T22:21:53.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.922+0000 7faa9e49d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa981024e0 0x7faa98074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.922+0000 7faa9e49d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa98102ea0 0x7faa98074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.922+0000 7faa9e49d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa981085f0 0x7faa98075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.922+0000 7faa9e49d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7faa98112790 con 0x7faa981085f0 2024-06-09T22:21:53.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.922+0000 7faa9e49d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faa98112610 con 0x7faa98102ea0 2024-06-09T22:21:53.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.922+0000 7faa9e49d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faa98112910 con 0x7faa981024e0 2024-06-09T22:21:53.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.922+0000 7faa9ca3a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa981085f0 0x7faa98075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.922+0000 7faa9ca3a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa981085f0 0x7faa98075210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44356/0 (socket says 172.21.15.104:44356) 2024-06-09T22:21:53.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.922+0000 7faa9ca3a700 1 -- 172.21.15.104:0/1373923207 learned_addr learned my addr 172.21.15.104:0/1373923207 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:53.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.923+0000 7faa97fff700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa981024e0 0x7faa98074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.923+0000 7faa977fe700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa98102ea0 0x7faa98074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.923+0000 7faa9ca3a700 1 -- 172.21.15.104:0/1373923207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa981024e0 msgr2=0x7faa98074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:53.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.923+0000 7faa9ca3a700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa981024e0 0x7faa98074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.923+0000 7faa9ca3a700 1 -- 172.21.15.104:0/1373923207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa98102ea0 msgr2=0x7faa98074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:53.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.923+0000 7faa9ca3a700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa98102ea0 0x7faa98074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.923+0000 7faa9ca3a700 1 -- 172.21.15.104:0/1373923207 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa981a7020 con 0x7faa981085f0 2024-06-09T22:21:53.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.923+0000 7faa97fff700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa981024e0 0x7faa98074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:21:53.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.923+0000 7faa9ca3a700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa981085f0 0x7faa98075210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7faa900062a0 tx=0x7faa90004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:53.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.923+0000 7faa957fa700 1 -- 172.21.15.104:0/1373923207 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa90021070 con 0x7faa981085f0 2024-06-09T22:21:53.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.924+0000 7faa957fa700 1 -- 172.21.15.104:0/1373923207 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa90003c00 con 0x7faa981085f0 2024-06-09T22:21:53.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.924+0000 7faa957fa700 1 -- 172.21.15.104:0/1373923207 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9001d450 con 0x7faa981085f0 2024-06-09T22:21:53.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.924+0000 7faa9e49d700 1 -- 172.21.15.104:0/1373923207 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa981a72b0 con 0x7faa981085f0 2024-06-09T22:21:53.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.924+0000 7faa9e49d700 1 -- 172.21.15.104:0/1373923207 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa981a7650 con 0x7faa981085f0 2024-06-09T22:21:53.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.926+0000 7faa957fa700 1 -- 172.21.15.104:0/1373923207 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7faa90004020 con 0x7faa981085f0 2024-06-09T22:21:53.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.927+0000 7faa957fa700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faa800a2f30 0x7faa800a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.927+0000 7faa957fa700 1 -- 172.21.15.104:0/1373923207 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7faa900d0d50 con 0x7faa981085f0 2024-06-09T22:21:53.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.927+0000 7faa97fff700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faa800a2f30 0x7faa800a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.928+0000 7faa97fff700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faa800a2f30 0x7faa800a53f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faa88005fd0 tx=0x7faa8800a5e0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:53.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.928+0000 7faaa8d20b80 1 Processor -- start 2024-06-09T22:21:53.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.928+0000 7faaa8d20b80 1 -- start start 2024-06-09T22:21:53.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.928+0000 7faaa8d20b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df5772f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.928+0000 7faaa8d20b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 0x55a8df571210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.928+0000 7faaa8d20b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df571750 0x55a8df571b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faaa8d20b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a8df463f10 con 0x55a8df576f10 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faaa8d20b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a8df4c2800 con 0x55a8df577830 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faaa8d20b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a8df42a410 con 0x55a8df571750 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faa97fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df5772f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faa97fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df5772f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44358/0 (socket says 172.21.15.104:44358) 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faa97fff700 1 -- 172.21.15.104:0/4011409121 learned_addr learned my addr 172.21.15.104:0/4011409121 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faa9ca3a700 1 --2- 172.21.15.104:0/4011409121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df571750 0x55a8df571b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faa9ca3a700 1 -- 172.21.15.104:0/4011409121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 msgr2=0x55a8df571210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faa9ca3a700 1 --2- 172.21.15.104:0/4011409121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 0x55a8df571210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faa9ca3a700 1 -- 172.21.15.104:0/4011409121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 msgr2=0x55a8df5772f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faa9ca3a700 1 --2- 172.21.15.104:0/4011409121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df5772f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faa9ca3a700 1 -- 172.21.15.104:0/4011409121 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8df411880 con 0x55a8df571750 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faa97fff700 1 --2- 172.21.15.104:0/4011409121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df5772f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:21:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.929+0000 7faa9ca3a700 1 --2- 172.21.15.104:0/4011409121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df571750 0x55a8df571b70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faa900cdfb0 tx=0x7faa9009c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=465dd74c5c4d6ebe server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.930+0000 7faa7e7fc700 1 -- 172.21.15.104:0/4011409121 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9002a070 con 0x55a8df571750 2024-06-09T22:21:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.930+0000 7faa7e7fc700 1 -- 172.21.15.104:0/4011409121 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa9009cd40 con 0x55a8df571750 2024-06-09T22:21:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.930+0000 7faa7e7fc700 1 -- 172.21.15.104:0/4011409121 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9009cfa0 con 0x55a8df571750 2024-06-09T22:21:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.930+0000 7faaa8d20b80 1 -- 172.21.15.104:0/4011409121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df571750 msgr2=0x55a8df571b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.930+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/4011409121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df571750 0x55a8df571b70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faa900cdfb0 tx=0x7faa9009c970 comp rx=0 tx=0).stop 2024-06-09T22:21:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.931+0000 7faaa8d20b80 1 -- 172.21.15.104:0/4011409121 shutdown_connections 2024-06-09T22:21:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.931+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/4011409121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df5772f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.931+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/4011409121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df571750 0x55a8df571b70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.931+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/4011409121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 0x55a8df571210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.931+0000 7faaa8d20b80 1 -- 172.21.15.104:0/4011409121 >> 172.21.15.104:0/4011409121 conn(0x55a8df49e030 msgr2=0x55a8df56c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.931+0000 7faaa8d20b80 1 -- 172.21.15.104:0/4011409121 shutdown_connections 2024-06-09T22:21:53.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.931+0000 7faaa8d20b80 1 -- 172.21.15.104:0/4011409121 wait complete. 2024-06-09T22:21:53.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.931+0000 7faaa8d20b80 1 -- start start 2024-06-09T22:21:53.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.931+0000 7faaa8d20b80 1 Processor -- start 2024-06-09T22:21:53.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faaa8d20b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df571750 0x55a8df57d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faaa8d20b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df57a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faaa8d20b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df577830 0x55a8df57aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faaa8d20b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a8df462f70 con 0x55a8df576f10 2024-06-09T22:21:53.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faaa8d20b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a8df4be7c0 con 0x55a8df571750 2024-06-09T22:21:53.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faaa8d20b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a8df57dfc0 con 0x55a8df577830 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faaa8d20b80 10 client.0 _ll_register_callbacks cb 0x55a8df458560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa977fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df57a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa977fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df57a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44366/0 (socket says 172.21.15.104:44366) 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa977fe700 1 -- 172.21.15.104:0/86034156 learned_addr learned my addr 172.21.15.104:0/86034156 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa9ca3a700 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df577830 0x55a8df57aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa97fff700 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df571750 0x55a8df57d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa977fe700 1 -- 172.21.15.104:0/86034156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df577830 msgr2=0x55a8df57aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa977fe700 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df577830 0x55a8df57aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa977fe700 1 -- 172.21.15.104:0/86034156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df571750 msgr2=0x55a8df57d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa977fe700 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df571750 0x55a8df57d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa977fe700 1 -- 172.21.15.104:0/86034156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8df488700 con 0x55a8df576f10 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa9ca3a700 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df577830 0x55a8df57aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa977fe700 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df57a9b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7faa8c007d50 tx=0x7faa8c0106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da0c755367c7a63 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.932+0000 7faa777fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faa777fe700 1 -- 172.21.15.104:0/86034156 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c010e40 con 0x55a8df576f10 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faaa8d20b80 1 -- 172.21.15.104:0/86034156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa9009cd40 con 0x55a8df576f10 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faa777fe700 1 -- 172.21.15.104:0/86034156 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7faa8c004020 con 0x55a8df576f10 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faa777fe700 1 -- 172.21.15.104:0/86034156 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faa8c023050 con 0x55a8df576f10 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faaa8d20b80 10 client.15672 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faaa8d20b80 1 -- 172.21.15.104:0/86034156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8df588820 con 0x55a8df576f10 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faa777fe700 1 -- 172.21.15.104:0/86034156 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa9009cd40 con 0x55a8df576f10 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faa777fe700 1 -- 172.21.15.104:0/86034156 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7faa8c01a480 con 0x55a8df576f10 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faaa8d20b80 10 client.15672 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faaa8d20b80 10 client.15672 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faaa8d20b80 20 client.15672 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8df58b2c0 0x55a8df58c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faaa8d20b80 4 client.15672 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:21:53.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.933+0000 7faaa8d20b80 1 -- 172.21.15.104:0/86034156 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8df4cfd00 con 0x55a8df58b2c0 2024-06-09T22:21:53.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.934+0000 7faa97fff700 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8df58b2c0 0x55a8df58c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.934+0000 7faa97fff700 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8df58b2c0 0x55a8df58c730 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=42b445bd07cd2aa7 server_cookie=d5162763ee92f9f8 in_seq=0 out_seq=0 2024-06-09T22:21:53.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.934+0000 7faa777fe700 10 client.15672 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:21:53.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faa777fe700 1 -- 172.21.15.104:0/86034156 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a8df4cfd00 con 0x55a8df58b2c0 2024-06-09T22:21:53.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faa777fe700 10 client.15672 handle_command_reply: tid=0 2024-06-09T22:21:53.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faaa8d20b80 1 client.15672 shutdown 2024-06-09T22:21:53.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faaa8d20b80 1 -- 172.21.15.104:0/86034156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 msgr2=0x55a8df57a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:53.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df57a9b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7faa8c007d50 tx=0x7faa8c0106b0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faaa8d20b80 1 -- 172.21.15.104:0/86034156 shutdown_connections 2024-06-09T22:21:53.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8df58b2c0 0x55a8df58c730 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df577830 0x55a8df57aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df571750 0x55a8df57d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/86034156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df57a9b0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faaa8d20b80 1 -- 172.21.15.104:0/86034156 >> 172.21.15.104:0/86034156 conn(0x55a8df49e030 msgr2=0x55a8df56b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:53.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faa777fe700 0 client.15672 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:21:53.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faaa8d20b80 1 -- 172.21.15.104:0/86034156 shutdown_connections 2024-06-09T22:21:53.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.935+0000 7faaa8d20b80 1 -- 172.21.15.104:0/86034156 wait complete. 2024-06-09T22:21:53.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.936+0000 7faaa8d20b80 20 client.15672 trim_cache size 0 max 16384 2024-06-09T22:21:53.955 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:53.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.959+0000 7faaa8d20b80 1 Processor -- start 2024-06-09T22:21:53.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.959+0000 7faaa8d20b80 1 -- start start 2024-06-09T22:21:53.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.959+0000 7faaa8d20b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df571750 0x55a8df571210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.959+0000 7faaa8d20b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df5784a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.959+0000 7faaa8d20b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 0x55a8df57d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.959+0000 7faaa8d20b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a8df58aa70 con 0x55a8df576f10 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.959+0000 7faaa8d20b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a8df58b140 con 0x55a8df577830 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.959+0000 7faaa8d20b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a8df58b810 con 0x55a8df571750 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.959+0000 7faa977fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df5784a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.959+0000 7faa977fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df5784a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44378/0 (socket says 172.21.15.104:44378) 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.959+0000 7faa977fe700 1 -- 172.21.15.104:0/1354043896 learned_addr learned my addr 172.21.15.104:0/1354043896 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.960+0000 7faa97fff700 1 --2- 172.21.15.104:0/1354043896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df571750 0x55a8df571210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.960+0000 7faa9ca3a700 1 --2- 172.21.15.104:0/1354043896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 0x55a8df57d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.960+0000 7faa977fe700 1 -- 172.21.15.104:0/1354043896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df571750 msgr2=0x55a8df571210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.960+0000 7faa977fe700 1 --2- 172.21.15.104:0/1354043896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df571750 0x55a8df571210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.960+0000 7faa977fe700 1 -- 172.21.15.104:0/1354043896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 msgr2=0x55a8df57d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.960+0000 7faa977fe700 1 --2- 172.21.15.104:0/1354043896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 0x55a8df57d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.960+0000 7faa977fe700 1 -- 172.21.15.104:0/1354043896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8df577540 con 0x55a8df576f10 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.960+0000 7faa977fe700 1 --2- 172.21.15.104:0/1354043896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df5784a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7faa8c01aee0 tx=0x7faa8c010e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4872db59b7738548 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:53.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.960+0000 7faa7dffb700 1 -- 172.21.15.104:0/1354043896 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c020040 con 0x55a8df576f10 2024-06-09T22:21:53.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.960+0000 7faa7dffb700 1 -- 172.21.15.104:0/1354043896 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa8c01acd0 con 0x55a8df576f10 2024-06-09T22:21:53.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.960+0000 7faa7dffb700 1 -- 172.21.15.104:0/1354043896 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c003a50 con 0x55a8df576f10 2024-06-09T22:21:53.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.961+0000 7faaa8d20b80 1 -- 172.21.15.104:0/1354043896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 msgr2=0x55a8df5784a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:53.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.961+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/1354043896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df5784a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7faa8c01aee0 tx=0x7faa8c010e50 comp rx=0 tx=0).stop 2024-06-09T22:21:53.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.961+0000 7faaa8d20b80 1 -- 172.21.15.104:0/1354043896 shutdown_connections 2024-06-09T22:21:53.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.961+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/1354043896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 0x55a8df57d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.961+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/1354043896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df571750 0x55a8df571210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.961+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/1354043896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df576f10 0x55a8df5784a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.961+0000 7faaa8d20b80 1 -- 172.21.15.104:0/1354043896 >> 172.21.15.104:0/1354043896 conn(0x55a8df49e030 msgr2=0x55a8df574df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:53.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.961+0000 7faaa8d20b80 1 -- 172.21.15.104:0/1354043896 shutdown_connections 2024-06-09T22:21:53.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.961+0000 7faaa8d20b80 1 -- 172.21.15.104:0/1354043896 wait complete. 2024-06-09T22:21:53.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.962+0000 7faaa8d20b80 1 -- start start 2024-06-09T22:21:53.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.962+0000 7faaa8d20b80 1 Processor -- start 2024-06-09T22:21:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.962+0000 7faaa8d20b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df571750 0x55a8df57c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.962+0000 7faaa8d20b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df576f10 0x55a8df57c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.962+0000 7faaa8d20b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 0x55a8df57cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.962+0000 7faaa8d20b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a8df58c5b0 con 0x55a8df571750 2024-06-09T22:21:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.962+0000 7faaa8d20b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a8df58bee0 con 0x55a8df577830 2024-06-09T22:21:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.962+0000 7faaa8d20b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a8df58cc80 con 0x55a8df576f10 2024-06-09T22:21:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.962+0000 7faaa8d20b80 10 client.0 _ll_register_callbacks cb 0x55a8df458560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:21:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.962+0000 7faa97fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df571750 0x55a8df57c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa97fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df571750 0x55a8df57c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:44384/0 (socket says 172.21.15.104:44384) 2024-06-09T22:21:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa97fff700 1 -- 172.21.15.104:0/3179196802 learned_addr learned my addr 172.21.15.104:0/3179196802 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa977fe700 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df576f10 0x55a8df57c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa9ca3a700 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 0x55a8df57cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa97fff700 1 -- 172.21.15.104:0/3179196802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df576f10 msgr2=0x55a8df57c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa97fff700 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df576f10 0x55a8df57c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa97fff700 1 -- 172.21.15.104:0/3179196802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 msgr2=0x55a8df57cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:21:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa97fff700 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 0x55a8df57cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa97fff700 1 -- 172.21.15.104:0/3179196802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8df594e50 con 0x55a8df571750 2024-06-09T22:21:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa97fff700 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df571750 0x55a8df57c190 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7faa88013040 tx=0x7faa880047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b719a99f0f4d280 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa777fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:21:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa777fe700 1 -- 172.21.15.104:0/3179196802 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa880106f0 con 0x55a8df571750 2024-06-09T22:21:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faa777fe700 1 -- 172.21.15.104:0/3179196802 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7faa88010890 con 0x55a8df571750 2024-06-09T22:21:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.963+0000 7faaa8d20b80 1 -- 172.21.15.104:0/3179196802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa8c01acd0 con 0x55a8df571750 2024-06-09T22:21:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.964+0000 7faa777fe700 1 -- 172.21.15.104:0/3179196802 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faa88010a90 con 0x55a8df571750 2024-06-09T22:21:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.964+0000 7faaa8d20b80 10 client.15678 fetch_fsmap learned FSMap version 29 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.964+0000 7faaa8d20b80 1 -- 172.21.15.104:0/3179196802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8df596150 con 0x55a8df571750 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.964+0000 7faa777fe700 1 -- 172.21.15.104:0/3179196802 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa88010c30 con 0x55a8df571750 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.964+0000 7faa777fe700 1 -- 172.21.15.104:0/3179196802 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7faa88027360 con 0x55a8df571750 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.964+0000 7faaa8d20b80 10 client.15678 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.964+0000 7faaa8d20b80 10 client.15678 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.964+0000 7faaa8d20b80 20 client.15678 populate_metadata read hostname 'smithi104' 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.964+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8df5fae70 0x55a8df5f9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.964+0000 7faaa8d20b80 4 client.15678 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.964+0000 7faaa8d20b80 1 -- 172.21.15.104:0/3179196802 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "status"}) -- 0x7faa8c01acd0 con 0x55a8df5fae70 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.965+0000 7faa977fe700 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8df5fae70 0x55a8df5f9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.965+0000 7faa977fe700 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8df5fae70 0x55a8df5f9b80 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9e4051a4a8882f8 server_cookie=2c0b9b3aabca7ee3 in_seq=0 out_seq=0 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.965+0000 7faa777fe700 10 client.15678 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.965+0000 7faa777fe700 1 -- 172.21.15.104:0/3179196802 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x7faa8c01acd0 con 0x55a8df5fae70 2024-06-09T22:21:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.965+0000 7faa777fe700 10 client.15678 handle_command_reply: tid=0 2024-06-09T22:21:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.965+0000 7faaa8d20b80 1 client.15678 shutdown 2024-06-09T22:21:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.966+0000 7faaa8d20b80 1 -- 172.21.15.104:0/3179196802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df571750 msgr2=0x55a8df57c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.966+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df571750 0x55a8df57c190 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7faa88013040 tx=0x7faa880047c0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.966+0000 7faaa8d20b80 1 -- 172.21.15.104:0/3179196802 shutdown_connections 2024-06-09T22:21:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.966+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8df5fae70 0x55a8df5f9b80 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.966+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8df571750 0x55a8df57c190 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.966+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8df577830 0x55a8df57cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.966+0000 7faaa8d20b80 1 --2- 172.21.15.104:0/3179196802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df576f10 0x55a8df57c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.966+0000 7faa777fe700 0 client.15678 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:21:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.966+0000 7faaa8d20b80 1 -- 172.21.15.104:0/3179196802 >> 172.21.15.104:0/3179196802 conn(0x55a8df49e030 msgr2=0x55a8df5784a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:53.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.966+0000 7faaa8d20b80 1 -- 172.21.15.104:0/3179196802 shutdown_connections 2024-06-09T22:21:53.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.966+0000 7faaa8d20b80 1 -- 172.21.15.104:0/3179196802 wait complete. 2024-06-09T22:21:53.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.966+0000 7faaa8d20b80 20 client.15678 trim_cache size 0 max 16384 2024-06-09T22:21:53.970 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:21:53.970 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:21:53.970 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 1, 2024-06-09T22:21:53.970 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24448, 2024-06-09T22:21:53.971 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:21:53.971 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:21:53.971 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:21:53.971 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 246.57507384499999, 2024-06-09T22:21:53.971 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 29, 2024-06-09T22:21:53.971 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 100, 2024-06-09T22:21:53.971 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 99, 2024-06-09T22:21:53.971 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 340.04994455899998 2024-06-09T22:21:53.971 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:21:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.967+0000 7faa9e49d700 1 -- 172.21.15.104:0/1373923207 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faa800a2f30 msgr2=0x7faa800a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.967+0000 7faa9e49d700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faa800a2f30 0x7faa800a53f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faa88005fd0 tx=0x7faa8800a5e0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.967+0000 7faa9e49d700 1 -- 172.21.15.104:0/1373923207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa981085f0 msgr2=0x7faa98075210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.967+0000 7faa9e49d700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa981085f0 0x7faa98075210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7faa900062a0 tx=0x7faa90004ce0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.967+0000 7faa9e49d700 1 -- 172.21.15.104:0/1373923207 shutdown_connections 2024-06-09T22:21:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.967+0000 7faa9e49d700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faa800a2f30 0x7faa800a53f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.967+0000 7faa9e49d700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa981085f0 0x7faa98075210 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.967+0000 7faa9e49d700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa981024e0 0x7faa98074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.967+0000 7faa9e49d700 1 --2- 172.21.15.104:0/1373923207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa98102ea0 0x7faa98074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.967+0000 7faa9e49d700 1 -- 172.21.15.104:0/1373923207 >> 172.21.15.104:0/1373923207 conn(0x7faa98079640 msgr2=0x7faa98100e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.967+0000 7faa9e49d700 1 -- 172.21.15.104:0/1373923207 shutdown_connections 2024-06-09T22:21:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:53.967+0000 7faa9e49d700 1 -- 172.21.15.104:0/1373923207 wait complete. 2024-06-09T22:21:55.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:55 smithi153 ceph-mon[36599]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-06-09T22:21:55.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:55 smithi104 ceph-mon[32389]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-06-09T22:21:55.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:55 smithi194 ceph-mon[36856]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-06-09T22:21:56.006 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-06-09T22:21:56.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.531+0000 7f2fb66fd700 1 -- 172.21.15.104:0/2536421212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb01066f0 msgr2=0x7f2fb010e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:56.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.531+0000 7f2fb66fd700 1 --2- 172.21.15.104:0/2536421212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb01066f0 0x7f2fb010e2c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800b0d0 tx=0x7f2fa800b3e0 comp rx=0 tx=0).stop 2024-06-09T22:21:56.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.531+0000 7f2fb66fd700 1 -- 172.21.15.104:0/2536421212 shutdown_connections 2024-06-09T22:21:56.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.531+0000 7f2fb66fd700 1 --2- 172.21.15.104:0/2536421212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb01066f0 0x7f2fb010e2c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:56.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.531+0000 7f2fb66fd700 1 --2- 172.21.15.104:0/2536421212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb0101610 0x7f2fb01019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:56.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.531+0000 7f2fb66fd700 1 --2- 172.21.15.104:0/2536421212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fb0101fc0 0x7f2fb0106010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:56.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.531+0000 7f2fb66fd700 1 -- 172.21.15.104:0/2536421212 >> 172.21.15.104:0/2536421212 conn(0x7f2fb0078430 msgr2=0x7f2fb0078840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:56.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.531+0000 7f2fb66fd700 1 -- 172.21.15.104:0/2536421212 shutdown_connections 2024-06-09T22:21:56.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.531+0000 7f2fb66fd700 1 -- 172.21.15.104:0/2536421212 wait complete. 2024-06-09T22:21:56.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.532+0000 7f2fb66fd700 1 Processor -- start 2024-06-09T22:21:56.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.532+0000 7f2fb66fd700 1 -- start start 2024-06-09T22:21:56.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2fb66fd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb0101610 0x7f2fb0199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:56.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2fb66fd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0101fc0 0x7f2fb0199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:56.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2fb66fd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fb01066f0 0x7f2fb01a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:56.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2fb66fd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb0111750 con 0x7f2fb01066f0 2024-06-09T22:21:56.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2fb66fd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01115d0 con 0x7f2fb0101610 2024-06-09T22:21:56.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2fb66fd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb01118d0 con 0x7f2fb0101fc0 2024-06-09T22:21:56.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2faf7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0101fc0 0x7f2fb0199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:56.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2faf7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0101fc0 0x7f2fb0199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:52628/0 (socket says 172.21.15.104:52628) 2024-06-09T22:21:56.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2faf7fe700 1 -- 172.21.15.104:0/3922034060 learned_addr learned my addr 172.21.15.104:0/3922034060 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:21:56.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2faffff700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb0101610 0x7f2fb0199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:56.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2faf7fe700 1 -- 172.21.15.104:0/3922034060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb0101610 msgr2=0x7f2fb0199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:56.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2faf7fe700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb0101610 0x7f2fb0199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:56.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2faf7fe700 1 -- 172.21.15.104:0/3922034060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fb01066f0 msgr2=0x7f2fb01a4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:56.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2fb4c9a700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fb01066f0 0x7f2fb01a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:56.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.533+0000 7f2faf7fe700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fb01066f0 0x7f2fb01a4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:56.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.534+0000 7f2faf7fe700 1 -- 172.21.15.104:0/3922034060 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb0197840 con 0x7f2fb0101fc0 2024-06-09T22:21:56.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.534+0000 7f2fb4c9a700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fb01066f0 0x7f2fb01a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:21:56.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.534+0000 7f2faf7fe700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0101fc0 0x7f2fb0199f90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400ea30 tx=0x7f2fa400ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:56.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.534+0000 7f2fad7fa700 1 -- 172.21.15.104:0/3922034060 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400cbd0 con 0x7f2fb0101fc0 2024-06-09T22:21:56.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.534+0000 7f2fad7fa700 1 -- 172.21.15.104:0/3922034060 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fa4004500 con 0x7f2fb0101fc0 2024-06-09T22:21:56.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.534+0000 7f2fb66fd700 1 -- 172.21.15.104:0/3922034060 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb0197b30 con 0x7f2fb0101fc0 2024-06-09T22:21:56.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.534+0000 7f2fad7fa700 1 -- 172.21.15.104:0/3922034060 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4010430 con 0x7f2fb0101fc0 2024-06-09T22:21:56.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.534+0000 7f2fb66fd700 1 -- 172.21.15.104:0/3922034060 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb0198060 con 0x7f2fb0101fc0 2024-06-09T22:21:56.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.535+0000 7f2fb66fd700 1 -- 172.21.15.104:0/3922034060 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb0069180 con 0x7f2fb0101fc0 2024-06-09T22:21:56.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.536+0000 7f2fad7fa700 1 -- 172.21.15.104:0/3922034060 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f2fa4004020 con 0x7f2fb0101fc0 2024-06-09T22:21:56.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.537+0000 7f2fad7fa700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2f980a2f30 0x7f2f980a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:21:56.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.537+0000 7f2fad7fa700 1 -- 172.21.15.104:0/3922034060 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f2fa40cfbc0 con 0x7f2fb0101fc0 2024-06-09T22:21:56.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.537+0000 7f2faffff700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2f980a2f30 0x7f2f980a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:21:56.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.538+0000 7f2faffff700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2f980a2f30 0x7f2f980a53f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0009c60 tx=0x7f2fa0009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:21:56.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.542+0000 7f2fad7fa700 1 -- 172.21.15.104:0/3922034060 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2fa409c220 con 0x7f2fb0101fc0 2024-06-09T22:21:56.565 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:56.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.805+0000 7f2fb66fd700 1 -- 172.21.15.104:0/3922034060 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fb006b210 con 0x7f2fb0101fc0 2024-06-09T22:21:56.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.806+0000 7f2fad7fa700 1 -- 172.21.15.104:0/3922034060 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11953 (secure 0 0 0) 0x7f2fa409bb20 con 0x7f2fb0101fc0 2024-06-09T22:21:56.806 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:21:56.806 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:20:55.680750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24460,24448,24442,24403,24419]},"id":1}]} 2024-06-09T22:21:56.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.809+0000 7f2fb66fd700 1 -- 172.21.15.104:0/3922034060 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2f980a2f30 msgr2=0x7f2f980a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:56.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.809+0000 7f2fb66fd700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2f980a2f30 0x7f2f980a53f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0009c60 tx=0x7f2fa0009260 comp rx=0 tx=0).stop 2024-06-09T22:21:56.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.809+0000 7f2fb66fd700 1 -- 172.21.15.104:0/3922034060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0101fc0 msgr2=0x7f2fb0199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:21:56.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.809+0000 7f2fb66fd700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0101fc0 0x7f2fb0199f90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400ea30 tx=0x7f2fa400ed40 comp rx=0 tx=0).stop 2024-06-09T22:21:56.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.809+0000 7f2fb66fd700 1 -- 172.21.15.104:0/3922034060 shutdown_connections 2024-06-09T22:21:56.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.809+0000 7f2fb66fd700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2f980a2f30 0x7f2f980a53f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:56.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.809+0000 7f2fb66fd700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fb01066f0 0x7f2fb01a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:56.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.809+0000 7f2fb66fd700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb0101610 0x7f2fb0199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:56.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.809+0000 7f2fb66fd700 1 --2- 172.21.15.104:0/3922034060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0101fc0 0x7f2fb0199f90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:21:56.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.809+0000 7f2fb66fd700 1 -- 172.21.15.104:0/3922034060 >> 172.21.15.104:0/3922034060 conn(0x7f2fb0078430 msgr2=0x7f2fb00ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:21:56.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.809+0000 7f2fb66fd700 1 -- 172.21.15.104:0/3922034060 shutdown_connections 2024-06-09T22:21:56.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:21:56.809+0000 7f2fb66fd700 1 -- 172.21.15.104:0/3922034060 wait complete. 2024-06-09T22:21:56.810 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 29 2024-06-09T22:21:57.554 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:57 smithi104 ceph-mon[32389]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 767 B/s wr, 11 op/s 2024-06-09T22:21:57.555 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:57 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3922034060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:57.635 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-09T22:21:57.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:57 smithi153 ceph-mon[36599]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 767 B/s wr, 11 op/s 2024-06-09T22:21:57.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:57 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3922034060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:57.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:57 smithi194 ceph-mon[36856]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 767 B/s wr, 11 op/s 2024-06-09T22:21:57.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:57 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3922034060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:21:58.857 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:21:59.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:21:59 smithi153 ceph-mon[36599]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-09T22:21:59.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:21:59 smithi104 ceph-mon[32389]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-09T22:21:59.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:21:59 smithi194 ceph-mon[36856]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-09T22:22:00.615 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:22:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:22:00.468+0000 7f2d27d6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:22:00.615 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:22:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:22:00.468+0000 7f2d27d6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:22:00.615 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:22:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:22:00.468+0000 7f2d27d6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:22:00.615 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:22:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:22:00.468+0000 7f2d27d6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:22:00.615 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:22:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:22:00.468+0000 7f2d27d6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-09T22:22:00.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.624+0000 7fc620f42700 1 -- 172.21.15.104:0/721828247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc61c101d50 msgr2=0x7fc61c10e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:00.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.624+0000 7fc620f42700 1 --2- 172.21.15.104:0/721828247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc61c101d50 0x7fc61c10e240 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc61400b0a0 tx=0x7fc61400b3b0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.624+0000 7fc620f42700 1 -- 172.21.15.104:0/721828247 shutdown_connections 2024-06-09T22:22:00.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.624+0000 7fc620f42700 1 --2- 172.21.15.104:0/721828247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc61c101d50 0x7fc61c10e240 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.624+0000 7fc620f42700 1 --2- 172.21.15.104:0/721828247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc61c107400 0x7fc61c1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.624+0000 7fc620f42700 1 --2- 172.21.15.104:0/721828247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc61c101390 0x7fc61c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.624+0000 7fc620f42700 1 -- 172.21.15.104:0/721828247 >> 172.21.15.104:0/721828247 conn(0x7fc61c078340 msgr2=0x7fc61c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:00.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.624+0000 7fc620f42700 1 -- 172.21.15.104:0/721828247 shutdown_connections 2024-06-09T22:22:00.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.624+0000 7fc620f42700 1 -- 172.21.15.104:0/721828247 wait complete. 2024-06-09T22:22:00.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.625+0000 7fc620f42700 1 Processor -- start 2024-06-09T22:22:00.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.625+0000 7fc620f42700 1 -- start start 2024-06-09T22:22:00.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc620f42700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc61c101390 0x7fc61c1978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc620f42700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc61c101d50 0x7fc61c197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc620f42700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc61c107400 0x7fc61c19c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc620f42700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1116a0 con 0x7fc61c107400 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc620f42700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c111520 con 0x7fc61c101d50 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc620f42700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c111820 con 0x7fc61c101390 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc61a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc61c101390 0x7fc61c1978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc61ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc61c107400 0x7fc61c19c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc61a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc61c101390 0x7fc61c1978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:52656/0 (socket says 172.21.15.104:52656) 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc61a59c700 1 -- 172.21.15.104:0/2815111114 learned_addr learned my addr 172.21.15.104:0/2815111114 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc619d9b700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc61c101d50 0x7fc61c197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc61a59c700 1 -- 172.21.15.104:0/2815111114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc61c101d50 msgr2=0x7fc61c197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc61a59c700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc61c101d50 0x7fc61c197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc61a59c700 1 -- 172.21.15.104:0/2815111114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc61c107400 msgr2=0x7fc61c19c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc61a59c700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc61c107400 0x7fc61c19c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc61a59c700 1 -- 172.21.15.104:0/2815111114 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c19ce40 con 0x7fc61c101390 2024-06-09T22:22:00.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.626+0000 7fc61ad9d700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc61c107400 0x7fc61c19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:00.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.627+0000 7fc61a59c700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc61c101390 0x7fc61c1978b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00d950 tx=0x7fc60c00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:00.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.627+0000 7fc60b7fe700 1 -- 172.21.15.104:0/2815111114 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c016070 con 0x7fc61c101390 2024-06-09T22:22:00.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.627+0000 7fc60b7fe700 1 -- 172.21.15.104:0/2815111114 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc60c00fd70 con 0x7fc61c101390 2024-06-09T22:22:00.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.627+0000 7fc620f42700 1 -- 172.21.15.104:0/2815111114 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c19d130 con 0x7fc61c101390 2024-06-09T22:22:00.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.627+0000 7fc60b7fe700 1 -- 172.21.15.104:0/2815111114 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c00e610 con 0x7fc61c101390 2024-06-09T22:22:00.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.627+0000 7fc620f42700 1 -- 172.21.15.104:0/2815111114 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61c19d670 con 0x7fc61c101390 2024-06-09T22:22:00.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.629+0000 7fc60b7fe700 1 -- 172.21.15.104:0/2815111114 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7fc60c019070 con 0x7fc61c101390 2024-06-09T22:22:00.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.630+0000 7fc60b7fe700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc6040a2eb0 0x7fc6040a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.631+0000 7fc619d9b700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc6040a2eb0 0x7fc6040a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.631+0000 7fc60b7fe700 1 -- 172.21.15.104:0/2815111114 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fc60c0cf480 con 0x7fc61c101390 2024-06-09T22:22:00.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.631+0000 7fc62b7c5b80 1 Processor -- start 2024-06-09T22:22:00.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.631+0000 7fc619d9b700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc6040a2eb0 0x7fc6040a5370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0787d0 tx=0x7fc61000b3c0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:00.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.631+0000 7fc62b7c5b80 1 -- start start 2024-06-09T22:22:00.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.631+0000 7fc62b7c5b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb0205f0 0x55dfcb0209d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.631+0000 7fc62b7c5b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb020f10 0x55dfcb01a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.631+0000 7fc62b7c5b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb01aed0 0x55dfcb01b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.631+0000 7fc62b7c5b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55dfcaf6c7c0 con 0x55dfcb01aed0 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.631+0000 7fc62b7c5b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dfcaf146d0 con 0x55dfcb0205f0 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.631+0000 7fc62b7c5b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dfcaf68780 con 0x55dfcb020f10 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.632+0000 7fc61ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb01aed0 0x55dfcb01b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.632+0000 7fc61ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb01aed0 0x55dfcb01b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45622/0 (socket says 172.21.15.104:45622) 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.632+0000 7fc61ad9d700 1 -- 172.21.15.104:0/3893195393 learned_addr learned my addr 172.21.15.104:0/3893195393 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.632+0000 7fc61a59c700 1 --2- 172.21.15.104:0/3893195393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb0205f0 0x55dfcb0209d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.632+0000 7fc619d9b700 1 --2- 172.21.15.104:0/3893195393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb020f10 0x55dfcb01a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.632+0000 7fc61ad9d700 1 -- 172.21.15.104:0/3893195393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb020f10 msgr2=0x55dfcb01a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.632+0000 7fc61ad9d700 1 --2- 172.21.15.104:0/3893195393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb020f10 0x55dfcb01a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.632+0000 7fc61ad9d700 1 -- 172.21.15.104:0/3893195393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb0205f0 msgr2=0x55dfcb0209d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:00.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.632+0000 7fc61ad9d700 1 --2- 172.21.15.104:0/3893195393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb0205f0 0x55dfcb0209d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.632+0000 7fc61ad9d700 1 -- 172.21.15.104:0/3893195393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfcaf0af00 con 0x55dfcb01aed0 2024-06-09T22:22:00.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.632+0000 7fc61ad9d700 1 --2- 172.21.15.104:0/3893195393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb01aed0 0x55dfcb01b2f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc614009e40 tx=0x7fc614009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71a360117398c6b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:00.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.633+0000 7fc608ff9700 1 -- 172.21.15.104:0/3893195393 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614004d30 con 0x55dfcb01aed0 2024-06-09T22:22:00.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.633+0000 7fc608ff9700 1 -- 172.21.15.104:0/3893195393 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6140079c0 con 0x55dfcb01aed0 2024-06-09T22:22:00.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.633+0000 7fc608ff9700 1 -- 172.21.15.104:0/3893195393 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614005620 con 0x55dfcb01aed0 2024-06-09T22:22:00.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.634+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3893195393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb01aed0 msgr2=0x55dfcb01b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:00.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.634+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/3893195393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb01aed0 0x55dfcb01b2f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc614009e40 tx=0x7fc614009500 comp rx=0 tx=0).stop 2024-06-09T22:22:00.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.634+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3893195393 shutdown_connections 2024-06-09T22:22:00.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.634+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/3893195393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb01aed0 0x55dfcb01b2f0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.634+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/3893195393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb0205f0 0x55dfcb0209d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.634+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/3893195393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb020f10 0x55dfcb01a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.634+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3893195393 >> 172.21.15.104:0/3893195393 conn(0x55dfcaf45c70 msgr2=0x55dfcb015800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:00.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.634+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3893195393 shutdown_connections 2024-06-09T22:22:00.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.634+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3893195393 wait complete. 2024-06-09T22:22:00.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc62b7c5b80 1 -- start start 2024-06-09T22:22:00.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc62b7c5b80 1 Processor -- start 2024-06-09T22:22:00.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc62b7c5b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 0x55dfcb026c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc62b7c5b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb0205f0 0x55dfcb023f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc62b7c5b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb020f10 0x55dfcb0244c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc62b7c5b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55dfcb027920 con 0x55dfcb0205f0 2024-06-09T22:22:00.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc62b7c5b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dfcaf0cba0 con 0x55dfcb01aed0 2024-06-09T22:22:00.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc62b7c5b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dfcb027aa0 con 0x55dfcb020f10 2024-06-09T22:22:00.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc62b7c5b80 10 client.0 _ll_register_callbacks cb 0x55dfcaf068d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:00.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc619d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb0205f0 0x55dfcb023f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc619d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb0205f0 0x55dfcb023f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45638/0 (socket says 172.21.15.104:45638) 2024-06-09T22:22:00.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc619d9b700 1 -- 172.21.15.104:0/2065673499 learned_addr learned my addr 172.21.15.104:0/2065673499 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:00.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc61a59c700 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 0x55dfcb026c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.635+0000 7fc61ad9d700 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb020f10 0x55dfcb0244c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc619d9b700 1 -- 172.21.15.104:0/2065673499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb020f10 msgr2=0x55dfcb0244c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:00.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc619d9b700 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb020f10 0x55dfcb0244c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc619d9b700 1 -- 172.21.15.104:0/2065673499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 msgr2=0x55dfcb026c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:00.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc619d9b700 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 0x55dfcb026c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc619d9b700 1 -- 172.21.15.104:0/2065673499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfcaebb880 con 0x55dfcb0205f0 2024-06-09T22:22:00.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc61a59c700 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 0x55dfcb026c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc619d9b700 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb0205f0 0x55dfcb023f80 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc61000b690 tx=0x7fc6100047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc19d57817d4ebb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc601ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc601ffb700 1 -- 172.21.15.104:0/2065673499 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610011d90 con 0x55dfcb0205f0 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc601ffb700 1 -- 172.21.15.104:0/2065673499 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fc610028470 con 0x55dfcb0205f0 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/2065673499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6140079c0 con 0x55dfcb0205f0 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc601ffb700 1 -- 172.21.15.104:0/2065673499 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc610031050 con 0x55dfcb0205f0 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc62b7c5b80 10 client.15699 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.636+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/2065673499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dfcb032300 con 0x55dfcb0205f0 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.637+0000 7fc601ffb700 1 -- 172.21.15.104:0/2065673499 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6140079c0 con 0x55dfcb0205f0 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.637+0000 7fc601ffb700 1 -- 172.21.15.104:0/2065673499 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc61001e690 con 0x55dfcb0205f0 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.637+0000 7fc62b7c5b80 10 client.15699 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.637+0000 7fc62b7c5b80 10 client.15699 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.637+0000 7fc62b7c5b80 20 client.15699 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:00.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.637+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dfcb034970 0x55dfcb035de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.637+0000 7fc62b7c5b80 4 client.15699 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:00.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.637+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/2065673499 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfcaf79cc0 con 0x55dfcb034970 2024-06-09T22:22:00.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.637+0000 7fc61a59c700 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dfcb034970 0x55dfcb035de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.638+0000 7fc601ffb700 10 client.15699 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:00.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.638+0000 7fc61a59c700 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dfcb034970 0x55dfcb035de0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34be78ea36efacb0 server_cookie=600135f35ef69bf6 in_seq=0 out_seq=0 2024-06-09T22:22:00.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.638+0000 7fc601ffb700 1 -- 172.21.15.104:0/2065673499 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55dfcaf79cc0 con 0x55dfcb034970 2024-06-09T22:22:00.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.638+0000 7fc601ffb700 10 client.15699 handle_command_reply: tid=0 2024-06-09T22:22:00.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.638+0000 7fc62b7c5b80 1 client.15699 shutdown 2024-06-09T22:22:00.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.638+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/2065673499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb0205f0 msgr2=0x55dfcb023f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:00.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.638+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb0205f0 0x55dfcb023f80 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc61000b690 tx=0x7fc6100047c0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.639+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/2065673499 shutdown_connections 2024-06-09T22:22:00.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.639+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dfcb034970 0x55dfcb035de0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.639+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb020f10 0x55dfcb0244c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.639+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 0x55dfcb026c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.639+0000 7fc601ffb700 0 client.15699 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:00.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.639+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/2065673499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb0205f0 0x55dfcb023f80 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.639+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/2065673499 >> 172.21.15.104:0/2065673499 conn(0x55dfcaf45c70 msgr2=0x55dfcb019980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:00.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.639+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/2065673499 shutdown_connections 2024-06-09T22:22:00.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.639+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/2065673499 wait complete. 2024-06-09T22:22:00.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.639+0000 7fc62b7c5b80 20 client.15699 trim_cache size 0 max 16384 2024-06-09T22:22:00.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.663+0000 7fc62b7c5b80 1 Processor -- start 2024-06-09T22:22:00.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.663+0000 7fc62b7c5b80 1 -- start start 2024-06-09T22:22:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.663+0000 7fc62b7c5b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb01aed0 0x55dfcb01a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.663+0000 7fc62b7c5b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb0205f0 0x55dfcb0212f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.663+0000 7fc62b7c5b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb020f10 0x55dfcb021830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.663+0000 7fc62b7c5b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55dfcb034120 con 0x55dfcb020f10 2024-06-09T22:22:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.663+0000 7fc62b7c5b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dfcb0347f0 con 0x55dfcb0205f0 2024-06-09T22:22:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.663+0000 7fc62b7c5b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dfcb034ec0 con 0x55dfcb01aed0 2024-06-09T22:22:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.663+0000 7fc61ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb020f10 0x55dfcb021830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc61ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb020f10 0x55dfcb021830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45652/0 (socket says 172.21.15.104:45652) 2024-06-09T22:22:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc61a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb01aed0 0x55dfcb01a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc61ad9d700 1 -- 172.21.15.104:0/393337694 learned_addr learned my addr 172.21.15.104:0/393337694 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc619d9b700 1 --2- 172.21.15.104:0/393337694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb0205f0 0x55dfcb0212f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc61ad9d700 1 -- 172.21.15.104:0/393337694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb01aed0 msgr2=0x55dfcb01a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc61ad9d700 1 --2- 172.21.15.104:0/393337694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb01aed0 0x55dfcb01a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc61ad9d700 1 -- 172.21.15.104:0/393337694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb0205f0 msgr2=0x55dfcb0212f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc61ad9d700 1 --2- 172.21.15.104:0/393337694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb0205f0 0x55dfcb0212f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc61ad9d700 1 -- 172.21.15.104:0/393337694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfcb0224f0 con 0x55dfcb020f10 2024-06-09T22:22:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc61ad9d700 1 --2- 172.21.15.104:0/393337694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb020f10 0x55dfcb021830 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc614004cc0 tx=0x7fc6140099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b8915f21c0afcd server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc603fff700 1 -- 172.21.15.104:0/393337694 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6140076a0 con 0x55dfcb020f10 2024-06-09T22:22:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc603fff700 1 -- 172.21.15.104:0/393337694 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc614007d20 con 0x55dfcb020f10 2024-06-09T22:22:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.664+0000 7fc603fff700 1 -- 172.21.15.104:0/393337694 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61401c410 con 0x55dfcb020f10 2024-06-09T22:22:00.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.665+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/393337694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb020f10 msgr2=0x55dfcb021830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:00.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.665+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/393337694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb020f10 0x55dfcb021830 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc614004cc0 tx=0x7fc6140099e0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.665+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/393337694 shutdown_connections 2024-06-09T22:22:00.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.665+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/393337694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb020f10 0x55dfcb021830 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.665+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/393337694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb01aed0 0x55dfcb01a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.665+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/393337694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb0205f0 0x55dfcb0212f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.665+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/393337694 >> 172.21.15.104:0/393337694 conn(0x55dfcaf45c70 msgr2=0x55dfcb022870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:00.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.665+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/393337694 shutdown_connections 2024-06-09T22:22:00.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.665+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/393337694 wait complete. 2024-06-09T22:22:00.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.665+0000 7fc62b7c5b80 1 -- start start 2024-06-09T22:22:00.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc62b7c5b80 1 Processor -- start 2024-06-09T22:22:00.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc62b7c5b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 0x55dfcb03f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc62b7c5b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb0205f0 0x55dfcb03f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc62b7c5b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb020f10 0x55dfcb03a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc62b7c5b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55dfcb035c60 con 0x55dfcb020f10 2024-06-09T22:22:00.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc62b7c5b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dfcb035590 con 0x55dfcb01aed0 2024-06-09T22:22:00.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc62b7c5b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dfcb036330 con 0x55dfcb0205f0 2024-06-09T22:22:00.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc62b7c5b80 10 client.0 _ll_register_callbacks cb 0x55dfcaf068d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:00.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc61a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 0x55dfcb03f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc61a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 0x55dfcb03f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:53072/0 (socket says 172.21.15.104:53072) 2024-06-09T22:22:00.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc61a59c700 1 -- 172.21.15.104:0/3210725874 learned_addr learned my addr 172.21.15.104:0/3210725874 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:00.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc61ad9d700 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb020f10 0x55dfcb03a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.666+0000 7fc619d9b700 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb0205f0 0x55dfcb03f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc61a59c700 1 -- 172.21.15.104:0/3210725874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb0205f0 msgr2=0x55dfcb03f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc61a59c700 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb0205f0 0x55dfcb03f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc61a59c700 1 -- 172.21.15.104:0/3210725874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb020f10 msgr2=0x55dfcb03a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc61a59c700 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb020f10 0x55dfcb03a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc61a59c700 1 -- 172.21.15.104:0/3210725874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfcb03fe20 con 0x55dfcb01aed0 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc61a59c700 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 0x55dfcb03f260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc60c09b370 tx=0x7fc60c097040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce6b906db6032c60 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc601ffb700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc601ffb700 1 -- 172.21.15.104:0/3210725874 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0b4060 con 0x55dfcb01aed0 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3210725874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc614007d20 con 0x55dfcb01aed0 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc601ffb700 1 -- 172.21.15.104:0/3210725874 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fc60c010710 con 0x55dfcb01aed0 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc601ffb700 1 -- 172.21.15.104:0/3210725874 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc60c014030 con 0x55dfcb01aed0 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc62b7c5b80 10 client.25193 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3210725874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dfcb0a0820 con 0x55dfcb01aed0 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.667+0000 7fc601ffb700 1 -- 172.21.15.104:0/3210725874 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc614007d20 con 0x55dfcb01aed0 2024-06-09T22:22:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.668+0000 7fc601ffb700 1 -- 172.21.15.104:0/3210725874 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc60c066260 con 0x55dfcb01aed0 2024-06-09T22:22:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.668+0000 7fc62b7c5b80 10 client.25193 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.668+0000 7fc62b7c5b80 10 client.25193 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-09T22:22:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.668+0000 7fc62b7c5b80 20 client.25193 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.668+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dfcb0a53b0 0x55dfcb0a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.668+0000 7fc62b7c5b80 4 client.25193 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-09T22:22:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.668+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3210725874 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "status"}) -- 0x55dfcaf74240 con 0x55dfcb0a53b0 2024-06-09T22:22:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.668+0000 7fc619d9b700 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dfcb0a53b0 0x55dfcb0a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc601ffb700 10 client.25193 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc619d9b700 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dfcb0a53b0 0x55dfcb0a40c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19a8d58cd0153dbb server_cookie=dc50daea8f68fb69 in_seq=0 out_seq=0 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc601ffb700 1 -- 172.21.15.104:0/3210725874 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x55dfcaf74240 con 0x55dfcb0a53b0 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc601ffb700 10 client.25193 handle_command_reply: tid=0 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc62b7c5b80 1 client.25193 shutdown 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3210725874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 msgr2=0x55dfcb03f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 0x55dfcb03f260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc60c09b370 tx=0x7fc60c097040 comp rx=0 tx=0).stop 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3210725874 shutdown_connections 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55dfcb0a53b0 0x55dfcb0a40c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dfcb020f10 0x55dfcb03a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dfcb01aed0 0x55dfcb03f260 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc62b7c5b80 1 --2- 172.21.15.104:0/3210725874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfcb0205f0 0x55dfcb03f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3210725874 >> 172.21.15.104:0/3210725874 conn(0x55dfcaf45c70 msgr2=0x55dfcb00edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc601ffb700 0 client.25193 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:00.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3210725874 shutdown_connections 2024-06-09T22:22:00.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc62b7c5b80 1 -- 172.21.15.104:0/3210725874 wait complete. 2024-06-09T22:22:00.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.669+0000 7fc62b7c5b80 20 client.25193 trim_cache size 0 max 16384 2024-06-09T22:22:00.674 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:22:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "a948c0ca-26ac-11ef-bc9b-c7b262605968", 2024-06-09T22:22:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2024-06-09T22:22:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 24460, 2024-06-09T22:22:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-06-09T22:22:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-06-09T22:22:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-06-09T22:22:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 315.46819217900003, 2024-06-09T22:22:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 29, 2024-06-09T22:22:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 100, 2024-06-09T22:22:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 99, 2024-06-09T22:22:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 340.98923288100002 2024-06-09T22:22:00.675 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:22:00.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.670+0000 7fc620f42700 1 -- 172.21.15.104:0/2815111114 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc6040a2eb0 msgr2=0x7fc6040a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:00.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.670+0000 7fc620f42700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc6040a2eb0 0x7fc6040a5370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0787d0 tx=0x7fc61000b3c0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.670+0000 7fc620f42700 1 -- 172.21.15.104:0/2815111114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc61c101390 msgr2=0x7fc61c1978b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:00.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.670+0000 7fc620f42700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc61c101390 0x7fc61c1978b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00d950 tx=0x7fc60c00dc60 comp rx=0 tx=0).stop 2024-06-09T22:22:00.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.670+0000 7fc620f42700 1 -- 172.21.15.104:0/2815111114 shutdown_connections 2024-06-09T22:22:00.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.670+0000 7fc620f42700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc6040a2eb0 0x7fc6040a5370 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.670+0000 7fc620f42700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc61c107400 0x7fc61c19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.670+0000 7fc620f42700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc61c101390 0x7fc61c1978b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.670+0000 7fc620f42700 1 --2- 172.21.15.104:0/2815111114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc61c101d50 0x7fc61c197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:00.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.670+0000 7fc620f42700 1 -- 172.21.15.104:0/2815111114 >> 172.21.15.104:0/2815111114 conn(0x7fc61c078340 msgr2=0x7fc61c0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:00.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.670+0000 7fc620f42700 1 -- 172.21.15.104:0/2815111114 shutdown_connections 2024-06-09T22:22:00.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:00.670+0000 7fc620f42700 1 -- 172.21.15.104:0/2815111114 wait complete. 2024-06-09T22:22:01.676 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:01 smithi104 ceph-mon[32389]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-06-09T22:22:01.703 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-06-09T22:22:01.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:01 smithi194 ceph-mon[36856]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-06-09T22:22:01.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.930+0000 7f6bdd44e700 1 -- 172.21.15.104:0/3569674552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bd81009e0 msgr2=0x7f6bd8100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:01.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.930+0000 7f6bdd44e700 1 --2- 172.21.15.104:0/3569674552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bd81009e0 0x7f6bd8100e60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00099c0 tx=0x7f6bc0009cd0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.930+0000 7f6bdd44e700 1 -- 172.21.15.104:0/3569674552 shutdown_connections 2024-06-09T22:22:01.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.930+0000 7f6bdd44e700 1 --2- 172.21.15.104:0/3569674552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bd81013a0 0x7f6bd810f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.930+0000 7f6bdd44e700 1 --2- 172.21.15.104:0/3569674552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bd8108560 0x7f6bd8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.930+0000 7f6bdd44e700 1 --2- 172.21.15.104:0/3569674552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bd81009e0 0x7f6bd8100e60 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.930+0000 7f6bdd44e700 1 -- 172.21.15.104:0/3569674552 >> 172.21.15.104:0/3569674552 conn(0x7f6bd8077990 msgr2=0x7f6bd80fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:01.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.930+0000 7f6bdd44e700 1 -- 172.21.15.104:0/3569674552 shutdown_connections 2024-06-09T22:22:01.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.930+0000 7f6bdd44e700 1 -- 172.21.15.104:0/3569674552 wait complete. 2024-06-09T22:22:01.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.931+0000 7f6bdd44e700 1 Processor -- start 2024-06-09T22:22:01.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.931+0000 7f6bdd44e700 1 -- start start 2024-06-09T22:22:01.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.931+0000 7f6bdd44e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bd81009e0 0x7f6bd8101bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.931+0000 7f6bdd44e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bd81013a0 0x7f6bd8102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.931+0000 7f6bdd44e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bd8108560 0x7f6bd8102640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.931+0000 7f6bdd44e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd81127c0 con 0x7f6bd8108560 2024-06-09T22:22:01.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.931+0000 7f6bdd44e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd8112640 con 0x7f6bd81013a0 2024-06-09T22:22:01.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.931+0000 7f6bdd44e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd8112940 con 0x7f6bd81009e0 2024-06-09T22:22:01.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.932+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bd8108560 0x7f6bd8102640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.932+0000 7f6bceffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bd81009e0 0x7f6bd8101bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.932+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bd8108560 0x7f6bd8102640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45672/0 (socket says 172.21.15.104:45672) 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.932+0000 7f6bd77fe700 1 -- 172.21.15.104:0/1564559801 learned_addr learned my addr 172.21.15.104:0/1564559801 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.932+0000 7f6bd6ffd700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bd81013a0 0x7f6bd8102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.932+0000 7f6bd77fe700 1 -- 172.21.15.104:0/1564559801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bd81009e0 msgr2=0x7f6bd8101bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.932+0000 7f6bd77fe700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bd81009e0 0x7f6bd8101bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.932+0000 7f6bd77fe700 1 -- 172.21.15.104:0/1564559801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bd81013a0 msgr2=0x7f6bd8102100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.932+0000 7f6bd77fe700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bd81013a0 0x7f6bd8102100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.932+0000 7f6bd77fe700 1 -- 172.21.15.104:0/1564559801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd81a7080 con 0x7f6bd8108560 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.932+0000 7f6bd6ffd700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bd81013a0 0x7f6bd8102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.932+0000 7f6bd77fe700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bd8108560 0x7f6bd8102640 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0007ee0 tx=0x7f6bd000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.933+0000 7f6bd4ff9700 1 -- 172.21.15.104:0/1564559801 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0004bb0 con 0x7f6bd8108560 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.933+0000 7f6bdd44e700 1 -- 172.21.15.104:0/1564559801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd81a7370 con 0x7f6bd8108560 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.933+0000 7f6bdd44e700 1 -- 172.21.15.104:0/1564559801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd81a78d0 con 0x7f6bd8108560 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.933+0000 7f6bd4ff9700 1 -- 172.21.15.104:0/1564559801 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bd0004d50 con 0x7f6bd8108560 2024-06-09T22:22:01.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.933+0000 7f6bd4ff9700 1 -- 172.21.15.104:0/1564559801 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0005960 con 0x7f6bd8108560 2024-06-09T22:22:01.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.935+0000 7f6bd4ff9700 1 -- 172.21.15.104:0/1564559801 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) ==== 138061+0+0 (secure 0 0 0) 0x7f6bd001a020 con 0x7f6bd8108560 2024-06-09T22:22:01.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.936+0000 7f6bd4ff9700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6bc40a2f30 0x7f6bc40a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.936+0000 7f6bceffd700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6bc40a2f30 0x7f6bc40a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.936+0000 7f6bd4ff9700 1 -- 172.21.15.104:0/1564559801 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6bd00cf130 con 0x7f6bd8108560 2024-06-09T22:22:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.937+0000 7f6be7cd1b80 1 Processor -- start 2024-06-09T22:22:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.937+0000 7f6bceffd700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6bc40a2f30 0x7f6bc40a53f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800abd0 tx=0x7f6bc800a380 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:01.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.937+0000 7f6be7cd1b80 1 -- start start 2024-06-09T22:22:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.937+0000 7f6be7cd1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 0x5579b2899340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.937+0000 7f6be7cd1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2899880 0x5579b2893260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.937+0000 7f6be7cd1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b2893bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.937+0000 7f6be7cd1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5579b2785f60 con 0x5579b2898f60 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.937+0000 7f6be7cd1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5579b27e4850 con 0x5579b28937a0 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.937+0000 7f6be7cd1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5579b274c460 con 0x5579b2899880 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bceffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 0x5579b2899340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bd6ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2899880 0x5579b2893260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bd6ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2899880 0x5579b2893260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:52714/0 (socket says 172.21.15.104:52714) 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bceffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 0x5579b2899340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45682/0 (socket says 172.21.15.104:45682) 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bd6ffd700 1 -- 172.21.15.104:0/396984577 learned_addr learned my addr 172.21.15.104:0/396984577 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bd77fe700 1 --2- 172.21.15.104:0/396984577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b2893bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bd6ffd700 1 -- 172.21.15.104:0/396984577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 msgr2=0x5579b2893bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bd6ffd700 1 --2- 172.21.15.104:0/396984577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b2893bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bd6ffd700 1 -- 172.21.15.104:0/396984577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 msgr2=0x5579b2899340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bd6ffd700 1 --2- 172.21.15.104:0/396984577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 0x5579b2899340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bd6ffd700 1 -- 172.21.15.104:0/396984577 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579b27338d0 con 0x5579b2899880 2024-06-09T22:22:01.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bd6ffd700 1 --2- 172.21.15.104:0/396984577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2899880 0x5579b2893260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00099c0 tx=0x7f6bc0005c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28493c789f1dd0f4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:01.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bcd7fa700 1 -- 172.21.15.104:0/396984577 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0028070 con 0x5579b2899880 2024-06-09T22:22:01.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bcd7fa700 1 -- 172.21.15.104:0/396984577 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bc0003e40 con 0x5579b2899880 2024-06-09T22:22:01.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.938+0000 7f6bcd7fa700 1 -- 172.21.15.104:0/396984577 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0023460 con 0x5579b2899880 2024-06-09T22:22:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.939+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/396984577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2899880 msgr2=0x5579b2893260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.939+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/396984577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2899880 0x5579b2893260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00099c0 tx=0x7f6bc0005c20 comp rx=0 tx=0).stop 2024-06-09T22:22:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.940+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/396984577 shutdown_connections 2024-06-09T22:22:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.940+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/396984577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 0x5579b2899340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.940+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/396984577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b2893bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.940+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/396984577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2899880 0x5579b2893260 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.940+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/396984577 >> 172.21.15.104:0/396984577 conn(0x5579b27c0080 msgr2=0x5579b288e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.940+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/396984577 shutdown_connections 2024-06-09T22:22:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.940+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/396984577 wait complete. 2024-06-09T22:22:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.940+0000 7f6be7cd1b80 1 -- start start 2024-06-09T22:22:01.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.940+0000 7f6be7cd1b80 1 Processor -- start 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.940+0000 7f6be7cd1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b289f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6be7cd1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2898f60 0x5579b289c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6be7cd1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 0x5579b289ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6be7cd1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5579b2784fc0 con 0x5579b2899880 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6be7cd1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5579b27e0810 con 0x5579b28937a0 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 0x5579b289ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 0x5579b289ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45684/0 (socket says 172.21.15.104:45684) 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bd77fe700 1 -- 172.21.15.104:0/682887131 learned_addr learned my addr 172.21.15.104:0/682887131 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/682887131 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5579b28a0200 con 0x5579b2898f60 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6be7cd1b80 10 client.0 _ll_register_callbacks cb 0x5579b277a5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bd6ffd700 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2898f60 0x5579b289c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bceffd700 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b289f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bd77fe700 1 -- 172.21.15.104:0/682887131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2898f60 msgr2=0x5579b289c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bd77fe700 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2898f60 0x5579b289c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bd77fe700 1 -- 172.21.15.104:0/682887131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 msgr2=0x5579b289f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bd77fe700 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b289f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bd77fe700 1 -- 172.21.15.104:0/682887131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579b27aa750 con 0x5579b2899880 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bceffd700 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b289f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bd77fe700 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 0x5579b289ce80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6bd009ad90 tx=0x7f6bd009aa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b107e1b9b4cbeee6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bbe7fc700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bbe7fc700 1 -- 172.21.15.104:0/682887131 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd00b4060 con 0x5579b2899880 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.941+0000 7f6bbe7fc700 1 -- 172.21.15.104:0/682887131 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6bd009b2f0 con 0x5579b2899880 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.942+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/682887131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5579b27f1d50 con 0x5579b2899880 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.942+0000 7f6bbe7fc700 1 -- 172.21.15.104:0/682887131 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6bd009f440 con 0x5579b2899880 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.942+0000 7f6be7cd1b80 10 client.15717 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.942+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/682887131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5579b28aaa40 con 0x5579b2899880 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.942+0000 7f6bbe7fc700 1 -- 172.21.15.104:0/682887131 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bd00a3d00 con 0x5579b2899880 2024-06-09T22:22:01.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.942+0000 7f6bbe7fc700 1 -- 172.21.15.104:0/682887131 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6bd00cc790 con 0x5579b2899880 2024-06-09T22:22:01.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.942+0000 7f6be7cd1b80 10 client.15717 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:01.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.942+0000 7f6be7cd1b80 10 client.15717 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:22:01.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.942+0000 7f6be7cd1b80 20 client.15717 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:01.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.942+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5579b28ad080 0x5579b28af470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.942+0000 7f6be7cd1b80 4 client.15717 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:01.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.942+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/682887131 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579b27f1d50 con 0x5579b28ad080 2024-06-09T22:22:01.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.943+0000 7f6bceffd700 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5579b28ad080 0x5579b28af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.943+0000 7f6bbe7fc700 10 client.15717 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:22:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.943+0000 7f6bceffd700 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5579b28ad080 0x5579b28af470 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4316c4068ae548cb server_cookie=7cecb6a93de95f2 in_seq=0 out_seq=0 2024-06-09T22:22:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6bbe7fc700 1 -- 172.21.15.104:0/682887131 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5579b27f1d50 con 0x5579b28ad080 2024-06-09T22:22:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6bbe7fc700 10 client.15717 handle_command_reply: tid=0 2024-06-09T22:22:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6be7cd1b80 1 client.15717 shutdown 2024-06-09T22:22:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/682887131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 msgr2=0x5579b289ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 0x5579b289ce80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6bd009ad90 tx=0x7f6bd009aa60 comp rx=0 tx=0).stop 2024-06-09T22:22:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/682887131 shutdown_connections 2024-06-09T22:22:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 0x5579b289ce80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5579b28ad080 0x5579b28af470 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b289f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/682887131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2898f60 0x5579b289c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/682887131 >> 172.21.15.104:0/682887131 conn(0x5579b27c0080 msgr2=0x5579b288d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6bbe7fc700 0 client.15717 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:22:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/682887131 shutdown_connections 2024-06-09T22:22:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/682887131 wait complete. 2024-06-09T22:22:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.944+0000 7f6be7cd1b80 20 client.15717 trim_cache size 0 max 16384 2024-06-09T22:22:01.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.973+0000 7f6be7cd1b80 1 Processor -- start 2024-06-09T22:22:01.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.973+0000 7f6be7cd1b80 1 -- start start 2024-06-09T22:22:01.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6be7cd1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b28937a0 0x5579b289a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6be7cd1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b2898f60 0x5579b288de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6be7cd1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 0x5579b288e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6be7cd1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd81b0660 con 0x5579b2899880 2024-06-09T22:22:01.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6be7cd1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd81b21a0 con 0x5579b2898f60 2024-06-09T22:22:01.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6be7cd1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd81b0d30 con 0x5579b28937a0 2024-06-09T22:22:01.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 0x5579b288e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 0x5579b288e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45686/0 (socket says 172.21.15.104:45686) 2024-06-09T22:22:01.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6bd77fe700 1 -- 172.21.15.104:0/2380410324 learned_addr learned my addr 172.21.15.104:0/2380410324 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:01.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6bceffd700 1 --2- 172.21.15.104:0/2380410324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b28937a0 0x5579b289a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6bd6ffd700 1 --2- 172.21.15.104:0/2380410324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b2898f60 0x5579b288de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6bd77fe700 1 -- 172.21.15.104:0/2380410324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b28937a0 msgr2=0x5579b289a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:01.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6bd77fe700 1 --2- 172.21.15.104:0/2380410324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b28937a0 0x5579b289a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6bd77fe700 1 -- 172.21.15.104:0/2380410324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b2898f60 msgr2=0x5579b288de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:01.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.974+0000 7f6bd77fe700 1 --2- 172.21.15.104:0/2380410324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b2898f60 0x5579b288de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.975+0000 7f6bd77fe700 1 -- 172.21.15.104:0/2380410324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579b288f9a0 con 0x5579b2899880 2024-06-09T22:22:01.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.975+0000 7f6bd77fe700 1 --2- 172.21.15.104:0/2380410324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 0x5579b288e3c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000d700 tx=0x7f6bd00bb2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef97ef5b016a8934 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:01.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.975+0000 7f6bccff9700 1 -- 172.21.15.104:0/2380410324 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd00b4040 con 0x5579b2899880 2024-06-09T22:22:01.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.975+0000 7f6bccff9700 1 -- 172.21.15.104:0/2380410324 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bd00a3d00 con 0x5579b2899880 2024-06-09T22:22:01.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.975+0000 7f6bccff9700 1 -- 172.21.15.104:0/2380410324 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd009b2f0 con 0x5579b2899880 2024-06-09T22:22:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.976+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2380410324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 msgr2=0x5579b288e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.976+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/2380410324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 0x5579b288e3c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000d700 tx=0x7f6bd00bb2c0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.976+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2380410324 shutdown_connections 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.976+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/2380410324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2899880 0x5579b288e3c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.976+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/2380410324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b28937a0 0x5579b289a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.976+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/2380410324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b2898f60 0x5579b288de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.976+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2380410324 >> 172.21.15.104:0/2380410324 conn(0x5579b27c0080 msgr2=0x5579b2899c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.976+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2380410324 shutdown_connections 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.976+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2380410324 wait complete. 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.976+0000 7f6be7cd1b80 1 -- start start 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6be7cd1b80 1 Processor -- start 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6be7cd1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b28a5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6be7cd1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 0x5579b28a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6be7cd1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2899880 0x5579b28b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6be7cd1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd81b1ad0 con 0x5579b2898f60 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6be7cd1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd81b1400 con 0x5579b28937a0 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6be7cd1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd81ae5e0 con 0x5579b2899880 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6bd6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 0x5579b28a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6be7cd1b80 10 client.0 _ll_register_callbacks cb 0x5579b277a5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6bd6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 0x5579b28a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45696/0 (socket says 172.21.15.104:45696) 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6bd6ffd700 1 -- 172.21.15.104:0/2671317035 learned_addr learned my addr 172.21.15.104:0/2671317035 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6bd77fe700 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2899880 0x5579b28b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6bceffd700 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b28a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6bd6ffd700 1 -- 172.21.15.104:0/2671317035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2899880 msgr2=0x5579b28b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6bd6ffd700 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2899880 0x5579b28b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6bd6ffd700 1 -- 172.21.15.104:0/2671317035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 msgr2=0x5579b28a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6bd6ffd700 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b28a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.977+0000 7f6bd6ffd700 1 -- 172.21.15.104:0/2671317035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579b28a6130 con 0x5579b2898f60 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.978+0000 7f6bd6ffd700 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 0x5579b28a5a60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00099c0 tx=0x7f6bc000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac831e2cf680f57 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:01.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.978+0000 7f6bbe7fc700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.978+0000 7f6bbe7fc700 1 -- 172.21.15.104:0/2671317035 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0028070 con 0x5579b2898f60 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.978+0000 7f6bbe7fc700 1 -- 172.21.15.104:0/2671317035 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6bc00054e0 con 0x5579b2898f60 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.978+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2671317035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bd00a3d00 con 0x5579b2898f60 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.978+0000 7f6bbe7fc700 1 -- 172.21.15.104:0/2671317035 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6bc003c050 con 0x5579b2898f60 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.978+0000 7f6be7cd1b80 10 client.15723 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.978+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2671317035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5579b2917290 con 0x5579b2898f60 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.978+0000 7f6bbe7fc700 1 -- 172.21.15.104:0/2671317035 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bd00a3d00 con 0x5579b2898f60 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.978+0000 7f6bbe7fc700 1 -- 172.21.15.104:0/2671317035 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6bc000fc10 con 0x5579b2898f60 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.979+0000 7f6be7cd1b80 10 client.15723 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.979+0000 7f6be7cd1b80 10 client.15723 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.979+0000 7f6be7cd1b80 20 client.15723 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.979+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5579b2917460 0x5579b2919850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.979+0000 7f6be7cd1b80 4 client.15723 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:22:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.979+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2671317035 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5579b27ec2d0 con 0x5579b2917460 2024-06-09T22:22:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.979+0000 7f6bceffd700 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5579b2917460 0x5579b2919850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.979+0000 7f6bceffd700 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5579b2917460 0x5579b2919850 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=219bf64186195935 server_cookie=de599e6e21fdcc02 in_seq=0 out_seq=0 2024-06-09T22:22:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.979+0000 7f6bbe7fc700 10 client.15723 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:22:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6bbe7fc700 1 -- 172.21.15.104:0/2671317035 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5579b27ec2d0 con 0x5579b2917460 2024-06-09T22:22:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6bbe7fc700 10 client.15723 handle_command_reply: tid=0 2024-06-09T22:22:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6be7cd1b80 1 client.15723 shutdown 2024-06-09T22:22:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2671317035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 msgr2=0x5579b28a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 0x5579b28a5a60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00099c0 tx=0x7f6bc000bd70 comp rx=0 tx=0).stop 2024-06-09T22:22:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2671317035 shutdown_connections 2024-06-09T22:22:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x5579b2917460 0x5579b2919850 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5579b2899880 0x5579b28b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6bbe7fc700 0 client.15723 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:22:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5579b28937a0 0x5579b28a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6be7cd1b80 1 --2- 172.21.15.104:0/2671317035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5579b2898f60 0x5579b28a5a60 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2671317035 >> 172.21.15.104:0/2671317035 conn(0x5579b27c0080 msgr2=0x5579b2899c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2671317035 shutdown_connections 2024-06-09T22:22:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6be7cd1b80 1 -- 172.21.15.104:0/2671317035 wait complete. 2024-06-09T22:22:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.980+0000 7f6be7cd1b80 20 client.15723 trim_cache size 0 max 16384 2024-06-09T22:22:01.984 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:22:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.981+0000 7f6bdd44e700 1 -- 172.21.15.104:0/1564559801 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6bc40a2f30 msgr2=0x7f6bc40a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.981+0000 7f6bdd44e700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6bc40a2f30 0x7f6bc40a53f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800abd0 tx=0x7f6bc800a380 comp rx=0 tx=0).stop 2024-06-09T22:22:01.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.981+0000 7f6bdd44e700 1 -- 172.21.15.104:0/1564559801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bd8108560 msgr2=0x7f6bd8102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:01.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.981+0000 7f6bdd44e700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bd8108560 0x7f6bd8102640 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0007ee0 tx=0x7f6bd000d3b0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.981+0000 7f6bdd44e700 1 -- 172.21.15.104:0/1564559801 shutdown_connections 2024-06-09T22:22:01.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.981+0000 7f6bdd44e700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6bc40a2f30 0x7f6bc40a53f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.981+0000 7f6bdd44e700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6bd8108560 0x7f6bd8102640 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.981+0000 7f6bdd44e700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bd81009e0 0x7f6bd8101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.981+0000 7f6bdd44e700 1 --2- 172.21.15.104:0/1564559801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bd81013a0 0x7f6bd8102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:01.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.981+0000 7f6bdd44e700 1 -- 172.21.15.104:0/1564559801 >> 172.21.15.104:0/1564559801 conn(0x7f6bd8077990 msgr2=0x7f6bd80fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:01.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.981+0000 7f6bdd44e700 1 -- 172.21.15.104:0/1564559801 shutdown_connections 2024-06-09T22:22:01.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:01.981+0000 7f6bdd44e700 1 -- 172.21.15.104:0/1564559801 wait complete. 2024-06-09T22:22:01.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:01 smithi153 ceph-mon[36599]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-06-09T22:22:02.844 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-09T22:22:03.108 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:03.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:03 smithi104 ceph-mon[32389]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-09T22:22:03.501 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:03 smithi104 ceph-mon[32389]: mgrmap e24: x(active, since 64s), standbys: y, z 2024-06-09T22:22:03.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:03 smithi194 ceph-mon[36856]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-09T22:22:03.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:03 smithi194 ceph-mon[36856]: mgrmap e24: x(active, since 64s), standbys: y, z 2024-06-09T22:22:03.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:03 smithi153 ceph-mon[36599]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-06-09T22:22:03.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:03 smithi153 ceph-mon[36599]: mgrmap e24: x(active, since 64s), standbys: y, z 2024-06-09T22:22:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.976+0000 7fdea1c8f700 1 -- 172.21.15.104:0/3965162057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde9c101440 msgr2=0x7fde9c10f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.976+0000 7fdea1c8f700 1 --2- 172.21.15.104:0/3965162057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde9c101440 0x7fde9c10f330 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fde9400b0a0 tx=0x7fde9400b3b0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.976+0000 7fdea1c8f700 1 -- 172.21.15.104:0/3965162057 shutdown_connections 2024-06-09T22:22:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.976+0000 7fdea1c8f700 1 --2- 172.21.15.104:0/3965162057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde9c101440 0x7fde9c10f330 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.976+0000 7fdea1c8f700 1 --2- 172.21.15.104:0/3965162057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde9c108510 0x7fde9c1088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.976+0000 7fdea1c8f700 1 --2- 172.21.15.104:0/3965162057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde9c100a80 0x7fde9c100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.976+0000 7fdea1c8f700 1 -- 172.21.15.104:0/3965162057 >> 172.21.15.104:0/3965162057 conn(0x7fde9c077a00 msgr2=0x7fde9c0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:04.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.976+0000 7fdea1c8f700 1 -- 172.21.15.104:0/3965162057 shutdown_connections 2024-06-09T22:22:04.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.976+0000 7fdea1c8f700 1 -- 172.21.15.104:0/3965162057 wait complete. 2024-06-09T22:22:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.977+0000 7fdea1c8f700 1 Processor -- start 2024-06-09T22:22:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.978+0000 7fdea1c8f700 1 -- start start 2024-06-09T22:22:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.978+0000 7fdea1c8f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde9c100a80 0x7fde9c19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.978+0000 7fdea1c8f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde9c101440 0x7fde9c19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.978+0000 7fdea1c8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde9c108510 0x7fde9c198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.978+0000 7fdea1c8f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c112860 con 0x7fde9c108510 2024-06-09T22:22:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.978+0000 7fdea1c8f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c1126e0 con 0x7fde9c101440 2024-06-09T22:22:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.978+0000 7fdea1c8f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c1129e0 con 0x7fde9c100a80 2024-06-09T22:22:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.978+0000 7fde9b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde9c100a80 0x7fde9c19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.978+0000 7fde9affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde9c101440 0x7fde9c19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:04.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.978+0000 7fde9b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde9c100a80 0x7fde9c19e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:52772/0 (socket says 172.21.15.104:52772) 2024-06-09T22:22:04.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.979+0000 7fde9b7fe700 1 -- 172.21.15.104:0/2365182181 learned_addr learned my addr 172.21.15.104:0/2365182181 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:04.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.979+0000 7fde9bfff700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde9c108510 0x7fde9c198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.979+0000 7fde9affd700 1 -- 172.21.15.104:0/2365182181 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde9c100a80 msgr2=0x7fde9c19e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.979+0000 7fde9affd700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde9c100a80 0x7fde9c19e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.979+0000 7fde9affd700 1 -- 172.21.15.104:0/2365182181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde9c108510 msgr2=0x7fde9c198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.979+0000 7fde9affd700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde9c108510 0x7fde9c198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.979+0000 7fde9affd700 1 -- 172.21.15.104:0/2365182181 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9c1991c0 con 0x7fde9c101440 2024-06-09T22:22:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.979+0000 7fde9bfff700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde9c108510 0x7fde9c198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.979+0000 7fde9affd700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde9c101440 0x7fde9c19ede0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fde9000b450 tx=0x7fde9000b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.980+0000 7fde98ff9700 1 -- 172.21.15.104:0/2365182181 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde90015020 con 0x7fde9c101440 2024-06-09T22:22:04.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.980+0000 7fde98ff9700 1 -- 172.21.15.104:0/2365182181 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde90011d10 con 0x7fde9c101440 2024-06-09T22:22:04.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.980+0000 7fdea1c8f700 1 -- 172.21.15.104:0/2365182181 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde9c1994b0 con 0x7fde9c101440 2024-06-09T22:22:04.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.980+0000 7fdea1c8f700 1 -- 172.21.15.104:0/2365182181 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde9c101b90 con 0x7fde9c101440 2024-06-09T22:22:04.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.980+0000 7fde98ff9700 1 -- 172.21.15.104:0/2365182181 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde90010d80 con 0x7fde9c101440 2024-06-09T22:22:04.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.983+0000 7fde98ff9700 1 -- 172.21.15.104:0/2365182181 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 24) ==== 138118+0+0 (secure 0 0 0) 0x7fde90018020 con 0x7fde9c101440 2024-06-09T22:22:04.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.983+0000 7fdeac512b80 1 Processor -- start 2024-06-09T22:22:04.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.983+0000 7fdeac512b80 1 -- start start 2024-06-09T22:22:04.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fdeac512b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca22770 0x563e5ca22b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:04.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fdeac512b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 0x563e5ca1ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:04.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fdeac512b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca1cfb0 0x563e5ca1d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:04.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fdeac512b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x563e5c906f60 con 0x563e5ca23090 2024-06-09T22:22:04.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fdeac512b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563e5c965850 con 0x563e5ca1cfb0 2024-06-09T22:22:04.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fdeac512b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563e5c8cd460 con 0x563e5ca22770 2024-06-09T22:22:04.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fde9b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 0x563e5ca1ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:04.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fde9b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 0x563e5ca1ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45728/0 (socket says 172.21.15.104:45728) 2024-06-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fde9b7fe700 1 -- 172.21.15.104:0/2343597505 learned_addr learned my addr 172.21.15.104:0/2343597505 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fde9bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca22770 0x563e5ca22b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fde98ff9700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fde840a2fc0 0x7fde840a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fde9bfff700 1 -- 172.21.15.104:0/2343597505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca1cfb0 msgr2=0x563e5ca1d3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fde9bfff700 1 --2- 172.21.15.104:0/2343597505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca1cfb0 0x563e5ca1d3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fde9bfff700 1 -- 172.21.15.104:0/2343597505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 msgr2=0x563e5ca1ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fde98ff9700 1 -- 172.21.15.104:0/2365182181 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fde900cef30 con 0x7fde9c101440 2024-06-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fde9bfff700 1 --2- 172.21.15.104:0/2343597505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 0x563e5ca1ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fde9bfff700 1 -- 172.21.15.104:0/2343597505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e5c8b48d0 con 0x563e5ca22770 2024-06-09T22:22:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.984+0000 7fde9bfff700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fde840a2fc0 0x7fde840a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.985+0000 7fde9bfff700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fde840a2fc0 0x7fde840a5480 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fde94004fd0 tx=0x7fde9401f000 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.985+0000 7fde9bfff700 1 --2- 172.21.15.104:0/2343597505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca22770 0x563e5ca22b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fde9400b070 tx=0x7fde94004b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa25f2f81a4fdf68 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.985+0000 7fde81ffb700 1 -- 172.21.15.104:0/2343597505 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9403d070 con 0x563e5ca22770 2024-06-09T22:22:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.985+0000 7fde81ffb700 1 -- 172.21.15.104:0/2343597505 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde94007bb0 con 0x563e5ca22770 2024-06-09T22:22:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.985+0000 7fde81ffb700 1 -- 172.21.15.104:0/2343597505 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9401e690 con 0x563e5ca22770 2024-06-09T22:22:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.986+0000 7fdeac512b80 1 -- 172.21.15.104:0/2343597505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca22770 msgr2=0x563e5ca22b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.986+0000 7fdeac512b80 1 --2- 172.21.15.104:0/2343597505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca22770 0x563e5ca22b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fde9400b070 tx=0x7fde94004b00 comp rx=0 tx=0).stop 2024-06-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.986+0000 7fdeac512b80 1 -- 172.21.15.104:0/2343597505 shutdown_connections 2024-06-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.986+0000 7fdeac512b80 1 --2- 172.21.15.104:0/2343597505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca1cfb0 0x563e5ca1d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.986+0000 7fdeac512b80 1 --2- 172.21.15.104:0/2343597505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca22770 0x563e5ca22b50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.986+0000 7fdeac512b80 1 --2- 172.21.15.104:0/2343597505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 0x563e5ca1ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.986+0000 7fdeac512b80 1 -- 172.21.15.104:0/2343597505 >> 172.21.15.104:0/2343597505 conn(0x563e5c941080 msgr2=0x563e5ca0f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.986+0000 7fdeac512b80 1 -- 172.21.15.104:0/2343597505 shutdown_connections 2024-06-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.986+0000 7fdeac512b80 1 -- 172.21.15.104:0/2343597505 wait complete. 2024-06-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.986+0000 7fdeac512b80 1 -- start start 2024-06-09T22:22:04.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fdeac512b80 1 Processor -- start 2024-06-09T22:22:04.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fdeac512b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca1cfb0 0x563e5ca28ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:04.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fdeac512b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca22770 0x563e5ca261a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:04.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fdeac512b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca23090 0x563e5ca266e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:04.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fdeac512b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x563e5c905fc0 con 0x563e5ca1cfb0 2024-06-09T22:22:04.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fdeac512b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563e5c961810 con 0x563e5ca22770 2024-06-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fdeac512b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563e5ca29a10 con 0x563e5ca23090 2024-06-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fdeac512b80 10 client.0 _ll_register_callbacks cb 0x563e5c8fb5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fde9affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca22770 0x563e5ca261a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fde9affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca22770 0x563e5ca261a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:53176/0 (socket says 172.21.15.104:53176) 2024-06-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fde9affd700 1 -- 172.21.15.104:0/606721018 learned_addr learned my addr 172.21.15.104:0/606721018 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fde9bfff700 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca23090 0x563e5ca266e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.987+0000 7fde9b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca1cfb0 0x563e5ca28ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.988+0000 7fde9affd700 1 -- 172.21.15.104:0/606721018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca23090 msgr2=0x563e5ca266e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.988+0000 7fde9affd700 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca23090 0x563e5ca266e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.988+0000 7fde9affd700 1 -- 172.21.15.104:0/606721018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca1cfb0 msgr2=0x563e5ca28ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.988+0000 7fde9affd700 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca1cfb0 0x563e5ca28ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.988+0000 7fde9affd700 1 -- 172.21.15.104:0/606721018 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e5c92b750 con 0x563e5ca22770 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.988+0000 7fde9b7fe700 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca1cfb0 0x563e5ca28ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.988+0000 7fde9affd700 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca22770 0x563e5ca261a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fde9009acb0 tx=0x7fde9009ae70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72a9989c6e1ff099 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.988+0000 7fde7affd700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.988+0000 7fde7affd700 1 -- 172.21.15.104:0/606721018 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9000be60 con 0x563e5ca22770 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.988+0000 7fdeac512b80 1 -- 172.21.15.104:0/606721018 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde94007bb0 con 0x563e5ca22770 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.988+0000 7fde7affd700 1 -- 172.21.15.104:0/606721018 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fde900a3990 con 0x563e5ca22770 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.989+0000 7fde7affd700 1 -- 172.21.15.104:0/606721018 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fde900bb0b0 con 0x563e5ca22770 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.989+0000 7fdeac512b80 10 client.25214 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.989+0000 7fdeac512b80 1 -- 172.21.15.104:0/606721018 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563e5ca34210 con 0x563e5ca22770 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.989+0000 7fde7affd700 1 -- 172.21.15.104:0/606721018 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde94007bb0 con 0x563e5ca22770 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.989+0000 7fde7affd700 1 -- 172.21.15.104:0/606721018 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fde9009a590 con 0x563e5ca22770 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.989+0000 7fdeac512b80 10 client.25214 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.989+0000 7fdeac512b80 10 client.25214 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.989+0000 7fdeac512b80 20 client.25214 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.989+0000 7fdeac512b80 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x563e5ca367a0 0x563e5ca38b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.989+0000 7fdeac512b80 4 client.25214 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.989+0000 7fdeac512b80 1 -- 172.21.15.104:0/606721018 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e5c972d50 con 0x563e5ca367a0 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.990+0000 7fde9b7fe700 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x563e5ca367a0 0x563e5ca38b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.990+0000 7fde9b7fe700 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x563e5ca367a0 0x563e5ca38b90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=72bd9a6130d000fa server_cookie=6648fbec6a048907 in_seq=0 out_seq=0 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.990+0000 7fde7affd700 10 client.25214 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.991+0000 7fde7affd700 1 -- 172.21.15.104:0/606721018 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x563e5c972d50 con 0x563e5ca367a0 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.991+0000 7fde7affd700 10 client.25214 handle_command_reply: tid=0 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.991+0000 7fdeac512b80 1 client.25214 shutdown 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.991+0000 7fdeac512b80 1 -- 172.21.15.104:0/606721018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca22770 msgr2=0x563e5ca261a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.991+0000 7fdeac512b80 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca22770 0x563e5ca261a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fde9009acb0 tx=0x7fde9009ae70 comp rx=0 tx=0).stop 2024-06-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.991+0000 7fdeac512b80 1 -- 172.21.15.104:0/606721018 shutdown_connections 2024-06-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.991+0000 7fdeac512b80 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x563e5ca367a0 0x563e5ca38b90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.991+0000 7fdeac512b80 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca1cfb0 0x563e5ca28ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.991+0000 7fdeac512b80 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca23090 0x563e5ca266e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.991+0000 7fde7affd700 0 client.25214 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.991+0000 7fdeac512b80 1 --2- 172.21.15.104:0/606721018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca22770 0x563e5ca261a0 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fde9009acb0 tx=0x7fde9009ae70 comp rx=0 tx=0).stop 2024-06-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.991+0000 7fdeac512b80 1 -- 172.21.15.104:0/606721018 >> 172.21.15.104:0/606721018 conn(0x563e5c941080 msgr2=0x563e5ca0e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.992+0000 7fdeac512b80 1 -- 172.21.15.104:0/606721018 shutdown_connections 2024-06-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.992+0000 7fdeac512b80 1 -- 172.21.15.104:0/606721018 wait complete. 2024-06-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:04.992+0000 7fdeac512b80 20 client.25214 trim_cache size 0 max 16384 2024-06-09T22:22:05.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fdeac512b80 1 Processor -- start 2024-06-09T22:22:05.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fdeac512b80 1 -- start start 2024-06-09T22:22:05.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fdeac512b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca1cfb0 0x563e5ca1ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fdeac512b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca23090 0x563e5ca23c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fdeac512b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca25a50 0x563e5ca0ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fdeac512b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c072970 con 0x563e5ca25a50 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fdeac512b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c0744b0 con 0x563e5ca23090 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fdeac512b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c073040 con 0x563e5ca1cfb0 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fde9bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca25a50 0x563e5ca0ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fde9bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca25a50 0x563e5ca0ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45746/0 (socket says 172.21.15.104:45746) 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fde9bfff700 1 -- 172.21.15.104:0/3672012453 learned_addr learned my addr 172.21.15.104:0/3672012453 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fde9affd700 1 --2- 172.21.15.104:0/3672012453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca23090 0x563e5ca23c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.019+0000 7fde9b7fe700 1 --2- 172.21.15.104:0/3672012453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca1cfb0 0x563e5ca1ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.020+0000 7fde9bfff700 1 -- 172.21.15.104:0/3672012453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca1cfb0 msgr2=0x563e5ca1ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.020+0000 7fde9bfff700 1 --2- 172.21.15.104:0/3672012453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca1cfb0 0x563e5ca1ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.020+0000 7fde9bfff700 1 -- 172.21.15.104:0/3672012453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca23090 msgr2=0x563e5ca23c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.020+0000 7fde9bfff700 1 --2- 172.21.15.104:0/3672012453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca23090 0x563e5ca23c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.020+0000 7fde9bfff700 1 -- 172.21.15.104:0/3672012453 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e5ca39170 con 0x563e5ca25a50 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.020+0000 7fde9bfff700 1 --2- 172.21.15.104:0/3672012453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca25a50 0x563e5ca0ee80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fde9403c040 tx=0x7fde94007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb675fa95210fe1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.020+0000 7fde817fa700 1 -- 172.21.15.104:0/3672012453 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9404a070 con 0x563e5ca25a50 2024-06-09T22:22:05.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.020+0000 7fde817fa700 1 -- 172.21.15.104:0/3672012453 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde94042450 con 0x563e5ca25a50 2024-06-09T22:22:05.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.020+0000 7fde817fa700 1 -- 172.21.15.104:0/3672012453 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9403d040 con 0x563e5ca25a50 2024-06-09T22:22:05.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.021+0000 7fdeac512b80 1 -- 172.21.15.104:0/3672012453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca25a50 msgr2=0x563e5ca0ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.021+0000 7fdeac512b80 1 --2- 172.21.15.104:0/3672012453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca25a50 0x563e5ca0ee80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fde9403c040 tx=0x7fde94007c70 comp rx=0 tx=0).stop 2024-06-09T22:22:05.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.021+0000 7fdeac512b80 1 -- 172.21.15.104:0/3672012453 shutdown_connections 2024-06-09T22:22:05.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.021+0000 7fdeac512b80 1 --2- 172.21.15.104:0/3672012453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca25a50 0x563e5ca0ee80 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.021+0000 7fdeac512b80 1 --2- 172.21.15.104:0/3672012453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca1cfb0 0x563e5ca1ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.021+0000 7fdeac512b80 1 --2- 172.21.15.104:0/3672012453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca23090 0x563e5ca23c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.021+0000 7fdeac512b80 1 -- 172.21.15.104:0/3672012453 >> 172.21.15.104:0/3672012453 conn(0x563e5c941080 msgr2=0x563e5ca23550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:05.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.021+0000 7fdeac512b80 1 -- 172.21.15.104:0/3672012453 shutdown_connections 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.022+0000 7fdeac512b80 1 -- 172.21.15.104:0/3672012453 wait complete. 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.022+0000 7fdeac512b80 1 -- start start 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.022+0000 7fdeac512b80 1 Processor -- start 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.022+0000 7fdeac512b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca1cfb0 0x563e5ca2ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.022+0000 7fdeac512b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 0x563e5ca2f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.022+0000 7fdeac512b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca25a50 0x563e5ca2bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.022+0000 7fdeac512b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c073de0 con 0x563e5ca23090 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.022+0000 7fdeac512b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c073710 con 0x563e5ca25a50 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.022+0000 7fdeac512b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c0708f0 con 0x563e5ca1cfb0 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.022+0000 7fdeac512b80 10 client.0 _ll_register_callbacks cb 0x563e5c8fb5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde9affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 0x563e5ca2f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde9b7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca1cfb0 0x563e5ca2ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde9affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 0x563e5ca2f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45750/0 (socket says 172.21.15.104:45750) 2024-06-09T22:22:05.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde9affd700 1 -- 172.21.15.104:0/2929172106 learned_addr learned my addr 172.21.15.104:0/2929172106 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:05.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde9bfff700 1 --2- 172.21.15.104:0/2929172106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca25a50 0x563e5ca2bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde9affd700 1 -- 172.21.15.104:0/2929172106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca1cfb0 msgr2=0x563e5ca2ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:05.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde9affd700 1 --2- 172.21.15.104:0/2929172106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca1cfb0 0x563e5ca2ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde9affd700 1 -- 172.21.15.104:0/2929172106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca25a50 msgr2=0x563e5ca2bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:05.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde9affd700 1 --2- 172.21.15.104:0/2929172106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca25a50 0x563e5ca2bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde9affd700 1 -- 172.21.15.104:0/2929172106 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e5ca2c1e0 con 0x563e5ca23090 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde9affd700 1 --2- 172.21.15.104:0/2929172106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 0x563e5ca2f210 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fde90011920 tx=0x7fde900970a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca3adb56937795d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde7affd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.023+0000 7fde7affd700 1 -- 172.21.15.104:0/2929172106 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9000be60 con 0x563e5ca23090 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fde7affd700 1 -- 172.21.15.104:0/2929172106 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fde9009a400 con 0x563e5ca23090 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fdeac512b80 1 -- 172.21.15.104:0/2929172106 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde94042450 con 0x563e5ca23090 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fde7affd700 1 -- 172.21.15.104:0/2929172106 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fde9009b6b0 con 0x563e5ca23090 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fdeac512b80 10 client.15741 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fdeac512b80 1 -- 172.21.15.104:0/2929172106 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563e5caa1c20 con 0x563e5ca23090 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fde7affd700 1 -- 172.21.15.104:0/2929172106 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde94042450 con 0x563e5ca23090 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fde7affd700 1 -- 172.21.15.104:0/2929172106 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fde90084080 con 0x563e5ca23090 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fdeac512b80 10 client.15741 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fdeac512b80 10 client.15741 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fdeac512b80 20 client.15741 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fdeac512b80 1 --2- 172.21.15.104:0/2929172106 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x563e5caa22f0 0x563e5caa4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fdeac512b80 4 client.15741 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.024+0000 7fdeac512b80 1 -- 172.21.15.104:0/2929172106 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563e5c96d2d0 con 0x563e5caa22f0 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.025+0000 7fde9b7fe700 1 --2- 172.21.15.104:0/2929172106 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x563e5caa22f0 0x563e5caa4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.025+0000 7fde7affd700 10 client.15741 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.025+0000 7fde9b7fe700 1 --2- 172.21.15.104:0/2929172106 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x563e5caa22f0 0x563e5caa4710 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=330df781e7b1c93f server_cookie=c244a4fc4847fc9a in_seq=0 out_seq=0 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.025+0000 7fde7affd700 1 -- 172.21.15.104:0/2929172106 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563e5c96d2d0 con 0x563e5caa22f0 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.025+0000 7fde7affd700 10 client.15741 handle_command_reply: tid=0 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.025+0000 7fdeac512b80 1 client.15741 shutdown 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.025+0000 7fdeac512b80 1 -- 172.21.15.104:0/2929172106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 msgr2=0x563e5ca2f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.025+0000 7fdeac512b80 1 --2- 172.21.15.104:0/2929172106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 0x563e5ca2f210 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fde90011920 tx=0x7fde900970a0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.025+0000 7fdeac512b80 1 -- 172.21.15.104:0/2929172106 shutdown_connections 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.025+0000 7fdeac512b80 1 --2- 172.21.15.104:0/2929172106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e5ca25a50 0x563e5ca2bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.025+0000 7fdeac512b80 1 --2- 172.21.15.104:0/2929172106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e5ca1cfb0 0x563e5ca2ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fdeac512b80 1 --2- 172.21.15.104:0/2929172106 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x563e5caa22f0 0x563e5caa4710 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fdeac512b80 1 --2- 172.21.15.104:0/2929172106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563e5ca23090 0x563e5ca2f210 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fdeac512b80 1 -- 172.21.15.104:0/2929172106 >> 172.21.15.104:0/2929172106 conn(0x563e5c941080 msgr2=0x563e5ca279d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:05.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fde7affd700 0 client.15741 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:22:05.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fdeac512b80 1 -- 172.21.15.104:0/2929172106 shutdown_connections 2024-06-09T22:22:05.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fdeac512b80 1 -- 172.21.15.104:0/2929172106 wait complete. 2024-06-09T22:22:05.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fdeac512b80 20 client.15741 trim_cache size 0 max 16384 2024-06-09T22:22:05.028 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:22:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fdea1c8f700 1 -- 172.21.15.104:0/2365182181 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fde840a2fc0 msgr2=0x7fde840a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fdea1c8f700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fde840a2fc0 0x7fde840a5480 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fde94004fd0 tx=0x7fde9401f000 comp rx=0 tx=0).stop 2024-06-09T22:22:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fdea1c8f700 1 -- 172.21.15.104:0/2365182181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde9c101440 msgr2=0x7fde9c19ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fdea1c8f700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde9c101440 0x7fde9c19ede0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fde9000b450 tx=0x7fde9000b760 comp rx=0 tx=0).stop 2024-06-09T22:22:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fdea1c8f700 1 -- 172.21.15.104:0/2365182181 shutdown_connections 2024-06-09T22:22:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.026+0000 7fdea1c8f700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fde840a2fc0 0x7fde840a5480 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.027+0000 7fdea1c8f700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde9c108510 0x7fde9c198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.027+0000 7fdea1c8f700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde9c100a80 0x7fde9c19e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.027+0000 7fdea1c8f700 1 --2- 172.21.15.104:0/2365182181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde9c101440 0x7fde9c19ede0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.027+0000 7fdea1c8f700 1 -- 172.21.15.104:0/2365182181 >> 172.21.15.104:0/2365182181 conn(0x7fde9c077a00 msgr2=0x7fde9c0fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:05.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.027+0000 7fdea1c8f700 1 -- 172.21.15.104:0/2365182181 shutdown_connections 2024-06-09T22:22:05.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:05.027+0000 7fdea1c8f700 1 -- 172.21.15.104:0/2365182181 wait complete. 2024-06-09T22:22:05.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:05 smithi104 ceph-mon[32389]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-06-09T22:22:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.786+0000 7f17808a5640 1 -- 172.21.15.104:0/2383342375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f177c101990 msgr2=0x7f177c101e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.786+0000 7f17808a5640 1 --2- 172.21.15.104:0/2383342375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f177c101990 0x7f177c101e10 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1770009930 tx=0x7f177002a420 comp rx=0 tx=0).stop 2024-06-09T22:22:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.786+0000 7f17808a5640 1 -- 172.21.15.104:0/2383342375 shutdown_connections 2024-06-09T22:22:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.786+0000 7f17808a5640 1 --2- 172.21.15.104:0/2383342375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c102350 0x7f177c10e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.786+0000 7f17808a5640 1 --2- 172.21.15.104:0/2383342375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f177c101990 0x7f177c101e10 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.786+0000 7f17808a5640 1 --2- 172.21.15.104:0/2383342375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f177c1079a0 0x7f177c107d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.787+0000 7f17808a5640 1 -- 172.21.15.104:0/2383342375 >> 172.21.15.104:0/2383342375 conn(0x7f177c0fd460 msgr2=0x7f177c0ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.787+0000 7f17808a5640 1 -- 172.21.15.104:0/2383342375 shutdown_connections 2024-06-09T22:22:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.787+0000 7f17808a5640 1 -- 172.21.15.104:0/2383342375 wait complete. 2024-06-09T22:22:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f17808a5640 1 Processor -- start 2024-06-09T22:22:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f17808a5640 1 -- start start 2024-06-09T22:22:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f17808a5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f177c101990 0x7f177c198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f17808a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c102350 0x7f177c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f17808a5640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f177c1079a0 0x7f177c19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f17808a5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f177c111d80 con 0x7f177c101990 2024-06-09T22:22:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f17808a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f177c111c00 con 0x7f177c102350 2024-06-09T22:22:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f17808a5640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f177c111f00 con 0x7f177c1079a0 2024-06-09T22:22:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f177a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f177c101990 0x7f177c198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f1779d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c102350 0x7f177c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f177a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f177c101990 0x7f177c198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45764/0 (socket says 172.21.15.104:45764) 2024-06-09T22:22:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f177a59c640 1 -- 172.21.15.104:0/301784625 learned_addr learned my addr 172.21.15.104:0/301784625 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f177ad9d640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f177c1079a0 0x7f177c19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f177a59c640 1 -- 172.21.15.104:0/301784625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f177c1079a0 msgr2=0x7f177c19cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f177a59c640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f177c1079a0 0x7f177c19cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f177a59c640 1 -- 172.21.15.104:0/301784625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c102350 msgr2=0x7f177c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f177a59c640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c102350 0x7f177c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.788+0000 7f177a59c640 1 -- 172.21.15.104:0/301784625 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177c19d210 con 0x7f177c101990 2024-06-09T22:22:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.789+0000 7f177a59c640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f177c101990 0x7f177c198220 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f176c00d950 tx=0x7f176c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.789+0000 7f176b7fe640 1 -- 172.21.15.104:0/301784625 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c017070 con 0x7f177c101990 2024-06-09T22:22:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.789+0000 7f17808a5640 1 -- 172.21.15.104:0/301784625 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f177c19d500 con 0x7f177c101990 2024-06-09T22:22:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.789+0000 7f176b7fe640 1 -- 172.21.15.104:0/301784625 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f176c009c30 con 0x7f177c101990 2024-06-09T22:22:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.789+0000 7f176b7fe640 1 -- 172.21.15.104:0/301784625 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c00f6b0 con 0x7f177c101990 2024-06-09T22:22:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.789+0000 7f17808a5640 1 -- 172.21.15.104:0/301784625 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f177c1a4d90 con 0x7f177c101990 2024-06-09T22:22:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.792+0000 7f17835b5740 1 Processor -- start 2024-06-09T22:22:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.792+0000 7f17835b5740 1 -- start start 2024-06-09T22:22:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.792+0000 7f17835b5740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26170 0x562a04b26550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.792+0000 7f176b7fe640 1 -- 172.21.15.104:0/301784625 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 24) ==== 138118+0+0 (secure 0 0 0) 0x7f176c0109d0 con 0x7f177c101990 2024-06-09T22:22:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.792+0000 7f17835b5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26a90 0x562a04b30e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.792+0000 7f17835b5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b33740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.792+0000 7f17835b5740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562a04a8a4f0 con 0x562a04b26170 2024-06-09T22:22:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.792+0000 7f17835b5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562a04a28870 con 0x562a04b26a90 2024-06-09T22:22:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.792+0000 7f17835b5740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562a04a864b0 con 0x562a04b31350 2024-06-09T22:22:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.792+0000 7f177ad9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26170 0x562a04b26550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.792+0000 7f177ad9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26170 0x562a04b26550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45772/0 (socket says 172.21.15.104:45772) 2024-06-09T22:22:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.792+0000 7f177ad9d640 1 -- 172.21.15.104:0/2466842551 learned_addr learned my addr 172.21.15.104:0/2466842551 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f1779d9b640 1 --2- 172.21.15.104:0/2466842551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b33740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f177a59c640 1 --2- 172.21.15.104:0/2466842551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26a90 0x562a04b30e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f177ad9d640 1 -- 172.21.15.104:0/2466842551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 msgr2=0x562a04b33740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f177ad9d640 1 --2- 172.21.15.104:0/2466842551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b33740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f177ad9d640 1 -- 172.21.15.104:0/2466842551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26a90 msgr2=0x562a04b30e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f177ad9d640 1 --2- 172.21.15.104:0/2466842551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26a90 0x562a04b30e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f177ad9d640 1 -- 172.21.15.104:0/2466842551 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a04a449b0 con 0x562a04b26170 2024-06-09T22:22:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f177ad9d640 1 --2- 172.21.15.104:0/2466842551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26170 0x562a04b26550 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1774009fd0 tx=0x7f17740118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b93f1632d7358a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f176b7fe640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f17640a2f50 0x7f17640a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f177a59c640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f17640a2f50 0x7f17640a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f1768ff9640 1 -- 172.21.15.104:0/2466842551 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1774004d20 con 0x562a04b26170 2024-06-09T22:22:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f176b7fe640 1 -- 172.21.15.104:0/301784625 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f176c0cf540 con 0x7f177c101990 2024-06-09T22:22:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f1768ff9640 1 -- 172.21.15.104:0/2466842551 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1774004050 con 0x562a04b26170 2024-06-09T22:22:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.793+0000 7f1768ff9640 1 -- 172.21.15.104:0/2466842551 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1774005240 con 0x562a04b26170 2024-06-09T22:22:05.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:05 smithi153 ceph-mon[36599]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-06-09T22:22:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.794+0000 7f17835b5740 1 -- 172.21.15.104:0/2466842551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26170 msgr2=0x562a04b26550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.794+0000 7f177a59c640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f17640a2f50 0x7f17640a5410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f176c0cc780 tx=0x7f176c086040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.794+0000 7f17835b5740 1 --2- 172.21.15.104:0/2466842551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26170 0x562a04b26550 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1774009fd0 tx=0x7f17740118d0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.794+0000 7f17835b5740 1 -- 172.21.15.104:0/2466842551 shutdown_connections 2024-06-09T22:22:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.794+0000 7f17835b5740 1 --2- 172.21.15.104:0/2466842551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b33740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.794+0000 7f17835b5740 1 --2- 172.21.15.104:0/2466842551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26a90 0x562a04b30e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.794+0000 7f17835b5740 1 --2- 172.21.15.104:0/2466842551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26170 0x562a04b26550 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.794+0000 7f17835b5740 1 -- 172.21.15.104:0/2466842551 >> 172.21.15.104:0/2466842551 conn(0x562a04b22680 msgr2=0x562a04b22a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f17835b5740 1 -- 172.21.15.104:0/2466842551 shutdown_connections 2024-06-09T22:22:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f17835b5740 1 -- 172.21.15.104:0/2466842551 wait complete. 2024-06-09T22:22:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f17835b5740 1 -- start start 2024-06-09T22:22:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f17835b5740 1 Processor -- start 2024-06-09T22:22:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f17835b5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b26170 0x562a04b2e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f17835b5740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b2e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f17835b5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b31350 0x562a04b2ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f17835b5740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562a04b33f80 con 0x562a04b26a90 2024-06-09T22:22:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f17835b5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562a04a13ae0 con 0x562a04b31350 2024-06-09T22:22:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f17835b5740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562a04b34100 con 0x562a04b26170 2024-06-09T22:22:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f17835b5740 10 client.0 _ll_register_callbacks cb 0x562a04a4c9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f1779d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b2e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f1779d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b2e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45784/0 (socket says 172.21.15.104:45784) 2024-06-09T22:22:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f1779d9b640 1 -- 172.21.15.104:0/565938033 learned_addr learned my addr 172.21.15.104:0/565938033 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.795+0000 7f177ad9d640 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b31350 0x562a04b2ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f177ad9d640 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b26170 0x562a04b2e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f1779d9b640 1 -- 172.21.15.104:0/565938033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b26170 msgr2=0x562a04b2e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f1779d9b640 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b26170 0x562a04b2e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f1779d9b640 1 -- 172.21.15.104:0/565938033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b31350 msgr2=0x562a04b2ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f1779d9b640 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b31350 0x562a04b2ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f1779d9b640 1 -- 172.21.15.104:0/565938033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a0479e210 con 0x562a04b26a90 2024-06-09T22:22:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f177ad9d640 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b31350 0x562a04b2ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f1779d9b640 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b2e7e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f177002a940 tx=0x7f17700317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7aaf9aaf5326c33 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f1761ffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f1761ffb640 1 -- 172.21.15.104:0/565938033 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770035040 con 0x562a04b26a90 2024-06-09T22:22:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f1761ffb640 1 -- 172.21.15.104:0/565938033 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1770004040 con 0x562a04b26a90 2024-06-09T22:22:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f17835b5740 1 -- 172.21.15.104:0/565938033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1774004050 con 0x562a04b26a90 2024-06-09T22:22:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f1761ffb640 1 -- 172.21.15.104:0/565938033 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f177004e050 con 0x562a04b26a90 2024-06-09T22:22:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f17835b5740 10 client.15753 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.796+0000 7f17835b5740 1 -- 172.21.15.104:0/565938033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562a0486a620 con 0x562a04b26a90 2024-06-09T22:22:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.797+0000 7f1761ffb640 1 -- 172.21.15.104:0/565938033 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1774004050 con 0x562a04b26a90 2024-06-09T22:22:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.797+0000 7f1761ffb640 1 -- 172.21.15.104:0/565938033 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f17700048c0 con 0x562a04b26a90 2024-06-09T22:22:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.797+0000 7f17835b5740 10 client.15753 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.797+0000 7f17835b5740 10 client.15753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.797+0000 7f17835b5740 20 client.15753 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.797+0000 7f17835b5740 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562a04b414d0 0x562a04b438c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.797+0000 7f17835b5740 4 client.15753 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.797+0000 7f17835b5740 1 -- 172.21.15.104:0/565938033 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a04a9df10 con 0x562a04b414d0 2024-06-09T22:22:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.797+0000 7f177ad9d640 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562a04b414d0 0x562a04b438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.798+0000 7f1761ffb640 10 client.15753 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.798+0000 7f177ad9d640 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562a04b414d0 0x562a04b438c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74529b0f653b808b server_cookie=bbab9eba91d860c0 in_seq=0 out_seq=0 2024-06-09T22:22:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.798+0000 7f1761ffb640 1 -- 172.21.15.104:0/565938033 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x562a04a9df10 con 0x562a04b414d0 2024-06-09T22:22:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.798+0000 7f1761ffb640 10 client.15753 handle_command_reply: tid=0 2024-06-09T22:22:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.798+0000 7f17835b5740 1 client.15753 shutdown 2024-06-09T22:22:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.799+0000 7f17835b5740 1 -- 172.21.15.104:0/565938033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 msgr2=0x562a04b2e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.799+0000 7f17835b5740 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b2e7e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f177002a940 tx=0x7f17700317e0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.799+0000 7f17835b5740 1 -- 172.21.15.104:0/565938033 shutdown_connections 2024-06-09T22:22:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.799+0000 7f17835b5740 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562a04b414d0 0x562a04b438c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.799+0000 7f17835b5740 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b31350 0x562a04b2ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.799+0000 7f17835b5740 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b2e7e0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.799+0000 7f17835b5740 1 --2- 172.21.15.104:0/565938033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b26170 0x562a04b2e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.799+0000 7f17835b5740 1 -- 172.21.15.104:0/565938033 >> 172.21.15.104:0/565938033 conn(0x562a04b22680 msgr2=0x562a04b28400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.799+0000 7f1761ffb640 0 client.15753 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.799+0000 7f17835b5740 1 -- 172.21.15.104:0/565938033 shutdown_connections 2024-06-09T22:22:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.799+0000 7f17835b5740 1 -- 172.21.15.104:0/565938033 wait complete. 2024-06-09T22:22:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.799+0000 7f17835b5740 20 client.15753 trim_cache size 0 max 16384 2024-06-09T22:22:05.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:05 smithi194 ceph-mon[36856]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-06-09T22:22:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.824+0000 7f17835b5740 1 Processor -- start 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.824+0000 7f17835b5740 1 -- start start 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f17835b5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26170 0x562a04b2eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f17835b5740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b49170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f17835b5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b496b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f17835b5740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f177c1accc0 con 0x562a04b26a90 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f17835b5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f177c1ae800 con 0x562a04b26170 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f17835b5740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f177c1ad390 con 0x562a04b31350 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f1779d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b49170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f1779d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b49170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45794/0 (socket says 172.21.15.104:45794) 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f1779d9b640 1 -- 172.21.15.104:0/501988086 learned_addr learned my addr 172.21.15.104:0/501988086 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f177ad9d640 1 --2- 172.21.15.104:0/501988086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f1779d9b640 1 -- 172.21.15.104:0/501988086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 msgr2=0x562a04b496b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f1779d9b640 1 --2- 172.21.15.104:0/501988086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b496b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f1779d9b640 1 -- 172.21.15.104:0/501988086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26170 msgr2=0x562a04b2eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f1779d9b640 1 --2- 172.21.15.104:0/501988086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26170 0x562a04b2eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f1779d9b640 1 -- 172.21.15.104:0/501988086 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a04b1cae0 con 0x562a04b26a90 2024-06-09T22:22:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f177ad9d640 1 --2- 172.21.15.104:0/501988086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f1779d9b640 1 --2- 172.21.15.104:0/501988086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b49170 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1770006010 tx=0x7f177002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36fab0b82d21faa server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.825+0000 7f1763fff640 1 -- 172.21.15.104:0/501988086 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770035070 con 0x562a04b26a90 2024-06-09T22:22:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.826+0000 7f1763fff640 1 -- 172.21.15.104:0/501988086 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17700035f0 con 0x562a04b26a90 2024-06-09T22:22:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.826+0000 7f1763fff640 1 -- 172.21.15.104:0/501988086 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17700049d0 con 0x562a04b26a90 2024-06-09T22:22:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.826+0000 7f17835b5740 1 -- 172.21.15.104:0/501988086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 msgr2=0x562a04b49170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.826+0000 7f17835b5740 1 --2- 172.21.15.104:0/501988086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b49170 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1770006010 tx=0x7f177002a420 comp rx=0 tx=0).stop 2024-06-09T22:22:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 -- 172.21.15.104:0/501988086 shutdown_connections 2024-06-09T22:22:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 --2- 172.21.15.104:0/501988086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 --2- 172.21.15.104:0/501988086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b49170 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 --2- 172.21.15.104:0/501988086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26170 0x562a04b2eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 -- 172.21.15.104:0/501988086 >> 172.21.15.104:0/501988086 conn(0x562a04b22680 msgr2=0x562a04b295d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 -- 172.21.15.104:0/501988086 shutdown_connections 2024-06-09T22:22:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 -- 172.21.15.104:0/501988086 wait complete. 2024-06-09T22:22:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 -- start start 2024-06-09T22:22:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 Processor -- start 2024-06-09T22:22:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26170 0x562a04b39350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b39890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b2a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f177c1ae130 con 0x562a04b26a90 2024-06-09T22:22:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f177c1ada60 con 0x562a04b26170 2024-06-09T22:22:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f177c1aac80 con 0x562a04b31350 2024-06-09T22:22:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.827+0000 7f17835b5740 10 client.0 _ll_register_callbacks cb 0x7f1770009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f1779d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b39890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f177ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26170 0x562a04b39350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f177ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26170 0x562a04b39350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:53230/0 (socket says 172.21.15.104:53230) 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f177ad9d640 1 -- 172.21.15.104:0/4088905131 learned_addr learned my addr 172.21.15.104:0/4088905131 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f1779d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b39890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45802/0 (socket says 172.21.15.104:45802) 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f177ad9d640 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b2a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f1779d9b640 1 -- 172.21.15.104:0/4088905131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 msgr2=0x562a04b2a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f1779d9b640 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b2a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f1779d9b640 1 -- 172.21.15.104:0/4088905131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26170 msgr2=0x562a04b39350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f1779d9b640 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26170 0x562a04b39350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f1779d9b640 1 -- 172.21.15.104:0/4088905131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a04b39fd0 con 0x562a04b26a90 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f177ad9d640 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26170 0x562a04b39350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f1779d9b640 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b39890 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x562a04b22280 tx=0x7f1770031a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9746f39309290835 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f1761ffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f1761ffb640 1 -- 172.21.15.104:0/4088905131 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177004b040 con 0x562a04b26a90 2024-06-09T22:22:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f1761ffb640 1 -- 172.21.15.104:0/4088905131 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1770005080 con 0x562a04b26a90 2024-06-09T22:22:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f17835b5740 1 -- 172.21.15.104:0/4088905131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17700035f0 con 0x562a04b26a90 2024-06-09T22:22:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.828+0000 7f1761ffb640 1 -- 172.21.15.104:0/4088905131 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1770035040 con 0x562a04b26a90 2024-06-09T22:22:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.829+0000 7f17835b5740 10 client.15759 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.829+0000 7f17835b5740 1 -- 172.21.15.104:0/4088905131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562a04bab6c0 con 0x562a04b26a90 2024-06-09T22:22:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.829+0000 7f1761ffb640 1 -- 172.21.15.104:0/4088905131 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1770005280 con 0x562a04b26a90 2024-06-09T22:22:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.831+0000 7f1761ffb640 1 -- 172.21.15.104:0/4088905131 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f177003c440 con 0x562a04b26a90 2024-06-09T22:22:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.831+0000 7f17835b5740 10 client.15759 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.831+0000 7f17835b5740 10 client.15759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.831+0000 7f17835b5740 20 client.15759 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.831+0000 7f17835b5740 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562a04babcc0 0x562a04bae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.831+0000 7f17835b5740 4 client.15759 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:22:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.831+0000 7f17835b5740 1 -- 172.21.15.104:0/4088905131 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f17700035f0 con 0x562a04babcc0 2024-06-09T22:22:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.831+0000 7f177ad9d640 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562a04babcc0 0x562a04bae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.832+0000 7f177ad9d640 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562a04babcc0 0x562a04bae0b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c7ac6aaaf28644d server_cookie=b78e7c0af8acc90 in_seq=0 out_seq=0 2024-06-09T22:22:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.832+0000 7f1761ffb640 10 client.15759 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.839+0000 7f1761ffb640 1 -- 172.21.15.104:0/4088905131 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7f17700035f0 con 0x562a04babcc0 2024-06-09T22:22:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.839+0000 7f1761ffb640 10 client.15759 handle_command_reply: tid=0 2024-06-09T22:22:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.839+0000 7f17835b5740 1 client.15759 shutdown 2024-06-09T22:22:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.839+0000 7f17835b5740 1 -- 172.21.15.104:0/4088905131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 msgr2=0x562a04b39890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.839+0000 7f17835b5740 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b39890 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x562a04b22280 tx=0x7f1770031a60 comp rx=0 tx=0).stop 2024-06-09T22:22:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.839+0000 7f17835b5740 1 -- 172.21.15.104:0/4088905131 shutdown_connections 2024-06-09T22:22:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.839+0000 7f17835b5740 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562a04babcc0 0x562a04bae0b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.839+0000 7f1761ffb640 0 client.15759 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.839+0000 7f17835b5740 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a04b31350 0x562a04b2a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.839+0000 7f17835b5740 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562a04b26a90 0x562a04b39890 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.839+0000 7f17835b5740 1 --2- 172.21.15.104:0/4088905131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a04b26170 0x562a04b39350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.839+0000 7f17835b5740 1 -- 172.21.15.104:0/4088905131 >> 172.21.15.104:0/4088905131 conn(0x562a04b22680 msgr2=0x562a04b20f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.840+0000 7f17835b5740 1 -- 172.21.15.104:0/4088905131 shutdown_connections 2024-06-09T22:22:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.840+0000 7f17835b5740 1 -- 172.21.15.104:0/4088905131 wait complete. 2024-06-09T22:22:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.840+0000 7f17835b5740 20 client.15759 trim_cache size 0 max 16384 2024-06-09T22:22:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:22:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:22:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:22:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.840+0000 7f17808a5640 1 -- 172.21.15.104:0/301784625 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f17640a2f50 msgr2=0x7f17640a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.840+0000 7f17808a5640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f17640a2f50 0x7f17640a5410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f176c0cc780 tx=0x7f176c086040 comp rx=0 tx=0).stop 2024-06-09T22:22:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.841+0000 7f17808a5640 1 -- 172.21.15.104:0/301784625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f177c101990 msgr2=0x7f177c198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.841+0000 7f17808a5640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f177c101990 0x7f177c198220 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f176c00d950 tx=0x7f176c00de10 comp rx=0 tx=0).stop 2024-06-09T22:22:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.841+0000 7f17808a5640 1 -- 172.21.15.104:0/301784625 shutdown_connections 2024-06-09T22:22:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.841+0000 7f17808a5640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f17640a2f50 0x7f17640a5410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.841+0000 7f17808a5640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f177c1079a0 0x7f177c19cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.841+0000 7f17808a5640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c102350 0x7f177c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.841+0000 7f17808a5640 1 --2- 172.21.15.104:0/301784625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f177c101990 0x7f177c198220 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.841+0000 7f17808a5640 1 -- 172.21.15.104:0/301784625 >> 172.21.15.104:0/301784625 conn(0x7f177c0fd460 msgr2=0x7f177c0fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.841+0000 7f17808a5640 1 -- 172.21.15.104:0/301784625 shutdown_connections 2024-06-09T22:22:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:05.841+0000 7f17808a5640 1 -- 172.21.15.104:0/301784625 wait complete. 2024-06-09T22:22:05.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-09T22:22:06.182 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:07.677 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:07 smithi104 ceph-mon[32389]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-06-09T22:22:07.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:07 smithi194 ceph-mon[36856]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-06-09T22:22:07.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:07 smithi153 ceph-mon[36599]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-06-09T22:22:08.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.093+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/2687118476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d806aa00 msgr2=0x7fb1d806ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:08.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.093+0000 7fb1dfc3b700 1 --2- 172.21.15.104:0/2687118476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d806aa00 0x7fb1d806ae80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8009990 tx=0x7fb1c8009ca0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.093+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/2687118476 shutdown_connections 2024-06-09T22:22:08.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.093+0000 7fb1dfc3b700 1 --2- 172.21.15.104:0/2687118476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1d806b3c0 0x7fb1d810f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.093+0000 7fb1dfc3b700 1 --2- 172.21.15.104:0/2687118476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1d81084f0 0x7fb1d81088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.093+0000 7fb1dfc3b700 1 --2- 172.21.15.104:0/2687118476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d806aa00 0x7fb1d806ae80 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.093+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/2687118476 >> 172.21.15.104:0/2687118476 conn(0x7fb1d8078300 msgr2=0x7fb1d8078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:08.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.093+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/2687118476 shutdown_connections 2024-06-09T22:22:08.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.093+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/2687118476 wait complete. 2024-06-09T22:22:08.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1dfc3b700 1 Processor -- start 2024-06-09T22:22:08.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1dfc3b700 1 -- start start 2024-06-09T22:22:08.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1dfc3b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d806aa00 0x7fb1d8101450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1dfc3b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1d806b3c0 0x7fb1d8101990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1dfc3b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1d81084f0 0x7fb1d8101ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1dfc3b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d8112770 con 0x7fb1d806aa00 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1dfc3b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d81125f0 con 0x7fb1d81084f0 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1dfc3b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d81128f0 con 0x7fb1d806b3c0 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1dd9d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d806aa00 0x7fb1d8101450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1dd9d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d806aa00 0x7fb1d8101450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54944/0 (socket says 172.21.15.104:54944) 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1dd9d7700 1 -- 172.21.15.104:0/1170312896 learned_addr learned my addr 172.21.15.104:0/1170312896 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1de1d8700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1d81084f0 0x7fb1d8101ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.094+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1d806b3c0 0x7fb1d8101990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.095+0000 7fb1de1d8700 1 -- 172.21.15.104:0/1170312896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1d806b3c0 msgr2=0x7fb1d8101990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.095+0000 7fb1de1d8700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1d806b3c0 0x7fb1d8101990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.095+0000 7fb1de1d8700 1 -- 172.21.15.104:0/1170312896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d806aa00 msgr2=0x7fb1d8101450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:08.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.095+0000 7fb1de1d8700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d806aa00 0x7fb1d8101450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.095+0000 7fb1de1d8700 1 -- 172.21.15.104:0/1170312896 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d81a71b0 con 0x7fb1d81084f0 2024-06-09T22:22:08.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.095+0000 7fb1de1d8700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1d81084f0 0x7fb1d8101ed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0007f10 tx=0x7fb1d000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:08.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.095+0000 7fb1ceffd700 1 -- 172.21.15.104:0/1170312896 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d000ddd0 con 0x7fb1d81084f0 2024-06-09T22:22:08.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.095+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/1170312896 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d81a74a0 con 0x7fb1d81084f0 2024-06-09T22:22:08.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.095+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/1170312896 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1d81a7a00 con 0x7fb1d81084f0 2024-06-09T22:22:08.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.095+0000 7fb1ceffd700 1 -- 172.21.15.104:0/1170312896 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d0009e20 con 0x7fb1d81084f0 2024-06-09T22:22:08.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.095+0000 7fb1ceffd700 1 -- 172.21.15.104:0/1170312896 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0015660 con 0x7fb1d81084f0 2024-06-09T22:22:08.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.099+0000 7fb1ceffd700 1 -- 172.21.15.104:0/1170312896 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 24) ==== 138118+0+0 (secure 0 0 0) 0x7fb1d0004ad0 con 0x7fb1d81084f0 2024-06-09T22:22:08.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.099+0000 7fb1ea4beb80 1 Processor -- start 2024-06-09T22:22:08.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1ea4beb80 1 -- start start 2024-06-09T22:22:08.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1ea4beb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b2f60 0x55bce26b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1ea4beb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 0x55bce26ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1ea4beb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26ad7a0 0x55bce26adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1ea4beb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55bce259ff60 con 0x55bce26b3880 2024-06-09T22:22:08.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1ea4beb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bce25fe850 con 0x55bce26ad7a0 2024-06-09T22:22:08.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1ea4beb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bce2566460 con 0x55bce26b2f60 2024-06-09T22:22:08.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1ceffd700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1c40a3040 0x7fb1c40a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1dd9d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 0x55bce26ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1dd1d6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26ad7a0 0x55bce26adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1dd1d6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26ad7a0 0x55bce26adbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:56580/0 (socket says 172.21.15.104:56580) 2024-06-09T22:22:08.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1dd9d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 0x55bce26ad260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54946/0 (socket says 172.21.15.104:54946) 2024-06-09T22:22:08.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/3324048569 learned_addr learned my addr 172.21.15.104:0/3324048569 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:08.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/3324048569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b2f60 0x55bce26b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/3324048569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b2f60 msgr2=0x55bce26b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:08.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/3324048569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b2f60 0x55bce26b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1de1d8700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1c40a3040 0x7fb1c40a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.100+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/3324048569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 msgr2=0x55bce26ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:08.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.101+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/3324048569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 0x55bce26ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.101+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/3324048569 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bce254d8d0 con 0x55bce26ad7a0 2024-06-09T22:22:08.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.101+0000 7fb1dd9d7700 1 --2- 172.21.15.104:0/3324048569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 0x55bce26ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:08.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.101+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/3324048569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26ad7a0 0x55bce26adbc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8009990 tx=0x7fb1c801a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f2915da68117f0d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:08.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.101+0000 7fb1c3fff700 1 -- 172.21.15.104:0/3324048569 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c802cb80 con 0x55bce26ad7a0 2024-06-09T22:22:08.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.101+0000 7fb1c3fff700 1 -- 172.21.15.104:0/3324048569 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1c8003e40 con 0x55bce26ad7a0 2024-06-09T22:22:08.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.101+0000 7fb1c3fff700 1 -- 172.21.15.104:0/3324048569 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c800b7d0 con 0x55bce26ad7a0 2024-06-09T22:22:08.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.101+0000 7fb1ceffd700 1 -- 172.21.15.104:0/1170312896 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb1d00cf340 con 0x7fb1d81084f0 2024-06-09T22:22:08.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.102+0000 7fb1de1d8700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1c40a3040 0x7fb1c40a5500 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00cc540 tx=0x7fb1d00d2a60 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:08.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.102+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3324048569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26ad7a0 msgr2=0x55bce26adbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:08.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.102+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3324048569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26ad7a0 0x55bce26adbc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8009990 tx=0x7fb1c801a040 comp rx=0 tx=0).stop 2024-06-09T22:22:08.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.102+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3324048569 shutdown_connections 2024-06-09T22:22:08.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.102+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3324048569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26ad7a0 0x55bce26adbc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.102+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3324048569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b2f60 0x55bce26b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.102+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3324048569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 0x55bce26ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.102+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3324048569 >> 172.21.15.104:0/3324048569 conn(0x55bce25da080 msgr2=0x55bce26a8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:08.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.102+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3324048569 shutdown_connections 2024-06-09T22:22:08.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.102+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3324048569 wait complete. 2024-06-09T22:22:08.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.103+0000 7fb1ea4beb80 1 -- start start 2024-06-09T22:22:08.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.103+0000 7fb1ea4beb80 1 Processor -- start 2024-06-09T22:22:08.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.103+0000 7fb1ea4beb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 0x55bce26b9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.103+0000 7fb1ea4beb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b2f60 0x55bce26b6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1ea4beb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b3880 0x55bce26b6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1ea4beb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55bce259efc0 con 0x55bce26ad7a0 2024-06-09T22:22:08.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1ea4beb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bce25fa810 con 0x55bce26b2f60 2024-06-09T22:22:08.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1ea4beb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bce26ba200 con 0x55bce26b3880 2024-06-09T22:22:08.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1dd1d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 0x55bce26b9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1dd1d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 0x55bce26b9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54952/0 (socket says 172.21.15.104:54952) 2024-06-09T22:22:08.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/2004463253 learned_addr learned my addr 172.21.15.104:0/2004463253 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:08.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1ea4beb80 10 client.0 _ll_register_callbacks cb 0x55bce25945b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:08.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1dd9d7700 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b2f60 0x55bce26b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b3880 0x55bce26b6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/2004463253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b3880 msgr2=0x55bce26b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b3880 0x55bce26b6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/2004463253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b2f60 msgr2=0x55bce26b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b2f60 0x55bce26b6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/2004463253 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bce25c4750 con 0x55bce26ad7a0 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 0x55bce26b9610 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8009350 tx=0x7fb1c8005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39be12a5f47d1483 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.104+0000 7fb1c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.105+0000 7fb1c17fa700 1 -- 172.21.15.104:0/2004463253 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8005560 con 0x55bce26ad7a0 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.105+0000 7fb1c17fa700 1 -- 172.21.15.104:0/2004463253 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb1c8021070 con 0x55bce26ad7a0 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.105+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/2004463253 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bce260bd50 con 0x55bce26ad7a0 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.105+0000 7fb1c17fa700 1 -- 172.21.15.104:0/2004463253 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb1c8005700 con 0x55bce26ad7a0 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.105+0000 7fb1ea4beb80 10 client.15771 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.105+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/2004463253 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bce26c49f0 con 0x55bce26ad7a0 2024-06-09T22:22:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.106+0000 7fb1c17fa700 1 -- 172.21.15.104:0/2004463253 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bce260bd50 con 0x55bce26ad7a0 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.106+0000 7fb1c17fa700 1 -- 172.21.15.104:0/2004463253 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb1c8049470 con 0x55bce26ad7a0 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.106+0000 7fb1ea4beb80 10 client.15771 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.106+0000 7fb1ea4beb80 10 client.15771 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.106+0000 7fb1ea4beb80 20 client.15771 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.106+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55bce26c6ec0 0x55bce26c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.106+0000 7fb1ea4beb80 4 client.15771 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.106+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/2004463253 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bce26062d0 con 0x55bce26c6ec0 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.106+0000 7fb1dd9d7700 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55bce26c6ec0 0x55bce26c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.107+0000 7fb1c17fa700 10 client.15771 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.107+0000 7fb1dd9d7700 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55bce26c6ec0 0x55bce26c92b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8794ad4345a5b4fb server_cookie=24a599c7558ff169 in_seq=0 out_seq=0 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.107+0000 7fb1c17fa700 1 -- 172.21.15.104:0/2004463253 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55bce26062d0 con 0x55bce26c6ec0 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.107+0000 7fb1c17fa700 10 client.15771 handle_command_reply: tid=0 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.107+0000 7fb1ea4beb80 1 client.15771 shutdown 2024-06-09T22:22:08.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.108+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/2004463253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 msgr2=0x55bce26b9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:08.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.108+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 0x55bce26b9610 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8009350 tx=0x7fb1c8005050 comp rx=0 tx=0).stop 2024-06-09T22:22:08.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.108+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/2004463253 shutdown_connections 2024-06-09T22:22:08.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.108+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55bce26c6ec0 0x55bce26c92b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.108+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 0x55bce26b9610 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.108+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b3880 0x55bce26b6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.108+0000 7fb1c17fa700 0 client.15771 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:22:08.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.108+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/2004463253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b2f60 0x55bce26b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.108+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/2004463253 >> 172.21.15.104:0/2004463253 conn(0x55bce25da080 msgr2=0x55bce26a7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:08.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.108+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/2004463253 shutdown_connections 2024-06-09T22:22:08.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.108+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/2004463253 wait complete. 2024-06-09T22:22:08.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.108+0000 7fb1ea4beb80 20 client.15771 trim_cache size 0 max 16384 2024-06-09T22:22:08.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1ea4beb80 1 Processor -- start 2024-06-09T22:22:08.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1ea4beb80 1 -- start start 2024-06-09T22:22:08.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1ea4beb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 0x55bce26ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1ea4beb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b2f60 0x55bce26b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1ea4beb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b3880 0x55bce26a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1ea4beb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc00ecc0 con 0x55bce26ad7a0 2024-06-09T22:22:08.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1ea4beb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc010800 con 0x55bce26b3880 2024-06-09T22:22:08.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1ea4beb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc00f390 con 0x55bce26b2f60 2024-06-09T22:22:08.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1dd1d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 0x55bce26ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1dd1d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 0x55bce26ad260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54954/0 (socket says 172.21.15.104:54954) 2024-06-09T22:22:08.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/3387735929 learned_addr learned my addr 172.21.15.104:0/3387735929 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:08.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/3387735929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b3880 0x55bce26a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1dd9d7700 1 --2- 172.21.15.104:0/3387735929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b2f60 0x55bce26b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/3387735929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b2f60 msgr2=0x55bce26b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:08.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.137+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/3387735929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b2f60 0x55bce26b0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.138+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/3387735929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b3880 msgr2=0x55bce26a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:08.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.138+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/3387735929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b3880 0x55bce26a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.138+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/3387735929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bce26a4810 con 0x55bce26ad7a0 2024-06-09T22:22:08.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.138+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/3387735929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 0x55bce26ad260 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80098e0 tx=0x7fb1c8009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d467c2c88247b5 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:08.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.138+0000 7fb1c37fe700 1 -- 172.21.15.104:0/3387735929 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8057080 con 0x55bce26ad7a0 2024-06-09T22:22:08.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.138+0000 7fb1c37fe700 1 -- 172.21.15.104:0/3387735929 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1c800b5c0 con 0x55bce26ad7a0 2024-06-09T22:22:08.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.138+0000 7fb1c37fe700 1 -- 172.21.15.104:0/3387735929 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8049470 con 0x55bce26ad7a0 2024-06-09T22:22:08.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.139+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3387735929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 msgr2=0x55bce26ad260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:08.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.139+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3387735929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 0x55bce26ad260 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80098e0 tx=0x7fb1c8009810 comp rx=0 tx=0).stop 2024-06-09T22:22:08.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.139+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3387735929 shutdown_connections 2024-06-09T22:22:08.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.139+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3387735929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26ad7a0 0x55bce26ad260 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.139+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3387735929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b3880 0x55bce26a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.139+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3387735929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26b2f60 0x55bce26b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.139+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3387735929 >> 172.21.15.104:0/3387735929 conn(0x55bce25da080 msgr2=0x55bce26b07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:08.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.139+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3387735929 shutdown_connections 2024-06-09T22:22:08.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.139+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3387735929 wait complete. 2024-06-09T22:22:08.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.139+0000 7fb1ea4beb80 1 -- start start 2024-06-09T22:22:08.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1ea4beb80 1 Processor -- start 2024-06-09T22:22:08.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1ea4beb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26ad7a0 0x55bce26cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1ea4beb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b2f60 0x55bce26c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1ea4beb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 0x55bce26c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1ea4beb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc010130 con 0x55bce26b3880 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1ea4beb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc00fa60 con 0x55bce26b2f60 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1ea4beb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc00cc40 con 0x55bce26ad7a0 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1ea4beb80 10 client.0 _ll_register_callbacks cb 0x55bce25945b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1dd1d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 0x55bce26c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1dd1d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 0x55bce26c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54962/0 (socket says 172.21.15.104:54962) 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1dd1d6700 1 -- 172.21.15.104:0/3538207550 learned_addr learned my addr 172.21.15.104:0/3538207550 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1dd9d7700 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b2f60 0x55bce26c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.140+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26ad7a0 0x55bce26cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1dd9d7700 1 -- 172.21.15.104:0/3538207550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26ad7a0 msgr2=0x55bce26cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1dd9d7700 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26ad7a0 0x55bce26cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1dd9d7700 1 -- 172.21.15.104:0/3538207550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 msgr2=0x55bce26c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1dd9d7700 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 0x55bce26c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1dd9d7700 1 -- 172.21.15.104:0/3538207550 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bce26b1420 con 0x55bce26b2f60 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1dd9d7700 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b2f60 0x55bce26c92b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x55bce26ba230 tx=0x7fb1d4004640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b0c30adae7090b3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1c17fa700 1 -- 172.21.15.104:0/3538207550 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d401c310 con 0x55bce26b2f60 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1c17fa700 1 -- 172.21.15.104:0/3538207550 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb1d401c4b0 con 0x55bce26b2f60 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3538207550 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1c800b5c0 con 0x55bce26b2f60 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1c17fa700 1 -- 172.21.15.104:0/3538207550 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb1d402d050 con 0x55bce26b2f60 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1ea4beb80 10 client.25253 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.141+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3538207550 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bce27306a0 con 0x55bce26b2f60 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.142+0000 7fb1c17fa700 1 -- 172.21.15.104:0/3538207550 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d4020030 con 0x55bce26b2f60 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.142+0000 7fb1c17fa700 1 -- 172.21.15.104:0/3538207550 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb1d4025940 con 0x55bce26b2f60 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.142+0000 7fb1ea4beb80 10 client.25253 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.142+0000 7fb1ea4beb80 10 client.25253 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:22:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.142+0000 7fb1ea4beb80 20 client.25253 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:08.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.142+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55bce26c6ec0 0x55bce2732e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:08.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.142+0000 7fb1ea4beb80 4 client.25253 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:22:08.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.142+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3538207550 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb1c800b5c0 con 0x55bce26c6ec0 2024-06-09T22:22:08.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.142+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55bce26c6ec0 0x55bce2732e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:08.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1c17fa700 10 client.25253 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:22:08.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1dd1d6700 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55bce26c6ec0 0x55bce2732e40 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=503460917c8fb76d server_cookie=5d5a8419cdc4e3df in_seq=0 out_seq=0 2024-06-09T22:22:08.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1c17fa700 1 -- 172.21.15.104:0/3538207550 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb1c800b5c0 con 0x55bce26c6ec0 2024-06-09T22:22:08.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1c17fa700 10 client.25253 handle_command_reply: tid=0 2024-06-09T22:22:08.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1ea4beb80 1 client.25253 shutdown 2024-06-09T22:22:08.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3538207550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b2f60 msgr2=0x55bce26c92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:08.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b2f60 0x55bce26c92b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x55bce26ba230 tx=0x7fb1d4004640 comp rx=0 tx=0).stop 2024-06-09T22:22:08.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3538207550 shutdown_connections 2024-06-09T22:22:08.146 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:22:08.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55bce26c6ec0 0x55bce2732e40 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bce26b3880 0x55bce26c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bce26ad7a0 0x55bce26cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1c17fa700 0 client.25253 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:22:08.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1ea4beb80 1 --2- 172.21.15.104:0/3538207550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bce26b2f60 0x55bce26c92b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.143+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3538207550 >> 172.21.15.104:0/3538207550 conn(0x55bce25da080 msgr2=0x55bce26b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:08.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3538207550 shutdown_connections 2024-06-09T22:22:08.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1ea4beb80 1 -- 172.21.15.104:0/3538207550 wait complete. 2024-06-09T22:22:08.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1ea4beb80 20 client.25253 trim_cache size 0 max 16384 2024-06-09T22:22:08.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/1170312896 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1c40a3040 msgr2=0x7fb1c40a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:08.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1dfc3b700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1c40a3040 0x7fb1c40a5500 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00cc540 tx=0x7fb1d00d2a60 comp rx=0 tx=0).stop 2024-06-09T22:22:08.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/1170312896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1d81084f0 msgr2=0x7fb1d8101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:08.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1dfc3b700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1d81084f0 0x7fb1d8101ed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0007f10 tx=0x7fb1d000d3b0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/1170312896 shutdown_connections 2024-06-09T22:22:08.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1dfc3b700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1c40a3040 0x7fb1c40a5500 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1dfc3b700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d806aa00 0x7fb1d8101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1dfc3b700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1d81084f0 0x7fb1d8101ed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1dfc3b700 1 --2- 172.21.15.104:0/1170312896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1d806b3c0 0x7fb1d8101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:08.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/1170312896 >> 172.21.15.104:0/1170312896 conn(0x7fb1d8078300 msgr2=0x7fb1d80ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:08.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.144+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/1170312896 shutdown_connections 2024-06-09T22:22:08.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:08.145+0000 7fb1dfc3b700 1 -- 172.21.15.104:0/1170312896 wait complete. 2024-06-09T22:22:09.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-09T22:22:09.330 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:09.728 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:09 smithi104 ceph-mon[32389]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-06-09T22:22:09.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:09 smithi194 ceph-mon[36856]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-06-09T22:22:09.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:09 smithi153 ceph-mon[36599]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-06-09T22:22:11.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.297+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/2157175334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec101390 msgr2=0x7fc0ec101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:11.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.297+0000 7fc0f0e1e700 1 --2- 172.21.15.104:0/2157175334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec101390 0x7fc0ec101810 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009990 tx=0x7fc0e0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.297+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/2157175334 shutdown_connections 2024-06-09T22:22:11.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.297+0000 7fc0f0e1e700 1 --2- 172.21.15.104:0/2157175334 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0ec101d50 0x7fc0ec10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.297+0000 7fc0f0e1e700 1 --2- 172.21.15.104:0/2157175334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec107400 0x7fc0ec1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.297+0000 7fc0f0e1e700 1 --2- 172.21.15.104:0/2157175334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec101390 0x7fc0ec101810 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.297+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/2157175334 >> 172.21.15.104:0/2157175334 conn(0x7fc0ec078340 msgr2=0x7fc0ec078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:11.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.297+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/2157175334 shutdown_connections 2024-06-09T22:22:11.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.298+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/2157175334 wait complete. 2024-06-09T22:22:11.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.298+0000 7fc0f0e1e700 1 Processor -- start 2024-06-09T22:22:11.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0f0e1e700 1 -- start start 2024-06-09T22:22:11.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0f0e1e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0ec101390 0x7fc0ec19e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0f0e1e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec101d50 0x7fc0ec19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0f0e1e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec107400 0x7fc0ec198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0ea59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0ec101390 0x7fc0ec19e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0f0e1e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1116c0 con 0x7fc0ec107400 2024-06-09T22:22:11.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0ea59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0ec101390 0x7fc0ec19e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:49556/0 (socket says 172.21.15.104:49556) 2024-06-09T22:22:11.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0f0e1e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec111540 con 0x7fc0ec101d50 2024-06-09T22:22:11.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/3746535371 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec111840 con 0x7fc0ec101390 2024-06-09T22:22:11.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0ea59c700 1 -- 172.21.15.104:0/3746535371 learned_addr learned my addr 172.21.15.104:0/3746535371 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:11.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0ead9d700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec107400 0x7fc0ec198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0e9d9b700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec101d50 0x7fc0ec19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0ea59c700 1 -- 172.21.15.104:0/3746535371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec101d50 msgr2=0x7fc0ec19ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:11.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0ea59c700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec101d50 0x7fc0ec19ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0ea59c700 1 -- 172.21.15.104:0/3746535371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec107400 msgr2=0x7fc0ec198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:11.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.299+0000 7fc0ea59c700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec107400 0x7fc0ec198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.300+0000 7fc0ea59c700 1 -- 172.21.15.104:0/3746535371 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ec1991f0 con 0x7fc0ec101390 2024-06-09T22:22:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.300+0000 7fc0ea59c700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0ec101390 0x7fc0ec19e8d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00d950 tx=0x7fc0dc00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.300+0000 7fc0db7fe700 1 -- 172.21.15.104:0/3746535371 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc016070 con 0x7fc0ec101390 2024-06-09T22:22:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.300+0000 7fc0db7fe700 1 -- 172.21.15.104:0/3746535371 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0dc00fd70 con 0x7fc0ec101390 2024-06-09T22:22:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.300+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/3746535371 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0ec1994e0 con 0x7fc0ec101390 2024-06-09T22:22:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.301+0000 7fc0db7fe700 1 -- 172.21.15.104:0/3746535371 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc00e610 con 0x7fc0ec101390 2024-06-09T22:22:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.301+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/3746535371 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0ec108210 con 0x7fc0ec101390 2024-06-09T22:22:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.303+0000 7fc0db7fe700 1 -- 172.21.15.104:0/3746535371 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 24) ==== 138118+0+0 (secure 0 0 0) 0x7fc0dc019070 con 0x7fc0ec101390 2024-06-09T22:22:11.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.304+0000 7fc0db7fe700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0d40a2f40 0x7fc0d40a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.305+0000 7fc0db7fe700 1 -- 172.21.15.104:0/3746535371 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fc0dc0cf870 con 0x7fc0ec101390 2024-06-09T22:22:11.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.305+0000 7fc0e9d9b700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0d40a2f40 0x7fc0d40a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.305+0000 7fc0e9d9b700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0d40a2f40 0x7fc0d40a5400 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0fe520 tx=0x7fc0e0005fd0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:11.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.305+0000 7fc0fb6a1b80 1 Processor -- start 2024-06-09T22:22:11.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.306+0000 7fc0fb6a1b80 1 -- start start 2024-06-09T22:22:11.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.306+0000 7fc0fb6a1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c0f60 0x562bf15c1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.306+0000 7fc0fb6a1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c1880 0x562bf15bb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.306+0000 7fc0fb6a1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15bb7a0 0x562bf15bbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.306+0000 7fc0fb6a1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562bf14adf60 con 0x562bf15c0f60 2024-06-09T22:22:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.306+0000 7fc0fb6a1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562bf150c850 con 0x562bf15bb7a0 2024-06-09T22:22:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.306+0000 7fc0fb6a1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562bf1474460 con 0x562bf15c1880 2024-06-09T22:22:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.306+0000 7fc0ead9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15bb7a0 0x562bf15bbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.306+0000 7fc0ead9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15bb7a0 0x562bf15bbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:56614/0 (socket says 172.21.15.104:56614) 2024-06-09T22:22:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.306+0000 7fc0ead9d700 1 -- 172.21.15.104:0/2332140998 learned_addr learned my addr 172.21.15.104:0/2332140998 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.306+0000 7fc0e9d9b700 1 --2- 172.21.15.104:0/2332140998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c1880 0x562bf15bb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.307+0000 7fc0ead9d700 1 -- 172.21.15.104:0/2332140998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c1880 msgr2=0x562bf15bb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.307+0000 7fc0ead9d700 1 --2- 172.21.15.104:0/2332140998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c1880 0x562bf15bb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.307+0000 7fc0ead9d700 1 -- 172.21.15.104:0/2332140998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c0f60 msgr2=0x562bf15c1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.307+0000 7fc0ead9d700 1 --2- 172.21.15.104:0/2332140998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c0f60 0x562bf15c1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.307+0000 7fc0ead9d700 1 -- 172.21.15.104:0/2332140998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bf145b8d0 con 0x562bf15bb7a0 2024-06-09T22:22:11.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.307+0000 7fc0ead9d700 1 --2- 172.21.15.104:0/2332140998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15bb7a0 0x562bf15bbbc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40098f0 tx=0x7fc0e4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b2307e3a52c0c86 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.307+0000 7fc0d8ff9700 1 -- 172.21.15.104:0/2332140998 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4004db0 con 0x562bf15bb7a0 2024-06-09T22:22:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.307+0000 7fc0d8ff9700 1 -- 172.21.15.104:0/2332140998 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e4007500 con 0x562bf15bb7a0 2024-06-09T22:22:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.307+0000 7fc0d8ff9700 1 -- 172.21.15.104:0/2332140998 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4005260 con 0x562bf15bb7a0 2024-06-09T22:22:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.308+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2332140998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15bb7a0 msgr2=0x562bf15bbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.308+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2332140998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15bb7a0 0x562bf15bbbc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40098f0 tx=0x7fc0e4009c00 comp rx=0 tx=0).stop 2024-06-09T22:22:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.308+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2332140998 shutdown_connections 2024-06-09T22:22:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.308+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2332140998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c0f60 0x562bf15c1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.308+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2332140998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15bb7a0 0x562bf15bbbc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.308+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2332140998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c1880 0x562bf15bb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.308+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2332140998 >> 172.21.15.104:0/2332140998 conn(0x562bf14e8080 msgr2=0x562bf15b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.308+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2332140998 shutdown_connections 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.309+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2332140998 wait complete. 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.309+0000 7fc0fb6a1b80 1 -- start start 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.309+0000 7fc0fb6a1b80 1 Processor -- start 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0fb6a1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15bb7a0 0x562bf15c7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0fb6a1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c0f60 0x562bf15c4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0fb6a1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15c1880 0x562bf15c4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0fb6a1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562bf14acfc0 con 0x562bf15c0f60 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0fb6a1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562bf1508810 con 0x562bf15c1880 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0fb6a1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562bf15c8260 con 0x562bf15bb7a0 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0fb6a1b80 10 client.0 _ll_register_callbacks cb 0x562bf14a25b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0e9d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c0f60 0x562bf15c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0e9d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c0f60 0x562bf15c4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55002/0 (socket says 172.21.15.104:55002) 2024-06-09T22:22:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0e9d9b700 1 -- 172.21.15.104:0/3526131900 learned_addr learned my addr 172.21.15.104:0/3526131900 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0ea59c700 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15bb7a0 0x562bf15c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0ead9d700 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15c1880 0x562bf15c4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0e9d9b700 1 -- 172.21.15.104:0/3526131900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15bb7a0 msgr2=0x562bf15c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0e9d9b700 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15bb7a0 0x562bf15c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0e9d9b700 1 -- 172.21.15.104:0/3526131900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15c1880 msgr2=0x562bf15c4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0e9d9b700 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15c1880 0x562bf15c4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.310+0000 7fc0e9d9b700 1 -- 172.21.15.104:0/3526131900 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bf14d2750 con 0x562bf15c0f60 2024-06-09T22:22:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.311+0000 7fc0e9d9b700 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c0f60 0x562bf15c4950 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0015000 tx=0x7fc0e0009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d5b59febb81ac5 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:11.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.311+0000 7fc0d1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:11.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.311+0000 7fc0d1ffb700 1 -- 172.21.15.104:0/3526131900 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e000fd60 con 0x562bf15c0f60 2024-06-09T22:22:11.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.311+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/3526131900 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0e4007500 con 0x562bf15c0f60 2024-06-09T22:22:11.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.311+0000 7fc0d1ffb700 1 -- 172.21.15.104:0/3526131900 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fc0e0035420 con 0x562bf15c0f60 2024-06-09T22:22:11.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.311+0000 7fc0d1ffb700 1 -- 172.21.15.104:0/3526131900 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc0e004f050 con 0x562bf15c0f60 2024-06-09T22:22:11.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.311+0000 7fc0fb6a1b80 10 client.15786 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:11.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.311+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/3526131900 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562bf15d2a90 con 0x562bf15c0f60 2024-06-09T22:22:11.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.312+0000 7fc0d1ffb700 1 -- 172.21.15.104:0/3526131900 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e4007500 con 0x562bf15c0f60 2024-06-09T22:22:11.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.312+0000 7fc0d1ffb700 1 -- 172.21.15.104:0/3526131900 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc0e0041610 con 0x562bf15c0f60 2024-06-09T22:22:11.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.312+0000 7fc0fb6a1b80 10 client.15786 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:11.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.312+0000 7fc0fb6a1b80 10 client.15786 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:22:11.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.312+0000 7fc0fb6a1b80 20 client.15786 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:11.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.312+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x562bf15d54b0 0x562bf15d78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.312+0000 7fc0fb6a1b80 4 client.15786 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:11.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.312+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/3526131900 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bf1519d50 con 0x562bf15d54b0 2024-06-09T22:22:11.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.312+0000 7fc0ea59c700 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x562bf15d54b0 0x562bf15d78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.313+0000 7fc0d1ffb700 10 client.15786 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:22:11.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.313+0000 7fc0ea59c700 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x562bf15d54b0 0x562bf15d78f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8033de6499ce0599 server_cookie=8af2f7e9ff1ac1b6 in_seq=0 out_seq=0 2024-06-09T22:22:11.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0d1ffb700 1 -- 172.21.15.104:0/3526131900 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x562bf1519d50 con 0x562bf15d54b0 2024-06-09T22:22:11.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0d1ffb700 10 client.15786 handle_command_reply: tid=0 2024-06-09T22:22:11.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0fb6a1b80 1 client.15786 shutdown 2024-06-09T22:22:11.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/3526131900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c0f60 msgr2=0x562bf15c4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:11.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c0f60 0x562bf15c4950 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0015000 tx=0x7fc0e0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/3526131900 shutdown_connections 2024-06-09T22:22:11.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x562bf15d54b0 0x562bf15d78f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15c1880 0x562bf15c4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0d1ffb700 0 client.15786 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:22:11.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15bb7a0 0x562bf15c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/3526131900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c0f60 0x562bf15c4950 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/3526131900 >> 172.21.15.104:0/3526131900 conn(0x562bf14e8080 msgr2=0x562bf15b5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:11.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/3526131900 shutdown_connections 2024-06-09T22:22:11.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/3526131900 wait complete. 2024-06-09T22:22:11.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.314+0000 7fc0fb6a1b80 20 client.15786 trim_cache size 0 max 16384 2024-06-09T22:22:11.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.343+0000 7fc0fb6a1b80 1 Processor -- start 2024-06-09T22:22:11.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.343+0000 7fc0fb6a1b80 1 -- start start 2024-06-09T22:22:11.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0fb6a1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15bb7a0 0x562bf15bb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0fb6a1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15c0f60 0x562bf15c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0fb6a1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c1880 0x562bf15beee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0fb6a1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1ac300 con 0x562bf15bb7a0 2024-06-09T22:22:11.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0fb6a1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1ade40 con 0x562bf15c0f60 2024-06-09T22:22:11.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0fb6a1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1ac9d0 con 0x562bf15c1880 2024-06-09T22:22:11.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0ea59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15bb7a0 0x562bf15bb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0ea59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15bb7a0 0x562bf15bb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55006/0 (socket says 172.21.15.104:55006) 2024-06-09T22:22:11.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0ea59c700 1 -- 172.21.15.104:0/2437112015 learned_addr learned my addr 172.21.15.104:0/2437112015 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:11.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0ea59c700 1 -- 172.21.15.104:0/2437112015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c1880 msgr2=0x562bf15beee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:11.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0ea59c700 1 --2- 172.21.15.104:0/2437112015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c1880 0x562bf15beee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0ea59c700 1 -- 172.21.15.104:0/2437112015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15c0f60 msgr2=0x562bf15c4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:11.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0ea59c700 1 --2- 172.21.15.104:0/2437112015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15c0f60 0x562bf15c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.344+0000 7fc0ea59c700 1 -- 172.21.15.104:0/2437112015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bf15bf420 con 0x562bf15bb7a0 2024-06-09T22:22:11.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.345+0000 7fc0ea59c700 1 --2- 172.21.15.104:0/2437112015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15bb7a0 0x562bf15bb260 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc010ac0 tx=0x7fc0dc0973c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866c8771e4b090bc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:11.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.345+0000 7fc0d3fff700 1 -- 172.21.15.104:0/2437112015 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc014070 con 0x562bf15bb7a0 2024-06-09T22:22:11.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.345+0000 7fc0d3fff700 1 -- 172.21.15.104:0/2437112015 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0dc0a4450 con 0x562bf15bb7a0 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.345+0000 7fc0d3fff700 1 -- 172.21.15.104:0/2437112015 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc020040 con 0x562bf15bb7a0 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.346+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2437112015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15bb7a0 msgr2=0x562bf15bb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.346+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2437112015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15bb7a0 0x562bf15bb260 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc010ac0 tx=0x7fc0dc0973c0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.346+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2437112015 shutdown_connections 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.346+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2437112015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15bb7a0 0x562bf15bb260 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.346+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2437112015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c1880 0x562bf15beee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.346+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2437112015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15c0f60 0x562bf15c4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.346+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2437112015 >> 172.21.15.104:0/2437112015 conn(0x562bf14e8080 msgr2=0x562bf15be7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.346+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2437112015 shutdown_connections 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.346+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2437112015 wait complete. 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.347+0000 7fc0fb6a1b80 1 -- start start 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.347+0000 7fc0fb6a1b80 1 Processor -- start 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.347+0000 7fc0fb6a1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15bb7a0 0x562bf15dd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.347+0000 7fc0fb6a1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c0f60 0x562bf15cd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.347+0000 7fc0fb6a1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c1880 0x562bf15cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.347+0000 7fc0fb6a1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1ad770 con 0x562bf15c1880 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.347+0000 7fc0fb6a1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1ad0a0 con 0x562bf15bb7a0 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.347+0000 7fc0fb6a1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1aa280 con 0x562bf15c0f60 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.347+0000 7fc0fb6a1b80 10 client.0 _ll_register_callbacks cb 0x562bf14a25b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0ead9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c1880 0x562bf15cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0ead9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c1880 0x562bf15cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55012/0 (socket says 172.21.15.104:55012) 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0ead9d700 1 -- 172.21.15.104:0/2260539401 learned_addr learned my addr 172.21.15.104:0/2260539401 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0e9d9b700 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c0f60 0x562bf15cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0ea59c700 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15bb7a0 0x562bf15dd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0ead9d700 1 -- 172.21.15.104:0/2260539401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c0f60 msgr2=0x562bf15cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0ead9d700 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c0f60 0x562bf15cd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0ead9d700 1 -- 172.21.15.104:0/2260539401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15bb7a0 msgr2=0x562bf15dd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0ead9d700 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15bb7a0 0x562bf15dd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0ead9d700 1 -- 172.21.15.104:0/2260539401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bf15ce0a0 con 0x562bf15c1880 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0e9d9b700 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c0f60 0x562bf15cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0ead9d700 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c1880 0x562bf15cda60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4004010 tx=0x7fc0e4005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51675ba2ebc64037 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.348+0000 7fc0d1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0d1ffb700 1 -- 172.21.15.104:0/2260539401 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4007700 con 0x562bf15c1880 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0d1ffb700 1 -- 172.21.15.104:0/2260539401 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fc0e40078a0 con 0x562bf15c1880 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2260539401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0dc0a4450 con 0x562bf15c1880 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0d1ffb700 1 -- 172.21.15.104:0/2260539401 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc0e402f050 con 0x562bf15c1880 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0fb6a1b80 10 client.15792 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2260539401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562bf163f1d0 con 0x562bf15c1880 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0d1ffb700 1 -- 172.21.15.104:0/2260539401 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0dc0a4450 con 0x562bf15c1880 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0d1ffb700 1 -- 172.21.15.104:0/2260539401 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc0e4012030 con 0x562bf15c1880 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0fb6a1b80 10 client.15792 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:11.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0fb6a1b80 10 client.15792 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0fb6a1b80 20 client.15792 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x562bf15d54b0 0x562bf1641740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0fb6a1b80 4 client.15792 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.349+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2260539401 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562bf15142d0 con 0x562bf15d54b0 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.350+0000 7fc0ea59c700 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x562bf15d54b0 0x562bf1641740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.350+0000 7fc0d1ffb700 10 client.15792 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.350+0000 7fc0ea59c700 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x562bf15d54b0 0x562bf1641740 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3736b873763527d2 server_cookie=3a4bd7432914831c in_seq=0 out_seq=0 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.350+0000 7fc0d1ffb700 1 -- 172.21.15.104:0/2260539401 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562bf15142d0 con 0x562bf15d54b0 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.350+0000 7fc0d1ffb700 10 client.15792 handle_command_reply: tid=0 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.350+0000 7fc0fb6a1b80 1 client.15792 shutdown 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.350+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2260539401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c1880 msgr2=0x562bf15cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.350+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c1880 0x562bf15cda60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4004010 tx=0x7fc0e4005690 comp rx=0 tx=0).stop 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.351+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2260539401 shutdown_connections 2024-06-09T22:22:11.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.351+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x562bf15d54b0 0x562bf1641740 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.351+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562bf15c1880 0x562bf15cda60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.351+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bf15bb7a0 0x562bf15dd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.351+0000 7fc0fb6a1b80 1 --2- 172.21.15.104:0/2260539401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bf15c0f60 0x562bf15cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.351+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2260539401 >> 172.21.15.104:0/2260539401 conn(0x562bf14e8080 msgr2=0x562bf15c1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:11.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.351+0000 7fc0d1ffb700 0 client.15792 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:22:11.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.351+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2260539401 shutdown_connections 2024-06-09T22:22:11.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.351+0000 7fc0fb6a1b80 1 -- 172.21.15.104:0/2260539401 wait complete. 2024-06-09T22:22:11.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.351+0000 7fc0fb6a1b80 20 client.15792 trim_cache size 0 max 16384 2024-06-09T22:22:11.354 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:22:11.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.352+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/3746535371 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0d40a2f40 msgr2=0x7fc0d40a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:11.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.352+0000 7fc0f0e1e700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0d40a2f40 0x7fc0d40a5400 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0fe520 tx=0x7fc0e0005fd0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.352+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/3746535371 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0ec101390 msgr2=0x7fc0ec19e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:11.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.352+0000 7fc0f0e1e700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0ec101390 0x7fc0ec19e8d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00d950 tx=0x7fc0dc00dc60 comp rx=0 tx=0).stop 2024-06-09T22:22:11.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.352+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/3746535371 shutdown_connections 2024-06-09T22:22:11.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.352+0000 7fc0f0e1e700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0d40a2f40 0x7fc0d40a5400 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.352+0000 7fc0f0e1e700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec107400 0x7fc0ec198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.352+0000 7fc0f0e1e700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0ec101390 0x7fc0ec19e8d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.352+0000 7fc0f0e1e700 1 --2- 172.21.15.104:0/3746535371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec101d50 0x7fc0ec19ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:11.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.352+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/3746535371 >> 172.21.15.104:0/3746535371 conn(0x7fc0ec078340 msgr2=0x7fc0ec0fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:11.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.353+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/3746535371 shutdown_connections 2024-06-09T22:22:11.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:11.353+0000 7fc0f0e1e700 1 -- 172.21.15.104:0/3746535371 wait complete. 2024-06-09T22:22:11.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:11 smithi194 ceph-mon[36856]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-06-09T22:22:11.951 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:11 smithi104 ceph-mon[32389]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-06-09T22:22:11.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:11 smithi153 ceph-mon[36599]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-06-09T22:22:12.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-09T22:22:12.757 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:12 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:22:12.768 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:12.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:12 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:22:12.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:12 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:22:13.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:13 smithi194 ceph-mon[36856]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-06-09T22:22:13.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:13 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:22:13.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:13 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:22:13.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:13 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:22:13.940 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:13 smithi104 ceph-mon[32389]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-06-09T22:22:13.940 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:13 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:22:13.941 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:13 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:22:13.941 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:13 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:22:13.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:13 smithi153 ceph-mon[36599]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-06-09T22:22:13.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:13 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:22:13.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:13 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:22:13.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:13 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:22:14.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.716+0000 7f1b9ba74700 1 -- 172.21.15.104:0/1207280680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b94101b90 msgr2=0x7f1b94102010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:14.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.716+0000 7f1b9ba74700 1 --2- 172.21.15.104:0/1207280680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b94101b90 0x7f1b94102010 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009990 tx=0x7f1b84009ca0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.716+0000 7f1b9ba74700 1 -- 172.21.15.104:0/1207280680 shutdown_connections 2024-06-09T22:22:14.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.716+0000 7f1b9ba74700 1 --2- 172.21.15.104:0/1207280680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b94102550 0x7f1b9410e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.716+0000 7f1b9ba74700 1 --2- 172.21.15.104:0/1207280680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b94106370 0x7f1b94106750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.716+0000 7f1b9ba74700 1 --2- 172.21.15.104:0/1207280680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b94101b90 0x7f1b94102010 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.716+0000 7f1b9ba74700 1 -- 172.21.15.104:0/1207280680 >> 172.21.15.104:0/1207280680 conn(0x7f1b94077a00 msgr2=0x7f1b940fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:14.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.716+0000 7f1b9ba74700 1 -- 172.21.15.104:0/1207280680 shutdown_connections 2024-06-09T22:22:14.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.716+0000 7f1b9ba74700 1 -- 172.21.15.104:0/1207280680 wait complete. 2024-06-09T22:22:14.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.717+0000 7f1b9ba74700 1 Processor -- start 2024-06-09T22:22:14.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.717+0000 7f1b9ba74700 1 -- start start 2024-06-09T22:22:14.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.717+0000 7f1b9ba74700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b94102550 0x7f1b9410dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.717+0000 7f1b9ba74700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b94106370 0x7f1b9410c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.717+0000 7f1b9ba74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b9410c860 0x7f1b9410cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b9ba74700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94111640 con 0x7f1b94102550 2024-06-09T22:22:14.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b9ba74700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941114c0 con 0x7f1b94106370 2024-06-09T22:22:14.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b9ba74700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941117c0 con 0x7f1b9410c860 2024-06-09T22:22:14.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b99810700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b94102550 0x7f1b9410dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b9a011700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b9410c860 0x7f1b9410cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b9a011700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b9410c860 0x7f1b9410cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:49622/0 (socket says 172.21.15.104:49622) 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b9a011700 1 -- 172.21.15.104:0/887366439 learned_addr learned my addr 172.21.15.104:0/887366439 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b9900f700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b94106370 0x7f1b9410c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b9a011700 1 -- 172.21.15.104:0/887366439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b94106370 msgr2=0x7f1b9410c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b9a011700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b94106370 0x7f1b9410c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b9a011700 1 -- 172.21.15.104:0/887366439 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b94102550 msgr2=0x7f1b9410dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b9a011700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b94102550 0x7f1b9410dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.718+0000 7f1b9a011700 1 -- 172.21.15.104:0/887366439 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b941abba0 con 0x7f1b9410c860 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.719+0000 7f1b9a011700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b9410c860 0x7f1b9410cd10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0094f0 tx=0x7f1b8c00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.719+0000 7f1b8affd700 1 -- 172.21.15.104:0/887366439 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c004db0 con 0x7f1b9410c860 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.719+0000 7f1b9ba74700 1 -- 172.21.15.104:0/887366439 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b941abe90 con 0x7f1b9410c860 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.719+0000 7f1b8affd700 1 -- 172.21.15.104:0/887366439 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b8c009e20 con 0x7f1b9410c860 2024-06-09T22:22:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.719+0000 7f1b9ba74700 1 -- 172.21.15.104:0/887366439 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b941ac350 con 0x7f1b9410c860 2024-06-09T22:22:14.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.719+0000 7f1b8affd700 1 -- 172.21.15.104:0/887366439 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c015400 con 0x7f1b9410c860 2024-06-09T22:22:14.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.722+0000 7f1b8affd700 1 -- 172.21.15.104:0/887366439 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 24) ==== 138118+0+0 (secure 0 0 0) 0x7f1b8c00db20 con 0x7f1b9410c860 2024-06-09T22:22:14.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.723+0000 7f1b8affd700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1b800a2f40 0x7f1b800a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.723+0000 7f1b8affd700 1 -- 172.21.15.104:0/887366439 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1b8c0cf4b0 con 0x7f1b9410c860 2024-06-09T22:22:14.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.723+0000 7f1b99810700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1b800a2f40 0x7f1b800a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.724+0000 7f1b99810700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1b800a2f40 0x7f1b800a5400 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009c30 tx=0x7f1b90009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:14.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.724+0000 7f1ba62f7b80 1 Processor -- start 2024-06-09T22:22:14.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.724+0000 7f1ba62f7b80 1 -- start start 2024-06-09T22:22:14.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.725+0000 7f1ba62f7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acecf60 0x562d4aced340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.725+0000 7f1ba62f7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4aced880 0x562d4ace7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.725+0000 7f1ba62f7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4ace7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.725+0000 7f1ba62f7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562d4abd9f60 con 0x562d4aced880 2024-06-09T22:22:14.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.725+0000 7f1b9900f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4aced880 0x562d4ace7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.725+0000 7f1ba62f7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562d4ac38850 con 0x562d4ace77a0 2024-06-09T22:22:14.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.725+0000 7f1ba62f7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562d4aba0460 con 0x562d4acecf60 2024-06-09T22:22:14.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.725+0000 7f1b9900f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4aced880 0x562d4ace7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55036/0 (socket says 172.21.15.104:55036) 2024-06-09T22:22:14.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.725+0000 7f1b9900f700 1 -- 172.21.15.104:0/3030532864 learned_addr learned my addr 172.21.15.104:0/3030532864 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:14.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.725+0000 7f1b9a011700 1 --2- 172.21.15.104:0/3030532864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4ace7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.725+0000 7f1b99810700 1 --2- 172.21.15.104:0/3030532864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acecf60 0x562d4aced340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.726+0000 7f1b9900f700 1 -- 172.21.15.104:0/3030532864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acecf60 msgr2=0x562d4aced340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:14.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.726+0000 7f1b9900f700 1 --2- 172.21.15.104:0/3030532864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acecf60 0x562d4aced340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.726+0000 7f1b9900f700 1 -- 172.21.15.104:0/3030532864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 msgr2=0x562d4ace7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:14.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.726+0000 7f1b9900f700 1 --2- 172.21.15.104:0/3030532864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4ace7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.726+0000 7f1b9900f700 1 -- 172.21.15.104:0/3030532864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d4ab878d0 con 0x562d4aced880 2024-06-09T22:22:14.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.726+0000 7f1b9a011700 1 --2- 172.21.15.104:0/3030532864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4ace7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:14.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.726+0000 7f1b9900f700 1 --2- 172.21.15.104:0/3030532864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4aced880 0x562d4ace7260 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400b2e0 tx=0x7f1b84005960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98df059a5090813 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:14.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.726+0000 7f1b7ffff700 1 -- 172.21.15.104:0/3030532864 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8400bce0 con 0x562d4aced880 2024-06-09T22:22:14.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.726+0000 7f1b7ffff700 1 -- 172.21.15.104:0/3030532864 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b8400be80 con 0x562d4aced880 2024-06-09T22:22:14.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.726+0000 7f1b7ffff700 1 -- 172.21.15.104:0/3030532864 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b84003cd0 con 0x562d4aced880 2024-06-09T22:22:14.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.727+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3030532864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4aced880 msgr2=0x562d4ace7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:14.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.727+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3030532864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4aced880 0x562d4ace7260 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400b2e0 tx=0x7f1b84005960 comp rx=0 tx=0).stop 2024-06-09T22:22:14.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.727+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3030532864 shutdown_connections 2024-06-09T22:22:14.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.727+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3030532864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4ace7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.727+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3030532864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acecf60 0x562d4aced340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.727+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3030532864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4aced880 0x562d4ace7260 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.727+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3030532864 >> 172.21.15.104:0/3030532864 conn(0x562d4ac14080 msgr2=0x562d4ace2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:14.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.727+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3030532864 shutdown_connections 2024-06-09T22:22:14.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.727+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3030532864 wait complete. 2024-06-09T22:22:14.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.728+0000 7f1ba62f7b80 1 -- start start 2024-06-09T22:22:14.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.728+0000 7f1ba62f7b80 1 Processor -- start 2024-06-09T22:22:14.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.728+0000 7f1ba62f7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4ace77a0 0x562d4acf38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1ba62f7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4acecf60 0x562d4acf0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1ba62f7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acf1150 0x562d4acf15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1ba62f7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562d4abd8fc0 con 0x562d4acf1150 2024-06-09T22:22:14.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1ba62f7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562d4ac34810 con 0x562d4acecf60 2024-06-09T22:22:14.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1ba62f7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562d4acf4200 con 0x562d4ace77a0 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1ba62f7b80 10 client.0 _ll_register_callbacks cb 0x562d4abce5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1b9a011700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acf1150 0x562d4acf15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1b9a011700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acf1150 0x562d4acf15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55038/0 (socket says 172.21.15.104:55038) 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1b9a011700 1 -- 172.21.15.104:0/3969753609 learned_addr learned my addr 172.21.15.104:0/3969753609 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1b9900f700 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4acecf60 0x562d4acf0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1b99810700 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4ace77a0 0x562d4acf38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1b9a011700 1 -- 172.21.15.104:0/3969753609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4ace77a0 msgr2=0x562d4acf38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1b9a011700 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4ace77a0 0x562d4acf38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1b9a011700 1 -- 172.21.15.104:0/3969753609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4acecf60 msgr2=0x562d4acf0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1b9a011700 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4acecf60 0x562d4acf0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1b9a011700 1 -- 172.21.15.104:0/3969753609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d4abfe750 con 0x562d4acf1150 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1b9a011700 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acf1150 0x562d4acf15b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009fd0 tx=0x7f1b8c09aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97edb1980c988552 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:14.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.729+0000 7f1b7d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.730+0000 7f1b7d7fa700 1 -- 172.21.15.104:0/3969753609 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c09f7d0 con 0x562d4acf1150 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.730+0000 7f1b7d7fa700 1 -- 172.21.15.104:0/3969753609 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1b8c0a3c00 con 0x562d4acf1150 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.730+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3969753609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562d4ac45d50 con 0x562d4acf1150 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.730+0000 7f1b7d7fa700 1 -- 172.21.15.104:0/3969753609 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1b8c0a4090 con 0x562d4acf1150 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.730+0000 7f1ba62f7b80 10 client.15804 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.730+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3969753609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562d4acfefc0 con 0x562d4acf1150 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.731+0000 7f1b7d7fa700 1 -- 172.21.15.104:0/3969753609 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562d4ac45d50 con 0x562d4acf1150 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.731+0000 7f1b7d7fa700 1 -- 172.21.15.104:0/3969753609 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1b8c012070 con 0x562d4acf1150 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.731+0000 7f1ba62f7b80 10 client.15804 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.731+0000 7f1ba62f7b80 10 client.15804 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.731+0000 7f1ba62f7b80 20 client.15804 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.731+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562d4ad01550 0x562d4ad03940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.731+0000 7f1ba62f7b80 4 client.15804 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.731+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3969753609 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d4ac402d0 con 0x562d4ad01550 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.731+0000 7f1b99810700 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562d4ad01550 0x562d4ad03940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.732+0000 7f1b99810700 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562d4ad01550 0x562d4ad03940 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e0b46fb9495e9ba server_cookie=e71e56d4d5379621 in_seq=0 out_seq=0 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.732+0000 7f1b7d7fa700 10 client.15804 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1b7d7fa700 1 -- 172.21.15.104:0/3969753609 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x562d4ac402d0 con 0x562d4ad01550 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1b7d7fa700 10 client.15804 handle_command_reply: tid=0 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1ba62f7b80 1 client.15804 shutdown 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3969753609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acf1150 msgr2=0x562d4acf15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acf1150 0x562d4acf15b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009fd0 tx=0x7f1b8c09aeb0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3969753609 shutdown_connections 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562d4ad01550 0x562d4ad03940 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acf1150 0x562d4acf15b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4ace77a0 0x562d4acf38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3969753609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4acecf60 0x562d4acf0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3969753609 >> 172.21.15.104:0/3969753609 conn(0x562d4ac14080 msgr2=0x562d4ace1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:14.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1b7d7fa700 0 client.15804 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:14.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3969753609 shutdown_connections 2024-06-09T22:22:14.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.733+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3969753609 wait complete. 2024-06-09T22:22:14.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.734+0000 7f1ba62f7b80 20 client.15804 trim_cache size 0 max 16384 2024-06-09T22:22:14.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.763+0000 7f1ba62f7b80 1 Processor -- start 2024-06-09T22:22:14.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.763+0000 7f1ba62f7b80 1 -- start start 2024-06-09T22:22:14.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.763+0000 7f1ba62f7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4acefbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.763+0000 7f1ba62f7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 0x562d4ad09170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.763+0000 7f1ba62f7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acf1150 0x562d4ad096b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.763+0000 7f1ba62f7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941b5090 con 0x562d4acecf60 2024-06-09T22:22:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.763+0000 7f1ba62f7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941b6bd0 con 0x562d4ace77a0 2024-06-09T22:22:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.763+0000 7f1ba62f7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941b5760 con 0x562d4acf1150 2024-06-09T22:22:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.763+0000 7f1b9900f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 0x562d4ad09170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.764+0000 7f1b9900f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 0x562d4ad09170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55040/0 (socket says 172.21.15.104:55040) 2024-06-09T22:22:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.764+0000 7f1b99810700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4acefbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.764+0000 7f1b9900f700 1 -- 172.21.15.104:0/1880372927 learned_addr learned my addr 172.21.15.104:0/1880372927 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.764+0000 7f1b9a011700 1 --2- 172.21.15.104:0/1880372927 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acf1150 0x562d4ad096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.764+0000 7f1b99810700 1 -- 172.21.15.104:0/1880372927 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acf1150 msgr2=0x562d4ad096b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.764+0000 7f1b99810700 1 --2- 172.21.15.104:0/1880372927 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acf1150 0x562d4ad096b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.764+0000 7f1b99810700 1 -- 172.21.15.104:0/1880372927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 msgr2=0x562d4ad09170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.764+0000 7f1b99810700 1 --2- 172.21.15.104:0/1880372927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 0x562d4ad09170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.764+0000 7f1b99810700 1 -- 172.21.15.104:0/1880372927 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d4acef150 con 0x562d4ace77a0 2024-06-09T22:22:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.764+0000 7f1b9900f700 1 --2- 172.21.15.104:0/1880372927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 0x562d4ad09170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.764+0000 7f1b99810700 1 --2- 172.21.15.104:0/1880372927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4acefbe0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c097da0 tx=0x7f1b90014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e54268fec0f0525d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.765+0000 7f1b7f7fe700 1 -- 172.21.15.104:0/1880372927 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9002e070 con 0x562d4ace77a0 2024-06-09T22:22:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.765+0000 7f1b7f7fe700 1 -- 172.21.15.104:0/1880372927 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b90003b60 con 0x562d4ace77a0 2024-06-09T22:22:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.765+0000 7f1b7f7fe700 1 -- 172.21.15.104:0/1880372927 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b90029ac0 con 0x562d4ace77a0 2024-06-09T22:22:14.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.765+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/1880372927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 msgr2=0x562d4acefbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:14.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.765+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/1880372927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4acefbe0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c097da0 tx=0x7f1b90014000 comp rx=0 tx=0).stop 2024-06-09T22:22:14.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.765+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/1880372927 shutdown_connections 2024-06-09T22:22:14.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.765+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/1880372927 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acf1150 0x562d4ad096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.765+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/1880372927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4acefbe0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.765+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/1880372927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 0x562d4ad09170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.765+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/1880372927 >> 172.21.15.104:0/1880372927 conn(0x562d4ac14080 msgr2=0x562d4acef4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:14.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.766+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/1880372927 shutdown_connections 2024-06-09T22:22:14.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.766+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/1880372927 wait complete. 2024-06-09T22:22:14.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.766+0000 7f1ba62f7b80 1 -- start start 2024-06-09T22:22:14.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.766+0000 7f1ba62f7b80 1 Processor -- start 2024-06-09T22:22:14.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1ba62f7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4ad08d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1ba62f7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 0x562d4ad09250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1ba62f7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acf1150 0x562d4ad09790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1ba62f7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941b6500 con 0x562d4acecf60 2024-06-09T22:22:14.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1ba62f7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941b5e30 con 0x562d4ace77a0 2024-06-09T22:22:14.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1ba62f7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941b3010 con 0x562d4acf1150 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1ba62f7b80 10 client.0 _ll_register_callbacks cb 0x562d4abce5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b9900f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 0x562d4ad09250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b99810700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4ad08d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b9900f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 0x562d4ad09250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55054/0 (socket says 172.21.15.104:55054) 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b9900f700 1 -- 172.21.15.104:0/3810018643 learned_addr learned my addr 172.21.15.104:0/3810018643 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b99810700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4ad08d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:56712/0 (socket says 172.21.15.104:56712) 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b9a011700 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acf1150 0x562d4ad09790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b9a011700 1 -- 172.21.15.104:0/3810018643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 msgr2=0x562d4ad08d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b9a011700 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4ad08d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b9a011700 1 -- 172.21.15.104:0/3810018643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 msgr2=0x562d4ad09250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b99810700 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4ad08d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b9a011700 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 0x562d4ad09250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b9a011700 1 -- 172.21.15.104:0/3810018643 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d4ad6b140 con 0x562d4acf1150 2024-06-09T22:22:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.767+0000 7f1b9900f700 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 0x562d4ad09250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.768+0000 7f1b9a011700 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acf1150 0x562d4ad09790 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c097df0 tx=0x7f1b8c005230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7da4e21bb197a24 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.768+0000 7f1b7d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:22:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.768+0000 7f1b7d7fa700 1 -- 172.21.15.104:0/3810018643 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c0a3e00 con 0x562d4acf1150 2024-06-09T22:22:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.768+0000 7f1b7d7fa700 1 -- 172.21.15.104:0/3810018643 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1b8c0971c0 con 0x562d4acf1150 2024-06-09T22:22:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.768+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3810018643 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b90003b60 con 0x562d4acf1150 2024-06-09T22:22:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.768+0000 7f1b7d7fa700 1 -- 172.21.15.104:0/3810018643 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1b8c09f7d0 con 0x562d4acf1150 2024-06-09T22:22:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.768+0000 7f1ba62f7b80 10 client.25384 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.768+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3810018643 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562d4ad6c380 con 0x562d4acf1150 2024-06-09T22:22:14.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.768+0000 7f1b7d7fa700 1 -- 172.21.15.104:0/3810018643 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b8c00de10 con 0x562d4acf1150 2024-06-09T22:22:14.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.769+0000 7f1b7d7fa700 1 -- 172.21.15.104:0/3810018643 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1b8c012040 con 0x562d4acf1150 2024-06-09T22:22:14.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.769+0000 7f1ba62f7b80 10 client.25384 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:14.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.769+0000 7f1ba62f7b80 10 client.25384 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:14.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.769+0000 7f1ba62f7b80 20 client.25384 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:14.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.769+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562d4ad6ca00 0x562d4ad6ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:14.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.769+0000 7f1ba62f7b80 4 client.25384 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:22:14.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.769+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3810018643 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f1b90003b60 con 0x562d4ad6ca00 2024-06-09T22:22:14.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.769+0000 7f1b99810700 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562d4ad6ca00 0x562d4ad6ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:14.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.769+0000 7f1b7d7fa700 10 client.25384 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.769+0000 7f1b99810700 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562d4ad6ca00 0x562d4ad6ee20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32f07daec9d59923 server_cookie=f8a7d66dc34465e in_seq=0 out_seq=0 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1b7d7fa700 1 -- 172.21.15.104:0/3810018643 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f1b90003b60 con 0x562d4ad6ca00 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1b7d7fa700 10 client.25384 handle_command_reply: tid=0 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1ba62f7b80 1 client.25384 shutdown 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3810018643 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acf1150 msgr2=0x562d4ad09790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acf1150 0x562d4ad09790 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c097df0 tx=0x7f1b8c005230 comp rx=0 tx=0).stop 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3810018643 shutdown_connections 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562d4ad6ca00 0x562d4ad6ee20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d4acf1150 0x562d4ad09790 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d4ace77a0 0x562d4ad08d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1ba62f7b80 1 --2- 172.21.15.104:0/3810018643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d4acecf60 0x562d4ad09250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3810018643 >> 172.21.15.104:0/3810018643 conn(0x562d4ac14080 msgr2=0x562d4acebcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:14.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1b7d7fa700 0 client.25384 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:14.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3810018643 shutdown_connections 2024-06-09T22:22:14.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.770+0000 7f1ba62f7b80 1 -- 172.21.15.104:0/3810018643 wait complete. 2024-06-09T22:22:14.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.771+0000 7f1ba62f7b80 20 client.25384 trim_cache size 0 max 16384 2024-06-09T22:22:14.774 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:22:14.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.771+0000 7f1b9ba74700 1 -- 172.21.15.104:0/887366439 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1b800a2f40 msgr2=0x7f1b800a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:14.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.771+0000 7f1b9ba74700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1b800a2f40 0x7f1b800a5400 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009c30 tx=0x7f1b90009260 comp rx=0 tx=0).stop 2024-06-09T22:22:14.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.771+0000 7f1b9ba74700 1 -- 172.21.15.104:0/887366439 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b9410c860 msgr2=0x7f1b9410cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:14.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.771+0000 7f1b9ba74700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b9410c860 0x7f1b9410cd10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0094f0 tx=0x7f1b8c00d3b0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.771+0000 7f1b9ba74700 1 -- 172.21.15.104:0/887366439 shutdown_connections 2024-06-09T22:22:14.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.771+0000 7f1b9ba74700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1b800a2f40 0x7f1b800a5400 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.771+0000 7f1b9ba74700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b94102550 0x7f1b9410dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.771+0000 7f1b9ba74700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b9410c860 0x7f1b9410cd10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.771+0000 7f1b9ba74700 1 --2- 172.21.15.104:0/887366439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b94106370 0x7f1b9410c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:14.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.771+0000 7f1b9ba74700 1 -- 172.21.15.104:0/887366439 >> 172.21.15.104:0/887366439 conn(0x7f1b94077a00 msgr2=0x7f1b941038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:14.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.772+0000 7f1b9ba74700 1 -- 172.21.15.104:0/887366439 shutdown_connections 2024-06-09T22:22:14.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:14.772+0000 7f1b9ba74700 1 -- 172.21.15.104:0/887366439 wait complete. 2024-06-09T22:22:15.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:15 smithi194 ceph-mon[36856]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 KiB/s wr, 30 op/s 2024-06-09T22:22:15.866 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:15 smithi104 ceph-mon[32389]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 KiB/s wr, 30 op/s 2024-06-09T22:22:15.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:15 smithi153 ceph-mon[36599]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 KiB/s wr, 30 op/s 2024-06-09T22:22:16.595 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-09T22:22:16.595 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-09T22:22:16.704 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-06-09T22:22:16.704 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-06-09T22:22:16.704 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:22:16.879 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:17.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:17 smithi104 ceph-mon[32389]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-06-09T22:22:17.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:17 smithi194 ceph-mon[36856]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-06-09T22:22:17.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:17 smithi153 ceph-mon[36599]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-06-09T22:22:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.934+0000 7feafcff9640 1 -- 172.21.15.104:0/663794313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaf8102c30 msgr2=0x7feaf8103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.934+0000 7feafcff9640 1 --2- 172.21.15.104:0/663794313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaf8102c30 0x7feaf8103010 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7feae8009900 tx=0x7feae802a510 comp rx=0 tx=0).stop 2024-06-09T22:22:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.934+0000 7feafcff9640 1 -- 172.21.15.104:0/663794313 shutdown_connections 2024-06-09T22:22:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.934+0000 7feafcff9640 1 --2- 172.21.15.104:0/663794313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feaf810bf70 0x7feaf810e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.934+0000 7feafcff9640 1 --2- 172.21.15.104:0/663794313 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feaf8103550 0x7feaf810ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.934+0000 7feafcff9640 1 --2- 172.21.15.104:0/663794313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaf8102c30 0x7feaf8103010 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.934+0000 7feafcff9640 1 -- 172.21.15.104:0/663794313 >> 172.21.15.104:0/663794313 conn(0x7feaf8077a20 msgr2=0x7feaf8077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.934+0000 7feafcff9640 1 -- 172.21.15.104:0/663794313 shutdown_connections 2024-06-09T22:22:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.934+0000 7feafcff9640 1 -- 172.21.15.104:0/663794313 wait complete. 2024-06-09T22:22:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.935+0000 7feafcff9640 1 Processor -- start 2024-06-09T22:22:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.935+0000 7feafcff9640 1 -- start start 2024-06-09T22:22:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feafcff9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feaf8102c30 0x7feaf81982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feafcff9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaf8103550 0x7feaf8198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feafcff9640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feaf810bf70 0x7feaf819cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feafcff9640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feaf8111d70 con 0x7feaf8103550 2024-06-09T22:22:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feaf5d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaf8103550 0x7feaf8198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feafcff9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feaf8111bf0 con 0x7feaf8102c30 2024-06-09T22:22:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feafcff9640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feaf8111ef0 con 0x7feaf810bf70 2024-06-09T22:22:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feaf5d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaf8103550 0x7feaf8198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58242/0 (socket says 172.21.15.104:58242) 2024-06-09T22:22:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feaf5d9b640 1 -- 172.21.15.104:0/2602672443 learned_addr learned my addr 172.21.15.104:0/2602672443 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feaf659c640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feaf8102c30 0x7feaf81982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feaf6d9d640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feaf810bf70 0x7feaf819cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feaf5d9b640 1 -- 172.21.15.104:0/2602672443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feaf810bf70 msgr2=0x7feaf819cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feaf5d9b640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feaf810bf70 0x7feaf819cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feaf5d9b640 1 -- 172.21.15.104:0/2602672443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feaf8102c30 msgr2=0x7feaf81982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feaf5d9b640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feaf8102c30 0x7feaf81982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feaf5d9b640 1 -- 172.21.15.104:0/2602672443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaf819d270 con 0x7feaf8103550 2024-06-09T22:22:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.936+0000 7feaf5d9b640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaf8103550 0x7feaf8198810 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feaec00d950 tx=0x7feaec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.937+0000 7feae77fe640 1 -- 172.21.15.104:0/2602672443 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec017030 con 0x7feaf8103550 2024-06-09T22:22:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.937+0000 7feafcff9640 1 -- 172.21.15.104:0/2602672443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaf819d560 con 0x7feaf8103550 2024-06-09T22:22:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.937+0000 7feae77fe640 1 -- 172.21.15.104:0/2602672443 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaec004020 con 0x7feaf8103550 2024-06-09T22:22:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.937+0000 7feae77fe640 1 -- 172.21.15.104:0/2602672443 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec0047a0 con 0x7feaf8103550 2024-06-09T22:22:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.938+0000 7feafcff9640 1 -- 172.21.15.104:0/2602672443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaf81a4df0 con 0x7feaf8103550 2024-06-09T22:22:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.940+0000 7feae77fe640 1 -- 172.21.15.104:0/2602672443 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 24) ==== 138118+0+0 (secure 0 0 0) 0x7feaec0041d0 con 0x7feaf8103550 2024-06-09T22:22:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.940+0000 7feaffde3740 1 Processor -- start 2024-06-09T22:22:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.940+0000 7feaffde3740 1 -- start start 2024-06-09T22:22:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaffde3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682170 0x55b89a682550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaffde3740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682a90 0x55b89a68ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaffde3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a68d350 0x55b89a68f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaffde3740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b89a5e64f0 con 0x55b89a682a90 2024-06-09T22:22:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaffde3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b89a584870 con 0x55b89a68d350 2024-06-09T22:22:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaffde3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b89a5e24b0 con 0x55b89a682170 2024-06-09T22:22:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaf659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682a90 0x55b89a68ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaf659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682a90 0x55b89a68ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58258/0 (socket says 172.21.15.104:58258) 2024-06-09T22:22:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaf659c640 1 -- 172.21.15.104:0/718008291 learned_addr learned my addr 172.21.15.104:0/718008291 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaf6d9d640 1 --2- 172.21.15.104:0/718008291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682170 0x55b89a682550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaf5d9b640 1 --2- 172.21.15.104:0/718008291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a68d350 0x55b89a68f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaf659c640 1 -- 172.21.15.104:0/718008291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682170 msgr2=0x55b89a682550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaf659c640 1 --2- 172.21.15.104:0/718008291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682170 0x55b89a682550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaf659c640 1 -- 172.21.15.104:0/718008291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a68d350 msgr2=0x55b89a68f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaf659c640 1 --2- 172.21.15.104:0/718008291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a68d350 0x55b89a68f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.941+0000 7feaf659c640 1 -- 172.21.15.104:0/718008291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b89a5a09b0 con 0x55b89a682a90 2024-06-09T22:22:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.942+0000 7feae77fe640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7feae00a2f50 0x7feae00a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.942+0000 7feaf6d9d640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7feae00a2f50 0x7feae00a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.942+0000 7feaf659c640 1 --2- 172.21.15.104:0/718008291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682a90 0x55b89a68ce10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feae8009900 tx=0x7feae8031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5417413635afa2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.942+0000 7feae77fe640 1 -- 172.21.15.104:0/2602672443 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7feaec0cef20 con 0x7feaf8103550 2024-06-09T22:22:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.942+0000 7feae4ff9640 1 -- 172.21.15.104:0/718008291 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae8035040 con 0x55b89a682a90 2024-06-09T22:22:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.942+0000 7feae4ff9640 1 -- 172.21.15.104:0/718008291 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feae8005020 con 0x55b89a682a90 2024-06-09T22:22:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.943+0000 7feaffde3740 1 -- 172.21.15.104:0/718008291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682a90 msgr2=0x55b89a68ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.943+0000 7feaf6d9d640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7feae00a2f50 0x7feae00a5410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feaf0008dc0 tx=0x7feaf0008570 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.943+0000 7feaffde3740 1 --2- 172.21.15.104:0/718008291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682a90 0x55b89a68ce10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feae8009900 tx=0x7feae8031630 comp rx=0 tx=0).stop 2024-06-09T22:22:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.943+0000 7feaffde3740 1 -- 172.21.15.104:0/718008291 shutdown_connections 2024-06-09T22:22:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.943+0000 7feaffde3740 1 --2- 172.21.15.104:0/718008291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a68d350 0x55b89a68f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.943+0000 7feaffde3740 1 --2- 172.21.15.104:0/718008291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682a90 0x55b89a68ce10 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.943+0000 7feaffde3740 1 --2- 172.21.15.104:0/718008291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682170 0x55b89a682550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.943+0000 7feaffde3740 1 -- 172.21.15.104:0/718008291 >> 172.21.15.104:0/718008291 conn(0x55b89a67e680 msgr2=0x55b89a67ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.943+0000 7feaffde3740 1 -- 172.21.15.104:0/718008291 shutdown_connections 2024-06-09T22:22:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.943+0000 7feaffde3740 1 -- 172.21.15.104:0/718008291 wait complete. 2024-06-09T22:22:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.944+0000 7feaffde3740 1 -- start start 2024-06-09T22:22:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.944+0000 7feaffde3740 1 Processor -- start 2024-06-09T22:22:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.944+0000 7feaffde3740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682170 0x55b89a68a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.944+0000 7feaffde3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682a90 0x55b89a68a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.944+0000 7feaffde3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a68d350 0x55b89a68ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.944+0000 7feaffde3740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b89a68ff70 con 0x55b89a682170 2024-06-09T22:22:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.944+0000 7feaffde3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b89a56fae0 con 0x55b89a68d350 2024-06-09T22:22:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.944+0000 7feaffde3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b89a6900f0 con 0x55b89a682a90 2024-06-09T22:22:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.944+0000 7feaf659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682170 0x55b89a68a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.944+0000 7feaf659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682170 0x55b89a68a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58272/0 (socket says 172.21.15.104:58272) 2024-06-09T22:22:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaf659c640 1 -- 172.21.15.104:0/3229186096 learned_addr learned my addr 172.21.15.104:0/3229186096 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaffde3740 10 client.0 _ll_register_callbacks cb 0x55b89a5a89b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaf5d9b640 1 --2- 172.21.15.104:0/3229186096 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682a90 0x55b89a68a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaf659c640 1 -- 172.21.15.104:0/3229186096 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682a90 msgr2=0x55b89a68a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaf659c640 1 --2- 172.21.15.104:0/3229186096 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682a90 0x55b89a68a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaf659c640 1 -- 172.21.15.104:0/3229186096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a68d350 msgr2=0x55b89a68ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaf659c640 1 --2- 172.21.15.104:0/3229186096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a68d350 0x55b89a68ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaf659c640 1 -- 172.21.15.104:0/3229186096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b89a2fa210 con 0x55b89a682170 2024-06-09T22:22:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaf659c640 1 --2- 172.21.15.104:0/3229186096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682170 0x55b89a68a1e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feae8006010 tx=0x7feae803ff10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446b953dd478e90e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaddffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaddffb640 1 -- 172.21.15.104:0/3229186096 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae8035050 con 0x55b89a682170 2024-06-09T22:22:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaddffb640 1 -- 172.21.15.104:0/3229186096 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7feae80051d0 con 0x55b89a682170 2024-06-09T22:22:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaffde3740 1 -- 172.21.15.104:0/3229186096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feae8005020 con 0x55b89a682170 2024-06-09T22:22:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaddffb640 1 -- 172.21.15.104:0/3229186096 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feae80053d0 con 0x55b89a682170 2024-06-09T22:22:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaffde3740 10 client.15819 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.945+0000 7feaffde3740 1 -- 172.21.15.104:0/3229186096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b89a3c6620 con 0x55b89a682170 2024-06-09T22:22:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.946+0000 7feaddffb640 1 -- 172.21.15.104:0/3229186096 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feae8005570 con 0x55b89a682170 2024-06-09T22:22:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.946+0000 7feaddffb640 1 -- 172.21.15.104:0/3229186096 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7feae8051910 con 0x55b89a682170 2024-06-09T22:22:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.946+0000 7feaffde3740 10 client.15819 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.946+0000 7feaffde3740 10 client.15819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.946+0000 7feaffde3740 20 client.15819 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.946+0000 7feaffde3740 1 --2- 172.21.15.104:0/3229186096 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b89a69d2c0 0x55b89a69f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.946+0000 7feaffde3740 4 client.15819 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.946+0000 7feaffde3740 1 -- 172.21.15.104:0/3229186096 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7feae8005020 con 0x55b89a69d2c0 2024-06-09T22:22:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.946+0000 7feaf5d9b640 1 --2- 172.21.15.104:0/3229186096 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b89a69d2c0 0x55b89a69f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.947+0000 7feaf5d9b640 1 --2- 172.21.15.104:0/3229186096 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b89a69d2c0 0x55b89a69f6b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23fba9fa2ebab879 server_cookie=796081c1784d4b2e in_seq=0 out_seq=0 2024-06-09T22:22:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.947+0000 7feaddffb640 10 client.15819 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.947+0000 7feaddffb640 1 -- 172.21.15.104:0/3229186096 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x7feae8005020 con 0x55b89a69d2c0 2024-06-09T22:22:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.947+0000 7feaddffb640 10 client.15819 handle_command_reply: tid=0 2024-06-09T22:22:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.947+0000 7feaffde3740 1 client.15819 shutdown 2024-06-09T22:22:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.948+0000 7feaffde3740 1 -- 172.21.15.104:0/3229186096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682170 msgr2=0x55b89a68a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.948+0000 7feaffde3740 1 --2- 172.21.15.104:0/3229186096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682170 0x55b89a68a1e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feae8006010 tx=0x7feae803ff10 comp rx=0 tx=0).stop 2024-06-09T22:22:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.948+0000 7feaffde3740 1 -- 172.21.15.104:0/3229186096 shutdown_connections 2024-06-09T22:22:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.948+0000 7feaffde3740 1 --2- 172.21.15.104:0/3229186096 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b89a69d2c0 0x55b89a69f6b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.948+0000 7feaddffb640 0 client.15819 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.948+0000 7feaffde3740 1 --2- 172.21.15.104:0/3229186096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a68d350 0x55b89a68ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.948+0000 7feaffde3740 1 --2- 172.21.15.104:0/3229186096 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682a90 0x55b89a68a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.948+0000 7feaffde3740 1 --2- 172.21.15.104:0/3229186096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a682170 0x55b89a68a1e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.948+0000 7feaffde3740 1 -- 172.21.15.104:0/3229186096 >> 172.21.15.104:0/3229186096 conn(0x55b89a67e680 msgr2=0x55b89a6843b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.948+0000 7feaffde3740 1 -- 172.21.15.104:0/3229186096 shutdown_connections 2024-06-09T22:22:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.948+0000 7feaffde3740 1 -- 172.21.15.104:0/3229186096 wait complete. 2024-06-09T22:22:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.948+0000 7feaffde3740 20 client.15819 trim_cache size 0 max 16384 2024-06-09T22:22:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.978+0000 7feaffde3740 1 Processor -- start 2024-06-09T22:22:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.978+0000 7feaffde3740 1 -- start start 2024-06-09T22:22:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.978+0000 7feaffde3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682170 0x55b89a6891c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.978+0000 7feaffde3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a682a90 0x55b89a6a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.979+0000 7feaffde3740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a68d350 0x55b89a6a5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.979+0000 7feaffde3740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feaf81acbe0 con 0x55b89a68d350 2024-06-09T22:22:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.979+0000 7feaf659c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682170 0x55b89a6891c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.979+0000 7feaf659c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682170 0x55b89a6891c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:41938/0 (socket says 172.21.15.104:41938) 2024-06-09T22:22:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.979+0000 7feaf659c640 1 -- 172.21.15.104:0/4261876414 learned_addr learned my addr 172.21.15.104:0/4261876414 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.979+0000 7feaf6d9d640 1 --2- 172.21.15.104:0/4261876414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a68d350 0x55b89a6a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.979+0000 7feaffde3740 1 -- 172.21.15.104:0/4261876414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feaf81ae720 con 0x55b89a682a90 2024-06-09T22:22:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.979+0000 7feaf5d9b640 1 --2- 172.21.15.104:0/4261876414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a682a90 0x55b89a6a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.979+0000 7feaffde3740 1 -- 172.21.15.104:0/4261876414 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feaf81ad2b0 con 0x55b89a682170 2024-06-09T22:22:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.980+0000 7feaf6d9d640 1 -- 172.21.15.104:0/4261876414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682170 msgr2=0x55b89a6891c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.980+0000 7feaf6d9d640 1 --2- 172.21.15.104:0/4261876414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682170 0x55b89a6891c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.980+0000 7feaf6d9d640 1 -- 172.21.15.104:0/4261876414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a682a90 msgr2=0x55b89a6a4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.980+0000 7feaf6d9d640 1 --2- 172.21.15.104:0/4261876414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a682a90 0x55b89a6a4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.980+0000 7feaf6d9d640 1 -- 172.21.15.104:0/4261876414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b89a6828c0 con 0x55b89a68d350 2024-06-09T22:22:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.980+0000 7feaf6d9d640 1 --2- 172.21.15.104:0/4261876414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a68d350 0x55b89a6a5190 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7feaf0008ba0 tx=0x7feaf0013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b4e5680db0ec24 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.980+0000 7feadffff640 1 -- 172.21.15.104:0/4261876414 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf000ec30 con 0x55b89a68d350 2024-06-09T22:22:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.980+0000 7feadffff640 1 -- 172.21.15.104:0/4261876414 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaf000edd0 con 0x55b89a68d350 2024-06-09T22:22:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.980+0000 7feadffff640 1 -- 172.21.15.104:0/4261876414 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf001d830 con 0x55b89a68d350 2024-06-09T22:22:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.981+0000 7feaffde3740 1 -- 172.21.15.104:0/4261876414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a68d350 msgr2=0x55b89a6a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.981+0000 7feaffde3740 1 --2- 172.21.15.104:0/4261876414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a68d350 0x55b89a6a5190 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7feaf0008ba0 tx=0x7feaf0013040 comp rx=0 tx=0).stop 2024-06-09T22:22:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.981+0000 7feaffde3740 1 -- 172.21.15.104:0/4261876414 shutdown_connections 2024-06-09T22:22:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.981+0000 7feaffde3740 1 --2- 172.21.15.104:0/4261876414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a68d350 0x55b89a6a5190 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.981+0000 7feaffde3740 1 --2- 172.21.15.104:0/4261876414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a682a90 0x55b89a6a4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.981+0000 7feaffde3740 1 --2- 172.21.15.104:0/4261876414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682170 0x55b89a6891c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.981+0000 7feaffde3740 1 -- 172.21.15.104:0/4261876414 >> 172.21.15.104:0/4261876414 conn(0x55b89a67e680 msgr2=0x55b89a688ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.981+0000 7feaffde3740 1 -- 172.21.15.104:0/4261876414 shutdown_connections 2024-06-09T22:22:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.981+0000 7feaffde3740 1 -- 172.21.15.104:0/4261876414 wait complete. 2024-06-09T22:22:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.982+0000 7feaffde3740 1 -- start start 2024-06-09T22:22:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.982+0000 7feaffde3740 1 Processor -- start 2024-06-09T22:22:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.982+0000 7feaffde3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a682170 0x55b89a695650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.982+0000 7feaffde3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682a90 0x55b89a695b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.982+0000 7feaffde3740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a68d350 0x55b89a690a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.982+0000 7feaffde3740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feaf81ae050 con 0x55b89a68d350 2024-06-09T22:22:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.982+0000 7feaffde3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feaf81ad980 con 0x55b89a682170 2024-06-09T22:22:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.982+0000 7feaffde3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feaf81aaba0 con 0x55b89a682a90 2024-06-09T22:22:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.982+0000 7feaf5d9b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682a90 0x55b89a695b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.982+0000 7feaf5d9b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682a90 0x55b89a695b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:41948/0 (socket says 172.21.15.104:41948) 2024-06-09T22:22:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.982+0000 7feaf5d9b640 1 -- 172.21.15.104:0/898556106 learned_addr learned my addr 172.21.15.104:0/898556106 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.982+0000 7feaf6d9d640 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a68d350 0x55b89a690a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaf659c640 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a682170 0x55b89a695650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaffde3740 10 client.0 _ll_register_callbacks cb 0x7feaf0008410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaf5d9b640 1 -- 172.21.15.104:0/898556106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a682170 msgr2=0x55b89a695650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaf5d9b640 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a682170 0x55b89a695650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaf5d9b640 1 -- 172.21.15.104:0/898556106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a68d350 msgr2=0x55b89a690a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaf5d9b640 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a68d350 0x55b89a690a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaf5d9b640 1 -- 172.21.15.104:0/898556106 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b89a691280 con 0x55b89a682a90 2024-06-09T22:22:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaf659c640 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a682170 0x55b89a695650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaf6d9d640 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a68d350 0x55b89a690a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaf5d9b640 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682a90 0x55b89a695b90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feaec0045f0 tx=0x7feaec000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2baa2e1734c7c7bc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaddffb640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:22:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaddffb640 1 -- 172.21.15.104:0/898556106 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec09ae90 con 0x55b89a682a90 2024-06-09T22:22:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaffde3740 1 -- 172.21.15.104:0/898556106 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaf000edd0 con 0x55b89a682a90 2024-06-09T22:22:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaddffb640 1 -- 172.21.15.104:0/898556106 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7feaec09a2e0 con 0x55b89a682a90 2024-06-09T22:22:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaddffb640 1 -- 172.21.15.104:0/898556106 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feaec09f240 con 0x55b89a682a90 2024-06-09T22:22:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaffde3740 10 client.25402 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.983+0000 7feaffde3740 1 -- 172.21.15.104:0/898556106 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b89a7077b0 con 0x55b89a682a90 2024-06-09T22:22:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.984+0000 7feaddffb640 1 -- 172.21.15.104:0/898556106 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaf000edd0 con 0x55b89a682a90 2024-06-09T22:22:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.985+0000 7feaddffb640 1 -- 172.21.15.104:0/898556106 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7feaec099d50 con 0x55b89a682a90 2024-06-09T22:22:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.985+0000 7feaffde3740 10 client.25402 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.985+0000 7feaffde3740 10 client.25402 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.985+0000 7feaffde3740 20 client.25402 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.985+0000 7feaffde3740 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b89a69d2c0 0x55b89a709f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.985+0000 7feaffde3740 4 client.25402 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:22:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.985+0000 7feaffde3740 1 -- 172.21.15.104:0/898556106 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b89a5f9f10 con 0x55b89a69d2c0 2024-06-09T22:22:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.985+0000 7feaf659c640 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b89a69d2c0 0x55b89a709f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.985+0000 7feaf659c640 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b89a69d2c0 0x55b89a709f40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=703d8694eda8e172 server_cookie=d8d03c51bb12d497 in_seq=0 out_seq=0 2024-06-09T22:22:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.985+0000 7feaddffb640 10 client.25402 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.993+0000 7feaddffb640 1 -- 172.21.15.104:0/898556106 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b89a5f9f10 con 0x55b89a69d2c0 2024-06-09T22:22:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.993+0000 7feaddffb640 10 client.25402 handle_command_reply: tid=0 2024-06-09T22:22:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.993+0000 7feaffde3740 1 client.25402 shutdown 2024-06-09T22:22:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.993+0000 7feaffde3740 1 -- 172.21.15.104:0/898556106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682a90 msgr2=0x55b89a695b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.993+0000 7feaffde3740 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682a90 0x55b89a695b90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feaec0045f0 tx=0x7feaec000bc0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.993+0000 7feaffde3740 1 -- 172.21.15.104:0/898556106 shutdown_connections 2024-06-09T22:22:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.993+0000 7feaffde3740 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b89a69d2c0 0x55b89a709f40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.993+0000 7feaffde3740 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b89a68d350 0x55b89a690a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.993+0000 7feaffde3740 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b89a682a90 0x55b89a695b90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.993+0000 7feaffde3740 1 --2- 172.21.15.104:0/898556106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b89a682170 0x55b89a695650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.993+0000 7feaddffb640 0 client.25402 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.993+0000 7feaffde3740 1 -- 172.21.15.104:0/898556106 >> 172.21.15.104:0/898556106 conn(0x55b89a67e680 msgr2=0x55b89a684320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feaffde3740 1 -- 172.21.15.104:0/898556106 shutdown_connections 2024-06-09T22:22:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feaffde3740 1 -- 172.21.15.104:0/898556106 wait complete. 2024-06-09T22:22:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feaffde3740 20 client.25402 trim_cache size 0 max 16384 2024-06-09T22:22:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:22:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:22:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:22:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feafcff9640 1 -- 172.21.15.104:0/2602672443 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7feae00a2f50 msgr2=0x7feae00a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feafcff9640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7feae00a2f50 0x7feae00a5410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feaf0008dc0 tx=0x7feaf0008570 comp rx=0 tx=0).stop 2024-06-09T22:22:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feafcff9640 1 -- 172.21.15.104:0/2602672443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaf8103550 msgr2=0x7feaf8198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feafcff9640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaf8103550 0x7feaf8198810 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feaec00d950 tx=0x7feaec00de10 comp rx=0 tx=0).stop 2024-06-09T22:22:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feafcff9640 1 -- 172.21.15.104:0/2602672443 shutdown_connections 2024-06-09T22:22:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feafcff9640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7feae00a2f50 0x7feae00a5410 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feafcff9640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feaf810bf70 0x7feaf819cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feafcff9640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaf8103550 0x7feaf8198810 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feafcff9640 1 --2- 172.21.15.104:0/2602672443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feaf8102c30 0x7feaf81982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.994+0000 7feafcff9640 1 -- 172.21.15.104:0/2602672443 >> 172.21.15.104:0/2602672443 conn(0x7feaf8077a20 msgr2=0x7feaf80fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.995+0000 7feafcff9640 1 -- 172.21.15.104:0/2602672443 shutdown_connections 2024-06-09T22:22:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:18.995+0000 7feafcff9640 1 -- 172.21.15.104:0/2602672443 wait complete. 2024-06-09T22:22:19.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.476+0000 7f879a41f700 1 -- 172.21.15.104:0/3416189162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f879406ab10 msgr2=0x7f879410f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:19.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.476+0000 7f879a41f700 1 --2- 172.21.15.104:0/3416189162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f879406ab10 0x7f879410f320 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f878c00b0d0 tx=0x7f878c00b3e0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.476+0000 7f879a41f700 1 -- 172.21.15.104:0/3416189162 shutdown_connections 2024-06-09T22:22:19.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.476+0000 7f879a41f700 1 --2- 172.21.15.104:0/3416189162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f879406ab10 0x7f879410f320 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.476+0000 7f879a41f700 1 --2- 172.21.15.104:0/3416189162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87941084e0 0x7f87941088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.476+0000 7f879a41f700 1 --2- 172.21.15.104:0/3416189162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f879406a170 0x7f879406a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.476+0000 7f879a41f700 1 -- 172.21.15.104:0/3416189162 >> 172.21.15.104:0/3416189162 conn(0x7f87940782c0 msgr2=0x7f87940786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:19.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.476+0000 7f879a41f700 1 -- 172.21.15.104:0/3416189162 shutdown_connections 2024-06-09T22:22:19.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.476+0000 7f879a41f700 1 -- 172.21.15.104:0/3416189162 wait complete. 2024-06-09T22:22:19.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.477+0000 7f879a41f700 1 Processor -- start 2024-06-09T22:22:19.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.477+0000 7f879a41f700 1 -- start start 2024-06-09T22:22:19.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.477+0000 7f879a41f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f879406a170 0x7f879419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.477+0000 7f879a41f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f879406ab10 0x7f879419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.477+0000 7f879a41f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87941084e0 0x7f8794198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.477+0000 7f879a41f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8794112730 con 0x7f87941084e0 2024-06-09T22:22:19.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.477+0000 7f879a41f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87941125b0 con 0x7f879406a170 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.477+0000 7f879a41f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f87941128b0 con 0x7f879406ab10 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.477+0000 7f87989bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87941084e0 0x7f8794198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.477+0000 7f87989bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87941084e0 0x7f8794198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58322/0 (socket says 172.21.15.104:58322) 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.477+0000 7f87989bc700 1 -- 172.21.15.104:0/3027456470 learned_addr learned my addr 172.21.15.104:0/3027456470 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.478+0000 7f87937fe700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f879406ab10 0x7f879419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.478+0000 7f8793fff700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f879406a170 0x7f879419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.478+0000 7f87937fe700 1 -- 172.21.15.104:0/3027456470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f879406a170 msgr2=0x7f879419e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.478+0000 7f87937fe700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f879406a170 0x7f879419e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.478+0000 7f87937fe700 1 -- 172.21.15.104:0/3027456470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87941084e0 msgr2=0x7f8794198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.478+0000 7f87937fe700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87941084e0 0x7f8794198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.478+0000 7f87937fe700 1 -- 172.21.15.104:0/3027456470 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8794199210 con 0x7f879406ab10 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.478+0000 7f87989bc700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87941084e0 0x7f8794198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.478+0000 7f87937fe700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f879406ab10 0x7f879419ee30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f878800d750 tx=0x7f878800da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:19.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.479+0000 7f87917fa700 1 -- 172.21.15.104:0/3027456470 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8788016070 con 0x7f879406ab10 2024-06-09T22:22:19.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.479+0000 7f879a41f700 1 -- 172.21.15.104:0/3027456470 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8794199500 con 0x7f879406ab10 2024-06-09T22:22:19.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.479+0000 7f879a41f700 1 -- 172.21.15.104:0/3027456470 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8794101390 con 0x7f879406ab10 2024-06-09T22:22:19.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.479+0000 7f87917fa700 1 -- 172.21.15.104:0/3027456470 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f878800fd70 con 0x7f879406ab10 2024-06-09T22:22:19.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.479+0000 7f87917fa700 1 -- 172.21.15.104:0/3027456470 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8788019360 con 0x7f879406ab10 2024-06-09T22:22:19.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.481+0000 7f87917fa700 1 -- 172.21.15.104:0/3027456470 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 24) ==== 138118+0+0 (secure 0 0 0) 0x7f8788019070 con 0x7f879406ab10 2024-06-09T22:22:19.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.482+0000 7f87917fa700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f877c0a2f40 0x7f877c0a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.482+0000 7f87a4ca2b80 1 Processor -- start 2024-06-09T22:22:19.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f87a4ca2b80 1 -- start start 2024-06-09T22:22:19.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f87917fa700 1 -- 172.21.15.104:0/3027456470 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f87880cf4b0 con 0x7f879406ab10 2024-06-09T22:22:19.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f87a4ca2b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 0x557f528c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f87a4ca2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c7fe0 0x557f528c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f87a4ca2b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c1fa0 0x557f528c23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f87a4ca2b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557f52813890 con 0x557f528c76c0 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f87a4ca2b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557f527bb7a0 con 0x557f528c1fa0 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f87a4ca2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557f5280f850 con 0x557f528c7fe0 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f8793fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 0x557f528c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f8793fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 0x557f528c7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58330/0 (socket says 172.21.15.104:58330) 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f8793fff700 1 -- 172.21.15.104:0/1336248814 learned_addr learned my addr 172.21.15.104:0/1336248814 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f8793fff700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f877c0a2f40 0x7f877c0a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f87989bc700 1 --2- 172.21.15.104:0/1336248814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c1fa0 0x557f528c23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.483+0000 7f87937fe700 1 --2- 172.21.15.104:0/1336248814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c7fe0 0x557f528c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.484+0000 7f87989bc700 1 -- 172.21.15.104:0/1336248814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c7fe0 msgr2=0x557f528c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.484+0000 7f87989bc700 1 --2- 172.21.15.104:0/1336248814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c7fe0 0x557f528c1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.484+0000 7f87989bc700 1 -- 172.21.15.104:0/1336248814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 msgr2=0x557f528c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:19.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.484+0000 7f87989bc700 1 --2- 172.21.15.104:0/1336248814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 0x557f528c7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.484+0000 7f87989bc700 1 -- 172.21.15.104:0/1336248814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f527b1fd0 con 0x557f528c1fa0 2024-06-09T22:22:19.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.484+0000 7f87989bc700 1 --2- 172.21.15.104:0/1336248814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c1fa0 0x557f528c23c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f878c0062a0 tx=0x7f878c006210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=787a97178ec2771e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:19.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.484+0000 7f8793fff700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f877c0a2f40 0x7f877c0a5400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8784013650 tx=0x7f878400b720 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:19.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.484+0000 7f877a7fc700 1 -- 172.21.15.104:0/1336248814 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c021d30 con 0x557f528c1fa0 2024-06-09T22:22:19.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.484+0000 7f877a7fc700 1 -- 172.21.15.104:0/1336248814 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f878c0074e0 con 0x557f528c1fa0 2024-06-09T22:22:19.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.484+0000 7f877a7fc700 1 -- 172.21.15.104:0/1336248814 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c0052a0 con 0x557f528c1fa0 2024-06-09T22:22:19.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.485+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1336248814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c1fa0 msgr2=0x557f528c23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:19.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.485+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1336248814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c1fa0 0x557f528c23c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f878c0062a0 tx=0x7f878c006210 comp rx=0 tx=0).stop 2024-06-09T22:22:19.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.485+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1336248814 shutdown_connections 2024-06-09T22:22:19.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.485+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1336248814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 0x557f528c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.485+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1336248814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c1fa0 0x557f528c23c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.485+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1336248814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c7fe0 0x557f528c1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.485+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1336248814 >> 172.21.15.104:0/1336248814 conn(0x557f527ecd40 msgr2=0x557f528bc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.485+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1336248814 shutdown_connections 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.485+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1336248814 wait complete. 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.486+0000 7f87a4ca2b80 1 -- start start 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.486+0000 7f87a4ca2b80 1 Processor -- start 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.486+0000 7f87a4ca2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c1fa0 0x557f528cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.486+0000 7f87a4ca2b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 0x557f528cb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.486+0000 7f87a4ca2b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c7fe0 0x557f528cb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87a4ca2b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557f528ce9f0 con 0x557f528c76c0 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f8793fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c1fa0 0x557f528cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87a4ca2b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557f527b3c70 con 0x557f528c7fe0 2024-06-09T22:22:19.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87937fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 0x557f528cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87a4ca2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557f528ceb70 con 0x557f528c1fa0 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f8793fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c1fa0 0x557f528cdd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:42004/0 (socket says 172.21.15.104:42004) 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f8793fff700 1 -- 172.21.15.104:0/1792618699 learned_addr learned my addr 172.21.15.104:0/1792618699 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87937fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 0x557f528cb050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58334/0 (socket says 172.21.15.104:58334) 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87989bc700 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c7fe0 0x557f528cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87a4ca2b80 10 client.0 _ll_register_callbacks cb 0x557f527ad9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87989bc700 1 -- 172.21.15.104:0/1792618699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c1fa0 msgr2=0x557f528cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87989bc700 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c1fa0 0x557f528cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87989bc700 1 -- 172.21.15.104:0/1792618699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 msgr2=0x557f528cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87989bc700 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 0x557f528cb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87989bc700 1 -- 172.21.15.104:0/1792618699 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f52762950 con 0x557f528c7fe0 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.487+0000 7f87937fe700 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 0x557f528cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.488+0000 7f87989bc700 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c7fe0 0x557f528cb590 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f878c007ee0 tx=0x7f878c0078c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=816f7505f31bf466 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:19.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.488+0000 7f87737fe700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:22:19.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.488+0000 7f87737fe700 1 -- 172.21.15.104:0/1792618699 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c031070 con 0x557f528c7fe0 2024-06-09T22:22:19.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.488+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1792618699 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f878c0074e0 con 0x557f528c7fe0 2024-06-09T22:22:19.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.488+0000 7f87737fe700 1 -- 172.21.15.104:0/1792618699 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f878c02b440 con 0x557f528c7fe0 2024-06-09T22:22:19.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.488+0000 7f87737fe700 1 -- 172.21.15.104:0/1792618699 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f878c009480 con 0x557f528c7fe0 2024-06-09T22:22:19.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.488+0000 7f87a4ca2b80 10 client.25304 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:19.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.488+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1792618699 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557f528ce9f0 con 0x557f528c7fe0 2024-06-09T22:22:19.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.488+0000 7f87737fe700 1 -- 172.21.15.104:0/1792618699 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f878c0074e0 con 0x557f528c7fe0 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.488+0000 7f87737fe700 1 -- 172.21.15.104:0/1792618699 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f878c029070 con 0x557f528c7fe0 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.489+0000 7f87a4ca2b80 10 client.25304 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.489+0000 7f87a4ca2b80 10 client.25304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.489+0000 7f87a4ca2b80 20 client.25304 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.489+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f528db610 0x557f528dda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.489+0000 7f87a4ca2b80 4 client.25304 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.489+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1792618699 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f52820d90 con 0x557f528db610 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.489+0000 7f8793fff700 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f528db610 0x557f528dda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.489+0000 7f8793fff700 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f528db610 0x557f528dda00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9a31730fe05825c server_cookie=1711e57e539c6b62 in_seq=0 out_seq=0 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.489+0000 7f87737fe700 10 client.25304 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.490+0000 7f87737fe700 1 -- 172.21.15.104:0/1792618699 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x557f52820d90 con 0x557f528db610 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.490+0000 7f87737fe700 10 client.25304 handle_command_reply: tid=0 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.490+0000 7f87a4ca2b80 1 client.25304 shutdown 2024-06-09T22:22:19.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.490+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1792618699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c7fe0 msgr2=0x557f528cb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.490+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c7fe0 0x557f528cb590 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f878c007ee0 tx=0x7f878c0078c0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.491+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1792618699 shutdown_connections 2024-06-09T22:22:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.491+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f528db610 0x557f528dda00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.491+0000 7f87737fe700 0 client.25304 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.491+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c7fe0 0x557f528cb590 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.491+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c1fa0 0x557f528cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.491+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1792618699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c76c0 0x557f528cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.491+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1792618699 >> 172.21.15.104:0/1792618699 conn(0x557f527ecd40 msgr2=0x557f528c0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.491+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1792618699 shutdown_connections 2024-06-09T22:22:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.491+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1792618699 wait complete. 2024-06-09T22:22:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.491+0000 7f87a4ca2b80 20 client.25304 trim_cache size 0 max 16384 2024-06-09T22:22:19.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.512+0000 7f87a4ca2b80 1 Processor -- start 2024-06-09T22:22:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.512+0000 7f87a4ca2b80 1 -- start start 2024-06-09T22:22:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.512+0000 7f87a4ca2b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 0x557f528c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.512+0000 7f87a4ca2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c76c0 0x557f528c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.512+0000 7f87a4ca2b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c7fe0 0x557f528ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.512+0000 7f87a4ca2b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f878000ec70 con 0x557f528c1fa0 2024-06-09T22:22:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.512+0000 7f87a4ca2b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87800107b0 con 0x557f528c7fe0 2024-06-09T22:22:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.512+0000 7f87a4ca2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f878000f340 con 0x557f528c76c0 2024-06-09T22:22:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.512+0000 7f8793fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 0x557f528c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.512+0000 7f8793fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 0x557f528c1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58348/0 (socket says 172.21.15.104:58348) 2024-06-09T22:22:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.512+0000 7f8793fff700 1 -- 172.21.15.104:0/1443011390 learned_addr learned my addr 172.21.15.104:0/1443011390 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.512+0000 7f87937fe700 1 --2- 172.21.15.104:0/1443011390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c76c0 0x557f528c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.513+0000 7f87989bc700 1 --2- 172.21.15.104:0/1443011390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c7fe0 0x557f528ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.513+0000 7f8793fff700 1 -- 172.21.15.104:0/1443011390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c76c0 msgr2=0x557f528c8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.513+0000 7f8793fff700 1 --2- 172.21.15.104:0/1443011390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c76c0 0x557f528c8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.513+0000 7f8793fff700 1 -- 172.21.15.104:0/1443011390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c7fe0 msgr2=0x557f528ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.513+0000 7f8793fff700 1 --2- 172.21.15.104:0/1443011390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c7fe0 0x557f528ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.513+0000 7f8793fff700 1 -- 172.21.15.104:0/1443011390 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f528dddc0 con 0x557f528c1fa0 2024-06-09T22:22:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.513+0000 7f87989bc700 1 --2- 172.21.15.104:0/1443011390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c7fe0 0x557f528ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.513+0000 7f8793fff700 1 --2- 172.21.15.104:0/1443011390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 0x557f528c1a60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8784013fd0 tx=0x7f878401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a590060e8ebde2b5 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.513+0000 7f8779ffb700 1 -- 172.21.15.104:0/1443011390 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8784005680 con 0x557f528c1fa0 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.513+0000 7f8779ffb700 1 -- 172.21.15.104:0/1443011390 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8784004500 con 0x557f528c1fa0 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.513+0000 7f8779ffb700 1 -- 172.21.15.104:0/1443011390 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8784003c30 con 0x557f528c1fa0 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.514+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1443011390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 msgr2=0x557f528c1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.514+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1443011390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 0x557f528c1a60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8784013fd0 tx=0x7f878401b000 comp rx=0 tx=0).stop 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.514+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1443011390 shutdown_connections 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.514+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1443011390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 0x557f528c1a60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.514+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1443011390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c7fe0 0x557f528ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.514+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1443011390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c76c0 0x557f528c8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.514+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1443011390 >> 172.21.15.104:0/1443011390 conn(0x557f527ecd40 msgr2=0x557f528b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.514+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1443011390 shutdown_connections 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.514+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1443011390 wait complete. 2024-06-09T22:22:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f87a4ca2b80 1 -- start start 2024-06-09T22:22:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f87a4ca2b80 1 Processor -- start 2024-06-09T22:22:19.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f87a4ca2b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 0x557f528e2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f87a4ca2b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c76c0 0x557f528e33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f87a4ca2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c7fe0 0x557f528e3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f87a4ca2b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f87800100e0 con 0x557f528c1fa0 2024-06-09T22:22:19.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f87a4ca2b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f878000fa10 con 0x557f528c76c0 2024-06-09T22:22:19.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f87a4ca2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f878000cbf0 con 0x557f528c7fe0 2024-06-09T22:22:19.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f87a4ca2b80 10 client.0 _ll_register_callbacks cb 0x557f527ad9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:19.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f8793fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 0x557f528e2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f8793fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 0x557f528e2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58364/0 (socket says 172.21.15.104:58364) 2024-06-09T22:22:19.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f8793fff700 1 -- 172.21.15.104:0/1633895978 learned_addr learned my addr 172.21.15.104:0/1633895978 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:19.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.515+0000 7f87989bc700 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c7fe0 0x557f528e3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.516+0000 7f87937fe700 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c76c0 0x557f528e33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.516+0000 7f8793fff700 1 -- 172.21.15.104:0/1633895978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c7fe0 msgr2=0x557f528e3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:19.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.516+0000 7f8793fff700 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c7fe0 0x557f528e3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.516+0000 7f8793fff700 1 -- 172.21.15.104:0/1633895978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c76c0 msgr2=0x557f528e33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:19.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.516+0000 7f8793fff700 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c76c0 0x557f528e33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.516+0000 7f8793fff700 1 -- 172.21.15.104:0/1633895978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f529452a0 con 0x557f528c1fa0 2024-06-09T22:22:19.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.516+0000 7f8793fff700 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 0x557f528e2eb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f87840039b0 tx=0x7f8784005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0e3180b572d3e3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:19.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.516+0000 7f87737fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:19.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.516+0000 7f87737fe700 1 -- 172.21.15.104:0/1633895978 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8784004e30 con 0x557f528c1fa0 2024-06-09T22:22:19.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.517+0000 7f87737fe700 1 -- 172.21.15.104:0/1633895978 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8784025040 con 0x557f528c1fa0 2024-06-09T22:22:19.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.517+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1633895978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8784004500 con 0x557f528c1fa0 2024-06-09T22:22:19.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.517+0000 7f87737fe700 1 -- 172.21.15.104:0/1633895978 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8784003680 con 0x557f528c1fa0 2024-06-09T22:22:19.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.517+0000 7f87a4ca2b80 10 client.15843 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:19.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.517+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1633895978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557f529464f0 con 0x557f528c1fa0 2024-06-09T22:22:19.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.517+0000 7f87737fe700 1 -- 172.21.15.104:0/1633895978 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8784004500 con 0x557f528c1fa0 2024-06-09T22:22:19.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.518+0000 7f87737fe700 1 -- 172.21.15.104:0/1633895978 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8784003a10 con 0x557f528c1fa0 2024-06-09T22:22:19.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.518+0000 7f87a4ca2b80 10 client.15843 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:19.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.518+0000 7f87a4ca2b80 10 client.15843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:19.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.518+0000 7f87a4ca2b80 20 client.15843 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:19.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.518+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f52946bc0 0x557f52948fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:19.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.518+0000 7f87a4ca2b80 4 client.15843 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-09T22:22:19.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.518+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1633895978 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557f5281b310 con 0x557f52946bc0 2024-06-09T22:22:19.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.519+0000 7f87937fe700 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f52946bc0 0x557f52948fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:19.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.519+0000 7f87937fe700 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f52946bc0 0x557f52948fe0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e24df1b5cd3b793b server_cookie=5394638117d1229e in_seq=0 out_seq=0 2024-06-09T22:22:19.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.519+0000 7f87737fe700 10 client.15843 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:19.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.520+0000 7f87737fe700 1 -- 172.21.15.104:0/1633895978 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557f5281b310 con 0x557f52946bc0 2024-06-09T22:22:19.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.520+0000 7f87737fe700 10 client.15843 handle_command_reply: tid=0 2024-06-09T22:22:19.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.520+0000 7f87a4ca2b80 1 client.15843 shutdown 2024-06-09T22:22:19.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.521+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1633895978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 msgr2=0x557f528e2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:19.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.521+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 0x557f528e2eb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f87840039b0 tx=0x7f8784005680 comp rx=0 tx=0).stop 2024-06-09T22:22:19.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.521+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1633895978 shutdown_connections 2024-06-09T22:22:19.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.521+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f52946bc0 0x557f52948fe0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.521+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f528c1fa0 0x557f528e2eb0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.523 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:22:19.523 INFO:teuthology.orchestra.run.smithi104.stdout: "return_code": 0, 2024-06-09T22:22:19.523 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_tag": "9ce7e540-ade9-462b-960e-178a2656dab2", 2024-06-09T22:22:19.523 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": "asynchronous" 2024-06-09T22:22:19.524 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:22:19.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.521+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f528c7fe0 0x557f528e3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.521+0000 7f87a4ca2b80 1 --2- 172.21.15.104:0/1633895978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f528c76c0 0x557f528e33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.521+0000 7f87737fe700 0 client.15843 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:19.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.521+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1633895978 >> 172.21.15.104:0/1633895978 conn(0x557f527ecd40 msgr2=0x557f528c8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:19.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.521+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1633895978 shutdown_connections 2024-06-09T22:22:19.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.521+0000 7f87a4ca2b80 1 -- 172.21.15.104:0/1633895978 wait complete. 2024-06-09T22:22:19.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.521+0000 7f87a4ca2b80 20 client.15843 trim_cache size 0 max 16384 2024-06-09T22:22:19.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.522+0000 7f879a41f700 1 -- 172.21.15.104:0/3027456470 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f877c0a2f40 msgr2=0x7f877c0a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:19.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.522+0000 7f879a41f700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f877c0a2f40 0x7f877c0a5400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8784013650 tx=0x7f878400b720 comp rx=0 tx=0).stop 2024-06-09T22:22:19.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.522+0000 7f879a41f700 1 -- 172.21.15.104:0/3027456470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f879406ab10 msgr2=0x7f879419ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:19.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.522+0000 7f879a41f700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f879406ab10 0x7f879419ee30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f878800d750 tx=0x7f878800da60 comp rx=0 tx=0).stop 2024-06-09T22:22:19.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.522+0000 7f879a41f700 1 -- 172.21.15.104:0/3027456470 shutdown_connections 2024-06-09T22:22:19.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.522+0000 7f879a41f700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f877c0a2f40 0x7f877c0a5400 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.522+0000 7f879a41f700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87941084e0 0x7f8794198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.522+0000 7f879a41f700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f879406a170 0x7f879419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.522+0000 7f879a41f700 1 --2- 172.21.15.104:0/3027456470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f879406ab10 0x7f879419ee30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:19.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.522+0000 7f879a41f700 1 -- 172.21.15.104:0/3027456470 >> 172.21.15.104:0/3027456470 conn(0x7f87940782c0 msgr2=0x7f87940ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:19.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.522+0000 7f879a41f700 1 -- 172.21.15.104:0/3027456470 shutdown_connections 2024-06-09T22:22:19.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:19.522+0000 7f879a41f700 1 -- 172.21.15.104:0/3027456470 wait complete. 2024-06-09T22:22:19.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:19 smithi104 ceph-mon[32389]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-06-09T22:22:19.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:19 smithi194 ceph-mon[36856]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-06-09T22:22:19.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:19 smithi153 ceph-mon[36599]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-06-09T22:22:20.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:20 smithi104 ceph-mon[32389]: scrub queued for path: / 2024-06-09T22:22:20.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:20 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:22:20.757 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:20 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:22:20.757 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:20 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:22:20.757 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:20 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:22:20.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:20 smithi194 ceph-mon[36856]: scrub queued for path: / 2024-06-09T22:22:20.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:20 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:22:20.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:20 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:22:20.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:20 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:22:20.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:20 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:22:20.814 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:22:20.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:20 smithi153 ceph-mon[36599]: scrub queued for path: / 2024-06-09T22:22:20.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:20 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:22:20.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:20 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:22:20.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:20 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:22:20.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:20 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:22:21.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:21 smithi194 ceph-mon[36856]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.2 KiB/s wr, 14 op/s 2024-06-09T22:22:21.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:21 smithi153 ceph-mon[36599]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.2 KiB/s wr, 14 op/s 2024-06-09T22:22:22.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:21 smithi104 ceph-mon[32389]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.2 KiB/s wr, 14 op/s 2024-06-09T22:22:23.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:23 smithi194 ceph-mon[36856]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-06-09T22:22:23.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:23 smithi153 ceph-mon[36599]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-06-09T22:22:24.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:23 smithi104 ceph-mon[32389]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-06-09T22:22:24.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.720+0000 7f59ef855700 1 -- 172.21.15.104:0/3147048733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e81073f0 msgr2=0x7f59e81077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:24.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.720+0000 7f59ef855700 1 --2- 172.21.15.104:0/3147048733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e81073f0 0x7f59e81077d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009990 tx=0x7f59e4009ca0 comp rx=0 tx=0).stop 2024-06-09T22:22:24.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.720+0000 7f59ef855700 1 -- 172.21.15.104:0/3147048733 shutdown_connections 2024-06-09T22:22:24.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.720+0000 7f59ef855700 1 --2- 172.21.15.104:0/3147048733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e81073f0 0x7f59e81077d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:24.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.720+0000 7f59ef855700 1 --2- 172.21.15.104:0/3147048733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59e8101d40 0x7f59e810e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:24.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.720+0000 7f59ef855700 1 --2- 172.21.15.104:0/3147048733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e8101380 0x7f59e8101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:24.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.720+0000 7f59ef855700 1 -- 172.21.15.104:0/3147048733 >> 172.21.15.104:0/3147048733 conn(0x7f59e8078300 msgr2=0x7f59e8078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:24.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.720+0000 7f59ef855700 1 -- 172.21.15.104:0/3147048733 shutdown_connections 2024-06-09T22:22:24.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.721+0000 7f59ef855700 1 -- 172.21.15.104:0/3147048733 wait complete. 2024-06-09T22:22:24.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.721+0000 7f59ef855700 1 Processor -- start 2024-06-09T22:22:24.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.721+0000 7f59ef855700 1 -- start start 2024-06-09T22:22:24.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.722+0000 7f59ef855700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e8101380 0x7f59e819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:24.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.722+0000 7f59ef855700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e8101d40 0x7f59e819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:24.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.722+0000 7f59ef855700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59e81073f0 0x7f59e81967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:24.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.722+0000 7f59ef855700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f59e8110570 con 0x7f59e8101d40 2024-06-09T22:22:24.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.722+0000 7f59ecdf0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e8101d40 0x7f59e819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:24.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.722+0000 7f59ed5f1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e8101380 0x7f59e819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:24.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.722+0000 7f59ecdf0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e8101d40 0x7f59e819cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58384/0 (socket says 172.21.15.104:58384) 2024-06-09T22:22:24.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.722+0000 7f59ecdf0700 1 -- 172.21.15.104:0/3281046344 learned_addr learned my addr 172.21.15.104:0/3281046344 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:24.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.722+0000 7f59eddf2700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59e81073f0 0x7f59e81967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:24.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.722+0000 7f59ef855700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59e81103f0 con 0x7f59e8101380 2024-06-09T22:22:24.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.722+0000 7f59ef855700 1 -- 172.21.15.104:0/3281046344 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f59e81106f0 con 0x7f59e81073f0 2024-06-09T22:22:24.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.723+0000 7f59ecdf0700 1 -- 172.21.15.104:0/3281046344 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59e81073f0 msgr2=0x7f59e81967f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:24.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.723+0000 7f59ecdf0700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59e81073f0 0x7f59e81967f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:24.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.723+0000 7f59ecdf0700 1 -- 172.21.15.104:0/3281046344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e8101380 msgr2=0x7f59e819c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:24.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.723+0000 7f59ecdf0700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e8101380 0x7f59e819c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:24.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.723+0000 7f59ecdf0700 1 -- 172.21.15.104:0/3281046344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e8196fd0 con 0x7f59e8101d40 2024-06-09T22:22:24.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.723+0000 7f59ed5f1700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e8101380 0x7f59e819c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:24.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.723+0000 7f59ecdf0700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e8101d40 0x7f59e819cbb0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f59d8010de0 tx=0x7f59d800e3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:24.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.723+0000 7f59de7fc700 1 -- 172.21.15.104:0/3281046344 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d801a020 con 0x7f59e8101d40 2024-06-09T22:22:24.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.723+0000 7f59de7fc700 1 -- 172.21.15.104:0/3281046344 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59d8016480 con 0x7f59e8101d40 2024-06-09T22:22:24.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.724+0000 7f59de7fc700 1 -- 172.21.15.104:0/3281046344 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d8015640 con 0x7f59e8101d40 2024-06-09T22:22:24.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.724+0000 7f59ef855700 1 -- 172.21.15.104:0/3281046344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e81972c0 con 0x7f59e8101d40 2024-06-09T22:22:24.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.725+0000 7f59ef855700 1 -- 172.21.15.104:0/3281046344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e81a34d0 con 0x7f59e8101d40 2024-06-09T22:22:24.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.726+0000 7f59ef855700 1 -- 172.21.15.104:0/3281046344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e8102aa0 con 0x7f59e8101d40 2024-06-09T22:22:24.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.727+0000 7f59de7fc700 1 -- 172.21.15.104:0/3281046344 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 24) ==== 138118+0+0 (secure 0 0 0) 0x7f59d801d020 con 0x7f59e8101d40 2024-06-09T22:22:24.652 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:22:24.652 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:20:55.680750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24460,24448,24442,24403,24419]},"id":1}]} 2024-06-09T22:22:24.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.728+0000 7f59de7fc700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f59d40a2ea0 0x7f59d40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:24.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.728+0000 7f59ed5f1700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f59d40a2ea0 0x7f59d40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:24.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.728+0000 7f59de7fc700 1 -- 172.21.15.104:0/3281046344 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f59d80d13f0 con 0x7f59e8101d40 2024-06-09T22:22:24.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.728+0000 7f59ed5f1700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f59d40a2ea0 0x7f59d40a5360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f59e400b2e0 tx=0x7f59e4005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:24.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:23.736+0000 7f59de7fc700 1 -- 172.21.15.104:0/3281046344 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f59d8024020 con 0x7f59e8101d40 2024-06-09T22:22:24.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.153+0000 7f59ef855700 1 -- 172.21.15.104:0/3281046344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59e8069180 con 0x7f59e8101d40 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.154+0000 7f59de7fc700 1 -- 172.21.15.104:0/3281046344 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11953 (secure 0 0 0) 0x7f59d8012040 con 0x7f59e8101d40 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.157+0000 7f59ef855700 1 -- 172.21.15.104:0/3281046344 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f59d40a2ea0 msgr2=0x7f59d40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.157+0000 7f59ef855700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f59d40a2ea0 0x7f59d40a5360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f59e400b2e0 tx=0x7f59e4005960 comp rx=0 tx=0).stop 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.157+0000 7f59ef855700 1 -- 172.21.15.104:0/3281046344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e8101d40 msgr2=0x7f59e819cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.157+0000 7f59ef855700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e8101d40 0x7f59e819cbb0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f59d8010de0 tx=0x7f59d800e3a0 comp rx=0 tx=0).stop 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.157+0000 7f59ef855700 1 -- 172.21.15.104:0/3281046344 shutdown_connections 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.157+0000 7f59ef855700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f59d40a2ea0 0x7f59d40a5360 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.157+0000 7f59ef855700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59e81073f0 0x7f59e81967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.157+0000 7f59ef855700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e8101380 0x7f59e819c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.157+0000 7f59ef855700 1 --2- 172.21.15.104:0/3281046344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e8101d40 0x7f59e819cbb0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.157+0000 7f59ef855700 1 -- 172.21.15.104:0/3281046344 >> 172.21.15.104:0/3281046344 conn(0x7f59e8078300 msgr2=0x7f59e81000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.157+0000 7f59ef855700 1 -- 172.21.15.104:0/3281046344 shutdown_connections 2024-06-09T22:22:24.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:24.157+0000 7f59ef855700 1 -- 172.21.15.104:0/3281046344 wait complete. 2024-06-09T22:22:24.654 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 29 2024-06-09T22:22:24.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:24 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3281046344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:22:24.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:24 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3281046344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:22:25.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:24 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3281046344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:22:25.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:25 smithi194 ceph-mon[36856]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.5 KiB/s wr, 15 op/s 2024-06-09T22:22:25.861 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:25 smithi104 ceph-mon[32389]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.5 KiB/s wr, 15 op/s 2024-06-09T22:22:25.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:25 smithi153 ceph-mon[36599]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.5 KiB/s wr, 15 op/s 2024-06-09T22:22:26.329 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:26.341 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24460 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-06-09T22:22:26.342 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24460 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-06-09T22:22:27.643 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:27.797 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:27 smithi104 ceph-mon[32389]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-06-09T22:22:27.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:27 smithi194 ceph-mon[36856]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-06-09T22:22:27.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:27 smithi153 ceph-mon[36599]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-06-09T22:22:28.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.870+0000 7fd835c8a700 1 -- 172.21.15.104:0/782244941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8301073f0 msgr2=0x7fd8301077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:28.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.870+0000 7fd835c8a700 1 --2- 172.21.15.104:0/782244941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8301073f0 0x7fd8301077d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd820009990 tx=0x7fd820009ca0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.870+0000 7fd835c8a700 1 -- 172.21.15.104:0/782244941 shutdown_connections 2024-06-09T22:22:28.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.870+0000 7fd835c8a700 1 --2- 172.21.15.104:0/782244941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8301073f0 0x7fd8301077d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.870+0000 7fd835c8a700 1 --2- 172.21.15.104:0/782244941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd830101d40 0x7fd83010e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.870+0000 7fd835c8a700 1 --2- 172.21.15.104:0/782244941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd830101380 0x7fd830101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.870+0000 7fd835c8a700 1 -- 172.21.15.104:0/782244941 >> 172.21.15.104:0/782244941 conn(0x7fd830078300 msgr2=0x7fd830078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:28.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.870+0000 7fd835c8a700 1 -- 172.21.15.104:0/782244941 shutdown_connections 2024-06-09T22:22:28.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.870+0000 7fd835c8a700 1 -- 172.21.15.104:0/782244941 wait complete. 2024-06-09T22:22:28.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.872+0000 7fd835c8a700 1 Processor -- start 2024-06-09T22:22:28.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.872+0000 7fd835c8a700 1 -- start start 2024-06-09T22:22:28.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.873+0000 7fd835c8a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd830101380 0x7fd83019e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.873+0000 7fd835c8a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd830101d40 0x7fd83019ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.873+0000 7fd835c8a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8301073f0 0x7fd830198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.873+0000 7fd835c8a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd830112780 con 0x7fd8301073f0 2024-06-09T22:22:28.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.873+0000 7fd835c8a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd830112600 con 0x7fd830101380 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.873+0000 7fd835c8a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd830112900 con 0x7fd830101d40 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.873+0000 7fd82f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd830101380 0x7fd83019e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.873+0000 7fd82f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd830101380 0x7fd83019e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:60436/0 (socket says 172.21.15.104:60436) 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.873+0000 7fd82f7fe700 1 -- 172.21.15.104:0/2461247813 learned_addr learned my addr 172.21.15.104:0/2461247813 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.873+0000 7fd82ffff700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8301073f0 0x7fd830198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.873+0000 7fd82effd700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd830101d40 0x7fd83019ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.874+0000 7fd82ffff700 1 -- 172.21.15.104:0/2461247813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd830101d40 msgr2=0x7fd83019ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.874+0000 7fd82ffff700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd830101d40 0x7fd83019ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.874+0000 7fd82ffff700 1 -- 172.21.15.104:0/2461247813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd830101380 msgr2=0x7fd83019e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.874+0000 7fd82ffff700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd830101380 0x7fd83019e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.874+0000 7fd82ffff700 1 -- 172.21.15.104:0/2461247813 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8301990c0 con 0x7fd8301073f0 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.874+0000 7fd82effd700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd830101d40 0x7fd83019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.874+0000 7fd82ffff700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8301073f0 0x7fd830198960 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd828007f10 tx=0x7fd82800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:28.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.874+0000 7fd82cff9700 1 -- 172.21.15.104:0/2461247813 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd828004db0 con 0x7fd8301073f0 2024-06-09T22:22:28.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.875+0000 7fd82cff9700 1 -- 172.21.15.104:0/2461247813 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd828009e20 con 0x7fd8301073f0 2024-06-09T22:22:28.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.875+0000 7fd835c8a700 1 -- 172.21.15.104:0/2461247813 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8301993b0 con 0x7fd8301073f0 2024-06-09T22:22:28.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.875+0000 7fd82cff9700 1 -- 172.21.15.104:0/2461247813 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd828015400 con 0x7fd8301073f0 2024-06-09T22:22:28.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.876+0000 7fd835c8a700 1 -- 172.21.15.104:0/2461247813 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8301a5670 con 0x7fd8301073f0 2024-06-09T22:22:28.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.878+0000 7fd82cff9700 1 -- 172.21.15.104:0/2461247813 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 24) ==== 138118+0+0 (secure 0 0 0) 0x7fd82800db20 con 0x7fd8301073f0 2024-06-09T22:22:28.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.879+0000 7fd82cff9700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd8180a2e20 0x7fd8180a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.879+0000 7fd82f7fe700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd8180a2e20 0x7fd8180a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.879+0000 7fd82cff9700 1 -- 172.21.15.104:0/2461247813 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fd8280cf2a0 con 0x7fd8301073f0 2024-06-09T22:22:28.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.880+0000 7fd82f7fe700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd8180a2e20 0x7fd8180a52e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd820009750 tx=0x7fd820005bc0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:28.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.880+0000 7fd84050db80 1 Processor -- start 2024-06-09T22:22:28.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.880+0000 7fd84050db80 1 -- start start 2024-06-09T22:22:28.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.880+0000 7fd84050db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 0x55786d7b2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd84050db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b2880 0x55786d7ac260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd84050db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7ac7a0 0x55786d7acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd84050db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55786d69ef60 con 0x55786d7b1f60 2024-06-09T22:22:28.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd84050db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55786d6fd850 con 0x55786d7ac7a0 2024-06-09T22:22:28.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd84050db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55786d665460 con 0x55786d7b2880 2024-06-09T22:22:28.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd82f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 0x55786d7b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd82f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 0x55786d7b2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53366/0 (socket says 172.21.15.104:53366) 2024-06-09T22:22:28.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd82f7fe700 1 -- 172.21.15.104:0/675999499 learned_addr learned my addr 172.21.15.104:0/675999499 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:28.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd82ffff700 1 --2- 172.21.15.104:0/675999499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7ac7a0 0x55786d7acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd82f7fe700 1 -- 172.21.15.104:0/675999499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b2880 msgr2=0x55786d7ac260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-09T22:22:28.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd82f7fe700 1 --2- 172.21.15.104:0/675999499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b2880 0x55786d7ac260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd82f7fe700 1 -- 172.21.15.104:0/675999499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7ac7a0 msgr2=0x55786d7acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:28.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd82f7fe700 1 --2- 172.21.15.104:0/675999499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7ac7a0 0x55786d7acbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd82f7fe700 1 -- 172.21.15.104:0/675999499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55786d64c8d0 con 0x55786d7b1f60 2024-06-09T22:22:28.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.881+0000 7fd82ffff700 1 --2- 172.21.15.104:0/675999499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7ac7a0 0x55786d7acbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:28.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.882+0000 7fd82f7fe700 1 --2- 172.21.15.104:0/675999499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 0x55786d7b2340 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd8200098e0 tx=0x7fd820020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3d4310a109e745 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:28.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.882+0000 7fd815ffb700 1 -- 172.21.15.104:0/675999499 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82000f820 con 0x55786d7b1f60 2024-06-09T22:22:28.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.882+0000 7fd815ffb700 1 -- 172.21.15.104:0/675999499 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd820031070 con 0x55786d7b1f60 2024-06-09T22:22:28.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.882+0000 7fd815ffb700 1 -- 172.21.15.104:0/675999499 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd820035410 con 0x55786d7b1f60 2024-06-09T22:22:28.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.883+0000 7fd84050db80 1 -- 172.21.15.104:0/675999499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 msgr2=0x55786d7b2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:28.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.883+0000 7fd84050db80 1 --2- 172.21.15.104:0/675999499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 0x55786d7b2340 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd8200098e0 tx=0x7fd820020000 comp rx=0 tx=0).stop 2024-06-09T22:22:28.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.883+0000 7fd84050db80 1 -- 172.21.15.104:0/675999499 shutdown_connections 2024-06-09T22:22:28.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.883+0000 7fd84050db80 1 --2- 172.21.15.104:0/675999499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 0x55786d7b2340 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.883+0000 7fd84050db80 1 --2- 172.21.15.104:0/675999499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7ac7a0 0x55786d7acbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.883+0000 7fd84050db80 1 --2- 172.21.15.104:0/675999499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b2880 0x55786d7ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.883+0000 7fd84050db80 1 -- 172.21.15.104:0/675999499 >> 172.21.15.104:0/675999499 conn(0x55786d6d9080 msgr2=0x55786d7a7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:28.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.884+0000 7fd84050db80 1 -- 172.21.15.104:0/675999499 shutdown_connections 2024-06-09T22:22:28.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.884+0000 7fd84050db80 1 -- 172.21.15.104:0/675999499 wait complete. 2024-06-09T22:22:28.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.884+0000 7fd84050db80 1 -- start start 2024-06-09T22:22:28.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.884+0000 7fd84050db80 1 Processor -- start 2024-06-09T22:22:28.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.885+0000 7fd84050db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7ac7a0 0x55786d7b8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.885+0000 7fd84050db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 0x55786d7b5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.885+0000 7fd84050db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b2880 0x55786d7b5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.885+0000 7fd84050db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55786d69dfc0 con 0x55786d7b1f60 2024-06-09T22:22:28.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.885+0000 7fd84050db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55786d6f9810 con 0x55786d7ac7a0 2024-06-09T22:22:28.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.885+0000 7fd84050db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55786d7b9200 con 0x55786d7b2880 2024-06-09T22:22:28.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.885+0000 7fd82ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b2880 0x55786d7b5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.886+0000 7fd82ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b2880 0x55786d7b5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:54134/0 (socket says 172.21.15.104:54134) 2024-06-09T22:22:28.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.886+0000 7fd82ffff700 1 -- 172.21.15.104:0/2005572557 learned_addr learned my addr 172.21.15.104:0/2005572557 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:28.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.886+0000 7fd84050db80 10 client.0 _ll_register_callbacks cb 0x55786d6935b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:28.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.886+0000 7fd82effd700 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 0x55786d7b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.886+0000 7fd82ffff700 1 -- 172.21.15.104:0/2005572557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7ac7a0 msgr2=0x55786d7b8640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-09T22:22:28.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.886+0000 7fd82ffff700 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7ac7a0 0x55786d7b8640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.886+0000 7fd82ffff700 1 -- 172.21.15.104:0/2005572557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 msgr2=0x55786d7b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:28.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.886+0000 7fd82ffff700 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 0x55786d7b5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.886+0000 7fd82ffff700 1 -- 172.21.15.104:0/2005572557 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55786d6c3750 con 0x55786d7b2880 2024-06-09T22:22:28.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.886+0000 7fd82effd700 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 0x55786d7b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:28.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.886+0000 7fd84050db80 1 -- 172.21.15.104:0/2005572557 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd820031070 con 0x55786d7b2880 2024-06-09T22:22:28.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.887+0000 7fd82ffff700 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b2880 0x55786d7b5e80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd82809b1a0 tx=0x7fd82809b340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=807d63b0ce4471dc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:28.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.887+0000 7fd80effd700 10 client.25435 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:22:28.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.887+0000 7fd80effd700 1 -- 172.21.15.104:0/2005572557 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8280b4210 con 0x55786d7b2880 2024-06-09T22:22:28.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.887+0000 7fd80effd700 1 -- 172.21.15.104:0/2005572557 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fd82809b670 con 0x55786d7b2880 2024-06-09T22:22:28.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.887+0000 7fd80effd700 1 -- 172.21.15.104:0/2005572557 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd82809f380 con 0x55786d7b2880 2024-06-09T22:22:28.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.887+0000 7fd84050db80 10 client.25435 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:28.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.887+0000 7fd84050db80 1 -- 172.21.15.104:0/2005572557 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55786d7cbf90 con 0x55786d7b2880 2024-06-09T22:22:28.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.888+0000 7fd80effd700 1 -- 172.21.15.104:0/2005572557 <== mon.1 v2:172.21.15.194:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd8280a39f0 con 0x55786d7b2880 2024-06-09T22:22:28.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.888+0000 7fd84050db80 10 client.25435 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:28.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.888+0000 7fd84050db80 10 client.25435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:28.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.888+0000 7fd84050db80 20 client.25435 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:28.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.888+0000 7fd84050db80 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55786d7ce460 0x55786d7d0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.888+0000 7fd84050db80 4 client.25435 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:28.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.888+0000 7fd84050db80 1 -- 172.21.15.104:0/2005572557 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55786d70ad50 con 0x55786d7ce460 2024-06-09T22:22:28.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.888+0000 7fd82f7fe700 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55786d7ce460 0x55786d7d0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.889+0000 7fd82f7fe700 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55786d7ce460 0x55786d7d0850 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e88356eaeb752191 server_cookie=89177900a3b693cf in_seq=0 out_seq=0 2024-06-09T22:22:28.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.889+0000 7fd80effd700 10 client.25435 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:28.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.890+0000 7fd80effd700 1 -- 172.21.15.104:0/2005572557 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55786d70ad50 con 0x55786d7ce460 2024-06-09T22:22:28.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.890+0000 7fd80effd700 10 client.25435 handle_command_reply: tid=0 2024-06-09T22:22:28.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.890+0000 7fd84050db80 1 client.25435 shutdown 2024-06-09T22:22:28.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.890+0000 7fd84050db80 1 -- 172.21.15.104:0/2005572557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b2880 msgr2=0x55786d7b5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:28.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.890+0000 7fd84050db80 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b2880 0x55786d7b5e80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd82809b1a0 tx=0x7fd82809b340 comp rx=0 tx=0).stop 2024-06-09T22:22:28.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.890+0000 7fd84050db80 1 -- 172.21.15.104:0/2005572557 shutdown_connections 2024-06-09T22:22:28.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.890+0000 7fd84050db80 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55786d7ce460 0x55786d7d0850 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.890+0000 7fd80effd700 0 client.25435 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:28.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.890+0000 7fd84050db80 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b2880 0x55786d7b5e80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.890+0000 7fd84050db80 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7ac7a0 0x55786d7b8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.891+0000 7fd84050db80 1 --2- 172.21.15.104:0/2005572557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b1f60 0x55786d7b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.891+0000 7fd84050db80 1 -- 172.21.15.104:0/2005572557 >> 172.21.15.104:0/2005572557 conn(0x55786d6d9080 msgr2=0x55786d7a6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:28.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.891+0000 7fd84050db80 1 -- 172.21.15.104:0/2005572557 shutdown_connections 2024-06-09T22:22:28.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.891+0000 7fd84050db80 1 -- 172.21.15.104:0/2005572557 wait complete. 2024-06-09T22:22:28.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.891+0000 7fd84050db80 20 client.25435 trim_cache size 0 max 16384 2024-06-09T22:22:28.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.932+0000 7fd84050db80 1 Processor -- start 2024-06-09T22:22:28.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.932+0000 7fd84050db80 1 -- start start 2024-06-09T22:22:28.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.932+0000 7fd84050db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7ac7a0 0x55786d7b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.932+0000 7fd84050db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b1f60 0x55786d7b3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.932+0000 7fd84050db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7b2880 0x55786d7a6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.932+0000 7fd82f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7ac7a0 0x55786d7b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.932+0000 7fd84050db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301ac000 con 0x55786d7ac7a0 2024-06-09T22:22:28.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.932+0000 7fd84050db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301adb40 con 0x55786d7b2880 2024-06-09T22:22:28.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.932+0000 7fd84050db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301ac6d0 con 0x55786d7b1f60 2024-06-09T22:22:28.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.932+0000 7fd82f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7ac7a0 0x55786d7b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53382/0 (socket says 172.21.15.104:53382) 2024-06-09T22:22:28.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.932+0000 7fd82f7fe700 1 -- 172.21.15.104:0/1826833587 learned_addr learned my addr 172.21.15.104:0/1826833587 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:28.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.932+0000 7fd82effd700 1 --2- 172.21.15.104:0/1826833587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b1f60 0x55786d7b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.933+0000 7fd82ffff700 1 --2- 172.21.15.104:0/1826833587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7b2880 0x55786d7a6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.933+0000 7fd82effd700 1 -- 172.21.15.104:0/1826833587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7b2880 msgr2=0x55786d7a6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:28.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.933+0000 7fd82effd700 1 --2- 172.21.15.104:0/1826833587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7b2880 0x55786d7a6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.933+0000 7fd82effd700 1 -- 172.21.15.104:0/1826833587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7ac7a0 msgr2=0x55786d7b4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:28.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.933+0000 7fd82effd700 1 --2- 172.21.15.104:0/1826833587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7ac7a0 0x55786d7b4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.933+0000 7fd82effd700 1 -- 172.21.15.104:0/1826833587 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55786d7b6880 con 0x55786d7b1f60 2024-06-09T22:22:28.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.933+0000 7fd82effd700 1 --2- 172.21.15.104:0/1826833587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b1f60 0x55786d7b3370 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55786d7729d0 tx=0x7fd8240116c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43c36177915b9f26 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:28.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.933+0000 7fd8157fa700 1 -- 172.21.15.104:0/1826833587 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824004020 con 0x55786d7b1f60 2024-06-09T22:22:28.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.933+0000 7fd8157fa700 1 -- 172.21.15.104:0/1826833587 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd824003680 con 0x55786d7b1f60 2024-06-09T22:22:28.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.933+0000 7fd8157fa700 1 -- 172.21.15.104:0/1826833587 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8240127f0 con 0x55786d7b1f60 2024-06-09T22:22:28.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.934+0000 7fd84050db80 1 -- 172.21.15.104:0/1826833587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b1f60 msgr2=0x55786d7b3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:28.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.934+0000 7fd84050db80 1 --2- 172.21.15.104:0/1826833587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b1f60 0x55786d7b3370 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55786d7729d0 tx=0x7fd8240116c0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.934+0000 7fd84050db80 1 -- 172.21.15.104:0/1826833587 shutdown_connections 2024-06-09T22:22:28.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.934+0000 7fd84050db80 1 --2- 172.21.15.104:0/1826833587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7ac7a0 0x55786d7b4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.934+0000 7fd84050db80 1 --2- 172.21.15.104:0/1826833587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7b2880 0x55786d7a6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.934+0000 7fd84050db80 1 --2- 172.21.15.104:0/1826833587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7b1f60 0x55786d7b3370 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.934+0000 7fd84050db80 1 -- 172.21.15.104:0/1826833587 >> 172.21.15.104:0/1826833587 conn(0x55786d6d9080 msgr2=0x55786d7b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:28.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.935+0000 7fd84050db80 1 -- 172.21.15.104:0/1826833587 shutdown_connections 2024-06-09T22:22:28.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.935+0000 7fd84050db80 1 -- 172.21.15.104:0/1826833587 wait complete. 2024-06-09T22:22:28.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.935+0000 7fd84050db80 1 -- start start 2024-06-09T22:22:28.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.935+0000 7fd84050db80 1 Processor -- start 2024-06-09T22:22:28.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd84050db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7ac7a0 0x55786d7c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd84050db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7b1f60 0x55786d7c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd84050db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b2880 0x55786d7c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd82f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7ac7a0 0x55786d7c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd82f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7ac7a0 0x55786d7c6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:54160/0 (socket says 172.21.15.104:54160) 2024-06-09T22:22:28.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd82f7fe700 1 -- 172.21.15.104:0/3319558844 learned_addr learned my addr 172.21.15.104:0/3319558844 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:28.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd84050db80 1 -- 172.21.15.104:0/3319558844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301ad470 con 0x55786d7b2880 2024-06-09T22:22:28.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd84050db80 1 -- 172.21.15.104:0/3319558844 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301acda0 con 0x55786d7b1f60 2024-06-09T22:22:28.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd82ffff700 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b2880 0x55786d7c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd82effd700 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7b1f60 0x55786d7c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd84050db80 1 -- 172.21.15.104:0/3319558844 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301a9f80 con 0x55786d7ac7a0 2024-06-09T22:22:28.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd84050db80 10 client.0 _ll_register_callbacks cb 0x55786d6935b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:28.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd82effd700 1 -- 172.21.15.104:0/3319558844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7ac7a0 msgr2=0x55786d7c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:28.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd82effd700 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7ac7a0 0x55786d7c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd82effd700 1 -- 172.21.15.104:0/3319558844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b2880 msgr2=0x55786d7c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:28.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd82effd700 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b2880 0x55786d7c3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.936+0000 7fd82effd700 1 -- 172.21.15.104:0/3319558844 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55786d7c4290 con 0x55786d7b1f60 2024-06-09T22:22:28.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.937+0000 7fd82effd700 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7b1f60 0x55786d7c72c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd824000c00 tx=0x7fd824005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b354fc3453a289d1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.937+0000 7fd80effd700 10 client.25325 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.937+0000 7fd84050db80 1 -- 172.21.15.104:0/3319558844 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd824003680 con 0x55786d7b1f60 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.937+0000 7fd80effd700 1 -- 172.21.15.104:0/3319558844 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824003a40 con 0x55786d7b1f60 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.937+0000 7fd80effd700 1 -- 172.21.15.104:0/3319558844 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fd82401f070 con 0x55786d7b1f60 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.937+0000 7fd80effd700 1 -- 172.21.15.104:0/3319558844 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd824003be0 con 0x55786d7b1f60 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.937+0000 7fd84050db80 10 client.25325 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.937+0000 7fd84050db80 1 -- 172.21.15.104:0/3319558844 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55786d839720 con 0x55786d7b1f60 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.938+0000 7fd80effd700 1 -- 172.21.15.104:0/3319558844 <== mon.2 v2:172.21.15.153:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd824005020 con 0x55786d7b1f60 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.938+0000 7fd84050db80 10 client.25325 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.938+0000 7fd84050db80 10 client.25325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.938+0000 7fd84050db80 20 client.25325 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.938+0000 7fd84050db80 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55786d7ce460 0x55786d83bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:28.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.938+0000 7fd84050db80 4 client.25325 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.938+0000 7fd84050db80 1 -- 172.21.15.104:0/3319558844 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fd824003680 con 0x55786d7ce460 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.938+0000 7fd82f7fe700 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55786d7ce460 0x55786d83bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.939+0000 7fd80effd700 10 client.25325 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.939+0000 7fd82f7fe700 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55786d7ce460 0x55786d83bc20 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63f0bdd494bc4baf server_cookie=a3a697d18ddacc1d in_seq=0 out_seq=0 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.939+0000 7fd80effd700 1 -- 172.21.15.104:0/3319558844 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fd824003680 con 0x55786d7ce460 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.939+0000 7fd80effd700 10 client.25325 handle_command_reply: tid=0 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.939+0000 7fd84050db80 1 client.25325 shutdown 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.939+0000 7fd84050db80 1 -- 172.21.15.104:0/3319558844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7b1f60 msgr2=0x55786d7c72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.939+0000 7fd84050db80 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7b1f60 0x55786d7c72c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd824000c00 tx=0x7fd824005800 comp rx=0 tx=0).stop 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.939+0000 7fd84050db80 1 -- 172.21.15.104:0/3319558844 shutdown_connections 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.939+0000 7fd84050db80 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55786d7ce460 0x55786d83bc20 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.939+0000 7fd80effd700 0 client.25325 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:28.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.939+0000 7fd84050db80 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55786d7b2880 0x55786d7c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.939+0000 7fd84050db80 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55786d7ac7a0 0x55786d7c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.940+0000 7fd84050db80 1 --2- 172.21.15.104:0/3319558844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55786d7b1f60 0x55786d7c72c0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd824000c00 tx=0x7fd824005800 comp rx=0 tx=0).stop 2024-06-09T22:22:28.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.940+0000 7fd84050db80 1 -- 172.21.15.104:0/3319558844 >> 172.21.15.104:0/3319558844 conn(0x55786d6d9080 msgr2=0x55786d7b34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:28.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.940+0000 7fd84050db80 1 -- 172.21.15.104:0/3319558844 shutdown_connections 2024-06-09T22:22:28.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.940+0000 7fd84050db80 1 -- 172.21.15.104:0/3319558844 wait complete. 2024-06-09T22:22:28.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.940+0000 7fd84050db80 20 client.25325 trim_cache size 0 max 16384 2024-06-09T22:22:28.946 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:22:28.946 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "no active scrubs running", 2024-06-09T22:22:28.946 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": {} 2024-06-09T22:22:28.946 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:22:28.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.941+0000 7fd835c8a700 1 -- 172.21.15.104:0/2461247813 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd8180a2e20 msgr2=0x7fd8180a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:28.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.941+0000 7fd835c8a700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd8180a2e20 0x7fd8180a52e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd820009750 tx=0x7fd820005bc0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.941+0000 7fd835c8a700 1 -- 172.21.15.104:0/2461247813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8301073f0 msgr2=0x7fd830198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:28.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.941+0000 7fd835c8a700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8301073f0 0x7fd830198960 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd828007f10 tx=0x7fd82800d3b0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.941+0000 7fd835c8a700 1 -- 172.21.15.104:0/2461247813 shutdown_connections 2024-06-09T22:22:28.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.941+0000 7fd835c8a700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd8180a2e20 0x7fd8180a52e0 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd820009750 tx=0x7fd820005bc0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.941+0000 7fd835c8a700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8301073f0 0x7fd830198960 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.941+0000 7fd835c8a700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd830101380 0x7fd83019e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.941+0000 7fd835c8a700 1 --2- 172.21.15.104:0/2461247813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd830101d40 0x7fd83019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:28.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.941+0000 7fd835c8a700 1 -- 172.21.15.104:0/2461247813 >> 172.21.15.104:0/2461247813 conn(0x7fd830078300 msgr2=0x7fd8300ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:28.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.941+0000 7fd835c8a700 1 -- 172.21.15.104:0/2461247813 shutdown_connections 2024-06-09T22:22:28.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:28.941+0000 7fd835c8a700 1 -- 172.21.15.104:0/2461247813 wait complete. 2024-06-09T22:22:29.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:29 smithi104 ceph-mon[32389]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-06-09T22:22:29.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:29 smithi194 ceph-mon[36856]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-06-09T22:22:29.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:29 smithi153 ceph-mon[36599]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-06-09T22:22:30.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.286+0000 7f2839837700 1 -- 172.21.15.104:0/3416264476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28341014a0 msgr2=0x7f283410d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:30.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.286+0000 7f2839837700 1 --2- 172.21.15.104:0/3416264476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28341014a0 0x7f283410d9a0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f282c00d330 tx=0x7f282c00d640 comp rx=0 tx=0).stop 2024-06-09T22:22:30.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.286+0000 7f2839837700 1 -- 172.21.15.104:0/3416264476 shutdown_connections 2024-06-09T22:22:30.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.286+0000 7f2839837700 1 --2- 172.21.15.104:0/3416264476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28341014a0 0x7f283410d9a0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.286+0000 7f2839837700 1 --2- 172.21.15.104:0/3416264476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2834106b60 0x7f2834106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.286+0000 7f2839837700 1 --2- 172.21.15.104:0/3416264476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2834100ae0 0x7f2834100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.286+0000 7f2839837700 1 -- 172.21.15.104:0/3416264476 >> 172.21.15.104:0/3416264476 conn(0x7f2834077a30 msgr2=0x7f28340fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.286+0000 7f2839837700 1 -- 172.21.15.104:0/3416264476 shutdown_connections 2024-06-09T22:22:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.286+0000 7f2839837700 1 -- 172.21.15.104:0/3416264476 wait complete. 2024-06-09T22:22:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.287+0000 7f2839837700 1 Processor -- start 2024-06-09T22:22:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.287+0000 7f2839837700 1 -- start start 2024-06-09T22:22:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.287+0000 7f2839837700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834100ae0 0x7f283419c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.287+0000 7f2839837700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28341014a0 0x7f283419cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.287+0000 7f2839837700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2834106b60 0x7f28341966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f2839837700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2834110590 con 0x7f2834100ae0 2024-06-09T22:22:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f2839837700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2834110410 con 0x7f2834106b60 2024-06-09T22:22:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f2839837700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2834110710 con 0x7f28341014a0 2024-06-09T22:22:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f2832ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834100ae0 0x7f283419c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f28327fc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28341014a0 0x7f283419cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f2832ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834100ae0 0x7f283419c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53392/0 (socket says 172.21.15.104:53392) 2024-06-09T22:22:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f2832ffd700 1 -- 172.21.15.104:0/1197854414 learned_addr learned my addr 172.21.15.104:0/1197854414 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f28337fe700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2834106b60 0x7f28341966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f28327fc700 1 -- 172.21.15.104:0/1197854414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2834106b60 msgr2=0x7f28341966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f28327fc700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2834106b60 0x7f28341966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f28327fc700 1 -- 172.21.15.104:0/1197854414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834100ae0 msgr2=0x7f283419c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f28327fc700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834100ae0 0x7f283419c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f28327fc700 1 -- 172.21.15.104:0/1197854414 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2834196fd0 con 0x7f28341014a0 2024-06-09T22:22:30.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.288+0000 7f28327fc700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28341014a0 0x7f283419cb10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f282800e9f0 tx=0x7f282800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:30.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.289+0000 7f2838835700 1 -- 172.21.15.104:0/1197854414 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2828004220 con 0x7f28341014a0 2024-06-09T22:22:30.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.289+0000 7f2839837700 1 -- 172.21.15.104:0/1197854414 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28341972c0 con 0x7f28341014a0 2024-06-09T22:22:30.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.289+0000 7f2838835700 1 -- 172.21.15.104:0/1197854414 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2828004d10 con 0x7f28341014a0 2024-06-09T22:22:30.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.289+0000 7f2838835700 1 -- 172.21.15.104:0/1197854414 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2828005020 con 0x7f28341014a0 2024-06-09T22:22:30.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.289+0000 7f2839837700 1 -- 172.21.15.104:0/1197854414 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28341a3550 con 0x7f28341014a0 2024-06-09T22:22:30.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.291+0000 7f2838835700 1 -- 172.21.15.104:0/1197854414 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 24) ==== 138118+0+0 (secure 0 0 0) 0x7f2828004750 con 0x7f28341014a0 2024-06-09T22:22:30.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.292+0000 7f2838835700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f281c0a2fe0 0x7f281c0a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.292+0000 7f2832ffd700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f281c0a2fe0 0x7f281c0a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.292+0000 7f2838835700 1 -- 172.21.15.104:0/1197854414 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f28280cf970 con 0x7f28341014a0 2024-06-09T22:22:30.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.293+0000 7f2832ffd700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f281c0a2fe0 0x7f281c0a54a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f282400a510 tx=0x7f282400a730 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:30.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.293+0000 7f28440bab80 1 Processor -- start 2024-06-09T22:22:30.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.293+0000 7f28440bab80 1 -- start start 2024-06-09T22:22:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.293+0000 7f28440bab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b2ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.293+0000 7f28440bab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b24bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.293+0000 7f28440bab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b250f0 0x55fdb6b25510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.293+0000 7f28440bab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fdb6a130f0 con 0x55fdb6b2b2c0 2024-06-09T22:22:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.293+0000 7f28440bab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fdb6a719e0 con 0x55fdb6b2a9a0 2024-06-09T22:22:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.293+0000 7f28440bab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fdb69d95f0 con 0x55fdb6b250f0 2024-06-09T22:22:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f28327fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b24bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f28327fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b24bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53398/0 (socket says 172.21.15.104:53398) 2024-06-09T22:22:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f28327fc700 1 -- 172.21.15.104:0/3406859577 learned_addr learned my addr 172.21.15.104:0/3406859577 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f2832ffd700 1 --2- 172.21.15.104:0/3406859577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b2ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f28327fc700 1 -- 172.21.15.104:0/3406859577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b250f0 msgr2=0x55fdb6b25510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f28327fc700 1 --2- 172.21.15.104:0/3406859577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b250f0 0x55fdb6b25510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f28327fc700 1 -- 172.21.15.104:0/3406859577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b2a9a0 msgr2=0x55fdb6b2ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f28327fc700 1 --2- 172.21.15.104:0/3406859577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b2ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f28327fc700 1 -- 172.21.15.104:0/3406859577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdb69c0a60 con 0x55fdb6b2b2c0 2024-06-09T22:22:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f28327fc700 1 --2- 172.21.15.104:0/3406859577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b24bb0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f28280982b0 tx=0x7f282809b2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634eeaac96e7c979 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f2819ffb700 1 -- 172.21.15.104:0/3406859577 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282801c040 con 0x55fdb6b2b2c0 2024-06-09T22:22:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f2819ffb700 1 -- 172.21.15.104:0/3406859577 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f282809b8e0 con 0x55fdb6b2b2c0 2024-06-09T22:22:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.294+0000 7f2819ffb700 1 -- 172.21.15.104:0/3406859577 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2828004a40 con 0x55fdb6b2b2c0 2024-06-09T22:22:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.295+0000 7f28440bab80 1 -- 172.21.15.104:0/3406859577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2b2c0 msgr2=0x55fdb6b24bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.295+0000 7f28440bab80 1 --2- 172.21.15.104:0/3406859577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b24bb0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f28280982b0 tx=0x7f282809b2c0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.295+0000 7f28440bab80 1 -- 172.21.15.104:0/3406859577 shutdown_connections 2024-06-09T22:22:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.295+0000 7f28440bab80 1 --2- 172.21.15.104:0/3406859577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b250f0 0x55fdb6b25510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.295+0000 7f28440bab80 1 --2- 172.21.15.104:0/3406859577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b2ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.295+0000 7f28440bab80 1 --2- 172.21.15.104:0/3406859577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b24bb0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.295+0000 7f28440bab80 1 -- 172.21.15.104:0/3406859577 >> 172.21.15.104:0/3406859577 conn(0x55fdb6a4d210 msgr2=0x55fdb6b1ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.295+0000 7f28440bab80 1 -- 172.21.15.104:0/3406859577 shutdown_connections 2024-06-09T22:22:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.295+0000 7f28440bab80 1 -- 172.21.15.104:0/3406859577 wait complete. 2024-06-09T22:22:30.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.295+0000 7f28440bab80 1 -- start start 2024-06-09T22:22:30.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.296+0000 7f28440bab80 1 Processor -- start 2024-06-09T22:22:30.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.296+0000 7f28440bab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b250f0 0x55fdb6b31090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.296+0000 7f28440bab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b2e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.296+0000 7f28440bab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b2e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.296+0000 7f28440bab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fdb6a12150 con 0x55fdb6b2a9a0 2024-06-09T22:22:30.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.296+0000 7f28440bab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fdb6a6d9a0 con 0x55fdb6b250f0 2024-06-09T22:22:30.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.296+0000 7f28440bab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fdb6b31ca0 con 0x55fdb6b2b2c0 2024-06-09T22:22:30.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.296+0000 7f28440bab80 10 client.0 _ll_register_callbacks cb 0x55fdb6a07740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:30.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.296+0000 7f2832ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b250f0 0x55fdb6b31090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.296+0000 7f2832ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b250f0 0x55fdb6b31090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:60514/0 (socket says 172.21.15.104:60514) 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.296+0000 7f2832ffd700 1 -- 172.21.15.104:0/39169641 learned_addr learned my addr 172.21.15.104:0/39169641 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.296+0000 7f28337fe700 1 --2- 172.21.15.104:0/39169641 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b2e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f2832ffd700 1 -- 172.21.15.104:0/39169641 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2b2c0 msgr2=0x55fdb6b2e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f2832ffd700 1 --2- 172.21.15.104:0/39169641 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b2e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f2832ffd700 1 -- 172.21.15.104:0/39169641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2a9a0 msgr2=0x55fdb6b2e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f2832ffd700 1 --2- 172.21.15.104:0/39169641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b2e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f2832ffd700 1 -- 172.21.15.104:0/39169641 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdb6a378e0 con 0x55fdb6b250f0 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f2832ffd700 1 --2- 172.21.15.104:0/39169641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b250f0 0x55fdb6b31090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2824004010 tx=0x7f2824004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17e8c2e6050462a9 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f2812ffd700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f2812ffd700 1 -- 172.21.15.104:0/39169641 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2824027420 con 0x55fdb6b250f0 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f28440bab80 1 -- 172.21.15.104:0/39169641 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f282809b8e0 con 0x55fdb6b250f0 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f2812ffd700 1 -- 172.21.15.104:0/39169641 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f2824008050 con 0x55fdb6b250f0 2024-06-09T22:22:30.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f2812ffd700 1 -- 172.21.15.104:0/39169641 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f28240275c0 con 0x55fdb6b250f0 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f28440bab80 10 client.25337 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.297+0000 7f28440bab80 1 -- 172.21.15.104:0/39169641 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fdb6b3c550 con 0x55fdb6b250f0 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.298+0000 7f2812ffd700 1 -- 172.21.15.104:0/39169641 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f282809b8e0 con 0x55fdb6b250f0 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.298+0000 7f2812ffd700 1 -- 172.21.15.104:0/39169641 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f282400b040 con 0x55fdb6b250f0 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.298+0000 7f28440bab80 10 client.25337 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.298+0000 7f28440bab80 10 client.25337 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.298+0000 7f28440bab80 20 client.25337 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.298+0000 7f28440bab80 1 --2- 172.21.15.104:0/39169641 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fdb6b3eb30 0x55fdb6b40fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.298+0000 7f28440bab80 4 client.25337 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.298+0000 7f28440bab80 1 -- 172.21.15.104:0/39169641 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdb6a7eee0 con 0x55fdb6b3eb30 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.298+0000 7f28327fc700 1 --2- 172.21.15.104:0/39169641 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fdb6b3eb30 0x55fdb6b40fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.299+0000 7f2812ffd700 10 client.25337 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.299+0000 7f28327fc700 1 --2- 172.21.15.104:0/39169641 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fdb6b3eb30 0x55fdb6b40fb0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21b53347f79f0f0a server_cookie=fb806cfd6ae90de4 in_seq=0 out_seq=0 2024-06-09T22:22:30.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.299+0000 7f2812ffd700 1 -- 172.21.15.104:0/39169641 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55fdb6a7eee0 con 0x55fdb6b3eb30 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.299+0000 7f2812ffd700 10 client.25337 handle_command_reply: tid=0 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f28440bab80 1 client.25337 shutdown 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f28440bab80 1 -- 172.21.15.104:0/39169641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b250f0 msgr2=0x55fdb6b31090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f28440bab80 1 --2- 172.21.15.104:0/39169641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b250f0 0x55fdb6b31090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2824004010 tx=0x7f2824004750 comp rx=0 tx=0).stop 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f28440bab80 1 -- 172.21.15.104:0/39169641 shutdown_connections 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f28440bab80 1 --2- 172.21.15.104:0/39169641 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fdb6b3eb30 0x55fdb6b40fb0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f28440bab80 1 --2- 172.21.15.104:0/39169641 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b2e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f2812ffd700 0 client.25337 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f28440bab80 1 --2- 172.21.15.104:0/39169641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b250f0 0x55fdb6b31090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f28440bab80 1 --2- 172.21.15.104:0/39169641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b2e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f28440bab80 1 -- 172.21.15.104:0/39169641 >> 172.21.15.104:0/39169641 conn(0x55fdb6a4d210 msgr2=0x55fdb6b1ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f28440bab80 1 -- 172.21.15.104:0/39169641 shutdown_connections 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f28440bab80 1 -- 172.21.15.104:0/39169641 wait complete. 2024-06-09T22:22:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.300+0000 7f28440bab80 20 client.25337 trim_cache size 0 max 16384 2024-06-09T22:22:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.332+0000 7f28440bab80 1 Processor -- start 2024-06-09T22:22:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.332+0000 7f28440bab80 1 -- start start 2024-06-09T22:22:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28440bab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b250f0 0x55fdb6b20170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28440bab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b30bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28440bab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b1b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28440bab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f28341ac230 con 0x55fdb6b2a9a0 2024-06-09T22:22:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28440bab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f28341add70 con 0x55fdb6b250f0 2024-06-09T22:22:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28440bab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f28341ac900 con 0x55fdb6b2b2c0 2024-06-09T22:22:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28327fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b30bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28327fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b30bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53410/0 (socket says 172.21.15.104:53410) 2024-06-09T22:22:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28327fc700 1 -- 172.21.15.104:0/2077164554 learned_addr learned my addr 172.21.15.104:0/2077164554 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f2832ffd700 1 --2- 172.21.15.104:0/2077164554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b250f0 0x55fdb6b20170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28337fe700 1 --2- 172.21.15.104:0/2077164554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b1b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28327fc700 1 -- 172.21.15.104:0/2077164554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2b2c0 msgr2=0x55fdb6b1b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28327fc700 1 --2- 172.21.15.104:0/2077164554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b1b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28327fc700 1 -- 172.21.15.104:0/2077164554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b250f0 msgr2=0x55fdb6b20170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28327fc700 1 --2- 172.21.15.104:0/2077164554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b250f0 0x55fdb6b20170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28327fc700 1 -- 172.21.15.104:0/2077164554 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdb6b2c920 con 0x55fdb6b2a9a0 2024-06-09T22:22:30.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.333+0000 7f28337fe700 1 --2- 172.21.15.104:0/2077164554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b1b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:30.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.334+0000 7f28327fc700 1 --2- 172.21.15.104:0/2077164554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b30bd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2828097d40 tx=0x7f2828086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e252fdac63d01201 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:30.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.334+0000 7f28197fa700 1 -- 172.21.15.104:0/2077164554 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282801c040 con 0x55fdb6b2a9a0 2024-06-09T22:22:30.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.334+0000 7f28197fa700 1 -- 172.21.15.104:0/2077164554 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28280a4550 con 0x55fdb6b2a9a0 2024-06-09T22:22:30.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.334+0000 7f28197fa700 1 -- 172.21.15.104:0/2077164554 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2828004b60 con 0x55fdb6b2a9a0 2024-06-09T22:22:30.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.335+0000 7f28440bab80 1 -- 172.21.15.104:0/2077164554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2a9a0 msgr2=0x55fdb6b30bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:30.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.335+0000 7f28440bab80 1 --2- 172.21.15.104:0/2077164554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b30bd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2828097d40 tx=0x7f2828086040 comp rx=0 tx=0).stop 2024-06-09T22:22:30.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.335+0000 7f28440bab80 1 -- 172.21.15.104:0/2077164554 shutdown_connections 2024-06-09T22:22:30.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.335+0000 7f28440bab80 1 --2- 172.21.15.104:0/2077164554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b1b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.335+0000 7f28440bab80 1 --2- 172.21.15.104:0/2077164554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b250f0 0x55fdb6b20170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.335+0000 7f28440bab80 1 --2- 172.21.15.104:0/2077164554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b30bd0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.335+0000 7f28440bab80 1 -- 172.21.15.104:0/2077164554 >> 172.21.15.104:0/2077164554 conn(0x55fdb6a4d210 msgr2=0x55fdb6b2e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:30.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.335+0000 7f28440bab80 1 -- 172.21.15.104:0/2077164554 shutdown_connections 2024-06-09T22:22:30.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.335+0000 7f28440bab80 1 -- 172.21.15.104:0/2077164554 wait complete. 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.335+0000 7f28440bab80 1 -- start start 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.335+0000 7f28440bab80 1 Processor -- start 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f28440bab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b250f0 0x55fdb6b376c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f28440bab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b33f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f28440bab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b34440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f28440bab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f28341ad6a0 con 0x55fdb6b250f0 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f28440bab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f28341acfd0 con 0x55fdb6b2b2c0 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f28440bab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f28341aa1b0 con 0x55fdb6b2a9a0 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f28440bab80 10 client.0 _ll_register_callbacks cb 0x55fdb6a07740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f2832ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b250f0 0x55fdb6b376c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f2832ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b250f0 0x55fdb6b376c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53426/0 (socket says 172.21.15.104:53426) 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f2832ffd700 1 -- 172.21.15.104:0/1595994096 learned_addr learned my addr 172.21.15.104:0/1595994096 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f28337fe700 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b34440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f28327fc700 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b33f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f2832ffd700 1 -- 172.21.15.104:0/1595994096 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2a9a0 msgr2=0x55fdb6b33f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f2832ffd700 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b33f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f2832ffd700 1 -- 172.21.15.104:0/1595994096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b2b2c0 msgr2=0x55fdb6b34440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f2832ffd700 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b34440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.336+0000 7f2832ffd700 1 -- 172.21.15.104:0/1595994096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdb6b34c80 con 0x55fdb6b250f0 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f2832ffd700 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b250f0 0x55fdb6b376c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f282401a8b0 tx=0x7f282401a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd66e7f408744bb6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f2812ffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f2812ffd700 1 -- 172.21.15.104:0/1595994096 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2824020040 con 0x55fdb6b250f0 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f2812ffd700 1 -- 172.21.15.104:0/1595994096 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f2824015030 con 0x55fdb6b250f0 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f28440bab80 1 -- 172.21.15.104:0/1595994096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28280a4550 con 0x55fdb6b250f0 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f2812ffd700 1 -- 172.21.15.104:0/1595994096 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2824027420 con 0x55fdb6b250f0 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f28440bab80 10 client.15879 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f28440bab80 1 -- 172.21.15.104:0/1595994096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fdb6baa820 con 0x55fdb6b250f0 2024-06-09T22:22:30.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f2812ffd700 1 -- 172.21.15.104:0/1595994096 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28280a4550 con 0x55fdb6b250f0 2024-06-09T22:22:30.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f2812ffd700 1 -- 172.21.15.104:0/1595994096 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f28240047c0 con 0x55fdb6b250f0 2024-06-09T22:22:30.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f28440bab80 10 client.15879 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:30.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f28440bab80 10 client.15879 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:22:30.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f28440bab80 20 client.15879 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:30.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f28440bab80 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fdb6b3eb30 0x55fdb6bad000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:30.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f28440bab80 4 client.15879 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-06-09T22:22:30.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.337+0000 7f28440bab80 1 -- 172.21.15.104:0/1595994096 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x55fdb6a79460 con 0x55fdb6b3eb30 2024-06-09T22:22:30.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.338+0000 7f28327fc700 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fdb6b3eb30 0x55fdb6bad000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:30.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.338+0000 7f2812ffd700 10 client.15879 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:30.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.338+0000 7f28327fc700 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fdb6b3eb30 0x55fdb6bad000 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca969fdfa35cf566 server_cookie=4448a287682b5ca9 in_seq=0 out_seq=0 2024-06-09T22:22:30.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f2812ffd700 1 -- 172.21.15.104:0/1595994096 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55fdb6a79460 con 0x55fdb6b3eb30 2024-06-09T22:22:30.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f2812ffd700 10 client.15879 handle_command_reply: tid=0 2024-06-09T22:22:30.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f28440bab80 1 client.15879 shutdown 2024-06-09T22:22:30.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f28440bab80 1 -- 172.21.15.104:0/1595994096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b250f0 msgr2=0x55fdb6b376c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:30.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f28440bab80 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b250f0 0x55fdb6b376c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f282401a8b0 tx=0x7f282401a630 comp rx=0 tx=0).stop 2024-06-09T22:22:30.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f28440bab80 1 -- 172.21.15.104:0/1595994096 shutdown_connections 2024-06-09T22:22:30.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f28440bab80 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fdb6b3eb30 0x55fdb6bad000 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f28440bab80 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fdb6b250f0 0x55fdb6b376c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f28440bab80 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fdb6b2b2c0 0x55fdb6b34440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f28440bab80 1 --2- 172.21.15.104:0/1595994096 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fdb6b2a9a0 0x55fdb6b33f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f28440bab80 1 -- 172.21.15.104:0/1595994096 >> 172.21.15.104:0/1595994096 conn(0x55fdb6a4d210 msgr2=0x55fdb6b29410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:30.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f2812ffd700 0 client.15879 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:30.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f28440bab80 1 -- 172.21.15.104:0/1595994096 shutdown_connections 2024-06-09T22:22:30.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f28440bab80 1 -- 172.21.15.104:0/1595994096 wait complete. 2024-06-09T22:22:30.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.347+0000 7f28440bab80 20 client.15879 trim_cache size 0 max 16384 2024-06-09T22:22:30.348 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 29, 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "leader": 24460, 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "set_version": 2, 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "sets": { 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "79edc90d": { 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "version": 2, 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "age_ref": 0.0, 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "QUIESCED", 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "timeout": 90.0, 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "expiration": 150.0, 2024-06-09T22:22:30.349 INFO:teuthology.orchestra.run.smithi104.stdout: "members": { 2024-06-09T22:22:30.350 INFO:teuthology.orchestra.run.smithi104.stdout: "file:/": { 2024-06-09T22:22:30.350 INFO:teuthology.orchestra.run.smithi104.stdout: "excluded": false, 2024-06-09T22:22:30.350 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:22:30.350 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "QUIESCED", 2024-06-09T22:22:30.350 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:22:30.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:22:30.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:22:30.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:22:30.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:22:30.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:22:30.350 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:22:30.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.348+0000 7f2839837700 1 -- 172.21.15.104:0/1197854414 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f281c0a2fe0 msgr2=0x7f281c0a54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:30.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.348+0000 7f2839837700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f281c0a2fe0 0x7f281c0a54a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f282400a510 tx=0x7f282400a730 comp rx=0 tx=0).stop 2024-06-09T22:22:30.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.348+0000 7f2839837700 1 -- 172.21.15.104:0/1197854414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28341014a0 msgr2=0x7f283419cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:30.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.348+0000 7f2839837700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28341014a0 0x7f283419cb10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f282800e9f0 tx=0x7f282800ed00 comp rx=0 tx=0).stop 2024-06-09T22:22:30.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.348+0000 7f2839837700 1 -- 172.21.15.104:0/1197854414 shutdown_connections 2024-06-09T22:22:30.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.348+0000 7f2839837700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f281c0a2fe0 0x7f281c0a54a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.348+0000 7f2839837700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834100ae0 0x7f283419c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.348+0000 7f2839837700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2834106b60 0x7f28341966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.348+0000 7f2839837700 1 --2- 172.21.15.104:0/1197854414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28341014a0 0x7f283419cb10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:30.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.348+0000 7f2839837700 1 -- 172.21.15.104:0/1197854414 >> 172.21.15.104:0/1197854414 conn(0x7f2834077a30 msgr2=0x7f2834104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:30.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.348+0000 7f2839837700 1 -- 172.21.15.104:0/1197854414 shutdown_connections 2024-06-09T22:22:30.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:30.348+0000 7f2839837700 1 -- 172.21.15.104:0/1197854414 wait complete. 2024-06-09T22:22:31.388 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-09T22:22:31.389 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:22:31.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:31 smithi194 ceph-mon[36856]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 2.3 KiB/s wr, 17 op/s 2024-06-09T22:22:31.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:31 smithi194 ceph-mon[36856]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-06-09T22:22:31.905 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:31 smithi104 ceph-mon[32389]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 2.3 KiB/s wr, 17 op/s 2024-06-09T22:22:31.905 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:31 smithi104 ceph-mon[32389]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-06-09T22:22:31.933 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '79edc90d', quiesce took 15.2 seconds. Will release after: 14.8 2024-06-09T22:22:31.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:31 smithi153 ceph-mon[36599]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 2.3 KiB/s wr, 17 op/s 2024-06-09T22:22:31.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:31 smithi153 ceph-mon[36599]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-06-09T22:22:32.070 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.095+0000 7f6982e4d640 1 -- 172.21.15.104:0/1671348888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c103540 msgr2=0x7f697c10ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.095+0000 7f6982e4d640 1 --2- 172.21.15.104:0/1671348888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c103540 0x7f697c10ba20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6968009900 tx=0x7f696802a480 comp rx=0 tx=0).stop 2024-06-09T22:22:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.095+0000 7f6982e4d640 1 -- 172.21.15.104:0/1671348888 shutdown_connections 2024-06-09T22:22:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.095+0000 7f6982e4d640 1 --2- 172.21.15.104:0/1671348888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f697c10bff0 0x7f697c10e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.095+0000 7f6982e4d640 1 --2- 172.21.15.104:0/1671348888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c103540 0x7f697c10ba20 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.095+0000 7f6982e4d640 1 --2- 172.21.15.104:0/1671348888 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f697c102c20 0x7f697c103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.095+0000 7f6982e4d640 1 -- 172.21.15.104:0/1671348888 >> 172.21.15.104:0/1671348888 conn(0x7f697c0779f0 msgr2=0x7f697c077e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.095+0000 7f6982e4d640 1 -- 172.21.15.104:0/1671348888 shutdown_connections 2024-06-09T22:22:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.095+0000 7f6982e4d640 1 -- 172.21.15.104:0/1671348888 wait complete. 2024-06-09T22:22:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f6982e4d640 1 Processor -- start 2024-06-09T22:22:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f6982e4d640 1 -- start start 2024-06-09T22:22:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f6982e4d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f697c102c20 0x7f697c1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f6982e4d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f697c103540 0x7f697c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f6982e4d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10bff0 0x7f697c19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f6982e4d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f697c111d80 con 0x7f697c10bff0 2024-06-09T22:22:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f6982e4d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f697c111c00 con 0x7f697c102c20 2024-06-09T22:22:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f6982e4d640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f697c111f00 con 0x7f697c103540 2024-06-09T22:22:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f69813ea640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10bff0 0x7f697c19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f69813ea640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10bff0 0x7f697c19ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53442/0 (socket says 172.21.15.104:53442) 2024-06-09T22:22:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f69813ea640 1 -- 172.21.15.104:0/2710006647 learned_addr learned my addr 172.21.15.104:0/2710006647 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f6980be9640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f697c102c20 0x7f697c1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f697bfff640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f697c103540 0x7f697c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f69813ea640 1 -- 172.21.15.104:0/2710006647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f697c103540 msgr2=0x7f697c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.096+0000 7f69813ea640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f697c103540 0x7f697c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.097+0000 7f69813ea640 1 -- 172.21.15.104:0/2710006647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f697c102c20 msgr2=0x7f697c1981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.097+0000 7f69813ea640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f697c102c20 0x7f697c1981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.097+0000 7f69813ea640 1 -- 172.21.15.104:0/2710006647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697c19d180 con 0x7f697c10bff0 2024-06-09T22:22:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.097+0000 7f69813ea640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10bff0 0x7f697c19ca50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f697400cce0 tx=0x7f697400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.097+0000 7f6979ffb640 1 -- 172.21.15.104:0/2710006647 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697400f450 con 0x7f697c10bff0 2024-06-09T22:22:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.097+0000 7f6982e4d640 1 -- 172.21.15.104:0/2710006647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f697c19d470 con 0x7f697c10bff0 2024-06-09T22:22:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.097+0000 7f6979ffb640 1 -- 172.21.15.104:0/2710006647 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f697400fad0 con 0x7f697c10bff0 2024-06-09T22:22:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.097+0000 7f6979ffb640 1 -- 172.21.15.104:0/2710006647 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974018c10 con 0x7f697c10bff0 2024-06-09T22:22:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.097+0000 7f6982e4d640 1 -- 172.21.15.104:0/2710006647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f697c1a4d00 con 0x7f697c10bff0 2024-06-09T22:22:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.099+0000 7f6979ffb640 1 -- 172.21.15.104:0/2710006647 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f697400f5f0 con 0x7f697c10bff0 2024-06-09T22:22:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f6979ffb640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f696c0a2da0 0x7f696c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f6985c38740 1 Processor -- start 2024-06-09T22:22:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f6985c38740 1 -- start start 2024-06-09T22:22:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f6980be9640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f696c0a2da0 0x7f696c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f6979ffb640 1 -- 172.21.15.104:0/2710006647 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f69740cff10 con 0x7f697c10bff0 2024-06-09T22:22:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f6985c38740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459317170 0x55d459317550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f6985c38740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317a90 0x55d459321e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f6985c38740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459322350 0x55d459324740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f6985c38740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d45927b4f0 con 0x55d459322350 2024-06-09T22:22:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f6985c38740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d459219870 con 0x55d459317170 2024-06-09T22:22:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f6985c38740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d4592774b0 con 0x55d459317a90 2024-06-09T22:22:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f69813ea640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459322350 0x55d459324740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f69813ea640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459322350 0x55d459324740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53456/0 (socket says 172.21.15.104:53456) 2024-06-09T22:22:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f69813ea640 1 -- 172.21.15.104:0/2344586538 learned_addr learned my addr 172.21.15.104:0/2344586538 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f6980be9640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f696c0a2da0 0x7f696c0a5260 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6970005fd0 tx=0x7f6970005850 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.101+0000 7f697bfff640 1 --2- 172.21.15.104:0/2344586538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317a90 0x55d459321e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.102+0000 7f6980be9640 1 --2- 172.21.15.104:0/2344586538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459317170 0x55d459317550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.102+0000 7f69813ea640 1 -- 172.21.15.104:0/2344586538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317a90 msgr2=0x55d459321e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.102+0000 7f69813ea640 1 --2- 172.21.15.104:0/2344586538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317a90 0x55d459321e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.102+0000 7f69813ea640 1 -- 172.21.15.104:0/2344586538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459317170 msgr2=0x55d459317550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.102+0000 7f69813ea640 1 --2- 172.21.15.104:0/2344586538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459317170 0x55d459317550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.102+0000 7f69813ea640 1 -- 172.21.15.104:0/2344586538 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4592359b0 con 0x55d459322350 2024-06-09T22:22:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.102+0000 7f69813ea640 1 --2- 172.21.15.104:0/2344586538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459322350 0x55d459324740 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6974098520 tx=0x7f697409ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35af8c7fd9a4d75e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.102+0000 7f6966ffd640 1 -- 172.21.15.104:0/2344586538 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69740a0470 con 0x55d459322350 2024-06-09T22:22:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.102+0000 7f6966ffd640 1 -- 172.21.15.104:0/2344586538 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69740a4d30 con 0x55d459322350 2024-06-09T22:22:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.102+0000 7f6966ffd640 1 -- 172.21.15.104:0/2344586538 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69740cd570 con 0x55d459322350 2024-06-09T22:22:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.103+0000 7f6985c38740 1 -- 172.21.15.104:0/2344586538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459322350 msgr2=0x55d459324740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.103+0000 7f6985c38740 1 --2- 172.21.15.104:0/2344586538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459322350 0x55d459324740 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6974098520 tx=0x7f697409ba20 comp rx=0 tx=0).stop 2024-06-09T22:22:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.103+0000 7f6985c38740 1 -- 172.21.15.104:0/2344586538 shutdown_connections 2024-06-09T22:22:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.103+0000 7f6985c38740 1 --2- 172.21.15.104:0/2344586538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459322350 0x55d459324740 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.103+0000 7f6985c38740 1 --2- 172.21.15.104:0/2344586538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317a90 0x55d459321e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.103+0000 7f6985c38740 1 --2- 172.21.15.104:0/2344586538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459317170 0x55d459317550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.103+0000 7f6985c38740 1 -- 172.21.15.104:0/2344586538 >> 172.21.15.104:0/2344586538 conn(0x55d459313680 msgr2=0x55d459313a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.103+0000 7f6985c38740 1 -- 172.21.15.104:0/2344586538 shutdown_connections 2024-06-09T22:22:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.103+0000 7f6985c38740 1 -- 172.21.15.104:0/2344586538 wait complete. 2024-06-09T22:22:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.103+0000 7f6985c38740 1 -- start start 2024-06-09T22:22:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f6985c38740 1 Processor -- start 2024-06-09T22:22:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f6985c38740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317170 0x55d45931f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f6985c38740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 0x55d45931f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f6985c38740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d45931fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f6985c38740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d459324f80 con 0x55d459317a90 2024-06-09T22:22:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f6985c38740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d459204ae0 con 0x55d459322350 2024-06-09T22:22:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f6985c38740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d459325100 con 0x55d459317170 2024-06-09T22:22:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f6985c38740 10 client.0 _ll_register_callbacks cb 0x55d45923d9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f697bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 0x55d45931f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f697bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 0x55d45931f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53458/0 (socket says 172.21.15.104:53458) 2024-06-09T22:22:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f697bfff640 1 -- 172.21.15.104:0/2417095256 learned_addr learned my addr 172.21.15.104:0/2417095256 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f6980be9640 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317170 0x55d45931f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f69813ea640 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d45931fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f697bfff640 1 -- 172.21.15.104:0/2417095256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317170 msgr2=0x55d45931f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f697bfff640 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317170 0x55d45931f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f697bfff640 1 -- 172.21.15.104:0/2417095256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 msgr2=0x55d45931fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f697bfff640 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d45931fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.104+0000 7f697bfff640 1 -- 172.21.15.104:0/2417095256 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d458f8f210 con 0x55d459317a90 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f69813ea640 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d45931fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f697bfff640 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 0x55d45931f880 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f6968009900 tx=0x7f6968031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=301bfd173a9aa24b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f695bfff640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f695bfff640 1 -- 172.21.15.104:0/2417095256 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968035040 con 0x55d459317a90 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f695bfff640 1 -- 172.21.15.104:0/2417095256 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6968004240 con 0x55d459317a90 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f6985c38740 1 -- 172.21.15.104:0/2417095256 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69740a4d30 con 0x55d459317a90 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f695bfff640 1 -- 172.21.15.104:0/2417095256 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6968005d50 con 0x55d459317a90 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f6985c38740 10 client.15891 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f6985c38740 1 -- 172.21.15.104:0/2417095256 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d45905b620 con 0x55d459317a90 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f695bfff640 1 -- 172.21.15.104:0/2417095256 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6968004440 con 0x55d459317a90 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f695bfff640 1 -- 172.21.15.104:0/2417095256 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f69680035f0 con 0x55d459317a90 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f6985c38740 10 client.15891 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f6985c38740 10 client.15891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f6985c38740 20 client.15891 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.105+0000 7f6985c38740 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d45933ade0 0x55d45933d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.106+0000 7f6985c38740 4 client.15891 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.106+0000 7f6985c38740 1 -- 172.21.15.104:0/2417095256 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f69740a4d30 con 0x55d45933ade0 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.106+0000 7f6980be9640 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d45933ade0 0x55d45933d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.106+0000 7f6980be9640 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d45933ade0 0x55d45933d1f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1210dff1dba4b97 server_cookie=fdac9496a3f00efe in_seq=0 out_seq=0 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.106+0000 7f695bfff640 10 client.15891 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.107+0000 7f695bfff640 1 -- 172.21.15.104:0/2417095256 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x7f69740a4d30 con 0x55d45933ade0 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.107+0000 7f695bfff640 10 client.15891 handle_command_reply: tid=0 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.107+0000 7f6985c38740 1 client.15891 shutdown 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.107+0000 7f6985c38740 1 -- 172.21.15.104:0/2417095256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 msgr2=0x55d45931f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.107+0000 7f6985c38740 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 0x55d45931f880 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f6968009900 tx=0x7f6968031a20 comp rx=0 tx=0).stop 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.107+0000 7f6985c38740 1 -- 172.21.15.104:0/2417095256 shutdown_connections 2024-06-09T22:22:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.107+0000 7f6985c38740 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d45933ade0 0x55d45933d1f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.107+0000 7f6985c38740 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d45931fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.107+0000 7f6985c38740 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 0x55d45931f880 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.107+0000 7f695bfff640 0 client.15891 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.107+0000 7f6985c38740 1 --2- 172.21.15.104:0/2417095256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317170 0x55d45931f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.107+0000 7f6985c38740 1 -- 172.21.15.104:0/2417095256 >> 172.21.15.104:0/2417095256 conn(0x55d459313680 msgr2=0x55d459319400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.108+0000 7f6985c38740 1 -- 172.21.15.104:0/2417095256 shutdown_connections 2024-06-09T22:22:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.108+0000 7f6985c38740 1 -- 172.21.15.104:0/2417095256 wait complete. 2024-06-09T22:22:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.108+0000 7f6985c38740 20 client.15891 trim_cache size 0 max 16384 2024-06-09T22:22:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f6985c38740 1 Processor -- start 2024-06-09T22:22:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f6985c38740 1 -- start start 2024-06-09T22:22:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f6985c38740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317170 0x55d459312f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f6985c38740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 0x55d459319b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f6985c38740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d459342ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f6985c38740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f697c1acb50 con 0x55d459317a90 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f6985c38740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f697c1ae690 con 0x55d459322350 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f6985c38740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f697c1ad220 con 0x55d459317170 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f697bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 0x55d459319b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f697bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 0x55d459319b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53468/0 (socket says 172.21.15.104:53468) 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f697bfff640 1 -- 172.21.15.104:0/2383001149 learned_addr learned my addr 172.21.15.104:0/2383001149 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f69813ea640 1 --2- 172.21.15.104:0/2383001149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d459342ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f6980be9640 1 --2- 172.21.15.104:0/2383001149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317170 0x55d459312f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f697bfff640 1 -- 172.21.15.104:0/2383001149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317170 msgr2=0x55d459312f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f697bfff640 1 --2- 172.21.15.104:0/2383001149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317170 0x55d459312f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f697bfff640 1 -- 172.21.15.104:0/2383001149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 msgr2=0x55d459342ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f697bfff640 1 --2- 172.21.15.104:0/2383001149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d459342ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f697bfff640 1 -- 172.21.15.104:0/2383001149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d459324ab0 con 0x55d459317a90 2024-06-09T22:22:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.123+0000 7f69813ea640 1 --2- 172.21.15.104:0/2383001149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d459342ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.124+0000 7f697bfff640 1 --2- 172.21.15.104:0/2383001149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 0x55d459319b30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6968009850 tx=0x7f696802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ecbd9ac17524dbc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.124+0000 7f69667fc640 1 -- 172.21.15.104:0/2383001149 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968057070 con 0x55d459317a90 2024-06-09T22:22:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.124+0000 7f69667fc640 1 -- 172.21.15.104:0/2383001149 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6968004ba0 con 0x55d459317a90 2024-06-09T22:22:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.124+0000 7f69667fc640 1 -- 172.21.15.104:0/2383001149 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968035070 con 0x55d459317a90 2024-06-09T22:22:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.124+0000 7f6985c38740 1 -- 172.21.15.104:0/2383001149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 msgr2=0x55d459319b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.124+0000 7f6985c38740 1 --2- 172.21.15.104:0/2383001149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 0x55d459319b30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6968009850 tx=0x7f696802a480 comp rx=0 tx=0).stop 2024-06-09T22:22:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 -- 172.21.15.104:0/2383001149 shutdown_connections 2024-06-09T22:22:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 --2- 172.21.15.104:0/2383001149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d459342ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 --2- 172.21.15.104:0/2383001149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317a90 0x55d459319b30 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 --2- 172.21.15.104:0/2383001149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317170 0x55d459312f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 -- 172.21.15.104:0/2383001149 >> 172.21.15.104:0/2383001149 conn(0x55d459313680 msgr2=0x55d45931fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 -- 172.21.15.104:0/2383001149 shutdown_connections 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 -- 172.21.15.104:0/2383001149 wait complete. 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 -- start start 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 Processor -- start 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317170 0x55d45931c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317a90 0x55d45931ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d45931cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f697c1adfc0 con 0x55d459317170 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f697c1ad8f0 con 0x55d459322350 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f697c1aab10 con 0x55d459317a90 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6985c38740 10 client.0 _ll_register_callbacks cb 0x7f69680314d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6980be9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317170 0x55d45931c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.125+0000 7f6980be9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317170 0x55d45931c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53470/0 (socket says 172.21.15.104:53470) 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f6980be9640 1 -- 172.21.15.104:0/2196743296 learned_addr learned my addr 172.21.15.104:0/2196743296 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f697bfff640 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317a90 0x55d45931ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f69813ea640 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d45931cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f6980be9640 1 -- 172.21.15.104:0/2196743296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317a90 msgr2=0x55d45931ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f6980be9640 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317a90 0x55d45931ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f6980be9640 1 -- 172.21.15.104:0/2196743296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 msgr2=0x55d45931cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f6980be9640 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d45931cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f6980be9640 1 -- 172.21.15.104:0/2196743296 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4593a4b60 con 0x55d459317170 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f6980be9640 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317170 0x55d45931c4d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f697000c010 tx=0x7f6970004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb382a54bbe93d56 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f695bfff640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f695bfff640 1 -- 172.21.15.104:0/2196743296 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970028800 con 0x55d459317170 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f695bfff640 1 -- 172.21.15.104:0/2196743296 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f69700289a0 con 0x55d459317170 2024-06-09T22:22:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f6985c38740 1 -- 172.21.15.104:0/2196743296 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6968004ba0 con 0x55d459317170 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f695bfff640 1 -- 172.21.15.104:0/2196743296 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6970039050 con 0x55d459317170 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f6985c38740 10 client.15897 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f6985c38740 1 -- 172.21.15.104:0/2196743296 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d4593a5e40 con 0x55d459317170 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.126+0000 7f695bfff640 1 -- 172.21.15.104:0/2196743296 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6968004ba0 con 0x55d459317170 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.127+0000 7f695bfff640 1 -- 172.21.15.104:0/2196743296 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6970031420 con 0x55d459317170 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.127+0000 7f6985c38740 10 client.15897 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.127+0000 7f6985c38740 10 client.15897 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.127+0000 7f6985c38740 20 client.15897 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.127+0000 7f6985c38740 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d4593a6570 0x55d4593a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.127+0000 7f6985c38740 4 client.15897 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.127+0000 7f6985c38740 1 -- 172.21.15.104:0/2196743296 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d45928ef10 con 0x55d4593a6570 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.127+0000 7f697bfff640 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d4593a6570 0x55d4593a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.127+0000 7f697bfff640 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d4593a6570 0x55d4593a8990 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9967bb486cd4a55b server_cookie=66b01166c7ce2cb4 in_seq=0 out_seq=0 2024-06-09T22:22:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.127+0000 7f695bfff640 10 client.15897 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.133+0000 7f695bfff640 1 -- 172.21.15.104:0/2196743296 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55d45928ef10 con 0x55d4593a6570 2024-06-09T22:22:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.133+0000 7f695bfff640 10 client.15897 handle_command_reply: tid=0 2024-06-09T22:22:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.133+0000 7f6985c38740 1 client.15897 shutdown 2024-06-09T22:22:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.133+0000 7f6985c38740 1 -- 172.21.15.104:0/2196743296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317170 msgr2=0x55d45931c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.133+0000 7f6985c38740 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317170 0x55d45931c4d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f697000c010 tx=0x7f6970004730 comp rx=0 tx=0).stop 2024-06-09T22:22:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.133+0000 7f6985c38740 1 -- 172.21.15.104:0/2196743296 shutdown_connections 2024-06-09T22:22:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.133+0000 7f6985c38740 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d4593a6570 0x55d4593a8990 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.133+0000 7f6985c38740 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d459322350 0x55d45931cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.133+0000 7f695bfff640 0 client.15897 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.133+0000 7f6985c38740 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d459317a90 0x55d45931ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.133+0000 7f6985c38740 1 --2- 172.21.15.104:0/2196743296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d459317170 0x55d45931c4d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.133+0000 7f6985c38740 1 -- 172.21.15.104:0/2196743296 >> 172.21.15.104:0/2196743296 conn(0x55d459313680 msgr2=0x55d459335a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6985c38740 1 -- 172.21.15.104:0/2196743296 shutdown_connections 2024-06-09T22:22:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6985c38740 1 -- 172.21.15.104:0/2196743296 wait complete. 2024-06-09T22:22:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6985c38740 20 client.15897 trim_cache size 0 max 16384 2024-06-09T22:22:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:22:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:22:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:22:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6982e4d640 1 -- 172.21.15.104:0/2710006647 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f696c0a2da0 msgr2=0x7f696c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6982e4d640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f696c0a2da0 0x7f696c0a5260 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6970005fd0 tx=0x7f6970005850 comp rx=0 tx=0).stop 2024-06-09T22:22:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6982e4d640 1 -- 172.21.15.104:0/2710006647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10bff0 msgr2=0x7f697c19ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6982e4d640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10bff0 0x7f697c19ca50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f697400cce0 tx=0x7f697400a620 comp rx=0 tx=0).stop 2024-06-09T22:22:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6982e4d640 1 -- 172.21.15.104:0/2710006647 shutdown_connections 2024-06-09T22:22:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6982e4d640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f696c0a2da0 0x7f696c0a5260 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6982e4d640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10bff0 0x7f697c19ca50 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6982e4d640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f697c103540 0x7f697c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6982e4d640 1 --2- 172.21.15.104:0/2710006647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f697c102c20 0x7f697c1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6982e4d640 1 -- 172.21.15.104:0/2710006647 >> 172.21.15.104:0/2710006647 conn(0x7f697c0779f0 msgr2=0x7f697c0fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6982e4d640 1 -- 172.21.15.104:0/2710006647 shutdown_connections 2024-06-09T22:22:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:32.134+0000 7f6982e4d640 1 -- 172.21.15.104:0/2710006647 wait complete. 2024-06-09T22:22:32.289 INFO:tasks.workunit.client.0.smithi104.stderr:Updating files: 95% (11991/12576) Updating files: 96% (12073/12576) Updating files: 97% (12199/12576) Updating files: 98% (12325/12576) Updating files: 99% (12451/12576) Updating files: 100% (12576/12576) Updating files: 100% (12576/12576), done. 2024-06-09T22:22:32.783 INFO:tasks.workunit.client.0.smithi104.stderr:Note: switching to '7b7a5d375cf4e4afab96b9d4d229b699ae8304bb'. 2024-06-09T22:22:32.783 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-06-09T22:22:32.783 INFO:tasks.workunit.client.0.smithi104.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-06-09T22:22:32.783 INFO:tasks.workunit.client.0.smithi104.stderr:changes and commit them, and you can discard any commits you make in this 2024-06-09T22:22:32.783 INFO:tasks.workunit.client.0.smithi104.stderr:state without impacting any branches by switching back to a branch. 2024-06-09T22:22:32.783 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-06-09T22:22:32.783 INFO:tasks.workunit.client.0.smithi104.stderr:If you want to create a new branch to retain commits you create, you may 2024-06-09T22:22:32.784 INFO:tasks.workunit.client.0.smithi104.stderr:do so (now or later) by using -c with the switch command. Example: 2024-06-09T22:22:32.784 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-06-09T22:22:32.784 INFO:tasks.workunit.client.0.smithi104.stderr: git switch -c 2024-06-09T22:22:32.784 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-06-09T22:22:32.784 INFO:tasks.workunit.client.0.smithi104.stderr:Or undo this operation with: 2024-06-09T22:22:32.784 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-06-09T22:22:32.784 INFO:tasks.workunit.client.0.smithi104.stderr: git switch - 2024-06-09T22:22:32.784 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-06-09T22:22:32.784 INFO:tasks.workunit.client.0.smithi104.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-06-09T22:22:32.784 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-06-09T22:22:32.784 INFO:tasks.workunit.client.0.smithi104.stderr:HEAD is now at 7b7a5d375cf Merge pull request #57941 from zdover23/wip-doc-2024-06-09-backport-57939-to-squid 2024-06-09T22:22:32.787 DEBUG:teuthology.orchestra.run.smithi104:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-06-09T22:22:32.862 INFO:tasks.workunit.client.0.smithi104.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-06-09T22:22:32.864 INFO:tasks.workunit.client.0.smithi104.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-06-09T22:22:32.864 INFO:tasks.workunit.client.0.smithi104.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-06-09T22:22:33.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:33 smithi104 ceph-mon[32389]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.3 KiB/s wr, 13 op/s 2024-06-09T22:22:33.808 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:33 smithi194 ceph-mon[36856]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.3 KiB/s wr, 13 op/s 2024-06-09T22:22:33.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:33 smithi153 ceph-mon[36599]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.3 KiB/s wr, 13 op/s 2024-06-09T22:22:34.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.124+0000 7fb93a290700 1 -- 172.21.15.104:0/2593992554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb934102000 msgr2=0x7fb934105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:34.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.124+0000 7fb93a290700 1 --2- 172.21.15.104:0/2593992554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb934102000 0x7fb934105fc0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb9280099c0 tx=0x7fb928009cd0 comp rx=0 tx=0).stop 2024-06-09T22:22:34.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.124+0000 7fb93a290700 1 -- 172.21.15.104:0/2593992554 shutdown_connections 2024-06-09T22:22:34.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.124+0000 7fb93a290700 1 --2- 172.21.15.104:0/2593992554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9341066a0 0x7fb93410e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.124+0000 7fb93a290700 1 --2- 172.21.15.104:0/2593992554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb934101650 0x7fb934101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.124+0000 7fb93a290700 1 --2- 172.21.15.104:0/2593992554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb934102000 0x7fb934105fc0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.124+0000 7fb93a290700 1 -- 172.21.15.104:0/2593992554 >> 172.21.15.104:0/2593992554 conn(0x7fb9340773e0 msgr2=0x7fb9340777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.125+0000 7fb93a290700 1 -- 172.21.15.104:0/2593992554 shutdown_connections 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.125+0000 7fb93a290700 1 -- 172.21.15.104:0/2593992554 wait complete. 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.125+0000 7fb93a290700 1 Processor -- start 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.125+0000 7fb93a290700 1 -- start start 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb93a290700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb934101650 0x7fb93419ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb93a290700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb934102000 0x7fb93419b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb93a290700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9341066a0 0x7fb93419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb93a290700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb9341116e0 con 0x7fb934101650 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb93a290700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb934111560 con 0x7fb9341066a0 2024-06-09T22:22:34.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb93a290700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb934111860 con 0x7fb934102000 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb933fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb934101650 0x7fb93419ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb933fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb934101650 0x7fb93419ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53486/0 (socket says 172.21.15.104:53486) 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb933fff700 1 -- 172.21.15.104:0/4253378934 learned_addr learned my addr 172.21.15.104:0/4253378934 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb93882d700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9341066a0 0x7fb93419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb9337fe700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb934102000 0x7fb93419b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb933fff700 1 -- 172.21.15.104:0/4253378934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb934102000 msgr2=0x7fb93419b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb933fff700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb934102000 0x7fb93419b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb933fff700 1 -- 172.21.15.104:0/4253378934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9341066a0 msgr2=0x7fb93419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb933fff700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9341066a0 0x7fb93419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb933fff700 1 -- 172.21.15.104:0/4253378934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb93419fe70 con 0x7fb934101650 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb933fff700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb934101650 0x7fb93419ae60 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb92400d950 tx=0x7fb92400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb9317fa700 1 -- 172.21.15.104:0/4253378934 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb924016070 con 0x7fb934101650 2024-06-09T22:22:34.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb93a290700 1 -- 172.21.15.104:0/4253378934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9341a0160 con 0x7fb934101650 2024-06-09T22:22:34.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb9317fa700 1 -- 172.21.15.104:0/4253378934 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb924009e20 con 0x7fb934101650 2024-06-09T22:22:34.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.126+0000 7fb9317fa700 1 -- 172.21.15.104:0/4253378934 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92400f620 con 0x7fb934101650 2024-06-09T22:22:34.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.127+0000 7fb93a290700 1 -- 172.21.15.104:0/4253378934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9341a79e0 con 0x7fb934101650 2024-06-09T22:22:34.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.127+0000 7fb93a290700 1 -- 172.21.15.104:0/4253378934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb934069180 con 0x7fb934101650 2024-06-09T22:22:34.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.128+0000 7fb9317fa700 1 -- 172.21.15.104:0/4253378934 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fb924019070 con 0x7fb934101650 2024-06-09T22:22:34.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.129+0000 7fb9317fa700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb91c0a2d90 0x7fb91c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:34.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.129+0000 7fb9337fe700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb91c0a2d90 0x7fb91c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:34.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.129+0000 7fb9317fa700 1 -- 172.21.15.104:0/4253378934 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb9240cf390 con 0x7fb934101650 2024-06-09T22:22:34.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.130+0000 7fb9337fe700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb91c0a2d90 0x7fb91c0a5250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb928000c00 tx=0x7fb928005fd0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:34.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.134+0000 7fb9317fa700 1 -- 172.21.15.104:0/4253378934 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb92409bc40 con 0x7fb934101650 2024-06-09T22:22:34.392 INFO:tasks.workunit.client.0.smithi104.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-06-09T22:22:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.395+0000 7fb93a290700 1 -- 172.21.15.104:0/4253378934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9341a7bb0 con 0x7fb934101650 2024-06-09T22:22:34.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.396+0000 7fb9317fa700 1 -- 172.21.15.104:0/4253378934 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11953 (secure 0 0 0) 0x7fb92409b540 con 0x7fb934101650 2024-06-09T22:22:34.395 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:22:34.396 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:20:55.680750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24460,24448,24442,24403,24419]},"id":1}]} 2024-06-09T22:22:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.398+0000 7fb93a290700 1 -- 172.21.15.104:0/4253378934 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb91c0a2d90 msgr2=0x7fb91c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.398+0000 7fb93a290700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb91c0a2d90 0x7fb91c0a5250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb928000c00 tx=0x7fb928005fd0 comp rx=0 tx=0).stop 2024-06-09T22:22:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.398+0000 7fb93a290700 1 -- 172.21.15.104:0/4253378934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb934101650 msgr2=0x7fb93419ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.398+0000 7fb93a290700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb934101650 0x7fb93419ae60 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb92400d950 tx=0x7fb92400dc60 comp rx=0 tx=0).stop 2024-06-09T22:22:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.399+0000 7fb93a290700 1 -- 172.21.15.104:0/4253378934 shutdown_connections 2024-06-09T22:22:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.399+0000 7fb93a290700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb91c0a2d90 0x7fb91c0a5250 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.399+0000 7fb93a290700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb934101650 0x7fb93419ae60 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.399+0000 7fb93a290700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9341066a0 0x7fb93419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.399+0000 7fb93a290700 1 --2- 172.21.15.104:0/4253378934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb934102000 0x7fb93419b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.399+0000 7fb93a290700 1 -- 172.21.15.104:0/4253378934 >> 172.21.15.104:0/4253378934 conn(0x7fb9340773e0 msgr2=0x7fb93406aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.399+0000 7fb93a290700 1 -- 172.21.15.104:0/4253378934 shutdown_connections 2024-06-09T22:22:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:34.399+0000 7fb93a290700 1 -- 172.21.15.104:0/4253378934 wait complete. 2024-06-09T22:22:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 29 2024-06-09T22:22:34.542 INFO:tasks.workunit.client.0.smithi104.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-06-09T22:22:34.582 INFO:tasks.workunit.client.0.smithi104.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-06-09T22:22:34.584 INFO:tasks.workunit.client.0.smithi104.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-06-09T22:22:34.584 INFO:tasks.workunit.client.0.smithi104.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-06-09T22:22:34.624 INFO:tasks.workunit.client.0.smithi104.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-06-09T22:22:34.628 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:22:34.628 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-06-09T22:22:34.644 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-06-09T22:22:34.645 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-06-09T22:22:34.645 DEBUG:teuthology.orchestra.run.smithi104:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7b7a5d375cf4e4afab96b9d4d229b699ae8304bb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-06-09T22:22:34.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:34 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/4253378934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:22:34.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:34 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/4253378934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:22:35.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:34 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/4253378934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:22:35.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-09T22:22:35.456 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:35.857 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:35 smithi104 ceph-mon[32389]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.6 KiB/s wr, 18 op/s 2024-06-09T22:22:35.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:35 smithi153 ceph-mon[36599]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.6 KiB/s wr, 18 op/s 2024-06-09T22:22:35.959 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:35 smithi194 ceph-mon[36856]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.6 KiB/s wr, 18 op/s 2024-06-09T22:22:36.936 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:36 smithi104 ceph-mon[32389]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-06-09T22:22:36.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:36 smithi153 ceph-mon[36599]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-06-09T22:22:37.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:36 smithi194 ceph-mon[36856]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-06-09T22:22:37.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.409+0000 7f3caa2fa700 1 -- 172.21.15.104:0/2336175835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ca41014c0 msgr2=0x7f3ca4101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:37.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.409+0000 7f3caa2fa700 1 --2- 172.21.15.104:0/2336175835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ca41014c0 0x7f3ca4101940 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009990 tx=0x7f3c98009ca0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.409+0000 7f3caa2fa700 1 -- 172.21.15.104:0/2336175835 shutdown_connections 2024-06-09T22:22:37.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.409+0000 7f3caa2fa700 1 --2- 172.21.15.104:0/2336175835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca4101e80 0x7f3ca410e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.409+0000 7f3caa2fa700 1 --2- 172.21.15.104:0/2336175835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ca4107470 0x7f3ca4107850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.409+0000 7f3caa2fa700 1 --2- 172.21.15.104:0/2336175835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ca41014c0 0x7f3ca4101940 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.409+0000 7f3caa2fa700 1 -- 172.21.15.104:0/2336175835 >> 172.21.15.104:0/2336175835 conn(0x7f3ca40772a0 msgr2=0x7f3ca40776b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:37.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.409+0000 7f3caa2fa700 1 -- 172.21.15.104:0/2336175835 shutdown_connections 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.409+0000 7f3caa2fa700 1 -- 172.21.15.104:0/2336175835 wait complete. 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.410+0000 7f3caa2fa700 1 Processor -- start 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.410+0000 7f3caa2fa700 1 -- start start 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.410+0000 7f3caa2fa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca41014c0 0x7f3ca4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.410+0000 7f3caa2fa700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ca4101e80 0x7f3ca4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.410+0000 7f3caa2fa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ca4107470 0x7f3ca41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.410+0000 7f3caa2fa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca4111750 con 0x7f3ca4107470 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.410+0000 7f3caa2fa700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca41115d0 con 0x7f3ca41014c0 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.410+0000 7f3caa2fa700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca41118d0 con 0x7f3ca4101e80 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca41014c0 0x7f3ca4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca41014c0 0x7f3ca4197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:36186/0 (socket says 172.21.15.104:36186) 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca3fff700 1 -- 172.21.15.104:0/376552568 learned_addr learned my addr 172.21.15.104:0/376552568 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:37.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca8897700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ca4107470 0x7f3ca41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca37fe700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ca4101e80 0x7f3ca4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca3fff700 1 -- 172.21.15.104:0/376552568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ca4101e80 msgr2=0x7f3ca4197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ca4101e80 0x7f3ca4197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca3fff700 1 -- 172.21.15.104:0/376552568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ca4107470 msgr2=0x7f3ca41984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ca4107470 0x7f3ca41984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca3fff700 1 -- 172.21.15.104:0/376552568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca41a72e0 con 0x7f3ca41014c0 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca8897700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ca4107470 0x7f3ca41984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca41014c0 0x7f3ca4197a60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400b530 tx=0x7f3c9400b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca17fa700 1 -- 172.21.15.104:0/376552568 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c94015020 con 0x7f3ca41014c0 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3caa2fa700 1 -- 172.21.15.104:0/376552568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ca41a75d0 con 0x7f3ca41014c0 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca17fa700 1 -- 172.21.15.104:0/376552568 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c94011d10 con 0x7f3ca41014c0 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.411+0000 7f3ca17fa700 1 -- 172.21.15.104:0/376552568 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400f2e0 con 0x7f3ca41014c0 2024-06-09T22:22:37.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.412+0000 7f3caa2fa700 1 -- 172.21.15.104:0/376552568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ca41a7b30 con 0x7f3ca41014c0 2024-06-09T22:22:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.414+0000 7f3ca17fa700 1 -- 172.21.15.104:0/376552568 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f3c94018020 con 0x7f3ca41014c0 2024-06-09T22:22:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3ca17fa700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f3c8c0a2d90 0x7f3c8c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3ca37fe700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f3c8c0a2d90 0x7f3c8c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3ca17fa700 1 -- 172.21.15.104:0/376552568 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f3c940cedb0 con 0x7f3ca41014c0 2024-06-09T22:22:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3cb4b7db80 1 Processor -- start 2024-06-09T22:22:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3cb4b7db80 1 -- start start 2024-06-09T22:22:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3ca37fe700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f3c8c0a2d90 0x7f3c8c0a5250 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800b5c0 tx=0x7f3c98005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3cb4b7db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4cf60 0x55d126e4d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3cb4b7db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 0x55d126e47260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3cb4b7db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e477a0 0x55d126e47bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3cb4b7db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d126d39f60 con 0x55d126e4d880 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3cb4b7db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d126d98850 con 0x55d126e4cf60 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3cb4b7db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d126d00460 con 0x55d126e477a0 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3ca37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 0x55d126e47260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3ca37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 0x55d126e47260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50626/0 (socket says 172.21.15.104:50626) 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.415+0000 7f3ca37fe700 1 -- 172.21.15.104:0/1944035813 learned_addr learned my addr 172.21.15.104:0/1944035813 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.416+0000 7f3ca8897700 1 --2- 172.21.15.104:0/1944035813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e477a0 0x55d126e47bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.416+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/1944035813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4cf60 0x55d126e4d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.416+0000 7f3ca37fe700 1 -- 172.21.15.104:0/1944035813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e477a0 msgr2=0x55d126e47bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.416+0000 7f3ca37fe700 1 --2- 172.21.15.104:0/1944035813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e477a0 0x55d126e47bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.416+0000 7f3ca37fe700 1 -- 172.21.15.104:0/1944035813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4cf60 msgr2=0x55d126e4d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.416+0000 7f3ca37fe700 1 --2- 172.21.15.104:0/1944035813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4cf60 0x55d126e4d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.416+0000 7f3ca37fe700 1 -- 172.21.15.104:0/1944035813 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d126ce78d0 con 0x55d126e4d880 2024-06-09T22:22:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.416+0000 7f3ca37fe700 1 --2- 172.21.15.104:0/1944035813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 0x55d126e47260 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3c98015000 tx=0x7f3c98020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb88dbf7be224b3e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.416+0000 7f3c8a7fc700 1 -- 172.21.15.104:0/1944035813 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9800bda0 con 0x55d126e4d880 2024-06-09T22:22:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.416+0000 7f3c8a7fc700 1 -- 172.21.15.104:0/1944035813 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c9800fd50 con 0x55d126e4d880 2024-06-09T22:22:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.416+0000 7f3c8a7fc700 1 -- 172.21.15.104:0/1944035813 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9802f400 con 0x55d126e4d880 2024-06-09T22:22:37.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.417+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/1944035813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 msgr2=0x55d126e47260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:37.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.417+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/1944035813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 0x55d126e47260 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3c98015000 tx=0x7f3c98020000 comp rx=0 tx=0).stop 2024-06-09T22:22:37.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.417+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/1944035813 shutdown_connections 2024-06-09T22:22:37.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.417+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/1944035813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e477a0 0x55d126e47bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.417+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/1944035813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4cf60 0x55d126e4d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.417+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/1944035813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 0x55d126e47260 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.417+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/1944035813 >> 172.21.15.104:0/1944035813 conn(0x55d126d74080 msgr2=0x55d126e42220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.417+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/1944035813 shutdown_connections 2024-06-09T22:22:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.417+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/1944035813 wait complete. 2024-06-09T22:22:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.417+0000 7f3cb4b7db80 1 -- start start 2024-06-09T22:22:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.417+0000 7f3cb4b7db80 1 Processor -- start 2024-06-09T22:22:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3cb4b7db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e477a0 0x55d126e53650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3cb4b7db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 0x55d126e50950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3cb4b7db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 0x55d126e50e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3cb4b7db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d126d38fc0 con 0x55d126e4d880 2024-06-09T22:22:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3cb4b7db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d126d94810 con 0x55d126e477a0 2024-06-09T22:22:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3cb4b7db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d126e541a0 con 0x55d126e4cf60 2024-06-09T22:22:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3cb4b7db80 10 client.0 _ll_register_callbacks cb 0x55d126d2e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3ca8897700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 0x55d126e50e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3ca8897700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 0x55d126e50e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50642/0 (socket says 172.21.15.104:50642) 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3ca8897700 1 -- 172.21.15.104:0/3379564603 learned_addr learned my addr 172.21.15.104:0/3379564603 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e477a0 0x55d126e53650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3ca37fe700 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 0x55d126e50950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3ca8897700 1 -- 172.21.15.104:0/3379564603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 msgr2=0x55d126e50950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3ca8897700 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 0x55d126e50950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3ca8897700 1 -- 172.21.15.104:0/3379564603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e477a0 msgr2=0x55d126e53650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3ca8897700 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e477a0 0x55d126e53650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3ca8897700 1 -- 172.21.15.104:0/3379564603 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d126d5e750 con 0x55d126e4d880 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3ca8897700 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 0x55d126e50e90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c00aed0 tx=0x7f3c9c012510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ba77e79b44262c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.418+0000 7f3c837fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3c837fe700 1 -- 172.21.15.104:0/3379564603 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c01a070 con 0x55d126e4d880 2024-06-09T22:22:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3c837fe700 1 -- 172.21.15.104:0/3379564603 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f3c9c012cb0 con 0x55d126e4d880 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3379564603 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c9800fd50 con 0x55d126e4d880 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3c837fe700 1 -- 172.21.15.104:0/3379564603 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c9c026050 con 0x55d126e4d880 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3cb4b7db80 10 client.15915 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3379564603 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d126e5ea10 con 0x55d126e4d880 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3c837fe700 1 -- 172.21.15.104:0/3379564603 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c9800fd50 con 0x55d126e4d880 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3c837fe700 1 -- 172.21.15.104:0/3379564603 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3c9c0083f0 con 0x55d126e4d880 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3cb4b7db80 10 client.15915 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3cb4b7db80 10 client.15915 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3cb4b7db80 20 client.15915 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55d126e60fa0 0x55d126e633f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3cb4b7db80 4 client.15915 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.419+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3379564603 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d126da5d50 con 0x55d126e60fa0 2024-06-09T22:22:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.420+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55d126e60fa0 0x55d126e633f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.420+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55d126e60fa0 0x55d126e633f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e52a0aa6c1a0121 server_cookie=aad2ef48d16e00de in_seq=0 out_seq=0 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.420+0000 7f3c837fe700 10 client.15915 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3c837fe700 1 -- 172.21.15.104:0/3379564603 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d126da5d50 con 0x55d126e60fa0 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3c837fe700 10 client.15915 handle_command_reply: tid=0 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3cb4b7db80 1 client.15915 shutdown 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3379564603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 msgr2=0x55d126e50e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 0x55d126e50e90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c00aed0 tx=0x7f3c9c012510 comp rx=0 tx=0).stop 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3379564603 shutdown_connections 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e4d880 0x55d126e50e90 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55d126e60fa0 0x55d126e633f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e477a0 0x55d126e53650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/3379564603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 0x55d126e50950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3c837fe700 0 client.15915 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:22:37.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3379564603 >> 172.21.15.104:0/3379564603 conn(0x55d126d74080 msgr2=0x55d126e41700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:37.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3379564603 shutdown_connections 2024-06-09T22:22:37.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3379564603 wait complete. 2024-06-09T22:22:37.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.421+0000 7f3cb4b7db80 20 client.15915 trim_cache size 0 max 16384 2024-06-09T22:22:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3cb4b7db80 1 Processor -- start 2024-06-09T22:22:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3cb4b7db80 1 -- start start 2024-06-09T22:22:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3cb4b7db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 0x55d126e47260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3cb4b7db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 0x55d126e4e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3cb4b7db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4d880 0x55d126e53190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3cb4b7db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca41b06f0 con 0x55d126e477a0 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3cb4b7db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca41b2230 con 0x55d126e4d880 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3cb4b7db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca41b0dc0 con 0x55d126e4cf60 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3ca3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 0x55d126e47260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3ca3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 0x55d126e47260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50656/0 (socket says 172.21.15.104:50656) 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3ca3fff700 1 -- 172.21.15.104:0/2470186116 learned_addr learned my addr 172.21.15.104:0/2470186116 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3ca37fe700 1 --2- 172.21.15.104:0/2470186116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 0x55d126e4e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3ca8897700 1 --2- 172.21.15.104:0/2470186116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4d880 0x55d126e53190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.436+0000 7f3ca3fff700 1 -- 172.21.15.104:0/2470186116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 msgr2=0x55d126e4e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/2470186116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 0x55d126e4e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3ca3fff700 1 -- 172.21.15.104:0/2470186116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4d880 msgr2=0x55d126e53190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:37.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/2470186116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4d880 0x55d126e53190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3ca3fff700 1 -- 172.21.15.104:0/2470186116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d126e63a70 con 0x55d126e477a0 2024-06-09T22:22:37.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/2470186116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 0x55d126e47260 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400b500 tx=0x7f3c9401c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5251918c09101181 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:37.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3c89ffb700 1 -- 172.21.15.104:0/2470186116 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9409f420 con 0x55d126e477a0 2024-06-09T22:22:37.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3c89ffb700 1 -- 172.21.15.104:0/2470186116 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c940a3ce0 con 0x55d126e477a0 2024-06-09T22:22:37.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3c89ffb700 1 -- 172.21.15.104:0/2470186116 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c940172c0 con 0x55d126e477a0 2024-06-09T22:22:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/2470186116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 msgr2=0x55d126e47260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/2470186116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 0x55d126e47260 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400b500 tx=0x7f3c9401c040 comp rx=0 tx=0).stop 2024-06-09T22:22:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/2470186116 shutdown_connections 2024-06-09T22:22:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/2470186116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 0x55d126e47260 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/2470186116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4d880 0x55d126e53190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.437+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/2470186116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 0x55d126e4e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/2470186116 >> 172.21.15.104:0/2470186116 conn(0x55d126d74080 msgr2=0x55d126e4ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/2470186116 shutdown_connections 2024-06-09T22:22:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/2470186116 wait complete. 2024-06-09T22:22:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3cb4b7db80 1 -- start start 2024-06-09T22:22:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3cb4b7db80 1 Processor -- start 2024-06-09T22:22:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3cb4b7db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 0x55d126e59c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3cb4b7db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 0x55d126e685d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3cb4b7db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4d880 0x55d126e68b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3cb4b7db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca41b1b60 con 0x55d126e477a0 2024-06-09T22:22:37.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3cb4b7db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca41b1490 con 0x55d126e4d880 2024-06-09T22:22:37.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3cb4b7db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca41ae670 con 0x55d126e4cf60 2024-06-09T22:22:37.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3cb4b7db80 10 client.0 _ll_register_callbacks cb 0x55d126d2e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:37.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3ca3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 0x55d126e59c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3ca3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 0x55d126e59c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50662/0 (socket says 172.21.15.104:50662) 2024-06-09T22:22:37.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.438+0000 7f3ca3fff700 1 -- 172.21.15.104:0/3330032193 learned_addr learned my addr 172.21.15.104:0/3330032193 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:37.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3ca37fe700 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 0x55d126e685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3ca8897700 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4d880 0x55d126e68b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3ca3fff700 1 -- 172.21.15.104:0/3330032193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 msgr2=0x55d126e685d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:37.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 0x55d126e685d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3ca3fff700 1 -- 172.21.15.104:0/3330032193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4d880 msgr2=0x55d126e68b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4d880 0x55d126e68b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3ca3fff700 1 -- 172.21.15.104:0/3330032193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d126e693c0 con 0x55d126e477a0 2024-06-09T22:22:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3ca3fff700 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 0x55d126e59c80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3c940048a0 tx=0x7f3c9409ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8906e349224e6633 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3c837fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3c837fe700 1 -- 172.21.15.104:0/3330032193 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c94099370 con 0x55d126e477a0 2024-06-09T22:22:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3c837fe700 1 -- 172.21.15.104:0/3330032193 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f3c940105a0 con 0x55d126e477a0 2024-06-09T22:22:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3330032193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c940a3ce0 con 0x55d126e477a0 2024-06-09T22:22:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3c837fe700 1 -- 172.21.15.104:0/3330032193 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c940660d0 con 0x55d126e477a0 2024-06-09T22:22:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3cb4b7db80 10 client.15921 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3330032193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d126ecbd10 con 0x55d126e477a0 2024-06-09T22:22:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.439+0000 7f3c837fe700 1 -- 172.21.15.104:0/3330032193 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c940a3ce0 con 0x55d126e477a0 2024-06-09T22:22:37.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.440+0000 7f3c837fe700 1 -- 172.21.15.104:0/3330032193 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3c9409b2d0 con 0x55d126e477a0 2024-06-09T22:22:37.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.440+0000 7f3cb4b7db80 10 client.15921 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:37.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.440+0000 7f3cb4b7db80 10 client.15921 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:22:37.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.440+0000 7f3cb4b7db80 20 client.15921 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:37.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.440+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55d126ecc2b0 0x55d126ece6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:37.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.440+0000 7f3cb4b7db80 4 client.15921 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:22:37.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.440+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3330032193 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d126da02d0 con 0x55d126ecc2b0 2024-06-09T22:22:37.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.440+0000 7f3ca37fe700 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55d126ecc2b0 0x55d126ece6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:37.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.440+0000 7f3c837fe700 10 client.15921 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:22:37.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.440+0000 7f3ca37fe700 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55d126ecc2b0 0x55d126ece6a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=27ce8cf7b3d8122c server_cookie=6571191e328d0b6 in_seq=0 out_seq=0 2024-06-09T22:22:37.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3c837fe700 1 -- 172.21.15.104:0/3330032193 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d126da02d0 con 0x55d126ecc2b0 2024-06-09T22:22:37.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3c837fe700 10 client.15921 handle_command_reply: tid=0 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3cb4b7db80 1 client.15921 shutdown 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3330032193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 msgr2=0x55d126e59c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 0x55d126e59c80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3c940048a0 tx=0x7f3c9409ac30 comp rx=0 tx=0).stop 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3330032193 shutdown_connections 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d126e477a0 0x55d126e59c80 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55d126ecc2b0 0x55d126ece6a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d126e4d880 0x55d126e68b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3cb4b7db80 1 --2- 172.21.15.104:0/3330032193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d126e4cf60 0x55d126e685d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3330032193 >> 172.21.15.104:0/3330032193 conn(0x55d126d74080 msgr2=0x55d126e4e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3c837fe700 0 client.15921 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3330032193 shutdown_connections 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3cb4b7db80 1 -- 172.21.15.104:0/3330032193 wait complete. 2024-06-09T22:22:37.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.441+0000 7f3cb4b7db80 20 client.15921 trim_cache size 0 max 16384 2024-06-09T22:22:37.448 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:22:37.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.442+0000 7f3caa2fa700 1 -- 172.21.15.104:0/376552568 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f3c8c0a2d90 msgr2=0x7f3c8c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:37.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.442+0000 7f3caa2fa700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f3c8c0a2d90 0x7f3c8c0a5250 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800b5c0 tx=0x7f3c98005960 comp rx=0 tx=0).stop 2024-06-09T22:22:37.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.442+0000 7f3caa2fa700 1 -- 172.21.15.104:0/376552568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca41014c0 msgr2=0x7f3ca4197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:37.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.442+0000 7f3caa2fa700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca41014c0 0x7f3ca4197a60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400b530 tx=0x7f3c9400b840 comp rx=0 tx=0).stop 2024-06-09T22:22:37.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.442+0000 7f3caa2fa700 1 -- 172.21.15.104:0/376552568 shutdown_connections 2024-06-09T22:22:37.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.442+0000 7f3caa2fa700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f3c8c0a2d90 0x7f3c8c0a5250 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.442+0000 7f3caa2fa700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ca4107470 0x7f3ca41984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.442+0000 7f3caa2fa700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ca41014c0 0x7f3ca4197a60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.442+0000 7f3caa2fa700 1 --2- 172.21.15.104:0/376552568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ca4101e80 0x7f3ca4197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:37.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.442+0000 7f3caa2fa700 1 -- 172.21.15.104:0/376552568 >> 172.21.15.104:0/376552568 conn(0x7f3ca40772a0 msgr2=0x7f3ca40ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:37.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.442+0000 7f3caa2fa700 1 -- 172.21.15.104:0/376552568 shutdown_connections 2024-06-09T22:22:37.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:37.442+0000 7f3caa2fa700 1 -- 172.21.15.104:0/376552568 wait complete. 2024-06-09T22:22:38.272 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-09T22:22:38.516 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:38.961 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:38 smithi104 ceph-mon[32389]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-06-09T22:22:38.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:38 smithi153 ceph-mon[36599]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-06-09T22:22:39.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:38 smithi194 ceph-mon[36856]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-06-09T22:22:40.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.250+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/4208980145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb01084f0 msgr2=0x7fbdb01088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:40.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.250+0000 7fbdb5ea7700 1 --2- 172.21.15.104:0/4208980145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb01084f0 0x7fbdb01088d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009990 tx=0x7fbda4009ca0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.251+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/4208980145 shutdown_connections 2024-06-09T22:22:40.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.251+0000 7fbdb5ea7700 1 --2- 172.21.15.104:0/4208980145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb01084f0 0x7fbdb01088d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.251+0000 7fbdb5ea7700 1 --2- 172.21.15.104:0/4208980145 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdb0102f10 0x7fbdb010f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.251+0000 7fbdb5ea7700 1 --2- 172.21.15.104:0/4208980145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdb0102550 0x7fbdb01029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.251+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/4208980145 >> 172.21.15.104:0/4208980145 conn(0x7fbdb0079650 msgr2=0x7fbdb0079a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:40.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.251+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/4208980145 shutdown_connections 2024-06-09T22:22:40.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.251+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/4208980145 wait complete. 2024-06-09T22:22:40.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.251+0000 7fbdb5ea7700 1 Processor -- start 2024-06-09T22:22:40.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.251+0000 7fbdb5ea7700 1 -- start start 2024-06-09T22:22:40.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdb5ea7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdb0102550 0x7fbdb019e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdb5ea7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdb0102f10 0x7fbdb019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdb5ea7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb01084f0 0x7fbdb0198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdb5ea7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb01127d0 con 0x7fbdb01084f0 2024-06-09T22:22:40.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdb5ea7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb0112650 con 0x7fbdb0102550 2024-06-09T22:22:40.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdb5ea7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb0112950 con 0x7fbdb0102f10 2024-06-09T22:22:40.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb01084f0 0x7fbdb0198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb01084f0 0x7fbdb0198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50680/0 (socket says 172.21.15.104:50680) 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaffff700 1 -- 172.21.15.104:0/614149013 learned_addr learned my addr 172.21.15.104:0/614149013 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaeffd700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdb0102f10 0x7fbdb019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdb0102550 0x7fbdb019e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaffff700 1 -- 172.21.15.104:0/614149013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdb0102f10 msgr2=0x7fbdb019ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaffff700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdb0102f10 0x7fbdb019ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaffff700 1 -- 172.21.15.104:0/614149013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdb0102550 msgr2=0x7fbdb019e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaffff700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdb0102550 0x7fbdb019e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaffff700 1 -- 172.21.15.104:0/614149013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb0199230 con 0x7fbdb01084f0 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdb0102550 0x7fbdb019e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaeffd700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdb0102f10 0x7fbdb019ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdaffff700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb01084f0 0x7fbdb0198ad0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbda800b970 tx=0x7fbda800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:40.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdacff9700 1 -- 172.21.15.104:0/614149013 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda8019070 con 0x7fbdb01084f0 2024-06-09T22:22:40.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdacff9700 1 -- 172.21.15.104:0/614149013 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbda8014440 con 0x7fbdb01084f0 2024-06-09T22:22:40.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.252+0000 7fbdacff9700 1 -- 172.21.15.104:0/614149013 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda80075d0 con 0x7fbdb01084f0 2024-06-09T22:22:40.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.253+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/614149013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdb0199520 con 0x7fbdb01084f0 2024-06-09T22:22:40.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.253+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/614149013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdb00746c0 con 0x7fbdb01084f0 2024-06-09T22:22:40.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.255+0000 7fbdacff9700 1 -- 172.21.15.104:0/614149013 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fbda801c070 con 0x7fbdb01084f0 2024-06-09T22:22:40.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.255+0000 7fbdacff9700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbd980a2d90 0x7fbd980a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.256+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbd980a2d90 0x7fbd980a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.256+0000 7fbdacff9700 1 -- 172.21.15.104:0/614149013 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fbda80ceb40 con 0x7fbdb01084f0 2024-06-09T22:22:40.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.256+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbd980a2d90 0x7fbd980a5250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbda4000c00 tx=0x7fbda4005fb0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:40.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.256+0000 7fbdc072ab80 1 Processor -- start 2024-06-09T22:22:40.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.256+0000 7fbdc072ab80 1 -- start start 2024-06-09T22:22:40.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdc072ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745b4f60 0x5601745b5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdc072ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 0x5601745af260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdc072ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745af7a0 0x5601745afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdc072ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5601744a1f60 con 0x5601745af7a0 2024-06-09T22:22:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdc072ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560174500850 con 0x5601745b5880 2024-06-09T22:22:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdc072ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560174468460 con 0x5601745b4f60 2024-06-09T22:22:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdaffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745af7a0 0x5601745afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdaffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745af7a0 0x5601745afbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50688/0 (socket says 172.21.15.104:50688) 2024-06-09T22:22:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdaffff700 1 -- 172.21.15.104:0/2610312799 learned_addr learned my addr 172.21.15.104:0/2610312799 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdaeffd700 1 --2- 172.21.15.104:0/2610312799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 0x5601745af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/2610312799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745b4f60 0x5601745b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdaffff700 1 -- 172.21.15.104:0/2610312799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745b4f60 msgr2=0x5601745b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdaffff700 1 --2- 172.21.15.104:0/2610312799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745b4f60 0x5601745b5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdaffff700 1 -- 172.21.15.104:0/2610312799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 msgr2=0x5601745af260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdaffff700 1 --2- 172.21.15.104:0/2610312799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 0x5601745af260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdaffff700 1 -- 172.21.15.104:0/2610312799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56017444f8d0 con 0x5601745af7a0 2024-06-09T22:22:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbdaffff700 1 --2- 172.21.15.104:0/2610312799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745af7a0 0x5601745afbc0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fbda80a3540 tx=0x7fbda809b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e546ebaf9254f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbd95ffb700 1 -- 172.21.15.104:0/2610312799 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda80a3a70 con 0x5601745af7a0 2024-06-09T22:22:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbd95ffb700 1 -- 172.21.15.104:0/2610312799 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbda8099100 con 0x5601745af7a0 2024-06-09T22:22:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.257+0000 7fbd95ffb700 1 -- 172.21.15.104:0/2610312799 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda80a35e0 con 0x5601745af7a0 2024-06-09T22:22:40.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.258+0000 7fbdc072ab80 1 -- 172.21.15.104:0/2610312799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745af7a0 msgr2=0x5601745afbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:40.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.258+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/2610312799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745af7a0 0x5601745afbc0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fbda80a3540 tx=0x7fbda809b580 comp rx=0 tx=0).stop 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.258+0000 7fbdc072ab80 1 -- 172.21.15.104:0/2610312799 shutdown_connections 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.258+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/2610312799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745af7a0 0x5601745afbc0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.258+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/2610312799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745b4f60 0x5601745b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.258+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/2610312799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 0x5601745af260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.258+0000 7fbdc072ab80 1 -- 172.21.15.104:0/2610312799 >> 172.21.15.104:0/2610312799 conn(0x5601744dc080 msgr2=0x5601745aa220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.258+0000 7fbdc072ab80 1 -- 172.21.15.104:0/2610312799 shutdown_connections 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.258+0000 7fbdc072ab80 1 -- 172.21.15.104:0/2610312799 wait complete. 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdc072ab80 1 -- start start 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdc072ab80 1 Processor -- start 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdc072ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745af7a0 0x5601745bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdc072ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745b8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdc072ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 0x5601745b8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdc072ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5601744a0fc0 con 0x5601745b4f60 2024-06-09T22:22:40.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdc072ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5601744fc810 con 0x5601745b5880 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdc072ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5601745bc260 con 0x5601745af7a0 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdc072ab80 10 client.0 _ll_register_callbacks cb 0x5601744965b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdaeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745b8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdaeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745b8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50698/0 (socket says 172.21.15.104:50698) 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdaeffd700 1 -- 172.21.15.104:0/4213708214 learned_addr learned my addr 172.21.15.104:0/4213708214 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdaffff700 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 0x5601745b8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.259+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745af7a0 0x5601745bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbdaeffd700 1 -- 172.21.15.104:0/4213708214 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745af7a0 msgr2=0x5601745bb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbdaeffd700 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745af7a0 0x5601745bb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbdaeffd700 1 -- 172.21.15.104:0/4213708214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 msgr2=0x5601745b8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbdaeffd700 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 0x5601745b8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbdaeffd700 1 -- 172.21.15.104:0/4213708214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601744c6750 con 0x5601745b4f60 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbdaeffd700 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745b8950 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fbda00117c0 tx=0x7fbda0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb2d23ecb4727d4a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:40.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbd8effd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbd8effd700 1 -- 172.21.15.104:0/4213708214 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda000fa90 con 0x5601745b4f60 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbdc072ab80 1 -- 172.21.15.104:0/4213708214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbda8099100 con 0x5601745b4f60 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbd8effd700 1 -- 172.21.15.104:0/4213708214 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fbda000fc30 con 0x5601745b4f60 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbd8effd700 1 -- 172.21.15.104:0/4213708214 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbda0024050 con 0x5601745b4f60 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbdc072ab80 10 client.15933 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbdc072ab80 1 -- 172.21.15.104:0/4213708214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5601745c6b10 con 0x5601745b4f60 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbd8effd700 1 -- 172.21.15.104:0/4213708214 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbda8099100 con 0x5601745b4f60 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.260+0000 7fbd8effd700 1 -- 172.21.15.104:0/4213708214 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fbda0018030 con 0x5601745b4f60 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.261+0000 7fbdc072ab80 10 client.15933 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.261+0000 7fbdc072ab80 10 client.15933 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.261+0000 7fbdc072ab80 20 client.15933 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.261+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5601745c9140 0x5601745cb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.261+0000 7fbdc072ab80 4 client.15933 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.261+0000 7fbdc072ab80 1 -- 172.21.15.104:0/4213708214 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56017450dd50 con 0x5601745c9140 2024-06-09T22:22:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.261+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5601745c9140 0x5601745cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.261+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5601745c9140 0x5601745cb530 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=abae39114792ef34 server_cookie=5c2588c357d6f821 in_seq=0 out_seq=0 2024-06-09T22:22:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.261+0000 7fbd8effd700 10 client.15933 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:22:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.262+0000 7fbd8effd700 1 -- 172.21.15.104:0/4213708214 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56017450dd50 con 0x5601745c9140 2024-06-09T22:22:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.262+0000 7fbd8effd700 10 client.15933 handle_command_reply: tid=0 2024-06-09T22:22:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.262+0000 7fbdc072ab80 1 client.15933 shutdown 2024-06-09T22:22:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.262+0000 7fbdc072ab80 1 -- 172.21.15.104:0/4213708214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 msgr2=0x5601745b8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.262+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745b8950 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fbda00117c0 tx=0x7fbda0011ad0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.262+0000 7fbdc072ab80 1 -- 172.21.15.104:0/4213708214 shutdown_connections 2024-06-09T22:22:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.262+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 0x5601745b8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.262+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745af7a0 0x5601745bb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.262+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5601745c9140 0x5601745cb530 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.262+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/4213708214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745b8950 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.262+0000 7fbdc072ab80 1 -- 172.21.15.104:0/4213708214 >> 172.21.15.104:0/4213708214 conn(0x5601744dc080 msgr2=0x5601745a9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:40.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.262+0000 7fbd8effd700 0 client.15933 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:22:40.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.263+0000 7fbdc072ab80 1 -- 172.21.15.104:0/4213708214 shutdown_connections 2024-06-09T22:22:40.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.263+0000 7fbdc072ab80 1 -- 172.21.15.104:0/4213708214 wait complete. 2024-06-09T22:22:40.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.263+0000 7fbdc072ab80 20 client.15933 trim_cache size 0 max 16384 2024-06-09T22:22:40.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.277+0000 7fbdc072ab80 1 Processor -- start 2024-06-09T22:22:40.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdc072ab80 1 -- start start 2024-06-09T22:22:40.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdc072ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745af7a0 0x5601745af260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdc072ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdc072ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745b5880 0x5601745b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdc072ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb01b0800 con 0x5601745b4f60 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdc072ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb01b2340 con 0x5601745af7a0 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdc072ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb01b0ed0 con 0x5601745b5880 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdaeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdaeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745b78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50712/0 (socket says 172.21.15.104:50712) 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdaeffd700 1 -- 172.21.15.104:0/732824910 learned_addr learned my addr 172.21.15.104:0/732824910 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/732824910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745af7a0 0x5601745af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdaeffd700 1 -- 172.21.15.104:0/732824910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745b5880 msgr2=0x5601745b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdaffff700 1 --2- 172.21.15.104:0/732824910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745b5880 0x5601745b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdaeffd700 1 --2- 172.21.15.104:0/732824910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745b5880 0x5601745b64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdaeffd700 1 -- 172.21.15.104:0/732824910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745af7a0 msgr2=0x5601745af260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdaeffd700 1 --2- 172.21.15.104:0/732824910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745af7a0 0x5601745af260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdaeffd700 1 -- 172.21.15.104:0/732824910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601745cbb60 con 0x5601745b4f60 2024-06-09T22:22:40.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/732824910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745af7a0 0x5601745af260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:40.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.278+0000 7fbdaeffd700 1 --2- 172.21.15.104:0/732824910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745b78d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbda0005630 tx=0x7fbda0005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808ee9bc4aed13c1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:40.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.279+0000 7fbd957fa700 1 -- 172.21.15.104:0/732824910 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda002e020 con 0x5601745b4f60 2024-06-09T22:22:40.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.279+0000 7fbd957fa700 1 -- 172.21.15.104:0/732824910 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbda0019580 con 0x5601745b4f60 2024-06-09T22:22:40.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.279+0000 7fbd957fa700 1 -- 172.21.15.104:0/732824910 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda002a680 con 0x5601745b4f60 2024-06-09T22:22:40.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.279+0000 7fbdc072ab80 1 -- 172.21.15.104:0/732824910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 msgr2=0x5601745b78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:40.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.279+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/732824910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745b78d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbda0005630 tx=0x7fbda0005b20 comp rx=0 tx=0).stop 2024-06-09T22:22:40.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.279+0000 7fbdc072ab80 1 -- 172.21.15.104:0/732824910 shutdown_connections 2024-06-09T22:22:40.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.279+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/732824910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745b5880 0x5601745b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.279+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/732824910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745af7a0 0x5601745af260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.279+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/732824910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745b78d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.279+0000 7fbdc072ab80 1 -- 172.21.15.104:0/732824910 >> 172.21.15.104:0/732824910 conn(0x5601744dc080 msgr2=0x5601745ba180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.279+0000 7fbdc072ab80 1 -- 172.21.15.104:0/732824910 shutdown_connections 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.279+0000 7fbdc072ab80 1 -- 172.21.15.104:0/732824910 wait complete. 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdc072ab80 1 -- start start 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdc072ab80 1 Processor -- start 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdc072ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745af7a0 0x5601745c17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdc072ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745c1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdc072ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 0x5601745d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdc072ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb01b1c70 con 0x5601745b4f60 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdc072ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb01b15a0 con 0x5601745b5880 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdc072ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb01ae780 con 0x5601745af7a0 2024-06-09T22:22:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdc072ab80 10 client.0 _ll_register_callbacks cb 0x5601744965b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdaeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745c1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdaeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745c1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50714/0 (socket says 172.21.15.104:50714) 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdaeffd700 1 -- 172.21.15.104:0/3601422436 learned_addr learned my addr 172.21.15.104:0/3601422436 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745af7a0 0x5601745c17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.280+0000 7fbdaffff700 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 0x5601745d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdaeffd700 1 -- 172.21.15.104:0/3601422436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745af7a0 msgr2=0x5601745c17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdaeffd700 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745af7a0 0x5601745c17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdaeffd700 1 -- 172.21.15.104:0/3601422436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 msgr2=0x5601745d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdaeffd700 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 0x5601745d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdaeffd700 1 -- 172.21.15.104:0/3601422436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601745d0cb0 con 0x5601745b4f60 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdaeffd700 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745c1ce0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbda0005950 tx=0x7fbda00199c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b56d629f6b31c7d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbd8effd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbd8effd700 1 -- 172.21.15.104:0/3601422436 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda002e030 con 0x5601745b4f60 2024-06-09T22:22:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbd8effd700 1 -- 172.21.15.104:0/3601422436 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fbda000fca0 con 0x5601745b4f60 2024-06-09T22:22:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdc072ab80 1 -- 172.21.15.104:0/3601422436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbda0019580 con 0x5601745b4f60 2024-06-09T22:22:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbd8effd700 1 -- 172.21.15.104:0/3601422436 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbda0043050 con 0x5601745b4f60 2024-06-09T22:22:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdc072ab80 10 client.15939 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdc072ab80 1 -- 172.21.15.104:0/3601422436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5601746338f0 con 0x5601745b4f60 2024-06-09T22:22:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbd8effd700 1 -- 172.21.15.104:0/3601422436 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbda0019580 con 0x5601745b4f60 2024-06-09T22:22:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbd8effd700 1 -- 172.21.15.104:0/3601422436 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fbda0018050 con 0x5601745b4f60 2024-06-09T22:22:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdc072ab80 10 client.15939 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdc072ab80 10 client.15939 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:22:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdc072ab80 20 client.15939 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x560174633e80 0x5601746362a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdc072ab80 4 client.15939 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.281+0000 7fbdc072ab80 1 -- 172.21.15.104:0/3601422436 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5601745b34a0 con 0x560174633e80 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.282+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x560174633e80 0x5601746362a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.282+0000 7fbdaf7fe700 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x560174633e80 0x5601746362a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8933a145947586de server_cookie=62ba3656d3a3a604 in_seq=0 out_seq=0 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.282+0000 7fbd8effd700 10 client.15939 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.282+0000 7fbd8effd700 1 -- 172.21.15.104:0/3601422436 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5601745b34a0 con 0x560174633e80 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.282+0000 7fbd8effd700 10 client.15939 handle_command_reply: tid=0 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.282+0000 7fbdc072ab80 1 client.15939 shutdown 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdc072ab80 1 -- 172.21.15.104:0/3601422436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 msgr2=0x5601745c1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745c1ce0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbda0005950 tx=0x7fbda00199c0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdc072ab80 1 -- 172.21.15.104:0/3601422436 shutdown_connections 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601745b5880 0x5601745d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601745af7a0 0x5601745c17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x560174633e80 0x5601746362a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdc072ab80 1 --2- 172.21.15.104:0/3601422436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601745b4f60 0x5601745c1ce0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdc072ab80 1 -- 172.21.15.104:0/3601422436 >> 172.21.15.104:0/3601422436 conn(0x5601744dc080 msgr2=0x5601745b6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:40.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbd8effd700 0 client.15939 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:22:40.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdc072ab80 1 -- 172.21.15.104:0/3601422436 shutdown_connections 2024-06-09T22:22:40.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdc072ab80 1 -- 172.21.15.104:0/3601422436 wait complete. 2024-06-09T22:22:40.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdc072ab80 20 client.15939 trim_cache size 0 max 16384 2024-06-09T22:22:40.288 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:22:40.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/614149013 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbd980a2d90 msgr2=0x7fbd980a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdb5ea7700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbd980a2d90 0x7fbd980a5250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbda4000c00 tx=0x7fbda4005fb0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/614149013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb01084f0 msgr2=0x7fbdb0198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdb5ea7700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb01084f0 0x7fbdb0198ad0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbda800b970 tx=0x7fbda800bd30 comp rx=0 tx=0).stop 2024-06-09T22:22:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/614149013 shutdown_connections 2024-06-09T22:22:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdb5ea7700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbd980a2d90 0x7fbd980a5250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdb5ea7700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdb01084f0 0x7fbdb0198ad0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdb5ea7700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdb0102550 0x7fbdb019e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdb5ea7700 1 --2- 172.21.15.104:0/614149013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdb0102f10 0x7fbdb019ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.283+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/614149013 >> 172.21.15.104:0/614149013 conn(0x7fbdb0079650 msgr2=0x7fbdb01005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.284+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/614149013 shutdown_connections 2024-06-09T22:22:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:40.284+0000 7fbdb5ea7700 1 -- 172.21.15.104:0/614149013 wait complete. 2024-06-09T22:22:40.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:40 smithi153 ceph-mon[36599]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-06-09T22:22:41.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:40 smithi104 ceph-mon[32389]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-06-09T22:22:41.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:40 smithi194 ceph-mon[36856]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-06-09T22:22:41.350 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-09T22:22:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:42.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:42 smithi153 ceph-mon[36599]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 9 op/s 2024-06-09T22:22:43.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:42 smithi104 ceph-mon[32389]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 9 op/s 2024-06-09T22:22:43.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:42 smithi194 ceph-mon[36856]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 9 op/s 2024-06-09T22:22:43.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.473+0000 7ffa33758700 1 -- 172.21.15.104:0/194530726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa2c108510 msgr2=0x7ffa2c1088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:43.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.473+0000 7ffa33758700 1 --2- 172.21.15.104:0/194530726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa2c108510 0x7ffa2c1088f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400b0a0 tx=0x7ffa2400b3b0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.473+0000 7ffa33758700 1 -- 172.21.15.104:0/194530726 shutdown_connections 2024-06-09T22:22:43.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.473+0000 7ffa33758700 1 --2- 172.21.15.104:0/194530726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa2c108510 0x7ffa2c1088f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.473+0000 7ffa33758700 1 --2- 172.21.15.104:0/194530726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa2c101440 0x7ffa2c10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.473+0000 7ffa33758700 1 --2- 172.21.15.104:0/194530726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2c100a80 0x7ffa2c100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.473+0000 7ffa33758700 1 -- 172.21.15.104:0/194530726 >> 172.21.15.104:0/194530726 conn(0x7ffa2c077a00 msgr2=0x7ffa2c0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:43.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.473+0000 7ffa33758700 1 -- 172.21.15.104:0/194530726 shutdown_connections 2024-06-09T22:22:43.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.473+0000 7ffa33758700 1 -- 172.21.15.104:0/194530726 wait complete. 2024-06-09T22:22:43.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.474+0000 7ffa33758700 1 Processor -- start 2024-06-09T22:22:43.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.474+0000 7ffa33758700 1 -- start start 2024-06-09T22:22:43.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.474+0000 7ffa33758700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2c100a80 0x7ffa2c19e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.474+0000 7ffa33758700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa2c101440 0x7ffa2c19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.474+0000 7ffa33758700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa2c108510 0x7ffa2c198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.474+0000 7ffa33758700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c112790 con 0x7ffa2c108510 2024-06-09T22:22:43.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.474+0000 7ffa33758700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c112610 con 0x7ffa2c100a80 2024-06-09T22:22:43.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.474+0000 7ffa33758700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c112910 con 0x7ffa2c101440 2024-06-09T22:22:43.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.474+0000 7ffa31cf5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa2c108510 0x7ffa2c198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa30cf3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa2c101440 0x7ffa2c19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa31cf5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa2c108510 0x7ffa2c198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50732/0 (socket says 172.21.15.104:50732) 2024-06-09T22:22:43.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa31cf5700 1 -- 172.21.15.104:0/2400966430 learned_addr learned my addr 172.21.15.104:0/2400966430 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:43.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa314f4700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2c100a80 0x7ffa2c19e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa31cf5700 1 -- 172.21.15.104:0/2400966430 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa2c101440 msgr2=0x7ffa2c19ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:43.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa31cf5700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa2c101440 0x7ffa2c19ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa31cf5700 1 -- 172.21.15.104:0/2400966430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2c100a80 msgr2=0x7ffa2c19e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:43.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa31cf5700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2c100a80 0x7ffa2c19e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa31cf5700 1 -- 172.21.15.104:0/2400966430 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa2c199170 con 0x7ffa2c108510 2024-06-09T22:22:43.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa31cf5700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa2c108510 0x7ffa2c198a10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00d980 tx=0x7ffa1c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:43.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa2a7fc700 1 -- 172.21.15.104:0/2400966430 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c017070 con 0x7ffa2c108510 2024-06-09T22:22:43.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa33758700 1 -- 172.21.15.104:0/2400966430 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa2c199460 con 0x7ffa2c108510 2024-06-09T22:22:43.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa2a7fc700 1 -- 172.21.15.104:0/2400966430 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa1c005020 con 0x7ffa2c108510 2024-06-09T22:22:43.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa2a7fc700 1 -- 172.21.15.104:0/2400966430 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c00fbf0 con 0x7ffa2c108510 2024-06-09T22:22:43.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.475+0000 7ffa33758700 1 -- 172.21.15.104:0/2400966430 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa2c1a5720 con 0x7ffa2c108510 2024-06-09T22:22:43.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.477+0000 7ffa2a7fc700 1 -- 172.21.15.104:0/2400966430 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ffa1c0051d0 con 0x7ffa2c108510 2024-06-09T22:22:43.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.478+0000 7ffa2a7fc700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa140a2d90 0x7ffa140a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.478+0000 7ffa314f4700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa140a2d90 0x7ffa140a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.479+0000 7ffa2a7fc700 1 -- 172.21.15.104:0/2400966430 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ffa1c0cf4f0 con 0x7ffa2c108510 2024-06-09T22:22:43.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.479+0000 7ffa314f4700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa140a2d90 0x7ffa140a5250 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009fd0 tx=0x7ffa2400b7b0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:43.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.479+0000 7ffa3dfdbb80 1 Processor -- start 2024-06-09T22:22:43.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.479+0000 7ffa3dfdbb80 1 -- start start 2024-06-09T22:22:43.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.479+0000 7ffa3dfdbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d092618fd0 0x55d0926193b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa3dfdbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d09261d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa3dfdbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d09261dc00 0x55d09261e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa3dfdbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d092566810 con 0x55d0926198f0 2024-06-09T22:22:43.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa3dfdbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d09250e720 con 0x55d09261dc00 2024-06-09T22:22:43.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa3dfdbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d0925627d0 con 0x55d092618fd0 2024-06-09T22:22:43.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa30cf3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d09261d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa30cf3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d09261d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50736/0 (socket says 172.21.15.104:50736) 2024-06-09T22:22:43.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa30cf3700 1 -- 172.21.15.104:0/698657729 learned_addr learned my addr 172.21.15.104:0/698657729 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:43.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa314f4700 1 --2- 172.21.15.104:0/698657729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d092618fd0 0x55d0926193b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa31cf5700 1 --2- 172.21.15.104:0/698657729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d09261dc00 0x55d09261e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa30cf3700 1 -- 172.21.15.104:0/698657729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d092618fd0 msgr2=0x55d0926193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:43.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa30cf3700 1 --2- 172.21.15.104:0/698657729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d092618fd0 0x55d0926193b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa30cf3700 1 -- 172.21.15.104:0/698657729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d09261dc00 msgr2=0x55d09261e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:43.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa30cf3700 1 --2- 172.21.15.104:0/698657729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d09261dc00 0x55d09261e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa30cf3700 1 -- 172.21.15.104:0/698657729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d092504f50 con 0x55d0926198f0 2024-06-09T22:22:43.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa30cf3700 1 --2- 172.21.15.104:0/698657729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d09261d6c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ffa20007c90 tx=0x7ffa200103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5783b6a8328c6693 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:43.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa137fe700 1 -- 172.21.15.104:0/698657729 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20010cb0 con 0x55d0926198f0 2024-06-09T22:22:43.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa137fe700 1 -- 172.21.15.104:0/698657729 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa20015070 con 0x55d0926198f0 2024-06-09T22:22:43.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.480+0000 7ffa137fe700 1 -- 172.21.15.104:0/698657729 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20003e30 con 0x55d0926198f0 2024-06-09T22:22:43.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.481+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/698657729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 msgr2=0x55d09261d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:43.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.481+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/698657729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d09261d6c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ffa20007c90 tx=0x7ffa200103d0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.481+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/698657729 shutdown_connections 2024-06-09T22:22:43.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.481+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/698657729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d09261dc00 0x55d09261e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.481+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/698657729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d092618fd0 0x55d0926193b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.481+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/698657729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d09261d6c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.481+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/698657729 >> 172.21.15.104:0/698657729 conn(0x55d09253fcc0 msgr2=0x55d09260f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:43.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.481+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/698657729 shutdown_connections 2024-06-09T22:22:43.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.481+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/698657729 wait complete. 2024-06-09T22:22:43.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.481+0000 7ffa3dfdbb80 1 -- start start 2024-06-09T22:22:43.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa3dfdbb80 1 Processor -- start 2024-06-09T22:22:43.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa3dfdbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d092618fd0 0x55d092625040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa3dfdbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092622340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa3dfdbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d09261dc00 0x55d092622880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa3dfdbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d092625dc0 con 0x55d0926198f0 2024-06-09T22:22:43.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa3dfdbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d092506bf0 con 0x55d09261dc00 2024-06-09T22:22:43.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa3dfdbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d092625f40 con 0x55d092618fd0 2024-06-09T22:22:43.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa3dfdbb80 10 client.0 _ll_register_callbacks cb 0x55d092500920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:43.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa30cf3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092622340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa30cf3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092622340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50744/0 (socket says 172.21.15.104:50744) 2024-06-09T22:22:43.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa30cf3700 1 -- 172.21.15.104:0/3996632422 learned_addr learned my addr 172.21.15.104:0/3996632422 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:43.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa314f4700 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d092618fd0 0x55d092625040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa31cf5700 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d09261dc00 0x55d092622880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa30cf3700 1 -- 172.21.15.104:0/3996632422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d092618fd0 msgr2=0x55d092625040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:43.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa30cf3700 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d092618fd0 0x55d092625040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa30cf3700 1 -- 172.21.15.104:0/3996632422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d09261dc00 msgr2=0x55d092622880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:43.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa30cf3700 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d09261dc00 0x55d092622880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa30cf3700 1 -- 172.21.15.104:0/3996632422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0924b58d0 con 0x55d0926198f0 2024-06-09T22:22:43.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.482+0000 7ffa31cf5700 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d09261dc00 0x55d092622880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:43.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa30cf3700 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092622340 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ffa20010720 tx=0x7ffa20003ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c65eacf5cc0cb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:43.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa10ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:43.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa10ff9700 1 -- 172.21.15.104:0/3996632422 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20028060 con 0x55d0926198f0 2024-06-09T22:22:43.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa10ff9700 1 -- 172.21.15.104:0/3996632422 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ffa20010ab0 con 0x55d0926198f0 2024-06-09T22:22:43.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/3996632422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa20015070 con 0x55d0926198f0 2024-06-09T22:22:43.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa10ff9700 1 -- 172.21.15.104:0/3996632422 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffa2002f050 con 0x55d0926198f0 2024-06-09T22:22:43.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa3dfdbb80 10 client.15951 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:43.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/3996632422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d092630610 con 0x55d0926198f0 2024-06-09T22:22:43.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa10ff9700 1 -- 172.21.15.104:0/3996632422 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa20015070 con 0x55d0926198f0 2024-06-09T22:22:43.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa10ff9700 1 -- 172.21.15.104:0/3996632422 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ffa2001a7f0 con 0x55d0926198f0 2024-06-09T22:22:43.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa3dfdbb80 10 client.15951 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:43.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa3dfdbb80 10 client.15951 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:22:43.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa3dfdbb80 20 client.15951 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:43.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.483+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d092632b90 0x55d092634f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.484+0000 7ffa3dfdbb80 4 client.15951 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:43.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.484+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/3996632422 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d092573d10 con 0x55d092632b90 2024-06-09T22:22:43.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.484+0000 7ffa314f4700 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d092632b90 0x55d092634f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.484+0000 7ffa314f4700 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d092632b90 0x55d092634f80 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=38e884aa6f1a5675 server_cookie=ab39edab235ff141 in_seq=0 out_seq=0 2024-06-09T22:22:43.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.484+0000 7ffa10ff9700 10 client.15951 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:22:43.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa10ff9700 1 -- 172.21.15.104:0/3996632422 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d092573d10 con 0x55d092632b90 2024-06-09T22:22:43.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa10ff9700 10 client.15951 handle_command_reply: tid=0 2024-06-09T22:22:43.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa3dfdbb80 1 client.15951 shutdown 2024-06-09T22:22:43.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/3996632422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 msgr2=0x55d092622340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:43.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092622340 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ffa20010720 tx=0x7ffa20003ab0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/3996632422 shutdown_connections 2024-06-09T22:22:43.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d092632b90 0x55d092634f80 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d09261dc00 0x55d092622880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d092618fd0 0x55d092625040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/3996632422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092622340 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/3996632422 >> 172.21.15.104:0/3996632422 conn(0x55d09253fcc0 msgr2=0x55d09260f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:43.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa10ff9700 0 client.15951 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:22:43.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.485+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/3996632422 shutdown_connections 2024-06-09T22:22:43.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.486+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/3996632422 wait complete. 2024-06-09T22:22:43.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.486+0000 7ffa3dfdbb80 20 client.15951 trim_cache size 0 max 16384 2024-06-09T22:22:43.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.500+0000 7ffa3dfdbb80 1 Processor -- start 2024-06-09T22:22:43.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.500+0000 7ffa3dfdbb80 1 -- start start 2024-06-09T22:22:43.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa3dfdbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d092618fd0 0x55d09261ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa3dfdbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092608e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa3dfdbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d09261dc00 0x55d092609340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa3dfdbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1ac350 con 0x55d0926198f0 2024-06-09T22:22:43.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa3dfdbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1ade90 con 0x55d092618fd0 2024-06-09T22:22:43.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa3dfdbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1aca20 con 0x55d09261dc00 2024-06-09T22:22:43.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa30cf3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092608e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa30cf3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092608e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50748/0 (socket says 172.21.15.104:50748) 2024-06-09T22:22:43.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa30cf3700 1 -- 172.21.15.104:0/4027784750 learned_addr learned my addr 172.21.15.104:0/4027784750 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:43.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa314f4700 1 --2- 172.21.15.104:0/4027784750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d092618fd0 0x55d09261ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa31cf5700 1 --2- 172.21.15.104:0/4027784750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d09261dc00 0x55d092609340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa30cf3700 1 -- 172.21.15.104:0/4027784750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d09261dc00 msgr2=0x55d092609340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:43.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa30cf3700 1 --2- 172.21.15.104:0/4027784750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d09261dc00 0x55d092609340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa30cf3700 1 -- 172.21.15.104:0/4027784750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d092618fd0 msgr2=0x55d09261ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:43.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa30cf3700 1 --2- 172.21.15.104:0/4027784750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d092618fd0 0x55d09261ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa30cf3700 1 -- 172.21.15.104:0/4027784750 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d09262f210 con 0x55d0926198f0 2024-06-09T22:22:43.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa31cf5700 1 --2- 172.21.15.104:0/4027784750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d09261dc00 0x55d092609340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:43.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa30cf3700 1 --2- 172.21.15.104:0/4027784750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092608e00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ffa20007510 tx=0x7ffa200103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9677a10eb8513d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:43.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa12ffd700 1 -- 172.21.15.104:0/4027784750 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20028040 con 0x55d0926198f0 2024-06-09T22:22:43.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa12ffd700 1 -- 172.21.15.104:0/4027784750 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa20015070 con 0x55d0926198f0 2024-06-09T22:22:43.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.501+0000 7ffa12ffd700 1 -- 172.21.15.104:0/4027784750 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2001a470 con 0x55d0926198f0 2024-06-09T22:22:43.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.502+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/4027784750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 msgr2=0x55d092608e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:43.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.502+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/4027784750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092608e00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ffa20007510 tx=0x7ffa200103d0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.502+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/4027784750 shutdown_connections 2024-06-09T22:22:43.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.502+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/4027784750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d09261dc00 0x55d092609340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.502+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/4027784750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d092618fd0 0x55d09261ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.502+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/4027784750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092608e00 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.502+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/4027784750 >> 172.21.15.104:0/4027784750 conn(0x55d09253fcc0 msgr2=0x55d09261f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:43.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.502+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/4027784750 shutdown_connections 2024-06-09T22:22:43.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.502+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/4027784750 wait complete. 2024-06-09T22:22:43.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.502+0000 7ffa3dfdbb80 1 -- start start 2024-06-09T22:22:43.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.502+0000 7ffa3dfdbb80 1 Processor -- start 2024-06-09T22:22:43.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa3dfdbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d092618fd0 0x55d092637ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa3dfdbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092638410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa3dfdbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d09261dc00 0x55d092621cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa3dfdbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1ad7c0 con 0x55d0926198f0 2024-06-09T22:22:43.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa3dfdbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1ad0f0 con 0x55d092618fd0 2024-06-09T22:22:43.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa3dfdbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1aa2d0 con 0x55d09261dc00 2024-06-09T22:22:43.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa3dfdbb80 10 client.0 _ll_register_callbacks cb 0x55d092500920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:43.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa30cf3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092638410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa30cf3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092638410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50752/0 (socket says 172.21.15.104:50752) 2024-06-09T22:22:43.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa30cf3700 1 -- 172.21.15.104:0/2484329537 learned_addr learned my addr 172.21.15.104:0/2484329537 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:43.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa314f4700 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d092618fd0 0x55d092637ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa31cf5700 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d09261dc00 0x55d092621cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa30cf3700 1 -- 172.21.15.104:0/2484329537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d09261dc00 msgr2=0x55d092621cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:43.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa30cf3700 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d09261dc00 0x55d092621cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa30cf3700 1 -- 172.21.15.104:0/2484329537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d092618fd0 msgr2=0x55d092637ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:43.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa30cf3700 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d092618fd0 0x55d092637ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.503+0000 7ffa30cf3700 1 -- 172.21.15.104:0/2484329537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0926224d0 con 0x55d0926198f0 2024-06-09T22:22:43.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa30cf3700 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092638410 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ffa20003960 tx=0x7ffa20005960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a2e39bf38ff96e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:43.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa10ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:43.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa10ff9700 1 -- 172.21.15.104:0/2484329537 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20005b00 con 0x55d0926198f0 2024-06-09T22:22:43.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa10ff9700 1 -- 172.21.15.104:0/2484329537 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ffa20005320 con 0x55d0926198f0 2024-06-09T22:22:43.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/2484329537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa20015070 con 0x55d0926198f0 2024-06-09T22:22:43.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa10ff9700 1 -- 172.21.15.104:0/2484329537 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffa20046050 con 0x55d0926198f0 2024-06-09T22:22:43.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa3dfdbb80 10 client.15957 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:43.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/2484329537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d09269d0a0 con 0x55d0926198f0 2024-06-09T22:22:43.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa10ff9700 1 -- 172.21.15.104:0/2484329537 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa20015070 con 0x55d0926198f0 2024-06-09T22:22:43.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa10ff9700 1 -- 172.21.15.104:0/2484329537 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ffa200116c0 con 0x55d0926198f0 2024-06-09T22:22:43.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa3dfdbb80 10 client.15957 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:43.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa3dfdbb80 10 client.15957 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:22:43.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa3dfdbb80 20 client.15957 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:43.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d09269d710 0x55d09269fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:43.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa3dfdbb80 4 client.15957 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:22:43.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.504+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/2484329537 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d09256e290 con 0x55d09269d710 2024-06-09T22:22:43.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa314f4700 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d09269d710 0x55d09269fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:43.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa314f4700 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d09269d710 0x55d09269fb30 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=47a90cf2b87f25b3 server_cookie=194633a43ab876a1 in_seq=0 out_seq=0 2024-06-09T22:22:43.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa10ff9700 10 client.15957 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:22:43.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa10ff9700 1 -- 172.21.15.104:0/2484329537 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d09256e290 con 0x55d09269d710 2024-06-09T22:22:43.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa10ff9700 10 client.15957 handle_command_reply: tid=0 2024-06-09T22:22:43.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa3dfdbb80 1 client.15957 shutdown 2024-06-09T22:22:43.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/2484329537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 msgr2=0x55d092638410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:43.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092638410 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ffa20003960 tx=0x7ffa20005960 comp rx=0 tx=0).stop 2024-06-09T22:22:43.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/2484329537 shutdown_connections 2024-06-09T22:22:43.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d09269d710 0x55d09269fb30 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d09261dc00 0x55d092621cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa10ff9700 0 client.15957 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:22:43.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d092618fd0 0x55d092637ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa3dfdbb80 1 --2- 172.21.15.104:0/2484329537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d0926198f0 0x55d092638410 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/2484329537 >> 172.21.15.104:0/2484329537 conn(0x55d09253fcc0 msgr2=0x55d092608e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:43.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/2484329537 shutdown_connections 2024-06-09T22:22:43.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.505+0000 7ffa3dfdbb80 1 -- 172.21.15.104:0/2484329537 wait complete. 2024-06-09T22:22:43.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa3dfdbb80 20 client.15957 trim_cache size 0 max 16384 2024-06-09T22:22:43.529 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:22:43.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa33758700 1 -- 172.21.15.104:0/2400966430 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa140a2d90 msgr2=0x7ffa140a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:43.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa33758700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa140a2d90 0x7ffa140a5250 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009fd0 tx=0x7ffa2400b7b0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa33758700 1 -- 172.21.15.104:0/2400966430 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa2c108510 msgr2=0x7ffa2c198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:43.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa33758700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa2c108510 0x7ffa2c198a10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00d980 tx=0x7ffa1c00dc90 comp rx=0 tx=0).stop 2024-06-09T22:22:43.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa33758700 1 -- 172.21.15.104:0/2400966430 shutdown_connections 2024-06-09T22:22:43.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa33758700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa140a2d90 0x7ffa140a5250 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa33758700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa2c108510 0x7ffa2c198a10 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa33758700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2c100a80 0x7ffa2c19e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa33758700 1 --2- 172.21.15.104:0/2400966430 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa2c101440 0x7ffa2c19ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:43.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa33758700 1 -- 172.21.15.104:0/2400966430 >> 172.21.15.104:0/2400966430 conn(0x7ffa2c077a00 msgr2=0x7ffa2c0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:43.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa33758700 1 -- 172.21.15.104:0/2400966430 shutdown_connections 2024-06-09T22:22:43.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:43.506+0000 7ffa33758700 1 -- 172.21.15.104:0/2400966430 wait complete. 2024-06-09T22:22:44.355 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-09T22:22:44.597 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:44.985 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:44 smithi104 ceph-mon[32389]: pgmap v56: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 13 op/s 2024-06-09T22:22:44.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:44 smithi153 ceph-mon[36599]: pgmap v56: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 13 op/s 2024-06-09T22:22:45.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:44 smithi194 ceph-mon[36856]: pgmap v56: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 13 op/s 2024-06-09T22:22:45.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.225+0000 7f12d4c38640 1 -- 172.21.15.104:0/1629027477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d0103550 msgr2=0x7f12d010ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:45.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.225+0000 7f12d4c38640 1 --2- 172.21.15.104:0/1629027477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d0103550 0x7f12d010ba30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009900 tx=0x7f12b802a510 comp rx=0 tx=0).stop 2024-06-09T22:22:45.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.225+0000 7f12d4c38640 1 -- 172.21.15.104:0/1629027477 shutdown_connections 2024-06-09T22:22:45.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.225+0000 7f12d4c38640 1 --2- 172.21.15.104:0/1629027477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d010bf70 0x7f12d010e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.225+0000 7f12d4c38640 1 --2- 172.21.15.104:0/1629027477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d0103550 0x7f12d010ba30 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.225+0000 7f12d4c38640 1 --2- 172.21.15.104:0/1629027477 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d0102c30 0x7f12d0103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.225+0000 7f12d4c38640 1 -- 172.21.15.104:0/1629027477 >> 172.21.15.104:0/1629027477 conn(0x7f12d0077a20 msgr2=0x7f12d0077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.225+0000 7f12d4c38640 1 -- 172.21.15.104:0/1629027477 shutdown_connections 2024-06-09T22:22:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.225+0000 7f12d4c38640 1 -- 172.21.15.104:0/1629027477 wait complete. 2024-06-09T22:22:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12d4c38640 1 Processor -- start 2024-06-09T22:22:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12d4c38640 1 -- start start 2024-06-09T22:22:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12d4c38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d0102c30 0x7f12d0100880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12d4c38640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d0103550 0x7f12d0100dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12d4c38640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d010bf70 0x7f12d0101300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12d4c38640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f12d0111d70 con 0x7f12d0103550 2024-06-09T22:22:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12d4c38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12d0111bf0 con 0x7f12d0102c30 2024-06-09T22:22:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12d4c38640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f12d0111ef0 con 0x7f12d010bf70 2024-06-09T22:22:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12c7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d0103550 0x7f12d0100dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12c7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d0103550 0x7f12d0100dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50766/0 (socket says 172.21.15.104:50766) 2024-06-09T22:22:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12c7fff640 1 -- 172.21.15.104:0/1847770898 learned_addr learned my addr 172.21.15.104:0/1847770898 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12ce59c640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d0102c30 0x7f12d0100880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12ced9d640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d010bf70 0x7f12d0101300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12c7fff640 1 -- 172.21.15.104:0/1847770898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d010bf70 msgr2=0x7f12d0101300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12c7fff640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d010bf70 0x7f12d0101300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12c7fff640 1 -- 172.21.15.104:0/1847770898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d0102c30 msgr2=0x7f12d0100880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12c7fff640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d0102c30 0x7f12d0100880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12c7fff640 1 -- 172.21.15.104:0/1847770898 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d01a6690 con 0x7f12d0103550 2024-06-09T22:22:45.230 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.226+0000 7f12ced9d640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d010bf70 0x7f12d0101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:45.230 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.227+0000 7f12c7fff640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d0103550 0x7f12d0100dc0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f12b8006010 tx=0x7f12b80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:45.230 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.227+0000 7f12c77fe640 1 -- 172.21.15.104:0/1847770898 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b803f030 con 0x7f12d0103550 2024-06-09T22:22:45.230 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.227+0000 7f12c77fe640 1 -- 172.21.15.104:0/1847770898 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12b8003ce0 con 0x7f12d0103550 2024-06-09T22:22:45.230 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.227+0000 7f12d4c38640 1 -- 172.21.15.104:0/1847770898 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12d01a6920 con 0x7f12d0103550 2024-06-09T22:22:45.230 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.227+0000 7f12c77fe640 1 -- 172.21.15.104:0/1847770898 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8035070 con 0x7f12d0103550 2024-06-09T22:22:45.230 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.227+0000 7f12d4c38640 1 -- 172.21.15.104:0/1847770898 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12d01a6e10 con 0x7f12d0103550 2024-06-09T22:22:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.229+0000 7f12c77fe640 1 -- 172.21.15.104:0/1847770898 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f12b8042030 con 0x7f12d0103550 2024-06-09T22:22:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.229+0000 7f12d7a25740 1 Processor -- start 2024-06-09T22:22:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12d7a25740 1 -- start start 2024-06-09T22:22:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12c77fe640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12bc0a2da0 0x7f12bc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12d7a25740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380e8170 0x5604380e8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12d7a25740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12d7a25740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380f3350 0x5604380f5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12d7a25740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56043804c4f0 con 0x5604380f3350 2024-06-09T22:22:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12d7a25740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560437fea870 con 0x5604380e8170 2024-06-09T22:22:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12d7a25740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5604380484b0 con 0x5604380e8a90 2024-06-09T22:22:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12ced9d640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12bc0a2da0 0x7f12bc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12c77fe640 1 -- 172.21.15.104:0/1847770898 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f12b80efea0 con 0x7f12d0103550 2024-06-09T22:22:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12ced9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380f3350 0x5604380f5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12ced9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380f3350 0x5604380f5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50780/0 (socket says 172.21.15.104:50780) 2024-06-09T22:22:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12c7fff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12ced9d640 1 -- 172.21.15.104:0/3069794659 learned_addr learned my addr 172.21.15.104:0/3069794659 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12ce59c640 1 --2- 172.21.15.104:0/3069794659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380e8170 0x5604380e8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12ced9d640 1 -- 172.21.15.104:0/3069794659 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 msgr2=0x5604380f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12ced9d640 1 --2- 172.21.15.104:0/3069794659 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380f2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12ced9d640 1 -- 172.21.15.104:0/3069794659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380e8170 msgr2=0x5604380e8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12ced9d640 1 --2- 172.21.15.104:0/3069794659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380e8170 0x5604380e8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12ced9d640 1 -- 172.21.15.104:0/3069794659 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604380069b0 con 0x5604380f3350 2024-06-09T22:22:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12ced9d640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12bc0a2da0 0x7f12bc0a5260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f12c8012ae0 tx=0x7f12c800ad30 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.230+0000 7f12ced9d640 1 --2- 172.21.15.104:0/3069794659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380f3350 0x5604380f5740 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f12c800fc60 tx=0x7f12c8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2efb9f1c57508a1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.231+0000 7f12c4ff9640 1 -- 172.21.15.104:0/3069794659 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c8022070 con 0x5604380f3350 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.231+0000 7f12c4ff9640 1 -- 172.21.15.104:0/3069794659 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12c800dde0 con 0x5604380f3350 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.231+0000 7f12c4ff9640 1 -- 172.21.15.104:0/3069794659 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c80105d0 con 0x5604380f3350 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.231+0000 7f12d7a25740 1 -- 172.21.15.104:0/3069794659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380f3350 msgr2=0x5604380f5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.231+0000 7f12d7a25740 1 --2- 172.21.15.104:0/3069794659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380f3350 0x5604380f5740 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f12c800fc60 tx=0x7f12c8019000 comp rx=0 tx=0).stop 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.231+0000 7f12d7a25740 1 -- 172.21.15.104:0/3069794659 shutdown_connections 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.231+0000 7f12d7a25740 1 --2- 172.21.15.104:0/3069794659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380f3350 0x5604380f5740 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.231+0000 7f12d7a25740 1 --2- 172.21.15.104:0/3069794659 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.231+0000 7f12d7a25740 1 --2- 172.21.15.104:0/3069794659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380e8170 0x5604380e8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.231+0000 7f12d7a25740 1 -- 172.21.15.104:0/3069794659 >> 172.21.15.104:0/3069794659 conn(0x5604380e4680 msgr2=0x5604380e4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12d7a25740 1 -- 172.21.15.104:0/3069794659 shutdown_connections 2024-06-09T22:22:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12d7a25740 1 -- 172.21.15.104:0/3069794659 wait complete. 2024-06-09T22:22:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12d7a25740 1 -- start start 2024-06-09T22:22:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12d7a25740 1 Processor -- start 2024-06-09T22:22:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12d7a25740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 0x5604380f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12d7a25740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380f0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12d7a25740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380f3350 0x5604380f0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12d7a25740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5604380f5f20 con 0x5604380e8170 2024-06-09T22:22:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12d7a25740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560437fd5ae0 con 0x5604380f3350 2024-06-09T22:22:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12d7a25740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5604380f60a0 con 0x5604380e8a90 2024-06-09T22:22:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12d7a25740 10 client.0 _ll_register_callbacks cb 0x56043800e9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12ce59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 0x5604380f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12ce59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 0x5604380f02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50788/0 (socket says 172.21.15.104:50788) 2024-06-09T22:22:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12ce59c640 1 -- 172.21.15.104:0/208413082 learned_addr learned my addr 172.21.15.104:0/208413082 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12ced9d640 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380f3350 0x5604380f0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12c7fff640 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380f0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12ce59c640 1 -- 172.21.15.104:0/208413082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 msgr2=0x5604380f0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12ce59c640 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380f0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.232+0000 7f12ce59c640 1 -- 172.21.15.104:0/208413082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380f3350 msgr2=0x5604380f0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12ce59c640 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380f3350 0x5604380f0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12ce59c640 1 -- 172.21.15.104:0/208413082 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560437d60210 con 0x5604380e8170 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12ce59c640 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 0x5604380f02d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f12c000bf40 tx=0x7f12c00128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8049a018c271979 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12b5ffb640 1 -- 172.21.15.104:0/208413082 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c001f060 con 0x5604380e8170 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12d7a25740 1 -- 172.21.15.104:0/208413082 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12c800dde0 con 0x5604380e8170 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12b5ffb640 1 -- 172.21.15.104:0/208413082 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f12c00108a0 con 0x5604380e8170 2024-06-09T22:22:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12b5ffb640 1 -- 172.21.15.104:0/208413082 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f12c0026050 con 0x5604380e8170 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12d7a25740 10 client.15969 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12d7a25740 1 -- 172.21.15.104:0/208413082 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560437e2c620 con 0x5604380e8170 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12b5ffb640 1 -- 172.21.15.104:0/208413082 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12c800dde0 con 0x5604380e8170 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12b5ffb640 1 -- 172.21.15.104:0/208413082 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f12c0019030 con 0x5604380e8170 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12d7a25740 10 client.15969 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12d7a25740 10 client.15969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12d7a25740 20 client.15969 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12d7a25740 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5604381034f0 0x5604381058e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12d7a25740 4 client.15969 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.233+0000 7f12d7a25740 1 -- 172.21.15.104:0/208413082 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56043805ff10 con 0x5604381034f0 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.234+0000 7f12c7fff640 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5604381034f0 0x5604381058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.234+0000 7f12b5ffb640 10 client.15969 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.234+0000 7f12c7fff640 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5604381034f0 0x5604381058e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18537326bc0c98e1 server_cookie=b1a9522b8000c680 in_seq=0 out_seq=0 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12b5ffb640 1 -- 172.21.15.104:0/208413082 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56043805ff10 con 0x5604381034f0 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12b5ffb640 10 client.15969 handle_command_reply: tid=0 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12d7a25740 1 client.15969 shutdown 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12d7a25740 1 -- 172.21.15.104:0/208413082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 msgr2=0x5604380f02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12d7a25740 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 0x5604380f02d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f12c000bf40 tx=0x7f12c00128e0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12d7a25740 1 -- 172.21.15.104:0/208413082 shutdown_connections 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12d7a25740 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5604381034f0 0x5604381058e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12d7a25740 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380f3350 0x5604380f0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12d7a25740 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380f0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12b5ffb640 0 client.15969 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12d7a25740 1 --2- 172.21.15.104:0/208413082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 0x5604380f02d0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12d7a25740 1 -- 172.21.15.104:0/208413082 >> 172.21.15.104:0/208413082 conn(0x5604380e4680 msgr2=0x5604380ea400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12d7a25740 1 -- 172.21.15.104:0/208413082 shutdown_connections 2024-06-09T22:22:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12d7a25740 1 -- 172.21.15.104:0/208413082 wait complete. 2024-06-09T22:22:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.235+0000 7f12d7a25740 20 client.15969 trim_cache size 0 max 16384 2024-06-09T22:22:45.255 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.250+0000 7f12d7a25740 1 Processor -- start 2024-06-09T22:22:45.255 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12d7a25740 1 -- start start 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12d7a25740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380e8170 0x5604380eab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12d7a25740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8a90 0x5604380f2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12d7a25740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380f3350 0x56043810b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12d7a25740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f12d01b0ee0 con 0x5604380e8a90 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12d7a25740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12d01b2a20 con 0x5604380e8170 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12d7a25740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f12d01b15b0 con 0x5604380f3350 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12c7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8a90 0x5604380f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12c7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8a90 0x5604380f2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50800/0 (socket says 172.21.15.104:50800) 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12c7fff640 1 -- 172.21.15.104:0/2568912873 learned_addr learned my addr 172.21.15.104:0/2568912873 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12ced9d640 1 --2- 172.21.15.104:0/2568912873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380f3350 0x56043810b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12ce59c640 1 --2- 172.21.15.104:0/2568912873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380e8170 0x5604380eab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12c7fff640 1 -- 172.21.15.104:0/2568912873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380f3350 msgr2=0x56043810b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:45.256 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12c7fff640 1 --2- 172.21.15.104:0/2568912873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380f3350 0x56043810b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12c7fff640 1 -- 172.21.15.104:0/2568912873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380e8170 msgr2=0x5604380eab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12c7fff640 1 --2- 172.21.15.104:0/2568912873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380e8170 0x5604380eab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12c7fff640 1 -- 172.21.15.104:0/2568912873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604380f5ab0 con 0x5604380e8a90 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12ce59c640 1 --2- 172.21.15.104:0/2568912873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380e8170 0x5604380eab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.251+0000 7f12c7fff640 1 --2- 172.21.15.104:0/2568912873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8a90 0x5604380f2210 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f12b80ed2e0 tx=0x7f12b80dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8228878e5e7221 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.252+0000 7f12b7fff640 1 -- 172.21.15.104:0/2568912873 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8049040 con 0x5604380e8a90 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.252+0000 7f12b7fff640 1 -- 172.21.15.104:0/2568912873 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12b803b580 con 0x5604380e8a90 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.252+0000 7f12b7fff640 1 -- 172.21.15.104:0/2568912873 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b80c4d80 con 0x5604380e8a90 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.252+0000 7f12d7a25740 1 -- 172.21.15.104:0/2568912873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8a90 msgr2=0x5604380f2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.252+0000 7f12d7a25740 1 --2- 172.21.15.104:0/2568912873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8a90 0x5604380f2210 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f12b80ed2e0 tx=0x7f12b80dc040 comp rx=0 tx=0).stop 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.252+0000 7f12d7a25740 1 -- 172.21.15.104:0/2568912873 shutdown_connections 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.252+0000 7f12d7a25740 1 --2- 172.21.15.104:0/2568912873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380f3350 0x56043810b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.252+0000 7f12d7a25740 1 --2- 172.21.15.104:0/2568912873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8a90 0x5604380f2210 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.252+0000 7f12d7a25740 1 --2- 172.21.15.104:0/2568912873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380e8170 0x5604380eab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.252+0000 7f12d7a25740 1 -- 172.21.15.104:0/2568912873 >> 172.21.15.104:0/2568912873 conn(0x5604380e4680 msgr2=0x5604380f1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.252+0000 7f12d7a25740 1 -- 172.21.15.104:0/2568912873 shutdown_connections 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.252+0000 7f12d7a25740 1 -- 172.21.15.104:0/2568912873 wait complete. 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12d7a25740 1 -- start start 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12d7a25740 1 Processor -- start 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12d7a25740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 0x5604380fb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12d7a25740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380fba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12d7a25740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380f3350 0x5604380f82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12d7a25740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f12d01b2350 con 0x5604380e8170 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12d7a25740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12d01b1c80 con 0x5604380f3350 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12d7a25740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f12d01aeea0 con 0x5604380e8a90 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12d7a25740 10 client.0 _ll_register_callbacks cb 0x7f12c000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12ce59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 0x5604380fb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12ce59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 0x5604380fb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50812/0 (socket says 172.21.15.104:50812) 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12ce59c640 1 -- 172.21.15.104:0/1632202772 learned_addr learned my addr 172.21.15.104:0/1632202772 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12ced9d640 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380f3350 0x5604380f82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12c7fff640 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380fba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12ce59c640 1 -- 172.21.15.104:0/1632202772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 msgr2=0x5604380fba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12ce59c640 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380fba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12ce59c640 1 -- 172.21.15.104:0/1632202772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380f3350 msgr2=0x5604380f82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12ce59c640 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380f3350 0x5604380f82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12ce59c640 1 -- 172.21.15.104:0/1632202772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604380f8950 con 0x5604380e8170 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12c7fff640 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380fba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.253+0000 7f12ced9d640 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380f3350 0x5604380f82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12ce59c640 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 0x5604380fb4e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f12c0004630 tx=0x7f12c0005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453a9549fd555300 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12b5ffb640 1 -- 172.21.15.104:0/1632202772 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c001f020 con 0x5604380e8170 2024-06-09T22:22:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12b5ffb640 1 -- 172.21.15.104:0/1632202772 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f12c0005250 con 0x5604380e8170 2024-06-09T22:22:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12d7a25740 1 -- 172.21.15.104:0/1632202772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560438059c20 con 0x5604380e8170 2024-06-09T22:22:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12b5ffb640 1 -- 172.21.15.104:0/1632202772 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f12c0037050 con 0x5604380e8170 2024-06-09T22:22:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12d7a25740 10 client.15975 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12d7a25740 1 -- 172.21.15.104:0/1632202772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56043816eab0 con 0x5604380e8170 2024-06-09T22:22:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12b5ffb640 1 -- 172.21.15.104:0/1632202772 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560438059c20 con 0x5604380e8170 2024-06-09T22:22:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12b5ffb640 1 -- 172.21.15.104:0/1632202772 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f12c0019040 con 0x5604380e8170 2024-06-09T22:22:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12d7a25740 10 client.15975 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12d7a25740 10 client.15975 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12d7a25740 20 client.15975 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12d7a25740 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5604381034f0 0x5604381712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12d7a25740 4 client.15975 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:22:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.254+0000 7f12d7a25740 1 -- 172.21.15.104:0/1632202772 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560438053f70 con 0x5604381034f0 2024-06-09T22:22:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.255+0000 7f12c7fff640 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5604381034f0 0x5604381712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.255+0000 7f12c7fff640 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5604381034f0 0x5604381712f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4002c965ee150f4 server_cookie=b1f101dec3a26b2d in_seq=0 out_seq=0 2024-06-09T22:22:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.255+0000 7f12b5ffb640 10 client.15975 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12b5ffb640 1 -- 172.21.15.104:0/1632202772 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x560438053f70 con 0x5604381034f0 2024-06-09T22:22:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12b5ffb640 10 client.15975 handle_command_reply: tid=0 2024-06-09T22:22:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12d7a25740 1 client.15975 shutdown 2024-06-09T22:22:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12d7a25740 1 -- 172.21.15.104:0/1632202772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 msgr2=0x5604380fb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12d7a25740 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 0x5604380fb4e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f12c0004630 tx=0x7f12c0005cc0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12d7a25740 1 -- 172.21.15.104:0/1632202772 shutdown_connections 2024-06-09T22:22:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12d7a25740 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5604381034f0 0x5604381712f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12d7a25740 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5604380f3350 0x5604380f82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12d7a25740 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604380e8a90 0x5604380fba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12b5ffb640 0 client.15975 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12d7a25740 1 --2- 172.21.15.104:0/1632202772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5604380e8170 0x5604380fb4e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12d7a25740 1 -- 172.21.15.104:0/1632202772 >> 172.21.15.104:0/1632202772 conn(0x5604380e4680 msgr2=0x5604380f2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12d7a25740 1 -- 172.21.15.104:0/1632202772 shutdown_connections 2024-06-09T22:22:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12d7a25740 1 -- 172.21.15.104:0/1632202772 wait complete. 2024-06-09T22:22:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.261+0000 7f12d7a25740 20 client.15975 trim_cache size 0 max 16384 2024-06-09T22:22:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:22:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:22:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:22:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.262+0000 7f12d4c38640 1 -- 172.21.15.104:0/1847770898 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12bc0a2da0 msgr2=0x7f12bc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.262+0000 7f12d4c38640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12bc0a2da0 0x7f12bc0a5260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f12c8012ae0 tx=0x7f12c800ad30 comp rx=0 tx=0).stop 2024-06-09T22:22:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.262+0000 7f12d4c38640 1 -- 172.21.15.104:0/1847770898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d0103550 msgr2=0x7f12d0100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.262+0000 7f12d4c38640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d0103550 0x7f12d0100dc0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f12b8006010 tx=0x7f12b80042f0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.262+0000 7f12d4c38640 1 -- 172.21.15.104:0/1847770898 shutdown_connections 2024-06-09T22:22:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.262+0000 7f12d4c38640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12bc0a2da0 0x7f12bc0a5260 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.262+0000 7f12d4c38640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12d010bf70 0x7f12d0101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.262+0000 7f12d4c38640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d0103550 0x7f12d0100dc0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.262+0000 7f12d4c38640 1 --2- 172.21.15.104:0/1847770898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12d0102c30 0x7f12d0100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.262+0000 7f12d4c38640 1 -- 172.21.15.104:0/1847770898 >> 172.21.15.104:0/1847770898 conn(0x7f12d0077a20 msgr2=0x7f12d00fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.262+0000 7f12d4c38640 1 -- 172.21.15.104:0/1847770898 shutdown_connections 2024-06-09T22:22:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:45.262+0000 7f12d4c38640 1 -- 172.21.15.104:0/1847770898 wait complete. 2024-06-09T22:22:46.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.467+0000 7f5b8aae0700 1 -- 172.21.15.104:0/1877353231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b84102770 msgr2=0x7f5b84102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:46.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.467+0000 7f5b8aae0700 1 --2- 172.21.15.104:0/1877353231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b84102770 0x7f5b84102b50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5b780099c0 tx=0x7f5b78009cd0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.467+0000 7f5b8aae0700 1 -- 172.21.15.104:0/1877353231 shutdown_connections 2024-06-09T22:22:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.467+0000 7f5b8aae0700 1 --2- 172.21.15.104:0/1877353231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b84102770 0x7f5b84102b50 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.467+0000 7f5b8aae0700 1 --2- 172.21.15.104:0/1877353231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b84107860 0x7f5b8410f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.467+0000 7f5b8aae0700 1 --2- 172.21.15.104:0/1877353231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b84103120 0x7f5b84107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.467+0000 7f5b8aae0700 1 -- 172.21.15.104:0/1877353231 >> 172.21.15.104:0/1877353231 conn(0x7f5b84079600 msgr2=0x7f5b84079a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.467+0000 7f5b8aae0700 1 -- 172.21.15.104:0/1877353231 shutdown_connections 2024-06-09T22:22:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.467+0000 7f5b8aae0700 1 -- 172.21.15.104:0/1877353231 wait complete. 2024-06-09T22:22:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.467+0000 7f5b8aae0700 1 Processor -- start 2024-06-09T22:22:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.467+0000 7f5b8aae0700 1 -- start start 2024-06-09T22:22:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8aae0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b84102770 0x7f5b8419e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8aae0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b84103120 0x7f5b8419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8aae0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b84107860 0x7f5b841989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8aae0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5b84112780 con 0x7f5b84107860 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8aae0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b84112600 con 0x7f5b84103120 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8aae0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5b84112900 con 0x7f5b84102770 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8907d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b84107860 0x7f5b841989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8907d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b84107860 0x7f5b841989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38030/0 (socket says 172.21.15.104:38030) 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8907d700 1 -- 172.21.15.104:0/3162021502 learned_addr learned my addr 172.21.15.104:0/3162021502 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b83fff700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b84103120 0x7f5b8419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8887c700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b84102770 0x7f5b8419e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8907d700 1 -- 172.21.15.104:0/3162021502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b84102770 msgr2=0x7f5b8419e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8907d700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b84102770 0x7f5b8419e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8907d700 1 -- 172.21.15.104:0/3162021502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b84103120 msgr2=0x7f5b8419ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8907d700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b84103120 0x7f5b8419ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8907d700 1 -- 172.21.15.104:0/3162021502 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b84199120 con 0x7f5b84107860 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8887c700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b84102770 0x7f5b8419e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b83fff700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b84103120 0x7f5b8419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8907d700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b84107860 0x7f5b841989c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00ef30 tx=0x7f5b7c00c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b81ffb700 1 -- 172.21.15.104:0/3162021502 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0092e0 con 0x7f5b84107860 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.468+0000 7f5b8aae0700 1 -- 172.21.15.104:0/3162021502 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b841993b0 con 0x7f5b84107860 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.469+0000 7f5b81ffb700 1 -- 172.21.15.104:0/3162021502 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b7c004bb0 con 0x7f5b84107860 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.469+0000 7f5b81ffb700 1 -- 172.21.15.104:0/3162021502 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c004280 con 0x7f5b84107860 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.469+0000 7f5b8aae0700 1 -- 172.21.15.104:0/3162021502 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b841a56d0 con 0x7f5b84107860 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.471+0000 7f5b81ffb700 1 -- 172.21.15.104:0/3162021502 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5b7c01b030 con 0x7f5b84107860 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.471+0000 7f5b81ffb700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5b740a2e10 0x7f5b740a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.471+0000 7f5b8887c700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5b740a2e10 0x7f5b740a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.471+0000 7f5b81ffb700 1 -- 172.21.15.104:0/3162021502 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f5b7c0cf340 con 0x7f5b84107860 2024-06-09T22:22:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.472+0000 7f5b8887c700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5b740a2e10 0x7f5b740a52d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800b580 tx=0x7f5b78003780 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.472+0000 7f5b95363b80 1 Processor -- start 2024-06-09T22:22:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.472+0000 7f5b95363b80 1 -- start start 2024-06-09T22:22:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b95363b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1a640 0x55a8bcd1aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b95363b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1af60 0x55a8bcd149e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b95363b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 0x55a8bcd15340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b95363b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a8bcc66810 con 0x55a8bcd14f20 2024-06-09T22:22:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b95363b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a8bcc0e720 con 0x55a8bcd1af60 2024-06-09T22:22:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b95363b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a8bcc627d0 con 0x55a8bcd1a640 2024-06-09T22:22:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b8907d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 0x55a8bcd15340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b8907d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 0x55a8bcd15340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38044/0 (socket says 172.21.15.104:38044) 2024-06-09T22:22:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b8907d700 1 -- 172.21.15.104:0/656527543 learned_addr learned my addr 172.21.15.104:0/656527543 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b8887c700 1 --2- 172.21.15.104:0/656527543 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1a640 0x55a8bcd1aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b83fff700 1 --2- 172.21.15.104:0/656527543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1af60 0x55a8bcd149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b8907d700 1 -- 172.21.15.104:0/656527543 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1a640 msgr2=0x55a8bcd1aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b8907d700 1 --2- 172.21.15.104:0/656527543 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1a640 0x55a8bcd1aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b8907d700 1 -- 172.21.15.104:0/656527543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1af60 msgr2=0x55a8bcd149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b8907d700 1 --2- 172.21.15.104:0/656527543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1af60 0x55a8bcd149e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b8907d700 1 -- 172.21.15.104:0/656527543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8bcc04f50 con 0x55a8bcd14f20 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b83fff700 1 --2- 172.21.15.104:0/656527543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1af60 0x55a8bcd149e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b8907d700 1 --2- 172.21.15.104:0/656527543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 0x55a8bcd15340 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c0a3c30 tx=0x7f5b7c004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad79b0896a862361 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b6effd700 1 -- 172.21.15.104:0/656527543 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0cf160 con 0x55a8bcd14f20 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b6effd700 1 -- 172.21.15.104:0/656527543 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b7c09b580 con 0x55a8bcd14f20 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.473+0000 7f5b6effd700 1 -- 172.21.15.104:0/656527543 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0a3ed0 con 0x55a8bcd14f20 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.474+0000 7f5b95363b80 1 -- 172.21.15.104:0/656527543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 msgr2=0x55a8bcd15340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.474+0000 7f5b95363b80 1 --2- 172.21.15.104:0/656527543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 0x55a8bcd15340 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c0a3c30 tx=0x7f5b7c004d60 comp rx=0 tx=0).stop 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.474+0000 7f5b95363b80 1 -- 172.21.15.104:0/656527543 shutdown_connections 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.474+0000 7f5b95363b80 1 --2- 172.21.15.104:0/656527543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 0x55a8bcd15340 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.474+0000 7f5b95363b80 1 --2- 172.21.15.104:0/656527543 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1a640 0x55a8bcd1aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.474+0000 7f5b95363b80 1 --2- 172.21.15.104:0/656527543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1af60 0x55a8bcd149e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.474+0000 7f5b95363b80 1 -- 172.21.15.104:0/656527543 >> 172.21.15.104:0/656527543 conn(0x55a8bcc3fcc0 msgr2=0x55a8bcd0f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.474+0000 7f5b95363b80 1 -- 172.21.15.104:0/656527543 shutdown_connections 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.474+0000 7f5b95363b80 1 -- 172.21.15.104:0/656527543 wait complete. 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b95363b80 1 -- start start 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b95363b80 1 Processor -- start 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b95363b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd14f20 0x55a8bcd20d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b95363b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1a640 0x55a8bcd1e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b95363b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1af60 0x55a8bcd1e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b95363b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a8bcd21970 con 0x55a8bcd1a640 2024-06-09T22:22:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b95363b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a8bcc06bf0 con 0x55a8bcd14f20 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b95363b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a8bcd21af0 con 0x55a8bcd1af60 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b95363b80 10 client.0 _ll_register_callbacks cb 0x55a8bcc00920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b83fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1a640 0x55a8bcd1e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b83fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1a640 0x55a8bcd1e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38052/0 (socket says 172.21.15.104:38052) 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b83fff700 1 -- 172.21.15.104:0/1890648729 learned_addr learned my addr 172.21.15.104:0/1890648729 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b8887c700 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd14f20 0x55a8bcd20d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.475+0000 7f5b8907d700 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1af60 0x55a8bcd1e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b83fff700 1 -- 172.21.15.104:0/1890648729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1af60 msgr2=0x55a8bcd1e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b83fff700 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1af60 0x55a8bcd1e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b83fff700 1 -- 172.21.15.104:0/1890648729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd14f20 msgr2=0x55a8bcd20d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b83fff700 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd14f20 0x55a8bcd20d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b83fff700 1 -- 172.21.15.104:0/1890648729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8bcbb58d0 con 0x55a8bcd1a640 2024-06-09T22:22:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b8907d700 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1af60 0x55a8bcd1e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b83fff700 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1a640 0x55a8bcd1e0e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5b70010700 tx=0x7f5b70010ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf846ad4bef43f6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b63fff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b63fff700 1 -- 172.21.15.104:0/1890648729 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70017070 con 0x55a8bcd1a640 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b63fff700 1 -- 172.21.15.104:0/1890648729 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f5b70011970 con 0x55a8bcd1a640 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b95363b80 1 -- 172.21.15.104:0/1890648729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b7c09b580 con 0x55a8bcd1a640 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b63fff700 1 -- 172.21.15.104:0/1890648729 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5b70026050 con 0x55a8bcd1a640 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b95363b80 10 client.15987 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b95363b80 1 -- 172.21.15.104:0/1890648729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8bcd2c360 con 0x55a8bcd1a640 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b63fff700 1 -- 172.21.15.104:0/1890648729 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b7c09b580 con 0x55a8bcd1a640 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b63fff700 1 -- 172.21.15.104:0/1890648729 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5b7001a3b0 con 0x55a8bcd1a640 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b95363b80 10 client.15987 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b95363b80 10 client.15987 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:22:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.476+0000 7f5b95363b80 20 client.15987 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.477+0000 7f5b95363b80 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8bcd2e9f0 0x55a8bcd30e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.477+0000 7f5b95363b80 4 client.15987 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.477+0000 7f5b95363b80 1 -- 172.21.15.104:0/1890648729 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8bcc73d10 con 0x55a8bcd2e9f0 2024-06-09T22:22:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.477+0000 7f5b8887c700 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8bcd2e9f0 0x55a8bcd30e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.477+0000 7f5b8887c700 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8bcd2e9f0 0x55a8bcd30e10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c9dc009f6ded2c27 server_cookie=5cef87a7efa9947d in_seq=0 out_seq=0 2024-06-09T22:22:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.477+0000 7f5b63fff700 10 client.15987 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:22:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b63fff700 1 -- 172.21.15.104:0/1890648729 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a8bcc73d10 con 0x55a8bcd2e9f0 2024-06-09T22:22:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b63fff700 10 client.15987 handle_command_reply: tid=0 2024-06-09T22:22:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b95363b80 1 client.15987 shutdown 2024-06-09T22:22:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b95363b80 1 -- 172.21.15.104:0/1890648729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1a640 msgr2=0x55a8bcd1e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b95363b80 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1a640 0x55a8bcd1e0e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5b70010700 tx=0x7f5b70010ac0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b95363b80 1 -- 172.21.15.104:0/1890648729 shutdown_connections 2024-06-09T22:22:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b95363b80 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8bcd2e9f0 0x55a8bcd30e10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b95363b80 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1af60 0x55a8bcd1e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b95363b80 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd14f20 0x55a8bcd20d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b63fff700 0 client.15987 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:22:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b95363b80 1 --2- 172.21.15.104:0/1890648729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1a640 0x55a8bcd1e0e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b95363b80 1 -- 172.21.15.104:0/1890648729 >> 172.21.15.104:0/1890648729 conn(0x55a8bcc3fcc0 msgr2=0x55a8bcd139d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b95363b80 1 -- 172.21.15.104:0/1890648729 shutdown_connections 2024-06-09T22:22:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b95363b80 1 -- 172.21.15.104:0/1890648729 wait complete. 2024-06-09T22:22:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.478+0000 7f5b95363b80 20 client.15987 trim_cache size 0 max 16384 2024-06-09T22:22:46.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b95363b80 1 Processor -- start 2024-06-09T22:22:46.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b95363b80 1 -- start start 2024-06-09T22:22:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b95363b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd14f20 0x55a8bcd149e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b95363b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1a640 0x55a8bcd2ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b95363b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1af60 0x55a8bcd1ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b95363b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5b841ae4e0 con 0x55a8bcd1af60 2024-06-09T22:22:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b95363b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b841b0020 con 0x55a8bcd1a640 2024-06-09T22:22:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b95363b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5b841aebb0 con 0x55a8bcd14f20 2024-06-09T22:22:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b8907d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1af60 0x55a8bcd1ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b8907d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1af60 0x55a8bcd1ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38056/0 (socket says 172.21.15.104:38056) 2024-06-09T22:22:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b8907d700 1 -- 172.21.15.104:0/1121983806 learned_addr learned my addr 172.21.15.104:0/1121983806 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b8887c700 1 --2- 172.21.15.104:0/1121983806 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd14f20 0x55a8bcd149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b83fff700 1 --2- 172.21.15.104:0/1121983806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1a640 0x55a8bcd2ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b8907d700 1 -- 172.21.15.104:0/1121983806 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd14f20 msgr2=0x55a8bcd149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b8907d700 1 --2- 172.21.15.104:0/1121983806 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd14f20 0x55a8bcd149e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b8907d700 1 -- 172.21.15.104:0/1121983806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1a640 msgr2=0x55a8bcd2ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b8907d700 1 --2- 172.21.15.104:0/1121983806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1a640 0x55a8bcd2ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.494+0000 7f5b8907d700 1 -- 172.21.15.104:0/1121983806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8bcd10520 con 0x55a8bcd1af60 2024-06-09T22:22:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.495+0000 7f5b8907d700 1 --2- 172.21.15.104:0/1121983806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1af60 0x55a8bcd1ba70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c004fd0 tx=0x7f5b7c009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab7565eaa330886 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.495+0000 7f5b6e7fc700 1 -- 172.21.15.104:0/1121983806 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c09b580 con 0x55a8bcd1af60 2024-06-09T22:22:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.495+0000 7f5b6e7fc700 1 -- 172.21.15.104:0/1121983806 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b7c0cf160 con 0x55a8bcd1af60 2024-06-09T22:22:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.495+0000 7f5b6e7fc700 1 -- 172.21.15.104:0/1121983806 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0a3cd0 con 0x55a8bcd1af60 2024-06-09T22:22:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.495+0000 7f5b95363b80 1 -- 172.21.15.104:0/1121983806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1af60 msgr2=0x55a8bcd1ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.495+0000 7f5b95363b80 1 --2- 172.21.15.104:0/1121983806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1af60 0x55a8bcd1ba70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c004fd0 tx=0x7f5b7c009ed0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.495+0000 7f5b95363b80 1 -- 172.21.15.104:0/1121983806 shutdown_connections 2024-06-09T22:22:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.495+0000 7f5b95363b80 1 --2- 172.21.15.104:0/1121983806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd1af60 0x55a8bcd1ba70 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.495+0000 7f5b95363b80 1 --2- 172.21.15.104:0/1121983806 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd14f20 0x55a8bcd149e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.495+0000 7f5b95363b80 1 --2- 172.21.15.104:0/1121983806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1a640 0x55a8bcd2ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.495+0000 7f5b95363b80 1 -- 172.21.15.104:0/1121983806 >> 172.21.15.104:0/1121983806 conn(0x55a8bcc3fcc0 msgr2=0x55a8bcd1c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b95363b80 1 -- 172.21.15.104:0/1121983806 shutdown_connections 2024-06-09T22:22:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b95363b80 1 -- 172.21.15.104:0/1121983806 wait complete. 2024-06-09T22:22:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b95363b80 1 -- start start 2024-06-09T22:22:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b95363b80 1 Processor -- start 2024-06-09T22:22:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b95363b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 0x55a8bcd27600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b95363b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1a640 0x55a8bcd22980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b95363b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1af60 0x55a8bcd22fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b95363b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5b841af950 con 0x55a8bcd14f20 2024-06-09T22:22:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b95363b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b841af280 con 0x55a8bcd1af60 2024-06-09T22:22:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b95363b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5b841ac460 con 0x55a8bcd1a640 2024-06-09T22:22:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b95363b80 10 client.0 _ll_register_callbacks cb 0x55a8bcc00920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b8887c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 0x55a8bcd27600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b8887c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 0x55a8bcd27600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38066/0 (socket says 172.21.15.104:38066) 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.496+0000 7f5b8887c700 1 -- 172.21.15.104:0/4071972212 learned_addr learned my addr 172.21.15.104:0/4071972212 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b8907d700 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1af60 0x55a8bcd22fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b83fff700 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1a640 0x55a8bcd22980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b8887c700 1 -- 172.21.15.104:0/4071972212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1a640 msgr2=0x55a8bcd22980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b8887c700 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1a640 0x55a8bcd22980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b8887c700 1 -- 172.21.15.104:0/4071972212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1af60 msgr2=0x55a8bcd22fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b8887c700 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1af60 0x55a8bcd22fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b8887c700 1 -- 172.21.15.104:0/4071972212 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8bcd98690 con 0x55a8bcd14f20 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b83fff700 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1a640 0x55a8bcd22980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b8887c700 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 0x55a8bcd27600 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5b78003750 tx=0x7f5b78009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f73129859f24cc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b63fff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b63fff700 1 -- 172.21.15.104:0/4071972212 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7803d060 con 0x55a8bcd14f20 2024-06-09T22:22:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b63fff700 1 -- 172.21.15.104:0/4071972212 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f5b78005500 con 0x55a8bcd14f20 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b95363b80 1 -- 172.21.15.104:0/4071972212 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b7c0cf160 con 0x55a8bcd14f20 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b63fff700 1 -- 172.21.15.104:0/4071972212 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5b78044050 con 0x55a8bcd14f20 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b95363b80 10 client.15993 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b95363b80 1 -- 172.21.15.104:0/4071972212 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8bcd99850 con 0x55a8bcd14f20 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.497+0000 7f5b63fff700 1 -- 172.21.15.104:0/4071972212 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b7c0cf160 con 0x55a8bcd14f20 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b63fff700 1 -- 172.21.15.104:0/4071972212 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5b78037030 con 0x55a8bcd14f20 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b95363b80 10 client.15993 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b95363b80 10 client.15993 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b95363b80 20 client.15993 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b95363b80 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8bcd99eb0 0x55a8bcd9c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b95363b80 4 client.15993 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b95363b80 1 -- 172.21.15.104:0/4071972212 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a8bcc6e290 con 0x55a8bcd99eb0 2024-06-09T22:22:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b83fff700 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8bcd99eb0 0x55a8bcd9c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b63fff700 10 client.15993 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:22:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b83fff700 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8bcd99eb0 0x55a8bcd9c2a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a3d3be4e3599a1c8 server_cookie=76ef38c9cb378aa2 in_seq=0 out_seq=0 2024-06-09T22:22:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b63fff700 1 -- 172.21.15.104:0/4071972212 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a8bcc6e290 con 0x55a8bcd99eb0 2024-06-09T22:22:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b63fff700 10 client.15993 handle_command_reply: tid=0 2024-06-09T22:22:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.498+0000 7f5b95363b80 1 client.15993 shutdown 2024-06-09T22:22:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b95363b80 1 -- 172.21.15.104:0/4071972212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 msgr2=0x55a8bcd27600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b95363b80 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 0x55a8bcd27600 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5b78003750 tx=0x7f5b78009cd0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b95363b80 1 -- 172.21.15.104:0/4071972212 shutdown_connections 2024-06-09T22:22:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b95363b80 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55a8bcd99eb0 0x55a8bcd9c2a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b95363b80 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a8bcd14f20 0x55a8bcd27600 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b95363b80 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a8bcd1af60 0x55a8bcd22fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b95363b80 1 --2- 172.21.15.104:0/4071972212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8bcd1a640 0x55a8bcd22980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b63fff700 0 client.15993 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:22:46.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b95363b80 1 -- 172.21.15.104:0/4071972212 >> 172.21.15.104:0/4071972212 conn(0x55a8bcc3fcc0 msgr2=0x55a8bcd19a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:46.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b95363b80 1 -- 172.21.15.104:0/4071972212 shutdown_connections 2024-06-09T22:22:46.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b95363b80 1 -- 172.21.15.104:0/4071972212 wait complete. 2024-06-09T22:22:46.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b95363b80 20 client.15993 trim_cache size 0 max 16384 2024-06-09T22:22:46.505 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:22:46.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b8aae0700 1 -- 172.21.15.104:0/3162021502 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5b740a2e10 msgr2=0x7f5b740a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:46.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b8aae0700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5b740a2e10 0x7f5b740a52d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800b580 tx=0x7f5b78003780 comp rx=0 tx=0).stop 2024-06-09T22:22:46.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b8aae0700 1 -- 172.21.15.104:0/3162021502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b84107860 msgr2=0x7f5b841989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:46.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.499+0000 7f5b8aae0700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b84107860 0x7f5b841989c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00ef30 tx=0x7f5b7c00c450 comp rx=0 tx=0).stop 2024-06-09T22:22:46.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.500+0000 7f5b8aae0700 1 -- 172.21.15.104:0/3162021502 shutdown_connections 2024-06-09T22:22:46.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.500+0000 7f5b8aae0700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5b740a2e10 0x7f5b740a52d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.500+0000 7f5b8aae0700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b84107860 0x7f5b841989c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.500+0000 7f5b8aae0700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b84102770 0x7f5b8419e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.500+0000 7f5b8aae0700 1 --2- 172.21.15.104:0/3162021502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b84103120 0x7f5b8419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:46.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.500+0000 7f5b8aae0700 1 -- 172.21.15.104:0/3162021502 >> 172.21.15.104:0/3162021502 conn(0x7f5b84079600 msgr2=0x7f5b84101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:46.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.500+0000 7f5b8aae0700 1 -- 172.21.15.104:0/3162021502 shutdown_connections 2024-06-09T22:22:46.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:46.500+0000 7f5b8aae0700 1 -- 172.21.15.104:0/3162021502 wait complete. 2024-06-09T22:22:46.734 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:22:46.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:46 smithi153 ceph-mon[36599]: pgmap v57: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-06-09T22:22:47.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:46 smithi104 ceph-mon[32389]: pgmap v57: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-06-09T22:22:47.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:46 smithi194 ceph-mon[36856]: pgmap v57: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-06-09T22:22:47.183 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-09T22:22:47.308 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:48.043 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:48.831 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:48 smithi104 ceph-mon[32389]: pgmap v58: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-06-09T22:22:48.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:48 smithi153 ceph-mon[36599]: pgmap v58: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-06-09T22:22:49.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:48 smithi194 ceph-mon[36856]: pgmap v58: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-06-09T22:22:50.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.343+0000 7f4854de9700 1 -- 172.21.15.104:0/3749830730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4850106720 msgr2=0x7f485010e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.343+0000 7f4854de9700 1 --2- 172.21.15.104:0/3749830730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4850106720 0x7f485010e2c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f484800b0d0 tx=0x7f484800b3e0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.343+0000 7f4854de9700 1 -- 172.21.15.104:0/3749830730 shutdown_connections 2024-06-09T22:22:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.343+0000 7f4854de9700 1 --2- 172.21.15.104:0/3749830730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4850106720 0x7f485010e2c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.343+0000 7f4854de9700 1 --2- 172.21.15.104:0/3749830730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48501019d0 0x7f4850101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.343+0000 7f4854de9700 1 --2- 172.21.15.104:0/3749830730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4850102380 0x7f4850105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.343+0000 7f4854de9700 1 -- 172.21.15.104:0/3749830730 >> 172.21.15.104:0/3749830730 conn(0x7f4850078340 msgr2=0x7f4850078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.344+0000 7f4854de9700 1 -- 172.21.15.104:0/3749830730 shutdown_connections 2024-06-09T22:22:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.344+0000 7f4854de9700 1 -- 172.21.15.104:0/3749830730 wait complete. 2024-06-09T22:22:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.344+0000 7f4854de9700 1 Processor -- start 2024-06-09T22:22:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.344+0000 7f4854de9700 1 -- start start 2024-06-09T22:22:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.344+0000 7f4854de9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48501019d0 0x7f485019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.344+0000 7f4854de9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4850102380 0x7f485019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f4854de9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4850106720 0x7f4850198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f4854de9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f48501116e0 con 0x7f4850106720 2024-06-09T22:22:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f4854de9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4850111560 con 0x7f48501019d0 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f4854de9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4850111860 con 0x7f4850102380 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4850106720 0x7f4850198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4850106720 0x7f4850198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38080/0 (socket says 172.21.15.104:38080) 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484ed9d700 1 -- 172.21.15.104:0/3022102212 learned_addr learned my addr 172.21.15.104:0/3022102212 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484dd9b700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4850102380 0x7f485019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484e59c700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48501019d0 0x7f485019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484ed9d700 1 -- 172.21.15.104:0/3022102212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4850102380 msgr2=0x7f485019edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484ed9d700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4850102380 0x7f485019edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484ed9d700 1 -- 172.21.15.104:0/3022102212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48501019d0 msgr2=0x7f485019e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484ed9d700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48501019d0 0x7f485019e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484ed9d700 1 -- 172.21.15.104:0/3022102212 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4850199190 con 0x7f4850106720 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484e59c700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48501019d0 0x7f485019e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484dd9b700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4850102380 0x7f485019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f484ed9d700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4850106720 0x7f4850198a30 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4848006070 tx=0x7f4848004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:50.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f48477fe700 1 -- 172.21.15.104:0/3022102212 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848021070 con 0x7f4850106720 2024-06-09T22:22:50.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f4854de9700 1 -- 172.21.15.104:0/3022102212 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4850199420 con 0x7f4850106720 2024-06-09T22:22:50.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f48477fe700 1 -- 172.21.15.104:0/3022102212 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4848007df0 con 0x7f4850106720 2024-06-09T22:22:50.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f48477fe700 1 -- 172.21.15.104:0/3022102212 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848005470 con 0x7f4850106720 2024-06-09T22:22:50.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.345+0000 7f4854de9700 1 -- 172.21.15.104:0/3022102212 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4850102fc0 con 0x7f4850106720 2024-06-09T22:22:50.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.347+0000 7f48477fe700 1 -- 172.21.15.104:0/3022102212 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f4848024070 con 0x7f4850106720 2024-06-09T22:22:50.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.348+0000 7f48477fe700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f483c0a2d90 0x7f483c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.348+0000 7f48477fe700 1 -- 172.21.15.104:0/3022102212 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f484800f020 con 0x7f4850106720 2024-06-09T22:22:50.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.348+0000 7f484e59c700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f483c0a2d90 0x7f483c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.349+0000 7f484e59c700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f483c0a2d90 0x7f483c0a5250 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4838005fd0 tx=0x7f4838005de0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:50.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.349+0000 7f485f66cb80 1 Processor -- start 2024-06-09T22:22:50.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.349+0000 7f485f66cb80 1 -- start start 2024-06-09T22:22:50.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.349+0000 7f485f66cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0e640 0x55db6ae0ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.349+0000 7f485f66cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 0x55db6ae089e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.349+0000 7f485f66cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 0x55db6ae09340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.349+0000 7f485f66cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55db6ad5a810 con 0x55db6ae0ef60 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.349+0000 7f485f66cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55db6ad02720 con 0x55db6ae08f20 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.349+0000 7f485f66cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55db6ad567d0 con 0x55db6ae0e640 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.349+0000 7f484dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 0x55db6ae089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f484dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 0x55db6ae089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38096/0 (socket says 172.21.15.104:38096) 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f484dd9b700 1 -- 172.21.15.104:0/4061396678 learned_addr learned my addr 172.21.15.104:0/4061396678 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f484e59c700 1 --2- 172.21.15.104:0/4061396678 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0e640 0x55db6ae0ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f484ed9d700 1 --2- 172.21.15.104:0/4061396678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 0x55db6ae09340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f484dd9b700 1 -- 172.21.15.104:0/4061396678 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0e640 msgr2=0x55db6ae0ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f484dd9b700 1 --2- 172.21.15.104:0/4061396678 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0e640 0x55db6ae0ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f484dd9b700 1 -- 172.21.15.104:0/4061396678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 msgr2=0x55db6ae09340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f484dd9b700 1 --2- 172.21.15.104:0/4061396678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 0x55db6ae09340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f484dd9b700 1 -- 172.21.15.104:0/4061396678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db6acf8f50 con 0x55db6ae0ef60 2024-06-09T22:22:50.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f484dd9b700 1 --2- 172.21.15.104:0/4061396678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 0x55db6ae089e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4840008b80 tx=0x7f4840008e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e368e5c8e9ebc0db server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:50.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f4844ff9700 1 -- 172.21.15.104:0/4061396678 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4840014070 con 0x55db6ae0ef60 2024-06-09T22:22:50.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f4844ff9700 1 -- 172.21.15.104:0/4061396678 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4840004d10 con 0x55db6ae0ef60 2024-06-09T22:22:50.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.350+0000 7f4844ff9700 1 -- 172.21.15.104:0/4061396678 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4840010430 con 0x55db6ae0ef60 2024-06-09T22:22:50.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.351+0000 7f485f66cb80 1 -- 172.21.15.104:0/4061396678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 msgr2=0x55db6ae089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:50.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.351+0000 7f485f66cb80 1 --2- 172.21.15.104:0/4061396678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 0x55db6ae089e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4840008b80 tx=0x7f4840008e90 comp rx=0 tx=0).stop 2024-06-09T22:22:50.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.351+0000 7f485f66cb80 1 -- 172.21.15.104:0/4061396678 shutdown_connections 2024-06-09T22:22:50.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.351+0000 7f485f66cb80 1 --2- 172.21.15.104:0/4061396678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 0x55db6ae09340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.351+0000 7f485f66cb80 1 --2- 172.21.15.104:0/4061396678 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0e640 0x55db6ae0ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.351+0000 7f485f66cb80 1 --2- 172.21.15.104:0/4061396678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 0x55db6ae089e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.351+0000 7f485f66cb80 1 -- 172.21.15.104:0/4061396678 >> 172.21.15.104:0/4061396678 conn(0x55db6ad33cc0 msgr2=0x55db6ae03850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.351+0000 7f485f66cb80 1 -- 172.21.15.104:0/4061396678 shutdown_connections 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.351+0000 7f485f66cb80 1 -- 172.21.15.104:0/4061396678 wait complete. 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f485f66cb80 1 -- start start 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f485f66cb80 1 Processor -- start 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f485f66cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 0x55db6ae14c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f485f66cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0e640 0x55db6ae11f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f485f66cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 0x55db6ae124c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f485f66cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55db6ae159d0 con 0x55db6ae0ef60 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f485f66cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55db6acfabf0 con 0x55db6ae08f20 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f485f66cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55db6ae15b50 con 0x55db6ae0e640 2024-06-09T22:22:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f484ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 0x55db6ae124c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f485f66cb80 10 client.0 _ll_register_callbacks cb 0x55db6acf4920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f484ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 0x55db6ae124c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38100/0 (socket says 172.21.15.104:38100) 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f484ed9d700 1 -- 172.21.15.104:0/852948391 learned_addr learned my addr 172.21.15.104:0/852948391 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f484dd9b700 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0e640 0x55db6ae11f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f484e59c700 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 0x55db6ae14c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f484ed9d700 1 -- 172.21.15.104:0/852948391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0e640 msgr2=0x55db6ae11f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f484ed9d700 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0e640 0x55db6ae11f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f484ed9d700 1 -- 172.21.15.104:0/852948391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 msgr2=0x55db6ae14c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f484ed9d700 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 0x55db6ae14c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.352+0000 7f484ed9d700 1 -- 172.21.15.104:0/852948391 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db6aca98d0 con 0x55db6ae0ef60 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f484ed9d700 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 0x55db6ae124c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f484809d140 tx=0x7f48480074e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe913935e6788e4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f4835ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f4835ffb700 1 -- 172.21.15.104:0/852948391 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48480a1560 con 0x55db6ae0ef60 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f4835ffb700 1 -- 172.21.15.104:0/852948391 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f484801db50 con 0x55db6ae0ef60 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f485f66cb80 1 -- 172.21.15.104:0/852948391 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4840004d10 con 0x55db6ae0ef60 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f4835ffb700 1 -- 172.21.15.104:0/852948391 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f48480a5d90 con 0x55db6ae0ef60 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f485f66cb80 10 client.16005 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f485f66cb80 1 -- 172.21.15.104:0/852948391 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55db6ae202e0 con 0x55db6ae0ef60 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f4835ffb700 1 -- 172.21.15.104:0/852948391 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48480b60d0 con 0x55db6ae0ef60 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f4835ffb700 1 -- 172.21.15.104:0/852948391 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4848011070 con 0x55db6ae0ef60 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f485f66cb80 10 client.16005 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f485f66cb80 10 client.16005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.353+0000 7f485f66cb80 20 client.16005 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.354+0000 7f485f66cb80 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55db6ae22800 0x55db6ae24bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.354+0000 7f485f66cb80 4 client.16005 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:50.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.354+0000 7f485f66cb80 1 -- 172.21.15.104:0/852948391 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4840004d10 con 0x55db6ae22800 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.354+0000 7f484e59c700 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55db6ae22800 0x55db6ae24bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.354+0000 7f4835ffb700 10 client.16005 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.354+0000 7f484e59c700 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55db6ae22800 0x55db6ae24bf0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=424f52b842435bcd server_cookie=6bfc60a1dc688bd7 in_seq=0 out_seq=0 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f4835ffb700 1 -- 172.21.15.104:0/852948391 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x7f4840004d10 con 0x55db6ae22800 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f4835ffb700 10 client.16005 handle_command_reply: tid=0 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f485f66cb80 1 client.16005 shutdown 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f485f66cb80 1 -- 172.21.15.104:0/852948391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 msgr2=0x55db6ae124c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f485f66cb80 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 0x55db6ae124c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f484809d140 tx=0x7f48480074e0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f485f66cb80 1 -- 172.21.15.104:0/852948391 shutdown_connections 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f485f66cb80 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55db6ae22800 0x55db6ae24bf0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f485f66cb80 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0ef60 0x55db6ae124c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f485f66cb80 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 0x55db6ae14c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f485f66cb80 1 --2- 172.21.15.104:0/852948391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0e640 0x55db6ae11f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f485f66cb80 1 -- 172.21.15.104:0/852948391 >> 172.21.15.104:0/852948391 conn(0x55db6ad33cc0 msgr2=0x55db6ae079d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f4835ffb700 0 client.16005 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.355+0000 7f485f66cb80 1 -- 172.21.15.104:0/852948391 shutdown_connections 2024-06-09T22:22:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.356+0000 7f485f66cb80 1 -- 172.21.15.104:0/852948391 wait complete. 2024-06-09T22:22:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.356+0000 7f485f66cb80 20 client.16005 trim_cache size 0 max 16384 2024-06-09T22:22:50.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f485f66cb80 1 Processor -- start 2024-06-09T22:22:50.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f485f66cb80 1 -- start start 2024-06-09T22:22:50.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f485f66cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 0x55db6ae089e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f485f66cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0e640 0x55db6ae10870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f485f66cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0ef60 0x55db6ae10db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f485f66cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f48501b0640 con 0x55db6ae0e640 2024-06-09T22:22:50.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f485f66cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f48501b2180 con 0x55db6ae08f20 2024-06-09T22:22:50.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f485f66cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f48501b0d10 con 0x55db6ae0ef60 2024-06-09T22:22:50.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f484dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0e640 0x55db6ae10870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f484dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0e640 0x55db6ae10870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38112/0 (socket says 172.21.15.104:38112) 2024-06-09T22:22:50.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f484dd9b700 1 -- 172.21.15.104:0/469903083 learned_addr learned my addr 172.21.15.104:0/469903083 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:50.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f484ed9d700 1 --2- 172.21.15.104:0/469903083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0ef60 0x55db6ae10db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f484e59c700 1 --2- 172.21.15.104:0/469903083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 0x55db6ae089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f484dd9b700 1 -- 172.21.15.104:0/469903083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0ef60 msgr2=0x55db6ae10db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:50.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.371+0000 7f484dd9b700 1 --2- 172.21.15.104:0/469903083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0ef60 0x55db6ae10db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.372+0000 7f484dd9b700 1 -- 172.21.15.104:0/469903083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 msgr2=0x55db6ae089e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:50.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.372+0000 7f484dd9b700 1 --2- 172.21.15.104:0/469903083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 0x55db6ae089e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.372+0000 7f484dd9b700 1 -- 172.21.15.104:0/469903083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db6ae0e0f0 con 0x55db6ae0e640 2024-06-09T22:22:50.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.372+0000 7f484ed9d700 1 --2- 172.21.15.104:0/469903083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0ef60 0x55db6ae10db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:50.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.372+0000 7f484dd9b700 1 --2- 172.21.15.104:0/469903083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0e640 0x55db6ae10870 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4840008fd0 tx=0x7f4840008f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a97e65d2b08ba9 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:50.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.372+0000 7f4837fff700 1 -- 172.21.15.104:0/469903083 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4840014050 con 0x55db6ae0e640 2024-06-09T22:22:50.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.372+0000 7f4837fff700 1 -- 172.21.15.104:0/469903083 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4840004500 con 0x55db6ae0e640 2024-06-09T22:22:50.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.372+0000 7f4837fff700 1 -- 172.21.15.104:0/469903083 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4840003c70 con 0x55db6ae0e640 2024-06-09T22:22:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 -- 172.21.15.104:0/469903083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0e640 msgr2=0x55db6ae10870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 --2- 172.21.15.104:0/469903083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0e640 0x55db6ae10870 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4840008fd0 tx=0x7f4840008f60 comp rx=0 tx=0).stop 2024-06-09T22:22:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 -- 172.21.15.104:0/469903083 shutdown_connections 2024-06-09T22:22:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 --2- 172.21.15.104:0/469903083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0ef60 0x55db6ae10db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 --2- 172.21.15.104:0/469903083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae08f20 0x55db6ae089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 --2- 172.21.15.104:0/469903083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae0e640 0x55db6ae10870 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 -- 172.21.15.104:0/469903083 >> 172.21.15.104:0/469903083 conn(0x55db6ad33cc0 msgr2=0x55db6ae03380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 -- 172.21.15.104:0/469903083 shutdown_connections 2024-06-09T22:22:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 -- 172.21.15.104:0/469903083 wait complete. 2024-06-09T22:22:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 -- start start 2024-06-09T22:22:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 Processor -- start 2024-06-09T22:22:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae08f20 0x55db6ae1ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae0e640 0x55db6ae1b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0ef60 0x55db6ae29570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f48501b1ab0 con 0x55db6ae08f20 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f48501b13e0 con 0x55db6ae0e640 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.373+0000 7f485f66cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f48501ae5c0 con 0x55db6ae0ef60 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f485f66cb80 10 client.0 _ll_register_callbacks cb 0x55db6acf4920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f484e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae08f20 0x55db6ae1ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f484e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae08f20 0x55db6ae1ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38124/0 (socket says 172.21.15.104:38124) 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f484e59c700 1 -- 172.21.15.104:0/977957981 learned_addr learned my addr 172.21.15.104:0/977957981 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f484ed9d700 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0ef60 0x55db6ae29570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f484dd9b700 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae0e640 0x55db6ae1b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f484e59c700 1 -- 172.21.15.104:0/977957981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0ef60 msgr2=0x55db6ae29570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f484e59c700 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0ef60 0x55db6ae29570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f484e59c700 1 -- 172.21.15.104:0/977957981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae0e640 msgr2=0x55db6ae1b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f484e59c700 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae0e640 0x55db6ae1b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f484e59c700 1 -- 172.21.15.104:0/977957981 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db6ae29ab0 con 0x55db6ae08f20 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f484dd9b700 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae0e640 0x55db6ae1b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f484e59c700 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae08f20 0x55db6ae1ae70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4838005950 tx=0x7f4838004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe029130bd0de1da server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f4835ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f4835ffb700 1 -- 172.21.15.104:0/977957981 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483801dae0 con 0x55db6ae08f20 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f4835ffb700 1 -- 172.21.15.104:0/977957981 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f483801dc80 con 0x55db6ae08f20 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f485f66cb80 1 -- 172.21.15.104:0/977957981 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4840004500 con 0x55db6ae08f20 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f4835ffb700 1 -- 172.21.15.104:0/977957981 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f483803a050 con 0x55db6ae08f20 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f485f66cb80 10 client.16011 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.374+0000 7f485f66cb80 1 -- 172.21.15.104:0/977957981 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55db6ae8cc70 con 0x55db6ae08f20 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.375+0000 7f4835ffb700 1 -- 172.21.15.104:0/977957981 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4840004500 con 0x55db6ae08f20 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.375+0000 7f4835ffb700 1 -- 172.21.15.104:0/977957981 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f48380317c0 con 0x55db6ae08f20 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.375+0000 7f485f66cb80 10 client.16011 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.375+0000 7f485f66cb80 10 client.16011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.375+0000 7f485f66cb80 20 client.16011 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.375+0000 7f485f66cb80 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55db6ae8d330 0x55db6ae8f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.375+0000 7f485f66cb80 4 client.16011 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.375+0000 7f485f66cb80 1 -- 172.21.15.104:0/977957981 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55db6ad67d10 con 0x55db6ae8d330 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.375+0000 7f484dd9b700 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55db6ae8d330 0x55db6ae8f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.375+0000 7f4835ffb700 10 client.16011 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.375+0000 7f484dd9b700 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55db6ae8d330 0x55db6ae8f750 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=799764b01e4b317d server_cookie=6a432fcb18e266ec in_seq=0 out_seq=0 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f4835ffb700 1 -- 172.21.15.104:0/977957981 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55db6ad67d10 con 0x55db6ae8d330 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f4835ffb700 10 client.16011 handle_command_reply: tid=0 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f485f66cb80 1 client.16011 shutdown 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f485f66cb80 1 -- 172.21.15.104:0/977957981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae08f20 msgr2=0x55db6ae1ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f485f66cb80 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae08f20 0x55db6ae1ae70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4838005950 tx=0x7f4838004750 comp rx=0 tx=0).stop 2024-06-09T22:22:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f485f66cb80 1 -- 172.21.15.104:0/977957981 shutdown_connections 2024-06-09T22:22:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f485f66cb80 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55db6ae8d330 0x55db6ae8f750 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f485f66cb80 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55db6ae08f20 0x55db6ae1ae70 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f485f66cb80 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db6ae0ef60 0x55db6ae29570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f4835ffb700 0 client.16011 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f485f66cb80 1 --2- 172.21.15.104:0/977957981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db6ae0e640 0x55db6ae1b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f485f66cb80 1 -- 172.21.15.104:0/977957981 >> 172.21.15.104:0/977957981 conn(0x55db6ad33cc0 msgr2=0x55db6ae0f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f485f66cb80 1 -- 172.21.15.104:0/977957981 shutdown_connections 2024-06-09T22:22:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f485f66cb80 1 -- 172.21.15.104:0/977957981 wait complete. 2024-06-09T22:22:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.376+0000 7f485f66cb80 20 client.16011 trim_cache size 0 max 16384 2024-06-09T22:22:50.381 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:22:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.377+0000 7f4854de9700 1 -- 172.21.15.104:0/3022102212 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f483c0a2d90 msgr2=0x7f483c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.377+0000 7f4854de9700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f483c0a2d90 0x7f483c0a5250 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4838005fd0 tx=0x7f4838005de0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.377+0000 7f4854de9700 1 -- 172.21.15.104:0/3022102212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4850106720 msgr2=0x7f4850198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.377+0000 7f4854de9700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4850106720 0x7f4850198a30 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4848006070 tx=0x7f4848004ef0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.377+0000 7f4854de9700 1 -- 172.21.15.104:0/3022102212 shutdown_connections 2024-06-09T22:22:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.377+0000 7f4854de9700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f483c0a2d90 0x7f483c0a5250 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.377+0000 7f4854de9700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4850106720 0x7f4850198a30 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.377+0000 7f4854de9700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48501019d0 0x7f485019e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.377+0000 7f4854de9700 1 --2- 172.21.15.104:0/3022102212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4850102380 0x7f485019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.377+0000 7f4854de9700 1 -- 172.21.15.104:0/3022102212 >> 172.21.15.104:0/3022102212 conn(0x7f4850078340 msgr2=0x7f485006ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.377+0000 7f4854de9700 1 -- 172.21.15.104:0/3022102212 shutdown_connections 2024-06-09T22:22:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.377+0000 7f4854de9700 1 -- 172.21.15.104:0/3022102212 wait complete. 2024-06-09T22:22:50.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.881+0000 7fda2726a700 1 -- 172.21.15.104:0/1960487731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda20101440 msgr2=0x7fda2010f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.881+0000 7fda2726a700 1 --2- 172.21.15.104:0/1960487731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda20101440 0x7fda2010f330 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fda1800b0a0 tx=0x7fda1800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.881+0000 7fda2726a700 1 -- 172.21.15.104:0/1960487731 shutdown_connections 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.881+0000 7fda2726a700 1 --2- 172.21.15.104:0/1960487731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda20101440 0x7fda2010f330 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.881+0000 7fda2726a700 1 --2- 172.21.15.104:0/1960487731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda20108510 0x7fda201088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.881+0000 7fda2726a700 1 --2- 172.21.15.104:0/1960487731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda20100a80 0x7fda20100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.881+0000 7fda2726a700 1 -- 172.21.15.104:0/1960487731 >> 172.21.15.104:0/1960487731 conn(0x7fda20077a00 msgr2=0x7fda200fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.881+0000 7fda2726a700 1 -- 172.21.15.104:0/1960487731 shutdown_connections 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.881+0000 7fda2726a700 1 -- 172.21.15.104:0/1960487731 wait complete. 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda2726a700 1 Processor -- start 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda2726a700 1 -- start start 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda2726a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda20100a80 0x7fda2019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda2726a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda20101440 0x7fda2019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda2726a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda20108510 0x7fda20198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda2726a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fda20112770 con 0x7fda20100a80 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda2726a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fda201125f0 con 0x7fda20101440 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda2726a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fda201128f0 con 0x7fda20108510 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda25006700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda20100a80 0x7fda2019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda25006700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda20100a80 0x7fda2019e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38138/0 (socket says 172.21.15.104:38138) 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda25006700 1 -- 172.21.15.104:0/2427631465 learned_addr learned my addr 172.21.15.104:0/2427631465 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda24805700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda20101440 0x7fda2019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda25807700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda20108510 0x7fda20198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda25006700 1 -- 172.21.15.104:0/2427631465 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda20108510 msgr2=0x7fda20198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda25006700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda20108510 0x7fda20198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda25006700 1 -- 172.21.15.104:0/2427631465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda20101440 msgr2=0x7fda2019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda25006700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda20101440 0x7fda2019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.882+0000 7fda25006700 1 -- 172.21.15.104:0/2427631465 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda201991d0 con 0x7fda20100a80 2024-06-09T22:22:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.883+0000 7fda25006700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda20100a80 0x7fda2019e860 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00d950 tx=0x7fda1c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.883+0000 7fda167fc700 1 -- 172.21.15.104:0/2427631465 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c016070 con 0x7fda20100a80 2024-06-09T22:22:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.883+0000 7fda167fc700 1 -- 172.21.15.104:0/2427631465 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda1c009b40 con 0x7fda20100a80 2024-06-09T22:22:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.883+0000 7fda2726a700 1 -- 172.21.15.104:0/2427631465 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda201994c0 con 0x7fda20100a80 2024-06-09T22:22:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.883+0000 7fda167fc700 1 -- 172.21.15.104:0/2427631465 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c00f620 con 0x7fda20100a80 2024-06-09T22:22:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.883+0000 7fda2726a700 1 -- 172.21.15.104:0/2427631465 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda201a5730 con 0x7fda20100a80 2024-06-09T22:22:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.884+0000 7fda2726a700 1 -- 172.21.15.104:0/2427631465 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda20069180 con 0x7fda20100a80 2024-06-09T22:22:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.885+0000 7fda167fc700 1 -- 172.21.15.104:0/2427631465 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fda1c019070 con 0x7fda20100a80 2024-06-09T22:22:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.886+0000 7fda167fc700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fda0c0a2d90 0x7fda0c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.886+0000 7fda24805700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fda0c0a2d90 0x7fda0c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.886+0000 7fda167fc700 1 -- 172.21.15.104:0/2427631465 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fda1c0cf750 con 0x7fda20100a80 2024-06-09T22:22:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.886+0000 7fda24805700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fda0c0a2d90 0x7fda0c0a5250 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fda10005fd0 tx=0x7fda10005f60 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:50.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:50.890+0000 7fda167fc700 1 -- 172.21.15.104:0/2427631465 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fda1c09c000 con 0x7fda20100a80 2024-06-09T22:22:50.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:50 smithi153 ceph-mon[36599]: pgmap v59: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1023 B/s wr, 13 op/s 2024-06-09T22:22:51.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:50 smithi104 ceph-mon[32389]: pgmap v59: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1023 B/s wr, 13 op/s 2024-06-09T22:22:51.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:50 smithi194 ceph-mon[36856]: pgmap v59: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1023 B/s wr, 13 op/s 2024-06-09T22:22:51.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.135+0000 7fda2726a700 1 -- 172.21.15.104:0/2427631465 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda2002dca0 con 0x7fda20100a80 2024-06-09T22:22:51.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.136+0000 7fda167fc700 1 -- 172.21.15.104:0/2427631465 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11953 (secure 0 0 0) 0x7fda1c0afe20 con 0x7fda20100a80 2024-06-09T22:22:51.136 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:22:51.136 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:20:55.680750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24460,24448,24442,24403,24419]},"id":1}]} 2024-06-09T22:22:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.138+0000 7fda2726a700 1 -- 172.21.15.104:0/2427631465 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fda0c0a2d90 msgr2=0x7fda0c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.138+0000 7fda2726a700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fda0c0a2d90 0x7fda0c0a5250 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fda10005fd0 tx=0x7fda10005f60 comp rx=0 tx=0).stop 2024-06-09T22:22:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.138+0000 7fda2726a700 1 -- 172.21.15.104:0/2427631465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda20100a80 msgr2=0x7fda2019e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.138+0000 7fda2726a700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda20100a80 0x7fda2019e860 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00d950 tx=0x7fda1c00dd10 comp rx=0 tx=0).stop 2024-06-09T22:22:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.138+0000 7fda2726a700 1 -- 172.21.15.104:0/2427631465 shutdown_connections 2024-06-09T22:22:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.138+0000 7fda2726a700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fda0c0a2d90 0x7fda0c0a5250 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.138+0000 7fda2726a700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda20100a80 0x7fda2019e860 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.138+0000 7fda2726a700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda20108510 0x7fda20198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:51.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.138+0000 7fda2726a700 1 --2- 172.21.15.104:0/2427631465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda20101440 0x7fda2019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:51.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.138+0000 7fda2726a700 1 -- 172.21.15.104:0/2427631465 >> 172.21.15.104:0/2427631465 conn(0x7fda20077a00 msgr2=0x7fda200ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:51.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.138+0000 7fda2726a700 1 -- 172.21.15.104:0/2427631465 shutdown_connections 2024-06-09T22:22:51.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:51.138+0000 7fda2726a700 1 -- 172.21.15.104:0/2427631465 wait complete. 2024-06-09T22:22:51.139 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 29 2024-06-09T22:22:51.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:51 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2427631465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:22:51.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:51 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2427631465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:22:52.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:51 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2427631465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:22:52.472 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24460 quiesce db --set-id 79edc90d --release --await' 2024-06-09T22:22:52.472 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24460 quiesce db --set-id 79edc90d --release --await 2024-06-09T22:22:52.721 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:52.966 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-09T22:22:52.966 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-09T22:22:52.988 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:52 smithi104 ceph-mon[32389]: pgmap v60: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-09T22:22:52.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:52 smithi153 ceph-mon[36599]: pgmap v60: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-09T22:22:53.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:52 smithi194 ceph-mon[36856]: pgmap v60: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-09T22:22:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.521+0000 7f95c286b700 1 -- 172.21.15.104:0/1451702598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95bc107440 msgr2=0x7f95bc107820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.521+0000 7f95c286b700 1 --2- 172.21.15.104:0/1451702598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95bc107440 0x7f95bc107820 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009990 tx=0x7f95ac009ca0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.522+0000 7f95c286b700 1 -- 172.21.15.104:0/1451702598 shutdown_connections 2024-06-09T22:22:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.522+0000 7f95c286b700 1 --2- 172.21.15.104:0/1451702598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95bc107440 0x7f95bc107820 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.522+0000 7f95c286b700 1 --2- 172.21.15.104:0/1451702598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95bc101d80 0x7f95bc10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.522+0000 7f95c286b700 1 --2- 172.21.15.104:0/1451702598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95bc1013c0 0x7f95bc101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.522+0000 7f95c286b700 1 -- 172.21.15.104:0/1451702598 >> 172.21.15.104:0/1451702598 conn(0x7f95bc077400 msgr2=0x7f95bc077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.522+0000 7f95c286b700 1 -- 172.21.15.104:0/1451702598 shutdown_connections 2024-06-09T22:22:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.522+0000 7f95c286b700 1 -- 172.21.15.104:0/1451702598 wait complete. 2024-06-09T22:22:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.522+0000 7f95c286b700 1 Processor -- start 2024-06-09T22:22:54.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.522+0000 7f95c286b700 1 -- start start 2024-06-09T22:22:54.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95c286b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95bc1013c0 0x7f95bc19e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95c286b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95bc101d80 0x7f95bc19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95c286b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95bc107440 0x7f95bc198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95c286b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc111680 con 0x7f95bc1013c0 2024-06-09T22:22:54.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95c286b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc111500 con 0x7f95bc107440 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95c286b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc111800 con 0x7f95bc101d80 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95bbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95bc1013c0 0x7f95bc19e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95bbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95bc1013c0 0x7f95bc19e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38150/0 (socket says 172.21.15.104:38150) 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95bbfff700 1 -- 172.21.15.104:0/3682252180 learned_addr learned my addr 172.21.15.104:0/3682252180 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95c0e08700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95bc107440 0x7f95bc198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95bb7fe700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95bc101d80 0x7f95bc19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95bbfff700 1 -- 172.21.15.104:0/3682252180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95bc101d80 msgr2=0x7f95bc19ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95bbfff700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95bc101d80 0x7f95bc19ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95bbfff700 1 -- 172.21.15.104:0/3682252180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95bc107440 msgr2=0x7f95bc198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95bbfff700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95bc107440 0x7f95bc198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95bbfff700 1 -- 172.21.15.104:0/3682252180 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95bc199220 con 0x7f95bc1013c0 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95c0e08700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95bc107440 0x7f95bc198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:22:54.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.523+0000 7f95bbfff700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95bc1013c0 0x7f95bc19e900 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f95ac000c00 tx=0x7f95ac00b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:54.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.524+0000 7f95b97fa700 1 -- 172.21.15.104:0/3682252180 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac027070 con 0x7f95bc1013c0 2024-06-09T22:22:54.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.524+0000 7f95c286b700 1 -- 172.21.15.104:0/3682252180 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95bc1994b0 con 0x7f95bc1013c0 2024-06-09T22:22:54.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.524+0000 7f95b97fa700 1 -- 172.21.15.104:0/3682252180 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95ac022470 con 0x7f95bc1013c0 2024-06-09T22:22:54.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.524+0000 7f95b97fa700 1 -- 172.21.15.104:0/3682252180 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac017630 con 0x7f95bc1013c0 2024-06-09T22:22:54.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.524+0000 7f95c286b700 1 -- 172.21.15.104:0/3682252180 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95bc1a57d0 con 0x7f95bc1013c0 2024-06-09T22:22:54.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.526+0000 7f95b97fa700 1 -- 172.21.15.104:0/3682252180 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f95ac02a070 con 0x7f95bc1013c0 2024-06-09T22:22:54.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.527+0000 7f95b97fa700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f95a80a2d90 0x7f95a80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.527+0000 7f95bb7fe700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f95a80a2d90 0x7f95a80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.527+0000 7f95b97fa700 1 -- 172.21.15.104:0/3682252180 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f95ac0d0770 con 0x7f95bc1013c0 2024-06-09T22:22:54.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.527+0000 7f95bb7fe700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f95a80a2d90 0x7f95a80a5250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f95bc077a20 tx=0x7f95a4009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:54.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95cd0eeb80 1 Processor -- start 2024-06-09T22:22:54.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95cd0eeb80 1 -- start start 2024-06-09T22:22:54.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95cd0eeb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ff998e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95cd0eeb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff99e20 0x55a04ff93710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95cd0eeb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff93c50 0x55a04ff94070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95cd0eeb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a04fe86060 con 0x55a04ff93c50 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95cd0eeb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a04fee4950 con 0x55a04ff99e20 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95cd0eeb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a04fe4c560 con 0x55a04ff99500 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95c0e08700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff93c50 0x55a04ff94070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95c0e08700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff93c50 0x55a04ff94070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38160/0 (socket says 172.21.15.104:38160) 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95c0e08700 1 -- 172.21.15.104:0/4229199874 learned_addr learned my addr 172.21.15.104:0/4229199874 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95bb7fe700 1 --2- 172.21.15.104:0/4229199874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff99e20 0x55a04ff93710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95bbfff700 1 --2- 172.21.15.104:0/4229199874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ff998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95c0e08700 1 -- 172.21.15.104:0/4229199874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 msgr2=0x55a04ff998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95c0e08700 1 --2- 172.21.15.104:0/4229199874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ff998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95c0e08700 1 -- 172.21.15.104:0/4229199874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff99e20 msgr2=0x55a04ff93710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:54.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95c0e08700 1 --2- 172.21.15.104:0/4229199874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff99e20 0x55a04ff93710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.528+0000 7f95c0e08700 1 -- 172.21.15.104:0/4229199874 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a04fe339d0 con 0x55a04ff93c50 2024-06-09T22:22:54.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.529+0000 7f95c0e08700 1 --2- 172.21.15.104:0/4229199874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff93c50 0x55a04ff94070 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f95b4009440 tx=0x7f95b4009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814d43e1ea1453db server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:54.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.529+0000 7f95b27fc700 1 -- 172.21.15.104:0/4229199874 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b4013750 con 0x55a04ff93c50 2024-06-09T22:22:54.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.529+0000 7f95b27fc700 1 -- 172.21.15.104:0/4229199874 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95b4013dd0 con 0x55a04ff93c50 2024-06-09T22:22:54.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.529+0000 7f95b27fc700 1 -- 172.21.15.104:0/4229199874 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b40143e0 con 0x55a04ff93c50 2024-06-09T22:22:54.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.530+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/4229199874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff93c50 msgr2=0x55a04ff94070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:54.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.530+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/4229199874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff93c50 0x55a04ff94070 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f95b4009440 tx=0x7f95b4009920 comp rx=0 tx=0).stop 2024-06-09T22:22:54.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.530+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/4229199874 shutdown_connections 2024-06-09T22:22:54.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.530+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/4229199874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff93c50 0x55a04ff94070 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.530+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/4229199874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ff998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.530+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/4229199874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff99e20 0x55a04ff93710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.530+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/4229199874 >> 172.21.15.104:0/4229199874 conn(0x55a04fec0180 msgr2=0x55a04ff8ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:54.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.530+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/4229199874 shutdown_connections 2024-06-09T22:22:54.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.530+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/4229199874 wait complete. 2024-06-09T22:22:54.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.530+0000 7f95cd0eeb80 1 -- start start 2024-06-09T22:22:54.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.530+0000 7f95cd0eeb80 1 Processor -- start 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95cd0eeb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ffa75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95cd0eeb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ffa7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95cd0eeb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99e20 0x55a04ffa1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95cd0eeb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a04fe850c0 con 0x55a04ff99e20 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95cd0eeb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a04fee0910 con 0x55a04ff93c50 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95cd0eeb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a04ffa0800 con 0x55a04ff99500 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95cd0eeb80 10 client.0 _ll_register_callbacks cb 0x55a04fe7a6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95bb7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ffa7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95c0e08700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99e20 0x55a04ffa1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95bb7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ffa7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:56614/0 (socket says 172.21.15.104:56614) 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95bb7fe700 1 -- 172.21.15.104:0/807714462 learned_addr learned my addr 172.21.15.104:0/807714462 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95bbfff700 1 --2- 172.21.15.104:0/807714462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ffa75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95c0e08700 1 -- 172.21.15.104:0/807714462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 msgr2=0x55a04ffa7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95c0e08700 1 --2- 172.21.15.104:0/807714462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ffa7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95c0e08700 1 -- 172.21.15.104:0/807714462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 msgr2=0x55a04ffa75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95c0e08700 1 --2- 172.21.15.104:0/807714462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ffa75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95c0e08700 1 -- 172.21.15.104:0/807714462 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a04feaa850 con 0x55a04ff99e20 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f95c0e08700 1 --2- 172.21.15.104:0/807714462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99e20 0x55a04ffa1940 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f95b4007ac0 tx=0x7f95b40156f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3acb221ef15fb62 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f959b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.531+0000 7f959b7fe700 1 -- 172.21.15.104:0/807714462 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b4013550 con 0x55a04ff99e20 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/807714462 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95b4013dd0 con 0x55a04ff99e20 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f959b7fe700 1 -- 172.21.15.104:0/807714462 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f95b40136f0 con 0x55a04ff99e20 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f959b7fe700 1 -- 172.21.15.104:0/807714462 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f95b40138f0 con 0x55a04ff99e20 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f95cd0eeb80 10 client.16029 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/807714462 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a04ffaf320 con 0x55a04ff99e20 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f959b7fe700 1 -- 172.21.15.104:0/807714462 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95b4013dd0 con 0x55a04ff99e20 2024-06-09T22:22:54.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f959b7fe700 1 -- 172.21.15.104:0/807714462 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f95b40057c0 con 0x55a04ff99e20 2024-06-09T22:22:54.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f95cd0eeb80 10 client.16029 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:54.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f95cd0eeb80 10 client.16029 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:22:54.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f95cd0eeb80 20 client.16029 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:54.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/807714462 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a04ffb1900 0x55a04ffb3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f95cd0eeb80 4 client.16029 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:54.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.532+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/807714462 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a04fef1e50 con 0x55a04ffb1900 2024-06-09T22:22:54.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.533+0000 7f95bbfff700 1 --2- 172.21.15.104:0/807714462 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a04ffb1900 0x55a04ffb3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.534+0000 7f95bbfff700 1 --2- 172.21.15.104:0/807714462 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a04ffb1900 0x55a04ffb3cf0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f60a19c0ca1d034d server_cookie=8342f694b58b969c in_seq=0 out_seq=0 2024-06-09T22:22:54.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.534+0000 7f959b7fe700 10 client.16029 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:54.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.534+0000 7f959b7fe700 1 -- 172.21.15.104:0/807714462 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a04fef1e50 con 0x55a04ffb1900 2024-06-09T22:22:54.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.534+0000 7f959b7fe700 10 client.16029 handle_command_reply: tid=0 2024-06-09T22:22:54.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.534+0000 7f95cd0eeb80 1 client.16029 shutdown 2024-06-09T22:22:54.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.535+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/807714462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99e20 msgr2=0x55a04ffa1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:54.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.535+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/807714462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99e20 0x55a04ffa1940 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f95b4007ac0 tx=0x7f95b40156f0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.535+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/807714462 shutdown_connections 2024-06-09T22:22:54.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.535+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/807714462 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a04ffb1900 0x55a04ffb3cf0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.535+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/807714462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99e20 0x55a04ffa1940 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.535+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/807714462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ffa75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.535+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/807714462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ffa7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.535+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/807714462 >> 172.21.15.104:0/807714462 conn(0x55a04fec0180 msgr2=0x55a04ff8dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:54.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.535+0000 7f959b7fe700 0 client.16029 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:54.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.535+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/807714462 shutdown_connections 2024-06-09T22:22:54.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.535+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/807714462 wait complete. 2024-06-09T22:22:54.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.535+0000 7f95cd0eeb80 20 client.16029 trim_cache size 0 max 16384 2024-06-09T22:22:54.540 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.556+0000 7f95cd0eeb80 1 Processor -- start 2024-06-09T22:22:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.556+0000 7f95cd0eeb80 1 -- start start 2024-06-09T22:22:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.556+0000 7f95cd0eeb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ff9f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.556+0000 7f95cd0eeb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99500 0x55a04ff9fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.556+0000 7f95cd0eeb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99e20 0x55a04ff92f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95cd0eeb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc1ac220 con 0x55a04ff99500 2024-06-09T22:22:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95cd0eeb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc1add60 con 0x55a04ff93c50 2024-06-09T22:22:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95cd0eeb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc1ac8f0 con 0x55a04ff99e20 2024-06-09T22:22:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95bb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99500 0x55a04ff9fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95bb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99500 0x55a04ff9fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38168/0 (socket says 172.21.15.104:38168) 2024-06-09T22:22:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95bb7fe700 1 -- 172.21.15.104:0/1026988386 learned_addr learned my addr 172.21.15.104:0/1026988386 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95c0e08700 1 --2- 172.21.15.104:0/1026988386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99e20 0x55a04ff92f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95bbfff700 1 --2- 172.21.15.104:0/1026988386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ff9f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95bb7fe700 1 -- 172.21.15.104:0/1026988386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99e20 msgr2=0x55a04ff92f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95bb7fe700 1 --2- 172.21.15.104:0/1026988386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99e20 0x55a04ff92f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95bb7fe700 1 -- 172.21.15.104:0/1026988386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 msgr2=0x55a04ff9f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95bb7fe700 1 --2- 172.21.15.104:0/1026988386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ff9f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95bb7fe700 1 -- 172.21.15.104:0/1026988386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a04ff8ffe0 con 0x55a04ff99500 2024-06-09T22:22:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95bb7fe700 1 --2- 172.21.15.104:0/1026988386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99500 0x55a04ff9fc20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f95a40097d0 tx=0x7f95a4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c729ccdff60337 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:54.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95b1ffb700 1 -- 172.21.15.104:0/1026988386 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a4003840 con 0x55a04ff99500 2024-06-09T22:22:54.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.557+0000 7f95b1ffb700 1 -- 172.21.15.104:0/1026988386 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95a4012450 con 0x55a04ff99500 2024-06-09T22:22:54.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.558+0000 7f95b1ffb700 1 -- 172.21.15.104:0/1026988386 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a401f660 con 0x55a04ff99500 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.558+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/1026988386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99500 msgr2=0x55a04ff9fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.558+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/1026988386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99500 0x55a04ff9fc20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f95a40097d0 tx=0x7f95a4014000 comp rx=0 tx=0).stop 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.558+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/1026988386 shutdown_connections 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.558+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/1026988386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99e20 0x55a04ff92f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.558+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/1026988386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ff9f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.558+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/1026988386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99500 0x55a04ff9fc20 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.558+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/1026988386 >> 172.21.15.104:0/1026988386 conn(0x55a04fec0180 msgr2=0x55a04ffa17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.558+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/1026988386 shutdown_connections 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.558+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/1026988386 wait complete. 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.559+0000 7f95cd0eeb80 1 -- start start 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.559+0000 7f95cd0eeb80 1 Processor -- start 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95cd0eeb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ffabde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95cd0eeb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ffac320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95cd0eeb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99e20 0x55a05001b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95cd0eeb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc1ad690 con 0x55a04ff99e20 2024-06-09T22:22:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95bbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ffabde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95cd0eeb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc1acfc0 con 0x55a04ff93c50 2024-06-09T22:22:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95c0e08700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99e20 0x55a05001b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95bbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ffabde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:41550/0 (socket says 172.21.15.104:41550) 2024-06-09T22:22:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95bbfff700 1 -- 172.21.15.104:0/3303864575 learned_addr learned my addr 172.21.15.104:0/3303864575 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95bb7fe700 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ffac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95cd0eeb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc1aa1c0 con 0x55a04ff99500 2024-06-09T22:22:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95cd0eeb80 10 client.0 _ll_register_callbacks cb 0x55a04fe7a6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95bbfff700 1 -- 172.21.15.104:0/3303864575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 msgr2=0x55a04ffac320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95bbfff700 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ffac320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95bbfff700 1 -- 172.21.15.104:0/3303864575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99e20 msgr2=0x55a05001b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:54.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95bbfff700 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99e20 0x55a05001b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.560+0000 7f95bbfff700 1 -- 172.21.15.104:0/3303864575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a05001b780 con 0x55a04ff93c50 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.561+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/3303864575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95a4012450 con 0x55a04ff93c50 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.561+0000 7f95c0e08700 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99e20 0x55a05001b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.561+0000 7f95bbfff700 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ffabde0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0cdbb0 tx=0x7f95ac02e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=581383df7220de0c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.561+0000 7f959b7fe700 10 client.25463 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.561+0000 7f959b7fe700 1 -- 172.21.15.104:0/3303864575 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac0bb070 con 0x55a04ff93c50 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.561+0000 7f959b7fe700 1 -- 172.21.15.104:0/3303864575 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f95ac0229b0 con 0x55a04ff93c50 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.561+0000 7f959b7fe700 1 -- 172.21.15.104:0/3303864575 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f95ac0a0e80 con 0x55a04ff93c50 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.561+0000 7f95cd0eeb80 10 client.25463 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.561+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/3303864575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55a05001c980 con 0x55a04ff93c50 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.562+0000 7f959b7fe700 1 -- 172.21.15.104:0/3303864575 <== mon.2 v2:172.21.15.153:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f95ac0d0380 con 0x55a04ff93c50 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.562+0000 7f95cd0eeb80 10 client.25463 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.562+0000 7f95cd0eeb80 10 client.25463 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:22:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.562+0000 7f95cd0eeb80 20 client.25463 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.562+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a05001cf70 0x55a05001f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.562+0000 7f95cd0eeb80 4 client.25463 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "79edc90d", "release": true, "await": true}] 2024-06-09T22:22:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.562+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/3303864575 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "quiesce db", "set_id": "79edc90d", "release": true, "await": true}) -- 0x55a04feec3d0 con 0x55a05001cf70 2024-06-09T22:22:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.562+0000 7f95bb7fe700 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a05001cf70 0x55a05001f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.562+0000 7f959b7fe700 10 client.25463 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.562+0000 7f95bb7fe700 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a05001cf70 0x55a05001f380 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7b2837eb2e97ab server_cookie=eb47bac0b750409e in_seq=0 out_seq=0 2024-06-09T22:22:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f959b7fe700 1 -- 172.21.15.104:0/3303864575 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55a04feec3d0 con 0x55a05001cf70 2024-06-09T22:22:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f959b7fe700 10 client.25463 handle_command_reply: tid=0 2024-06-09T22:22:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f95cd0eeb80 1 client.25463 shutdown 2024-06-09T22:22:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/3303864575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 msgr2=0x55a04ffabde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ffabde0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0cdbb0 tx=0x7f95ac02e040 comp rx=0 tx=0).stop 2024-06-09T22:22:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/3303864575 shutdown_connections 2024-06-09T22:22:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a05001cf70 0x55a05001f380 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f959b7fe700 0 client.25463 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a04ff99e20 0x55a05001b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a04ff93c50 0x55a04ffabde0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f95cd0eeb80 1 --2- 172.21.15.104:0/3303864575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a04ff99500 0x55a04ffac320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/3303864575 >> 172.21.15.104:0/3303864575 conn(0x55a04fec0180 msgr2=0x55a04ff92f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.566+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/3303864575 shutdown_connections 2024-06-09T22:22:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.567+0000 7f95cd0eeb80 1 -- 172.21.15.104:0/3303864575 wait complete. 2024-06-09T22:22:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.567+0000 7f95cd0eeb80 20 client.25463 trim_cache size 0 max 16384 2024-06-09T22:22:54.568 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:22:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 29, 2024-06-09T22:22:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "leader": 24460, 2024-06-09T22:22:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "set_version": 4, 2024-06-09T22:22:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "sets": { 2024-06-09T22:22:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "79edc90d": { 2024-06-09T22:22:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "version": 4, 2024-06-09T22:22:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "age_ref": 0.0, 2024-06-09T22:22:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:22:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RELEASED", 2024-06-09T22:22:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "timeout": 90.0, 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "expiration": 150.0, 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "members": { 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "file:/": { 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "excluded": false, 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RELEASED", 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:22:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:22:54.570 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:22:54.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.567+0000 7f95c286b700 1 -- 172.21.15.104:0/3682252180 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f95a80a2d90 msgr2=0x7f95a80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:54.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.567+0000 7f95c286b700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f95a80a2d90 0x7f95a80a5250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f95bc077a20 tx=0x7f95a4009260 comp rx=0 tx=0).stop 2024-06-09T22:22:54.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.568+0000 7f95c286b700 1 -- 172.21.15.104:0/3682252180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95bc1013c0 msgr2=0x7f95bc19e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:54.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.568+0000 7f95c286b700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95bc1013c0 0x7f95bc19e900 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f95ac000c00 tx=0x7f95ac00b6c0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.568+0000 7f95c286b700 1 -- 172.21.15.104:0/3682252180 shutdown_connections 2024-06-09T22:22:54.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.568+0000 7f95c286b700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f95a80a2d90 0x7f95a80a5250 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.568+0000 7f95c286b700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95bc1013c0 0x7f95bc19e900 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.568+0000 7f95c286b700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95bc107440 0x7f95bc198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.568+0000 7f95c286b700 1 --2- 172.21.15.104:0/3682252180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95bc101d80 0x7f95bc19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:54.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.568+0000 7f95c286b700 1 -- 172.21.15.104:0/3682252180 >> 172.21.15.104:0/3682252180 conn(0x7f95bc077400 msgr2=0x7f95bc100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:54.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.568+0000 7f95c286b700 1 -- 172.21.15.104:0/3682252180 shutdown_connections 2024-06-09T22:22:54.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:54.568+0000 7f95c286b700 1 -- 172.21.15.104:0/3682252180 wait complete. 2024-06-09T22:22:54.571 INFO:tasks.workunit.client.0.smithi104.stderr:+ dbench 1 2024-06-09T22:22:54.604 INFO:tasks.workunit.client.0.smithi104.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-06-09T22:22:54.605 INFO:tasks.workunit.client.0.smithi104.stdout: 2024-06-09T22:22:54.605 INFO:tasks.workunit.client.0.smithi104.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-06-09T22:22:54.615 INFO:tasks.workunit.client.0.smithi104.stdout:0 of 1 processes prepared for launch 0 sec 2024-06-09T22:22:54.715 INFO:tasks.workunit.client.0.smithi104.stdout:1 of 1 processes prepared for launch 0 sec 2024-06-09T22:22:54.715 INFO:tasks.workunit.client.0.smithi104.stdout:releasing clients 2024-06-09T22:22:54.837 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:54 smithi104 ceph-mon[32389]: pgmap v61: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 13 op/s 2024-06-09T22:22:54.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:54 smithi153 ceph-mon[36599]: pgmap v61: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 13 op/s 2024-06-09T22:22:55.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:54 smithi194 ceph-mon[36856]: pgmap v61: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 13 op/s 2024-06-09T22:22:55.715 INFO:tasks.workunit.client.0.smithi104.stdout: 1 3556 38.42 MB/sec warmup 1 sec latency 7.287 ms 2024-06-09T22:22:56.084 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '79edc90d', total seconds elapsed: 39.4 2024-06-09T22:22:56.084 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-06-09T22:22:56.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:22:56.715 INFO:tasks.workunit.client.0.smithi104.stdout: 1 6477 26.30 MB/sec warmup 2 sec latency 13.901 ms 2024-06-09T22:22:56.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:56 smithi153 ceph-mon[36599]: pgmap v62: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-06-09T22:22:57.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:56 smithi104 ceph-mon[32389]: pgmap v62: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-06-09T22:22:57.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:56 smithi194 ceph-mon[36856]: pgmap v62: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-06-09T22:22:57.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.263+0000 7f6aafe5a700 1 -- 172.21.15.104:0/536842662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aa8101510 msgr2=0x7f6aa810d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:57.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.263+0000 7f6aafe5a700 1 --2- 172.21.15.104:0/536842662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aa8101510 0x7f6aa810d9d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6aa000b0a0 tx=0x7f6aa000b3b0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.263+0000 7f6aafe5a700 1 -- 172.21.15.104:0/536842662 shutdown_connections 2024-06-09T22:22:57.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.263+0000 7f6aafe5a700 1 --2- 172.21.15.104:0/536842662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aa8101510 0x7f6aa810d9d0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.263+0000 7f6aafe5a700 1 --2- 172.21.15.104:0/536842662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6aa8106b90 0x7f6aa8106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.263+0000 7f6aafe5a700 1 --2- 172.21.15.104:0/536842662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6aa8100b50 0x7f6aa8100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.263+0000 7f6aafe5a700 1 -- 172.21.15.104:0/536842662 >> 172.21.15.104:0/536842662 conn(0x7f6aa8077a00 msgr2=0x7f6aa80fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:57.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.263+0000 7f6aafe5a700 1 -- 172.21.15.104:0/536842662 shutdown_connections 2024-06-09T22:22:57.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.263+0000 7f6aafe5a700 1 -- 172.21.15.104:0/536842662 wait complete. 2024-06-09T22:22:57.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aafe5a700 1 Processor -- start 2024-06-09T22:22:57.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aafe5a700 1 -- start start 2024-06-09T22:22:57.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aafe5a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6aa8100b50 0x7f6aa80fce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aafe5a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aa8101510 0x7f6aa80fd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aafe5a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6aa8106b90 0x7f6aa80fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aafe5a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa81105c0 con 0x7f6aa8101510 2024-06-09T22:22:57.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aafe5a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa8110440 con 0x7f6aa8106b90 2024-06-09T22:22:57.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aafe5a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa8110740 con 0x7f6aa8100b50 2024-06-09T22:22:57.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aad3f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aa8101510 0x7f6aa80fd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aadbf6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6aa8100b50 0x7f6aa80fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aadbf6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6aa8100b50 0x7f6aa80fce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:41522/0 (socket says 172.21.15.104:41522) 2024-06-09T22:22:57.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aad3f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aa8101510 0x7f6aa80fd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38590/0 (socket says 172.21.15.104:38590) 2024-06-09T22:22:57.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aadbf6700 1 -- 172.21.15.104:0/21635496 learned_addr learned my addr 172.21.15.104:0/21635496 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:57.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.264+0000 7f6aae3f7700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6aa8106b90 0x7f6aa80fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.265+0000 7f6aadbf6700 1 -- 172.21.15.104:0/21635496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6aa8106b90 msgr2=0x7f6aa80fd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:57.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.265+0000 7f6aadbf6700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6aa8106b90 0x7f6aa80fd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.265+0000 7f6aadbf6700 1 -- 172.21.15.104:0/21635496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aa8101510 msgr2=0x7f6aa80fd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:57.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.265+0000 7f6aadbf6700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aa8101510 0x7f6aa80fd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.265+0000 7f6aadbf6700 1 -- 172.21.15.104:0/21635496 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa81a70d0 con 0x7f6aa8100b50 2024-06-09T22:22:57.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.265+0000 7f6aad3f5700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aa8101510 0x7f6aa80fd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:57.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.265+0000 7f6aadbf6700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6aa8100b50 0x7f6aa80fce70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400d980 tx=0x7f6aa400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:57.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.265+0000 7f6a9effd700 1 -- 172.21.15.104:0/21635496 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4016070 con 0x7f6aa8100b50 2024-06-09T22:22:57.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.265+0000 7f6aafe5a700 1 -- 172.21.15.104:0/21635496 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa81a73c0 con 0x7f6aa8100b50 2024-06-09T22:22:57.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.265+0000 7f6a9effd700 1 -- 172.21.15.104:0/21635496 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6aa4004d10 con 0x7f6aa8100b50 2024-06-09T22:22:57.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.265+0000 7f6a9effd700 1 -- 172.21.15.104:0/21635496 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4005670 con 0x7f6aa8100b50 2024-06-09T22:22:57.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.265+0000 7f6aafe5a700 1 -- 172.21.15.104:0/21635496 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa81a7950 con 0x7f6aa8100b50 2024-06-09T22:22:57.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.268+0000 7f6a9effd700 1 -- 172.21.15.104:0/21635496 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f6aa4004750 con 0x7f6aa8100b50 2024-06-09T22:22:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.268+0000 7f6a9effd700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6a940a2e10 0x7f6a940a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.268+0000 7f6aad3f5700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6a940a2e10 0x7f6a940a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.268+0000 7f6a9effd700 1 -- 172.21.15.104:0/21635496 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6aa40cf500 con 0x7f6aa8100b50 2024-06-09T22:22:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aad3f5700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6a940a2e10 0x7f6a940a52d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8105330 tx=0x7f6a98010040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aba6ddb80 1 Processor -- start 2024-06-09T22:22:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aba6ddb80 1 -- start start 2024-06-09T22:22:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aba6ddb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b86c0 0x558a662b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aba6ddb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aba6ddb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aba6ddb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x558a66204890 con 0x558a662b8fe0 2024-06-09T22:22:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aba6ddb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558a661ac7a0 con 0x558a662b2fa0 2024-06-09T22:22:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aba6ddb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558a66200850 con 0x558a662b86c0 2024-06-09T22:22:57.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aad3f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aad3f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662b2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38600/0 (socket says 172.21.15.104:38600) 2024-06-09T22:22:57.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aad3f5700 1 -- 172.21.15.104:0/1490092860 learned_addr learned my addr 172.21.15.104:0/1490092860 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:57.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.269+0000 7f6aadbf6700 1 --2- 172.21.15.104:0/1490092860 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b86c0 0x558a662b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.270+0000 7f6aae3f7700 1 --2- 172.21.15.104:0/1490092860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.270+0000 7f6aad3f5700 1 -- 172.21.15.104:0/1490092860 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b86c0 msgr2=0x558a662b8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:57.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.270+0000 7f6aad3f5700 1 --2- 172.21.15.104:0/1490092860 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b86c0 0x558a662b8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.270+0000 7f6aad3f5700 1 -- 172.21.15.104:0/1490092860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 msgr2=0x558a662b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:57.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.270+0000 7f6aad3f5700 1 --2- 172.21.15.104:0/1490092860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662b33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.270+0000 7f6aad3f5700 1 -- 172.21.15.104:0/1490092860 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a661a2fd0 con 0x558a662b8fe0 2024-06-09T22:22:57.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.270+0000 7f6aad3f5700 1 --2- 172.21.15.104:0/1490092860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662b2a60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6a980043c0 tx=0x7f6a98014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec1b95aa5dcaab3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:57.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.270+0000 7f6a93fff700 1 -- 172.21.15.104:0/1490092860 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a98003b10 con 0x558a662b8fe0 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.270+0000 7f6a93fff700 1 -- 172.21.15.104:0/1490092860 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a98003cb0 con 0x558a662b8fe0 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.270+0000 7f6a93fff700 1 -- 172.21.15.104:0/1490092860 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9801c760 con 0x558a662b8fe0 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.271+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/1490092860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 msgr2=0x558a662b2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.271+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/1490092860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662b2a60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6a980043c0 tx=0x7f6a98014000 comp rx=0 tx=0).stop 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.271+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/1490092860 shutdown_connections 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.271+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/1490092860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.271+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/1490092860 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b86c0 0x558a662b8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.271+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/1490092860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662b2a60 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.271+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/1490092860 >> 172.21.15.104:0/1490092860 conn(0x558a661ddd40 msgr2=0x558a662ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.271+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/1490092860 shutdown_connections 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.271+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/1490092860 wait complete. 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.271+0000 7f6aba6ddb80 1 -- start start 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.271+0000 7f6aba6ddb80 1 Processor -- start 2024-06-09T22:22:57.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aba6ddb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b2fa0 0x558a662bed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aba6ddb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b86c0 0x558a662bc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aba6ddb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662bc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aba6ddb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x558a662bfa50 con 0x558a662b8fe0 2024-06-09T22:22:57.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aba6ddb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558a661a4c70 con 0x558a662b86c0 2024-06-09T22:22:57.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aba6ddb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558a662bfbd0 con 0x558a662b2fa0 2024-06-09T22:22:57.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aba6ddb80 10 client.0 _ll_register_callbacks cb 0x558a6619e9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:57.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aae3f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662bc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aae3f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662bc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38616/0 (socket says 172.21.15.104:38616) 2024-06-09T22:22:57.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aae3f7700 1 -- 172.21.15.104:0/149481821 learned_addr learned my addr 172.21.15.104:0/149481821 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:57.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aad3f5700 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b86c0 0x558a662bc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aadbf6700 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b2fa0 0x558a662bed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aae3f7700 1 -- 172.21.15.104:0/149481821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b2fa0 msgr2=0x558a662bed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:57.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aae3f7700 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b2fa0 0x558a662bed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aae3f7700 1 -- 172.21.15.104:0/149481821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b86c0 msgr2=0x558a662bc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:57.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aae3f7700 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b86c0 0x558a662bc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aae3f7700 1 -- 172.21.15.104:0/149481821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a66153950 con 0x558a662b8fe0 2024-06-09T22:22:57.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6aae3f7700 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662bc610 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0009610 tx=0x7f6aa000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3caf4fb740adc86 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:57.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.272+0000 7f6a917fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:57.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6a917fa700 1 -- 172.21.15.104:0/149481821 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa00074e0 con 0x558a662b8fe0 2024-06-09T22:22:57.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6a917fa700 1 -- 172.21.15.104:0/149481821 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6aa0007680 con 0x558a662b8fe0 2024-06-09T22:22:57.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/149481821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a98003cb0 con 0x558a662b8fe0 2024-06-09T22:22:57.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6a917fa700 1 -- 172.21.15.104:0/149481821 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6aa0007880 con 0x558a662b8fe0 2024-06-09T22:22:57.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6aba6ddb80 10 client.16047 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:57.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/149481821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558a662ca3d0 con 0x558a662b8fe0 2024-06-09T22:22:57.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6a917fa700 1 -- 172.21.15.104:0/149481821 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a98003cb0 con 0x558a662b8fe0 2024-06-09T22:22:57.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6a917fa700 1 -- 172.21.15.104:0/149481821 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6aa0011070 con 0x558a662b8fe0 2024-06-09T22:22:57.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6aba6ddb80 10 client.16047 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:57.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6aba6ddb80 10 client.16047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:57.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6aba6ddb80 20 client.16047 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:57.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558a662cca10 0x558a662cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6aba6ddb80 4 client.16047 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:57.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/149481821 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a66211d90 con 0x558a662cca10 2024-06-09T22:22:57.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.273+0000 7f6aadbf6700 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558a662cca10 0x558a662cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.274+0000 7f6aadbf6700 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558a662cca10 0x558a662cee00 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acd5aa7bcc0338ac server_cookie=969bc42b3e6b5d1b in_seq=0 out_seq=0 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.274+0000 7f6a917fa700 10 client.16047 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6a917fa700 1 -- 172.21.15.104:0/149481821 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x558a66211d90 con 0x558a662cca10 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6a917fa700 10 client.16047 handle_command_reply: tid=0 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6aba6ddb80 1 client.16047 shutdown 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/149481821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 msgr2=0x558a662bc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662bc610 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0009610 tx=0x7f6aa000be20 comp rx=0 tx=0).stop 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/149481821 shutdown_connections 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558a662cca10 0x558a662cee00 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662bc610 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b2fa0 0x558a662bed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6a917fa700 0 client.16047 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:57.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/149481821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b86c0 0x558a662bc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/149481821 >> 172.21.15.104:0/149481821 conn(0x558a661ddd40 msgr2=0x558a662ad670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:57.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/149481821 shutdown_connections 2024-06-09T22:22:57.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/149481821 wait complete. 2024-06-09T22:22:57.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.275+0000 7f6aba6ddb80 20 client.16047 trim_cache size 0 max 16384 2024-06-09T22:22:57.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.298+0000 7f6aba6ddb80 1 Processor -- start 2024-06-09T22:22:57.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.298+0000 7f6aba6ddb80 1 -- start start 2024-06-09T22:22:57.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aba6ddb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aba6ddb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b86c0 0x558a662b9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aba6ddb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b8fe0 0x558a662d4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aba6ddb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa81b0660 con 0x558a662b86c0 2024-06-09T22:22:57.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aba6ddb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa81b21a0 con 0x558a662b2fa0 2024-06-09T22:22:57.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aba6ddb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa81b0d30 con 0x558a662b8fe0 2024-06-09T22:22:57.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aad3f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b86c0 0x558a662b9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aad3f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b86c0 0x558a662b9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38626/0 (socket says 172.21.15.104:38626) 2024-06-09T22:22:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aad3f5700 1 -- 172.21.15.104:0/3738723233 learned_addr learned my addr 172.21.15.104:0/3738723233 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aae3f7700 1 --2- 172.21.15.104:0/3738723233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b8fe0 0x558a662d4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aadbf6700 1 --2- 172.21.15.104:0/3738723233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aad3f5700 1 -- 172.21.15.104:0/3738723233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b8fe0 msgr2=0x558a662d4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aad3f5700 1 --2- 172.21.15.104:0/3738723233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b8fe0 0x558a662d4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aad3f5700 1 -- 172.21.15.104:0/3738723233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 msgr2=0x558a662b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aad3f5700 1 --2- 172.21.15.104:0/3738723233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662b2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aad3f5700 1 -- 172.21.15.104:0/3738723233 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a662cf460 con 0x558a662b86c0 2024-06-09T22:22:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aae3f7700 1 --2- 172.21.15.104:0/3738723233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b8fe0 0x558a662d4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.299+0000 7f6aad3f5700 1 --2- 172.21.15.104:0/3738723233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b86c0 0x558a662b9b90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6a98003960 tx=0x7f6a9801bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ebd6513adb12f4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.300+0000 7f6a937fe700 1 -- 172.21.15.104:0/3738723233 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a98003b10 con 0x558a662b86c0 2024-06-09T22:22:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.300+0000 7f6a937fe700 1 -- 172.21.15.104:0/3738723233 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a98003cb0 con 0x558a662b86c0 2024-06-09T22:22:57.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.300+0000 7f6a937fe700 1 -- 172.21.15.104:0/3738723233 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a98026730 con 0x558a662b86c0 2024-06-09T22:22:57.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.301+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3738723233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b86c0 msgr2=0x558a662b9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:57.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.301+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/3738723233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b86c0 0x558a662b9b90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6a98003960 tx=0x7f6a9801bf50 comp rx=0 tx=0).stop 2024-06-09T22:22:57.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.301+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3738723233 shutdown_connections 2024-06-09T22:22:57.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.301+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/3738723233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b8fe0 0x558a662d4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.301+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/3738723233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662b2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.301+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/3738723233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b86c0 0x558a662b9b90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.301+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3738723233 >> 172.21.15.104:0/3738723233 conn(0x558a661ddd40 msgr2=0x558a662b9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:57.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.301+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3738723233 shutdown_connections 2024-06-09T22:22:57.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.301+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3738723233 wait complete. 2024-06-09T22:22:57.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.301+0000 7f6aba6ddb80 1 -- start start 2024-06-09T22:22:57.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aba6ddb80 1 Processor -- start 2024-06-09T22:22:57.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aba6ddb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662c1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aba6ddb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b86c0 0x558a662c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aba6ddb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662c28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aba6ddb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa81b1ad0 con 0x558a662b8fe0 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aba6ddb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa81b1400 con 0x558a662b2fa0 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aba6ddb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa81ae5e0 con 0x558a662b86c0 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aba6ddb80 10 client.0 _ll_register_callbacks cb 0x558a6619e9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aae3f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662c28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aad3f5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b86c0 0x558a662c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aae3f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662c28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38634/0 (socket says 172.21.15.104:38634) 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aadbf6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662c1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.302+0000 7f6aae3f7700 1 -- 172.21.15.104:0/3870206455 learned_addr learned my addr 172.21.15.104:0/3870206455 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6aae3f7700 1 -- 172.21.15.104:0/3870206455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b86c0 msgr2=0x558a662c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:57.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6aae3f7700 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b86c0 0x558a662c23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6aae3f7700 1 -- 172.21.15.104:0/3870206455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 msgr2=0x558a662c1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:57.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6aae3f7700 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662c1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6aae3f7700 1 -- 172.21.15.104:0/3870206455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a66337670 con 0x558a662b8fe0 2024-06-09T22:22:57.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6aad3f5700 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b86c0 0x558a662c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:57.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6aadbf6700 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662c1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:57.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6aae3f7700 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662c28e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0009fd0 tx=0x7f6aa0006250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3118f5fef7194e5f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:57.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6a917fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:57.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6a917fa700 1 -- 172.21.15.104:0/3870206455 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0039060 con 0x558a662b8fe0 2024-06-09T22:22:57.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6a917fa700 1 -- 172.21.15.104:0/3870206455 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6aa000f040 con 0x558a662b8fe0 2024-06-09T22:22:57.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3870206455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a98003cb0 con 0x558a662b8fe0 2024-06-09T22:22:57.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6a917fa700 1 -- 172.21.15.104:0/3870206455 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6aa0040050 con 0x558a662b8fe0 2024-06-09T22:22:57.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6aba6ddb80 10 client.16053 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:57.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3870206455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558a66338900 con 0x558a662b8fe0 2024-06-09T22:22:57.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.303+0000 7f6a917fa700 1 -- 172.21.15.104:0/3870206455 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a98003cb0 con 0x558a662b8fe0 2024-06-09T22:22:57.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.304+0000 7f6a917fa700 1 -- 172.21.15.104:0/3870206455 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6aa0011040 con 0x558a662b8fe0 2024-06-09T22:22:57.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.304+0000 7f6aba6ddb80 10 client.16053 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:57.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.304+0000 7f6aba6ddb80 10 client.16053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:57.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.305+0000 7f6aba6ddb80 20 client.16053 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:57.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.305+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558a66338ef0 0x558a6633b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:57.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.305+0000 7f6aba6ddb80 4 client.16053 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-09T22:22:57.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.305+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3870206455 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558a6620c310 con 0x558a66338ef0 2024-06-09T22:22:57.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.305+0000 7f6aadbf6700 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558a66338ef0 0x558a6633b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:57.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.305+0000 7f6aadbf6700 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558a66338ef0 0x558a6633b310 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57a773dd06b2f564 server_cookie=613ed10b4122ba2c in_seq=0 out_seq=0 2024-06-09T22:22:57.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.305+0000 7f6a917fa700 10 client.16053 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:57.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.307+0000 7f6a917fa700 1 -- 172.21.15.104:0/3870206455 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558a6620c310 con 0x558a66338ef0 2024-06-09T22:22:57.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.307+0000 7f6a917fa700 10 client.16053 handle_command_reply: tid=0 2024-06-09T22:22:57.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.307+0000 7f6aba6ddb80 1 client.16053 shutdown 2024-06-09T22:22:57.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.308+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3870206455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 msgr2=0x558a662c28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:57.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.308+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662c28e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0009fd0 tx=0x7f6aa0006250 comp rx=0 tx=0).stop 2024-06-09T22:22:57.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.308+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3870206455 shutdown_connections 2024-06-09T22:22:57.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.308+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558a66338ef0 0x558a6633b310 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.308+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558a662b8fe0 0x558a662c28e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.308+0000 7f6a917fa700 0 client.16053 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:57.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.308+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558a662b2fa0 0x558a662c1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.308+0000 7f6aba6ddb80 1 --2- 172.21.15.104:0/3870206455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558a662b86c0 0x558a662c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.308+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3870206455 >> 172.21.15.104:0/3870206455 conn(0x558a661ddd40 msgr2=0x558a662a6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:57.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.308+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3870206455 shutdown_connections 2024-06-09T22:22:57.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.308+0000 7f6aba6ddb80 1 -- 172.21.15.104:0/3870206455 wait complete. 2024-06-09T22:22:57.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.308+0000 7f6aba6ddb80 20 client.16053 trim_cache size 0 max 16384 2024-06-09T22:22:57.311 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:22:57.311 INFO:teuthology.orchestra.run.smithi104.stdout: "return_code": 0, 2024-06-09T22:22:57.311 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_tag": "398b55fc-c28d-4c44-98a6-2bd4948ea268", 2024-06-09T22:22:57.311 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": "asynchronous" 2024-06-09T22:22:57.311 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:22:57.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.309+0000 7f6aafe5a700 1 -- 172.21.15.104:0/21635496 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6a940a2e10 msgr2=0x7f6a940a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:57.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.309+0000 7f6aafe5a700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6a940a2e10 0x7f6a940a52d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8105330 tx=0x7f6a98010040 comp rx=0 tx=0).stop 2024-06-09T22:22:57.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.309+0000 7f6aafe5a700 1 -- 172.21.15.104:0/21635496 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6aa8100b50 msgr2=0x7f6aa80fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:57.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.309+0000 7f6aafe5a700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6aa8100b50 0x7f6aa80fce70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400d980 tx=0x7f6aa400dc90 comp rx=0 tx=0).stop 2024-06-09T22:22:57.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.309+0000 7f6aafe5a700 1 -- 172.21.15.104:0/21635496 shutdown_connections 2024-06-09T22:22:57.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.309+0000 7f6aafe5a700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6a940a2e10 0x7f6a940a52d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.309+0000 7f6aafe5a700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6aa8106b90 0x7f6aa80fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.309+0000 7f6aafe5a700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6aa8100b50 0x7f6aa80fce70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.309+0000 7f6aafe5a700 1 --2- 172.21.15.104:0/21635496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aa8101510 0x7f6aa80fd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:57.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.309+0000 7f6aafe5a700 1 -- 172.21.15.104:0/21635496 >> 172.21.15.104:0/21635496 conn(0x7f6aa8077a00 msgr2=0x7f6aa80ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:57.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.310+0000 7f6aafe5a700 1 -- 172.21.15.104:0/21635496 shutdown_connections 2024-06-09T22:22:57.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:22:57.310+0000 7f6aafe5a700 1 -- 172.21.15.104:0/21635496 wait complete. 2024-06-09T22:22:57.324 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:57.715 INFO:tasks.workunit.client.0.smithi104.stdout: 1 9581 22.32 MB/sec warmup 3 sec latency 14.227 ms 2024-06-09T22:22:58.116 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:22:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.358+0000 7f8268612640 1 -- 172.21.15.104:0/3203343133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8260102350 msgr2=0x7f826010e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.358+0000 7f8268612640 1 --2- 172.21.15.104:0/3203343133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8260102350 0x7f826010e860 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f8258009cd0 tx=0x7f825802a8e0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.359+0000 7f8268612640 1 -- 172.21.15.104:0/3203343133 shutdown_connections 2024-06-09T22:22:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.359+0000 7f8268612640 1 --2- 172.21.15.104:0/3203343133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8260102350 0x7f826010e860 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.359+0000 7f8268612640 1 --2- 172.21.15.104:0/3203343133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8260101990 0x7f8260101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.359+0000 7f8268612640 1 --2- 172.21.15.104:0/3203343133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82601079a0 0x7f8260107d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.359+0000 7f8268612640 1 -- 172.21.15.104:0/3203343133 >> 172.21.15.104:0/3203343133 conn(0x7f82600fd460 msgr2=0x7f82600ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.359+0000 7f8268612640 1 -- 172.21.15.104:0/3203343133 shutdown_connections 2024-06-09T22:22:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.359+0000 7f8268612640 1 -- 172.21.15.104:0/3203343133 wait complete. 2024-06-09T22:22:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.359+0000 7f8268612640 1 Processor -- start 2024-06-09T22:22:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.359+0000 7f8268612640 1 -- start start 2024-06-09T22:22:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.359+0000 7f8268612640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8260101990 0x7f82601981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.359+0000 7f8268612640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8260102350 0x7f82601986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f8268612640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82601079a0 0x7f826019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f8268612640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8260111db0 con 0x7f8260102350 2024-06-09T22:22:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f8268612640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8260111c30 con 0x7f82601079a0 2024-06-09T22:22:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f8268612640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8260111f30 con 0x7f8260101990 2024-06-09T22:22:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f8265bad640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8260102350 0x7f82601986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f82663ae640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8260101990 0x7f82601981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f8265bad640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8260102350 0x7f82601986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38646/0 (socket says 172.21.15.104:38646) 2024-06-09T22:22:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f8265bad640 1 -- 172.21.15.104:0/1718102103 learned_addr learned my addr 172.21.15.104:0/1718102103 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f8266baf640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82601079a0 0x7f826019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f82663ae640 1 -- 172.21.15.104:0/1718102103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82601079a0 msgr2=0x7f826019ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f82663ae640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82601079a0 0x7f826019ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f82663ae640 1 -- 172.21.15.104:0/1718102103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8260102350 msgr2=0x7f82601986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f82663ae640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8260102350 0x7f82601986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f82663ae640 1 -- 172.21.15.104:0/1718102103 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826019d1a0 con 0x7f8260101990 2024-06-09T22:22:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f8265bad640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8260102350 0x7f82601986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f82663ae640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8260101990 0x7f82601981b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f825c00e9f0 tx=0x7f825c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f82577fe640 1 -- 172.21.15.104:0/1718102103 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c00ccc0 con 0x7f8260101990 2024-06-09T22:22:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f8268612640 1 -- 172.21.15.104:0/1718102103 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826019d490 con 0x7f8260101990 2024-06-09T22:22:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f82577fe640 1 -- 172.21.15.104:0/1718102103 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f825c010910 con 0x7f8260101990 2024-06-09T22:22:58.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f82577fe640 1 -- 172.21.15.104:0/1718102103 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c018960 con 0x7f8260101990 2024-06-09T22:22:58.364 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.360+0000 7f8268612640 1 -- 172.21.15.104:0/1718102103 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82601a4d20 con 0x7f8260101990 2024-06-09T22:22:58.368 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.363+0000 7f82577fe640 1 -- 172.21.15.104:0/1718102103 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f825c010ac0 con 0x7f8260101990 2024-06-09T22:22:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.363+0000 7f826b3f7740 1 Processor -- start 2024-06-09T22:22:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f826b3f7740 1 -- start start 2024-06-09T22:22:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f826b3f7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89d2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f826b3f7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89d2a90 0x5649a89dce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f826b3f7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89dd350 0x5649a89df740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f826b3f7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5649a89364f0 con 0x5649a89d2170 2024-06-09T22:22:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f826b3f7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5649a88d4870 con 0x5649a89dd350 2024-06-09T22:22:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f826b3f7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5649a89324b0 con 0x5649a89d2a90 2024-06-09T22:22:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f8266baf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89d2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f82663ae640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89d2a90 0x5649a89dce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f8266baf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89d2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38650/0 (socket says 172.21.15.104:38650) 2024-06-09T22:22:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f8266baf640 1 -- 172.21.15.104:0/1828807360 learned_addr learned my addr 172.21.15.104:0/1828807360 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f8265bad640 1 --2- 172.21.15.104:0/1828807360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89dd350 0x5649a89df740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f82577fe640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f824c0a2e20 0x7f824c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f8266baf640 1 -- 172.21.15.104:0/1828807360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89d2a90 msgr2=0x5649a89dce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f8266baf640 1 --2- 172.21.15.104:0/1828807360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89d2a90 0x5649a89dce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f8266baf640 1 -- 172.21.15.104:0/1828807360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89dd350 msgr2=0x5649a89df740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f8266baf640 1 --2- 172.21.15.104:0/1828807360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89dd350 0x5649a89df740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.364+0000 7f8266baf640 1 -- 172.21.15.104:0/1828807360 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649a88f09b0 con 0x5649a89d2170 2024-06-09T22:22:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.365+0000 7f82577fe640 1 -- 172.21.15.104:0/1718102103 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f825c0cfa50 con 0x7f8260101990 2024-06-09T22:22:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.365+0000 7f8266baf640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f824c0a2e20 0x7f824c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.365+0000 7f8266baf640 1 --2- 172.21.15.104:0/1828807360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89d2550 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f8258009cd0 tx=0x7f8258009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7775596594ac8a9e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.365+0000 7f8254bfe640 1 -- 172.21.15.104:0/1828807360 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258031a40 con 0x5649a89d2170 2024-06-09T22:22:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.365+0000 7f8254bfe640 1 -- 172.21.15.104:0/1828807360 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8258036070 con 0x5649a89d2170 2024-06-09T22:22:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.365+0000 7f8254bfe640 1 -- 172.21.15.104:0/1828807360 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258030be0 con 0x5649a89d2170 2024-06-09T22:22:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.366+0000 7f826b3f7740 1 -- 172.21.15.104:0/1828807360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 msgr2=0x5649a89d2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.366+0000 7f8266baf640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f824c0a2e20 0x7f824c0a52e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8258049000 tx=0x7f825804a000 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.366+0000 7f826b3f7740 1 --2- 172.21.15.104:0/1828807360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89d2550 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f8258009cd0 tx=0x7f8258009a20 comp rx=0 tx=0).stop 2024-06-09T22:22:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.366+0000 7f826b3f7740 1 -- 172.21.15.104:0/1828807360 shutdown_connections 2024-06-09T22:22:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.366+0000 7f826b3f7740 1 --2- 172.21.15.104:0/1828807360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89dd350 0x5649a89df740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.366+0000 7f826b3f7740 1 --2- 172.21.15.104:0/1828807360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89d2a90 0x5649a89dce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.366+0000 7f826b3f7740 1 --2- 172.21.15.104:0/1828807360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89d2550 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.366+0000 7f826b3f7740 1 -- 172.21.15.104:0/1828807360 >> 172.21.15.104:0/1828807360 conn(0x5649a89ce680 msgr2=0x5649a89cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.366+0000 7f826b3f7740 1 -- 172.21.15.104:0/1828807360 shutdown_connections 2024-06-09T22:22:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.366+0000 7f826b3f7740 1 -- 172.21.15.104:0/1828807360 wait complete. 2024-06-09T22:22:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.366+0000 7f826b3f7740 1 -- start start 2024-06-09T22:22:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f826b3f7740 1 Processor -- start 2024-06-09T22:22:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f826b3f7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89da350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f826b3f7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89d2a90 0x5649a89da890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f826b3f7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89dd350 0x5649a89dadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f826b3f7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5649a89dff80 con 0x5649a89d2170 2024-06-09T22:22:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f826b3f7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5649a88bfae0 con 0x5649a89d2a90 2024-06-09T22:22:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f826b3f7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5649a89e0100 con 0x5649a89dd350 2024-06-09T22:22:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f826b3f7740 10 client.0 _ll_register_callbacks cb 0x5649a88f89b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:58.373 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f82663ae640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89da350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f82663ae640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89da350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38662/0 (socket says 172.21.15.104:38662) 2024-06-09T22:22:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f82663ae640 1 -- 172.21.15.104:0/421359640 learned_addr learned my addr 172.21.15.104:0/421359640 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f8266baf640 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89dd350 0x5649a89dadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f8265bad640 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89d2a90 0x5649a89da890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f82663ae640 1 -- 172.21.15.104:0/421359640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89dd350 msgr2=0x5649a89dadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f82663ae640 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89dd350 0x5649a89dadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f82663ae640 1 -- 172.21.15.104:0/421359640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89d2a90 msgr2=0x5649a89da890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f82663ae640 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89d2a90 0x5649a89da890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f82663ae640 1 -- 172.21.15.104:0/421359640 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649a864a210 con 0x5649a89d2170 2024-06-09T22:22:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f82663ae640 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89da350 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f825c00e9c0 tx=0x7f825c005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c9f16018e407620 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f8249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:22:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.367+0000 7f8249ffb640 1 -- 172.21.15.104:0/421359640 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c09b610 con 0x5649a89d2170 2024-06-09T22:22:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f8249ffb640 1 -- 172.21.15.104:0/421359640 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f825c09b300 con 0x5649a89d2170 2024-06-09T22:22:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f826b3f7740 1 -- 172.21.15.104:0/421359640 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8258036070 con 0x5649a89d2170 2024-06-09T22:22:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f8249ffb640 1 -- 172.21.15.104:0/421359640 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f825c0cd2c0 con 0x5649a89d2170 2024-06-09T22:22:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f826b3f7740 10 client.16065 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f826b3f7740 1 -- 172.21.15.104:0/421359640 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5649a8716620 con 0x5649a89d2170 2024-06-09T22:22:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f8249ffb640 1 -- 172.21.15.104:0/421359640 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8258036070 con 0x5649a89d2170 2024-06-09T22:22:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f8249ffb640 1 -- 172.21.15.104:0/421359640 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f825c018b00 con 0x5649a89d2170 2024-06-09T22:22:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f826b3f7740 10 client.16065 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f826b3f7740 10 client.16065 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f826b3f7740 20 client.16065 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f826b3f7740 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5649a89ed580 0x5649a89ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f826b3f7740 4 client.16065 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:22:58.378 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f826b3f7740 1 -- 172.21.15.104:0/421359640 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649a8949f10 con 0x5649a89ed580 2024-06-09T22:22:58.378 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.368+0000 7f8265bad640 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5649a89ed580 0x5649a89ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.378 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.369+0000 7f8265bad640 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5649a89ed580 0x5649a89ef9b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=837ddbc5e9d9ea02 server_cookie=ecd56e552d2b28a3 in_seq=0 out_seq=0 2024-06-09T22:22:58.378 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.369+0000 7f8249ffb640 10 client.16065 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f8249ffb640 1 -- 172.21.15.104:0/421359640 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5649a8949f10 con 0x5649a89ed580 2024-06-09T22:22:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f8249ffb640 10 client.16065 handle_command_reply: tid=0 2024-06-09T22:22:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f826b3f7740 1 client.16065 shutdown 2024-06-09T22:22:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f826b3f7740 1 -- 172.21.15.104:0/421359640 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 msgr2=0x5649a89da350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f826b3f7740 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89da350 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f825c00e9c0 tx=0x7f825c005f10 comp rx=0 tx=0).stop 2024-06-09T22:22:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f826b3f7740 1 -- 172.21.15.104:0/421359640 shutdown_connections 2024-06-09T22:22:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f826b3f7740 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5649a89ed580 0x5649a89ef9b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f826b3f7740 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89dd350 0x5649a89dadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f826b3f7740 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89d2a90 0x5649a89da890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f826b3f7740 1 --2- 172.21.15.104:0/421359640 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89da350 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f8249ffb640 0 client.16065 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f826b3f7740 1 -- 172.21.15.104:0/421359640 >> 172.21.15.104:0/421359640 conn(0x5649a89ce680 msgr2=0x5649a89d43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f826b3f7740 1 -- 172.21.15.104:0/421359640 shutdown_connections 2024-06-09T22:22:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f826b3f7740 1 -- 172.21.15.104:0/421359640 wait complete. 2024-06-09T22:22:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.370+0000 7f826b3f7740 20 client.16065 trim_cache size 0 max 16384 2024-06-09T22:22:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.387+0000 7f826b3f7740 1 Processor -- start 2024-06-09T22:22:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.387+0000 7f826b3f7740 1 -- start start 2024-06-09T22:22:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.387+0000 7f826b3f7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89d4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.387+0000 7f826b3f7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89d2a90 0x5649a89f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.387+0000 7f826b3f7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89dd350 0x5649a89f5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.387+0000 7f826b3f7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f82601acba0 con 0x5649a89d2170 2024-06-09T22:22:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.387+0000 7f826b3f7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82601ae6e0 con 0x5649a89dd350 2024-06-09T22:22:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.387+0000 7f826b3f7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f82601ad270 con 0x5649a89d2a90 2024-06-09T22:22:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f82663ae640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89d4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f8265bad640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89d2a90 0x5649a89f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f82663ae640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89d4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38678/0 (socket says 172.21.15.104:38678) 2024-06-09T22:22:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f82663ae640 1 -- 172.21.15.104:0/4217032323 learned_addr learned my addr 172.21.15.104:0/4217032323 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f8266baf640 1 --2- 172.21.15.104:0/4217032323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89dd350 0x5649a89f5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f8265bad640 1 -- 172.21.15.104:0/4217032323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89dd350 msgr2=0x5649a89f5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f8265bad640 1 --2- 172.21.15.104:0/4217032323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89dd350 0x5649a89f5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f8265bad640 1 -- 172.21.15.104:0/4217032323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 msgr2=0x5649a89d4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f8265bad640 1 --2- 172.21.15.104:0/4217032323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89d4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f8265bad640 1 -- 172.21.15.104:0/4217032323 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649a89e91e0 con 0x5649a89d2a90 2024-06-09T22:22:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f82663ae640 1 --2- 172.21.15.104:0/4217032323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89d4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:22:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f8265bad640 1 --2- 172.21.15.104:0/4217032323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89d2a90 0x5649a89f52c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x5649a89cfaa0 tx=0x7f825000b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1daafb879223ad66 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.388+0000 7f824bfff640 1 -- 172.21.15.104:0/4217032323 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8250011640 con 0x5649a89d2a90 2024-06-09T22:22:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.389+0000 7f824bfff640 1 -- 172.21.15.104:0/4217032323 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f825001d8d0 con 0x5649a89d2a90 2024-06-09T22:22:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.389+0000 7f824bfff640 1 -- 172.21.15.104:0/4217032323 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8250010b00 con 0x5649a89d2a90 2024-06-09T22:22:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.389+0000 7f826b3f7740 1 -- 172.21.15.104:0/4217032323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89d2a90 msgr2=0x5649a89f52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.389+0000 7f826b3f7740 1 --2- 172.21.15.104:0/4217032323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89d2a90 0x5649a89f52c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x5649a89cfaa0 tx=0x7f825000b480 comp rx=0 tx=0).stop 2024-06-09T22:22:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.390+0000 7f826b3f7740 1 -- 172.21.15.104:0/4217032323 shutdown_connections 2024-06-09T22:22:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.390+0000 7f826b3f7740 1 --2- 172.21.15.104:0/4217032323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89dd350 0x5649a89f5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.390+0000 7f826b3f7740 1 --2- 172.21.15.104:0/4217032323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89d2a90 0x5649a89f52c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.390+0000 7f826b3f7740 1 --2- 172.21.15.104:0/4217032323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2170 0x5649a89d4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.390+0000 7f826b3f7740 1 -- 172.21.15.104:0/4217032323 >> 172.21.15.104:0/4217032323 conn(0x5649a89ce680 msgr2=0x5649a89d3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.390+0000 7f826b3f7740 1 -- 172.21.15.104:0/4217032323 shutdown_connections 2024-06-09T22:22:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.390+0000 7f826b3f7740 1 -- 172.21.15.104:0/4217032323 wait complete. 2024-06-09T22:22:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.390+0000 7f826b3f7740 1 -- start start 2024-06-09T22:22:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.390+0000 7f826b3f7740 1 Processor -- start 2024-06-09T22:22:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f826b3f7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89d2170 0x5649a89e1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f826b3f7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2a90 0x5649a89e17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f826b3f7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89dd350 0x5649a89e1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f826b3f7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f82601ae010 con 0x5649a89d2a90 2024-06-09T22:22:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f826b3f7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82601ad940 con 0x5649a89d2170 2024-06-09T22:22:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f826b3f7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f82601aab60 con 0x5649a89dd350 2024-06-09T22:22:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f826b3f7740 10 client.0 _ll_register_callbacks cb 0x7f825c00ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:22:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f82663ae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89d2170 0x5649a89e1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f8265bad640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2a90 0x5649a89e17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f8266baf640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89dd350 0x5649a89e1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f8265bad640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2a90 0x5649a89e17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38682/0 (socket says 172.21.15.104:38682) 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f8265bad640 1 -- 172.21.15.104:0/1847841958 learned_addr learned my addr 172.21.15.104:0/1847841958 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f82663ae640 1 --2- 172.21.15.104:0/1847841958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89d2170 0x5649a89e1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:43242/0 (socket says 172.21.15.104:43242) 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f8266baf640 1 -- 172.21.15.104:0/1847841958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89d2170 msgr2=0x5649a89e1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f8266baf640 1 --2- 172.21.15.104:0/1847841958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89d2170 0x5649a89e1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f8266baf640 1 -- 172.21.15.104:0/1847841958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2a90 msgr2=0x5649a89e17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f8266baf640 1 --2- 172.21.15.104:0/1847841958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2a90 0x5649a89e17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f8266baf640 1 -- 172.21.15.104:0/1847841958 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649a8a573f0 con 0x5649a89dd350 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f8266baf640 1 --2- 172.21.15.104:0/1847841958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89dd350 0x5649a89e1cf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f825802ae00 tx=0x7f8258050000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8757839a67a67088 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f8249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.391+0000 7f8249ffb640 1 -- 172.21.15.104:0/1847841958 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258062060 con 0x5649a89dd350 2024-06-09T22:22:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f826b3f7740 1 -- 172.21.15.104:0/1847841958 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f825001d8d0 con 0x5649a89dd350 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f8249ffb640 1 -- 172.21.15.104:0/1847841958 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f82580046a0 con 0x5649a89dd350 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f8249ffb640 1 -- 172.21.15.104:0/1847841958 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8258069050 con 0x5649a89dd350 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f826b3f7740 10 client.25615 fetch_fsmap learned FSMap version 29 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f826b3f7740 1 -- 172.21.15.104:0/1847841958 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5649a8a586e0 con 0x5649a89dd350 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f8249ffb640 1 -- 172.21.15.104:0/1847841958 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f825001d8d0 con 0x5649a89dd350 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f8249ffb640 1 -- 172.21.15.104:0/1847841958 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f825805c070 con 0x5649a89dd350 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f826b3f7740 10 client.25615 fetch_fsmap finished waiting for FSMap version 29 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f826b3f7740 10 client.25615 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f826b3f7740 20 client.25615 populate_metadata read hostname 'smithi104' 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f826b3f7740 1 --2- 172.21.15.104:0/1847841958 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5649a8a58960 0x5649a8a5ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f826b3f7740 4 client.25615 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f826b3f7740 1 -- 172.21.15.104:0/1847841958 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5649a8943c20 con 0x5649a8a58960 2024-06-09T22:22:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.392+0000 7f82663ae640 1 --2- 172.21.15.104:0/1847841958 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5649a8a58960 0x5649a8a5ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:22:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.393+0000 7f82663ae640 1 --2- 172.21.15.104:0/1847841958 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5649a8a58960 0x5649a8a5ad50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c90c5c7cf0ea11d9 server_cookie=6cb4c46c6cc6049d in_seq=0 out_seq=0 2024-06-09T22:22:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.393+0000 7f8249ffb640 10 client.25615 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f8249ffb640 1 -- 172.21.15.104:0/1847841958 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5649a8943c20 con 0x5649a8a58960 2024-06-09T22:22:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f8249ffb640 10 client.25615 handle_command_reply: tid=0 2024-06-09T22:22:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f826b3f7740 1 client.25615 shutdown 2024-06-09T22:22:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f826b3f7740 1 -- 172.21.15.104:0/1847841958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89dd350 msgr2=0x5649a89e1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f826b3f7740 1 --2- 172.21.15.104:0/1847841958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89dd350 0x5649a89e1cf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f825802ae00 tx=0x7f8258050000 comp rx=0 tx=0).stop 2024-06-09T22:22:58.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f826b3f7740 1 -- 172.21.15.104:0/1847841958 shutdown_connections 2024-06-09T22:22:58.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f826b3f7740 1 --2- 172.21.15.104:0/1847841958 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5649a8a58960 0x5649a8a5ad50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f826b3f7740 1 --2- 172.21.15.104:0/1847841958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649a89dd350 0x5649a89e1cf0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f8249ffb640 0 client.25615 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:22:58.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f826b3f7740 1 --2- 172.21.15.104:0/1847841958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5649a89d2a90 0x5649a89e17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f826b3f7740 1 --2- 172.21.15.104:0/1847841958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5649a89d2170 0x5649a89e1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f826b3f7740 1 -- 172.21.15.104:0/1847841958 >> 172.21.15.104:0/1847841958 conn(0x5649a89ce680 msgr2=0x5649a89d43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:58.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f826b3f7740 1 -- 172.21.15.104:0/1847841958 shutdown_connections 2024-06-09T22:22:58.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f826b3f7740 1 -- 172.21.15.104:0/1847841958 wait complete. 2024-06-09T22:22:58.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.534+0000 7f826b3f7740 20 client.25615 trim_cache size 0 max 16384 2024-06-09T22:22:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:22:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:22:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:22:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.535+0000 7f8268612640 1 -- 172.21.15.104:0/1718102103 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f824c0a2e20 msgr2=0x7f824c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.535+0000 7f8268612640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f824c0a2e20 0x7f824c0a52e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8258049000 tx=0x7f825804a000 comp rx=0 tx=0).stop 2024-06-09T22:22:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.535+0000 7f8268612640 1 -- 172.21.15.104:0/1718102103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8260101990 msgr2=0x7f82601981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:22:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.535+0000 7f8268612640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8260101990 0x7f82601981b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f825c00e9f0 tx=0x7f825c00eeb0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.535+0000 7f8268612640 1 -- 172.21.15.104:0/1718102103 shutdown_connections 2024-06-09T22:22:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.535+0000 7f8268612640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f824c0a2e20 0x7f824c0a52e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.535+0000 7f8268612640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82601079a0 0x7f826019ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.535+0000 7f8268612640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8260102350 0x7f82601986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.535+0000 7f8268612640 1 --2- 172.21.15.104:0/1718102103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8260101990 0x7f82601981b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:22:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.535+0000 7f8268612640 1 -- 172.21.15.104:0/1718102103 >> 172.21.15.104:0/1718102103 conn(0x7f82600fd460 msgr2=0x7f82600fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:22:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.535+0000 7f8268612640 1 -- 172.21.15.104:0/1718102103 shutdown_connections 2024-06-09T22:22:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:22:58.535+0000 7f8268612640 1 -- 172.21.15.104:0/1718102103 wait complete. 2024-06-09T22:22:58.715 INFO:tasks.workunit.client.0.smithi104.stdout: 1 12102 20.84 MB/sec warmup 4 sec latency 53.682 ms 2024-06-09T22:22:58.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:58 smithi153 ceph-mon[36599]: scrub queued for path: / 2024-06-09T22:22:58.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:58 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:22:58.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:58 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:22:58.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:58 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:22:58.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:22:58 smithi153 ceph-mon[36599]: pgmap v63: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-06-09T22:22:59.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:58 smithi104 ceph-mon[32389]: scrub queued for path: / 2024-06-09T22:22:59.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:58 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:22:59.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:58 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:22:59.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:58 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:22:59.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:22:58 smithi104 ceph-mon[32389]: pgmap v63: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-06-09T22:22:59.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:58 smithi194 ceph-mon[36856]: scrub queued for path: / 2024-06-09T22:22:59.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:58 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:22:59.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:58 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:22:59.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:58 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:22:59.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:22:58 smithi194 ceph-mon[36856]: pgmap v63: 129 pgs: 129 active+clean; 1.4 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-06-09T22:22:59.459 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:22:59.715 INFO:tasks.workunit.client.0.smithi104.stdout: 1 14985 19.57 MB/sec warmup 5 sec latency 15.687 ms 2024-06-09T22:23:00.715 INFO:tasks.workunit.client.0.smithi104.stdout: 1 17911 19.22 MB/sec warmup 6 sec latency 40.890 ms 2024-06-09T22:23:00.976 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:00 smithi104 ceph-mon[32389]: pgmap v64: 129 pgs: 129 active+clean; 7.6 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 641 KiB/s wr, 181 op/s 2024-06-09T22:23:00.976 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:00 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:23:00.976 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:00 smithi104 ceph-mon[32389]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:23:00.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:00 smithi153 ceph-mon[36599]: pgmap v64: 129 pgs: 129 active+clean; 7.6 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 641 KiB/s wr, 181 op/s 2024-06-09T22:23:00.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:00 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:23:00.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:00 smithi153 ceph-mon[36599]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:23:01.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:00 smithi194 ceph-mon[36856]: pgmap v64: 129 pgs: 129 active+clean; 7.6 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 641 KiB/s wr, 181 op/s 2024-06-09T22:23:01.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:00 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:23:01.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:00 smithi194 ceph-mon[36856]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:23:01.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.535+0000 7f58b05bb700 1 -- 172.21.15.104:0/2442851924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a81073f0 msgr2=0x7f58a81077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:01.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.535+0000 7f58b05bb700 1 --2- 172.21.15.104:0/2442851924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a81073f0 0x7f58a81077d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5898009990 tx=0x7f5898009ca0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.535+0000 7f58b05bb700 1 -- 172.21.15.104:0/2442851924 shutdown_connections 2024-06-09T22:23:01.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.535+0000 7f58b05bb700 1 --2- 172.21.15.104:0/2442851924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a81073f0 0x7f58a81077d0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.535+0000 7f58b05bb700 1 --2- 172.21.15.104:0/2442851924 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a8101d40 0x7f58a810e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.535+0000 7f58b05bb700 1 --2- 172.21.15.104:0/2442851924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a8101380 0x7f58a8101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.535+0000 7f58b05bb700 1 -- 172.21.15.104:0/2442851924 >> 172.21.15.104:0/2442851924 conn(0x7f58a8078300 msgr2=0x7f58a8078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:01.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.535+0000 7f58b05bb700 1 -- 172.21.15.104:0/2442851924 shutdown_connections 2024-06-09T22:23:01.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.535+0000 7f58b05bb700 1 -- 172.21.15.104:0/2442851924 wait complete. 2024-06-09T22:23:01.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.536+0000 7f58b05bb700 1 Processor -- start 2024-06-09T22:23:01.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.536+0000 7f58b05bb700 1 -- start start 2024-06-09T22:23:01.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.536+0000 7f58b05bb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a8101380 0x7f58a8199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.536+0000 7f58b05bb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a8101d40 0x7f58a819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.536+0000 7f58b05bb700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a81073f0 0x7f58a81a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.536+0000 7f58b05bb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f58a8111690 con 0x7f58a8101d40 2024-06-09T22:23:01.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.536+0000 7f58b05bb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58a8111510 con 0x7f58a8101380 2024-06-09T22:23:01.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.536+0000 7f58b05bb700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f58a8111810 con 0x7f58a81073f0 2024-06-09T22:23:01.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58adb56700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a8101d40 0x7f58a819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58adb56700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a8101d40 0x7f58a819a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38690/0 (socket says 172.21.15.104:38690) 2024-06-09T22:23:01.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58adb56700 1 -- 172.21.15.104:0/3382597906 learned_addr learned my addr 172.21.15.104:0/3382597906 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:01.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58ae357700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a8101380 0x7f58a8199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58aeb58700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a81073f0 0x7f58a81a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58ae357700 1 -- 172.21.15.104:0/3382597906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a81073f0 msgr2=0x7f58a81a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:01.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58ae357700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a81073f0 0x7f58a81a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58ae357700 1 -- 172.21.15.104:0/3382597906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a8101d40 msgr2=0x7f58a819a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:01.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58ae357700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a8101d40 0x7f58a819a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58ae357700 1 -- 172.21.15.104:0/3382597906 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a81a5090 con 0x7f58a8101380 2024-06-09T22:23:01.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58adb56700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a8101d40 0x7f58a819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:01.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58aeb58700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a81073f0 0x7f58a81a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:01.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58ae357700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a8101380 0x7f58a8199ad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f58980059d0 tx=0x7f589800bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:01.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f589f7fe700 1 -- 172.21.15.104:0/3382597906 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898027070 con 0x7f58a8101380 2024-06-09T22:23:01.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f58b05bb700 1 -- 172.21.15.104:0/3382597906 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a81a5260 con 0x7f58a8101380 2024-06-09T22:23:01.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.537+0000 7f589f7fe700 1 -- 172.21.15.104:0/3382597906 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5898003d70 con 0x7f58a8101380 2024-06-09T22:23:01.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.538+0000 7f589f7fe700 1 -- 172.21.15.104:0/3382597906 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898022e30 con 0x7f58a8101380 2024-06-09T22:23:01.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.538+0000 7f58b05bb700 1 -- 172.21.15.104:0/3382597906 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a81a5710 con 0x7f58a8101380 2024-06-09T22:23:01.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.540+0000 7f589f7fe700 1 -- 172.21.15.104:0/3382597906 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f589802a070 con 0x7f58a8101380 2024-06-09T22:23:01.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.541+0000 7f589f7fe700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f58940a2d90 0x7f58940a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.541+0000 7f58adb56700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f58940a2d90 0x7f58940a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.541+0000 7f589f7fe700 1 -- 172.21.15.104:0/3382597906 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f58980d04d0 con 0x7f58a8101380 2024-06-09T22:23:01.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.541+0000 7f58bae3eb80 1 Processor -- start 2024-06-09T22:23:01.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.541+0000 7f58adb56700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f58940a2d90 0x7f58940a5250 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f58a8078920 tx=0x7f58a4009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:01.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58bae3eb80 1 -- start start 2024-06-09T22:23:01.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58bae3eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251fdf60 0x559b251fe340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58bae3eb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251fe880 0x559b251f8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58bae3eb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251f87a0 0x559b251f8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58bae3eb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559b250eaf60 con 0x559b251f87a0 2024-06-09T22:23:01.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58bae3eb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559b25149850 con 0x559b251fe880 2024-06-09T22:23:01.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58bae3eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559b250b1460 con 0x559b251fdf60 2024-06-09T22:23:01.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58aeb58700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251f87a0 0x559b251f8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58ae357700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251fdf60 0x559b251fe340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58adb56700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251fe880 0x559b251f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58aeb58700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251f87a0 0x559b251f8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38694/0 (socket says 172.21.15.104:38694) 2024-06-09T22:23:01.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58ae357700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251fdf60 0x559b251fe340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:41656/0 (socket says 172.21.15.104:41656) 2024-06-09T22:23:01.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58aeb58700 1 -- 172.21.15.104:0/3218998662 learned_addr learned my addr 172.21.15.104:0/3218998662 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:01.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58aeb58700 1 -- 172.21.15.104:0/3218998662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251fdf60 msgr2=0x559b251fe340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:01.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58aeb58700 1 --2- 172.21.15.104:0/3218998662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251fdf60 0x559b251fe340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58aeb58700 1 -- 172.21.15.104:0/3218998662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251fe880 msgr2=0x559b251f8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:01.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58aeb58700 1 --2- 172.21.15.104:0/3218998662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251fe880 0x559b251f8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58aeb58700 1 -- 172.21.15.104:0/3218998662 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b250988d0 con 0x559b251f87a0 2024-06-09T22:23:01.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58ae357700 1 --2- 172.21.15.104:0/3218998662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251fdf60 0x559b251fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:01.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.542+0000 7f58adb56700 1 --2- 172.21.15.104:0/3218998662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251fe880 0x559b251f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:01.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.543+0000 7f58aeb58700 1 --2- 172.21.15.104:0/3218998662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251f87a0 0x559b251f8bc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f58a0003eb0 tx=0x7f58a0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5809f1088b9f9a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:01.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.543+0000 7f589cff9700 1 -- 172.21.15.104:0/3218998662 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a0019020 con 0x559b251f87a0 2024-06-09T22:23:01.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.543+0000 7f589cff9700 1 -- 172.21.15.104:0/3218998662 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58a00092d0 con 0x559b251f87a0 2024-06-09T22:23:01.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.543+0000 7f589cff9700 1 -- 172.21.15.104:0/3218998662 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a000e870 con 0x559b251f87a0 2024-06-09T22:23:01.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3218998662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251f87a0 msgr2=0x559b251f8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:01.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/3218998662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251f87a0 0x559b251f8bc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f58a0003eb0 tx=0x7f58a0009920 comp rx=0 tx=0).stop 2024-06-09T22:23:01.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3218998662 shutdown_connections 2024-06-09T22:23:01.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/3218998662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251f87a0 0x559b251f8bc0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/3218998662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251fdf60 0x559b251fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/3218998662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251fe880 0x559b251f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3218998662 >> 172.21.15.104:0/3218998662 conn(0x559b25125080 msgr2=0x559b251f3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:01.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3218998662 shutdown_connections 2024-06-09T22:23:01.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3218998662 wait complete. 2024-06-09T22:23:01.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 -- start start 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 Processor -- start 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251f87a0 0x559b25204500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251fdf60 0x559b25201800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.544+0000 7f58bae3eb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fe880 0x559b25201d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58bae3eb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559b250e9fc0 con 0x559b251fe880 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58bae3eb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559b25145810 con 0x559b251fdf60 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58bae3eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559b25205260 con 0x559b251f87a0 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58bae3eb80 10 client.0 _ll_register_callbacks cb 0x559b250df5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58ae357700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251f87a0 0x559b25204500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58aeb58700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fe880 0x559b25201d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58ae357700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251f87a0 0x559b25204500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:41664/0 (socket says 172.21.15.104:41664) 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58aeb58700 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fe880 0x559b25201d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38696/0 (socket says 172.21.15.104:38696) 2024-06-09T22:23:01.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58adb56700 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251fdf60 0x559b25201800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58ae357700 1 -- 172.21.15.104:0/1839649625 learned_addr learned my addr 172.21.15.104:0/1839649625 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:01.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58ae357700 1 -- 172.21.15.104:0/1839649625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251fdf60 msgr2=0x559b25201800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:01.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58ae357700 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251fdf60 0x559b25201800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58ae357700 1 -- 172.21.15.104:0/1839649625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fe880 msgr2=0x559b25201d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:01.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58ae357700 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fe880 0x559b25201d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58ae357700 1 -- 172.21.15.104:0/1839649625 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b2510f750 con 0x559b251f87a0 2024-06-09T22:23:01.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58ae357700 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251f87a0 0x559b25204500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5898099120 tx=0x7f589809c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e14546699138878 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:01.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f5891ffb700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:23:01.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f5891ffb700 1 -- 172.21.15.104:0/1839649625 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589809c0b0 con 0x559b251f87a0 2024-06-09T22:23:01.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f58bae3eb80 1 -- 172.21.15.104:0/1839649625 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58a00092d0 con 0x559b251f87a0 2024-06-09T22:23:01.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.545+0000 7f5891ffb700 1 -- 172.21.15.104:0/1839649625 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f58980a4ea0 con 0x559b251f87a0 2024-06-09T22:23:01.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.546+0000 7f5891ffb700 1 -- 172.21.15.104:0/1839649625 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f58980095d0 con 0x559b251f87a0 2024-06-09T22:23:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.546+0000 7f58bae3eb80 10 client.25627 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.546+0000 7f58bae3eb80 1 -- 172.21.15.104:0/1839649625 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559b2520f970 con 0x559b251f87a0 2024-06-09T22:23:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.546+0000 7f5891ffb700 1 -- 172.21.15.104:0/1839649625 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58a00092d0 con 0x559b251f87a0 2024-06-09T22:23:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.546+0000 7f5891ffb700 1 -- 172.21.15.104:0/1839649625 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f589809bba0 con 0x559b251f87a0 2024-06-09T22:23:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.546+0000 7f58bae3eb80 10 client.25627 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.546+0000 7f58bae3eb80 10 client.25627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.546+0000 7f58bae3eb80 20 client.25627 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.546+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559b25211f90 0x559b252143d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.546+0000 7f58bae3eb80 4 client.25627 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.547+0000 7f58bae3eb80 1 -- 172.21.15.104:0/1839649625 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b25156d50 con 0x559b25211f90 2024-06-09T22:23:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.547+0000 7f58adb56700 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559b25211f90 0x559b252143d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.547+0000 7f58adb56700 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559b25211f90 0x559b252143d0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f006f8fa736ff041 server_cookie=3b605ac3f84d48dc in_seq=0 out_seq=0 2024-06-09T22:23:01.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.547+0000 7f5891ffb700 10 client.25627 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:01.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f5891ffb700 1 -- 172.21.15.104:0/1839649625 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x559b25156d50 con 0x559b25211f90 2024-06-09T22:23:01.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f5891ffb700 10 client.25627 handle_command_reply: tid=0 2024-06-09T22:23:01.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f58bae3eb80 1 client.25627 shutdown 2024-06-09T22:23:01.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f58bae3eb80 1 -- 172.21.15.104:0/1839649625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251f87a0 msgr2=0x559b25204500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:01.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251f87a0 0x559b25204500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5898099120 tx=0x7f589809c5f0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f58bae3eb80 1 -- 172.21.15.104:0/1839649625 shutdown_connections 2024-06-09T22:23:01.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559b25211f90 0x559b252143d0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fe880 0x559b25201d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b251f87a0 0x559b25204500 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/1839649625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251fdf60 0x559b25201800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f58bae3eb80 1 -- 172.21.15.104:0/1839649625 >> 172.21.15.104:0/1839649625 conn(0x559b25125080 msgr2=0x559b251f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:01.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f5891ffb700 0 client.25627 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:01.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f58bae3eb80 1 -- 172.21.15.104:0/1839649625 shutdown_connections 2024-06-09T22:23:01.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f58bae3eb80 1 -- 172.21.15.104:0/1839649625 wait complete. 2024-06-09T22:23:01.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.548+0000 7f58bae3eb80 20 client.25627 trim_cache size 0 max 16384 2024-06-09T22:23:01.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.576+0000 7f58bae3eb80 1 Processor -- start 2024-06-09T22:23:01.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.576+0000 7f58bae3eb80 1 -- start start 2024-06-09T22:23:01.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.577+0000 7f58bae3eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b25125080 0x559b251f8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.577+0000 7f58bae3eb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 0x559b251f2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.577+0000 7f58bae3eb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fdf60 0x559b251f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.577+0000 7f58bae3eb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f58a81ae420 con 0x559b251fdf60 2024-06-09T22:23:01.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.577+0000 7f58bae3eb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58a81affc0 con 0x559b251f87a0 2024-06-09T22:23:01.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.577+0000 7f58bae3eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f58a81aeaf0 con 0x559b25125080 2024-06-09T22:23:01.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.577+0000 7f58adb56700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 0x559b251f2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.577+0000 7f58adb56700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 0x559b251f2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:43290/0 (socket says 172.21.15.104:43290) 2024-06-09T22:23:01.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.577+0000 7f58adb56700 1 -- 172.21.15.104:0/622807240 learned_addr learned my addr 172.21.15.104:0/622807240 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:01.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.578+0000 7f58aeb58700 1 --2- 172.21.15.104:0/622807240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fdf60 0x559b251f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.578+0000 7f58ae357700 1 --2- 172.21.15.104:0/622807240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b25125080 0x559b251f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.578+0000 7f58adb56700 1 -- 172.21.15.104:0/622807240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b25125080 msgr2=0x559b251f8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:01.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.578+0000 7f58adb56700 1 --2- 172.21.15.104:0/622807240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b25125080 0x559b251f8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.578+0000 7f58adb56700 1 -- 172.21.15.104:0/622807240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fdf60 msgr2=0x559b251f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:01.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.578+0000 7f58adb56700 1 --2- 172.21.15.104:0/622807240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fdf60 0x559b251f33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.578+0000 7f58adb56700 1 -- 172.21.15.104:0/622807240 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b251ffd70 con 0x559b251f87a0 2024-06-09T22:23:01.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.578+0000 7f58aeb58700 1 --2- 172.21.15.104:0/622807240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fdf60 0x559b251f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:01.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.578+0000 7f58ae357700 1 --2- 172.21.15.104:0/622807240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b25125080 0x559b251f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:23:01.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.579+0000 7f58adb56700 1 --2- 172.21.15.104:0/622807240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 0x559b251f2e80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f58a400e460 tx=0x7f58a4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae90e986a0b63 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:01.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.579+0000 7f5893fff700 1 -- 172.21.15.104:0/622807240 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a4004830 con 0x559b251f87a0 2024-06-09T22:23:01.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.579+0000 7f5893fff700 1 -- 172.21.15.104:0/622807240 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58a401d7f0 con 0x559b251f87a0 2024-06-09T22:23:01.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.579+0000 7f5893fff700 1 -- 172.21.15.104:0/622807240 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a4005280 con 0x559b251f87a0 2024-06-09T22:23:01.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.580+0000 7f58bae3eb80 1 -- 172.21.15.104:0/622807240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 msgr2=0x559b251f2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:01.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.580+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/622807240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 0x559b251f2e80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f58a400e460 tx=0x7f58a4014000 comp rx=0 tx=0).stop 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.580+0000 7f58bae3eb80 1 -- 172.21.15.104:0/622807240 shutdown_connections 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.580+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/622807240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fdf60 0x559b251f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.580+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/622807240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b25125080 0x559b251f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.580+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/622807240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 0x559b251f2e80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.580+0000 7f58bae3eb80 1 -- 172.21.15.104:0/622807240 >> 172.21.15.104:0/622807240 conn(0x559b251fbd90 msgr2=0x559b252038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.581+0000 7f58bae3eb80 1 -- 172.21.15.104:0/622807240 shutdown_connections 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.581+0000 7f58bae3eb80 1 -- 172.21.15.104:0/622807240 wait complete. 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.581+0000 7f58bae3eb80 1 -- start start 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.581+0000 7f58bae3eb80 1 Processor -- start 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.582+0000 7f58bae3eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b25125080 0x559b2520a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.582+0000 7f58bae3eb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 0x559b2520aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.582+0000 7f58bae3eb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fdf60 0x559b25207550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.582+0000 7f58bae3eb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f58a81af890 con 0x559b251fdf60 2024-06-09T22:23:01.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.582+0000 7f58bae3eb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58a81af1c0 con 0x559b251f87a0 2024-06-09T22:23:01.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.582+0000 7f58bae3eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f58a81ac3a0 con 0x559b25125080 2024-06-09T22:23:01.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.582+0000 7f58bae3eb80 10 client.0 _ll_register_callbacks cb 0x559b250df5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:01.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.583+0000 7f58adb56700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 0x559b2520aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.583+0000 7f58adb56700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 0x559b2520aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:43302/0 (socket says 172.21.15.104:43302) 2024-06-09T22:23:01.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.583+0000 7f58adb56700 1 -- 172.21.15.104:0/3724571474 learned_addr learned my addr 172.21.15.104:0/3724571474 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:01.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.583+0000 7f58adb56700 1 -- 172.21.15.104:0/3724571474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b25125080 msgr2=0x559b2520a580 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-09T22:23:01.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.583+0000 7f58adb56700 1 --2- 172.21.15.104:0/3724571474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b25125080 0x559b2520a580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.583+0000 7f58adb56700 1 -- 172.21.15.104:0/3724571474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fdf60 msgr2=0x559b25207550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:01.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.583+0000 7f58adb56700 1 --2- 172.21.15.104:0/3724571474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fdf60 0x559b25207550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.583+0000 7f58adb56700 1 -- 172.21.15.104:0/3724571474 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b25207a90 con 0x559b251f87a0 2024-06-09T22:23:01.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.583+0000 7f58adb56700 1 --2- 172.21.15.104:0/3724571474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 0x559b2520aac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009cf0 tx=0x7f58a400e880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed593d41c627ff07 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:01.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.583+0000 7f5891ffb700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:23:01.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.584+0000 7f5891ffb700 1 -- 172.21.15.104:0/3724571474 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a40046b0 con 0x559b251f87a0 2024-06-09T22:23:01.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.584+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3724571474 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58a401d7f0 con 0x559b251f87a0 2024-06-09T22:23:01.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.584+0000 7f5891ffb700 1 -- 172.21.15.104:0/3724571474 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f58a4004850 con 0x559b251f87a0 2024-06-09T22:23:01.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.584+0000 7f5891ffb700 1 -- 172.21.15.104:0/3724571474 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f58a4047050 con 0x559b251f87a0 2024-06-09T22:23:01.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.584+0000 7f58bae3eb80 10 client.25517 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:01.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.584+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3724571474 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559b2527d450 con 0x559b251f87a0 2024-06-09T22:23:01.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.585+0000 7f5891ffb700 1 -- 172.21.15.104:0/3724571474 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58a401d7f0 con 0x559b251f87a0 2024-06-09T22:23:01.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.585+0000 7f5891ffb700 1 -- 172.21.15.104:0/3724571474 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f58a40394a0 con 0x559b251f87a0 2024-06-09T22:23:01.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.585+0000 7f58bae3eb80 10 client.25517 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:01.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.585+0000 7f58bae3eb80 10 client.25517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:01.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.585+0000 7f58bae3eb80 20 client.25517 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:01.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.585+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/3724571474 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559b251fe880 0x559b2527fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:01.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.585+0000 7f58bae3eb80 4 client.25517 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:23:01.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.585+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3724571474 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559b251512d0 con 0x559b251fe880 2024-06-09T22:23:01.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.585+0000 7f58ae357700 1 --2- 172.21.15.104:0/3724571474 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559b251fe880 0x559b2527fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:01.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f5891ffb700 10 client.25517 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58ae357700 1 --2- 172.21.15.104:0/3724571474 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559b251fe880 0x559b2527fb80 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=547e9e02b7769323 server_cookie=c159c4fad3c276ba in_seq=0 out_seq=0 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f5891ffb700 1 -- 172.21.15.104:0/3724571474 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559b251512d0 con 0x559b251fe880 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f5891ffb700 10 client.25517 handle_command_reply: tid=0 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58bae3eb80 1 client.25517 shutdown 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3724571474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 msgr2=0x559b2520aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/3724571474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 0x559b2520aac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009cf0 tx=0x7f58a400e880 comp rx=0 tx=0).stop 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3724571474 shutdown_connections 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/3724571474 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559b251fe880 0x559b2527fb80 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/3724571474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559b251fdf60 0x559b25207550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/3724571474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b25125080 0x559b2520a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58bae3eb80 1 --2- 172.21.15.104:0/3724571474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559b251f87a0 0x559b2520aac0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3724571474 >> 172.21.15.104:0/3724571474 conn(0x559b251fbd90 msgr2=0x559b25203030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:01.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f5891ffb700 0 client.25517 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:01.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3724571474 shutdown_connections 2024-06-09T22:23:01.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58bae3eb80 1 -- 172.21.15.104:0/3724571474 wait complete. 2024-06-09T22:23:01.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.586+0000 7f58bae3eb80 20 client.25517 trim_cache size 0 max 16384 2024-06-09T22:23:01.591 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:23:01.591 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "no active scrubs running", 2024-06-09T22:23:01.591 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": {} 2024-06-09T22:23:01.591 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:23:01.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.587+0000 7f58b05bb700 1 -- 172.21.15.104:0/3382597906 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f58940a2d90 msgr2=0x7f58940a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:01.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.587+0000 7f58b05bb700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f58940a2d90 0x7f58940a5250 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f58a8078920 tx=0x7f58a4009260 comp rx=0 tx=0).stop 2024-06-09T22:23:01.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.587+0000 7f58b05bb700 1 -- 172.21.15.104:0/3382597906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a8101380 msgr2=0x7f58a8199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:01.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.587+0000 7f58b05bb700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a8101380 0x7f58a8199ad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f58980059d0 tx=0x7f589800bab0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.587+0000 7f58b05bb700 1 -- 172.21.15.104:0/3382597906 shutdown_connections 2024-06-09T22:23:01.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.587+0000 7f58b05bb700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f58940a2d90 0x7f58940a5250 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.587+0000 7f58b05bb700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58a81073f0 0x7f58a81a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.587+0000 7f58b05bb700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a8101380 0x7f58a8199ad0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.587+0000 7f58b05bb700 1 --2- 172.21.15.104:0/3382597906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a8101d40 0x7f58a819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:01.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.587+0000 7f58b05bb700 1 -- 172.21.15.104:0/3382597906 >> 172.21.15.104:0/3382597906 conn(0x7f58a8078300 msgr2=0x7f58a8100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:01.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.588+0000 7f58b05bb700 1 -- 172.21.15.104:0/3382597906 shutdown_connections 2024-06-09T22:23:01.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:01.588+0000 7f58b05bb700 1 -- 172.21.15.104:0/3382597906 wait complete. 2024-06-09T22:23:01.715 INFO:tasks.workunit.client.0.smithi104.stdout: 1 18363 16.65 MB/sec warmup 7 sec latency 51.460 ms 2024-06-09T22:23:02.715 INFO:tasks.workunit.client.0.smithi104.stdout: 1 18466 14.60 MB/sec warmup 8 sec latency 937.257 ms 2024-06-09T22:23:02.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:02 smithi153 ceph-mon[36599]: pgmap v65: 129 pgs: 129 active+clean; 7.6 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 641 KiB/s wr, 177 op/s 2024-06-09T22:23:03.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:02 smithi104 ceph-mon[32389]: pgmap v65: 129 pgs: 129 active+clean; 7.6 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 641 KiB/s wr, 177 op/s 2024-06-09T22:23:03.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:02 smithi194 ceph-mon[36856]: pgmap v65: 129 pgs: 129 active+clean; 7.6 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 641 KiB/s wr, 177 op/s 2024-06-09T22:23:03.716 INFO:tasks.workunit.client.0.smithi104.stdout: 1 18466 12.98 MB/sec warmup 9 sec latency 1937.350 ms 2024-06-09T22:23:04.716 INFO:tasks.workunit.client.0.smithi104.stdout: 1 18466 11.68 MB/sec warmup 10 sec latency 2937.395 ms 2024-06-09T22:23:04.737 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-09T22:23:04.737 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:23:04.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 -- 172.21.15.104:0/2055392561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13cc06a170 msgr2=0x7f13cc06a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:04.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 --2- 172.21.15.104:0/2055392561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13cc06a170 0x7f13cc06a5f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009990 tx=0x7f13b4009ca0 comp rx=0 tx=0).stop 2024-06-09T22:23:04.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 -- 172.21.15.104:0/2055392561 shutdown_connections 2024-06-09T22:23:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 --2- 172.21.15.104:0/2055392561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc06ab30 0x7f13cc10f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 --2- 172.21.15.104:0/2055392561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13cc108500 0x7f13cc1088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 --2- 172.21.15.104:0/2055392561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13cc06a170 0x7f13cc06a5f0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 -- 172.21.15.104:0/2055392561 >> 172.21.15.104:0/2055392561 conn(0x7f13cc078340 msgr2=0x7f13cc078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 -- 172.21.15.104:0/2055392561 shutdown_connections 2024-06-09T22:23:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 -- 172.21.15.104:0/2055392561 wait complete. 2024-06-09T22:23:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 Processor -- start 2024-06-09T22:23:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 -- start start 2024-06-09T22:23:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13cc06a170 0x7f13cc19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13cc06ab30 0x7f13cc19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc108500 0x7f13cc198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc112760 con 0x7f13cc06ab30 2024-06-09T22:23:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc1125e0 con 0x7f13cc06a170 2024-06-09T22:23:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.299+0000 7f13d21ae700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc1128e0 con 0x7f13cc108500 2024-06-09T22:23:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13cc06ab30 0x7f13cc19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13cc06ab30 0x7f13cc19eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38744/0 (socket says 172.21.15.104:38744) 2024-06-09T22:23:04.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13c3fff700 1 -- 172.21.15.104:0/1531596027 learned_addr learned my addr 172.21.15.104:0/1531596027 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:04.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13cbfff700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc108500 0x7f13cc198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:04.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13cb7fe700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13cc06a170 0x7f13cc19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:04.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13c3fff700 1 -- 172.21.15.104:0/1531596027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc108500 msgr2=0x7f13cc198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:04.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13c3fff700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc108500 0x7f13cc198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:04.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13c3fff700 1 -- 172.21.15.104:0/1531596027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13cc06a170 msgr2=0x7f13cc19e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:04.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13c3fff700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13cc06a170 0x7f13cc19e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:04.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13c3fff700 1 -- 172.21.15.104:0/1531596027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13cc199180 con 0x7f13cc06ab30 2024-06-09T22:23:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:23:04.829 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:22:59.692018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24419,24403,24442,24460,24448]},"id":1}]} 2024-06-09T22:23:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13c3fff700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13cc06ab30 0x7f13cc19eda0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f13b400b5c0 tx=0x7f13b40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13c97fa700 1 -- 172.21.15.104:0/1531596027 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4027070 con 0x7f13cc06ab30 2024-06-09T22:23:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13d21ae700 1 -- 172.21.15.104:0/1531596027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13cc199410 con 0x7f13cc06ab30 2024-06-09T22:23:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13c97fa700 1 -- 172.21.15.104:0/1531596027 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13b4004030 con 0x7f13cc06ab30 2024-06-09T22:23:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13c97fa700 1 -- 172.21.15.104:0/1531596027 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b400f460 con 0x7f13cc06ab30 2024-06-09T22:23:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.300+0000 7f13d21ae700 1 -- 172.21.15.104:0/1531596027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13cc1a5730 con 0x7f13cc06ab30 2024-06-09T22:23:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.301+0000 7f13d21ae700 1 -- 172.21.15.104:0/1531596027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13cc103bc0 con 0x7f13cc06ab30 2024-06-09T22:23:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.302+0000 7f13c97fa700 1 -- 172.21.15.104:0/1531596027 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f13b402a070 con 0x7f13cc06ab30 2024-06-09T22:23:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.303+0000 7f13c97fa700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13b80a2d90 0x7f13b80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.303+0000 7f13cb7fe700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13b80a2d90 0x7f13b80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.303+0000 7f13c97fa700 1 -- 172.21.15.104:0/1531596027 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f13b40d0670 con 0x7f13cc06ab30 2024-06-09T22:23:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.303+0000 7f13cb7fe700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13b80a2d90 0x7f13b80a5250 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f13cc199b60 tx=0x7f13bc00b5b0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.308+0000 7f13c97fa700 1 -- 172.21.15.104:0/1531596027 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f13b405e850 con 0x7f13cc06ab30 2024-06-09T22:23:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.553+0000 7f13d21ae700 1 -- 172.21.15.104:0/1531596027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13cc02dca0 con 0x7f13cc06ab30 2024-06-09T22:23:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.553+0000 7f13c97fa700 1 -- 172.21.15.104:0/1531596027 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11956 (secure 0 0 0) 0x7f13b4013070 con 0x7f13cc06ab30 2024-06-09T22:23:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.556+0000 7f13d21ae700 1 -- 172.21.15.104:0/1531596027 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13b80a2d90 msgr2=0x7f13b80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.556+0000 7f13d21ae700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13b80a2d90 0x7f13b80a5250 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f13cc199b60 tx=0x7f13bc00b5b0 comp rx=0 tx=0).stop 2024-06-09T22:23:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.556+0000 7f13d21ae700 1 -- 172.21.15.104:0/1531596027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13cc06ab30 msgr2=0x7f13cc19eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.556+0000 7f13d21ae700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13cc06ab30 0x7f13cc19eda0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f13b400b5c0 tx=0x7f13b40047b0 comp rx=0 tx=0).stop 2024-06-09T22:23:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.556+0000 7f13d21ae700 1 -- 172.21.15.104:0/1531596027 shutdown_connections 2024-06-09T22:23:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.556+0000 7f13d21ae700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13b80a2d90 0x7f13b80a5250 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.556+0000 7f13d21ae700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc108500 0x7f13cc198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.556+0000 7f13d21ae700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13cc06a170 0x7f13cc19e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.556+0000 7f13d21ae700 1 --2- 172.21.15.104:0/1531596027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13cc06ab30 0x7f13cc19eda0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.556+0000 7f13d21ae700 1 -- 172.21.15.104:0/1531596027 >> 172.21.15.104:0/1531596027 conn(0x7f13cc078340 msgr2=0x7f13cc0fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.556+0000 7f13d21ae700 1 -- 172.21.15.104:0/1531596027 shutdown_connections 2024-06-09T22:23:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:04.556+0000 7f13d21ae700 1 -- 172.21.15.104:0/1531596027 wait complete. 2024-06-09T22:23:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 30 2024-06-09T22:23:04.832 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:04 smithi104 ceph-mon[32389]: pgmap v66: 129 pgs: 129 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 1.1 MiB/s wr, 287 op/s 2024-06-09T22:23:04.832 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:04 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1531596027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:23:04.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:04 smithi153 ceph-mon[36599]: pgmap v66: 129 pgs: 129 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 1.1 MiB/s wr, 287 op/s 2024-06-09T22:23:04.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:04 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1531596027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:23:05.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:04 smithi194 ceph-mon[36856]: pgmap v66: 129 pgs: 129 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 1.1 MiB/s wr, 287 op/s 2024-06-09T22:23:05.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:04 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1531596027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:23:05.580 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24460 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-06-09T22:23:05.580 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24460 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-06-09T22:23:05.631 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:23:05.716 INFO:tasks.workunit.client.0.smithi104.stdout: 1 18466 10.62 MB/sec warmup 11 sec latency 3937.422 ms 2024-06-09T22:23:06.715 INFO:tasks.workunit.client.0.smithi104.stdout: 1 20130 10.30 MB/sec warmup 12 sec latency 4334.621 ms 2024-06-09T22:23:06.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:06 smithi153 ceph-mon[36599]: pgmap v67: 129 pgs: 129 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 283 op/s 2024-06-09T22:23:07.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:06 smithi104 ceph-mon[32389]: pgmap v67: 129 pgs: 129 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 283 op/s 2024-06-09T22:23:07.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:06 smithi194 ceph-mon[36856]: pgmap v67: 129 pgs: 129 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 283 op/s 2024-06-09T22:23:07.199 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:23:07.716 INFO:tasks.workunit.client.0.smithi104.stdout: 1 21924 10.44 MB/sec warmup 13 sec latency 13.745 ms 2024-06-09T22:23:07.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.770+0000 7f61be5f5700 1 -- 172.21.15.104:0/3837983600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61b806aac0 msgr2=0x7f61b810ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:07.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.770+0000 7f61be5f5700 1 --2- 172.21.15.104:0/3837983600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61b806aac0 0x7f61b810ea00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009990 tx=0x7f61ac009ca0 comp rx=0 tx=0).stop 2024-06-09T22:23:07.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.770+0000 7f61be5f5700 1 -- 172.21.15.104:0/3837983600 shutdown_connections 2024-06-09T22:23:07.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.770+0000 7f61be5f5700 1 --2- 172.21.15.104:0/3837983600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b810ef40 0x7f61b8111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:07.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.770+0000 7f61be5f5700 1 --2- 172.21.15.104:0/3837983600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61b806a1a0 0x7f61b806a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:07.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.770+0000 7f61be5f5700 1 --2- 172.21.15.104:0/3837983600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61b806aac0 0x7f61b810ea00 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:07.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.770+0000 7f61be5f5700 1 -- 172.21.15.104:0/3837983600 >> 172.21.15.104:0/3837983600 conn(0x7f61b8078340 msgr2=0x7f61b8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:07.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.770+0000 7f61be5f5700 1 -- 172.21.15.104:0/3837983600 shutdown_connections 2024-06-09T22:23:07.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.770+0000 7f61be5f5700 1 -- 172.21.15.104:0/3837983600 wait complete. 2024-06-09T22:23:07.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.771+0000 7f61be5f5700 1 Processor -- start 2024-06-09T22:23:07.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.771+0000 7f61be5f5700 1 -- start start 2024-06-09T22:23:07.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.771+0000 7f61be5f5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61b806a1a0 0x7f61b819af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:07.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.771+0000 7f61be5f5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b806aac0 0x7f61b819b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:07.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61be5f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61b810ef40 0x7f61b819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:07.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61be5f5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8114a10 con 0x7f61b810ef40 2024-06-09T22:23:07.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61be5f5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8114890 con 0x7f61b806aac0 2024-06-09T22:23:07.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61be5f5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8114b90 con 0x7f61b806a1a0 2024-06-09T22:23:07.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b7fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61b806a1a0 0x7f61b819af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:07.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b806aac0 0x7f61b819b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:07.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b7fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61b806a1a0 0x7f61b819af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:38462/0 (socket says 172.21.15.104:38462) 2024-06-09T22:23:07.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b806aac0 0x7f61b819b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:47420/0 (socket says 172.21.15.104:47420) 2024-06-09T22:23:07.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b7fff700 1 -- 172.21.15.104:0/3414476811 learned_addr learned my addr 172.21.15.104:0/3414476811 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:07.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61bcb92700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61b810ef40 0x7f61b819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:07.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b7fff700 1 -- 172.21.15.104:0/3414476811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b806aac0 msgr2=0x7f61b819b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:07.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b7fff700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b806aac0 0x7f61b819b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:07.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b7fff700 1 -- 172.21.15.104:0/3414476811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61b810ef40 msgr2=0x7f61b819f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:07.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b7fff700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61b810ef40 0x7f61b819f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:07.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b7fff700 1 -- 172.21.15.104:0/3414476811 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b819ff00 con 0x7f61b806a1a0 2024-06-09T22:23:07.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b77fe700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b806aac0 0x7f61b819b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:07.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b7fff700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61b806a1a0 0x7f61b819af40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f61a800e9c0 tx=0x7f61a800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:07.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b57fa700 1 -- 172.21.15.104:0/3414476811 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a800cbd0 con 0x7f61b806a1a0 2024-06-09T22:23:07.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b57fa700 1 -- 172.21.15.104:0/3414476811 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61a800ee60 con 0x7f61b806a1a0 2024-06-09T22:23:07.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61be5f5700 1 -- 172.21.15.104:0/3414476811 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b81a01f0 con 0x7f61b806a1a0 2024-06-09T22:23:07.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61b57fa700 1 -- 172.21.15.104:0/3414476811 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a8005630 con 0x7f61b806a1a0 2024-06-09T22:23:07.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.772+0000 7f61be5f5700 1 -- 172.21.15.104:0/3414476811 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b81a7a70 con 0x7f61b806a1a0 2024-06-09T22:23:07.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.773+0000 7f61be5f5700 1 -- 172.21.15.104:0/3414476811 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b8069180 con 0x7f61b806a1a0 2024-06-09T22:23:07.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.775+0000 7f61b57fa700 1 -- 172.21.15.104:0/3414476811 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f61a8044bb0 con 0x7f61b806a1a0 2024-06-09T22:23:07.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.775+0000 7f61b57fa700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f61a00a2d90 0x7f61a00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:07.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.776+0000 7f61b77fe700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f61a00a2d90 0x7f61a00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:07.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.776+0000 7f61b57fa700 1 -- 172.21.15.104:0/3414476811 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f61a80cfa10 con 0x7f61b806a1a0 2024-06-09T22:23:07.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.776+0000 7f61b77fe700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f61a00a2d90 0x7f61a00a5250 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f61b8102910 tx=0x7f61ac005fd0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:07.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:07.779+0000 7f61b57fa700 1 -- 172.21.15.104:0/3414476811 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f61a809c2c0 con 0x7f61b806a1a0 2024-06-09T22:23:08.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.024+0000 7f61be5f5700 1 -- 172.21.15.104:0/3414476811 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61b819c250 con 0x7f61b806a1a0 2024-06-09T22:23:08.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.024+0000 7f61b57fa700 1 -- 172.21.15.104:0/3414476811 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11956 (secure 0 0 0) 0x7f61a809bbc0 con 0x7f61b806a1a0 2024-06-09T22:23:08.024 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:23:08.025 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:22:59.692018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24419,24403,24442,24460,24448]},"id":1}]} 2024-06-09T22:23:08.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.027+0000 7f61be5f5700 1 -- 172.21.15.104:0/3414476811 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f61a00a2d90 msgr2=0x7f61a00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:08.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.027+0000 7f61be5f5700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f61a00a2d90 0x7f61a00a5250 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f61b8102910 tx=0x7f61ac005fd0 comp rx=0 tx=0).stop 2024-06-09T22:23:08.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.027+0000 7f61be5f5700 1 -- 172.21.15.104:0/3414476811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61b806a1a0 msgr2=0x7f61b819af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:08.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.027+0000 7f61be5f5700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61b806a1a0 0x7f61b819af40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f61a800e9c0 tx=0x7f61a800ecd0 comp rx=0 tx=0).stop 2024-06-09T22:23:08.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.027+0000 7f61be5f5700 1 -- 172.21.15.104:0/3414476811 shutdown_connections 2024-06-09T22:23:08.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.027+0000 7f61be5f5700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f61a00a2d90 0x7f61a00a5250 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:08.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.027+0000 7f61be5f5700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61b810ef40 0x7f61b819f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:08.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.027+0000 7f61be5f5700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61b806a1a0 0x7f61b819af40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:08.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.027+0000 7f61be5f5700 1 --2- 172.21.15.104:0/3414476811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b806aac0 0x7f61b819b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:08.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.027+0000 7f61be5f5700 1 -- 172.21.15.104:0/3414476811 >> 172.21.15.104:0/3414476811 conn(0x7f61b8078340 msgr2=0x7f61b8110390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:08.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.027+0000 7f61be5f5700 1 -- 172.21.15.104:0/3414476811 shutdown_connections 2024-06-09T22:23:08.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:08.027+0000 7f61be5f5700 1 -- 172.21.15.104:0/3414476811 wait complete. 2024-06-09T22:23:08.027 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 30 2024-06-09T22:23:08.716 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22047 9.71 MB/sec warmup 14 sec latency 915.092 ms 2024-06-09T22:23:08.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:08 smithi153 ceph-mon[36599]: pgmap v68: 129 pgs: 129 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 283 op/s 2024-06-09T22:23:08.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:08 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3414476811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:23:09.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:08 smithi104 ceph-mon[32389]: pgmap v68: 129 pgs: 129 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 283 op/s 2024-06-09T22:23:09.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:08 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3414476811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:23:09.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:08 smithi194 ceph-mon[36856]: pgmap v68: 129 pgs: 129 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 283 op/s 2024-06-09T22:23:09.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:08 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3414476811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:23:09.250 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-09T22:23:09.716 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22047 9.06 MB/sec warmup 15 sec latency 1915.164 ms 2024-06-09T22:23:10.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.363+0000 7f9a178f7700 1 -- 172.21.15.104:0/2134507499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a10106b90 msgr2=0x7f9a10106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:10.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.363+0000 7f9a178f7700 1 --2- 172.21.15.104:0/2134507499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a10106b90 0x7f9a10106f70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009990 tx=0x7f9a0c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.363+0000 7f9a178f7700 1 -- 172.21.15.104:0/2134507499 shutdown_connections 2024-06-09T22:23:10.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.363+0000 7f9a178f7700 1 --2- 172.21.15.104:0/2134507499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a10106b90 0x7f9a10106f70 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.363+0000 7f9a178f7700 1 --2- 172.21.15.104:0/2134507499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a101014c0 0x7f9a1010d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.363+0000 7f9a178f7700 1 --2- 172.21.15.104:0/2134507499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a10100b00 0x7f9a10100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.363+0000 7f9a178f7700 1 -- 172.21.15.104:0/2134507499 >> 172.21.15.104:0/2134507499 conn(0x7f9a10077a30 msgr2=0x7f9a100fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:10.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.363+0000 7f9a178f7700 1 -- 172.21.15.104:0/2134507499 shutdown_connections 2024-06-09T22:23:10.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.363+0000 7f9a178f7700 1 -- 172.21.15.104:0/2134507499 wait complete. 2024-06-09T22:23:10.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a178f7700 1 Processor -- start 2024-06-09T22:23:10.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a178f7700 1 -- start start 2024-06-09T22:23:10.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a178f7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a10100b00 0x7f9a1019e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a178f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a101014c0 0x7f9a1019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a178f7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a10106b90 0x7f9a10198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a178f7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9a10111690 con 0x7f9a101014c0 2024-06-09T22:23:10.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a178f7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a10111510 con 0x7f9a10106b90 2024-06-09T22:23:10.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a178f7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9a10111810 con 0x7f9a10100b00 2024-06-09T22:23:10.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a14e92700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a101014c0 0x7f9a1019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a14e92700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a101014c0 0x7f9a1019edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59874/0 (socket says 172.21.15.104:59874) 2024-06-09T22:23:10.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a14e92700 1 -- 172.21.15.104:0/4147080958 learned_addr learned my addr 172.21.15.104:0/4147080958 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:10.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a15693700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a10100b00 0x7f9a1019e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a15e94700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a10106b90 0x7f9a10198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a14e92700 1 -- 172.21.15.104:0/4147080958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a10100b00 msgr2=0x7f9a1019e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:10.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a14e92700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a10100b00 0x7f9a1019e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a14e92700 1 -- 172.21.15.104:0/4147080958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a10106b90 msgr2=0x7f9a10198d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:10.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a14e92700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a10106b90 0x7f9a10198d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a14e92700 1 -- 172.21.15.104:0/4147080958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a10199650 con 0x7f9a101014c0 2024-06-09T22:23:10.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.364+0000 7f9a15e94700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a10106b90 0x7f9a10198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:10.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.365+0000 7f9a14e92700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a101014c0 0x7f9a1019edd0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000d950 tx=0x7f9a0000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:10.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.365+0000 7f9a067fc700 1 -- 172.21.15.104:0/4147080958 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a00016070 con 0x7f9a101014c0 2024-06-09T22:23:10.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.365+0000 7f9a178f7700 1 -- 172.21.15.104:0/4147080958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a101998e0 con 0x7f9a101014c0 2024-06-09T22:23:10.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.365+0000 7f9a067fc700 1 -- 172.21.15.104:0/4147080958 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a00009e20 con 0x7f9a101014c0 2024-06-09T22:23:10.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.365+0000 7f9a067fc700 1 -- 172.21.15.104:0/4147080958 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0000f620 con 0x7f9a101014c0 2024-06-09T22:23:10.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.365+0000 7f9a178f7700 1 -- 172.21.15.104:0/4147080958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a101a5850 con 0x7f9a101014c0 2024-06-09T22:23:10.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.368+0000 7f9a067fc700 1 -- 172.21.15.104:0/4147080958 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f9a00019070 con 0x7f9a101014c0 2024-06-09T22:23:10.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.369+0000 7f9a067fc700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f99fc0a2e30 0x7f99fc0a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.369+0000 7f9a067fc700 1 -- 172.21.15.104:0/4147080958 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f9a000cf790 con 0x7f9a101014c0 2024-06-09T22:23:10.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.369+0000 7f9a15693700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f99fc0a2e30 0x7f99fc0a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.369+0000 7f9a2217ab80 1 Processor -- start 2024-06-09T22:23:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.369+0000 7f9a2217ab80 1 -- start start 2024-06-09T22:23:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a15693700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f99fc0a2e30 0x7f99fc0a52f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c0098e0 tx=0x7f9a0c005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a2217ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b974c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a2217ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15b97a00 0x564c15b91330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a2217ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 0x564c15b91cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a2217ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564c15ade9a0 con 0x564c15b91870 2024-06-09T22:23:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a2217ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564c15a868b0 con 0x564c15b97a00 2024-06-09T22:23:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a2217ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564c15ada960 con 0x564c15b970e0 2024-06-09T22:23:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a15e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 0x564c15b91cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a15e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 0x564c15b91cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59884/0 (socket says 172.21.15.104:59884) 2024-06-09T22:23:10.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a15e94700 1 -- 172.21.15.104:0/2575562418 learned_addr learned my addr 172.21.15.104:0/2575562418 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a14e92700 1 --2- 172.21.15.104:0/2575562418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15b97a00 0x564c15b91330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a15693700 1 --2- 172.21.15.104:0/2575562418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a15e94700 1 -- 172.21.15.104:0/2575562418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 msgr2=0x564c15b974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a15e94700 1 --2- 172.21.15.104:0/2575562418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a15e94700 1 -- 172.21.15.104:0/2575562418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15b97a00 msgr2=0x564c15b91330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a15e94700 1 --2- 172.21.15.104:0/2575562418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15b97a00 0x564c15b91330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a15e94700 1 -- 172.21.15.104:0/2575562418 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c15a7d0e0 con 0x564c15b91870 2024-06-09T22:23:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a15693700 1 --2- 172.21.15.104:0/2575562418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.370+0000 7f9a15e94700 1 --2- 172.21.15.104:0/2575562418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 0x564c15b91cb0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9a08009d70 tx=0x7f9a080077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67f8b70653c5cf1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.371+0000 7f99fb7fe700 1 -- 172.21.15.104:0/2575562418 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a08004bb0 con 0x564c15b91870 2024-06-09T22:23:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.371+0000 7f99fb7fe700 1 -- 172.21.15.104:0/2575562418 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a08004d50 con 0x564c15b91870 2024-06-09T22:23:10.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.371+0000 7f99fb7fe700 1 -- 172.21.15.104:0/2575562418 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080053a0 con 0x564c15b91870 2024-06-09T22:23:10.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.372+0000 7f9a2217ab80 1 -- 172.21.15.104:0/2575562418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 msgr2=0x564c15b91cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:10.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.372+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/2575562418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 0x564c15b91cb0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9a08009d70 tx=0x7f9a080077e0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.372+0000 7f9a2217ab80 1 -- 172.21.15.104:0/2575562418 shutdown_connections 2024-06-09T22:23:10.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.372+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/2575562418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 0x564c15b91cb0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.372+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/2575562418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.372+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/2575562418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15b97a00 0x564c15b91330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.372+0000 7f9a2217ab80 1 -- 172.21.15.104:0/2575562418 >> 172.21.15.104:0/2575562418 conn(0x564c15ab7e50 msgr2=0x564c15b881b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:10.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.372+0000 7f9a2217ab80 1 -- 172.21.15.104:0/2575562418 shutdown_connections 2024-06-09T22:23:10.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.372+0000 7f9a2217ab80 1 -- 172.21.15.104:0/2575562418 wait complete. 2024-06-09T22:23:10.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.372+0000 7f9a2217ab80 1 -- start start 2024-06-09T22:23:10.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.372+0000 7f9a2217ab80 1 Processor -- start 2024-06-09T22:23:10.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a2217ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 0x564c15b9d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a2217ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b9aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a2217ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15b97a00 0x564c15b9af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a2217ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564c15b9e470 con 0x564c15b91870 2024-06-09T22:23:10.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a2217ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564c15a7ed80 con 0x564c15b97a00 2024-06-09T22:23:10.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a2217ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564c15b9e5f0 con 0x564c15b970e0 2024-06-09T22:23:10.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a2217ab80 10 client.0 _ll_register_callbacks cb 0x564c15a78ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:10.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a15693700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 0x564c15b9d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a15693700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 0x564c15b9d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59896/0 (socket says 172.21.15.104:59896) 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a15693700 1 -- 172.21.15.104:0/3000409743 learned_addr learned my addr 172.21.15.104:0/3000409743 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a15e94700 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15b97a00 0x564c15b9af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a14e92700 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b9aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a15693700 1 -- 172.21.15.104:0/3000409743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 msgr2=0x564c15b9aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a15693700 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b9aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a15693700 1 -- 172.21.15.104:0/3000409743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15b97a00 msgr2=0x564c15b9af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a15693700 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15b97a00 0x564c15b9af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a15693700 1 -- 172.21.15.104:0/3000409743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c15a2da60 con 0x564c15b91870 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.373+0000 7f9a14e92700 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b9aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f9a15693700 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 0x564c15b9d700 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c015000 tx=0x7f9a0c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a144ee3ce3d8440 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f99f8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f99f8ff9700 1 -- 172.21.15.104:0/3000409743 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c035060 con 0x564c15b91870 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f9a2217ab80 1 -- 172.21.15.104:0/3000409743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a08004d50 con 0x564c15b91870 2024-06-09T22:23:10.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f99f8ff9700 1 -- 172.21.15.104:0/3000409743 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f9a0c030630 con 0x564c15b91870 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f99f8ff9700 1 -- 172.21.15.104:0/3000409743 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9a0c03c050 con 0x564c15b91870 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f9a2217ab80 10 client.16110 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f9a2217ab80 1 -- 172.21.15.104:0/3000409743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564c15ba8d80 con 0x564c15b91870 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f99f8ff9700 1 -- 172.21.15.104:0/3000409743 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a08004d50 con 0x564c15b91870 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f99f8ff9700 1 -- 172.21.15.104:0/3000409743 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9a0c02f070 con 0x564c15b91870 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f9a2217ab80 10 client.16110 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f9a2217ab80 10 client.16110 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f9a2217ab80 20 client.16110 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.374+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564c15bab740 0x564c15badbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.375+0000 7f9a2217ab80 4 client.16110 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.375+0000 7f9a2217ab80 1 -- 172.21.15.104:0/3000409743 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c15aebea0 con 0x564c15bab740 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.375+0000 7f9a14e92700 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564c15bab740 0x564c15badbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.375+0000 7f99f8ff9700 10 client.16110 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:10.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.375+0000 7f9a14e92700 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564c15bab740 0x564c15badbc0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55dcdfcb585f31ff server_cookie=aa30edbf3bd031f9 in_seq=0 out_seq=0 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.376+0000 7f99f8ff9700 1 -- 172.21.15.104:0/3000409743 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x564c15aebea0 con 0x564c15bab740 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.376+0000 7f99f8ff9700 10 client.16110 handle_command_reply: tid=0 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.376+0000 7f9a2217ab80 1 client.16110 shutdown 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.376+0000 7f9a2217ab80 1 -- 172.21.15.104:0/3000409743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 msgr2=0x564c15b9d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.376+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 0x564c15b9d700 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c015000 tx=0x7f9a0c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.376+0000 7f9a2217ab80 1 -- 172.21.15.104:0/3000409743 shutdown_connections 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.376+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564c15bab740 0x564c15badbc0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.376+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b91870 0x564c15b9d700 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.376+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15b97a00 0x564c15b9af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.376+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/3000409743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b9aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.376+0000 7f9a2217ab80 1 -- 172.21.15.104:0/3000409743 >> 172.21.15.104:0/3000409743 conn(0x564c15ab7e50 msgr2=0x564c15b87da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.376+0000 7f99f8ff9700 0 client.16110 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.377+0000 7f9a2217ab80 1 -- 172.21.15.104:0/3000409743 shutdown_connections 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.377+0000 7f9a2217ab80 1 -- 172.21.15.104:0/3000409743 wait complete. 2024-06-09T22:23:10.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.377+0000 7f9a2217ab80 20 client.16110 trim_cache size 0 max 16384 2024-06-09T22:23:10.458 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:23:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.398+0000 7f9a2217ab80 1 Processor -- start 2024-06-09T22:23:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.398+0000 7f9a2217ab80 1 -- start start 2024-06-09T22:23:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.398+0000 7f9a2217ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b91330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.398+0000 7f9a2217ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 0x564c15b87fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.398+0000 7f9a2217ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15ba7770 0x564c15b97de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.398+0000 7f9a2217ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9a101ac320 con 0x564c15b97a00 2024-06-09T22:23:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.398+0000 7f9a2217ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a101adec0 con 0x564c15ba7770 2024-06-09T22:23:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.398+0000 7f9a2217ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9a101ac9f0 con 0x564c15b970e0 2024-06-09T22:23:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.398+0000 7f9a14e92700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 0x564c15b87fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.398+0000 7f9a14e92700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 0x564c15b87fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59908/0 (socket says 172.21.15.104:59908) 2024-06-09T22:23:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.398+0000 7f9a14e92700 1 -- 172.21.15.104:0/617175182 learned_addr learned my addr 172.21.15.104:0/617175182 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.398+0000 7f9a15693700 1 --2- 172.21.15.104:0/617175182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b91330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.399+0000 7f9a15e94700 1 --2- 172.21.15.104:0/617175182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15ba7770 0x564c15b97de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.399+0000 7f9a15693700 1 -- 172.21.15.104:0/617175182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15ba7770 msgr2=0x564c15b97de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.399+0000 7f9a15693700 1 --2- 172.21.15.104:0/617175182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15ba7770 0x564c15b97de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.399+0000 7f9a15693700 1 -- 172.21.15.104:0/617175182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 msgr2=0x564c15b87fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.399+0000 7f9a15693700 1 --2- 172.21.15.104:0/617175182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 0x564c15b87fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.399+0000 7f9a15693700 1 -- 172.21.15.104:0/617175182 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c15b97830 con 0x564c15b970e0 2024-06-09T22:23:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.399+0000 7f9a14e92700 1 --2- 172.21.15.104:0/617175182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 0x564c15b87fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.399+0000 7f9a15693700 1 --2- 172.21.15.104:0/617175182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b91330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c0175a0 tx=0x7f9a0c0173f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7698c58c7541c202 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.399+0000 7f99faffd700 1 -- 172.21.15.104:0/617175182 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c035070 con 0x564c15b970e0 2024-06-09T22:23:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.399+0000 7f99faffd700 1 -- 172.21.15.104:0/617175182 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a0c030430 con 0x564c15b970e0 2024-06-09T22:23:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.399+0000 7f99faffd700 1 -- 172.21.15.104:0/617175182 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c022620 con 0x564c15b970e0 2024-06-09T22:23:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.400+0000 7f9a2217ab80 1 -- 172.21.15.104:0/617175182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 msgr2=0x564c15b91330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.400+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/617175182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b91330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c0175a0 tx=0x7f9a0c0173f0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.400+0000 7f9a2217ab80 1 -- 172.21.15.104:0/617175182 shutdown_connections 2024-06-09T22:23:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.400+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/617175182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15ba7770 0x564c15b97de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.400+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/617175182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15b91330 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.400+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/617175182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 0x564c15b87fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.401+0000 7f9a2217ab80 1 -- 172.21.15.104:0/617175182 >> 172.21.15.104:0/617175182 conn(0x564c15ab7e50 msgr2=0x564c15b878c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.401+0000 7f9a2217ab80 1 -- 172.21.15.104:0/617175182 shutdown_connections 2024-06-09T22:23:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.401+0000 7f9a2217ab80 1 -- 172.21.15.104:0/617175182 wait complete. 2024-06-09T22:23:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.401+0000 7f9a2217ab80 1 -- start start 2024-06-09T22:23:10.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.401+0000 7f9a2217ab80 1 Processor -- start 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a2217ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15ba3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a2217ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 0x564c15bb2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a14e92700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 0x564c15bb2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a15693700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15ba3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a15693700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15ba3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:38524/0 (socket says 172.21.15.104:38524) 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a14e92700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 0x564c15bb2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59916/0 (socket says 172.21.15.104:59916) 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a15693700 1 -- 172.21.15.104:0/4120135528 learned_addr learned my addr 172.21.15.104:0/4120135528 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15bb3080 0x564c15bb3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a2217ab80 1 -- 172.21.15.104:0/4120135528 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9a101ad790 con 0x564c15b97a00 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a2217ab80 1 -- 172.21.15.104:0/4120135528 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a101ad0c0 con 0x564c15bb3080 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a2217ab80 1 -- 172.21.15.104:0/4120135528 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9a101aa2a0 con 0x564c15b970e0 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a2217ab80 10 client.0 _ll_register_callbacks cb 0x564c15a78ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:10.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a15e94700 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15bb3080 0x564c15bb3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a14e92700 1 -- 172.21.15.104:0/4120135528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 msgr2=0x564c15ba3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a14e92700 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15ba3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a14e92700 1 -- 172.21.15.104:0/4120135528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15bb3080 msgr2=0x564c15bb3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a14e92700 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15bb3080 0x564c15bb3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.402+0000 7f9a14e92700 1 -- 172.21.15.104:0/4120135528 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c15c15460 con 0x564c15b97a00 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.403+0000 7f9a14e92700 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 0x564c15bb2b40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000b2f0 tx=0x7f9a00005910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94954a4958676e4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.403+0000 7f99f8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.403+0000 7f99f8ff9700 1 -- 172.21.15.104:0/4120135528 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a00020040 con 0x564c15b97a00 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.403+0000 7f99f8ff9700 1 -- 172.21.15.104:0/4120135528 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f9a00010780 con 0x564c15b97a00 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.403+0000 7f9a2217ab80 1 -- 172.21.15.104:0/4120135528 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a0c030430 con 0x564c15b97a00 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.403+0000 7f99f8ff9700 1 -- 172.21.15.104:0/4120135528 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9a00097f70 con 0x564c15b97a00 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.403+0000 7f9a2217ab80 10 client.16116 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:10.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.403+0000 7f9a2217ab80 1 -- 172.21.15.104:0/4120135528 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564c15c16650 con 0x564c15b97a00 2024-06-09T22:23:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.404+0000 7f99f8ff9700 1 -- 172.21.15.104:0/4120135528 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a000ccdb0 con 0x564c15b97a00 2024-06-09T22:23:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.404+0000 7f99f8ff9700 1 -- 172.21.15.104:0/4120135528 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9a000cf040 con 0x564c15b97a00 2024-06-09T22:23:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.404+0000 7f9a2217ab80 10 client.16116 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.404+0000 7f9a2217ab80 10 client.16116 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:23:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.404+0000 7f9a2217ab80 20 client.16116 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.404+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564c15c16de0 0x564c15c19200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.404+0000 7f9a2217ab80 4 client.16116 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-06-09T22:23:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.404+0000 7f9a2217ab80 1 -- 172.21.15.104:0/4120135528 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x7f9a0c030430 con 0x564c15c16de0 2024-06-09T22:23:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.404+0000 7f9a15693700 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564c15c16de0 0x564c15c19200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.405+0000 7f99f8ff9700 10 client.16116 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:10.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.405+0000 7f9a15693700 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564c15c16de0 0x564c15c19200 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1df3e631774c3f2f server_cookie=7a367860b35a34a5 in_seq=0 out_seq=0 2024-06-09T22:23:10.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.472+0000 7f99f8ff9700 1 -- 172.21.15.104:0/4120135528 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x7f9a0c030430 con 0x564c15c16de0 2024-06-09T22:23:10.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f99f8ff9700 10 client.16116 handle_command_reply: tid=0 2024-06-09T22:23:10.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f9a2217ab80 1 client.16116 shutdown 2024-06-09T22:23:10.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f9a2217ab80 1 -- 172.21.15.104:0/4120135528 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 msgr2=0x564c15bb2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:10.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 0x564c15bb2b40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000b2f0 tx=0x7f9a00005910 comp rx=0 tx=0).stop 2024-06-09T22:23:10.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f9a2217ab80 1 -- 172.21.15.104:0/4120135528 shutdown_connections 2024-06-09T22:23:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564c15c16de0 0x564c15c19200 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564c15bb3080 0x564c15bb3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c15b970e0 0x564c15ba3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f9a2217ab80 1 --2- 172.21.15.104:0/4120135528 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564c15b97a00 0x564c15bb2b40 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f9a2217ab80 1 -- 172.21.15.104:0/4120135528 >> 172.21.15.104:0/4120135528 conn(0x564c15ab7e50 msgr2=0x564c15b98f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f99f8ff9700 0 client.16116 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f9a2217ab80 1 -- 172.21.15.104:0/4120135528 shutdown_connections 2024-06-09T22:23:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f9a2217ab80 1 -- 172.21.15.104:0/4120135528 wait complete. 2024-06-09T22:23:10.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.473+0000 7f9a2217ab80 20 client.16116 trim_cache size 0 max 16384 2024-06-09T22:23:10.474 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:23:10.475 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 30, 2024-06-09T22:23:10.475 INFO:teuthology.orchestra.run.smithi104.stdout: "leader": 24460, 2024-06-09T22:23:10.475 INFO:teuthology.orchestra.run.smithi104.stdout: "set_version": 6, 2024-06-09T22:23:10.475 INFO:teuthology.orchestra.run.smithi104.stdout: "sets": { 2024-06-09T22:23:10.475 INFO:teuthology.orchestra.run.smithi104.stdout: "6f4b8fc4": { 2024-06-09T22:23:10.475 INFO:teuthology.orchestra.run.smithi104.stdout: "version": 6, 2024-06-09T22:23:10.475 INFO:teuthology.orchestra.run.smithi104.stdout: "age_ref": 0.0, 2024-06-09T22:23:10.475 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:23:10.475 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "QUIESCED", 2024-06-09T22:23:10.475 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:23:10.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:23:10.476 INFO:teuthology.orchestra.run.smithi104.stdout: "timeout": 90.0, 2024-06-09T22:23:10.476 INFO:teuthology.orchestra.run.smithi104.stdout: "expiration": 150.0, 2024-06-09T22:23:10.476 INFO:teuthology.orchestra.run.smithi104.stdout: "members": { 2024-06-09T22:23:10.476 INFO:teuthology.orchestra.run.smithi104.stdout: "file:/": { 2024-06-09T22:23:10.476 INFO:teuthology.orchestra.run.smithi104.stdout: "excluded": false, 2024-06-09T22:23:10.476 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:23:10.476 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "QUIESCED", 2024-06-09T22:23:10.477 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:23:10.477 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:23:10.477 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:23:10.477 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:23:10.477 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:23:10.477 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:23:10.477 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:23:10.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.474+0000 7f9a178f7700 1 -- 172.21.15.104:0/4147080958 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f99fc0a2e30 msgr2=0x7f99fc0a52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:10.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.474+0000 7f9a178f7700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f99fc0a2e30 0x7f99fc0a52f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c0098e0 tx=0x7f9a0c005960 comp rx=0 tx=0).stop 2024-06-09T22:23:10.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.474+0000 7f9a178f7700 1 -- 172.21.15.104:0/4147080958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a101014c0 msgr2=0x7f9a1019edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:10.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.474+0000 7f9a178f7700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a101014c0 0x7f9a1019edd0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000d950 tx=0x7f9a0000dc60 comp rx=0 tx=0).stop 2024-06-09T22:23:10.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.474+0000 7f9a178f7700 1 -- 172.21.15.104:0/4147080958 shutdown_connections 2024-06-09T22:23:10.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.474+0000 7f9a178f7700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f99fc0a2e30 0x7f99fc0a52f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.474+0000 7f9a178f7700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a10106b90 0x7f9a10198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.474+0000 7f9a178f7700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a10100b00 0x7f9a1019e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.474+0000 7f9a178f7700 1 --2- 172.21.15.104:0/4147080958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a101014c0 0x7f9a1019edd0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:10.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.474+0000 7f9a178f7700 1 -- 172.21.15.104:0/4147080958 >> 172.21.15.104:0/4147080958 conn(0x7f9a10077a30 msgr2=0x7f9a10102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:10.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.474+0000 7f9a178f7700 1 -- 172.21.15.104:0/4147080958 shutdown_connections 2024-06-09T22:23:10.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:10.474+0000 7f9a178f7700 1 -- 172.21.15.104:0/4147080958 wait complete. 2024-06-09T22:23:10.716 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 8.51 MB/sec warmup 16 sec latency 2220.024 ms 2024-06-09T22:23:10.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:10 smithi153 ceph-mon[36599]: pgmap v69: 129 pgs: 129 active+clean; 33 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 4.1 MiB/s wr, 382 op/s 2024-06-09T22:23:11.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:10 smithi104 ceph-mon[32389]: pgmap v69: 129 pgs: 129 active+clean; 33 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 4.1 MiB/s wr, 382 op/s 2024-06-09T22:23:11.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:10 smithi194 ceph-mon[36856]: pgmap v69: 129 pgs: 129 active+clean; 33 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 4.1 MiB/s wr, 382 op/s 2024-06-09T22:23:11.633 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.631+0000 7f663f31d640 1 -- 172.21.15.104:0/3986645803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6638102c50 msgr2=0x7f6638103030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:11.633 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.631+0000 7f663f31d640 1 --2- 172.21.15.104:0/3986645803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6638102c50 0x7f6638103030 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6628009900 tx=0x7f662802a480 comp rx=0 tx=0).stop 2024-06-09T22:23:11.633 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.631+0000 7f663f31d640 1 -- 172.21.15.104:0/3986645803 shutdown_connections 2024-06-09T22:23:11.633 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.631+0000 7f663f31d640 1 --2- 172.21.15.104:0/3986645803 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f663810c020 0x7f663810e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.631+0000 7f663f31d640 1 --2- 172.21.15.104:0/3986645803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6638103570 0x7f663810ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.631+0000 7f663f31d640 1 --2- 172.21.15.104:0/3986645803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6638102c50 0x7f6638103030 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.631+0000 7f663f31d640 1 -- 172.21.15.104:0/3986645803 >> 172.21.15.104:0/3986645803 conn(0x7f6638076c20 msgr2=0x7f6638077030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.631+0000 7f663f31d640 1 -- 172.21.15.104:0/3986645803 shutdown_connections 2024-06-09T22:23:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.631+0000 7f663f31d640 1 -- 172.21.15.104:0/3986645803 wait complete. 2024-06-09T22:23:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663f31d640 1 Processor -- start 2024-06-09T22:23:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663f31d640 1 -- start start 2024-06-09T22:23:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663f31d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6638102c50 0x7f66381981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663f31d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6638103570 0x7f6638198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663f31d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f663810c020 0x7f663819cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663f31d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6638111db0 con 0x7f663810c020 2024-06-09T22:23:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663f31d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6638111c30 con 0x7f6638103570 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663f31d640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6638111f30 con 0x7f6638102c50 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663d8ba640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f663810c020 0x7f663819cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663d8ba640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f663810c020 0x7f663819cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59946/0 (socket says 172.21.15.104:59946) 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663d8ba640 1 -- 172.21.15.104:0/828989039 learned_addr learned my addr 172.21.15.104:0/828989039 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663d0b9640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6638102c50 0x7f66381981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663c8b8640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6638103570 0x7f6638198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663d8ba640 1 -- 172.21.15.104:0/828989039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6638102c50 msgr2=0x7f66381981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663d8ba640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6638102c50 0x7f66381981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663d8ba640 1 -- 172.21.15.104:0/828989039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6638103570 msgr2=0x7f6638198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663d8ba640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6638103570 0x7f6638198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663d8ba640 1 -- 172.21.15.104:0/828989039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663819d1e0 con 0x7f663810c020 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663d0b9640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6638102c50 0x7f66381981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.632+0000 7f663d8ba640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f663810c020 0x7f663819cab0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f663000e9d0 tx=0x7f663000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.633+0000 7f66367fc640 1 -- 172.21.15.104:0/828989039 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663000cd80 con 0x7f663810c020 2024-06-09T22:23:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.633+0000 7f663f31d640 1 -- 172.21.15.104:0/828989039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f663819d4d0 con 0x7f663810c020 2024-06-09T22:23:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.633+0000 7f66367fc640 1 -- 172.21.15.104:0/828989039 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6630005020 con 0x7f663810c020 2024-06-09T22:23:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.633+0000 7f66367fc640 1 -- 172.21.15.104:0/828989039 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66300116d0 con 0x7f663810c020 2024-06-09T22:23:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.633+0000 7f663f31d640 1 -- 172.21.15.104:0/828989039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66381a4d60 con 0x7f663810c020 2024-06-09T22:23:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f66367fc640 1 -- 172.21.15.104:0/828989039 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f66300051d0 con 0x7f663810c020 2024-06-09T22:23:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f6642020740 1 Processor -- start 2024-06-09T22:23:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f6642020740 1 -- start start 2024-06-09T22:23:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f6642020740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 0x5652f20f1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f6642020740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1a90 0x5652f20fbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f6642020740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 0x5652f20fe740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f6642020740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5652f20554f0 con 0x5652f20fc350 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f6642020740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5652f1ff3870 con 0x5652f20f1170 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f6642020740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5652f20514b0 con 0x5652f20f1a90 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f663c8b8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 0x5652f20fe740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f663c8b8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 0x5652f20fe740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59952/0 (socket says 172.21.15.104:59952) 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f663c8b8640 1 -- 172.21.15.104:0/2290619403 learned_addr learned my addr 172.21.15.104:0/2290619403 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f663c8b8640 1 --2- 172.21.15.104:0/2290619403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 0x5652f20f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.636+0000 7f663d0b9640 1 --2- 172.21.15.104:0/2290619403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1a90 0x5652f20fbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f66367fc640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f66200a2da0 0x7f66200a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f663c8b8640 1 -- 172.21.15.104:0/2290619403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1a90 msgr2=0x5652f20fbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f663c8b8640 1 --2- 172.21.15.104:0/2290619403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1a90 0x5652f20fbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f663c8b8640 1 -- 172.21.15.104:0/2290619403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 msgr2=0x5652f20f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f663c8b8640 1 --2- 172.21.15.104:0/2290619403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 0x5652f20f1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f663c8b8640 1 -- 172.21.15.104:0/2290619403 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652f200f9b0 con 0x5652f20fc350 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f663d0b9640 1 --2- 172.21.15.104:0/2290619403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1a90 0x5652f20fbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f663d8ba640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f66200a2da0 0x7f66200a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f663c8b8640 1 --2- 172.21.15.104:0/2290619403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 0x5652f20fe740 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f662c013630 tx=0x7f662c00c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18951fe0ea6bb4e8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f66367fc640 1 -- 172.21.15.104:0/828989039 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f66300cfd70 con 0x7f663810c020 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f661f7fe640 1 -- 172.21.15.104:0/2290619403 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c005020 con 0x5652f20fc350 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f661f7fe640 1 -- 172.21.15.104:0/2290619403 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f662c0047a0 con 0x5652f20fc350 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.637+0000 7f661f7fe640 1 -- 172.21.15.104:0/2290619403 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c00f780 con 0x5652f20fc350 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.638+0000 7f6642020740 1 -- 172.21.15.104:0/2290619403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 msgr2=0x5652f20fe740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.638+0000 7f663d8ba640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f66200a2da0 0x7f66200a5260 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f66300988f0 tx=0x7f663008f040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.638+0000 7f6642020740 1 --2- 172.21.15.104:0/2290619403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 0x5652f20fe740 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f662c013630 tx=0x7f662c00c4b0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.638+0000 7f6642020740 1 -- 172.21.15.104:0/2290619403 shutdown_connections 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.638+0000 7f6642020740 1 --2- 172.21.15.104:0/2290619403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 0x5652f20fe740 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.638+0000 7f6642020740 1 --2- 172.21.15.104:0/2290619403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1a90 0x5652f20fbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.638+0000 7f6642020740 1 --2- 172.21.15.104:0/2290619403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 0x5652f20f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.638+0000 7f6642020740 1 -- 172.21.15.104:0/2290619403 >> 172.21.15.104:0/2290619403 conn(0x5652f20ed680 msgr2=0x5652f20eda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.638+0000 7f6642020740 1 -- 172.21.15.104:0/2290619403 shutdown_connections 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.638+0000 7f6642020740 1 -- 172.21.15.104:0/2290619403 wait complete. 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.638+0000 7f6642020740 1 -- start start 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.638+0000 7f6642020740 1 Processor -- start 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f6642020740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 0x5652f20f9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f6642020740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1a90 0x5652f20f97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f6642020740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 0x5652f20f9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f6642020740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5652f20fef80 con 0x5652f20fc350 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f6642020740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5652f1fdeae0 con 0x5652f20f1170 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f6642020740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5652f20ff100 con 0x5652f20f1a90 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f6642020740 10 client.0 _ll_register_callbacks cb 0x5652f20179b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f663c8b8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 0x5652f20f9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f663c8b8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 0x5652f20f9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59958/0 (socket says 172.21.15.104:59958) 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f663c8b8640 1 -- 172.21.15.104:0/2555719327 learned_addr learned my addr 172.21.15.104:0/2555719327 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f663c8b8640 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 0x5652f20f9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f663d0b9640 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1a90 0x5652f20f97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f663c8b8640 1 -- 172.21.15.104:0/2555719327 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1a90 msgr2=0x5652f20f97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f663c8b8640 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1a90 0x5652f20f97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f663c8b8640 1 -- 172.21.15.104:0/2555719327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 msgr2=0x5652f20f9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f663c8b8640 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 0x5652f20f9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f663c8b8640 1 -- 172.21.15.104:0/2555719327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652f1d69210 con 0x5652f20fc350 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f663c8b8640 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 0x5652f20f9d10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f662c005fd0 tx=0x7f662c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c73f05acd9649f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.639+0000 7f661cff9640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f661cff9640 1 -- 172.21.15.104:0/2555719327 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c005310 con 0x5652f20fc350 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f661cff9640 1 -- 172.21.15.104:0/2555719327 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f662c0054b0 con 0x5652f20fc350 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f6642020740 1 -- 172.21.15.104:0/2555719327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f662c0047a0 con 0x5652f20fc350 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f661cff9640 1 -- 172.21.15.104:0/2555719327 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f662c036050 con 0x5652f20fc350 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f6642020740 10 client.16128 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f6642020740 1 -- 172.21.15.104:0/2555719327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5652f1e35620 con 0x5652f20fc350 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f661cff9640 1 -- 172.21.15.104:0/2555719327 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f662c0047a0 con 0x5652f20fc350 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f661cff9640 1 -- 172.21.15.104:0/2555719327 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f662c003bc0 con 0x5652f20fc350 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f6642020740 10 client.16128 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f6642020740 10 client.16128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f6642020740 20 client.16128 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f6642020740 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5652f210c440 0x5652f210e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f6642020740 4 client.16128 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.640+0000 7f6642020740 1 -- 172.21.15.104:0/2555719327 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652f2068f10 con 0x5652f210c440 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.641+0000 7f663d0b9640 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5652f210c440 0x5652f210e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.641+0000 7f661cff9640 10 client.16128 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.641+0000 7f663d0b9640 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5652f210c440 0x5652f210e830 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b89d0ec7bbfc7c3 server_cookie=6c879f8845d86ce1 in_seq=0 out_seq=0 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f661cff9640 1 -- 172.21.15.104:0/2555719327 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5652f2068f10 con 0x5652f210c440 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f661cff9640 10 client.16128 handle_command_reply: tid=0 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f6642020740 1 client.16128 shutdown 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f6642020740 1 -- 172.21.15.104:0/2555719327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 msgr2=0x5652f20f9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f6642020740 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 0x5652f20f9d10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f662c005fd0 tx=0x7f662c005850 comp rx=0 tx=0).stop 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f6642020740 1 -- 172.21.15.104:0/2555719327 shutdown_connections 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f6642020740 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5652f210c440 0x5652f210e830 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f6642020740 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20fc350 0x5652f20f9d10 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f6642020740 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1a90 0x5652f20f97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f6642020740 1 --2- 172.21.15.104:0/2555719327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 0x5652f20f9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f6642020740 1 -- 172.21.15.104:0/2555719327 >> 172.21.15.104:0/2555719327 conn(0x5652f20ed680 msgr2=0x5652f20f33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f661cff9640 0 client.16128 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f6642020740 1 -- 172.21.15.104:0/2555719327 shutdown_connections 2024-06-09T22:23:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f6642020740 1 -- 172.21.15.104:0/2555719327 wait complete. 2024-06-09T22:23:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.642+0000 7f6642020740 20 client.16128 trim_cache size 0 max 16384 2024-06-09T22:23:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.657+0000 7f6642020740 1 Processor -- start 2024-06-09T22:23:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f6642020740 1 -- start start 2024-06-09T22:23:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f6642020740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1170 0x5652f20f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f6642020740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 0x5652f2113f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f6642020740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20fc350 0x5652f2114490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f6642020740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f66381aedc0 con 0x5652f20f1a90 2024-06-09T22:23:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f6642020740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f66381b0960 con 0x5652f20fc350 2024-06-09T22:23:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f6642020740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f66381af490 con 0x5652f20f1170 2024-06-09T22:23:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f663d0b9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 0x5652f2113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f663d0b9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 0x5652f2113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59972/0 (socket says 172.21.15.104:59972) 2024-06-09T22:23:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f663d0b9640 1 -- 172.21.15.104:0/3858105861 learned_addr learned my addr 172.21.15.104:0/3858105861 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f663d0b9640 1 -- 172.21.15.104:0/3858105861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1170 msgr2=0x5652f20f3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f663c8b8640 1 --2- 172.21.15.104:0/3858105861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1170 0x5652f20f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f663d0b9640 1 --2- 172.21.15.104:0/3858105861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1170 0x5652f20f3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f663d0b9640 1 -- 172.21.15.104:0/3858105861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20fc350 msgr2=0x5652f2114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f663d0b9640 1 --2- 172.21.15.104:0/3858105861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20fc350 0x5652f2114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f663d0b9640 1 -- 172.21.15.104:0/3858105861 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652f20f4070 con 0x5652f20f1a90 2024-06-09T22:23:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f663d0b9640 1 --2- 172.21.15.104:0/3858105861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 0x5652f2113f50 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f662802a9a0 tx=0x7f6628005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=372aaa61c7862237 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f661effd640 1 -- 172.21.15.104:0/3858105861 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628049070 con 0x5652f20f1a90 2024-06-09T22:23:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f661effd640 1 -- 172.21.15.104:0/3858105861 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6628005520 con 0x5652f20f1a90 2024-06-09T22:23:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.658+0000 7f661effd640 1 -- 172.21.15.104:0/3858105861 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628035040 con 0x5652f20f1a90 2024-06-09T22:23:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.659+0000 7f6642020740 1 -- 172.21.15.104:0/3858105861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 msgr2=0x5652f2113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.659+0000 7f6642020740 1 --2- 172.21.15.104:0/3858105861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 0x5652f2113f50 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f662802a9a0 tx=0x7f6628005a20 comp rx=0 tx=0).stop 2024-06-09T22:23:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.659+0000 7f6642020740 1 -- 172.21.15.104:0/3858105861 shutdown_connections 2024-06-09T22:23:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.659+0000 7f6642020740 1 --2- 172.21.15.104:0/3858105861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20fc350 0x5652f2114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.659+0000 7f6642020740 1 --2- 172.21.15.104:0/3858105861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 0x5652f2113f50 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.659+0000 7f6642020740 1 --2- 172.21.15.104:0/3858105861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20f1170 0x5652f20f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.659+0000 7f6642020740 1 -- 172.21.15.104:0/3858105861 >> 172.21.15.104:0/3858105861 conn(0x5652f20ed680 msgr2=0x5652f20f4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.659+0000 7f6642020740 1 -- 172.21.15.104:0/3858105861 shutdown_connections 2024-06-09T22:23:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.659+0000 7f6642020740 1 -- 172.21.15.104:0/3858105861 wait complete. 2024-06-09T22:23:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f6642020740 1 -- start start 2024-06-09T22:23:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f6642020740 1 Processor -- start 2024-06-09T22:23:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f6642020740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 0x5652f2104380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f6642020740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 0x5652f21048c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f6642020740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20fc350 0x5652f20f5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f6642020740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f66381b0230 con 0x5652f20f1a90 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f6642020740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f66381afb60 con 0x5652f20f1170 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f6642020740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f66381acd80 con 0x5652f20fc350 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f6642020740 10 client.0 _ll_register_callbacks cb 0x7f662c013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f663d0b9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 0x5652f21048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f663d0b9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 0x5652f21048c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59974/0 (socket says 172.21.15.104:59974) 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f663d0b9640 1 -- 172.21.15.104:0/206560467 learned_addr learned my addr 172.21.15.104:0/206560467 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.660+0000 7f663c8b8640 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 0x5652f2104380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f663c8b8640 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20fc350 0x5652f20f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f663d0b9640 1 -- 172.21.15.104:0/206560467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20fc350 msgr2=0x5652f20f5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f663d0b9640 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20fc350 0x5652f20f5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f663d0b9640 1 -- 172.21.15.104:0/206560467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 msgr2=0x5652f2104380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f663d0b9640 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 0x5652f2104380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f663d0b9640 1 -- 172.21.15.104:0/206560467 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652f20f5b60 con 0x5652f20f1a90 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f663d0b9640 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 0x5652f21048c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6628031550 tx=0x7f66280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e54ea539876d42d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f661cff9640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f661cff9640 1 -- 172.21.15.104:0/206560467 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628049040 con 0x5652f20f1a90 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f661cff9640 1 -- 172.21.15.104:0/206560467 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6628004b40 con 0x5652f20f1a90 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f6642020740 1 -- 172.21.15.104:0/206560467 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6628005520 con 0x5652f20f1a90 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f661cff9640 1 -- 172.21.15.104:0/206560467 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6628035040 con 0x5652f20f1a90 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f6642020740 10 client.16134 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f6642020740 1 -- 172.21.15.104:0/206560467 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5652f21765c0 con 0x5652f20f1a90 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.661+0000 7f661cff9640 1 -- 172.21.15.104:0/206560467 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6628005520 con 0x5652f20f1a90 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.662+0000 7f661cff9640 1 -- 172.21.15.104:0/206560467 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f662804c070 con 0x5652f20f1a90 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.662+0000 7f6642020740 10 client.16134 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.662+0000 7f6642020740 10 client.16134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.662+0000 7f6642020740 20 client.16134 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.662+0000 7f6642020740 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5652f2176c80 0x5652f21790a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.662+0000 7f6642020740 4 client.16134 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:23:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.662+0000 7f6642020740 1 -- 172.21.15.104:0/206560467 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5652f2062c20 con 0x5652f2176c80 2024-06-09T22:23:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.662+0000 7f663c8b8640 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5652f2176c80 0x5652f21790a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.662+0000 7f663c8b8640 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5652f2176c80 0x5652f21790a0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b54a63107834db6b server_cookie=1d3fba1910a73787 in_seq=0 out_seq=0 2024-06-09T22:23:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.662+0000 7f661cff9640 10 client.16134 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:11.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f661cff9640 1 -- 172.21.15.104:0/206560467 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5652f2062c20 con 0x5652f2176c80 2024-06-09T22:23:11.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f661cff9640 10 client.16134 handle_command_reply: tid=0 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f6642020740 1 client.16134 shutdown 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f6642020740 1 -- 172.21.15.104:0/206560467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 msgr2=0x5652f21048c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f6642020740 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 0x5652f21048c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6628031550 tx=0x7f66280046c0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f6642020740 1 -- 172.21.15.104:0/206560467 shutdown_connections 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f6642020740 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5652f2176c80 0x5652f21790a0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f6642020740 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652f20fc350 0x5652f20f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f6642020740 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5652f20f1a90 0x5652f21048c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f661cff9640 0 client.16134 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f6642020740 1 --2- 172.21.15.104:0/206560467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652f20f1170 0x5652f2104380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f6642020740 1 -- 172.21.15.104:0/206560467 >> 172.21.15.104:0/206560467 conn(0x5652f20ed680 msgr2=0x5652f20ebf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f6642020740 1 -- 172.21.15.104:0/206560467 shutdown_connections 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f6642020740 1 -- 172.21.15.104:0/206560467 wait complete. 2024-06-09T22:23:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.683+0000 7f6642020740 20 client.16134 trim_cache size 0 max 16384 2024-06-09T22:23:11.685 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:23:11.685 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:23:11.685 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:23:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.684+0000 7f663f31d640 1 -- 172.21.15.104:0/828989039 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f66200a2da0 msgr2=0x7f66200a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.684+0000 7f663f31d640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f66200a2da0 0x7f66200a5260 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f66300988f0 tx=0x7f663008f040 comp rx=0 tx=0).stop 2024-06-09T22:23:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.684+0000 7f663f31d640 1 -- 172.21.15.104:0/828989039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f663810c020 msgr2=0x7f663819cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.684+0000 7f663f31d640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f663810c020 0x7f663819cab0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f663000e9d0 tx=0x7f663000ee90 comp rx=0 tx=0).stop 2024-06-09T22:23:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.684+0000 7f663f31d640 1 -- 172.21.15.104:0/828989039 shutdown_connections 2024-06-09T22:23:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.684+0000 7f663f31d640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f66200a2da0 0x7f66200a5260 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.684+0000 7f663f31d640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f663810c020 0x7f663819cab0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.684+0000 7f663f31d640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6638103570 0x7f6638198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.684+0000 7f663f31d640 1 --2- 172.21.15.104:0/828989039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6638102c50 0x7f66381981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.684+0000 7f663f31d640 1 -- 172.21.15.104:0/828989039 >> 172.21.15.104:0/828989039 conn(0x7f6638076c20 msgr2=0x7f66380fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:11.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.684+0000 7f663f31d640 1 -- 172.21.15.104:0/828989039 shutdown_connections 2024-06-09T22:23:11.687 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:11.684+0000 7f663f31d640 1 -- 172.21.15.104:0/828989039 wait complete. 2024-06-09T22:23:11.716 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 8.01 MB/sec warmup 17 sec latency 1242.880 ms 2024-06-09T22:23:12.065 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '6f4b8fc4', quiesce took 16.0 seconds. Will release after: 14.0 2024-06-09T22:23:12.716 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 7.56 MB/sec warmup 18 sec latency 2242.954 ms 2024-06-09T22:23:12.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:12 smithi153 ceph-mon[36599]: pgmap v70: 129 pgs: 129 active+clean; 33 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 3.5 MiB/s wr, 208 op/s 2024-06-09T22:23:13.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:12 smithi104 ceph-mon[32389]: pgmap v70: 129 pgs: 129 active+clean; 33 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 3.5 MiB/s wr, 208 op/s 2024-06-09T22:23:13.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:12 smithi194 ceph-mon[36856]: pgmap v70: 129 pgs: 129 active+clean; 33 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 3.5 MiB/s wr, 208 op/s 2024-06-09T22:23:13.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.279+0000 7f1f626ca700 1 -- 172.21.15.104:0/4266067530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f5c102ef0 msgr2=0x7f1f5c10f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:13.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.279+0000 7f1f626ca700 1 --2- 172.21.15.104:0/4266067530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f5c102ef0 0x7f1f5c10f3a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400b0d0 tx=0x7f1f5400b3e0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.279+0000 7f1f626ca700 1 -- 172.21.15.104:0/4266067530 shutdown_connections 2024-06-09T22:23:13.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.279+0000 7f1f626ca700 1 --2- 172.21.15.104:0/4266067530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f5c102ef0 0x7f1f5c10f3a0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.279+0000 7f1f626ca700 1 --2- 172.21.15.104:0/4266067530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f5c1085d0 0x7f1f5c1089b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.279+0000 7f1f626ca700 1 --2- 172.21.15.104:0/4266067530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f5c102530 0x7f1f5c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.279+0000 7f1f626ca700 1 -- 172.21.15.104:0/4266067530 >> 172.21.15.104:0/4266067530 conn(0x7f1f5c079640 msgr2=0x7f1f5c079a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.279+0000 7f1f626ca700 1 -- 172.21.15.104:0/4266067530 shutdown_connections 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.279+0000 7f1f626ca700 1 -- 172.21.15.104:0/4266067530 wait complete. 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.280+0000 7f1f626ca700 1 Processor -- start 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.280+0000 7f1f626ca700 1 -- start start 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.280+0000 7f1f626ca700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f5c102530 0x7f1f5c19e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.280+0000 7f1f626ca700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f5c102ef0 0x7f1f5c19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.280+0000 7f1f626ca700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f5c1085d0 0x7f1f5c1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.280+0000 7f1f626ca700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c112840 con 0x7f1f5c1085d0 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.280+0000 7f1f626ca700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1126c0 con 0x7f1f5c102530 2024-06-09T22:23:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.280+0000 7f1f626ca700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1129c0 con 0x7f1f5c102ef0 2024-06-09T22:23:13.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.280+0000 7f1f60c67700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f5c1085d0 0x7f1f5c1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.280+0000 7f1f53fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f5c102ef0 0x7f1f5c19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.280+0000 7f1f60c67700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f5c1085d0 0x7f1f5c1989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59988/0 (socket says 172.21.15.104:59988) 2024-06-09T22:23:13.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f60c67700 1 -- 172.21.15.104:0/2672765908 learned_addr learned my addr 172.21.15.104:0/2672765908 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:13.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f5c102530 0x7f1f5c19e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f53fff700 1 -- 172.21.15.104:0/2672765908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f5c102530 msgr2=0x7f1f5c19e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f53fff700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f5c102530 0x7f1f5c19e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f53fff700 1 -- 172.21.15.104:0/2672765908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f5c1085d0 msgr2=0x7f1f5c1989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f53fff700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f5c1085d0 0x7f1f5c1989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f53fff700 1 -- 172.21.15.104:0/2672765908 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5c199140 con 0x7f1f5c102ef0 2024-06-09T22:23:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f60c67700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f5c1085d0 0x7f1f5c1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f5c102530 0x7f1f5c19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f53fff700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f5c102ef0 0x7f1f5c19ed10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400e9f0 tx=0x7f1f4400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f59ffb700 1 -- 172.21.15.104:0/2672765908 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44004220 con 0x7f1f5c102ef0 2024-06-09T22:23:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f626ca700 1 -- 172.21.15.104:0/2672765908 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f5c199430 con 0x7f1f5c102ef0 2024-06-09T22:23:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.281+0000 7f1f59ffb700 1 -- 172.21.15.104:0/2672765908 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f44004d10 con 0x7f1f5c102ef0 2024-06-09T22:23:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.282+0000 7f1f59ffb700 1 -- 172.21.15.104:0/2672765908 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44005020 con 0x7f1f5c102ef0 2024-06-09T22:23:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.282+0000 7f1f626ca700 1 -- 172.21.15.104:0/2672765908 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f5c1a56a0 con 0x7f1f5c102ef0 2024-06-09T22:23:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.284+0000 7f1f59ffb700 1 -- 172.21.15.104:0/2672765908 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f1f44004750 con 0x7f1f5c102ef0 2024-06-09T22:23:13.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.284+0000 7f1f59ffb700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1f480a2e10 0x7f1f480a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.285+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1f480a2e10 0x7f1f480a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.285+0000 7f1f59ffb700 1 -- 172.21.15.104:0/2672765908 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1f440cf630 con 0x7f1f5c102ef0 2024-06-09T22:23:13.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.285+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1f480a2e10 0x7f1f480a52d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c079bc0 tx=0x7f1f4c0073d0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:13.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.285+0000 7f1f6cf4db80 1 Processor -- start 2024-06-09T22:23:13.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.285+0000 7f1f6cf4db80 1 -- start start 2024-06-09T22:23:13.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f6cf4db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c2f60 0x562d842c3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f6cf4db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c3880 0x562d842bd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f6cf4db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 0x562d842bdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f6cf4db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562d841aff60 con 0x562d842bd7a0 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f6cf4db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562d8420e850 con 0x562d842c2f60 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f6cf4db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562d84176460 con 0x562d842c3880 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f60c67700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 0x562d842bdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f5bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c2f60 0x562d842c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f60c67700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 0x562d842bdbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59992/0 (socket says 172.21.15.104:59992) 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f5bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c2f60 0x562d842c3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:47564/0 (socket says 172.21.15.104:47564) 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f60c67700 1 -- 172.21.15.104:0/2655562088 learned_addr learned my addr 172.21.15.104:0/2655562088 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f53fff700 1 --2- 172.21.15.104:0/2655562088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c3880 0x562d842bd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f60c67700 1 -- 172.21.15.104:0/2655562088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c3880 msgr2=0x562d842bd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f60c67700 1 --2- 172.21.15.104:0/2655562088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c3880 0x562d842bd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f60c67700 1 -- 172.21.15.104:0/2655562088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c2f60 msgr2=0x562d842c3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f60c67700 1 --2- 172.21.15.104:0/2655562088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c2f60 0x562d842c3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f60c67700 1 -- 172.21.15.104:0/2655562088 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d8415d8d0 con 0x562d842bd7a0 2024-06-09T22:23:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/2655562088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c2f60 0x562d842c3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f60c67700 1 --2- 172.21.15.104:0/2655562088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 0x562d842bdbc0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1f54009d90 tx=0x7f1f54009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b43bf7e6c957076 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.286+0000 7f1f527fc700 1 -- 172.21.15.104:0/2655562088 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f54004c50 con 0x562d842bd7a0 2024-06-09T22:23:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.287+0000 7f1f527fc700 1 -- 172.21.15.104:0/2655562088 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f54009460 con 0x562d842bd7a0 2024-06-09T22:23:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.287+0000 7f1f527fc700 1 -- 172.21.15.104:0/2655562088 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f54005330 con 0x562d842bd7a0 2024-06-09T22:23:13.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.287+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/2655562088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 msgr2=0x562d842bdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:13.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.287+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/2655562088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 0x562d842bdbc0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1f54009d90 tx=0x7f1f54009dc0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.287+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/2655562088 shutdown_connections 2024-06-09T22:23:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.287+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/2655562088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 0x562d842bdbc0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.287+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/2655562088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c2f60 0x562d842c3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.287+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/2655562088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c3880 0x562d842bd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.287+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/2655562088 >> 172.21.15.104:0/2655562088 conn(0x562d841ea080 msgr2=0x562d842b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.287+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/2655562088 shutdown_connections 2024-06-09T22:23:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.287+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/2655562088 wait complete. 2024-06-09T22:23:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f6cf4db80 1 -- start start 2024-06-09T22:23:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f6cf4db80 1 Processor -- start 2024-06-09T22:23:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f6cf4db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842bd7a0 0x562d842c9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f6cf4db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c2f60 0x562d842c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f6cf4db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 0x562d842c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f6cf4db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562d841aefc0 con 0x562d842c2f60 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f6cf4db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562d8420a810 con 0x562d842c3880 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f6cf4db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562d842ce610 con 0x562d842bd7a0 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f53fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c2f60 0x562d842c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f6cf4db80 10 client.0 _ll_register_callbacks cb 0x562d841a45b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f60c67700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 0x562d842c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f60c67700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 0x562d842c6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:47574/0 (socket says 172.21.15.104:47574) 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f53fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c2f60 0x562d842c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59996/0 (socket says 172.21.15.104:59996) 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f60c67700 1 -- 172.21.15.104:0/1194219867 learned_addr learned my addr 172.21.15.104:0/1194219867 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.288+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842bd7a0 0x562d842c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f60c67700 1 -- 172.21.15.104:0/1194219867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842bd7a0 msgr2=0x562d842c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f60c67700 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842bd7a0 0x562d842c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f60c67700 1 -- 172.21.15.104:0/1194219867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c2f60 msgr2=0x562d842c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f60c67700 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c2f60 0x562d842c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f60c67700 1 -- 172.21.15.104:0/1194219867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d841d4750 con 0x562d842c3880 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f53fff700 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c2f60 0x562d842c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f60c67700 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 0x562d842c6e60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1f54007780 tx=0x7f1f540077b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8c84bb6c8a70092 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f3b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f3b7fe700 1 -- 172.21.15.104:0/1194219867 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5402a070 con 0x562d842c3880 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1194219867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f54009460 con 0x562d842c3880 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f3b7fe700 1 -- 172.21.15.104:0/1194219867 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1f540052b0 con 0x562d842c3880 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f3b7fe700 1 -- 172.21.15.104:0/1194219867 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1f54038050 con 0x562d842c3880 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f6cf4db80 10 client.25574 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1194219867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562d842d8df0 con 0x562d842c3880 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.289+0000 7f1f3b7fe700 1 -- 172.21.15.104:0/1194219867 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f54009460 con 0x562d842c3880 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.290+0000 7f1f3b7fe700 1 -- 172.21.15.104:0/1194219867 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f1f54011070 con 0x562d842c3880 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.290+0000 7f1f6cf4db80 10 client.25574 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.290+0000 7f1f6cf4db80 10 client.25574 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:23:13.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.290+0000 7f1f6cf4db80 20 client.25574 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.290+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x562d842db2c0 0x562d842dd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.290+0000 7f1f6cf4db80 4 client.25574 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.290+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1194219867 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d8421bd50 con 0x562d842db2c0 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.290+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x562d842db2c0 0x562d842dd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.290+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x562d842db2c0 0x562d842dd6b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=87734220a22a1560 server_cookie=ecffb119d98d44c1 in_seq=0 out_seq=0 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.290+0000 7f1f3b7fe700 10 client.25574 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f3b7fe700 1 -- 172.21.15.104:0/1194219867 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x562d8421bd50 con 0x562d842db2c0 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f3b7fe700 10 client.25574 handle_command_reply: tid=0 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f6cf4db80 1 client.25574 shutdown 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1194219867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 msgr2=0x562d842c6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 0x562d842c6e60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1f54007780 tx=0x7f1f540077b0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1194219867 shutdown_connections 2024-06-09T22:23:13.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x562d842db2c0 0x562d842dd6b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 0x562d842c6e60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f3b7fe700 0 client.25574 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:23:13.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842bd7a0 0x562d842c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/1194219867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c2f60 0x562d842c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1194219867 >> 172.21.15.104:0/1194219867 conn(0x562d841ea080 msgr2=0x562d842b7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:13.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1194219867 shutdown_connections 2024-06-09T22:23:13.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1194219867 wait complete. 2024-06-09T22:23:13.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.291+0000 7f1f6cf4db80 20 client.25574 trim_cache size 0 max 16384 2024-06-09T22:23:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.306+0000 7f1f6cf4db80 1 Processor -- start 2024-06-09T22:23:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f6cf4db80 1 -- start start 2024-06-09T22:23:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f6cf4db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842bd7a0 0x562d842bd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f6cf4db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c2f60 0x562d842c5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f6cf4db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c3880 0x562d842b7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f6cf4db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1ac100 con 0x562d842c3880 2024-06-09T22:23:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f6cf4db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1adca0 con 0x562d842bd7a0 2024-06-09T22:23:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f6cf4db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1ac7d0 con 0x562d842c2f60 2024-06-09T22:23:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f5bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842bd7a0 0x562d842bd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f5bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842bd7a0 0x562d842bd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:47586/0 (socket says 172.21.15.104:47586) 2024-06-09T22:23:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f60c67700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c3880 0x562d842b7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f5bfff700 1 -- 172.21.15.104:0/3956980854 learned_addr learned my addr 172.21.15.104:0/3956980854 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f53fff700 1 --2- 172.21.15.104:0/3956980854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c2f60 0x562d842c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f5bfff700 1 -- 172.21.15.104:0/3956980854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c2f60 msgr2=0x562d842c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/3956980854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c2f60 0x562d842c5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f5bfff700 1 -- 172.21.15.104:0/3956980854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c3880 msgr2=0x562d842b7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/3956980854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c3880 0x562d842b7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f5bfff700 1 -- 172.21.15.104:0/3956980854 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d842c4e90 con 0x562d842bd7a0 2024-06-09T22:23:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.307+0000 7f1f60c67700 1 --2- 172.21.15.104:0/3956980854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c3880 0x562d842b7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.308+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/3956980854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842bd7a0 0x562d842bd260 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c007fd0 tx=0x7f1f4c013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fd6b2eb71a099a9 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.308+0000 7f1f51ffb700 1 -- 172.21.15.104:0/3956980854 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c02a5b0 con 0x562d842bd7a0 2024-06-09T22:23:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.308+0000 7f1f51ffb700 1 -- 172.21.15.104:0/3956980854 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f4c011e40 con 0x562d842bd7a0 2024-06-09T22:23:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.308+0000 7f1f51ffb700 1 -- 172.21.15.104:0/3956980854 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c01fd20 con 0x562d842bd7a0 2024-06-09T22:23:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.308+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/3956980854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842bd7a0 msgr2=0x562d842bd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.308+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/3956980854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842bd7a0 0x562d842bd260 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c007fd0 tx=0x7f1f4c013000 comp rx=0 tx=0).stop 2024-06-09T22:23:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.308+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/3956980854 shutdown_connections 2024-06-09T22:23:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.308+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/3956980854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842c3880 0x562d842b7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.308+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/3956980854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842bd7a0 0x562d842bd260 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.308+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/3956980854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c2f60 0x562d842c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.309+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/3956980854 >> 172.21.15.104:0/3956980854 conn(0x562d841ea080 msgr2=0x562d842c5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.309+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/3956980854 shutdown_connections 2024-06-09T22:23:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.309+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/3956980854 wait complete. 2024-06-09T22:23:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.309+0000 7f1f6cf4db80 1 -- start start 2024-06-09T22:23:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.309+0000 7f1f6cf4db80 1 Processor -- start 2024-06-09T22:23:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.309+0000 7f1f6cf4db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 0x562d842dd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.309+0000 7f1f6cf4db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c2f60 0x562d842ddc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.309+0000 7f1f6cf4db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 0x562d842de180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.309+0000 7f1f6cf4db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1ad570 con 0x562d842bd7a0 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.309+0000 7f1f6cf4db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1acea0 con 0x562d842c3880 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.309+0000 7f1f6cf4db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c1aa080 con 0x562d842c2f60 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f6cf4db80 10 client.0 _ll_register_callbacks cb 0x562d841a45b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f5bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 0x562d842dd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f53fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c2f60 0x562d842ddc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f60c67700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 0x562d842de180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f5bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 0x562d842dd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60022/0 (socket says 172.21.15.104:60022) 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f5bfff700 1 -- 172.21.15.104:0/1752965603 learned_addr learned my addr 172.21.15.104:0/1752965603 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f53fff700 1 -- 172.21.15.104:0/1752965603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 msgr2=0x562d842de180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f53fff700 1 --2- 172.21.15.104:0/1752965603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 0x562d842de180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f53fff700 1 -- 172.21.15.104:0/1752965603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 msgr2=0x562d842dd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f53fff700 1 --2- 172.21.15.104:0/1752965603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 0x562d842dd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f53fff700 1 -- 172.21.15.104:0/1752965603 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d84343e50 con 0x562d842c2f60 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f60c67700 1 --2- 172.21.15.104:0/1752965603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 0x562d842de180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f53fff700 1 --2- 172.21.15.104:0/1752965603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c2f60 0x562d842ddc40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1f440cca10 tx=0x7f1f4409bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5b30d3a658a2e1a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f3b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f3b7fe700 1 -- 172.21.15.104:0/1752965603 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f440b9060 con 0x562d842c2f60 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f3b7fe700 1 -- 172.21.15.104:0/1752965603 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1f44004a40 con 0x562d842c2f60 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.310+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1752965603 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562d842162d0 con 0x562d842c2f60 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.311+0000 7f1f3b7fe700 1 -- 172.21.15.104:0/1752965603 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1f4401c040 con 0x562d842c2f60 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.311+0000 7f1f6cf4db80 10 client.25690 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.311+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1752965603 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562d84345170 con 0x562d842c2f60 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.311+0000 7f1f3b7fe700 1 -- 172.21.15.104:0/1752965603 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562d842162d0 con 0x562d842c2f60 2024-06-09T22:23:13.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.311+0000 7f1f3b7fe700 1 -- 172.21.15.104:0/1752965603 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f1f4409ba40 con 0x562d842c2f60 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.311+0000 7f1f6cf4db80 10 client.25690 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.311+0000 7f1f6cf4db80 10 client.25690 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.311+0000 7f1f6cf4db80 20 client.25690 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.311+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/1752965603 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x562d84345800 0x562d84347c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.311+0000 7f1f6cf4db80 4 client.25690 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.311+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1752965603 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562d841c9f50 con 0x562d84345800 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.311+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/1752965603 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x562d84345800 0x562d84347c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f3b7fe700 10 client.25690 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f5bfff700 1 --2- 172.21.15.104:0/1752965603 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x562d84345800 0x562d84347c20 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6933fbc402ddc6d0 server_cookie=d056a81e91179a82 in_seq=0 out_seq=0 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f3b7fe700 1 -- 172.21.15.104:0/1752965603 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562d841c9f50 con 0x562d84345800 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f3b7fe700 10 client.25690 handle_command_reply: tid=0 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f6cf4db80 1 client.25690 shutdown 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1752965603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c2f60 msgr2=0x562d842ddc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:13.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/1752965603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c2f60 0x562d842ddc40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1f440cca10 tx=0x7f1f4409bee0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1752965603 shutdown_connections 2024-06-09T22:23:13.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/1752965603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562d842bd7a0 0x562d842dd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/1752965603 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x562d84345800 0x562d84347c20 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/1752965603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d842c3880 0x562d842de180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f3b7fe700 0 client.25690 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:23:13.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f6cf4db80 1 --2- 172.21.15.104:0/1752965603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d842c2f60 0x562d842ddc40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1752965603 >> 172.21.15.104:0/1752965603 conn(0x562d841ea080 msgr2=0x562d842c89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:13.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1752965603 shutdown_connections 2024-06-09T22:23:13.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f6cf4db80 1 -- 172.21.15.104:0/1752965603 wait complete. 2024-06-09T22:23:13.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.312+0000 7f1f6cf4db80 20 client.25690 trim_cache size 0 max 16384 2024-06-09T22:23:13.316 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:23:13.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.313+0000 7f1f626ca700 1 -- 172.21.15.104:0/2672765908 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1f480a2e10 msgr2=0x7f1f480a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:13.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.313+0000 7f1f626ca700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1f480a2e10 0x7f1f480a52d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c079bc0 tx=0x7f1f4c0073d0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.313+0000 7f1f626ca700 1 -- 172.21.15.104:0/2672765908 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f5c102ef0 msgr2=0x7f1f5c19ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:13.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.313+0000 7f1f626ca700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f5c102ef0 0x7f1f5c19ed10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400e9f0 tx=0x7f1f4400ed00 comp rx=0 tx=0).stop 2024-06-09T22:23:13.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.313+0000 7f1f626ca700 1 -- 172.21.15.104:0/2672765908 shutdown_connections 2024-06-09T22:23:13.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.313+0000 7f1f626ca700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1f480a2e10 0x7f1f480a52d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.313+0000 7f1f626ca700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f5c1085d0 0x7f1f5c1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.313+0000 7f1f626ca700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f5c102530 0x7f1f5c19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.313+0000 7f1f626ca700 1 --2- 172.21.15.104:0/2672765908 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f5c102ef0 0x7f1f5c19ed10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:13.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.313+0000 7f1f626ca700 1 -- 172.21.15.104:0/2672765908 >> 172.21.15.104:0/2672765908 conn(0x7f1f5c079640 msgr2=0x7f1f5c1061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:13.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.313+0000 7f1f626ca700 1 -- 172.21.15.104:0/2672765908 shutdown_connections 2024-06-09T22:23:13.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:13.313+0000 7f1f626ca700 1 -- 172.21.15.104:0/2672765908 wait complete. 2024-06-09T22:23:13.716 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 7.16 MB/sec warmup 19 sec latency 3243.029 ms 2024-06-09T22:23:13.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:13 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:23:13.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:13 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:23:13.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:13 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:23:13.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:13 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:23:14.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:13 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:23:14.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:13 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:23:14.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:13 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:23:14.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:13 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:23:14.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:13 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:23:14.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:13 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:23:14.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:13 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:23:14.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:13 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:23:14.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-09T22:23:14.361 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:23:14.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 6.81 MB/sec warmup 20 sec latency 4243.141 ms 2024-06-09T22:23:15.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 6.48 MB/sec warmup 21 sec latency 5243.196 ms 2024-06-09T22:23:15.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:15 smithi104 ceph-mon[32389]: pgmap v71: 129 pgs: 129 active+clean; 29 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 3.5 MiB/s wr, 223 op/s 2024-06-09T22:23:15.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:15 smithi194 ceph-mon[36856]: pgmap v71: 129 pgs: 129 active+clean; 29 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 3.5 MiB/s wr, 223 op/s 2024-06-09T22:23:15.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:15 smithi153 ceph-mon[36599]: pgmap v71: 129 pgs: 129 active+clean; 29 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 3.5 MiB/s wr, 223 op/s 2024-06-09T22:23:16.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.584+0000 7f7fa1a63700 1 -- 172.21.15.104:0/2416735780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9c107870 msgr2=0x7f7f9c10f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:16.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.584+0000 7f7fa1a63700 1 --2- 172.21.15.104:0/2416735780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9c107870 0x7f7f9c10f3f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400b0d0 tx=0x7f7f9400b3e0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.584+0000 7f7fa1a63700 1 -- 172.21.15.104:0/2416735780 shutdown_connections 2024-06-09T22:23:16.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.584+0000 7f7fa1a63700 1 --2- 172.21.15.104:0/2416735780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9c107870 0x7f7f9c10f3f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.584+0000 7f7fa1a63700 1 --2- 172.21.15.104:0/2416735780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f9c1027a0 0x7f7f9c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.584+0000 7f7fa1a63700 1 --2- 172.21.15.104:0/2416735780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9c103150 0x7f7f9c107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.584+0000 7f7fa1a63700 1 -- 172.21.15.104:0/2416735780 >> 172.21.15.104:0/2416735780 conn(0x7f7f9c079520 msgr2=0x7f7f9c079930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:16.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.584+0000 7f7fa1a63700 1 -- 172.21.15.104:0/2416735780 shutdown_connections 2024-06-09T22:23:16.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.584+0000 7f7fa1a63700 1 -- 172.21.15.104:0/2416735780 wait complete. 2024-06-09T22:23:16.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.585+0000 7f7fa1a63700 1 Processor -- start 2024-06-09T22:23:16.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.585+0000 7f7fa1a63700 1 -- start start 2024-06-09T22:23:16.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.585+0000 7f7fa1a63700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f9c1027a0 0x7f7f9c19e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.585+0000 7f7fa1a63700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9c103150 0x7f7f9c19ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.585+0000 7f7fa1a63700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9c107870 0x7f7f9c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.585+0000 7f7fa1a63700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c112790 con 0x7f7f9c103150 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.585+0000 7f7fa1a63700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c112610 con 0x7f7f9c107870 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.585+0000 7f7fa1a63700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c112910 con 0x7f7f9c1027a0 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f9affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9c103150 0x7f7f9c19ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f9affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9c103150 0x7f7f9c19ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46744/0 (socket says 172.21.15.104:46744) 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f9affd700 1 -- 172.21.15.104:0/3617959448 learned_addr learned my addr 172.21.15.104:0/3617959448 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f9b7fe700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f9c1027a0 0x7f7f9c19e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f9bfff700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9c107870 0x7f7f9c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f9affd700 1 -- 172.21.15.104:0/3617959448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f9c1027a0 msgr2=0x7f7f9c19e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f9affd700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f9c1027a0 0x7f7f9c19e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f9affd700 1 -- 172.21.15.104:0/3617959448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9c107870 msgr2=0x7f7f9c198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f9affd700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9c107870 0x7f7f9c198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f9affd700 1 -- 172.21.15.104:0/3617959448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9c1990f0 con 0x7f7f9c103150 2024-06-09T22:23:16.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f9affd700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9c103150 0x7f7f9c19ecc0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00d950 tx=0x7f7f8c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:16.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f98ff9700 1 -- 172.21.15.104:0/3617959448 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c016070 con 0x7f7f9c103150 2024-06-09T22:23:16.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7fa1a63700 1 -- 172.21.15.104:0/3617959448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f9c1993e0 con 0x7f7f9c103150 2024-06-09T22:23:16.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f98ff9700 1 -- 172.21.15.104:0/3617959448 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f8c00fd70 con 0x7f7f9c103150 2024-06-09T22:23:16.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7f98ff9700 1 -- 172.21.15.104:0/3617959448 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c00e610 con 0x7f7f9c103150 2024-06-09T22:23:16.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.586+0000 7f7fa1a63700 1 -- 172.21.15.104:0/3617959448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f9c1a5650 con 0x7f7f9c103150 2024-06-09T22:23:16.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.590+0000 7f7f98ff9700 1 -- 172.21.15.104:0/3617959448 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7f8c019070 con 0x7f7f9c103150 2024-06-09T22:23:16.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7fac2e6b80 1 Processor -- start 2024-06-09T22:23:16.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7fac2e6b80 1 -- start start 2024-06-09T22:23:16.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7fac2e6b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badd01a70 0x562badd01e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7fac2e6b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd02390 0x562badcfbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7fac2e6b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 0x562badcfc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7fac2e6b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562badc49810 con 0x562badcfc330 2024-06-09T22:23:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7fac2e6b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562badbf1720 con 0x562badd02390 2024-06-09T22:23:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7fac2e6b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562badc457d0 con 0x562badd01a70 2024-06-09T22:23:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7f98ff9700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f840a2e10 0x7f7f840a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7f9affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 0x562badcfc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7f9bfff700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f840a2e10 0x7f7f840a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7f9affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 0x562badcfc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46750/0 (socket says 172.21.15.104:46750) 2024-06-09T22:23:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7f9affd700 1 -- 172.21.15.104:0/1373836406 learned_addr learned my addr 172.21.15.104:0/1373836406 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.591+0000 7f7f9bfff700 1 --2- 172.21.15.104:0/1373836406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badd01a70 0x562badd01e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.592+0000 7f7f98ff9700 1 -- 172.21.15.104:0/3617959448 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7f8c0cf230 con 0x7f7f9c103150 2024-06-09T22:23:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.592+0000 7f7f9b7fe700 1 --2- 172.21.15.104:0/1373836406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd02390 0x562badcfbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.592+0000 7f7f9affd700 1 -- 172.21.15.104:0/1373836406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badd01a70 msgr2=0x562badd01e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.592+0000 7f7f9affd700 1 --2- 172.21.15.104:0/1373836406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badd01a70 0x562badd01e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.592+0000 7f7f9affd700 1 -- 172.21.15.104:0/1373836406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd02390 msgr2=0x562badcfbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.592+0000 7f7f9affd700 1 --2- 172.21.15.104:0/1373836406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd02390 0x562badcfbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.592+0000 7f7f9affd700 1 -- 172.21.15.104:0/1373836406 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562badbe7f50 con 0x562badcfc330 2024-06-09T22:23:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.592+0000 7f7f9bfff700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f840a2e10 0x7f7f840a52d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7f940060d0 tx=0x7f7f94009c40 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.592+0000 7f7f9affd700 1 --2- 172.21.15.104:0/1373836406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 0x562badcfc750 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c010040 tx=0x7f7f8c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6e1794cbe56825 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.592+0000 7f7f81ffb700 1 -- 172.21.15.104:0/1373836406 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c020070 con 0x562badcfc330 2024-06-09T22:23:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.592+0000 7f7f81ffb700 1 -- 172.21.15.104:0/1373836406 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f8c09b3a0 con 0x562badcfc330 2024-06-09T22:23:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.592+0000 7f7f81ffb700 1 -- 172.21.15.104:0/1373836406 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c010710 con 0x562badcfc330 2024-06-09T22:23:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.593+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/1373836406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 msgr2=0x562badcfc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.593+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/1373836406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 0x562badcfc750 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c010040 tx=0x7f7f8c000bc0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.593+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/1373836406 shutdown_connections 2024-06-09T22:23:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.593+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/1373836406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 0x562badcfc750 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.593+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/1373836406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badd01a70 0x562badd01e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.593+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/1373836406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd02390 0x562badcfbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.593+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/1373836406 >> 172.21.15.104:0/1373836406 conn(0x562badc22cc0 msgr2=0x562badcf2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.593+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/1373836406 shutdown_connections 2024-06-09T22:23:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.593+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/1373836406 wait complete. 2024-06-09T22:23:16.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.593+0000 7f7fac2e6b80 1 -- start start 2024-06-09T22:23:16.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7fac2e6b80 1 Processor -- start 2024-06-09T22:23:16.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7fac2e6b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badcfc330 0x562badd08160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7fac2e6b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd01a70 0x562badd05460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7fac2e6b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd02390 0x562badd059a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7fac2e6b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562badd08d70 con 0x562badd02390 2024-06-09T22:23:16.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7fac2e6b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562badbe9bf0 con 0x562badd01a70 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7fac2e6b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562badd08ef0 con 0x562badcfc330 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7fac2e6b80 10 client.0 _ll_register_callbacks cb 0x562badbe3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7f9bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd02390 0x562badd059a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7f9bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd02390 0x562badd059a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46762/0 (socket says 172.21.15.104:46762) 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7f9bfff700 1 -- 172.21.15.104:0/3527979035 learned_addr learned my addr 172.21.15.104:0/3527979035 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7f9b7fe700 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badcfc330 0x562badd08160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7f9affd700 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd01a70 0x562badd05460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7f9bfff700 1 -- 172.21.15.104:0/3527979035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badcfc330 msgr2=0x562badd08160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7f9bfff700 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badcfc330 0x562badd08160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7f9bfff700 1 -- 172.21.15.104:0/3527979035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd01a70 msgr2=0x562badd05460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7f9bfff700 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd01a70 0x562badd05460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.594+0000 7f7f9bfff700 1 -- 172.21.15.104:0/3527979035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562badb988d0 con 0x562badd02390 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7f9bfff700 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd02390 0x562badd059a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400b0d0 tx=0x7f7f94026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=101346e982dc4e23 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7f7affd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7f7affd700 1 -- 172.21.15.104:0/3527979035 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94007e30 con 0x562badd02390 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7f7affd700 1 -- 172.21.15.104:0/3527979035 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7f9401e460 con 0x562badd02390 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3527979035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f8c09b3a0 con 0x562badd02390 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7f7affd700 1 -- 172.21.15.104:0/3527979035 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7f94037050 con 0x562badd02390 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7fac2e6b80 10 client.16164 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3527979035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562badd136f0 con 0x562badd02390 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7f7affd700 1 -- 172.21.15.104:0/3527979035 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f8c09b3a0 con 0x562badd02390 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7f7affd700 1 -- 172.21.15.104:0/3527979035 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f7f94011070 con 0x562badd02390 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7fac2e6b80 10 client.16164 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7fac2e6b80 10 client.16164 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7fac2e6b80 20 client.16164 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.595+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x562badd15d30 0x562badd18150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.596+0000 7f7fac2e6b80 4 client.16164 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.596+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3527979035 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562badc56d10 con 0x562badd15d30 2024-06-09T22:23:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.596+0000 7f7f9b7fe700 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x562badd15d30 0x562badd18150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.596+0000 7f7f7affd700 10 client.16164 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.596+0000 7f7f9b7fe700 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x562badd15d30 0x562badd18150 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5029a95efdb7eb49 server_cookie=74dd2abef4d32dd in_seq=0 out_seq=0 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7f7affd700 1 -- 172.21.15.104:0/3527979035 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x562badc56d10 con 0x562badd15d30 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7f7affd700 10 client.16164 handle_command_reply: tid=0 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7fac2e6b80 1 client.16164 shutdown 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3527979035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd02390 msgr2=0x562badd059a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd02390 0x562badd059a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400b0d0 tx=0x7f7f94026000 comp rx=0 tx=0).stop 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3527979035 shutdown_connections 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x562badd15d30 0x562badd18150 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7f7affd700 0 client.16164 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd02390 0x562badd059a0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badcfc330 0x562badd08160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/3527979035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd01a70 0x562badd05460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3527979035 >> 172.21.15.104:0/3527979035 conn(0x562badc22cc0 msgr2=0x562badcf69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3527979035 shutdown_connections 2024-06-09T22:23:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.597+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3527979035 wait complete. 2024-06-09T22:23:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.598+0000 7f7fac2e6b80 20 client.16164 trim_cache size 0 max 16384 2024-06-09T22:23:16.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.626+0000 7f7fac2e6b80 1 Processor -- start 2024-06-09T22:23:16.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.626+0000 7f7fac2e6b80 1 -- start start 2024-06-09T22:23:16.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.626+0000 7f7fac2e6b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 0x562badcf70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.626+0000 7f7fac2e6b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd01a70 0x562badd1da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.626+0000 7f7fac2e6b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badd02390 0x562badd1df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.626+0000 7f7fac2e6b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c1ac2c0 con 0x562badcfc330 2024-06-09T22:23:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.626+0000 7f7fac2e6b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c1ade60 con 0x562badd01a70 2024-06-09T22:23:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.626+0000 7f7fac2e6b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c1ac990 con 0x562badd02390 2024-06-09T22:23:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.626+0000 7f7f9b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 0x562badcf70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.626+0000 7f7f9b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 0x562badcf70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46776/0 (socket says 172.21.15.104:46776) 2024-06-09T22:23:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.626+0000 7f7f9b7fe700 1 -- 172.21.15.104:0/3041000312 learned_addr learned my addr 172.21.15.104:0/3041000312 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.627+0000 7f7f9affd700 1 --2- 172.21.15.104:0/3041000312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd01a70 0x562badd1da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.627+0000 7f7f9bfff700 1 --2- 172.21.15.104:0/3041000312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badd02390 0x562badd1df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.627+0000 7f7f9b7fe700 1 -- 172.21.15.104:0/3041000312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badd02390 msgr2=0x562badd1df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.627+0000 7f7f9b7fe700 1 --2- 172.21.15.104:0/3041000312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badd02390 0x562badd1df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.627+0000 7f7f9b7fe700 1 -- 172.21.15.104:0/3041000312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd01a70 msgr2=0x562badd1da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:16.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.627+0000 7f7f9b7fe700 1 --2- 172.21.15.104:0/3041000312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd01a70 0x562badd1da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.627+0000 7f7f9b7fe700 1 -- 172.21.15.104:0/3041000312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562badcf7620 con 0x562badcfc330 2024-06-09T22:23:16.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.627+0000 7f7f9bfff700 1 --2- 172.21.15.104:0/3041000312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badd02390 0x562badd1df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:16.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.627+0000 7f7f9b7fe700 1 --2- 172.21.15.104:0/3041000312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 0x562badcf70e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000bbe0 tx=0x7f7f9000b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409c55cb2d18f68d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:16.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.627+0000 7f7f817fa700 1 -- 172.21.15.104:0/3041000312 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90003cf0 con 0x562badcfc330 2024-06-09T22:23:16.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.627+0000 7f7f817fa700 1 -- 172.21.15.104:0/3041000312 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f90003e90 con 0x562badcfc330 2024-06-09T22:23:16.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.627+0000 7f7f817fa700 1 -- 172.21.15.104:0/3041000312 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900266c0 con 0x562badcfc330 2024-06-09T22:23:16.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.628+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3041000312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 msgr2=0x562badcf70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.628+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/3041000312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 0x562badcf70e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000bbe0 tx=0x7f7f9000b410 comp rx=0 tx=0).stop 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.628+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3041000312 shutdown_connections 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.628+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/3041000312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badcfc330 0x562badcf70e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.628+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/3041000312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badd02390 0x562badd1df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.628+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/3041000312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd01a70 0x562badd1da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.628+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3041000312 >> 172.21.15.104:0/3041000312 conn(0x562badc22cc0 msgr2=0x562badcf69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.628+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3041000312 shutdown_connections 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.628+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/3041000312 wait complete. 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.628+0000 7f7fac2e6b80 1 -- start start 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7fac2e6b80 1 Processor -- start 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7fac2e6b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badcfc330 0x562badd0e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7fac2e6b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd01a70 0x562badd1d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7fac2e6b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd02390 0x562badd1d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7fac2e6b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c1ad730 con 0x562badd01a70 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7fac2e6b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c1ad060 con 0x562badd02390 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7fac2e6b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c1aa240 con 0x562badcfc330 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7fac2e6b80 10 client.0 _ll_register_callbacks cb 0x562badbe3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7f9affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd01a70 0x562badd1d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7f9affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd01a70 0x562badd1d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46782/0 (socket says 172.21.15.104:46782) 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7f9affd700 1 -- 172.21.15.104:0/2800114398 learned_addr learned my addr 172.21.15.104:0/2800114398 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7f9b7fe700 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badcfc330 0x562badd0e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7f9bfff700 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd02390 0x562badd1d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7f9b7fe700 1 -- 172.21.15.104:0/2800114398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd02390 msgr2=0x562badd1d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7f9b7fe700 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd02390 0x562badd1d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7f9b7fe700 1 -- 172.21.15.104:0/2800114398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd01a70 msgr2=0x562badd1d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7f9b7fe700 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd01a70 0x562badd1d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7f9b7fe700 1 -- 172.21.15.104:0/2800114398 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562badd1dec0 con 0x562badcfc330 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.629+0000 7f7f9affd700 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd01a70 0x562badd1d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.630+0000 7f7f9b7fe700 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badcfc330 0x562badd0e810 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7f90005950 tx=0x7f7f900052b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=582742ad92796fac server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.630+0000 7f7f7affd700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:23:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.630+0000 7f7f7affd700 1 -- 172.21.15.104:0/2800114398 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90003af0 con 0x562badcfc330 2024-06-09T22:23:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.630+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/2800114398 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f90003e90 con 0x562badcfc330 2024-06-09T22:23:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.630+0000 7f7f7affd700 1 -- 172.21.15.104:0/2800114398 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7f90003c90 con 0x562badcfc330 2024-06-09T22:23:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.630+0000 7f7f7affd700 1 -- 172.21.15.104:0/2800114398 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7f90039050 con 0x562badcfc330 2024-06-09T22:23:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.630+0000 7f7fac2e6b80 10 client.25708 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.630+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/2800114398 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562badd808a0 con 0x562badcfc330 2024-06-09T22:23:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.630+0000 7f7f7affd700 1 -- 172.21.15.104:0/2800114398 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f90003e90 con 0x562badcfc330 2024-06-09T22:23:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.630+0000 7f7f7affd700 1 -- 172.21.15.104:0/2800114398 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f7f900267e0 con 0x562badcfc330 2024-06-09T22:23:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.631+0000 7f7fac2e6b80 10 client.25708 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.631+0000 7f7fac2e6b80 10 client.25708 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:23:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.631+0000 7f7fac2e6b80 20 client.25708 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:16.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.631+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x562badd15d30 0x562badd82f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:16.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.631+0000 7f7fac2e6b80 4 client.25708 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:23:16.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.631+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/2800114398 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562badc51290 con 0x562badd15d30 2024-06-09T22:23:16.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.631+0000 7f7f9affd700 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x562badd15d30 0x562badd82f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:16.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.631+0000 7f7f9affd700 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x562badd15d30 0x562badd82f20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4197f4d738381507 server_cookie=c957d353a48a2fa7 in_seq=0 out_seq=0 2024-06-09T22:23:16.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.631+0000 7f7f7affd700 10 client.25708 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:23:16.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7f7affd700 1 -- 172.21.15.104:0/2800114398 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562badc51290 con 0x562badd15d30 2024-06-09T22:23:16.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7f7affd700 10 client.25708 handle_command_reply: tid=0 2024-06-09T22:23:16.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7fac2e6b80 1 client.25708 shutdown 2024-06-09T22:23:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/2800114398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badcfc330 msgr2=0x562badd0e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badcfc330 0x562badd0e810 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7f90005950 tx=0x7f7f900052b0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/2800114398 shutdown_connections 2024-06-09T22:23:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562badd02390 0x562badd1d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562badcfc330 0x562badd0e810 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x562badd15d30 0x562badd82f20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7fac2e6b80 1 --2- 172.21.15.104:0/2800114398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562badd01a70 0x562badd1d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7f7affd700 0 client.25708 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:23:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/2800114398 >> 172.21.15.104:0/2800114398 conn(0x562badc22cc0 msgr2=0x562badcebe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/2800114398 shutdown_connections 2024-06-09T22:23:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7fac2e6b80 1 -- 172.21.15.104:0/2800114398 wait complete. 2024-06-09T22:23:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.632+0000 7f7fac2e6b80 20 client.25708 trim_cache size 0 max 16384 2024-06-09T22:23:16.637 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:23:16.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.633+0000 7f7fa1a63700 1 -- 172.21.15.104:0/3617959448 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f840a2e10 msgr2=0x7f7f840a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:16.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.633+0000 7f7fa1a63700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f840a2e10 0x7f7f840a52d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7f940060d0 tx=0x7f7f94009c40 comp rx=0 tx=0).stop 2024-06-09T22:23:16.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.633+0000 7f7fa1a63700 1 -- 172.21.15.104:0/3617959448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9c103150 msgr2=0x7f7f9c19ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:16.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.633+0000 7f7fa1a63700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9c103150 0x7f7f9c19ecc0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00d950 tx=0x7f7f8c00dc60 comp rx=0 tx=0).stop 2024-06-09T22:23:16.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.633+0000 7f7fa1a63700 1 -- 172.21.15.104:0/3617959448 shutdown_connections 2024-06-09T22:23:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.633+0000 7f7fa1a63700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f840a2e10 0x7f7f840a52d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.633+0000 7f7fa1a63700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9c107870 0x7f7f9c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.633+0000 7f7fa1a63700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f9c1027a0 0x7f7f9c19e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.633+0000 7f7fa1a63700 1 --2- 172.21.15.104:0/3617959448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f9c103150 0x7f7f9c19ecc0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.633+0000 7f7fa1a63700 1 -- 172.21.15.104:0/3617959448 >> 172.21.15.104:0/3617959448 conn(0x7f7f9c079520 msgr2=0x7f7f9c1004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.633+0000 7f7fa1a63700 1 -- 172.21.15.104:0/3617959448 shutdown_connections 2024-06-09T22:23:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:16.633+0000 7f7fa1a63700 1 -- 172.21.15.104:0/3617959448 wait complete. 2024-06-09T22:23:16.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 6.19 MB/sec warmup 22 sec latency 6243.250 ms 2024-06-09T22:23:17.475 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-09T22:23:17.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 5.92 MB/sec warmup 23 sec latency 7243.277 ms 2024-06-09T22:23:17.719 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:23:17.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:17 smithi104 ceph-mon[32389]: pgmap v72: 129 pgs: 129 active+clean; 29 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-06-09T22:23:17.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:17 smithi194 ceph-mon[36856]: pgmap v72: 129 pgs: 129 active+clean; 29 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-06-09T22:23:17.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:17 smithi153 ceph-mon[36599]: pgmap v72: 129 pgs: 129 active+clean; 29 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-06-09T22:23:18.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 5.67 MB/sec warmup 24 sec latency 8243.323 ms 2024-06-09T22:23:19.555 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:19 smithi104 ceph-mon[32389]: pgmap v73: 129 pgs: 129 active+clean; 29 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-06-09T22:23:19.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.549+0000 7ff26c289700 1 -- 172.21.15.104:0/1315626135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff264101d80 msgr2=0x7ff26410e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:19.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.549+0000 7ff26c289700 1 --2- 172.21.15.104:0/1315626135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff264101d80 0x7ff26410e340 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00b0a0 tx=0x7ff25c00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.549+0000 7ff26c289700 1 -- 172.21.15.104:0/1315626135 shutdown_connections 2024-06-09T22:23:19.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.549+0000 7ff26c289700 1 --2- 172.21.15.104:0/1315626135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff264101d80 0x7ff26410e340 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.549+0000 7ff26c289700 1 --2- 172.21.15.104:0/1315626135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2641032e0 0x7ff2641036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.549+0000 7ff26c289700 1 --2- 172.21.15.104:0/1315626135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2641013c0 0x7ff264101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.549+0000 7ff26c289700 1 -- 172.21.15.104:0/1315626135 >> 172.21.15.104:0/1315626135 conn(0x7ff264077400 msgr2=0x7ff264077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.549+0000 7ff26c289700 1 -- 172.21.15.104:0/1315626135 shutdown_connections 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.549+0000 7ff26c289700 1 -- 172.21.15.104:0/1315626135 wait complete. 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff26c289700 1 Processor -- start 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff26c289700 1 -- start start 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff26c289700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2641013c0 0x7ff26419e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff26c289700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff264101d80 0x7ff26419ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff26c289700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2641032e0 0x7ff264198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff26c289700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff264111770 con 0x7ff2641032e0 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff26c289700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff2641115f0 con 0x7ff2641013c0 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff26c289700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff2641118f0 con 0x7ff264101d80 2024-06-09T22:23:19.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff26a826700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2641032e0 0x7ff264198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff26a826700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2641032e0 0x7ff264198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46806/0 (socket says 172.21.15.104:46806) 2024-06-09T22:23:19.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff26a826700 1 -- 172.21.15.104:0/768939417 learned_addr learned my addr 172.21.15.104:0/768939417 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:19.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff26a025700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2641013c0 0x7ff26419e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.550+0000 7ff269824700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff264101d80 0x7ff26419ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.551+0000 7ff26a826700 1 -- 172.21.15.104:0/768939417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff264101d80 msgr2=0x7ff26419ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:19.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.551+0000 7ff26a826700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff264101d80 0x7ff26419ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.551+0000 7ff26a826700 1 -- 172.21.15.104:0/768939417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2641013c0 msgr2=0x7ff26419e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:19.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.551+0000 7ff26a826700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2641013c0 0x7ff26419e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.551+0000 7ff26a826700 1 -- 172.21.15.104:0/768939417 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2641991c0 con 0x7ff2641032e0 2024-06-09T22:23:19.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.551+0000 7ff26a826700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2641032e0 0x7ff264198a60 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00b070 tx=0x7ff25c007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:19.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.551+0000 7ff25b7fe700 1 -- 172.21.15.104:0/768939417 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c021070 con 0x7ff2641032e0 2024-06-09T22:23:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.551+0000 7ff25b7fe700 1 -- 172.21.15.104:0/768939417 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff25c007e50 con 0x7ff2641032e0 2024-06-09T22:23:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.551+0000 7ff26c289700 1 -- 172.21.15.104:0/768939417 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff264199450 con 0x7ff2641032e0 2024-06-09T22:23:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.551+0000 7ff25b7fe700 1 -- 172.21.15.104:0/768939417 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c01bac0 con 0x7ff2641032e0 2024-06-09T22:23:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.551+0000 7ff26c289700 1 -- 172.21.15.104:0/768939417 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2641a5720 con 0x7ff2641032e0 2024-06-09T22:23:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.553+0000 7ff25b7fe700 1 -- 172.21.15.104:0/768939417 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ff25c024070 con 0x7ff2641032e0 2024-06-09T22:23:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.554+0000 7ff25b7fe700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff2500a2d90 0x7ff2500a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.554+0000 7ff26a025700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff2500a2d90 0x7ff2500a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.554+0000 7ff25b7fe700 1 -- 172.21.15.104:0/768939417 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff25c00f020 con 0x7ff2641032e0 2024-06-09T22:23:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.554+0000 7ff26a025700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff2500a2d90 0x7ff2500a5250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff264107a00 tx=0x7ff26000b410 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.554+0000 7ff276b0cb80 1 Processor -- start 2024-06-09T22:23:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff276b0cb80 1 -- start start 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff276b0cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada8c640 0x564eada8ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff276b0cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 0x564eada869e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff276b0cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada86f20 0x564eada87340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff276b0cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564ead9d8810 con 0x564eada8cf60 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff276b0cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564ead980720 con 0x564eada8c640 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff276b0cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564ead9d47d0 con 0x564eada86f20 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff269824700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 0x564eada869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff269824700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 0x564eada869e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46808/0 (socket says 172.21.15.104:46808) 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff269824700 1 -- 172.21.15.104:0/2278811882 learned_addr learned my addr 172.21.15.104:0/2278811882 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff26a025700 1 --2- 172.21.15.104:0/2278811882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada8c640 0x564eada8ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff26a826700 1 --2- 172.21.15.104:0/2278811882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada86f20 0x564eada87340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff269824700 1 -- 172.21.15.104:0/2278811882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada86f20 msgr2=0x564eada87340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff269824700 1 --2- 172.21.15.104:0/2278811882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada86f20 0x564eada87340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff269824700 1 -- 172.21.15.104:0/2278811882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada8c640 msgr2=0x564eada8ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff269824700 1 --2- 172.21.15.104:0/2278811882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada8c640 0x564eada8ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff269824700 1 -- 172.21.15.104:0/2278811882 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ead976f50 con 0x564eada8cf60 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.555+0000 7ff26a826700 1 --2- 172.21.15.104:0/2278811882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada86f20 0x564eada87340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.556+0000 7ff269824700 1 --2- 172.21.15.104:0/2278811882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 0x564eada869e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff25400bd30 tx=0x7ff2540106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de319f3f9ee0ee6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.556+0000 7ff258d70700 1 -- 172.21.15.104:0/2278811882 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254010e10 con 0x564eada8cf60 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.556+0000 7ff258d70700 1 -- 172.21.15.104:0/2278811882 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff254004510 con 0x564eada8cf60 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.556+0000 7ff258d70700 1 -- 172.21.15.104:0/2278811882 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2540116a0 con 0x564eada8cf60 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.556+0000 7ff276b0cb80 1 -- 172.21.15.104:0/2278811882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 msgr2=0x564eada869e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.556+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/2278811882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 0x564eada869e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff25400bd30 tx=0x7ff2540106b0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 -- 172.21.15.104:0/2278811882 shutdown_connections 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/2278811882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada86f20 0x564eada87340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/2278811882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada8c640 0x564eada8ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/2278811882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 0x564eada869e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 -- 172.21.15.104:0/2278811882 >> 172.21.15.104:0/2278811882 conn(0x564ead9b1cc0 msgr2=0x564eada81850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 -- 172.21.15.104:0/2278811882 shutdown_connections 2024-06-09T22:23:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 -- 172.21.15.104:0/2278811882 wait complete. 2024-06-09T22:23:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 -- start start 2024-06-09T22:23:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 Processor -- start 2024-06-09T22:23:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada86f20 0x564eada92c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada8c640 0x564eada8ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 0x564eada904d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564eada939d0 con 0x564eada8cf60 2024-06-09T22:23:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564ead978bf0 con 0x564eada86f20 2024-06-09T22:23:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564eada93b50 con 0x564eada8c640 2024-06-09T22:23:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff276b0cb80 10 client.0 _ll_register_callbacks cb 0x564ead972920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.557+0000 7ff26a826700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 0x564eada904d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff26a826700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 0x564eada904d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46818/0 (socket says 172.21.15.104:46818) 2024-06-09T22:23:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff26a826700 1 -- 172.21.15.104:0/1347532628 learned_addr learned my addr 172.21.15.104:0/1347532628 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff269824700 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada8c640 0x564eada8ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff26a025700 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada86f20 0x564eada92c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff26a826700 1 -- 172.21.15.104:0/1347532628 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada8c640 msgr2=0x564eada8ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff26a826700 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada8c640 0x564eada8ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff26a826700 1 -- 172.21.15.104:0/1347532628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada86f20 msgr2=0x564eada92c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff26a826700 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada86f20 0x564eada92c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff26a826700 1 -- 172.21.15.104:0/1347532628 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ead9278d0 con 0x564eada8cf60 2024-06-09T22:23:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff26a826700 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 0x564eada904d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00b830 tx=0x7ff25c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc730701f5d9f22 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff24dffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff24dffb700 1 -- 172.21.15.104:0/1347532628 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c0a1390 con 0x564eada8cf60 2024-06-09T22:23:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff24dffb700 1 -- 172.21.15.104:0/1347532628 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff25c024360 con 0x564eada8cf60 2024-06-09T22:23:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1347532628 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff254004510 con 0x564eada8cf60 2024-06-09T22:23:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff24dffb700 1 -- 172.21.15.104:0/1347532628 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff25c024560 con 0x564eada8cf60 2024-06-09T22:23:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff276b0cb80 10 client.16182 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.558+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1347532628 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564eada9e2c0 con 0x564eada8cf60 2024-06-09T22:23:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.559+0000 7ff24dffb700 1 -- 172.21.15.104:0/1347532628 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff254004510 con 0x564eada8cf60 2024-06-09T22:23:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.559+0000 7ff24dffb700 1 -- 172.21.15.104:0/1347532628 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7ff25c011070 con 0x564eada8cf60 2024-06-09T22:23:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.559+0000 7ff276b0cb80 10 client.16182 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.559+0000 7ff276b0cb80 10 client.16182 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:23:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.559+0000 7ff276b0cb80 20 client.16182 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.559+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564eadaa0880 0x564eadaa2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.559+0000 7ff276b0cb80 4 client.16182 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.559+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1347532628 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ead9e5d10 con 0x564eadaa0880 2024-06-09T22:23:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.559+0000 7ff26a025700 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564eadaa0880 0x564eadaa2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.559+0000 7ff26a025700 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564eadaa0880 0x564eadaa2c70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b202bdb95be387e8 server_cookie=e3990b111aeb668 in_seq=0 out_seq=0 2024-06-09T22:23:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.559+0000 7ff24dffb700 10 client.16182 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:23:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff24dffb700 1 -- 172.21.15.104:0/1347532628 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x564ead9e5d10 con 0x564eadaa0880 2024-06-09T22:23:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff24dffb700 10 client.16182 handle_command_reply: tid=0 2024-06-09T22:23:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff276b0cb80 1 client.16182 shutdown 2024-06-09T22:23:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1347532628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 msgr2=0x564eada904d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 0x564eada904d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00b830 tx=0x7ff25c000f40 comp rx=0 tx=0).stop 2024-06-09T22:23:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1347532628 shutdown_connections 2024-06-09T22:23:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564eadaa0880 0x564eadaa2c70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8cf60 0x564eada904d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada86f20 0x564eada92c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/1347532628 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada8c640 0x564eada8ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff24dffb700 0 client.16182 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:23:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1347532628 >> 172.21.15.104:0/1347532628 conn(0x564ead9b1cc0 msgr2=0x564eada859d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1347532628 shutdown_connections 2024-06-09T22:23:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1347532628 wait complete. 2024-06-09T22:23:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.560+0000 7ff276b0cb80 20 client.16182 trim_cache size 0 max 16384 2024-06-09T22:23:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff276b0cb80 1 Processor -- start 2024-06-09T22:23:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff276b0cb80 1 -- start start 2024-06-09T22:23:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff276b0cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ead9b1cc0 0x564eada869e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff276b0cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada86f20 0x564eada7ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff276b0cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8c640 0x564eada7b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff276b0cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff2641ae3c0 con 0x564eada8c640 2024-06-09T22:23:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff276b0cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff2641aff60 con 0x564ead9b1cc0 2024-06-09T22:23:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff276b0cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff2641aea90 con 0x564eada86f20 2024-06-09T22:23:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff26a826700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8c640 0x564eada7b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff26a826700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8c640 0x564eada7b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46822/0 (socket says 172.21.15.104:46822) 2024-06-09T22:23:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff26a826700 1 -- 172.21.15.104:0/1954371885 learned_addr learned my addr 172.21.15.104:0/1954371885 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff26a025700 1 --2- 172.21.15.104:0/1954371885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ead9b1cc0 0x564eada869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff269824700 1 --2- 172.21.15.104:0/1954371885 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada86f20 0x564eada7ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff26a826700 1 -- 172.21.15.104:0/1954371885 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada86f20 msgr2=0x564eada7ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:19.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff26a826700 1 --2- 172.21.15.104:0/1954371885 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada86f20 0x564eada7ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff26a826700 1 -- 172.21.15.104:0/1954371885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ead9b1cc0 msgr2=0x564eada869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:19.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff26a826700 1 --2- 172.21.15.104:0/1954371885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ead9b1cc0 0x564eada869e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.576+0000 7ff26a826700 1 -- 172.21.15.104:0/1954371885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eadaa32c0 con 0x564eada8c640 2024-06-09T22:23:19.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.577+0000 7ff26a826700 1 --2- 172.21.15.104:0/1954371885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8c640 0x564eada7b340 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0999b0 tx=0x7ff25c09b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4ce6a77c43b9d5 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.577+0000 7ff24ffff700 1 -- 172.21.15.104:0/1954371885 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c0a1390 con 0x564eada8c640 2024-06-09T22:23:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.577+0000 7ff24ffff700 1 -- 172.21.15.104:0/1954371885 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff25c0a5730 con 0x564eada8c640 2024-06-09T22:23:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.577+0000 7ff24ffff700 1 -- 172.21.15.104:0/1954371885 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c09d4a0 con 0x564eada8c640 2024-06-09T22:23:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1954371885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8c640 msgr2=0x564eada7b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/1954371885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8c640 0x564eada7b340 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0999b0 tx=0x7ff25c09b4f0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1954371885 shutdown_connections 2024-06-09T22:23:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/1954371885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564eada8c640 0x564eada7b340 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/1954371885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ead9b1cc0 0x564eada869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/1954371885 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada86f20 0x564eada7ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1954371885 >> 172.21.15.104:0/1954371885 conn(0x564eada8a820 msgr2=0x564eada8e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1954371885 shutdown_connections 2024-06-09T22:23:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 -- 172.21.15.104:0/1954371885 wait complete. 2024-06-09T22:23:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 -- start start 2024-06-09T22:23:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 Processor -- start 2024-06-09T22:23:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ead9b1cc0 0x564eadaa8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada86f20 0x564eadaa87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada8c640 0x564eadaa8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff2641af830 con 0x564ead9b1cc0 2024-06-09T22:23:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff2641af160 con 0x564eada86f20 2024-06-09T22:23:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff2641ac340 con 0x564eada8c640 2024-06-09T22:23:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff276b0cb80 10 client.0 _ll_register_callbacks cb 0x564ead972920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff26a025700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ead9b1cc0 0x564eadaa8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff26a025700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ead9b1cc0 0x564eadaa8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46824/0 (socket says 172.21.15.104:46824) 2024-06-09T22:23:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.578+0000 7ff26a025700 1 -- 172.21.15.104:0/688714097 learned_addr learned my addr 172.21.15.104:0/688714097 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff269824700 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada86f20 0x564eadaa87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff26a826700 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada8c640 0x564eadaa8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff26a025700 1 -- 172.21.15.104:0/688714097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada8c640 msgr2=0x564eadaa8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff26a025700 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada8c640 0x564eadaa8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff26a025700 1 -- 172.21.15.104:0/688714097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada86f20 msgr2=0x564eadaa87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff26a025700 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada86f20 0x564eadaa87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff26a025700 1 -- 172.21.15.104:0/688714097 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eadb0a620 con 0x564ead9b1cc0 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff26a025700 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ead9b1cc0 0x564eadaa8270 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff26000ba80 tx=0x7ff260013070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b662ccf347ca14 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff24dffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff24dffb700 1 -- 172.21.15.104:0/688714097 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26002c070 con 0x564ead9b1cc0 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff24dffb700 1 -- 172.21.15.104:0/688714097 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff260011440 con 0x564ead9b1cc0 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff276b0cb80 1 -- 172.21.15.104:0/688714097 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff25c0a5730 con 0x564ead9b1cc0 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff24dffb700 1 -- 172.21.15.104:0/688714097 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff260039050 con 0x564ead9b1cc0 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff276b0cb80 10 client.16188 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.579+0000 7ff276b0cb80 1 -- 172.21.15.104:0/688714097 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564eadb0b980 con 0x564ead9b1cc0 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.580+0000 7ff24dffb700 1 -- 172.21.15.104:0/688714097 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff25c0a5730 con 0x564ead9b1cc0 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.580+0000 7ff24dffb700 1 -- 172.21.15.104:0/688714097 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7ff26001e7b0 con 0x564ead9b1cc0 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.580+0000 7ff276b0cb80 10 client.16188 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.580+0000 7ff276b0cb80 10 client.16188 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:23:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.580+0000 7ff276b0cb80 20 client.16188 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.580+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564eada8cf60 0x564eadb0e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.580+0000 7ff276b0cb80 4 client.16188 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:23:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.580+0000 7ff276b0cb80 1 -- 172.21.15.104:0/688714097 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564ead9e0290 con 0x564eada8cf60 2024-06-09T22:23:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.580+0000 7ff269824700 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564eada8cf60 0x564eadb0e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.580+0000 7ff269824700 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564eada8cf60 0x564eadb0e0b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c4ee899dfd62fc2a server_cookie=7f37f7ebea0d38f5 in_seq=0 out_seq=0 2024-06-09T22:23:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.580+0000 7ff24dffb700 10 client.16188 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:23:19.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff24dffb700 1 -- 172.21.15.104:0/688714097 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564ead9e0290 con 0x564eada8cf60 2024-06-09T22:23:19.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff24dffb700 10 client.16188 handle_command_reply: tid=0 2024-06-09T22:23:19.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff276b0cb80 1 client.16188 shutdown 2024-06-09T22:23:19.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff276b0cb80 1 -- 172.21.15.104:0/688714097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ead9b1cc0 msgr2=0x564eadaa8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:19.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ead9b1cc0 0x564eadaa8270 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff26000ba80 tx=0x7ff260013070 comp rx=0 tx=0).stop 2024-06-09T22:23:19.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff276b0cb80 1 -- 172.21.15.104:0/688714097 shutdown_connections 2024-06-09T22:23:19.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x564eada8cf60 0x564eadb0e0b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ead9b1cc0 0x564eadaa8270 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564eada8c640 0x564eadaa8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff24dffb700 0 client.16188 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:23:19.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff276b0cb80 1 --2- 172.21.15.104:0/688714097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564eada86f20 0x564eadaa87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff276b0cb80 1 -- 172.21.15.104:0/688714097 >> 172.21.15.104:0/688714097 conn(0x564eada8a820 msgr2=0x564eada8d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:19.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff276b0cb80 1 -- 172.21.15.104:0/688714097 shutdown_connections 2024-06-09T22:23:19.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff276b0cb80 1 -- 172.21.15.104:0/688714097 wait complete. 2024-06-09T22:23:19.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.581+0000 7ff276b0cb80 20 client.16188 trim_cache size 0 max 16384 2024-06-09T22:23:19.601 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:23:19.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.582+0000 7ff26c289700 1 -- 172.21.15.104:0/768939417 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff2500a2d90 msgr2=0x7ff2500a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:19.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.582+0000 7ff26c289700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff2500a2d90 0x7ff2500a5250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff264107a00 tx=0x7ff26000b410 comp rx=0 tx=0).stop 2024-06-09T22:23:19.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.582+0000 7ff26c289700 1 -- 172.21.15.104:0/768939417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2641032e0 msgr2=0x7ff264198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:19.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.582+0000 7ff26c289700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2641032e0 0x7ff264198a60 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00b070 tx=0x7ff25c007a20 comp rx=0 tx=0).stop 2024-06-09T22:23:19.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.582+0000 7ff26c289700 1 -- 172.21.15.104:0/768939417 shutdown_connections 2024-06-09T22:23:19.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.582+0000 7ff26c289700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff2500a2d90 0x7ff2500a5250 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.582+0000 7ff26c289700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2641032e0 0x7ff264198a60 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.582+0000 7ff26c289700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2641013c0 0x7ff26419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.582+0000 7ff26c289700 1 --2- 172.21.15.104:0/768939417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff264101d80 0x7ff26419ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:19.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.582+0000 7ff26c289700 1 -- 172.21.15.104:0/768939417 >> 172.21.15.104:0/768939417 conn(0x7ff264077400 msgr2=0x7ff26406aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:19.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.582+0000 7ff26c289700 1 -- 172.21.15.104:0/768939417 shutdown_connections 2024-06-09T22:23:19.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:19.582+0000 7ff26c289700 1 -- 172.21.15.104:0/768939417 wait complete. 2024-06-09T22:23:19.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 5.45 MB/sec warmup 25 sec latency 9243.655 ms 2024-06-09T22:23:19.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:19 smithi194 ceph-mon[36856]: pgmap v73: 129 pgs: 129 active+clean; 29 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-06-09T22:23:19.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:19 smithi153 ceph-mon[36599]: pgmap v73: 129 pgs: 129 active+clean; 29 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-06-09T22:23:20.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-09T22:23:20.595 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:23:20.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 5.24 MB/sec warmup 26 sec latency 10243.681 ms 2024-06-09T22:23:21.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 5.04 MB/sec warmup 27 sec latency 11243.707 ms 2024-06-09T22:23:21.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:21 smithi104 ceph-mon[32389]: pgmap v74: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 3.1 MiB/s wr, 118 op/s 2024-06-09T22:23:21.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:21 smithi194 ceph-mon[36856]: pgmap v74: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 3.1 MiB/s wr, 118 op/s 2024-06-09T22:23:21.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:21 smithi153 ceph-mon[36599]: pgmap v74: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 3.1 MiB/s wr, 118 op/s 2024-06-09T22:23:22.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.459+0000 7f453e6f2700 1 -- 172.21.15.104:0/2694730208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45381019d0 msgr2=0x7f4538101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:22.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.459+0000 7f453e6f2700 1 --2- 172.21.15.104:0/2694730208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45381019d0 0x7f4538101db0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f45280099c0 tx=0x7f4528009cd0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.459+0000 7f453e6f2700 1 -- 172.21.15.104:0/2694730208 shutdown_connections 2024-06-09T22:23:22.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.459+0000 7f453e6f2700 1 --2- 172.21.15.104:0/2694730208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45381019d0 0x7f4538101db0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.459+0000 7f453e6f2700 1 --2- 172.21.15.104:0/2694730208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4538106720 0x7f453810e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.459+0000 7f453e6f2700 1 --2- 172.21.15.104:0/2694730208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4538102380 0x7f4538105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.459+0000 7f453e6f2700 1 -- 172.21.15.104:0/2694730208 >> 172.21.15.104:0/2694730208 conn(0x7f4538078340 msgr2=0x7f4538078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:22.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.459+0000 7f453e6f2700 1 -- 172.21.15.104:0/2694730208 shutdown_connections 2024-06-09T22:23:22.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.459+0000 7f453e6f2700 1 -- 172.21.15.104:0/2694730208 wait complete. 2024-06-09T22:23:22.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.460+0000 7f453e6f2700 1 Processor -- start 2024-06-09T22:23:22.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.460+0000 7f453e6f2700 1 -- start start 2024-06-09T22:23:22.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453e6f2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45381019d0 0x7f4538199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453e6f2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4538102380 0x7f453819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453e6f2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4538106720 0x7f45381a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453e6f2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4538111730 con 0x7f4538106720 2024-06-09T22:23:22.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453e6f2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45381115b0 con 0x7f45381019d0 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453e6f2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f45381118b0 con 0x7f4538102380 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453cc8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4538106720 0x7f45381a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453cc8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4538106720 0x7f45381a4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46846/0 (socket says 172.21.15.104:46846) 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453cc8f700 1 -- 172.21.15.104:0/3817609476 learned_addr learned my addr 172.21.15.104:0/3817609476 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f4537fff700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45381019d0 0x7f4538199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f45377fe700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4538102380 0x7f453819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453cc8f700 1 -- 172.21.15.104:0/3817609476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4538102380 msgr2=0x7f453819a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453cc8f700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4538102380 0x7f453819a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453cc8f700 1 -- 172.21.15.104:0/3817609476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45381019d0 msgr2=0x7f4538199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453cc8f700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45381019d0 0x7f4538199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453cc8f700 1 -- 172.21.15.104:0/3817609476 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45381a5090 con 0x7f4538106720 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f453cc8f700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4538106720 0x7f45381a4b50 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4530007ee0 tx=0x7f453000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:22.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.461+0000 7f45357fa700 1 -- 172.21.15.104:0/3817609476 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4530004bb0 con 0x7f4538106720 2024-06-09T22:23:22.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.462+0000 7f453e6f2700 1 -- 172.21.15.104:0/3817609476 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45381a5320 con 0x7f4538106720 2024-06-09T22:23:22.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.462+0000 7f45357fa700 1 -- 172.21.15.104:0/3817609476 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4530004d50 con 0x7f4538106720 2024-06-09T22:23:22.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.462+0000 7f45357fa700 1 -- 172.21.15.104:0/3817609476 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4530005960 con 0x7f4538106720 2024-06-09T22:23:22.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.462+0000 7f453e6f2700 1 -- 172.21.15.104:0/3817609476 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45381a5830 con 0x7f4538106720 2024-06-09T22:23:22.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.464+0000 7f45357fa700 1 -- 172.21.15.104:0/3817609476 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f453001a020 con 0x7f4538106720 2024-06-09T22:23:22.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.465+0000 7f45357fa700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45200a2d90 0x7f45200a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.465+0000 7f4537fff700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45200a2d90 0x7f45200a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.465+0000 7f45357fa700 1 -- 172.21.15.104:0/3817609476 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f45300cef00 con 0x7f4538106720 2024-06-09T22:23:22.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.465+0000 7f4537fff700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45200a2d90 0x7f45200a5250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4528009990 tx=0x7f4528000bc0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:22.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f4548f75b80 1 Processor -- start 2024-06-09T22:23:22.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f4548f75b80 1 -- start start 2024-06-09T22:23:22.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f4548f75b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902640 0x563a60902a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f4548f75b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a60902f60 0x563a608fc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f4548f75b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 0x563a608fd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f4548f75b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x563a6084e810 con 0x563a608fcf20 2024-06-09T22:23:22.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f4548f75b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563a607f6720 con 0x563a60902640 2024-06-09T22:23:22.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f4548f75b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563a6084a7d0 con 0x563a60902f60 2024-06-09T22:23:22.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f453cc8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 0x563a608fd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f453cc8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 0x563a608fd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46854/0 (socket says 172.21.15.104:46854) 2024-06-09T22:23:22.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f453cc8f700 1 -- 172.21.15.104:0/2793386829 learned_addr learned my addr 172.21.15.104:0/2793386829 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:22.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f4537fff700 1 --2- 172.21.15.104:0/2793386829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902640 0x563a60902a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f45377fe700 1 --2- 172.21.15.104:0/2793386829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a60902f60 0x563a608fc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f453cc8f700 1 -- 172.21.15.104:0/2793386829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a60902f60 msgr2=0x563a608fc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:22.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f453cc8f700 1 --2- 172.21.15.104:0/2793386829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a60902f60 0x563a608fc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f453cc8f700 1 -- 172.21.15.104:0/2793386829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902640 msgr2=0x563a60902a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:22.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f453cc8f700 1 --2- 172.21.15.104:0/2793386829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902640 0x563a60902a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f453cc8f700 1 -- 172.21.15.104:0/2793386829 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a607ecf50 con 0x563a608fcf20 2024-06-09T22:23:22.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.466+0000 7f4537fff700 1 --2- 172.21.15.104:0/2793386829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902640 0x563a60902a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:22.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.467+0000 7f453cc8f700 1 --2- 172.21.15.104:0/2793386829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 0x563a608fd340 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4530004b80 tx=0x7f453009ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ea67f7bef9c30c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.467+0000 7f451e7fc700 1 -- 172.21.15.104:0/2793386829 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453009b070 con 0x563a608fcf20 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.467+0000 7f451e7fc700 1 -- 172.21.15.104:0/2793386829 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f453009f420 con 0x563a608fcf20 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.467+0000 7f451e7fc700 1 -- 172.21.15.104:0/2793386829 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45300976e0 con 0x563a608fcf20 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.467+0000 7f4548f75b80 1 -- 172.21.15.104:0/2793386829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 msgr2=0x563a608fd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.467+0000 7f4548f75b80 1 --2- 172.21.15.104:0/2793386829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 0x563a608fd340 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4530004b80 tx=0x7f453009ac30 comp rx=0 tx=0).stop 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.467+0000 7f4548f75b80 1 -- 172.21.15.104:0/2793386829 shutdown_connections 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.467+0000 7f4548f75b80 1 --2- 172.21.15.104:0/2793386829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 0x563a608fd340 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.467+0000 7f4548f75b80 1 --2- 172.21.15.104:0/2793386829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902640 0x563a60902a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.467+0000 7f4548f75b80 1 --2- 172.21.15.104:0/2793386829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a60902f60 0x563a608fc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.467+0000 7f4548f75b80 1 -- 172.21.15.104:0/2793386829 >> 172.21.15.104:0/2793386829 conn(0x563a60827cc0 msgr2=0x563a608f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f4548f75b80 1 -- 172.21.15.104:0/2793386829 shutdown_connections 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f4548f75b80 1 -- 172.21.15.104:0/2793386829 wait complete. 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f4548f75b80 1 -- start start 2024-06-09T22:23:22.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f4548f75b80 1 Processor -- start 2024-06-09T22:23:22.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f4548f75b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a608fcf20 0x563a60908d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f4548f75b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 0x563a609060e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f4548f75b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 0x563a60906620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f4548f75b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x563a609099d0 con 0x563a60902640 2024-06-09T22:23:22.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f4548f75b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563a607eebf0 con 0x563a60902f60 2024-06-09T22:23:22.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f4548f75b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563a60909b50 con 0x563a608fcf20 2024-06-09T22:23:22.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f4548f75b80 10 client.0 _ll_register_callbacks cb 0x563a607e8920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:22.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f45377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 0x563a609060e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f45377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 0x563a609060e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46866/0 (socket says 172.21.15.104:46866) 2024-06-09T22:23:22.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f45377fe700 1 -- 172.21.15.104:0/3452782955 learned_addr learned my addr 172.21.15.104:0/3452782955 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:22.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.468+0000 7f4537fff700 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a608fcf20 0x563a60908d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f453cc8f700 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 0x563a60906620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f45377fe700 1 -- 172.21.15.104:0/3452782955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a608fcf20 msgr2=0x563a60908d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:22.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f45377fe700 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a608fcf20 0x563a60908d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f45377fe700 1 -- 172.21.15.104:0/3452782955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 msgr2=0x563a60906620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:22.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f45377fe700 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 0x563a60906620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f45377fe700 1 -- 172.21.15.104:0/3452782955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6079d8d0 con 0x563a60902640 2024-06-09T22:23:22.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f4537fff700 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a608fcf20 0x563a60908d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:22.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f45377fe700 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 0x563a609060e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f452c009f50 tx=0x7f452c011540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=818db035f422a065 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:22.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f45177fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:22.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f45177fe700 1 -- 172.21.15.104:0/3452782955 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452c019070 con 0x563a60902640 2024-06-09T22:23:22.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f45177fe700 1 -- 172.21.15.104:0/3452782955 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f452c015460 con 0x563a60902640 2024-06-09T22:23:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f4548f75b80 1 -- 172.21.15.104:0/3452782955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f453009f420 con 0x563a60902640 2024-06-09T22:23:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f45177fe700 1 -- 172.21.15.104:0/3452782955 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f452c025050 con 0x563a60902640 2024-06-09T22:23:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f4548f75b80 10 client.16200 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.469+0000 7f4548f75b80 1 -- 172.21.15.104:0/3452782955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563a60918810 con 0x563a60902640 2024-06-09T22:23:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.470+0000 7f45177fe700 1 -- 172.21.15.104:0/3452782955 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f453009f420 con 0x563a60902640 2024-06-09T22:23:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.470+0000 7f45177fe700 1 -- 172.21.15.104:0/3452782955 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f452c0148d0 con 0x563a60902640 2024-06-09T22:23:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.470+0000 7f4548f75b80 10 client.16200 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.470+0000 7f4548f75b80 10 client.16200 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:23:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.470+0000 7f4548f75b80 20 client.16200 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.470+0000 7f4548f75b80 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x563a6091ada0 0x563a6091d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.470+0000 7f4548f75b80 4 client.16200 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.470+0000 7f4548f75b80 1 -- 172.21.15.104:0/3452782955 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a6085bd10 con 0x563a6091ada0 2024-06-09T22:23:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.471+0000 7f4537fff700 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x563a6091ada0 0x563a6091d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.472+0000 7f45177fe700 10 client.16200 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:23:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.472+0000 7f4537fff700 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x563a6091ada0 0x563a6091d1f0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2e7d69ee9d765d6d server_cookie=889292f9fb96c1e6 in_seq=0 out_seq=0 2024-06-09T22:23:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f45177fe700 1 -- 172.21.15.104:0/3452782955 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x563a6085bd10 con 0x563a6091ada0 2024-06-09T22:23:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f45177fe700 10 client.16200 handle_command_reply: tid=0 2024-06-09T22:23:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f4548f75b80 1 client.16200 shutdown 2024-06-09T22:23:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f4548f75b80 1 -- 172.21.15.104:0/3452782955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 msgr2=0x563a609060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f4548f75b80 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 0x563a609060e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f452c009f50 tx=0x7f452c011540 comp rx=0 tx=0).stop 2024-06-09T22:23:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f4548f75b80 1 -- 172.21.15.104:0/3452782955 shutdown_connections 2024-06-09T22:23:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f4548f75b80 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x563a6091ada0 0x563a6091d1f0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f4548f75b80 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 0x563a60906620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f45177fe700 0 client.16200 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:23:22.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f4548f75b80 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a608fcf20 0x563a60908d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f4548f75b80 1 --2- 172.21.15.104:0/3452782955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 0x563a609060e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f4548f75b80 1 -- 172.21.15.104:0/3452782955 >> 172.21.15.104:0/3452782955 conn(0x563a60827cc0 msgr2=0x563a608fb9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:22.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f4548f75b80 1 -- 172.21.15.104:0/3452782955 shutdown_connections 2024-06-09T22:23:22.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f4548f75b80 1 -- 172.21.15.104:0/3452782955 wait complete. 2024-06-09T22:23:22.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.474+0000 7f4548f75b80 20 client.16200 trim_cache size 0 max 16384 2024-06-09T22:23:22.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.489+0000 7f4548f75b80 1 Processor -- start 2024-06-09T22:23:22.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.489+0000 7f4548f75b80 1 -- start start 2024-06-09T22:23:22.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.489+0000 7f4548f75b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a608fcf20 0x563a608fc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.489+0000 7f4548f75b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 0x563a6091d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.489+0000 7f4548f75b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 0x563a60922b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.489+0000 7f4548f75b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f45381ac470 con 0x563a60902640 2024-06-09T22:23:22.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.489+0000 7f4548f75b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45381ae010 con 0x563a60902f60 2024-06-09T22:23:22.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.489+0000 7f4548f75b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f45381acb40 con 0x563a608fcf20 2024-06-09T22:23:22.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.489+0000 7f45377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 0x563a6091d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.489+0000 7f45377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 0x563a6091d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46878/0 (socket says 172.21.15.104:46878) 2024-06-09T22:23:22.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.489+0000 7f45377fe700 1 -- 172.21.15.104:0/1917824569 learned_addr learned my addr 172.21.15.104:0/1917824569 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:22.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.490+0000 7f4537fff700 1 --2- 172.21.15.104:0/1917824569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a608fcf20 0x563a608fc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.490+0000 7f45377fe700 1 -- 172.21.15.104:0/1917824569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a608fcf20 msgr2=0x563a608fc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:22.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.490+0000 7f45377fe700 1 --2- 172.21.15.104:0/1917824569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a608fcf20 0x563a608fc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.490+0000 7f45377fe700 1 -- 172.21.15.104:0/1917824569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 msgr2=0x563a60922b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:22.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.490+0000 7f45377fe700 1 --2- 172.21.15.104:0/1917824569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 0x563a60922b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.490+0000 7f45377fe700 1 -- 172.21.15.104:0/1917824569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a60906c20 con 0x563a60902640 2024-06-09T22:23:22.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.490+0000 7f45377fe700 1 --2- 172.21.15.104:0/1917824569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 0x563a6091d190 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f452c000c00 tx=0x7f452c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cafaa0d86cc4a8c7 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:22.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.490+0000 7f451dffb700 1 -- 172.21.15.104:0/1917824569 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452c019040 con 0x563a60902640 2024-06-09T22:23:22.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.490+0000 7f451dffb700 1 -- 172.21.15.104:0/1917824569 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f452c01d3b0 con 0x563a60902640 2024-06-09T22:23:22.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.490+0000 7f451dffb700 1 -- 172.21.15.104:0/1917824569 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452c01dc10 con 0x563a60902640 2024-06-09T22:23:22.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 -- 172.21.15.104:0/1917824569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 msgr2=0x563a6091d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:22.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 --2- 172.21.15.104:0/1917824569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 0x563a6091d190 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f452c000c00 tx=0x7f452c011940 comp rx=0 tx=0).stop 2024-06-09T22:23:22.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 -- 172.21.15.104:0/1917824569 shutdown_connections 2024-06-09T22:23:22.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 --2- 172.21.15.104:0/1917824569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 0x563a60922b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 --2- 172.21.15.104:0/1917824569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a608fcf20 0x563a608fc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 --2- 172.21.15.104:0/1917824569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a60902640 0x563a6091d190 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 -- 172.21.15.104:0/1917824569 >> 172.21.15.104:0/1917824569 conn(0x563a60827cc0 msgr2=0x563a609049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:22.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 -- 172.21.15.104:0/1917824569 shutdown_connections 2024-06-09T22:23:22.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 -- 172.21.15.104:0/1917824569 wait complete. 2024-06-09T22:23:22.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 -- start start 2024-06-09T22:23:22.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 Processor -- start 2024-06-09T22:23:22.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 0x563a6090efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a60902640 0x563a6090f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 0x563a60921ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f45381ad8e0 con 0x563a608fcf20 2024-06-09T22:23:22.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45381ad210 con 0x563a60902f60 2024-06-09T22:23:22.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.491+0000 7f4548f75b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f45381aa410 con 0x563a60902640 2024-06-09T22:23:22.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4548f75b80 10 client.0 _ll_register_callbacks cb 0x563a607e8920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:22.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4537fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 0x563a6090efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4537fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 0x563a6090efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46894/0 (socket says 172.21.15.104:46894) 2024-06-09T22:23:22.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4537fff700 1 -- 172.21.15.104:0/3010298856 learned_addr learned my addr 172.21.15.104:0/3010298856 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:22.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f45377fe700 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a60902640 0x563a6090f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4537fff700 1 -- 172.21.15.104:0/3010298856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a60902640 msgr2=0x563a6090f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:22.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4537fff700 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a60902640 0x563a6090f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4537fff700 1 -- 172.21.15.104:0/3010298856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 msgr2=0x563a60921ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4537fff700 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 0x563a60921ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4537fff700 1 -- 172.21.15.104:0/3010298856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a60922630 con 0x563a608fcf20 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f45377fe700 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a60902640 0x563a6090f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4537fff700 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 0x563a6090efc0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f45280038e0 tx=0x7f4528009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec2c5c36f818c66 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f45177fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f45177fe700 1 -- 172.21.15.104:0/3010298856 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452803f060 con 0x563a608fcf20 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f45177fe700 1 -- 172.21.15.104:0/3010298856 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f452800f930 con 0x563a608fcf20 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4548f75b80 1 -- 172.21.15.104:0/3010298856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f452c01d3b0 con 0x563a608fcf20 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f45177fe700 1 -- 172.21.15.104:0/3010298856 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4528046050 con 0x563a608fcf20 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4548f75b80 10 client.16206 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.492+0000 7f4548f75b80 1 -- 172.21.15.104:0/3010298856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563a60985550 con 0x563a608fcf20 2024-06-09T22:23:22.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.493+0000 7f45177fe700 1 -- 172.21.15.104:0/3010298856 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f452c01d3b0 con 0x563a608fcf20 2024-06-09T22:23:22.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.493+0000 7f45177fe700 1 -- 172.21.15.104:0/3010298856 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f4528024bf0 con 0x563a608fcf20 2024-06-09T22:23:22.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.493+0000 7f4548f75b80 10 client.16206 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:22.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.493+0000 7f4548f75b80 10 client.16206 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:23:22.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.493+0000 7f4548f75b80 20 client.16206 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:22.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.493+0000 7f4548f75b80 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x563a60985c00 0x563a60987ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:22.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.493+0000 7f4548f75b80 4 client.16206 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:23:22.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.493+0000 7f4548f75b80 1 -- 172.21.15.104:0/3010298856 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a60856290 con 0x563a60985c00 2024-06-09T22:23:22.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.493+0000 7f45377fe700 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x563a60985c00 0x563a60987ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:22.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.493+0000 7f45377fe700 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x563a60985c00 0x563a60987ff0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1a3cead2c1c6368f server_cookie=496953189df8e07b in_seq=0 out_seq=0 2024-06-09T22:23:22.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.493+0000 7f45177fe700 10 client.16206 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:23:22.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f45177fe700 1 -- 172.21.15.104:0/3010298856 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a60856290 con 0x563a60985c00 2024-06-09T22:23:22.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f45177fe700 10 client.16206 handle_command_reply: tid=0 2024-06-09T22:23:22.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f4548f75b80 1 client.16206 shutdown 2024-06-09T22:23:22.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f4548f75b80 1 -- 172.21.15.104:0/3010298856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 msgr2=0x563a6090efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:22.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f4548f75b80 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 0x563a6090efc0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f45280038e0 tx=0x7f4528009cd0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f4548f75b80 1 -- 172.21.15.104:0/3010298856 shutdown_connections 2024-06-09T22:23:22.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f4548f75b80 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x563a60985c00 0x563a60987ff0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f4548f75b80 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563a608fcf20 0x563a6090efc0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f4548f75b80 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563a60902f60 0x563a60921ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f4548f75b80 1 --2- 172.21.15.104:0/3010298856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a60902640 0x563a6090f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f4548f75b80 1 -- 172.21.15.104:0/3010298856 >> 172.21.15.104:0/3010298856 conn(0x563a60827cc0 msgr2=0x563a608f0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:22.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f45177fe700 0 client.16206 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:23:22.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f4548f75b80 1 -- 172.21.15.104:0/3010298856 shutdown_connections 2024-06-09T22:23:22.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f4548f75b80 1 -- 172.21.15.104:0/3010298856 wait complete. 2024-06-09T22:23:22.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f4548f75b80 20 client.16206 trim_cache size 0 max 16384 2024-06-09T22:23:22.508 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:23:22.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f453e6f2700 1 -- 172.21.15.104:0/3817609476 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45200a2d90 msgr2=0x7f45200a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:22.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.494+0000 7f453e6f2700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45200a2d90 0x7f45200a5250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4528009990 tx=0x7f4528000bc0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.495+0000 7f453e6f2700 1 -- 172.21.15.104:0/3817609476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4538106720 msgr2=0x7f45381a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:22.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.495+0000 7f453e6f2700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4538106720 0x7f45381a4b50 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4530007ee0 tx=0x7f453000d3b0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.495+0000 7f453e6f2700 1 -- 172.21.15.104:0/3817609476 shutdown_connections 2024-06-09T22:23:22.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.495+0000 7f453e6f2700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f45200a2d90 0x7f45200a5250 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.495+0000 7f453e6f2700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4538106720 0x7f45381a4b50 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.495+0000 7f453e6f2700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45381019d0 0x7f4538199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.495+0000 7f453e6f2700 1 --2- 172.21.15.104:0/3817609476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4538102380 0x7f453819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:22.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.495+0000 7f453e6f2700 1 -- 172.21.15.104:0/3817609476 >> 172.21.15.104:0/3817609476 conn(0x7f4538078340 msgr2=0x7f4538107870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:22.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.495+0000 7f453e6f2700 1 -- 172.21.15.104:0/3817609476 shutdown_connections 2024-06-09T22:23:22.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:22.495+0000 7f453e6f2700 1 -- 172.21.15.104:0/3817609476 wait complete. 2024-06-09T22:23:22.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 4.86 MB/sec warmup 28 sec latency 12243.736 ms 2024-06-09T22:23:23.227 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-09T22:23:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:23:23.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 4.69 MB/sec warmup 29 sec latency 13243.759 ms 2024-06-09T22:23:23.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:23 smithi194 ceph-mon[36856]: pgmap v75: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 41 KiB/s wr, 20 op/s 2024-06-09T22:23:23.832 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:23 smithi104 ceph-mon[32389]: pgmap v75: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 41 KiB/s wr, 20 op/s 2024-06-09T22:23:23.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:23 smithi153 ceph-mon[36599]: pgmap v75: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 41 KiB/s wr, 20 op/s 2024-06-09T22:23:24.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 4.54 MB/sec warmup 30 sec latency 14243.810 ms 2024-06-09T22:23:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.781+0000 7f8d90ae1640 1 -- 172.21.15.104:0/2498790633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c103540 msgr2=0x7f8d8c10ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.781+0000 7f8d90ae1640 1 --2- 172.21.15.104:0/2498790633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c103540 0x7f8d8c10ba20 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f8d80009900 tx=0x7f8d8002a480 comp rx=0 tx=0).stop 2024-06-09T22:23:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.781+0000 7f8d90ae1640 1 -- 172.21.15.104:0/2498790633 shutdown_connections 2024-06-09T22:23:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.781+0000 7f8d90ae1640 1 --2- 172.21.15.104:0/2498790633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d8c10bff0 0x7f8d8c10e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.781+0000 7f8d90ae1640 1 --2- 172.21.15.104:0/2498790633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c103540 0x7f8d8c10ba20 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.781+0000 7f8d90ae1640 1 --2- 172.21.15.104:0/2498790633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d8c102c20 0x7f8d8c103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.781+0000 7f8d90ae1640 1 -- 172.21.15.104:0/2498790633 >> 172.21.15.104:0/2498790633 conn(0x7f8d8c0779f0 msgr2=0x7f8d8c077e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.781+0000 7f8d90ae1640 1 -- 172.21.15.104:0/2498790633 shutdown_connections 2024-06-09T22:23:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.781+0000 7f8d90ae1640 1 -- 172.21.15.104:0/2498790633 wait complete. 2024-06-09T22:23:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.782+0000 7f8d90ae1640 1 Processor -- start 2024-06-09T22:23:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.782+0000 7f8d90ae1640 1 -- start start 2024-06-09T22:23:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.782+0000 7f8d90ae1640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d8c102c20 0x7f8d8c100870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.782+0000 7f8d90ae1640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c103540 0x7f8d8c100db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.782+0000 7f8d90ae1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d8c10bff0 0x7f8d8c1012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.782+0000 7f8d90ae1640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c111d80 con 0x7f8d8c103540 2024-06-09T22:23:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.782+0000 7f8d90ae1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c111c00 con 0x7f8d8c10bff0 2024-06-09T22:23:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.782+0000 7f8d90ae1640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c111f00 con 0x7f8d8c102c20 2024-06-09T22:23:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.782+0000 7f8d89d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c103540 0x7f8d8c100db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.782+0000 7f8d89d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c103540 0x7f8d8c100db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46902/0 (socket says 172.21.15.104:46902) 2024-06-09T22:23:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.782+0000 7f8d89d9b640 1 -- 172.21.15.104:0/3337997039 learned_addr learned my addr 172.21.15.104:0/3337997039 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.782+0000 7f8d8a59c640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d8c102c20 0x7f8d8c100870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.783+0000 7f8d8ad9d640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d8c10bff0 0x7f8d8c1012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.783+0000 7f8d89d9b640 1 -- 172.21.15.104:0/3337997039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d8c102c20 msgr2=0x7f8d8c100870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.783+0000 7f8d89d9b640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d8c102c20 0x7f8d8c100870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.783+0000 7f8d89d9b640 1 -- 172.21.15.104:0/3337997039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d8c10bff0 msgr2=0x7f8d8c1012f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.783+0000 7f8d89d9b640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d8c10bff0 0x7f8d8c1012f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.783+0000 7f8d89d9b640 1 -- 172.21.15.104:0/3337997039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c1a66a0 con 0x7f8d8c103540 2024-06-09T22:23:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.783+0000 7f8d89d9b640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c103540 0x7f8d8c100db0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8d80006010 tx=0x7f8d800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.783+0000 7f8d7b7fe640 1 -- 172.21.15.104:0/3337997039 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8003f030 con 0x7f8d8c103540 2024-06-09T22:23:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.783+0000 7f8d7b7fe640 1 -- 172.21.15.104:0/3337997039 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d80003ce0 con 0x7f8d8c103540 2024-06-09T22:23:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.783+0000 7f8d7b7fe640 1 -- 172.21.15.104:0/3337997039 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80035070 con 0x7f8d8c103540 2024-06-09T22:23:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.783+0000 7f8d90ae1640 1 -- 172.21.15.104:0/3337997039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8c1a6930 con 0x7f8d8c103540 2024-06-09T22:23:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.783+0000 7f8d90ae1640 1 -- 172.21.15.104:0/3337997039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8c1a6e20 con 0x7f8d8c103540 2024-06-09T22:23:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.786+0000 7f8d7b7fe640 1 -- 172.21.15.104:0/3337997039 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f8d80042030 con 0x7f8d8c103540 2024-06-09T22:23:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.786+0000 7f8d7b7fe640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8d740a2dd0 0x7f8d740a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.787+0000 7f8d8a59c640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8d740a2dd0 0x7f8d740a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.787+0000 7f8d7b7fe640 1 -- 172.21.15.104:0/3337997039 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8d800efd80 con 0x7f8d8c103540 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.787+0000 7f8d937f3740 1 Processor -- start 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.787+0000 7f8d8a59c640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8d740a2dd0 0x7f8d740a5290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c005d00 tx=0x7f8d7c00b410 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.787+0000 7f8d937f3740 1 -- start start 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.787+0000 7f8d937f3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x5639790ef550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.787+0000 7f8d937f3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790efa90 0x5639790f9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.787+0000 7f8d937f3740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 0x5639790fc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.787+0000 7f8d937f3740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5639790534f0 con 0x5639790fa350 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.787+0000 7f8d937f3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563978ff1870 con 0x5639790efa90 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.787+0000 7f8d937f3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56397904f4b0 con 0x5639790ef170 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d8ad9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 0x5639790fc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d8ad9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 0x5639790fc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46914/0 (socket says 172.21.15.104:46914) 2024-06-09T22:23:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d8ad9d640 1 -- 172.21.15.104:0/163602648 learned_addr learned my addr 172.21.15.104:0/163602648 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d8a59c640 1 --2- 172.21.15.104:0/163602648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x5639790ef550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d89d9b640 1 --2- 172.21.15.104:0/163602648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790efa90 0x5639790f9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d8ad9d640 1 -- 172.21.15.104:0/163602648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 msgr2=0x5639790ef550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d8ad9d640 1 --2- 172.21.15.104:0/163602648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x5639790ef550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d8ad9d640 1 -- 172.21.15.104:0/163602648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790efa90 msgr2=0x5639790f9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d8ad9d640 1 --2- 172.21.15.104:0/163602648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790efa90 0x5639790f9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d8ad9d640 1 -- 172.21.15.104:0/163602648 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56397900d9b0 con 0x5639790fa350 2024-06-09T22:23:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d8ad9d640 1 --2- 172.21.15.104:0/163602648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 0x5639790fc740 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009f40 tx=0x7f8d840107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c33220bd136673c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d78ff9640 1 -- 172.21.15.104:0/163602648 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84003b70 con 0x5639790fa350 2024-06-09T22:23:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d78ff9640 1 -- 172.21.15.104:0/163602648 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d84003d10 con 0x5639790fa350 2024-06-09T22:23:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.788+0000 7f8d78ff9640 1 -- 172.21.15.104:0/163602648 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84011a20 con 0x5639790fa350 2024-06-09T22:23:24.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.789+0000 7f8d937f3740 1 -- 172.21.15.104:0/163602648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 msgr2=0x5639790fc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.789+0000 7f8d937f3740 1 --2- 172.21.15.104:0/163602648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 0x5639790fc740 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009f40 tx=0x7f8d840107a0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.789+0000 7f8d937f3740 1 -- 172.21.15.104:0/163602648 shutdown_connections 2024-06-09T22:23:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.789+0000 7f8d937f3740 1 --2- 172.21.15.104:0/163602648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 0x5639790fc740 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.789+0000 7f8d937f3740 1 --2- 172.21.15.104:0/163602648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790efa90 0x5639790f9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.789+0000 7f8d937f3740 1 --2- 172.21.15.104:0/163602648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x5639790ef550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.789+0000 7f8d937f3740 1 -- 172.21.15.104:0/163602648 >> 172.21.15.104:0/163602648 conn(0x5639790eb680 msgr2=0x5639790eba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.789+0000 7f8d937f3740 1 -- 172.21.15.104:0/163602648 shutdown_connections 2024-06-09T22:23:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.789+0000 7f8d937f3740 1 -- 172.21.15.104:0/163602648 wait complete. 2024-06-09T22:23:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.789+0000 7f8d937f3740 1 -- start start 2024-06-09T22:23:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d937f3740 1 Processor -- start 2024-06-09T22:23:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d937f3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x5639790f7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d937f3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790efa90 0x5639790f78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d937f3740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 0x5639790f7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d89d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790efa90 0x5639790f78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d8ad9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 0x5639790f7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d937f3740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5639790fcf80 con 0x5639790fa350 2024-06-09T22:23:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d89d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790efa90 0x5639790f78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:49306/0 (socket says 172.21.15.104:49306) 2024-06-09T22:23:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d89d9b640 1 -- 172.21.15.104:0/1588684892 learned_addr learned my addr 172.21.15.104:0/1588684892 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d8ad9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 0x5639790f7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46922/0 (socket says 172.21.15.104:46922) 2024-06-09T22:23:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d937f3740 1 -- 172.21.15.104:0/1588684892 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563978fdcae0 con 0x5639790efa90 2024-06-09T22:23:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d8a59c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x5639790f7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d937f3740 1 -- 172.21.15.104:0/1588684892 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5639790fd100 con 0x5639790ef170 2024-06-09T22:23:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.790+0000 7f8d937f3740 10 client.0 _ll_register_callbacks cb 0x5639790159b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d8a59c640 1 -- 172.21.15.104:0/1588684892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790efa90 msgr2=0x5639790f78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d8a59c640 1 --2- 172.21.15.104:0/1588684892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790efa90 0x5639790f78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d8a59c640 1 -- 172.21.15.104:0/1588684892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 msgr2=0x5639790f7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d8a59c640 1 --2- 172.21.15.104:0/1588684892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 0x5639790f7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d8a59c640 1 -- 172.21.15.104:0/1588684892 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563978d67210 con 0x5639790ef170 2024-06-09T22:23:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d8a59c640 1 --2- 172.21.15.104:0/1588684892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x5639790f7360 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c011010 tx=0x7f8d7c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c516f0e3b5e966ea server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:23:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d71ffb640 1 -- 172.21.15.104:0/1588684892 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c01fa00 con 0x5639790ef170 2024-06-09T22:23:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d937f3740 1 -- 172.21.15.104:0/1588684892 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d84003d10 con 0x5639790ef170 2024-06-09T22:23:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d71ffb640 1 -- 172.21.15.104:0/1588684892 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8d7c01fba0 con 0x5639790ef170 2024-06-09T22:23:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d71ffb640 1 -- 172.21.15.104:0/1588684892 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8d7c01fda0 con 0x5639790ef170 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d937f3740 10 client.25753 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.791+0000 7f8d937f3740 1 -- 172.21.15.104:0/1588684892 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563978e33620 con 0x5639790ef170 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.792+0000 7f8d71ffb640 1 -- 172.21.15.104:0/1588684892 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d84003d10 con 0x5639790ef170 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.792+0000 7f8d71ffb640 1 -- 172.21.15.104:0/1588684892 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8d7c026070 con 0x5639790ef170 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.792+0000 7f8d937f3740 10 client.25753 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.792+0000 7f8d937f3740 10 client.25753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.792+0000 7f8d937f3740 20 client.25753 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.792+0000 7f8d937f3740 1 --2- 172.21.15.104:0/1588684892 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56397910a5d0 0x56397910ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.792+0000 7f8d937f3740 4 client.25753 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.792+0000 7f8d937f3740 1 -- 172.21.15.104:0/1588684892 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563979066f10 con 0x56397910a5d0 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.793+0000 7f8d89d9b640 1 --2- 172.21.15.104:0/1588684892 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56397910a5d0 0x56397910ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.793+0000 7f8d89d9b640 1 --2- 172.21.15.104:0/1588684892 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56397910a5d0 0x56397910ca40 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86eb060cc40a7b0c server_cookie=fabdfc3132fc0e25 in_seq=0 out_seq=0 2024-06-09T22:23:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.793+0000 7f8d71ffb640 10 client.25753 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d71ffb640 1 -- 172.21.15.104:0/1588684892 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x563979066f10 con 0x56397910a5d0 2024-06-09T22:23:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d71ffb640 10 client.25753 handle_command_reply: tid=0 2024-06-09T22:23:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d937f3740 1 client.25753 shutdown 2024-06-09T22:23:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d937f3740 1 -- 172.21.15.104:0/1588684892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 msgr2=0x5639790f7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d937f3740 1 --2- 172.21.15.104:0/1588684892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x5639790f7360 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c011010 tx=0x7f8d7c004730 comp rx=0 tx=0).stop 2024-06-09T22:23:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d937f3740 1 -- 172.21.15.104:0/1588684892 shutdown_connections 2024-06-09T22:23:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d937f3740 1 --2- 172.21.15.104:0/1588684892 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56397910a5d0 0x56397910ca40 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d937f3740 1 --2- 172.21.15.104:0/1588684892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790fa350 0x5639790f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d937f3740 1 --2- 172.21.15.104:0/1588684892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790efa90 0x5639790f78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d937f3740 1 --2- 172.21.15.104:0/1588684892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x5639790f7360 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d71ffb640 0 client.25753 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d937f3740 1 -- 172.21.15.104:0/1588684892 >> 172.21.15.104:0/1588684892 conn(0x5639790eb680 msgr2=0x5639790f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d937f3740 1 -- 172.21.15.104:0/1588684892 shutdown_connections 2024-06-09T22:23:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d937f3740 1 -- 172.21.15.104:0/1588684892 wait complete. 2024-06-09T22:23:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.794+0000 7f8d937f3740 20 client.25753 trim_cache size 0 max 16384 2024-06-09T22:23:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.809+0000 7f8d937f3740 1 Processor -- start 2024-06-09T22:23:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.809+0000 7f8d937f3740 1 -- start start 2024-06-09T22:23:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d937f3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790ef170 0x5639790eaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d937f3740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 0x563979112350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d937f3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790fa350 0x563979112890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d937f3740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1b0f60 con 0x5639790efa90 2024-06-09T22:23:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d937f3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1b2b00 con 0x5639790ef170 2024-06-09T22:23:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d937f3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1b1630 con 0x5639790fa350 2024-06-09T22:23:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d89d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 0x563979112350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d89d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 0x563979112350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46936/0 (socket says 172.21.15.104:46936) 2024-06-09T22:23:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d89d9b640 1 -- 172.21.15.104:0/1949947273 learned_addr learned my addr 172.21.15.104:0/1949947273 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d8a59c640 1 --2- 172.21.15.104:0/1949947273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790ef170 0x5639790eaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d8ad9d640 1 --2- 172.21.15.104:0/1949947273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790fa350 0x563979112890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d89d9b640 1 -- 172.21.15.104:0/1949947273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790fa350 msgr2=0x563979112890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d89d9b640 1 --2- 172.21.15.104:0/1949947273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790fa350 0x563979112890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d89d9b640 1 -- 172.21.15.104:0/1949947273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790ef170 msgr2=0x5639790eaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d89d9b640 1 --2- 172.21.15.104:0/1949947273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790ef170 0x5639790eaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d89d9b640 1 -- 172.21.15.104:0/1949947273 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563979109110 con 0x5639790efa90 2024-06-09T22:23:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d89d9b640 1 --2- 172.21.15.104:0/1949947273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 0x563979112350 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8d80004650 tx=0x7f8d80031a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12474159bf3f5bb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d73fff640 1 -- 172.21.15.104:0/1949947273 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80049070 con 0x5639790efa90 2024-06-09T22:23:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d73fff640 1 -- 172.21.15.104:0/1949947273 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d800bc6f0 con 0x5639790efa90 2024-06-09T22:23:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.810+0000 7f8d73fff640 1 -- 172.21.15.104:0/1949947273 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d800ed3e0 con 0x5639790efa90 2024-06-09T22:23:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.811+0000 7f8d937f3740 1 -- 172.21.15.104:0/1949947273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 msgr2=0x563979112350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.811+0000 7f8d937f3740 1 --2- 172.21.15.104:0/1949947273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 0x563979112350 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8d80004650 tx=0x7f8d80031a60 comp rx=0 tx=0).stop 2024-06-09T22:23:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.811+0000 7f8d937f3740 1 -- 172.21.15.104:0/1949947273 shutdown_connections 2024-06-09T22:23:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.811+0000 7f8d937f3740 1 --2- 172.21.15.104:0/1949947273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790fa350 0x563979112890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.811+0000 7f8d937f3740 1 --2- 172.21.15.104:0/1949947273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 0x563979112350 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.811+0000 7f8d937f3740 1 --2- 172.21.15.104:0/1949947273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790ef170 0x5639790eaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.811+0000 7f8d937f3740 1 -- 172.21.15.104:0/1949947273 >> 172.21.15.104:0/1949947273 conn(0x5639790eb680 msgr2=0x5639790f9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.811+0000 7f8d937f3740 1 -- 172.21.15.104:0/1949947273 shutdown_connections 2024-06-09T22:23:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.811+0000 7f8d937f3740 1 -- 172.21.15.104:0/1949947273 wait complete. 2024-06-09T22:23:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d937f3740 1 -- start start 2024-06-09T22:23:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d937f3740 1 Processor -- start 2024-06-09T22:23:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d937f3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x563979102350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d937f3740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 0x563979102890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d937f3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790fa350 0x5639790ff320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d937f3740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1b23d0 con 0x5639790efa90 2024-06-09T22:23:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d937f3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1b1d00 con 0x5639790fa350 2024-06-09T22:23:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d937f3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1aef20 con 0x5639790ef170 2024-06-09T22:23:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d937f3740 10 client.0 _ll_register_callbacks cb 0x7f8d7c005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d89d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 0x563979102890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d89d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 0x563979102890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46952/0 (socket says 172.21.15.104:46952) 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d89d9b640 1 -- 172.21.15.104:0/406420216 learned_addr learned my addr 172.21.15.104:0/406420216 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d8a59c640 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x563979102350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d8ad9d640 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790fa350 0x5639790ff320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d89d9b640 1 -- 172.21.15.104:0/406420216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 msgr2=0x563979102350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d89d9b640 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x563979102350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d89d9b640 1 -- 172.21.15.104:0/406420216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790fa350 msgr2=0x5639790ff320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d89d9b640 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790fa350 0x5639790ff320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d89d9b640 1 -- 172.21.15.104:0/406420216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563979102ed0 con 0x5639790efa90 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.812+0000 7f8d89d9b640 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 0x563979102890 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f8d800bbcf0 tx=0x7f8d800c4de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac88c879de60ab1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d71ffb640 1 -- 172.21.15.104:0/406420216 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80049050 con 0x5639790efa90 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d71ffb640 1 -- 172.21.15.104:0/406420216 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8d800b8660 con 0x5639790efa90 2024-06-09T22:23:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d937f3740 1 -- 172.21.15.104:0/406420216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d800bc6f0 con 0x5639790efa90 2024-06-09T22:23:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d71ffb640 1 -- 172.21.15.104:0/406420216 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8d800bbff0 con 0x5639790efa90 2024-06-09T22:23:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d937f3740 10 client.16224 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d937f3740 1 -- 172.21.15.104:0/406420216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563979175840 con 0x5639790efa90 2024-06-09T22:23:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d71ffb640 1 -- 172.21.15.104:0/406420216 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d800bc6f0 con 0x5639790efa90 2024-06-09T22:23:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d71ffb640 1 -- 172.21.15.104:0/406420216 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8d800c48d0 con 0x5639790efa90 2024-06-09T22:23:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d937f3740 10 client.16224 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d937f3740 10 client.16224 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d937f3740 20 client.16224 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d937f3740 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x563979175df0 0x563979178210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d937f3740 4 client.16224 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:23:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d937f3740 1 -- 172.21.15.104:0/406420216 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563979060c20 con 0x563979175df0 2024-06-09T22:23:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.813+0000 7f8d8a59c640 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x563979175df0 0x563979178210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.814+0000 7f8d71ffb640 10 client.16224 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.814+0000 7f8d8a59c640 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x563979175df0 0x563979178210 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24b9a92b1f3c9066 server_cookie=be2b720733edebd in_seq=0 out_seq=0 2024-06-09T22:23:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d71ffb640 1 -- 172.21.15.104:0/406420216 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x563979060c20 con 0x563979175df0 2024-06-09T22:23:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d71ffb640 10 client.16224 handle_command_reply: tid=0 2024-06-09T22:23:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d937f3740 1 client.16224 shutdown 2024-06-09T22:23:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d937f3740 1 -- 172.21.15.104:0/406420216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 msgr2=0x563979102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d937f3740 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 0x563979102890 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f8d800bbcf0 tx=0x7f8d800c4de0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d937f3740 1 -- 172.21.15.104:0/406420216 shutdown_connections 2024-06-09T22:23:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d937f3740 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x563979175df0 0x563979178210 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d937f3740 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5639790fa350 0x5639790ff320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d937f3740 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5639790efa90 0x563979102890 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d71ffb640 0 client.16224 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d937f3740 1 --2- 172.21.15.104:0/406420216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639790ef170 0x563979102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d937f3740 1 -- 172.21.15.104:0/406420216 >> 172.21.15.104:0/406420216 conn(0x5639790eb680 msgr2=0x5639790f7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d937f3740 1 -- 172.21.15.104:0/406420216 shutdown_connections 2024-06-09T22:23:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.825+0000 7f8d937f3740 1 -- 172.21.15.104:0/406420216 wait complete. 2024-06-09T22:23:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d937f3740 20 client.16224 trim_cache size 0 max 16384 2024-06-09T22:23:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:23:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:23:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:23:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d90ae1640 1 -- 172.21.15.104:0/3337997039 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8d740a2dd0 msgr2=0x7f8d740a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d90ae1640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8d740a2dd0 0x7f8d740a5290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c005d00 tx=0x7f8d7c00b410 comp rx=0 tx=0).stop 2024-06-09T22:23:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d90ae1640 1 -- 172.21.15.104:0/3337997039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c103540 msgr2=0x7f8d8c100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d90ae1640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c103540 0x7f8d8c100db0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8d80006010 tx=0x7f8d800042f0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d90ae1640 1 -- 172.21.15.104:0/3337997039 shutdown_connections 2024-06-09T22:23:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d90ae1640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8d740a2dd0 0x7f8d740a5290 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d90ae1640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d8c10bff0 0x7f8d8c1012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d90ae1640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c103540 0x7f8d8c100db0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d90ae1640 1 --2- 172.21.15.104:0/3337997039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d8c102c20 0x7f8d8c100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d90ae1640 1 -- 172.21.15.104:0/3337997039 >> 172.21.15.104:0/3337997039 conn(0x7f8d8c0779f0 msgr2=0x7f8d8c0fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d90ae1640 1 -- 172.21.15.104:0/3337997039 shutdown_connections 2024-06-09T22:23:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:24.826+0000 7f8d90ae1640 1 -- 172.21.15.104:0/3337997039 wait complete. 2024-06-09T22:23:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.288+0000 7f934811f700 1 -- 172.21.15.104:0/1226691135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9340108530 msgr2=0x7f9340108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.288+0000 7f934811f700 1 --2- 172.21.15.104:0/1226691135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9340108530 0x7f9340108910 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f933c009990 tx=0x7f933c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.288+0000 7f934811f700 1 -- 172.21.15.104:0/1226691135 shutdown_connections 2024-06-09T22:23:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.288+0000 7f934811f700 1 --2- 172.21.15.104:0/1226691135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9340108530 0x7f9340108910 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.288+0000 7f934811f700 1 --2- 172.21.15.104:0/1226691135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f934006b5c0 0x7f934010f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.288+0000 7f934811f700 1 --2- 172.21.15.104:0/1226691135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f934006ac00 0x7f934006b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.288+0000 7f934811f700 1 -- 172.21.15.104:0/1226691135 >> 172.21.15.104:0/1226691135 conn(0x7f9340077400 msgr2=0x7f9340077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.288+0000 7f934811f700 1 -- 172.21.15.104:0/1226691135 shutdown_connections 2024-06-09T22:23:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.288+0000 7f934811f700 1 -- 172.21.15.104:0/1226691135 wait complete. 2024-06-09T22:23:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f934811f700 1 Processor -- start 2024-06-09T22:23:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f934811f700 1 -- start start 2024-06-09T22:23:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f934811f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f934006ac00 0x7f934019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f934811f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f934006b5c0 0x7f934019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f934811f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9340108530 0x7f93401989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f934811f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9340112780 con 0x7f9340108530 2024-06-09T22:23:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f934811f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9340112600 con 0x7f934006ac00 2024-06-09T22:23:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f934811f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9340112900 con 0x7f934006b5c0 2024-06-09T22:23:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f93466bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9340108530 0x7f93401989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f93466bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9340108530 0x7f93401989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46960/0 (socket says 172.21.15.104:46960) 2024-06-09T22:23:25.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f93466bc700 1 -- 172.21.15.104:0/2064561085 learned_addr learned my addr 172.21.15.104:0/2064561085 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:25.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f9345ebb700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f934006ac00 0x7f934019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f93456ba700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f934006b5c0 0x7f934019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f93466bc700 1 -- 172.21.15.104:0/2064561085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f934006b5c0 msgr2=0x7f934019ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:25.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f93466bc700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f934006b5c0 0x7f934019ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f93466bc700 1 -- 172.21.15.104:0/2064561085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f934006ac00 msgr2=0x7f934019e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:25.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f93466bc700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f934006ac00 0x7f934019e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f93466bc700 1 -- 172.21.15.104:0/2064561085 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9340199140 con 0x7f9340108530 2024-06-09T22:23:25.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.289+0000 7f93456ba700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f934006b5c0 0x7f934019ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:23:25.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.290+0000 7f93466bc700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9340108530 0x7f93401989e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f933800ada0 tx=0x7f933800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:25.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.290+0000 7f9336ffd700 1 -- 172.21.15.104:0/2064561085 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9338017020 con 0x7f9340108530 2024-06-09T22:23:25.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.290+0000 7f9336ffd700 1 -- 172.21.15.104:0/2064561085 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9338004bb0 con 0x7f9340108530 2024-06-09T22:23:25.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.290+0000 7f934811f700 1 -- 172.21.15.104:0/2064561085 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9340199430 con 0x7f9340108530 2024-06-09T22:23:25.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.290+0000 7f9336ffd700 1 -- 172.21.15.104:0/2064561085 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9338005520 con 0x7f9340108530 2024-06-09T22:23:25.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.290+0000 7f934811f700 1 -- 172.21.15.104:0/2064561085 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93401a56f0 con 0x7f9340108530 2024-06-09T22:23:25.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.292+0000 7f9336ffd700 1 -- 172.21.15.104:0/2064561085 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f93380056c0 con 0x7f9340108530 2024-06-09T22:23:25.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.293+0000 7f9336ffd700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f932c0a2e10 0x7f932c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.293+0000 7f9345ebb700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f932c0a2e10 0x7f932c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.293+0000 7f9336ffd700 1 -- 172.21.15.104:0/2064561085 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f93380cee60 con 0x7f9340108530 2024-06-09T22:23:25.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.293+0000 7f9345ebb700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f932c0a2e10 0x7f932c0a52d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f933c006010 tx=0x7f933c00b540 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:25.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.293+0000 7f93529a2b80 1 Processor -- start 2024-06-09T22:23:25.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f93529a2b80 1 -- start start 2024-06-09T22:23:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f93529a2b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9105a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f93529a2b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff90ff9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f93529a2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff9100340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f93529a2b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564ff9051810 con 0x564ff9105640 2024-06-09T22:23:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f93529a2b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564ff8ff9720 con 0x564ff9105f60 2024-06-09T22:23:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f93529a2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564ff904d7d0 con 0x564ff90fff20 2024-06-09T22:23:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f9345ebb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9105a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f9345ebb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9105a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46976/0 (socket says 172.21.15.104:46976) 2024-06-09T22:23:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f9345ebb700 1 -- 172.21.15.104:0/2308875786 learned_addr learned my addr 172.21.15.104:0/2308875786 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f93456ba700 1 --2- 172.21.15.104:0/2308875786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff90ff9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f93466bc700 1 --2- 172.21.15.104:0/2308875786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff9100340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f9345ebb700 1 -- 172.21.15.104:0/2308875786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 msgr2=0x564ff9100340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f9345ebb700 1 --2- 172.21.15.104:0/2308875786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff9100340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f9345ebb700 1 -- 172.21.15.104:0/2308875786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 msgr2=0x564ff90ff9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f9345ebb700 1 --2- 172.21.15.104:0/2308875786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff90ff9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.294+0000 7f9345ebb700 1 -- 172.21.15.104:0/2308875786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ff8feff50 con 0x564ff9105640 2024-06-09T22:23:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.295+0000 7f9345ebb700 1 --2- 172.21.15.104:0/2308875786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9105a20 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f933c015000 tx=0x7f933c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e899b88065dcd75f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.295+0000 7f932bfff700 1 -- 172.21.15.104:0/2308875786 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c00bdc0 con 0x564ff9105640 2024-06-09T22:23:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.295+0000 7f932bfff700 1 -- 172.21.15.104:0/2308875786 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f933c00f940 con 0x564ff9105640 2024-06-09T22:23:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.295+0000 7f932bfff700 1 -- 172.21.15.104:0/2308875786 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c0248b0 con 0x564ff9105640 2024-06-09T22:23:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.295+0000 7f93529a2b80 1 -- 172.21.15.104:0/2308875786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 msgr2=0x564ff9105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.295+0000 7f93529a2b80 1 --2- 172.21.15.104:0/2308875786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9105a20 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f933c015000 tx=0x7f933c01f000 comp rx=0 tx=0).stop 2024-06-09T22:23:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.296+0000 7f93529a2b80 1 -- 172.21.15.104:0/2308875786 shutdown_connections 2024-06-09T22:23:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.296+0000 7f93529a2b80 1 --2- 172.21.15.104:0/2308875786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9105a20 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.296+0000 7f93529a2b80 1 --2- 172.21.15.104:0/2308875786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff9100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.296+0000 7f93529a2b80 1 --2- 172.21.15.104:0/2308875786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff90ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.296+0000 7f93529a2b80 1 -- 172.21.15.104:0/2308875786 >> 172.21.15.104:0/2308875786 conn(0x564ff902acc0 msgr2=0x564ff90fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.296+0000 7f93529a2b80 1 -- 172.21.15.104:0/2308875786 shutdown_connections 2024-06-09T22:23:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.296+0000 7f93529a2b80 1 -- 172.21.15.104:0/2308875786 wait complete. 2024-06-09T22:23:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.296+0000 7f93529a2b80 1 -- start start 2024-06-09T22:23:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.296+0000 7f93529a2b80 1 Processor -- start 2024-06-09T22:23:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93529a2b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff90fff20 0x564ff910bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93529a2b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9108fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93529a2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff9105f60 0x564ff91094f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93529a2b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564ff910c910 con 0x564ff9105640 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93529a2b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564ff8ff1bf0 con 0x564ff90fff20 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93529a2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564ff910ca90 con 0x564ff9105f60 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93529a2b80 10 client.0 _ll_register_callbacks cb 0x564ff8feb920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93456ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9108fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93456ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9108fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46988/0 (socket says 172.21.15.104:46988) 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93456ba700 1 -- 172.21.15.104:0/1358900531 learned_addr learned my addr 172.21.15.104:0/1358900531 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f9345ebb700 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff90fff20 0x564ff910bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93466bc700 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff9105f60 0x564ff91094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93456ba700 1 -- 172.21.15.104:0/1358900531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff9105f60 msgr2=0x564ff91094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93456ba700 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff9105f60 0x564ff91094f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93456ba700 1 -- 172.21.15.104:0/1358900531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff90fff20 msgr2=0x564ff910bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93456ba700 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff90fff20 0x564ff910bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93456ba700 1 -- 172.21.15.104:0/1358900531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ff8fa08d0 con 0x564ff9105640 2024-06-09T22:23:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93466bc700 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff9105f60 0x564ff91094f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93456ba700 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9108fb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f9330010a50 tx=0x7f9330010d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7e22fad45f07fb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93297fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93297fa700 1 -- 172.21.15.104:0/1358900531 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933001d060 con 0x564ff9105640 2024-06-09T22:23:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93297fa700 1 -- 172.21.15.104:0/1358900531 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f933000ecb0 con 0x564ff9105640 2024-06-09T22:23:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.297+0000 7f93529a2b80 1 -- 172.21.15.104:0/1358900531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f933c00f940 con 0x564ff9105640 2024-06-09T22:23:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.298+0000 7f93297fa700 1 -- 172.21.15.104:0/1358900531 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9330024050 con 0x564ff9105640 2024-06-09T22:23:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.298+0000 7f93529a2b80 10 client.16236 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.298+0000 7f93529a2b80 1 -- 172.21.15.104:0/1358900531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564ff9117280 con 0x564ff9105640 2024-06-09T22:23:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.298+0000 7f93297fa700 1 -- 172.21.15.104:0/1358900531 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f933c00f940 con 0x564ff9105640 2024-06-09T22:23:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.298+0000 7f93297fa700 1 -- 172.21.15.104:0/1358900531 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f933000d3a0 con 0x564ff9105640 2024-06-09T22:23:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.298+0000 7f93529a2b80 10 client.16236 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.298+0000 7f93529a2b80 10 client.16236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.298+0000 7f93529a2b80 20 client.16236 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.298+0000 7f93529a2b80 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564ff9119960 0x564ff911bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.298+0000 7f93529a2b80 4 client.16236 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.298+0000 7f93529a2b80 1 -- 172.21.15.104:0/1358900531 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ff905ed10 con 0x564ff9119960 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.298+0000 7f9345ebb700 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564ff9119960 0x564ff911bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.299+0000 7f9345ebb700 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564ff9119960 0x564ff911bd50 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd4c59ccea093f55 server_cookie=71afa4dc3e243a19 in_seq=0 out_seq=0 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.299+0000 7f93297fa700 10 client.16236 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93297fa700 1 -- 172.21.15.104:0/1358900531 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x564ff905ed10 con 0x564ff9119960 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93297fa700 10 client.16236 handle_command_reply: tid=0 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93529a2b80 1 client.16236 shutdown 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93529a2b80 1 -- 172.21.15.104:0/1358900531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 msgr2=0x564ff9108fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:25.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93529a2b80 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9108fb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f9330010a50 tx=0x7f9330010d60 comp rx=0 tx=0).stop 2024-06-09T22:23:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93529a2b80 1 -- 172.21.15.104:0/1358900531 shutdown_connections 2024-06-09T22:23:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93529a2b80 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564ff9119960 0x564ff911bd50 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93529a2b80 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff9105f60 0x564ff91094f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93297fa700 0 client.16236 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93529a2b80 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff90fff20 0x564ff910bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93529a2b80 1 --2- 172.21.15.104:0/1358900531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9108fb0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93529a2b80 1 -- 172.21.15.104:0/1358900531 >> 172.21.15.104:0/1358900531 conn(0x564ff902acc0 msgr2=0x564ff90fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93529a2b80 1 -- 172.21.15.104:0/1358900531 shutdown_connections 2024-06-09T22:23:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93529a2b80 1 -- 172.21.15.104:0/1358900531 wait complete. 2024-06-09T22:23:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.300+0000 7f93529a2b80 20 client.16236 trim_cache size 0 max 16384 2024-06-09T22:23:25.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93529a2b80 1 Processor -- start 2024-06-09T22:23:25.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93529a2b80 1 -- start start 2024-06-09T22:23:25.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93529a2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff90ff9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93529a2b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9109520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93529a2b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff9106a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93529a2b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f93401ae580 con 0x564ff9105640 2024-06-09T22:23:25.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93529a2b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f93401b0120 con 0x564ff9105f60 2024-06-09T22:23:25.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93529a2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93401aec50 con 0x564ff90fff20 2024-06-09T22:23:25.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93456ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93456ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46992/0 (socket says 172.21.15.104:46992) 2024-06-09T22:23:25.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93456ba700 1 -- 172.21.15.104:0/527418110 learned_addr learned my addr 172.21.15.104:0/527418110 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:25.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93466bc700 1 --2- 172.21.15.104:0/527418110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff9106a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f9345ebb700 1 --2- 172.21.15.104:0/527418110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff90ff9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93456ba700 1 -- 172.21.15.104:0/527418110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 msgr2=0x564ff90ff9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:25.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93456ba700 1 --2- 172.21.15.104:0/527418110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff90ff9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93456ba700 1 -- 172.21.15.104:0/527418110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 msgr2=0x564ff9106a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:25.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93456ba700 1 --2- 172.21.15.104:0/527418110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff9106a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f93456ba700 1 -- 172.21.15.104:0/527418110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ff9106f90 con 0x564ff9105640 2024-06-09T22:23:25.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.315+0000 7f9345ebb700 1 --2- 172.21.15.104:0/527418110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff90ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:25.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.316+0000 7f93456ba700 1 --2- 172.21.15.104:0/527418110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9109520 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f933000d930 tx=0x7f933000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a3c8c7d919e1f5 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:25.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.316+0000 7f932b7fe700 1 -- 172.21.15.104:0/527418110 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933000eab0 con 0x564ff9105640 2024-06-09T22:23:25.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.316+0000 7f932b7fe700 1 -- 172.21.15.104:0/527418110 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f933002d070 con 0x564ff9105640 2024-06-09T22:23:25.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.316+0000 7f932b7fe700 1 -- 172.21.15.104:0/527418110 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93300288b0 con 0x564ff9105640 2024-06-09T22:23:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.316+0000 7f93529a2b80 1 -- 172.21.15.104:0/527418110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 msgr2=0x564ff9109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.316+0000 7f93529a2b80 1 --2- 172.21.15.104:0/527418110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9109520 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f933000d930 tx=0x7f933000b8e0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.316+0000 7f93529a2b80 1 -- 172.21.15.104:0/527418110 shutdown_connections 2024-06-09T22:23:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.316+0000 7f93529a2b80 1 --2- 172.21.15.104:0/527418110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff9106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.316+0000 7f93529a2b80 1 --2- 172.21.15.104:0/527418110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff90ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.316+0000 7f93529a2b80 1 --2- 172.21.15.104:0/527418110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9109520 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.316+0000 7f93529a2b80 1 -- 172.21.15.104:0/527418110 >> 172.21.15.104:0/527418110 conn(0x564ff902acc0 msgr2=0x564ff9106340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.317+0000 7f93529a2b80 1 -- 172.21.15.104:0/527418110 shutdown_connections 2024-06-09T22:23:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.317+0000 7f93529a2b80 1 -- 172.21.15.104:0/527418110 wait complete. 2024-06-09T22:23:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.317+0000 7f93529a2b80 1 -- start start 2024-06-09T22:23:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.317+0000 7f93529a2b80 1 Processor -- start 2024-06-09T22:23:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.317+0000 7f93529a2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff91122c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.317+0000 7f93529a2b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9120cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.317+0000 7f93529a2b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff9121230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.317+0000 7f93529a2b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f93401af9f0 con 0x564ff9105640 2024-06-09T22:23:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.317+0000 7f93529a2b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f93401af320 con 0x564ff9105f60 2024-06-09T22:23:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.317+0000 7f93529a2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93401ac500 con 0x564ff90fff20 2024-06-09T22:23:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93529a2b80 10 client.0 _ll_register_callbacks cb 0x564ff8feb920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93456ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9120cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93456ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9120cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46994/0 (socket says 172.21.15.104:46994) 2024-06-09T22:23:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93456ba700 1 -- 172.21.15.104:0/2491121545 learned_addr learned my addr 172.21.15.104:0/2491121545 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f9345ebb700 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff91122c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93466bc700 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff9121230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93456ba700 1 -- 172.21.15.104:0/2491121545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 msgr2=0x564ff91122c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93456ba700 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff91122c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93456ba700 1 -- 172.21.15.104:0/2491121545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 msgr2=0x564ff9121230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93456ba700 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff9121230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93456ba700 1 -- 172.21.15.104:0/2491121545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ff91217f0 con 0x564ff9105640 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93466bc700 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff9121230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93456ba700 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9120cf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f933000ba10 tx=0x7f9330010d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e9c917ad7cc6a3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93297fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93297fa700 1 -- 172.21.15.104:0/2491121545 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933001d050 con 0x564ff9105640 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93529a2b80 1 -- 172.21.15.104:0/2491121545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f933002d070 con 0x564ff9105640 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93297fa700 1 -- 172.21.15.104:0/2491121545 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f933000d9b0 con 0x564ff9105640 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93297fa700 1 -- 172.21.15.104:0/2491121545 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f933000dbb0 con 0x564ff9105640 2024-06-09T22:23:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93529a2b80 10 client.16242 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.318+0000 7f93529a2b80 1 -- 172.21.15.104:0/2491121545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564ff9184380 con 0x564ff9105640 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.319+0000 7f93297fa700 1 -- 172.21.15.104:0/2491121545 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f933002d070 con 0x564ff9105640 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.319+0000 7f93297fa700 1 -- 172.21.15.104:0/2491121545 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9330031450 con 0x564ff9105640 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.319+0000 7f93529a2b80 10 client.16242 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.319+0000 7f93529a2b80 10 client.16242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.319+0000 7f93529a2b80 20 client.16242 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.319+0000 7f93529a2b80 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564ff9119960 0x564ff9186a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.319+0000 7f93529a2b80 4 client.16242 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.319+0000 7f93529a2b80 1 -- 172.21.15.104:0/2491121545 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564ff9059290 con 0x564ff9119960 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.319+0000 7f9345ebb700 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564ff9119960 0x564ff9186a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f9345ebb700 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564ff9119960 0x564ff9186a10 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11222754ee0030cc server_cookie=b1f8734c14d3fbb6 in_seq=0 out_seq=0 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93297fa700 10 client.16242 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:25.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93297fa700 1 -- 172.21.15.104:0/2491121545 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564ff9059290 con 0x564ff9119960 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93297fa700 10 client.16242 handle_command_reply: tid=0 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93529a2b80 1 client.16242 shutdown 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93529a2b80 1 -- 172.21.15.104:0/2491121545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 msgr2=0x564ff9120cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93529a2b80 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9120cf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f933000ba10 tx=0x7f9330010d60 comp rx=0 tx=0).stop 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93529a2b80 1 -- 172.21.15.104:0/2491121545 shutdown_connections 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93529a2b80 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564ff9119960 0x564ff9186a10 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93529a2b80 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ff9105f60 0x564ff9121230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93297fa700 0 client.16242 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93529a2b80 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564ff90fff20 0x564ff91122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93529a2b80 1 --2- 172.21.15.104:0/2491121545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564ff9105640 0x564ff9120cf0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93529a2b80 1 -- 172.21.15.104:0/2491121545 >> 172.21.15.104:0/2491121545 conn(0x564ff902acc0 msgr2=0x564ff90f3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93529a2b80 1 -- 172.21.15.104:0/2491121545 shutdown_connections 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93529a2b80 1 -- 172.21.15.104:0/2491121545 wait complete. 2024-06-09T22:23:25.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.320+0000 7f93529a2b80 20 client.16242 trim_cache size 0 max 16384 2024-06-09T22:23:25.326 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:23:25.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.321+0000 7f934811f700 1 -- 172.21.15.104:0/2064561085 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f932c0a2e10 msgr2=0x7f932c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:25.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.321+0000 7f934811f700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f932c0a2e10 0x7f932c0a52d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f933c006010 tx=0x7f933c00b540 comp rx=0 tx=0).stop 2024-06-09T22:23:25.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.321+0000 7f934811f700 1 -- 172.21.15.104:0/2064561085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9340108530 msgr2=0x7f93401989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:25.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.321+0000 7f934811f700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9340108530 0x7f93401989e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f933800ada0 tx=0x7f933800c330 comp rx=0 tx=0).stop 2024-06-09T22:23:25.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.321+0000 7f934811f700 1 -- 172.21.15.104:0/2064561085 shutdown_connections 2024-06-09T22:23:25.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.321+0000 7f934811f700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f932c0a2e10 0x7f932c0a52d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.321+0000 7f934811f700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9340108530 0x7f93401989e0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.321+0000 7f934811f700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f934006ac00 0x7f934019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.321+0000 7f934811f700 1 --2- 172.21.15.104:0/2064561085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f934006b5c0 0x7f934019ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:25.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.321+0000 7f934811f700 1 -- 172.21.15.104:0/2064561085 >> 172.21.15.104:0/2064561085 conn(0x7f9340077400 msgr2=0x7f9340100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:25.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.321+0000 7f934811f700 1 -- 172.21.15.104:0/2064561085 shutdown_connections 2024-06-09T22:23:25.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:25.321+0000 7f934811f700 1 -- 172.21.15.104:0/2064561085 wait complete. 2024-06-09T22:23:25.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 4.39 MB/sec warmup 31 sec latency 15243.917 ms 2024-06-09T22:23:25.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:25 smithi194 ceph-mon[36856]: pgmap v76: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 42 KiB/s wr, 24 op/s 2024-06-09T22:23:25.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:25 smithi153 ceph-mon[36599]: pgmap v76: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 42 KiB/s wr, 24 op/s 2024-06-09T22:23:26.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:25 smithi104 ceph-mon[32389]: pgmap v76: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 42 KiB/s wr, 24 op/s 2024-06-09T22:23:26.066 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:23:26.322 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:23:26.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 4.25 MB/sec warmup 32 sec latency 16243.979 ms 2024-06-09T22:23:27.170 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-09T22:23:27.170 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-09T22:23:27.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 4.13 MB/sec warmup 33 sec latency 17244.078 ms 2024-06-09T22:23:27.792 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:27 smithi104 ceph-mon[32389]: pgmap v77: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-06-09T22:23:27.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:27 smithi194 ceph-mon[36856]: pgmap v77: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-06-09T22:23:27.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:27 smithi153 ceph-mon[36599]: pgmap v77: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-06-09T22:23:28.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.155+0000 7f27256d5700 1 -- 172.21.15.104:0/189662750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2720101680 msgr2=0x7f2720101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:28.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.155+0000 7f27256d5700 1 --2- 172.21.15.104:0/189662750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2720101680 0x7f2720101a60 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f27100099c0 tx=0x7f2710009cd0 comp rx=0 tx=0).stop 2024-06-09T22:23:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.155+0000 7f27256d5700 1 -- 172.21.15.104:0/189662750 shutdown_connections 2024-06-09T22:23:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.155+0000 7f27256d5700 1 --2- 172.21.15.104:0/189662750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2720101680 0x7f2720101a60 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.155+0000 7f27256d5700 1 --2- 172.21.15.104:0/189662750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27201066d0 0x7f272010e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.155+0000 7f27256d5700 1 --2- 172.21.15.104:0/189662750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2720102030 0x7f2720105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.155+0000 7f27256d5700 1 -- 172.21.15.104:0/189662750 >> 172.21.15.104:0/189662750 conn(0x7f27200773e0 msgr2=0x7f27200777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.155+0000 7f27256d5700 1 -- 172.21.15.104:0/189662750 shutdown_connections 2024-06-09T22:23:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.156+0000 7f27256d5700 1 -- 172.21.15.104:0/189662750 wait complete. 2024-06-09T22:23:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.156+0000 7f27256d5700 1 Processor -- start 2024-06-09T22:23:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.156+0000 7f27256d5700 1 -- start start 2024-06-09T22:23:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.156+0000 7f27256d5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2720101680 0x7f2720199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.156+0000 7f27256d5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2720102030 0x7f272019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.156+0000 7f27256d5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27201066d0 0x7f27201a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.156+0000 7f27256d5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f27201117b0 con 0x7f2720102030 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.156+0000 7f27256d5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2720111630 con 0x7f2720101680 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.156+0000 7f27256d5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2720111930 con 0x7f27201066d0 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.156+0000 7f2717fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2720102030 0x7f272019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.156+0000 7f2717fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2720102030 0x7f272019a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57188/0 (socket says 172.21.15.104:57188) 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.156+0000 7f2717fff700 1 -- 172.21.15.104:0/1072188873 learned_addr learned my addr 172.21.15.104:0/1072188873 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f271f7fe700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27201066d0 0x7f27201a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f271effd700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2720101680 0x7f2720199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f2717fff700 1 -- 172.21.15.104:0/1072188873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27201066d0 msgr2=0x7f27201a4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f2717fff700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27201066d0 0x7f27201a4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f2717fff700 1 -- 172.21.15.104:0/1072188873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2720101680 msgr2=0x7f2720199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f2717fff700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2720101680 0x7f2720199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:28.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f2717fff700 1 -- 172.21.15.104:0/1072188873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27201a51f0 con 0x7f2720102030 2024-06-09T22:23:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f2717fff700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2720102030 0x7f272019a170 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f270800b500 tx=0x7f270800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f271cff9700 1 -- 172.21.15.104:0/1072188873 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708015020 con 0x7f2720102030 2024-06-09T22:23:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f27256d5700 1 -- 172.21.15.104:0/1072188873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27201a5420 con 0x7f2720102030 2024-06-09T22:23:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f271cff9700 1 -- 172.21.15.104:0/1072188873 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2708011d10 con 0x7f2720102030 2024-06-09T22:23:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f271cff9700 1 -- 172.21.15.104:0/1072188873 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270800f2e0 con 0x7f2720102030 2024-06-09T22:23:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.157+0000 7f27256d5700 1 -- 172.21.15.104:0/1072188873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27201a58e0 con 0x7f2720102030 2024-06-09T22:23:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.158+0000 7f27256d5700 1 -- 172.21.15.104:0/1072188873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2720069180 con 0x7f2720102030 2024-06-09T22:23:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.159+0000 7f271cff9700 1 -- 172.21.15.104:0/1072188873 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f2708018020 con 0x7f2720102030 2024-06-09T22:23:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.160+0000 7f271cff9700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f270c0a2d90 0x7f270c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.160+0000 7f271effd700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f270c0a2d90 0x7f270c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.160+0000 7f271cff9700 1 -- 172.21.15.104:0/1072188873 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f27080cf3b0 con 0x7f2720102030 2024-06-09T22:23:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.160+0000 7f271effd700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f270c0a2d90 0x7f270c0a5250 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2710006010 tx=0x7f2710000bc0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:28.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.164+0000 7f271cff9700 1 -- 172.21.15.104:0/1072188873 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f270801f020 con 0x7f2720102030 2024-06-09T22:23:28.253 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:23:28.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.423+0000 7f27256d5700 1 -- 172.21.15.104:0/1072188873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f272002dca0 con 0x7f2720102030 2024-06-09T22:23:28.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.423+0000 7f271cff9700 1 -- 172.21.15.104:0/1072188873 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11956 (secure 0 0 0) 0x7f270809bd70 con 0x7f2720102030 2024-06-09T22:23:28.424 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:23:28.424 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:22:59.692018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24419,24403,24442,24460,24448]},"id":1}]} 2024-06-09T22:23:28.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.426+0000 7f27256d5700 1 -- 172.21.15.104:0/1072188873 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f270c0a2d90 msgr2=0x7f270c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:28.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.426+0000 7f27256d5700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f270c0a2d90 0x7f270c0a5250 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2710006010 tx=0x7f2710000bc0 comp rx=0 tx=0).stop 2024-06-09T22:23:28.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.426+0000 7f27256d5700 1 -- 172.21.15.104:0/1072188873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2720102030 msgr2=0x7f272019a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:28.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.426+0000 7f27256d5700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2720102030 0x7f272019a170 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f270800b500 tx=0x7f270800b810 comp rx=0 tx=0).stop 2024-06-09T22:23:28.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.426+0000 7f27256d5700 1 -- 172.21.15.104:0/1072188873 shutdown_connections 2024-06-09T22:23:28.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.426+0000 7f27256d5700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f270c0a2d90 0x7f270c0a5250 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:28.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.426+0000 7f27256d5700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27201066d0 0x7f27201a4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:28.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.426+0000 7f27256d5700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2720101680 0x7f2720199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:28.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.426+0000 7f27256d5700 1 --2- 172.21.15.104:0/1072188873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2720102030 0x7f272019a170 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:28.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.426+0000 7f27256d5700 1 -- 172.21.15.104:0/1072188873 >> 172.21.15.104:0/1072188873 conn(0x7f27200773e0 msgr2=0x7f2720105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:28.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.426+0000 7f27256d5700 1 -- 172.21.15.104:0/1072188873 shutdown_connections 2024-06-09T22:23:28.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:28.426+0000 7f27256d5700 1 -- 172.21.15.104:0/1072188873 wait complete. 2024-06-09T22:23:28.427 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 30 2024-06-09T22:23:28.717 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 4.00 MB/sec warmup 34 sec latency 18244.116 ms 2024-06-09T22:23:28.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:28 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1072188873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:23:28.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:28 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1072188873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:23:28.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:28 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1072188873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:23:29.241 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24460 quiesce db --set-id 6f4b8fc4 --release --await' 2024-06-09T22:23:29.241 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24460 quiesce db --set-id 6f4b8fc4 --release --await 2024-06-09T22:23:29.718 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 3.89 MB/sec warmup 35 sec latency 19244.152 ms 2024-06-09T22:23:29.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:29 smithi194 ceph-mon[36856]: pgmap v78: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-06-09T22:23:29.810 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:29 smithi104 ceph-mon[32389]: pgmap v78: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-06-09T22:23:29.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:29 smithi153 ceph-mon[36599]: pgmap v78: 129 pgs: 129 active+clean; 29 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-06-09T22:23:30.250 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:23:30.718 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 3.78 MB/sec warmup 36 sec latency 20244.227 ms 2024-06-09T22:23:31.718 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22116 3.68 MB/sec warmup 37 sec latency 21244.253 ms 2024-06-09T22:23:31.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:31 smithi194 ceph-mon[36856]: pgmap v79: 129 pgs: 129 active+clean; 30 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 87 KiB/s wr, 17 op/s 2024-06-09T22:23:31.819 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:31 smithi104 ceph-mon[32389]: pgmap v79: 129 pgs: 129 active+clean; 30 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 87 KiB/s wr, 17 op/s 2024-06-09T22:23:31.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:31 smithi153 ceph-mon[36599]: pgmap v79: 129 pgs: 129 active+clean; 30 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 87 KiB/s wr, 17 op/s 2024-06-09T22:23:32.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.226+0000 7f2337939700 1 -- 172.21.15.104:0/1416634415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2330107570 msgr2=0x7f2330107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.226+0000 7f2337939700 1 --2- 172.21.15.104:0/1416634415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2330107570 0x7f2330107950 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f232c009990 tx=0x7f232c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.226+0000 7f2337939700 1 -- 172.21.15.104:0/1416634415 shutdown_connections 2024-06-09T22:23:32.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.226+0000 7f2337939700 1 --2- 172.21.15.104:0/1416634415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2330107570 0x7f2330107950 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.226+0000 7f2337939700 1 --2- 172.21.15.104:0/1416634415 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2330101810 0x7f233010f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.226+0000 7f2337939700 1 --2- 172.21.15.104:0/1416634415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2330100e50 0x7f23301012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.226+0000 7f2337939700 1 -- 172.21.15.104:0/1416634415 >> 172.21.15.104:0/1416634415 conn(0x7f23300766b0 msgr2=0x7f2330076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.226+0000 7f2337939700 1 -- 172.21.15.104:0/1416634415 shutdown_connections 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.226+0000 7f2337939700 1 -- 172.21.15.104:0/1416634415 wait complete. 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f2337939700 1 Processor -- start 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f2337939700 1 -- start start 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f2337939700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2330100e50 0x7f233019e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f2337939700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2330101810 0x7f233019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f2337939700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2330107570 0x7f2330198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f2337939700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f23301127d0 con 0x7f2330100e50 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f2337939700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2330112650 con 0x7f2330107570 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f2337939700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2330112950 con 0x7f2330101810 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f23356d5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2330100e50 0x7f233019e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f23356d5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2330100e50 0x7f233019e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57204/0 (socket says 172.21.15.104:57204) 2024-06-09T22:23:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f23356d5700 1 -- 172.21.15.104:0/3674443086 learned_addr learned my addr 172.21.15.104:0/3674443086 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f2334ed4700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2330101810 0x7f233019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f2335ed6700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2330107570 0x7f2330198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f23356d5700 1 -- 172.21.15.104:0/3674443086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2330101810 msgr2=0x7f233019ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f23356d5700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2330101810 0x7f233019ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f23356d5700 1 -- 172.21.15.104:0/3674443086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2330107570 msgr2=0x7f2330198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f23356d5700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2330107570 0x7f2330198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f23356d5700 1 -- 172.21.15.104:0/3674443086 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2330199230 con 0x7f2330100e50 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.227+0000 7f23356d5700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2330100e50 0x7f233019e8c0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f232c000c00 tx=0x7f232c00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.228+0000 7f23267fc700 1 -- 172.21.15.104:0/3674443086 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c027070 con 0x7f2330100e50 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.228+0000 7f2337939700 1 -- 172.21.15.104:0/3674443086 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23301994c0 con 0x7f2330100e50 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.228+0000 7f23267fc700 1 -- 172.21.15.104:0/3674443086 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f232c003d70 con 0x7f2330100e50 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.228+0000 7f23267fc700 1 -- 172.21.15.104:0/3674443086 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c00f6c0 con 0x7f2330100e50 2024-06-09T22:23:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.228+0000 7f2337939700 1 -- 172.21.15.104:0/3674443086 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23301a5790 con 0x7f2330100e50 2024-06-09T22:23:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.230+0000 7f23267fc700 1 -- 172.21.15.104:0/3674443086 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f232c02a070 con 0x7f2330100e50 2024-06-09T22:23:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.230+0000 7f23267fc700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f231c0a2d90 0x7f231c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.231+0000 7f2334ed4700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f231c0a2d90 0x7f231c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.231+0000 7f23267fc700 1 -- 172.21.15.104:0/3674443086 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f232c0d0800 con 0x7f2330100e50 2024-06-09T22:23:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.231+0000 7f2334ed4700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f231c0a2d90 0x7f231c0a5250 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2330076bb0 tx=0x7f232000b3c0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f23421bcb80 1 Processor -- start 2024-06-09T22:23:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f23421bcb80 1 -- start start 2024-06-09T22:23:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f23421bcb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f63f0 0x5575858f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f23421bcb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f6d10 0x5575858f06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f23421bcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x5575858f1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f23421bcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5575857defe0 con 0x5575858f0c30 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f23421bcb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55758583d8d0 con 0x5575858f63f0 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f23421bcb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5575857a54e0 con 0x5575858f6d10 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f2335ed6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x5575858f1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f2335ed6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x5575858f1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57212/0 (socket says 172.21.15.104:57212) 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f2335ed6700 1 -- 172.21.15.104:0/1200865056 learned_addr learned my addr 172.21.15.104:0/1200865056 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f23356d5700 1 --2- 172.21.15.104:0/1200865056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f63f0 0x5575858f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f2334ed4700 1 --2- 172.21.15.104:0/1200865056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f6d10 0x5575858f06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f2335ed6700 1 -- 172.21.15.104:0/1200865056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f6d10 msgr2=0x5575858f06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f2335ed6700 1 --2- 172.21.15.104:0/1200865056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f6d10 0x5575858f06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f2335ed6700 1 -- 172.21.15.104:0/1200865056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f63f0 msgr2=0x5575858f67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f2335ed6700 1 --2- 172.21.15.104:0/1200865056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f63f0 0x5575858f67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f2335ed6700 1 -- 172.21.15.104:0/1200865056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55758578c950 con 0x5575858f0c30 2024-06-09T22:23:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f23356d5700 1 --2- 172.21.15.104:0/1200865056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f63f0 0x5575858f67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.232+0000 7f2335ed6700 1 --2- 172.21.15.104:0/1200865056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x5575858f1070 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2328009440 tx=0x7f2328009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884bb8312526a883 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.233+0000 7f231b7fe700 1 -- 172.21.15.104:0/1200865056 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2328013750 con 0x5575858f0c30 2024-06-09T22:23:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.233+0000 7f231b7fe700 1 -- 172.21.15.104:0/1200865056 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2328013dd0 con 0x5575858f0c30 2024-06-09T22:23:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.233+0000 7f231b7fe700 1 -- 172.21.15.104:0/1200865056 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23280143e0 con 0x5575858f0c30 2024-06-09T22:23:32.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.233+0000 7f23421bcb80 1 -- 172.21.15.104:0/1200865056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 msgr2=0x5575858f1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.233+0000 7f23421bcb80 1 --2- 172.21.15.104:0/1200865056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x5575858f1070 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2328009440 tx=0x7f2328009920 comp rx=0 tx=0).stop 2024-06-09T22:23:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.233+0000 7f23421bcb80 1 -- 172.21.15.104:0/1200865056 shutdown_connections 2024-06-09T22:23:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.233+0000 7f23421bcb80 1 --2- 172.21.15.104:0/1200865056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x5575858f1070 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.233+0000 7f23421bcb80 1 --2- 172.21.15.104:0/1200865056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f63f0 0x5575858f67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.233+0000 7f23421bcb80 1 --2- 172.21.15.104:0/1200865056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f6d10 0x5575858f06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.233+0000 7f23421bcb80 1 -- 172.21.15.104:0/1200865056 >> 172.21.15.104:0/1200865056 conn(0x557585819100 msgr2=0x5575858e72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.233+0000 7f23421bcb80 1 -- 172.21.15.104:0/1200865056 shutdown_connections 2024-06-09T22:23:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.233+0000 7f23421bcb80 1 -- 172.21.15.104:0/1200865056 wait complete. 2024-06-09T22:23:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f23421bcb80 1 -- start start 2024-06-09T22:23:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f23421bcb80 1 Processor -- start 2024-06-09T22:23:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f23421bcb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f0c30 0x5575858fca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f23421bcb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 0x5575858f9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f23421bcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f6d10 0x5575858fa290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f23421bcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5575857de040 con 0x5575858f6d10 2024-06-09T22:23:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f23421bcb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557585839890 con 0x5575858f0c30 2024-06-09T22:23:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f23421bcb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5575858fd700 con 0x5575858f63f0 2024-06-09T22:23:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f23421bcb80 10 client.0 _ll_register_callbacks cb 0x5575857d3630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f2334ed4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 0x5575858f9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f2335ed6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f6d10 0x5575858fa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f2334ed4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 0x5575858f9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:57602/0 (socket says 172.21.15.104:57602) 2024-06-09T22:23:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.234+0000 7f2334ed4700 1 -- 172.21.15.104:0/2710981997 learned_addr learned my addr 172.21.15.104:0/2710981997 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f23356d5700 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f0c30 0x5575858fca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f2335ed6700 1 -- 172.21.15.104:0/2710981997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 msgr2=0x5575858f9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f2335ed6700 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 0x5575858f9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f2335ed6700 1 -- 172.21.15.104:0/2710981997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f0c30 msgr2=0x5575858fca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f2335ed6700 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f0c30 0x5575858fca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f2335ed6700 1 -- 172.21.15.104:0/2710981997 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575858037d0 con 0x5575858f6d10 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f2334ed4700 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 0x5575858f9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f2335ed6700 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f6d10 0x5575858fa290 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2328000f80 tx=0x7f2328016dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63177acf28a31ab9 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f2318ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f2318ff9700 1 -- 172.21.15.104:0/2710981997 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2328023070 con 0x5575858f6d10 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f2318ff9700 1 -- 172.21.15.104:0/2710981997 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f2328013550 con 0x5575858f6d10 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f23421bcb80 1 -- 172.21.15.104:0/2710981997 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2328013dd0 con 0x5575858f6d10 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f2318ff9700 1 -- 172.21.15.104:0/2710981997 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2328031050 con 0x5575858f6d10 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f23421bcb80 10 client.16260 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f23421bcb80 1 -- 172.21.15.104:0/2710981997 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557585907e50 con 0x5575858f6d10 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.235+0000 7f2318ff9700 1 -- 172.21.15.104:0/2710981997 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2328013dd0 con 0x5575858f6d10 2024-06-09T22:23:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.236+0000 7f2318ff9700 1 -- 172.21.15.104:0/2710981997 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f2328005650 con 0x5575858f6d10 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.236+0000 7f23421bcb80 10 client.16260 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.236+0000 7f23421bcb80 10 client.16260 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.236+0000 7f23421bcb80 20 client.16260 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.236+0000 7f23421bcb80 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55758590a3d0 0x55758590c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.236+0000 7f23421bcb80 4 client.16260 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.236+0000 7f23421bcb80 1 -- 172.21.15.104:0/2710981997 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55758584add0 con 0x55758590a3d0 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.236+0000 7f23356d5700 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55758590a3d0 0x55758590c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.236+0000 7f2318ff9700 10 client.16260 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.236+0000 7f23356d5700 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55758590a3d0 0x55758590c7c0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6619c1e4b1ac2de server_cookie=7dbd36d1abc117ed in_seq=0 out_seq=0 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.237+0000 7f2318ff9700 1 -- 172.21.15.104:0/2710981997 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55758584add0 con 0x55758590a3d0 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.237+0000 7f2318ff9700 10 client.16260 handle_command_reply: tid=0 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.237+0000 7f23421bcb80 1 client.16260 shutdown 2024-06-09T22:23:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.237+0000 7f23421bcb80 1 -- 172.21.15.104:0/2710981997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f6d10 msgr2=0x5575858fa290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.237+0000 7f23421bcb80 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f6d10 0x5575858fa290 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2328000f80 tx=0x7f2328016dc0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.237+0000 7f23421bcb80 1 -- 172.21.15.104:0/2710981997 shutdown_connections 2024-06-09T22:23:32.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.237+0000 7f23421bcb80 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55758590a3d0 0x55758590c7c0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.237+0000 7f23421bcb80 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f6d10 0x5575858fa290 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.237+0000 7f23421bcb80 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f0c30 0x5575858fca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.237+0000 7f2318ff9700 0 client.16260 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:32.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.237+0000 7f23421bcb80 1 --2- 172.21.15.104:0/2710981997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 0x5575858f9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.237+0000 7f23421bcb80 1 -- 172.21.15.104:0/2710981997 >> 172.21.15.104:0/2710981997 conn(0x557585819100 msgr2=0x5575858e6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:32.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.238+0000 7f23421bcb80 1 -- 172.21.15.104:0/2710981997 shutdown_connections 2024-06-09T22:23:32.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.238+0000 7f23421bcb80 1 -- 172.21.15.104:0/2710981997 wait complete. 2024-06-09T22:23:32.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.238+0000 7f23421bcb80 20 client.16260 trim_cache size 0 max 16384 2024-06-09T22:23:32.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.258+0000 7f23421bcb80 1 Processor -- start 2024-06-09T22:23:32.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.258+0000 7f23421bcb80 1 -- start start 2024-06-09T22:23:32.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.258+0000 7f23421bcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x5575858f8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.258+0000 7f23421bcb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f63f0 0x5575858e6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.258+0000 7f23421bcb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f6d10 0x5575858e7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.258+0000 7f23421bcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f23301ac260 con 0x5575858f0c30 2024-06-09T22:23:32.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.258+0000 7f23421bcb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23301ade00 con 0x5575858f63f0 2024-06-09T22:23:32.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.258+0000 7f23421bcb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23301ac930 con 0x5575858f6d10 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.258+0000 7f23356d5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x5575858f8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.258+0000 7f23356d5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x5575858f8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57240/0 (socket says 172.21.15.104:57240) 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.258+0000 7f23356d5700 1 -- 172.21.15.104:0/3192116255 learned_addr learned my addr 172.21.15.104:0/3192116255 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.259+0000 7f2335ed6700 1 --2- 172.21.15.104:0/3192116255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f6d10 0x5575858e7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.259+0000 7f2334ed4700 1 --2- 172.21.15.104:0/3192116255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f63f0 0x5575858e6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.259+0000 7f23356d5700 1 -- 172.21.15.104:0/3192116255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f6d10 msgr2=0x5575858e7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.259+0000 7f23356d5700 1 --2- 172.21.15.104:0/3192116255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f6d10 0x5575858e7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.259+0000 7f23356d5700 1 -- 172.21.15.104:0/3192116255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f63f0 msgr2=0x5575858e6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.259+0000 7f23356d5700 1 --2- 172.21.15.104:0/3192116255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f63f0 0x5575858e6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.259+0000 7f23356d5700 1 -- 172.21.15.104:0/3192116255 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575858f0a30 con 0x5575858f0c30 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.259+0000 7f2335ed6700 1 --2- 172.21.15.104:0/3192116255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f6d10 0x5575858e7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.259+0000 7f23356d5700 1 --2- 172.21.15.104:0/3192116255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x5575858f8f00 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f232c098ea0 tx=0x7f232c022ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b50a27528717423 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:32.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.259+0000 7f231affd700 1 -- 172.21.15.104:0/3192116255 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c0ba070 con 0x5575858f0c30 2024-06-09T22:23:32.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.259+0000 7f231affd700 1 -- 172.21.15.104:0/3192116255 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f232c0a5620 con 0x5575858f0c30 2024-06-09T22:23:32.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.259+0000 7f231affd700 1 -- 172.21.15.104:0/3192116255 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c09cf00 con 0x5575858f0c30 2024-06-09T22:23:32.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.260+0000 7f23421bcb80 1 -- 172.21.15.104:0/3192116255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 msgr2=0x5575858f8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.260+0000 7f23421bcb80 1 --2- 172.21.15.104:0/3192116255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x5575858f8f00 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f232c098ea0 tx=0x7f232c022ba0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.260+0000 7f23421bcb80 1 -- 172.21.15.104:0/3192116255 shutdown_connections 2024-06-09T22:23:32.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.260+0000 7f23421bcb80 1 --2- 172.21.15.104:0/3192116255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x5575858f8f00 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.260+0000 7f23421bcb80 1 --2- 172.21.15.104:0/3192116255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f6d10 0x5575858e7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.260+0000 7f23421bcb80 1 --2- 172.21.15.104:0/3192116255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f63f0 0x5575858e6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.260+0000 7f23421bcb80 1 -- 172.21.15.104:0/3192116255 >> 172.21.15.104:0/3192116255 conn(0x557585819100 msgr2=0x5575858ebb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:32.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.260+0000 7f23421bcb80 1 -- 172.21.15.104:0/3192116255 shutdown_connections 2024-06-09T22:23:32.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.260+0000 7f23421bcb80 1 -- 172.21.15.104:0/3192116255 wait complete. 2024-06-09T22:23:32.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.260+0000 7f23421bcb80 1 -- start start 2024-06-09T22:23:32.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.260+0000 7f23421bcb80 1 Processor -- start 2024-06-09T22:23:32.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23421bcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x557585902b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23421bcb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 0x5575859030b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23421bcb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f6d10 0x5575858ff960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23421bcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f23301ad6d0 con 0x5575858f0c30 2024-06-09T22:23:32.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23421bcb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23301ad000 con 0x5575858f6d10 2024-06-09T22:23:32.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23421bcb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23301aa1e0 con 0x5575858f63f0 2024-06-09T22:23:32.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23421bcb80 10 client.0 _ll_register_callbacks cb 0x5575857d3630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:32.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23356d5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x557585902b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23356d5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x557585902b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57246/0 (socket says 172.21.15.104:57246) 2024-06-09T22:23:32.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23356d5700 1 -- 172.21.15.104:0/2311684287 learned_addr learned my addr 172.21.15.104:0/2311684287 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:32.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f2335ed6700 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f6d10 0x5575858ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f2334ed4700 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 0x5575859030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23356d5700 1 -- 172.21.15.104:0/2311684287 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 msgr2=0x5575859030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23356d5700 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 0x5575859030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23356d5700 1 -- 172.21.15.104:0/2311684287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f6d10 msgr2=0x5575858ff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23356d5700 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f6d10 0x5575858ff960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23356d5700 1 -- 172.21.15.104:0/2311684287 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575859001a0 con 0x5575858f0c30 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f2334ed4700 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 0x5575859030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23356d5700 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x557585902b70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f232c0b2040 tx=0x7f232c09cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e16822ff720f4dc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f2318ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f2318ff9700 1 -- 172.21.15.104:0/2311684287 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c0ba070 con 0x5575858f0c30 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f2318ff9700 1 -- 172.21.15.104:0/2311684287 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f232c022800 con 0x5575858f0c30 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.261+0000 7f23421bcb80 1 -- 172.21.15.104:0/2311684287 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f232c0a5620 con 0x5575858f0c30 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.262+0000 7f2318ff9700 1 -- 172.21.15.104:0/2311684287 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f232c09cca0 con 0x5575858f0c30 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.262+0000 7f23421bcb80 10 client.16266 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.262+0000 7f23421bcb80 1 -- 172.21.15.104:0/2311684287 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557585975860 con 0x5575858f0c30 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.262+0000 7f2318ff9700 1 -- 172.21.15.104:0/2311684287 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f232c0a5620 con 0x5575858f0c30 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.262+0000 7f2318ff9700 1 -- 172.21.15.104:0/2311684287 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f232c0b7c80 con 0x5575858f0c30 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.262+0000 7f23421bcb80 10 client.16266 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.262+0000 7f23421bcb80 10 client.16266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.262+0000 7f23421bcb80 20 client.16266 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.262+0000 7f23421bcb80 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557585975f70 0x557585978360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.262+0000 7f23421bcb80 4 client.16266 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.262+0000 7f23421bcb80 1 -- 172.21.15.104:0/2311684287 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557585845350 con 0x557585975f70 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.262+0000 7f2334ed4700 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557585975f70 0x557585978360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.263+0000 7f2318ff9700 10 client.16266 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.263+0000 7f2334ed4700 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557585975f70 0x557585978360 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59a854530dd03b8e server_cookie=f7241788e87db83e in_seq=0 out_seq=0 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f2318ff9700 1 -- 172.21.15.104:0/2311684287 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557585845350 con 0x557585975f70 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f2318ff9700 10 client.16266 handle_command_reply: tid=0 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f23421bcb80 1 client.16266 shutdown 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f23421bcb80 1 -- 172.21.15.104:0/2311684287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 msgr2=0x557585902b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f23421bcb80 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x557585902b70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f232c0b2040 tx=0x7f232c09cab0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f23421bcb80 1 -- 172.21.15.104:0/2311684287 shutdown_connections 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f23421bcb80 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557585975f70 0x557585978360 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f23421bcb80 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5575858f0c30 0x557585902b70 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f23421bcb80 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5575858f6d10 0x5575858ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f2318ff9700 0 client.16266 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:32.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f23421bcb80 1 --2- 172.21.15.104:0/2311684287 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575858f63f0 0x5575859030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f23421bcb80 1 -- 172.21.15.104:0/2311684287 >> 172.21.15.104:0/2311684287 conn(0x557585819100 msgr2=0x5575858fb580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:32.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f23421bcb80 1 -- 172.21.15.104:0/2311684287 shutdown_connections 2024-06-09T22:23:32.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f23421bcb80 1 -- 172.21.15.104:0/2311684287 wait complete. 2024-06-09T22:23:32.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.264+0000 7f23421bcb80 20 client.16266 trim_cache size 0 max 16384 2024-06-09T22:23:32.269 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:23:32.269 INFO:teuthology.orchestra.run.smithi104.stdout: "return_code": 0, 2024-06-09T22:23:32.269 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_tag": "a422c545-4000-44e7-be61-bf57605714cd", 2024-06-09T22:23:32.269 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": "asynchronous" 2024-06-09T22:23:32.270 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:23:32.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.265+0000 7f2337939700 1 -- 172.21.15.104:0/3674443086 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f231c0a2d90 msgr2=0x7f231c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.265+0000 7f2337939700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f231c0a2d90 0x7f231c0a5250 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2330076bb0 tx=0x7f232000b3c0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.265+0000 7f2337939700 1 -- 172.21.15.104:0/3674443086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2330100e50 msgr2=0x7f233019e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.265+0000 7f2337939700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2330100e50 0x7f233019e8c0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f232c000c00 tx=0x7f232c00bab0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.265+0000 7f2337939700 1 -- 172.21.15.104:0/3674443086 shutdown_connections 2024-06-09T22:23:32.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.265+0000 7f2337939700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f231c0a2d90 0x7f231c0a5250 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.265+0000 7f2337939700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2330100e50 0x7f233019e8c0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.265+0000 7f2337939700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2330107570 0x7f2330198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.265+0000 7f2337939700 1 --2- 172.21.15.104:0/3674443086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2330101810 0x7f233019ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.265+0000 7f2337939700 1 -- 172.21.15.104:0/3674443086 >> 172.21.15.104:0/3674443086 conn(0x7f23300766b0 msgr2=0x7f23300ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:32.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.265+0000 7f2337939700 1 -- 172.21.15.104:0/3674443086 shutdown_connections 2024-06-09T22:23:32.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.265+0000 7f2337939700 1 -- 172.21.15.104:0/3674443086 wait complete. 2024-06-09T22:23:32.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.450+0000 7f39d4eba700 1 -- 172.21.15.104:0/1897686123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39d006b3c0 msgr2=0x7f39d010f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.450+0000 7f39d4eba700 1 --2- 172.21.15.104:0/1897686123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39d006b3c0 0x7f39d010f310 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f39c800b0a0 tx=0x7f39c800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.450+0000 7f39d4eba700 1 -- 172.21.15.104:0/1897686123 shutdown_connections 2024-06-09T22:23:32.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.450+0000 7f39d4eba700 1 --2- 172.21.15.104:0/1897686123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39d006b3c0 0x7f39d010f310 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.450+0000 7f39d4eba700 1 --2- 172.21.15.104:0/1897686123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39d01084f0 0x7f39d01088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.450+0000 7f39d4eba700 1 --2- 172.21.15.104:0/1897686123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39d006aa00 0x7f39d006ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.450+0000 7f39d4eba700 1 -- 172.21.15.104:0/1897686123 >> 172.21.15.104:0/1897686123 conn(0x7f39d0078300 msgr2=0x7f39d0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:32.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.451+0000 7f39d4eba700 1 -- 172.21.15.104:0/1897686123 shutdown_connections 2024-06-09T22:23:32.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.451+0000 7f39d4eba700 1 -- 172.21.15.104:0/1897686123 wait complete. 2024-06-09T22:23:32.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.451+0000 7f39d4eba700 1 Processor -- start 2024-06-09T22:23:32.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.451+0000 7f39d4eba700 1 -- start start 2024-06-09T22:23:32.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.451+0000 7f39d4eba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39d006aa00 0x7f39d0101450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.451+0000 7f39d4eba700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39d006b3c0 0x7f39d0101990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39d4eba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39d01084f0 0x7f39d0101ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39d4eba700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f39d0112770 con 0x7f39d01084f0 2024-06-09T22:23:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39d4eba700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39d01125f0 con 0x7f39d006aa00 2024-06-09T22:23:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39d4eba700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f39d01128f0 con 0x7f39d006b3c0 2024-06-09T22:23:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39ced9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39d01084f0 0x7f39d0101ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39ced9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39d01084f0 0x7f39d0101ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57256/0 (socket says 172.21.15.104:57256) 2024-06-09T22:23:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39ced9d700 1 -- 172.21.15.104:0/1800461538 learned_addr learned my addr 172.21.15.104:0/1800461538 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39cdd9b700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39d006b3c0 0x7f39d0101990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39ce59c700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39d006aa00 0x7f39d0101450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39ced9d700 1 -- 172.21.15.104:0/1800461538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39d006b3c0 msgr2=0x7f39d0101990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39ced9d700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39d006b3c0 0x7f39d0101990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39ced9d700 1 -- 172.21.15.104:0/1800461538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39d006aa00 msgr2=0x7f39d0101450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39ced9d700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39d006aa00 0x7f39d0101450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39ced9d700 1 -- 172.21.15.104:0/1800461538 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d01a70f0 con 0x7f39d01084f0 2024-06-09T22:23:32.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39ce59c700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39d006aa00 0x7f39d0101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:32.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39ced9d700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39d01084f0 0x7f39d0101ed0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f39c80062a0 tx=0x7f39c8004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:32.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39bf7fe700 1 -- 172.21.15.104:0/1800461538 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c8021070 con 0x7f39d01084f0 2024-06-09T22:23:32.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.452+0000 7f39d4eba700 1 -- 172.21.15.104:0/1800461538 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d01a7380 con 0x7f39d01084f0 2024-06-09T22:23:32.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.453+0000 7f39bf7fe700 1 -- 172.21.15.104:0/1800461538 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39c8007c00 con 0x7f39d01084f0 2024-06-09T22:23:32.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.453+0000 7f39d4eba700 1 -- 172.21.15.104:0/1800461538 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d01a7880 con 0x7f39d01084f0 2024-06-09T22:23:32.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.453+0000 7f39bf7fe700 1 -- 172.21.15.104:0/1800461538 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c801d450 con 0x7f39d01084f0 2024-06-09T22:23:32.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.455+0000 7f39bf7fe700 1 -- 172.21.15.104:0/1800461538 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f39c801d5f0 con 0x7f39d01084f0 2024-06-09T22:23:32.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.456+0000 7f39bf7fe700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f39b80a2d90 0x7f39b80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.456+0000 7f39ce59c700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f39b80a2d90 0x7f39b80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.456+0000 7f39bf7fe700 1 -- 172.21.15.104:0/1800461538 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f39c800f020 con 0x7f39d01084f0 2024-06-09T22:23:32.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.456+0000 7f39df73db80 1 Processor -- start 2024-06-09T22:23:32.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39ce59c700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f39b80a2d90 0x7f39b80a5250 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f39c4005e10 tx=0x7f39c4005d60 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:32.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39df73db80 1 -- start start 2024-06-09T22:23:32.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39df73db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 0x56177c41c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39df73db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c416710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39df73db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c416c50 0x56177c417070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39df73db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56177c309060 con 0x56177c41c500 2024-06-09T22:23:32.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39df73db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56177c367950 con 0x56177c41ce20 2024-06-09T22:23:32.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39df73db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56177c2cf560 con 0x56177c416c50 2024-06-09T22:23:32.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39ce59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 0x56177c41c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39ce59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 0x56177c41c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57264/0 (socket says 172.21.15.104:57264) 2024-06-09T22:23:32.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39ce59c700 1 -- 172.21.15.104:0/151959970 learned_addr learned my addr 172.21.15.104:0/151959970 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:32.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39cdd9b700 1 --2- 172.21.15.104:0/151959970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c416710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39ced9d700 1 --2- 172.21.15.104:0/151959970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c416c50 0x56177c417070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39ce59c700 1 -- 172.21.15.104:0/151959970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c416c50 msgr2=0x56177c417070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39ce59c700 1 --2- 172.21.15.104:0/151959970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c416c50 0x56177c417070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39ce59c700 1 -- 172.21.15.104:0/151959970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 msgr2=0x56177c416710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39ce59c700 1 --2- 172.21.15.104:0/151959970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c416710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.457+0000 7f39ce59c700 1 -- 172.21.15.104:0/151959970 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56177c2b69d0 con 0x56177c41c500 2024-06-09T22:23:32.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.458+0000 7f39ce59c700 1 --2- 172.21.15.104:0/151959970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 0x56177c41c8e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f39c40044e0 tx=0x7f39c4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a9c48252a1a757a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:32.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.458+0000 7f39bcff9700 1 -- 172.21.15.104:0/151959970 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c4003880 con 0x56177c41c500 2024-06-09T22:23:32.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.458+0000 7f39bcff9700 1 -- 172.21.15.104:0/151959970 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39c4024070 con 0x56177c41c500 2024-06-09T22:23:32.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.458+0000 7f39bcff9700 1 -- 172.21.15.104:0/151959970 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c401fce0 con 0x56177c41c500 2024-06-09T22:23:32.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.459+0000 7f39df73db80 1 -- 172.21.15.104:0/151959970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 msgr2=0x56177c41c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.459+0000 7f39df73db80 1 --2- 172.21.15.104:0/151959970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 0x56177c41c8e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f39c40044e0 tx=0x7f39c4012000 comp rx=0 tx=0).stop 2024-06-09T22:23:32.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.459+0000 7f39df73db80 1 -- 172.21.15.104:0/151959970 shutdown_connections 2024-06-09T22:23:32.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.459+0000 7f39df73db80 1 --2- 172.21.15.104:0/151959970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 0x56177c41c8e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.459+0000 7f39df73db80 1 --2- 172.21.15.104:0/151959970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c416c50 0x56177c417070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.459+0000 7f39df73db80 1 --2- 172.21.15.104:0/151959970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c416710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.459+0000 7f39df73db80 1 -- 172.21.15.104:0/151959970 >> 172.21.15.104:0/151959970 conn(0x56177c343180 msgr2=0x56177c411a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:32.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.459+0000 7f39df73db80 1 -- 172.21.15.104:0/151959970 shutdown_connections 2024-06-09T22:23:32.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.459+0000 7f39df73db80 1 -- 172.21.15.104:0/151959970 wait complete. 2024-06-09T22:23:32.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.459+0000 7f39df73db80 1 -- start start 2024-06-09T22:23:32.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.459+0000 7f39df73db80 1 Processor -- start 2024-06-09T22:23:32.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39df73db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c416c50 0x56177c422af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39df73db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 0x56177c41fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39df73db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c420380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39df73db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56177c3080c0 con 0x56177c41c500 2024-06-09T22:23:32.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39df73db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56177c363910 con 0x56177c41ce20 2024-06-09T22:23:32.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39df73db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56177c423800 con 0x56177c416c50 2024-06-09T22:23:32.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39cdd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 0x56177c41fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39df73db80 10 client.0 _ll_register_callbacks cb 0x56177c2fd6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:32.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39cdd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 0x56177c41fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57276/0 (socket says 172.21.15.104:57276) 2024-06-09T22:23:32.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39cdd9b700 1 -- 172.21.15.104:0/1104070906 learned_addr learned my addr 172.21.15.104:0/1104070906 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:32.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39ce59c700 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c416c50 0x56177c422af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39ced9d700 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c420380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39cdd9b700 1 -- 172.21.15.104:0/1104070906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c416c50 msgr2=0x56177c422af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39cdd9b700 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c416c50 0x56177c422af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39cdd9b700 1 -- 172.21.15.104:0/1104070906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 msgr2=0x56177c420380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39cdd9b700 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c420380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39cdd9b700 1 -- 172.21.15.104:0/1104070906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56177c32d850 con 0x56177c41c500 2024-06-09T22:23:32.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39ced9d700 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c420380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39cdd9b700 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 0x56177c41fe40 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f39c0010a50 tx=0x7f39c0010e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d155e3a1766f4f6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39b5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39b5ffb700 1 -- 172.21.15.104:0/1104070906 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c000faa0 con 0x56177c41c500 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.460+0000 7f39df73db80 1 -- 172.21.15.104:0/1104070906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39c4024070 con 0x56177c41c500 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39b5ffb700 1 -- 172.21.15.104:0/1104070906 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f39c000fc40 con 0x56177c41c500 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39b5ffb700 1 -- 172.21.15.104:0/1104070906 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f39c0024050 con 0x56177c41c500 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39df73db80 10 client.16278 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39df73db80 1 -- 172.21.15.104:0/1104070906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56177c42dfa0 con 0x56177c41c500 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39b5ffb700 1 -- 172.21.15.104:0/1104070906 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39c4024070 con 0x56177c41c500 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39b5ffb700 1 -- 172.21.15.104:0/1104070906 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f39c0005460 con 0x56177c41c500 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39df73db80 10 client.16278 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39df73db80 10 client.16278 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:23:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39df73db80 20 client.16278 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:32.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39df73db80 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56177c430630 0x56177c432a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39df73db80 4 client.16278 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:32.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39df73db80 1 -- 172.21.15.104:0/1104070906 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56177c374e50 con 0x56177c430630 2024-06-09T22:23:32.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.461+0000 7f39ce59c700 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56177c430630 0x56177c432a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.462+0000 7f39ce59c700 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56177c430630 0x56177c432a50 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f916c910297d61e9 server_cookie=7f06eac842264200 in_seq=0 out_seq=0 2024-06-09T22:23:32.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.462+0000 7f39b5ffb700 10 client.16278 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:32.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39b5ffb700 1 -- 172.21.15.104:0/1104070906 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56177c374e50 con 0x56177c430630 2024-06-09T22:23:32.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39b5ffb700 10 client.16278 handle_command_reply: tid=0 2024-06-09T22:23:32.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39df73db80 1 client.16278 shutdown 2024-06-09T22:23:32.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39df73db80 1 -- 172.21.15.104:0/1104070906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 msgr2=0x56177c41fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39df73db80 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 0x56177c41fe40 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f39c0010a50 tx=0x7f39c0010e10 comp rx=0 tx=0).stop 2024-06-09T22:23:32.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39df73db80 1 -- 172.21.15.104:0/1104070906 shutdown_connections 2024-06-09T22:23:32.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39df73db80 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56177c430630 0x56177c432a50 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39df73db80 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c420380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39df73db80 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c416c50 0x56177c422af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39df73db80 1 --2- 172.21.15.104:0/1104070906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c41c500 0x56177c41fe40 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39df73db80 1 -- 172.21.15.104:0/1104070906 >> 172.21.15.104:0/1104070906 conn(0x56177c343180 msgr2=0x56177c410bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39b5ffb700 0 client.16278 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39df73db80 1 -- 172.21.15.104:0/1104070906 shutdown_connections 2024-06-09T22:23:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39df73db80 1 -- 172.21.15.104:0/1104070906 wait complete. 2024-06-09T22:23:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.463+0000 7f39df73db80 20 client.16278 trim_cache size 0 max 16384 2024-06-09T22:23:32.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.483+0000 7f39df73db80 1 Processor -- start 2024-06-09T22:23:32.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.483+0000 7f39df73db80 1 -- start start 2024-06-09T22:23:32.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.483+0000 7f39df73db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 0x56177c421d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.483+0000 7f39df73db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c41c500 0x56177c411330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.483+0000 7f39df73db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c411870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.483+0000 7f39df73db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f39d01b0710 con 0x56177c416c50 2024-06-09T22:23:32.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.483+0000 7f39df73db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39d01b22b0 con 0x56177c41ce20 2024-06-09T22:23:32.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.483+0000 7f39df73db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f39d01b0de0 con 0x56177c41c500 2024-06-09T22:23:32.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.483+0000 7f39ce59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 0x56177c421d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.483+0000 7f39ce59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 0x56177c421d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57288/0 (socket says 172.21.15.104:57288) 2024-06-09T22:23:32.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.483+0000 7f39ce59c700 1 -- 172.21.15.104:0/3232525983 learned_addr learned my addr 172.21.15.104:0/3232525983 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:32.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.483+0000 7f39cdd9b700 1 --2- 172.21.15.104:0/3232525983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c41c500 0x56177c411330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.484+0000 7f39ced9d700 1 --2- 172.21.15.104:0/3232525983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c411870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.484+0000 7f39ce59c700 1 -- 172.21.15.104:0/3232525983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c41c500 msgr2=0x56177c411330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.484+0000 7f39ce59c700 1 --2- 172.21.15.104:0/3232525983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c41c500 0x56177c411330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.484+0000 7f39ce59c700 1 -- 172.21.15.104:0/3232525983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 msgr2=0x56177c411870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.484+0000 7f39ce59c700 1 --2- 172.21.15.104:0/3232525983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c411870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.484+0000 7f39ce59c700 1 -- 172.21.15.104:0/3232525983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56177c41de00 con 0x56177c416c50 2024-06-09T22:23:32.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.484+0000 7f39cdd9b700 1 --2- 172.21.15.104:0/3232525983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c41c500 0x56177c411330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:23:32.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.484+0000 7f39ced9d700 1 --2- 172.21.15.104:0/3232525983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c411870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:32.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.484+0000 7f39ce59c700 1 --2- 172.21.15.104:0/3232525983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 0x56177c421d30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f39c40044e0 tx=0x7f39c400b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961827f9bf72e1c5 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:32.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.484+0000 7f39b7fff700 1 -- 172.21.15.104:0/3232525983 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c401f700 con 0x56177c416c50 2024-06-09T22:23:32.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.484+0000 7f39b7fff700 1 -- 172.21.15.104:0/3232525983 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39c4024030 con 0x56177c416c50 2024-06-09T22:23:32.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.484+0000 7f39b7fff700 1 -- 172.21.15.104:0/3232525983 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c4004710 con 0x56177c416c50 2024-06-09T22:23:32.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.485+0000 7f39df73db80 1 -- 172.21.15.104:0/3232525983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 msgr2=0x56177c421d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.485+0000 7f39df73db80 1 --2- 172.21.15.104:0/3232525983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 0x56177c421d30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f39c40044e0 tx=0x7f39c400b2e0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.485+0000 7f39df73db80 1 -- 172.21.15.104:0/3232525983 shutdown_connections 2024-06-09T22:23:32.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.485+0000 7f39df73db80 1 --2- 172.21.15.104:0/3232525983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 0x56177c421d30 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.485+0000 7f39df73db80 1 --2- 172.21.15.104:0/3232525983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41ce20 0x56177c411870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.485+0000 7f39df73db80 1 --2- 172.21.15.104:0/3232525983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c41c500 0x56177c411330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.485+0000 7f39df73db80 1 -- 172.21.15.104:0/3232525983 >> 172.21.15.104:0/3232525983 conn(0x56177c343180 msgr2=0x56177c421620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.485+0000 7f39df73db80 1 -- 172.21.15.104:0/3232525983 shutdown_connections 2024-06-09T22:23:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.485+0000 7f39df73db80 1 -- 172.21.15.104:0/3232525983 wait complete. 2024-06-09T22:23:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.485+0000 7f39df73db80 1 -- start start 2024-06-09T22:23:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.485+0000 7f39df73db80 1 Processor -- start 2024-06-09T22:23:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39df73db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 0x56177c428ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39df73db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41c500 0x56177c429000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39df73db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c41ce20 0x56177c425a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39df73db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f39d01b1b80 con 0x56177c416c50 2024-06-09T22:23:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39df73db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39d01b14b0 con 0x56177c41c500 2024-06-09T22:23:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39df73db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f39d01ae690 con 0x56177c41ce20 2024-06-09T22:23:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39df73db80 10 client.0 _ll_register_callbacks cb 0x56177c2fd6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39ce59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 0x56177c428ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39ce59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 0x56177c428ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57300/0 (socket says 172.21.15.104:57300) 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39ce59c700 1 -- 172.21.15.104:0/3911142477 learned_addr learned my addr 172.21.15.104:0/3911142477 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39cdd9b700 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41c500 0x56177c429000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39ced9d700 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c41ce20 0x56177c425a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39ce59c700 1 -- 172.21.15.104:0/3911142477 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c41ce20 msgr2=0x56177c425a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39ce59c700 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c41ce20 0x56177c425a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39ce59c700 1 -- 172.21.15.104:0/3911142477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41c500 msgr2=0x56177c429000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39ce59c700 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41c500 0x56177c429000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39ce59c700 1 -- 172.21.15.104:0/3911142477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56177c425fd0 con 0x56177c416c50 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39ced9d700 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c41ce20 0x56177c425a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39ce59c700 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 0x56177c428ac0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f39c40044b0 tx=0x7f39c40040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbb6f7816394489 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.486+0000 7f39b5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:32.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39b5ffb700 1 -- 172.21.15.104:0/3911142477 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c401f700 con 0x56177c416c50 2024-06-09T22:23:32.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39b5ffb700 1 -- 172.21.15.104:0/3911142477 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f39c4005330 con 0x56177c416c50 2024-06-09T22:23:32.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39df73db80 1 -- 172.21.15.104:0/3911142477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39c4024030 con 0x56177c416c50 2024-06-09T22:23:32.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39b5ffb700 1 -- 172.21.15.104:0/3911142477 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f39c4048050 con 0x56177c416c50 2024-06-09T22:23:32.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39df73db80 10 client.16284 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:32.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39df73db80 1 -- 172.21.15.104:0/3911142477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56177c49b9b0 con 0x56177c416c50 2024-06-09T22:23:32.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39b5ffb700 1 -- 172.21.15.104:0/3911142477 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39c4024030 con 0x56177c416c50 2024-06-09T22:23:32.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39b5ffb700 1 -- 172.21.15.104:0/3911142477 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f39c4003680 con 0x56177c416c50 2024-06-09T22:23:32.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39df73db80 10 client.16284 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:32.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39df73db80 10 client.16284 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:23:32.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39df73db80 20 client.16284 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:32.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39df73db80 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56177c49be70 0x56177c49e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:32.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39df73db80 4 client.16284 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6f4b8fc4", "release": true, "await": true}] 2024-06-09T22:23:32.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.487+0000 7f39df73db80 1 -- 172.21.15.104:0/3911142477 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6f4b8fc4", "release": true, "await": true}) -- 0x56177c36f3d0 con 0x56177c49be70 2024-06-09T22:23:32.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.488+0000 7f39cdd9b700 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56177c49be70 0x56177c49e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:32.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.488+0000 7f39b5ffb700 10 client.16284 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:32.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.488+0000 7f39cdd9b700 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56177c49be70 0x56177c49e290 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aada6f10a9fc48b6 server_cookie=e9b5e191210428c7 in_seq=0 out_seq=0 2024-06-09T22:23:32.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39b5ffb700 1 -- 172.21.15.104:0/3911142477 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x56177c36f3d0 con 0x56177c49be70 2024-06-09T22:23:32.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39b5ffb700 10 client.16284 handle_command_reply: tid=0 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39df73db80 1 client.16284 shutdown 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39df73db80 1 -- 172.21.15.104:0/3911142477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 msgr2=0x56177c428ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39df73db80 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 0x56177c428ac0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f39c40044b0 tx=0x7f39c40040e0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39df73db80 1 -- 172.21.15.104:0/3911142477 shutdown_connections 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39df73db80 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56177c49be70 0x56177c49e290 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39df73db80 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56177c416c50 0x56177c428ac0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39df73db80 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56177c41ce20 0x56177c425a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39df73db80 1 --2- 172.21.15.104:0/3911142477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56177c41c500 0x56177c429000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39df73db80 1 -- 172.21.15.104:0/3911142477 >> 172.21.15.104:0/3911142477 conn(0x56177c343180 msgr2=0x56177c415f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39b5ffb700 0 client.16284 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39df73db80 1 -- 172.21.15.104:0/3911142477 shutdown_connections 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39df73db80 1 -- 172.21.15.104:0/3911142477 wait complete. 2024-06-09T22:23:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.499+0000 7f39df73db80 20 client.16284 trim_cache size 0 max 16384 2024-06-09T22:23:32.501 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:23:32.501 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 30, 2024-06-09T22:23:32.501 INFO:teuthology.orchestra.run.smithi104.stdout: "leader": 24460, 2024-06-09T22:23:32.501 INFO:teuthology.orchestra.run.smithi104.stdout: "set_version": 8, 2024-06-09T22:23:32.501 INFO:teuthology.orchestra.run.smithi104.stdout: "sets": { 2024-06-09T22:23:32.501 INFO:teuthology.orchestra.run.smithi104.stdout: "6f4b8fc4": { 2024-06-09T22:23:32.501 INFO:teuthology.orchestra.run.smithi104.stdout: "version": 8, 2024-06-09T22:23:32.501 INFO:teuthology.orchestra.run.smithi104.stdout: "age_ref": 0.0, 2024-06-09T22:23:32.502 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:23:32.502 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RELEASED", 2024-06-09T22:23:32.502 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:23:32.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:23:32.502 INFO:teuthology.orchestra.run.smithi104.stdout: "timeout": 90.0, 2024-06-09T22:23:32.502 INFO:teuthology.orchestra.run.smithi104.stdout: "expiration": 150.0, 2024-06-09T22:23:32.502 INFO:teuthology.orchestra.run.smithi104.stdout: "members": { 2024-06-09T22:23:32.502 INFO:teuthology.orchestra.run.smithi104.stdout: "file:/": { 2024-06-09T22:23:32.502 INFO:teuthology.orchestra.run.smithi104.stdout: "excluded": false, 2024-06-09T22:23:32.502 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:23:32.502 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RELEASED", 2024-06-09T22:23:32.502 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:23:32.503 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:23:32.503 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:23:32.503 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:23:32.503 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:23:32.503 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:23:32.503 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:23:32.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.500+0000 7f39d4eba700 1 -- 172.21.15.104:0/1800461538 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f39b80a2d90 msgr2=0x7f39b80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.500+0000 7f39d4eba700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f39b80a2d90 0x7f39b80a5250 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f39c4005e10 tx=0x7f39c4005d60 comp rx=0 tx=0).stop 2024-06-09T22:23:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.500+0000 7f39d4eba700 1 -- 172.21.15.104:0/1800461538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39d01084f0 msgr2=0x7f39d0101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.500+0000 7f39d4eba700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39d01084f0 0x7f39d0101ed0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f39c80062a0 tx=0x7f39c8004d70 comp rx=0 tx=0).stop 2024-06-09T22:23:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.500+0000 7f39d4eba700 1 -- 172.21.15.104:0/1800461538 shutdown_connections 2024-06-09T22:23:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.500+0000 7f39d4eba700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f39b80a2d90 0x7f39b80a5250 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.500+0000 7f39d4eba700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39d01084f0 0x7f39d0101ed0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.500+0000 7f39d4eba700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39d006aa00 0x7f39d0101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.500+0000 7f39d4eba700 1 --2- 172.21.15.104:0/1800461538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39d006b3c0 0x7f39d0101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.500+0000 7f39d4eba700 1 -- 172.21.15.104:0/1800461538 >> 172.21.15.104:0/1800461538 conn(0x7f39d0078300 msgr2=0x7f39d00ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.500+0000 7f39d4eba700 1 -- 172.21.15.104:0/1800461538 shutdown_connections 2024-06-09T22:23:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:32.500+0000 7f39d4eba700 1 -- 172.21.15.104:0/1800461538 wait complete. 2024-06-09T22:23:32.718 INFO:tasks.workunit.client.0.smithi104.stdout: 1 22583 3.63 MB/sec warmup 38 sec latency 22029.125 ms 2024-06-09T22:23:33.298 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:23:33.718 INFO:tasks.workunit.client.0.smithi104.stdout: 1 25243 3.97 MB/sec warmup 39 sec latency 10.186 ms 2024-06-09T22:23:33.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:33 smithi194 ceph-mon[36856]: pgmap v80: 129 pgs: 129 active+clean; 30 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 84 KiB/s wr, 12 op/s 2024-06-09T22:23:33.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:33 smithi194 ceph-mon[36856]: scrub queued for path: / 2024-06-09T22:23:33.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:33 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:33 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:23:33.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:33 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:23:33.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:33 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:23:33.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:33 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:33 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:23:33.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:33 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:33 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:33 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.992 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:33 smithi104 ceph-mon[32389]: pgmap v80: 129 pgs: 129 active+clean; 30 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 84 KiB/s wr, 12 op/s 2024-06-09T22:23:33.992 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:33 smithi104 ceph-mon[32389]: scrub queued for path: / 2024-06-09T22:23:33.992 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:33 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.992 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:33 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:23:33.992 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:33 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:23:33.992 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:33 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:23:33.993 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:33 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.993 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:33 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:23:33.993 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:33 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.993 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:33 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.993 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:33 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:33 smithi153 ceph-mon[36599]: pgmap v80: 129 pgs: 129 active+clean; 30 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 84 KiB/s wr, 12 op/s 2024-06-09T22:23:33.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:33 smithi153 ceph-mon[36599]: scrub queued for path: / 2024-06-09T22:23:33.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:33 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:33 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:23:33.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:33 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:23:33.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:33 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:23:33.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:33 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:33 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:23:33.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:33 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:33 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:33.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:33 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:23:34.020 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '6f4b8fc4', total seconds elapsed: 37.9 2024-06-09T22:23:34.022 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:23:34.718 INFO:tasks.workunit.client.0.smithi104.stdout: 1 25597 3.88 MB/sec warmup 40 sec latency 642.137 ms 2024-06-09T22:23:35.718 INFO:tasks.workunit.client.0.smithi104.stdout: 1 25597 3.78 MB/sec warmup 41 sec latency 1642.175 ms 2024-06-09T22:23:35.731 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:35 smithi104 ceph-mon[32389]: pgmap v81: 129 pgs: 129 active+clean; 32 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 232 KiB/s wr, 46 op/s 2024-06-09T22:23:35.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:35 smithi194 ceph-mon[36856]: pgmap v81: 129 pgs: 129 active+clean; 32 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 232 KiB/s wr, 46 op/s 2024-06-09T22:23:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.835+0000 7fac478fc700 1 -- 172.21.15.104:0/3447846948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac4006ac00 msgr2=0x7fac4006b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.835+0000 7fac478fc700 1 --2- 172.21.15.104:0/3447846948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac4006ac00 0x7fac4006b080 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fac300099c0 tx=0x7fac30009cd0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.835+0000 7fac478fc700 1 -- 172.21.15.104:0/3447846948 shutdown_connections 2024-06-09T22:23:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.835+0000 7fac478fc700 1 --2- 172.21.15.104:0/3447846948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac4006b5c0 0x7fac4010f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.835+0000 7fac478fc700 1 --2- 172.21.15.104:0/3447846948 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac40108560 0x7fac40108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.835+0000 7fac478fc700 1 --2- 172.21.15.104:0/3447846948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac4006ac00 0x7fac4006b080 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.835+0000 7fac478fc700 1 -- 172.21.15.104:0/3447846948 >> 172.21.15.104:0/3447846948 conn(0x7fac400773e0 msgr2=0x7fac400777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.835+0000 7fac478fc700 1 -- 172.21.15.104:0/3447846948 shutdown_connections 2024-06-09T22:23:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.835+0000 7fac478fc700 1 -- 172.21.15.104:0/3447846948 wait complete. 2024-06-09T22:23:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac478fc700 1 Processor -- start 2024-06-09T22:23:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac478fc700 1 -- start start 2024-06-09T22:23:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac478fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac4006ac00 0x7fac401014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac478fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac4006b5c0 0x7fac40101a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac478fc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac40108560 0x7fac40101f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac478fc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fac40112810 con 0x7fac4006ac00 2024-06-09T22:23:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac478fc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac40112690 con 0x7fac4006b5c0 2024-06-09T22:23:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac478fc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fac40112990 con 0x7fac40108560 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac45698700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac4006ac00 0x7fac401014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac45698700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac4006ac00 0x7fac401014f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57308/0 (socket says 172.21.15.104:57308) 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac45698700 1 -- 172.21.15.104:0/602770 learned_addr learned my addr 172.21.15.104:0/602770 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac44e97700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac4006b5c0 0x7fac40101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.836+0000 7fac45e99700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac40108560 0x7fac40101f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.837+0000 7fac45698700 1 -- 172.21.15.104:0/602770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac40108560 msgr2=0x7fac40101f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.837+0000 7fac45698700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac40108560 0x7fac40101f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.837+0000 7fac45698700 1 -- 172.21.15.104:0/602770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac4006b5c0 msgr2=0x7fac40101a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.837+0000 7fac45698700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac4006b5c0 0x7fac40101a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.837+0000 7fac45698700 1 -- 172.21.15.104:0/602770 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac401a7150 con 0x7fac4006ac00 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.837+0000 7fac45698700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac4006ac00 0x7fac401014f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00b530 tx=0x7fac3c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.837+0000 7fac367fc700 1 -- 172.21.15.104:0/602770 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c015020 con 0x7fac4006ac00 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.837+0000 7fac478fc700 1 -- 172.21.15.104:0/602770 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac401a7440 con 0x7fac4006ac00 2024-06-09T22:23:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.837+0000 7fac367fc700 1 -- 172.21.15.104:0/602770 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac3c011d10 con 0x7fac4006ac00 2024-06-09T22:23:35.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.837+0000 7fac367fc700 1 -- 172.21.15.104:0/602770 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c00f2e0 con 0x7fac4006ac00 2024-06-09T22:23:35.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.837+0000 7fac478fc700 1 -- 172.21.15.104:0/602770 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac401a79a0 con 0x7fac4006ac00 2024-06-09T22:23:35.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.839+0000 7fac367fc700 1 -- 172.21.15.104:0/602770 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fac3c018020 con 0x7fac4006ac00 2024-06-09T22:23:35.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.840+0000 7fac367fc700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fac2c0a2e10 0x7fac2c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.840+0000 7fac367fc700 1 -- 172.21.15.104:0/602770 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fac3c0ceed0 con 0x7fac4006ac00 2024-06-09T22:23:35.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.840+0000 7fac44e97700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fac2c0a2e10 0x7fac2c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac44e97700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fac2c0a2e10 0x7fac2c0a52d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fac40102f80 tx=0x7fac30005fb0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:35.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac5217fb80 1 Processor -- start 2024-06-09T22:23:35.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac5217fb80 1 -- start start 2024-06-09T22:23:35.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac5217fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dc640 0x56329d1dca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac5217fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dcf60 0x56329d1d69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac5217fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1d7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac5217fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56329d128810 con 0x56329d1dcf60 2024-06-09T22:23:35.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac5217fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56329d0d0720 con 0x56329d1dc640 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac5217fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56329d1247d0 con 0x56329d1d6f20 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac44e97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dcf60 0x56329d1d69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac44e97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dcf60 0x56329d1d69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57314/0 (socket says 172.21.15.104:57314) 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac44e97700 1 -- 172.21.15.104:0/1392730784 learned_addr learned my addr 172.21.15.104:0/1392730784 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac45698700 1 --2- 172.21.15.104:0/1392730784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dc640 0x56329d1dca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.841+0000 7fac45e99700 1 --2- 172.21.15.104:0/1392730784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1d7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.842+0000 7fac44e97700 1 -- 172.21.15.104:0/1392730784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 msgr2=0x56329d1d7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.842+0000 7fac44e97700 1 --2- 172.21.15.104:0/1392730784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1d7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.842+0000 7fac44e97700 1 -- 172.21.15.104:0/1392730784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dc640 msgr2=0x56329d1dca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.842+0000 7fac44e97700 1 --2- 172.21.15.104:0/1392730784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dc640 0x56329d1dca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.842+0000 7fac44e97700 1 -- 172.21.15.104:0/1392730784 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56329d0c6f50 con 0x56329d1dcf60 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.842+0000 7fac45698700 1 --2- 172.21.15.104:0/1392730784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dc640 0x56329d1dca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.842+0000 7fac45e99700 1 --2- 172.21.15.104:0/1392730784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1d7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.842+0000 7fac44e97700 1 --2- 172.21.15.104:0/1392730784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dcf60 0x56329d1d69e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fac30015000 tx=0x7fac30020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9f06af84666e27 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.842+0000 7fac2b7fe700 1 -- 172.21.15.104:0/1392730784 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3000bdb0 con 0x56329d1dcf60 2024-06-09T22:23:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.842+0000 7fac2b7fe700 1 -- 172.21.15.104:0/1392730784 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac3000fd50 con 0x56329d1dcf60 2024-06-09T22:23:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.842+0000 7fac2b7fe700 1 -- 172.21.15.104:0/1392730784 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac30024d70 con 0x56329d1dcf60 2024-06-09T22:23:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.843+0000 7fac5217fb80 1 -- 172.21.15.104:0/1392730784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dcf60 msgr2=0x56329d1d69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.843+0000 7fac5217fb80 1 --2- 172.21.15.104:0/1392730784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dcf60 0x56329d1d69e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fac30015000 tx=0x7fac30020000 comp rx=0 tx=0).stop 2024-06-09T22:23:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.843+0000 7fac5217fb80 1 -- 172.21.15.104:0/1392730784 shutdown_connections 2024-06-09T22:23:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.843+0000 7fac5217fb80 1 --2- 172.21.15.104:0/1392730784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1d7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.843+0000 7fac5217fb80 1 --2- 172.21.15.104:0/1392730784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dc640 0x56329d1dca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.843+0000 7fac5217fb80 1 --2- 172.21.15.104:0/1392730784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dcf60 0x56329d1d69e0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.843+0000 7fac5217fb80 1 -- 172.21.15.104:0/1392730784 >> 172.21.15.104:0/1392730784 conn(0x56329d101cc0 msgr2=0x56329d1d1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.843+0000 7fac5217fb80 1 -- 172.21.15.104:0/1392730784 shutdown_connections 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.843+0000 7fac5217fb80 1 -- 172.21.15.104:0/1392730784 wait complete. 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.843+0000 7fac5217fb80 1 -- start start 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.843+0000 7fac5217fb80 1 Processor -- start 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac5217fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1d6f20 0x56329d1e2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac5217fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1e0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac5217fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1dcf60 0x56329d1e0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac5217fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56329d1e3910 con 0x56329d1dc640 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac5217fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56329d0c8bf0 con 0x56329d1d6f20 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac5217fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56329d1e3a90 con 0x56329d1dcf60 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac5217fb80 10 client.0 _ll_register_callbacks cb 0x56329d0c2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac44e97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1e0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac44e97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1e0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57316/0 (socket says 172.21.15.104:57316) 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac44e97700 1 -- 172.21.15.104:0/995579183 learned_addr learned my addr 172.21.15.104:0/995579183 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac45698700 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1d6f20 0x56329d1e2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac45e99700 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1dcf60 0x56329d1e0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac44e97700 1 -- 172.21.15.104:0/995579183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1dcf60 msgr2=0x56329d1e0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac44e97700 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1dcf60 0x56329d1e0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac44e97700 1 -- 172.21.15.104:0/995579183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1d6f20 msgr2=0x56329d1e2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac44e97700 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1d6f20 0x56329d1e2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac44e97700 1 -- 172.21.15.104:0/995579183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56329d0778d0 con 0x56329d1dc640 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac45698700 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1d6f20 0x56329d1e2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac45e99700 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1dcf60 0x56329d1e0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac44e97700 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1e0010 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fac300248f0 tx=0x7fac3000b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef2ed10e188ed5f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac28ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.844+0000 7fac28ff9700 1 -- 172.21.15.104:0/995579183 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac30024410 con 0x56329d1dc640 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac28ff9700 1 -- 172.21.15.104:0/995579183 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fac30024c00 con 0x56329d1dc640 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac5217fb80 1 -- 172.21.15.104:0/995579183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac3000fd50 con 0x56329d1dc640 2024-06-09T22:23:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac28ff9700 1 -- 172.21.15.104:0/995579183 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fac30024e00 con 0x56329d1dc640 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac5217fb80 10 client.16296 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac5217fb80 1 -- 172.21.15.104:0/995579183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56329d1ee210 con 0x56329d1dc640 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac28ff9700 1 -- 172.21.15.104:0/995579183 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac3000fd50 con 0x56329d1dc640 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac28ff9700 1 -- 172.21.15.104:0/995579183 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fac3002f520 con 0x56329d1dc640 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac5217fb80 10 client.16296 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac5217fb80 10 client.16296 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac5217fb80 20 client.16296 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac5217fb80 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56329d1f06e0 0x56329d1f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac5217fb80 4 client.16296 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac5217fb80 1 -- 172.21.15.104:0/995579183 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56329d135d10 con 0x56329d1f06e0 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.845+0000 7fac45698700 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56329d1f06e0 0x56329d1f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.846+0000 7fac45698700 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56329d1f06e0 0x56329d1f2ad0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaec50825585d055 server_cookie=71ea9d627c5ca311 in_seq=0 out_seq=0 2024-06-09T22:23:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.846+0000 7fac28ff9700 10 client.16296 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac28ff9700 1 -- 172.21.15.104:0/995579183 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56329d135d10 con 0x56329d1f06e0 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac28ff9700 10 client.16296 handle_command_reply: tid=0 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac5217fb80 1 client.16296 shutdown 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac5217fb80 1 -- 172.21.15.104:0/995579183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 msgr2=0x56329d1e0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac5217fb80 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1e0010 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fac300248f0 tx=0x7fac3000b540 comp rx=0 tx=0).stop 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac5217fb80 1 -- 172.21.15.104:0/995579183 shutdown_connections 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac5217fb80 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56329d1f06e0 0x56329d1f2ad0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac5217fb80 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1dcf60 0x56329d1e0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac5217fb80 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1d6f20 0x56329d1e2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac28ff9700 0 client.16296 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac5217fb80 1 --2- 172.21.15.104:0/995579183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1e0010 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac5217fb80 1 -- 172.21.15.104:0/995579183 >> 172.21.15.104:0/995579183 conn(0x56329d101cc0 msgr2=0x56329d1d59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac5217fb80 1 -- 172.21.15.104:0/995579183 shutdown_connections 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac5217fb80 1 -- 172.21.15.104:0/995579183 wait complete. 2024-06-09T22:23:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.847+0000 7fac5217fb80 20 client.16296 trim_cache size 0 max 16384 2024-06-09T22:23:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.867+0000 7fac5217fb80 1 Processor -- start 2024-06-09T22:23:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.867+0000 7fac5217fb80 1 -- start start 2024-06-09T22:23:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.867+0000 7fac5217fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1d69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.867+0000 7fac5217fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1cb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.867+0000 7fac5217fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dcf60 0x56329d1df050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.867+0000 7fac5217fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fac401b0750 con 0x56329d1dc640 2024-06-09T22:23:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.867+0000 7fac5217fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac401b22f0 con 0x56329d1dcf60 2024-06-09T22:23:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.867+0000 7fac5217fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fac401b0e20 con 0x56329d1d6f20 2024-06-09T22:23:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.867+0000 7fac44e97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.867+0000 7fac44e97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57330/0 (socket says 172.21.15.104:57330) 2024-06-09T22:23:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.867+0000 7fac44e97700 1 -- 172.21.15.104:0/1782781459 learned_addr learned my addr 172.21.15.104:0/1782781459 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac45698700 1 --2- 172.21.15.104:0/1782781459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1d69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac45e99700 1 --2- 172.21.15.104:0/1782781459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dcf60 0x56329d1df050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac44e97700 1 -- 172.21.15.104:0/1782781459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 msgr2=0x56329d1d69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac44e97700 1 --2- 172.21.15.104:0/1782781459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1d69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac44e97700 1 -- 172.21.15.104:0/1782781459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dcf60 msgr2=0x56329d1df050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac44e97700 1 --2- 172.21.15.104:0/1782781459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dcf60 0x56329d1df050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac44e97700 1 -- 172.21.15.104:0/1782781459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56329d1e3990 con 0x56329d1dc640 2024-06-09T22:23:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac45698700 1 --2- 172.21.15.104:0/1782781459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1d69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:23:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac45e99700 1 --2- 172.21.15.104:0/1782781459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dcf60 0x56329d1df050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:23:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac44e97700 1 --2- 172.21.15.104:0/1782781459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1cb690 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fac3000b9f0 tx=0x7fac30024d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5e318d66961a27 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac2affd700 1 -- 172.21.15.104:0/1782781459 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3000fd00 con 0x56329d1dc640 2024-06-09T22:23:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac2affd700 1 -- 172.21.15.104:0/1782781459 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac3003c070 con 0x56329d1dc640 2024-06-09T22:23:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.868+0000 7fac2affd700 1 -- 172.21.15.104:0/1782781459 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac30037660 con 0x56329d1dc640 2024-06-09T22:23:35.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 -- 172.21.15.104:0/1782781459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 msgr2=0x56329d1cb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:35.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 --2- 172.21.15.104:0/1782781459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1cb690 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fac3000b9f0 tx=0x7fac30024d90 comp rx=0 tx=0).stop 2024-06-09T22:23:35.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 -- 172.21.15.104:0/1782781459 shutdown_connections 2024-06-09T22:23:35.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 --2- 172.21.15.104:0/1782781459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dcf60 0x56329d1df050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 --2- 172.21.15.104:0/1782781459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1d69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 --2- 172.21.15.104:0/1782781459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1cb690 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 -- 172.21.15.104:0/1782781459 >> 172.21.15.104:0/1782781459 conn(0x56329d101cc0 msgr2=0x56329d1de940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 -- 172.21.15.104:0/1782781459 shutdown_connections 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 -- 172.21.15.104:0/1782781459 wait complete. 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 -- start start 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 Processor -- start 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1f6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.869+0000 7fac5217fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1f3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac5217fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dcf60 0x56329d1f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac5217fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fac401b1bc0 con 0x56329d1dc640 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac5217fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac401b14f0 con 0x56329d1dcf60 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac5217fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fac401ae6d0 con 0x56329d1d6f20 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac5217fb80 10 client.0 _ll_register_callbacks cb 0x56329d0c2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac44e97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac44e97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1f3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57336/0 (socket says 172.21.15.104:57336) 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac44e97700 1 -- 172.21.15.104:0/852436316 learned_addr learned my addr 172.21.15.104:0/852436316 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac45698700 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac45e99700 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dcf60 0x56329d1f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac44e97700 1 -- 172.21.15.104:0/852436316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 msgr2=0x56329d1f6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac44e97700 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1f6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac44e97700 1 -- 172.21.15.104:0/852436316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dcf60 msgr2=0x56329d1f3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac44e97700 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dcf60 0x56329d1f3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac44e97700 1 -- 172.21.15.104:0/852436316 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56329d1f4150 con 0x56329d1dc640 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac45e99700 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dcf60 0x56329d1f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac44e97700 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1f3500 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fac3002f9c0 tx=0x7fac300243e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1753c2ad9ad5683b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac28ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac28ff9700 1 -- 172.21.15.104:0/852436316 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3004e060 con 0x56329d1dc640 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac28ff9700 1 -- 172.21.15.104:0/852436316 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fac30049430 con 0x56329d1dc640 2024-06-09T22:23:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac5217fb80 1 -- 172.21.15.104:0/852436316 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac3003c070 con 0x56329d1dc640 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac28ff9700 1 -- 172.21.15.104:0/852436316 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fac30049630 con 0x56329d1dc640 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac5217fb80 10 client.16302 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.870+0000 7fac5217fb80 1 -- 172.21.15.104:0/852436316 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56329d25aee0 con 0x56329d1dc640 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.871+0000 7fac28ff9700 1 -- 172.21.15.104:0/852436316 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac3003c070 con 0x56329d1dc640 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.871+0000 7fac28ff9700 1 -- 172.21.15.104:0/852436316 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fac30037960 con 0x56329d1dc640 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.871+0000 7fac5217fb80 10 client.16302 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.871+0000 7fac5217fb80 10 client.16302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.871+0000 7fac5217fb80 20 client.16302 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.871+0000 7fac5217fb80 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56329d25b320 0x56329d25d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.871+0000 7fac5217fb80 4 client.16302 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.871+0000 7fac5217fb80 1 -- 172.21.15.104:0/852436316 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56329d130290 con 0x56329d25b320 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.871+0000 7fac45698700 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56329d25b320 0x56329d25d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.871+0000 7fac45698700 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56329d25b320 0x56329d25d740 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bd30079c43d51ee server_cookie=6ae5b3e771772d50 in_seq=0 out_seq=0 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.871+0000 7fac28ff9700 10 client.16302 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac28ff9700 1 -- 172.21.15.104:0/852436316 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56329d130290 con 0x56329d25b320 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac28ff9700 10 client.16302 handle_command_reply: tid=0 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac5217fb80 1 client.16302 shutdown 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac5217fb80 1 -- 172.21.15.104:0/852436316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 msgr2=0x56329d1f3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac5217fb80 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1f3500 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fac3002f9c0 tx=0x7fac300243e0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac5217fb80 1 -- 172.21.15.104:0/852436316 shutdown_connections 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac5217fb80 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56329d25b320 0x56329d25d740 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac5217fb80 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56329d1dcf60 0x56329d1f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac5217fb80 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56329d1d6f20 0x56329d1f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac5217fb80 1 --2- 172.21.15.104:0/852436316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56329d1dc640 0x56329d1f3500 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac5217fb80 1 -- 172.21.15.104:0/852436316 >> 172.21.15.104:0/852436316 conn(0x56329d101cc0 msgr2=0x56329d1db9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac28ff9700 0 client.16302 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:35.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac5217fb80 1 -- 172.21.15.104:0/852436316 shutdown_connections 2024-06-09T22:23:35.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac5217fb80 1 -- 172.21.15.104:0/852436316 wait complete. 2024-06-09T22:23:35.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.872+0000 7fac5217fb80 20 client.16302 trim_cache size 0 max 16384 2024-06-09T22:23:35.879 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:23:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-09T22:23:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": { 2024-06-09T22:23:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "a422c545-4000-44e7-be61-bf57605714cd": { 2024-06-09T22:23:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "path": "/", 2024-06-09T22:23:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "tag": "a422c545-4000-44e7-be61-bf57605714cd", 2024-06-09T22:23:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "options": "recursive,force" 2024-06-09T22:23:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:23:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:23:35.880 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:23:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.873+0000 7fac478fc700 1 -- 172.21.15.104:0/602770 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fac2c0a2e10 msgr2=0x7fac2c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.873+0000 7fac478fc700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fac2c0a2e10 0x7fac2c0a52d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fac40102f80 tx=0x7fac30005fb0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.873+0000 7fac478fc700 1 -- 172.21.15.104:0/602770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac4006ac00 msgr2=0x7fac401014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.873+0000 7fac478fc700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac4006ac00 0x7fac401014f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00b530 tx=0x7fac3c00b840 comp rx=0 tx=0).stop 2024-06-09T22:23:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.873+0000 7fac478fc700 1 -- 172.21.15.104:0/602770 shutdown_connections 2024-06-09T22:23:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.873+0000 7fac478fc700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fac2c0a2e10 0x7fac2c0a52d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.873+0000 7fac478fc700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac4006ac00 0x7fac401014f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.873+0000 7fac478fc700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac40108560 0x7fac40101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.873+0000 7fac478fc700 1 --2- 172.21.15.104:0/602770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac4006b5c0 0x7fac40101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.873+0000 7fac478fc700 1 -- 172.21.15.104:0/602770 >> 172.21.15.104:0/602770 conn(0x7fac400773e0 msgr2=0x7fac400fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.873+0000 7fac478fc700 1 -- 172.21.15.104:0/602770 shutdown_connections 2024-06-09T22:23:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:23:35.873+0000 7fac478fc700 1 -- 172.21.15.104:0/602770 wait complete. 2024-06-09T22:23:35.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:35 smithi153 ceph-mon[36599]: pgmap v81: 129 pgs: 129 active+clean; 32 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 232 KiB/s wr, 46 op/s 2024-06-09T22:23:36.529 INFO:tasks.cephfs.filesystem:scrub status for tag:a422c545-4000-44e7-be61-bf57605714cd - {'path': '/', 'tag': 'a422c545-4000-44e7-be61-bf57605714cd', 'options': 'recursive,force'} 2024-06-09T22:23:36.718 INFO:tasks.workunit.client.0.smithi104.stdout: 1 25665 3.70 MB/sec warmup 42 sec latency 2033.764 ms 2024-06-09T22:23:37.718 INFO:tasks.workunit.client.0.smithi104.stdout: 1 25665 3.61 MB/sec warmup 43 sec latency 1556.700 ms 2024-06-09T22:23:37.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:37 smithi153 ceph-mon[36599]: pgmap v82: 129 pgs: 129 active+clean; 32 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 231 KiB/s wr, 41 op/s 2024-06-09T22:23:37.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:37 smithi194 ceph-mon[36856]: pgmap v82: 129 pgs: 129 active+clean; 32 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 231 KiB/s wr, 41 op/s 2024-06-09T22:23:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.923+0000 7f06bd6fc640 1 -- 172.21.15.104:0/984216270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06b810bf70 msgr2=0x7f06b810e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.923+0000 7f06bd6fc640 1 --2- 172.21.15.104:0/984216270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06b810bf70 0x7f06b810e410 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f06b0009900 tx=0x7f06b002a510 comp rx=0 tx=0).stop 2024-06-09T22:23:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.923+0000 7f06bd6fc640 1 -- 172.21.15.104:0/984216270 shutdown_connections 2024-06-09T22:23:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.923+0000 7f06bd6fc640 1 --2- 172.21.15.104:0/984216270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06b810bf70 0x7f06b810e410 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.923+0000 7f06bd6fc640 1 --2- 172.21.15.104:0/984216270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b8103550 0x7f06b810ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.923+0000 7f06bd6fc640 1 --2- 172.21.15.104:0/984216270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b8102c30 0x7f06b8103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.923+0000 7f06bd6fc640 1 -- 172.21.15.104:0/984216270 >> 172.21.15.104:0/984216270 conn(0x7f06b8077a20 msgr2=0x7f06b8077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.923+0000 7f06bd6fc640 1 -- 172.21.15.104:0/984216270 shutdown_connections 2024-06-09T22:23:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.923+0000 7f06bd6fc640 1 -- 172.21.15.104:0/984216270 wait complete. 2024-06-09T22:23:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06bd6fc640 1 Processor -- start 2024-06-09T22:23:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06bd6fc640 1 -- start start 2024-06-09T22:23:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06bd6fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b8102c30 0x7f06b81981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06bd6fc640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b8103550 0x7f06b8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06bd6fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06b810bf70 0x7f06b819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06bd6fc640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f06b8111dc0 con 0x7f06b810bf70 2024-06-09T22:23:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06bd6fc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f06b8111c40 con 0x7f06b8102c30 2024-06-09T22:23:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06bd6fc640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f06b8111f40 con 0x7f06b8103550 2024-06-09T22:23:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06b77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06b810bf70 0x7f06b819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06b6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b8102c30 0x7f06b81981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06b77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06b810bf70 0x7f06b819ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58198/0 (socket says 172.21.15.104:58198) 2024-06-09T22:23:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06b77fe640 1 -- 172.21.15.104:0/2104694249 learned_addr learned my addr 172.21.15.104:0/2104694249 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:37.928 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.924+0000 7f06b67fc640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b8103550 0x7f06b8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.928 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.925+0000 7f06b77fe640 1 -- 172.21.15.104:0/2104694249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b8103550 msgr2=0x7f06b8198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:37.928 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.925+0000 7f06b77fe640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b8103550 0x7f06b8198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.928 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.925+0000 7f06b77fe640 1 -- 172.21.15.104:0/2104694249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b8102c30 msgr2=0x7f06b81981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:37.928 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.925+0000 7f06b77fe640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b8102c30 0x7f06b81981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.928 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.925+0000 7f06b77fe640 1 -- 172.21.15.104:0/2104694249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b819d160 con 0x7f06b810bf70 2024-06-09T22:23:37.928 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.925+0000 7f06b67fc640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b8103550 0x7f06b8198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:23:37.928 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.925+0000 7f06b77fe640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06b810bf70 0x7f06b819ca30 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f06b0004770 tx=0x7f06b0003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:37.928 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.925+0000 7f06affff640 1 -- 172.21.15.104:0/2104694249 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b003f070 con 0x7f06b810bf70 2024-06-09T22:23:37.928 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.925+0000 7f06affff640 1 -- 172.21.15.104:0/2104694249 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06b0031c90 con 0x7f06b810bf70 2024-06-09T22:23:37.928 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.925+0000 7f06bd6fc640 1 -- 172.21.15.104:0/2104694249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b819d3f0 con 0x7f06b810bf70 2024-06-09T22:23:37.929 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.925+0000 7f06affff640 1 -- 172.21.15.104:0/2104694249 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b0035030 con 0x7f06b810bf70 2024-06-09T22:23:37.929 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.925+0000 7f06bd6fc640 1 -- 172.21.15.104:0/2104694249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b81a4ce0 con 0x7f06b810bf70 2024-06-09T22:23:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.927+0000 7f06affff640 1 -- 172.21.15.104:0/2104694249 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f06b0042070 con 0x7f06b810bf70 2024-06-09T22:23:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06c0406740 1 Processor -- start 2024-06-09T22:23:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06c0406740 1 -- start start 2024-06-09T22:23:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06c0406740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e7639ef550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06c0406740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06c0406740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e7639fc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06affff640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f06a40a2da0 0x7f06a40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06c0406740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e7639534f0 con 0x55e7639ef170 2024-06-09T22:23:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06c0406740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e7638f1870 con 0x55e7639efa90 2024-06-09T22:23:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06c0406740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e76394f4b0 con 0x55e7639fa350 2024-06-09T22:23:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06b67fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e7639ef550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06b77fe640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f06a40a2da0 0x7f06a40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06b67fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e7639ef550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58204/0 (socket says 172.21.15.104:58204) 2024-06-09T22:23:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06b67fc640 1 -- 172.21.15.104:0/2358552016 learned_addr learned my addr 172.21.15.104:0/2358552016 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06b6ffd640 1 --2- 172.21.15.104:0/2358552016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.928+0000 7f06b67fc640 1 --2- 172.21.15.104:0/2358552016 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e7639fc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.929+0000 7f06affff640 1 -- 172.21.15.104:0/2104694249 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f06b00eff20 con 0x7f06b810bf70 2024-06-09T22:23:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.929+0000 7f06b6ffd640 1 -- 172.21.15.104:0/2358552016 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 msgr2=0x55e7639fc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.929+0000 7f06b6ffd640 1 --2- 172.21.15.104:0/2358552016 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e7639fc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.929+0000 7f06b6ffd640 1 -- 172.21.15.104:0/2358552016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 msgr2=0x55e7639ef550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.929+0000 7f06b6ffd640 1 --2- 172.21.15.104:0/2358552016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e7639ef550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.929+0000 7f06b6ffd640 1 -- 172.21.15.104:0/2358552016 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e76390d9b0 con 0x55e7639efa90 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.929+0000 7f06b77fe640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f06a40a2da0 0x7f06a40a5260 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f06b0006010 tx=0x7f06b008e150 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.929+0000 7f06b6ffd640 1 --2- 172.21.15.104:0/2358552016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f9e10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009fd0 tx=0x7f06a8003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cac6d9505eb26342 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.929+0000 7f06ad7fa640 1 -- 172.21.15.104:0/2358552016 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8011440 con 0x55e7639efa90 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.929+0000 7f06ad7fa640 1 -- 172.21.15.104:0/2358552016 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06a8015030 con 0x55e7639efa90 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.929+0000 7f06ad7fa640 1 -- 172.21.15.104:0/2358552016 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8010950 con 0x55e7639efa90 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.930+0000 7f06c0406740 1 -- 172.21.15.104:0/2358552016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 msgr2=0x55e7639f9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.930+0000 7f06c0406740 1 --2- 172.21.15.104:0/2358552016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f9e10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009fd0 tx=0x7f06a8003e00 comp rx=0 tx=0).stop 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.930+0000 7f06c0406740 1 -- 172.21.15.104:0/2358552016 shutdown_connections 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.930+0000 7f06c0406740 1 --2- 172.21.15.104:0/2358552016 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e7639fc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.930+0000 7f06c0406740 1 --2- 172.21.15.104:0/2358552016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f9e10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.930+0000 7f06c0406740 1 --2- 172.21.15.104:0/2358552016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e7639ef550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.930+0000 7f06c0406740 1 -- 172.21.15.104:0/2358552016 >> 172.21.15.104:0/2358552016 conn(0x55e7639eb680 msgr2=0x55e7639eba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.930+0000 7f06c0406740 1 -- 172.21.15.104:0/2358552016 shutdown_connections 2024-06-09T22:23:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.930+0000 7f06c0406740 1 -- 172.21.15.104:0/2358552016 wait complete. 2024-06-09T22:23:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.930+0000 7f06c0406740 1 -- start start 2024-06-09T22:23:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06c0406740 1 Processor -- start 2024-06-09T22:23:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06c0406740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e7639f71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06c0406740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06c0406740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e7639f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06c0406740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e7639fcf80 con 0x55e7639ef170 2024-06-09T22:23:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06c0406740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e7638dcae0 con 0x55e7639efa90 2024-06-09T22:23:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06c0406740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e7639fd100 con 0x55e7639fa350 2024-06-09T22:23:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06c0406740 10 client.0 _ll_register_callbacks cb 0x55e7639159b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06b67fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e7639f71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.938 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06b67fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e7639f71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58220/0 (socket says 172.21.15.104:58220) 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06b67fc640 1 -- 172.21.15.104:0/3498992467 learned_addr learned my addr 172.21.15.104:0/3498992467 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06b6ffd640 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06b67fc640 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e7639f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06b67fc640 1 -- 172.21.15.104:0/3498992467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 msgr2=0x55e7639f7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06b67fc640 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e7639f7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06b67fc640 1 -- 172.21.15.104:0/3498992467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 msgr2=0x55e7639f7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06b67fc640 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06b67fc640 1 -- 172.21.15.104:0/3498992467 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e763667210 con 0x55e7639ef170 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f06b67fc640 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e7639f71f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f06a0013e30 tx=0x7f06a0015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa99b0af15dab46c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.931+0000 7f069e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f069e7fc640 1 -- 172.21.15.104:0/3498992467 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a00115f0 con 0x55e7639ef170 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f069e7fc640 1 -- 172.21.15.104:0/3498992467 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f06a0011790 con 0x55e7639ef170 2024-06-09T22:23:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f06c0406740 1 -- 172.21.15.104:0/3498992467 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06a8015030 con 0x55e7639ef170 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f069e7fc640 1 -- 172.21.15.104:0/3498992467 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f06a002a050 con 0x55e7639ef170 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f06c0406740 10 client.16314 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f06c0406740 1 -- 172.21.15.104:0/3498992467 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e763733620 con 0x55e7639ef170 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f069e7fc640 1 -- 172.21.15.104:0/3498992467 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06a8015030 con 0x55e7639ef170 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f069e7fc640 1 -- 172.21.15.104:0/3498992467 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f06a001f020 con 0x55e7639ef170 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f06c0406740 10 client.16314 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f06c0406740 10 client.16314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f06c0406740 20 client.16314 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f06c0406740 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e763a0a520 0x55e763a0c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f06c0406740 4 client.16314 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f06c0406740 1 -- 172.21.15.104:0/3498992467 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e763966f10 con 0x55e763a0a520 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.932+0000 7f06b6ffd640 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e763a0a520 0x55e763a0c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.933+0000 7f069e7fc640 10 client.16314 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.933+0000 7f06b6ffd640 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e763a0a520 0x55e763a0c910 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ed3f53bdfc505ce server_cookie=1ea2650a74a5f59c in_seq=0 out_seq=0 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f069e7fc640 1 -- 172.21.15.104:0/3498992467 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55e763966f10 con 0x55e763a0a520 2024-06-09T22:23:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f069e7fc640 10 client.16314 handle_command_reply: tid=0 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f06c0406740 1 client.16314 shutdown 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f06c0406740 1 -- 172.21.15.104:0/3498992467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 msgr2=0x55e7639f71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f06c0406740 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e7639f71f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f06a0013e30 tx=0x7f06a0015040 comp rx=0 tx=0).stop 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f06c0406740 1 -- 172.21.15.104:0/3498992467 shutdown_connections 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f06c0406740 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e763a0a520 0x55e763a0c910 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f06c0406740 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e7639f7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f06c0406740 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f069e7fc640 0 client.16314 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f06c0406740 1 --2- 172.21.15.104:0/3498992467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e7639f71f0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f06c0406740 1 -- 172.21.15.104:0/3498992467 >> 172.21.15.104:0/3498992467 conn(0x55e7639eb680 msgr2=0x55e7639f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f06c0406740 1 -- 172.21.15.104:0/3498992467 shutdown_connections 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f06c0406740 1 -- 172.21.15.104:0/3498992467 wait complete. 2024-06-09T22:23:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.934+0000 7f06c0406740 20 client.16314 trim_cache size 0 max 16384 2024-06-09T22:23:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.949+0000 7f06c0406740 1 Processor -- start 2024-06-09T22:23:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.949+0000 7f06c0406740 1 -- start start 2024-06-09T22:23:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.949+0000 7f06c0406740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e763a06850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.949+0000 7f06c0406740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.949+0000 7f06c0406740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e763a02760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.949+0000 7f06c0406740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f06b81aecf0 con 0x55e7639ef170 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.949+0000 7f06c0406740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f06b81b0890 con 0x55e7639efa90 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.949+0000 7f06c0406740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f06b81af3c0 con 0x55e7639fa350 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.949+0000 7f06b67fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e763a06850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.949+0000 7f06b67fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e763a06850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58226/0 (socket says 172.21.15.104:58226) 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.949+0000 7f06b67fc640 1 -- 172.21.15.104:0/110486609 learned_addr learned my addr 172.21.15.104:0/110486609 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.950+0000 7f06b6ffd640 1 --2- 172.21.15.104:0/110486609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.950+0000 7f06b67fc640 1 -- 172.21.15.104:0/110486609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 msgr2=0x55e763a02760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.950+0000 7f06b67fc640 1 --2- 172.21.15.104:0/110486609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e763a02760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.950+0000 7f06b67fc640 1 -- 172.21.15.104:0/110486609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 msgr2=0x55e7639f61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.950+0000 7f06b67fc640 1 --2- 172.21.15.104:0/110486609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.950+0000 7f06b67fc640 1 -- 172.21.15.104:0/110486609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e763a0cc80 con 0x55e7639ef170 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.950+0000 7f06b67fc640 1 --2- 172.21.15.104:0/110486609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e763a06850 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f06a0005b90 tx=0x7f06a0013c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa939991f6442f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.950+0000 7f06acff9640 1 -- 172.21.15.104:0/110486609 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a0032070 con 0x55e7639ef170 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.950+0000 7f06acff9640 1 -- 172.21.15.104:0/110486609 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06a00087e0 con 0x55e7639ef170 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.950+0000 7f06acff9640 1 -- 172.21.15.104:0/110486609 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a0005040 con 0x55e7639ef170 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 -- 172.21.15.104:0/110486609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 msgr2=0x55e763a06850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 --2- 172.21.15.104:0/110486609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e763a06850 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f06a0005b90 tx=0x7f06a0013c20 comp rx=0 tx=0).stop 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 -- 172.21.15.104:0/110486609 shutdown_connections 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 --2- 172.21.15.104:0/110486609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e763a02760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 --2- 172.21.15.104:0/110486609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e7639f61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 --2- 172.21.15.104:0/110486609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e763a06850 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 -- 172.21.15.104:0/110486609 >> 172.21.15.104:0/110486609 conn(0x55e7639eb680 msgr2=0x55e7639f5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 -- 172.21.15.104:0/110486609 shutdown_connections 2024-06-09T22:23:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 -- 172.21.15.104:0/110486609 wait complete. 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 -- start start 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 Processor -- start 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e763a117b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e763a11cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e763a77200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f06b81b0160 con 0x55e7639ef170 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f06b81afa90 con 0x55e7639efa90 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f06b81accb0 con 0x55e7639fa350 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06c0406740 10 client.0 _ll_register_callbacks cb 0x7f06a0013cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06b67fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e763a117b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06b67fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e763a117b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58236/0 (socket says 172.21.15.104:58236) 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.951+0000 7f06b67fc640 1 -- 172.21.15.104:0/554157569 learned_addr learned my addr 172.21.15.104:0/554157569 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f06b6ffd640 1 --2- 172.21.15.104:0/554157569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e763a11cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f06b67fc640 1 -- 172.21.15.104:0/554157569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 msgr2=0x55e763a77200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f06b67fc640 1 --2- 172.21.15.104:0/554157569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e763a77200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f06b67fc640 1 -- 172.21.15.104:0/554157569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 msgr2=0x55e763a11cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f06b67fc640 1 --2- 172.21.15.104:0/554157569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e763a11cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f06b67fc640 1 -- 172.21.15.104:0/554157569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e763a77810 con 0x55e7639ef170 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f06b67fc640 1 --2- 172.21.15.104:0/554157569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e763a117b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f06a0008ef0 tx=0x7f06a0008f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e48923723cd6dbb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f069e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f069e7fc640 1 -- 172.21.15.104:0/554157569 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a0032030 con 0x55e7639ef170 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f069e7fc640 1 -- 172.21.15.104:0/554157569 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f06a0010eb0 con 0x55e7639ef170 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f06c0406740 1 -- 172.21.15.104:0/554157569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06a00087e0 con 0x55e7639ef170 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f069e7fc640 1 -- 172.21.15.104:0/554157569 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f06a00110b0 con 0x55e7639ef170 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f06c0406740 10 client.16320 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f06c0406740 1 -- 172.21.15.104:0/554157569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e763a78b20 con 0x55e7639ef170 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f069e7fc640 1 -- 172.21.15.104:0/554157569 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06a00087e0 con 0x55e7639ef170 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.952+0000 7f069e7fc640 1 -- 172.21.15.104:0/554157569 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f06a001f050 con 0x55e7639ef170 2024-06-09T22:23:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.953+0000 7f06c0406740 10 client.16320 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.953+0000 7f06c0406740 10 client.16320 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.953+0000 7f06c0406740 20 client.16320 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.953+0000 7f06c0406740 1 --2- 172.21.15.104:0/554157569 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e763a79180 0x55e763a7b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.953+0000 7f06c0406740 4 client.16320 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:23:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.953+0000 7f06c0406740 1 -- 172.21.15.104:0/554157569 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e763960c20 con 0x55e763a79180 2024-06-09T22:23:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.953+0000 7f06b6ffd640 1 --2- 172.21.15.104:0/554157569 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e763a79180 0x55e763a7b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.953+0000 7f06b6ffd640 1 --2- 172.21.15.104:0/554157569 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e763a79180 0x55e763a7b5a0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2d8ee4194a317aa server_cookie=458dcba3aa32d5b4 in_seq=0 out_seq=0 2024-06-09T22:23:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.953+0000 7f069e7fc640 10 client.16320 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.962+0000 7f069e7fc640 1 -- 172.21.15.104:0/554157569 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55e763960c20 con 0x55e763a79180 2024-06-09T22:23:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.962+0000 7f069e7fc640 10 client.16320 handle_command_reply: tid=0 2024-06-09T22:23:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.962+0000 7f06c0406740 1 client.16320 shutdown 2024-06-09T22:23:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.962+0000 7f06c0406740 1 -- 172.21.15.104:0/554157569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 msgr2=0x55e763a117b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.962+0000 7f06c0406740 1 --2- 172.21.15.104:0/554157569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e763a117b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f06a0008ef0 tx=0x7f06a0008f20 comp rx=0 tx=0).stop 2024-06-09T22:23:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06c0406740 1 -- 172.21.15.104:0/554157569 shutdown_connections 2024-06-09T22:23:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06c0406740 1 --2- 172.21.15.104:0/554157569 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e763a79180 0x55e763a7b5a0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06c0406740 1 --2- 172.21.15.104:0/554157569 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7639fa350 0x55e763a77200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06c0406740 1 --2- 172.21.15.104:0/554157569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e7639efa90 0x55e763a11cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06c0406740 1 --2- 172.21.15.104:0/554157569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e7639ef170 0x55e763a117b0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f069e7fc640 0 client.16320 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06c0406740 1 -- 172.21.15.104:0/554157569 >> 172.21.15.104:0/554157569 conn(0x55e7639eb680 msgr2=0x55e7639f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06c0406740 1 -- 172.21.15.104:0/554157569 shutdown_connections 2024-06-09T22:23:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06c0406740 1 -- 172.21.15.104:0/554157569 wait complete. 2024-06-09T22:23:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06c0406740 20 client.16320 trim_cache size 0 max 16384 2024-06-09T22:23:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:23:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:23:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:23:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06bd6fc640 1 -- 172.21.15.104:0/2104694249 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f06a40a2da0 msgr2=0x7f06a40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06bd6fc640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f06a40a2da0 0x7f06a40a5260 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f06b0006010 tx=0x7f06b008e150 comp rx=0 tx=0).stop 2024-06-09T22:23:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06bd6fc640 1 -- 172.21.15.104:0/2104694249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06b810bf70 msgr2=0x7f06b819ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06bd6fc640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06b810bf70 0x7f06b819ca30 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f06b0004770 tx=0x7f06b0003b90 comp rx=0 tx=0).stop 2024-06-09T22:23:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06bd6fc640 1 -- 172.21.15.104:0/2104694249 shutdown_connections 2024-06-09T22:23:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06bd6fc640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f06a40a2da0 0x7f06a40a5260 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06bd6fc640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06b810bf70 0x7f06b819ca30 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06bd6fc640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b8103550 0x7f06b8198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06bd6fc640 1 --2- 172.21.15.104:0/2104694249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b8102c30 0x7f06b81981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06bd6fc640 1 -- 172.21.15.104:0/2104694249 >> 172.21.15.104:0/2104694249 conn(0x7f06b8077a20 msgr2=0x7f06b80fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06bd6fc640 1 -- 172.21.15.104:0/2104694249 shutdown_connections 2024-06-09T22:23:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:37.963+0000 7f06bd6fc640 1 -- 172.21.15.104:0/2104694249 wait complete. 2024-06-09T22:23:38.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:37 smithi104 ceph-mon[32389]: pgmap v82: 129 pgs: 129 active+clean; 32 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 231 KiB/s wr, 41 op/s 2024-06-09T22:23:38.718 INFO:tasks.workunit.client.0.smithi104.stdout: 1 25665 3.53 MB/sec warmup 44 sec latency 2556.814 ms 2024-06-09T22:23:39.719 INFO:tasks.workunit.client.0.smithi104.stdout: 1 25665 3.45 MB/sec warmup 45 sec latency 3556.915 ms 2024-06-09T22:23:39.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:39 smithi194 ceph-mon[36856]: pgmap v83: 129 pgs: 129 active+clean; 32 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 231 KiB/s wr, 41 op/s 2024-06-09T22:23:39.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:39 smithi153 ceph-mon[36599]: pgmap v83: 129 pgs: 129 active+clean; 32 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 231 KiB/s wr, 41 op/s 2024-06-09T22:23:40.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:39 smithi104 ceph-mon[32389]: pgmap v83: 129 pgs: 129 active+clean; 32 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 231 KiB/s wr, 41 op/s 2024-06-09T22:23:40.719 INFO:tasks.workunit.client.0.smithi104.stdout: 1 27751 3.61 MB/sec warmup 46 sec latency 3865.198 ms 2024-06-09T22:23:41.719 INFO:tasks.workunit.client.0.smithi104.stdout: 1 29083 3.70 MB/sec warmup 47 sec latency 220.406 ms 2024-06-09T22:23:41.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:41 smithi194 ceph-mon[36856]: pgmap v84: 129 pgs: 129 active+clean; 32 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 297 KiB/s wr, 58 op/s 2024-06-09T22:23:41.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:41 smithi153 ceph-mon[36599]: pgmap v84: 129 pgs: 129 active+clean; 32 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 297 KiB/s wr, 58 op/s 2024-06-09T22:23:42.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:41 smithi104 ceph-mon[32389]: pgmap v84: 129 pgs: 129 active+clean; 32 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 297 KiB/s wr, 58 op/s 2024-06-09T22:23:42.719 INFO:tasks.workunit.client.0.smithi104.stdout: 1 29083 3.63 MB/sec warmup 48 sec latency 1220.491 ms 2024-06-09T22:23:43.719 INFO:tasks.workunit.client.0.smithi104.stdout: 1 29083 3.55 MB/sec warmup 49 sec latency 2220.586 ms 2024-06-09T22:23:43.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:43 smithi194 ceph-mon[36856]: pgmap v85: 129 pgs: 129 active+clean; 32 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 215 KiB/s wr, 50 op/s 2024-06-09T22:23:43.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:43 smithi153 ceph-mon[36599]: pgmap v85: 129 pgs: 129 active+clean; 32 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 215 KiB/s wr, 50 op/s 2024-06-09T22:23:44.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:43 smithi104 ceph-mon[32389]: pgmap v85: 129 pgs: 129 active+clean; 32 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 215 KiB/s wr, 50 op/s 2024-06-09T22:23:44.719 INFO:tasks.workunit.client.0.smithi104.stdout: 1 29083 3.48 MB/sec warmup 50 sec latency 3220.650 ms 2024-06-09T22:23:45.719 INFO:tasks.workunit.client.0.smithi104.stdout: 1 29083 3.41 MB/sec warmup 51 sec latency 4220.762 ms 2024-06-09T22:23:45.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:45 smithi194 ceph-mon[36856]: pgmap v86: 129 pgs: 129 active+clean; 34 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 655 KiB/s wr, 93 op/s 2024-06-09T22:23:45.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:45 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:23:45.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:45 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:23:45.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:45 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:23:45.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:45 smithi153 ceph-mon[36599]: pgmap v86: 129 pgs: 129 active+clean; 34 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 655 KiB/s wr, 93 op/s 2024-06-09T22:23:45.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:45 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:23:45.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:45 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:23:45.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:45 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:23:46.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:45 smithi104 ceph-mon[32389]: pgmap v86: 129 pgs: 129 active+clean; 34 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 655 KiB/s wr, 93 op/s 2024-06-09T22:23:46.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:45 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:23:46.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:45 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:23:46.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:45 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:23:46.719 INFO:tasks.workunit.client.0.smithi104.stdout: 1 30847 3.48 MB/sec warmup 52 sec latency 4611.577 ms 2024-06-09T22:23:47.719 INFO:tasks.workunit.client.0.smithi104.stdout: 1 32686 3.65 MB/sec warmup 53 sec latency 14.204 ms 2024-06-09T22:23:47.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:47 smithi194 ceph-mon[36856]: pgmap v87: 129 pgs: 129 active+clean; 34 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 506 KiB/s wr, 59 op/s 2024-06-09T22:23:47.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:47 smithi153 ceph-mon[36599]: pgmap v87: 129 pgs: 129 active+clean; 34 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 506 KiB/s wr, 59 op/s 2024-06-09T22:23:48.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:47 smithi104 ceph-mon[32389]: pgmap v87: 129 pgs: 129 active+clean; 34 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 506 KiB/s wr, 59 op/s 2024-06-09T22:23:48.719 INFO:tasks.workunit.client.0.smithi104.stdout: 1 32687 3.58 MB/sec warmup 54 sec latency 999.655 ms 2024-06-09T22:23:49.719 INFO:tasks.workunit.client.0.smithi104.stdout: 1 32687 3.52 MB/sec warmup 55 sec latency 1999.726 ms 2024-06-09T22:23:49.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:49 smithi194 ceph-mon[36856]: pgmap v88: 129 pgs: 129 active+clean; 34 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 506 KiB/s wr, 59 op/s 2024-06-09T22:23:49.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:49 smithi153 ceph-mon[36599]: pgmap v88: 129 pgs: 129 active+clean; 34 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 506 KiB/s wr, 59 op/s 2024-06-09T22:23:50.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:49 smithi104 ceph-mon[32389]: pgmap v88: 129 pgs: 129 active+clean; 34 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 506 KiB/s wr, 59 op/s 2024-06-09T22:23:50.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 32755 3.46 MB/sec warmup 56 sec latency 2302.871 ms 2024-06-09T22:23:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.056+0000 7f45070ca640 1 -- 172.21.15.104:0/796160066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4500100bf0 msgr2=0x7f4500101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.056+0000 7f45070ca640 1 --2- 172.21.15.104:0/796160066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4500100bf0 0x7f4500101070 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009900 tx=0x7f44e802a410 comp rx=0 tx=0).stop 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.056+0000 7f45070ca640 1 -- 172.21.15.104:0/796160066 shutdown_connections 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.056+0000 7f45070ca640 1 --2- 172.21.15.104:0/796160066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45001015b0 0x7f450010d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.056+0000 7f45070ca640 1 --2- 172.21.15.104:0/796160066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4500100bf0 0x7f4500101070 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.056+0000 7f45070ca640 1 --2- 172.21.15.104:0/796160066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4500105840 0x7f4500105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.056+0000 7f45070ca640 1 -- 172.21.15.104:0/796160066 >> 172.21.15.104:0/796160066 conn(0x7f45000766a0 msgr2=0x7f4500076ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.056+0000 7f45070ca640 1 -- 172.21.15.104:0/796160066 shutdown_connections 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.056+0000 7f45070ca640 1 -- 172.21.15.104:0/796160066 wait complete. 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f45070ca640 1 Processor -- start 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f45070ca640 1 -- start start 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f45070ca640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4500100bf0 0x7f45000fe650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f45070ca640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45001015b0 0x7f45000feb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f45070ca640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4500105840 0x7f45000ff0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f45070ca640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4500110c70 con 0x7f45001015b0 2024-06-09T22:23:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f45070ca640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4500110af0 con 0x7f4500105840 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f45070ca640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4500110df0 con 0x7f4500100bf0 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f44fffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45001015b0 0x7f45000feb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f44fffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45001015b0 0x7f45000feb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55634/0 (socket says 172.21.15.104:55634) 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f44fffff640 1 -- 172.21.15.104:0/4240713776 learned_addr learned my addr 172.21.15.104:0/4240713776 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f4504e66640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4500100bf0 0x7f45000fe650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f4505667640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4500105840 0x7f45000ff0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f44fffff640 1 -- 172.21.15.104:0/4240713776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4500100bf0 msgr2=0x7f45000fe650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f44fffff640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4500100bf0 0x7f45000fe650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f44fffff640 1 -- 172.21.15.104:0/4240713776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4500105840 msgr2=0x7f45000ff0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f44fffff640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4500105840 0x7f45000ff0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.057+0000 7f44fffff640 1 -- 172.21.15.104:0/4240713776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45001a6640 con 0x7f45001015b0 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.058+0000 7f44fffff640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45001015b0 0x7f45000feb90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f44e8006010 tx=0x7f44e80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.058+0000 7f44fdffb640 1 -- 172.21.15.104:0/4240713776 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e803f030 con 0x7f45001015b0 2024-06-09T22:23:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.058+0000 7f44fdffb640 1 -- 172.21.15.104:0/4240713776 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44e8003ce0 con 0x7f45001015b0 2024-06-09T22:23:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.058+0000 7f45070ca640 1 -- 172.21.15.104:0/4240713776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45001a68d0 con 0x7f45001015b0 2024-06-09T22:23:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.058+0000 7f44fdffb640 1 -- 172.21.15.104:0/4240713776 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8035070 con 0x7f45001015b0 2024-06-09T22:23:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.058+0000 7f45070ca640 1 -- 172.21.15.104:0/4240713776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45001a6e10 con 0x7f45001015b0 2024-06-09T22:23:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.060+0000 7f44fdffb640 1 -- 172.21.15.104:0/4240713776 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f44e8042030 con 0x7f45001015b0 2024-06-09T22:23:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4509dd7740 1 Processor -- start 2024-06-09T22:23:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4509dd7740 1 -- start start 2024-06-09T22:23:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4509dd7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a48f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4509dd7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 0x559e6a499e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4509dd7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 0x559e6a49c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4509dd7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559e6a3f34f0 con 0x559e6a48fa90 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4509dd7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559e6a391870 con 0x559e6a48f170 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4509dd7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559e6a3ef4b0 con 0x559e6a49a350 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4504e66640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 0x559e6a499e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4504e66640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 0x559e6a499e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55648/0 (socket says 172.21.15.104:55648) 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4504e66640 1 -- 172.21.15.104:0/2915866097 learned_addr learned my addr 172.21.15.104:0/2915866097 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f44fdffb640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44ec0a2e20 0x7f44ec0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f44fffff640 1 --2- 172.21.15.104:0/2915866097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 0x559e6a49c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4505667640 1 --2- 172.21.15.104:0/2915866097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a48f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4505667640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44ec0a2e20 0x7f44ec0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.068 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4504e66640 1 -- 172.21.15.104:0/2915866097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 msgr2=0x559e6a49c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4504e66640 1 --2- 172.21.15.104:0/2915866097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 0x559e6a49c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4504e66640 1 -- 172.21.15.104:0/2915866097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 msgr2=0x559e6a48f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4504e66640 1 --2- 172.21.15.104:0/2915866097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a48f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f4504e66640 1 -- 172.21.15.104:0/2915866097 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e6a3ad9b0 con 0x559e6a48fa90 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.061+0000 7f44fdffb640 1 -- 172.21.15.104:0/4240713776 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f44e80efd70 con 0x7f45001015b0 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.062+0000 7f4504e66640 1 --2- 172.21.15.104:0/2915866097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 0x559e6a499e10 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f44f000b710 tx=0x7f44f0004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41be72c0f5c6f72d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.062+0000 7f44f6ffd640 1 -- 172.21.15.104:0/2915866097 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f0003bd0 con 0x559e6a48fa90 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.062+0000 7f4505667640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44ec0a2e20 0x7f44ec0a52e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f44f8009fd0 tx=0x7f44f8005bd0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.062+0000 7f44f6ffd640 1 -- 172.21.15.104:0/2915866097 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f0015070 con 0x559e6a48fa90 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.062+0000 7f44f6ffd640 1 -- 172.21.15.104:0/2915866097 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f0004e40 con 0x559e6a48fa90 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.062+0000 7f4509dd7740 1 -- 172.21.15.104:0/2915866097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 msgr2=0x559e6a499e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.062+0000 7f4509dd7740 1 --2- 172.21.15.104:0/2915866097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 0x559e6a499e10 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f44f000b710 tx=0x7f44f0004610 comp rx=0 tx=0).stop 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.062+0000 7f4509dd7740 1 -- 172.21.15.104:0/2915866097 shutdown_connections 2024-06-09T22:23:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.062+0000 7f4509dd7740 1 --2- 172.21.15.104:0/2915866097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 0x559e6a49c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.062+0000 7f4509dd7740 1 --2- 172.21.15.104:0/2915866097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 0x559e6a499e10 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.062+0000 7f4509dd7740 1 --2- 172.21.15.104:0/2915866097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a48f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.062+0000 7f4509dd7740 1 -- 172.21.15.104:0/2915866097 >> 172.21.15.104:0/2915866097 conn(0x559e6a48b680 msgr2=0x559e6a48ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f4509dd7740 1 -- 172.21.15.104:0/2915866097 shutdown_connections 2024-06-09T22:23:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f4509dd7740 1 -- 172.21.15.104:0/2915866097 wait complete. 2024-06-09T22:23:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f4509dd7740 1 -- start start 2024-06-09T22:23:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f4509dd7740 1 Processor -- start 2024-06-09T22:23:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f4509dd7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a4972b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f4509dd7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 0x559e6a4977f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f4509dd7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 0x559e6a497d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f4509dd7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559e6a49cf80 con 0x559e6a48fa90 2024-06-09T22:23:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f4509dd7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559e6a37cae0 con 0x559e6a48f170 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f4509dd7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559e6a49d100 con 0x559e6a49a350 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f4509dd7740 10 client.0 _ll_register_callbacks cb 0x559e6a3b59b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f44fffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 0x559e6a4977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f44fffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 0x559e6a4977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55662/0 (socket says 172.21.15.104:55662) 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.063+0000 7f44fffff640 1 -- 172.21.15.104:0/1792766323 learned_addr learned my addr 172.21.15.104:0/1792766323 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f4505667640 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 0x559e6a497d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f4504e66640 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a4972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f44fffff640 1 -- 172.21.15.104:0/1792766323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 msgr2=0x559e6a497d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f44fffff640 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 0x559e6a497d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f44fffff640 1 -- 172.21.15.104:0/1792766323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 msgr2=0x559e6a4972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f44fffff640 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a4972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f44fffff640 1 -- 172.21.15.104:0/1792766323 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e6a107210 con 0x559e6a48fa90 2024-06-09T22:23:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f44fffff640 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 0x559e6a4977f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f44e80c4720 tx=0x7f44e808afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a644f12f577fe67 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f44dffff640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f44dffff640 1 -- 172.21.15.104:0/1792766323 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8049070 con 0x559e6a48fa90 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f44dffff640 1 -- 172.21.15.104:0/1792766323 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f44e80bc240 con 0x559e6a48fa90 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f4509dd7740 1 -- 172.21.15.104:0/1792766323 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44f0015070 con 0x559e6a48fa90 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f44dffff640 1 -- 172.21.15.104:0/1792766323 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f44e80bbdb0 con 0x559e6a48fa90 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f4509dd7740 10 client.16332 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f4509dd7740 1 -- 172.21.15.104:0/1792766323 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559e6a1d3620 con 0x559e6a48fa90 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.064+0000 7f44dffff640 1 -- 172.21.15.104:0/1792766323 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f0015070 con 0x559e6a48fa90 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.065+0000 7f44dffff640 1 -- 172.21.15.104:0/1792766323 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f44e80c47c0 con 0x559e6a48fa90 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.065+0000 7f4509dd7740 10 client.16332 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.065+0000 7f4509dd7740 10 client.16332 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.065+0000 7f4509dd7740 20 client.16332 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.065+0000 7f4509dd7740 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559e6a4aa3e0 0x559e6a4ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.065+0000 7f4509dd7740 4 client.16332 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:23:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.065+0000 7f4509dd7740 1 -- 172.21.15.104:0/1792766323 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e6a406f10 con 0x559e6a4aa3e0 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.065+0000 7f4504e66640 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559e6a4aa3e0 0x559e6a4ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.065+0000 7f4504e66640 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559e6a4aa3e0 0x559e6a4ac8a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da40a1f92994eefc server_cookie=c45090b3f95949e4 in_seq=0 out_seq=0 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.065+0000 7f44dffff640 10 client.16332 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f44dffff640 1 -- 172.21.15.104:0/1792766323 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x559e6a406f10 con 0x559e6a4aa3e0 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f44dffff640 10 client.16332 handle_command_reply: tid=0 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f4509dd7740 1 client.16332 shutdown 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f4509dd7740 1 -- 172.21.15.104:0/1792766323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 msgr2=0x559e6a4977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f4509dd7740 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 0x559e6a4977f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f44e80c4720 tx=0x7f44e808afd0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f4509dd7740 1 -- 172.21.15.104:0/1792766323 shutdown_connections 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f4509dd7740 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559e6a4aa3e0 0x559e6a4ac8a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f4509dd7740 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 0x559e6a497d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f4509dd7740 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48fa90 0x559e6a4977f0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f44dffff640 0 client.16332 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f4509dd7740 1 --2- 172.21.15.104:0/1792766323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a4972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f4509dd7740 1 -- 172.21.15.104:0/1792766323 >> 172.21.15.104:0/1792766323 conn(0x559e6a48b680 msgr2=0x559e6a491400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f4509dd7740 1 -- 172.21.15.104:0/1792766323 shutdown_connections 2024-06-09T22:23:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.066+0000 7f4509dd7740 1 -- 172.21.15.104:0/1792766323 wait complete. 2024-06-09T22:23:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.067+0000 7f4509dd7740 20 client.16332 trim_cache size 0 max 16384 2024-06-09T22:23:51.083 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.081+0000 7f4509dd7740 1 Processor -- start 2024-06-09T22:23:51.084 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4509dd7740 1 -- start start 2024-06-09T22:23:51.084 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4509dd7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a4b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.084 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4509dd7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a48fa90 0x559e6a4b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.084 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4509dd7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a49a350 0x559e6a489f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.084 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4509dd7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f45001b0ee0 con 0x559e6a49a350 2024-06-09T22:23:51.084 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4509dd7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45001b2a80 con 0x559e6a48f170 2024-06-09T22:23:51.084 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4509dd7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f45001b15b0 con 0x559e6a48fa90 2024-06-09T22:23:51.084 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4505667640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a49a350 0x559e6a489f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.084 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4505667640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a49a350 0x559e6a489f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55676/0 (socket says 172.21.15.104:55676) 2024-06-09T22:23:51.084 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4505667640 1 -- 172.21.15.104:0/72325594 learned_addr learned my addr 172.21.15.104:0/72325594 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:51.084 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f44fffff640 1 --2- 172.21.15.104:0/72325594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a48fa90 0x559e6a4b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.084 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4504e66640 1 --2- 172.21.15.104:0/72325594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a4b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4505667640 1 -- 172.21.15.104:0/72325594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a48fa90 msgr2=0x559e6a4b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4505667640 1 --2- 172.21.15.104:0/72325594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a48fa90 0x559e6a4b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4505667640 1 -- 172.21.15.104:0/72325594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 msgr2=0x559e6a4b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4505667640 1 --2- 172.21.15.104:0/72325594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a4b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4505667640 1 -- 172.21.15.104:0/72325594 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e6a48f8c0 con 0x559e6a49a350 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f44fffff640 1 --2- 172.21.15.104:0/72325594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a48fa90 0x559e6a4b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4504e66640 1 --2- 172.21.15.104:0/72325594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a4b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f4505667640 1 --2- 172.21.15.104:0/72325594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a49a350 0x559e6a489f50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f44f8010910 tx=0x7f44f8009720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d80e8f97e9f4b6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f44f67fc640 1 -- 172.21.15.104:0/72325594 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8004b20 con 0x559e6a49a350 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f44f67fc640 1 -- 172.21.15.104:0/72325594 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f8004cc0 con 0x559e6a49a350 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.082+0000 7f44f67fc640 1 -- 172.21.15.104:0/72325594 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8005810 con 0x559e6a49a350 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.083+0000 7f4509dd7740 1 -- 172.21.15.104:0/72325594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a49a350 msgr2=0x559e6a489f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:51.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.083+0000 7f4509dd7740 1 --2- 172.21.15.104:0/72325594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a49a350 0x559e6a489f50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f44f8010910 tx=0x7f44f8009720 comp rx=0 tx=0).stop 2024-06-09T22:23:51.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.083+0000 7f4509dd7740 1 -- 172.21.15.104:0/72325594 shutdown_connections 2024-06-09T22:23:51.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.083+0000 7f4509dd7740 1 --2- 172.21.15.104:0/72325594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a49a350 0x559e6a489f50 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.083+0000 7f4509dd7740 1 --2- 172.21.15.104:0/72325594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a48fa90 0x559e6a4b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.083+0000 7f4509dd7740 1 --2- 172.21.15.104:0/72325594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48f170 0x559e6a4b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.083+0000 7f4509dd7740 1 -- 172.21.15.104:0/72325594 >> 172.21.15.104:0/72325594 conn(0x559e6a48b680 msgr2=0x559e6a492590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:51.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.083+0000 7f4509dd7740 1 -- 172.21.15.104:0/72325594 shutdown_connections 2024-06-09T22:23:51.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.083+0000 7f4509dd7740 1 -- 172.21.15.104:0/72325594 wait complete. 2024-06-09T22:23:51.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.083+0000 7f4509dd7740 1 -- start start 2024-06-09T22:23:51.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.083+0000 7f4509dd7740 1 Processor -- start 2024-06-09T22:23:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4509dd7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48f170 0x559e6a4a28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4509dd7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48fa90 0x559e6a49dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4509dd7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 0x559e6a49e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4509dd7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f45001b2350 con 0x559e6a48f170 2024-06-09T22:23:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4509dd7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45001b1c80 con 0x559e6a48fa90 2024-06-09T22:23:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4509dd7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f45001aeea0 con 0x559e6a49a350 2024-06-09T22:23:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4509dd7740 10 client.0 _ll_register_callbacks cb 0x7f44e8032ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:23:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4504e66640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48f170 0x559e6a4a28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4504e66640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48f170 0x559e6a4a28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55686/0 (socket says 172.21.15.104:55686) 2024-06-09T22:23:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4504e66640 1 -- 172.21.15.104:0/2579694673 learned_addr learned my addr 172.21.15.104:0/2579694673 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:23:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f44fffff640 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48fa90 0x559e6a49dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4505667640 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 0x559e6a49e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4504e66640 1 -- 172.21.15.104:0/2579694673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 msgr2=0x559e6a49e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4504e66640 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 0x559e6a49e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4504e66640 1 -- 172.21.15.104:0/2579694673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48fa90 msgr2=0x559e6a49dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:23:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4504e66640 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48fa90 0x559e6a49dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4504e66640 1 -- 172.21.15.104:0/2579694673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e6a49e7c0 con 0x559e6a48f170 2024-06-09T22:23:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f4504e66640 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48f170 0x559e6a4a28d0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f44f000b710 tx=0x7f44f0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e321b53de794980 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:23:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.084+0000 7f44dffff640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:23:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f44dffff640 1 -- 172.21.15.104:0/2579694673 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f00047c0 con 0x559e6a48f170 2024-06-09T22:23:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f44dffff640 1 -- 172.21.15.104:0/2579694673 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f44f0003de0 con 0x559e6a48f170 2024-06-09T22:23:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f4509dd7740 1 -- 172.21.15.104:0/2579694673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44f8004cc0 con 0x559e6a48f170 2024-06-09T22:23:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f44dffff640 1 -- 172.21.15.104:0/2579694673 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f44f0036050 con 0x559e6a48f170 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f4509dd7740 10 client.16338 fetch_fsmap learned FSMap version 30 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f4509dd7740 1 -- 172.21.15.104:0/2579694673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559e6a514f70 con 0x559e6a48f170 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f44dffff640 1 -- 172.21.15.104:0/2579694673 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f8004cc0 con 0x559e6a48f170 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f44dffff640 1 -- 172.21.15.104:0/2579694673 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f44f00274d0 con 0x559e6a48f170 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f4509dd7740 10 client.16338 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f4509dd7740 10 client.16338 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f4509dd7740 20 client.16338 populate_metadata read hostname 'smithi104' 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f4509dd7740 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559e6a515630 0x559e6a517a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f4509dd7740 4 client.16338 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f4509dd7740 1 -- 172.21.15.104:0/2579694673 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559e6a400c20 con 0x559e6a515630 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.085+0000 7f44fffff640 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559e6a515630 0x559e6a517a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.086+0000 7f44dffff640 10 client.16338 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.086+0000 7f44fffff640 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559e6a515630 0x559e6a517a50 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c0c2877143999f0 server_cookie=485dca1c698a5291 in_seq=0 out_seq=0 2024-06-09T22:23:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f44dffff640 1 -- 172.21.15.104:0/2579694673 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x559e6a400c20 con 0x559e6a515630 2024-06-09T22:23:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f44dffff640 10 client.16338 handle_command_reply: tid=0 2024-06-09T22:23:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f4509dd7740 1 client.16338 shutdown 2024-06-09T22:23:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f4509dd7740 1 -- 172.21.15.104:0/2579694673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48f170 msgr2=0x559e6a4a28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f4509dd7740 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48f170 0x559e6a4a28d0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f44f000b710 tx=0x7f44f0005f90 comp rx=0 tx=0).stop 2024-06-09T22:23:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f4509dd7740 1 -- 172.21.15.104:0/2579694673 shutdown_connections 2024-06-09T22:23:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f4509dd7740 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559e6a515630 0x559e6a517a50 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f4509dd7740 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559e6a49a350 0x559e6a49e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f44dffff640 0 client.16338 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:23:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f4509dd7740 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e6a48fa90 0x559e6a49dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f4509dd7740 1 --2- 172.21.15.104:0/2579694673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559e6a48f170 0x559e6a4a28d0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f4509dd7740 1 -- 172.21.15.104:0/2579694673 >> 172.21.15.104:0/2579694673 conn(0x559e6a48b680 msgr2=0x559e6a497270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f4509dd7740 1 -- 172.21.15.104:0/2579694673 shutdown_connections 2024-06-09T22:23:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f4509dd7740 1 -- 172.21.15.104:0/2579694673 wait complete. 2024-06-09T22:23:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.095+0000 7f4509dd7740 20 client.16338 trim_cache size 0 max 16384 2024-06-09T22:23:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:23:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:23:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:23:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.096+0000 7f45070ca640 1 -- 172.21.15.104:0/4240713776 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44ec0a2e20 msgr2=0x7f44ec0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.096+0000 7f45070ca640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44ec0a2e20 0x7f44ec0a52e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f44f8009fd0 tx=0x7f44f8005bd0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.096+0000 7f45070ca640 1 -- 172.21.15.104:0/4240713776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45001015b0 msgr2=0x7f45000feb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:23:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.096+0000 7f45070ca640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45001015b0 0x7f45000feb90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f44e8006010 tx=0x7f44e80042f0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.096+0000 7f45070ca640 1 -- 172.21.15.104:0/4240713776 shutdown_connections 2024-06-09T22:23:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.096+0000 7f45070ca640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44ec0a2e20 0x7f44ec0a52e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.096+0000 7f45070ca640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4500105840 0x7f45000ff0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.096+0000 7f45070ca640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45001015b0 0x7f45000feb90 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.096+0000 7f45070ca640 1 --2- 172.21.15.104:0/4240713776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4500100bf0 0x7f45000fe650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:23:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.096+0000 7f45070ca640 1 -- 172.21.15.104:0/4240713776 >> 172.21.15.104:0/4240713776 conn(0x7f45000766a0 msgr2=0x7f450006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:23:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.096+0000 7f45070ca640 1 -- 172.21.15.104:0/4240713776 shutdown_connections 2024-06-09T22:23:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:23:51.096+0000 7f45070ca640 1 -- 172.21.15.104:0/4240713776 wait complete. 2024-06-09T22:23:51.719 INFO:tasks.workunit.client.0.smithi104.stdout: 1 34523 3.53 MB/sec warmup 57 sec latency 1029.675 ms 2024-06-09T22:23:51.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:51 smithi153 ceph-mon[36599]: pgmap v89: 129 pgs: 129 active+clean; 36 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 725 KiB/s wr, 95 op/s 2024-06-09T22:23:52.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:51 smithi104 ceph-mon[32389]: pgmap v89: 129 pgs: 129 active+clean; 36 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 725 KiB/s wr, 95 op/s 2024-06-09T22:23:52.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:51 smithi194 ceph-mon[36856]: pgmap v89: 129 pgs: 129 active+clean; 36 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 725 KiB/s wr, 95 op/s 2024-06-09T22:23:52.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 36174 3.66 MB/sec warmup 58 sec latency 89.779 ms 2024-06-09T22:23:53.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 36174 3.60 MB/sec warmup 59 sec latency 1089.816 ms 2024-06-09T22:23:53.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:53 smithi153 ceph-mon[36599]: pgmap v90: 129 pgs: 129 active+clean; 36 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 659 KiB/s wr, 78 op/s 2024-06-09T22:23:54.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:53 smithi104 ceph-mon[32389]: pgmap v90: 129 pgs: 129 active+clean; 36 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 659 KiB/s wr, 78 op/s 2024-06-09T22:23:54.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:53 smithi194 ceph-mon[36856]: pgmap v90: 129 pgs: 129 active+clean; 36 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 659 KiB/s wr, 78 op/s 2024-06-09T22:23:54.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 36174 3.54 MB/sec warmup 60 sec latency 2089.906 ms 2024-06-09T22:23:55.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 36174 3.48 MB/sec warmup 61 sec latency 3089.949 ms 2024-06-09T22:23:55.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:55 smithi153 ceph-mon[36599]: pgmap v91: 129 pgs: 129 active+clean; 37 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 860 KiB/s wr, 116 op/s 2024-06-09T22:23:56.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:55 smithi104 ceph-mon[32389]: pgmap v91: 129 pgs: 129 active+clean; 37 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 860 KiB/s wr, 116 op/s 2024-06-09T22:23:56.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:55 smithi194 ceph-mon[36856]: pgmap v91: 129 pgs: 129 active+clean; 37 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 860 KiB/s wr, 116 op/s 2024-06-09T22:23:56.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 36298 3.43 MB/sec warmup 62 sec latency 3479.947 ms 2024-06-09T22:23:57.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 36298 3.37 MB/sec warmup 63 sec latency 1503.411 ms 2024-06-09T22:23:57.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:57 smithi153 ceph-mon[36599]: pgmap v92: 129 pgs: 129 active+clean; 37 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 420 KiB/s wr, 73 op/s 2024-06-09T22:23:58.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:57 smithi104 ceph-mon[32389]: pgmap v92: 129 pgs: 129 active+clean; 37 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 420 KiB/s wr, 73 op/s 2024-06-09T22:23:58.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:57 smithi194 ceph-mon[36856]: pgmap v92: 129 pgs: 129 active+clean; 37 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 420 KiB/s wr, 73 op/s 2024-06-09T22:23:58.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 36298 3.32 MB/sec warmup 64 sec latency 2503.496 ms 2024-06-09T22:23:59.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 36298 3.27 MB/sec warmup 65 sec latency 3503.587 ms 2024-06-09T22:23:59.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:23:59 smithi153 ceph-mon[36599]: pgmap v93: 129 pgs: 129 active+clean; 37 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 420 KiB/s wr, 73 op/s 2024-06-09T22:24:00.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:23:59 smithi104 ceph-mon[32389]: pgmap v93: 129 pgs: 129 active+clean; 37 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 420 KiB/s wr, 73 op/s 2024-06-09T22:24:00.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:23:59 smithi194 ceph-mon[36856]: pgmap v93: 129 pgs: 129 active+clean; 37 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 420 KiB/s wr, 73 op/s 2024-06-09T22:24:00.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 37993 3.33 MB/sec warmup 66 sec latency 3808.997 ms 2024-06-09T22:24:01.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 39162 3.43 MB/sec warmup 67 sec latency 54.543 ms 2024-06-09T22:24:01.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:01 smithi153 ceph-mon[36599]: pgmap v94: 129 pgs: 129 active+clean; 38 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 449 KiB/s wr, 81 op/s 2024-06-09T22:24:02.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:01 smithi104 ceph-mon[32389]: pgmap v94: 129 pgs: 129 active+clean; 38 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 449 KiB/s wr, 81 op/s 2024-06-09T22:24:02.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:01 smithi194 ceph-mon[36856]: pgmap v94: 129 pgs: 129 active+clean; 38 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 449 KiB/s wr, 81 op/s 2024-06-09T22:24:02.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 39846 3.41 MB/sec warmup 68 sec latency 327.733 ms 2024-06-09T22:24:03.720 INFO:tasks.workunit.client.0.smithi104.stdout: 1 39846 3.36 MB/sec warmup 69 sec latency 1327.848 ms 2024-06-09T22:24:03.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:03 smithi153 ceph-mon[36599]: pgmap v95: 129 pgs: 129 active+clean; 38 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 230 KiB/s wr, 46 op/s 2024-06-09T22:24:04.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:03 smithi104 ceph-mon[32389]: pgmap v95: 129 pgs: 129 active+clean; 38 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 230 KiB/s wr, 46 op/s 2024-06-09T22:24:04.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:03 smithi194 ceph-mon[36856]: pgmap v95: 129 pgs: 129 active+clean; 38 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 230 KiB/s wr, 46 op/s 2024-06-09T22:24:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.192+0000 7f220fe31640 1 -- 172.21.15.104:0/3717088265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2208107990 msgr2=0x7f2208107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.192+0000 7f220fe31640 1 --2- 172.21.15.104:0/3717088265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2208107990 0x7f2208107d70 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2204009900 tx=0x7f220402a410 comp rx=0 tx=0).stop 2024-06-09T22:24:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.192+0000 7f220fe31640 1 -- 172.21.15.104:0/3717088265 shutdown_connections 2024-06-09T22:24:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.192+0000 7f220fe31640 1 --2- 172.21.15.104:0/3717088265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f220806abb0 0x7f220810e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.192+0000 7f220fe31640 1 --2- 172.21.15.104:0/3717088265 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f220806a210 0x7f220806a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.192+0000 7f220fe31640 1 --2- 172.21.15.104:0/3717088265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2208107990 0x7f2208107d70 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.192+0000 7f220fe31640 1 -- 172.21.15.104:0/3717088265 >> 172.21.15.104:0/3717088265 conn(0x7f22080766a0 msgr2=0x7f2208076ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.192+0000 7f220fe31640 1 -- 172.21.15.104:0/3717088265 shutdown_connections 2024-06-09T22:24:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.192+0000 7f220fe31640 1 -- 172.21.15.104:0/3717088265 wait complete. 2024-06-09T22:24:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.192+0000 7f220fe31640 1 Processor -- start 2024-06-09T22:24:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.192+0000 7f220fe31640 1 -- start start 2024-06-09T22:24:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220fe31640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220806a210 0x7f2208198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220fe31640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f220806abb0 0x7f22081986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220fe31640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2208107990 0x7f220819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220fe31640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2208111d80 con 0x7f220806a210 2024-06-09T22:24:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220fe31640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2208111c00 con 0x7f2208107990 2024-06-09T22:24:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220fe31640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2208111f00 con 0x7f220806abb0 2024-06-09T22:24:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220dbcd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220806a210 0x7f2208198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220dbcd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220806a210 0x7f2208198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50150/0 (socket says 172.21.15.104:50150) 2024-06-09T22:24:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220dbcd640 1 -- 172.21.15.104:0/172401623 learned_addr learned my addr 172.21.15.104:0/172401623 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220d3cc640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f220806abb0 0x7f22081986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220dbcd640 1 -- 172.21.15.104:0/172401623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f220806abb0 msgr2=0x7f22081986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220dbcd640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f220806abb0 0x7f22081986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220dbcd640 1 -- 172.21.15.104:0/172401623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2208107990 msgr2=0x7f220819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220dbcd640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2208107990 0x7f220819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220dbcd640 1 -- 172.21.15.104:0/172401623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220819d170 con 0x7f220806a210 2024-06-09T22:24:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220dbcd640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220806a210 0x7f2208198180 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2204006010 tx=0x7f22040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f21feffd640 1 -- 172.21.15.104:0/172401623 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220403f030 con 0x7f220806a210 2024-06-09T22:24:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f21feffd640 1 -- 172.21.15.104:0/172401623 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2204003ce0 con 0x7f220806a210 2024-06-09T22:24:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220fe31640 1 -- 172.21.15.104:0/172401623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220819d400 con 0x7f220806a210 2024-06-09T22:24:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f21feffd640 1 -- 172.21.15.104:0/172401623 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204035070 con 0x7f220806a210 2024-06-09T22:24:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.193+0000 7f220fe31640 1 -- 172.21.15.104:0/172401623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22081a4cf0 con 0x7f220806a210 2024-06-09T22:24:04.203 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.196+0000 7f21feffd640 1 -- 172.21.15.104:0/172401623 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f2204042030 con 0x7f220806a210 2024-06-09T22:24:04.203 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.196+0000 7f2212c1b740 1 Processor -- start 2024-06-09T22:24:04.203 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.196+0000 7f2212c1b740 1 -- start start 2024-06-09T22:24:04.203 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f2212c1b740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bad550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f2212c1b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bada90 0x55b116bb7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f2212c1b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bb8350 0x55b116bba740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f2212c1b740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b116b114f0 con 0x55b116bad170 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f2212c1b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b116aaf870 con 0x55b116bada90 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f2212c1b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b116b0d4b0 con 0x55b116bb8350 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f21feffd640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f21f40a2e20 0x7f21f40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220e3ce640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220e3ce640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bad550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50162/0 (socket says 172.21.15.104:50162) 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220e3ce640 1 -- 172.21.15.104:0/4291015474 learned_addr learned my addr 172.21.15.104:0/4291015474 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220e3ce640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f21f40a2e20 0x7f21f40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f21feffd640 1 -- 172.21.15.104:0/172401623 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f22040efd30 con 0x7f220806a210 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220d3cc640 1 --2- 172.21.15.104:0/4291015474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bb8350 0x55b116bba740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220dbcd640 1 --2- 172.21.15.104:0/4291015474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bada90 0x55b116bb7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220e3ce640 1 -- 172.21.15.104:0/4291015474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bb8350 msgr2=0x55b116bba740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220e3ce640 1 --2- 172.21.15.104:0/4291015474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bb8350 0x55b116bba740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220e3ce640 1 -- 172.21.15.104:0/4291015474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bada90 msgr2=0x55b116bb7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220e3ce640 1 --2- 172.21.15.104:0/4291015474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bada90 0x55b116bb7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220e3ce640 1 -- 172.21.15.104:0/4291015474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b116acb9b0 con 0x55b116bad170 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220d3cc640 1 --2- 172.21.15.104:0/4291015474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bb8350 0x55b116bba740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.197+0000 7f220e3ce640 1 --2- 172.21.15.104:0/4291015474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bad550 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2200012d20 tx=0x7f220000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8601705878ea1815 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.198+0000 7f220e3ce640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f21f40a2e20 0x7f21f40a52e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f220000fc30 tx=0x7f220001a000 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.198+0000 7f21f3fff640 1 -- 172.21.15.104:0/4291015474 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200020070 con 0x55b116bad170 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.198+0000 7f21f3fff640 1 -- 172.21.15.104:0/4291015474 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2200009bc0 con 0x55b116bad170 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.198+0000 7f21f3fff640 1 -- 172.21.15.104:0/4291015474 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220001ce30 con 0x55b116bad170 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.198+0000 7f2212c1b740 1 -- 172.21.15.104:0/4291015474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 msgr2=0x55b116bad550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.198+0000 7f2212c1b740 1 --2- 172.21.15.104:0/4291015474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bad550 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2200012d20 tx=0x7f220000f730 comp rx=0 tx=0).stop 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.198+0000 7f2212c1b740 1 -- 172.21.15.104:0/4291015474 shutdown_connections 2024-06-09T22:24:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.198+0000 7f2212c1b740 1 --2- 172.21.15.104:0/4291015474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bb8350 0x55b116bba740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.198+0000 7f2212c1b740 1 --2- 172.21.15.104:0/4291015474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bada90 0x55b116bb7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.198+0000 7f2212c1b740 1 --2- 172.21.15.104:0/4291015474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bad550 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.198+0000 7f2212c1b740 1 -- 172.21.15.104:0/4291015474 >> 172.21.15.104:0/4291015474 conn(0x55b116ba9680 msgr2=0x55b116ba9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.199+0000 7f2212c1b740 1 -- 172.21.15.104:0/4291015474 shutdown_connections 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.199+0000 7f2212c1b740 1 -- 172.21.15.104:0/4291015474 wait complete. 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.199+0000 7f2212c1b740 1 -- start start 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.199+0000 7f2212c1b740 1 Processor -- start 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.199+0000 7f2212c1b740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bb53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.199+0000 7f2212c1b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bada90 0x55b116bb58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.199+0000 7f2212c1b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bb8350 0x55b116bb5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.199+0000 7f2212c1b740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b116bbaf20 con 0x55b116bad170 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.199+0000 7f2212c1b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b116a9aae0 con 0x55b116bb8350 2024-06-09T22:24:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.199+0000 7f2212c1b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b116bbb0a0 con 0x55b116bada90 2024-06-09T22:24:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.199+0000 7f2212c1b740 10 client.0 _ll_register_callbacks cb 0x55b116ad39b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f220dbcd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bb53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f220dbcd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bb53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50168/0 (socket says 172.21.15.104:50168) 2024-06-09T22:24:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f220dbcd640 1 -- 172.21.15.104:0/1044548868 learned_addr learned my addr 172.21.15.104:0/1044548868 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f220d3cc640 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bada90 0x55b116bb58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f220e3ce640 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bb8350 0x55b116bb5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f220dbcd640 1 -- 172.21.15.104:0/1044548868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bada90 msgr2=0x55b116bb58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f220dbcd640 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bada90 0x55b116bb58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f220dbcd640 1 -- 172.21.15.104:0/1044548868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bb8350 msgr2=0x55b116bb5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f220dbcd640 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bb8350 0x55b116bb5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f220dbcd640 1 -- 172.21.15.104:0/1044548868 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b116825210 con 0x55b116bad170 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f220dbcd640 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bb53b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f22040bbf30 tx=0x7f22040bbba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e32aa4dc7f2903 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f21f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f21f17fa640 1 -- 172.21.15.104:0/1044548868 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204049050 con 0x55b116bad170 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f21f17fa640 1 -- 172.21.15.104:0/1044548868 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f22040bc200 con 0x55b116bad170 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f2212c1b740 1 -- 172.21.15.104:0/1044548868 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2200009bc0 con 0x55b116bad170 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f21f17fa640 1 -- 172.21.15.104:0/1044548868 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f22040044e0 con 0x55b116bad170 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f2212c1b740 10 client.16350 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.200+0000 7f2212c1b740 1 -- 172.21.15.104:0/1044548868 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b1168f1620 con 0x55b116bad170 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.201+0000 7f21f17fa640 1 -- 172.21.15.104:0/1044548868 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2200009bc0 con 0x55b116bad170 2024-06-09T22:24:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.201+0000 7f21f17fa640 1 -- 172.21.15.104:0/1044548868 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f22040c4780 con 0x55b116bad170 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.201+0000 7f2212c1b740 10 client.16350 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.201+0000 7f2212c1b740 10 client.16350 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.201+0000 7f2212c1b740 20 client.16350 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.201+0000 7f2212c1b740 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b116bc85d0 0x55b116bcaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.201+0000 7f2212c1b740 4 client.16350 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.201+0000 7f2212c1b740 1 -- 172.21.15.104:0/1044548868 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b116b24f10 con 0x55b116bc85d0 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.201+0000 7f220d3cc640 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b116bc85d0 0x55b116bcaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.201+0000 7f21f17fa640 10 client.16350 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.201+0000 7f220d3cc640 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b116bc85d0 0x55b116bcaa50 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd31670bf805bcb9 server_cookie=3dc8813ff619df2c in_seq=0 out_seq=0 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f21f17fa640 1 -- 172.21.15.104:0/1044548868 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b116b24f10 con 0x55b116bc85d0 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f21f17fa640 10 client.16350 handle_command_reply: tid=0 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f2212c1b740 1 client.16350 shutdown 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f2212c1b740 1 -- 172.21.15.104:0/1044548868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 msgr2=0x55b116bb53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:04.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f2212c1b740 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bb53b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f22040bbf30 tx=0x7f22040bbba0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f2212c1b740 1 -- 172.21.15.104:0/1044548868 shutdown_connections 2024-06-09T22:24:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f2212c1b740 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b116bc85d0 0x55b116bcaa50 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f2212c1b740 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bb8350 0x55b116bb5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f2212c1b740 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bada90 0x55b116bb58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f21f17fa640 0 client.16350 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f2212c1b740 1 --2- 172.21.15.104:0/1044548868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bb53b0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f2212c1b740 1 -- 172.21.15.104:0/1044548868 >> 172.21.15.104:0/1044548868 conn(0x55b116ba9680 msgr2=0x55b116baf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f2212c1b740 1 -- 172.21.15.104:0/1044548868 shutdown_connections 2024-06-09T22:24:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.202+0000 7f2212c1b740 1 -- 172.21.15.104:0/1044548868 wait complete. 2024-06-09T22:24:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.203+0000 7f2212c1b740 20 client.16350 trim_cache size 0 max 16384 2024-06-09T22:24:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.217+0000 7f2212c1b740 1 Processor -- start 2024-06-09T22:24:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f2212c1b740 1 -- start start 2024-06-09T22:24:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f2212c1b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bad170 0x55b116ba8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f2212c1b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bada90 0x55b116bd0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f2212c1b740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bb8350 0x55b116bd0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f2212c1b740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f22081b0f90 con 0x55b116bb8350 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f2212c1b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f22081b2b30 con 0x55b116bad170 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f2212c1b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f22081b1660 con 0x55b116bada90 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f220e3ce640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bb8350 0x55b116bd0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f220e3ce640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bb8350 0x55b116bd0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50178/0 (socket says 172.21.15.104:50178) 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f220e3ce640 1 -- 172.21.15.104:0/377035048 learned_addr learned my addr 172.21.15.104:0/377035048 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f220dbcd640 1 --2- 172.21.15.104:0/377035048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bad170 0x55b116ba8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f220d3cc640 1 --2- 172.21.15.104:0/377035048 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bada90 0x55b116bd0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f220e3ce640 1 -- 172.21.15.104:0/377035048 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bada90 msgr2=0x55b116bd0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f220e3ce640 1 --2- 172.21.15.104:0/377035048 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bada90 0x55b116bd0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f220e3ce640 1 -- 172.21.15.104:0/377035048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bad170 msgr2=0x55b116ba8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f220e3ce640 1 --2- 172.21.15.104:0/377035048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bad170 0x55b116ba8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f220e3ce640 1 -- 172.21.15.104:0/377035048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b116bc4180 con 0x55b116bb8350 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f220e3ce640 1 --2- 172.21.15.104:0/377035048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bb8350 0x55b116bd0860 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2200012530 tx=0x7f220001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07bc6ab5a73f24b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.218+0000 7f21f37fe640 1 -- 172.21.15.104:0/377035048 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200020040 con 0x55b116bb8350 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.219+0000 7f21f37fe640 1 -- 172.21.15.104:0/377035048 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f220001c430 con 0x55b116bb8350 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.219+0000 7f21f37fe640 1 -- 172.21.15.104:0/377035048 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200019df0 con 0x55b116bb8350 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.219+0000 7f2212c1b740 1 -- 172.21.15.104:0/377035048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bb8350 msgr2=0x55b116bd0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.219+0000 7f2212c1b740 1 --2- 172.21.15.104:0/377035048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bb8350 0x55b116bd0860 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2200012530 tx=0x7f220001c9d0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.219+0000 7f2212c1b740 1 -- 172.21.15.104:0/377035048 shutdown_connections 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.219+0000 7f2212c1b740 1 --2- 172.21.15.104:0/377035048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bb8350 0x55b116bd0860 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.219+0000 7f2212c1b740 1 --2- 172.21.15.104:0/377035048 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bada90 0x55b116bd0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.219+0000 7f2212c1b740 1 --2- 172.21.15.104:0/377035048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bad170 0x55b116ba8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.219+0000 7f2212c1b740 1 -- 172.21.15.104:0/377035048 >> 172.21.15.104:0/377035048 conn(0x55b116ba9680 msgr2=0x55b116ba7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.219+0000 7f2212c1b740 1 -- 172.21.15.104:0/377035048 shutdown_connections 2024-06-09T22:24:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.219+0000 7f2212c1b740 1 -- 172.21.15.104:0/377035048 wait complete. 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.219+0000 7f2212c1b740 1 -- start start 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f2212c1b740 1 Processor -- start 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f2212c1b740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bb25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f2212c1b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bada90 0x55b116bb2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f2212c1b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bb8350 0x55b116bb3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f2212c1b740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f22081b2400 con 0x55b116bad170 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f2212c1b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f22081b1d30 con 0x55b116bada90 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f2212c1b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f22081aef50 con 0x55b116bb8350 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f2212c1b740 10 client.0 _ll_register_callbacks cb 0x7f22040dc050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f220dbcd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bb25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f220dbcd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bb25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50186/0 (socket says 172.21.15.104:50186) 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f220dbcd640 1 -- 172.21.15.104:0/2438724280 learned_addr learned my addr 172.21.15.104:0/2438724280 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f220d3cc640 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bada90 0x55b116bb2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f220e3ce640 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bb8350 0x55b116bb3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f220dbcd640 1 -- 172.21.15.104:0/2438724280 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bb8350 msgr2=0x55b116bb3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f220dbcd640 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bb8350 0x55b116bb3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f220dbcd640 1 -- 172.21.15.104:0/2438724280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bada90 msgr2=0x55b116bb2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f220dbcd640 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bada90 0x55b116bb2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.220+0000 7f220dbcd640 1 -- 172.21.15.104:0/2438724280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b116c324a0 con 0x55b116bad170 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f220dbcd640 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bb25b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f22040bc160 tx=0x7f22040f0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0d5e85f33e5059 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f21f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f21f17fa640 1 -- 172.21.15.104:0/2438724280 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204049040 con 0x55b116bad170 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f21f17fa640 1 -- 172.21.15.104:0/2438724280 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f22040bb3f0 con 0x55b116bad170 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f2212c1b740 1 -- 172.21.15.104:0/2438724280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f220001c430 con 0x55b116bad170 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f21f17fa640 1 -- 172.21.15.104:0/2438724280 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f22040c4e60 con 0x55b116bad170 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f2212c1b740 10 client.16356 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f2212c1b740 1 -- 172.21.15.104:0/2438724280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b116c324a0 con 0x55b116bad170 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f21f17fa640 1 -- 172.21.15.104:0/2438724280 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f220001c430 con 0x55b116bad170 2024-06-09T22:24:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f21f17fa640 1 -- 172.21.15.104:0/2438724280 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f2204003800 con 0x55b116bad170 2024-06-09T22:24:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f2212c1b740 10 client.16356 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f2212c1b740 10 client.16356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f2212c1b740 20 client.16356 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f2212c1b740 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b116c33c40 0x55b116c36060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f2212c1b740 4 client.16356 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:24:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f2212c1b740 1 -- 172.21.15.104:0/2438724280 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b116ba94d0 con 0x55b116c33c40 2024-06-09T22:24:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.221+0000 7f220d3cc640 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b116c33c40 0x55b116c36060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.222+0000 7f220d3cc640 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b116c33c40 0x55b116c36060 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=342914bb197c67db server_cookie=16929e010d3c8818 in_seq=0 out_seq=0 2024-06-09T22:24:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.222+0000 7f21f17fa640 10 client.16356 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.231+0000 7f21f17fa640 1 -- 172.21.15.104:0/2438724280 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b116ba94d0 con 0x55b116c33c40 2024-06-09T22:24:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.231+0000 7f21f17fa640 10 client.16356 handle_command_reply: tid=0 2024-06-09T22:24:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.231+0000 7f2212c1b740 1 client.16356 shutdown 2024-06-09T22:24:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.231+0000 7f2212c1b740 1 -- 172.21.15.104:0/2438724280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 msgr2=0x55b116bb25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.231+0000 7f2212c1b740 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bb25b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f22040bc160 tx=0x7f22040f0040 comp rx=0 tx=0).stop 2024-06-09T22:24:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f2212c1b740 1 -- 172.21.15.104:0/2438724280 shutdown_connections 2024-06-09T22:24:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f2212c1b740 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b116c33c40 0x55b116c36060 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f2212c1b740 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b116bb8350 0x55b116bb3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f2212c1b740 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b116bada90 0x55b116bb2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f2212c1b740 1 --2- 172.21.15.104:0/2438724280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b116bad170 0x55b116bb25b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f21f17fa640 0 client.16356 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f2212c1b740 1 -- 172.21.15.104:0/2438724280 >> 172.21.15.104:0/2438724280 conn(0x55b116ba9680 msgr2=0x55b116bc3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f2212c1b740 1 -- 172.21.15.104:0/2438724280 shutdown_connections 2024-06-09T22:24:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f2212c1b740 1 -- 172.21.15.104:0/2438724280 wait complete. 2024-06-09T22:24:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f2212c1b740 20 client.16356 trim_cache size 0 max 16384 2024-06-09T22:24:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:24:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:24:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:24:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f220fe31640 1 -- 172.21.15.104:0/172401623 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f21f40a2e20 msgr2=0x7f21f40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f220fe31640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f21f40a2e20 0x7f21f40a52e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f220000fc30 tx=0x7f220001a000 comp rx=0 tx=0).stop 2024-06-09T22:24:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f220fe31640 1 -- 172.21.15.104:0/172401623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220806a210 msgr2=0x7f2208198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f220fe31640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220806a210 0x7f2208198180 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2204006010 tx=0x7f22040042f0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f220fe31640 1 -- 172.21.15.104:0/172401623 shutdown_connections 2024-06-09T22:24:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f220fe31640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f21f40a2e20 0x7f21f40a52e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f220fe31640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2208107990 0x7f220819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f220fe31640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f220806abb0 0x7f22081986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f220fe31640 1 --2- 172.21.15.104:0/172401623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220806a210 0x7f2208198180 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f220fe31640 1 -- 172.21.15.104:0/172401623 >> 172.21.15.104:0/172401623 conn(0x7f22080766a0 msgr2=0x7f22080fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f220fe31640 1 -- 172.21.15.104:0/172401623 shutdown_connections 2024-06-09T22:24:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:04.232+0000 7f220fe31640 1 -- 172.21.15.104:0/172401623 wait complete. 2024-06-09T22:24:04.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 39846 3.31 MB/sec warmup 70 sec latency 2327.934 ms 2024-06-09T22:24:05.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 39846 3.26 MB/sec warmup 71 sec latency 3328.037 ms 2024-06-09T22:24:05.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:05 smithi153 ceph-mon[36599]: pgmap v96: 129 pgs: 129 active+clean; 40 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 566 KiB/s wr, 118 op/s 2024-06-09T22:24:06.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:05 smithi104 ceph-mon[32389]: pgmap v96: 129 pgs: 129 active+clean; 40 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 566 KiB/s wr, 118 op/s 2024-06-09T22:24:06.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:05 smithi194 ceph-mon[36856]: pgmap v96: 129 pgs: 129 active+clean; 40 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 566 KiB/s wr, 118 op/s 2024-06-09T22:24:06.531 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:24:06.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 41548 3.32 MB/sec warmup 72 sec latency 3716.797 ms 2024-06-09T22:24:06.791 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:24:06.928 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:06 smithi104 ceph-mon[32389]: pgmap v97: 129 pgs: 129 active+clean; 40 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 365 KiB/s wr, 80 op/s 2024-06-09T22:24:06.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:06 smithi153 ceph-mon[36599]: pgmap v97: 129 pgs: 129 active+clean; 40 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 365 KiB/s wr, 80 op/s 2024-06-09T22:24:07.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:06 smithi194 ceph-mon[36856]: pgmap v97: 129 pgs: 129 active+clean; 40 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 365 KiB/s wr, 80 op/s 2024-06-09T22:24:07.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 43259 3.43 MB/sec warmup 73 sec latency 19.262 ms 2024-06-09T22:24:08.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.611+0000 7efdc9763700 1 -- 172.21.15.104:0/1295366148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc4107400 msgr2=0x7efdc41077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:08.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.611+0000 7efdc9763700 1 --2- 172.21.15.104:0/1295366148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc4107400 0x7efdc41077e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7efdb400b0a0 tx=0x7efdb400b3b0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.611+0000 7efdc9763700 1 -- 172.21.15.104:0/1295366148 shutdown_connections 2024-06-09T22:24:08.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.611+0000 7efdc9763700 1 --2- 172.21.15.104:0/1295366148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc4107400 0x7efdc41077e0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.611+0000 7efdc9763700 1 --2- 172.21.15.104:0/1295366148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdc4101d50 0x7efdc410e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.611+0000 7efdc9763700 1 --2- 172.21.15.104:0/1295366148 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdc4101390 0x7efdc4101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.611+0000 7efdc9763700 1 -- 172.21.15.104:0/1295366148 >> 172.21.15.104:0/1295366148 conn(0x7efdc4078340 msgr2=0x7efdc4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:08.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.611+0000 7efdc9763700 1 -- 172.21.15.104:0/1295366148 shutdown_connections 2024-06-09T22:24:08.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.611+0000 7efdc9763700 1 -- 172.21.15.104:0/1295366148 wait complete. 2024-06-09T22:24:08.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.612+0000 7efdc9763700 1 Processor -- start 2024-06-09T22:24:08.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.612+0000 7efdc9763700 1 -- start start 2024-06-09T22:24:08.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc9763700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc4101390 0x7efdc419ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc9763700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdc4101d50 0x7efdc419b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc9763700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdc4107400 0x7efdc419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc9763700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efdc41116a0 con 0x7efdc4101390 2024-06-09T22:24:08.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc9763700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efdc4111520 con 0x7efdc4101d50 2024-06-09T22:24:08.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc9763700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efdc4111820 con 0x7efdc4107400 2024-06-09T22:24:08.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc4101390 0x7efdc419ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc27fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdc4101d50 0x7efdc419b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc4101390 0x7efdc419ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37986/0 (socket says 172.21.15.104:37986) 2024-06-09T22:24:08.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc2ffd700 1 -- 172.21.15.104:0/137494520 learned_addr learned my addr 172.21.15.104:0/137494520 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:08.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc37fe700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdc4107400 0x7efdc419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc27fc700 1 -- 172.21.15.104:0/137494520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdc4107400 msgr2=0x7efdc419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc27fc700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdc4107400 0x7efdc419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc27fc700 1 -- 172.21.15.104:0/137494520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc4101390 msgr2=0x7efdc419ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc27fc700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc4101390 0x7efdc419ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc27fc700 1 -- 172.21.15.104:0/137494520 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc419fe60 con 0x7efdc4101d50 2024-06-09T22:24:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.613+0000 7efdc37fe700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdc4107400 0x7efdc419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.614+0000 7efdc27fc700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdc4101d50 0x7efdc419b390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efdac00e9f0 tx=0x7efdac00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.614+0000 7efdbbfff700 1 -- 172.21.15.104:0/137494520 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac004220 con 0x7efdc4101d50 2024-06-09T22:24:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.614+0000 7efdbbfff700 1 -- 172.21.15.104:0/137494520 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac004d10 con 0x7efdc4101d50 2024-06-09T22:24:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.614+0000 7efdc9763700 1 -- 172.21.15.104:0/137494520 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdc41a0150 con 0x7efdc4101d50 2024-06-09T22:24:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.614+0000 7efdbbfff700 1 -- 172.21.15.104:0/137494520 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac005020 con 0x7efdc4101d50 2024-06-09T22:24:08.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.614+0000 7efdc9763700 1 -- 172.21.15.104:0/137494520 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdc41a79d0 con 0x7efdc4101d50 2024-06-09T22:24:08.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.617+0000 7efdbbfff700 1 -- 172.21.15.104:0/137494520 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7efdac005200 con 0x7efdc4101d50 2024-06-09T22:24:08.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.617+0000 7efdbbfff700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efdb00a2d90 0x7efdb00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.618+0000 7efdbbfff700 1 -- 172.21.15.104:0/137494520 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7efdac0cf6e0 con 0x7efdc4101d50 2024-06-09T22:24:08.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.618+0000 7efdc2ffd700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efdb00a2d90 0x7efdb00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.618+0000 7efdd3fe6b80 1 Processor -- start 2024-06-09T22:24:08.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.618+0000 7efdc2ffd700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efdb00a2d90 0x7efdb00a5250 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009fd0 tx=0x7efdb4006210 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:08.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.618+0000 7efdd3fe6b80 1 -- start start 2024-06-09T22:24:08.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdd3fe6b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d31a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdd3fe6b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31f60 0x55e984d2b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdd3fe6b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 0x55e984d2c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdd3fe6b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e984c7d810 con 0x55e984d31f60 2024-06-09T22:24:08.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdd3fe6b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e984c25720 con 0x55e984d2bf20 2024-06-09T22:24:08.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdd3fe6b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e984c797d0 con 0x55e984d31640 2024-06-09T22:24:08.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d31a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d31a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:42408/0 (socket says 172.21.15.104:42408) 2024-06-09T22:24:08.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc2ffd700 1 -- 172.21.15.104:0/3958665616 learned_addr learned my addr 172.21.15.104:0/3958665616 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc27fc700 1 --2- 172.21.15.104:0/3958665616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31f60 0x55e984d2b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc37fe700 1 --2- 172.21.15.104:0/3958665616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 0x55e984d2c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc27fc700 1 -- 172.21.15.104:0/3958665616 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 msgr2=0x55e984d31a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc27fc700 1 --2- 172.21.15.104:0/3958665616 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d31a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc27fc700 1 -- 172.21.15.104:0/3958665616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 msgr2=0x55e984d2c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc27fc700 1 --2- 172.21.15.104:0/3958665616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 0x55e984d2c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc27fc700 1 -- 172.21.15.104:0/3958665616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e984c1bf50 con 0x55e984d31f60 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc2ffd700 1 --2- 172.21.15.104:0/3958665616 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d31a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc37fe700 1 --2- 172.21.15.104:0/3958665616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 0x55e984d2c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.619+0000 7efdc27fc700 1 --2- 172.21.15.104:0/3958665616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31f60 0x55e984d2b9e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7efdac0cca80 tx=0x7efdac09b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1fb5d77b398df2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.620+0000 7efdb97fa700 1 -- 172.21.15.104:0/3958665616 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac01c040 con 0x55e984d31f60 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.620+0000 7efdb97fa700 1 -- 172.21.15.104:0/3958665616 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac09b850 con 0x55e984d31f60 2024-06-09T22:24:08.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.620+0000 7efdb97fa700 1 -- 172.21.15.104:0/3958665616 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac0cf520 con 0x55e984d31f60 2024-06-09T22:24:08.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.620+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3958665616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31f60 msgr2=0x55e984d2b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:08.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.620+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/3958665616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31f60 0x55e984d2b9e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7efdac0cca80 tx=0x7efdac09b1b0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.620+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3958665616 shutdown_connections 2024-06-09T22:24:08.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.620+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/3958665616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 0x55e984d2c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.620+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/3958665616 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d31a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.620+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/3958665616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31f60 0x55e984d2b9e0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.620+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3958665616 >> 172.21.15.104:0/3958665616 conn(0x55e984c56cc0 msgr2=0x55e984d26850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:08.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3958665616 shutdown_connections 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3958665616 wait complete. 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdd3fe6b80 1 -- start start 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdd3fe6b80 1 Processor -- start 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdd3fe6b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d2bf20 0x55e984d37d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdd3fe6b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31640 0x55e984d35050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdd3fe6b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d31f60 0x55e984d35590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdd3fe6b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e984d389d0 con 0x55e984d31640 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdd3fe6b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e984c1dbf0 con 0x55e984d31f60 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdd3fe6b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e984d38b50 con 0x55e984d2bf20 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdd3fe6b80 10 client.0 _ll_register_callbacks cb 0x55e984c17920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdc2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d2bf20 0x55e984d37d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdc27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31640 0x55e984d35050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdc2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d2bf20 0x55e984d37d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:42412/0 (socket says 172.21.15.104:42412) 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdc27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31640 0x55e984d35050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38012/0 (socket says 172.21.15.104:38012) 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.621+0000 7efdc2ffd700 1 -- 172.21.15.104:0/1323111483 learned_addr learned my addr 172.21.15.104:0/1323111483 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdc37fe700 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d31f60 0x55e984d35590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdc2ffd700 1 -- 172.21.15.104:0/1323111483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d31f60 msgr2=0x55e984d35590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdc2ffd700 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d31f60 0x55e984d35590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdc2ffd700 1 -- 172.21.15.104:0/1323111483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31640 msgr2=0x55e984d35050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdc2ffd700 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31640 0x55e984d35050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdc2ffd700 1 -- 172.21.15.104:0/1323111483 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e984bcc8d0 con 0x55e984d2bf20 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdc27fc700 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31640 0x55e984d35050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdc37fe700 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d31f60 0x55e984d35590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdc2ffd700 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d2bf20 0x55e984d37d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efdb40062a0 tx=0x7efdb40099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f209d378fe336e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdaa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:24:08.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdaa7fc700 1 -- 172.21.15.104:0/1323111483 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb4007b70 con 0x55e984d2bf20 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/1323111483 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdac09b850 con 0x55e984d2bf20 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdaa7fc700 1 -- 172.21.15.104:0/1323111483 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7efdb4007d10 con 0x55e984d2bf20 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdaa7fc700 1 -- 172.21.15.104:0/1323111483 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efdb4036580 con 0x55e984d2bf20 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdd3fe6b80 10 client.25855 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.622+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/1323111483 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e984d433d0 con 0x55e984d2bf20 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.623+0000 7efdaa7fc700 1 -- 172.21.15.104:0/1323111483 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac09b850 con 0x55e984d2bf20 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.623+0000 7efdaa7fc700 1 -- 172.21.15.104:0/1323111483 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7efdb4011070 con 0x55e984d2bf20 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.623+0000 7efdd3fe6b80 10 client.25855 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.623+0000 7efdd3fe6b80 10 client.25855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.623+0000 7efdd3fe6b80 20 client.25855 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.623+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e984d45960 0x55e984d47d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.623+0000 7efdd3fe6b80 4 client.25855 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:08.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.623+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/1323111483 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e984c8ad10 con 0x55e984d45960 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.623+0000 7efdc27fc700 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e984d45960 0x55e984d47d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdc27fc700 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e984d45960 0x55e984d47d80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c6a70624e47532f server_cookie=f891cbd875788730 in_seq=0 out_seq=0 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdaa7fc700 10 client.25855 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdaa7fc700 1 -- 172.21.15.104:0/1323111483 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55e984c8ad10 con 0x55e984d45960 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdaa7fc700 10 client.25855 handle_command_reply: tid=0 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdd3fe6b80 1 client.25855 shutdown 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/1323111483 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d2bf20 msgr2=0x55e984d37d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d2bf20 0x55e984d37d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efdb40062a0 tx=0x7efdb40099e0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/1323111483 shutdown_connections 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e984d45960 0x55e984d47d80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d31f60 0x55e984d35590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d2bf20 0x55e984d37d50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/1323111483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31640 0x55e984d35050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdaa7fc700 0 client.25855 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:08.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/1323111483 >> 172.21.15.104:0/1323111483 conn(0x55e984c56cc0 msgr2=0x55e984d2a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:08.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.624+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/1323111483 shutdown_connections 2024-06-09T22:24:08.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.625+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/1323111483 wait complete. 2024-06-09T22:24:08.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.625+0000 7efdd3fe6b80 20 client.25855 trim_cache size 0 max 16384 2024-06-09T22:24:08.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.647+0000 7efdd3fe6b80 1 Processor -- start 2024-06-09T22:24:08.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.647+0000 7efdd3fe6b80 1 -- start start 2024-06-09T22:24:08.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.647+0000 7efdd3fe6b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d2bf20 0x55e984d2b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.647+0000 7efdd3fe6b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d41c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.647+0000 7efdd3fe6b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d31f60 0x55e984d1fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.647+0000 7efdd3fe6b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efdc41b07e0 con 0x55e984d2bf20 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.647+0000 7efdd3fe6b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efdc41b2380 con 0x55e984d31f60 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.647+0000 7efdd3fe6b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efdc41b0eb0 con 0x55e984d31640 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.647+0000 7efdc27fc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d41c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.647+0000 7efdc2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d2bf20 0x55e984d2b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.647+0000 7efdc27fc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d41c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:42420/0 (socket says 172.21.15.104:42420) 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.648+0000 7efdc37fe700 1 --2- 172.21.15.104:0/2315166468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d31f60 0x55e984d1fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.647+0000 7efdc27fc700 1 -- 172.21.15.104:0/2315166468 learned_addr learned my addr 172.21.15.104:0/2315166468 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.648+0000 7efdc27fc700 1 -- 172.21.15.104:0/2315166468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d31f60 msgr2=0x55e984d1fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.648+0000 7efdc27fc700 1 --2- 172.21.15.104:0/2315166468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d31f60 0x55e984d1fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.648+0000 7efdc27fc700 1 -- 172.21.15.104:0/2315166468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d2bf20 msgr2=0x55e984d2b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.648+0000 7efdc27fc700 1 --2- 172.21.15.104:0/2315166468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d2bf20 0x55e984d2b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.648+0000 7efdc27fc700 1 -- 172.21.15.104:0/2315166468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e984d483e0 con 0x55e984d31640 2024-06-09T22:24:08.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.648+0000 7efdc37fe700 1 --2- 172.21.15.104:0/2315166468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d31f60 0x55e984d1fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:08.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.648+0000 7efdc2ffd700 1 --2- 172.21.15.104:0/2315166468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d2bf20 0x55e984d2b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:08.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.648+0000 7efdc27fc700 1 --2- 172.21.15.104:0/2315166468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d41c70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efdac099b00 tx=0x7efdac097c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16e7c4d4b8b83db8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:08.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.648+0000 7efdb8ff9700 1 -- 172.21.15.104:0/2315166468 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac066070 con 0x55e984d31640 2024-06-09T22:24:08.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.648+0000 7efdb8ff9700 1 -- 172.21.15.104:0/2315166468 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac09b850 con 0x55e984d31640 2024-06-09T22:24:08.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.648+0000 7efdb8ff9700 1 -- 172.21.15.104:0/2315166468 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac01c040 con 0x55e984d31640 2024-06-09T22:24:08.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.649+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/2315166468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 msgr2=0x55e984d41c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:08.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.649+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/2315166468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d41c70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efdac099b00 tx=0x7efdac097c20 comp rx=0 tx=0).stop 2024-06-09T22:24:08.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.649+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/2315166468 shutdown_connections 2024-06-09T22:24:08.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.649+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/2315166468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d2bf20 0x55e984d2b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.649+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/2315166468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d31f60 0x55e984d1fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.649+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/2315166468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d41c70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.649+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/2315166468 >> 172.21.15.104:0/2315166468 conn(0x55e984c56cc0 msgr2=0x55e984d2f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:08.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.649+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/2315166468 shutdown_connections 2024-06-09T22:24:08.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.649+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/2315166468 wait complete. 2024-06-09T22:24:08.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdd3fe6b80 1 -- start start 2024-06-09T22:24:08.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdd3fe6b80 1 Processor -- start 2024-06-09T22:24:08.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdd3fe6b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 0x55e984d4d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdd3fe6b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d4d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdd3fe6b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31f60 0x55e984d4dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdd3fe6b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efdc41b1c50 con 0x55e984d31f60 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdd3fe6b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efdc41b1580 con 0x55e984d2bf20 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdd3fe6b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efdc41ae760 con 0x55e984d31640 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdd3fe6b80 10 client.0 _ll_register_callbacks cb 0x55e984c17920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdc2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 0x55e984d4d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdc37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31f60 0x55e984d4dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdc2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 0x55e984d4d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:36506/0 (socket says 172.21.15.104:36506) 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdc2ffd700 1 -- 172.21.15.104:0/3123826679 learned_addr learned my addr 172.21.15.104:0/3123826679 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdc27fc700 1 --2- 172.21.15.104:0/3123826679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d4d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdc2ffd700 1 -- 172.21.15.104:0/3123826679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 msgr2=0x55e984d4d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdc2ffd700 1 --2- 172.21.15.104:0/3123826679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d4d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdc2ffd700 1 -- 172.21.15.104:0/3123826679 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31f60 msgr2=0x55e984d4dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:08.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdc2ffd700 1 --2- 172.21.15.104:0/3123826679 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31f60 0x55e984d4dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.650+0000 7efdc2ffd700 1 -- 172.21.15.104:0/3123826679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e984daf690 con 0x55e984d2bf20 2024-06-09T22:24:08.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdc2ffd700 1 --2- 172.21.15.104:0/3123826679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 0x55e984d4d1b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efdb40052d0 tx=0x7efdb4005e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24a1d1d7e52897aa server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:08.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdaa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:24:08.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdaa7fc700 1 -- 172.21.15.104:0/3123826679 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb40056e0 con 0x55e984d2bf20 2024-06-09T22:24:08.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdaa7fc700 1 -- 172.21.15.104:0/3123826679 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7efdb4005880 con 0x55e984d2bf20 2024-06-09T22:24:08.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3123826679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdac09b850 con 0x55e984d2bf20 2024-06-09T22:24:08.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdaa7fc700 1 -- 172.21.15.104:0/3123826679 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efdb40077c0 con 0x55e984d2bf20 2024-06-09T22:24:08.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdd3fe6b80 10 client.25772 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:08.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3123826679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e984db0830 con 0x55e984d2bf20 2024-06-09T22:24:08.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdaa7fc700 1 -- 172.21.15.104:0/3123826679 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac09b850 con 0x55e984d2bf20 2024-06-09T22:24:08.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdaa7fc700 1 -- 172.21.15.104:0/3123826679 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7efdb4011040 con 0x55e984d2bf20 2024-06-09T22:24:08.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdd3fe6b80 10 client.25772 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:08.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdd3fe6b80 10 client.25772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:08.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdd3fe6b80 20 client.25772 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:08.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/3123826679 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e984db0f00 0x55e984db3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:08.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdd3fe6b80 4 client.25772 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:24:08.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.651+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3123826679 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e984c85290 con 0x55e984db0f00 2024-06-09T22:24:08.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdc27fc700 1 --2- 172.21.15.104:0/3123826679 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e984db0f00 0x55e984db3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:08.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdc27fc700 1 --2- 172.21.15.104:0/3123826679 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e984db0f00 0x55e984db3320 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5edb22058dc24fa2 server_cookie=9520c7f0e1cc832d in_seq=0 out_seq=0 2024-06-09T22:24:08.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdaa7fc700 10 client.25772 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:08.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdaa7fc700 1 -- 172.21.15.104:0/3123826679 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e984c85290 con 0x55e984db0f00 2024-06-09T22:24:08.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdaa7fc700 10 client.25772 handle_command_reply: tid=0 2024-06-09T22:24:08.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdd3fe6b80 1 client.25772 shutdown 2024-06-09T22:24:08.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3123826679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 msgr2=0x55e984d4d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:08.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/3123826679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 0x55e984d4d1b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efdb40052d0 tx=0x7efdb4005e90 comp rx=0 tx=0).stop 2024-06-09T22:24:08.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3123826679 shutdown_connections 2024-06-09T22:24:08.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/3123826679 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e984db0f00 0x55e984db3320 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/3123826679 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e984d31f60 0x55e984d4dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/3123826679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e984d2bf20 0x55e984d4d1b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdaa7fc700 0 client.25772 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:08.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdd3fe6b80 1 --2- 172.21.15.104:0/3123826679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e984d31640 0x55e984d4d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3123826679 >> 172.21.15.104:0/3123826679 conn(0x55e984c56cc0 msgr2=0x55e984d1fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:08.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3123826679 shutdown_connections 2024-06-09T22:24:08.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.652+0000 7efdd3fe6b80 1 -- 172.21.15.104:0/3123826679 wait complete. 2024-06-09T22:24:08.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdd3fe6b80 20 client.25772 trim_cache size 0 max 16384 2024-06-09T22:24:08.659 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:24:08.659 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "no active scrubs running", 2024-06-09T22:24:08.659 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": {} 2024-06-09T22:24:08.659 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:24:08.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdc9763700 1 -- 172.21.15.104:0/137494520 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efdb00a2d90 msgr2=0x7efdb00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:08.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdc9763700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efdb00a2d90 0x7efdb00a5250 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009fd0 tx=0x7efdb4006210 comp rx=0 tx=0).stop 2024-06-09T22:24:08.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdc9763700 1 -- 172.21.15.104:0/137494520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdc4101d50 msgr2=0x7efdc419b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:08.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdc9763700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdc4101d50 0x7efdc419b390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efdac00e9f0 tx=0x7efdac00ed00 comp rx=0 tx=0).stop 2024-06-09T22:24:08.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdc9763700 1 -- 172.21.15.104:0/137494520 shutdown_connections 2024-06-09T22:24:08.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdc9763700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efdb00a2d90 0x7efdb00a5250 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdc9763700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efdc4101390 0x7efdc419ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdc9763700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdc4107400 0x7efdc419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdc9763700 1 --2- 172.21.15.104:0/137494520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdc4101d50 0x7efdc419b390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:08.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdc9763700 1 -- 172.21.15.104:0/137494520 >> 172.21.15.104:0/137494520 conn(0x7efdc4078340 msgr2=0x7efdc406a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:08.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdc9763700 1 -- 172.21.15.104:0/137494520 shutdown_connections 2024-06-09T22:24:08.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:08.653+0000 7efdc9763700 1 -- 172.21.15.104:0/137494520 wait complete. 2024-06-09T22:24:08.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 46098 3.62 MB/sec warmup 74 sec latency 14.238 ms 2024-06-09T22:24:08.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:08 smithi153 ceph-mon[36599]: pgmap v98: 129 pgs: 129 active+clean; 40 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 365 KiB/s wr, 80 op/s 2024-06-09T22:24:09.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:08 smithi104 ceph-mon[32389]: pgmap v98: 129 pgs: 129 active+clean; 40 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 365 KiB/s wr, 80 op/s 2024-06-09T22:24:09.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:08 smithi194 ceph-mon[36856]: pgmap v98: 129 pgs: 129 active+clean; 40 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 365 KiB/s wr, 80 op/s 2024-06-09T22:24:09.574 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-09T22:24:09.575 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:24:09.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 46938 3.60 MB/sec warmup 75 sec latency 230.123 ms 2024-06-09T22:24:09.838 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:24:10.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 49077 3.71 MB/sec warmup 76 sec latency 531.902 ms 2024-06-09T22:24:10.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:10 smithi153 ceph-mon[36599]: pgmap v99: 129 pgs: 129 active+clean; 44 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 144 op/s 2024-06-09T22:24:11.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:10 smithi104 ceph-mon[32389]: pgmap v99: 129 pgs: 129 active+clean; 44 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 144 op/s 2024-06-09T22:24:11.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:10 smithi194 ceph-mon[36856]: pgmap v99: 129 pgs: 129 active+clean; 44 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 144 op/s 2024-06-09T22:24:11.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 50355 3.75 MB/sec warmup 77 sec latency 169.909 ms 2024-06-09T22:24:11.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.736+0000 7f5d5b89f700 1 -- 172.21.15.104:0/3941035531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d54101780 msgr2=0x7f5d54112d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:11.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.736+0000 7f5d5b89f700 1 --2- 172.21.15.104:0/3941035531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d54101780 0x7f5d54112d20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5d44009990 tx=0x7f5d44009ca0 comp rx=0 tx=0).stop 2024-06-09T22:24:11.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.736+0000 7f5d5b89f700 1 -- 172.21.15.104:0/3941035531 shutdown_connections 2024-06-09T22:24:11.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.736+0000 7f5d5b89f700 1 --2- 172.21.15.104:0/3941035531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d54113260 0x7f5d54115650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:11.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.736+0000 7f5d5b89f700 1 --2- 172.21.15.104:0/3941035531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d54100dd0 0x7f5d541011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:11.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.736+0000 7f5d5b89f700 1 --2- 172.21.15.104:0/3941035531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d54101780 0x7f5d54112d20 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:11.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.736+0000 7f5d5b89f700 1 -- 172.21.15.104:0/3941035531 >> 172.21.15.104:0/3941035531 conn(0x7f5d540766d0 msgr2=0x7f5d54076ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:11.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.736+0000 7f5d5b89f700 1 -- 172.21.15.104:0/3941035531 shutdown_connections 2024-06-09T22:24:11.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.736+0000 7f5d5b89f700 1 -- 172.21.15.104:0/3941035531 wait complete. 2024-06-09T22:24:11.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.736+0000 7f5d5b89f700 1 Processor -- start 2024-06-09T22:24:11.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.736+0000 7f5d5b89f700 1 -- start start 2024-06-09T22:24:11.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d5b89f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d54100dd0 0x7f5d5419f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:11.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d5b89f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d54101780 0x7f5d5419f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:11.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d5b89f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d54113260 0x7f5d541a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:11.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d5b89f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5d54118e10 con 0x7f5d54113260 2024-06-09T22:24:11.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d5b89f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5d54118c90 con 0x7f5d54100dd0 2024-06-09T22:24:11.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d5b89f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5d54118f90 con 0x7f5d54101780 2024-06-09T22:24:11.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d59e3c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d54113260 0x7f5d541a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:11.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d59e3c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d54113260 0x7f5d541a3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38040/0 (socket says 172.21.15.104:38040) 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d59e3c700 1 -- 172.21.15.104:0/345546385 learned_addr learned my addr 172.21.15.104:0/345546385 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d5963b700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d54100dd0 0x7f5d5419f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d58e3a700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d54101780 0x7f5d5419f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d59e3c700 1 -- 172.21.15.104:0/345546385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d54101780 msgr2=0x7f5d5419f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d59e3c700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d54101780 0x7f5d5419f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d59e3c700 1 -- 172.21.15.104:0/345546385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d54100dd0 msgr2=0x7f5d5419f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d59e3c700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d54100dd0 0x7f5d5419f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d59e3c700 1 -- 172.21.15.104:0/345546385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d541a4150 con 0x7f5d54113260 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d59e3c700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d54113260 0x7f5d541a3a10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c007d40 tx=0x7f5d4c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d4a7fc700 1 -- 172.21.15.104:0/345546385 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c016020 con 0x7f5d54113260 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d5b89f700 1 -- 172.21.15.104:0/345546385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d541a4440 con 0x7f5d54113260 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d4a7fc700 1 -- 172.21.15.104:0/345546385 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d4c00ac90 con 0x7f5d54113260 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d4a7fc700 1 -- 172.21.15.104:0/345546385 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c0115e0 con 0x7f5d54113260 2024-06-09T22:24:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.737+0000 7f5d5b89f700 1 -- 172.21.15.104:0/345546385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d541abcc0 con 0x7f5d54113260 2024-06-09T22:24:11.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.738+0000 7f5d5b89f700 1 -- 172.21.15.104:0/345546385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d54069180 con 0x7f5d54113260 2024-06-09T22:24:11.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.740+0000 7f5d4a7fc700 1 -- 172.21.15.104:0/345546385 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5d4c019020 con 0x7f5d54113260 2024-06-09T22:24:11.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.740+0000 7f5d4a7fc700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5d400a2d90 0x7f5d400a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:11.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.740+0000 7f5d5963b700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5d400a2d90 0x7f5d400a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:11.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.740+0000 7f5d4a7fc700 1 -- 172.21.15.104:0/345546385 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f5d4c0cf920 con 0x7f5d54113260 2024-06-09T22:24:11.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.741+0000 7f5d5963b700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5d400a2d90 0x7f5d400a5250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5d50005950 tx=0x7f5d5000b410 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:11.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.744+0000 7f5d4a7fc700 1 -- 172.21.15.104:0/345546385 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5d4c020020 con 0x7f5d54113260 2024-06-09T22:24:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.995+0000 7f5d5b89f700 1 -- 172.21.15.104:0/345546385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d541abfb0 con 0x7f5d54113260 2024-06-09T22:24:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.996+0000 7f5d4a7fc700 1 -- 172.21.15.104:0/345546385 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11956 (secure 0 0 0) 0x7f5d4c086090 con 0x7f5d54113260 2024-06-09T22:24:11.996 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:24:11.996 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:22:59.692018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24419,24403,24442,24460,24448]},"id":1}]} 2024-06-09T22:24:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.998+0000 7f5d5b89f700 1 -- 172.21.15.104:0/345546385 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5d400a2d90 msgr2=0x7f5d400a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.998+0000 7f5d5b89f700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5d400a2d90 0x7f5d400a5250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5d50005950 tx=0x7f5d5000b410 comp rx=0 tx=0).stop 2024-06-09T22:24:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.998+0000 7f5d5b89f700 1 -- 172.21.15.104:0/345546385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d54113260 msgr2=0x7f5d541a3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.998+0000 7f5d5b89f700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d54113260 0x7f5d541a3a10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c007d40 tx=0x7f5d4c00a340 comp rx=0 tx=0).stop 2024-06-09T22:24:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.998+0000 7f5d5b89f700 1 -- 172.21.15.104:0/345546385 shutdown_connections 2024-06-09T22:24:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.998+0000 7f5d5b89f700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5d400a2d90 0x7f5d400a5250 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.998+0000 7f5d5b89f700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d54113260 0x7f5d541a3a10 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.998+0000 7f5d5b89f700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d54100dd0 0x7f5d5419f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.998+0000 7f5d5b89f700 1 --2- 172.21.15.104:0/345546385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d54101780 0x7f5d5419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.998+0000 7f5d5b89f700 1 -- 172.21.15.104:0/345546385 >> 172.21.15.104:0/345546385 conn(0x7f5d540766d0 msgr2=0x7f5d54110d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.999+0000 7f5d5b89f700 1 -- 172.21.15.104:0/345546385 shutdown_connections 2024-06-09T22:24:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:11.999+0000 7f5d5b89f700 1 -- 172.21.15.104:0/345546385 wait complete. 2024-06-09T22:24:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 30 2024-06-09T22:24:12.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 50355 3.70 MB/sec warmup 78 sec latency 1169.951 ms 2024-06-09T22:24:12.823 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-09T22:24:12.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:12 smithi153 ceph-mon[36599]: pgmap v100: 129 pgs: 129 active+clean; 44 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 135 op/s 2024-06-09T22:24:12.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:12 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/345546385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:24:13.006 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:12 smithi104 ceph-mon[32389]: pgmap v100: 129 pgs: 129 active+clean; 44 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 135 op/s 2024-06-09T22:24:13.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:12 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/345546385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:24:13.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:12 smithi194 ceph-mon[36856]: pgmap v100: 129 pgs: 129 active+clean; 44 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 135 op/s 2024-06-09T22:24:13.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:12 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/345546385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:24:13.062 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:24:13.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 50355 3.66 MB/sec warmup 79 sec latency 2169.993 ms 2024-06-09T22:24:13.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:13 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:24:14.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:13 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:24:14.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:13 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:24:14.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 50355 3.61 MB/sec warmup 80 sec latency 3170.018 ms 2024-06-09T22:24:14.757 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:14 smithi104 ceph-mon[32389]: pgmap v101: 129 pgs: 129 active+clean; 50 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 189 op/s 2024-06-09T22:24:14.757 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:14 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:24:14.757 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:14 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:24:14.757 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:14 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:24:14.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:14 smithi153 ceph-mon[36599]: pgmap v101: 129 pgs: 129 active+clean; 50 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 189 op/s 2024-06-09T22:24:14.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:14 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:24:14.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:14 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:24:14.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:14 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:24:15.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.031+0000 7fbf215c4700 1 -- 172.21.15.104:0/206687888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c10a6d0 msgr2=0x7fbf1c10aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:15.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.031+0000 7fbf215c4700 1 --2- 172.21.15.104:0/206687888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c10a6d0 0x7fbf1c10aab0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c009990 tx=0x7fbf0c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.031+0000 7fbf215c4700 1 -- 172.21.15.104:0/206687888 shutdown_connections 2024-06-09T22:24:15.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.031+0000 7fbf215c4700 1 --2- 172.21.15.104:0/206687888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c10a6d0 0x7fbf1c10aab0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.031+0000 7fbf215c4700 1 --2- 172.21.15.104:0/206687888 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf1c105090 0x7fbf1c1115a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.031+0000 7fbf215c4700 1 --2- 172.21.15.104:0/206687888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf1c1046d0 0x7fbf1c104b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.031+0000 7fbf215c4700 1 -- 172.21.15.104:0/206687888 >> 172.21.15.104:0/206687888 conn(0x7fbf1c1001c0 msgr2=0x7fbf1c1025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:15.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.031+0000 7fbf215c4700 1 -- 172.21.15.104:0/206687888 shutdown_connections 2024-06-09T22:24:15.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.031+0000 7fbf215c4700 1 -- 172.21.15.104:0/206687888 wait complete. 2024-06-09T22:24:15.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.032+0000 7fbf215c4700 1 Processor -- start 2024-06-09T22:24:15.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.032+0000 7fbf215c4700 1 -- start start 2024-06-09T22:24:15.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.032+0000 7fbf215c4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c1046d0 0x7fbf1c19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.032+0000 7fbf215c4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf1c105090 0x7fbf1c19b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf215c4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf1c10a6d0 0x7fbf1c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf215c4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c114a20 con 0x7fbf1c1046d0 2024-06-09T22:24:15.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf215c4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1148a0 con 0x7fbf1c105090 2024-06-09T22:24:15.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf215c4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c114ba0 con 0x7fbf1c10a6d0 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf1affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c1046d0 0x7fbf1c19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf1affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c1046d0 0x7fbf1c19ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38068/0 (socket says 172.21.15.104:38068) 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf1affd700 1 -- 172.21.15.104:0/919175257 learned_addr learned my addr 172.21.15.104:0/919175257 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf1c105090 0x7fbf1c19b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf1c10a6d0 0x7fbf1c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf1affd700 1 -- 172.21.15.104:0/919175257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf1c10a6d0 msgr2=0x7fbf1c19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf1affd700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf1c10a6d0 0x7fbf1c19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf1affd700 1 -- 172.21.15.104:0/919175257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf1c105090 msgr2=0x7fbf1c19b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf1affd700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf1c105090 0x7fbf1c19b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf1affd700 1 -- 172.21.15.104:0/919175257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1c19fe20 con 0x7fbf1c1046d0 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf1c10a6d0 0x7fbf1c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf1affd700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c1046d0 0x7fbf1c19ae10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c000c00 tx=0x7fbf0c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf03fff700 1 -- 172.21.15.104:0/919175257 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c027070 con 0x7fbf1c1046d0 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf215c4700 1 -- 172.21.15.104:0/919175257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf1c1a00b0 con 0x7fbf1c1046d0 2024-06-09T22:24:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf03fff700 1 -- 172.21.15.104:0/919175257 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf0c003780 con 0x7fbf1c1046d0 2024-06-09T22:24:15.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.033+0000 7fbf03fff700 1 -- 172.21.15.104:0/919175257 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c022dd0 con 0x7fbf1c1046d0 2024-06-09T22:24:15.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.034+0000 7fbf215c4700 1 -- 172.21.15.104:0/919175257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf1c1a7990 con 0x7fbf1c1046d0 2024-06-09T22:24:15.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.036+0000 7fbf03fff700 1 -- 172.21.15.104:0/919175257 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fbf0c02a070 con 0x7fbf1c1046d0 2024-06-09T22:24:15.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.036+0000 7fbf03fff700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbf040a2d90 0x7fbf040a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.036+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbf040a2d90 0x7fbf040a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.036+0000 7fbf03fff700 1 -- 172.21.15.104:0/919175257 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fbf0c0d04a0 con 0x7fbf1c1046d0 2024-06-09T22:24:15.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.037+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbf040a2d90 0x7fbf040a5250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c109cf0 tx=0x7fbf100073d0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:15.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf2be47b80 1 Processor -- start 2024-06-09T22:24:15.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf2be47b80 1 -- start start 2024-06-09T22:24:15.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf2be47b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947640 0x55fd74947a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf2be47b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd749419e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf2be47b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd74942340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf2be47b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fd74893810 con 0x55fd74947f60 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf2be47b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fd7483b720 con 0x55fd74947640 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf2be47b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fd7488f7d0 con 0x55fd74941f20 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf1a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd749419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf1a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd749419e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38074/0 (socket says 172.21.15.104:38074) 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf1a7fc700 1 -- 172.21.15.104:0/2120908715 learned_addr learned my addr 172.21.15.104:0/2120908715 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf1affd700 1 --2- 172.21.15.104:0/2120908715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947640 0x55fd74947a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.038+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/2120908715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd74942340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.039+0000 7fbf1a7fc700 1 -- 172.21.15.104:0/2120908715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 msgr2=0x55fd74942340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.039+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/2120908715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd74942340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.039+0000 7fbf1a7fc700 1 -- 172.21.15.104:0/2120908715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947640 msgr2=0x55fd74947a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.039+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/2120908715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947640 0x55fd74947a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.039+0000 7fbf1a7fc700 1 -- 172.21.15.104:0/2120908715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd74831f50 con 0x55fd74947f60 2024-06-09T22:24:15.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.039+0000 7fbf1affd700 1 --2- 172.21.15.104:0/2120908715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947640 0x55fd74947a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:15.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.039+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/2120908715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd749419e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fbf10010e40 tx=0x7fbf10013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906a7609b92f5aec server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:15.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.039+0000 7fbf017fa700 1 -- 172.21.15.104:0/2120908715 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf10024070 con 0x55fd74947f60 2024-06-09T22:24:15.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.039+0000 7fbf017fa700 1 -- 172.21.15.104:0/2120908715 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf10015070 con 0x55fd74947f60 2024-06-09T22:24:15.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.039+0000 7fbf017fa700 1 -- 172.21.15.104:0/2120908715 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf100118e0 con 0x55fd74947f60 2024-06-09T22:24:15.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.040+0000 7fbf2be47b80 1 -- 172.21.15.104:0/2120908715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 msgr2=0x55fd749419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:15.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.040+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/2120908715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd749419e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fbf10010e40 tx=0x7fbf10013000 comp rx=0 tx=0).stop 2024-06-09T22:24:15.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.040+0000 7fbf2be47b80 1 -- 172.21.15.104:0/2120908715 shutdown_connections 2024-06-09T22:24:15.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.040+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/2120908715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd74942340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.040+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/2120908715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947640 0x55fd74947a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.040+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/2120908715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd749419e0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.040+0000 7fbf2be47b80 1 -- 172.21.15.104:0/2120908715 >> 172.21.15.104:0/2120908715 conn(0x55fd7486ccc0 msgr2=0x55fd7493c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:15.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.040+0000 7fbf2be47b80 1 -- 172.21.15.104:0/2120908715 shutdown_connections 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.040+0000 7fbf2be47b80 1 -- 172.21.15.104:0/2120908715 wait complete. 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.040+0000 7fbf2be47b80 1 -- start start 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.040+0000 7fbf2be47b80 1 Processor -- start 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf2be47b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd7494dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf2be47b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947640 0x55fd7494afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf2be47b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947f60 0x55fd7494b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf2be47b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fd7494e9d0 con 0x55fd74947640 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf2be47b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fd74833bf0 con 0x55fd74947f60 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf2be47b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fd7494eb50 con 0x55fd74941f20 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf2be47b80 10 client.0 _ll_register_callbacks cb 0x55fd7482d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf1a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947640 0x55fd7494afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf1a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947640 0x55fd7494afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38088/0 (socket says 172.21.15.104:38088) 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf1affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd7494dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf1a7fc700 1 -- 172.21.15.104:0/1405724870 learned_addr learned my addr 172.21.15.104:0/1405724870 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947f60 0x55fd7494b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf1a7fc700 1 -- 172.21.15.104:0/1405724870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 msgr2=0x55fd7494dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd7494dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf1a7fc700 1 -- 172.21.15.104:0/1405724870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947f60 msgr2=0x55fd7494b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947f60 0x55fd7494b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf1a7fc700 1 -- 172.21.15.104:0/1405724870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd747e28d0 con 0x55fd74947640 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf1affd700 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd7494dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947640 0x55fd7494afd0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fbf100079c0 tx=0x7fbf100038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba051cee634e06a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbefeff5700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbefeff5700 1 -- 172.21.15.104:0/1405724870 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1002e020 con 0x55fd74947640 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbefeff5700 1 -- 172.21.15.104:0/1405724870 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fbf10011580 con 0x55fd74947640 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.041+0000 7fbf2be47b80 1 -- 172.21.15.104:0/1405724870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf10015070 con 0x55fd74947640 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.042+0000 7fbefeff5700 1 -- 172.21.15.104:0/1405724870 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbf10039050 con 0x55fd74947640 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.042+0000 7fbf2be47b80 10 client.16392 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:15.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.042+0000 7fbf2be47b80 1 -- 172.21.15.104:0/1405724870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fd749592d0 con 0x55fd74947640 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.042+0000 7fbefeff5700 1 -- 172.21.15.104:0/1405724870 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf10015070 con 0x55fd74947640 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.042+0000 7fbefeff5700 1 -- 172.21.15.104:0/1405724870 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fbf1001f650 con 0x55fd74947640 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.042+0000 7fbf2be47b80 10 client.16392 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.042+0000 7fbf2be47b80 10 client.16392 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.042+0000 7fbf2be47b80 20 client.16392 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.042+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55fd7495b8b0 0x55fd7495dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.042+0000 7fbf2be47b80 4 client.16392 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.042+0000 7fbf2be47b80 1 -- 172.21.15.104:0/1405724870 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd748a0d10 con 0x55fd7495b8b0 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.042+0000 7fbf1affd700 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55fd7495b8b0 0x55fd7495dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.043+0000 7fbefeff5700 10 client.16392 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.043+0000 7fbf1affd700 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55fd7495b8b0 0x55fd7495dca0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fe18d00e25a76678 server_cookie=401c978729cf2a62 in_seq=0 out_seq=0 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.043+0000 7fbefeff5700 1 -- 172.21.15.104:0/1405724870 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55fd748a0d10 con 0x55fd7495b8b0 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.043+0000 7fbefeff5700 10 client.16392 handle_command_reply: tid=0 2024-06-09T22:24:15.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.043+0000 7fbf2be47b80 1 client.16392 shutdown 2024-06-09T22:24:15.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.044+0000 7fbf2be47b80 1 -- 172.21.15.104:0/1405724870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947640 msgr2=0x55fd7494afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:15.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.044+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947640 0x55fd7494afd0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fbf100079c0 tx=0x7fbf100038b0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.044+0000 7fbf2be47b80 1 -- 172.21.15.104:0/1405724870 shutdown_connections 2024-06-09T22:24:15.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.044+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55fd7495b8b0 0x55fd7495dca0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.044+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947f60 0x55fd7494b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.044+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd7494dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.044+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/1405724870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947640 0x55fd7494afd0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.044+0000 7fbf2be47b80 1 -- 172.21.15.104:0/1405724870 >> 172.21.15.104:0/1405724870 conn(0x55fd7486ccc0 msgr2=0x55fd7493c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:15.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.044+0000 7fbefeff5700 0 client.16392 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:24:15.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.044+0000 7fbf2be47b80 1 -- 172.21.15.104:0/1405724870 shutdown_connections 2024-06-09T22:24:15.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.044+0000 7fbf2be47b80 1 -- 172.21.15.104:0/1405724870 wait complete. 2024-06-09T22:24:15.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.044+0000 7fbf2be47b80 20 client.16392 trim_cache size 0 max 16384 2024-06-09T22:24:15.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf2be47b80 1 Processor -- start 2024-06-09T22:24:15.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf2be47b80 1 -- start start 2024-06-09T22:24:15.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf2be47b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd749419e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf2be47b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947640 0x55fd7494af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf2be47b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd7494b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf2be47b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1b0620 con 0x55fd74947f60 2024-06-09T22:24:15.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf2be47b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1b21c0 con 0x55fd74947640 2024-06-09T22:24:15.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf2be47b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1b0cf0 con 0x55fd74941f20 2024-06-09T22:24:15.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf1b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd7494b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf1b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd7494b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38090/0 (socket says 172.21.15.104:38090) 2024-06-09T22:24:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf1affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd749419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/2104193318 learned_addr learned my addr 172.21.15.104:0/2104193318 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/2104193318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 msgr2=0x55fd749419e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/2104193318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947640 0x55fd7494af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/2104193318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd749419e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/2104193318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947640 msgr2=0x55fd7494af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/2104193318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947640 0x55fd7494af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/2104193318 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd7494ba10 con 0x55fd74947f60 2024-06-09T22:24:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.059+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/2104193318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd7494b4d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009f20 tx=0x7fbf140077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5bafb3a96b59ba server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.060+0000 7fbf00ff9700 1 -- 172.21.15.104:0/2104193318 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1401b070 con 0x55fd74947f60 2024-06-09T22:24:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.060+0000 7fbf00ff9700 1 -- 172.21.15.104:0/2104193318 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf14017920 con 0x55fd74947f60 2024-06-09T22:24:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.060+0000 7fbf00ff9700 1 -- 172.21.15.104:0/2104193318 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf14016ab0 con 0x55fd74947f60 2024-06-09T22:24:15.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:14 smithi194 ceph-mon[36856]: pgmap v101: 129 pgs: 129 active+clean; 50 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 189 op/s 2024-06-09T22:24:15.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:14 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:24:15.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:14 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:24:15.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:14 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:24:15.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.060+0000 7fbf2be47b80 1 -- 172.21.15.104:0/2104193318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 msgr2=0x55fd7494b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:15.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.060+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/2104193318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd7494b4d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009f20 tx=0x7fbf140077e0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.060+0000 7fbf2be47b80 1 -- 172.21.15.104:0/2104193318 shutdown_connections 2024-06-09T22:24:15.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.060+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/2104193318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd7494b4d0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.060+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/2104193318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74941f20 0x55fd749419e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.060+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/2104193318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74947640 0x55fd7494af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.060+0000 7fbf2be47b80 1 -- 172.21.15.104:0/2104193318 >> 172.21.15.104:0/2104193318 conn(0x55fd7486ccc0 msgr2=0x55fd749498e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.060+0000 7fbf2be47b80 1 -- 172.21.15.104:0/2104193318 shutdown_connections 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.060+0000 7fbf2be47b80 1 -- 172.21.15.104:0/2104193318 wait complete. 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf2be47b80 1 -- start start 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf2be47b80 1 Processor -- start 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf2be47b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74941f20 0x55fd7495e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf2be47b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74947640 0x55fd7495e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf2be47b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd749c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf2be47b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1b1a90 con 0x55fd74947f60 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf2be47b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1b13c0 con 0x55fd74941f20 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf2be47b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1ae5a0 con 0x55fd74947640 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf2be47b80 10 client.0 _ll_register_callbacks cb 0x55fd7482d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf1b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd749c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf1b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd749c41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38096/0 (socket says 172.21.15.104:38096) 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/970604180 learned_addr learned my addr 172.21.15.104:0/970604180 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74947640 0x55fd7495e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf1affd700 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74941f20 0x55fd7495e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/970604180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74947640 msgr2=0x55fd7495e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74947640 0x55fd7495e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/970604180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74941f20 msgr2=0x55fd7495e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74941f20 0x55fd7495e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/970604180 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd749c47f0 con 0x55fd74947f60 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.061+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74947640 0x55fd7495e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd749c41e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009320 tx=0x7fbf14017670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a27a6f5d9e7882f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbefeff5700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbefeff5700 1 -- 172.21.15.104:0/970604180 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf14028060 con 0x55fd74947f60 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbefeff5700 1 -- 172.21.15.104:0/970604180 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fbf14016650 con 0x55fd74947f60 2024-06-09T22:24:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbf2be47b80 1 -- 172.21.15.104:0/970604180 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf14017920 con 0x55fd74947f60 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbefeff5700 1 -- 172.21.15.104:0/970604180 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbf1402f050 con 0x55fd74947f60 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbf2be47b80 10 client.16398 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbf2be47b80 1 -- 172.21.15.104:0/970604180 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fd749c5ad0 con 0x55fd74947f60 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbefeff5700 1 -- 172.21.15.104:0/970604180 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf14017920 con 0x55fd74947f60 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbefeff5700 1 -- 172.21.15.104:0/970604180 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fbf14005630 con 0x55fd74947f60 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbf2be47b80 10 client.16398 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbf2be47b80 10 client.16398 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbf2be47b80 20 client.16398 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55fd749c60f0 0x55fd749c8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbf2be47b80 4 client.16398 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.062+0000 7fbf2be47b80 1 -- 172.21.15.104:0/970604180 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd7489b290 con 0x55fd749c60f0 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf1affd700 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55fd749c60f0 0x55fd749c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf1affd700 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55fd749c60f0 0x55fd749c8510 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=36498df654e8e7bd server_cookie=b16cf036886c1670 in_seq=0 out_seq=0 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbefeff5700 10 client.16398 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:24:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbefeff5700 1 -- 172.21.15.104:0/970604180 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd7489b290 con 0x55fd749c60f0 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbefeff5700 10 client.16398 handle_command_reply: tid=0 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf2be47b80 1 client.16398 shutdown 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf2be47b80 1 -- 172.21.15.104:0/970604180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 msgr2=0x55fd749c41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd749c41e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009320 tx=0x7fbf14017670 comp rx=0 tx=0).stop 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf2be47b80 1 -- 172.21.15.104:0/970604180 shutdown_connections 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd74947f60 0x55fd749c41e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55fd749c60f0 0x55fd749c8510 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd74941f20 0x55fd7495e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf2be47b80 1 --2- 172.21.15.104:0/970604180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd74947640 0x55fd7495e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf2be47b80 1 -- 172.21.15.104:0/970604180 >> 172.21.15.104:0/970604180 conn(0x55fd7486ccc0 msgr2=0x55fd74949870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbefeff5700 0 client.16398 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf2be47b80 1 -- 172.21.15.104:0/970604180 shutdown_connections 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.063+0000 7fbf2be47b80 1 -- 172.21.15.104:0/970604180 wait complete. 2024-06-09T22:24:15.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf2be47b80 20 client.16398 trim_cache size 0 max 16384 2024-06-09T22:24:15.070 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:24:15.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf215c4700 1 -- 172.21.15.104:0/919175257 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbf040a2d90 msgr2=0x7fbf040a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:15.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf215c4700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbf040a2d90 0x7fbf040a5250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c109cf0 tx=0x7fbf100073d0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf215c4700 1 -- 172.21.15.104:0/919175257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c1046d0 msgr2=0x7fbf1c19ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:15.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf215c4700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c1046d0 0x7fbf1c19ae10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c000c00 tx=0x7fbf0c0047b0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf215c4700 1 -- 172.21.15.104:0/919175257 shutdown_connections 2024-06-09T22:24:15.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf215c4700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbf040a2d90 0x7fbf040a5250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf215c4700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c1046d0 0x7fbf1c19ae10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf215c4700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf1c10a6d0 0x7fbf1c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf215c4700 1 --2- 172.21.15.104:0/919175257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf1c105090 0x7fbf1c19b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:15.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf215c4700 1 -- 172.21.15.104:0/919175257 >> 172.21.15.104:0/919175257 conn(0x7fbf1c1001c0 msgr2=0x7fbf1c100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:15.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf215c4700 1 -- 172.21.15.104:0/919175257 shutdown_connections 2024-06-09T22:24:15.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:15.064+0000 7fbf215c4700 1 -- 172.21.15.104:0/919175257 wait complete. 2024-06-09T22:24:15.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 50355 3.57 MB/sec warmup 81 sec latency 4170.044 ms 2024-06-09T22:24:15.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-09T22:24:16.130 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:24:16.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 50411 3.52 MB/sec warmup 82 sec latency 4573.825 ms 2024-06-09T22:24:16.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:16 smithi153 ceph-mon[36599]: pgmap v102: 129 pgs: 129 active+clean; 50 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-06-09T22:24:17.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:16 smithi104 ceph-mon[32389]: pgmap v102: 129 pgs: 129 active+clean; 50 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-06-09T22:24:17.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:16 smithi194 ceph-mon[36856]: pgmap v102: 129 pgs: 129 active+clean; 50 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-06-09T22:24:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.322+0000 7f8bb25ce640 1 -- 172.21.15.104:0/1300335393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bac101980 msgr2=0x7f8bac101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.322+0000 7f8bb25ce640 1 --2- 172.21.15.104:0/1300335393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bac101980 0x7f8bac101e00 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009930 tx=0x7f8b9402a420 comp rx=0 tx=0).stop 2024-06-09T22:24:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.322+0000 7f8bb25ce640 1 -- 172.21.15.104:0/1300335393 shutdown_connections 2024-06-09T22:24:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.322+0000 7f8bb25ce640 1 --2- 172.21.15.104:0/1300335393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8bac102340 0x7f8bac10e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.322+0000 7f8bb25ce640 1 --2- 172.21.15.104:0/1300335393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bac101980 0x7f8bac101e00 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.322+0000 7f8bb25ce640 1 --2- 172.21.15.104:0/1300335393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac107990 0x7f8bac107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.322+0000 7f8bb25ce640 1 -- 172.21.15.104:0/1300335393 >> 172.21.15.104:0/1300335393 conn(0x7f8bac078e30 msgr2=0x7f8bac079240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.322+0000 7f8bb25ce640 1 -- 172.21.15.104:0/1300335393 shutdown_connections 2024-06-09T22:24:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.322+0000 7f8bb25ce640 1 -- 172.21.15.104:0/1300335393 wait complete. 2024-06-09T22:24:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.323+0000 7f8bb25ce640 1 Processor -- start 2024-06-09T22:24:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.323+0000 7f8bb25ce640 1 -- start start 2024-06-09T22:24:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.323+0000 7f8bb25ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac101980 0x7f8bac1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.323+0000 7f8bb25ce640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8bac102340 0x7f8bac1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.323+0000 7f8bb25ce640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bac107990 0x7f8bac19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.323+0000 7f8bb25ce640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac111d70 con 0x7f8bac107990 2024-06-09T22:24:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.323+0000 7f8bb25ce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac111bf0 con 0x7f8bac101980 2024-06-09T22:24:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.323+0000 7f8bb25ce640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac111ef0 con 0x7f8bac102340 2024-06-09T22:24:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.323+0000 7f8bb0b6b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bac107990 0x7f8bac19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.323+0000 7f8bb0b6b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bac107990 0x7f8bac19ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46464/0 (socket says 172.21.15.104:46464) 2024-06-09T22:24:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.323+0000 7f8bb0b6b640 1 -- 172.21.15.104:0/2809996761 learned_addr learned my addr 172.21.15.104:0/2809996761 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.323+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8bac102340 0x7f8bac1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8babfff640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac101980 0x7f8bac1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8bb0b6b640 1 -- 172.21.15.104:0/2809996761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8bac102340 msgr2=0x7f8bac1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8bb0b6b640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8bac102340 0x7f8bac1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8bb0b6b640 1 -- 172.21.15.104:0/2809996761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac101980 msgr2=0x7f8bac1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8bb0b6b640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac101980 0x7f8bac1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8bb0b6b640 1 -- 172.21.15.104:0/2809996761 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bac19d150 con 0x7f8bac107990 2024-06-09T22:24:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8bac102340 0x7f8bac1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8bb0b6b640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bac107990 0x7f8bac19ca20 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400da40 tx=0x7f8ba400df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8ba9ffb640 1 -- 172.21.15.104:0/2809996761 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4017070 con 0x7f8bac107990 2024-06-09T22:24:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8ba9ffb640 1 -- 172.21.15.104:0/2809996761 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ba4003e50 con 0x7f8bac107990 2024-06-09T22:24:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8bb25ce640 1 -- 172.21.15.104:0/2809996761 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bac19d440 con 0x7f8bac107990 2024-06-09T22:24:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8ba9ffb640 1 -- 172.21.15.104:0/2809996761 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4010460 con 0x7f8bac107990 2024-06-09T22:24:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.324+0000 7f8bb25ce640 1 -- 172.21.15.104:0/2809996761 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bac1a4cd0 con 0x7f8bac107990 2024-06-09T22:24:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.326+0000 7f8ba9ffb640 1 -- 172.21.15.104:0/2809996761 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f8ba4003970 con 0x7f8bac107990 2024-06-09T22:24:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8bb53b8740 1 Processor -- start 2024-06-09T22:24:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8bb53b8740 1 -- start start 2024-06-09T22:24:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8bb53b8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 0x55a7dd72f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8bb53b8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd72fa90 0x55a7dd739e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8bb53b8740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 0x55a7dd73c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8ba9ffb640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8b980a2da0 0x7f8b980a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8bb53b8740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a7dd6934f0 con 0x55a7dd73a350 2024-06-09T22:24:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8bb53b8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a7dd631870 con 0x55a7dd72f170 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8bb53b8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a7dd68f4b0 con 0x55a7dd72fa90 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 0x55a7dd73c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8bb0b6b640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8b980a2da0 0x7f8b980a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 0x55a7dd73c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46478/0 (socket says 172.21.15.104:46478) 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8ba3fff640 1 -- 172.21.15.104:0/1127764650 learned_addr learned my addr 172.21.15.104:0/1127764650 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/1127764650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 0x55a7dd72f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8babfff640 1 --2- 172.21.15.104:0/1127764650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd72fa90 0x55a7dd739e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8ba9ffb640 1 -- 172.21.15.104:0/2809996761 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8ba40cf4b0 con 0x7f8bac107990 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8ba3fff640 1 -- 172.21.15.104:0/1127764650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd72fa90 msgr2=0x55a7dd739e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/1127764650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd72fa90 0x55a7dd739e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8ba3fff640 1 -- 172.21.15.104:0/1127764650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 msgr2=0x55a7dd72f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.327+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/1127764650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 0x55a7dd72f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.328+0000 7f8ba3fff640 1 -- 172.21.15.104:0/1127764650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7dd64d9b0 con 0x55a7dd73a350 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.328+0000 7f8bb0b6b640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8b980a2da0 0x7f8b980a5260 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8ba409b5f0 tx=0x7f8ba4086040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.328+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/1127764650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 0x55a7dd73c740 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009930 tx=0x7f8b940056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad461b15762293a4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.328+0000 7f8ba27fc640 1 -- 172.21.15.104:0/1127764650 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94035040 con 0x55a7dd73a350 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.328+0000 7f8ba27fc640 1 -- 172.21.15.104:0/1127764650 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b940035f0 con 0x55a7dd73a350 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.328+0000 7f8ba27fc640 1 -- 172.21.15.104:0/1127764650 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9403d6e0 con 0x55a7dd73a350 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.328+0000 7f8bb53b8740 1 -- 172.21.15.104:0/1127764650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 msgr2=0x55a7dd73c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.328+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/1127764650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 0x55a7dd73c740 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009930 tx=0x7f8b940056d0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.328+0000 7f8bb53b8740 1 -- 172.21.15.104:0/1127764650 shutdown_connections 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.328+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/1127764650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 0x55a7dd73c740 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.328+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/1127764650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd72fa90 0x55a7dd739e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.328+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/1127764650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 0x55a7dd72f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8bb53b8740 1 -- 172.21.15.104:0/1127764650 >> 172.21.15.104:0/1127764650 conn(0x55a7dd72b680 msgr2=0x55a7dd72ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8bb53b8740 1 -- 172.21.15.104:0/1127764650 shutdown_connections 2024-06-09T22:24:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8bb53b8740 1 -- 172.21.15.104:0/1127764650 wait complete. 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8bb53b8740 1 -- start start 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8bb53b8740 1 Processor -- start 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8bb53b8740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72f170 0x55a7dd737290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8bb53b8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72fa90 0x55a7dd7377d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8bb53b8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd73a350 0x55a7dd737d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8bb53b8740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a7dd73cf20 con 0x55a7dd72f170 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8bb53b8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a7dd61cae0 con 0x55a7dd72fa90 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8bb53b8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a7dd73d0a0 con 0x55a7dd73a350 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8bb53b8740 10 client.0 _ll_register_callbacks cb 0x55a7dd6559b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72f170 0x55a7dd737290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72f170 0x55a7dd737290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46490/0 (socket says 172.21.15.104:46490) 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.329+0000 7f8ba3fff640 1 -- 172.21.15.104:0/407564365 learned_addr learned my addr 172.21.15.104:0/407564365 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd73a350 0x55a7dd737d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8babfff640 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72fa90 0x55a7dd7377d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8ba3fff640 1 -- 172.21.15.104:0/407564365 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd73a350 msgr2=0x55a7dd737d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd73a350 0x55a7dd737d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8ba3fff640 1 -- 172.21.15.104:0/407564365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72fa90 msgr2=0x55a7dd7377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72fa90 0x55a7dd7377d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8ba3fff640 1 -- 172.21.15.104:0/407564365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7dd3a7210 con 0x55a7dd72f170 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8babfff640 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72fa90 0x55a7dd7377d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72f170 0x55a7dd737290 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8b94006010 tx=0x7f8b94005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46280466c20a2a2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8b8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8b8b7fe640 1 -- 172.21.15.104:0/407564365 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94000e60 con 0x55a7dd72f170 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8b8b7fe640 1 -- 172.21.15.104:0/407564365 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8b94004210 con 0x55a7dd72f170 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8bb53b8740 1 -- 172.21.15.104:0/407564365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b940035f0 con 0x55a7dd72f170 2024-06-09T22:24:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8b8b7fe640 1 -- 172.21.15.104:0/407564365 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8b94035050 con 0x55a7dd72f170 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8bb53b8740 10 client.16410 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8bb53b8740 1 -- 172.21.15.104:0/407564365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a7dd473620 con 0x55a7dd72f170 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8b8b7fe640 1 -- 172.21.15.104:0/407564365 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b940035f0 con 0x55a7dd72f170 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.330+0000 7f8b8b7fe640 1 -- 172.21.15.104:0/407564365 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8b9403a040 con 0x55a7dd72f170 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.331+0000 7f8bb53b8740 10 client.16410 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.331+0000 7f8bb53b8740 10 client.16410 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.331+0000 7f8bb53b8740 20 client.16410 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.331+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a7dd74a320 0x55a7dd74c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.331+0000 7f8bb53b8740 4 client.16410 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.331+0000 7f8bb53b8740 1 -- 172.21.15.104:0/407564365 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7dd6a6f10 con 0x55a7dd74a320 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.331+0000 7f8babfff640 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a7dd74a320 0x55a7dd74c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.331+0000 7f8b8b7fe640 10 client.16410 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.331+0000 7f8babfff640 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a7dd74a320 0x55a7dd74c710 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c53c107c3fcf92a5 server_cookie=7f97d87395a9b8a8 in_seq=0 out_seq=0 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8b8b7fe640 1 -- 172.21.15.104:0/407564365 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a7dd6a6f10 con 0x55a7dd74a320 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8b8b7fe640 10 client.16410 handle_command_reply: tid=0 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8bb53b8740 1 client.16410 shutdown 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8bb53b8740 1 -- 172.21.15.104:0/407564365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72f170 msgr2=0x55a7dd737290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72f170 0x55a7dd737290 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8b94006010 tx=0x7f8b94005a20 comp rx=0 tx=0).stop 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8bb53b8740 1 -- 172.21.15.104:0/407564365 shutdown_connections 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a7dd74a320 0x55a7dd74c710 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd73a350 0x55a7dd737d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72fa90 0x55a7dd7377d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/407564365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72f170 0x55a7dd737290 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8b8b7fe640 0 client.16410 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8bb53b8740 1 -- 172.21.15.104:0/407564365 >> 172.21.15.104:0/407564365 conn(0x55a7dd72b680 msgr2=0x55a7dd7313b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8bb53b8740 1 -- 172.21.15.104:0/407564365 shutdown_connections 2024-06-09T22:24:17.342 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8bb53b8740 1 -- 172.21.15.104:0/407564365 wait complete. 2024-06-09T22:24:17.342 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.332+0000 7f8bb53b8740 20 client.16410 trim_cache size 0 max 16384 2024-06-09T22:24:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.347+0000 7f8bb53b8740 1 Processor -- start 2024-06-09T22:24:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.347+0000 7f8bb53b8740 1 -- start start 2024-06-09T22:24:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8bb53b8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 0x55a7dd731b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8bb53b8740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72fa90 0x55a7dd745a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8bb53b8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd73a350 0x55a7dd74f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8bb53b8740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1aecb0 con 0x55a7dd72fa90 2024-06-09T22:24:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8bb53b8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1b0850 con 0x55a7dd72f170 2024-06-09T22:24:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8bb53b8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1af380 con 0x55a7dd73a350 2024-06-09T22:24:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8babfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72fa90 0x55a7dd745a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8babfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72fa90 0x55a7dd745a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46502/0 (socket says 172.21.15.104:46502) 2024-06-09T22:24:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8babfff640 1 -- 172.21.15.104:0/3908051911 learned_addr learned my addr 172.21.15.104:0/3908051911 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8babfff640 1 -- 172.21.15.104:0/3908051911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd73a350 msgr2=0x55a7dd74f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8babfff640 1 --2- 172.21.15.104:0/3908051911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd73a350 0x55a7dd74f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8babfff640 1 -- 172.21.15.104:0/3908051911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 msgr2=0x55a7dd731b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8babfff640 1 --2- 172.21.15.104:0/3908051911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 0x55a7dd731b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8babfff640 1 -- 172.21.15.104:0/3908051911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7dd7357e0 con 0x55a7dd72fa90 2024-06-09T22:24:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8babfff640 1 --2- 172.21.15.104:0/3908051911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72fa90 0x55a7dd745a30 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x55a7dd73d150 tx=0x7f8b9c00ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7730eba27f65b2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8ba1ffb640 1 -- 172.21.15.104:0/3908051911 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c021030 con 0x55a7dd72fa90 2024-06-09T22:24:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8ba1ffb640 1 -- 172.21.15.104:0/3908051911 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b9c005020 con 0x55a7dd72fa90 2024-06-09T22:24:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.348+0000 7f8ba1ffb640 1 -- 172.21.15.104:0/3908051911 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c004dd0 con 0x55a7dd72fa90 2024-06-09T22:24:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.349+0000 7f8bb53b8740 1 -- 172.21.15.104:0/3908051911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72fa90 msgr2=0x55a7dd745a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.349+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/3908051911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72fa90 0x55a7dd745a30 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x55a7dd73d150 tx=0x7f8b9c00ce40 comp rx=0 tx=0).stop 2024-06-09T22:24:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.349+0000 7f8bb53b8740 1 -- 172.21.15.104:0/3908051911 shutdown_connections 2024-06-09T22:24:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.349+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/3908051911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd73a350 0x55a7dd74f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.349+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/3908051911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd72fa90 0x55a7dd745a30 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.349+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/3908051911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 0x55a7dd731b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.349+0000 7f8bb53b8740 1 -- 172.21.15.104:0/3908051911 >> 172.21.15.104:0/3908051911 conn(0x55a7dd72b680 msgr2=0x55a7dd745320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.349+0000 7f8bb53b8740 1 -- 172.21.15.104:0/3908051911 shutdown_connections 2024-06-09T22:24:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.349+0000 7f8bb53b8740 1 -- 172.21.15.104:0/3908051911 wait complete. 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.349+0000 7f8bb53b8740 1 -- start start 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.349+0000 7f8bb53b8740 1 Processor -- start 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8bb53b8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 0x55a7dd74fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8bb53b8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd72fa90 0x55a7dd74ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8bb53b8740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 0x55a7dd74cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8bb53b8740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1b0120 con 0x55a7dd73a350 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8bb53b8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1afa50 con 0x55a7dd72f170 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8bb53b8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1acc70 con 0x55a7dd72fa90 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8bb53b8740 10 client.0 _ll_register_callbacks cb 0x7f8b94032de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 0x55a7dd74cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 0x55a7dd74cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46510/0 (socket says 172.21.15.104:46510) 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8ba3fff640 1 -- 172.21.15.104:0/641681982 learned_addr learned my addr 172.21.15.104:0/641681982 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 0x55a7dd74fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8ba3fff640 1 -- 172.21.15.104:0/641681982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd72fa90 msgr2=0x55a7dd74ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8babfff640 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd72fa90 0x55a7dd74ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd72fa90 0x55a7dd74ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8ba3fff640 1 -- 172.21.15.104:0/641681982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 msgr2=0x55a7dd74fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 0x55a7dd74fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8ba3fff640 1 -- 172.21.15.104:0/641681982 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7dd74d710 con 0x55a7dd73a350 2024-06-09T22:24:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8ba3fff640 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 0x55a7dd74cee0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8b94031bf0 tx=0x7f8b94005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd24aa406fee0f7 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8b8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8b8b7fe640 1 -- 172.21.15.104:0/641681982 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94005d50 con 0x55a7dd73a350 2024-06-09T22:24:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8b8b7fe640 1 -- 172.21.15.104:0/641681982 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8b9404c990 con 0x55a7dd73a350 2024-06-09T22:24:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8bb53b8740 1 -- 172.21.15.104:0/641681982 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b9c005020 con 0x55a7dd73a350 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8b8b7fe640 1 -- 172.21.15.104:0/641681982 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8b940035f0 con 0x55a7dd73a350 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8bb53b8740 10 client.16416 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.350+0000 7f8bb53b8740 1 -- 172.21.15.104:0/641681982 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a7dd7b8c80 con 0x55a7dd73a350 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.351+0000 7f8b8b7fe640 1 -- 172.21.15.104:0/641681982 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b9c005020 con 0x55a7dd73a350 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.351+0000 7f8b8b7fe640 1 -- 172.21.15.104:0/641681982 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8b9403d960 con 0x55a7dd73a350 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.351+0000 7f8bb53b8740 10 client.16416 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.351+0000 7f8bb53b8740 10 client.16416 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.351+0000 7f8bb53b8740 20 client.16416 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.351+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a7dd7b9180 0x55a7dd7bb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.351+0000 7f8bb53b8740 4 client.16416 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.351+0000 7f8bb53b8740 1 -- 172.21.15.104:0/641681982 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a7dd6a0c20 con 0x55a7dd7b9180 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.351+0000 7f8babfff640 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a7dd7b9180 0x55a7dd7bb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.351+0000 7f8b8b7fe640 10 client.16416 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.351+0000 7f8babfff640 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a7dd7b9180 0x55a7dd7bb5a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d30345d3abb6b88 server_cookie=fa4494286fa7600e in_seq=0 out_seq=0 2024-06-09T22:24:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8b8b7fe640 1 -- 172.21.15.104:0/641681982 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a7dd6a0c20 con 0x55a7dd7b9180 2024-06-09T22:24:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8b8b7fe640 10 client.16416 handle_command_reply: tid=0 2024-06-09T22:24:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8bb53b8740 1 client.16416 shutdown 2024-06-09T22:24:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8bb53b8740 1 -- 172.21.15.104:0/641681982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 msgr2=0x55a7dd74cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 0x55a7dd74cee0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8b94031bf0 tx=0x7f8b94005970 comp rx=0 tx=0).stop 2024-06-09T22:24:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8bb53b8740 1 -- 172.21.15.104:0/641681982 shutdown_connections 2024-06-09T22:24:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a7dd7b9180 0x55a7dd7bb5a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a7dd73a350 0x55a7dd74cee0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7dd72fa90 0x55a7dd74ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8b8b7fe640 0 client.16416 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8bb53b8740 1 --2- 172.21.15.104:0/641681982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7dd72f170 0x55a7dd74fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8bb53b8740 1 -- 172.21.15.104:0/641681982 >> 172.21.15.104:0/641681982 conn(0x55a7dd72b680 msgr2=0x55a7dd737ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8bb53b8740 1 -- 172.21.15.104:0/641681982 shutdown_connections 2024-06-09T22:24:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.359+0000 7f8bb53b8740 1 -- 172.21.15.104:0/641681982 wait complete. 2024-06-09T22:24:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb53b8740 20 client.16416 trim_cache size 0 max 16384 2024-06-09T22:24:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:24:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:24:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:24:17.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb25ce640 1 -- 172.21.15.104:0/2809996761 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8b980a2da0 msgr2=0x7f8b980a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb25ce640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8b980a2da0 0x7f8b980a5260 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8ba409b5f0 tx=0x7f8ba4086040 comp rx=0 tx=0).stop 2024-06-09T22:24:17.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb25ce640 1 -- 172.21.15.104:0/2809996761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bac107990 msgr2=0x7f8bac19ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb25ce640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bac107990 0x7f8bac19ca20 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400da40 tx=0x7f8ba400df00 comp rx=0 tx=0).stop 2024-06-09T22:24:17.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb25ce640 1 -- 172.21.15.104:0/2809996761 shutdown_connections 2024-06-09T22:24:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb25ce640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8b980a2da0 0x7f8b980a5260 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb25ce640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bac107990 0x7f8bac19ca20 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb25ce640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8bac102340 0x7f8bac1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb25ce640 1 --2- 172.21.15.104:0/2809996761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac101980 0x7f8bac1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb25ce640 1 -- 172.21.15.104:0/2809996761 >> 172.21.15.104:0/2809996761 conn(0x7f8bac078e30 msgr2=0x7f8bac0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb25ce640 1 -- 172.21.15.104:0/2809996761 shutdown_connections 2024-06-09T22:24:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:17.360+0000 7f8bb25ce640 1 -- 172.21.15.104:0/2809996761 wait complete. 2024-06-09T22:24:17.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 50411 3.48 MB/sec warmup 83 sec latency 1544.060 ms 2024-06-09T22:24:17.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.959+0000 7f9f209aa700 1 -- 172.21.15.104:0/950800799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f1c1084f0 msgr2=0x7f9f1c1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.959+0000 7f9f209aa700 1 --2- 172.21.15.104:0/950800799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f1c1084f0 0x7f9f1c1088d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009990 tx=0x7f9f0c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.959+0000 7f9f209aa700 1 -- 172.21.15.104:0/950800799 shutdown_connections 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.959+0000 7f9f209aa700 1 --2- 172.21.15.104:0/950800799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f1c1084f0 0x7f9f1c1088d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.959+0000 7f9f209aa700 1 --2- 172.21.15.104:0/950800799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f1c102f10 0x7f9f1c10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.959+0000 7f9f209aa700 1 --2- 172.21.15.104:0/950800799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9f1c102550 0x7f9f1c1029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.959+0000 7f9f209aa700 1 -- 172.21.15.104:0/950800799 >> 172.21.15.104:0/950800799 conn(0x7f9f1c079650 msgr2=0x7f9f1c079a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.959+0000 7f9f209aa700 1 -- 172.21.15.104:0/950800799 shutdown_connections 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.959+0000 7f9f209aa700 1 -- 172.21.15.104:0/950800799 wait complete. 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f209aa700 1 Processor -- start 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f209aa700 1 -- start start 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f209aa700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9f1c102550 0x7f9f1c19e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f209aa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f1c102f10 0x7f9f1c19ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f209aa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f1c1084f0 0x7f9f1c198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f209aa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c112770 con 0x7f9f1c1084f0 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f209aa700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1125f0 con 0x7f9f1c102f10 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f209aa700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1128f0 con 0x7f9f1c102550 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f1ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f1c1084f0 0x7f9f1c198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f1ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f1c1084f0 0x7f9f1c198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46532/0 (socket says 172.21.15.104:46532) 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/582488032 learned_addr learned my addr 172.21.15.104:0/582488032 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9f1c102550 0x7f9f1c19e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f19d9b700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f1c102f10 0x7f9f1c19ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/582488032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9f1c102550 msgr2=0x7f9f1c19e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9f1c102550 0x7f9f1c19e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/582488032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f1c102f10 msgr2=0x7f9f1c19ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.960+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f1c102f10 0x7f9f1c19ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.961+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/582488032 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1c199190 con 0x7f9f1c1084f0 2024-06-09T22:24:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.961+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f1c1084f0 0x7f9f1c198a30 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9f14007ee0 tx=0x7f9f1400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.961+0000 7f9f0b7fe700 1 -- 172.21.15.104:0/582488032 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f14004bb0 con 0x7f9f1c1084f0 2024-06-09T22:24:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.961+0000 7f9f209aa700 1 -- 172.21.15.104:0/582488032 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f1c199480 con 0x7f9f1c1084f0 2024-06-09T22:24:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.961+0000 7f9f0b7fe700 1 -- 172.21.15.104:0/582488032 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f14004d50 con 0x7f9f1c1084f0 2024-06-09T22:24:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.961+0000 7f9f0b7fe700 1 -- 172.21.15.104:0/582488032 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f14005960 con 0x7f9f1c1084f0 2024-06-09T22:24:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.961+0000 7f9f209aa700 1 -- 172.21.15.104:0/582488032 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f1c1a56f0 con 0x7f9f1c1084f0 2024-06-09T22:24:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.964+0000 7f9f0b7fe700 1 -- 172.21.15.104:0/582488032 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f9f1401a020 con 0x7f9f1c1084f0 2024-06-09T22:24:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.964+0000 7f9f0b7fe700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9f040a2d90 0x7f9f040a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.965+0000 7f9f0b7fe700 1 -- 172.21.15.104:0/582488032 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f9f140cee90 con 0x7f9f1c1084f0 2024-06-09T22:24:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.965+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9f040a2d90 0x7f9f040a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.965+0000 7f9f2b22db80 1 Processor -- start 2024-06-09T22:24:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f2b22db80 1 -- start start 2024-06-09T22:24:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9f040a2d90 0x7f9f040a5250 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00b5c0 tx=0x7f9f0c005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f2b22db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a172166f60 0x55a172167340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f2b22db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172167880 0x55a172161260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f2b22db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a1721617a0 0x55a172161bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f2b22db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a172053f60 con 0x55a1721617a0 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f2b22db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a1720b2850 con 0x55a172166f60 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f2b22db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a17201a460 con 0x55a172167880 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f1ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a1721617a0 0x55a172161bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f1ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a1721617a0 0x55a172161bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46542/0 (socket says 172.21.15.104:46542) 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/2058407307 learned_addr learned my addr 172.21.15.104:0/2058407307 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/2058407307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a172166f60 0x55a172167340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f19d9b700 1 --2- 172.21.15.104:0/2058407307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172167880 0x55a172161260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/2058407307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172167880 msgr2=0x55a172161260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/2058407307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172167880 0x55a172161260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/2058407307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a172166f60 msgr2=0x55a172167340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/2058407307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a172166f60 0x55a172167340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/2058407307 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1720018d0 con 0x55a1721617a0 2024-06-09T22:24:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/2058407307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a172166f60 0x55a172167340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.966+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/2058407307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a1721617a0 0x55a172161bc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f9f140974a0 tx=0x7f9f1409ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94657a3ad315eb5 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.967+0000 7f9f08ff9700 1 -- 172.21.15.104:0/2058407307 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1409b040 con 0x55a1721617a0 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.967+0000 7f9f08ff9700 1 -- 172.21.15.104:0/2058407307 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f1409f3f0 con 0x55a1721617a0 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.967+0000 7f9f08ff9700 1 -- 172.21.15.104:0/2058407307 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f140976b0 con 0x55a1721617a0 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.967+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2058407307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a1721617a0 msgr2=0x55a172161bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.967+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/2058407307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a1721617a0 0x55a172161bc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f9f140974a0 tx=0x7f9f1409ac00 comp rx=0 tx=0).stop 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.967+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2058407307 shutdown_connections 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.967+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/2058407307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a1721617a0 0x55a172161bc0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.967+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/2058407307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a172166f60 0x55a172167340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.967+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/2058407307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172167880 0x55a172161260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.967+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2058407307 >> 172.21.15.104:0/2058407307 conn(0x55a17208e080 msgr2=0x55a17215c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2058407307 shutdown_connections 2024-06-09T22:24:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2058407307 wait complete. 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f2b22db80 1 -- start start 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f2b22db80 1 Processor -- start 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f2b22db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1721617a0 0x55a17216d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f2b22db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172166f60 0x55a17216a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f2b22db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 0x55a17216aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f2b22db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a172052fc0 con 0x55a172167880 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f2b22db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a1720ae810 con 0x55a1721617a0 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f2b22db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a17216e260 con 0x55a172166f60 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f2b22db80 10 client.0 _ll_register_callbacks cb 0x55a1720485b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f1ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 0x55a17216aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f1ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 0x55a17216aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46546/0 (socket says 172.21.15.104:46546) 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/2368002590 learned_addr learned my addr 172.21.15.104:0/2368002590 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.968+0000 7f9f19d9b700 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172166f60 0x55a17216a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1721617a0 0x55a17216d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/2368002590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172166f60 msgr2=0x55a17216a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172166f60 0x55a17216a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/2368002590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1721617a0 msgr2=0x55a17216d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1721617a0 0x55a17216d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/2368002590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a172078750 con 0x55a172167880 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1721617a0 0x55a17216d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 0x55a17216aeb0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009510 tx=0x7f9f140cc260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9f231cdefb60ca server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f01ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f01ffb700 1 -- 172.21.15.104:0/2368002590 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f140cc4f0 con 0x55a172167880 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f01ffb700 1 -- 172.21.15.104:0/2368002590 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f9f14022250 con 0x55a172167880 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2368002590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f1409f3f0 con 0x55a172167880 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f01ffb700 1 -- 172.21.15.104:0/2368002590 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9f14079e60 con 0x55a172167880 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f2b22db80 10 client.16428 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2368002590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a1721789e0 con 0x55a172167880 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.969+0000 7f9f01ffb700 1 -- 172.21.15.104:0/2368002590 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f1409f3f0 con 0x55a172167880 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.970+0000 7f9f01ffb700 1 -- 172.21.15.104:0/2368002590 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9f140a3820 con 0x55a172167880 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.970+0000 7f9f2b22db80 10 client.16428 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.970+0000 7f9f2b22db80 10 client.16428 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.970+0000 7f9f2b22db80 20 client.16428 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.970+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55a17217aeb0 0x55a17217d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.970+0000 7f9f2b22db80 4 client.16428 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.970+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2368002590 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1720bfd50 con 0x55a17217aeb0 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.970+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55a17217aeb0 0x55a17217d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.970+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55a17217aeb0 0x55a17217d2a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7f79dd0eb55a5e57 server_cookie=1daf7cd4619b3307 in_seq=0 out_seq=0 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.970+0000 7f9f01ffb700 10 client.16428 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.971+0000 7f9f01ffb700 1 -- 172.21.15.104:0/2368002590 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a1720bfd50 con 0x55a17217aeb0 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.971+0000 7f9f01ffb700 10 client.16428 handle_command_reply: tid=0 2024-06-09T22:24:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.971+0000 7f9f2b22db80 1 client.16428 shutdown 2024-06-09T22:24:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.971+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2368002590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 msgr2=0x55a17216aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.971+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 0x55a17216aeb0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009510 tx=0x7f9f140cc260 comp rx=0 tx=0).stop 2024-06-09T22:24:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.971+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2368002590 shutdown_connections 2024-06-09T22:24:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.971+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55a17217aeb0 0x55a17217d2a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.971+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 0x55a17216aeb0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.971+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1721617a0 0x55a17216d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.971+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/2368002590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172166f60 0x55a17216a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.971+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2368002590 >> 172.21.15.104:0/2368002590 conn(0x55a17208e080 msgr2=0x55a17215b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.971+0000 7f9f01ffb700 0 client.16428 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:24:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.972+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2368002590 shutdown_connections 2024-06-09T22:24:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.972+0000 7f9f2b22db80 1 -- 172.21.15.104:0/2368002590 wait complete. 2024-06-09T22:24:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.972+0000 7f9f2b22db80 20 client.16428 trim_cache size 0 max 16384 2024-06-09T22:24:17.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.986+0000 7f9f2b22db80 1 Processor -- start 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f2b22db80 1 -- start start 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f2b22db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1721617a0 0x55a172161260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f2b22db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172166f60 0x55a172168370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f2b22db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 0x55a17215be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f2b22db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1ae480 con 0x55a172167880 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f2b22db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1b0020 con 0x55a1721617a0 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f2b22db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1aeb50 con 0x55a172166f60 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f1ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 0x55a17215be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f1ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 0x55a17215be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46558/0 (socket says 172.21.15.104:46558) 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/1425853185 learned_addr learned my addr 172.21.15.104:0/1425853185 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/1425853185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1721617a0 0x55a172161260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/1425853185 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172166f60 msgr2=0x55a172168370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f19d9b700 1 --2- 172.21.15.104:0/1425853185 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172166f60 0x55a172168370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/1425853185 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172166f60 0x55a172168370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/1425853185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1721617a0 msgr2=0x55a172161260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/1425853185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1721617a0 0x55a172161260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f1ad9d700 1 -- 172.21.15.104:0/1425853185 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a172168ee0 con 0x55a172167880 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/1425853185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1721617a0 0x55a172161260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/1425853185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 0x55a17215be80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9f140052c0 tx=0x7f9f1409aa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121e819ad9266af server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f03fff700 1 -- 172.21.15.104:0/1425853185 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1409f3f0 con 0x55a172167880 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f03fff700 1 -- 172.21.15.104:0/1425853185 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f1409b2a0 con 0x55a172167880 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.987+0000 7f9f03fff700 1 -- 172.21.15.104:0/1425853185 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f14022250 con 0x55a172167880 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.988+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1425853185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 msgr2=0x55a17215be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.988+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/1425853185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 0x55a17215be80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9f140052c0 tx=0x7f9f1409aa60 comp rx=0 tx=0).stop 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.988+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1425853185 shutdown_connections 2024-06-09T22:24:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.988+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/1425853185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172167880 0x55a17215be80 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.988+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/1425853185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1721617a0 0x55a172161260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.988+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/1425853185 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a172166f60 0x55a172168370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.988+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1425853185 >> 172.21.15.104:0/1425853185 conn(0x55a17208e080 msgr2=0x55a172167c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.988+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1425853185 shutdown_connections 2024-06-09T22:24:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.988+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1425853185 wait complete. 2024-06-09T22:24:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.988+0000 7f9f2b22db80 1 -- start start 2024-06-09T22:24:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.988+0000 7f9f2b22db80 1 Processor -- start 2024-06-09T22:24:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f2b22db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a1721617a0 0x55a17217cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f2b22db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172166f60 0x55a17217d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f2b22db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a172167880 0x55a17217d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f2b22db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1af8f0 con 0x55a172166f60 2024-06-09T22:24:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f2b22db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1af220 con 0x55a172167880 2024-06-09T22:24:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f2b22db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1ac400 con 0x55a1721617a0 2024-06-09T22:24:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f2b22db80 10 client.0 _ll_register_callbacks cb 0x55a1720485b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f19d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172166f60 0x55a17217d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f19d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172166f60 0x55a17217d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46560/0 (socket says 172.21.15.104:46560) 2024-06-09T22:24:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f19d9b700 1 -- 172.21.15.104:0/1977101198 learned_addr learned my addr 172.21.15.104:0/1977101198 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a1721617a0 0x55a17217cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a172167880 0x55a17217d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f19d9b700 1 -- 172.21.15.104:0/1977101198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a1721617a0 msgr2=0x55a17217cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f19d9b700 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a1721617a0 0x55a17217cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f19d9b700 1 -- 172.21.15.104:0/1977101198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a172167880 msgr2=0x55a17217d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f19d9b700 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a172167880 0x55a17217d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f19d9b700 1 -- 172.21.15.104:0/1977101198 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1721e7e50 con 0x55a172166f60 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f1ad9d700 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a172167880 0x55a17217d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f19d9b700 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172166f60 0x55a17217d440 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x55a17215d2b0 tx=0x7f9f1000bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7089515acc54d0fc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f01ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.989+0000 7f9f01ffb700 1 -- 172.21.15.104:0/1977101198 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f10003c50 con 0x55a172166f60 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f01ffb700 1 -- 172.21.15.104:0/1977101198 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f9f10003df0 con 0x55a172166f60 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1977101198 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f1409b2a0 con 0x55a172166f60 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f01ffb700 1 -- 172.21.15.104:0/1977101198 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9f10011440 con 0x55a172166f60 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f2b22db80 10 client.16434 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1977101198 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a1721e9130 con 0x55a172166f60 2024-06-09T22:24:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f01ffb700 1 -- 172.21.15.104:0/1977101198 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f1409b2a0 con 0x55a172166f60 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f01ffb700 1 -- 172.21.15.104:0/1977101198 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9f10010630 con 0x55a172166f60 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f2b22db80 10 client.16434 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f2b22db80 10 client.16434 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f2b22db80 20 client.16434 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55a1721e98f0 0x55a1721ebce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f2b22db80 4 client.16434 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1977101198 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a1720ba2d0 con 0x55a1721e98f0 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.990+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55a1721e98f0 0x55a1721ebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f1a59c700 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55a1721e98f0 0x55a1721ebce0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=51c8189e887193cd server_cookie=8451232344716462 in_seq=0 out_seq=0 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f01ffb700 10 client.16434 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f01ffb700 1 -- 172.21.15.104:0/1977101198 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a1720ba2d0 con 0x55a1721e98f0 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f01ffb700 10 client.16434 handle_command_reply: tid=0 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f2b22db80 1 client.16434 shutdown 2024-06-09T22:24:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1977101198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172166f60 msgr2=0x55a17217d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172166f60 0x55a17217d440 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x55a17215d2b0 tx=0x7f9f1000bc60 comp rx=0 tx=0).stop 2024-06-09T22:24:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1977101198 shutdown_connections 2024-06-09T22:24:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a172167880 0x55a17217d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a1721617a0 0x55a17217cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55a1721e98f0 0x55a1721ebce0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f2b22db80 1 --2- 172.21.15.104:0/1977101198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a172166f60 0x55a17217d440 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1977101198 >> 172.21.15.104:0/1977101198 conn(0x55a17208e080 msgr2=0x55a1721684f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f01ffb700 0 client.16434 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:24:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1977101198 shutdown_connections 2024-06-09T22:24:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f2b22db80 1 -- 172.21.15.104:0/1977101198 wait complete. 2024-06-09T22:24:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.991+0000 7f9f2b22db80 20 client.16434 trim_cache size 0 max 16384 2024-06-09T22:24:17.997 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:24:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.992+0000 7f9f209aa700 1 -- 172.21.15.104:0/582488032 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9f040a2d90 msgr2=0x7f9f040a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.992+0000 7f9f209aa700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9f040a2d90 0x7f9f040a5250 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00b5c0 tx=0x7f9f0c005960 comp rx=0 tx=0).stop 2024-06-09T22:24:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.992+0000 7f9f209aa700 1 -- 172.21.15.104:0/582488032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f1c1084f0 msgr2=0x7f9f1c198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.992+0000 7f9f209aa700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f1c1084f0 0x7f9f1c198a30 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9f14007ee0 tx=0x7f9f1400d3b0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.992+0000 7f9f209aa700 1 -- 172.21.15.104:0/582488032 shutdown_connections 2024-06-09T22:24:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.992+0000 7f9f209aa700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9f040a2d90 0x7f9f040a5250 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.992+0000 7f9f209aa700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f1c1084f0 0x7f9f1c198a30 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.992+0000 7f9f209aa700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9f1c102550 0x7f9f1c19e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.992+0000 7f9f209aa700 1 --2- 172.21.15.104:0/582488032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f1c102f10 0x7f9f1c19ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.992+0000 7f9f209aa700 1 -- 172.21.15.104:0/582488032 >> 172.21.15.104:0/582488032 conn(0x7f9f1c079650 msgr2=0x7f9f1c1005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.992+0000 7f9f209aa700 1 -- 172.21.15.104:0/582488032 shutdown_connections 2024-06-09T22:24:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:17.992+0000 7f9f209aa700 1 -- 172.21.15.104:0/582488032 wait complete. 2024-06-09T22:24:18.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 50411 3.44 MB/sec warmup 84 sec latency 2544.132 ms 2024-06-09T22:24:18.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-09T22:24:18.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:18 smithi153 ceph-mon[36599]: pgmap v103: 129 pgs: 129 active+clean; 50 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-06-09T22:24:19.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:18 smithi104 ceph-mon[32389]: pgmap v103: 129 pgs: 129 active+clean; 50 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-06-09T22:24:19.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:18 smithi194 ceph-mon[36856]: pgmap v103: 129 pgs: 129 active+clean; 50 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-06-09T22:24:19.059 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:24:19.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 50411 3.40 MB/sec warmup 85 sec latency 3544.184 ms 2024-06-09T22:24:20.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 52169 3.45 MB/sec warmup 86 sec latency 3850.749 ms 2024-06-09T22:24:20.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.847+0000 7f0e74f25700 1 -- 172.21.15.104:0/3133457455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70101d50 msgr2=0x7f0e7010e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:20.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.847+0000 7f0e74f25700 1 --2- 172.21.15.104:0/3133457455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70101d50 0x7f0e7010e240 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800b0a0 tx=0x7f0e6800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.847+0000 7f0e74f25700 1 -- 172.21.15.104:0/3133457455 shutdown_connections 2024-06-09T22:24:20.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.847+0000 7f0e74f25700 1 --2- 172.21.15.104:0/3133457455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70101d50 0x7f0e7010e240 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.847+0000 7f0e74f25700 1 --2- 172.21.15.104:0/3133457455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e70107400 0x7f0e701077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.847+0000 7f0e74f25700 1 --2- 172.21.15.104:0/3133457455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e70101390 0x7f0e70101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.848+0000 7f0e74f25700 1 -- 172.21.15.104:0/3133457455 >> 172.21.15.104:0/3133457455 conn(0x7f0e70078340 msgr2=0x7f0e70078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:20.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.848+0000 7f0e74f25700 1 -- 172.21.15.104:0/3133457455 shutdown_connections 2024-06-09T22:24:20.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.848+0000 7f0e74f25700 1 -- 172.21.15.104:0/3133457455 wait complete. 2024-06-09T22:24:20.850 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:20 smithi104 ceph-mon[32389]: pgmap v104: 129 pgs: 129 active+clean; 49 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 139 op/s 2024-06-09T22:24:20.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e74f25700 1 Processor -- start 2024-06-09T22:24:20.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e74f25700 1 -- start start 2024-06-09T22:24:20.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e74f25700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e70101390 0x7f0e7019d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e74f25700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e70101d50 0x7f0e7019dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e74f25700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70107400 0x7f0e70197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e74f25700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0e701116f0 con 0x7f0e70107400 2024-06-09T22:24:20.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e74f25700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e70111570 con 0x7f0e70101390 2024-06-09T22:24:20.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e74f25700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0e70111870 con 0x7f0e70101d50 2024-06-09T22:24:20.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e6ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70107400 0x7f0e70197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e6dd9b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e70101d50 0x7f0e7019dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e6dd9b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e70101d50 0x7f0e7019dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:37096/0 (socket says 172.21.15.104:37096) 2024-06-09T22:24:20.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e6ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70107400 0x7f0e70197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46578/0 (socket says 172.21.15.104:46578) 2024-06-09T22:24:20.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e6dd9b700 1 -- 172.21.15.104:0/625410206 learned_addr learned my addr 172.21.15.104:0/625410206 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:20.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e70101390 0x7f0e7019d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e6ed9d700 1 -- 172.21.15.104:0/625410206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e70101d50 msgr2=0x7f0e7019dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:20.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e6ed9d700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e70101d50 0x7f0e7019dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e6ed9d700 1 -- 172.21.15.104:0/625410206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e70101390 msgr2=0x7f0e7019d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:20.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e6ed9d700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e70101390 0x7f0e7019d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.849+0000 7f0e6ed9d700 1 -- 172.21.15.104:0/625410206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e701980f0 con 0x7f0e70107400 2024-06-09T22:24:20.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.850+0000 7f0e6ed9d700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70107400 0x7f0e70197990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800b070 tx=0x7f0e68004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:20.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.850+0000 7f0e5f7fe700 1 -- 172.21.15.104:0/625410206 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e68021070 con 0x7f0e70107400 2024-06-09T22:24:20.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.850+0000 7f0e5f7fe700 1 -- 172.21.15.104:0/625410206 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e68007b80 con 0x7f0e70107400 2024-06-09T22:24:20.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.850+0000 7f0e5f7fe700 1 -- 172.21.15.104:0/625410206 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e680052c0 con 0x7f0e70107400 2024-06-09T22:24:20.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.850+0000 7f0e74f25700 1 -- 172.21.15.104:0/625410206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e70198380 con 0x7f0e70107400 2024-06-09T22:24:20.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.850+0000 7f0e74f25700 1 -- 172.21.15.104:0/625410206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e7006a170 con 0x7f0e70107400 2024-06-09T22:24:20.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.852+0000 7f0e5f7fe700 1 -- 172.21.15.104:0/625410206 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f0e68024070 con 0x7f0e70107400 2024-06-09T22:24:20.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.853+0000 7f0e5f7fe700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0e580a2e10 0x7f0e580a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.853+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0e580a2e10 0x7f0e580a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.853+0000 7f0e5f7fe700 1 -- 172.21.15.104:0/625410206 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f0e6800f020 con 0x7f0e70107400 2024-06-09T22:24:20.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0e580a2e10 0x7f0e580a52d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e70106d60 tx=0x7f0e6400b410 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:20.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e7f7a8b80 1 Processor -- start 2024-06-09T22:24:20.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e7f7a8b80 1 -- start start 2024-06-09T22:24:20.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e7f7a8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32bb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e7f7a8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32bb880 0x55a5a32b5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e7f7a8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32b57a0 0x55a5a32b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e7f7a8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a5a31a7f60 con 0x55a5a32baf60 2024-06-09T22:24:20.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e7f7a8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a5a3206850 con 0x55a5a32b57a0 2024-06-09T22:24:20.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e7f7a8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a5a316e460 con 0x55a5a32bb880 2024-06-09T22:24:20.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e6e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32bb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e6e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32bb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46590/0 (socket says 172.21.15.104:46590) 2024-06-09T22:24:20.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e6dd9b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32bb880 0x55a5a32b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.854+0000 7f0e6e59c700 1 -- 172.21.15.104:0/1892321051 learned_addr learned my addr 172.21.15.104:0/1892321051 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:20.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.855+0000 7f0e6ed9d700 1 --2- 172.21.15.104:0/1892321051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32b57a0 0x55a5a32b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.855+0000 7f0e6e59c700 1 -- 172.21.15.104:0/1892321051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32bb880 msgr2=0x55a5a32b5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:20.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.855+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/1892321051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32bb880 0x55a5a32b5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.855+0000 7f0e6e59c700 1 -- 172.21.15.104:0/1892321051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32b57a0 msgr2=0x55a5a32b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:20.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.855+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/1892321051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32b57a0 0x55a5a32b5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.855+0000 7f0e6e59c700 1 -- 172.21.15.104:0/1892321051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a31558d0 con 0x55a5a32baf60 2024-06-09T22:24:20.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.855+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/1892321051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32bb340 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0e64010e00 tx=0x7f0e64013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45db363bde1f6df3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:20.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.855+0000 7f0e5cff9700 1 -- 172.21.15.104:0/1892321051 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64011bb0 con 0x55a5a32baf60 2024-06-09T22:24:20.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.855+0000 7f0e5cff9700 1 -- 172.21.15.104:0/1892321051 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e64011d50 con 0x55a5a32baf60 2024-06-09T22:24:20.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.855+0000 7f0e5cff9700 1 -- 172.21.15.104:0/1892321051 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6401c6e0 con 0x55a5a32baf60 2024-06-09T22:24:20.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.856+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/1892321051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 msgr2=0x55a5a32bb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:20.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.856+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/1892321051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32bb340 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0e64010e00 tx=0x7f0e64013000 comp rx=0 tx=0).stop 2024-06-09T22:24:20.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.856+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/1892321051 shutdown_connections 2024-06-09T22:24:20.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.856+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/1892321051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32bb340 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.856+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/1892321051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32b57a0 0x55a5a32b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.856+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/1892321051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32bb880 0x55a5a32b5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.856+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/1892321051 >> 172.21.15.104:0/1892321051 conn(0x55a5a31e2080 msgr2=0x55a5a32b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:20.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.856+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/1892321051 shutdown_connections 2024-06-09T22:24:20.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.856+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/1892321051 wait complete. 2024-06-09T22:24:20.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.856+0000 7f0e7f7a8b80 1 -- start start 2024-06-09T22:24:20.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e7f7a8b80 1 Processor -- start 2024-06-09T22:24:20.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e7f7a8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32b57a0 0x55a5a32c15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e7f7a8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32be8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e7f7a8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32bb880 0x55a5a32bedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e7f7a8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a5a31a6fc0 con 0x55a5a32baf60 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e7f7a8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a5a3202810 con 0x55a5a32b57a0 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e7f7a8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a5a32c2260 con 0x55a5a32bb880 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e7f7a8b80 10 client.0 _ll_register_callbacks cb 0x55a5a319c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e6dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32be8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e6ed9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32bb880 0x55a5a32bedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e6dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32be8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46594/0 (socket says 172.21.15.104:46594) 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e6ed9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32bb880 0x55a5a32bedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:37124/0 (socket says 172.21.15.104:37124) 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e6dd9b700 1 -- 172.21.15.104:0/317959973 learned_addr learned my addr 172.21.15.104:0/317959973 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32b57a0 0x55a5a32c15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e6dd9b700 1 -- 172.21.15.104:0/317959973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32bb880 msgr2=0x55a5a32bedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:20.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e6dd9b700 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32bb880 0x55a5a32bedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e6dd9b700 1 -- 172.21.15.104:0/317959973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32b57a0 msgr2=0x55a5a32c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:20.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e6dd9b700 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32b57a0 0x55a5a32c15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e6dd9b700 1 -- 172.21.15.104:0/317959973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5a31cc750 con 0x55a5a32baf60 2024-06-09T22:24:20.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.857+0000 7f0e6ed9d700 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32bb880 0x55a5a32bedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:20.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e6dd9b700 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32be8b0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0e60012980 tx=0x7f0e60012c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29ba8c6c85f62ec server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:20.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e55ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:20.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e55ffb700 1 -- 172.21.15.104:0/317959973 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6000f5e0 con 0x55a5a32baf60 2024-06-09T22:24:20.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/317959973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e64011d50 con 0x55a5a32baf60 2024-06-09T22:24:20.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e55ffb700 1 -- 172.21.15.104:0/317959973 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f0e6000f780 con 0x55a5a32baf60 2024-06-09T22:24:20.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e55ffb700 1 -- 172.21.15.104:0/317959973 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0e6000f9c0 con 0x55a5a32baf60 2024-06-09T22:24:20.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e7f7a8b80 10 client.16446 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:20.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/317959973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a5a32ccab0 con 0x55a5a32baf60 2024-06-09T22:24:20.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e55ffb700 1 -- 172.21.15.104:0/317959973 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e64011d50 con 0x55a5a32baf60 2024-06-09T22:24:20.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e55ffb700 1 -- 172.21.15.104:0/317959973 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f0e60010cb0 con 0x55a5a32baf60 2024-06-09T22:24:20.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e7f7a8b80 10 client.16446 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:20.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e7f7a8b80 10 client.16446 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:24:20.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e7f7a8b80 20 client.16446 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:20.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a5a32cf190 0x55a5a32d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e7f7a8b80 4 client.16446 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:20.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.858+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/317959973 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5a3213d50 con 0x55a5a32cf190 2024-06-09T22:24:20.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.859+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a5a32cf190 0x55a5a32d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.859+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a5a32cf190 0x55a5a32d15b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e2f7095e0539d90 server_cookie=fca36ee5a90195c4 in_seq=0 out_seq=0 2024-06-09T22:24:20.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.859+0000 7f0e55ffb700 10 client.16446 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:24:20.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e55ffb700 1 -- 172.21.15.104:0/317959973 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a5a3213d50 con 0x55a5a32cf190 2024-06-09T22:24:20.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e55ffb700 10 client.16446 handle_command_reply: tid=0 2024-06-09T22:24:20.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e7f7a8b80 1 client.16446 shutdown 2024-06-09T22:24:20.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/317959973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 msgr2=0x55a5a32be8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:20.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32be8b0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0e60012980 tx=0x7f0e60012c90 comp rx=0 tx=0).stop 2024-06-09T22:24:20.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/317959973 shutdown_connections 2024-06-09T22:24:20.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a5a32cf190 0x55a5a32d15b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e55ffb700 0 client.16446 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:24:20.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32bb880 0x55a5a32bedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32b57a0 0x55a5a32c15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/317959973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32be8b0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/317959973 >> 172.21.15.104:0/317959973 conn(0x55a5a31e2080 msgr2=0x55a5a32af700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:20.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/317959973 shutdown_connections 2024-06-09T22:24:20.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/317959973 wait complete. 2024-06-09T22:24:20.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.861+0000 7f0e7f7a8b80 20 client.16446 trim_cache size 0 max 16384 2024-06-09T22:24:20.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.877+0000 7f0e7f7a8b80 1 Processor -- start 2024-06-09T22:24:20.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.877+0000 7f0e7f7a8b80 1 -- start start 2024-06-09T22:24:20.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.877+0000 7f0e7f7a8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32b57a0 0x55a5a32b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.877+0000 7f0e7f7a8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32baf60 0x55a5a32bc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.877+0000 7f0e7f7a8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32bb880 0x55a5a32afe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.877+0000 7f0e7f7a8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0e701b05c0 con 0x55a5a32bb880 2024-06-09T22:24:20.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.877+0000 7f0e7f7a8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e701b2160 con 0x55a5a32baf60 2024-06-09T22:24:20.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.877+0000 7f0e7f7a8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0e701b0c90 con 0x55a5a32b57a0 2024-06-09T22:24:20.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e6ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32bb880 0x55a5a32afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e6ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32bb880 0x55a5a32afe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46606/0 (socket says 172.21.15.104:46606) 2024-06-09T22:24:20.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e6ed9d700 1 -- 172.21.15.104:0/141223801 learned_addr learned my addr 172.21.15.104:0/141223801 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:20.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/141223801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32b57a0 0x55a5a32b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e6dd9b700 1 --2- 172.21.15.104:0/141223801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32baf60 0x55a5a32bc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e6ed9d700 1 -- 172.21.15.104:0/141223801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32b57a0 msgr2=0x55a5a32b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:20.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e6ed9d700 1 --2- 172.21.15.104:0/141223801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32b57a0 0x55a5a32b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e6ed9d700 1 -- 172.21.15.104:0/141223801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32baf60 msgr2=0x55a5a32bc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:20.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e6ed9d700 1 --2- 172.21.15.104:0/141223801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32baf60 0x55a5a32bc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e6ed9d700 1 -- 172.21.15.104:0/141223801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a32bce20 con 0x55a5a32bb880 2024-06-09T22:24:20.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e6dd9b700 1 --2- 172.21.15.104:0/141223801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32baf60 0x55a5a32bc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:20.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e6ed9d700 1 --2- 172.21.15.104:0/141223801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32bb880 0x55a5a32afe80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f0e680062a0 tx=0x7f0e6801dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d6896c4e5695e4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:20.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e57fff700 1 -- 172.21.15.104:0/141223801 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6809d470 con 0x55a5a32bb880 2024-06-09T22:24:20.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e57fff700 1 -- 172.21.15.104:0/141223801 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e6809cd70 con 0x55a5a32bb880 2024-06-09T22:24:20.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.878+0000 7f0e57fff700 1 -- 172.21.15.104:0/141223801 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e680993e0 con 0x55a5a32bb880 2024-06-09T22:24:20.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.879+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/141223801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32bb880 msgr2=0x55a5a32afe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:20.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.879+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/141223801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32bb880 0x55a5a32afe80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f0e680062a0 tx=0x7f0e6801dc30 comp rx=0 tx=0).stop 2024-06-09T22:24:20.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.879+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/141223801 shutdown_connections 2024-06-09T22:24:20.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.879+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/141223801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32bb880 0x55a5a32afe80 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.879+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/141223801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32b57a0 0x55a5a32b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.879+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/141223801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32baf60 0x55a5a32bc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.879+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/141223801 >> 172.21.15.104:0/141223801 conn(0x55a5a31e2080 msgr2=0x55a5a32c0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.879+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/141223801 shutdown_connections 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.879+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/141223801 wait complete. 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e7f7a8b80 1 -- start start 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e7f7a8b80 1 Processor -- start 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e7f7a8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32b57a0 0x55a5a32c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e7f7a8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32c5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e7f7a8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32bb880 0x55a5a3331ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e7f7a8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0e701b1a30 con 0x55a5a32baf60 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e7f7a8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e701b1360 con 0x55a5a32bb880 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e7f7a8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0e701ae560 con 0x55a5a32b57a0 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e7f7a8b80 10 client.0 _ll_register_callbacks cb 0x55a5a319c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:20.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e6dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e6dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32c5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46620/0 (socket says 172.21.15.104:46620) 2024-06-09T22:24:20.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e6dd9b700 1 -- 172.21.15.104:0/4033372973 learned_addr learned my addr 172.21.15.104:0/4033372973 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:20.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32b57a0 0x55a5a32c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e6ed9d700 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32bb880 0x55a5a3331ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e6dd9b700 1 -- 172.21.15.104:0/4033372973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32b57a0 msgr2=0x55a5a32c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:20.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e6dd9b700 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32b57a0 0x55a5a32c4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e6dd9b700 1 -- 172.21.15.104:0/4033372973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32bb880 msgr2=0x55a5a3331ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:20.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e6dd9b700 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32bb880 0x55a5a3331ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e6dd9b700 1 -- 172.21.15.104:0/4033372973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5a3332510 con 0x55a5a32baf60 2024-06-09T22:24:20.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.880+0000 7f0e6ed9d700 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32bb880 0x55a5a3331ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:20.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e6dd9b700 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32c5250 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0e60000c00 tx=0x7f0e60009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67365a15a5d0bcd4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:20.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e55ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e55ffb700 1 -- 172.21.15.104:0/4033372973 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6002f060 con 0x55a5a32baf60 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e55ffb700 1 -- 172.21.15.104:0/4033372973 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f0e6000fd70 con 0x55a5a32baf60 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/4033372973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e6809cd70 con 0x55a5a32baf60 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e55ffb700 1 -- 172.21.15.104:0/4033372973 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0e60036050 con 0x55a5a32baf60 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e7f7a8b80 10 client.16452 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/4033372973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a5a3339860 con 0x55a5a32baf60 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e55ffb700 1 -- 172.21.15.104:0/4033372973 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e6809cd70 con 0x55a5a32baf60 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e55ffb700 1 -- 172.21.15.104:0/4033372973 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f0e60004710 con 0x55a5a32baf60 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e7f7a8b80 10 client.16452 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e7f7a8b80 10 client.16452 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e7f7a8b80 20 client.16452 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a5a3339f40 0x55a5a333c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:20.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e7f7a8b80 4 client.16452 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:24:20.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.881+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/4033372973 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a5a320e2d0 con 0x55a5a3339f40 2024-06-09T22:24:20.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a5a3339f40 0x55a5a333c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:20.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e6e59c700 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a5a3339f40 0x55a5a333c360 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=63946bd23c154f17 server_cookie=ce4cd99b7594dd8d in_seq=0 out_seq=0 2024-06-09T22:24:20.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e55ffb700 10 client.16452 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:24:20.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e55ffb700 1 -- 172.21.15.104:0/4033372973 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a5a320e2d0 con 0x55a5a3339f40 2024-06-09T22:24:20.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e55ffb700 10 client.16452 handle_command_reply: tid=0 2024-06-09T22:24:20.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e7f7a8b80 1 client.16452 shutdown 2024-06-09T22:24:20.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/4033372973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 msgr2=0x55a5a32c5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:20.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32c5250 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0e60000c00 tx=0x7f0e60009fd0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/4033372973 shutdown_connections 2024-06-09T22:24:20.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a5a3339f40 0x55a5a333c360 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a5a32bb880 0x55a5a3331ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a5a32b57a0 0x55a5a32c4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e7f7a8b80 1 --2- 172.21.15.104:0/4033372973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a5a32baf60 0x55a5a32c5250 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/4033372973 >> 172.21.15.104:0/4033372973 conn(0x55a5a31e2080 msgr2=0x55a5a32b4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:20.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.882+0000 7f0e55ffb700 0 client.16452 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:24:20.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/4033372973 shutdown_connections 2024-06-09T22:24:20.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e7f7a8b80 1 -- 172.21.15.104:0/4033372973 wait complete. 2024-06-09T22:24:20.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e7f7a8b80 20 client.16452 trim_cache size 0 max 16384 2024-06-09T22:24:20.890 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:24:20.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e74f25700 1 -- 172.21.15.104:0/625410206 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0e580a2e10 msgr2=0x7f0e580a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:20.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e74f25700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0e580a2e10 0x7f0e580a52d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e70106d60 tx=0x7f0e6400b410 comp rx=0 tx=0).stop 2024-06-09T22:24:20.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e74f25700 1 -- 172.21.15.104:0/625410206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70107400 msgr2=0x7f0e70197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:20.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e74f25700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70107400 0x7f0e70197990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800b070 tx=0x7f0e68004c80 comp rx=0 tx=0).stop 2024-06-09T22:24:20.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e74f25700 1 -- 172.21.15.104:0/625410206 shutdown_connections 2024-06-09T22:24:20.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e74f25700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0e580a2e10 0x7f0e580a52d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e74f25700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70107400 0x7f0e70197990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e74f25700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e70101390 0x7f0e7019d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e74f25700 1 --2- 172.21.15.104:0/625410206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e70101d50 0x7f0e7019dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:20.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e74f25700 1 -- 172.21.15.104:0/625410206 >> 172.21.15.104:0/625410206 conn(0x7f0e70078340 msgr2=0x7f0e700ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:20.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e74f25700 1 -- 172.21.15.104:0/625410206 shutdown_connections 2024-06-09T22:24:20.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:20.883+0000 7f0e74f25700 1 -- 172.21.15.104:0/625410206 wait complete. 2024-06-09T22:24:20.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:20 smithi153 ceph-mon[36599]: pgmap v104: 129 pgs: 129 active+clean; 49 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 139 op/s 2024-06-09T22:24:21.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:20 smithi194 ceph-mon[36856]: pgmap v104: 129 pgs: 129 active+clean; 49 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 139 op/s 2024-06-09T22:24:21.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 53920 3.54 MB/sec warmup 87 sec latency 14.157 ms 2024-06-09T22:24:21.811 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-09T22:24:22.064 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:24:22.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 56624 3.66 MB/sec warmup 88 sec latency 13.025 ms 2024-06-09T22:24:22.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:22 smithi153 ceph-mon[36599]: pgmap v105: 129 pgs: 129 active+clean; 49 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 796 KiB/s wr, 75 op/s 2024-06-09T22:24:23.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:22 smithi104 ceph-mon[32389]: pgmap v105: 129 pgs: 129 active+clean; 49 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 796 KiB/s wr, 75 op/s 2024-06-09T22:24:23.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:22 smithi194 ceph-mon[36856]: pgmap v105: 129 pgs: 129 active+clean; 49 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 796 KiB/s wr, 75 op/s 2024-06-09T22:24:23.721 INFO:tasks.workunit.client.0.smithi104.stdout: 1 57447 3.67 MB/sec warmup 89 sec latency 303.533 ms 2024-06-09T22:24:23.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.895+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/1452634066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cc0103150 msgr2=0x7f8cc0107140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:23.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.895+0000 7f8cc6cdc700 1 --2- 172.21.15.104:0/1452634066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cc0103150 0x7f8cc0107140 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00099c0 tx=0x7f8cb0009cd0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.895+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/1452634066 shutdown_connections 2024-06-09T22:24:23.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.895+0000 7f8cc6cdc700 1 --2- 172.21.15.104:0/1452634066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cc0107870 0x7f8cc010f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.895+0000 7f8cc6cdc700 1 --2- 172.21.15.104:0/1452634066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cc01027a0 0x7f8cc0102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.895+0000 7f8cc6cdc700 1 --2- 172.21.15.104:0/1452634066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cc0103150 0x7f8cc0107140 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.895+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/1452634066 >> 172.21.15.104:0/1452634066 conn(0x7f8cc0079520 msgr2=0x7f8cc0079930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:23.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.895+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/1452634066 shutdown_connections 2024-06-09T22:24:23.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.895+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/1452634066 wait complete. 2024-06-09T22:24:23.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc6cdc700 1 Processor -- start 2024-06-09T22:24:23.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc6cdc700 1 -- start start 2024-06-09T22:24:23.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc6cdc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cc01027a0 0x7f8cc0074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc6cdc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cc0103150 0x7f8cc0074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc6cdc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cc0107870 0x7f8cc0075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc6cdc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc0112780 con 0x7f8cc01027a0 2024-06-09T22:24:23.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc6cdc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc0112600 con 0x7f8cc0103150 2024-06-09T22:24:23.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc6cdc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc0112900 con 0x7f8cc0107870 2024-06-09T22:24:23.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc4a78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cc01027a0 0x7f8cc0074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc4a78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cc01027a0 0x7f8cc0074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46634/0 (socket says 172.21.15.104:46634) 2024-06-09T22:24:23.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc4a78700 1 -- 172.21.15.104:0/4065050518 learned_addr learned my addr 172.21.15.104:0/4065050518 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:23.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cb7fff700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cc0103150 0x7f8cc0074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc5279700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cc0107870 0x7f8cc0075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc4a78700 1 -- 172.21.15.104:0/4065050518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cc0107870 msgr2=0x7f8cc0075210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:23.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cc0107870 0x7f8cc0075210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc4a78700 1 -- 172.21.15.104:0/4065050518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cc0103150 msgr2=0x7f8cc0074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:23.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cc0103150 0x7f8cc0074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.896+0000 7f8cc4a78700 1 -- 172.21.15.104:0/4065050518 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc01a70b0 con 0x7f8cc01027a0 2024-06-09T22:24:23.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.897+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cc01027a0 0x7f8cc0074790 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0060b0 tx=0x7f8cbc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:23.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.897+0000 7f8cb5ffb700 1 -- 172.21.15.104:0/4065050518 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc01a070 con 0x7f8cc01027a0 2024-06-09T22:24:23.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.897+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/4065050518 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc01a73a0 con 0x7f8cc01027a0 2024-06-09T22:24:23.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.897+0000 7f8cb5ffb700 1 -- 172.21.15.104:0/4065050518 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cbc00ddd0 con 0x7f8cc01027a0 2024-06-09T22:24:23.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.897+0000 7f8cb5ffb700 1 -- 172.21.15.104:0/4065050518 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc0147a0 con 0x7f8cc01027a0 2024-06-09T22:24:23.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.897+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/4065050518 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc01a7900 con 0x7f8cc01027a0 2024-06-09T22:24:23.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.899+0000 7f8cb5ffb700 1 -- 172.21.15.104:0/4065050518 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f8cbc01d070 con 0x7f8cc01027a0 2024-06-09T22:24:23.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.900+0000 7f8cb5ffb700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8ca80a2d90 0x7f8ca80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.900+0000 7f8cb7fff700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8ca80a2d90 0x7f8ca80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.900+0000 7f8cb5ffb700 1 -- 172.21.15.104:0/4065050518 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8cbc0cf0b0 con 0x7f8cc01027a0 2024-06-09T22:24:23.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.900+0000 7f8cb7fff700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8ca80a2d90 0x7f8ca80a5250 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00761d0 tx=0x7f8cb0005fb0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:23.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cd155fb80 1 Processor -- start 2024-06-09T22:24:23.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cd155fb80 1 -- start start 2024-06-09T22:24:23.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cd155fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8640 0x55ef74cd8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cd155fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8f60 0x55ef74cd29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cd155fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd2f20 0x55ef74cd3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cd155fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55ef74c24810 con 0x55ef74cd8640 2024-06-09T22:24:23.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cd155fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ef74bcc720 con 0x55ef74cd8f60 2024-06-09T22:24:23.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cd155fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ef74c207d0 con 0x55ef74cd2f20 2024-06-09T22:24:23.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cc4a78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8640 0x55ef74cd8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cc4a78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8640 0x55ef74cd8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46642/0 (socket says 172.21.15.104:46642) 2024-06-09T22:24:23.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cc4a78700 1 -- 172.21.15.104:0/4178075739 learned_addr learned my addr 172.21.15.104:0/4178075739 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:23.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cb7fff700 1 --2- 172.21.15.104:0/4178075739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8f60 0x55ef74cd29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cc5279700 1 --2- 172.21.15.104:0/4178075739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd2f20 0x55ef74cd3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cc4a78700 1 -- 172.21.15.104:0/4178075739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd2f20 msgr2=0x55ef74cd3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:23.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/4178075739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd2f20 0x55ef74cd3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cc4a78700 1 -- 172.21.15.104:0/4178075739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8f60 msgr2=0x55ef74cd29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:23.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/4178075739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8f60 0x55ef74cd29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.901+0000 7f8cc4a78700 1 -- 172.21.15.104:0/4178075739 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef74bc2f50 con 0x55ef74cd8640 2024-06-09T22:24:23.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.902+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/4178075739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8640 0x55ef74cd8a20 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc097c30 tx=0x7f8cbc09ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c499ea719b390e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:23.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.902+0000 7f8ca6ffd700 1 -- 172.21.15.104:0/4178075739 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc0a3ed0 con 0x55ef74cd8640 2024-06-09T22:24:23.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.902+0000 7f8ca6ffd700 1 -- 172.21.15.104:0/4178075739 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cbc015ae0 con 0x55ef74cd8640 2024-06-09T22:24:23.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.902+0000 7f8ca6ffd700 1 -- 172.21.15.104:0/4178075739 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc09b720 con 0x55ef74cd8640 2024-06-09T22:24:23.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.902+0000 7f8cd155fb80 1 -- 172.21.15.104:0/4178075739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8640 msgr2=0x55ef74cd8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:23.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.902+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/4178075739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8640 0x55ef74cd8a20 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc097c30 tx=0x7f8cbc09ba30 comp rx=0 tx=0).stop 2024-06-09T22:24:23.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.902+0000 7f8cd155fb80 1 -- 172.21.15.104:0/4178075739 shutdown_connections 2024-06-09T22:24:23.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.902+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/4178075739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8640 0x55ef74cd8a20 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.902+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/4178075739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd2f20 0x55ef74cd3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/4178075739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8f60 0x55ef74cd29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 1 -- 172.21.15.104:0/4178075739 >> 172.21.15.104:0/4178075739 conn(0x55ef74bfdcc0 msgr2=0x55ef74ccd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:23.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 1 -- 172.21.15.104:0/4178075739 shutdown_connections 2024-06-09T22:24:23.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 1 -- 172.21.15.104:0/4178075739 wait complete. 2024-06-09T22:24:23.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 1 -- start start 2024-06-09T22:24:23.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 1 Processor -- start 2024-06-09T22:24:23.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd2f20 0x55ef74cded80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8640 0x55ef74cdc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd8f60 0x55ef74cdc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55ef74cdf9d0 con 0x55ef74cd2f20 2024-06-09T22:24:23.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ef74bc4bf0 con 0x55ef74cd8640 2024-06-09T22:24:23.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ef74cdfb50 con 0x55ef74cd8f60 2024-06-09T22:24:23.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cd155fb80 10 client.0 _ll_register_callbacks cb 0x55ef74bbe920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:23.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.903+0000 7f8cc4a78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd2f20 0x55ef74cded80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cc4a78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd2f20 0x55ef74cded80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46654/0 (socket says 172.21.15.104:46654) 2024-06-09T22:24:23.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cc4a78700 1 -- 172.21.15.104:0/2521851902 learned_addr learned my addr 172.21.15.104:0/2521851902 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:23.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cb7fff700 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8640 0x55ef74cdc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cc5279700 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd8f60 0x55ef74cdc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cc4a78700 1 -- 172.21.15.104:0/2521851902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd8f60 msgr2=0x55ef74cdc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:23.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd8f60 0x55ef74cdc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cc4a78700 1 -- 172.21.15.104:0/2521851902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8640 msgr2=0x55ef74cdc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:23.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8640 0x55ef74cdc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cc4a78700 1 -- 172.21.15.104:0/2521851902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef74b738d0 con 0x55ef74cd2f20 2024-06-09T22:24:23.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd2f20 0x55ef74cded80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0cf350 tx=0x7f8cbc0cce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2dd7ada6a0dc67 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:23.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8c9ffff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:23.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8c9ffff700 1 -- 172.21.15.104:0/2521851902 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc0b8070 con 0x55ef74cd2f20 2024-06-09T22:24:23.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8c9ffff700 1 -- 172.21.15.104:0/2521851902 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8cbc099560 con 0x55ef74cd2f20 2024-06-09T22:24:23.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cd155fb80 1 -- 172.21.15.104:0/2521851902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cbc015ae0 con 0x55ef74cd2f20 2024-06-09T22:24:23.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8c9ffff700 1 -- 172.21.15.104:0/2521851902 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8cbc0a3ed0 con 0x55ef74cd2f20 2024-06-09T22:24:23.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cd155fb80 10 client.16464 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:23.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.904+0000 7f8cd155fb80 1 -- 172.21.15.104:0/2521851902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ef74cea390 con 0x55ef74cd2f20 2024-06-09T22:24:23.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.905+0000 7f8c9ffff700 1 -- 172.21.15.104:0/2521851902 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cbc015ae0 con 0x55ef74cd2f20 2024-06-09T22:24:23.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.905+0000 7f8c9ffff700 1 -- 172.21.15.104:0/2521851902 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8cbc0a3ad0 con 0x55ef74cd2f20 2024-06-09T22:24:23.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.905+0000 7f8cd155fb80 10 client.16464 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:23.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.905+0000 7f8cd155fb80 10 client.16464 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:24:23.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.905+0000 7f8cd155fb80 20 client.16464 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:23.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.905+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55ef74cec860 0x55ef74ceec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.905+0000 7f8cd155fb80 4 client.16464 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:23.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.905+0000 7f8cd155fb80 1 -- 172.21.15.104:0/2521851902 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef74c31d10 con 0x55ef74cec860 2024-06-09T22:24:23.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.905+0000 7f8cb7fff700 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55ef74cec860 0x55ef74ceec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.905+0000 7f8c9ffff700 10 client.16464 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:24:23.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.905+0000 7f8cb7fff700 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55ef74cec860 0x55ef74ceec50 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2aa63a3b63d9f6bf server_cookie=2593f08390d77a5b in_seq=0 out_seq=0 2024-06-09T22:24:23.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8c9ffff700 1 -- 172.21.15.104:0/2521851902 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55ef74c31d10 con 0x55ef74cec860 2024-06-09T22:24:23.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8c9ffff700 10 client.16464 handle_command_reply: tid=0 2024-06-09T22:24:23.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8cd155fb80 1 client.16464 shutdown 2024-06-09T22:24:23.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8cd155fb80 1 -- 172.21.15.104:0/2521851902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd2f20 msgr2=0x55ef74cded80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:23.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd2f20 0x55ef74cded80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0cf350 tx=0x7f8cbc0cce70 comp rx=0 tx=0).stop 2024-06-09T22:24:23.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8cd155fb80 1 -- 172.21.15.104:0/2521851902 shutdown_connections 2024-06-09T22:24:23.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55ef74cec860 0x55ef74ceec50 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd2f20 0x55ef74cded80 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd8f60 0x55ef74cdc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/2521851902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8640 0x55ef74cdc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8cd155fb80 1 -- 172.21.15.104:0/2521851902 >> 172.21.15.104:0/2521851902 conn(0x55ef74bfdcc0 msgr2=0x55ef74ccd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:23.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8c9ffff700 0 client.16464 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:24:23.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8cd155fb80 1 -- 172.21.15.104:0/2521851902 shutdown_connections 2024-06-09T22:24:23.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8cd155fb80 1 -- 172.21.15.104:0/2521851902 wait complete. 2024-06-09T22:24:23.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.906+0000 7f8cd155fb80 20 client.16464 trim_cache size 0 max 16384 2024-06-09T22:24:23.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.921+0000 7f8cd155fb80 1 Processor -- start 2024-06-09T22:24:23.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.921+0000 7f8cd155fb80 1 -- start start 2024-06-09T22:24:23.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.921+0000 7f8cd155fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd2f20 0x55ef74cd29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.921+0000 7f8cd155fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd8640 0x55ef74cd9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cd155fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 0x55ef74cda840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cd155fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01b0750 con 0x55ef74cd8f60 2024-06-09T22:24:23.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cd155fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01b22f0 con 0x55ef74cd2f20 2024-06-09T22:24:23.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cd155fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01b0e20 con 0x55ef74cd8640 2024-06-09T22:24:23.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cc5279700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 0x55ef74cda840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cc5279700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 0x55ef74cda840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46668/0 (socket says 172.21.15.104:46668) 2024-06-09T22:24:23.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cc5279700 1 -- 172.21.15.104:0/3065869455 learned_addr learned my addr 172.21.15.104:0/3065869455 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/3065869455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd2f20 0x55ef74cd29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cb7fff700 1 --2- 172.21.15.104:0/3065869455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd8640 0x55ef74cd9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cc5279700 1 -- 172.21.15.104:0/3065869455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd8640 msgr2=0x55ef74cd9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cc5279700 1 --2- 172.21.15.104:0/3065869455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd8640 0x55ef74cd9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cc5279700 1 -- 172.21.15.104:0/3065869455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd2f20 msgr2=0x55ef74cd29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cc5279700 1 --2- 172.21.15.104:0/3065869455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd2f20 0x55ef74cd29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cc5279700 1 -- 172.21.15.104:0/3065869455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef74cda5f0 con 0x55ef74cd8f60 2024-06-09T22:24:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cb7fff700 1 --2- 172.21.15.104:0/3065869455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd8640 0x55ef74cd9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:24:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8cc5279700 1 --2- 172.21.15.104:0/3065869455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 0x55ef74cda840 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800dd70 tx=0x7f8cb8012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2254fc8a69cc4467 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8ca67fc700 1 -- 172.21.15.104:0/3065869455 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb801f020 con 0x55ef74cd8f60 2024-06-09T22:24:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8ca67fc700 1 -- 172.21.15.104:0/3065869455 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cb8004d10 con 0x55ef74cd8f60 2024-06-09T22:24:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.922+0000 7f8ca67fc700 1 -- 172.21.15.104:0/3065869455 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8005600 con 0x55ef74cd8f60 2024-06-09T22:24:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.923+0000 7f8cd155fb80 1 -- 172.21.15.104:0/3065869455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 msgr2=0x55ef74cda840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.923+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/3065869455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 0x55ef74cda840 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800dd70 tx=0x7f8cb8012730 comp rx=0 tx=0).stop 2024-06-09T22:24:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.923+0000 7f8cd155fb80 1 -- 172.21.15.104:0/3065869455 shutdown_connections 2024-06-09T22:24:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.923+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/3065869455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 0x55ef74cda840 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.923+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/3065869455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd2f20 0x55ef74cd29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.923+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/3065869455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd8640 0x55ef74cd9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.923+0000 7f8cd155fb80 1 -- 172.21.15.104:0/3065869455 >> 172.21.15.104:0/3065869455 conn(0x55ef74bfdcc0 msgr2=0x55ef74cd9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.923+0000 7f8cd155fb80 1 -- 172.21.15.104:0/3065869455 shutdown_connections 2024-06-09T22:24:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.923+0000 7f8cd155fb80 1 -- 172.21.15.104:0/3065869455 wait complete. 2024-06-09T22:24:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.923+0000 7f8cd155fb80 1 -- start start 2024-06-09T22:24:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.923+0000 7f8cd155fb80 1 Processor -- start 2024-06-09T22:24:23.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cd155fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd2f20 0x55ef74cef030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cd155fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8640 0x55ef74cef570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cd155fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 0x55ef74d4f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cd155fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01b1bc0 con 0x55ef74cd8f60 2024-06-09T22:24:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cd155fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01b14f0 con 0x55ef74cd8640 2024-06-09T22:24:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cd155fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01ae6d0 con 0x55ef74cd2f20 2024-06-09T22:24:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cd155fb80 10 client.0 _ll_register_callbacks cb 0x55ef74bbe920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cc5279700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 0x55ef74d4f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cc5279700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 0x55ef74d4f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46676/0 (socket says 172.21.15.104:46676) 2024-06-09T22:24:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cc5279700 1 -- 172.21.15.104:0/1170117521 learned_addr learned my addr 172.21.15.104:0/1170117521 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cb7fff700 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8640 0x55ef74cef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd2f20 0x55ef74cef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cc5279700 1 -- 172.21.15.104:0/1170117521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd2f20 msgr2=0x55ef74cef030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cc5279700 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd2f20 0x55ef74cef030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cc5279700 1 -- 172.21.15.104:0/1170117521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8640 msgr2=0x55ef74cef570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cc5279700 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8640 0x55ef74cef570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cc5279700 1 -- 172.21.15.104:0/1170117521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef74d4f7c0 con 0x55ef74cd8f60 2024-06-09T22:24:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cb7fff700 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8640 0x55ef74cef570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8cc5279700 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 0x55ef74d4f1d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80051a0 tx=0x7f8cb8012e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183a187fbb54ef81 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.924+0000 7f8c9ffff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8c9ffff700 1 -- 172.21.15.104:0/1170117521 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb801f040 con 0x55ef74cd8f60 2024-06-09T22:24:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8c9ffff700 1 -- 172.21.15.104:0/1170117521 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8cb8004960 con 0x55ef74cd8f60 2024-06-09T22:24:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8cd155fb80 1 -- 172.21.15.104:0/1170117521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cb8004d10 con 0x55ef74cd8f60 2024-06-09T22:24:23.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8c9ffff700 1 -- 172.21.15.104:0/1170117521 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8cb8005200 con 0x55ef74cd8f60 2024-06-09T22:24:23.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8cd155fb80 10 client.16470 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:23.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8cd155fb80 1 -- 172.21.15.104:0/1170117521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ef74d56ad0 con 0x55ef74cd8f60 2024-06-09T22:24:23.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8c9ffff700 1 -- 172.21.15.104:0/1170117521 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cb8004d10 con 0x55ef74cd8f60 2024-06-09T22:24:23.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8c9ffff700 1 -- 172.21.15.104:0/1170117521 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8cb801a070 con 0x55ef74cd8f60 2024-06-09T22:24:23.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8cd155fb80 10 client.16470 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:23.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8cd155fb80 10 client.16470 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:24:23.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8cd155fb80 20 client.16470 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:23.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55ef74d57120 0x55ef74d59540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:23.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8cd155fb80 4 client.16470 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:24:23.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8cd155fb80 1 -- 172.21.15.104:0/1170117521 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef74c2c290 con 0x55ef74d57120 2024-06-09T22:24:23.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.925+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55ef74d57120 0x55ef74d59540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:23.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8c9ffff700 10 client.16470 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:24:23.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cc4a78700 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55ef74d57120 0x55ef74d59540 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fc600fce1bdf1234 server_cookie=3b1c46f30e7b9b2c in_seq=0 out_seq=0 2024-06-09T22:24:23.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8c9ffff700 1 -- 172.21.15.104:0/1170117521 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef74c2c290 con 0x55ef74d57120 2024-06-09T22:24:23.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8c9ffff700 10 client.16470 handle_command_reply: tid=0 2024-06-09T22:24:23.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cd155fb80 1 client.16470 shutdown 2024-06-09T22:24:23.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cd155fb80 1 -- 172.21.15.104:0/1170117521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 msgr2=0x55ef74d4f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:23.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 0x55ef74d4f1d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80051a0 tx=0x7f8cb8012e70 comp rx=0 tx=0).stop 2024-06-09T22:24:23.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cd155fb80 1 -- 172.21.15.104:0/1170117521 shutdown_connections 2024-06-09T22:24:23.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55ef74d57120 0x55ef74d59540 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ef74cd8f60 0x55ef74d4f1d0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef74cd2f20 0x55ef74cef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8c9ffff700 0 client.16470 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:24:23.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cd155fb80 1 --2- 172.21.15.104:0/1170117521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ef74cd8640 0x55ef74cef570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cd155fb80 1 -- 172.21.15.104:0/1170117521 >> 172.21.15.104:0/1170117521 conn(0x55ef74bfdcc0 msgr2=0x55ef74cd9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:23.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cd155fb80 1 -- 172.21.15.104:0/1170117521 shutdown_connections 2024-06-09T22:24:23.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cd155fb80 1 -- 172.21.15.104:0/1170117521 wait complete. 2024-06-09T22:24:23.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.926+0000 7f8cd155fb80 20 client.16470 trim_cache size 0 max 16384 2024-06-09T22:24:23.935 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:24:23.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.927+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/4065050518 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8ca80a2d90 msgr2=0x7f8ca80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:23.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.927+0000 7f8cc6cdc700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8ca80a2d90 0x7f8ca80a5250 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00761d0 tx=0x7f8cb0005fb0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.927+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/4065050518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cc01027a0 msgr2=0x7f8cc0074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:23.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.927+0000 7f8cc6cdc700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cc01027a0 0x7f8cc0074790 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0060b0 tx=0x7f8cbc00d3b0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.927+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/4065050518 shutdown_connections 2024-06-09T22:24:23.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.927+0000 7f8cc6cdc700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8ca80a2d90 0x7f8ca80a5250 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.927+0000 7f8cc6cdc700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cc01027a0 0x7f8cc0074790 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.927+0000 7f8cc6cdc700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8cc0107870 0x7f8cc0075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.927+0000 7f8cc6cdc700 1 --2- 172.21.15.104:0/4065050518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cc0103150 0x7f8cc0074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:23.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.927+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/4065050518 >> 172.21.15.104:0/4065050518 conn(0x7f8cc0079520 msgr2=0x7f8cc0100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:23.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.927+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/4065050518 shutdown_connections 2024-06-09T22:24:23.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:23.927+0000 7f8cc6cdc700 1 -- 172.21.15.104:0/4065050518 wait complete. 2024-06-09T22:24:24.656 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-09T22:24:24.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 57447 3.63 MB/sec warmup 90 sec latency 1303.594 ms 2024-06-09T22:24:24.897 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:24:24.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:24 smithi153 ceph-mon[36599]: pgmap v106: 129 pgs: 129 active+clean; 51 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-06-09T22:24:25.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:24 smithi104 ceph-mon[32389]: pgmap v106: 129 pgs: 129 active+clean; 51 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-06-09T22:24:25.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:24 smithi194 ceph-mon[36856]: pgmap v106: 129 pgs: 129 active+clean; 51 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-06-09T22:24:25.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 57447 3.59 MB/sec warmup 91 sec latency 2303.668 ms 2024-06-09T22:24:26.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 57503 3.56 MB/sec warmup 92 sec latency 2692.171 ms 2024-06-09T22:24:26.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.741+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/1370148840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df4101440 msgr2=0x7f8df410f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:26.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.741+0000 7f8dfa7cb700 1 --2- 172.21.15.104:0/1370148840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df4101440 0x7f8df410f530 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f8dec00b0a0 tx=0x7f8dec00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.741+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/1370148840 shutdown_connections 2024-06-09T22:24:26.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.741+0000 7f8dfa7cb700 1 --2- 172.21.15.104:0/1370148840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df4101440 0x7f8df410f530 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.741+0000 7f8dfa7cb700 1 --2- 172.21.15.104:0/1370148840 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8df41086f0 0x7f8df4108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.741+0000 7f8dfa7cb700 1 --2- 172.21.15.104:0/1370148840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8df4100a80 0x7f8df4100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.741+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/1370148840 >> 172.21.15.104:0/1370148840 conn(0x7f8df4077a00 msgr2=0x7f8df40fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:26.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.741+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/1370148840 shutdown_connections 2024-06-09T22:24:26.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.741+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/1370148840 wait complete. 2024-06-09T22:24:26.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8dfa7cb700 1 Processor -- start 2024-06-09T22:24:26.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8dfa7cb700 1 -- start start 2024-06-09T22:24:26.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8dfa7cb700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8df4100a80 0x7f8df4101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8dfa7cb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8df4101440 0x7f8df41021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8dfa7cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df41086f0 0x7f8df41026e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8dfa7cb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41129f0 con 0x7f8df41086f0 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8dfa7cb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8df4112870 con 0x7f8df4101440 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8dfa7cb700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8df4112b70 con 0x7f8df4100a80 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8df8d68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df41086f0 0x7f8df41026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8df8d68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df41086f0 0x7f8df41026e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54200/0 (socket says 172.21.15.104:54200) 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8df8d68700 1 -- 172.21.15.104:0/2415653932 learned_addr learned my addr 172.21.15.104:0/2415653932 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8debfff700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8df4101440 0x7f8df41021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.742+0000 7f8df3fff700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8df4100a80 0x7f8df4101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.743+0000 7f8df8d68700 1 -- 172.21.15.104:0/2415653932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8df4100a80 msgr2=0x7f8df4101c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.743+0000 7f8df8d68700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8df4100a80 0x7f8df4101c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.743+0000 7f8df8d68700 1 -- 172.21.15.104:0/2415653932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8df4101440 msgr2=0x7f8df41021a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:26.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.743+0000 7f8df8d68700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8df4101440 0x7f8df41021a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.743+0000 7f8df8d68700 1 -- 172.21.15.104:0/2415653932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df41a72e0 con 0x7f8df41086f0 2024-06-09T22:24:26.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.743+0000 7f8df8d68700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df41086f0 0x7f8df41026e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0062a0 tx=0x7f8dec004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:26.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.743+0000 7f8df1ffb700 1 -- 172.21.15.104:0/2415653932 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec021070 con 0x7f8df41086f0 2024-06-09T22:24:26.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.743+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/2415653932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df41a7570 con 0x7f8df41086f0 2024-06-09T22:24:26.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.743+0000 7f8df1ffb700 1 -- 172.21.15.104:0/2415653932 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dec007c80 con 0x7f8df41086f0 2024-06-09T22:24:26.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.743+0000 7f8df1ffb700 1 -- 172.21.15.104:0/2415653932 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec01d450 con 0x7f8df41086f0 2024-06-09T22:24:26.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.743+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/2415653932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df41a7a70 con 0x7f8df41086f0 2024-06-09T22:24:26.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.746+0000 7f8df1ffb700 1 -- 172.21.15.104:0/2415653932 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f8dec004020 con 0x7f8df41086f0 2024-06-09T22:24:26.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.746+0000 7f8df1ffb700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8de00a2d90 0x7f8de00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8df3fff700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8de00a2d90 0x7f8de00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8df1ffb700 1 -- 172.21.15.104:0/2415653932 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8dec00f020 con 0x7f8df41086f0 2024-06-09T22:24:26.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8df3fff700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8de00a2d90 0x7f8de00a5250 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8de4006fd0 tx=0x7f8de4008040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:26.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8e0504eb80 1 Processor -- start 2024-06-09T22:24:26.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8e0504eb80 1 -- start start 2024-06-09T22:24:26.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8e0504eb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392640 0x5606c7392a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8e0504eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c7392f60 0x5606c738c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8e0504eb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c738cf20 0x5606c738d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8e0504eb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5606c72de810 con 0x5606c738cf20 2024-06-09T22:24:26.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8e0504eb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5606c7286720 con 0x5606c7392640 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8e0504eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5606c72da7d0 con 0x5606c7392f60 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8df3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392640 0x5606c7392a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8df8d68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c738cf20 0x5606c738d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8df8d68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c738cf20 0x5606c738d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54206/0 (socket says 172.21.15.104:54206) 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8df3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392640 0x5606c7392a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:52068/0 (socket says 172.21.15.104:52068) 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.747+0000 7f8df8d68700 1 -- 172.21.15.104:0/1512955541 learned_addr learned my addr 172.21.15.104:0/1512955541 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.748+0000 7f8debfff700 1 --2- 172.21.15.104:0/1512955541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c7392f60 0x5606c738c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.748+0000 7f8df8d68700 1 -- 172.21.15.104:0/1512955541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c7392f60 msgr2=0x5606c738c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.748+0000 7f8df8d68700 1 --2- 172.21.15.104:0/1512955541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c7392f60 0x5606c738c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.748+0000 7f8df8d68700 1 -- 172.21.15.104:0/1512955541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392640 msgr2=0x5606c7392a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.748+0000 7f8df8d68700 1 --2- 172.21.15.104:0/1512955541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392640 0x5606c7392a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.748+0000 7f8df8d68700 1 -- 172.21.15.104:0/1512955541 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606c727cf50 con 0x5606c738cf20 2024-06-09T22:24:26.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.748+0000 7f8df3fff700 1 --2- 172.21.15.104:0/1512955541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392640 0x5606c7392a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.748+0000 7f8df8d68700 1 --2- 172.21.15.104:0/1512955541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c738cf20 0x5606c738d340 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0999d0 tx=0x7f8dec0993b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b26ea63e364a49 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.748+0000 7f8dea7fc700 1 -- 172.21.15.104:0/1512955541 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec09d700 con 0x5606c738cf20 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.748+0000 7f8dea7fc700 1 -- 172.21.15.104:0/1512955541 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dec09d000 con 0x5606c738cf20 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.748+0000 7f8dea7fc700 1 -- 172.21.15.104:0/1512955541 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec02d350 con 0x5606c738cf20 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.749+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1512955541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c738cf20 msgr2=0x5606c738d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.749+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/1512955541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c738cf20 0x5606c738d340 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0999d0 tx=0x7f8dec0993b0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.749+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1512955541 shutdown_connections 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.749+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/1512955541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c738cf20 0x5606c738d340 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.749+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/1512955541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392640 0x5606c7392a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.749+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/1512955541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c7392f60 0x5606c738c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.749+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1512955541 >> 172.21.15.104:0/1512955541 conn(0x5606c72b7cc0 msgr2=0x5606c7387850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.749+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1512955541 shutdown_connections 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.749+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1512955541 wait complete. 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.749+0000 7f8e0504eb80 1 -- start start 2024-06-09T22:24:26.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8e0504eb80 1 Processor -- start 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8e0504eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c738cf20 0x5606c7398d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8e0504eb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392640 0x5606c7396050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8e0504eb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392f60 0x5606c7396590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8e0504eb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5606c73996f0 con 0x5606c7392640 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8e0504eb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5606c727ebf0 con 0x5606c7392f60 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8e0504eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5606c7399870 con 0x5606c738cf20 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8e0504eb80 10 client.0 _ll_register_callbacks cb 0x5606c7278920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8debfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392640 0x5606c7396050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8debfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392640 0x5606c7396050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54214/0 (socket says 172.21.15.104:54214) 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8debfff700 1 -- 172.21.15.104:0/1963863842 learned_addr learned my addr 172.21.15.104:0/1963863842 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8df8d68700 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392f60 0x5606c7396590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8df3fff700 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c738cf20 0x5606c7398d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8debfff700 1 -- 172.21.15.104:0/1963863842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c738cf20 msgr2=0x5606c7398d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8debfff700 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c738cf20 0x5606c7398d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8debfff700 1 -- 172.21.15.104:0/1963863842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392f60 msgr2=0x5606c7396590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8debfff700 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392f60 0x5606c7396590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8debfff700 1 -- 172.21.15.104:0/1963863842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606c722d8d0 con 0x5606c7392640 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8debfff700 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392640 0x5606c7396050 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00bef0 tx=0x7f8ddc012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd98f79e951465d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.750+0000 7f8dd37fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8dd37fe700 1 -- 172.21.15.104:0/1963863842 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc004240 con 0x5606c7392640 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8dd37fe700 1 -- 172.21.15.104:0/1963863842 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8ddc0043e0 con 0x5606c7392640 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1963863842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dec09d000 con 0x5606c7392640 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8dd37fe700 1 -- 172.21.15.104:0/1963863842 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8ddc003680 con 0x5606c7392640 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8e0504eb80 10 client.16482 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1963863842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5606c73a4080 con 0x5606c7392640 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8dd37fe700 1 -- 172.21.15.104:0/1963863842 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dec09d000 con 0x5606c7392640 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8dd37fe700 1 -- 172.21.15.104:0/1963863842 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8ddc019020 con 0x5606c7392640 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8e0504eb80 10 client.16482 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:26.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8e0504eb80 10 client.16482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8e0504eb80 20 client.16482 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5606c73a66c0 0x5606c73a8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8e0504eb80 4 client.16482 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1963863842 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606c72ebd10 con 0x5606c73a66c0 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.751+0000 7f8df3fff700 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5606c73a66c0 0x5606c73a8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.752+0000 7f8df3fff700 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5606c73a66c0 0x5606c73a8ab0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ec1bf70c8f8ed56 server_cookie=21ad12214c4ec661 in_seq=0 out_seq=0 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.752+0000 7f8dd37fe700 10 client.16482 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8dd37fe700 1 -- 172.21.15.104:0/1963863842 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5606c72ebd10 con 0x5606c73a66c0 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8dd37fe700 10 client.16482 handle_command_reply: tid=0 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8e0504eb80 1 client.16482 shutdown 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1963863842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392640 msgr2=0x5606c7396050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392640 0x5606c7396050 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00bef0 tx=0x7f8ddc012730 comp rx=0 tx=0).stop 2024-06-09T22:24:26.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1963863842 shutdown_connections 2024-06-09T22:24:26.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5606c73a66c0 0x5606c73a8ab0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392f60 0x5606c7396590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c738cf20 0x5606c7398d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/1963863842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392640 0x5606c7396050 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1963863842 >> 172.21.15.104:0/1963863842 conn(0x5606c72b7cc0 msgr2=0x5606c738b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:26.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8dd37fe700 0 client.16482 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:26.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1963863842 shutdown_connections 2024-06-09T22:24:26.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8e0504eb80 1 -- 172.21.15.104:0/1963863842 wait complete. 2024-06-09T22:24:26.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.753+0000 7f8e0504eb80 20 client.16482 trim_cache size 0 max 16384 2024-06-09T22:24:26.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.768+0000 7f8e0504eb80 1 Processor -- start 2024-06-09T22:24:26.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.768+0000 7f8e0504eb80 1 -- start start 2024-06-09T22:24:26.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.768+0000 7f8e0504eb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c738cf20 0x5606c738c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.768+0000 7f8e0504eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c7392640 0x5606c7390800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.768+0000 7f8e0504eb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 0x5606c73a2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.768+0000 7f8e0504eb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41b06c0 con 0x5606c7392f60 2024-06-09T22:24:26.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.768+0000 7f8e0504eb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41b2260 con 0x5606c738cf20 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.768+0000 7f8e0504eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41b0d90 con 0x5606c7392640 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.768+0000 7f8df8d68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 0x5606c73a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.768+0000 7f8df8d68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 0x5606c73a2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54224/0 (socket says 172.21.15.104:54224) 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.769+0000 7f8df8d68700 1 -- 172.21.15.104:0/3849039262 learned_addr learned my addr 172.21.15.104:0/3849039262 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.769+0000 7f8debfff700 1 --2- 172.21.15.104:0/3849039262 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c7392640 0x5606c7390800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.769+0000 7f8df3fff700 1 --2- 172.21.15.104:0/3849039262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c738cf20 0x5606c738c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.769+0000 7f8df8d68700 1 -- 172.21.15.104:0/3849039262 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c7392640 msgr2=0x5606c7390800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.769+0000 7f8df8d68700 1 --2- 172.21.15.104:0/3849039262 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c7392640 0x5606c7390800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.769+0000 7f8df8d68700 1 -- 172.21.15.104:0/3849039262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c738cf20 msgr2=0x5606c738c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.769+0000 7f8df8d68700 1 --2- 172.21.15.104:0/3849039262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c738cf20 0x5606c738c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.769+0000 7f8df8d68700 1 -- 172.21.15.104:0/3849039262 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606c73945c0 con 0x5606c7392f60 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.769+0000 7f8df8d68700 1 --2- 172.21.15.104:0/3849039262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 0x5606c73a2950 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f8dec009fd0 tx=0x7f8dec0099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58c9f69703cdf01 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:26.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.769+0000 7f8de9ffb700 1 -- 172.21.15.104:0/3849039262 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec0b7070 con 0x5606c7392f60 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.769+0000 7f8de9ffb700 1 -- 172.21.15.104:0/3849039262 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dec09d700 con 0x5606c7392f60 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.769+0000 7f8de9ffb700 1 -- 172.21.15.104:0/3849039262 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec099670 con 0x5606c7392f60 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.770+0000 7f8e0504eb80 1 -- 172.21.15.104:0/3849039262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 msgr2=0x5606c73a2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.770+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/3849039262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 0x5606c73a2950 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f8dec009fd0 tx=0x7f8dec0099e0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.770+0000 7f8e0504eb80 1 -- 172.21.15.104:0/3849039262 shutdown_connections 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.770+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/3849039262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 0x5606c73a2950 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.770+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/3849039262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c738cf20 0x5606c738c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.770+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/3849039262 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c7392640 0x5606c7390800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.770+0000 7f8e0504eb80 1 -- 172.21.15.104:0/3849039262 >> 172.21.15.104:0/3849039262 conn(0x5606c72b7cc0 msgr2=0x5606c7394940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.770+0000 7f8e0504eb80 1 -- 172.21.15.104:0/3849039262 shutdown_connections 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.770+0000 7f8e0504eb80 1 -- 172.21.15.104:0/3849039262 wait complete. 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.770+0000 7f8e0504eb80 1 -- start start 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.770+0000 7f8e0504eb80 1 Processor -- start 2024-06-09T22:24:26.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8e0504eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c738cf20 0x5606c739ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8e0504eb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392640 0x5606c739f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8e0504eb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 0x5606c73ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8e0504eb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41b1b30 con 0x5606c7392f60 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8e0504eb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41b1460 con 0x5606c7392640 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8e0504eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41ae640 con 0x5606c738cf20 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8e0504eb80 10 client.0 _ll_register_callbacks cb 0x5606c7278920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8df8d68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 0x5606c73ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8df8d68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 0x5606c73ad9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54234/0 (socket says 172.21.15.104:54234) 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8df8d68700 1 -- 172.21.15.104:0/882959664 learned_addr learned my addr 172.21.15.104:0/882959664 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8debfff700 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392640 0x5606c739f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8df3fff700 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c738cf20 0x5606c739ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8df8d68700 1 -- 172.21.15.104:0/882959664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c738cf20 msgr2=0x5606c739ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8df8d68700 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c738cf20 0x5606c739ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8df8d68700 1 -- 172.21.15.104:0/882959664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392640 msgr2=0x5606c739f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:26.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8df8d68700 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392640 0x5606c739f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8df8d68700 1 -- 172.21.15.104:0/882959664 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606c73ae260 con 0x5606c7392f60 2024-06-09T22:24:26.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8df8d68700 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 0x5606c73ad9f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f8dec009e60 tx=0x7f8dec004380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21df2fb300267a1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:26.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.771+0000 7f8dd37fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:26.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8dd37fe700 1 -- 172.21.15.104:0/882959664 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec0b7070 con 0x5606c7392f60 2024-06-09T22:24:26.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8dd37fe700 1 -- 172.21.15.104:0/882959664 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f8dec09d260 con 0x5606c7392f60 2024-06-09T22:24:26.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8e0504eb80 1 -- 172.21.15.104:0/882959664 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dec09d700 con 0x5606c7392f60 2024-06-09T22:24:26.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8dd37fe700 1 -- 172.21.15.104:0/882959664 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8dec02d550 con 0x5606c7392f60 2024-06-09T22:24:26.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8e0504eb80 10 client.16488 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8e0504eb80 1 -- 172.21.15.104:0/882959664 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5606c7410f30 con 0x5606c7392f60 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8dd37fe700 1 -- 172.21.15.104:0/882959664 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dec09d700 con 0x5606c7392f60 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8dd37fe700 1 -- 172.21.15.104:0/882959664 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8dec011070 con 0x5606c7392f60 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8e0504eb80 10 client.16488 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8e0504eb80 10 client.16488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8e0504eb80 20 client.16488 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5606c7411250 0x5606c7413670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8e0504eb80 4 client.16488 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8e0504eb80 1 -- 172.21.15.104:0/882959664 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5606c72e6290 con 0x5606c7411250 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.772+0000 7f8df3fff700 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5606c7411250 0x5606c7413670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8df3fff700 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5606c7411250 0x5606c7413670 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6ce63439477986d server_cookie=9255bbdcabec84e2 in_seq=0 out_seq=0 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8dd37fe700 10 client.16488 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8dd37fe700 1 -- 172.21.15.104:0/882959664 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5606c72e6290 con 0x5606c7411250 2024-06-09T22:24:26.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8dd37fe700 10 client.16488 handle_command_reply: tid=0 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8e0504eb80 1 client.16488 shutdown 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8e0504eb80 1 -- 172.21.15.104:0/882959664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 msgr2=0x5606c73ad9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 0x5606c73ad9f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f8dec009e60 tx=0x7f8dec004380 comp rx=0 tx=0).stop 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8e0504eb80 1 -- 172.21.15.104:0/882959664 shutdown_connections 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5606c7411250 0x5606c7413670 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5606c7392f60 0x5606c73ad9f0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5606c738cf20 0x5606c739ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8e0504eb80 1 --2- 172.21.15.104:0/882959664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606c7392640 0x5606c739f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8e0504eb80 1 -- 172.21.15.104:0/882959664 >> 172.21.15.104:0/882959664 conn(0x5606c72b7cc0 msgr2=0x5606c7380e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8dd37fe700 0 client.16488 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8e0504eb80 1 -- 172.21.15.104:0/882959664 shutdown_connections 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8e0504eb80 1 -- 172.21.15.104:0/882959664 wait complete. 2024-06-09T22:24:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.773+0000 7f8e0504eb80 20 client.16488 trim_cache size 0 max 16384 2024-06-09T22:24:26.781 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:24:26.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.774+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/2415653932 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8de00a2d90 msgr2=0x7f8de00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:26.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.774+0000 7f8dfa7cb700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8de00a2d90 0x7f8de00a5250 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8de4006fd0 tx=0x7f8de4008040 comp rx=0 tx=0).stop 2024-06-09T22:24:26.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.774+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/2415653932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df41086f0 msgr2=0x7f8df41026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:26.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.774+0000 7f8dfa7cb700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df41086f0 0x7f8df41026e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0062a0 tx=0x7f8dec004e20 comp rx=0 tx=0).stop 2024-06-09T22:24:26.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.774+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/2415653932 shutdown_connections 2024-06-09T22:24:26.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.774+0000 7f8dfa7cb700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f8de00a2d90 0x7f8de00a5250 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.774+0000 7f8dfa7cb700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df41086f0 0x7f8df41026e0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.774+0000 7f8dfa7cb700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8df4100a80 0x7f8df4101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.774+0000 7f8dfa7cb700 1 --2- 172.21.15.104:0/2415653932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8df4101440 0x7f8df41021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:26.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.774+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/2415653932 >> 172.21.15.104:0/2415653932 conn(0x7f8df4077a00 msgr2=0x7f8df40fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:26.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.774+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/2415653932 shutdown_connections 2024-06-09T22:24:26.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:26.774+0000 7f8dfa7cb700 1 -- 172.21.15.104:0/2415653932 wait complete. 2024-06-09T22:24:26.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:26 smithi153 ceph-mon[36599]: pgmap v107: 129 pgs: 129 active+clean; 51 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 965 KiB/s wr, 96 op/s 2024-06-09T22:24:27.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:26 smithi104 ceph-mon[32389]: pgmap v107: 129 pgs: 129 active+clean; 51 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 965 KiB/s wr, 96 op/s 2024-06-09T22:24:27.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:26 smithi194 ceph-mon[36856]: pgmap v107: 129 pgs: 129 active+clean; 51 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 965 KiB/s wr, 96 op/s 2024-06-09T22:24:27.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 57503 3.52 MB/sec warmup 93 sec latency 1559.791 ms 2024-06-09T22:24:28.605 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-09T22:24:28.606 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-09T22:24:28.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 57503 3.48 MB/sec warmup 94 sec latency 2559.855 ms 2024-06-09T22:24:28.860 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:24:28.982 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:28 smithi104 ceph-mon[32389]: pgmap v108: 129 pgs: 129 active+clean; 51 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 965 KiB/s wr, 96 op/s 2024-06-09T22:24:28.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:28 smithi153 ceph-mon[36599]: pgmap v108: 129 pgs: 129 active+clean; 51 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 965 KiB/s wr, 96 op/s 2024-06-09T22:24:29.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:28 smithi194 ceph-mon[36856]: pgmap v108: 129 pgs: 129 active+clean; 51 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 965 KiB/s wr, 96 op/s 2024-06-09T22:24:29.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 57503 3.44 MB/sec warmup 95 sec latency 3559.884 ms 2024-06-09T22:24:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.451+0000 7fc9d5902640 1 -- 172.21.15.104:0/3217465904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d0106890 msgr2=0x7fc9d0106c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.451+0000 7fc9d5902640 1 --2- 172.21.15.104:0/3217465904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d0106890 0x7fc9d0106c70 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0009cd0 tx=0x7fc9c002a8e0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.451+0000 7fc9d5902640 1 -- 172.21.15.104:0/3217465904 shutdown_connections 2024-06-09T22:24:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.451+0000 7fc9d5902640 1 --2- 172.21.15.104:0/3217465904 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9d0101240 0x7fc9d010d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.451+0000 7fc9d5902640 1 --2- 172.21.15.104:0/3217465904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9d0100880 0x7fc9d0100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.451+0000 7fc9d5902640 1 --2- 172.21.15.104:0/3217465904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d0106890 0x7fc9d0106c70 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.451+0000 7fc9d5902640 1 -- 172.21.15.104:0/3217465904 >> 172.21.15.104:0/3217465904 conn(0x7fc9d00fc370 msgr2=0x7fc9d00fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.451+0000 7fc9d5902640 1 -- 172.21.15.104:0/3217465904 shutdown_connections 2024-06-09T22:24:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.451+0000 7fc9d5902640 1 -- 172.21.15.104:0/3217465904 wait complete. 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9d5902640 1 Processor -- start 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9d5902640 1 -- start start 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9d5902640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9d0100880 0x7fc9d00fc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9d5902640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d0101240 0x7fc9d00fcd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9d5902640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9d0106890 0x7fc9d00fd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9d5902640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d0110ca0 con 0x7fc9d0101240 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9d5902640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d0110b20 con 0x7fc9d0106890 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9d5902640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d0110e20 con 0x7fc9d0100880 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9ce7fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d0101240 0x7fc9d00fcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9ce7fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d0101240 0x7fc9d00fcd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54266/0 (socket says 172.21.15.104:54266) 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9ce7fc640 1 -- 172.21.15.104:0/1094834235 learned_addr learned my addr 172.21.15.104:0/1094834235 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9d0106890 0x7fc9d00fd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9ceffd640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9d0100880 0x7fc9d00fc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9ce7fc640 1 -- 172.21.15.104:0/1094834235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9d0100880 msgr2=0x7fc9d00fc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9ce7fc640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9d0100880 0x7fc9d00fc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9ce7fc640 1 -- 172.21.15.104:0/1094834235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9d0106890 msgr2=0x7fc9d00fd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9ce7fc640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9d0106890 0x7fc9d00fd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9ce7fc640 1 -- 172.21.15.104:0/1094834235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d01a66f0 con 0x7fc9d0101240 2024-06-09T22:24:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.452+0000 7fc9ceffd640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9d0100880 0x7fc9d00fc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:24:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.453+0000 7fc9ce7fc640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d0101240 0x7fc9d00fcd60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400e9f0 tx=0x7fc9c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.453+0000 7fc9d4900640 1 -- 172.21.15.104:0/1094834235 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c400cd80 con 0x7fc9d0101240 2024-06-09T22:24:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.453+0000 7fc9d4900640 1 -- 172.21.15.104:0/1094834235 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c4010910 con 0x7fc9d0101240 2024-06-09T22:24:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.453+0000 7fc9d5902640 1 -- 172.21.15.104:0/1094834235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d01a69e0 con 0x7fc9d0101240 2024-06-09T22:24:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.453+0000 7fc9d4900640 1 -- 172.21.15.104:0/1094834235 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4018990 con 0x7fc9d0101240 2024-06-09T22:24:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.453+0000 7fc9d5902640 1 -- 172.21.15.104:0/1094834235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d01a6f30 con 0x7fc9d0101240 2024-06-09T22:24:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.455+0000 7fc9d4900640 1 -- 172.21.15.104:0/1094834235 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc9c4010ac0 con 0x7fc9d0101240 2024-06-09T22:24:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.455+0000 7fc9d8605740 1 Processor -- start 2024-06-09T22:24:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.455+0000 7fc9d8605740 1 -- start start 2024-06-09T22:24:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9d8605740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919d170 0x556a3919d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9d8605740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919da90 0x556a391a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9d8605740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a391a8350 0x556a391aa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9d8605740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x556a391014f0 con 0x556a3919d170 2024-06-09T22:24:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9d8605740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556a3909f870 con 0x556a3919da90 2024-06-09T22:24:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9d8605740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556a390fd4b0 con 0x556a391a8350 2024-06-09T22:24:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9d4900640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc9b80a2da0 0x7fc9b80a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9cf7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919d170 0x556a3919d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9cf7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919d170 0x556a3919d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54276/0 (socket says 172.21.15.104:54276) 2024-06-09T22:24:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9cf7fe640 1 -- 172.21.15.104:0/3124199423 learned_addr learned my addr 172.21.15.104:0/3124199423 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9d4900640 1 -- 172.21.15.104:0/1094834235 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fc9c40cfc30 con 0x7fc9d0101240 2024-06-09T22:24:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9ce7fc640 1 --2- 172.21.15.104:0/3124199423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a391a8350 0x556a391aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc9b80a2da0 0x7fc9b80a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9ceffd640 1 --2- 172.21.15.104:0/3124199423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919da90 0x556a391a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9cf7fe640 1 -- 172.21.15.104:0/3124199423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a391a8350 msgr2=0x556a391aa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/3124199423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a391a8350 0x556a391aa740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9cf7fe640 1 -- 172.21.15.104:0/3124199423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919da90 msgr2=0x556a391a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/3124199423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919da90 0x556a391a7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9cf7fe640 1 -- 172.21.15.104:0/3124199423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a390bb9b0 con 0x556a3919d170 2024-06-09T22:24:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9ce7fc640 1 --2- 172.21.15.104:0/3124199423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a391a8350 0x556a391aa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/3124199423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919d170 0x556a3919d550 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc9c80137c0 tx=0x7fc9c8013c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ed8f1d86ea2e6b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9b5ffb640 1 -- 172.21.15.104:0/3124199423 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c801d7c0 con 0x556a3919d170 2024-06-09T22:24:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9b5ffb640 1 -- 172.21.15.104:0/3124199423 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c8022070 con 0x556a3919d170 2024-06-09T22:24:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.456+0000 7fc9b5ffb640 1 -- 172.21.15.104:0/3124199423 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c801c920 con 0x556a3919d170 2024-06-09T22:24:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.457+0000 7fc9d8605740 1 -- 172.21.15.104:0/3124199423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919d170 msgr2=0x556a3919d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.457+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc9b80a2da0 0x7fc9b80a5260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8021000 tx=0x7fc9c801f000 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.457+0000 7fc9d8605740 1 --2- 172.21.15.104:0/3124199423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919d170 0x556a3919d550 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc9c80137c0 tx=0x7fc9c8013c80 comp rx=0 tx=0).stop 2024-06-09T22:24:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.457+0000 7fc9d8605740 1 -- 172.21.15.104:0/3124199423 shutdown_connections 2024-06-09T22:24:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.457+0000 7fc9d8605740 1 --2- 172.21.15.104:0/3124199423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a391a8350 0x556a391aa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.457+0000 7fc9d8605740 1 --2- 172.21.15.104:0/3124199423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919da90 0x556a391a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.457+0000 7fc9d8605740 1 --2- 172.21.15.104:0/3124199423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919d170 0x556a3919d550 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.457+0000 7fc9d8605740 1 -- 172.21.15.104:0/3124199423 >> 172.21.15.104:0/3124199423 conn(0x556a39199680 msgr2=0x556a39199a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.457+0000 7fc9d8605740 1 -- 172.21.15.104:0/3124199423 shutdown_connections 2024-06-09T22:24:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.457+0000 7fc9d8605740 1 -- 172.21.15.104:0/3124199423 wait complete. 2024-06-09T22:24:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9d8605740 1 -- start start 2024-06-09T22:24:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9d8605740 1 Processor -- start 2024-06-09T22:24:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9d8605740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919d170 0x556a391a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9d8605740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919da90 0x556a391a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9d8605740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a391a8350 0x556a391a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9d8605740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x556a391aaf80 con 0x556a3919da90 2024-06-09T22:24:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9d8605740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556a3908aae0 con 0x556a391a8350 2024-06-09T22:24:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9d8605740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556a391ab100 con 0x556a3919d170 2024-06-09T22:24:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9d8605740 10 client.0 _ll_register_callbacks cb 0x556a390c39b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9ce7fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919da90 0x556a391a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9ce7fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919da90 0x556a391a57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54282/0 (socket says 172.21.15.104:54282) 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9ce7fc640 1 -- 172.21.15.104:0/1229669173 learned_addr learned my addr 172.21.15.104:0/1229669173 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a391a8350 0x556a391a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9ceffd640 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919d170 0x556a391a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9ce7fc640 1 -- 172.21.15.104:0/1229669173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919d170 msgr2=0x556a391a52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9ce7fc640 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919d170 0x556a391a52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9ce7fc640 1 -- 172.21.15.104:0/1229669173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a391a8350 msgr2=0x556a391a5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9ce7fc640 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a391a8350 0x556a391a5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.458+0000 7fc9ce7fc640 1 -- 172.21.15.104:0/1229669173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a38e15210 con 0x556a3919da90 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9ce7fc640 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919da90 0x556a391a57e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400c6a0 tx=0x7fc9c409c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b92c83fd4eb3bb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9aeffd640 1 -- 172.21.15.104:0/1229669173 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c40cd4a0 con 0x556a3919da90 2024-06-09T22:24:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9aeffd640 1 -- 172.21.15.104:0/1229669173 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fc9c4018d30 con 0x556a3919da90 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9d8605740 1 -- 172.21.15.104:0/1229669173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9c8022070 con 0x556a3919da90 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9aeffd640 1 -- 172.21.15.104:0/1229669173 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc9c409b230 con 0x556a3919da90 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9d8605740 10 client.16500 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9d8605740 1 -- 172.21.15.104:0/1229669173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556a38ee1620 con 0x556a3919da90 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9aeffd640 1 -- 172.21.15.104:0/1229669173 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c8022070 con 0x556a3919da90 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9aeffd640 1 -- 172.21.15.104:0/1229669173 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fc9c409aca0 con 0x556a3919da90 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9d8605740 10 client.16500 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9d8605740 10 client.16500 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9d8605740 20 client.16500 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9d8605740 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a391b8480 0x556a391ba870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9d8605740 4 client.16500 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.459+0000 7fc9d8605740 1 -- 172.21.15.104:0/1229669173 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a39114f10 con 0x556a391b8480 2024-06-09T22:24:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.460+0000 7fc9ceffd640 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a391b8480 0x556a391ba870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.460+0000 7fc9aeffd640 10 client.16500 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.460+0000 7fc9ceffd640 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a391b8480 0x556a391ba870 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5faab7448322b1ad server_cookie=609909f3bc562637 in_seq=0 out_seq=0 2024-06-09T22:24:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9aeffd640 1 -- 172.21.15.104:0/1229669173 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x556a39114f10 con 0x556a391b8480 2024-06-09T22:24:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9aeffd640 10 client.16500 handle_command_reply: tid=0 2024-06-09T22:24:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9d8605740 1 client.16500 shutdown 2024-06-09T22:24:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9d8605740 1 -- 172.21.15.104:0/1229669173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919da90 msgr2=0x556a391a57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9d8605740 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919da90 0x556a391a57e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400c6a0 tx=0x7fc9c409c620 comp rx=0 tx=0).stop 2024-06-09T22:24:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9d8605740 1 -- 172.21.15.104:0/1229669173 shutdown_connections 2024-06-09T22:24:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9d8605740 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a391b8480 0x556a391ba870 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9d8605740 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a391a8350 0x556a391a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9d8605740 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a3919da90 0x556a391a57e0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9aeffd640 0 client.16500 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9d8605740 1 --2- 172.21.15.104:0/1229669173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919d170 0x556a391a52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9d8605740 1 -- 172.21.15.104:0/1229669173 >> 172.21.15.104:0/1229669173 conn(0x556a39199680 msgr2=0x556a3919f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9d8605740 1 -- 172.21.15.104:0/1229669173 shutdown_connections 2024-06-09T22:24:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9d8605740 1 -- 172.21.15.104:0/1229669173 wait complete. 2024-06-09T22:24:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.461+0000 7fc9d8605740 20 client.16500 trim_cache size 0 max 16384 2024-06-09T22:24:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.476+0000 7fc9d8605740 1 Processor -- start 2024-06-09T22:24:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.476+0000 7fc9d8605740 1 -- start start 2024-06-09T22:24:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.476+0000 7fc9d8605740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919d170 0x556a3919fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.476+0000 7fc9d8605740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919da90 0x556a391a71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.476+0000 7fc9d8605740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 0x556a391c0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.476+0000 7fc9d8605740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01b1060 con 0x556a391a8350 2024-06-09T22:24:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.476+0000 7fc9d8605740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01b2c00 con 0x556a3919da90 2024-06-09T22:24:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.476+0000 7fc9d8605740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01b1730 con 0x556a3919d170 2024-06-09T22:24:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.476+0000 7fc9cf7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 0x556a391c0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.476+0000 7fc9cf7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 0x556a391c0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54286/0 (socket says 172.21.15.104:54286) 2024-06-09T22:24:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.476+0000 7fc9cf7fe640 1 -- 172.21.15.104:0/653141475 learned_addr learned my addr 172.21.15.104:0/653141475 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9ceffd640 1 --2- 172.21.15.104:0/653141475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919d170 0x556a3919fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9cf7fe640 1 -- 172.21.15.104:0/653141475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919d170 msgr2=0x556a3919fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9ce7fc640 1 --2- 172.21.15.104:0/653141475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919da90 0x556a391a71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/653141475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919d170 0x556a3919fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9cf7fe640 1 -- 172.21.15.104:0/653141475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919da90 msgr2=0x556a391a71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/653141475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919da90 0x556a391a71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9cf7fe640 1 -- 172.21.15.104:0/653141475 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a391babe0 con 0x556a391a8350 2024-06-09T22:24:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9ceffd640 1 --2- 172.21.15.104:0/653141475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919d170 0x556a3919fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:24:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/653141475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 0x556a391c0160 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc9c80137c0 tx=0x7fc9c801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a07827cddf31214 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9b57fa640 1 -- 172.21.15.104:0/653141475 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8034070 con 0x556a391a8350 2024-06-09T22:24:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9b57fa640 1 -- 172.21.15.104:0/653141475 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c8022030 con 0x556a391a8350 2024-06-09T22:24:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9b57fa640 1 -- 172.21.15.104:0/653141475 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8005050 con 0x556a391a8350 2024-06-09T22:24:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9d8605740 1 -- 172.21.15.104:0/653141475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 msgr2=0x556a391c0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.477+0000 7fc9d8605740 1 --2- 172.21.15.104:0/653141475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 0x556a391c0160 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc9c80137c0 tx=0x7fc9c801c500 comp rx=0 tx=0).stop 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 -- 172.21.15.104:0/653141475 shutdown_connections 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 --2- 172.21.15.104:0/653141475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 0x556a391c0160 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 --2- 172.21.15.104:0/653141475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919da90 0x556a391a71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 --2- 172.21.15.104:0/653141475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919d170 0x556a3919fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 -- 172.21.15.104:0/653141475 >> 172.21.15.104:0/653141475 conn(0x556a39199680 msgr2=0x556a391a6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 -- 172.21.15.104:0/653141475 shutdown_connections 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 -- 172.21.15.104:0/653141475 wait complete. 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 -- start start 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 Processor -- start 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919d170 0x556a391a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919da90 0x556a391a2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 0x556a391a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01b24d0 con 0x556a391a8350 2024-06-09T22:24:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01b1e00 con 0x556a3919d170 2024-06-09T22:24:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01af020 con 0x556a3919da90 2024-06-09T22:24:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9d8605740 10 client.0 _ll_register_callbacks cb 0x7fc9c4009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9cf7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 0x556a391a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9cf7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 0x556a391a2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54298/0 (socket says 172.21.15.104:54298) 2024-06-09T22:24:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9cf7fe640 1 -- 172.21.15.104:0/2459091251 learned_addr learned my addr 172.21.15.104:0/2459091251 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.478+0000 7fc9ce7fc640 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919da90 0x556a391a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9ceffd640 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919d170 0x556a391a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9cf7fe640 1 -- 172.21.15.104:0/2459091251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919da90 msgr2=0x556a391a2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919da90 0x556a391a2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9cf7fe640 1 -- 172.21.15.104:0/2459091251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919d170 msgr2=0x556a391a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919d170 0x556a391a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9cf7fe640 1 -- 172.21.15.104:0/2459091251 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a39222160 con 0x556a391a8350 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9ceffd640 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919d170 0x556a391a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9cf7fe640 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 0x556a391a2e40 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8011750 tx=0x7fc9c8011fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c3151f0636ca4a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9aeffd640 1 -- 172.21.15.104:0/2459091251 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8034070 con 0x556a391a8350 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9aeffd640 1 -- 172.21.15.104:0/2459091251 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fc9c801cc40 con 0x556a391a8350 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9d8605740 1 -- 172.21.15.104:0/2459091251 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9c8022030 con 0x556a391a8350 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9aeffd640 1 -- 172.21.15.104:0/2459091251 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc9c801c910 con 0x556a391a8350 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9d8605740 10 client.16506 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9d8605740 1 -- 172.21.15.104:0/2459091251 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556a392233c0 con 0x556a391a8350 2024-06-09T22:24:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9aeffd640 1 -- 172.21.15.104:0/2459091251 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c8022030 con 0x556a391a8350 2024-06-09T22:24:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9aeffd640 1 -- 172.21.15.104:0/2459091251 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fc9c8040030 con 0x556a391a8350 2024-06-09T22:24:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9d8605740 10 client.16506 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9d8605740 10 client.16506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9d8605740 20 client.16506 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.479+0000 7fc9d8605740 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a392238e0 0x556a39225d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.480+0000 7fc9d8605740 4 client.16506 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:24:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.480+0000 7fc9d8605740 1 -- 172.21.15.104:0/2459091251 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556a3910ec20 con 0x556a392238e0 2024-06-09T22:24:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.480+0000 7fc9ceffd640 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a392238e0 0x556a39225d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.480+0000 7fc9aeffd640 10 client.16506 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.480+0000 7fc9ceffd640 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a392238e0 0x556a39225d00 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6e35fdeb7cca2d8 server_cookie=9c421e380095d003 in_seq=0 out_seq=0 2024-06-09T22:24:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.489+0000 7fc9aeffd640 1 -- 172.21.15.104:0/2459091251 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x556a3910ec20 con 0x556a392238e0 2024-06-09T22:24:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.489+0000 7fc9aeffd640 10 client.16506 handle_command_reply: tid=0 2024-06-09T22:24:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.489+0000 7fc9d8605740 1 client.16506 shutdown 2024-06-09T22:24:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d8605740 1 -- 172.21.15.104:0/2459091251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 msgr2=0x556a391a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d8605740 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 0x556a391a2e40 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8011750 tx=0x7fc9c8011fc0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d8605740 1 -- 172.21.15.104:0/2459091251 shutdown_connections 2024-06-09T22:24:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d8605740 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a392238e0 0x556a39225d00 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d8605740 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a391a8350 0x556a391a2e40 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d8605740 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a3919da90 0x556a391a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d8605740 1 --2- 172.21.15.104:0/2459091251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a3919d170 0x556a391a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9aeffd640 0 client.16506 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d8605740 1 -- 172.21.15.104:0/2459091251 >> 172.21.15.104:0/2459091251 conn(0x556a39199680 msgr2=0x556a391a5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d8605740 1 -- 172.21.15.104:0/2459091251 shutdown_connections 2024-06-09T22:24:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d8605740 1 -- 172.21.15.104:0/2459091251 wait complete. 2024-06-09T22:24:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d8605740 20 client.16506 trim_cache size 0 max 16384 2024-06-09T22:24:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:24:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:24:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:24:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d5902640 1 -- 172.21.15.104:0/1094834235 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc9b80a2da0 msgr2=0x7fc9b80a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d5902640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc9b80a2da0 0x7fc9b80a5260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8021000 tx=0x7fc9c801f000 comp rx=0 tx=0).stop 2024-06-09T22:24:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d5902640 1 -- 172.21.15.104:0/1094834235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d0101240 msgr2=0x7fc9d00fcd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d5902640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d0101240 0x7fc9d00fcd60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400e9f0 tx=0x7fc9c400eeb0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d5902640 1 -- 172.21.15.104:0/1094834235 shutdown_connections 2024-06-09T22:24:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d5902640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc9b80a2da0 0x7fc9b80a5260 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d5902640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9d0106890 0x7fc9d00fd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d5902640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d0101240 0x7fc9d00fcd60 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d5902640 1 --2- 172.21.15.104:0/1094834235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9d0100880 0x7fc9d00fc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.490+0000 7fc9d5902640 1 -- 172.21.15.104:0/1094834235 >> 172.21.15.104:0/1094834235 conn(0x7fc9d00fc370 msgr2=0x7fc9d006a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.491+0000 7fc9d5902640 1 -- 172.21.15.104:0/1094834235 shutdown_connections 2024-06-09T22:24:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:30.491+0000 7fc9d5902640 1 -- 172.21.15.104:0/1094834235 wait complete. 2024-06-09T22:24:30.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 57571 3.41 MB/sec warmup 96 sec latency 3866.124 ms 2024-06-09T22:24:30.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.831+0000 7faf4337e700 1 -- 172.21.15.104:0/759028260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3c102770 msgr2=0x7faf3c102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.831+0000 7faf4337e700 1 --2- 172.21.15.104:0/759028260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3c102770 0x7faf3c102b50 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7faf380099c0 tx=0x7faf38009cd0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.831+0000 7faf4337e700 1 -- 172.21.15.104:0/759028260 shutdown_connections 2024-06-09T22:24:30.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.831+0000 7faf4337e700 1 --2- 172.21.15.104:0/759028260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3c102770 0x7faf3c102b50 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.831+0000 7faf4337e700 1 --2- 172.21.15.104:0/759028260 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf3c107860 0x7faf3c10f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.831+0000 7faf4337e700 1 --2- 172.21.15.104:0/759028260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf3c103120 0x7faf3c107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.831+0000 7faf4337e700 1 -- 172.21.15.104:0/759028260 >> 172.21.15.104:0/759028260 conn(0x7faf3c079600 msgr2=0x7faf3c079a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:30.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.831+0000 7faf4337e700 1 -- 172.21.15.104:0/759028260 shutdown_connections 2024-06-09T22:24:30.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.831+0000 7faf4337e700 1 -- 172.21.15.104:0/759028260 wait complete. 2024-06-09T22:24:30.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.832+0000 7faf4337e700 1 Processor -- start 2024-06-09T22:24:30.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.832+0000 7faf4337e700 1 -- start start 2024-06-09T22:24:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.832+0000 7faf4337e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3c102770 0x7faf3c19e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.832+0000 7faf4337e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf3c103120 0x7faf3c19ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.832+0000 7faf4337e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf3c107860 0x7faf3c198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.832+0000 7faf4337e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c1127a0 con 0x7faf3c102770 2024-06-09T22:24:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.832+0000 7faf4337e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c112620 con 0x7faf3c103120 2024-06-09T22:24:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.832+0000 7faf4337e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c112920 con 0x7faf3c107860 2024-06-09T22:24:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.832+0000 7faf4111a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3c102770 0x7faf3c19e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.832+0000 7faf4111a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3c102770 0x7faf3c19e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54308/0 (socket says 172.21.15.104:54308) 2024-06-09T22:24:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.832+0000 7faf4111a700 1 -- 172.21.15.104:0/899373716 learned_addr learned my addr 172.21.15.104:0/899373716 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.832+0000 7faf40919700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf3c103120 0x7faf3c19ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf4111a700 1 -- 172.21.15.104:0/899373716 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf3c107860 msgr2=0x7faf3c198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf4191b700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf3c107860 0x7faf3c198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf4111a700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf3c107860 0x7faf3c198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf4111a700 1 -- 172.21.15.104:0/899373716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf3c103120 msgr2=0x7faf3c19ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf4111a700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf3c103120 0x7faf3c19ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf4111a700 1 -- 172.21.15.104:0/899373716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3c1990c0 con 0x7faf3c102770 2024-06-09T22:24:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf4191b700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf3c107860 0x7faf3c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:24:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf4111a700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3c102770 0x7faf3c19e7a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7faf38006010 tx=0x7faf38009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf327fc700 1 -- 172.21.15.104:0/899373716 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf38027070 con 0x7faf3c102770 2024-06-09T22:24:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf4337e700 1 -- 172.21.15.104:0/899373716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf3c199350 con 0x7faf3c102770 2024-06-09T22:24:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf327fc700 1 -- 172.21.15.104:0/899373716 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf38003780 con 0x7faf3c102770 2024-06-09T22:24:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf327fc700 1 -- 172.21.15.104:0/899373716 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf38022da0 con 0x7faf3c102770 2024-06-09T22:24:30.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.833+0000 7faf4337e700 1 -- 172.21.15.104:0/899373716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf3c199740 con 0x7faf3c102770 2024-06-09T22:24:30.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.835+0000 7faf327fc700 1 -- 172.21.15.104:0/899373716 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7faf3802a070 con 0x7faf3c102770 2024-06-09T22:24:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.836+0000 7faf327fc700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faf280a2d90 0x7faf280a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.836+0000 7faf40919700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faf280a2d90 0x7faf280a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.836+0000 7faf327fc700 1 -- 172.21.15.104:0/899373716 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7faf380d0470 con 0x7faf3c102770 2024-06-09T22:24:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.836+0000 7faf40919700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faf280a2d90 0x7faf280a5250 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faf3c108bd0 tx=0x7faf2c00a380 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf4dc01b80 1 Processor -- start 2024-06-09T22:24:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf4dc01b80 1 -- start start 2024-06-09T22:24:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf4dc01b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 0x555a1640c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf4dc01b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 0x555a164062e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf4dc01b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a16406c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf4dc01b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x555a162f8fe0 con 0x555a1640c900 2024-06-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf4dc01b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555a163578d0 con 0x555a1640bfe0 2024-06-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf4dc01b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555a162bf4e0 con 0x555a16406820 2024-06-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf40919700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 0x555a164062e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf40919700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 0x555a164062e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54324/0 (socket says 172.21.15.104:54324) 2024-06-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf40919700 1 -- 172.21.15.104:0/3859750259 learned_addr learned my addr 172.21.15.104:0/3859750259 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf4191b700 1 --2- 172.21.15.104:0/3859750259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a16406c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf4111a700 1 --2- 172.21.15.104:0/3859750259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 0x555a1640c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf40919700 1 -- 172.21.15.104:0/3859750259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 msgr2=0x555a16406c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf40919700 1 --2- 172.21.15.104:0/3859750259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a16406c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf40919700 1 -- 172.21.15.104:0/3859750259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 msgr2=0x555a1640c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf40919700 1 --2- 172.21.15.104:0/3859750259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 0x555a1640c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf40919700 1 -- 172.21.15.104:0/3859750259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a162a6950 con 0x555a1640c900 2024-06-09T22:24:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf4191b700 1 --2- 172.21.15.104:0/3859750259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a16406c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.837+0000 7faf40919700 1 --2- 172.21.15.104:0/3859750259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 0x555a164062e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7faf2c00fea0 tx=0x7faf2c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5697866c91fd0c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.838+0000 7faf277fe700 1 -- 172.21.15.104:0/3859750259 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf2c016070 con 0x555a1640c900 2024-06-09T22:24:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.838+0000 7faf277fe700 1 -- 172.21.15.104:0/3859750259 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf2c014070 con 0x555a1640c900 2024-06-09T22:24:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.838+0000 7faf277fe700 1 -- 172.21.15.104:0/3859750259 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf2c01d400 con 0x555a1640c900 2024-06-09T22:24:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.838+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3859750259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 msgr2=0x555a164062e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.838+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3859750259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 0x555a164062e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7faf2c00fea0 tx=0x7faf2c012000 comp rx=0 tx=0).stop 2024-06-09T22:24:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.838+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3859750259 shutdown_connections 2024-06-09T22:24:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.838+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3859750259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a16406c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.838+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3859750259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 0x555a1640c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.838+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3859750259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 0x555a164062e0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.838+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3859750259 >> 172.21.15.104:0/3859750259 conn(0x555a16333100 msgr2=0x555a164012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.839+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3859750259 shutdown_connections 2024-06-09T22:24:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.839+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3859750259 wait complete. 2024-06-09T22:24:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.839+0000 7faf4dc01b80 1 -- start start 2024-06-09T22:24:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.839+0000 7faf4dc01b80 1 Processor -- start 2024-06-09T22:24:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.839+0000 7faf4dc01b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a16406820 0x555a16412620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.839+0000 7faf4dc01b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 0x555a1640f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.839+0000 7faf4dc01b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a1640c900 0x555a1640fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.839+0000 7faf4dc01b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x555a162f8040 con 0x555a16406820 2024-06-09T22:24:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.839+0000 7faf4dc01b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555a16353890 con 0x555a1640bfe0 2024-06-09T22:24:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.839+0000 7faf4dc01b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555a164132e0 con 0x555a1640c900 2024-06-09T22:24:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.839+0000 7faf4dc01b80 10 client.0 _ll_register_callbacks cb 0x555a162ed630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4111a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a16406820 0x555a16412620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4111a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a16406820 0x555a16412620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54336/0 (socket says 172.21.15.104:54336) 2024-06-09T22:24:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4111a700 1 -- 172.21.15.104:0/3672150049 learned_addr learned my addr 172.21.15.104:0/3672150049 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4191b700 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a1640c900 0x555a1640fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf40919700 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 0x555a1640f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4111a700 1 -- 172.21.15.104:0/3672150049 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a1640c900 msgr2=0x555a1640fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4111a700 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a1640c900 0x555a1640fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4111a700 1 -- 172.21.15.104:0/3672150049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 msgr2=0x555a1640f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4111a700 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 0x555a1640f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4111a700 1 -- 172.21.15.104:0/3672150049 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a1631d7d0 con 0x555a16406820 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4111a700 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a16406820 0x555a16412620 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7faf38000c00 tx=0x7faf3809c090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2733f86eed274899 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf24ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf24ff9700 1 -- 172.21.15.104:0/3672150049 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3809cb10 con 0x555a16406820 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3672150049 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf2c014070 con 0x555a16406820 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf24ff9700 1 -- 172.21.15.104:0/3672150049 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7faf380a4e00 con 0x555a16406820 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf24ff9700 1 -- 172.21.15.104:0/3672150049 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faf38087070 con 0x555a16406820 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4dc01b80 10 client.16518 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.840+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3672150049 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555a1641da40 con 0x555a16406820 2024-06-09T22:24:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.841+0000 7faf24ff9700 1 -- 172.21.15.104:0/3672150049 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf2c014070 con 0x555a16406820 2024-06-09T22:24:30.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.841+0000 7faf24ff9700 1 -- 172.21.15.104:0/3672150049 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7faf3809bb00 con 0x555a16406820 2024-06-09T22:24:30.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.841+0000 7faf4dc01b80 10 client.16518 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:30.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.841+0000 7faf4dc01b80 10 client.16518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:30.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.841+0000 7faf4dc01b80 20 client.16518 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:30.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.841+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555a1641ff10 0x555a16422320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.841+0000 7faf4dc01b80 4 client.16518 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:30.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.841+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3672150049 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a16364dd0 con 0x555a1641ff10 2024-06-09T22:24:30.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.841+0000 7faf40919700 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555a1641ff10 0x555a16422320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf40919700 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555a1641ff10 0x555a16422320 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e79db1aa258318db server_cookie=c2f5ea9fb228c913 in_seq=0 out_seq=0 2024-06-09T22:24:30.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf24ff9700 10 client.16518 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:30.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf24ff9700 1 -- 172.21.15.104:0/3672150049 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x555a16364dd0 con 0x555a1641ff10 2024-06-09T22:24:30.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf24ff9700 10 client.16518 handle_command_reply: tid=0 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf4dc01b80 1 client.16518 shutdown 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3672150049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a16406820 msgr2=0x555a16412620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a16406820 0x555a16412620 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7faf38000c00 tx=0x7faf3809c090 comp rx=0 tx=0).stop 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3672150049 shutdown_connections 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555a1641ff10 0x555a16422320 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a16406820 0x555a16412620 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a1640c900 0x555a1640fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3672150049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 0x555a1640f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3672150049 >> 172.21.15.104:0/3672150049 conn(0x555a16333100 msgr2=0x555a16400780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.842+0000 7faf24ff9700 0 client.16518 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.843+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3672150049 shutdown_connections 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.843+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3672150049 wait complete. 2024-06-09T22:24:30.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.843+0000 7faf4dc01b80 20 client.16518 trim_cache size 0 max 16384 2024-06-09T22:24:30.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.863+0000 7faf4dc01b80 1 Processor -- start 2024-06-09T22:24:30.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf4dc01b80 1 -- start start 2024-06-09T22:24:30.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf4dc01b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a1640d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf4dc01b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640bfe0 0x555a16411860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf4dc01b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640c900 0x555a16400f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf4dc01b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c1b0550 con 0x555a1640bfe0 2024-06-09T22:24:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf4dc01b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c1b20f0 con 0x555a1640c900 2024-06-09T22:24:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf4dc01b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c1b0c20 con 0x555a16406820 2024-06-09T22:24:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf40919700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640bfe0 0x555a16411860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf40919700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640bfe0 0x555a16411860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54344/0 (socket says 172.21.15.104:54344) 2024-06-09T22:24:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf40919700 1 -- 172.21.15.104:0/2518302594 learned_addr learned my addr 172.21.15.104:0/2518302594 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf4191b700 1 --2- 172.21.15.104:0/2518302594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640c900 0x555a16400f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf4111a700 1 --2- 172.21.15.104:0/2518302594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a1640d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf40919700 1 -- 172.21.15.104:0/2518302594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 msgr2=0x555a1640d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf40919700 1 --2- 172.21.15.104:0/2518302594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a1640d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf40919700 1 -- 172.21.15.104:0/2518302594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640c900 msgr2=0x555a16400f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf40919700 1 --2- 172.21.15.104:0/2518302594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640c900 0x555a16400f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf40919700 1 -- 172.21.15.104:0/2518302594 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a1640deb0 con 0x555a1640bfe0 2024-06-09T22:24:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf4111a700 1 --2- 172.21.15.104:0/2518302594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a1640d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:24:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf40919700 1 --2- 172.21.15.104:0/2518302594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640bfe0 0x555a16411860 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7faf2c010010 tx=0x7faf2c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1def5d37a3e4fa8c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.864+0000 7faf26ffd700 1 -- 172.21.15.104:0/2518302594 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf2c010430 con 0x555a1640bfe0 2024-06-09T22:24:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.865+0000 7faf26ffd700 1 -- 172.21.15.104:0/2518302594 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf2c014040 con 0x555a1640bfe0 2024-06-09T22:24:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.865+0000 7faf26ffd700 1 -- 172.21.15.104:0/2518302594 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf2c0052b0 con 0x555a1640bfe0 2024-06-09T22:24:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.865+0000 7faf4dc01b80 1 -- 172.21.15.104:0/2518302594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640bfe0 msgr2=0x555a16411860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.865+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/2518302594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640bfe0 0x555a16411860 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7faf2c010010 tx=0x7faf2c016000 comp rx=0 tx=0).stop 2024-06-09T22:24:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.865+0000 7faf4dc01b80 1 -- 172.21.15.104:0/2518302594 shutdown_connections 2024-06-09T22:24:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.865+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/2518302594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640c900 0x555a16400f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.865+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/2518302594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a1640d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.865+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/2518302594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640bfe0 0x555a16411860 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.865+0000 7faf4dc01b80 1 -- 172.21.15.104:0/2518302594 >> 172.21.15.104:0/2518302594 conn(0x555a16333100 msgr2=0x555a16411150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.865+0000 7faf4dc01b80 1 -- 172.21.15.104:0/2518302594 shutdown_connections 2024-06-09T22:24:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.865+0000 7faf4dc01b80 1 -- 172.21.15.104:0/2518302594 wait complete. 2024-06-09T22:24:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4dc01b80 1 -- start start 2024-06-09T22:24:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4dc01b80 1 Processor -- start 2024-06-09T22:24:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4dc01b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a164187b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4dc01b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 0x555a16418cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4dc01b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 0x555a164155a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4dc01b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c1b19c0 con 0x555a1640c900 2024-06-09T22:24:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4dc01b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c1b12f0 con 0x555a1640bfe0 2024-06-09T22:24:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4dc01b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c1ae4d0 con 0x555a16406820 2024-06-09T22:24:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4dc01b80 10 client.0 _ll_register_callbacks cb 0x555a162ed630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4191b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 0x555a164155a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4191b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 0x555a164155a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54352/0 (socket says 172.21.15.104:54352) 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4191b700 1 -- 172.21.15.104:0/3033851743 learned_addr learned my addr 172.21.15.104:0/3033851743 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf4111a700 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a164187b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.866+0000 7faf40919700 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 0x555a16418cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf4191b700 1 -- 172.21.15.104:0/3033851743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 msgr2=0x555a164187b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf4191b700 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a164187b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf4191b700 1 -- 172.21.15.104:0/3033851743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 msgr2=0x555a16418cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf4191b700 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 0x555a16418cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf4191b700 1 -- 172.21.15.104:0/3033851743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a16415be0 con 0x555a1640c900 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf4191b700 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 0x555a164155a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7faf34007bd0 tx=0x7faf34007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f05d0a2f173202f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf24ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf24ff9700 1 -- 172.21.15.104:0/3033851743 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3401c030 con 0x555a1640c900 2024-06-09T22:24:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf24ff9700 1 -- 172.21.15.104:0/3033851743 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7faf34012040 con 0x555a1640c900 2024-06-09T22:24:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3033851743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf2c014040 con 0x555a1640c900 2024-06-09T22:24:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf24ff9700 1 -- 172.21.15.104:0/3033851743 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faf34029050 con 0x555a1640c900 2024-06-09T22:24:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf4dc01b80 10 client.16524 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3033851743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555a1648b510 con 0x555a1640c900 2024-06-09T22:24:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf24ff9700 1 -- 172.21.15.104:0/3033851743 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf2c014040 con 0x555a1640c900 2024-06-09T22:24:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.867+0000 7faf24ff9700 1 -- 172.21.15.104:0/3033851743 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7faf34018cc0 con 0x555a1640c900 2024-06-09T22:24:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.868+0000 7faf4dc01b80 10 client.16524 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.868+0000 7faf4dc01b80 10 client.16524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.868+0000 7faf4dc01b80 20 client.16524 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.868+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555a1648bc70 0x555a1648e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.868+0000 7faf4dc01b80 4 client.16524 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-09T22:24:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.868+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3033851743 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555a1635f350 con 0x555a1648bc70 2024-06-09T22:24:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.868+0000 7faf4111a700 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555a1648bc70 0x555a1648e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.868+0000 7faf24ff9700 10 client.16524 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.868+0000 7faf4111a700 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555a1648bc70 0x555a1648e090 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd5eef4251e406c1 server_cookie=3b021e45a41b5210 in_seq=0 out_seq=0 2024-06-09T22:24:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.869+0000 7faf24ff9700 1 -- 172.21.15.104:0/3033851743 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555a1635f350 con 0x555a1648bc70 2024-06-09T22:24:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.869+0000 7faf24ff9700 10 client.16524 handle_command_reply: tid=0 2024-06-09T22:24:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.869+0000 7faf4dc01b80 1 client.16524 shutdown 2024-06-09T22:24:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.869+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3033851743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 msgr2=0x555a164155a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.869+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 0x555a164155a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7faf34007bd0 tx=0x7faf34007ee0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.870+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3033851743 shutdown_connections 2024-06-09T22:24:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.870+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x555a1648bc70 0x555a1648e090 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.870+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555a1640c900 0x555a164155a0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.870+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a16406820 0x555a164187b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.870+0000 7faf4dc01b80 1 --2- 172.21.15.104:0/3033851743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a1640bfe0 0x555a16418cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.870+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3033851743 >> 172.21.15.104:0/3033851743 conn(0x555a16333100 msgr2=0x555a164119e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:30.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.870+0000 7faf24ff9700 0 client.16524 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:30.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.870+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3033851743 shutdown_connections 2024-06-09T22:24:30.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.870+0000 7faf4dc01b80 1 -- 172.21.15.104:0/3033851743 wait complete. 2024-06-09T22:24:30.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.870+0000 7faf4dc01b80 20 client.16524 trim_cache size 0 max 16384 2024-06-09T22:24:30.876 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:24:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: "return_code": 0, 2024-06-09T22:24:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_tag": "193b45d0-2fe9-451c-8d02-69d7b60abc8c", 2024-06-09T22:24:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": "asynchronous" 2024-06-09T22:24:30.877 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:24:30.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.870+0000 7faf4337e700 1 -- 172.21.15.104:0/899373716 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faf280a2d90 msgr2=0x7faf280a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.871+0000 7faf4337e700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faf280a2d90 0x7faf280a5250 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faf3c108bd0 tx=0x7faf2c00a380 comp rx=0 tx=0).stop 2024-06-09T22:24:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.871+0000 7faf4337e700 1 -- 172.21.15.104:0/899373716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3c102770 msgr2=0x7faf3c19e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.871+0000 7faf4337e700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3c102770 0x7faf3c19e7a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7faf38006010 tx=0x7faf38009f80 comp rx=0 tx=0).stop 2024-06-09T22:24:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.871+0000 7faf4337e700 1 -- 172.21.15.104:0/899373716 shutdown_connections 2024-06-09T22:24:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.871+0000 7faf4337e700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faf280a2d90 0x7faf280a5250 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.871+0000 7faf4337e700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3c102770 0x7faf3c19e7a0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.871+0000 7faf4337e700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faf3c107860 0x7faf3c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.871+0000 7faf4337e700 1 --2- 172.21.15.104:0/899373716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf3c103120 0x7faf3c19ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.871+0000 7faf4337e700 1 -- 172.21.15.104:0/899373716 >> 172.21.15.104:0/899373716 conn(0x7faf3c079600 msgr2=0x7faf3c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.871+0000 7faf4337e700 1 -- 172.21.15.104:0/899373716 shutdown_connections 2024-06-09T22:24:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:30.871+0000 7faf4337e700 1 -- 172.21.15.104:0/899373716 wait complete. 2024-06-09T22:24:30.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:30 smithi153 ceph-mon[36599]: pgmap v109: 129 pgs: 129 active+clean; 51 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 1012 KiB/s wr, 105 op/s 2024-06-09T22:24:31.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:30 smithi104 ceph-mon[32389]: pgmap v109: 129 pgs: 129 active+clean; 51 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 1012 KiB/s wr, 105 op/s 2024-06-09T22:24:31.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:30 smithi194 ceph-mon[36856]: pgmap v109: 129 pgs: 129 active+clean; 51 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 1012 KiB/s wr, 105 op/s 2024-06-09T22:24:31.684 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:24:31.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 59271 3.45 MB/sec warmup 97 sec latency 1035.555 ms 2024-06-09T22:24:31.985 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:24:32.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:31 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:24:32.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:31 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:24:32.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:31 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:32.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:31 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:24:32.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:31 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:32.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:31 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:32.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:31 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:24:32.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:31 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:24:32.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:31 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:32.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:31 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:24:32.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:31 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:32.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:31 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:32.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:31 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:24:32.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:31 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:24:32.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:31 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:32.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:31 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:24:32.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:31 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:32.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:31 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:32.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 61039 3.53 MB/sec warmup 98 sec latency 29.004 ms 2024-06-09T22:24:33.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:32 smithi104 ceph-mon[32389]: scrub queued for path: / 2024-06-09T22:24:33.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:32 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:33.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:32 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:24:33.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:32 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:33.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:32 smithi104 ceph-mon[32389]: pgmap v110: 129 pgs: 129 active+clean; 51 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 753 KiB/s wr, 84 op/s 2024-06-09T22:24:33.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:32 smithi194 ceph-mon[36856]: scrub queued for path: / 2024-06-09T22:24:33.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:32 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:33.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:32 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:24:33.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:32 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:33.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:32 smithi194 ceph-mon[36856]: pgmap v110: 129 pgs: 129 active+clean; 51 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 753 KiB/s wr, 84 op/s 2024-06-09T22:24:33.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:32 smithi153 ceph-mon[36599]: scrub queued for path: / 2024-06-09T22:24:33.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:32 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:33.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:32 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:24:33.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:32 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:24:33.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:32 smithi153 ceph-mon[36599]: pgmap v110: 129 pgs: 129 active+clean; 51 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 753 KiB/s wr, 84 op/s 2024-06-09T22:24:33.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 61051 3.50 MB/sec warmup 99 sec latency 988.639 ms 2024-06-09T22:24:33.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.835+0000 7f39966e2700 1 -- 172.21.15.104:0/1579763467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3990105030 msgr2=0x7f3990111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:33.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.835+0000 7f39966e2700 1 --2- 172.21.15.104:0/1579763467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3990105030 0x7f3990111540 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f398800b0a0 tx=0x7f398800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.835+0000 7f39966e2700 1 -- 172.21.15.104:0/1579763467 shutdown_connections 2024-06-09T22:24:33.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.835+0000 7f39966e2700 1 --2- 172.21.15.104:0/1579763467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3990105030 0x7f3990111540 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.835+0000 7f39966e2700 1 --2- 172.21.15.104:0/1579763467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f399010a670 0x7f399010aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.835+0000 7f39966e2700 1 --2- 172.21.15.104:0/1579763467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3990104670 0x7f3990104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.835+0000 7f39966e2700 1 -- 172.21.15.104:0/1579763467 >> 172.21.15.104:0/1579763467 conn(0x7f3990100140 msgr2=0x7f3990102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:33.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.835+0000 7f39966e2700 1 -- 172.21.15.104:0/1579763467 shutdown_connections 2024-06-09T22:24:33.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.835+0000 7f39966e2700 1 -- 172.21.15.104:0/1579763467 wait complete. 2024-06-09T22:24:33.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f39966e2700 1 Processor -- start 2024-06-09T22:24:33.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f39966e2700 1 -- start start 2024-06-09T22:24:33.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f39966e2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3990104670 0x7f39901a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f39966e2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3990105030 0x7f39901a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f39966e2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f399010a670 0x7f399019ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f39966e2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f39901149a0 con 0x7f399010a670 2024-06-09T22:24:33.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f39966e2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3990114820 con 0x7f3990104670 2024-06-09T22:24:33.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f39966e2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3990114b20 con 0x7f3990105030 2024-06-09T22:24:33.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f3994c7f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f399010a670 0x7f399019ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f3994c7f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f399010a670 0x7f399019ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54356/0 (socket says 172.21.15.104:54356) 2024-06-09T22:24:33.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f3994c7f700 1 -- 172.21.15.104:0/445950268 learned_addr learned my addr 172.21.15.104:0/445950268 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:33.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f398ffff700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3990104670 0x7f39901a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f3987fff700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3990105030 0x7f39901a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f3994c7f700 1 -- 172.21.15.104:0/445950268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3990105030 msgr2=0x7f39901a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:33.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f3994c7f700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3990105030 0x7f39901a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f3994c7f700 1 -- 172.21.15.104:0/445950268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3990104670 msgr2=0x7f39901a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:33.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f3994c7f700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3990104670 0x7f39901a0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f3994c7f700 1 -- 172.21.15.104:0/445950268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399019b400 con 0x7f399010a670 2024-06-09T22:24:33.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.836+0000 7f3994c7f700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f399010a670 0x7f399019ab20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f39880062a0 tx=0x7f3988004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:33.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.837+0000 7f398dffb700 1 -- 172.21.15.104:0/445950268 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988021070 con 0x7f399010a670 2024-06-09T22:24:33.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.837+0000 7f39966e2700 1 -- 172.21.15.104:0/445950268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f399019b690 con 0x7f399010a670 2024-06-09T22:24:33.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.837+0000 7f398dffb700 1 -- 172.21.15.104:0/445950268 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3988007c80 con 0x7f399010a670 2024-06-09T22:24:33.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.837+0000 7f398dffb700 1 -- 172.21.15.104:0/445950268 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398801d450 con 0x7f399010a670 2024-06-09T22:24:33.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.837+0000 7f39966e2700 1 -- 172.21.15.104:0/445950268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39901a79d0 con 0x7f399010a670 2024-06-09T22:24:33.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.840+0000 7f398dffb700 1 -- 172.21.15.104:0/445950268 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f3988004020 con 0x7f399010a670 2024-06-09T22:24:33.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.840+0000 7f398dffb700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f397c0a2d90 0x7f397c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.840+0000 7f398ffff700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f397c0a2d90 0x7f397c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.840+0000 7f398dffb700 1 -- 172.21.15.104:0/445950268 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f39880d05d0 con 0x7f399010a670 2024-06-09T22:24:33.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f39a0f65b80 1 Processor -- start 2024-06-09T22:24:33.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f398ffff700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f397c0a2d90 0x7f397c0a5250 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3980005fd0 tx=0x7f398000af20 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:33.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f39a0f65b80 1 -- start start 2024-06-09T22:24:33.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f39a0f65b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482640 0x56205e482a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f39a0f65b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482f60 0x56205e47c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f39a0f65b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e47d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f39a0f65b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56205e3ce810 con 0x56205e47cf20 2024-06-09T22:24:33.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f39a0f65b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56205e376720 con 0x56205e482640 2024-06-09T22:24:33.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f39a0f65b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56205e3ca7d0 con 0x56205e482f60 2024-06-09T22:24:33.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f3994c7f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e47d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f3994c7f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e47d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54358/0 (socket says 172.21.15.104:54358) 2024-06-09T22:24:33.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f3994c7f700 1 -- 172.21.15.104:0/3421104184 learned_addr learned my addr 172.21.15.104:0/3421104184 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:33.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f398ffff700 1 --2- 172.21.15.104:0/3421104184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482640 0x56205e482a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.841+0000 7f3987fff700 1 --2- 172.21.15.104:0/3421104184 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482f60 0x56205e47c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.842+0000 7f3994c7f700 1 -- 172.21.15.104:0/3421104184 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482f60 msgr2=0x56205e47c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:33.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.842+0000 7f3994c7f700 1 --2- 172.21.15.104:0/3421104184 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482f60 0x56205e47c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.842+0000 7f3994c7f700 1 -- 172.21.15.104:0/3421104184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482640 msgr2=0x56205e482a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:33.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.842+0000 7f3994c7f700 1 --2- 172.21.15.104:0/3421104184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482640 0x56205e482a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.842+0000 7f3994c7f700 1 -- 172.21.15.104:0/3421104184 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56205e36cf50 con 0x56205e47cf20 2024-06-09T22:24:33.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.842+0000 7f398ffff700 1 --2- 172.21.15.104:0/3421104184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482640 0x56205e482a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:33.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.842+0000 7f3994c7f700 1 --2- 172.21.15.104:0/3421104184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e47d340 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f39880a4ec0 tx=0x7f398800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4201b08271d0d5e1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:33.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.842+0000 7f39867fc700 1 -- 172.21.15.104:0/3421104184 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398802a070 con 0x56205e47cf20 2024-06-09T22:24:33.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.842+0000 7f39867fc700 1 -- 172.21.15.104:0/3421104184 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f398809c810 con 0x56205e47cf20 2024-06-09T22:24:33.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.842+0000 7f39867fc700 1 -- 172.21.15.104:0/3421104184 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988098e80 con 0x56205e47cf20 2024-06-09T22:24:33.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.843+0000 7f39a0f65b80 1 -- 172.21.15.104:0/3421104184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 msgr2=0x56205e47d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:33.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.843+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/3421104184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e47d340 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f39880a4ec0 tx=0x7f398800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.843+0000 7f39a0f65b80 1 -- 172.21.15.104:0/3421104184 shutdown_connections 2024-06-09T22:24:33.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.843+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/3421104184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e47d340 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.843+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/3421104184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482640 0x56205e482a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.843+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/3421104184 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482f60 0x56205e47c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.843+0000 7f39a0f65b80 1 -- 172.21.15.104:0/3421104184 >> 172.21.15.104:0/3421104184 conn(0x56205e3a7cc0 msgr2=0x56205e477850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:33.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.843+0000 7f39a0f65b80 1 -- 172.21.15.104:0/3421104184 shutdown_connections 2024-06-09T22:24:33.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.843+0000 7f39a0f65b80 1 -- 172.21.15.104:0/3421104184 wait complete. 2024-06-09T22:24:33.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.843+0000 7f39a0f65b80 1 -- start start 2024-06-09T22:24:33.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f39a0f65b80 1 Processor -- start 2024-06-09T22:24:33.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f39a0f65b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e488d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f39a0f65b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482640 0x56205e486050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f39a0f65b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482f60 0x56205e486590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f39a0f65b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56205e4896f0 con 0x56205e47cf20 2024-06-09T22:24:33.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f39a0f65b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56205e36ebf0 con 0x56205e482f60 2024-06-09T22:24:33.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f39a0f65b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56205e489870 con 0x56205e482640 2024-06-09T22:24:33.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f39a0f65b80 10 client.0 _ll_register_callbacks cb 0x56205e368920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:33.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f398ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e488d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f398ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e488d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54374/0 (socket says 172.21.15.104:54374) 2024-06-09T22:24:33.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f398ffff700 1 -- 172.21.15.104:0/318802206 learned_addr learned my addr 172.21.15.104:0/318802206 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:33.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f3987fff700 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482640 0x56205e486050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f3994c7f700 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482f60 0x56205e486590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f398ffff700 1 -- 172.21.15.104:0/318802206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482640 msgr2=0x56205e486050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:33.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f398ffff700 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482640 0x56205e486050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f398ffff700 1 -- 172.21.15.104:0/318802206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482f60 msgr2=0x56205e486590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:33.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f398ffff700 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482f60 0x56205e486590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f398ffff700 1 -- 172.21.15.104:0/318802206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56205e31d8d0 con 0x56205e47cf20 2024-06-09T22:24:33.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f3994c7f700 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482f60 0x56205e486590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:24:33.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f398ffff700 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e488d50 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f39800109e0 tx=0x7f39800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58eac290556d9113 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:33.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.844+0000 7f396f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:33.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f396f7fe700 1 -- 172.21.15.104:0/318802206 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398001dc60 con 0x56205e47cf20 2024-06-09T22:24:33.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f396f7fe700 1 -- 172.21.15.104:0/318802206 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f398001de00 con 0x56205e47cf20 2024-06-09T22:24:33.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f39a0f65b80 1 -- 172.21.15.104:0/318802206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f398809c810 con 0x56205e47cf20 2024-06-09T22:24:33.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f396f7fe700 1 -- 172.21.15.104:0/318802206 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3980031050 con 0x56205e47cf20 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f39a0f65b80 10 client.16536 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f39a0f65b80 1 -- 172.21.15.104:0/318802206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56205e4940d0 con 0x56205e47cf20 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f396f7fe700 1 -- 172.21.15.104:0/318802206 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f398809c810 con 0x56205e47cf20 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f396f7fe700 1 -- 172.21.15.104:0/318802206 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f39800265b0 con 0x56205e47cf20 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f39a0f65b80 10 client.16536 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f39a0f65b80 10 client.16536 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f39a0f65b80 20 client.16536 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56205e4967b0 0x56205e498ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f39a0f65b80 4 client.16536 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.845+0000 7f39a0f65b80 1 -- 172.21.15.104:0/318802206 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56205e3dbd10 con 0x56205e4967b0 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.846+0000 7f3987fff700 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56205e4967b0 0x56205e498ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.846+0000 7f3987fff700 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56205e4967b0 0x56205e498ba0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b62beae1a1e611bc server_cookie=627aa197b824e98d in_seq=0 out_seq=0 2024-06-09T22:24:33.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.846+0000 7f396f7fe700 10 client.16536 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:33.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f396f7fe700 1 -- 172.21.15.104:0/318802206 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56205e3dbd10 con 0x56205e4967b0 2024-06-09T22:24:33.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f396f7fe700 10 client.16536 handle_command_reply: tid=0 2024-06-09T22:24:33.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f39a0f65b80 1 client.16536 shutdown 2024-06-09T22:24:33.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f39a0f65b80 1 -- 172.21.15.104:0/318802206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 msgr2=0x56205e488d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:33.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e488d50 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f39800109e0 tx=0x7f39800047c0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f39a0f65b80 1 -- 172.21.15.104:0/318802206 shutdown_connections 2024-06-09T22:24:33.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56205e4967b0 0x56205e498ba0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e488d50 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482f60 0x56205e486590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f396f7fe700 0 client.16536 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:33.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/318802206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482640 0x56205e486050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f39a0f65b80 1 -- 172.21.15.104:0/318802206 >> 172.21.15.104:0/318802206 conn(0x56205e3a7cc0 msgr2=0x56205e47b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:33.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f39a0f65b80 1 -- 172.21.15.104:0/318802206 shutdown_connections 2024-06-09T22:24:33.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f39a0f65b80 1 -- 172.21.15.104:0/318802206 wait complete. 2024-06-09T22:24:33.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.847+0000 7f39a0f65b80 20 client.16536 trim_cache size 0 max 16384 2024-06-09T22:24:33.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.867+0000 7f39a0f65b80 1 Processor -- start 2024-06-09T22:24:33.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.867+0000 7f39a0f65b80 1 -- start start 2024-06-09T22:24:33.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.867+0000 7f39a0f65b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e47c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.867+0000 7f39a0f65b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482640 0x56205e483a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.867+0000 7f39a0f65b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482f60 0x56205e49e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.867+0000 7f39a0f65b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f39901b05b0 con 0x56205e47cf20 2024-06-09T22:24:33.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.867+0000 7f39a0f65b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39901b2150 con 0x56205e482f60 2024-06-09T22:24:33.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.867+0000 7f39a0f65b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f39901b0c80 con 0x56205e482640 2024-06-09T22:24:33.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f398ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e47c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f398ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e47c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54378/0 (socket says 172.21.15.104:54378) 2024-06-09T22:24:33.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f398ffff700 1 -- 172.21.15.104:0/1676474896 learned_addr learned my addr 172.21.15.104:0/1676474896 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:33.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f3987fff700 1 --2- 172.21.15.104:0/1676474896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482640 0x56205e483a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f3994c7f700 1 --2- 172.21.15.104:0/1676474896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482f60 0x56205e49e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f398ffff700 1 -- 172.21.15.104:0/1676474896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482640 msgr2=0x56205e483a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:33.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f398ffff700 1 --2- 172.21.15.104:0/1676474896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482640 0x56205e483a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f398ffff700 1 -- 172.21.15.104:0/1676474896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482f60 msgr2=0x56205e49e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:33.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f398ffff700 1 --2- 172.21.15.104:0/1676474896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482f60 0x56205e49e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f398ffff700 1 -- 172.21.15.104:0/1676474896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56205e478500 con 0x56205e47cf20 2024-06-09T22:24:33.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f398ffff700 1 --2- 172.21.15.104:0/1676474896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e47c9e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3980013040 tx=0x7f398000a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1d00b9b7c224ba server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:33.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f3985ffb700 1 -- 172.21.15.104:0/1676474896 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39800265f0 con 0x56205e47cf20 2024-06-09T22:24:33.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f3985ffb700 1 -- 172.21.15.104:0/1676474896 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f398003a070 con 0x56205e47cf20 2024-06-09T22:24:33.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.868+0000 7f3985ffb700 1 -- 172.21.15.104:0/1676474896 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3980035bb0 con 0x56205e47cf20 2024-06-09T22:24:33.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.869+0000 7f39a0f65b80 1 -- 172.21.15.104:0/1676474896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 msgr2=0x56205e47c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:33.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.869+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/1676474896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e47c9e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3980013040 tx=0x7f398000a370 comp rx=0 tx=0).stop 2024-06-09T22:24:33.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.869+0000 7f39a0f65b80 1 -- 172.21.15.104:0/1676474896 shutdown_connections 2024-06-09T22:24:33.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.869+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/1676474896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e47c9e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.869+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/1676474896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482f60 0x56205e49e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.869+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/1676474896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482640 0x56205e483a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.869+0000 7f39a0f65b80 1 -- 172.21.15.104:0/1676474896 >> 172.21.15.104:0/1676474896 conn(0x56205e3a7cc0 msgr2=0x56205e483340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:33.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.869+0000 7f39a0f65b80 1 -- 172.21.15.104:0/1676474896 shutdown_connections 2024-06-09T22:24:33.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.869+0000 7f39a0f65b80 1 -- 172.21.15.104:0/1676474896 wait complete. 2024-06-09T22:24:33.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.869+0000 7f39a0f65b80 1 -- start start 2024-06-09T22:24:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f39a0f65b80 1 Processor -- start 2024-06-09T22:24:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f39a0f65b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e48eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f39a0f65b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482640 0x56205e48f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f39a0f65b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482f60 0x56205e48bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f39a0f65b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f39901b1a20 con 0x56205e47cf20 2024-06-09T22:24:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f39a0f65b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39901b1350 con 0x56205e482640 2024-06-09T22:24:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f39a0f65b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f39901ae530 con 0x56205e482f60 2024-06-09T22:24:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f39a0f65b80 10 client.0 _ll_register_callbacks cb 0x56205e368920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f398ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e48eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f398ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e48eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54384/0 (socket says 172.21.15.104:54384) 2024-06-09T22:24:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f398ffff700 1 -- 172.21.15.104:0/2584288362 learned_addr learned my addr 172.21.15.104:0/2584288362 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f3987fff700 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482640 0x56205e48f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f3994c7f700 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482f60 0x56205e48bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f398ffff700 1 -- 172.21.15.104:0/2584288362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482f60 msgr2=0x56205e48bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f398ffff700 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482f60 0x56205e48bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f398ffff700 1 -- 172.21.15.104:0/2584288362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482640 msgr2=0x56205e48f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f398ffff700 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482640 0x56205e48f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f398ffff700 1 -- 172.21.15.104:0/2584288362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56205e48c140 con 0x56205e47cf20 2024-06-09T22:24:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f3994c7f700 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482f60 0x56205e48bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f398ffff700 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e48eb30 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f39800357d0 tx=0x7f39800356d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19a5ee0552f11c3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.870+0000 7f396f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f396f7fe700 1 -- 172.21.15.104:0/2584288362 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3980042020 con 0x56205e47cf20 2024-06-09T22:24:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f396f7fe700 1 -- 172.21.15.104:0/2584288362 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f398001da60 con 0x56205e47cf20 2024-06-09T22:24:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f39a0f65b80 1 -- 172.21.15.104:0/2584288362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f398003a070 con 0x56205e47cf20 2024-06-09T22:24:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f396f7fe700 1 -- 172.21.15.104:0/2584288362 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f398004d050 con 0x56205e47cf20 2024-06-09T22:24:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f39a0f65b80 10 client.16542 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f39a0f65b80 1 -- 172.21.15.104:0/2584288362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56205e501d40 con 0x56205e47cf20 2024-06-09T22:24:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f396f7fe700 1 -- 172.21.15.104:0/2584288362 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f398003a070 con 0x56205e47cf20 2024-06-09T22:24:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f396f7fe700 1 -- 172.21.15.104:0/2584288362 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f3980005320 con 0x56205e47cf20 2024-06-09T22:24:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f39a0f65b80 10 client.16542 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f39a0f65b80 10 client.16542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f39a0f65b80 20 client.16542 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56205e502330 0x56205e504750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:33.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f39a0f65b80 4 client.16542 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:24:33.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.871+0000 7f39a0f65b80 1 -- 172.21.15.104:0/2584288362 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56205e3d6290 con 0x56205e502330 2024-06-09T22:24:33.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f3987fff700 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56205e502330 0x56205e504750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:33.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f3987fff700 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56205e502330 0x56205e504750 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60a1b4e62e51a7d9 server_cookie=2e05bd0bbd663a54 in_seq=0 out_seq=0 2024-06-09T22:24:33.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f396f7fe700 10 client.16542 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:33.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f396f7fe700 1 -- 172.21.15.104:0/2584288362 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56205e3d6290 con 0x56205e502330 2024-06-09T22:24:33.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f396f7fe700 10 client.16542 handle_command_reply: tid=0 2024-06-09T22:24:33.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f39a0f65b80 1 client.16542 shutdown 2024-06-09T22:24:33.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f39a0f65b80 1 -- 172.21.15.104:0/2584288362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 msgr2=0x56205e48eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:33.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e48eb30 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f39800357d0 tx=0x7f39800356d0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f39a0f65b80 1 -- 172.21.15.104:0/2584288362 shutdown_connections 2024-06-09T22:24:33.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56205e502330 0x56205e504750 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56205e47cf20 0x56205e48eb30 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56205e482f60 0x56205e48bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f396f7fe700 0 client.16542 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:33.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f39a0f65b80 1 --2- 172.21.15.104:0/2584288362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56205e482640 0x56205e48f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f39a0f65b80 1 -- 172.21.15.104:0/2584288362 >> 172.21.15.104:0/2584288362 conn(0x56205e3a7cc0 msgr2=0x56205e483bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:33.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f39a0f65b80 1 -- 172.21.15.104:0/2584288362 shutdown_connections 2024-06-09T22:24:33.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.872+0000 7f39a0f65b80 1 -- 172.21.15.104:0/2584288362 wait complete. 2024-06-09T22:24:33.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39a0f65b80 20 client.16542 trim_cache size 0 max 16384 2024-06-09T22:24:33.881 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:24:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-09T22:24:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": { 2024-06-09T22:24:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "193b45d0-2fe9-451c-8d02-69d7b60abc8c": { 2024-06-09T22:24:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "path": "/", 2024-06-09T22:24:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "tag": "193b45d0-2fe9-451c-8d02-69d7b60abc8c", 2024-06-09T22:24:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "options": "recursive,force" 2024-06-09T22:24:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:24:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:24:33.881 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:24:33.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39966e2700 1 -- 172.21.15.104:0/445950268 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f397c0a2d90 msgr2=0x7f397c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:33.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39966e2700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f397c0a2d90 0x7f397c0a5250 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3980005fd0 tx=0x7f398000af20 comp rx=0 tx=0).stop 2024-06-09T22:24:33.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39966e2700 1 -- 172.21.15.104:0/445950268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f399010a670 msgr2=0x7f399019ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:33.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39966e2700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f399010a670 0x7f399019ab20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f39880062a0 tx=0x7f3988004e20 comp rx=0 tx=0).stop 2024-06-09T22:24:33.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39966e2700 1 -- 172.21.15.104:0/445950268 shutdown_connections 2024-06-09T22:24:33.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39966e2700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f397c0a2d90 0x7f397c0a5250 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39966e2700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f399010a670 0x7f399019ab20 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39966e2700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3990104670 0x7f39901a0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39966e2700 1 --2- 172.21.15.104:0/445950268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3990105030 0x7f39901a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:33.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39966e2700 1 -- 172.21.15.104:0/445950268 >> 172.21.15.104:0/445950268 conn(0x7f3990100140 msgr2=0x7f39901005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:33.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39966e2700 1 -- 172.21.15.104:0/445950268 shutdown_connections 2024-06-09T22:24:33.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:24:33.873+0000 7f39966e2700 1 -- 172.21.15.104:0/445950268 wait complete. 2024-06-09T22:24:34.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 61051 3.46 MB/sec warmup 100 sec latency 1988.728 ms 2024-06-09T22:24:34.937 INFO:tasks.cephfs.filesystem:scrub status for tag:193b45d0-2fe9-451c-8d02-69d7b60abc8c - {'path': '/', 'tag': '193b45d0-2fe9-451c-8d02-69d7b60abc8c', 'options': 'recursive,force'} 2024-06-09T22:24:35.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:34 smithi104 ceph-mon[32389]: pgmap v111: 129 pgs: 129 active+clean; 52 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-06-09T22:24:35.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:34 smithi194 ceph-mon[36856]: pgmap v111: 129 pgs: 129 active+clean; 52 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-06-09T22:24:35.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:34 smithi153 ceph-mon[36599]: pgmap v111: 129 pgs: 129 active+clean; 52 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-06-09T22:24:35.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 61051 3.43 MB/sec warmup 101 sec latency 2988.835 ms 2024-06-09T22:24:36.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 61119 3.40 MB/sec warmup 102 sec latency 3377.277 ms 2024-06-09T22:24:37.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:36 smithi104 ceph-mon[32389]: pgmap v112: 129 pgs: 129 active+clean; 52 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 514 KiB/s wr, 62 op/s 2024-06-09T22:24:37.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:36 smithi194 ceph-mon[36856]: pgmap v112: 129 pgs: 129 active+clean; 52 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 514 KiB/s wr, 62 op/s 2024-06-09T22:24:37.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:36 smithi153 ceph-mon[36599]: pgmap v112: 129 pgs: 129 active+clean; 52 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 514 KiB/s wr, 62 op/s 2024-06-09T22:24:37.722 INFO:tasks.workunit.client.0.smithi104.stdout: 1 61119 3.36 MB/sec warmup 103 sec latency 1560.252 ms 2024-06-09T22:24:38.723 INFO:tasks.workunit.client.0.smithi104.stdout: 1 61119 3.33 MB/sec warmup 104 sec latency 2560.352 ms 2024-06-09T22:24:39.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:38 smithi153 ceph-mon[36599]: pgmap v113: 129 pgs: 129 active+clean; 52 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 514 KiB/s wr, 62 op/s 2024-06-09T22:24:39.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:38 smithi194 ceph-mon[36856]: pgmap v113: 129 pgs: 129 active+clean; 52 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 514 KiB/s wr, 62 op/s 2024-06-09T22:24:39.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:38 smithi104 ceph-mon[32389]: pgmap v113: 129 pgs: 129 active+clean; 52 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 514 KiB/s wr, 62 op/s 2024-06-09T22:24:39.723 INFO:tasks.workunit.client.0.smithi104.stdout: 1 61119 3.30 MB/sec warmup 105 sec latency 3560.449 ms 2024-06-09T22:24:40.723 INFO:tasks.workunit.client.0.smithi104.stdout: 1 63095 3.37 MB/sec warmup 106 sec latency 3864.402 ms 2024-06-09T22:24:41.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:40 smithi194 ceph-mon[36856]: pgmap v114: 129 pgs: 129 active+clean; 55 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 842 KiB/s wr, 75 op/s 2024-06-09T22:24:41.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:40 smithi153 ceph-mon[36599]: pgmap v114: 129 pgs: 129 active+clean; 55 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 842 KiB/s wr, 75 op/s 2024-06-09T22:24:41.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:40 smithi104 ceph-mon[32389]: pgmap v114: 129 pgs: 129 active+clean; 55 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 842 KiB/s wr, 75 op/s 2024-06-09T22:24:41.723 INFO:tasks.workunit.client.0.smithi104.stdout: 1 64679 3.42 MB/sec warmup 107 sec latency 14.112 ms 2024-06-09T22:24:42.723 INFO:tasks.workunit.client.0.smithi104.stdout: 1 67525 3.55 MB/sec warmup 108 sec latency 45.586 ms 2024-06-09T22:24:43.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:42 smithi194 ceph-mon[36856]: pgmap v115: 129 pgs: 129 active+clean; 55 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 795 KiB/s wr, 65 op/s 2024-06-09T22:24:43.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:42 smithi153 ceph-mon[36599]: pgmap v115: 129 pgs: 129 active+clean; 55 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 795 KiB/s wr, 65 op/s 2024-06-09T22:24:43.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:42 smithi104 ceph-mon[32389]: pgmap v115: 129 pgs: 129 active+clean; 55 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 795 KiB/s wr, 65 op/s 2024-06-09T22:24:43.580 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.579+0000 7fcc299d5640 1 -- 172.21.15.104:0/1316421124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc241008d0 msgr2=0x7fcc24100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:43.580 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.579+0000 7fcc299d5640 1 --2- 172.21.15.104:0/1316421124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc241008d0 0x7fcc24100d50 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009930 tx=0x7fcc1802a420 comp rx=0 tx=0).stop 2024-06-09T22:24:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.579+0000 7fcc299d5640 1 -- 172.21.15.104:0/1316421124 shutdown_connections 2024-06-09T22:24:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.579+0000 7fcc299d5640 1 --2- 172.21.15.104:0/1316421124 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc24101290 0x7fcc2410d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.579+0000 7fcc299d5640 1 --2- 172.21.15.104:0/1316421124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc241008d0 0x7fcc24100d50 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.579+0000 7fcc299d5640 1 --2- 172.21.15.104:0/1316421124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc24106890 0x7fcc24106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.579+0000 7fcc299d5640 1 -- 172.21.15.104:0/1316421124 >> 172.21.15.104:0/1316421124 conn(0x7fcc24077a20 msgr2=0x7fcc24077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.579+0000 7fcc299d5640 1 -- 172.21.15.104:0/1316421124 shutdown_connections 2024-06-09T22:24:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.579+0000 7fcc299d5640 1 -- 172.21.15.104:0/1316421124 wait complete. 2024-06-09T22:24:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.579+0000 7fcc299d5640 1 Processor -- start 2024-06-09T22:24:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.579+0000 7fcc299d5640 1 -- start start 2024-06-09T22:24:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc299d5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc241008d0 0x7fcc24198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc299d5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc24101290 0x7fcc241994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc299d5640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc24106890 0x7fcc241a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc299d5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcc24110ca0 con 0x7fcc241008d0 2024-06-09T22:24:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc299d5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcc24110b20 con 0x7fcc24101290 2024-06-09T22:24:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc299d5640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcc24110e20 con 0x7fcc24106890 2024-06-09T22:24:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc22ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc241008d0 0x7fcc24198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc22ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc241008d0 0x7fcc24198f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35978/0 (socket says 172.21.15.104:35978) 2024-06-09T22:24:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc22ffd640 1 -- 172.21.15.104:0/3793188796 learned_addr learned my addr 172.21.15.104:0/3793188796 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc237fe640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc24106890 0x7fcc241a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc22ffd640 1 -- 172.21.15.104:0/3793188796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc24106890 msgr2=0x7fcc241a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc227fc640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc24101290 0x7fcc241994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc22ffd640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc24106890 0x7fcc241a4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc22ffd640 1 -- 172.21.15.104:0/3793188796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc24101290 msgr2=0x7fcc241994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc22ffd640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc24101290 0x7fcc241994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc22ffd640 1 -- 172.21.15.104:0/3793188796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc241a4540 con 0x7fcc241008d0 2024-06-09T22:24:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc22ffd640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc241008d0 0x7fcc24198f80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400c920 tx=0x7fcc1400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc289d3640 1 -- 172.21.15.104:0/3793188796 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14007c70 con 0x7fcc241008d0 2024-06-09T22:24:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc299d5640 1 -- 172.21.15.104:0/3793188796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc241a4770 con 0x7fcc241008d0 2024-06-09T22:24:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc289d3640 1 -- 172.21.15.104:0/3793188796 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc14007e10 con 0x7fcc241008d0 2024-06-09T22:24:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc289d3640 1 -- 172.21.15.104:0/3793188796 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14018890 con 0x7fcc241008d0 2024-06-09T22:24:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.580+0000 7fcc299d5640 1 -- 172.21.15.104:0/3793188796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc241a4ce0 con 0x7fcc241008d0 2024-06-09T22:24:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc289d3640 1 -- 172.21.15.104:0/3793188796 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fcc14018a70 con 0x7fcc241008d0 2024-06-09T22:24:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc2c7b7740 1 Processor -- start 2024-06-09T22:24:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc2c7b7740 1 -- start start 2024-06-09T22:24:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc2c7b7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7edf170 0x55c2a7edf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc2c7b7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7ee9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc2c7b7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7eea350 0x55c2a7eec740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc2c7b7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55c2a7e434f0 con 0x55c2a7eea350 2024-06-09T22:24:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc2c7b7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c2a7de1870 con 0x55c2a7edfa90 2024-06-09T22:24:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc2c7b7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c2a7e3f4b0 con 0x55c2a7edf170 2024-06-09T22:24:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc237fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7edf170 0x55c2a7edf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc237fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7edf170 0x55c2a7edf550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:55864/0 (socket says 172.21.15.104:55864) 2024-06-09T22:24:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc237fe640 1 -- 172.21.15.104:0/1777674766 learned_addr learned my addr 172.21.15.104:0/1777674766 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc227fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7eea350 0x55c2a7eec740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc289d3640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fcc0c0a2da0 0x7fcc0c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.583+0000 7fcc22ffd640 1 --2- 172.21.15.104:0/1777674766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7ee9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc237fe640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fcc0c0a2da0 0x7fcc0c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc289d3640 1 -- 172.21.15.104:0/3793188796 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fcc140cf940 con 0x7fcc241008d0 2024-06-09T22:24:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc227fc640 1 -- 172.21.15.104:0/1777674766 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7edf170 msgr2=0x55c2a7edf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc227fc640 1 --2- 172.21.15.104:0/1777674766 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7edf170 0x55c2a7edf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc227fc640 1 -- 172.21.15.104:0/1777674766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 msgr2=0x55c2a7ee9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc227fc640 1 --2- 172.21.15.104:0/1777674766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7ee9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc227fc640 1 -- 172.21.15.104:0/1777674766 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2a7dfd9b0 con 0x55c2a7eea350 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc237fe640 1 --2- 172.21.15.104:0/1777674766 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7edf170 0x55c2a7edf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc237fe640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fcc0c0a2da0 0x7fcc0c0a5260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009fd0 tx=0x7fcc1c004ab0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc227fc640 1 --2- 172.21.15.104:0/1777674766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7eea350 0x55c2a7eec740 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009230 tx=0x7fcc18005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269ff0360227bc87 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc09ffb640 1 -- 172.21.15.104:0/1777674766 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1803f030 con 0x55c2a7eea350 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc09ffb640 1 -- 172.21.15.104:0/1777674766 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc18032d30 con 0x55c2a7eea350 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.584+0000 7fcc09ffb640 1 -- 172.21.15.104:0/1777674766 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18035070 con 0x55c2a7eea350 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/1777674766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7eea350 msgr2=0x55c2a7eec740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/1777674766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7eea350 0x55c2a7eec740 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009230 tx=0x7fcc18005dd0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/1777674766 shutdown_connections 2024-06-09T22:24:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/1777674766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7eea350 0x55c2a7eec740 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/1777674766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7ee9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/1777674766 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7edf170 0x55c2a7edf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/1777674766 >> 172.21.15.104:0/1777674766 conn(0x55c2a7edb680 msgr2=0x55c2a7edba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/1777674766 shutdown_connections 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/1777674766 wait complete. 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 -- start start 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 Processor -- start 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 0x55c2a7ee72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7ee7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7eea350 0x55c2a7ee7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.585+0000 7fcc2c7b7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55c2a7eecf20 con 0x55c2a7edf170 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc2c7b7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c2a7dccae0 con 0x55c2a7edfa90 2024-06-09T22:24:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc2c7b7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c2a7eed0a0 con 0x55c2a7eea350 2024-06-09T22:24:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc2c7b7740 10 client.0 _ll_register_callbacks cb 0x55c2a7e059b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc22ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 0x55c2a7ee72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc22ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 0x55c2a7ee72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35998/0 (socket says 172.21.15.104:35998) 2024-06-09T22:24:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc22ffd640 1 -- 172.21.15.104:0/3887374159 learned_addr learned my addr 172.21.15.104:0/3887374159 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc227fc640 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7ee7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc237fe640 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7eea350 0x55c2a7ee7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc22ffd640 1 -- 172.21.15.104:0/3887374159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7eea350 msgr2=0x55c2a7ee7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc22ffd640 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7eea350 0x55c2a7ee7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc22ffd640 1 -- 172.21.15.104:0/3887374159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 msgr2=0x55c2a7ee7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc22ffd640 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7ee7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc22ffd640 1 -- 172.21.15.104:0/3887374159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2a7b57210 con 0x55c2a7edf170 2024-06-09T22:24:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc22ffd640 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 0x55c2a7ee72f0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcc140984c0 tx=0x7fcc1409c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3abdf48c2f46020 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc02ffd640 1 -- 172.21.15.104:0/3887374159 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc140a4760 con 0x55c2a7edf170 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc02ffd640 1 -- 172.21.15.104:0/3887374159 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fcc1400fb30 con 0x55c2a7edf170 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/3887374159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc18032d30 con 0x55c2a7edf170 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc02ffd640 1 -- 172.21.15.104:0/3887374159 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcc140ccfa0 con 0x55c2a7edf170 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc2c7b7740 10 client.16554 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/3887374159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c2a7c23620 con 0x55c2a7edf170 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.586+0000 7fcc02ffd640 1 -- 172.21.15.104:0/3887374159 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc18032d30 con 0x55c2a7edf170 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.587+0000 7fcc02ffd640 1 -- 172.21.15.104:0/3887374159 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fcc140ba270 con 0x55c2a7edf170 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.587+0000 7fcc2c7b7740 10 client.16554 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.587+0000 7fcc2c7b7740 10 client.16554 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.587+0000 7fcc2c7b7740 20 client.16554 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.587+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c2a7efa490 0x55c2a7efc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.587+0000 7fcc2c7b7740 4 client.16554 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.587+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/3887374159 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2a7e56f10 con 0x55c2a7efa490 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.587+0000 7fcc227fc640 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c2a7efa490 0x55c2a7efc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.587+0000 7fcc227fc640 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c2a7efa490 0x55c2a7efc8a0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d418f0361247df3 server_cookie=92f75c708fd0ac50 in_seq=0 out_seq=0 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.587+0000 7fcc02ffd640 10 client.16554 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc02ffd640 1 -- 172.21.15.104:0/3887374159 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55c2a7e56f10 con 0x55c2a7efa490 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc02ffd640 10 client.16554 handle_command_reply: tid=0 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc2c7b7740 1 client.16554 shutdown 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/3887374159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 msgr2=0x55c2a7ee72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 0x55c2a7ee72f0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcc140984c0 tx=0x7fcc1409c280 comp rx=0 tx=0).stop 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/3887374159 shutdown_connections 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c2a7efa490 0x55c2a7efc8a0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7eea350 0x55c2a7ee7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7ee7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc02ffd640 0 client.16554 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/3887374159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 0x55c2a7ee72f0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/3887374159 >> 172.21.15.104:0/3887374159 conn(0x55c2a7edb680 msgr2=0x55c2a7ee13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/3887374159 shutdown_connections 2024-06-09T22:24:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/3887374159 wait complete. 2024-06-09T22:24:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.588+0000 7fcc2c7b7740 20 client.16554 trim_cache size 0 max 16384 2024-06-09T22:24:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.603+0000 7fcc2c7b7740 1 Processor -- start 2024-06-09T22:24:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.603+0000 7fcc2c7b7740 1 -- start start 2024-06-09T22:24:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.603+0000 7fcc2c7b7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7edf170 0x55c2a7f021b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.603+0000 7fcc2c7b7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edfa90 0x55c2a7f026f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.603+0000 7fcc2c7b7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7eea350 0x55c2a7ee8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.603+0000 7fcc2c7b7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcc241aec70 con 0x55c2a7edfa90 2024-06-09T22:24:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.603+0000 7fcc2c7b7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcc241b0810 con 0x55c2a7eea350 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.603+0000 7fcc2c7b7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcc241af340 con 0x55c2a7edf170 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.603+0000 7fcc227fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edfa90 0x55c2a7f026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.603+0000 7fcc227fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edfa90 0x55c2a7f026f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36002/0 (socket says 172.21.15.104:36002) 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.603+0000 7fcc227fc640 1 -- 172.21.15.104:0/4040318765 learned_addr learned my addr 172.21.15.104:0/4040318765 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.603+0000 7fcc22ffd640 1 --2- 172.21.15.104:0/4040318765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7edf170 0x55c2a7f021b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc227fc640 1 -- 172.21.15.104:0/4040318765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7edf170 msgr2=0x55c2a7f021b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc227fc640 1 --2- 172.21.15.104:0/4040318765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7edf170 0x55c2a7f021b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc227fc640 1 -- 172.21.15.104:0/4040318765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7eea350 msgr2=0x55c2a7ee8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc227fc640 1 --2- 172.21.15.104:0/4040318765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7eea350 0x55c2a7ee8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc227fc640 1 -- 172.21.15.104:0/4040318765 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2a7ee0830 con 0x55c2a7edfa90 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc227fc640 1 --2- 172.21.15.104:0/4040318765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edfa90 0x55c2a7f026f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fcc18031550 tx=0x7fcc1802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1b9b3f606f1c96 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc097fa640 1 -- 172.21.15.104:0/4040318765 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1803f040 con 0x55c2a7edfa90 2024-06-09T22:24:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc097fa640 1 -- 172.21.15.104:0/4040318765 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc1804e070 con 0x55c2a7edfa90 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc097fa640 1 -- 172.21.15.104:0/4040318765 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18035040 con 0x55c2a7edfa90 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/4040318765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edfa90 msgr2=0x55c2a7f026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/4040318765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edfa90 0x55c2a7f026f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fcc18031550 tx=0x7fcc1802a420 comp rx=0 tx=0).stop 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/4040318765 shutdown_connections 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/4040318765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7eea350 0x55c2a7ee8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/4040318765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edfa90 0x55c2a7f026f0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/4040318765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7edf170 0x55c2a7f021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/4040318765 >> 172.21.15.104:0/4040318765 conn(0x55c2a7edb680 msgr2=0x55c2a7ee5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/4040318765 shutdown_connections 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.604+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/4040318765 wait complete. 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc2c7b7740 1 -- start start 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc2c7b7740 1 Processor -- start 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc2c7b7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 0x55c2a7eed350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc2c7b7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7eed890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc2c7b7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7eea350 0x55c2a7eeddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc2c7b7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcc241b00e0 con 0x55c2a7edf170 2024-06-09T22:24:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc2c7b7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcc241afa10 con 0x55c2a7edfa90 2024-06-09T22:24:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc2c7b7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcc241acc30 con 0x55c2a7eea350 2024-06-09T22:24:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc2c7b7740 10 client.0 _ll_register_callbacks cb 0x7fcc1409b540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc22ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 0x55c2a7eed350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc22ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 0x55c2a7eed350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36004/0 (socket says 172.21.15.104:36004) 2024-06-09T22:24:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc22ffd640 1 -- 172.21.15.104:0/2000698363 learned_addr learned my addr 172.21.15.104:0/2000698363 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc227fc640 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7eed890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.605+0000 7fcc237fe640 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7eea350 0x55c2a7eeddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc22ffd640 1 -- 172.21.15.104:0/2000698363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7eea350 msgr2=0x55c2a7eeddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc22ffd640 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7eea350 0x55c2a7eeddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc22ffd640 1 -- 172.21.15.104:0/2000698363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 msgr2=0x55c2a7eed890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc22ffd640 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7eed890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc22ffd640 1 -- 172.21.15.104:0/2000698363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2a7f63410 con 0x55c2a7edf170 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc227fc640 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7eed890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc22ffd640 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 0x55c2a7eed350 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fcc14098470 tx=0x7fcc1409b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01caaf513c89240 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc02ffd640 1 -- 172.21.15.104:0/2000698363 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc140a4760 con 0x55c2a7edf170 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc02ffd640 1 -- 172.21.15.104:0/2000698363 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fcc140ba2a0 con 0x55c2a7edf170 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/2000698363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc1804e070 con 0x55c2a7edf170 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc02ffd640 1 -- 172.21.15.104:0/2000698363 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcc1409baf0 con 0x55c2a7edf170 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc2c7b7740 10 client.16560 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/2000698363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c2a7f646e0 con 0x55c2a7edf170 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc02ffd640 1 -- 172.21.15.104:0/2000698363 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc1804e070 con 0x55c2a7edf170 2024-06-09T22:24:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc02ffd640 1 -- 172.21.15.104:0/2000698363 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fcc14097840 con 0x55c2a7edf170 2024-06-09T22:24:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc2c7b7740 10 client.16560 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc2c7b7740 10 client.16560 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc2c7b7740 20 client.16560 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c2a7f64c80 0x55c2a7f670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc2c7b7740 4 client.16560 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:24:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.606+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/2000698363 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c2a7e50c20 con 0x55c2a7f64c80 2024-06-09T22:24:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.607+0000 7fcc227fc640 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c2a7f64c80 0x55c2a7f670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.607+0000 7fcc02ffd640 10 client.16560 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.607+0000 7fcc227fc640 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c2a7f64c80 0x55c2a7f670a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf04228654df9f81 server_cookie=bc3b7c70d9e423e in_seq=0 out_seq=0 2024-06-09T22:24:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc02ffd640 1 -- 172.21.15.104:0/2000698363 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55c2a7e50c20 con 0x55c2a7f64c80 2024-06-09T22:24:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc02ffd640 10 client.16560 handle_command_reply: tid=0 2024-06-09T22:24:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc2c7b7740 1 client.16560 shutdown 2024-06-09T22:24:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/2000698363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 msgr2=0x55c2a7eed350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 0x55c2a7eed350 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fcc14098470 tx=0x7fcc1409b4d0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/2000698363 shutdown_connections 2024-06-09T22:24:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c2a7f64c80 0x55c2a7f670a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2a7eea350 0x55c2a7eeddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc02ffd640 0 client.16560 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2a7edfa90 0x55c2a7eed890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc2c7b7740 1 --2- 172.21.15.104:0/2000698363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c2a7edf170 0x55c2a7eed350 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/2000698363 >> 172.21.15.104:0/2000698363 conn(0x55c2a7edb680 msgr2=0x55c2a7ed9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/2000698363 shutdown_connections 2024-06-09T22:24:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc2c7b7740 1 -- 172.21.15.104:0/2000698363 wait complete. 2024-06-09T22:24:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.614+0000 7fcc2c7b7740 20 client.16560 trim_cache size 0 max 16384 2024-06-09T22:24:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:24:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:24:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:24:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.615+0000 7fcc299d5640 1 -- 172.21.15.104:0/3793188796 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fcc0c0a2da0 msgr2=0x7fcc0c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.615+0000 7fcc299d5640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fcc0c0a2da0 0x7fcc0c0a5260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009fd0 tx=0x7fcc1c004ab0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.615+0000 7fcc299d5640 1 -- 172.21.15.104:0/3793188796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc241008d0 msgr2=0x7fcc24198f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.615+0000 7fcc299d5640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc241008d0 0x7fcc24198f80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400c920 tx=0x7fcc1400cde0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.615+0000 7fcc299d5640 1 -- 172.21.15.104:0/3793188796 shutdown_connections 2024-06-09T22:24:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.615+0000 7fcc299d5640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fcc0c0a2da0 0x7fcc0c0a5260 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.615+0000 7fcc299d5640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc24106890 0x7fcc241a4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.615+0000 7fcc299d5640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc24101290 0x7fcc241994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.615+0000 7fcc299d5640 1 --2- 172.21.15.104:0/3793188796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc241008d0 0x7fcc24198f80 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.615+0000 7fcc299d5640 1 -- 172.21.15.104:0/3793188796 >> 172.21.15.104:0/3793188796 conn(0x7fcc24077a20 msgr2=0x7fcc241032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.615+0000 7fcc299d5640 1 -- 172.21.15.104:0/3793188796 shutdown_connections 2024-06-09T22:24:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:43.615+0000 7fcc299d5640 1 -- 172.21.15.104:0/3793188796 wait complete. 2024-06-09T22:24:43.723 INFO:tasks.workunit.client.0.smithi104.stdout: 1 68087 3.53 MB/sec warmup 109 sec latency 435.755 ms 2024-06-09T22:24:44.723 INFO:tasks.workunit.client.0.smithi104.stdout: 1 68087 3.49 MB/sec warmup 110 sec latency 1435.850 ms 2024-06-09T22:24:45.058 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:44 smithi104 ceph-mon[32389]: pgmap v116: 129 pgs: 129 active+clean; 62 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 142 op/s 2024-06-09T22:24:45.059 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:44 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:24:45.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:44 smithi194 ceph-mon[36856]: pgmap v116: 129 pgs: 129 active+clean; 62 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 142 op/s 2024-06-09T22:24:45.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:44 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:24:45.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:44 smithi153 ceph-mon[36599]: pgmap v116: 129 pgs: 129 active+clean; 62 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 142 op/s 2024-06-09T22:24:45.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:44 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:24:45.723 INFO:tasks.workunit.client.0.smithi104.stdout: 1 68087 3.46 MB/sec warmup 111 sec latency 2435.922 ms 2024-06-09T22:24:46.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:45 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:24:46.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:45 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:24:46.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:45 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:24:46.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:45 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:24:46.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:45 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:24:46.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:45 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:24:46.723 INFO:tasks.workunit.client.0.smithi104.stdout: 1 68143 3.43 MB/sec warmup 112 sec latency 2824.389 ms 2024-06-09T22:24:47.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:46 smithi194 ceph-mon[36856]: pgmap v117: 129 pgs: 129 active+clean; 62 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-06-09T22:24:47.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:46 smithi153 ceph-mon[36599]: pgmap v117: 129 pgs: 129 active+clean; 62 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-06-09T22:24:47.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:46 smithi104 ceph-mon[32389]: pgmap v117: 129 pgs: 129 active+clean; 62 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-06-09T22:24:47.724 INFO:tasks.workunit.client.0.smithi104.stdout: 1 68143 3.40 MB/sec warmup 113 sec latency 1558.012 ms 2024-06-09T22:24:48.724 INFO:tasks.workunit.client.0.smithi104.stdout: 1 68143 3.37 MB/sec warmup 114 sec latency 2558.066 ms 2024-06-09T22:24:49.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:48 smithi194 ceph-mon[36856]: pgmap v118: 129 pgs: 129 active+clean; 62 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-06-09T22:24:49.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:48 smithi153 ceph-mon[36599]: pgmap v118: 129 pgs: 129 active+clean; 62 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-06-09T22:24:49.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:48 smithi104 ceph-mon[32389]: pgmap v118: 129 pgs: 129 active+clean; 62 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-06-09T22:24:49.724 INFO:tasks.workunit.client.0.smithi104.stdout: 1 68143 3.34 MB/sec warmup 115 sec latency 3558.161 ms 2024-06-09T22:24:50.724 INFO:tasks.workunit.client.0.smithi104.stdout: 1 68211 3.32 MB/sec warmup 116 sec latency 3861.707 ms 2024-06-09T22:24:51.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:50 smithi194 ceph-mon[36856]: pgmap v119: 129 pgs: 129 active+clean; 55 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 107 op/s 2024-06-09T22:24:51.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:50 smithi153 ceph-mon[36599]: pgmap v119: 129 pgs: 129 active+clean; 55 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 107 op/s 2024-06-09T22:24:51.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:50 smithi104 ceph-mon[32389]: pgmap v119: 129 pgs: 129 active+clean; 55 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 107 op/s 2024-06-09T22:24:51.724 INFO:tasks.workunit.client.0.smithi104.stdout: 1 70002 3.35 MB/sec warmup 117 sec latency 1036.098 ms 2024-06-09T22:24:52.724 INFO:tasks.workunit.client.0.smithi104.stdout: 1 71753 3.42 MB/sec warmup 118 sec latency 15.573 ms 2024-06-09T22:24:53.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:52 smithi194 ceph-mon[36856]: pgmap v120: 129 pgs: 129 active+clean; 55 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 954 KiB/s wr, 94 op/s 2024-06-09T22:24:53.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:52 smithi153 ceph-mon[36599]: pgmap v120: 129 pgs: 129 active+clean; 55 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 954 KiB/s wr, 94 op/s 2024-06-09T22:24:53.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:52 smithi104 ceph-mon[32389]: pgmap v120: 129 pgs: 129 active+clean; 55 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 954 KiB/s wr, 94 op/s 2024-06-09T22:24:53.724 INFO:tasks.workunit.client.0.smithi104.stdout: 1 71753 3.39 MB/sec warmup 119 sec latency 1006.793 ms 2024-06-09T22:24:55.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:54 smithi194 ceph-mon[36856]: pgmap v121: 129 pgs: 129 active+clean; 63 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 139 op/s 2024-06-09T22:24:55.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:54 smithi153 ceph-mon[36599]: pgmap v121: 129 pgs: 129 active+clean; 63 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 139 op/s 2024-06-09T22:24:55.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:54 smithi104 ceph-mon[32389]: pgmap v121: 129 pgs: 129 active+clean; 63 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 139 op/s 2024-06-09T22:24:55.724 INFO:tasks.workunit.client.0.smithi104.stdout: 1 71753 0.00 MB/sec execute 1 sec latency 3007.007 ms 2024-06-09T22:24:56.724 INFO:tasks.workunit.client.0.smithi104.stdout: 1 73487 3.71 MB/sec execute 2 sec latency 3394.581 ms 2024-06-09T22:24:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.729+0000 7efeeab22640 1 -- 172.21.15.104:0/3242350742 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efee4106cd0 msgr2=0x7efee410e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.729+0000 7efeeab22640 1 --2- 172.21.15.104:0/3242350742 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efee4106cd0 0x7efee410e8a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7efed4009900 tx=0x7efed402a410 comp rx=0 tx=0).stop 2024-06-09T22:24:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.729+0000 7efeeab22640 1 -- 172.21.15.104:0/3242350742 shutdown_connections 2024-06-09T22:24:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.729+0000 7efeeab22640 1 --2- 172.21.15.104:0/3242350742 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efee4106cd0 0x7efee410e8a0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.729+0000 7efeeab22640 1 --2- 172.21.15.104:0/3242350742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efee4100ac0 0x7efee41065f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.729+0000 7efeeab22640 1 --2- 172.21.15.104:0/3242350742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee4100110 0x7efee41004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.729+0000 7efeeab22640 1 -- 172.21.15.104:0/3242350742 >> 172.21.15.104:0/3242350742 conn(0x7efee4076dc0 msgr2=0x7efee40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.729+0000 7efeeab22640 1 -- 172.21.15.104:0/3242350742 shutdown_connections 2024-06-09T22:24:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.729+0000 7efeeab22640 1 -- 172.21.15.104:0/3242350742 wait complete. 2024-06-09T22:24:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efeeab22640 1 Processor -- start 2024-06-09T22:24:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efeeab22640 1 -- start start 2024-06-09T22:24:56.733 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efeeab22640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efee4100110 0x7efee41981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.733 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efeeab22640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee4106cd0 0x7efee4198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.733 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efeeab22640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efee419caa0 0x7efee419cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.733 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efeeab22640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efee4111d80 con 0x7efee419caa0 2024-06-09T22:24:56.733 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efeeab22640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efee4111c00 con 0x7efee4106cd0 2024-06-09T22:24:56.733 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efeeab22640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efee4111f00 con 0x7efee4100110 2024-06-09T22:24:56.733 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efee90bf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efee419caa0 0x7efee419cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.733 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efee90bf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efee419caa0 0x7efee419cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53238/0 (socket says 172.21.15.104:53238) 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efee88be640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efee4100110 0x7efee41981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efee90bf640 1 -- 172.21.15.104:0/1646167572 learned_addr learned my addr 172.21.15.104:0/1646167572 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efee3fff640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee4106cd0 0x7efee4198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efee90bf640 1 -- 172.21.15.104:0/1646167572 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efee4100110 msgr2=0x7efee41981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efee90bf640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efee4100110 0x7efee41981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efee90bf640 1 -- 172.21.15.104:0/1646167572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee4106cd0 msgr2=0x7efee4198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efee90bf640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee4106cd0 0x7efee4198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efee90bf640 1 -- 172.21.15.104:0/1646167572 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee419d590 con 0x7efee419caa0 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.730+0000 7efee3fff640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee4106cd0 0x7efee4198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.731+0000 7efee90bf640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efee419caa0 0x7efee419cf50 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7efed4005ab0 tx=0x7efed4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.731+0000 7efee1ffb640 1 -- 172.21.15.104:0/1646167572 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed403f030 con 0x7efee419caa0 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.731+0000 7efee1ffb640 1 -- 172.21.15.104:0/1646167572 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efed4004440 con 0x7efee419caa0 2024-06-09T22:24:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.731+0000 7efeeab22640 1 -- 172.21.15.104:0/1646167572 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee419d820 con 0x7efee419caa0 2024-06-09T22:24:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.731+0000 7efee1ffb640 1 -- 172.21.15.104:0/1646167572 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed4035070 con 0x7efee419caa0 2024-06-09T22:24:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.731+0000 7efeeab22640 1 -- 172.21.15.104:0/1646167572 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee41a4fe0 con 0x7efee419caa0 2024-06-09T22:24:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.733+0000 7efee1ffb640 1 -- 172.21.15.104:0/1646167572 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7efed4042030 con 0x7efee419caa0 2024-06-09T22:24:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efeed832740 1 Processor -- start 2024-06-09T22:24:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efee1ffb640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efecc0a2e20 0x7efecc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efeed832740 1 -- start start 2024-06-09T22:24:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efeed832740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24170 0x561320b24550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efeed832740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b2ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efee1ffb640 1 -- 172.21.15.104:0/1646167572 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7efed40efa50 con 0x7efee419caa0 2024-06-09T22:24:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efeed832740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b2f350 0x561320b31740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efeed832740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561320a884f0 con 0x561320b24a90 2024-06-09T22:24:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efeed832740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561320a26870 con 0x561320b24170 2024-06-09T22:24:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efeed832740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561320a844b0 con 0x561320b2f350 2024-06-09T22:24:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efee88be640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24170 0x561320b24550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efee88be640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24170 0x561320b24550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:35852/0 (socket says 172.21.15.104:35852) 2024-06-09T22:24:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efee88be640 1 -- 172.21.15.104:0/3076105289 learned_addr learned my addr 172.21.15.104:0/3076105289 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efee3fff640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efecc0a2e20 0x7efecc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.734+0000 7efee90bf640 1 --2- 172.21.15.104:0/3076105289 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b2f350 0x561320b31740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.735+0000 7efee88be640 1 -- 172.21.15.104:0/3076105289 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b2f350 msgr2=0x561320b31740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.735+0000 7efee88be640 1 --2- 172.21.15.104:0/3076105289 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b2f350 0x561320b31740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.735+0000 7efee88be640 1 -- 172.21.15.104:0/3076105289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 msgr2=0x561320b2ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.735+0000 7efee3fff640 1 --2- 172.21.15.104:0/3076105289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b2ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.735+0000 7efee88be640 1 --2- 172.21.15.104:0/3076105289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b2ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.735+0000 7efee88be640 1 -- 172.21.15.104:0/3076105289 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561320a429b0 con 0x561320b24170 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.735+0000 7efee3fff640 1 --2- 172.21.15.104:0/3076105289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b2ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.735+0000 7efee88be640 1 --2- 172.21.15.104:0/3076105289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24170 0x561320b24550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efedc0114f0 tx=0x7efedc0119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36a3c3d3988c90fb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.735+0000 7efee3fff640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efecc0a2e20 0x7efecc0a52e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x561320b25710 tx=0x7efed8009ba0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.735+0000 7efecaffd640 1 -- 172.21.15.104:0/3076105289 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc01e070 con 0x561320b24170 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.735+0000 7efecaffd640 1 -- 172.21.15.104:0/3076105289 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efedc00fb00 con 0x561320b24170 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.735+0000 7efecaffd640 1 -- 172.21.15.104:0/3076105289 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc019b90 con 0x561320b24170 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.736+0000 7efeed832740 1 -- 172.21.15.104:0/3076105289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24170 msgr2=0x561320b24550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.736+0000 7efeed832740 1 --2- 172.21.15.104:0/3076105289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24170 0x561320b24550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efedc0114f0 tx=0x7efedc0119b0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.736+0000 7efeed832740 1 -- 172.21.15.104:0/3076105289 shutdown_connections 2024-06-09T22:24:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.736+0000 7efeed832740 1 --2- 172.21.15.104:0/3076105289 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b2f350 0x561320b31740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.736+0000 7efeed832740 1 --2- 172.21.15.104:0/3076105289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b2ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.736+0000 7efeed832740 1 --2- 172.21.15.104:0/3076105289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24170 0x561320b24550 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.736+0000 7efeed832740 1 -- 172.21.15.104:0/3076105289 >> 172.21.15.104:0/3076105289 conn(0x561320b20680 msgr2=0x561320b20a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.736+0000 7efeed832740 1 -- 172.21.15.104:0/3076105289 shutdown_connections 2024-06-09T22:24:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.736+0000 7efeed832740 1 -- 172.21.15.104:0/3076105289 wait complete. 2024-06-09T22:24:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.736+0000 7efeed832740 1 -- start start 2024-06-09T22:24:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.736+0000 7efeed832740 1 Processor -- start 2024-06-09T22:24:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efeed832740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b2c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efeed832740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b2c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efeed832740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b2f350 0x561320b2cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efeed832740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561320b36390 con 0x561320b24a90 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efeed832740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561320a11ae0 con 0x561320b2f350 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efeed832740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561320b36510 con 0x561320b24170 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efeed832740 10 client.0 _ll_register_callbacks cb 0x561320a4a9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efee3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b2c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efee3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b2c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53262/0 (socket says 172.21.15.104:53262) 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efee3fff640 1 -- 172.21.15.104:0/3675318513 learned_addr learned my addr 172.21.15.104:0/3675318513 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efee88be640 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b2c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efee90bf640 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b2f350 0x561320b2cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efee90bf640 1 -- 172.21.15.104:0/3675318513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 msgr2=0x561320b2c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efee90bf640 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b2c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efee90bf640 1 -- 172.21.15.104:0/3675318513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 msgr2=0x561320b2c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efee90bf640 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b2c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efee90bf640 1 -- 172.21.15.104:0/3675318513 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56132079c210 con 0x561320b2f350 2024-06-09T22:24:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efee3fff640 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b2c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efee90bf640 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b2f350 0x561320b2cdd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efed40ecdf0 tx=0x7efed40bb7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b92afe7686caa21 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.737+0000 7efec3fff640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:24:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efec3fff640 1 -- 172.21.15.104:0/3675318513 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed4049040 con 0x561320b2f350 2024-06-09T22:24:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efeed832740 1 -- 172.21.15.104:0/3675318513 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efedc00fb00 con 0x561320b2f350 2024-06-09T22:24:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efec3fff640 1 -- 172.21.15.104:0/3675318513 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7efed40ed070 con 0x561320b2f350 2024-06-09T22:24:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efec3fff640 1 -- 172.21.15.104:0/3675318513 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efed40ed270 con 0x561320b2f350 2024-06-09T22:24:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efeed832740 10 client.25937 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efeed832740 1 -- 172.21.15.104:0/3675318513 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561320868620 con 0x561320b2f350 2024-06-09T22:24:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efec3fff640 1 -- 172.21.15.104:0/3675318513 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efedc00fb00 con 0x561320b2f350 2024-06-09T22:24:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efec3fff640 1 -- 172.21.15.104:0/3675318513 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7efed40d9070 con 0x561320b2f350 2024-06-09T22:24:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efeed832740 10 client.25937 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efeed832740 10 client.25937 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efeed832740 20 client.25937 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efeed832740 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561320b439a0 0x561320b45e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efeed832740 4 client.25937 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:24:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efeed832740 1 -- 172.21.15.104:0/3675318513 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561320a9bf10 con 0x561320b439a0 2024-06-09T22:24:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.738+0000 7efee88be640 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561320b439a0 0x561320b45e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.739+0000 7efec3fff640 10 client.25937 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.739+0000 7efee88be640 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561320b439a0 0x561320b45e00 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85a5b2e76c74820d server_cookie=30e0fda342d4b133 in_seq=0 out_seq=0 2024-06-09T22:24:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.739+0000 7efec3fff640 1 -- 172.21.15.104:0/3675318513 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x561320a9bf10 con 0x561320b439a0 2024-06-09T22:24:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.739+0000 7efec3fff640 10 client.25937 handle_command_reply: tid=0 2024-06-09T22:24:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.739+0000 7efeed832740 1 client.25937 shutdown 2024-06-09T22:24:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.740+0000 7efeed832740 1 -- 172.21.15.104:0/3675318513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b2f350 msgr2=0x561320b2cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.740+0000 7efeed832740 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b2f350 0x561320b2cdd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efed40ecdf0 tx=0x7efed40bb7b0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.740+0000 7efeed832740 1 -- 172.21.15.104:0/3675318513 shutdown_connections 2024-06-09T22:24:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.740+0000 7efeed832740 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561320b439a0 0x561320b45e00 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.740+0000 7efeed832740 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b2f350 0x561320b2cdd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.740+0000 7efeed832740 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b2c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.740+0000 7efeed832740 1 --2- 172.21.15.104:0/3675318513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b2c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.740+0000 7efec3fff640 0 client.25937 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.740+0000 7efeed832740 1 -- 172.21.15.104:0/3675318513 >> 172.21.15.104:0/3675318513 conn(0x561320b20680 msgr2=0x561320b26400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.740+0000 7efeed832740 1 -- 172.21.15.104:0/3675318513 shutdown_connections 2024-06-09T22:24:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.740+0000 7efeed832740 1 -- 172.21.15.104:0/3675318513 wait complete. 2024-06-09T22:24:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.740+0000 7efeed832740 20 client.25937 trim_cache size 0 max 16384 2024-06-09T22:24:56.758 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efeed832740 1 Processor -- start 2024-06-09T22:24:56.758 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efeed832740 1 -- start start 2024-06-09T22:24:56.758 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efeed832740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b45d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.758 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efeed832740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24a90 0x561320b4b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.758 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efeed832740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b2f350 0x561320b4bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.758 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efeed832740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efee41acb30 con 0x561320b2f350 2024-06-09T22:24:56.758 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efeed832740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efee41ae6d0 con 0x561320b24a90 2024-06-09T22:24:56.758 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efeed832740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efee41ad200 con 0x561320b24170 2024-06-09T22:24:56.758 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efee3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24a90 0x561320b4b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.758 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efee90bf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b2f350 0x561320b4bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.759 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efee88be640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b45d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.759 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efee3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24a90 0x561320b4b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:35870/0 (socket says 172.21.15.104:35870) 2024-06-09T22:24:56.759 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.756+0000 7efee3fff640 1 -- 172.21.15.104:0/3449419035 learned_addr learned my addr 172.21.15.104:0/3449419035 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:56.759 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.757+0000 7efee90bf640 1 -- 172.21.15.104:0/3449419035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 msgr2=0x561320b45d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:56.759 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.757+0000 7efee90bf640 1 --2- 172.21.15.104:0/3449419035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b45d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.759 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.757+0000 7efee90bf640 1 -- 172.21.15.104:0/3449419035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24a90 msgr2=0x561320b4b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:56.759 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.757+0000 7efee90bf640 1 --2- 172.21.15.104:0/3449419035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24a90 0x561320b4b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.759 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.757+0000 7efee90bf640 1 -- 172.21.15.104:0/3449419035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561320b248c0 con 0x561320b2f350 2024-06-09T22:24:56.759 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.757+0000 7efee90bf640 1 --2- 172.21.15.104:0/3449419035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b2f350 0x561320b4bc50 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7efed4009a20 tx=0x7efed40bbdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6791d7e991cfa04 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:56.759 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.757+0000 7efeca7fc640 1 -- 172.21.15.104:0/3449419035 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed40bba40 con 0x561320b2f350 2024-06-09T22:24:56.759 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.757+0000 7efeca7fc640 1 -- 172.21.15.104:0/3449419035 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efed40d2020 con 0x561320b2f350 2024-06-09T22:24:56.759 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.757+0000 7efeca7fc640 1 -- 172.21.15.104:0/3449419035 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed40ed070 con 0x561320b2f350 2024-06-09T22:24:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.758+0000 7efeed832740 1 -- 172.21.15.104:0/3449419035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b2f350 msgr2=0x561320b4bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.758+0000 7efeed832740 1 --2- 172.21.15.104:0/3449419035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b2f350 0x561320b4bc50 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7efed4009a20 tx=0x7efed40bbdf0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.758+0000 7efeed832740 1 -- 172.21.15.104:0/3449419035 shutdown_connections 2024-06-09T22:24:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.758+0000 7efeed832740 1 --2- 172.21.15.104:0/3449419035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b2f350 0x561320b4bc50 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.758+0000 7efeed832740 1 --2- 172.21.15.104:0/3449419035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b24a90 0x561320b4b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.758+0000 7efeed832740 1 --2- 172.21.15.104:0/3449419035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b45d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.758+0000 7efeed832740 1 -- 172.21.15.104:0/3449419035 >> 172.21.15.104:0/3449419035 conn(0x561320b20680 msgr2=0x561320b2ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.758+0000 7efeed832740 1 -- 172.21.15.104:0/3449419035 shutdown_connections 2024-06-09T22:24:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.758+0000 7efeed832740 1 -- 172.21.15.104:0/3449419035 wait complete. 2024-06-09T22:24:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.758+0000 7efeed832740 1 -- start start 2024-06-09T22:24:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.758+0000 7efeed832740 1 Processor -- start 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efeed832740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b36760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efeed832740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b36ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efeed832740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b2f350 0x561320b371e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efeed832740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efee41adfa0 con 0x561320b24a90 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efeed832740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efee41ad8d0 con 0x561320b2f350 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efeed832740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efee41aaaf0 con 0x561320b24170 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efeed832740 10 client.0 _ll_register_callbacks cb 0x7efed4009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b36ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee88be640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b36760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b36ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53274/0 (socket says 172.21.15.104:53274) 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee3fff640 1 -- 172.21.15.104:0/80784786 learned_addr learned my addr 172.21.15.104:0/80784786 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee90bf640 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b2f350 0x561320b371e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee3fff640 1 -- 172.21.15.104:0/80784786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 msgr2=0x561320b36760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee3fff640 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b36760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee3fff640 1 -- 172.21.15.104:0/80784786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b2f350 msgr2=0x561320b371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee3fff640 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b2f350 0x561320b371e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee3fff640 1 -- 172.21.15.104:0/80784786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561320bac8f0 con 0x561320b24a90 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee88be640 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b36760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee90bf640 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b2f350 0x561320b371e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efee3fff640 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b36ca0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7efed800fb30 tx=0x7efed8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a3a4eee8b7b8cd server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efec3fff640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efec3fff640 1 -- 172.21.15.104:0/80784786 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed801a070 con 0x561320b24a90 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efec3fff640 1 -- 172.21.15.104:0/80784786 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7efed8003d00 con 0x561320b24a90 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efeed832740 1 -- 172.21.15.104:0/80784786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efed40d2020 con 0x561320b24a90 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efec3fff640 1 -- 172.21.15.104:0/80784786 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efed8034050 con 0x561320b24a90 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.759+0000 7efeed832740 10 client.16575 fetch_fsmap learned FSMap version 30 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.760+0000 7efeed832740 1 -- 172.21.15.104:0/80784786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561320badc40 con 0x561320b24a90 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.760+0000 7efec3fff640 1 -- 172.21.15.104:0/80784786 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efed40d2020 con 0x561320b24a90 2024-06-09T22:24:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.760+0000 7efec3fff640 1 -- 172.21.15.104:0/80784786 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7efed80104c0 con 0x561320b24a90 2024-06-09T22:24:56.766 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.760+0000 7efeed832740 10 client.16575 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:24:56.766 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.760+0000 7efeed832740 10 client.16575 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:24:56.766 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.760+0000 7efeed832740 20 client.16575 populate_metadata read hostname 'smithi104' 2024-06-09T22:24:56.766 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.760+0000 7efeed832740 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561320bae480 0x561320bb08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:24:56.766 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.760+0000 7efeed832740 4 client.16575 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:24:56.766 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.760+0000 7efeed832740 1 -- 172.21.15.104:0/80784786 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561320a95c20 con 0x561320bae480 2024-06-09T22:24:56.766 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.760+0000 7efee88be640 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561320bae480 0x561320bb08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:24:56.766 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.760+0000 7efee88be640 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561320bae480 0x561320bb08a0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81136a1a3eeb91c server_cookie=7ea7e854086e3017 in_seq=0 out_seq=0 2024-06-09T22:24:56.766 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.760+0000 7efec3fff640 10 client.16575 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.768+0000 7efec3fff640 1 -- 172.21.15.104:0/80784786 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x561320a95c20 con 0x561320bae480 2024-06-09T22:24:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.768+0000 7efec3fff640 10 client.16575 handle_command_reply: tid=0 2024-06-09T22:24:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeed832740 1 client.16575 shutdown 2024-06-09T22:24:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeed832740 1 -- 172.21.15.104:0/80784786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 msgr2=0x561320b36ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeed832740 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b36ca0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7efed800fb30 tx=0x7efed8005f90 comp rx=0 tx=0).stop 2024-06-09T22:24:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeed832740 1 -- 172.21.15.104:0/80784786 shutdown_connections 2024-06-09T22:24:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeed832740 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561320bae480 0x561320bb08a0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeed832740 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561320b2f350 0x561320b371e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeed832740 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561320b24a90 0x561320b36ca0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efec3fff640 0 client.16575 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:24:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeed832740 1 --2- 172.21.15.104:0/80784786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561320b24170 0x561320b36760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeed832740 1 -- 172.21.15.104:0/80784786 >> 172.21.15.104:0/80784786 conn(0x561320b20680 msgr2=0x561320b25b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeed832740 1 -- 172.21.15.104:0/80784786 shutdown_connections 2024-06-09T22:24:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeed832740 1 -- 172.21.15.104:0/80784786 wait complete. 2024-06-09T22:24:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeed832740 20 client.16575 trim_cache size 0 max 16384 2024-06-09T22:24:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:24:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:24:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:24:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeeab22640 1 -- 172.21.15.104:0/1646167572 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efecc0a2e20 msgr2=0x7efecc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeeab22640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efecc0a2e20 0x7efecc0a52e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x561320b25710 tx=0x7efed8009ba0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeeab22640 1 -- 172.21.15.104:0/1646167572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efee419caa0 msgr2=0x7efee419cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:24:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.769+0000 7efeeab22640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efee419caa0 0x7efee419cf50 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7efed4005ab0 tx=0x7efed4004040 comp rx=0 tx=0).stop 2024-06-09T22:24:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.770+0000 7efeeab22640 1 -- 172.21.15.104:0/1646167572 shutdown_connections 2024-06-09T22:24:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.770+0000 7efeeab22640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7efecc0a2e20 0x7efecc0a52e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.770+0000 7efeeab22640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efee419caa0 0x7efee419cf50 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.770+0000 7efeeab22640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee4106cd0 0x7efee4198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.770+0000 7efeeab22640 1 --2- 172.21.15.104:0/1646167572 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efee4100110 0x7efee41981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:24:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.770+0000 7efeeab22640 1 -- 172.21.15.104:0/1646167572 >> 172.21.15.104:0/1646167572 conn(0x7efee4076dc0 msgr2=0x7efee41054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:24:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.770+0000 7efeeab22640 1 -- 172.21.15.104:0/1646167572 shutdown_connections 2024-06-09T22:24:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:24:56.770+0000 7efeeab22640 1 -- 172.21.15.104:0/1646167572 wait complete. 2024-06-09T22:24:57.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:56 smithi194 ceph-mon[36856]: pgmap v122: 129 pgs: 129 active+clean; 63 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 722 KiB/s wr, 62 op/s 2024-06-09T22:24:57.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:56 smithi153 ceph-mon[36599]: pgmap v122: 129 pgs: 129 active+clean; 63 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 722 KiB/s wr, 62 op/s 2024-06-09T22:24:57.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:56 smithi104 ceph-mon[32389]: pgmap v122: 129 pgs: 129 active+clean; 63 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 722 KiB/s wr, 62 op/s 2024-06-09T22:24:57.724 INFO:tasks.workunit.client.0.smithi104.stdout: 1 75151 6.26 MB/sec execute 3 sec latency 58.636 ms 2024-06-09T22:24:58.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 75210 4.72 MB/sec execute 4 sec latency 980.521 ms 2024-06-09T22:24:59.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:24:58 smithi194 ceph-mon[36856]: pgmap v123: 129 pgs: 129 active+clean; 63 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 722 KiB/s wr, 62 op/s 2024-06-09T22:24:59.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:24:58 smithi153 ceph-mon[36599]: pgmap v123: 129 pgs: 129 active+clean; 63 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 722 KiB/s wr, 62 op/s 2024-06-09T22:24:59.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:24:58 smithi104 ceph-mon[32389]: pgmap v123: 129 pgs: 129 active+clean; 63 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 722 KiB/s wr, 62 op/s 2024-06-09T22:24:59.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 75210 3.78 MB/sec execute 5 sec latency 1980.607 ms 2024-06-09T22:25:00.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 75298 3.19 MB/sec execute 6 sec latency 2282.570 ms 2024-06-09T22:25:01.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:00 smithi194 ceph-mon[36856]: pgmap v124: 129 pgs: 129 active+clean; 59 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 976 KiB/s wr, 105 op/s 2024-06-09T22:25:01.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:00 smithi153 ceph-mon[36599]: pgmap v124: 129 pgs: 129 active+clean; 59 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 976 KiB/s wr, 105 op/s 2024-06-09T22:25:01.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:00 smithi104 ceph-mon[32389]: pgmap v124: 129 pgs: 129 active+clean; 59 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 976 KiB/s wr, 105 op/s 2024-06-09T22:25:01.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 75574 2.91 MB/sec execute 7 sec latency 372.498 ms 2024-06-09T22:25:02.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 77905 4.16 MB/sec execute 8 sec latency 57.932 ms 2024-06-09T22:25:03.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:02 smithi194 ceph-mon[36856]: pgmap v125: 129 pgs: 129 active+clean; 59 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 949 KiB/s wr, 87 op/s 2024-06-09T22:25:03.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:02 smithi153 ceph-mon[36599]: pgmap v125: 129 pgs: 129 active+clean; 59 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 949 KiB/s wr, 87 op/s 2024-06-09T22:25:03.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:02 smithi104 ceph-mon[32389]: pgmap v125: 129 pgs: 129 active+clean; 59 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 949 KiB/s wr, 87 op/s 2024-06-09T22:25:03.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 78877 4.26 MB/sec execute 9 sec latency 100.245 ms 2024-06-09T22:25:04.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 78877 3.83 MB/sec execute 10 sec latency 1100.344 ms 2024-06-09T22:25:04.938 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:25:05.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:04 smithi194 ceph-mon[36856]: pgmap v126: 129 pgs: 129 active+clean; 65 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 167 op/s 2024-06-09T22:25:05.195 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:05.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:04 smithi153 ceph-mon[36599]: pgmap v126: 129 pgs: 129 active+clean; 65 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 167 op/s 2024-06-09T22:25:05.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:04 smithi104 ceph-mon[32389]: pgmap v126: 129 pgs: 129 active+clean; 65 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 167 op/s 2024-06-09T22:25:05.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 78877 3.48 MB/sec execute 11 sec latency 2100.393 ms 2024-06-09T22:25:06.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 80554 3.80 MB/sec execute 12 sec latency 2486.902 ms 2024-06-09T22:25:06.966 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:06 smithi104 ceph-mon[32389]: pgmap v127: 129 pgs: 129 active+clean; 65 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 909 KiB/s wr, 122 op/s 2024-06-09T22:25:07.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:06 smithi194 ceph-mon[36856]: pgmap v127: 129 pgs: 129 active+clean; 65 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 909 KiB/s wr, 122 op/s 2024-06-09T22:25:07.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.076+0000 7f23e0217700 1 -- 172.21.15.104:0/4005562458 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23d8101380 msgr2=0x7f23d8101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:07.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.076+0000 7f23e0217700 1 --2- 172.21.15.104:0/4005562458 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23d8101380 0x7f23d8101800 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009990 tx=0x7f23c8009ca0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.076+0000 7f23e0217700 1 -- 172.21.15.104:0/4005562458 shutdown_connections 2024-06-09T22:25:07.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.076+0000 7f23e0217700 1 --2- 172.21.15.104:0/4005562458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23d8101d40 0x7f23d810e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.076+0000 7f23e0217700 1 --2- 172.21.15.104:0/4005562458 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23d81073f0 0x7f23d81077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.076+0000 7f23e0217700 1 --2- 172.21.15.104:0/4005562458 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23d8101380 0x7f23d8101800 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.076+0000 7f23e0217700 1 -- 172.21.15.104:0/4005562458 >> 172.21.15.104:0/4005562458 conn(0x7f23d8078300 msgr2=0x7f23d8078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:07.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.077+0000 7f23e0217700 1 -- 172.21.15.104:0/4005562458 shutdown_connections 2024-06-09T22:25:07.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.077+0000 7f23e0217700 1 -- 172.21.15.104:0/4005562458 wait complete. 2024-06-09T22:25:07.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.077+0000 7f23e0217700 1 Processor -- start 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.077+0000 7f23e0217700 1 -- start start 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.077+0000 7f23e0217700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23d8101380 0x7f23d819e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.077+0000 7f23e0217700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23d8101d40 0x7f23d819ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.077+0000 7f23e0217700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23d81073f0 0x7f23d81989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.077+0000 7f23e0217700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f23d8111660 con 0x7f23d8101380 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.077+0000 7f23e0217700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23d81114e0 con 0x7f23d81073f0 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.077+0000 7f23e0217700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23d81117e0 con 0x7f23d8101d40 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23ddfb3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23d8101380 0x7f23d819e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23ddfb3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23d8101380 0x7f23d819e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58072/0 (socket says 172.21.15.104:58072) 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23ddfb3700 1 -- 172.21.15.104:0/3349503472 learned_addr learned my addr 172.21.15.104:0/3349503472 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23dd7b2700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23d8101d40 0x7f23d819ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23de7b4700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23d81073f0 0x7f23d81989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23ddfb3700 1 -- 172.21.15.104:0/3349503472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23d8101d40 msgr2=0x7f23d819ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:07.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23d8101d40 0x7f23d819ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23ddfb3700 1 -- 172.21.15.104:0/3349503472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23d81073f0 msgr2=0x7f23d81989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:07.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23d81073f0 0x7f23d81989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23ddfb3700 1 -- 172.21.15.104:0/3349503472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d8199100 con 0x7f23d8101380 2024-06-09T22:25:07.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23d8101380 0x7f23d819e7e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f23d400e9a0 tx=0x7f23d400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:07.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23ceffd700 1 -- 172.21.15.104:0/3349503472 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d400ca10 con 0x7f23d8101380 2024-06-09T22:25:07.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23ceffd700 1 -- 172.21.15.104:0/3349503472 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23d400cbb0 con 0x7f23d8101380 2024-06-09T22:25:07.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23ceffd700 1 -- 172.21.15.104:0/3349503472 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4018680 con 0x7f23d8101380 2024-06-09T22:25:07.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23e0217700 1 -- 172.21.15.104:0/3349503472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23d81993f0 con 0x7f23d8101380 2024-06-09T22:25:07.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.078+0000 7f23e0217700 1 -- 172.21.15.104:0/3349503472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23d81a56b0 con 0x7f23d8101380 2024-06-09T22:25:07.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.081+0000 7f23ceffd700 1 -- 172.21.15.104:0/3349503472 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f23d4010430 con 0x7f23d8101380 2024-06-09T22:25:07.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.081+0000 7f23ceffd700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f23c40a2d90 0x7f23c40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.081+0000 7f23dd7b2700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f23c40a2d90 0x7f23c40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.081+0000 7f23ceffd700 1 -- 172.21.15.104:0/3349503472 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f23d40cf970 con 0x7f23d8101380 2024-06-09T22:25:07.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.082+0000 7f23dd7b2700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f23c40a2d90 0x7f23c40a5250 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f23c800b2e0 tx=0x7f23c8005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:07.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.082+0000 7f23eaa9ab80 1 Processor -- start 2024-06-09T22:25:07.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.082+0000 7f23eaa9ab80 1 -- start start 2024-06-09T22:25:07.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.082+0000 7f23eaa9ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 0x5564d63aaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.082+0000 7f23eaa9ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aaf60 0x5564d63a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23eaa9ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63a4f20 0x5564d63a5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23eaa9ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5564d62f6810 con 0x5564d63aa640 2024-06-09T22:25:07.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23eaa9ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5564d629e720 con 0x5564d63aaf60 2024-06-09T22:25:07.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23eaa9ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5564d62f27d0 con 0x5564d63a4f20 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23ddfb3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 0x5564d63aaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23ddfb3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 0x5564d63aaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58084/0 (socket says 172.21.15.104:58084) 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23ddfb3700 1 -- 172.21.15.104:0/3754357206 learned_addr learned my addr 172.21.15.104:0/3754357206 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23de7b4700 1 --2- 172.21.15.104:0/3754357206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63a4f20 0x5564d63a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23dd7b2700 1 --2- 172.21.15.104:0/3754357206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aaf60 0x5564d63a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23ddfb3700 1 -- 172.21.15.104:0/3754357206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63a4f20 msgr2=0x5564d63a5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/3754357206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63a4f20 0x5564d63a5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23ddfb3700 1 -- 172.21.15.104:0/3754357206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aaf60 msgr2=0x5564d63a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/3754357206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aaf60 0x5564d63a49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23ddfb3700 1 -- 172.21.15.104:0/3754357206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564d6294f50 con 0x5564d63aa640 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23de7b4700 1 --2- 172.21.15.104:0/3754357206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63a4f20 0x5564d63a5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/3754357206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 0x5564d63aaa20 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f23d40984f0 tx=0x7f23d409bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec3c9e354e889f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23c3fff700 1 -- 172.21.15.104:0/3754357206 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d40a4790 con 0x5564d63aa640 2024-06-09T22:25:07.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23c3fff700 1 -- 172.21.15.104:0/3754357206 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23d409ce50 con 0x5564d63aa640 2024-06-09T22:25:07.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.083+0000 7f23c3fff700 1 -- 172.21.15.104:0/3754357206 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4010980 con 0x5564d63aa640 2024-06-09T22:25:07.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.084+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/3754357206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 msgr2=0x5564d63aaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:07.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.084+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/3754357206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 0x5564d63aaa20 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f23d40984f0 tx=0x7f23d409bb20 comp rx=0 tx=0).stop 2024-06-09T22:25:07.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.084+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/3754357206 shutdown_connections 2024-06-09T22:25:07.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.084+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/3754357206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 0x5564d63aaa20 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.084+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/3754357206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63a4f20 0x5564d63a5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.084+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/3754357206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aaf60 0x5564d63a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.084+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/3754357206 >> 172.21.15.104:0/3754357206 conn(0x5564d62cfcc0 msgr2=0x5564d639f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:07.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.084+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/3754357206 shutdown_connections 2024-06-09T22:25:07.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.084+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/3754357206 wait complete. 2024-06-09T22:25:07.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.085+0000 7f23eaa9ab80 1 -- start start 2024-06-09T22:25:07.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.085+0000 7f23eaa9ab80 1 Processor -- start 2024-06-09T22:25:07.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.085+0000 7f23eaa9ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63a4f20 0x5564d63b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.085+0000 7f23eaa9ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63aa640 0x5564d63ae010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.085+0000 7f23eaa9ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aaf60 0x5564d63ae550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.085+0000 7f23eaa9ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5564d63b19d0 con 0x5564d63aaf60 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.085+0000 7f23eaa9ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5564d6296bf0 con 0x5564d63a4f20 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.085+0000 7f23eaa9ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5564d63b1b50 con 0x5564d63aa640 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23eaa9ab80 10 client.0 _ll_register_callbacks cb 0x5564d6290920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23ddfb3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63a4f20 0x5564d63b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23ddfb3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63a4f20 0x5564d63b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:38832/0 (socket says 172.21.15.104:38832) 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23ddfb3700 1 -- 172.21.15.104:0/17306085 learned_addr learned my addr 172.21.15.104:0/17306085 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23dd7b2700 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63aa640 0x5564d63ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23de7b4700 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aaf60 0x5564d63ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23ddfb3700 1 -- 172.21.15.104:0/17306085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63aa640 msgr2=0x5564d63ae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63aa640 0x5564d63ae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23ddfb3700 1 -- 172.21.15.104:0/17306085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aaf60 msgr2=0x5564d63ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aaf60 0x5564d63ae550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23ddfb3700 1 -- 172.21.15.104:0/17306085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564d62458d0 con 0x5564d63a4f20 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63a4f20 0x5564d63b0d10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009230 tx=0x7f23d40ccd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49fb80209544fd61 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23c17fa700 1 -- 172.21.15.104:0/17306085 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d40b8020 con 0x5564d63a4f20 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23c17fa700 1 -- 172.21.15.104:0/17306085 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f23d409bf80 con 0x5564d63a4f20 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/17306085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23d409ce50 con 0x5564d63a4f20 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23c17fa700 1 -- 172.21.15.104:0/17306085 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f23d40ba2b0 con 0x5564d63a4f20 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23eaa9ab80 10 client.25955 fetch_fsmap learned FSMap version 30 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.086+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/17306085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5564d63c0730 con 0x5564d63a4f20 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.087+0000 7f23c17fa700 1 -- 172.21.15.104:0/17306085 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23d409ce50 con 0x5564d63a4f20 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.087+0000 7f23c17fa700 1 -- 172.21.15.104:0/17306085 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f23d409b090 con 0x5564d63a4f20 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.087+0000 7f23eaa9ab80 10 client.25955 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.087+0000 7f23eaa9ab80 10 client.25955 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:07.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.087+0000 7f23eaa9ab80 20 client.25955 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.087+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5564d63c2c00 0x5564d63c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.087+0000 7f23eaa9ab80 4 client.25955 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.087+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/17306085 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564d6303d10 con 0x5564d63c2c00 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.087+0000 7f23dd7b2700 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5564d63c2c00 0x5564d63c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.088+0000 7f23dd7b2700 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5564d63c2c00 0x5564d63c4ff0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=714bd62e8e877166 server_cookie=25313172ec88daf in_seq=0 out_seq=0 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.088+0000 7f23c17fa700 10 client.25955 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.088+0000 7f23c17fa700 1 -- 172.21.15.104:0/17306085 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5564d6303d10 con 0x5564d63c2c00 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.088+0000 7f23c17fa700 10 client.25955 handle_command_reply: tid=0 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.088+0000 7f23eaa9ab80 1 client.25955 shutdown 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.088+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/17306085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63a4f20 msgr2=0x5564d63b0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.088+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63a4f20 0x5564d63b0d10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009230 tx=0x7f23d40ccd70 comp rx=0 tx=0).stop 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.089+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/17306085 shutdown_connections 2024-06-09T22:25:07.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.089+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5564d63c2c00 0x5564d63c4ff0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.089+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aaf60 0x5564d63ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.089+0000 7f23c17fa700 0 client.25955 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:07.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.089+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63a4f20 0x5564d63b0d10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.089+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/17306085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63aa640 0x5564d63ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.089+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/17306085 >> 172.21.15.104:0/17306085 conn(0x5564d62cfcc0 msgr2=0x5564d63a39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:07.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.089+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/17306085 shutdown_connections 2024-06-09T22:25:07.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.089+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/17306085 wait complete. 2024-06-09T22:25:07.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.089+0000 7f23eaa9ab80 20 client.25955 trim_cache size 0 max 16384 2024-06-09T22:25:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.110+0000 7f23eaa9ab80 1 Processor -- start 2024-06-09T22:25:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.110+0000 7f23eaa9ab80 1 -- start start 2024-06-09T22:25:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.110+0000 7f23eaa9ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63a4f20 0x5564d63a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.110+0000 7f23eaa9ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 0x5564d63abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.110+0000 7f23eaa9ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aaf60 0x5564d63ad010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.110+0000 7f23eaa9ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f23d81ac3b0 con 0x5564d63aa640 2024-06-09T22:25:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.110+0000 7f23eaa9ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23d81adf50 con 0x5564d63aaf60 2024-06-09T22:25:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.110+0000 7f23eaa9ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23d81aca80 con 0x5564d63a4f20 2024-06-09T22:25:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23dd7b2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 0x5564d63abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23dd7b2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 0x5564d63abbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58104/0 (socket says 172.21.15.104:58104) 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23dd7b2700 1 -- 172.21.15.104:0/759962524 learned_addr learned my addr 172.21.15.104:0/759962524 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/759962524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63a4f20 0x5564d63a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23de7b4700 1 --2- 172.21.15.104:0/759962524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aaf60 0x5564d63ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23de7b4700 1 -- 172.21.15.104:0/759962524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63a4f20 msgr2=0x5564d63a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23de7b4700 1 --2- 172.21.15.104:0/759962524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63a4f20 0x5564d63a49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23de7b4700 1 -- 172.21.15.104:0/759962524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 msgr2=0x5564d63abbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23de7b4700 1 --2- 172.21.15.104:0/759962524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 0x5564d63abbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23de7b4700 1 -- 172.21.15.104:0/759962524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564d63ac580 con 0x5564d63aaf60 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23dd7b2700 1 --2- 172.21.15.104:0/759962524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 0x5564d63abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23de7b4700 1 --2- 172.21.15.104:0/759962524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aaf60 0x5564d63ad010 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009080 tx=0x7f23d0009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74b53ed0fb8adb5f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23c37fe700 1 -- 172.21.15.104:0/759962524 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d00046b0 con 0x5564d63aaf60 2024-06-09T22:25:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23c37fe700 1 -- 172.21.15.104:0/759962524 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23d0003ae0 con 0x5564d63aaf60 2024-06-09T22:25:07.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.111+0000 7f23c37fe700 1 -- 172.21.15.104:0/759962524 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d00208e0 con 0x5564d63aaf60 2024-06-09T22:25:07.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.112+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/759962524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aaf60 msgr2=0x5564d63ad010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:07.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.112+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/759962524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aaf60 0x5564d63ad010 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009080 tx=0x7f23d0009920 comp rx=0 tx=0).stop 2024-06-09T22:25:07.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.112+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/759962524 shutdown_connections 2024-06-09T22:25:07.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.112+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/759962524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aaf60 0x5564d63ad010 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.112+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/759962524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63a4f20 0x5564d63a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.112+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/759962524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63aa640 0x5564d63abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.112+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/759962524 >> 172.21.15.104:0/759962524 conn(0x5564d62cfcc0 msgr2=0x5564d63ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:07.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.112+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/759962524 shutdown_connections 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.112+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/759962524 wait complete. 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23eaa9ab80 1 -- start start 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23eaa9ab80 1 Processor -- start 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23eaa9ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63a4f20 0x5564d63c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23eaa9ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aa640 0x5564d63c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23eaa9ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63aaf60 0x5564d63c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23eaa9ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f23d81ad820 con 0x5564d63a4f20 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23eaa9ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23d81ad150 con 0x5564d63aa640 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23eaa9ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23d81aa330 con 0x5564d63aaf60 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23eaa9ab80 10 client.0 _ll_register_callbacks cb 0x5564d6290920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23ddfb3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63a4f20 0x5564d63c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23ddfb3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63a4f20 0x5564d63c5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58118/0 (socket says 172.21.15.104:58118) 2024-06-09T22:25:07.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23ddfb3700 1 -- 172.21.15.104:0/2630014994 learned_addr learned my addr 172.21.15.104:0/2630014994 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23dd7b2700 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aa640 0x5564d63c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.113+0000 7f23de7b4700 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63aaf60 0x5564d63c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23ddfb3700 1 -- 172.21.15.104:0/2630014994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63aaf60 msgr2=0x5564d63c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63aaf60 0x5564d63c68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23ddfb3700 1 -- 172.21.15.104:0/2630014994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aa640 msgr2=0x5564d63c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aa640 0x5564d63c63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23ddfb3700 1 -- 172.21.15.104:0/2630014994 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564d642c5f0 con 0x5564d63a4f20 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23dd7b2700 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aa640 0x5564d63c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23ddfb3700 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63a4f20 0x5564d63c5e60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f23d409a9e0 tx=0x7f23d40cce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f359e75dd965c6ed server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23c17fa700 1 -- 172.21.15.104:0/2630014994 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d40b8070 con 0x5564d63a4f20 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23c17fa700 1 -- 172.21.15.104:0/2630014994 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f23d409af70 con 0x5564d63a4f20 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/2630014994 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5564d62fe290 con 0x5564d63a4f20 2024-06-09T22:25:07.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23c17fa700 1 -- 172.21.15.104:0/2630014994 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f23d409bb20 con 0x5564d63a4f20 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23eaa9ab80 10 client.16593 fetch_fsmap learned FSMap version 30 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/2630014994 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5564d642d830 con 0x5564d63a4f20 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.114+0000 7f23c17fa700 1 -- 172.21.15.104:0/2630014994 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5564d62fe290 con 0x5564d63a4f20 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23c17fa700 1 -- 172.21.15.104:0/2630014994 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f23d40a1260 con 0x5564d63a4f20 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23eaa9ab80 10 client.16593 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23eaa9ab80 10 client.16593 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23eaa9ab80 20 client.16593 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5564d642ddb0 0x5564d64301d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23eaa9ab80 4 client.16593 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/2630014994 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5564d6178250 con 0x5564d642ddb0 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23dd7b2700 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5564d642ddb0 0x5564d64301d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23dd7b2700 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5564d642ddb0 0x5564d64301d0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0dd4d29fe782276 server_cookie=c0cd1d1fb134dfdc in_seq=0 out_seq=0 2024-06-09T22:25:07.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23c17fa700 10 client.16593 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23c17fa700 1 -- 172.21.15.104:0/2630014994 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5564d6178250 con 0x5564d642ddb0 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23c17fa700 10 client.16593 handle_command_reply: tid=0 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23eaa9ab80 1 client.16593 shutdown 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/2630014994 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63a4f20 msgr2=0x5564d63c5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.115+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63a4f20 0x5564d63c5e60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f23d409a9e0 tx=0x7f23d40cce30 comp rx=0 tx=0).stop 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/2630014994 shutdown_connections 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5564d642ddb0 0x5564d64301d0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5564d63a4f20 0x5564d63c5e60 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564d63aaf60 0x5564d63c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23eaa9ab80 1 --2- 172.21.15.104:0/2630014994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564d63aa640 0x5564d63c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/2630014994 >> 172.21.15.104:0/2630014994 conn(0x5564d62cfcc0 msgr2=0x5564d63aba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23c17fa700 0 client.16593 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:07.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/2630014994 shutdown_connections 2024-06-09T22:25:07.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23eaa9ab80 1 -- 172.21.15.104:0/2630014994 wait complete. 2024-06-09T22:25:07.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23eaa9ab80 20 client.16593 trim_cache size 0 max 16384 2024-06-09T22:25:07.122 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:25:07.122 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "no active scrubs running", 2024-06-09T22:25:07.122 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": {} 2024-06-09T22:25:07.122 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:25:07.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23e0217700 1 -- 172.21.15.104:0/3349503472 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f23c40a2d90 msgr2=0x7f23c40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:07.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23e0217700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f23c40a2d90 0x7f23c40a5250 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f23c800b2e0 tx=0x7f23c8005960 comp rx=0 tx=0).stop 2024-06-09T22:25:07.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23e0217700 1 -- 172.21.15.104:0/3349503472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23d8101380 msgr2=0x7f23d819e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:07.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23e0217700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23d8101380 0x7f23d819e7e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f23d400e9a0 tx=0x7f23d400ecb0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23e0217700 1 -- 172.21.15.104:0/3349503472 shutdown_connections 2024-06-09T22:25:07.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23e0217700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f23c40a2d90 0x7f23c40a5250 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23e0217700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23d8101380 0x7f23d819e7e0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23e0217700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23d81073f0 0x7f23d81989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23e0217700 1 --2- 172.21.15.104:0/3349503472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23d8101d40 0x7f23d819ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:07.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23e0217700 1 -- 172.21.15.104:0/3349503472 >> 172.21.15.104:0/3349503472 conn(0x7f23d8078300 msgr2=0x7f23d8100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:07.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.116+0000 7f23e0217700 1 -- 172.21.15.104:0/3349503472 shutdown_connections 2024-06-09T22:25:07.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:07.117+0000 7f23e0217700 1 -- 172.21.15.104:0/3349503472 wait complete. 2024-06-09T22:25:07.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:06 smithi153 ceph-mon[36599]: pgmap v127: 129 pgs: 129 active+clean; 65 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 909 KiB/s wr, 122 op/s 2024-06-09T22:25:07.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 82174 4.39 MB/sec execute 13 sec latency 19.271 ms 2024-06-09T22:25:08.134 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-09T22:25:08.135 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:25:08.382 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:08.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 82302 4.08 MB/sec execute 14 sec latency 932.850 ms 2024-06-09T22:25:09.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:08 smithi194 ceph-mon[36856]: pgmap v128: 129 pgs: 129 active+clean; 65 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 909 KiB/s wr, 122 op/s 2024-06-09T22:25:09.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:08 smithi153 ceph-mon[36599]: pgmap v128: 129 pgs: 129 active+clean; 65 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 909 KiB/s wr, 122 op/s 2024-06-09T22:25:09.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:08 smithi104 ceph-mon[32389]: pgmap v128: 129 pgs: 129 active+clean; 65 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 909 KiB/s wr, 122 op/s 2024-06-09T22:25:09.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 82302 3.81 MB/sec execute 15 sec latency 1932.877 ms 2024-06-09T22:25:09.864 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.861+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/3034754760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2b281019b0 msgr2=0x7f2b28101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.861+0000 7f2b2e2cd640 1 --2- 172.21.15.104:0/3034754760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2b281019b0 0x7f2b28101e30 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f2b18009930 tx=0x7f2b1802a420 comp rx=0 tx=0).stop 2024-06-09T22:25:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.861+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/3034754760 shutdown_connections 2024-06-09T22:25:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.861+0000 7f2b2e2cd640 1 --2- 172.21.15.104:0/3034754760 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b28102370 0x7f2b2810e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.861+0000 7f2b2e2cd640 1 --2- 172.21.15.104:0/3034754760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2b281019b0 0x7f2b28101e30 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.861+0000 7f2b2e2cd640 1 --2- 172.21.15.104:0/3034754760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b281079c0 0x7f2b28107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.861+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/3034754760 >> 172.21.15.104:0/3034754760 conn(0x7f2b28078e30 msgr2=0x7f2b28079240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.861+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/3034754760 shutdown_connections 2024-06-09T22:25:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.861+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/3034754760 wait complete. 2024-06-09T22:25:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.862+0000 7f2b2e2cd640 1 Processor -- start 2024-06-09T22:25:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.862+0000 7f2b2e2cd640 1 -- start start 2024-06-09T22:25:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.862+0000 7f2b2e2cd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b281019b0 0x7f2b28198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.862+0000 7f2b2e2cd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2b28102370 0x7f2b28198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.862+0000 7f2b2e2cd640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b281079c0 0x7f2b2819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.862+0000 7f2b2e2cd640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2b28111e20 con 0x7f2b28102370 2024-06-09T22:25:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.862+0000 7f2b2e2cd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2b28111ca0 con 0x7f2b281019b0 2024-06-09T22:25:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.862+0000 7f2b2e2cd640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2b28111fa0 con 0x7f2b281079c0 2024-06-09T22:25:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b277fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2b28102370 0x7f2b28198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b277fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2b28102370 0x7f2b28198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58130/0 (socket says 172.21.15.104:58130) 2024-06-09T22:25:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b277fe640 1 -- 172.21.15.104:0/4082478502 learned_addr learned my addr 172.21.15.104:0/4082478502 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b2c86a640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b281079c0 0x7f2b2819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b27fff640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b281019b0 0x7f2b28198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b277fe640 1 -- 172.21.15.104:0/4082478502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b281079c0 msgr2=0x7f2b2819ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b277fe640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b281079c0 0x7f2b2819ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b277fe640 1 -- 172.21.15.104:0/4082478502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b281019b0 msgr2=0x7f2b28198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b277fe640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b281019b0 0x7f2b28198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b277fe640 1 -- 172.21.15.104:0/4082478502 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b2819d1c0 con 0x7f2b28102370 2024-06-09T22:25:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b277fe640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2b28102370 0x7f2b28198760 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f2b1802a940 tx=0x7f2b18003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b257fa640 1 -- 172.21.15.104:0/4082478502 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1803f070 con 0x7f2b28102370 2024-06-09T22:25:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/4082478502 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b2819d450 con 0x7f2b28102370 2024-06-09T22:25:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b257fa640 1 -- 172.21.15.104:0/4082478502 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b18031c90 con 0x7f2b28102370 2024-06-09T22:25:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b257fa640 1 -- 172.21.15.104:0/4082478502 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b18035030 con 0x7f2b28102370 2024-06-09T22:25:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.863+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/4082478502 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b281a4d40 con 0x7f2b28102370 2024-06-09T22:25:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b257fa640 1 -- 172.21.15.104:0/4082478502 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f2b18042070 con 0x7f2b28102370 2024-06-09T22:25:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b30fd9740 1 Processor -- start 2024-06-09T22:25:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b30fd9740 1 -- start start 2024-06-09T22:25:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b30fd9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0b3170 0x56050d0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b30fd9740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 0x56050d0bde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b30fd9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0be350 0x56050d0c0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b30fd9740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56050d0174f0 con 0x56050d0b3a90 2024-06-09T22:25:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b257fa640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2b140a2da0 0x7f2b140a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b30fd9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56050cfb5870 con 0x56050d0be350 2024-06-09T22:25:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b30fd9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56050d0134b0 con 0x56050d0b3170 2024-06-09T22:25:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b27fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 0x56050d0bde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b27fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 0x56050d0bde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58144/0 (socket says 172.21.15.104:58144) 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b2c86a640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2b140a2da0 0x7f2b140a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b27fff640 1 -- 172.21.15.104:0/1872038006 learned_addr learned my addr 172.21.15.104:0/1872038006 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b277fe640 1 --2- 172.21.15.104:0/1872038006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0be350 0x56050d0c0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.866+0000 7f2b2c86a640 1 --2- 172.21.15.104:0/1872038006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0b3170 0x56050d0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.867+0000 7f2b257fa640 1 -- 172.21.15.104:0/4082478502 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f2b180efea0 con 0x7f2b28102370 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.867+0000 7f2b27fff640 1 -- 172.21.15.104:0/1872038006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0b3170 msgr2=0x56050d0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.867+0000 7f2b27fff640 1 --2- 172.21.15.104:0/1872038006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0b3170 0x56050d0b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.867+0000 7f2b27fff640 1 -- 172.21.15.104:0/1872038006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0be350 msgr2=0x56050d0c0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.867+0000 7f2b27fff640 1 --2- 172.21.15.104:0/1872038006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0be350 0x56050d0c0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.867+0000 7f2b27fff640 1 -- 172.21.15.104:0/1872038006 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56050cfd19b0 con 0x56050d0b3a90 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.867+0000 7f2b2c86a640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2b140a2da0 0x7f2b140a5260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2b20005fd0 tx=0x7f2b20009dd0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.867+0000 7f2b27fff640 1 --2- 172.21.15.104:0/1872038006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 0x56050d0bde10 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2b10009c70 tx=0x7f2b10011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916889ab06acad20 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.867+0000 7f2b1e7fc640 1 -- 172.21.15.104:0/1872038006 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10004230 con 0x56050d0b3a90 2024-06-09T22:25:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.867+0000 7f2b1e7fc640 1 -- 172.21.15.104:0/1872038006 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b100043d0 con 0x56050d0b3a90 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.867+0000 7f2b1e7fc640 1 -- 172.21.15.104:0/1872038006 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10019800 con 0x56050d0b3a90 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.868+0000 7f2b30fd9740 1 -- 172.21.15.104:0/1872038006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 msgr2=0x56050d0bde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.868+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/1872038006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 0x56050d0bde10 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2b10009c70 tx=0x7f2b10011810 comp rx=0 tx=0).stop 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.868+0000 7f2b30fd9740 1 -- 172.21.15.104:0/1872038006 shutdown_connections 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.868+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/1872038006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0be350 0x56050d0c0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.868+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/1872038006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 0x56050d0bde10 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.868+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/1872038006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0b3170 0x56050d0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.868+0000 7f2b30fd9740 1 -- 172.21.15.104:0/1872038006 >> 172.21.15.104:0/1872038006 conn(0x56050d0af680 msgr2=0x56050d0afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.868+0000 7f2b30fd9740 1 -- 172.21.15.104:0/1872038006 shutdown_connections 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.868+0000 7f2b30fd9740 1 -- 172.21.15.104:0/1872038006 wait complete. 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.868+0000 7f2b30fd9740 1 -- start start 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.868+0000 7f2b30fd9740 1 Processor -- start 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b30fd9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3170 0x56050d0bb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b30fd9740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 0x56050d0bb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b30fd9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0be350 0x56050d0bbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b30fd9740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56050d0c0f80 con 0x56050d0b3a90 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b30fd9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56050cfa0ae0 con 0x56050d0b3170 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b30fd9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56050d0c1100 con 0x56050d0be350 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b30fd9740 10 client.0 _ll_register_callbacks cb 0x56050cfd99b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b277fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 0x56050d0bb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b277fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 0x56050d0bb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58146/0 (socket says 172.21.15.104:58146) 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b277fe640 1 -- 172.21.15.104:0/325012241 learned_addr learned my addr 172.21.15.104:0/325012241 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b27fff640 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3170 0x56050d0bb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b2c86a640 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0be350 0x56050d0bbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b277fe640 1 -- 172.21.15.104:0/325012241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0be350 msgr2=0x56050d0bbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b277fe640 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0be350 0x56050d0bbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b277fe640 1 -- 172.21.15.104:0/325012241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3170 msgr2=0x56050d0bb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b277fe640 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3170 0x56050d0bb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b277fe640 1 -- 172.21.15.104:0/325012241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56050cd2b210 con 0x56050d0b3a90 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b277fe640 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 0x56050d0bb780 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f2b18000c00 tx=0x7f2b1802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97edf8b6b13bbe0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b077fe640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b077fe640 1 -- 172.21.15.104:0/325012241 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b18049070 con 0x56050d0b3a90 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b077fe640 1 -- 172.21.15.104:0/325012241 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f2b180c0080 con 0x56050d0b3a90 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.869+0000 7f2b30fd9740 1 -- 172.21.15.104:0/325012241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b100043d0 con 0x56050d0b3a90 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.870+0000 7f2b077fe640 1 -- 172.21.15.104:0/325012241 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2b180bc100 con 0x56050d0b3a90 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.870+0000 7f2b30fd9740 10 client.16605 fetch_fsmap learned FSMap version 30 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.870+0000 7f2b30fd9740 1 -- 172.21.15.104:0/325012241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56050cdf7620 con 0x56050d0b3a90 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.870+0000 7f2b077fe640 1 -- 172.21.15.104:0/325012241 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b100043d0 con 0x56050d0b3a90 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.870+0000 7f2b077fe640 1 -- 172.21.15.104:0/325012241 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f2b1803bce0 con 0x56050d0b3a90 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.870+0000 7f2b30fd9740 10 client.16605 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.870+0000 7f2b30fd9740 10 client.16605 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.870+0000 7f2b30fd9740 20 client.16605 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.870+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56050d0ce400 0x56050d0d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.870+0000 7f2b30fd9740 4 client.16605 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.870+0000 7f2b30fd9740 1 -- 172.21.15.104:0/325012241 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56050d02af10 con 0x56050d0ce400 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.870+0000 7f2b27fff640 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56050d0ce400 0x56050d0d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.871+0000 7f2b077fe640 10 client.16605 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.871+0000 7f2b27fff640 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56050d0ce400 0x56050d0d08c0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ed7d0dfba461693 server_cookie=a90352508e17b753 in_seq=0 out_seq=0 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.871+0000 7f2b077fe640 1 -- 172.21.15.104:0/325012241 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56050d02af10 con 0x56050d0ce400 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.871+0000 7f2b077fe640 10 client.16605 handle_command_reply: tid=0 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.871+0000 7f2b30fd9740 1 client.16605 shutdown 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.872+0000 7f2b30fd9740 1 -- 172.21.15.104:0/325012241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 msgr2=0x56050d0bb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.872+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 0x56050d0bb780 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f2b18000c00 tx=0x7f2b1802a420 comp rx=0 tx=0).stop 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.872+0000 7f2b30fd9740 1 -- 172.21.15.104:0/325012241 shutdown_connections 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.872+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56050d0ce400 0x56050d0d08c0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.872+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0be350 0x56050d0bbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.872+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3a90 0x56050d0bb780 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.872+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/325012241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3170 0x56050d0bb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.872+0000 7f2b077fe640 0 client.16605 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.872+0000 7f2b30fd9740 1 -- 172.21.15.104:0/325012241 >> 172.21.15.104:0/325012241 conn(0x56050d0af680 msgr2=0x56050d0b53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.872+0000 7f2b30fd9740 1 -- 172.21.15.104:0/325012241 shutdown_connections 2024-06-09T22:25:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.872+0000 7f2b30fd9740 1 -- 172.21.15.104:0/325012241 wait complete. 2024-06-09T22:25:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.872+0000 7f2b30fd9740 20 client.16605 trim_cache size 0 max 16384 2024-06-09T22:25:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b30fd9740 1 Processor -- start 2024-06-09T22:25:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b30fd9740 1 -- start start 2024-06-09T22:25:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b30fd9740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3170 0x56050d0aef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b30fd9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3a90 0x56050d0b5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b30fd9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0be350 0x56050d0d61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b30fd9740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2b281aec20 con 0x56050d0b3170 2024-06-09T22:25:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b30fd9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2b281b07c0 con 0x56050d0b3a90 2024-06-09T22:25:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b30fd9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2b281af2f0 con 0x56050d0be350 2024-06-09T22:25:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b27fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3170 0x56050d0aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b27fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3170 0x56050d0aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58162/0 (socket says 172.21.15.104:58162) 2024-06-09T22:25:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b27fff640 1 -- 172.21.15.104:0/2695662354 learned_addr learned my addr 172.21.15.104:0/2695662354 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b2c86a640 1 --2- 172.21.15.104:0/2695662354 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0be350 0x56050d0d61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b277fe640 1 --2- 172.21.15.104:0/2695662354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3a90 0x56050d0b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b27fff640 1 -- 172.21.15.104:0/2695662354 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0be350 msgr2=0x56050d0d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b27fff640 1 --2- 172.21.15.104:0/2695662354 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0be350 0x56050d0d61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b27fff640 1 -- 172.21.15.104:0/2695662354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3a90 msgr2=0x56050d0b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b27fff640 1 --2- 172.21.15.104:0/2695662354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3a90 0x56050d0b5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b27fff640 1 -- 172.21.15.104:0/2695662354 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56050d0ca1e0 con 0x56050d0b3170 2024-06-09T22:25:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b277fe640 1 --2- 172.21.15.104:0/2695662354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3a90 0x56050d0b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:25:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.887+0000 7f2b2c86a640 1 --2- 172.21.15.104:0/2695662354 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0be350 0x56050d0d61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:25:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b27fff640 1 --2- 172.21.15.104:0/2695662354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3170 0x56050d0aef60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x56050d0b4710 tx=0x7f2b100097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8107a5b9b6062404 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b1dffb640 1 -- 172.21.15.104:0/2695662354 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b100035f0 con 0x56050d0b3170 2024-06-09T22:25:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b1dffb640 1 -- 172.21.15.104:0/2695662354 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b10005030 con 0x56050d0b3170 2024-06-09T22:25:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b1dffb640 1 -- 172.21.15.104:0/2695662354 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10019730 con 0x56050d0b3170 2024-06-09T22:25:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2695662354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3170 msgr2=0x56050d0aef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/2695662354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3170 0x56050d0aef60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x56050d0b4710 tx=0x7f2b100097a0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2695662354 shutdown_connections 2024-06-09T22:25:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/2695662354 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0be350 0x56050d0d61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/2695662354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3a90 0x56050d0b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/2695662354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0b3170 0x56050d0aef60 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2695662354 >> 172.21.15.104:0/2695662354 conn(0x56050d0af680 msgr2=0x56050d0b4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2695662354 shutdown_connections 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.888+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2695662354 wait complete. 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b30fd9740 1 -- start start 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b30fd9740 1 Processor -- start 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b30fd9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3170 0x56050d0c6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b30fd9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0b3a90 0x56050d0c6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b30fd9740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0be350 0x56050d0c3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b30fd9740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2b281b0090 con 0x56050d0be350 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b30fd9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2b281af9c0 con 0x56050d0b3170 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b30fd9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2b281acbe0 con 0x56050d0b3a90 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b30fd9740 10 client.0 _ll_register_callbacks cb 0x7f2b18009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b2c86a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0be350 0x56050d0c3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b2c86a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0be350 0x56050d0c3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58164/0 (socket says 172.21.15.104:58164) 2024-06-09T22:25:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b2c86a640 1 -- 172.21.15.104:0/2125063463 learned_addr learned my addr 172.21.15.104:0/2125063463 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b277fe640 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0b3a90 0x56050d0c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b27fff640 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3170 0x56050d0c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b2c86a640 1 -- 172.21.15.104:0/2125063463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0b3a90 msgr2=0x56050d0c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b2c86a640 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0b3a90 0x56050d0c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b2c86a640 1 -- 172.21.15.104:0/2125063463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3170 msgr2=0x56050d0c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b2c86a640 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3170 0x56050d0c6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.889+0000 7f2b2c86a640 1 -- 172.21.15.104:0/2125063463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56050d0c3860 con 0x56050d0be350 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b2c86a640 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0be350 0x56050d0c3320 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2b20010d20 tx=0x7f2b2001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7930ea711051ee3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b077fe640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b077fe640 1 -- 172.21.15.104:0/2125063463 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b20003e00 con 0x56050d0be350 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b077fe640 1 -- 172.21.15.104:0/2125063463 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f2b20004530 con 0x56050d0be350 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2125063463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b10005030 con 0x56050d0be350 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b077fe640 1 -- 172.21.15.104:0/2125063463 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2b20032050 con 0x56050d0be350 2024-06-09T22:25:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b30fd9740 10 client.16611 fetch_fsmap learned FSMap version 30 2024-06-09T22:25:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2125063463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56050d13dce0 con 0x56050d0be350 2024-06-09T22:25:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b077fe640 1 -- 172.21.15.104:0/2125063463 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b10005030 con 0x56050d0be350 2024-06-09T22:25:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b077fe640 1 -- 172.21.15.104:0/2125063463 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f2b2001f070 con 0x56050d0be350 2024-06-09T22:25:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b30fd9740 10 client.16611 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:25:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b30fd9740 10 client.16611 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b30fd9740 20 client.16611 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56050d13e4f0 0x56050d140910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b30fd9740 4 client.16611 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:25:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2125063463 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56050d024c20 con 0x56050d13e4f0 2024-06-09T22:25:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.890+0000 7f2b27fff640 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56050d13e4f0 0x56050d140910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.891+0000 7f2b27fff640 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56050d13e4f0 0x56050d140910 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be705ca98be897ad server_cookie=493e1458f05cbe74 in_seq=0 out_seq=0 2024-06-09T22:25:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.891+0000 7f2b077fe640 10 client.16611 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b077fe640 1 -- 172.21.15.104:0/2125063463 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56050d024c20 con 0x56050d13e4f0 2024-06-09T22:25:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b077fe640 10 client.16611 handle_command_reply: tid=0 2024-06-09T22:25:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b30fd9740 1 client.16611 shutdown 2024-06-09T22:25:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2125063463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0be350 msgr2=0x56050d0c3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0be350 0x56050d0c3320 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2b20010d20 tx=0x7f2b2001c000 comp rx=0 tx=0).stop 2024-06-09T22:25:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2125063463 shutdown_connections 2024-06-09T22:25:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56050d13e4f0 0x56050d140910 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56050d0be350 0x56050d0c3320 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b077fe640 0 client.16611 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56050d0b3a90 0x56050d0c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b30fd9740 1 --2- 172.21.15.104:0/2125063463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56050d0b3170 0x56050d0c6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2125063463 >> 172.21.15.104:0/2125063463 conn(0x56050d0af680 msgr2=0x56050d0b53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2125063463 shutdown_connections 2024-06-09T22:25:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b30fd9740 1 -- 172.21.15.104:0/2125063463 wait complete. 2024-06-09T22:25:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.901+0000 7f2b30fd9740 20 client.16611 trim_cache size 0 max 16384 2024-06-09T22:25:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:25:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:25:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:25:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.902+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/4082478502 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2b140a2da0 msgr2=0x7f2b140a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.902+0000 7f2b2e2cd640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2b140a2da0 0x7f2b140a5260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2b20005fd0 tx=0x7f2b20009dd0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.902+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/4082478502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2b28102370 msgr2=0x7f2b28198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.902+0000 7f2b2e2cd640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2b28102370 0x7f2b28198760 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f2b1802a940 tx=0x7f2b18003b90 comp rx=0 tx=0).stop 2024-06-09T22:25:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.902+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/4082478502 shutdown_connections 2024-06-09T22:25:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.902+0000 7f2b2e2cd640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2b140a2da0 0x7f2b140a5260 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.902+0000 7f2b2e2cd640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b281079c0 0x7f2b2819ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.902+0000 7f2b2e2cd640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2b28102370 0x7f2b28198760 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.902+0000 7f2b2e2cd640 1 --2- 172.21.15.104:0/4082478502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b281019b0 0x7f2b28198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.902+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/4082478502 >> 172.21.15.104:0/4082478502 conn(0x7f2b28078e30 msgr2=0x7f2b280ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.902+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/4082478502 shutdown_connections 2024-06-09T22:25:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:09.902+0000 7f2b2e2cd640 1 -- 172.21.15.104:0/4082478502 wait complete. 2024-06-09T22:25:10.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.207+0000 7ffa60272700 1 -- 172.21.15.104:0/4025664349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa58103150 msgr2=0x7ffa58107140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:10.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.207+0000 7ffa60272700 1 --2- 172.21.15.104:0/4025664349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa58103150 0x7ffa58107140 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ffa480099c0 tx=0x7ffa48009cd0 comp rx=0 tx=0).stop 2024-06-09T22:25:10.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.207+0000 7ffa60272700 1 -- 172.21.15.104:0/4025664349 shutdown_connections 2024-06-09T22:25:10.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.207+0000 7ffa60272700 1 --2- 172.21.15.104:0/4025664349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa58107870 0x7ffa5810f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:10.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.207+0000 7ffa60272700 1 --2- 172.21.15.104:0/4025664349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa581027a0 0x7ffa58102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:10.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.207+0000 7ffa60272700 1 --2- 172.21.15.104:0/4025664349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa58103150 0x7ffa58107140 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:10.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.207+0000 7ffa60272700 1 -- 172.21.15.104:0/4025664349 >> 172.21.15.104:0/4025664349 conn(0x7ffa58079570 msgr2=0x7ffa58079980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:10.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.207+0000 7ffa60272700 1 -- 172.21.15.104:0/4025664349 shutdown_connections 2024-06-09T22:25:10.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.207+0000 7ffa60272700 1 -- 172.21.15.104:0/4025664349 wait complete. 2024-06-09T22:25:10.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa60272700 1 Processor -- start 2024-06-09T22:25:10.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa60272700 1 -- start start 2024-06-09T22:25:10.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa60272700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa581027a0 0x7ffa5819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:10.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa60272700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa58103150 0x7ffa5819ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:10.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa60272700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa58107870 0x7ffa58198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:10.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa60272700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ffa58112800 con 0x7ffa58107870 2024-06-09T22:25:10.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa60272700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa58112680 con 0x7ffa581027a0 2024-06-09T22:25:10.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa60272700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffa58112980 con 0x7ffa58103150 2024-06-09T22:25:10.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa5e80f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa58107870 0x7ffa58198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:10.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa5e80f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa58107870 0x7ffa58198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58184/0 (socket says 172.21.15.104:58184) 2024-06-09T22:25:10.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa5e80f700 1 -- 172.21.15.104:0/924193455 learned_addr learned my addr 172.21.15.104:0/924193455 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:10.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa5d80d700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa58103150 0x7ffa5819ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:10.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.208+0000 7ffa5e00e700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa581027a0 0x7ffa5819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.209+0000 7ffa5e80f700 1 -- 172.21.15.104:0/924193455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa58103150 msgr2=0x7ffa5819ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.209+0000 7ffa5e80f700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa58103150 0x7ffa5819ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.209+0000 7ffa5e80f700 1 -- 172.21.15.104:0/924193455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa581027a0 msgr2=0x7ffa5819e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.209+0000 7ffa5e80f700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa581027a0 0x7ffa5819e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.209+0000 7ffa5e80f700 1 -- 172.21.15.104:0/924193455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa581991c0 con 0x7ffa58107870 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.209+0000 7ffa5e80f700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa58107870 0x7ffa58198a60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ffa5000b970 tx=0x7ffa5000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.209+0000 7ffa4f7fe700 1 -- 172.21.15.104:0/924193455 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50019070 con 0x7ffa58107870 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.209+0000 7ffa60272700 1 -- 172.21.15.104:0/924193455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa581994b0 con 0x7ffa58107870 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.209+0000 7ffa4f7fe700 1 -- 172.21.15.104:0/924193455 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa50014440 con 0x7ffa58107870 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.209+0000 7ffa4f7fe700 1 -- 172.21.15.104:0/924193455 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa500075d0 con 0x7ffa58107870 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.209+0000 7ffa60272700 1 -- 172.21.15.104:0/924193455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa581a5720 con 0x7ffa58107870 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.210+0000 7ffa60272700 1 -- 172.21.15.104:0/924193455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa58069180 con 0x7ffa58107870 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.212+0000 7ffa4f7fe700 1 -- 172.21.15.104:0/924193455 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ffa5001c070 con 0x7ffa58107870 2024-06-09T22:25:10.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.213+0000 7ffa4f7fe700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa440a2d90 0x7ffa440a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:10.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.213+0000 7ffa5e00e700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa440a2d90 0x7ffa440a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:10.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.213+0000 7ffa4f7fe700 1 -- 172.21.15.104:0/924193455 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ffa500cefe0 con 0x7ffa58107870 2024-06-09T22:25:10.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.213+0000 7ffa5e00e700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa440a2d90 0x7ffa440a5250 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffa54005950 tx=0x7ffa540058e0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:10.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.216+0000 7ffa4f7fe700 1 -- 172.21.15.104:0/924193455 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffa5009b9a0 con 0x7ffa58107870 2024-06-09T22:25:10.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.458+0000 7ffa60272700 1 -- 172.21.15.104:0/924193455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa58199f40 con 0x7ffa58107870 2024-06-09T22:25:10.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.458+0000 7ffa4f7fe700 1 -- 172.21.15.104:0/924193455 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11956 (secure 0 0 0) 0x7ffa5009b2a0 con 0x7ffa58107870 2024-06-09T22:25:10.459 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:25:10.459 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:22:59.692018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24419,24403,24442,24460,24448]},"id":1}]} 2024-06-09T22:25:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.461+0000 7ffa60272700 1 -- 172.21.15.104:0/924193455 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa440a2d90 msgr2=0x7ffa440a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.461+0000 7ffa60272700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa440a2d90 0x7ffa440a5250 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffa54005950 tx=0x7ffa540058e0 comp rx=0 tx=0).stop 2024-06-09T22:25:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.461+0000 7ffa60272700 1 -- 172.21.15.104:0/924193455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa58107870 msgr2=0x7ffa58198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.461+0000 7ffa60272700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa58107870 0x7ffa58198a60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ffa5000b970 tx=0x7ffa5000bd30 comp rx=0 tx=0).stop 2024-06-09T22:25:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.461+0000 7ffa60272700 1 -- 172.21.15.104:0/924193455 shutdown_connections 2024-06-09T22:25:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.461+0000 7ffa60272700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa440a2d90 0x7ffa440a5250 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.461+0000 7ffa60272700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa58107870 0x7ffa58198a60 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.461+0000 7ffa60272700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa581027a0 0x7ffa5819e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:10.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.461+0000 7ffa60272700 1 --2- 172.21.15.104:0/924193455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa58103150 0x7ffa5819ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.461+0000 7ffa60272700 1 -- 172.21.15.104:0/924193455 >> 172.21.15.104:0/924193455 conn(0x7ffa58079570 msgr2=0x7ffa58100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.461+0000 7ffa60272700 1 -- 172.21.15.104:0/924193455 shutdown_connections 2024-06-09T22:25:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:10.461+0000 7ffa60272700 1 -- 172.21.15.104:0/924193455 wait complete. 2024-06-09T22:25:10.463 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 30 2024-06-09T22:25:10.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 82426 3.59 MB/sec execute 16 sec latency 2233.599 ms 2024-06-09T22:25:11.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:10 smithi194 ceph-mon[36856]: pgmap v129: 129 pgs: 129 active+clean; 64 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 162 op/s 2024-06-09T22:25:11.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:10 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/924193455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:25:11.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:10 smithi153 ceph-mon[36599]: pgmap v129: 129 pgs: 129 active+clean; 64 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 162 op/s 2024-06-09T22:25:11.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:10 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/924193455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:25:11.244 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:10 smithi104 ceph-mon[32389]: pgmap v129: 129 pgs: 129 active+clean; 64 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 162 op/s 2024-06-09T22:25:11.245 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:10 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/924193455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:25:11.275 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-09T22:25:11.526 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:11.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 84127 3.81 MB/sec execute 17 sec latency 984.369 ms 2024-06-09T22:25:12.725 INFO:tasks.workunit.client.0.smithi104.stdout: 1 85887 4.24 MB/sec execute 18 sec latency 17.390 ms 2024-06-09T22:25:12.963 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:12 smithi104 ceph-mon[32389]: pgmap v130: 129 pgs: 129 active+clean; 64 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 884 KiB/s wr, 119 op/s 2024-06-09T22:25:13.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:12 smithi194 ceph-mon[36856]: pgmap v130: 129 pgs: 129 active+clean; 64 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 884 KiB/s wr, 119 op/s 2024-06-09T22:25:13.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:12 smithi153 ceph-mon[36599]: pgmap v130: 129 pgs: 129 active+clean; 64 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 884 KiB/s wr, 119 op/s 2024-06-09T22:25:13.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.398+0000 7f8464b1e700 1 -- 172.21.15.104:0/560818688 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8460113330 msgr2=0x7f8460115720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:13.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.398+0000 7f8464b1e700 1 --2- 172.21.15.104:0/560818688 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8460113330 0x7f8460115720 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f845800b0a0 tx=0x7f845800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.398+0000 7f8464b1e700 1 -- 172.21.15.104:0/560818688 shutdown_connections 2024-06-09T22:25:13.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.398+0000 7f8464b1e700 1 --2- 172.21.15.104:0/560818688 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8460113330 0x7f8460115720 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.398+0000 7f8464b1e700 1 --2- 172.21.15.104:0/560818688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8460100610 0x7f84601009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.398+0000 7f8464b1e700 1 --2- 172.21.15.104:0/560818688 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8460100f30 0x7f8460112df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.398+0000 7f8464b1e700 1 -- 172.21.15.104:0/560818688 >> 172.21.15.104:0/560818688 conn(0x7f8460077590 msgr2=0x7f84600fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:13.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.398+0000 7f8464b1e700 1 -- 172.21.15.104:0/560818688 shutdown_connections 2024-06-09T22:25:13.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.398+0000 7f8464b1e700 1 -- 172.21.15.104:0/560818688 wait complete. 2024-06-09T22:25:13.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.399+0000 7f8464b1e700 1 Processor -- start 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.399+0000 7f8464b1e700 1 -- start start 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.400+0000 7f8464b1e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8460100610 0x7f846019f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.400+0000 7f8464b1e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8460100f30 0x7f846019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.400+0000 7f8464b1e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8460113330 0x7f84601a3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.400+0000 7f845e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8460100610 0x7f846019f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.400+0000 7f845e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8460100610 0x7f846019f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58200/0 (socket says 172.21.15.104:58200) 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.400+0000 7f845e59c700 1 -- 172.21.15.104:0/1468085317 learned_addr learned my addr 172.21.15.104:0/1468085317 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.400+0000 7f8464b1e700 1 -- 172.21.15.104:0/1468085317 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8460118dd0 con 0x7f8460100610 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.400+0000 7f845dd9b700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8460100f30 0x7f846019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.400+0000 7f8464b1e700 1 -- 172.21.15.104:0/1468085317 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8460118c50 con 0x7f8460100f30 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.400+0000 7f8464b1e700 1 -- 172.21.15.104:0/1468085317 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8460118f50 con 0x7f8460113330 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.401+0000 7f845ed9d700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8460113330 0x7f84601a3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.401+0000 7f845e59c700 1 -- 172.21.15.104:0/1468085317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8460113330 msgr2=0x7f84601a3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:13.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.401+0000 7f845e59c700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8460113330 0x7f84601a3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.401+0000 7f845e59c700 1 -- 172.21.15.104:0/1468085317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8460100f30 msgr2=0x7f846019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:13.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.401+0000 7f845e59c700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8460100f30 0x7f846019f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.401+0000 7f845e59c700 1 -- 172.21.15.104:0/1468085317 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84601a4230 con 0x7f8460100610 2024-06-09T22:25:13.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.402+0000 7f845e59c700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8460100610 0x7f846019f220 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f845000e9a0 tx=0x7f845000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:13.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.402+0000 7f845dd9b700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8460100f30 0x7f846019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:13.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.402+0000 7f845ed9d700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8460113330 0x7f84601a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:13.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.402+0000 7f844f7fe700 1 -- 172.21.15.104:0/1468085317 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845000cbd0 con 0x7f8460100610 2024-06-09T22:25:13.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.402+0000 7f844f7fe700 1 -- 172.21.15.104:0/1468085317 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f845000cd70 con 0x7f8460100610 2024-06-09T22:25:13.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.402+0000 7f8464b1e700 1 -- 172.21.15.104:0/1468085317 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84601a4520 con 0x7f8460100610 2024-06-09T22:25:13.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.402+0000 7f844f7fe700 1 -- 172.21.15.104:0/1468085317 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8450018840 con 0x7f8460100610 2024-06-09T22:25:13.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.402+0000 7f8464b1e700 1 -- 172.21.15.104:0/1468085317 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84601abda0 con 0x7f8460100610 2024-06-09T22:25:13.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.406+0000 7f844f7fe700 1 -- 172.21.15.104:0/1468085317 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f84500189e0 con 0x7f8460100610 2024-06-09T22:25:13.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.406+0000 7f844f7fe700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84480a2f60 0x7f84480a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.407+0000 7f844f7fe700 1 -- 172.21.15.104:0/1468085317 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f84500cf9e0 con 0x7f8460100610 2024-06-09T22:25:13.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.407+0000 7f845dd9b700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84480a2f60 0x7f84480a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.407+0000 7f846f3a1b80 1 Processor -- start 2024-06-09T22:25:13.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.407+0000 7f846f3a1b80 1 -- start start 2024-06-09T22:25:13.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f845dd9b700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84480a2f60 0x7f84480a5420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8454005e10 tx=0x7f8454005d60 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:13.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f846f3a1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f846f3a1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f846f3a1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8cef20 0x55ad1e8cf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f846f3a1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55ad1e820810 con 0x55ad1e8d4640 2024-06-09T22:25:13.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f846f3a1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ad1e7c8720 con 0x55ad1e8d4f60 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f846f3a1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ad1e81c7d0 con 0x55ad1e8cef20 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f845e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f845e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58216/0 (socket says 172.21.15.104:58216) 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f845e59c700 1 -- 172.21.15.104:0/2564109616 learned_addr learned my addr 172.21.15.104:0/2564109616 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f845dd9b700 1 --2- 172.21.15.104:0/2564109616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f845e59c700 1 -- 172.21.15.104:0/2564109616 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8cef20 msgr2=0x55ad1e8cf340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f845e59c700 1 --2- 172.21.15.104:0/2564109616 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8cef20 0x55ad1e8cf340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f845e59c700 1 -- 172.21.15.104:0/2564109616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4f60 msgr2=0x55ad1e8ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f845e59c700 1 --2- 172.21.15.104:0/2564109616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f845e59c700 1 -- 172.21.15.104:0/2564109616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad1e7bef50 con 0x55ad1e8d4640 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f845e59c700 1 --2- 172.21.15.104:0/2564109616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d4a20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8450097fb0 tx=0x7f8450097f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b744c6572b870d7 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.408+0000 7f844cff9700 1 -- 172.21.15.104:0/2564109616 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845009ff00 con 0x55ad1e8d4640 2024-06-09T22:25:13.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.409+0000 7f844cff9700 1 -- 172.21.15.104:0/2564109616 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84500a47c0 con 0x55ad1e8d4640 2024-06-09T22:25:13.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.409+0000 7f844cff9700 1 -- 172.21.15.104:0/2564109616 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84500a4590 con 0x55ad1e8d4640 2024-06-09T22:25:13.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.409+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2564109616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 msgr2=0x55ad1e8d4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:13.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.409+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/2564109616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d4a20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8450097fb0 tx=0x7f8450097f20 comp rx=0 tx=0).stop 2024-06-09T22:25:13.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.410+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2564109616 shutdown_connections 2024-06-09T22:25:13.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.410+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/2564109616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d4a20 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.410+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/2564109616 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8cef20 0x55ad1e8cf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.410+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/2564109616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.410+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2564109616 >> 172.21.15.104:0/2564109616 conn(0x55ad1e7f9cc0 msgr2=0x55ad1e8c9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:13.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.410+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2564109616 shutdown_connections 2024-06-09T22:25:13.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.410+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2564109616 wait complete. 2024-06-09T22:25:13.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.410+0000 7f846f3a1b80 1 -- start start 2024-06-09T22:25:13.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.410+0000 7f846f3a1b80 1 Processor -- start 2024-06-09T22:25:13.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f846f3a1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 0x55ad1e8dad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f846f3a1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f846f3a1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8d8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f846f3a1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55ad1e8dba70 con 0x55ad1e8cef20 2024-06-09T22:25:13.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f846f3a1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ad1e7c0bf0 con 0x55ad1e8d4f60 2024-06-09T22:25:13.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f846f3a1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ad1e8dbbf0 con 0x55ad1e8d4640 2024-06-09T22:25:13.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f846f3a1b80 10 client.0 _ll_register_callbacks cb 0x55ad1e7ba920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:13.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f845e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 0x55ad1e8dad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f845ed9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8d8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f845e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 0x55ad1e8dad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58228/0 (socket says 172.21.15.104:58228) 2024-06-09T22:25:13.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f845ed9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8d8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:38926/0 (socket says 172.21.15.104:38926) 2024-06-09T22:25:13.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f845e59c700 1 -- 172.21.15.104:0/588940872 learned_addr learned my addr 172.21.15.104:0/588940872 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:13.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f845dd9b700 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f845ed9d700 1 -- 172.21.15.104:0/588940872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4640 msgr2=0x55ad1e8d8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f845ed9d700 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f845ed9d700 1 -- 172.21.15.104:0/588940872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 msgr2=0x55ad1e8dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f845ed9d700 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 0x55ad1e8dad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f845ed9d700 1 -- 172.21.15.104:0/588940872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad1e76f8d0 con 0x55ad1e8d4f60 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.411+0000 7f845e59c700 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 0x55ad1e8dad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.412+0000 7f845ed9d700 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8d8570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8458009fd0 tx=0x7f845800bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95a0d6258e157c5d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.412+0000 7f8445ffb700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.412+0000 7f8445ffb700 1 -- 172.21.15.104:0/588940872 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8458022070 con 0x55ad1e8d4f60 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.412+0000 7f846f3a1b80 1 -- 172.21.15.104:0/588940872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84500a47c0 con 0x55ad1e8d4f60 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.412+0000 7f8445ffb700 1 -- 172.21.15.104:0/588940872 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f84580074e0 con 0x55ad1e8d4f60 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.412+0000 7f8445ffb700 1 -- 172.21.15.104:0/588940872 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f845802b050 con 0x55ad1e8d4f60 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.412+0000 7f846f3a1b80 10 client.25991 fetch_fsmap learned FSMap version 30 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.412+0000 7f846f3a1b80 1 -- 172.21.15.104:0/588940872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ad1e8e63f0 con 0x55ad1e8d4f60 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.412+0000 7f8445ffb700 1 -- 172.21.15.104:0/588940872 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84500a47c0 con 0x55ad1e8d4f60 2024-06-09T22:25:13.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.412+0000 7f8445ffb700 1 -- 172.21.15.104:0/588940872 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8458011070 con 0x55ad1e8d4f60 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.413+0000 7f846f3a1b80 10 client.25991 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.413+0000 7f846f3a1b80 10 client.25991 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.413+0000 7f846f3a1b80 20 client.25991 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.413+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55ad1e8e8980 0x55ad1e8eada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.413+0000 7f846f3a1b80 4 client.25991 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.413+0000 7f846f3a1b80 1 -- 172.21.15.104:0/588940872 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad1e82dd10 con 0x55ad1e8e8980 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.413+0000 7f845e59c700 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55ad1e8e8980 0x55ad1e8eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.413+0000 7f8445ffb700 10 client.25991 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.413+0000 7f845e59c700 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55ad1e8e8980 0x55ad1e8eada0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dd88ceb18b54492f server_cookie=6a0bd19ab3afe52 in_seq=0 out_seq=0 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f8445ffb700 1 -- 172.21.15.104:0/588940872 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55ad1e82dd10 con 0x55ad1e8e8980 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f8445ffb700 10 client.25991 handle_command_reply: tid=0 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f846f3a1b80 1 client.25991 shutdown 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f846f3a1b80 1 -- 172.21.15.104:0/588940872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4f60 msgr2=0x55ad1e8d8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8d8570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8458009fd0 tx=0x7f845800bce0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f846f3a1b80 1 -- 172.21.15.104:0/588940872 shutdown_connections 2024-06-09T22:25:13.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 0x55ad1e8dad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55ad1e8e8980 0x55ad1e8eada0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f8445ffb700 0 client.25991 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:25:13.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8d8570 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/588940872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f846f3a1b80 1 -- 172.21.15.104:0/588940872 >> 172.21.15.104:0/588940872 conn(0x55ad1e7f9cc0 msgr2=0x55ad1e8c95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:13.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f846f3a1b80 1 -- 172.21.15.104:0/588940872 shutdown_connections 2024-06-09T22:25:13.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f846f3a1b80 1 -- 172.21.15.104:0/588940872 wait complete. 2024-06-09T22:25:13.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.414+0000 7f846f3a1b80 20 client.25991 trim_cache size 0 max 16384 2024-06-09T22:25:13.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.429+0000 7f846f3a1b80 1 Processor -- start 2024-06-09T22:25:13.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.429+0000 7f846f3a1b80 1 -- start start 2024-06-09T22:25:13.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.429+0000 7f846f3a1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8cef20 0x55ad1e8ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.429+0000 7f846f3a1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.429+0000 7f846f3a1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8f0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.429+0000 7f846f3a1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f84601b4b70 con 0x55ad1e8d4640 2024-06-09T22:25:13.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.429+0000 7f846f3a1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84601b6710 con 0x55ad1e8cef20 2024-06-09T22:25:13.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.429+0000 7f846f3a1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f84601b5240 con 0x55ad1e8d4f60 2024-06-09T22:25:13.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f845dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f845dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58236/0 (socket says 172.21.15.104:58236) 2024-06-09T22:25:13.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f845dd9b700 1 -- 172.21.15.104:0/1113501699 learned_addr learned my addr 172.21.15.104:0/1113501699 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:13.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f845ed9d700 1 --2- 172.21.15.104:0/1113501699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8f0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f845e59c700 1 --2- 172.21.15.104:0/1113501699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8cef20 0x55ad1e8ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f845ed9d700 1 -- 172.21.15.104:0/1113501699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8cef20 msgr2=0x55ad1e8ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:13.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f845ed9d700 1 --2- 172.21.15.104:0/1113501699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8cef20 0x55ad1e8ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f845ed9d700 1 -- 172.21.15.104:0/1113501699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 msgr2=0x55ad1e8d5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:13.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f845ed9d700 1 --2- 172.21.15.104:0/1113501699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f845ed9d700 1 -- 172.21.15.104:0/1113501699 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad1e8eb420 con 0x55ad1e8d4f60 2024-06-09T22:25:13.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f845ed9d700 1 --2- 172.21.15.104:0/1113501699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8f0670 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8458009a20 tx=0x7f845800be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b5cbafadc7ee876 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:13.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f8447fff700 1 -- 172.21.15.104:0/1113501699 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8458034070 con 0x55ad1e8d4f60 2024-06-09T22:25:13.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f8447fff700 1 -- 172.21.15.104:0/1113501699 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8458004ce0 con 0x55ad1e8d4f60 2024-06-09T22:25:13.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.430+0000 7f8447fff700 1 -- 172.21.15.104:0/1113501699 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8458022070 con 0x55ad1e8d4f60 2024-06-09T22:25:13.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.431+0000 7f846f3a1b80 1 -- 172.21.15.104:0/1113501699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4f60 msgr2=0x55ad1e8f0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:13.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.431+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/1113501699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8f0670 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8458009a20 tx=0x7f845800be00 comp rx=0 tx=0).stop 2024-06-09T22:25:13.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.431+0000 7f846f3a1b80 1 -- 172.21.15.104:0/1113501699 shutdown_connections 2024-06-09T22:25:13.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.431+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/1113501699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8f0670 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.431+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/1113501699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8cef20 0x55ad1e8ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.431+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/1113501699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8d4640 0x55ad1e8d5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.431+0000 7f846f3a1b80 1 -- 172.21.15.104:0/1113501699 >> 172.21.15.104:0/1113501699 conn(0x55ad1e7f9cc0 msgr2=0x55ad1e8d5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:13.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.431+0000 7f846f3a1b80 1 -- 172.21.15.104:0/1113501699 shutdown_connections 2024-06-09T22:25:13.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.431+0000 7f846f3a1b80 1 -- 172.21.15.104:0/1113501699 wait complete. 2024-06-09T22:25:13.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f846f3a1b80 1 -- start start 2024-06-09T22:25:13.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f846f3a1b80 1 Processor -- start 2024-06-09T22:25:13.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f846f3a1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 0x55ad1e8e1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f846f3a1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4640 0x55ad1e8e1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f846f3a1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8efc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f846f3a1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f84601b5fe0 con 0x55ad1e8cef20 2024-06-09T22:25:13.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f846f3a1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84601b5910 con 0x55ad1e8d4640 2024-06-09T22:25:13.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f846f3a1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f84601b2af0 con 0x55ad1e8d4f60 2024-06-09T22:25:13.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f845e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 0x55ad1e8e1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f846f3a1b80 10 client.0 _ll_register_callbacks cb 0x55ad1e7ba920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:13.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f845e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 0x55ad1e8e1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58248/0 (socket says 172.21.15.104:58248) 2024-06-09T22:25:13.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f845e59c700 1 -- 172.21.15.104:0/2753088189 learned_addr learned my addr 172.21.15.104:0/2753088189 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f845dd9b700 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4640 0x55ad1e8e1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.432+0000 7f845ed9d700 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8efc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f845e59c700 1 -- 172.21.15.104:0/2753088189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4f60 msgr2=0x55ad1e8efc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f845e59c700 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8efc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f845e59c700 1 -- 172.21.15.104:0/2753088189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4640 msgr2=0x55ad1e8e1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f845e59c700 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4640 0x55ad1e8e1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f845e59c700 1 -- 172.21.15.104:0/2753088189 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad1e8f0480 con 0x55ad1e8cef20 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f845e59c700 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 0x55ad1e8e1110 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f845009c400 tx=0x7f84500977c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f3814aa18e2aba server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f8445ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f8445ffb700 1 -- 172.21.15.104:0/2753088189 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84500a47c0 con 0x55ad1e8cef20 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f8445ffb700 1 -- 172.21.15.104:0/2753088189 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f84500a4330 con 0x55ad1e8cef20 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2753088189 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8458004ce0 con 0x55ad1e8cef20 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f8445ffb700 1 -- 172.21.15.104:0/2753088189 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f84500a4530 con 0x55ad1e8cef20 2024-06-09T22:25:13.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f846f3a1b80 10 client.16635 fetch_fsmap learned FSMap version 30 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2753088189 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ad1e953100 con 0x55ad1e8cef20 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.433+0000 7f8445ffb700 1 -- 172.21.15.104:0/2753088189 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8458004ce0 con 0x55ad1e8cef20 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.434+0000 7f8445ffb700 1 -- 172.21.15.104:0/2753088189 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f845009afa0 con 0x55ad1e8cef20 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.434+0000 7f846f3a1b80 10 client.16635 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.434+0000 7f846f3a1b80 10 client.16635 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.434+0000 7f846f3a1b80 20 client.16635 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.434+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55ad1e953440 0x55ad1e955830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.434+0000 7f846f3a1b80 4 client.16635 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.434+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2753088189 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad1e828290 con 0x55ad1e953440 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.434+0000 7f845dd9b700 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55ad1e953440 0x55ad1e955830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.434+0000 7f845dd9b700 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55ad1e953440 0x55ad1e955830 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3804c0549fcb92af server_cookie=8fe5d39ab3b9994e in_seq=0 out_seq=0 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.434+0000 7f8445ffb700 10 client.16635 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f8445ffb700 1 -- 172.21.15.104:0/2753088189 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad1e828290 con 0x55ad1e953440 2024-06-09T22:25:13.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f8445ffb700 10 client.16635 handle_command_reply: tid=0 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f846f3a1b80 1 client.16635 shutdown 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2753088189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 msgr2=0x55ad1e8e1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 0x55ad1e8e1110 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f845009c400 tx=0x7f84500977c0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2753088189 shutdown_connections 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad1e8cef20 0x55ad1e8e1110 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55ad1e953440 0x55ad1e955830 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f8445ffb700 0 client.16635 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad1e8d4f60 0x55ad1e8efc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f846f3a1b80 1 --2- 172.21.15.104:0/2753088189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad1e8d4640 0x55ad1e8e1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2753088189 >> 172.21.15.104:0/2753088189 conn(0x55ad1e7f9cc0 msgr2=0x55ad1e8d5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2753088189 shutdown_connections 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f846f3a1b80 1 -- 172.21.15.104:0/2753088189 wait complete. 2024-06-09T22:25:13.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f846f3a1b80 20 client.16635 trim_cache size 0 max 16384 2024-06-09T22:25:13.442 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:25:13.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f8464b1e700 1 -- 172.21.15.104:0/1468085317 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84480a2f60 msgr2=0x7f84480a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:13.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f8464b1e700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84480a2f60 0x7f84480a5420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8454005e10 tx=0x7f8454005d60 comp rx=0 tx=0).stop 2024-06-09T22:25:13.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f8464b1e700 1 -- 172.21.15.104:0/1468085317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8460100610 msgr2=0x7f846019f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:13.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.435+0000 7f8464b1e700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8460100610 0x7f846019f220 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f845000e9a0 tx=0x7f845000ed60 comp rx=0 tx=0).stop 2024-06-09T22:25:13.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.436+0000 7f8464b1e700 1 -- 172.21.15.104:0/1468085317 shutdown_connections 2024-06-09T22:25:13.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.436+0000 7f8464b1e700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84480a2f60 0x7f84480a5420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.436+0000 7f8464b1e700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8460100610 0x7f846019f220 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.436+0000 7f8464b1e700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8460113330 0x7f84601a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.436+0000 7f8464b1e700 1 --2- 172.21.15.104:0/1468085317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8460100f30 0x7f846019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:13.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.436+0000 7f8464b1e700 1 -- 172.21.15.104:0/1468085317 >> 172.21.15.104:0/1468085317 conn(0x7f8460077590 msgr2=0x7f8460111b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:13.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.436+0000 7f8464b1e700 1 -- 172.21.15.104:0/1468085317 shutdown_connections 2024-06-09T22:25:13.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:13.436+0000 7f8464b1e700 1 -- 172.21.15.104:0/1468085317 wait complete. 2024-06-09T22:25:13.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 85899 4.02 MB/sec execute 19 sec latency 986.566 ms 2024-06-09T22:25:14.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-09T22:25:14.553 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:14.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 85899 3.82 MB/sec execute 20 sec latency 1986.651 ms 2024-06-09T22:25:15.004 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:14 smithi104 ceph-mon[32389]: pgmap v131: 129 pgs: 129 active+clean; 73 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 167 op/s 2024-06-09T22:25:15.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:14 smithi153 ceph-mon[36599]: pgmap v131: 129 pgs: 129 active+clean; 73 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 167 op/s 2024-06-09T22:25:15.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:14 smithi194 ceph-mon[36856]: pgmap v131: 129 pgs: 129 active+clean; 73 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 167 op/s 2024-06-09T22:25:15.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 85899 3.64 MB/sec execute 21 sec latency 2986.747 ms 2024-06-09T22:25:16.053 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:15 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:25:16.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:15 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:25:16.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:15 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:25:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.442+0000 7f87c02a6700 1 -- 172.21.15.104:0/4143780802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87b8102530 msgr2=0x7f87b81029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.442+0000 7f87c02a6700 1 --2- 172.21.15.104:0/4143780802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87b8102530 0x7f87b81029b0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f87a80099c0 tx=0x7f87a8009cd0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.442+0000 7f87c02a6700 1 -- 172.21.15.104:0/4143780802 shutdown_connections 2024-06-09T22:25:16.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.442+0000 7f87c02a6700 1 --2- 172.21.15.104:0/4143780802 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f87b8102ef0 0x7f87b810f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.442+0000 7f87c02a6700 1 --2- 172.21.15.104:0/4143780802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87b81085d0 0x7f87b81089b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.442+0000 7f87c02a6700 1 --2- 172.21.15.104:0/4143780802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87b8102530 0x7f87b81029b0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.442+0000 7f87c02a6700 1 -- 172.21.15.104:0/4143780802 >> 172.21.15.104:0/4143780802 conn(0x7f87b8079640 msgr2=0x7f87b8079a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:16.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.442+0000 7f87c02a6700 1 -- 172.21.15.104:0/4143780802 shutdown_connections 2024-06-09T22:25:16.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.442+0000 7f87c02a6700 1 -- 172.21.15.104:0/4143780802 wait complete. 2024-06-09T22:25:16.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.442+0000 7f87c02a6700 1 Processor -- start 2024-06-09T22:25:16.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.442+0000 7f87c02a6700 1 -- start start 2024-06-09T22:25:16.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87c02a6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87b8102530 0x7f87b8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87c02a6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f87b8102ef0 0x7f87b8074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87c02a6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87b81085d0 0x7f87b8075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87c02a6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f87b8112810 con 0x7f87b81085d0 2024-06-09T22:25:16.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87c02a6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87b8112690 con 0x7f87b8102530 2024-06-09T22:25:16.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87c02a6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f87b8112990 con 0x7f87b8102ef0 2024-06-09T22:25:16.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87be042700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87b8102530 0x7f87b8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87be843700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87b81085d0 0x7f87b8075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87be042700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87b8102530 0x7f87b8074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:40080/0 (socket says 172.21.15.104:40080) 2024-06-09T22:25:16.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87be042700 1 -- 172.21.15.104:0/4248542776 learned_addr learned my addr 172.21.15.104:0/4248542776 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:16.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87bd841700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f87b8102ef0 0x7f87b8074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87be843700 1 -- 172.21.15.104:0/4248542776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f87b8102ef0 msgr2=0x7f87b8074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:16.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87be843700 1 --2- 172.21.15.104:0/4248542776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f87b8102ef0 0x7f87b8074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87be843700 1 -- 172.21.15.104:0/4248542776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87b8102530 msgr2=0x7f87b8074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:16.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87be843700 1 --2- 172.21.15.104:0/4248542776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87b8102530 0x7f87b8074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.443+0000 7f87be843700 1 -- 172.21.15.104:0/4248542776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b81a7150 con 0x7f87b81085d0 2024-06-09T22:25:16.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.444+0000 7f87be843700 1 --2- 172.21.15.104:0/4248542776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87b81085d0 0x7f87b8075210 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f87b000efc0 tx=0x7f87b000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:16.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.444+0000 7f87af7fe700 1 -- 172.21.15.104:0/4248542776 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b00092e0 con 0x7f87b81085d0 2024-06-09T22:25:16.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.444+0000 7f87af7fe700 1 -- 172.21.15.104:0/4248542776 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87b0004bb0 con 0x7f87b81085d0 2024-06-09T22:25:16.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.444+0000 7f87c02a6700 1 -- 172.21.15.104:0/4248542776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87b81a73e0 con 0x7f87b81085d0 2024-06-09T22:25:16.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.444+0000 7f87af7fe700 1 -- 172.21.15.104:0/4248542776 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0004280 con 0x7f87b81085d0 2024-06-09T22:25:16.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.444+0000 7f87c02a6700 1 -- 172.21.15.104:0/4248542776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87b81a78f0 con 0x7f87b81085d0 2024-06-09T22:25:16.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.446+0000 7f87af7fe700 1 -- 172.21.15.104:0/4248542776 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f87b001b030 con 0x7f87b81085d0 2024-06-09T22:25:16.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.447+0000 7f87af7fe700 1 --2- 172.21.15.104:0/4248542776 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f87a40a2e10 0x7f87a40a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.447+0000 7f87be042700 1 --2- 172.21.15.104:0/4248542776 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f87a40a2e10 0x7f87a40a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.447+0000 7f87af7fe700 1 -- 172.21.15.104:0/4248542776 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f87b00cf260 con 0x7f87b81085d0 2024-06-09T22:25:16.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.447+0000 7f87be042700 1 --2- 172.21.15.104:0/4248542776 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f87a40a2e10 0x7f87a40a52d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f87b4007c10 tx=0x7f87b4007480 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:16.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87cab29b80 1 Processor -- start 2024-06-09T22:25:16.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87cab29b80 1 -- start start 2024-06-09T22:25:16.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87cab29b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f374640 0x55ca2f374a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87cab29b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374f60 0x55ca2f36e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87cab29b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f36ef20 0x55ca2f36f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87cab29b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55ca2f2c0810 con 0x55ca2f374640 2024-06-09T22:25:16.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87cab29b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ca2f268720 con 0x55ca2f36ef20 2024-06-09T22:25:16.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87cab29b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ca2f2bc7d0 con 0x55ca2f374f60 2024-06-09T22:25:16.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87be042700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f374640 0x55ca2f374a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87bd841700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374f60 0x55ca2f36e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87bd841700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374f60 0x55ca2f36e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:52208/0 (socket says 172.21.15.104:52208) 2024-06-09T22:25:16.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87bd841700 1 -- 172.21.15.104:0/1559912339 learned_addr learned my addr 172.21.15.104:0/1559912339 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:16.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87be843700 1 --2- 172.21.15.104:0/1559912339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f36ef20 0x55ca2f36f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87bd841700 1 -- 172.21.15.104:0/1559912339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f36ef20 msgr2=0x55ca2f36f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:16.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87bd841700 1 --2- 172.21.15.104:0/1559912339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f36ef20 0x55ca2f36f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87bd841700 1 -- 172.21.15.104:0/1559912339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f374640 msgr2=0x55ca2f374a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:16.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87bd841700 1 --2- 172.21.15.104:0/1559912339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f374640 0x55ca2f374a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.448+0000 7f87bd841700 1 -- 172.21.15.104:0/1559912339 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca2f25ef50 con 0x55ca2f374f60 2024-06-09T22:25:16.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.449+0000 7f87bd841700 1 --2- 172.21.15.104:0/1559912339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374f60 0x55ca2f36e9e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f87a80099c0 tx=0x7f87a8003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81aa512d22c7ce8e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:16.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.449+0000 7f87acd70700 1 -- 172.21.15.104:0/1559912339 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8003a60 con 0x55ca2f374f60 2024-06-09T22:25:16.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.449+0000 7f87acd70700 1 -- 172.21.15.104:0/1559912339 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87a8026070 con 0x55ca2f374f60 2024-06-09T22:25:16.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.449+0000 7f87acd70700 1 -- 172.21.15.104:0/1559912339 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a800bad0 con 0x55ca2f374f60 2024-06-09T22:25:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.450+0000 7f87cab29b80 1 -- 172.21.15.104:0/1559912339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374f60 msgr2=0x55ca2f36e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.450+0000 7f87cab29b80 1 --2- 172.21.15.104:0/1559912339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374f60 0x55ca2f36e9e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f87a80099c0 tx=0x7f87a8003680 comp rx=0 tx=0).stop 2024-06-09T22:25:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.450+0000 7f87cab29b80 1 -- 172.21.15.104:0/1559912339 shutdown_connections 2024-06-09T22:25:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.450+0000 7f87cab29b80 1 --2- 172.21.15.104:0/1559912339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f374640 0x55ca2f374a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.450+0000 7f87cab29b80 1 --2- 172.21.15.104:0/1559912339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f36ef20 0x55ca2f36f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.450+0000 7f87cab29b80 1 --2- 172.21.15.104:0/1559912339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374f60 0x55ca2f36e9e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.450+0000 7f87cab29b80 1 -- 172.21.15.104:0/1559912339 >> 172.21.15.104:0/1559912339 conn(0x55ca2f299cc0 msgr2=0x55ca2f369850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.450+0000 7f87cab29b80 1 -- 172.21.15.104:0/1559912339 shutdown_connections 2024-06-09T22:25:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.450+0000 7f87cab29b80 1 -- 172.21.15.104:0/1559912339 wait complete. 2024-06-09T22:25:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.450+0000 7f87cab29b80 1 -- start start 2024-06-09T22:25:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.450+0000 7f87cab29b80 1 Processor -- start 2024-06-09T22:25:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.450+0000 7f87cab29b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 0x55ca2f37aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87cab29b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f377fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87cab29b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f374f60 0x55ca2f3784e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87cab29b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55ca2f37b970 con 0x55ca2f36ef20 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87cab29b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ca2f260bf0 con 0x55ca2f374f60 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87cab29b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ca2f37baf0 con 0x55ca2f374640 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87cab29b80 10 client.0 _ll_register_callbacks cb 0x55ca2f25a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87be042700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 0x55ca2f37aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87be042700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 0x55ca2f37aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33574/0 (socket says 172.21.15.104:33574) 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87be042700 1 -- 172.21.15.104:0/2136514846 learned_addr learned my addr 172.21.15.104:0/2136514846 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87bd841700 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f377fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87be843700 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f374f60 0x55ca2f3784e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87be042700 1 -- 172.21.15.104:0/2136514846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 msgr2=0x55ca2f377fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87be042700 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f377fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87be042700 1 -- 172.21.15.104:0/2136514846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f374f60 msgr2=0x55ca2f3784e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87be042700 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f374f60 0x55ca2f3784e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87be042700 1 -- 172.21.15.104:0/2136514846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca2f20f8d0 con 0x55ca2f36ef20 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87be042700 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 0x55ca2f37aca0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f87b40116f0 tx=0x7f87b40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bbec9ec86a377b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87a1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87a1ffb700 1 -- 172.21.15.104:0/2136514846 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4028470 con 0x55ca2f36ef20 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87a1ffb700 1 -- 172.21.15.104:0/2136514846 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f87b4028610 con 0x55ca2f36ef20 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.451+0000 7f87cab29b80 1 -- 172.21.15.104:0/2136514846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87a8026070 con 0x55ca2f36ef20 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.452+0000 7f87a1ffb700 1 -- 172.21.15.104:0/2136514846 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f87b4028810 con 0x55ca2f36ef20 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.452+0000 7f87cab29b80 10 client.16647 fetch_fsmap learned FSMap version 30 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.452+0000 7f87cab29b80 1 -- 172.21.15.104:0/2136514846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ca2f3862b0 con 0x55ca2f36ef20 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.452+0000 7f87a1ffb700 1 -- 172.21.15.104:0/2136514846 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87a8026070 con 0x55ca2f36ef20 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.452+0000 7f87a1ffb700 1 -- 172.21.15.104:0/2136514846 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f87b401fc00 con 0x55ca2f36ef20 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.452+0000 7f87cab29b80 10 client.16647 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:25:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.452+0000 7f87cab29b80 10 client.16647 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.452+0000 7f87cab29b80 20 client.16647 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.452+0000 7f87cab29b80 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55ca2f3888f0 0x55ca2f38ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.452+0000 7f87cab29b80 4 client.16647 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.452+0000 7f87cab29b80 1 -- 172.21.15.104:0/2136514846 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca2f2cdd10 con 0x55ca2f3888f0 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.452+0000 7f87bd841700 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55ca2f3888f0 0x55ca2f38ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.453+0000 7f87bd841700 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55ca2f3888f0 0x55ca2f38ace0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=27b9ee4220ab5120 server_cookie=60e4e2aaa228706b in_seq=0 out_seq=0 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.453+0000 7f87a1ffb700 10 client.16647 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.453+0000 7f87a1ffb700 1 -- 172.21.15.104:0/2136514846 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55ca2f2cdd10 con 0x55ca2f3888f0 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.453+0000 7f87a1ffb700 10 client.16647 handle_command_reply: tid=0 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.453+0000 7f87cab29b80 1 client.16647 shutdown 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.453+0000 7f87cab29b80 1 -- 172.21.15.104:0/2136514846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 msgr2=0x55ca2f37aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.453+0000 7f87cab29b80 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 0x55ca2f37aca0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f87b40116f0 tx=0x7f87b40047c0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.454+0000 7f87cab29b80 1 -- 172.21.15.104:0/2136514846 shutdown_connections 2024-06-09T22:25:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.454+0000 7f87cab29b80 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55ca2f3888f0 0x55ca2f38ace0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.454+0000 7f87cab29b80 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 0x55ca2f37aca0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.454+0000 7f87cab29b80 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f374f60 0x55ca2f3784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.454+0000 7f87cab29b80 1 --2- 172.21.15.104:0/2136514846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f377fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.454+0000 7f87a1ffb700 0 client.16647 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:25:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.454+0000 7f87cab29b80 1 -- 172.21.15.104:0/2136514846 >> 172.21.15.104:0/2136514846 conn(0x55ca2f299cc0 msgr2=0x55ca2f36d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.454+0000 7f87cab29b80 1 -- 172.21.15.104:0/2136514846 shutdown_connections 2024-06-09T22:25:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.454+0000 7f87cab29b80 1 -- 172.21.15.104:0/2136514846 wait complete. 2024-06-09T22:25:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.454+0000 7f87cab29b80 20 client.16647 trim_cache size 0 max 16384 2024-06-09T22:25:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87cab29b80 1 Processor -- start 2024-06-09T22:25:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87cab29b80 1 -- start start 2024-06-09T22:25:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87cab29b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f36ef20 0x55ca2f36e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87cab29b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f363510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87cab29b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f374f60 0x55ca2f3905b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87cab29b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f87b81b0670 con 0x55ca2f374f60 2024-06-09T22:25:16.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87cab29b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87b81b2210 con 0x55ca2f36ef20 2024-06-09T22:25:16.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87cab29b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f87b81b0d40 con 0x55ca2f374640 2024-06-09T22:25:16.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87be843700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f374f60 0x55ca2f3905b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87bd841700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f363510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87be843700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f374f60 0x55ca2f3905b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33580/0 (socket says 172.21.15.104:33580) 2024-06-09T22:25:16.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87be843700 1 -- 172.21.15.104:0/113107687 learned_addr learned my addr 172.21.15.104:0/113107687 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:16.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87be042700 1 --2- 172.21.15.104:0/113107687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f36ef20 0x55ca2f36e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87be843700 1 -- 172.21.15.104:0/113107687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 msgr2=0x55ca2f363510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.470+0000 7f87be843700 1 --2- 172.21.15.104:0/113107687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f363510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.471+0000 7f87be843700 1 -- 172.21.15.104:0/113107687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f36ef20 msgr2=0x55ca2f36e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.471+0000 7f87be843700 1 --2- 172.21.15.104:0/113107687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f36ef20 0x55ca2f36e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.471+0000 7f87be843700 1 -- 172.21.15.104:0/113107687 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca2f38b360 con 0x55ca2f374f60 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.471+0000 7f87be843700 1 --2- 172.21.15.104:0/113107687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f374f60 0x55ca2f3905b0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f87b0009510 tx=0x7f87b009b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718f0dfd87848081 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.471+0000 7f87a3fff700 1 -- 172.21.15.104:0/113107687 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b00b3070 con 0x55ca2f374f60 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.471+0000 7f87a3fff700 1 -- 172.21.15.104:0/113107687 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87b00cf0b0 con 0x55ca2f374f60 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.471+0000 7f87a3fff700 1 -- 172.21.15.104:0/113107687 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b009b700 con 0x55ca2f374f60 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.472+0000 7f87cab29b80 1 -- 172.21.15.104:0/113107687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f374f60 msgr2=0x55ca2f3905b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.472+0000 7f87cab29b80 1 --2- 172.21.15.104:0/113107687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f374f60 0x55ca2f3905b0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f87b0009510 tx=0x7f87b009b050 comp rx=0 tx=0).stop 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.472+0000 7f87cab29b80 1 -- 172.21.15.104:0/113107687 shutdown_connections 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.472+0000 7f87cab29b80 1 --2- 172.21.15.104:0/113107687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f374f60 0x55ca2f3905b0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.472+0000 7f87cab29b80 1 --2- 172.21.15.104:0/113107687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f36ef20 0x55ca2f36e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.472+0000 7f87cab29b80 1 --2- 172.21.15.104:0/113107687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f363510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.472+0000 7f87cab29b80 1 -- 172.21.15.104:0/113107687 >> 172.21.15.104:0/113107687 conn(0x55ca2f299cc0 msgr2=0x55ca2f362e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:16.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.472+0000 7f87cab29b80 1 -- 172.21.15.104:0/113107687 shutdown_connections 2024-06-09T22:25:16.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.472+0000 7f87cab29b80 1 -- 172.21.15.104:0/113107687 wait complete. 2024-06-09T22:25:16.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.472+0000 7f87cab29b80 1 -- start start 2024-06-09T22:25:16.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87cab29b80 1 Processor -- start 2024-06-09T22:25:16.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87cab29b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 0x55ca2f381150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87cab29b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f381690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87cab29b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f374f60 0x55ca2f37c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87cab29b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f87b81b1ae0 con 0x55ca2f36ef20 2024-06-09T22:25:16.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87cab29b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87b81b1410 con 0x55ca2f374f60 2024-06-09T22:25:16.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87cab29b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f87b81ae5f0 con 0x55ca2f374640 2024-06-09T22:25:16.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87be042700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 0x55ca2f381150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87cab29b80 10 client.0 _ll_register_callbacks cb 0x55ca2f25a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:16.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87bd841700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f381690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87be042700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 0x55ca2f381150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33588/0 (socket says 172.21.15.104:33588) 2024-06-09T22:25:16.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87be042700 1 -- 172.21.15.104:0/4175331284 learned_addr learned my addr 172.21.15.104:0/4175331284 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:16.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87be843700 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f374f60 0x55ca2f37c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87be042700 1 -- 172.21.15.104:0/4175331284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 msgr2=0x55ca2f381690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:16.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87be042700 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f381690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87be042700 1 -- 172.21.15.104:0/4175331284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f374f60 msgr2=0x55ca2f37c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:16.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87be042700 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f374f60 0x55ca2f37c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87be042700 1 -- 172.21.15.104:0/4175331284 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca2f37cec0 con 0x55ca2f36ef20 2024-06-09T22:25:16.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87bd841700 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f381690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:16.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87be843700 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f374f60 0x55ca2f37c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:16.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.473+0000 7f87be042700 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 0x55ca2f381150 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f87b401ff70 tx=0x7f87b4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5654be3b51d8ceda server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:16.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87a1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:16.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87a1ffb700 1 -- 172.21.15.104:0/4175331284 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b401ec40 con 0x55ca2f36ef20 2024-06-09T22:25:16.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87a1ffb700 1 -- 172.21.15.104:0/4175331284 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f87b401ede0 con 0x55ca2f36ef20 2024-06-09T22:25:16.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87cab29b80 1 -- 172.21.15.104:0/4175331284 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87b00cf0b0 con 0x55ca2f36ef20 2024-06-09T22:25:16.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87a1ffb700 1 -- 172.21.15.104:0/4175331284 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f87b4047050 con 0x55ca2f36ef20 2024-06-09T22:25:16.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87cab29b80 10 client.16653 fetch_fsmap learned FSMap version 30 2024-06-09T22:25:16.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87cab29b80 1 -- 172.21.15.104:0/4175331284 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ca2f3f31e0 con 0x55ca2f36ef20 2024-06-09T22:25:16.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87a1ffb700 1 -- 172.21.15.104:0/4175331284 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87b00cf0b0 con 0x55ca2f36ef20 2024-06-09T22:25:16.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87a1ffb700 1 -- 172.21.15.104:0/4175331284 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f87b401fb30 con 0x55ca2f36ef20 2024-06-09T22:25:16.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87cab29b80 10 client.16653 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:25:16.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87cab29b80 10 client.16653 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:25:16.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87cab29b80 20 client.16653 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:16.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87cab29b80 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55ca2f3f38f0 0x55ca2f3f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:16.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87cab29b80 4 client.16653 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:25:16.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.474+0000 7f87cab29b80 1 -- 172.21.15.104:0/4175331284 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ca2f2c8290 con 0x55ca2f3f38f0 2024-06-09T22:25:16.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87bd841700 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55ca2f3f38f0 0x55ca2f3f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:16.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87a1ffb700 10 client.16653 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:25:16.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87bd841700 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55ca2f3f38f0 0x55ca2f3f5d10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cefd0e3cd372b971 server_cookie=dd3214b41115cd0f in_seq=0 out_seq=0 2024-06-09T22:25:16.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87a1ffb700 1 -- 172.21.15.104:0/4175331284 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ca2f2c8290 con 0x55ca2f3f38f0 2024-06-09T22:25:16.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87a1ffb700 10 client.16653 handle_command_reply: tid=0 2024-06-09T22:25:16.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87cab29b80 1 client.16653 shutdown 2024-06-09T22:25:16.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87cab29b80 1 -- 172.21.15.104:0/4175331284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 msgr2=0x55ca2f381150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:16.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87cab29b80 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 0x55ca2f381150 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f87b401ff70 tx=0x7f87b4011ab0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87cab29b80 1 -- 172.21.15.104:0/4175331284 shutdown_connections 2024-06-09T22:25:16.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87cab29b80 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55ca2f3f38f0 0x55ca2f3f5d10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87cab29b80 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ca2f36ef20 0x55ca2f381150 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87cab29b80 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ca2f374f60 0x55ca2f37c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87cab29b80 1 --2- 172.21.15.104:0/4175331284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca2f374640 0x55ca2f381690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87cab29b80 1 -- 172.21.15.104:0/4175331284 >> 172.21.15.104:0/4175331284 conn(0x55ca2f299cc0 msgr2=0x55ca2f373110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:16.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87a1ffb700 0 client.16653 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:25:16.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87cab29b80 1 -- 172.21.15.104:0/4175331284 shutdown_connections 2024-06-09T22:25:16.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87cab29b80 1 -- 172.21.15.104:0/4175331284 wait complete. 2024-06-09T22:25:16.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.475+0000 7f87cab29b80 20 client.16653 trim_cache size 0 max 16384 2024-06-09T22:25:16.482 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:25:16.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.476+0000 7f87c02a6700 1 -- 172.21.15.104:0/4248542776 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f87a40a2e10 msgr2=0x7f87a40a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:16.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.476+0000 7f87c02a6700 1 --2- 172.21.15.104:0/4248542776 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f87a40a2e10 0x7f87a40a52d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f87b4007c10 tx=0x7f87b4007480 comp rx=0 tx=0).stop 2024-06-09T22:25:16.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.476+0000 7f87c02a6700 1 -- 172.21.15.104:0/4248542776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87b81085d0 msgr2=0x7f87b8075210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:16.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.476+0000 7f87c02a6700 1 --2- 172.21.15.104:0/4248542776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87b81085d0 0x7f87b8075210 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f87b000efc0 tx=0x7f87b000c3a0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.476+0000 7f87c02a6700 1 -- 172.21.15.104:0/4248542776 shutdown_connections 2024-06-09T22:25:16.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.476+0000 7f87c02a6700 1 --2- 172.21.15.104:0/4248542776 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f87a40a2e10 0x7f87a40a52d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.476+0000 7f87c02a6700 1 --2- 172.21.15.104:0/4248542776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87b81085d0 0x7f87b8075210 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.476+0000 7f87c02a6700 1 --2- 172.21.15.104:0/4248542776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87b8102530 0x7f87b8074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.476+0000 7f87c02a6700 1 --2- 172.21.15.104:0/4248542776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f87b8102ef0 0x7f87b8074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:16.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.476+0000 7f87c02a6700 1 -- 172.21.15.104:0/4248542776 >> 172.21.15.104:0/4248542776 conn(0x7f87b8079640 msgr2=0x7f87b8101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:16.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.477+0000 7f87c02a6700 1 -- 172.21.15.104:0/4248542776 shutdown_connections 2024-06-09T22:25:16.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:16.477+0000 7f87c02a6700 1 -- 172.21.15.104:0/4248542776 wait complete. 2024-06-09T22:25:16.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 87545 3.78 MB/sec execute 22 sec latency 3380.459 ms 2024-06-09T22:25:17.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:16 smithi153 ceph-mon[36599]: pgmap v132: 129 pgs: 129 active+clean; 73 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 87 op/s 2024-06-09T22:25:17.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:16 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:25:17.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:16 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:25:17.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:16 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:25:17.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:16 smithi104 ceph-mon[32389]: pgmap v132: 129 pgs: 129 active+clean; 73 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 87 op/s 2024-06-09T22:25:17.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:16 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:25:17.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:16 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:25:17.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:16 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:25:17.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:16 smithi194 ceph-mon[36856]: pgmap v132: 129 pgs: 129 active+clean; 73 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 87 op/s 2024-06-09T22:25:17.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:16 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:25:17.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:16 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:25:17.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:16 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:25:17.398 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-09T22:25:17.662 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:17.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 89188 4.14 MB/sec execute 23 sec latency 15.149 ms 2024-06-09T22:25:18.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 89392 3.97 MB/sec execute 24 sec latency 841.515 ms 2024-06-09T22:25:19.117 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:18 smithi104 ceph-mon[32389]: pgmap v133: 129 pgs: 129 active+clean; 73 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 87 op/s 2024-06-09T22:25:19.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:18 smithi153 ceph-mon[36599]: pgmap v133: 129 pgs: 129 active+clean; 73 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 87 op/s 2024-06-09T22:25:19.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:18 smithi194 ceph-mon[36856]: pgmap v133: 129 pgs: 129 active+clean; 73 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 87 op/s 2024-06-09T22:25:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.478+0000 7f685a6cd700 1 -- 172.21.15.104:0/734221488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685406ac10 msgr2=0x7f685406b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.478+0000 7f685a6cd700 1 --2- 172.21.15.104:0/734221488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685406ac10 0x7f685406b090 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f683c0099c0 tx=0x7f683c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.478+0000 7f685a6cd700 1 -- 172.21.15.104:0/734221488 shutdown_connections 2024-06-09T22:25:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.478+0000 7f685a6cd700 1 --2- 172.21.15.104:0/734221488 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f685406b5d0 0x7f685410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.478+0000 7f685a6cd700 1 --2- 172.21.15.104:0/734221488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6854107390 0x7f6854107770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.478+0000 7f685a6cd700 1 --2- 172.21.15.104:0/734221488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685406ac10 0x7f685406b090 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.478+0000 7f685a6cd700 1 -- 172.21.15.104:0/734221488 >> 172.21.15.104:0/734221488 conn(0x7f68540783a0 msgr2=0x7f68540787b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.478+0000 7f685a6cd700 1 -- 172.21.15.104:0/734221488 shutdown_connections 2024-06-09T22:25:19.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.478+0000 7f685a6cd700 1 -- 172.21.15.104:0/734221488 wait complete. 2024-06-09T22:25:19.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f685a6cd700 1 Processor -- start 2024-06-09T22:25:19.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f685a6cd700 1 -- start start 2024-06-09T22:25:19.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f685a6cd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f685406ac10 0x7f6854198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f685a6cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685406b5d0 0x7f68541992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f685a6cd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6854107390 0x7f685419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f685a6cd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6854112850 con 0x7f685406b5d0 2024-06-09T22:25:19.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f685a6cd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f68541126d0 con 0x7f6854107390 2024-06-09T22:25:19.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f685a6cd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f68541129d0 con 0x7f685406ac10 2024-06-09T22:25:19.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f684bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685406b5d0 0x7f68541992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f684bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685406b5d0 0x7f68541992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33604/0 (socket says 172.21.15.104:33604) 2024-06-09T22:25:19.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f684bfff700 1 -- 172.21.15.104:0/2630087897 learned_addr learned my addr 172.21.15.104:0/2630087897 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:19.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f6858c6a700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6854107390 0x7f685419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f6853fff700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f685406ac10 0x7f6854198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f684bfff700 1 -- 172.21.15.104:0/2630087897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f685406ac10 msgr2=0x7f6854198d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f684bfff700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f685406ac10 0x7f6854198d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f684bfff700 1 -- 172.21.15.104:0/2630087897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6854107390 msgr2=0x7f685419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f684bfff700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6854107390 0x7f685419d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f684bfff700 1 -- 172.21.15.104:0/2630087897 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685419dd40 con 0x7f685406b5d0 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f6858c6a700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6854107390 0x7f685419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f6853fff700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f685406ac10 0x7f6854198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.479+0000 7f684bfff700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685406b5d0 0x7f68541992c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f683c009fd0 tx=0x7f683c003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.480+0000 7f6851ffb700 1 -- 172.21.15.104:0/2630087897 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c027070 con 0x7f685406b5d0 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.480+0000 7f685a6cd700 1 -- 172.21.15.104:0/2630087897 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f685419dfd0 con 0x7f685406b5d0 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.480+0000 7f6851ffb700 1 -- 172.21.15.104:0/2630087897 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c004150 con 0x7f685406b5d0 2024-06-09T22:25:19.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.480+0000 7f6851ffb700 1 -- 172.21.15.104:0/2630087897 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c00f460 con 0x7f685406b5d0 2024-06-09T22:25:19.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.480+0000 7f685a6cd700 1 -- 172.21.15.104:0/2630087897 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68541a58b0 con 0x7f685406b5d0 2024-06-09T22:25:19.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.483+0000 7f6851ffb700 1 -- 172.21.15.104:0/2630087897 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f683c02a070 con 0x7f685406b5d0 2024-06-09T22:25:19.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.483+0000 7f6851ffb700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f68400a2d90 0x7f68400a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.483+0000 7f6853fff700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f68400a2d90 0x7f68400a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.483+0000 7f6851ffb700 1 -- 172.21.15.104:0/2630087897 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f683c0d0740 con 0x7f685406b5d0 2024-06-09T22:25:19.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f6853fff700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f68400a2d90 0x7f68400a5250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6844007900 tx=0x7f6844008040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:19.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f6864f50b80 1 Processor -- start 2024-06-09T22:25:19.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f6864f50b80 1 -- start start 2024-06-09T22:25:19.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f6864f50b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af119a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f6864f50b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 0x5622af1139e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f6864f50b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 0x5622af114340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f6864f50b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5622af065810 con 0x5622af119f60 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f6864f50b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5622af00d720 con 0x5622af113f20 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f6864f50b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5622af0617d0 con 0x5622af119640 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f684bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 0x5622af1139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f684bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 0x5622af1139e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33610/0 (socket says 172.21.15.104:33610) 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f684bfff700 1 -- 172.21.15.104:0/3884254123 learned_addr learned my addr 172.21.15.104:0/3884254123 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f6858c6a700 1 --2- 172.21.15.104:0/3884254123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 0x5622af114340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f6853fff700 1 --2- 172.21.15.104:0/3884254123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af119a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f684bfff700 1 -- 172.21.15.104:0/3884254123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 msgr2=0x5622af119a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f684bfff700 1 --2- 172.21.15.104:0/3884254123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af119a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f684bfff700 1 -- 172.21.15.104:0/3884254123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 msgr2=0x5622af114340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f684bfff700 1 --2- 172.21.15.104:0/3884254123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 0x5622af114340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.484+0000 7f684bfff700 1 -- 172.21.15.104:0/3884254123 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622af003f50 con 0x5622af119f60 2024-06-09T22:25:19.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.485+0000 7f684bfff700 1 --2- 172.21.15.104:0/3884254123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 0x5622af1139e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f683c099350 tx=0x7f683c004300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a530a01da20a9b4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.485+0000 7f684a7fc700 1 -- 172.21.15.104:0/3884254123 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c09c980 con 0x5622af119f60 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.485+0000 7f684a7fc700 1 -- 172.21.15.104:0/3884254123 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c0a0d30 con 0x5622af119f60 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.485+0000 7f684a7fc700 1 -- 172.21.15.104:0/3884254123 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c09cbe0 con 0x5622af119f60 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.485+0000 7f6864f50b80 1 -- 172.21.15.104:0/3884254123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 msgr2=0x5622af1139e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.485+0000 7f6864f50b80 1 --2- 172.21.15.104:0/3884254123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 0x5622af1139e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f683c099350 tx=0x7f683c004300 comp rx=0 tx=0).stop 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 -- 172.21.15.104:0/3884254123 shutdown_connections 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 --2- 172.21.15.104:0/3884254123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 0x5622af114340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 --2- 172.21.15.104:0/3884254123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af119a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 --2- 172.21.15.104:0/3884254123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 0x5622af1139e0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 -- 172.21.15.104:0/3884254123 >> 172.21.15.104:0/3884254123 conn(0x5622af03ecc0 msgr2=0x5622af10e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 -- 172.21.15.104:0/3884254123 shutdown_connections 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 -- 172.21.15.104:0/3884254123 wait complete. 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 -- start start 2024-06-09T22:25:19.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 Processor -- start 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 0x5622af11fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af11d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 0x5622af11d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5622af1206f0 con 0x5622af119f60 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5622af005bf0 con 0x5622af113f20 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5622af120870 con 0x5622af119640 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6864f50b80 10 client.0 _ll_register_callbacks cb 0x5622aefff920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.486+0000 7f6858c6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 0x5622af11d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f6858c6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 0x5622af11d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33620/0 (socket says 172.21.15.104:33620) 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f6858c6a700 1 -- 172.21.15.104:0/1734230425 learned_addr learned my addr 172.21.15.104:0/1734230425 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f684bfff700 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af11d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f6853fff700 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 0x5622af11fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f6858c6a700 1 -- 172.21.15.104:0/1734230425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 msgr2=0x5622af11d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:19.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f6858c6a700 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af11d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f6858c6a700 1 -- 172.21.15.104:0/1734230425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 msgr2=0x5622af11fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f6858c6a700 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 0x5622af11fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f6858c6a700 1 -- 172.21.15.104:0/1734230425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622aefb48d0 con 0x5622af119f60 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f6858c6a700 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 0x5622af11d570 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f684c00ea90 tx=0x7f684c00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f81efed39565589 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f68337fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f68337fe700 1 -- 172.21.15.104:0/1734230425 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684c00cbc0 con 0x5622af119f60 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f68337fe700 1 -- 172.21.15.104:0/1734230425 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f684c00cd60 con 0x5622af119f60 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f6864f50b80 1 -- 172.21.15.104:0/1734230425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f683c0a0d30 con 0x5622af119f60 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f68337fe700 1 -- 172.21.15.104:0/1734230425 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f684c023050 con 0x5622af119f60 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f6864f50b80 10 client.16665 fetch_fsmap learned FSMap version 30 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.487+0000 7f6864f50b80 1 -- 172.21.15.104:0/1734230425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5622af12b050 con 0x5622af119f60 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.488+0000 7f68337fe700 1 -- 172.21.15.104:0/1734230425 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c0a0d30 con 0x5622af119f60 2024-06-09T22:25:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.488+0000 7f68337fe700 1 -- 172.21.15.104:0/1734230425 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f684c01a4a0 con 0x5622af119f60 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.488+0000 7f6864f50b80 10 client.16665 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.488+0000 7f6864f50b80 10 client.16665 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.488+0000 7f6864f50b80 20 client.16665 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.488+0000 7f6864f50b80 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5622af12d690 0x5622af12fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.488+0000 7f6864f50b80 4 client.16665 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.488+0000 7f6864f50b80 1 -- 172.21.15.104:0/1734230425 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622af072d10 con 0x5622af12d690 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.488+0000 7f6853fff700 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5622af12d690 0x5622af12fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.489+0000 7f68337fe700 10 client.16665 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.489+0000 7f6853fff700 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5622af12d690 0x5622af12fa80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5849c752c6547827 server_cookie=def97ab31deac798 in_seq=0 out_seq=0 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.489+0000 7f68337fe700 1 -- 172.21.15.104:0/1734230425 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5622af072d10 con 0x5622af12d690 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.489+0000 7f68337fe700 10 client.16665 handle_command_reply: tid=0 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.489+0000 7f6864f50b80 1 client.16665 shutdown 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.490+0000 7f6864f50b80 1 -- 172.21.15.104:0/1734230425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 msgr2=0x5622af11d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.490+0000 7f6864f50b80 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 0x5622af11d570 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f684c00ea90 tx=0x7f684c00eda0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.490+0000 7f6864f50b80 1 -- 172.21.15.104:0/1734230425 shutdown_connections 2024-06-09T22:25:19.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.490+0000 7f6864f50b80 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5622af12d690 0x5622af12fa80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.490+0000 7f6864f50b80 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119f60 0x5622af11d570 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.490+0000 7f6864f50b80 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 0x5622af11fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.490+0000 7f6864f50b80 1 --2- 172.21.15.104:0/1734230425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af11d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.490+0000 7f68337fe700 0 client.16665 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:25:19.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.490+0000 7f6864f50b80 1 -- 172.21.15.104:0/1734230425 >> 172.21.15.104:0/1734230425 conn(0x5622af03ecc0 msgr2=0x5622af10e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:19.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.490+0000 7f6864f50b80 1 -- 172.21.15.104:0/1734230425 shutdown_connections 2024-06-09T22:25:19.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.490+0000 7f6864f50b80 1 -- 172.21.15.104:0/1734230425 wait complete. 2024-06-09T22:25:19.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.490+0000 7f6864f50b80 20 client.16665 trim_cache size 0 max 16384 2024-06-09T22:25:19.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.505+0000 7f6864f50b80 1 Processor -- start 2024-06-09T22:25:19.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.505+0000 7f6864f50b80 1 -- start start 2024-06-09T22:25:19.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.505+0000 7f6864f50b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 0x5622af1139e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.505+0000 7f6864f50b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119640 0x5622af117730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.505+0000 7f6864f50b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119f60 0x5622af11a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.505+0000 7f6864f50b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f68541ae4d0 con 0x5622af119640 2024-06-09T22:25:19.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.505+0000 7f6864f50b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f68541b0070 con 0x5622af113f20 2024-06-09T22:25:19.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.505+0000 7f6864f50b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f68541aeba0 con 0x5622af119f60 2024-06-09T22:25:19.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.505+0000 7f684bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119640 0x5622af117730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.505+0000 7f684bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119640 0x5622af117730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33636/0 (socket says 172.21.15.104:33636) 2024-06-09T22:25:19.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.505+0000 7f684bfff700 1 -- 172.21.15.104:0/1509557560 learned_addr learned my addr 172.21.15.104:0/1509557560 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:19.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.506+0000 7f6858c6a700 1 --2- 172.21.15.104:0/1509557560 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119f60 0x5622af11a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.506+0000 7f6853fff700 1 --2- 172.21.15.104:0/1509557560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 0x5622af1139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.506+0000 7f684bfff700 1 -- 172.21.15.104:0/1509557560 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119f60 msgr2=0x5622af11a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:19.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.506+0000 7f684bfff700 1 --2- 172.21.15.104:0/1509557560 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119f60 0x5622af11a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.506+0000 7f684bfff700 1 -- 172.21.15.104:0/1509557560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 msgr2=0x5622af1139e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:19.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.506+0000 7f684bfff700 1 --2- 172.21.15.104:0/1509557560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 0x5622af1139e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.506+0000 7f684bfff700 1 -- 172.21.15.104:0/1509557560 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622af120330 con 0x5622af119640 2024-06-09T22:25:19.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.506+0000 7f684bfff700 1 --2- 172.21.15.104:0/1509557560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119640 0x5622af117730 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f683c0059d0 tx=0x7f683c087040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed09c5aa04fce66b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:19.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.506+0000 7f6849ffb700 1 -- 172.21.15.104:0/1509557560 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c09c980 con 0x5622af119640 2024-06-09T22:25:19.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.506+0000 7f6849ffb700 1 -- 172.21.15.104:0/1509557560 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c0b4070 con 0x5622af119640 2024-06-09T22:25:19.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.506+0000 7f6849ffb700 1 -- 172.21.15.104:0/1509557560 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c0a3050 con 0x5622af119640 2024-06-09T22:25:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.507+0000 7f6864f50b80 1 -- 172.21.15.104:0/1509557560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119640 msgr2=0x5622af117730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.507+0000 7f6864f50b80 1 --2- 172.21.15.104:0/1509557560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119640 0x5622af117730 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f683c0059d0 tx=0x7f683c087040 comp rx=0 tx=0).stop 2024-06-09T22:25:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.507+0000 7f6864f50b80 1 -- 172.21.15.104:0/1509557560 shutdown_connections 2024-06-09T22:25:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.507+0000 7f6864f50b80 1 --2- 172.21.15.104:0/1509557560 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119f60 0x5622af11a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.507+0000 7f6864f50b80 1 --2- 172.21.15.104:0/1509557560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af113f20 0x5622af1139e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.507+0000 7f6864f50b80 1 --2- 172.21.15.104:0/1509557560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af119640 0x5622af117730 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.507+0000 7f6864f50b80 1 -- 172.21.15.104:0/1509557560 >> 172.21.15.104:0/1509557560 conn(0x5622af03ecc0 msgr2=0x5622af11b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.507+0000 7f6864f50b80 1 -- 172.21.15.104:0/1509557560 shutdown_connections 2024-06-09T22:25:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.507+0000 7f6864f50b80 1 -- 172.21.15.104:0/1509557560 wait complete. 2024-06-09T22:25:19.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.507+0000 7f6864f50b80 1 -- start start 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.507+0000 7f6864f50b80 1 Processor -- start 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6864f50b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af113f20 0x5622af125ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6864f50b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af1261e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6864f50b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af119f60 0x5622af134710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6864f50b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f68541af940 con 0x5622af113f20 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6864f50b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f68541af270 con 0x5622af119f60 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6864f50b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f68541ac450 con 0x5622af119640 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6864f50b80 10 client.0 _ll_register_callbacks cb 0x5622aefff920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6853fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af113f20 0x5622af125ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6853fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af113f20 0x5622af125ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33642/0 (socket says 172.21.15.104:33642) 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6853fff700 1 -- 172.21.15.104:0/4261761964 learned_addr learned my addr 172.21.15.104:0/4261761964 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6858c6a700 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af119f60 0x5622af134710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f684bfff700 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af1261e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6853fff700 1 -- 172.21.15.104:0/4261761964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 msgr2=0x5622af1261e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6853fff700 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af1261e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6853fff700 1 -- 172.21.15.104:0/4261761964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af119f60 msgr2=0x5622af134710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6853fff700 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af119f60 0x5622af134710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6853fff700 1 -- 172.21.15.104:0/4261761964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622af134f80 con 0x5622af113f20 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f684bfff700 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af1261e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6858c6a700 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af119f60 0x5622af134710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6853fff700 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af113f20 0x5622af125ca0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6844007540 tx=0x7f684400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9b47f404b5088d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f68337fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f68337fe700 1 -- 172.21.15.104:0/4261761964 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684400ce80 con 0x5622af113f20 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f68337fe700 1 -- 172.21.15.104:0/4261761964 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6844021d90 con 0x5622af113f20 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.508+0000 7f6864f50b80 1 -- 172.21.15.104:0/4261761964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f683c0b4070 con 0x5622af113f20 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.509+0000 7f68337fe700 1 -- 172.21.15.104:0/4261761964 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6844039050 con 0x5622af113f20 2024-06-09T22:25:19.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.509+0000 7f6864f50b80 10 client.16671 fetch_fsmap learned FSMap version 30 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.509+0000 7f6864f50b80 1 -- 172.21.15.104:0/4261761964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5622af197d90 con 0x5622af113f20 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.509+0000 7f68337fe700 1 -- 172.21.15.104:0/4261761964 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c0b4070 con 0x5622af113f20 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.509+0000 7f68337fe700 1 -- 172.21.15.104:0/4261761964 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f68440135b0 con 0x5622af113f20 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.509+0000 7f6864f50b80 10 client.16671 fetch_fsmap finished waiting for FSMap version 30 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.509+0000 7f6864f50b80 10 client.16671 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.509+0000 7f6864f50b80 20 client.16671 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.509+0000 7f6864f50b80 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5622af198480 0x5622af19a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.509+0000 7f6864f50b80 4 client.16671 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.509+0000 7f6864f50b80 1 -- 172.21.15.104:0/4261761964 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5622af06d290 con 0x5622af198480 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.509+0000 7f684bfff700 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5622af198480 0x5622af19a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f68337fe700 10 client.16671 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f684bfff700 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5622af198480 0x5622af19a8a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4e279a34db5d0916 server_cookie=f24b8e2fb39d32e6 in_seq=0 out_seq=0 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f68337fe700 1 -- 172.21.15.104:0/4261761964 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5622af06d290 con 0x5622af198480 2024-06-09T22:25:19.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f68337fe700 10 client.16671 handle_command_reply: tid=0 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f6864f50b80 1 client.16671 shutdown 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f6864f50b80 1 -- 172.21.15.104:0/4261761964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af113f20 msgr2=0x5622af125ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f6864f50b80 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af113f20 0x5622af125ca0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6844007540 tx=0x7f684400f000 comp rx=0 tx=0).stop 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f6864f50b80 1 -- 172.21.15.104:0/4261761964 shutdown_connections 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f6864f50b80 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5622af198480 0x5622af19a8a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f6864f50b80 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5622af113f20 0x5622af125ca0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f6864f50b80 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622af119f60 0x5622af134710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f68337fe700 0 client.16671 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f6864f50b80 1 --2- 172.21.15.104:0/4261761964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5622af119640 0x5622af1261e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f6864f50b80 1 -- 172.21.15.104:0/4261761964 >> 172.21.15.104:0/4261761964 conn(0x5622af03ecc0 msgr2=0x5622af107e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f6864f50b80 1 -- 172.21.15.104:0/4261761964 shutdown_connections 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f6864f50b80 1 -- 172.21.15.104:0/4261761964 wait complete. 2024-06-09T22:25:19.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.510+0000 7f6864f50b80 20 client.16671 trim_cache size 0 max 16384 2024-06-09T22:25:19.517 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:25:19.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.511+0000 7f685a6cd700 1 -- 172.21.15.104:0/2630087897 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f68400a2d90 msgr2=0x7f68400a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:19.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.511+0000 7f685a6cd700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f68400a2d90 0x7f68400a5250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6844007900 tx=0x7f6844008040 comp rx=0 tx=0).stop 2024-06-09T22:25:19.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.511+0000 7f685a6cd700 1 -- 172.21.15.104:0/2630087897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685406b5d0 msgr2=0x7f68541992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:19.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.511+0000 7f685a6cd700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685406b5d0 0x7f68541992c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f683c009fd0 tx=0x7f683c003730 comp rx=0 tx=0).stop 2024-06-09T22:25:19.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.511+0000 7f685a6cd700 1 -- 172.21.15.104:0/2630087897 shutdown_connections 2024-06-09T22:25:19.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.511+0000 7f685a6cd700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f68400a2d90 0x7f68400a5250 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.511+0000 7f685a6cd700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6854107390 0x7f685419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.511+0000 7f685a6cd700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f685406ac10 0x7f6854198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.511+0000 7f685a6cd700 1 --2- 172.21.15.104:0/2630087897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685406b5d0 0x7f68541992c0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:19.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.511+0000 7f685a6cd700 1 -- 172.21.15.104:0/2630087897 >> 172.21.15.104:0/2630087897 conn(0x7f68540783a0 msgr2=0x7f68540ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:19.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.511+0000 7f685a6cd700 1 -- 172.21.15.104:0/2630087897 shutdown_connections 2024-06-09T22:25:19.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:19.511+0000 7f685a6cd700 1 -- 172.21.15.104:0/2630087897 wait complete. 2024-06-09T22:25:19.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 89392 3.81 MB/sec execute 25 sec latency 1841.605 ms 2024-06-09T22:25:20.342 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-09T22:25:20.582 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:20.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 89448 3.67 MB/sec execute 26 sec latency 2142.617 ms 2024-06-09T22:25:20.960 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:20 smithi104 ceph-mon[32389]: pgmap v134: 129 pgs: 129 active+clean; 69 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 129 op/s 2024-06-09T22:25:20.961 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:20 smithi104 ceph-mon[32389]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:25:21.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:20 smithi153 ceph-mon[36599]: pgmap v134: 129 pgs: 129 active+clean; 69 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 129 op/s 2024-06-09T22:25:21.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:20 smithi153 ceph-mon[36599]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:25:21.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:20 smithi194 ceph-mon[36856]: pgmap v134: 129 pgs: 129 active+clean; 69 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 129 op/s 2024-06-09T22:25:21.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:20 smithi194 ceph-mon[36856]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:25:21.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 91162 3.79 MB/sec execute 27 sec latency 1036.854 ms 2024-06-09T22:25:22.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.327+0000 7f669c330700 1 -- 172.21.15.104:0/1368274754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66941073d0 msgr2=0x7f66941077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:22.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.327+0000 7f669c330700 1 --2- 172.21.15.104:0/1368274754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66941073d0 0x7f66941077b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f6690009990 tx=0x7f6690009ca0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.327+0000 7f669c330700 1 -- 172.21.15.104:0/1368274754 shutdown_connections 2024-06-09T22:25:22.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.327+0000 7f669c330700 1 --2- 172.21.15.104:0/1368274754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66941073d0 0x7f66941077b0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.327+0000 7f669c330700 1 --2- 172.21.15.104:0/1368274754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6694101d30 0x7f669410e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.327+0000 7f669c330700 1 --2- 172.21.15.104:0/1368274754 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6694101390 0x7f66941017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.327+0000 7f669c330700 1 -- 172.21.15.104:0/1368274754 >> 172.21.15.104:0/1368274754 conn(0x7f66940782c0 msgr2=0x7f66940786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:22.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.327+0000 7f669c330700 1 -- 172.21.15.104:0/1368274754 shutdown_connections 2024-06-09T22:25:22.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.327+0000 7f669c330700 1 -- 172.21.15.104:0/1368274754 wait complete. 2024-06-09T22:25:22.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669c330700 1 Processor -- start 2024-06-09T22:25:22.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669c330700 1 -- start start 2024-06-09T22:25:22.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669c330700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6694101390 0x7f669419e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669c330700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6694101d30 0x7f669419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669c330700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66941073d0 0x7f66941989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669c330700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6694111610 con 0x7f6694101d30 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669c330700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6694111490 con 0x7f6694101390 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669c330700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6694111790 con 0x7f66941073d0 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f66998cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6694101d30 0x7f669419ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669a0cc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6694101390 0x7f669419e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669a0cc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6694101390 0x7f669419e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:40188/0 (socket says 172.21.15.104:40188) 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f66998cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6694101d30 0x7f669419ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33654/0 (socket says 172.21.15.104:33654) 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669a0cc700 1 -- 172.21.15.104:0/1795917353 learned_addr learned my addr 172.21.15.104:0/1795917353 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669a8cd700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66941073d0 0x7f66941989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669a0cc700 1 -- 172.21.15.104:0/1795917353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66941073d0 msgr2=0x7f66941989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669a0cc700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66941073d0 0x7f66941989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669a0cc700 1 -- 172.21.15.104:0/1795917353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6694101d30 msgr2=0x7f669419ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:22.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669a0cc700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6694101d30 0x7f669419ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669a0cc700 1 -- 172.21.15.104:0/1795917353 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6694199150 con 0x7f6694101390 2024-06-09T22:25:22.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f66998cb700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6694101d30 0x7f669419ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:22.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.328+0000 7f669a0cc700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6694101390 0x7f669419e830 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6690000c00 tx=0x7f669000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:22.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.329+0000 7f668b7fe700 1 -- 172.21.15.104:0/1795917353 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6690027070 con 0x7f6694101390 2024-06-09T22:25:22.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.329+0000 7f669c330700 1 -- 172.21.15.104:0/1795917353 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66941993e0 con 0x7f6694101390 2024-06-09T22:25:22.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.329+0000 7f668b7fe700 1 -- 172.21.15.104:0/1795917353 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6690003d70 con 0x7f6694101390 2024-06-09T22:25:22.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.329+0000 7f669c330700 1 -- 172.21.15.104:0/1795917353 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66941a5700 con 0x7f6694101390 2024-06-09T22:25:22.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.329+0000 7f668b7fe700 1 -- 172.21.15.104:0/1795917353 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669000f460 con 0x7f6694101390 2024-06-09T22:25:22.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.331+0000 7f668b7fe700 1 -- 172.21.15.104:0/1795917353 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f669002a070 con 0x7f6694101390 2024-06-09T22:25:22.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.332+0000 7f668b7fe700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f66800a2d90 0x7f66800a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.332+0000 7f668b7fe700 1 -- 172.21.15.104:0/1795917353 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f66900d0550 con 0x7f6694101390 2024-06-09T22:25:22.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.332+0000 7f66998cb700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f66800a2d90 0x7f66800a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.333+0000 7f66a6bb3b80 1 Processor -- start 2024-06-09T22:25:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.333+0000 7f66a6bb3b80 1 -- start start 2024-06-09T22:25:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.333+0000 7f66998cb700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f66800a2d90 0x7f66800a5250 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f66940ffb20 tx=0x7f6684009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.333+0000 7f66a6bb3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855a88c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.333+0000 7f66a6bb3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855a8cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.333+0000 7f66a6bb3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855a8d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.333+0000 7f66a6bb3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557855976f60 con 0x557855a89170 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.333+0000 7f66a6bb3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5578559d5850 con 0x557855a88850 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.333+0000 7f66a6bb3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55785593d460 con 0x557855a8d510 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f669a8cd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855a8d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f669a8cd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855a8d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:52326/0 (socket says 172.21.15.104:52326) 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f66998cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855a8cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f669a8cd700 1 -- 172.21.15.104:0/962718519 learned_addr learned my addr 172.21.15.104:0/962718519 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f669a0cc700 1 --2- 172.21.15.104:0/962718519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855a88c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f669a8cd700 1 -- 172.21.15.104:0/962718519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 msgr2=0x557855a88c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f669a8cd700 1 --2- 172.21.15.104:0/962718519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855a88c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f669a8cd700 1 -- 172.21.15.104:0/962718519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 msgr2=0x557855a8cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f669a8cd700 1 --2- 172.21.15.104:0/962718519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855a8cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f669a8cd700 1 -- 172.21.15.104:0/962718519 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578559248d0 con 0x557855a8d510 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f66998cb700 1 --2- 172.21.15.104:0/962718519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855a8cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f669a8cd700 1 --2- 172.21.15.104:0/962718519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855a8d970 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f668c011680 tx=0x7f668c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b24c6462a18866 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f6688d70700 1 -- 172.21.15.104:0/962718519 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c019030 con 0x557855a8d510 2024-06-09T22:25:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f6688d70700 1 -- 172.21.15.104:0/962718519 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f668c0092e0 con 0x557855a8d510 2024-06-09T22:25:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.334+0000 7f6688d70700 1 -- 172.21.15.104:0/962718519 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c007760 con 0x557855a8d510 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.335+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/962718519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 msgr2=0x557855a8d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.335+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/962718519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855a8d970 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f668c011680 tx=0x7f668c009920 comp rx=0 tx=0).stop 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.335+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/962718519 shutdown_connections 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.335+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/962718519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855a8d970 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.335+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/962718519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855a88c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.335+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/962718519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855a8cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.335+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/962718519 >> 172.21.15.104:0/962718519 conn(0x5578559b1080 msgr2=0x557855a7f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.335+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/962718519 shutdown_connections 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.335+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/962718519 wait complete. 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f66a6bb3b80 1 -- start start 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f66a6bb3b80 1 Processor -- start 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f66a6bb3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855a94a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f66a6bb3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855a91d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f66a6bb3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855a92280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f66a6bb3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557855975fc0 con 0x557855a89170 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f66a6bb3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5578559d1810 con 0x557855a88850 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f66a6bb3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557855a95660 con 0x557855a8d510 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f66a6bb3b80 10 client.0 _ll_register_callbacks cb 0x55785596b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f669a8cd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855a92280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f669a0cc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855a94a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f66998cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855a91d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f669a8cd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855a92280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:52332/0 (socket says 172.21.15.104:52332) 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f669a0cc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855a94a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:40212/0 (socket says 172.21.15.104:40212) 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f669a8cd700 1 -- 172.21.15.104:0/2525584656 learned_addr learned my addr 172.21.15.104:0/2525584656 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.336+0000 7f66998cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855a91d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33658/0 (socket says 172.21.15.104:33658) 2024-06-09T22:25:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f669a8cd700 1 -- 172.21.15.104:0/2525584656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 msgr2=0x557855a94a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f669a8cd700 1 --2- 172.21.15.104:0/2525584656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855a94a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f669a8cd700 1 -- 172.21.15.104:0/2525584656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 msgr2=0x557855a91d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f669a8cd700 1 --2- 172.21.15.104:0/2525584656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855a91d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f669a8cd700 1 -- 172.21.15.104:0/2525584656 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55785599b750 con 0x557855a8d510 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f66998cb700 1 --2- 172.21.15.104:0/2525584656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855a91d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f669a8cd700 1 --2- 172.21.15.104:0/2525584656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855a92280 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f668c0045d0 tx=0x7f668c004900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1f3924b5448542c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f667dffb700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f667dffb700 1 -- 172.21.15.104:0/2525584656 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c003a00 con 0x557855a8d510 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f667dffb700 1 -- 172.21.15.104:0/2525584656 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f668c003ba0 con 0x557855a8d510 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/2525584656 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f668c0092e0 con 0x557855a8d510 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f667dffb700 1 -- 172.21.15.104:0/2525584656 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f668c030050 con 0x557855a8d510 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f66a6bb3b80 10 client.26125 fetch_fsmap learned FSMap version 31 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.337+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/2525584656 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557855a9fde0 con 0x557855a8d510 2024-06-09T22:25:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.338+0000 7f667dffb700 1 -- 172.21.15.104:0/2525584656 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f668c0092e0 con 0x557855a8d510 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.338+0000 7f667dffb700 1 -- 172.21.15.104:0/2525584656 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f668c005050 con 0x557855a8d510 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.338+0000 7f66a6bb3b80 10 client.26125 fetch_fsmap finished waiting for FSMap version 31 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.338+0000 7f66a6bb3b80 10 client.26125 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.338+0000 7f66a6bb3b80 20 client.26125 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.338+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/2525584656 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x557855aa2360 0x557855aa4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.338+0000 7f66a6bb3b80 4 client.26125 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.338+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/2525584656 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578559e2d50 con 0x557855aa2360 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.338+0000 7f669a0cc700 1 --2- 172.21.15.104:0/2525584656 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x557855aa2360 0x557855aa4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.338+0000 7f669a0cc700 1 --2- 172.21.15.104:0/2525584656 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x557855aa2360 0x557855aa4750 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=def2e4093cea3c6d server_cookie=bfba1c4528468e72 in_seq=0 out_seq=0 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.338+0000 7f667dffb700 10 client.26125 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.339+0000 7f667dffb700 1 -- 172.21.15.104:0/2525584656 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5578559e2d50 con 0x557855aa2360 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.339+0000 7f667dffb700 10 client.26125 handle_command_reply: tid=0 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.339+0000 7f66a6bb3b80 1 client.26125 shutdown 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.339+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/2525584656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 msgr2=0x557855a92280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.339+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/2525584656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855a92280 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f668c0045d0 tx=0x7f668c004900 comp rx=0 tx=0).stop 2024-06-09T22:25:22.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.339+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/2525584656 shutdown_connections 2024-06-09T22:25:22.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.339+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/2525584656 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x557855aa2360 0x557855aa4750 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.339+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/2525584656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855a92280 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.339+0000 7f667dffb700 0 client.26125 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:25:22.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.339+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/2525584656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855a94a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.339+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/2525584656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855a91d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.339+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/2525584656 >> 172.21.15.104:0/2525584656 conn(0x5578559b1080 msgr2=0x557855a83250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:22.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.340+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/2525584656 shutdown_connections 2024-06-09T22:25:22.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.340+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/2525584656 wait complete. 2024-06-09T22:25:22.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.340+0000 7f66a6bb3b80 20 client.26125 trim_cache size 0 max 16384 2024-06-09T22:25:22.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f66a6bb3b80 1 Processor -- start 2024-06-09T22:25:22.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f66a6bb3b80 1 -- start start 2024-06-09T22:25:22.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f66a6bb3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a88850 0x557855a8f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f66a6bb3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a89170 0x557855a838c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f66a6bb3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a8d510 0x557855a7ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f66a6bb3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f66941ac290 con 0x557855a88850 2024-06-09T22:25:22.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f66a6bb3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f66941ade00 con 0x557855a8d510 2024-06-09T22:25:22.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f66a6bb3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f66941ac960 con 0x557855a89170 2024-06-09T22:25:22.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f669a0cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a88850 0x557855a8f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f669a0cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a88850 0x557855a8f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33662/0 (socket says 172.21.15.104:33662) 2024-06-09T22:25:22.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f669a0cc700 1 -- 172.21.15.104:0/845318996 learned_addr learned my addr 172.21.15.104:0/845318996 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:22.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f66998cb700 1 --2- 172.21.15.104:0/845318996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a89170 0x557855a838c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.357+0000 7f669a8cd700 1 --2- 172.21.15.104:0/845318996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a8d510 0x557855a7ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.358+0000 7f669a0cc700 1 -- 172.21.15.104:0/845318996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a89170 msgr2=0x557855a838c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:22.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.358+0000 7f669a0cc700 1 --2- 172.21.15.104:0/845318996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a89170 0x557855a838c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.358+0000 7f669a0cc700 1 -- 172.21.15.104:0/845318996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a8d510 msgr2=0x557855a7ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:22.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.358+0000 7f669a0cc700 1 --2- 172.21.15.104:0/845318996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a8d510 0x557855a7ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.358+0000 7f669a0cc700 1 -- 172.21.15.104:0/845318996 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557855a902b0 con 0x557855a88850 2024-06-09T22:25:22.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.358+0000 7f669a0cc700 1 --2- 172.21.15.104:0/845318996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a88850 0x557855a8f9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f66900cd990 tx=0x7f669002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631e96a94df1b5c4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:22.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.358+0000 7f667ffff700 1 -- 172.21.15.104:0/845318996 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669009c0f0 con 0x557855a88850 2024-06-09T22:25:22.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.358+0000 7f667ffff700 1 -- 172.21.15.104:0/845318996 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66900043d0 con 0x557855a88850 2024-06-09T22:25:22.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.358+0000 7f667ffff700 1 -- 172.21.15.104:0/845318996 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66900225e0 con 0x557855a88850 2024-06-09T22:25:22.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/845318996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a88850 msgr2=0x557855a8f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:22.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/845318996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a88850 0x557855a8f9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f66900cd990 tx=0x7f669002e040 comp rx=0 tx=0).stop 2024-06-09T22:25:22.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/845318996 shutdown_connections 2024-06-09T22:25:22.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/845318996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a88850 0x557855a8f9c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/845318996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a8d510 0x557855a7ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/845318996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a89170 0x557855a838c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/845318996 >> 172.21.15.104:0/845318996 conn(0x5578559b1080 msgr2=0x557855a93e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:22.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/845318996 shutdown_connections 2024-06-09T22:25:22.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/845318996 wait complete. 2024-06-09T22:25:22.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 -- start start 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 Processor -- start 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855aa4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855aa52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855b04c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f66941ad700 con 0x557855a89170 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f66941ad030 con 0x557855a88850 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f66941aa210 con 0x557855a8d510 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.359+0000 7f66a6bb3b80 10 client.0 _ll_register_callbacks cb 0x55785596b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f66998cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855aa52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f66998cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855aa52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33678/0 (socket says 172.21.15.104:33678) 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f66998cb700 1 -- 172.21.15.104:0/85825240 learned_addr learned my addr 172.21.15.104:0/85825240 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f669a0cc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855aa4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f66998cb700 1 -- 172.21.15.104:0/85825240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 msgr2=0x557855b04c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f66998cb700 1 --2- 172.21.15.104:0/85825240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855b04c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f66998cb700 1 -- 172.21.15.104:0/85825240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 msgr2=0x557855aa4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f66998cb700 1 --2- 172.21.15.104:0/85825240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855aa4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f66998cb700 1 -- 172.21.15.104:0/85825240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557855b052a0 con 0x557855a89170 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f669a0cc700 1 --2- 172.21.15.104:0/85825240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855aa4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f66998cb700 1 --2- 172.21.15.104:0/85825240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855aa52e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f6684011bb0 tx=0x7f66840047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6617e3f5bcc693f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f667dffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f667dffb700 1 -- 172.21.15.104:0/85825240 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668402e060 con 0x557855a89170 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f667dffb700 1 -- 172.21.15.104:0/85825240 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f668401ed10 con 0x557855a89170 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/85825240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66900043d0 con 0x557855a89170 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f667dffb700 1 -- 172.21.15.104:0/85825240 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6684035050 con 0x557855a89170 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f66a6bb3b80 10 client.16689 fetch_fsmap learned FSMap version 31 2024-06-09T22:25:22.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.360+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/85825240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557855b0c6c0 con 0x557855a89170 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.361+0000 7f667dffb700 1 -- 172.21.15.104:0/85825240 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66900043d0 con 0x557855a89170 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.361+0000 7f667dffb700 1 -- 172.21.15.104:0/85825240 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f6684028030 con 0x557855a89170 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.361+0000 7f66a6bb3b80 10 client.16689 fetch_fsmap finished waiting for FSMap version 31 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.361+0000 7f66a6bb3b80 10 client.16689 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.361+0000 7f66a6bb3b80 20 client.16689 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.361+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/85825240 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x557855b0ce30 0x557855b0f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.361+0000 7f66a6bb3b80 4 client.16689 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.361+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/85825240 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5578559dd2d0 con 0x557855b0ce30 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.361+0000 7f669a0cc700 1 --2- 172.21.15.104:0/85825240 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x557855b0ce30 0x557855b0f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.361+0000 7f667dffb700 10 client.16689 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.361+0000 7f669a0cc700 1 --2- 172.21.15.104:0/85825240 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x557855b0ce30 0x557855b0f250 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5358ef589a30d52c server_cookie=fcf1bef0d29a5a42 in_seq=0 out_seq=0 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f667dffb700 1 -- 172.21.15.104:0/85825240 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5578559dd2d0 con 0x557855b0ce30 2024-06-09T22:25:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f667dffb700 10 client.16689 handle_command_reply: tid=0 2024-06-09T22:25:22.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f66a6bb3b80 1 client.16689 shutdown 2024-06-09T22:25:22.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/85825240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 msgr2=0x557855aa52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:22.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/85825240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855aa52e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f6684011bb0 tx=0x7f66840047c0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/85825240 shutdown_connections 2024-06-09T22:25:22.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/85825240 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x557855b0ce30 0x557855b0f250 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/85825240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557855a8d510 0x557855b04c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/85825240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557855a88850 0x557855aa4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f66a6bb3b80 1 --2- 172.21.15.104:0/85825240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557855a89170 0x557855aa52e0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/85825240 >> 172.21.15.104:0/85825240 conn(0x5578559b1080 msgr2=0x557855a8c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:22.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f667dffb700 0 client.16689 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:25:22.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/85825240 shutdown_connections 2024-06-09T22:25:22.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f66a6bb3b80 1 -- 172.21.15.104:0/85825240 wait complete. 2024-06-09T22:25:22.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.362+0000 7f66a6bb3b80 20 client.16689 trim_cache size 0 max 16384 2024-06-09T22:25:22.370 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:25:22.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.363+0000 7f669c330700 1 -- 172.21.15.104:0/1795917353 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f66800a2d90 msgr2=0x7f66800a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:22.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.363+0000 7f669c330700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f66800a2d90 0x7f66800a5250 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f66940ffb20 tx=0x7f6684009260 comp rx=0 tx=0).stop 2024-06-09T22:25:22.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.363+0000 7f669c330700 1 -- 172.21.15.104:0/1795917353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6694101390 msgr2=0x7f669419e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:22.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.363+0000 7f669c330700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6694101390 0x7f669419e830 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6690000c00 tx=0x7f669000bab0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.363+0000 7f669c330700 1 -- 172.21.15.104:0/1795917353 shutdown_connections 2024-06-09T22:25:22.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.363+0000 7f669c330700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f66800a2d90 0x7f66800a5250 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.363+0000 7f669c330700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66941073d0 0x7f66941989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.363+0000 7f669c330700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6694101390 0x7f669419e830 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.363+0000 7f669c330700 1 --2- 172.21.15.104:0/1795917353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6694101d30 0x7f669419ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.363+0000 7f669c330700 1 -- 172.21.15.104:0/1795917353 >> 172.21.15.104:0/1795917353 conn(0x7f66940782c0 msgr2=0x7f66941006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:22.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.363+0000 7f669c330700 1 -- 172.21.15.104:0/1795917353 shutdown_connections 2024-06-09T22:25:22.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:22.363+0000 7f669c330700 1 -- 172.21.15.104:0/1795917353 wait complete. 2024-06-09T22:25:22.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 92867 4.09 MB/sec execute 28 sec latency 12.147 ms 2024-06-09T22:25:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.990+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2091652096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb8101980 msgr2=0x7f0cb8101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.990+0000 7f0cbe4dd640 1 --2- 172.21.15.104:0/2091652096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb8101980 0x7f0cb8101e00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009930 tx=0x7f0cac02a420 comp rx=0 tx=0).stop 2024-06-09T22:25:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.990+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2091652096 shutdown_connections 2024-06-09T22:25:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.990+0000 7f0cbe4dd640 1 --2- 172.21.15.104:0/2091652096 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0cb8102340 0x7f0cb810e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.990+0000 7f0cbe4dd640 1 --2- 172.21.15.104:0/2091652096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb8101980 0x7f0cb8101e00 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.990+0000 7f0cbe4dd640 1 --2- 172.21.15.104:0/2091652096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb8107990 0x7f0cb8107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.990+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2091652096 >> 172.21.15.104:0/2091652096 conn(0x7f0cb80fd450 msgr2=0x7f0cb80ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.990+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2091652096 shutdown_connections 2024-06-09T22:25:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.990+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2091652096 wait complete. 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.991+0000 7f0cbe4dd640 1 Processor -- start 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.991+0000 7f0cbe4dd640 1 -- start start 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.991+0000 7f0cbe4dd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb8101980 0x7f0cb8198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.991+0000 7f0cbe4dd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb8102340 0x7f0cb8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.991+0000 7f0cbe4dd640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0cb8107990 0x7f0cb819cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.991+0000 7f0cbe4dd640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb8111da0 con 0x7f0cb8101980 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.991+0000 7f0cbe4dd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb8111c20 con 0x7f0cb8102340 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.991+0000 7f0cbe4dd640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb8111f20 con 0x7f0cb8107990 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.991+0000 7f0cb7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb8101980 0x7f0cb8198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cb7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb8101980 0x7f0cb8198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33690/0 (socket says 172.21.15.104:33690) 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cb7fff640 1 -- 172.21.15.104:0/2046237870 learned_addr learned my addr 172.21.15.104:0/2046237870 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cbca7a640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0cb8107990 0x7f0cb819cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cb77fe640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb8102340 0x7f0cb8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cb7fff640 1 -- 172.21.15.104:0/2046237870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0cb8107990 msgr2=0x7f0cb819cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0cb8107990 0x7f0cb819cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cb7fff640 1 -- 172.21.15.104:0/2046237870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb8102340 msgr2=0x7f0cb8198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb8102340 0x7f0cb8198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cb7fff640 1 -- 172.21.15.104:0/2046237870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb819d1f0 con 0x7f0cb8101980 2024-06-09T22:25:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb8101980 0x7f0cb8198200 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800e9f0 tx=0x7f0ca800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cb57fa640 1 -- 172.21.15.104:0/2046237870 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca800cd80 con 0x7f0cb8101980 2024-06-09T22:25:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2046237870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb819d4e0 con 0x7f0cb8101980 2024-06-09T22:25:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cb57fa640 1 -- 172.21.15.104:0/2046237870 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ca8010910 con 0x7f0cb8101980 2024-06-09T22:25:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cb57fa640 1 -- 172.21.15.104:0/2046237870 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca80188f0 con 0x7f0cb8101980 2024-06-09T22:25:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.992+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2046237870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb81a4d70 con 0x7f0cb8101980 2024-06-09T22:25:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.994+0000 7f0cb57fa640 1 -- 172.21.15.104:0/2046237870 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f0ca8010ac0 con 0x7f0cb8101980 2024-06-09T22:25:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cc11ea740 1 Processor -- start 2024-06-09T22:25:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cc11ea740 1 -- start start 2024-06-09T22:25:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cb57fa640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0ca00a2da0 0x7f0ca00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cc11ea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19170 0x55c4f8b19550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cc11ea740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b23e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cbca7a640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0ca00a2da0 0x7f0ca00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cc11ea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 0x55c4f8b26740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cc11ea740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55c4f8a7d4f0 con 0x55c4f8b19a90 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cc11ea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c4f8a1b870 con 0x55c4f8b19170 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cc11ea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c4f8a794b0 con 0x55c4f8b24350 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cb57fa640 1 -- 172.21.15.104:0/2046237870 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f0ca80cfb20 con 0x7f0cb8101980 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cb77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b23e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cb77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b23e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33698/0 (socket says 172.21.15.104:33698) 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cb7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19170 0x55c4f8b19550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cb77fe640 1 -- 172.21.15.104:0/2378622191 learned_addr learned my addr 172.21.15.104:0/2378622191 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cbca7a640 1 --2- 172.21.15.104:0/2378622191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 0x55c4f8b26740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cb7fff640 1 -- 172.21.15.104:0/2378622191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 msgr2=0x55c4f8b26740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/2378622191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 0x55c4f8b26740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cb7fff640 1 -- 172.21.15.104:0/2378622191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 msgr2=0x55c4f8b23e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/2378622191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b23e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cb7fff640 1 -- 172.21.15.104:0/2378622191 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4f8a379b0 con 0x55c4f8b19170 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.995+0000 7f0cb77fe640 1 --2- 172.21.15.104:0/2378622191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b23e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.996+0000 7f0cbca7a640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0ca00a2da0 0x7f0ca00a5260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0005fd0 tx=0x7f0cb0014bf0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.996+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/2378622191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19170 0x55c4f8b19550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80ccf00 tx=0x7f0ca809be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=902c064fb3f6e0e8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.996+0000 7f0c9e7fc640 1 -- 172.21.15.104:0/2378622191 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca80983f0 con 0x55c4f8b19170 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.996+0000 7f0c9e7fc640 1 -- 172.21.15.104:0/2378622191 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ca80cd180 con 0x55c4f8b19170 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.996+0000 7f0c9e7fc640 1 -- 172.21.15.104:0/2378622191 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8097d30 con 0x55c4f8b19170 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.997+0000 7f0cc11ea740 1 -- 172.21.15.104:0/2378622191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19170 msgr2=0x55c4f8b19550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.997+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/2378622191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19170 0x55c4f8b19550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80ccf00 tx=0x7f0ca809be40 comp rx=0 tx=0).stop 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.997+0000 7f0cc11ea740 1 -- 172.21.15.104:0/2378622191 shutdown_connections 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.997+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/2378622191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 0x55c4f8b26740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.997+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/2378622191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b23e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.997+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/2378622191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19170 0x55c4f8b19550 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.997+0000 7f0cc11ea740 1 -- 172.21.15.104:0/2378622191 >> 172.21.15.104:0/2378622191 conn(0x55c4f8b15680 msgr2=0x55c4f8b15a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.997+0000 7f0cc11ea740 1 -- 172.21.15.104:0/2378622191 shutdown_connections 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.997+0000 7f0cc11ea740 1 -- 172.21.15.104:0/2378622191 wait complete. 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.997+0000 7f0cc11ea740 1 -- start start 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.997+0000 7f0cc11ea740 1 Processor -- start 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cc11ea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b19170 0x55c4f8b212d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cc11ea740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b21810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cc11ea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b24350 0x55c4f8b21d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cc11ea740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55c4f8b26f20 con 0x55c4f8b19a90 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cc11ea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c4f8a06ae0 con 0x55c4f8b24350 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cc11ea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c4f8b270a0 con 0x55c4f8b19170 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cc11ea740 10 client.0 _ll_register_callbacks cb 0x55c4f8a3f9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cb77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b21810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cb77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b21810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33710/0 (socket says 172.21.15.104:33710) 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cb77fe640 1 -- 172.21.15.104:0/928435260 learned_addr learned my addr 172.21.15.104:0/928435260 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cbca7a640 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b24350 0x55c4f8b21d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b19170 0x55c4f8b212d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cb77fe640 1 -- 172.21.15.104:0/928435260 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b19170 msgr2=0x55c4f8b212d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cb77fe640 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b19170 0x55c4f8b212d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cb77fe640 1 -- 172.21.15.104:0/928435260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b24350 msgr2=0x55c4f8b21d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cb77fe640 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b24350 0x55c4f8b21d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cb77fe640 1 -- 172.21.15.104:0/928435260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4f8791210 con 0x55c4f8b19a90 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cb77fe640 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b21810 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009230 tx=0x7f0cac031850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba384706a0ca7ae server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0c977fe640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0c977fe640 1 -- 172.21.15.104:0/928435260 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac035040 con 0x55c4f8b19a90 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0c977fe640 1 -- 172.21.15.104:0/928435260 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f0cac004460 con 0x55c4f8b19a90 2024-06-09T22:25:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cc11ea740 1 -- 172.21.15.104:0/928435260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ca80cd180 con 0x55c4f8b19a90 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0c977fe640 1 -- 172.21.15.104:0/928435260 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0cac04f050 con 0x55c4f8b19a90 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cc11ea740 10 client.16701 fetch_fsmap learned FSMap version 31 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.998+0000 7f0cc11ea740 1 -- 172.21.15.104:0/928435260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c4f885d620 con 0x55c4f8b19a90 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.999+0000 7f0c977fe640 1 -- 172.21.15.104:0/928435260 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ca80cd180 con 0x55c4f8b19a90 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.999+0000 7f0c977fe640 1 -- 172.21.15.104:0/928435260 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f0cac004c50 con 0x55c4f8b19a90 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.999+0000 7f0cc11ea740 10 client.16701 fetch_fsmap finished waiting for FSMap version 31 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.999+0000 7f0cc11ea740 10 client.16701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.999+0000 7f0cc11ea740 20 client.16701 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.999+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c4f8b344a0 0x55c4f8b36890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.999+0000 7f0cc11ea740 4 client.16701 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.999+0000 7f0cc11ea740 1 -- 172.21.15.104:0/928435260 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4f8a90f10 con 0x55c4f8b344a0 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.999+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c4f8b344a0 0x55c4f8b36890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.999+0000 7f0c977fe640 10 client.16701 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:22.999+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c4f8b344a0 0x55c4f8b36890 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c405bf989ab7c524 server_cookie=2fb10e0ee360718e in_seq=0 out_seq=0 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0c977fe640 1 -- 172.21.15.104:0/928435260 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55c4f8a90f10 con 0x55c4f8b344a0 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0c977fe640 10 client.16701 handle_command_reply: tid=0 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0cc11ea740 1 client.16701 shutdown 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0cc11ea740 1 -- 172.21.15.104:0/928435260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 msgr2=0x55c4f8b21810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b21810 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009230 tx=0x7f0cac031850 comp rx=0 tx=0).stop 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0cc11ea740 1 -- 172.21.15.104:0/928435260 shutdown_connections 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c4f8b344a0 0x55c4f8b36890 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b24350 0x55c4f8b21d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0c977fe640 0 client.16701 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b21810 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/928435260 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b19170 0x55c4f8b212d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0cc11ea740 1 -- 172.21.15.104:0/928435260 >> 172.21.15.104:0/928435260 conn(0x55c4f8b15680 msgr2=0x55c4f8b1b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0cc11ea740 1 -- 172.21.15.104:0/928435260 shutdown_connections 2024-06-09T22:25:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.000+0000 7f0cc11ea740 1 -- 172.21.15.104:0/928435260 wait complete. 2024-06-09T22:25:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.001+0000 7f0cc11ea740 20 client.16701 trim_cache size 0 max 16384 2024-06-09T22:25:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.015+0000 7f0cc11ea740 1 Processor -- start 2024-06-09T22:25:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.015+0000 7f0cc11ea740 1 -- start start 2024-06-09T22:25:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cc11ea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19170 0x55c4f8b1bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cc11ea740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b3c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cc11ea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 0x55c4f8b3c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cc11ea740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81aee00 con 0x55c4f8b19a90 2024-06-09T22:25:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cc11ea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81b0970 con 0x55c4f8b19170 2024-06-09T22:25:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cc11ea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81af4d0 con 0x55c4f8b24350 2024-06-09T22:25:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cb77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b3c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cb77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b3c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33722/0 (socket says 172.21.15.104:33722) 2024-06-09T22:25:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cb77fe640 1 -- 172.21.15.104:0/3645826109 learned_addr learned my addr 172.21.15.104:0/3645826109 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cbca7a640 1 --2- 172.21.15.104:0/3645826109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 0x55c4f8b3c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cb77fe640 1 -- 172.21.15.104:0/3645826109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 msgr2=0x55c4f8b3c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cb77fe640 1 --2- 172.21.15.104:0/3645826109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 0x55c4f8b3c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cb77fe640 1 -- 172.21.15.104:0/3645826109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19170 msgr2=0x55c4f8b1bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/3645826109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19170 0x55c4f8b1bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cb77fe640 1 --2- 172.21.15.104:0/3645826109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19170 0x55c4f8b1bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cb77fe640 1 -- 172.21.15.104:0/3645826109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4f8b36f10 con 0x55c4f8b19a90 2024-06-09T22:25:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0cb77fe640 1 --2- 172.21.15.104:0/3645826109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b3c160 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f0cac003730 tx=0x7f0cac02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985b815adda75d21 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0c9dffb640 1 -- 172.21.15.104:0/3645826109 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac000e60 con 0x55c4f8b19a90 2024-06-09T22:25:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0c9dffb640 1 -- 172.21.15.104:0/3645826109 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cac041070 con 0x55c4f8b19a90 2024-06-09T22:25:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.016+0000 7f0c9dffb640 1 -- 172.21.15.104:0/3645826109 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac035030 con 0x55c4f8b19a90 2024-06-09T22:25:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.017+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3645826109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 msgr2=0x55c4f8b3c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.017+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/3645826109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b3c160 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f0cac003730 tx=0x7f0cac02a420 comp rx=0 tx=0).stop 2024-06-09T22:25:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.017+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3645826109 shutdown_connections 2024-06-09T22:25:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.017+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/3645826109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 0x55c4f8b3c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.017+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/3645826109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19a90 0x55c4f8b3c160 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.017+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/3645826109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19170 0x55c4f8b1bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.017+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3645826109 >> 172.21.15.104:0/3645826109 conn(0x55c4f8b15680 msgr2=0x55c4f8b15a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.017+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3645826109 shutdown_connections 2024-06-09T22:25:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.017+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3645826109 wait complete. 2024-06-09T22:25:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.017+0000 7f0cc11ea740 1 -- start start 2024-06-09T22:25:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.017+0000 7f0cc11ea740 1 Processor -- start 2024-06-09T22:25:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.017+0000 7f0cc11ea740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19170 0x55c4f8b2c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cc11ea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19a90 0x55c4f8b1dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cc11ea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 0x55c4f8b1e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cc11ea740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81b0270 con 0x55c4f8b19170 2024-06-09T22:25:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cc11ea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81afba0 con 0x55c4f8b19a90 2024-06-09T22:25:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cc11ea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81acdc0 con 0x55c4f8b24350 2024-06-09T22:25:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cc11ea740 10 client.0 _ll_register_callbacks cb 0x7f0cac005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cb7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19170 0x55c4f8b2c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cb7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19170 0x55c4f8b2c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33738/0 (socket says 172.21.15.104:33738) 2024-06-09T22:25:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cb7fff640 1 -- 172.21.15.104:0/3727146943 learned_addr learned my addr 172.21.15.104:0/3727146943 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cb77fe640 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19a90 0x55c4f8b1dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cbca7a640 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 0x55c4f8b1e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cb7fff640 1 -- 172.21.15.104:0/3727146943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 msgr2=0x55c4f8b1e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 0x55c4f8b1e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cb7fff640 1 -- 172.21.15.104:0/3727146943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19a90 msgr2=0x55c4f8b1dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19a90 0x55c4f8b1dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cb7fff640 1 -- 172.21.15.104:0/3727146943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4f8b1e7c0 con 0x55c4f8b19170 2024-06-09T22:25:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cb7fff640 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19170 0x55c4f8b2c7f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80a0240 tx=0x7f0ca8098150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4766c617bb149921 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0c977fe640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0c977fe640 1 -- 172.21.15.104:0/3727146943 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca80cd180 con 0x55c4f8b19170 2024-06-09T22:25:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0c977fe640 1 -- 172.21.15.104:0/3727146943 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f0ca80cd320 con 0x55c4f8b19170 2024-06-09T22:25:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3727146943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cac041070 con 0x55c4f8b19170 2024-06-09T22:25:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0c977fe640 1 -- 172.21.15.104:0/3727146943 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0ca8010430 con 0x55c4f8b19170 2024-06-09T22:25:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cc11ea740 10 client.16707 fetch_fsmap learned FSMap version 31 2024-06-09T22:25:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3727146943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c4f8b9ee50 con 0x55c4f8b19170 2024-06-09T22:25:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.018+0000 7f0c977fe640 1 -- 172.21.15.104:0/3727146943 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cac041070 con 0x55c4f8b19170 2024-06-09T22:25:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.019+0000 7f0c977fe640 1 -- 172.21.15.104:0/3727146943 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f0ca80b3020 con 0x55c4f8b19170 2024-06-09T22:25:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.019+0000 7f0cc11ea740 10 client.16707 fetch_fsmap finished waiting for FSMap version 31 2024-06-09T22:25:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.019+0000 7f0cc11ea740 10 client.16707 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.019+0000 7f0cc11ea740 20 client.16707 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.019+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c4f8b9f150 0x55c4f8ba1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.019+0000 7f0cc11ea740 4 client.16707 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:25:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.019+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3727146943 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c4f8a8ac20 con 0x55c4f8b9f150 2024-06-09T22:25:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.019+0000 7f0cb77fe640 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c4f8b9f150 0x55c4f8ba1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.019+0000 7f0c977fe640 10 client.16707 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.019+0000 7f0cb77fe640 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c4f8b9f150 0x55c4f8ba1540 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=424d48c736962921 server_cookie=1400e5a43dc5e539 in_seq=0 out_seq=0 2024-06-09T22:25:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0c977fe640 1 -- 172.21.15.104:0/3727146943 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55c4f8a8ac20 con 0x55c4f8b9f150 2024-06-09T22:25:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0c977fe640 10 client.16707 handle_command_reply: tid=0 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0cc11ea740 1 client.16707 shutdown 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3727146943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19170 msgr2=0x55c4f8b2c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19170 0x55c4f8b2c7f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80a0240 tx=0x7f0ca8098150 comp rx=0 tx=0).stop 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3727146943 shutdown_connections 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c4f8b9f150 0x55c4f8ba1540 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4f8b24350 0x55c4f8b1e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c4f8b19a90 0x55c4f8b1dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0cc11ea740 1 --2- 172.21.15.104:0/3727146943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c4f8b19170 0x55c4f8b2c7f0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0c977fe640 0 client.16707 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3727146943 >> 172.21.15.104:0/3727146943 conn(0x55c4f8b15680 msgr2=0x55c4f8b22b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3727146943 shutdown_connections 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0cc11ea740 1 -- 172.21.15.104:0/3727146943 wait complete. 2024-06-09T22:25:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.027+0000 7f0cc11ea740 20 client.16707 trim_cache size 0 max 16384 2024-06-09T22:25:23.030 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:25:23.030 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:25:23.030 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:25:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.028+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2046237870 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0ca00a2da0 msgr2=0x7f0ca00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.028+0000 7f0cbe4dd640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0ca00a2da0 0x7f0ca00a5260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0005fd0 tx=0x7f0cb0014bf0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.028+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2046237870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb8101980 msgr2=0x7f0cb8198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.028+0000 7f0cbe4dd640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb8101980 0x7f0cb8198200 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800e9f0 tx=0x7f0ca800eeb0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.028+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2046237870 shutdown_connections 2024-06-09T22:25:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.028+0000 7f0cbe4dd640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0ca00a2da0 0x7f0ca00a5260 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.028+0000 7f0cbe4dd640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0cb8107990 0x7f0cb819cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.028+0000 7f0cbe4dd640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb8102340 0x7f0cb8198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.028+0000 7f0cbe4dd640 1 --2- 172.21.15.104:0/2046237870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb8101980 0x7f0cb8198200 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.028+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2046237870 >> 172.21.15.104:0/2046237870 conn(0x7f0cb80fd450 msgr2=0x7f0cb80fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.028+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2046237870 shutdown_connections 2024-06-09T22:25:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:23.028+0000 7f0cbe4dd640 1 -- 172.21.15.104:0/2046237870 wait complete. 2024-06-09T22:25:23.169 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:22 smithi104 ceph-mon[32389]: pgmap v135: 129 pgs: 129 active+clean; 69 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 1.1 MiB/s wr, 89 op/s 2024-06-09T22:25:23.195 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-09T22:25:23.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:22 smithi153 ceph-mon[36599]: pgmap v135: 129 pgs: 129 active+clean; 69 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 1.1 MiB/s wr, 89 op/s 2024-06-09T22:25:23.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:22 smithi194 ceph-mon[36856]: pgmap v135: 129 pgs: 129 active+clean; 69 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 1.1 MiB/s wr, 89 op/s 2024-06-09T22:25:23.436 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:23.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 92992 3.95 MB/sec execute 29 sec latency 925.915 ms 2024-06-09T22:25:24.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 92992 3.82 MB/sec execute 30 sec latency 1925.945 ms 2024-06-09T22:25:25.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:24 smithi104 ceph-mon[32389]: pgmap v136: 129 pgs: 129 active+clean; 74 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 134 op/s 2024-06-09T22:25:25.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:24 smithi153 ceph-mon[36599]: pgmap v136: 129 pgs: 129 active+clean; 74 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 134 op/s 2024-06-09T22:25:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.303+0000 7fec41f0a700 1 -- 172.21.15.104:0/2366559913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec3c107660 msgr2=0x7fec3c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.303+0000 7fec41f0a700 1 --2- 172.21.15.104:0/2366559913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec3c107660 0x7fec3c107a40 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fec2c009990 tx=0x7fec2c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.303+0000 7fec41f0a700 1 -- 172.21.15.104:0/2366559913 shutdown_connections 2024-06-09T22:25:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.303+0000 7fec41f0a700 1 --2- 172.21.15.104:0/2366559913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec3c107660 0x7fec3c107a40 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.303+0000 7fec41f0a700 1 --2- 172.21.15.104:0/2366559913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec3c1024b0 0x7fec3c10e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.303+0000 7fec41f0a700 1 --2- 172.21.15.104:0/2366559913 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec3c101af0 0x7fec3c101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.303+0000 7fec41f0a700 1 -- 172.21.15.104:0/2366559913 >> 172.21.15.104:0/2366559913 conn(0x7fec3c077990 msgr2=0x7fec3c0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.303+0000 7fec41f0a700 1 -- 172.21.15.104:0/2366559913 shutdown_connections 2024-06-09T22:25:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.303+0000 7fec41f0a700 1 -- 172.21.15.104:0/2366559913 wait complete. 2024-06-09T22:25:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec41f0a700 1 Processor -- start 2024-06-09T22:25:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec41f0a700 1 -- start start 2024-06-09T22:25:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec41f0a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec3c101af0 0x7fec3c198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec41f0a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec3c1024b0 0x7fec3c199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec41f0a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec3c107660 0x7fec3c19d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec41f0a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c111900 con 0x7fec3c107660 2024-06-09T22:25:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec41f0a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c111780 con 0x7fec3c101af0 2024-06-09T22:25:25.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec41f0a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c111a80 con 0x7fec3c1024b0 2024-06-09T22:25:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec3bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec3c107660 0x7fec3c19d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec3bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec3c107660 0x7fec3c19d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33750/0 (socket says 172.21.15.104:33750) 2024-06-09T22:25:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec3bfff700 1 -- 172.21.15.104:0/2099681295 learned_addr learned my addr 172.21.15.104:0/2099681295 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec3c101af0 0x7fec3c198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.304+0000 7fec33fff700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec3c1024b0 0x7fec3c199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.305+0000 7fec3bfff700 1 -- 172.21.15.104:0/2099681295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec3c1024b0 msgr2=0x7fec3c199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.305+0000 7fec3bfff700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec3c1024b0 0x7fec3c199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.305+0000 7fec3bfff700 1 -- 172.21.15.104:0/2099681295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec3c101af0 msgr2=0x7fec3c198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.305+0000 7fec3bfff700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec3c101af0 0x7fec3c198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.305+0000 7fec3bfff700 1 -- 172.21.15.104:0/2099681295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3c19ddd0 con 0x7fec3c107660 2024-06-09T22:25:25.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.305+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec3c101af0 0x7fec3c198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.305+0000 7fec3bfff700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec3c107660 0x7fec3c19d690 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fec3400cca0 tx=0x7fec3400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.305+0000 7fec397fa700 1 -- 172.21.15.104:0/2099681295 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec34016020 con 0x7fec3c107660 2024-06-09T22:25:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.305+0000 7fec397fa700 1 -- 172.21.15.104:0/2099681295 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec34012420 con 0x7fec3c107660 2024-06-09T22:25:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.305+0000 7fec41f0a700 1 -- 172.21.15.104:0/2099681295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec3c19e0c0 con 0x7fec3c107660 2024-06-09T22:25:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.305+0000 7fec397fa700 1 -- 172.21.15.104:0/2099681295 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec340115e0 con 0x7fec3c107660 2024-06-09T22:25:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.305+0000 7fec41f0a700 1 -- 172.21.15.104:0/2099681295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec3c1a5940 con 0x7fec3c107660 2024-06-09T22:25:25.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.307+0000 7fec397fa700 1 -- 172.21.15.104:0/2099681295 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fec34019020 con 0x7fec3c107660 2024-06-09T22:25:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.308+0000 7fec397fa700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fec280a2d90 0x7fec280a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.308+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fec280a2d90 0x7fec280a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.308+0000 7fec397fa700 1 -- 172.21.15.104:0/2099681295 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fec340cf540 con 0x7fec3c107660 2024-06-09T22:25:25.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.308+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fec280a2d90 0x7fec280a5250 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fec2c006010 tx=0x7fec2c005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:25.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:24 smithi194 ceph-mon[36856]: pgmap v136: 129 pgs: 129 active+clean; 74 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 134 op/s 2024-06-09T22:25:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.309+0000 7fec4c78db80 1 Processor -- start 2024-06-09T22:25:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.309+0000 7fec4c78db80 1 -- start start 2024-06-09T22:25:25.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec4c78db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196b9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec4c78db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b9880 0x5624196b3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec4c78db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b37a0 0x5624196b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec4c78db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5624195a5f60 con 0x5624196b8f60 2024-06-09T22:25:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec4c78db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562419604850 con 0x5624196b37a0 2024-06-09T22:25:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec4c78db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56241956c460 con 0x5624196b9880 2024-06-09T22:25:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec3b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196b9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec3b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196b9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33764/0 (socket says 172.21.15.104:33764) 2024-06-09T22:25:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec3b7fe700 1 -- 172.21.15.104:0/1683487838 learned_addr learned my addr 172.21.15.104:0/1683487838 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec33fff700 1 --2- 172.21.15.104:0/1683487838 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b9880 0x5624196b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec3b7fe700 1 -- 172.21.15.104:0/1683487838 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b9880 msgr2=0x5624196b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/1683487838 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b9880 0x5624196b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec3b7fe700 1 -- 172.21.15.104:0/1683487838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b37a0 msgr2=0x5624196b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/1683487838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b37a0 0x5624196b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec3b7fe700 1 -- 172.21.15.104:0/1683487838 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624195538d0 con 0x5624196b8f60 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/1683487838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196b9340 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fec3c06a5f0 tx=0x7fec2c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb69b4b2843040e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec31ffb700 1 -- 172.21.15.104:0/1683487838 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2c0050c0 con 0x5624196b8f60 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec31ffb700 1 -- 172.21.15.104:0/1683487838 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec2c00f460 con 0x5624196b8f60 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.310+0000 7fec31ffb700 1 -- 172.21.15.104:0/1683487838 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2c024670 con 0x5624196b8f60 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.311+0000 7fec4c78db80 1 -- 172.21.15.104:0/1683487838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 msgr2=0x5624196b9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.311+0000 7fec4c78db80 1 --2- 172.21.15.104:0/1683487838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196b9340 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fec3c06a5f0 tx=0x7fec2c020000 comp rx=0 tx=0).stop 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.311+0000 7fec4c78db80 1 -- 172.21.15.104:0/1683487838 shutdown_connections 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.311+0000 7fec4c78db80 1 --2- 172.21.15.104:0/1683487838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196b9340 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.311+0000 7fec4c78db80 1 --2- 172.21.15.104:0/1683487838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b37a0 0x5624196b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.311+0000 7fec4c78db80 1 --2- 172.21.15.104:0/1683487838 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b9880 0x5624196b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.311+0000 7fec4c78db80 1 -- 172.21.15.104:0/1683487838 >> 172.21.15.104:0/1683487838 conn(0x5624195e0080 msgr2=0x5624196ae220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:25.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.311+0000 7fec4c78db80 1 -- 172.21.15.104:0/1683487838 shutdown_connections 2024-06-09T22:25:25.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.311+0000 7fec4c78db80 1 -- 172.21.15.104:0/1683487838 wait complete. 2024-06-09T22:25:25.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec4c78db80 1 -- start start 2024-06-09T22:25:25.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec4c78db80 1 Processor -- start 2024-06-09T22:25:25.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec4c78db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b37a0 0x5624196bf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec4c78db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196bc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec4c78db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b9880 0x5624196bcee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec4c78db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5624195a4fc0 con 0x5624196b8f60 2024-06-09T22:25:25.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec4c78db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562419600810 con 0x5624196b9880 2024-06-09T22:25:25.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec4c78db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5624196c0200 con 0x5624196b37a0 2024-06-09T22:25:25.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec4c78db80 10 client.0 _ll_register_callbacks cb 0x56241959a5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:25.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec33fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196bc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec33fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196bc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33768/0 (socket says 172.21.15.104:33768) 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec33fff700 1 -- 172.21.15.104:0/4205196684 learned_addr learned my addr 172.21.15.104:0/4205196684 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b37a0 0x5624196bf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.312+0000 7fec3bfff700 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b9880 0x5624196bcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec33fff700 1 -- 172.21.15.104:0/4205196684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b37a0 msgr2=0x5624196bf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec33fff700 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b37a0 0x5624196bf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec33fff700 1 -- 172.21.15.104:0/4205196684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b9880 msgr2=0x5624196bcee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec33fff700 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b9880 0x5624196bcee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec33fff700 1 -- 172.21.15.104:0/4205196684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624195ca750 con 0x5624196b8f60 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec3bfff700 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b9880 0x5624196bcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec33fff700 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196bc9a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fec2400bac0 tx=0x7fec2400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc09a1cbc8acc33 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec1affd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:25.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec1affd700 1 -- 172.21.15.104:0/4205196684 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec24004830 con 0x5624196b8f60 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec1affd700 1 -- 172.21.15.104:0/4205196684 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fec240049d0 con 0x5624196b8f60 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec4c78db80 1 -- 172.21.15.104:0/4205196684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec2c00f460 con 0x5624196b8f60 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec1affd700 1 -- 172.21.15.104:0/4205196684 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fec24023050 con 0x5624196b8f60 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec4c78db80 10 client.16719 fetch_fsmap learned FSMap version 31 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec4c78db80 1 -- 172.21.15.104:0/4205196684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5624196caac0 con 0x5624196b8f60 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec1affd700 1 -- 172.21.15.104:0/4205196684 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec2c00f460 con 0x5624196b8f60 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec1affd700 1 -- 172.21.15.104:0/4205196684 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fec24019030 con 0x5624196b8f60 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec4c78db80 10 client.16719 fetch_fsmap finished waiting for FSMap version 31 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec4c78db80 10 client.16719 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.313+0000 7fec4c78db80 20 client.16719 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.314+0000 7fec4c78db80 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5624196cd1a0 0x5624196cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.314+0000 7fec4c78db80 4 client.16719 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.314+0000 7fec4c78db80 1 -- 172.21.15.104:0/4205196684 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562419611d50 con 0x5624196cd1a0 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.314+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5624196cd1a0 0x5624196cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.314+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5624196cd1a0 0x5624196cf5c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64cddb27faa88135 server_cookie=671de5611b06a30e in_seq=0 out_seq=0 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.314+0000 7fec1affd700 10 client.16719 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec1affd700 1 -- 172.21.15.104:0/4205196684 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x562419611d50 con 0x5624196cd1a0 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec1affd700 10 client.16719 handle_command_reply: tid=0 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec4c78db80 1 client.16719 shutdown 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec4c78db80 1 -- 172.21.15.104:0/4205196684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 msgr2=0x5624196bc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec4c78db80 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196bc9a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fec2400bac0 tx=0x7fec2400bdd0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec4c78db80 1 -- 172.21.15.104:0/4205196684 shutdown_connections 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec4c78db80 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5624196cd1a0 0x5624196cf5c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec4c78db80 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b9880 0x5624196bcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec4c78db80 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b37a0 0x5624196bf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec4c78db80 1 --2- 172.21.15.104:0/4205196684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196bc9a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec4c78db80 1 -- 172.21.15.104:0/4205196684 >> 172.21.15.104:0/4205196684 conn(0x5624195e0080 msgr2=0x5624196ad700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec1affd700 0 client.16719 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec4c78db80 1 -- 172.21.15.104:0/4205196684 shutdown_connections 2024-06-09T22:25:25.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec4c78db80 1 -- 172.21.15.104:0/4205196684 wait complete. 2024-06-09T22:25:25.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.315+0000 7fec4c78db80 20 client.16719 trim_cache size 0 max 16384 2024-06-09T22:25:25.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.330+0000 7fec4c78db80 1 Processor -- start 2024-06-09T22:25:25.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.330+0000 7fec4c78db80 1 -- start start 2024-06-09T22:25:25.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.330+0000 7fec4c78db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b37a0 0x5624196c9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.330+0000 7fec4c78db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b8f60 0x5624196bbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.330+0000 7fec4c78db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b9880 0x5624196b8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.330+0000 7fec4c78db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c1ae4b0 con 0x5624196b37a0 2024-06-09T22:25:25.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.330+0000 7fec4c78db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c1b0020 con 0x5624196b8f60 2024-06-09T22:25:25.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.330+0000 7fec4c78db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c1aeb80 con 0x5624196b9880 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.330+0000 7fec3b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b37a0 0x5624196c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.330+0000 7fec3b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b37a0 0x5624196c9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33770/0 (socket says 172.21.15.104:33770) 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.330+0000 7fec3b7fe700 1 -- 172.21.15.104:0/2089985842 learned_addr learned my addr 172.21.15.104:0/2089985842 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.331+0000 7fec33fff700 1 --2- 172.21.15.104:0/2089985842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b8f60 0x5624196bbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.331+0000 7fec3bfff700 1 --2- 172.21.15.104:0/2089985842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b9880 0x5624196b8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.331+0000 7fec3b7fe700 1 -- 172.21.15.104:0/2089985842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b9880 msgr2=0x5624196b8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.331+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/2089985842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b9880 0x5624196b8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.331+0000 7fec3b7fe700 1 -- 172.21.15.104:0/2089985842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b8f60 msgr2=0x5624196bbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.331+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/2089985842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b8f60 0x5624196bbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.331+0000 7fec3b7fe700 1 -- 172.21.15.104:0/2089985842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624196af9c0 con 0x5624196b37a0 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.331+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/2089985842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b37a0 0x5624196c9350 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fec2c009750 tx=0x7fec2c005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104a22ae252d62a8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.331+0000 7fec317fa700 1 -- 172.21.15.104:0/2089985842 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2c005410 con 0x5624196b37a0 2024-06-09T22:25:25.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.331+0000 7fec317fa700 1 -- 172.21.15.104:0/2089985842 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec2c0055b0 con 0x5624196b37a0 2024-06-09T22:25:25.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.331+0000 7fec317fa700 1 -- 172.21.15.104:0/2089985842 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2c02f670 con 0x5624196b37a0 2024-06-09T22:25:25.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.332+0000 7fec4c78db80 1 -- 172.21.15.104:0/2089985842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b37a0 msgr2=0x5624196c9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:25.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.332+0000 7fec4c78db80 1 --2- 172.21.15.104:0/2089985842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b37a0 0x5624196c9350 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fec2c009750 tx=0x7fec2c005040 comp rx=0 tx=0).stop 2024-06-09T22:25:25.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.332+0000 7fec4c78db80 1 -- 172.21.15.104:0/2089985842 shutdown_connections 2024-06-09T22:25:25.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.332+0000 7fec4c78db80 1 --2- 172.21.15.104:0/2089985842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b37a0 0x5624196c9350 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.332+0000 7fec4c78db80 1 --2- 172.21.15.104:0/2089985842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b9880 0x5624196b8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.332+0000 7fec4c78db80 1 --2- 172.21.15.104:0/2089985842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b8f60 0x5624196bbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.332+0000 7fec4c78db80 1 -- 172.21.15.104:0/2089985842 >> 172.21.15.104:0/2089985842 conn(0x5624195e0080 msgr2=0x5624196b7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:25.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.332+0000 7fec4c78db80 1 -- 172.21.15.104:0/2089985842 shutdown_connections 2024-06-09T22:25:25.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.332+0000 7fec4c78db80 1 -- 172.21.15.104:0/2089985842 wait complete. 2024-06-09T22:25:25.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.332+0000 7fec4c78db80 1 -- start start 2024-06-09T22:25:25.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.332+0000 7fec4c78db80 1 Processor -- start 2024-06-09T22:25:25.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec4c78db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b37a0 0x5624196c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec4c78db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec4c78db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b9880 0x5624196d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec4c78db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c1af920 con 0x5624196b8f60 2024-06-09T22:25:25.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec4c78db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c1af250 con 0x5624196b9880 2024-06-09T22:25:25.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec4c78db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c1ac430 con 0x5624196b37a0 2024-06-09T22:25:25.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec4c78db80 10 client.0 _ll_register_callbacks cb 0x56241959a5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec33fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec33fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196c5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33784/0 (socket says 172.21.15.104:33784) 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec33fff700 1 -- 172.21.15.104:0/2134246290 learned_addr learned my addr 172.21.15.104:0/2134246290 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec3bfff700 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b9880 0x5624196d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b37a0 0x5624196c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec33fff700 1 -- 172.21.15.104:0/2134246290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b37a0 msgr2=0x5624196c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec33fff700 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b37a0 0x5624196c54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec33fff700 1 -- 172.21.15.104:0/2134246290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b9880 msgr2=0x5624196d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec33fff700 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b9880 0x5624196d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec33fff700 1 -- 172.21.15.104:0/2134246290 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624196d5930 con 0x5624196b8f60 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec33fff700 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196c5a00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fec24004570 tx=0x7fec24003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48408ca66aa32c57 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec1affd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec1affd700 1 -- 172.21.15.104:0/2134246290 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec24005250 con 0x5624196b8f60 2024-06-09T22:25:25.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec1affd700 1 -- 172.21.15.104:0/2134246290 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fec240053f0 con 0x5624196b8f60 2024-06-09T22:25:25.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.333+0000 7fec4c78db80 1 -- 172.21.15.104:0/2134246290 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec2c0055b0 con 0x5624196b8f60 2024-06-09T22:25:25.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.334+0000 7fec1affd700 1 -- 172.21.15.104:0/2134246290 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fec24038050 con 0x5624196b8f60 2024-06-09T22:25:25.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.334+0000 7fec4c78db80 10 client.16725 fetch_fsmap learned FSMap version 31 2024-06-09T22:25:25.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.334+0000 7fec4c78db80 1 -- 172.21.15.104:0/2134246290 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562419737330 con 0x5624196b8f60 2024-06-09T22:25:25.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.334+0000 7fec1affd700 1 -- 172.21.15.104:0/2134246290 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec2c0055b0 con 0x5624196b8f60 2024-06-09T22:25:25.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.334+0000 7fec1affd700 1 -- 172.21.15.104:0/2134246290 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fec24019040 con 0x5624196b8f60 2024-06-09T22:25:25.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.334+0000 7fec4c78db80 10 client.16725 fetch_fsmap finished waiting for FSMap version 31 2024-06-09T22:25:25.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.334+0000 7fec4c78db80 10 client.16725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:25.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.334+0000 7fec4c78db80 20 client.16725 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:25.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.334+0000 7fec4c78db80 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562419737aa0 0x562419739ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:25.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.334+0000 7fec4c78db80 4 client.16725 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:25:25.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.334+0000 7fec4c78db80 1 -- 172.21.15.104:0/2134246290 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56241960c2d0 con 0x562419737aa0 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.334+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562419737aa0 0x562419739ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec3b7fe700 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562419737aa0 0x562419739ec0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25311de9628189cc server_cookie=96812f5b1c95a73a in_seq=0 out_seq=0 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec1affd700 10 client.16725 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec1affd700 1 -- 172.21.15.104:0/2134246290 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56241960c2d0 con 0x562419737aa0 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec1affd700 10 client.16725 handle_command_reply: tid=0 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec4c78db80 1 client.16725 shutdown 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec4c78db80 1 -- 172.21.15.104:0/2134246290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 msgr2=0x5624196c5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec4c78db80 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196c5a00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fec24004570 tx=0x7fec24003fb0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec4c78db80 1 -- 172.21.15.104:0/2134246290 shutdown_connections 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec4c78db80 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x562419737aa0 0x562419739ec0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec4c78db80 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624196b9880 0x5624196d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec4c78db80 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5624196b37a0 0x5624196c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec4c78db80 1 --2- 172.21.15.104:0/2134246290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5624196b8f60 0x5624196c5a00 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec4c78db80 1 -- 172.21.15.104:0/2134246290 >> 172.21.15.104:0/2134246290 conn(0x5624195e0080 msgr2=0x5624196bb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:25.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec1affd700 0 client.16725 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:25.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec4c78db80 1 -- 172.21.15.104:0/2134246290 shutdown_connections 2024-06-09T22:25:25.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec4c78db80 1 -- 172.21.15.104:0/2134246290 wait complete. 2024-06-09T22:25:25.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.335+0000 7fec4c78db80 20 client.16725 trim_cache size 0 max 16384 2024-06-09T22:25:25.342 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:25:25.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.336+0000 7fec41f0a700 1 -- 172.21.15.104:0/2099681295 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fec280a2d90 msgr2=0x7fec280a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:25.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.336+0000 7fec41f0a700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fec280a2d90 0x7fec280a5250 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fec2c006010 tx=0x7fec2c005960 comp rx=0 tx=0).stop 2024-06-09T22:25:25.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.336+0000 7fec41f0a700 1 -- 172.21.15.104:0/2099681295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec3c107660 msgr2=0x7fec3c19d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:25.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.336+0000 7fec41f0a700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec3c107660 0x7fec3c19d690 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fec3400cca0 tx=0x7fec3400a060 comp rx=0 tx=0).stop 2024-06-09T22:25:25.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.336+0000 7fec41f0a700 1 -- 172.21.15.104:0/2099681295 shutdown_connections 2024-06-09T22:25:25.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.336+0000 7fec41f0a700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fec280a2d90 0x7fec280a5250 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.336+0000 7fec41f0a700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec3c107660 0x7fec3c19d690 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.336+0000 7fec41f0a700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec3c101af0 0x7fec3c198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.336+0000 7fec41f0a700 1 --2- 172.21.15.104:0/2099681295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec3c1024b0 0x7fec3c199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:25.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.336+0000 7fec41f0a700 1 -- 172.21.15.104:0/2099681295 >> 172.21.15.104:0/2099681295 conn(0x7fec3c077990 msgr2=0x7fec3c0ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:25.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.336+0000 7fec41f0a700 1 -- 172.21.15.104:0/2099681295 shutdown_connections 2024-06-09T22:25:25.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:25.336+0000 7fec41f0a700 1 -- 172.21.15.104:0/2099681295 wait complete. 2024-06-09T22:25:25.449 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.2 2024-06-09T22:25:25.449 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:25:25.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 92992 3.70 MB/sec execute 31 sec latency 2926.081 ms 2024-06-09T22:25:26.424 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:26.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 93060 3.59 MB/sec execute 32 sec latency 3312.096 ms 2024-06-09T22:25:27.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:26 smithi153 ceph-mon[36599]: pgmap v137: 129 pgs: 129 active+clean; 74 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 775 KiB/s wr, 87 op/s 2024-06-09T22:25:27.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:26 smithi104 ceph-mon[32389]: pgmap v137: 129 pgs: 129 active+clean; 74 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 775 KiB/s wr, 87 op/s 2024-06-09T22:25:27.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:26 smithi194 ceph-mon[36856]: pgmap v137: 129 pgs: 129 active+clean; 74 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 775 KiB/s wr, 87 op/s 2024-06-09T22:25:27.340 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-09T22:25:27.340 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-09T22:25:27.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 93060 3.48 MB/sec execute 33 sec latency 1553.296 ms 2024-06-09T22:25:28.248 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:28.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.264+0000 7f078a4f7700 1 -- 172.21.15.104:0/3750882680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0784106b90 msgr2=0x7f0784106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:28.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.264+0000 7f078a4f7700 1 --2- 172.21.15.104:0/3750882680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0784106b90 0x7f0784106f70 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f0774009990 tx=0x7f0774009ca0 comp rx=0 tx=0).stop 2024-06-09T22:25:28.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.264+0000 7f078a4f7700 1 -- 172.21.15.104:0/3750882680 shutdown_connections 2024-06-09T22:25:28.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.264+0000 7f078a4f7700 1 --2- 172.21.15.104:0/3750882680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0784106b90 0x7f0784106f70 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:28.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.264+0000 7f078a4f7700 1 --2- 172.21.15.104:0/3750882680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0784101510 0x7f078410d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:28.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.264+0000 7f078a4f7700 1 --2- 172.21.15.104:0/3750882680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0784100b50 0x7f0784100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:28.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.264+0000 7f078a4f7700 1 -- 172.21.15.104:0/3750882680 >> 172.21.15.104:0/3750882680 conn(0x7f0784077a00 msgr2=0x7f07840fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:28.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.264+0000 7f078a4f7700 1 -- 172.21.15.104:0/3750882680 shutdown_connections 2024-06-09T22:25:28.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.264+0000 7f078a4f7700 1 -- 172.21.15.104:0/3750882680 wait complete. 2024-06-09T22:25:28.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f078a4f7700 1 Processor -- start 2024-06-09T22:25:28.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f078a4f7700 1 -- start start 2024-06-09T22:25:28.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f078a4f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0784100b50 0x7f07840fce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:28.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f078a4f7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0784101510 0x7f07840fd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:28.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f078a4f7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0784106b90 0x7f07840fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:28.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f078a4f7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f07841105a0 con 0x7f0784100b50 2024-06-09T22:25:28.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f078a4f7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0784110420 con 0x7f0784106b90 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f078a4f7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0784110720 con 0x7f0784101510 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f0783fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0784100b50 0x7f07840fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f0783fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0784100b50 0x7f07840fce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55296/0 (socket says 172.21.15.104:55296) 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f0788a94700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0784106b90 0x7f07840fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f0783fff700 1 -- 172.21.15.104:0/1391424369 learned_addr learned my addr 172.21.15.104:0/1391424369 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.265+0000 7f07837fe700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0784101510 0x7f07840fd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.266+0000 7f0783fff700 1 -- 172.21.15.104:0/1391424369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0784101510 msgr2=0x7f07840fd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.266+0000 7f0783fff700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0784101510 0x7f07840fd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.266+0000 7f0783fff700 1 -- 172.21.15.104:0/1391424369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0784106b90 msgr2=0x7f07840fd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.266+0000 7f0783fff700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0784106b90 0x7f07840fd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.266+0000 7f0783fff700 1 -- 172.21.15.104:0/1391424369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07841a70b0 con 0x7f0784100b50 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.266+0000 7f0788a94700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0784106b90 0x7f07840fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:28.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.266+0000 7f0783fff700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0784100b50 0x7f07840fce70 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f0774000c00 tx=0x7f07740047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:28.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.266+0000 7f07817fa700 1 -- 172.21.15.104:0/1391424369 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774027070 con 0x7f0784100b50 2024-06-09T22:25:28.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.266+0000 7f078a4f7700 1 -- 172.21.15.104:0/1391424369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07841a7340 con 0x7f0784100b50 2024-06-09T22:25:28.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.266+0000 7f078a4f7700 1 -- 172.21.15.104:0/1391424369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07841a77f0 con 0x7f0784100b50 2024-06-09T22:25:28.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.266+0000 7f07817fa700 1 -- 172.21.15.104:0/1391424369 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0774003e40 con 0x7f0784100b50 2024-06-09T22:25:28.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.266+0000 7f07817fa700 1 -- 172.21.15.104:0/1391424369 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774022d10 con 0x7f0784100b50 2024-06-09T22:25:28.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.268+0000 7f078a4f7700 1 -- 172.21.15.104:0/1391424369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0784069180 con 0x7f0784100b50 2024-06-09T22:25:28.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.269+0000 7f07817fa700 1 -- 172.21.15.104:0/1391424369 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f077402a070 con 0x7f0784100b50 2024-06-09T22:25:28.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.270+0000 7f07817fa700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f07700a2e10 0x7f07700a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:28.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.270+0000 7f07817fa700 1 -- 172.21.15.104:0/1391424369 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f07740d0500 con 0x7f0784100b50 2024-06-09T22:25:28.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.270+0000 7f07837fe700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f07700a2e10 0x7f07700a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:28.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.271+0000 7f07837fe700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f07700a2e10 0x7f07700a52d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0784105340 tx=0x7f076c00a380 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:28.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.275+0000 7f07817fa700 1 -- 172.21.15.104:0/1391424369 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f077405e850 con 0x7f0784100b50 2024-06-09T22:25:28.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.525+0000 7f078a4f7700 1 -- 172.21.15.104:0/1391424369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f078402dca0 con 0x7f0784100b50 2024-06-09T22:25:28.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.526+0000 7f07817fa700 1 -- 172.21.15.104:0/1391424369 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11954 (secure 0 0 0) 0x7f0774013070 con 0x7f0784100b50 2024-06-09T22:25:28.527 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:25:28.527 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:19.695382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24448,24460,24442,24419,24403]},"id":1}]} 2024-06-09T22:25:28.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.528+0000 7f078a4f7700 1 -- 172.21.15.104:0/1391424369 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f07700a2e10 msgr2=0x7f07700a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:28.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.528+0000 7f078a4f7700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f07700a2e10 0x7f07700a52d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0784105340 tx=0x7f076c00a380 comp rx=0 tx=0).stop 2024-06-09T22:25:28.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.528+0000 7f078a4f7700 1 -- 172.21.15.104:0/1391424369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0784100b50 msgr2=0x7f07840fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:28.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.528+0000 7f078a4f7700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0784100b50 0x7f07840fce70 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f0774000c00 tx=0x7f07740047b0 comp rx=0 tx=0).stop 2024-06-09T22:25:28.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.529+0000 7f078a4f7700 1 -- 172.21.15.104:0/1391424369 shutdown_connections 2024-06-09T22:25:28.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.529+0000 7f078a4f7700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f07700a2e10 0x7f07700a52d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:28.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.529+0000 7f078a4f7700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0784100b50 0x7f07840fce70 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:28.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.529+0000 7f078a4f7700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0784106b90 0x7f07840fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:28.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.529+0000 7f078a4f7700 1 --2- 172.21.15.104:0/1391424369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0784101510 0x7f07840fd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:28.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.529+0000 7f078a4f7700 1 -- 172.21.15.104:0/1391424369 >> 172.21.15.104:0/1391424369 conn(0x7f0784077a00 msgr2=0x7f07840ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:28.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.529+0000 7f078a4f7700 1 -- 172.21.15.104:0/1391424369 shutdown_connections 2024-06-09T22:25:28.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:28.529+0000 7f078a4f7700 1 -- 172.21.15.104:0/1391424369 wait complete. 2024-06-09T22:25:28.531 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 31 2024-06-09T22:25:28.726 INFO:tasks.workunit.client.0.smithi104.stdout: 1 93060 3.38 MB/sec execute 34 sec latency 2553.328 ms 2024-06-09T22:25:29.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:28 smithi153 ceph-mon[36599]: pgmap v138: 129 pgs: 129 active+clean; 74 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 775 KiB/s wr, 87 op/s 2024-06-09T22:25:29.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:28 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1391424369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:25:29.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:28 smithi104 ceph-mon[32389]: pgmap v138: 129 pgs: 129 active+clean; 74 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 775 KiB/s wr, 87 op/s 2024-06-09T22:25:29.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:28 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1391424369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:25:29.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:28 smithi194 ceph-mon[36856]: pgmap v138: 129 pgs: 129 active+clean; 74 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 775 KiB/s wr, 87 op/s 2024-06-09T22:25:29.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:28 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1391424369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:25:29.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 93060 3.28 MB/sec execute 35 sec latency 3553.365 ms 2024-06-09T22:25:29.728 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24460 quiesce db / --timeout 90 --expiration 138.2 --await-for 45.0' 2024-06-09T22:25:29.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24460 quiesce db / --timeout 90 --expiration 138.2 --await-for 45.0 2024-06-09T22:25:30.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 94846 3.40 MB/sec execute 36 sec latency 3854.664 ms 2024-06-09T22:25:30.750 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:30.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.754+0000 7f4b72cd8700 1 -- 172.21.15.104:0/2642004964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c108520 msgr2=0x7f4b6c108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:30.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.754+0000 7f4b72cd8700 1 --2- 172.21.15.104:0/2642004964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c108520 0x7f4b6c108900 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009990 tx=0x7f4b60009ca0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.754+0000 7f4b72cd8700 1 -- 172.21.15.104:0/2642004964 shutdown_connections 2024-06-09T22:25:30.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.754+0000 7f4b72cd8700 1 --2- 172.21.15.104:0/2642004964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c108520 0x7f4b6c108900 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.754+0000 7f4b72cd8700 1 --2- 172.21.15.104:0/2642004964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b6c102ed0 0x7f4b6c10f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.754+0000 7f4b72cd8700 1 --2- 172.21.15.104:0/2642004964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b6c102510 0x7f4b6c102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.754+0000 7f4b72cd8700 1 -- 172.21.15.104:0/2642004964 >> 172.21.15.104:0/2642004964 conn(0x7f4b6c079560 msgr2=0x7f4b6c079970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:30.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.754+0000 7f4b72cd8700 1 -- 172.21.15.104:0/2642004964 shutdown_connections 2024-06-09T22:25:30.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.754+0000 7f4b72cd8700 1 -- 172.21.15.104:0/2642004964 wait complete. 2024-06-09T22:25:30.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.755+0000 7f4b72cd8700 1 Processor -- start 2024-06-09T22:25:30.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.755+0000 7f4b72cd8700 1 -- start start 2024-06-09T22:25:30.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b72cd8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b6c102510 0x7f4b6c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b72cd8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c102ed0 0x7f4b6c074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b72cd8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b6c108520 0x7f4b6c075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b72cd8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c112800 con 0x7f4b6c102ed0 2024-06-09T22:25:30.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b72cd8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c112680 con 0x7f4b6c108520 2024-06-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b72cd8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c112980 con 0x7f4b6c102510 2024-06-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b70a74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b6c102510 0x7f4b6c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b70a74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b6c102510 0x7f4b6c074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:45652/0 (socket says 172.21.15.104:45652) 2024-06-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b70a74700 1 -- 172.21.15.104:0/365387762 learned_addr learned my addr 172.21.15.104:0/365387762 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b6bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c102ed0 0x7f4b6c074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b71275700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b6c108520 0x7f4b6c075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b70a74700 1 -- 172.21.15.104:0/365387762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b6c108520 msgr2=0x7f4b6c075210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b70a74700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b6c108520 0x7f4b6c075210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b70a74700 1 -- 172.21.15.104:0/365387762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c102ed0 msgr2=0x7f4b6c074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b70a74700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c102ed0 0x7f4b6c074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b70a74700 1 -- 172.21.15.104:0/365387762 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c1a7100 con 0x7f4b6c102510 2024-06-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.756+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c102ed0 0x7f4b6c074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.757+0000 7f4b70a74700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b6c102510 0x7f4b6c074790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4b60005910 tx=0x7f4b6000bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.757+0000 7f4b69ffb700 1 -- 172.21.15.104:0/365387762 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60027070 con 0x7f4b6c102510 2024-06-09T22:25:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.757+0000 7f4b69ffb700 1 -- 172.21.15.104:0/365387762 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b60004650 con 0x7f4b6c102510 2024-06-09T22:25:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.757+0000 7f4b69ffb700 1 -- 172.21.15.104:0/365387762 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6000f460 con 0x7f4b6c102510 2024-06-09T22:25:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.757+0000 7f4b72cd8700 1 -- 172.21.15.104:0/365387762 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6c1a7390 con 0x7f4b6c102510 2024-06-09T22:25:30.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.757+0000 7f4b72cd8700 1 -- 172.21.15.104:0/365387762 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b6c1a7840 con 0x7f4b6c102510 2024-06-09T22:25:30.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.760+0000 7f4b69ffb700 1 -- 172.21.15.104:0/365387762 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f4b6002a070 con 0x7f4b6c102510 2024-06-09T22:25:30.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.761+0000 7f4b69ffb700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4b5c0a2e10 0x7f4b5c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b69ffb700 1 -- 172.21.15.104:0/365387762 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f4b600d0610 con 0x7f4b6c102510 2024-06-09T22:25:30.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b7d55bb80 1 Processor -- start 2024-06-09T22:25:30.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4b5c0a2e10 0x7f4b5c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b7d55bb80 1 -- start start 2024-06-09T22:25:30.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b7d55bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf8fe0 0x5644dabf93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b7d55bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf9900 0x5644dabf32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b7d55bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf3820 0x5644dabf3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b7d55bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5644daae5fe0 con 0x5644dabf3820 2024-06-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b7d55bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5644dab448d0 con 0x5644dabf9900 2024-06-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b7d55bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5644daaac4e0 con 0x5644dabf8fe0 2024-06-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b70a74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf8fe0 0x5644dabf93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b70a74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf8fe0 0x5644dabf93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:45662/0 (socket says 172.21.15.104:45662) 2024-06-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b70a74700 1 -- 172.21.15.104:0/1140437930 learned_addr learned my addr 172.21.15.104:0/1140437930 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/1140437930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf9900 0x5644dabf32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b71275700 1 --2- 172.21.15.104:0/1140437930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf3820 0x5644dabf3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.762+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4b5c0a2e10 0x7f4b5c0a52d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c079b60 tx=0x7f4b58009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.763+0000 7f4b70a74700 1 -- 172.21.15.104:0/1140437930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf9900 msgr2=0x5644dabf32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.763+0000 7f4b70a74700 1 --2- 172.21.15.104:0/1140437930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf9900 0x5644dabf32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.763+0000 7f4b70a74700 1 -- 172.21.15.104:0/1140437930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf3820 msgr2=0x5644dabf3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.763+0000 7f4b70a74700 1 --2- 172.21.15.104:0/1140437930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf3820 0x5644dabf3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.763+0000 7f4b70a74700 1 -- 172.21.15.104:0/1140437930 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644daa93950 con 0x5644dabf8fe0 2024-06-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.763+0000 7f4b71275700 1 --2- 172.21.15.104:0/1140437930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf3820 0x5644dabf3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.763+0000 7f4b70a74700 1 --2- 172.21.15.104:0/1140437930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf8fe0 0x5644dabf93c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4b600a4ec0 tx=0x7f4b6009c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1af9b563140074f0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.763+0000 7f4b56ffd700 1 -- 172.21.15.104:0/1140437930 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600043d0 con 0x5644dabf8fe0 2024-06-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.763+0000 7f4b56ffd700 1 -- 172.21.15.104:0/1140437930 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b6009cf10 con 0x5644dabf8fe0 2024-06-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.763+0000 7f4b56ffd700 1 -- 172.21.15.104:0/1140437930 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600d0220 con 0x5644dabf8fe0 2024-06-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.764+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/1140437930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf8fe0 msgr2=0x5644dabf93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.764+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/1140437930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf8fe0 0x5644dabf93c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4b600a4ec0 tx=0x7f4b6009c710 comp rx=0 tx=0).stop 2024-06-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.764+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/1140437930 shutdown_connections 2024-06-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.764+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/1140437930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf3820 0x5644dabf3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.764+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/1140437930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf8fe0 0x5644dabf93c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.764+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/1140437930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf9900 0x5644dabf32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.764+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/1140437930 >> 172.21.15.104:0/1140437930 conn(0x5644dab20100 msgr2=0x5644dabee2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.764+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/1140437930 shutdown_connections 2024-06-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.764+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/1140437930 wait complete. 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.765+0000 7f4b7d55bb80 1 -- start start 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.765+0000 7f4b7d55bb80 1 Processor -- start 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b7d55bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf3820 0x5644dabff6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b7d55bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf8fe0 0x5644dabfc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b70a74700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf3820 0x5644dabff6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b70a74700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf3820 0x5644dabff6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:35406/0 (socket says 172.21.15.104:35406) 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b7d55bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf9900 0x5644dabfcf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b6bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf8fe0 0x5644dabfc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3834292183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5644daae5040 con 0x5644dabf8fe0 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b70a74700 1 -- 172.21.15.104:0/3834292183 learned_addr learned my addr 172.21.15.104:0/3834292183 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3834292183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5644dab40890 con 0x5644dabf3820 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3834292183 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5644dac00090 con 0x5644dabf9900 2024-06-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b7d55bb80 10 client.0 _ll_register_callbacks cb 0x5644daada630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b70a74700 1 -- 172.21.15.104:0/3834292183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf9900 msgr2=0x5644dabfcf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b71275700 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf9900 0x5644dabfcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.766+0000 7f4b70a74700 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf9900 0x5644dabfcf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.767+0000 7f4b70a74700 1 -- 172.21.15.104:0/3834292183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf8fe0 msgr2=0x5644dabfc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.767+0000 7f4b70a74700 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf8fe0 0x5644dabfc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.767+0000 7f4b70a74700 1 -- 172.21.15.104:0/3834292183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644dab0a7d0 con 0x5644dabf3820 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.767+0000 7f4b71275700 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf9900 0x5644dabfcf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.767+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf8fe0 0x5644dabfc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.767+0000 7f4b70a74700 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf3820 0x5644dabff6c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4b6009c7a0 tx=0x7f4b6009ad20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfc98232cde331c8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.767+0000 7f4b4bfff700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.767+0000 7f4b4bfff700 1 -- 172.21.15.104:0/3834292183 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600ba020 con 0x5644dabf3820 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.768+0000 7f4b4bfff700 1 -- 172.21.15.104:0/3834292183 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f4b6009cc70 con 0x5644dabf3820 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.768+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3834292183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b6009cf10 con 0x5644dabf3820 2024-06-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.768+0000 7f4b4bfff700 1 -- 172.21.15.104:0/3834292183 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4b600043d0 con 0x5644dabf3820 2024-06-09T22:25:30.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.768+0000 7f4b7d55bb80 10 client.26090 fetch_fsmap learned FSMap version 31 2024-06-09T22:25:30.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.768+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3834292183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5644dac0a7c0 con 0x5644dabf3820 2024-06-09T22:25:30.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.768+0000 7f4b4bfff700 1 -- 172.21.15.104:0/3834292183 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b6009cf10 con 0x5644dabf3820 2024-06-09T22:25:30.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.769+0000 7f4b4bfff700 1 -- 172.21.15.104:0/3834292183 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f4b6009db40 con 0x5644dabf3820 2024-06-09T22:25:30.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.769+0000 7f4b7d55bb80 10 client.26090 fetch_fsmap finished waiting for FSMap version 31 2024-06-09T22:25:30.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.769+0000 7f4b7d55bb80 10 client.26090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.769+0000 7f4b7d55bb80 20 client.26090 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.769+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5644dac0cac0 0x5644dac0eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.769+0000 7f4b7d55bb80 4 client.26090 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.769+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3834292183 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644dab51dd0 con 0x5644dac0cac0 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.769+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5644dac0cac0 0x5644dac0eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.769+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5644dac0cac0 0x5644dac0eeb0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cee12b2b863121c5 server_cookie=1d9b008fe01839a5 in_seq=0 out_seq=0 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.769+0000 7f4b4bfff700 10 client.26090 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.770+0000 7f4b4bfff700 1 -- 172.21.15.104:0/3834292183 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5644dab51dd0 con 0x5644dac0cac0 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.770+0000 7f4b4bfff700 10 client.26090 handle_command_reply: tid=0 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.770+0000 7f4b7d55bb80 1 client.26090 shutdown 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.770+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3834292183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf3820 msgr2=0x5644dabff6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.770+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf3820 0x5644dabff6c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4b6009c7a0 tx=0x7f4b6009ad20 comp rx=0 tx=0).stop 2024-06-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.770+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3834292183 shutdown_connections 2024-06-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.770+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5644dac0cac0 0x5644dac0eeb0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.770+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf9900 0x5644dabfcf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.770+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf3820 0x5644dabff6c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.770+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/3834292183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf8fe0 0x5644dabfc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.770+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3834292183 >> 172.21.15.104:0/3834292183 conn(0x5644dab20100 msgr2=0x5644dabed780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.770+0000 7f4b4bfff700 0 client.26090 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.771+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3834292183 shutdown_connections 2024-06-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.771+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3834292183 wait complete. 2024-06-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.771+0000 7f4b7d55bb80 20 client.26090 trim_cache size 0 max 16384 2024-06-09T22:25:30.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.797+0000 7f4b7d55bb80 1 Processor -- start 2024-06-09T22:25:30.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.797+0000 7f4b7d55bb80 1 -- start start 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b7d55bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 0x5644dabedf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b7d55bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf8fe0 0x5644dabee440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b7d55bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf9900 0x5644dabf81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b7d55bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1b05d0 con 0x5644dabf8fe0 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b7d55bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1b2140 con 0x5644dabf9900 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b7d55bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1b0ca0 con 0x5644dabf3820 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b6bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf8fe0 0x5644dabee440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b70a74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 0x5644dabedf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b6bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf8fe0 0x5644dabee440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55348/0 (socket says 172.21.15.104:55348) 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b6bfff700 1 -- 172.21.15.104:0/3814955376 learned_addr learned my addr 172.21.15.104:0/3814955376 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b71275700 1 --2- 172.21.15.104:0/3814955376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf9900 0x5644dabf81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b70a74700 1 -- 172.21.15.104:0/3814955376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf9900 msgr2=0x5644dabf81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b70a74700 1 --2- 172.21.15.104:0/3814955376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf9900 0x5644dabf81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b70a74700 1 -- 172.21.15.104:0/3814955376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf8fe0 msgr2=0x5644dabee440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b70a74700 1 --2- 172.21.15.104:0/3814955376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf8fe0 0x5644dabee440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b70a74700 1 -- 172.21.15.104:0/3814955376 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644dabf3620 con 0x5644dabf3820 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b70a74700 1 --2- 172.21.15.104:0/3814955376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 0x5644dabedf00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b6009df30 tx=0x7f4b6009c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ab123999c6dde9d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.798+0000 7f4b567fc700 1 -- 172.21.15.104:0/3814955376 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600d0430 con 0x5644dabf3820 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.799+0000 7f4b567fc700 1 -- 172.21.15.104:0/3814955376 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b6009cf10 con 0x5644dabf3820 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.799+0000 7f4b567fc700 1 -- 172.21.15.104:0/3814955376 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600a4f60 con 0x5644dabf3820 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.799+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3814955376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 msgr2=0x5644dabedf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.799+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/3814955376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 0x5644dabedf00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b6009df30 tx=0x7f4b6009c710 comp rx=0 tx=0).stop 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.799+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3814955376 shutdown_connections 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.799+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/3814955376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf9900 0x5644dabf81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.799+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/3814955376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 0x5644dabedf00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.799+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/3814955376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf8fe0 0x5644dabee440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.799+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3814955376 >> 172.21.15.104:0/3814955376 conn(0x5644dab20100 msgr2=0x5644dabf7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:30.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.799+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3814955376 shutdown_connections 2024-06-09T22:25:30.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.799+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/3814955376 wait complete. 2024-06-09T22:25:30.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b7d55bb80 1 -- start start 2024-06-09T22:25:30.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b7d55bb80 1 Processor -- start 2024-06-09T22:25:30.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b7d55bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 0x5644dac05820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b7d55bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf8fe0 0x5644dac022b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b7d55bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf9900 0x5644dac027f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b7d55bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1b1a40 con 0x5644dabf9900 2024-06-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b7d55bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1b1370 con 0x5644dabf8fe0 2024-06-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b7d55bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1ae550 con 0x5644dabf3820 2024-06-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b7d55bb80 10 client.0 _ll_register_callbacks cb 0x5644daada630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b70a74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 0x5644dac05820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b6bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf8fe0 0x5644dac022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b70a74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 0x5644dac05820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:45694/0 (socket says 172.21.15.104:45694) 2024-06-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b70a74700 1 -- 172.21.15.104:0/2987481216 learned_addr learned my addr 172.21.15.104:0/2987481216 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.800+0000 7f4b71275700 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf9900 0x5644dac027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b6bfff700 1 -- 172.21.15.104:0/2987481216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 msgr2=0x5644dac05820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 0x5644dac05820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b6bfff700 1 -- 172.21.15.104:0/2987481216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf9900 msgr2=0x5644dac027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf9900 0x5644dac027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b6bfff700 1 -- 172.21.15.104:0/2987481216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644dac02db0 con 0x5644dabf8fe0 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b70a74700 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 0x5644dac05820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf8fe0 0x5644dac022b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x5644dabee980 tx=0x7f4b580047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d26e0a0ba2f63229 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b4bfff700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b4bfff700 1 -- 172.21.15.104:0/2987481216 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58038060 con 0x5644dabf8fe0 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/2987481216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b6009cf10 con 0x5644dabf8fe0 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b4bfff700 1 -- 172.21.15.104:0/2987481216 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f4b5802b6b0 con 0x5644dabf8fe0 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b4bfff700 1 -- 172.21.15.104:0/2987481216 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4b5803f050 con 0x5644dabf8fe0 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b7d55bb80 10 client.26096 fetch_fsmap learned FSMap version 31 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.801+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/2987481216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5644dac785e0 con 0x5644dabf8fe0 2024-06-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.802+0000 7f4b4bfff700 1 -- 172.21.15.104:0/2987481216 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b6009cf10 con 0x5644dabf8fe0 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.802+0000 7f4b4bfff700 1 -- 172.21.15.104:0/2987481216 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f4b5801fc70 con 0x5644dabf8fe0 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.802+0000 7f4b7d55bb80 10 client.26096 fetch_fsmap finished waiting for FSMap version 31 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.802+0000 7f4b7d55bb80 10 client.26096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.802+0000 7f4b7d55bb80 20 client.26096 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.802+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5644dac78d00 0x5644dac7b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.802+0000 7f4b7d55bb80 4 client.26096 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.802+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/2987481216 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5644dab4c350 con 0x5644dac78d00 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.802+0000 7f4b70a74700 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5644dac78d00 0x5644dac7b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.802+0000 7f4b4bfff700 10 client.26096 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.802+0000 7f4b70a74700 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5644dac78d00 0x5644dac7b120 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f74ae42ace6bc53a server_cookie=2a6ecc7cf6abc1f5 in_seq=0 out_seq=0 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.803+0000 7f4b4bfff700 1 -- 172.21.15.104:0/2987481216 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5644dab4c350 con 0x5644dac78d00 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.803+0000 7f4b4bfff700 10 client.26096 handle_command_reply: tid=0 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.803+0000 7f4b7d55bb80 1 client.26096 shutdown 2024-06-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.804+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/2987481216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf8fe0 msgr2=0x5644dac022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.804+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf8fe0 0x5644dac022b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x5644dabee980 tx=0x7f4b580047c0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.804+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/2987481216 shutdown_connections 2024-06-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.804+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5644dac78d00 0x5644dac7b120 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.804+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5644dabf9900 0x5644dac027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.804+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5644dabf3820 0x5644dac05820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.804+0000 7f4b4bfff700 0 client.26096 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.804+0000 7f4b7d55bb80 1 --2- 172.21.15.104:0/2987481216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644dabf8fe0 0x5644dac022b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.804+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/2987481216 >> 172.21.15.104:0/2987481216 conn(0x5644dab20100 msgr2=0x5644dabf8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.804+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/2987481216 shutdown_connections 2024-06-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.804+0000 7f4b7d55bb80 1 -- 172.21.15.104:0/2987481216 wait complete. 2024-06-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.804+0000 7f4b7d55bb80 20 client.26096 trim_cache size 0 max 16384 2024-06-09T22:25:30.810 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:25:30.810 INFO:teuthology.orchestra.run.smithi104.stdout: "return_code": 0, 2024-06-09T22:25:30.810 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_tag": "2ca81206-bae8-4b07-8e88-a9a09aee077e", 2024-06-09T22:25:30.810 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": "asynchronous" 2024-06-09T22:25:30.810 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:25:30.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.805+0000 7f4b72cd8700 1 -- 172.21.15.104:0/365387762 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4b5c0a2e10 msgr2=0x7f4b5c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.805+0000 7f4b72cd8700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4b5c0a2e10 0x7f4b5c0a52d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c079b60 tx=0x7f4b58009260 comp rx=0 tx=0).stop 2024-06-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.805+0000 7f4b72cd8700 1 -- 172.21.15.104:0/365387762 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b6c102510 msgr2=0x7f4b6c074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.805+0000 7f4b72cd8700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b6c102510 0x7f4b6c074790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4b60005910 tx=0x7f4b6000bab0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.805+0000 7f4b72cd8700 1 -- 172.21.15.104:0/365387762 shutdown_connections 2024-06-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.805+0000 7f4b72cd8700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4b5c0a2e10 0x7f4b5c0a52d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.805+0000 7f4b72cd8700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b6c108520 0x7f4b6c075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.805+0000 7f4b72cd8700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b6c102510 0x7f4b6c074790 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.805+0000 7f4b72cd8700 1 --2- 172.21.15.104:0/365387762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c102ed0 0x7f4b6c074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.805+0000 7f4b72cd8700 1 -- 172.21.15.104:0/365387762 >> 172.21.15.104:0/365387762 conn(0x7f4b6c079560 msgr2=0x7f4b6c101040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.805+0000 7f4b72cd8700 1 -- 172.21.15.104:0/365387762 shutdown_connections 2024-06-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:30.805+0000 7f4b72cd8700 1 -- 172.21.15.104:0/365387762 wait complete. 2024-06-09T22:25:31.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:30 smithi104 ceph-mon[32389]: pgmap v139: 129 pgs: 129 active+clean; 71 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 811 KiB/s wr, 99 op/s 2024-06-09T22:25:31.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:30 smithi153 ceph-mon[36599]: pgmap v139: 129 pgs: 129 active+clean; 71 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 811 KiB/s wr, 99 op/s 2024-06-09T22:25:31.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:30 smithi194 ceph-mon[36856]: pgmap v139: 129 pgs: 129 active+clean; 71 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 811 KiB/s wr, 99 op/s 2024-06-09T22:25:31.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96527 3.61 MB/sec execute 37 sec latency 24.207 ms 2024-06-09T22:25:31.904 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:25:32.122 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:31 smithi104 ceph-mon[32389]: scrub queued for path: / 2024-06-09T22:25:32.122 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:31 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.123 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:31 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:25:32.123 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:31 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:25:32.123 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:31 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:25:32.123 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:31 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.123 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:31 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:25:32.123 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:31 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.123 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:31 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.123 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:31 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:31 smithi153 ceph-mon[36599]: scrub queued for path: / 2024-06-09T22:25:32.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:31 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:31 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:25:32.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:31 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:25:32.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:31 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:25:32.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:31 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:31 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:25:32.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:31 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:31 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:31 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:31 smithi194 ceph-mon[36856]: scrub queued for path: / 2024-06-09T22:25:32.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:31 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:31 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:25:32.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:31 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:25:32.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:31 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:25:32.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:31 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:31 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:25:32.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:31 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:31 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:31 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:25:32.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96539 3.52 MB/sec execute 38 sec latency 988.542 ms 2024-06-09T22:25:33.169 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:32 smithi104 ceph-mon[32389]: pgmap v140: 129 pgs: 129 active+clean; 71 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 541 KiB/s wr, 57 op/s 2024-06-09T22:25:33.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:32 smithi153 ceph-mon[36599]: pgmap v140: 129 pgs: 129 active+clean; 71 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 541 KiB/s wr, 57 op/s 2024-06-09T22:25:33.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:32 smithi194 ceph-mon[36856]: pgmap v140: 129 pgs: 129 active+clean; 71 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 541 KiB/s wr, 57 op/s 2024-06-09T22:25:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.437+0000 7f44d2fbd700 1 -- 172.21.15.104:0/1677960985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44cc101ec0 msgr2=0x7f44cc1022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.437+0000 7f44d2fbd700 1 --2- 172.21.15.104:0/1677960985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44cc101ec0 0x7f44cc1022a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009990 tx=0x7f44c8009ca0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.437+0000 7f44d2fbd700 1 -- 172.21.15.104:0/1677960985 shutdown_connections 2024-06-09T22:25:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.437+0000 7f44d2fbd700 1 --2- 172.21.15.104:0/1677960985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44cc101ec0 0x7f44cc1022a0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.437+0000 7f44d2fbd700 1 --2- 172.21.15.104:0/1677960985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc10ef40 0x7f44cc111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.437+0000 7f44d2fbd700 1 --2- 172.21.15.104:0/1677960985 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44cc1027e0 0x7f44cc10ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.437+0000 7f44d2fbd700 1 -- 172.21.15.104:0/1677960985 >> 172.21.15.104:0/1677960985 conn(0x7f44cc078330 msgr2=0x7f44cc078740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.437+0000 7f44d2fbd700 1 -- 172.21.15.104:0/1677960985 shutdown_connections 2024-06-09T22:25:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.437+0000 7f44d2fbd700 1 -- 172.21.15.104:0/1677960985 wait complete. 2024-06-09T22:25:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.438+0000 7f44d2fbd700 1 Processor -- start 2024-06-09T22:25:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.438+0000 7f44d2fbd700 1 -- start start 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.438+0000 7f44d2fbd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc101ec0 0x7f44cc1a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.438+0000 7f44d2fbd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44cc1027e0 0x7f44cc1a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.438+0000 7f44d2fbd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44cc10ef40 0x7f44cc19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.438+0000 7f44d2fbd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc1149a0 con 0x7f44cc10ef40 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.438+0000 7f44d2fbd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc114820 con 0x7f44cc101ec0 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.438+0000 7f44d2fbd700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc114b20 con 0x7f44cc1027e0 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44d155a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44cc10ef40 0x7f44cc19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44d0d59700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc101ec0 0x7f44cc1a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44d155a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44cc10ef40 0x7f44cc19ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55376/0 (socket says 172.21.15.104:55376) 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44d0d59700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc101ec0 0x7f44cc1a0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:35450/0 (socket says 172.21.15.104:35450) 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44d155a700 1 -- 172.21.15.104:0/312602963 learned_addr learned my addr 172.21.15.104:0/312602963 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44c3fff700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44cc1027e0 0x7f44cc1a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44d155a700 1 -- 172.21.15.104:0/312602963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44cc1027e0 msgr2=0x7f44cc1a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:33.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44d155a700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44cc1027e0 0x7f44cc1a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44d155a700 1 -- 172.21.15.104:0/312602963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc101ec0 msgr2=0x7f44cc1a0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:33.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44d155a700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc101ec0 0x7f44cc1a0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44d155a700 1 -- 172.21.15.104:0/312602963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44cc19b460 con 0x7f44cc10ef40 2024-06-09T22:25:33.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44d155a700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44cc10ef40 0x7f44cc19ab80 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f44c400ada0 tx=0x7f44c400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:33.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44c1ffb700 1 -- 172.21.15.104:0/312602963 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c4017020 con 0x7f44cc10ef40 2024-06-09T22:25:33.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44c1ffb700 1 -- 172.21.15.104:0/312602963 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44c400cdb0 con 0x7f44cc10ef40 2024-06-09T22:25:33.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44d2fbd700 1 -- 172.21.15.104:0/312602963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44cc19b750 con 0x7f44cc10ef40 2024-06-09T22:25:33.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.439+0000 7f44c1ffb700 1 -- 172.21.15.104:0/312602963 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c40053a0 con 0x7f44cc10ef40 2024-06-09T22:25:33.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.440+0000 7f44d2fbd700 1 -- 172.21.15.104:0/312602963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44cc1a79e0 con 0x7f44cc10ef40 2024-06-09T22:25:33.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.441+0000 7f44c1ffb700 1 -- 172.21.15.104:0/312602963 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f44c400c8d0 con 0x7f44cc10ef40 2024-06-09T22:25:33.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.442+0000 7f44c1ffb700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44b40a2e30 0x7f44b40a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.442+0000 7f44d0d59700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44b40a2e30 0x7f44b40a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.443+0000 7f44c1ffb700 1 -- 172.21.15.104:0/312602963 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f44c40cee40 con 0x7f44cc10ef40 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.443+0000 7f44d0d59700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44b40a2e30 0x7f44b40a52f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f44c80098e0 tx=0x7f44c8005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.443+0000 7f44dd840b80 1 Processor -- start 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.443+0000 7f44dd840b80 1 -- start start 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44dd840b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55928995fcd0 0x5592899600b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44dd840b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5592899605f0 0x559289959f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44dd840b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995a460 0x55928995a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44dd840b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5592898ab9a0 con 0x5592899605f0 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44dd840b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5592898538b0 con 0x55928995fcd0 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44dd840b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5592898a7960 con 0x55928995a460 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5592899605f0 0x559289959f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5592899605f0 0x559289959f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55386/0 (socket says 172.21.15.104:55386) 2024-06-09T22:25:33.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44c3fff700 1 -- 172.21.15.104:0/1251000045 learned_addr learned my addr 172.21.15.104:0/1251000045 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44d0d59700 1 --2- 172.21.15.104:0/1251000045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55928995fcd0 0x5592899600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44d155a700 1 --2- 172.21.15.104:0/1251000045 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995a460 0x55928995a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44c3fff700 1 -- 172.21.15.104:0/1251000045 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995a460 msgr2=0x55928995a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44c3fff700 1 --2- 172.21.15.104:0/1251000045 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995a460 0x55928995a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44c3fff700 1 -- 172.21.15.104:0/1251000045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55928995fcd0 msgr2=0x5592899600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44c3fff700 1 --2- 172.21.15.104:0/1251000045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55928995fcd0 0x5592899600b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44c3fff700 1 -- 172.21.15.104:0/1251000045 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55928984a0e0 con 0x5592899605f0 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44d0d59700 1 --2- 172.21.15.104:0/1251000045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55928995fcd0 0x5592899600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44c3fff700 1 --2- 172.21.15.104:0/1251000045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5592899605f0 0x559289959f20 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00aca0 tx=0x7f44bc00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d007ea702f46043 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44b2ffd700 1 -- 172.21.15.104:0/1251000045 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc004830 con 0x5592899605f0 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44b2ffd700 1 -- 172.21.15.104:0/1251000045 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44bc005020 con 0x5592899605f0 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.444+0000 7f44b2ffd700 1 -- 172.21.15.104:0/1251000045 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc018640 con 0x5592899605f0 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.445+0000 7f44dd840b80 1 -- 172.21.15.104:0/1251000045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5592899605f0 msgr2=0x559289959f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:33.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.445+0000 7f44dd840b80 1 --2- 172.21.15.104:0/1251000045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5592899605f0 0x559289959f20 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00aca0 tx=0x7f44bc00f3c0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.445+0000 7f44dd840b80 1 -- 172.21.15.104:0/1251000045 shutdown_connections 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.445+0000 7f44dd840b80 1 --2- 172.21.15.104:0/1251000045 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995a460 0x55928995a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.445+0000 7f44dd840b80 1 --2- 172.21.15.104:0/1251000045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55928995fcd0 0x5592899600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.445+0000 7f44dd840b80 1 --2- 172.21.15.104:0/1251000045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5592899605f0 0x559289959f20 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.445+0000 7f44dd840b80 1 -- 172.21.15.104:0/1251000045 >> 172.21.15.104:0/1251000045 conn(0x559289884e50 msgr2=0x5592899551b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.445+0000 7f44dd840b80 1 -- 172.21.15.104:0/1251000045 shutdown_connections 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.445+0000 7f44dd840b80 1 -- 172.21.15.104:0/1251000045 wait complete. 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.445+0000 7f44dd840b80 1 -- start start 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44dd840b80 1 Processor -- start 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44dd840b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 0x5592899662b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44dd840b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995fcd0 0x5592899635b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44dd840b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592899605f0 0x559289963af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44dd840b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559289967060 con 0x55928995a460 2024-06-09T22:25:33.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44dd840b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55928984bd80 con 0x5592899605f0 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44dd840b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5592899671e0 con 0x55928995fcd0 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44dd840b80 10 client.0 _ll_register_callbacks cb 0x559289845ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44d0d59700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 0x5592899662b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44d0d59700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 0x5592899662b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55400/0 (socket says 172.21.15.104:55400) 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44d0d59700 1 -- 172.21.15.104:0/2818831596 learned_addr learned my addr 172.21.15.104:0/2818831596 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44c3fff700 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995fcd0 0x5592899635b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44d155a700 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592899605f0 0x559289963af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44d0d59700 1 -- 172.21.15.104:0/2818831596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995fcd0 msgr2=0x5592899635b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44d0d59700 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995fcd0 0x5592899635b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44d0d59700 1 -- 172.21.15.104:0/2818831596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592899605f0 msgr2=0x559289963af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44d0d59700 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592899605f0 0x559289963af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44d0d59700 1 -- 172.21.15.104:0/2818831596 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592897faa60 con 0x55928995a460 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.446+0000 7f44d155a700 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592899605f0 0x559289963af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:33.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44d0d59700 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 0x5592899662b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009990 tx=0x7f44c8009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc1cc8c25b3216b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44abfff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44abfff700 1 -- 172.21.15.104:0/2818831596 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8035060 con 0x55928995a460 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44abfff700 1 -- 172.21.15.104:0/2818831596 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f44c8030630 con 0x55928995a460 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44dd840b80 1 -- 172.21.15.104:0/2818831596 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44bc005020 con 0x55928995a460 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44abfff700 1 -- 172.21.15.104:0/2818831596 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f44c803c050 con 0x55928995a460 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44dd840b80 10 client.16761 fetch_fsmap learned FSMap version 31 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44dd840b80 1 -- 172.21.15.104:0/2818831596 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5592899718e0 con 0x55928995a460 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44abfff700 1 -- 172.21.15.104:0/2818831596 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44bc005020 con 0x55928995a460 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44abfff700 1 -- 172.21.15.104:0/2818831596 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f44c802f070 con 0x55928995a460 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44dd840b80 10 client.16761 fetch_fsmap finished waiting for FSMap version 31 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44dd840b80 10 client.16761 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44dd840b80 20 client.16761 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:33.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44dd840b80 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559289973f20 0x559289976350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44dd840b80 4 client.16761 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44dd840b80 1 -- 172.21.15.104:0/2818831596 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592898b8ea0 con 0x559289973f20 2024-06-09T22:25:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.447+0000 7f44c3fff700 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559289973f20 0x559289976350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.448+0000 7f44c3fff700 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559289973f20 0x559289976350 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=849aa6ab780043a3 server_cookie=b771fbdc5ad76c99 in_seq=0 out_seq=0 2024-06-09T22:25:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.448+0000 7f44abfff700 10 client.16761 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44abfff700 1 -- 172.21.15.104:0/2818831596 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5592898b8ea0 con 0x559289973f20 2024-06-09T22:25:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44abfff700 10 client.16761 handle_command_reply: tid=0 2024-06-09T22:25:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44dd840b80 1 client.16761 shutdown 2024-06-09T22:25:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44dd840b80 1 -- 172.21.15.104:0/2818831596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 msgr2=0x5592899662b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44dd840b80 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 0x5592899662b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009990 tx=0x7f44c8009ca0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44dd840b80 1 -- 172.21.15.104:0/2818831596 shutdown_connections 2024-06-09T22:25:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44dd840b80 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559289973f20 0x559289976350 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44dd840b80 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 0x5592899662b0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44abfff700 0 client.16761 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44dd840b80 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592899605f0 0x559289963af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44dd840b80 1 --2- 172.21.15.104:0/2818831596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995fcd0 0x5592899635b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44dd840b80 1 -- 172.21.15.104:0/2818831596 >> 172.21.15.104:0/2818831596 conn(0x559289884e50 msgr2=0x559289958f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44dd840b80 1 -- 172.21.15.104:0/2818831596 shutdown_connections 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44dd840b80 1 -- 172.21.15.104:0/2818831596 wait complete. 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.449+0000 7f44dd840b80 20 client.16761 trim_cache size 0 max 16384 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.469+0000 7f44dd840b80 1 Processor -- start 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.469+0000 7f44dd840b80 1 -- start start 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.469+0000 7f44dd840b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 0x5592899609d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.469+0000 7f44dd840b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55928995fcd0 0x559289960f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.469+0000 7f44dd840b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592899605f0 0x55928997bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.469+0000 7f44dd840b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc1b05c0 con 0x55928995a460 2024-06-09T22:25:33.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.469+0000 7f44dd840b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc1b2130 con 0x55928995fcd0 2024-06-09T22:25:33.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.469+0000 7f44dd840b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc1b0c90 con 0x5592899605f0 2024-06-09T22:25:33.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.469+0000 7f44d0d59700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 0x5592899609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.469+0000 7f44d0d59700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 0x5592899609d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55416/0 (socket says 172.21.15.104:55416) 2024-06-09T22:25:33.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.469+0000 7f44d0d59700 1 -- 172.21.15.104:0/3716532334 learned_addr learned my addr 172.21.15.104:0/3716532334 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:33.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.469+0000 7f44d155a700 1 --2- 172.21.15.104:0/3716532334 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592899605f0 0x55928997bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.470+0000 7f44c3fff700 1 --2- 172.21.15.104:0/3716532334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55928995fcd0 0x559289960f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.470+0000 7f44d0d59700 1 -- 172.21.15.104:0/3716532334 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592899605f0 msgr2=0x55928997bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:33.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.470+0000 7f44d0d59700 1 --2- 172.21.15.104:0/3716532334 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592899605f0 0x55928997bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.470+0000 7f44d0d59700 1 -- 172.21.15.104:0/3716532334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55928995fcd0 msgr2=0x559289960f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:33.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.470+0000 7f44d0d59700 1 --2- 172.21.15.104:0/3716532334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55928995fcd0 0x559289960f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.470+0000 7f44d0d59700 1 -- 172.21.15.104:0/3716532334 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559289961ba0 con 0x55928995a460 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.470+0000 7f44d0d59700 1 --2- 172.21.15.104:0/3716532334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 0x5592899609d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f44c8000c00 tx=0x7f44c8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6400ff39eae5e1f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.470+0000 7f44b27fc700 1 -- 172.21.15.104:0/3716532334 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8028070 con 0x55928995a460 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.470+0000 7f44b27fc700 1 -- 172.21.15.104:0/3716532334 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44c80173f0 con 0x55928995a460 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.470+0000 7f44b27fc700 1 -- 172.21.15.104:0/3716532334 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8035050 con 0x55928995a460 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 -- 172.21.15.104:0/3716532334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 msgr2=0x5592899609d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 --2- 172.21.15.104:0/3716532334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 0x5592899609d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f44c8000c00 tx=0x7f44c8009310 comp rx=0 tx=0).stop 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 -- 172.21.15.104:0/3716532334 shutdown_connections 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 --2- 172.21.15.104:0/3716532334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995a460 0x5592899609d0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 --2- 172.21.15.104:0/3716532334 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592899605f0 0x55928997bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 --2- 172.21.15.104:0/3716532334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55928995fcd0 0x559289960f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 -- 172.21.15.104:0/3716532334 >> 172.21.15.104:0/3716532334 conn(0x559289884e50 msgr2=0x55928995d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 -- 172.21.15.104:0/3716532334 shutdown_connections 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 -- 172.21.15.104:0/3716532334 wait complete. 2024-06-09T22:25:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 -- start start 2024-06-09T22:25:33.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 Processor -- start 2024-06-09T22:25:33.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995a460 0x55928996ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995fcd0 0x5592899694a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592899605f0 0x5592899699e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc1b1a30 con 0x55928995fcd0 2024-06-09T22:25:33.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc1b1360 con 0x5592899605f0 2024-06-09T22:25:33.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.471+0000 7f44dd840b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc1ae540 con 0x55928995a460 2024-06-09T22:25:33.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44dd840b80 10 client.0 _ll_register_callbacks cb 0x559289845ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:33.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995fcd0 0x5592899694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995fcd0 0x5592899694a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55426/0 (socket says 172.21.15.104:55426) 2024-06-09T22:25:33.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44c3fff700 1 -- 172.21.15.104:0/1268397676 learned_addr learned my addr 172.21.15.104:0/1268397676 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:33.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44d155a700 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592899605f0 0x5592899699e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44d0d59700 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995a460 0x55928996ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44c3fff700 1 -- 172.21.15.104:0/1268397676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995a460 msgr2=0x55928996ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44c3fff700 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995a460 0x55928996ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44c3fff700 1 -- 172.21.15.104:0/1268397676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592899605f0 msgr2=0x5592899699e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44c3fff700 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592899605f0 0x5592899699e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44c3fff700 1 -- 172.21.15.104:0/1268397676 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55928996a240 con 0x55928995fcd0 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44c3fff700 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995fcd0 0x5592899694a0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00afd0 tx=0x7f44bc005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8876f79254c4b1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44abfff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44abfff700 1 -- 172.21.15.104:0/1268397676 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc003680 con 0x55928995fcd0 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44dd840b80 1 -- 172.21.15.104:0/1268397676 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5592898b3420 con 0x55928995fcd0 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44abfff700 1 -- 172.21.15.104:0/1268397676 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f44bc003820 con 0x55928995fcd0 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44abfff700 1 -- 172.21.15.104:0/1268397676 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f44bc038050 con 0x55928995fcd0 2024-06-09T22:25:33.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44dd840b80 10 client.16767 fetch_fsmap learned FSMap version 31 2024-06-09T22:25:33.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44dd840b80 1 -- 172.21.15.104:0/1268397676 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5592899dfc50 con 0x55928995fcd0 2024-06-09T22:25:33.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.472+0000 7f44abfff700 1 -- 172.21.15.104:0/1268397676 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5592898b3420 con 0x55928995fcd0 2024-06-09T22:25:33.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.473+0000 7f44abfff700 1 -- 172.21.15.104:0/1268397676 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f44bc018bb0 con 0x55928995fcd0 2024-06-09T22:25:33.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.473+0000 7f44dd840b80 10 client.16767 fetch_fsmap finished waiting for FSMap version 31 2024-06-09T22:25:33.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.473+0000 7f44dd840b80 10 client.16767 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:25:33.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.473+0000 7f44dd840b80 20 client.16767 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:33.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.473+0000 7f44dd840b80 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5592899e0230 0x5592899e2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:33.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.473+0000 7f44dd840b80 4 client.16767 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.2, "await_for": 45.0}] 2024-06-09T22:25:33.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.473+0000 7f44dd840b80 1 -- 172.21.15.104:0/1268397676 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.2, "await_for": 45.0}) -- 0x55928972d3e0 con 0x5592899e0230 2024-06-09T22:25:33.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.473+0000 7f44d0d59700 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5592899e0230 0x5592899e2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:33.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.473+0000 7f44abfff700 10 client.16767 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:33.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.473+0000 7f44d0d59700 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5592899e0230 0x5592899e2650 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fc8e9c01fb185d0 server_cookie=dcf190dd7af8aba6 in_seq=0 out_seq=0 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.554+0000 7f44abfff700 1 -- 172.21.15.104:0/1268397676 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55928972d3e0 con 0x5592899e0230 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.554+0000 7f44abfff700 10 client.16767 handle_command_reply: tid=0 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.554+0000 7f44dd840b80 1 client.16767 shutdown 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44dd840b80 1 -- 172.21.15.104:0/1268397676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995fcd0 msgr2=0x5592899694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44dd840b80 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995fcd0 0x5592899694a0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00afd0 tx=0x7f44bc005e10 comp rx=0 tx=0).stop 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44dd840b80 1 -- 172.21.15.104:0/1268397676 shutdown_connections 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44dd840b80 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5592899e0230 0x5592899e2650 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44dd840b80 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592899605f0 0x5592899699e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44dd840b80 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55928995a460 0x55928996ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44abfff700 0 client.16767 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44dd840b80 1 --2- 172.21.15.104:0/1268397676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55928995fcd0 0x5592899694a0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44dd840b80 1 -- 172.21.15.104:0/1268397676 >> 172.21.15.104:0/1268397676 conn(0x559289884e50 msgr2=0x5592899609d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:33.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44dd840b80 1 -- 172.21.15.104:0/1268397676 shutdown_connections 2024-06-09T22:25:33.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44dd840b80 1 -- 172.21.15.104:0/1268397676 wait complete. 2024-06-09T22:25:33.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44dd840b80 20 client.16767 trim_cache size 0 max 16384 2024-06-09T22:25:33.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44d2fbd700 1 -- 172.21.15.104:0/312602963 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44b40a2e30 msgr2=0x7f44b40a52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:33.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.555+0000 7f44d2fbd700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44b40a2e30 0x7f44b40a52f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f44c80098e0 tx=0x7f44c8005960 comp rx=0 tx=0).stop 2024-06-09T22:25:33.616 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:25:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 31, 2024-06-09T22:25:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: "leader": 24460, 2024-06-09T22:25:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: "set_version": 10, 2024-06-09T22:25:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: "sets": { 2024-06-09T22:25:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: "4224fc39": { 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "version": 10, 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "age_ref": 0.0, 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "QUIESCED", 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "timeout": 90.0, 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "expiration": 138.2, 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "members": { 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "file:/": { 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "excluded": false, 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "QUIESCED", 2024-06-09T22:25:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:25:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:25:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:25:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:25:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:25:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:25:33.618 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:25:33.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.556+0000 7f44d2fbd700 1 -- 172.21.15.104:0/312602963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44cc10ef40 msgr2=0x7f44cc19ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:33.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.556+0000 7f44d2fbd700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44cc10ef40 0x7f44cc19ab80 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f44c400ada0 tx=0x7f44c400c330 comp rx=0 tx=0).stop 2024-06-09T22:25:33.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.556+0000 7f44d2fbd700 1 -- 172.21.15.104:0/312602963 shutdown_connections 2024-06-09T22:25:33.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.556+0000 7f44d2fbd700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f44b40a2e30 0x7f44b40a52f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.556+0000 7f44d2fbd700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44cc10ef40 0x7f44cc19ab80 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.556+0000 7f44d2fbd700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc101ec0 0x7f44cc1a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.556+0000 7f44d2fbd700 1 --2- 172.21.15.104:0/312602963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44cc1027e0 0x7f44cc1a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:33.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.556+0000 7f44d2fbd700 1 -- 172.21.15.104:0/312602963 >> 172.21.15.104:0/312602963 conn(0x7f44cc078330 msgr2=0x7f44cc110250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:33.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.556+0000 7f44d2fbd700 1 -- 172.21.15.104:0/312602963 shutdown_connections 2024-06-09T22:25:33.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:33.556+0000 7f44d2fbd700 1 -- 172.21.15.104:0/312602963 wait complete. 2024-06-09T22:25:33.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 3.43 MB/sec execute 39 sec latency 1812.364 ms 2024-06-09T22:25:33.861 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:34.676 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '4224fc39', quiesce took 9.2 seconds. Will release after: 9.0 2024-06-09T22:25:34.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 3.34 MB/sec execute 40 sec latency 1176.244 ms 2024-06-09T22:25:35.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:34 smithi153 ceph-mon[36599]: pgmap v141: 129 pgs: 129 active+clean; 74 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-06-09T22:25:35.245 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:34 smithi104 ceph-mon[32389]: pgmap v141: 129 pgs: 129 active+clean; 74 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-06-09T22:25:35.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:34 smithi194 ceph-mon[36856]: pgmap v141: 129 pgs: 129 active+clean; 74 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-06-09T22:25:35.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 3.26 MB/sec execute 41 sec latency 2176.309 ms 2024-06-09T22:25:36.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:35 smithi104 ceph-mon[32389]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:25:36.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.117+0000 7f12bb532640 1 -- 172.21.15.104:0/2230603737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12b41008d0 msgr2=0x7f12b4100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.117+0000 7f12bb532640 1 --2- 172.21.15.104:0/2230603737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12b41008d0 0x7f12b4100d50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009930 tx=0x7f12a402a420 comp rx=0 tx=0).stop 2024-06-09T22:25:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.117+0000 7f12bb532640 1 -- 172.21.15.104:0/2230603737 shutdown_connections 2024-06-09T22:25:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.117+0000 7f12bb532640 1 --2- 172.21.15.104:0/2230603737 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12b4101290 0x7f12b410d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.117+0000 7f12bb532640 1 --2- 172.21.15.104:0/2230603737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12b41008d0 0x7f12b4100d50 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.117+0000 7f12bb532640 1 --2- 172.21.15.104:0/2230603737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12b4106890 0x7f12b4106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.117+0000 7f12bb532640 1 -- 172.21.15.104:0/2230603737 >> 172.21.15.104:0/2230603737 conn(0x7f12b4077a20 msgr2=0x7f12b4077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.117+0000 7f12bb532640 1 -- 172.21.15.104:0/2230603737 shutdown_connections 2024-06-09T22:25:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.117+0000 7f12bb532640 1 -- 172.21.15.104:0/2230603737 wait complete. 2024-06-09T22:25:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.117+0000 7f12bb532640 1 Processor -- start 2024-06-09T22:25:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.117+0000 7f12bb532640 1 -- start start 2024-06-09T22:25:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12bb532640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12b41008d0 0x7f12b4197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12bb532640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12b4101290 0x7f12b41976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12bb532640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12b4106890 0x7f12b419b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12bb532640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f12b4110ca0 con 0x7f12b4106890 2024-06-09T22:25:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12bb532640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12b4110b20 con 0x7f12b4101290 2024-06-09T22:25:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12bb532640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f12b4110e20 con 0x7f12b41008d0 2024-06-09T22:25:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b9acf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12b4106890 0x7f12b419b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b9acf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12b4106890 0x7f12b419b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47236/0 (socket says 172.21.15.104:47236) 2024-06-09T22:25:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b9acf640 1 -- 172.21.15.104:0/1780858672 learned_addr learned my addr 172.21.15.104:0/1780858672 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b8acd640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12b4101290 0x7f12b41976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b9acf640 1 -- 172.21.15.104:0/1780858672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12b41008d0 msgr2=0x7f12b4197180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b92ce640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12b41008d0 0x7f12b4197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b9acf640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12b41008d0 0x7f12b4197180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b9acf640 1 -- 172.21.15.104:0/1780858672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12b4101290 msgr2=0x7f12b41976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b9acf640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12b4101290 0x7f12b41976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b9acf640 1 -- 172.21.15.104:0/1780858672 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b419c0d0 con 0x7f12b4106890 2024-06-09T22:25:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b92ce640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12b41008d0 0x7f12b4197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:25:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b8acd640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12b4101290 0x7f12b41976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12b9acf640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12b4106890 0x7f12b419b9a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009fd0 tx=0x7f12ac00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12aa7fc640 1 -- 172.21.15.104:0/1780858672 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ac017070 con 0x7f12b4106890 2024-06-09T22:25:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12aa7fc640 1 -- 172.21.15.104:0/1780858672 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12ac003e50 con 0x7f12b4106890 2024-06-09T22:25:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12bb532640 1 -- 172.21.15.104:0/1780858672 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b419c3c0 con 0x7f12b4106890 2024-06-09T22:25:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12aa7fc640 1 -- 172.21.15.104:0/1780858672 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ac0106c0 con 0x7f12b4106890 2024-06-09T22:25:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.118+0000 7f12bb532640 1 -- 172.21.15.104:0/1780858672 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b41a3c50 con 0x7f12b4106890 2024-06-09T22:25:36.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.121+0000 7f12aa7fc640 1 -- 172.21.15.104:0/1780858672 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f12ac003970 con 0x7f12b4106890 2024-06-09T22:25:36.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.121+0000 7f12be23d740 1 Processor -- start 2024-06-09T22:25:36.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.121+0000 7f12be23d740 1 -- start start 2024-06-09T22:25:36.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.121+0000 7f12be23d740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847050 0x560d24847430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.121+0000 7f12be23d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d24854020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.131 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.121+0000 7f12be23d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24854560 0x560d24856950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.121+0000 7f12be23d740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x560d247ab4f0 con 0x560d24847050 2024-06-09T22:25:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.121+0000 7f12be23d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560d24749870 con 0x560d24847970 2024-06-09T22:25:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.121+0000 7f12be23d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560d247a74b0 con 0x560d24854560 2024-06-09T22:25:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b8acd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847050 0x560d24847430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12aa7fc640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12a00a2da0 0x7f12a00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b8acd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847050 0x560d24847430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47240/0 (socket says 172.21.15.104:47240) 2024-06-09T22:25:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b8acd640 1 -- 172.21.15.104:0/2265482984 learned_addr learned my addr 172.21.15.104:0/2265482984 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b92ce640 1 --2- 172.21.15.104:0/2265482984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d24854020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b8acd640 1 --2- 172.21.15.104:0/2265482984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24854560 0x560d24856950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b9acf640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12a00a2da0 0x7f12a00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12aa7fc640 1 -- 172.21.15.104:0/1780858672 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f12ac0cf610 con 0x7f12b4106890 2024-06-09T22:25:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b8acd640 1 -- 172.21.15.104:0/2265482984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24854560 msgr2=0x560d24856950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b8acd640 1 --2- 172.21.15.104:0/2265482984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24854560 0x560d24856950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b8acd640 1 -- 172.21.15.104:0/2265482984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 msgr2=0x560d24854020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b8acd640 1 --2- 172.21.15.104:0/2265482984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d24854020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b8acd640 1 -- 172.21.15.104:0/2265482984 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d247659b0 con 0x560d24847050 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b8acd640 1 --2- 172.21.15.104:0/2265482984 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847050 0x560d24847430 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f12a402a940 tx=0x7f12a4031480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5daed06389fafd server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f12b9acf640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12a00a2da0 0x7f12a00a5260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f12ac097c20 tx=0x7f12ac086040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f129f7fe640 1 -- 172.21.15.104:0/2265482984 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4035070 con 0x560d24847050 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f129f7fe640 1 -- 172.21.15.104:0/2265482984 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a4044070 con 0x560d24847050 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.122+0000 7f129f7fe640 1 -- 172.21.15.104:0/2265482984 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a40048c0 con 0x560d24847050 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.123+0000 7f12be23d740 1 -- 172.21.15.104:0/2265482984 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847050 msgr2=0x560d24847430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.123+0000 7f12be23d740 1 --2- 172.21.15.104:0/2265482984 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847050 0x560d24847430 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f12a402a940 tx=0x7f12a4031480 comp rx=0 tx=0).stop 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.123+0000 7f12be23d740 1 -- 172.21.15.104:0/2265482984 shutdown_connections 2024-06-09T22:25:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.123+0000 7f12be23d740 1 --2- 172.21.15.104:0/2265482984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24854560 0x560d24856950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.123+0000 7f12be23d740 1 --2- 172.21.15.104:0/2265482984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d24854020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.123+0000 7f12be23d740 1 --2- 172.21.15.104:0/2265482984 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847050 0x560d24847430 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.123+0000 7f12be23d740 1 -- 172.21.15.104:0/2265482984 >> 172.21.15.104:0/2265482984 conn(0x560d24843680 msgr2=0x560d24843a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.123+0000 7f12be23d740 1 -- 172.21.15.104:0/2265482984 shutdown_connections 2024-06-09T22:25:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.123+0000 7f12be23d740 1 -- 172.21.15.104:0/2265482984 wait complete. 2024-06-09T22:25:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.123+0000 7f12be23d740 1 -- start start 2024-06-09T22:25:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.123+0000 7f12be23d740 1 Processor -- start 2024-06-09T22:25:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12be23d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 0x560d24851540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12be23d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d24851a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12be23d740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 0x560d24851fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12be23d740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x560d24857130 con 0x560d24854560 2024-06-09T22:25:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12be23d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560d24734ae0 con 0x560d24847970 2024-06-09T22:25:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12be23d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560d248572b0 con 0x560d24847050 2024-06-09T22:25:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12be23d740 10 client.0 _ll_register_callbacks cb 0x560d2476d9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12b8acd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 0x560d24851fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12b8acd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 0x560d24851fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47252/0 (socket says 172.21.15.104:47252) 2024-06-09T22:25:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12b8acd640 1 -- 172.21.15.104:0/3690677163 learned_addr learned my addr 172.21.15.104:0/3690677163 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12b8acd640 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 0x560d24851540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12b92ce640 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d24851a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12b8acd640 1 -- 172.21.15.104:0/3690677163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 msgr2=0x560d24851540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12b8acd640 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 0x560d24851540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12b8acd640 1 -- 172.21.15.104:0/3690677163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 msgr2=0x560d24851a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12b8acd640 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d24851a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12b8acd640 1 -- 172.21.15.104:0/3690677163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d244bf210 con 0x560d24854560 2024-06-09T22:25:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f12b8acd640 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 0x560d24851fc0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f12a4005ec0 tx=0x7f12a40055b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7b58e806293789 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.124+0000 7f129cff9640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f129cff9640 1 -- 172.21.15.104:0/3690677163 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4035070 con 0x560d24854560 2024-06-09T22:25:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f129cff9640 1 -- 172.21.15.104:0/3690677163 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f12a4004220 con 0x560d24854560 2024-06-09T22:25:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f12be23d740 1 -- 172.21.15.104:0/3690677163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12a4044070 con 0x560d24854560 2024-06-09T22:25:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f129cff9640 1 -- 172.21.15.104:0/3690677163 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f12a405b050 con 0x560d24854560 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f12be23d740 10 client.16779 fetch_fsmap learned FSMap version 32 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f12be23d740 1 -- 172.21.15.104:0/3690677163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d2458b620 con 0x560d24854560 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f129cff9640 1 -- 172.21.15.104:0/3690677163 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a4044070 con 0x560d24854560 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f129cff9640 1 -- 172.21.15.104:0/3690677163 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f12a40039f0 con 0x560d24854560 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f12be23d740 10 client.16779 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f12be23d740 10 client.16779 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f12be23d740 20 client.16779 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f12be23d740 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560d248645f0 0x560d248669e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f12be23d740 4 client.16779 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.125+0000 7f12be23d740 1 -- 172.21.15.104:0/3690677163 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d247bef10 con 0x560d248645f0 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.126+0000 7f12b92ce640 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560d248645f0 0x560d248669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.126+0000 7f129cff9640 10 client.16779 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.126+0000 7f12b92ce640 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560d248645f0 0x560d248669e0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=465c2d4f81d61640 server_cookie=bef7b65c75eedd50 in_seq=0 out_seq=0 2024-06-09T22:25:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f129cff9640 1 -- 172.21.15.104:0/3690677163 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x560d247bef10 con 0x560d248645f0 2024-06-09T22:25:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f129cff9640 10 client.16779 handle_command_reply: tid=0 2024-06-09T22:25:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f12be23d740 1 client.16779 shutdown 2024-06-09T22:25:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f12be23d740 1 -- 172.21.15.104:0/3690677163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 msgr2=0x560d24851fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f12be23d740 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 0x560d24851fc0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f12a4005ec0 tx=0x7f12a40055b0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f12be23d740 1 -- 172.21.15.104:0/3690677163 shutdown_connections 2024-06-09T22:25:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f12be23d740 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560d248645f0 0x560d248669e0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f12be23d740 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 0x560d24851fc0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f12be23d740 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d24851a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f129cff9640 0 client.16779 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f12be23d740 1 --2- 172.21.15.104:0/3690677163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 0x560d24851540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f12be23d740 1 -- 172.21.15.104:0/3690677163 >> 172.21.15.104:0/3690677163 conn(0x560d24843680 msgr2=0x560d24849490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f12be23d740 1 -- 172.21.15.104:0/3690677163 shutdown_connections 2024-06-09T22:25:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f12be23d740 1 -- 172.21.15.104:0/3690677163 wait complete. 2024-06-09T22:25:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.127+0000 7f12be23d740 20 client.16779 trim_cache size 0 max 16384 2024-06-09T22:25:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.142+0000 7f12be23d740 1 Processor -- start 2024-06-09T22:25:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.142+0000 7f12be23d740 1 -- start start 2024-06-09T22:25:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.142+0000 7f12be23d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 0x560d24842660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.142+0000 7f12be23d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d2486c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.142+0000 7f12be23d740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 0x560d2486c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.142+0000 7f12be23d740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f12b41aca00 con 0x560d24854560 2024-06-09T22:25:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.142+0000 7f12be23d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12b41ae540 con 0x560d24847970 2024-06-09T22:25:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.142+0000 7f12be23d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f12b41ad0d0 con 0x560d24847050 2024-06-09T22:25:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f12b8acd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 0x560d2486c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f12b8acd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 0x560d2486c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47262/0 (socket says 172.21.15.104:47262) 2024-06-09T22:25:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f12b8acd640 1 -- 172.21.15.104:0/3290774590 learned_addr learned my addr 172.21.15.104:0/3290774590 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f12b92ce640 1 --2- 172.21.15.104:0/3290774590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d2486c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f12b8acd640 1 --2- 172.21.15.104:0/3290774590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 0x560d24842660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f12b8acd640 1 -- 172.21.15.104:0/3290774590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 msgr2=0x560d24842660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f12b8acd640 1 --2- 172.21.15.104:0/3290774590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 0x560d24842660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f12b8acd640 1 -- 172.21.15.104:0/3290774590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 msgr2=0x560d2486c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f12b8acd640 1 --2- 172.21.15.104:0/3290774590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d2486c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f12b8acd640 1 -- 172.21.15.104:0/3290774590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d24856950 con 0x560d24854560 2024-06-09T22:25:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f12b92ce640 1 --2- 172.21.15.104:0/3290774590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d2486c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:25:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f12b8acd640 1 --2- 172.21.15.104:0/3290774590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 0x560d2486c580 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f12a4001eb0 tx=0x7f12a4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3715200bbd348c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f129effd640 1 -- 172.21.15.104:0/3290774590 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4035040 con 0x560d24854560 2024-06-09T22:25:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f129effd640 1 -- 172.21.15.104:0/3290774590 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a4044040 con 0x560d24854560 2024-06-09T22:25:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.143+0000 7f129effd640 1 -- 172.21.15.104:0/3290774590 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4005110 con 0x560d24854560 2024-06-09T22:25:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.144+0000 7f12be23d740 1 -- 172.21.15.104:0/3290774590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 msgr2=0x560d2486c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.144+0000 7f12be23d740 1 --2- 172.21.15.104:0/3290774590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 0x560d2486c580 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f12a4001eb0 tx=0x7f12a4000bc0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.144+0000 7f12be23d740 1 -- 172.21.15.104:0/3290774590 shutdown_connections 2024-06-09T22:25:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.144+0000 7f12be23d740 1 --2- 172.21.15.104:0/3290774590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24854560 0x560d2486c580 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.144+0000 7f12be23d740 1 --2- 172.21.15.104:0/3290774590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24847970 0x560d2486c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.144+0000 7f12be23d740 1 --2- 172.21.15.104:0/3290774590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 0x560d24842660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.144+0000 7f12be23d740 1 -- 172.21.15.104:0/3290774590 >> 172.21.15.104:0/3290774590 conn(0x560d24843680 msgr2=0x560d24841f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.144+0000 7f12be23d740 1 -- 172.21.15.104:0/3290774590 shutdown_connections 2024-06-09T22:25:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.144+0000 7f12be23d740 1 -- 172.21.15.104:0/3290774590 wait complete. 2024-06-09T22:25:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.144+0000 7f12be23d740 1 -- start start 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.144+0000 7f12be23d740 1 Processor -- start 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12be23d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 0x560d24858260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12be23d740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847970 0x560d248587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12be23d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24858ce0 0x560d248ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12be23d740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f12b41ade70 con 0x560d24847970 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12be23d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12b41ad7a0 con 0x560d24858ce0 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12be23d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f12b41aa9c0 con 0x560d24847050 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12be23d740 10 client.0 _ll_register_callbacks cb 0x7f12b0005f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12b92ce640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847970 0x560d248587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12b92ce640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847970 0x560d248587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47272/0 (socket says 172.21.15.104:47272) 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12b92ce640 1 -- 172.21.15.104:0/2847041663 learned_addr learned my addr 172.21.15.104:0/2847041663 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12b8acd640 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24858ce0 0x560d248ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12b92ce640 1 -- 172.21.15.104:0/2847041663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 msgr2=0x560d24858260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12b92ce640 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 0x560d24858260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12b92ce640 1 -- 172.21.15.104:0/2847041663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24858ce0 msgr2=0x560d248ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12b92ce640 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24858ce0 0x560d248ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12b92ce640 1 -- 172.21.15.104:0/2847041663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d2485d1b0 con 0x560d24847970 2024-06-09T22:25:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12b8acd640 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24858ce0 0x560d248ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12b92ce640 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847970 0x560d248587a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f12b000bb60 tx=0x7f12b000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7febbd271bf1af61 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f129cff9640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f129cff9640 1 -- 172.21.15.104:0/2847041663 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b001c880 con 0x560d24847970 2024-06-09T22:25:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f129cff9640 1 -- 172.21.15.104:0/2847041663 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f12b001ca20 con 0x560d24847970 2024-06-09T22:25:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12be23d740 1 -- 172.21.15.104:0/2847041663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12a4044040 con 0x560d24847970 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f129cff9640 1 -- 172.21.15.104:0/2847041663 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f12b002c050 con 0x560d24847970 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12be23d740 10 client.16785 fetch_fsmap learned FSMap version 32 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.145+0000 7f12be23d740 1 -- 172.21.15.104:0/2847041663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d248cf610 con 0x560d24847970 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.146+0000 7f129cff9640 1 -- 172.21.15.104:0/2847041663 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a4044040 con 0x560d24847970 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.146+0000 7f129cff9640 1 -- 172.21.15.104:0/2847041663 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f12b001b3d0 con 0x560d24847970 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.146+0000 7f12be23d740 10 client.16785 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.146+0000 7f12be23d740 10 client.16785 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.146+0000 7f12be23d740 20 client.16785 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.146+0000 7f12be23d740 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560d248cfd70 0x560d248d2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.146+0000 7f12be23d740 4 client.16785 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.146+0000 7f12be23d740 1 -- 172.21.15.104:0/2847041663 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560d247b8c20 con 0x560d248cfd70 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.146+0000 7f12b8acd640 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560d248cfd70 0x560d248d2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.146+0000 7f129cff9640 10 client.16785 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.146+0000 7f12b8acd640 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560d248cfd70 0x560d248d2190 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dd50fa9869ca086 server_cookie=7d5fddfd4c50aca0 in_seq=0 out_seq=0 2024-06-09T22:25:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f129cff9640 1 -- 172.21.15.104:0/2847041663 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x560d247b8c20 con 0x560d248cfd70 2024-06-09T22:25:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f129cff9640 10 client.16785 handle_command_reply: tid=0 2024-06-09T22:25:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f12be23d740 1 client.16785 shutdown 2024-06-09T22:25:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f12be23d740 1 -- 172.21.15.104:0/2847041663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847970 msgr2=0x560d248587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f12be23d740 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847970 0x560d248587a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f12b000bb60 tx=0x7f12b000f7f0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f12be23d740 1 -- 172.21.15.104:0/2847041663 shutdown_connections 2024-06-09T22:25:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f12be23d740 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560d248cfd70 0x560d248d2190 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f12be23d740 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560d24858ce0 0x560d248ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f12be23d740 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560d24847970 0x560d248587a0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f129cff9640 0 client.16785 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f12be23d740 1 --2- 172.21.15.104:0/2847041663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d24847050 0x560d24858260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f12be23d740 1 -- 172.21.15.104:0/2847041663 >> 172.21.15.104:0/2847041663 conn(0x560d24843680 msgr2=0x560d24841f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f12be23d740 1 -- 172.21.15.104:0/2847041663 shutdown_connections 2024-06-09T22:25:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f12be23d740 1 -- 172.21.15.104:0/2847041663 wait complete. 2024-06-09T22:25:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.156+0000 7f12be23d740 20 client.16785 trim_cache size 0 max 16384 2024-06-09T22:25:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:25:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:25:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:25:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.157+0000 7f12bb532640 1 -- 172.21.15.104:0/1780858672 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12a00a2da0 msgr2=0x7f12a00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.157+0000 7f12bb532640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12a00a2da0 0x7f12a00a5260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f12ac097c20 tx=0x7f12ac086040 comp rx=0 tx=0).stop 2024-06-09T22:25:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.157+0000 7f12bb532640 1 -- 172.21.15.104:0/1780858672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12b4106890 msgr2=0x7f12b419b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.157+0000 7f12bb532640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12b4106890 0x7f12b419b9a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009fd0 tx=0x7f12ac00b260 comp rx=0 tx=0).stop 2024-06-09T22:25:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.157+0000 7f12bb532640 1 -- 172.21.15.104:0/1780858672 shutdown_connections 2024-06-09T22:25:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.157+0000 7f12bb532640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f12a00a2da0 0x7f12a00a5260 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.157+0000 7f12bb532640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12b4106890 0x7f12b419b9a0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.157+0000 7f12bb532640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12b4101290 0x7f12b41976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.157+0000 7f12bb532640 1 --2- 172.21.15.104:0/1780858672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12b41008d0 0x7f12b4197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.157+0000 7f12bb532640 1 -- 172.21.15.104:0/1780858672 >> 172.21.15.104:0/1780858672 conn(0x7f12b4077a20 msgr2=0x7f12b41033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.157+0000 7f12bb532640 1 -- 172.21.15.104:0/1780858672 shutdown_connections 2024-06-09T22:25:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:36.157+0000 7f12bb532640 1 -- 172.21.15.104:0/1780858672 wait complete. 2024-06-09T22:25:36.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:35 smithi153 ceph-mon[36599]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:25:36.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:35 smithi194 ceph-mon[36856]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-06-09T22:25:36.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.479+0000 7fb14b8f4700 1 -- 172.21.15.104:0/2348126013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb144101390 msgr2=0x7fb144101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.479+0000 7fb14b8f4700 1 --2- 172.21.15.104:0/2348126013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb144101390 0x7fb144101810 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb134009990 tx=0x7fb134009ca0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.479+0000 7fb14b8f4700 1 -- 172.21.15.104:0/2348126013 shutdown_connections 2024-06-09T22:25:36.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.479+0000 7fb14b8f4700 1 --2- 172.21.15.104:0/2348126013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb144101d50 0x7fb14410e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.479+0000 7fb14b8f4700 1 --2- 172.21.15.104:0/2348126013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb144107400 0x7fb1441077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.479+0000 7fb14b8f4700 1 --2- 172.21.15.104:0/2348126013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb144101390 0x7fb144101810 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.479+0000 7fb14b8f4700 1 -- 172.21.15.104:0/2348126013 >> 172.21.15.104:0/2348126013 conn(0x7fb144078340 msgr2=0x7fb144078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:36.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.480+0000 7fb14b8f4700 1 -- 172.21.15.104:0/2348126013 shutdown_connections 2024-06-09T22:25:36.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.480+0000 7fb14b8f4700 1 -- 172.21.15.104:0/2348126013 wait complete. 2024-06-09T22:25:36.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.480+0000 7fb14b8f4700 1 Processor -- start 2024-06-09T22:25:36.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.480+0000 7fb14b8f4700 1 -- start start 2024-06-09T22:25:36.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.480+0000 7fb14b8f4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb144101390 0x7fb144103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.480+0000 7fb14b8f4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb144101d50 0x7fb1441035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.480+0000 7fb14b8f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb144107400 0x7fb144103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.480+0000 7fb14b8f4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb144111690 con 0x7fb144107400 2024-06-09T22:25:36.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.480+0000 7fb14b8f4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb144111510 con 0x7fb144101390 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.480+0000 7fb14b8f4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb144111810 con 0x7fb144101d50 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb149e91700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb144107400 0x7fb144103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb149e91700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb144107400 0x7fb144103af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47282/0 (socket says 172.21.15.104:47282) 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb149e91700 1 -- 172.21.15.104:0/1111792292 learned_addr learned my addr 172.21.15.104:0/1111792292 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb148e8f700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb144101d50 0x7fb1441035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb149690700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb144101390 0x7fb144103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb149e91700 1 -- 172.21.15.104:0/1111792292 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb144101d50 msgr2=0x7fb1441035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb149e91700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb144101d50 0x7fb1441035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb149e91700 1 -- 172.21.15.104:0/1111792292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb144101390 msgr2=0x7fb144103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb149e91700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb144101390 0x7fb144103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb149e91700 1 -- 172.21.15.104:0/1111792292 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1441a70b0 con 0x7fb144107400 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb149690700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb144101390 0x7fb144103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:25:36.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb149e91700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb144107400 0x7fb144103af0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb13c007d70 tx=0x7fb13c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb13a7fc700 1 -- 172.21.15.104:0/1111792292 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c016020 con 0x7fb144107400 2024-06-09T22:25:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb14b8f4700 1 -- 172.21.15.104:0/1111792292 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1441a73a0 con 0x7fb144107400 2024-06-09T22:25:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb13a7fc700 1 -- 172.21.15.104:0/1111792292 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb13c00ad40 con 0x7fb144107400 2024-06-09T22:25:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb13a7fc700 1 -- 172.21.15.104:0/1111792292 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c0116f0 con 0x7fb144107400 2024-06-09T22:25:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.481+0000 7fb14b8f4700 1 -- 172.21.15.104:0/1111792292 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1441a7930 con 0x7fb144107400 2024-06-09T22:25:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.483+0000 7fb13a7fc700 1 -- 172.21.15.104:0/1111792292 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fb13c019020 con 0x7fb144107400 2024-06-09T22:25:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.484+0000 7fb13a7fc700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1300a2d90 0x7fb1300a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.485+0000 7fb149690700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1300a2d90 0x7fb1300a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.485+0000 7fb13a7fc700 1 -- 172.21.15.104:0/1111792292 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb13c0cf6f0 con 0x7fb144107400 2024-06-09T22:25:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.485+0000 7fb149690700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1300a2d90 0x7fb1300a5250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb140009b50 tx=0x7fb140009550 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.485+0000 7fb156177b80 1 Processor -- start 2024-06-09T22:25:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.485+0000 7fb156177b80 1 -- start start 2024-06-09T22:25:36.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb156177b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a24f60 0x55ad57a25340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb156177b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a25880 0x55ad57a1f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb156177b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 0x55ad57a1fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb156177b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55ad57911f60 con 0x55ad57a1f7a0 2024-06-09T22:25:36.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb156177b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ad57970850 con 0x55ad57a25880 2024-06-09T22:25:36.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb156177b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ad578d8460 con 0x55ad57a24f60 2024-06-09T22:25:36.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb149690700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a24f60 0x55ad57a25340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb149690700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a24f60 0x55ad57a25340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:34666/0 (socket says 172.21.15.104:34666) 2024-06-09T22:25:36.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb149690700 1 -- 172.21.15.104:0/228393156 learned_addr learned my addr 172.21.15.104:0/228393156 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:36.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb149e91700 1 --2- 172.21.15.104:0/228393156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 0x55ad57a1fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb148e8f700 1 --2- 172.21.15.104:0/228393156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a25880 0x55ad57a1f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb149e91700 1 -- 172.21.15.104:0/228393156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a24f60 msgr2=0x55ad57a25340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb149e91700 1 --2- 172.21.15.104:0/228393156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a24f60 0x55ad57a25340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb149e91700 1 -- 172.21.15.104:0/228393156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a25880 msgr2=0x55ad57a1f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb149e91700 1 --2- 172.21.15.104:0/228393156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a25880 0x55ad57a1f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb149e91700 1 -- 172.21.15.104:0/228393156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad578bf8d0 con 0x55ad57a1f7a0 2024-06-09T22:25:36.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb148e8f700 1 --2- 172.21.15.104:0/228393156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a25880 0x55ad57a1f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:36.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.486+0000 7fb149e91700 1 --2- 172.21.15.104:0/228393156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 0x55ad57a1fbc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0ccba0 tx=0x7fb13c007c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8efd456cf12c3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:36.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.487+0000 7fb12f7fe700 1 -- 172.21.15.104:0/228393156 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c09fd20 con 0x55ad57a1f7a0 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.487+0000 7fb12f7fe700 1 -- 172.21.15.104:0/228393156 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb13c0a45e0 con 0x55ad57a1f7a0 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.487+0000 7fb12f7fe700 1 -- 172.21.15.104:0/228393156 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c09be30 con 0x55ad57a1f7a0 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.487+0000 7fb156177b80 1 -- 172.21.15.104:0/228393156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 msgr2=0x55ad57a1fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.487+0000 7fb156177b80 1 --2- 172.21.15.104:0/228393156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 0x55ad57a1fbc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0ccba0 tx=0x7fb13c007c10 comp rx=0 tx=0).stop 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.487+0000 7fb156177b80 1 -- 172.21.15.104:0/228393156 shutdown_connections 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.487+0000 7fb156177b80 1 --2- 172.21.15.104:0/228393156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 0x55ad57a1fbc0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.487+0000 7fb156177b80 1 --2- 172.21.15.104:0/228393156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a24f60 0x55ad57a25340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.487+0000 7fb156177b80 1 --2- 172.21.15.104:0/228393156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a25880 0x55ad57a1f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.487+0000 7fb156177b80 1 -- 172.21.15.104:0/228393156 >> 172.21.15.104:0/228393156 conn(0x55ad5794c080 msgr2=0x55ad57a1a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.487+0000 7fb156177b80 1 -- 172.21.15.104:0/228393156 shutdown_connections 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.487+0000 7fb156177b80 1 -- 172.21.15.104:0/228393156 wait complete. 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.488+0000 7fb156177b80 1 -- start start 2024-06-09T22:25:36.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.488+0000 7fb156177b80 1 Processor -- start 2024-06-09T22:25:36.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.488+0000 7fb156177b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 0x55ad57a2b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.488+0000 7fb156177b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a24f60 0x55ad57a28970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.488+0000 7fb156177b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a25880 0x55ad57a28eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.488+0000 7fb156177b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55ad57910fc0 con 0x55ad57a1f7a0 2024-06-09T22:25:36.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.488+0000 7fb156177b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ad5796c810 con 0x55ad57a24f60 2024-06-09T22:25:36.670 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:25:36.671 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-09T22:25:36.671 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": { 2024-06-09T22:25:36.671 INFO:teuthology.orchestra.run.smithi104.stdout: "2ca81206-bae8-4b07-8e88-a9a09aee077e": { 2024-06-09T22:25:36.671 INFO:teuthology.orchestra.run.smithi104.stdout: "path": "/", 2024-06-09T22:25:36.671 INFO:teuthology.orchestra.run.smithi104.stdout: "tag": "2ca81206-bae8-4b07-8e88-a9a09aee077e", 2024-06-09T22:25:36.672 INFO:teuthology.orchestra.run.smithi104.stdout: "options": "recursive,force" 2024-06-09T22:25:36.672 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:25:36.672 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:25:36.672 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:25:36.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.488+0000 7fb156177b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ad57a2c260 con 0x55ad57a25880 2024-06-09T22:25:36.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.488+0000 7fb156177b80 10 client.0 _ll_register_callbacks cb 0x55ad579065b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:36.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.488+0000 7fb149690700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 0x55ad57a2b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.488+0000 7fb149690700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 0x55ad57a2b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47304/0 (socket says 172.21.15.104:47304) 2024-06-09T22:25:36.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.488+0000 7fb149690700 1 -- 172.21.15.104:0/1439817500 learned_addr learned my addr 172.21.15.104:0/1439817500 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:36.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb148e8f700 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a24f60 0x55ad57a28970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb149e91700 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a25880 0x55ad57a28eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb149690700 1 -- 172.21.15.104:0/1439817500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a25880 msgr2=0x55ad57a28eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb149690700 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a25880 0x55ad57a28eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb149690700 1 -- 172.21.15.104:0/1439817500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a24f60 msgr2=0x55ad57a28970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb149690700 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a24f60 0x55ad57a28970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb149690700 1 -- 172.21.15.104:0/1439817500 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad57936750 con 0x55ad57a1f7a0 2024-06-09T22:25:36.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb149690700 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 0x55ad57a2b670 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fb1400116f0 tx=0x7fb140004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc25de5d63bf816c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:36.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb12cff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:36.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb12cff9700 1 -- 172.21.15.104:0/1439817500 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140027600 con 0x55ad57a1f7a0 2024-06-09T22:25:36.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb12cff9700 1 -- 172.21.15.104:0/1439817500 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb14001f3f0 con 0x55ad57a1f7a0 2024-06-09T22:25:36.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb156177b80 1 -- 172.21.15.104:0/1439817500 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb13c0a45e0 con 0x55ad57a1f7a0 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb12cff9700 1 -- 172.21.15.104:0/1439817500 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1400277a0 con 0x55ad57a1f7a0 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb156177b80 10 client.16797 fetch_fsmap learned FSMap version 32 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb156177b80 1 -- 172.21.15.104:0/1439817500 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ad57a36a30 con 0x55ad57a1f7a0 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.489+0000 7fb12cff9700 1 -- 172.21.15.104:0/1439817500 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb13c0a45e0 con 0x55ad57a1f7a0 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.490+0000 7fb12cff9700 1 -- 172.21.15.104:0/1439817500 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb14000f040 con 0x55ad57a1f7a0 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.490+0000 7fb156177b80 10 client.16797 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.490+0000 7fb156177b80 10 client.16797 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.490+0000 7fb156177b80 20 client.16797 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.490+0000 7fb156177b80 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ad57a39070 0x55ad57a3b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.490+0000 7fb156177b80 4 client.16797 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.490+0000 7fb156177b80 1 -- 172.21.15.104:0/1439817500 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad5797dd50 con 0x55ad57a39070 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.490+0000 7fb148e8f700 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ad57a39070 0x55ad57a3b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.491+0000 7fb12cff9700 10 client.16797 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.491+0000 7fb148e8f700 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ad57a39070 0x55ad57a3b460 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2734f5e76f6073c server_cookie=8026fcd97eaa70b8 in_seq=0 out_seq=0 2024-06-09T22:25:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.491+0000 7fb12cff9700 1 -- 172.21.15.104:0/1439817500 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55ad5797dd50 con 0x55ad57a39070 2024-06-09T22:25:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.491+0000 7fb12cff9700 10 client.16797 handle_command_reply: tid=0 2024-06-09T22:25:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.491+0000 7fb156177b80 1 client.16797 shutdown 2024-06-09T22:25:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.491+0000 7fb156177b80 1 -- 172.21.15.104:0/1439817500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 msgr2=0x55ad57a2b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.491+0000 7fb156177b80 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 0x55ad57a2b670 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fb1400116f0 tx=0x7fb140004750 comp rx=0 tx=0).stop 2024-06-09T22:25:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.492+0000 7fb156177b80 1 -- 172.21.15.104:0/1439817500 shutdown_connections 2024-06-09T22:25:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.492+0000 7fb156177b80 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ad57a39070 0x55ad57a3b460 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.492+0000 7fb156177b80 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a1f7a0 0x55ad57a2b670 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.492+0000 7fb156177b80 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a25880 0x55ad57a28eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.492+0000 7fb12cff9700 0 client.16797 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:36.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.492+0000 7fb156177b80 1 --2- 172.21.15.104:0/1439817500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a24f60 0x55ad57a28970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.492+0000 7fb156177b80 1 -- 172.21.15.104:0/1439817500 >> 172.21.15.104:0/1439817500 conn(0x55ad5794c080 msgr2=0x55ad57a19700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.492+0000 7fb156177b80 1 -- 172.21.15.104:0/1439817500 shutdown_connections 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.492+0000 7fb156177b80 1 -- 172.21.15.104:0/1439817500 wait complete. 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.492+0000 7fb156177b80 20 client.16797 trim_cache size 0 max 16384 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.512+0000 7fb156177b80 1 Processor -- start 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb156177b80 1 -- start start 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb156177b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a1f7a0 0x55ad57a35380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb156177b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 0x55ad57a2b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb156177b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a25880 0x55ad57a19e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb156177b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb1441b07b0 con 0x55ad57a24f60 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb156177b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1441b22f0 con 0x55ad57a25880 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb156177b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1441b0e80 con 0x55ad57a1f7a0 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb148e8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 0x55ad57a2b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb148e8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 0x55ad57a2b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47320/0 (socket says 172.21.15.104:47320) 2024-06-09T22:25:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb148e8f700 1 -- 172.21.15.104:0/102598731 learned_addr learned my addr 172.21.15.104:0/102598731 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb149690700 1 --2- 172.21.15.104:0/102598731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a1f7a0 0x55ad57a35380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb148e8f700 1 -- 172.21.15.104:0/102598731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a1f7a0 msgr2=0x55ad57a35380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb148e8f700 1 --2- 172.21.15.104:0/102598731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a1f7a0 0x55ad57a35380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb149e91700 1 --2- 172.21.15.104:0/102598731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a25880 0x55ad57a19e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb148e8f700 1 -- 172.21.15.104:0/102598731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a25880 msgr2=0x55ad57a19e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb148e8f700 1 --2- 172.21.15.104:0/102598731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a25880 0x55ad57a19e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb148e8f700 1 -- 172.21.15.104:0/102598731 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad57a20120 con 0x55ad57a24f60 2024-06-09T22:25:36.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb148e8f700 1 --2- 172.21.15.104:0/102598731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 0x55ad57a2b1b0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fb134006010 tx=0x7fb134009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d99e87ed876542dd server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.513+0000 7fb12effd700 1 -- 172.21.15.104:0/102598731 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1340050b0 con 0x55ad57a24f60 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.514+0000 7fb12effd700 1 -- 172.21.15.104:0/102598731 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb134029070 con 0x55ad57a24f60 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.514+0000 7fb12effd700 1 -- 172.21.15.104:0/102598731 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13402f480 con 0x55ad57a24f60 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.514+0000 7fb156177b80 1 -- 172.21.15.104:0/102598731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 msgr2=0x55ad57a2b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.514+0000 7fb156177b80 1 --2- 172.21.15.104:0/102598731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 0x55ad57a2b1b0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fb134006010 tx=0x7fb134009310 comp rx=0 tx=0).stop 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.514+0000 7fb156177b80 1 -- 172.21.15.104:0/102598731 shutdown_connections 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.514+0000 7fb156177b80 1 --2- 172.21.15.104:0/102598731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a25880 0x55ad57a19e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.514+0000 7fb156177b80 1 --2- 172.21.15.104:0/102598731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a1f7a0 0x55ad57a35380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.514+0000 7fb156177b80 1 --2- 172.21.15.104:0/102598731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 0x55ad57a2b1b0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.514+0000 7fb156177b80 1 -- 172.21.15.104:0/102598731 >> 172.21.15.104:0/102598731 conn(0x55ad5794c080 msgr2=0x55ad57a242e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.514+0000 7fb156177b80 1 -- 172.21.15.104:0/102598731 shutdown_connections 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.514+0000 7fb156177b80 1 -- 172.21.15.104:0/102598731 wait complete. 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb156177b80 1 -- start start 2024-06-09T22:25:36.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb156177b80 1 Processor -- start 2024-06-09T22:25:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb156177b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a1f7a0 0x55ad57a40b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb156177b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 0x55ad57a41080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb156177b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a25880 0x55ad57a415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb156177b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb1441b1c20 con 0x55ad57a24f60 2024-06-09T22:25:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb156177b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1441b1550 con 0x55ad57a1f7a0 2024-06-09T22:25:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb156177b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1441ae730 con 0x55ad57a25880 2024-06-09T22:25:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb156177b80 10 client.0 _ll_register_callbacks cb 0x55ad579065b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb148e8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 0x55ad57a41080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb148e8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 0x55ad57a41080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47330/0 (socket says 172.21.15.104:47330) 2024-06-09T22:25:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb148e8f700 1 -- 172.21.15.104:0/1321952356 learned_addr learned my addr 172.21.15.104:0/1321952356 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb149690700 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a1f7a0 0x55ad57a40b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb149e91700 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a25880 0x55ad57a415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb148e8f700 1 -- 172.21.15.104:0/1321952356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a25880 msgr2=0x55ad57a415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb148e8f700 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a25880 0x55ad57a415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb148e8f700 1 -- 172.21.15.104:0/1321952356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a1f7a0 msgr2=0x55ad57a40b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:36.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb148e8f700 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a1f7a0 0x55ad57a40b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.515+0000 7fb148e8f700 1 -- 172.21.15.104:0/1321952356 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad57aa2eb0 con 0x55ad57a24f60 2024-06-09T22:25:36.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb148e8f700 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 0x55ad57a41080 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb134000c00 tx=0x7fb134009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4bad3083801d511 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:36.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb12cff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:36.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb12cff9700 1 -- 172.21.15.104:0/1321952356 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13400be80 con 0x55ad57a24f60 2024-06-09T22:25:36.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb156177b80 1 -- 172.21.15.104:0/1321952356 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb134029070 con 0x55ad57a24f60 2024-06-09T22:25:36.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb12cff9700 1 -- 172.21.15.104:0/1321952356 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb1340050c0 con 0x55ad57a24f60 2024-06-09T22:25:36.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb12cff9700 1 -- 172.21.15.104:0/1321952356 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb134040050 con 0x55ad57a24f60 2024-06-09T22:25:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb156177b80 10 client.16803 fetch_fsmap learned FSMap version 32 2024-06-09T22:25:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb156177b80 1 -- 172.21.15.104:0/1321952356 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ad57aa41a0 con 0x55ad57a24f60 2024-06-09T22:25:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb12cff9700 1 -- 172.21.15.104:0/1321952356 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb134029070 con 0x55ad57a24f60 2024-06-09T22:25:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb12cff9700 1 -- 172.21.15.104:0/1321952356 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb13402fbc0 con 0x55ad57a24f60 2024-06-09T22:25:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb156177b80 10 client.16803 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:25:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb156177b80 10 client.16803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb156177b80 20 client.16803 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb156177b80 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ad57aa4770 0x55ad57aa6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:36.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb156177b80 4 client.16803 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb156177b80 1 -- 172.21.15.104:0/1321952356 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ad579782d0 con 0x55ad57aa4770 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.516+0000 7fb149690700 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ad57aa4770 0x55ad57aa6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb12cff9700 10 client.16803 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb149690700 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ad57aa4770 0x55ad57aa6b90 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c02e88145d18e766 server_cookie=e5651f528e4602c9 in_seq=0 out_seq=0 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb12cff9700 1 -- 172.21.15.104:0/1321952356 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ad579782d0 con 0x55ad57aa4770 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb12cff9700 10 client.16803 handle_command_reply: tid=0 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb156177b80 1 client.16803 shutdown 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb156177b80 1 -- 172.21.15.104:0/1321952356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 msgr2=0x55ad57a41080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb156177b80 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 0x55ad57a41080 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb134000c00 tx=0x7fb134009fd0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb156177b80 1 -- 172.21.15.104:0/1321952356 shutdown_connections 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb156177b80 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ad57aa4770 0x55ad57aa6b90 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb156177b80 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ad57a25880 0x55ad57a415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb156177b80 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad57a1f7a0 0x55ad57a40b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb12cff9700 0 client.16803 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:36.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb156177b80 1 --2- 172.21.15.104:0/1321952356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ad57a24f60 0x55ad57a41080 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb156177b80 1 -- 172.21.15.104:0/1321952356 >> 172.21.15.104:0/1321952356 conn(0x55ad5794c080 msgr2=0x55ad57a281b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:36.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb156177b80 1 -- 172.21.15.104:0/1321952356 shutdown_connections 2024-06-09T22:25:36.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.517+0000 7fb156177b80 1 -- 172.21.15.104:0/1321952356 wait complete. 2024-06-09T22:25:36.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb156177b80 20 client.16803 trim_cache size 0 max 16384 2024-06-09T22:25:36.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb14b8f4700 1 -- 172.21.15.104:0/1111792292 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1300a2d90 msgr2=0x7fb1300a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb14b8f4700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1300a2d90 0x7fb1300a5250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb140009b50 tx=0x7fb140009550 comp rx=0 tx=0).stop 2024-06-09T22:25:36.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb14b8f4700 1 -- 172.21.15.104:0/1111792292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb144107400 msgr2=0x7fb144103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:36.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb14b8f4700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb144107400 0x7fb144103af0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb13c007d70 tx=0x7fb13c00a340 comp rx=0 tx=0).stop 2024-06-09T22:25:36.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb14b8f4700 1 -- 172.21.15.104:0/1111792292 shutdown_connections 2024-06-09T22:25:36.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb14b8f4700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1300a2d90 0x7fb1300a5250 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb14b8f4700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb144107400 0x7fb144103af0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb14b8f4700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb144101390 0x7fb144103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb14b8f4700 1 --2- 172.21.15.104:0/1111792292 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb144101d50 0x7fb1441035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:36.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb14b8f4700 1 -- 172.21.15.104:0/1111792292 >> 172.21.15.104:0/1111792292 conn(0x7fb144078340 msgr2=0x7fb1441004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:36.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb14b8f4700 1 -- 172.21.15.104:0/1111792292 shutdown_connections 2024-06-09T22:25:36.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:36.518+0000 7fb14b8f4700 1 -- 172.21.15.104:0/1111792292 wait complete. 2024-06-09T22:25:36.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 3.18 MB/sec execute 42 sec latency 3176.344 ms 2024-06-09T22:25:37.098 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:36 smithi104 ceph-mon[32389]: pgmap v142: 129 pgs: 129 active+clean; 74 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 546 KiB/s wr, 67 op/s 2024-06-09T22:25:37.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:36 smithi153 ceph-mon[36599]: pgmap v142: 129 pgs: 129 active+clean; 74 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 546 KiB/s wr, 67 op/s 2024-06-09T22:25:37.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:36 smithi194 ceph-mon[36856]: pgmap v142: 129 pgs: 129 active+clean; 74 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 546 KiB/s wr, 67 op/s 2024-06-09T22:25:37.527 INFO:tasks.cephfs.filesystem:scrub status for tag:2ca81206-bae8-4b07-8e88-a9a09aee077e - {'path': '/', 'tag': '2ca81206-bae8-4b07-8e88-a9a09aee077e', 'options': 'recursive,force'} 2024-06-09T22:25:37.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 3.11 MB/sec execute 43 sec latency 4176.395 ms 2024-06-09T22:25:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 3.04 MB/sec execute 44 sec latency 5176.470 ms 2024-06-09T22:25:39.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:38 smithi153 ceph-mon[36599]: pgmap v143: 129 pgs: 129 active+clean; 74 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 546 KiB/s wr, 67 op/s 2024-06-09T22:25:39.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:38 smithi104 ceph-mon[32389]: pgmap v143: 129 pgs: 129 active+clean; 74 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 546 KiB/s wr, 67 op/s 2024-06-09T22:25:39.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:38 smithi194 ceph-mon[36856]: pgmap v143: 129 pgs: 129 active+clean; 74 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 546 KiB/s wr, 67 op/s 2024-06-09T22:25:39.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 2.97 MB/sec execute 45 sec latency 6176.562 ms 2024-06-09T22:25:40.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 2.91 MB/sec execute 46 sec latency 7176.617 ms 2024-06-09T22:25:41.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:40 smithi153 ceph-mon[36599]: pgmap v144: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 804 KiB/s wr, 85 op/s 2024-06-09T22:25:41.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:40 smithi104 ceph-mon[32389]: pgmap v144: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 804 KiB/s wr, 85 op/s 2024-06-09T22:25:41.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:40 smithi194 ceph-mon[36856]: pgmap v144: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 804 KiB/s wr, 85 op/s 2024-06-09T22:25:41.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 2.85 MB/sec execute 47 sec latency 8176.760 ms 2024-06-09T22:25:42.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 2.79 MB/sec execute 48 sec latency 9176.909 ms 2024-06-09T22:25:43.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:42 smithi153 ceph-mon[36599]: pgmap v145: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 768 KiB/s wr, 73 op/s 2024-06-09T22:25:43.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:42 smithi104 ceph-mon[32389]: pgmap v145: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 768 KiB/s wr, 73 op/s 2024-06-09T22:25:43.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:42 smithi194 ceph-mon[36856]: pgmap v145: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 768 KiB/s wr, 73 op/s 2024-06-09T22:25:43.677 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:25:43.727 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 2.73 MB/sec execute 49 sec latency 10176.980 ms 2024-06-09T22:25:43.935 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:44.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 2.67 MB/sec execute 50 sec latency 11177.027 ms 2024-06-09T22:25:45.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:44 smithi104 ceph-mon[32389]: pgmap v146: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 1.0 MiB/s wr, 87 op/s 2024-06-09T22:25:45.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:44 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:25:45.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:44 smithi153 ceph-mon[36599]: pgmap v146: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 1.0 MiB/s wr, 87 op/s 2024-06-09T22:25:45.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:44 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:25:45.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:44 smithi194 ceph-mon[36856]: pgmap v146: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 1.0 MiB/s wr, 87 op/s 2024-06-09T22:25:45.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:44 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:25:45.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.663+0000 7fe3bd752700 1 -- 172.21.15.104:0/3593852150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3b81066b0 msgr2=0x7fe3b810e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:45.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.663+0000 7fe3bd752700 1 --2- 172.21.15.104:0/3593852150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3b81066b0 0x7fe3b810e280 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000b0d0 tx=0x7fe3b000b3e0 comp rx=0 tx=0).stop 2024-06-09T22:25:45.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.663+0000 7fe3bd752700 1 -- 172.21.15.104:0/3593852150 shutdown_connections 2024-06-09T22:25:45.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.663+0000 7fe3bd752700 1 --2- 172.21.15.104:0/3593852150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3b81066b0 0x7fe3b810e280 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:45.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.663+0000 7fe3bd752700 1 --2- 172.21.15.104:0/3593852150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b8101620 0x7fe3b8101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:45.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.663+0000 7fe3bd752700 1 --2- 172.21.15.104:0/3593852150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8101fd0 0x7fe3b8105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:45.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.663+0000 7fe3bd752700 1 -- 172.21.15.104:0/3593852150 >> 172.21.15.104:0/3593852150 conn(0x7fe3b8078340 msgr2=0x7fe3b8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:45.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.663+0000 7fe3bd752700 1 -- 172.21.15.104:0/3593852150 shutdown_connections 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.663+0000 7fe3bd752700 1 -- 172.21.15.104:0/3593852150 wait complete. 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.663+0000 7fe3bd752700 1 Processor -- start 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3bd752700 1 -- start start 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3bd752700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b8101620 0x7fe3b819d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3bd752700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8101fd0 0x7fe3b819dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3bd752700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3b81066b0 0x7fe3b81979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3bd752700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b81116a0 con 0x7fe3b81066b0 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3bd752700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b8111520 con 0x7fe3b8101fd0 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3bd752700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b8111820 con 0x7fe3b8101620 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3b81066b0 0x7fe3b81979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3b81066b0 0x7fe3b81979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47346/0 (socket says 172.21.15.104:47346) 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b77fe700 1 -- 172.21.15.104:0/452801743 learned_addr learned my addr 172.21.15.104:0/452801743 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b67fc700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8101fd0 0x7fe3b819dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:45.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b6ffd700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b8101620 0x7fe3b819d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b77fe700 1 -- 172.21.15.104:0/452801743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b8101620 msgr2=0x7fe3b819d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b77fe700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b8101620 0x7fe3b819d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b77fe700 1 -- 172.21.15.104:0/452801743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8101fd0 msgr2=0x7fe3b819dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b77fe700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8101fd0 0x7fe3b819dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b77fe700 1 -- 172.21.15.104:0/452801743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b8198100 con 0x7fe3b81066b0 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b6ffd700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b8101620 0x7fe3b819d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b67fc700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8101fd0 0x7fe3b819dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.664+0000 7fe3b77fe700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3b81066b0 0x7fe3b81979a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0006070 tx=0x7fe3b0004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.665+0000 7fe39ffff700 1 -- 172.21.15.104:0/452801743 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0021070 con 0x7fe3b81066b0 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.665+0000 7fe3bd752700 1 -- 172.21.15.104:0/452801743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3b8198390 con 0x7fe3b81066b0 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.665+0000 7fe39ffff700 1 -- 172.21.15.104:0/452801743 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b0007df0 con 0x7fe3b81066b0 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.665+0000 7fe3bd752700 1 -- 172.21.15.104:0/452801743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3b81a46b0 con 0x7fe3b81066b0 2024-06-09T22:25:45.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.665+0000 7fe39ffff700 1 -- 172.21.15.104:0/452801743 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0005470 con 0x7fe3b81066b0 2024-06-09T22:25:45.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.666+0000 7fe3bd752700 1 -- 172.21.15.104:0/452801743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3b8069180 con 0x7fe3b81066b0 2024-06-09T22:25:45.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.667+0000 7fe39ffff700 1 -- 172.21.15.104:0/452801743 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fe3b0024070 con 0x7fe3b81066b0 2024-06-09T22:25:45.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.668+0000 7fe39ffff700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe3a00a2e10 0x7fe3a00a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:45.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.668+0000 7fe3b6ffd700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe3a00a2e10 0x7fe3a00a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:45.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.668+0000 7fe39ffff700 1 -- 172.21.15.104:0/452801743 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fe3b00d09d0 con 0x7fe3b81066b0 2024-06-09T22:25:45.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.668+0000 7fe3b6ffd700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe3a00a2e10 0x7fe3a00a52d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8005fd0 tx=0x7fe3a8005de0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:45.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.672+0000 7fe39ffff700 1 -- 172.21.15.104:0/452801743 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe3b009d310 con 0x7fe3b81066b0 2024-06-09T22:25:45.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 2.62 MB/sec execute 51 sec latency 12177.054 ms 2024-06-09T22:25:45.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.914+0000 7fe3bd752700 1 -- 172.21.15.104:0/452801743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3b8198ed0 con 0x7fe3b81066b0 2024-06-09T22:25:45.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.914+0000 7fe39ffff700 1 -- 172.21.15.104:0/452801743 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7fe3b009cc10 con 0x7fe3b81066b0 2024-06-09T22:25:45.916 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:25:45.916 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:25:45.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.916+0000 7fe3bd752700 1 -- 172.21.15.104:0/452801743 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe3a00a2e10 msgr2=0x7fe3a00a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:45.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.916+0000 7fe3bd752700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe3a00a2e10 0x7fe3a00a52d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8005fd0 tx=0x7fe3a8005de0 comp rx=0 tx=0).stop 2024-06-09T22:25:45.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.916+0000 7fe3bd752700 1 -- 172.21.15.104:0/452801743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3b81066b0 msgr2=0x7fe3b81979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:45.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.916+0000 7fe3bd752700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3b81066b0 0x7fe3b81979a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0006070 tx=0x7fe3b0004ef0 comp rx=0 tx=0).stop 2024-06-09T22:25:45.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.916+0000 7fe3bd752700 1 -- 172.21.15.104:0/452801743 shutdown_connections 2024-06-09T22:25:45.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.916+0000 7fe3bd752700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe3a00a2e10 0x7fe3a00a52d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:45.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.916+0000 7fe3bd752700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3b81066b0 0x7fe3b81979a0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:45.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.916+0000 7fe3bd752700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3b8101620 0x7fe3b819d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:45.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.916+0000 7fe3bd752700 1 --2- 172.21.15.104:0/452801743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8101fd0 0x7fe3b819dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:45.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.916+0000 7fe3bd752700 1 -- 172.21.15.104:0/452801743 >> 172.21.15.104:0/452801743 conn(0x7fe3b8078340 msgr2=0x7fe3b80ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:45.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.917+0000 7fe3bd752700 1 -- 172.21.15.104:0/452801743 shutdown_connections 2024-06-09T22:25:45.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:45.917+0000 7fe3bd752700 1 -- 172.21.15.104:0/452801743 wait complete. 2024-06-09T22:25:45.919 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:25:46.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:45 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:25:46.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:45 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:25:46.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:45 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:25:46.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:45 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:25:46.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:45 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:25:46.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:45 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:25:46.676 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24460 quiesce db --set-id 4224fc39 --release --await' 2024-06-09T22:25:46.676 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24460 quiesce db --set-id 4224fc39 --release --await 2024-06-09T22:25:46.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 2.57 MB/sec execute 52 sec latency 13177.097 ms 2024-06-09T22:25:46.932 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:25:47.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:46 smithi104 ceph-mon[32389]: pgmap v147: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 514 KiB/s wr, 32 op/s 2024-06-09T22:25:47.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:46 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/452801743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:25:47.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:46 smithi153 ceph-mon[36599]: pgmap v147: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 514 KiB/s wr, 32 op/s 2024-06-09T22:25:47.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:46 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/452801743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:25:47.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:46 smithi194 ceph-mon[36856]: pgmap v147: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 514 KiB/s wr, 32 op/s 2024-06-09T22:25:47.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:46 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/452801743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:25:47.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 2.52 MB/sec execute 53 sec latency 14177.137 ms 2024-06-09T22:25:48.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 96540 2.48 MB/sec execute 54 sec latency 15177.176 ms 2024-06-09T22:25:48.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.764+0000 7f4f9774b700 1 -- 172.21.15.104:0/974794997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f90107480 msgr2=0x7f4f90107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:48.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.764+0000 7f4f9774b700 1 --2- 172.21.15.104:0/974794997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f90107480 0x7f4f90107860 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009990 tx=0x7f4f8c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.764+0000 7f4f9774b700 1 -- 172.21.15.104:0/974794997 shutdown_connections 2024-06-09T22:25:48.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.764+0000 7f4f9774b700 1 --2- 172.21.15.104:0/974794997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f90107480 0x7f4f90107860 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.764+0000 7f4f9774b700 1 --2- 172.21.15.104:0/974794997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f901024d0 0x7f4f9010e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.764+0000 7f4f9774b700 1 --2- 172.21.15.104:0/974794997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f90101b10 0x7f4f90101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.764+0000 7f4f9774b700 1 -- 172.21.15.104:0/974794997 >> 172.21.15.104:0/974794997 conn(0x7f4f90077990 msgr2=0x7f4f900fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:48.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.764+0000 7f4f9774b700 1 -- 172.21.15.104:0/974794997 shutdown_connections 2024-06-09T22:25:48.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.764+0000 7f4f9774b700 1 -- 172.21.15.104:0/974794997 wait complete. 2024-06-09T22:25:48.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f9774b700 1 Processor -- start 2024-06-09T22:25:48.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f9774b700 1 -- start start 2024-06-09T22:25:48.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f9774b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f90101b10 0x7f4f90103040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f9774b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f901024d0 0x7f4f90103580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f9774b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f90107480 0x7f4f90103ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f9774b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4f90111740 con 0x7f4f90107480 2024-06-09T22:25:48.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f9774b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901115c0 con 0x7f4f90101b10 2024-06-09T22:25:48.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f9774b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901118c0 con 0x7f4f901024d0 2024-06-09T22:25:48.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f95ce8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f90107480 0x7f4f90103ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f94ce6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f901024d0 0x7f4f90103580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f95ce8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f90107480 0x7f4f90103ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55208/0 (socket says 172.21.15.104:55208) 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f94ce6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f901024d0 0x7f4f90103580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:54096/0 (socket says 172.21.15.104:54096) 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f95ce8700 1 -- 172.21.15.104:0/582141608 learned_addr learned my addr 172.21.15.104:0/582141608 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.765+0000 7f4f954e7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f90101b10 0x7f4f90103040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.766+0000 7f4f95ce8700 1 -- 172.21.15.104:0/582141608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f901024d0 msgr2=0x7f4f90103580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.766+0000 7f4f95ce8700 1 --2- 172.21.15.104:0/582141608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f901024d0 0x7f4f90103580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.766+0000 7f4f95ce8700 1 -- 172.21.15.104:0/582141608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f90101b10 msgr2=0x7f4f90103040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.766+0000 7f4f95ce8700 1 --2- 172.21.15.104:0/582141608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f90101b10 0x7f4f90103040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.766+0000 7f4f95ce8700 1 -- 172.21.15.104:0/582141608 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f901a7120 con 0x7f4f90107480 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.766+0000 7f4f95ce8700 1 --2- 172.21.15.104:0/582141608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f90107480 0x7f4f90103ac0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4f8800b9e0 tx=0x7f4f8800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.766+0000 7f4f867fc700 1 -- 172.21.15.104:0/582141608 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8800d820 con 0x7f4f90107480 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.766+0000 7f4f9774b700 1 -- 172.21.15.104:0/582141608 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f901a7410 con 0x7f4f90107480 2024-06-09T22:25:48.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.766+0000 7f4f867fc700 1 -- 172.21.15.104:0/582141608 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f88015420 con 0x7f4f90107480 2024-06-09T22:25:48.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.766+0000 7f4f867fc700 1 -- 172.21.15.104:0/582141608 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88014630 con 0x7f4f90107480 2024-06-09T22:25:48.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.766+0000 7f4f9774b700 1 -- 172.21.15.104:0/582141608 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f901a7900 con 0x7f4f90107480 2024-06-09T22:25:48.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.768+0000 7f4f867fc700 1 -- 172.21.15.104:0/582141608 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f4f880155d0 con 0x7f4f90107480 2024-06-09T22:25:48.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.769+0000 7f4f867fc700 1 --2- 172.21.15.104:0/582141608 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4f7c0a2d90 0x7f4f7c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.769+0000 7f4f954e7700 1 --2- 172.21.15.104:0/582141608 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4f7c0a2d90 0x7f4f7c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.769+0000 7f4f867fc700 1 -- 172.21.15.104:0/582141608 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f4f880cf050 con 0x7f4f90107480 2024-06-09T22:25:48.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.770+0000 7f4f954e7700 1 --2- 172.21.15.104:0/582141608 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4f7c0a2d90 0x7f4f7c0a5250 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00b5c0 tx=0x7f4f8c005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:48.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.770+0000 7f4fa1fceb80 1 Processor -- start 2024-06-09T22:25:48.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.770+0000 7f4fa1fceb80 1 -- start start 2024-06-09T22:25:48.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.770+0000 7f4fa1fceb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb2c40 0x5602a1bb3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.770+0000 7f4fa1fceb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb3560 0x5602a1bace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.770+0000 7f4fa1fceb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bad3d0 0x5602a1bad810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.770+0000 7f4fa1fceb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5602a1afe910 con 0x5602a1bb2c40 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.770+0000 7f4fa1fceb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5602a1aa6820 con 0x5602a1bad3d0 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.770+0000 7f4fa1fceb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5602a1afa8d0 con 0x5602a1bb3560 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.770+0000 7f4f954e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb2c40 0x5602a1bb3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.770+0000 7f4f954e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb2c40 0x5602a1bb3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55224/0 (socket says 172.21.15.104:55224) 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.770+0000 7f4f954e7700 1 -- 172.21.15.104:0/754903130 learned_addr learned my addr 172.21.15.104:0/754903130 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.771+0000 7f4f94ce6700 1 --2- 172.21.15.104:0/754903130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb3560 0x5602a1bace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.771+0000 7f4f95ce8700 1 --2- 172.21.15.104:0/754903130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bad3d0 0x5602a1bad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.771+0000 7f4f954e7700 1 -- 172.21.15.104:0/754903130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb3560 msgr2=0x5602a1bace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.771+0000 7f4f954e7700 1 --2- 172.21.15.104:0/754903130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb3560 0x5602a1bace90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.771+0000 7f4f954e7700 1 -- 172.21.15.104:0/754903130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bad3d0 msgr2=0x5602a1bad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:48.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.771+0000 7f4f954e7700 1 --2- 172.21.15.104:0/754903130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bad3d0 0x5602a1bad810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.771+0000 7f4f954e7700 1 -- 172.21.15.104:0/754903130 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602a1a9d050 con 0x5602a1bb2c40 2024-06-09T22:25:48.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.771+0000 7f4f954e7700 1 --2- 172.21.15.104:0/754903130 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb2c40 0x5602a1bb3020 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c015000 tx=0x7f4f8c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=973b165d7388b632 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:48.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.771+0000 7f4f7b7fe700 1 -- 172.21.15.104:0/754903130 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c00bda0 con 0x5602a1bb2c40 2024-06-09T22:25:48.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.771+0000 7f4f7b7fe700 1 -- 172.21.15.104:0/754903130 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f8c00fd50 con 0x5602a1bb2c40 2024-06-09T22:25:48.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.771+0000 7f4f7b7fe700 1 -- 172.21.15.104:0/754903130 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c02f400 con 0x5602a1bb2c40 2024-06-09T22:25:48.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.772+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/754903130 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb2c40 msgr2=0x5602a1bb3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:48.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.772+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/754903130 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb2c40 0x5602a1bb3020 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c015000 tx=0x7f4f8c020000 comp rx=0 tx=0).stop 2024-06-09T22:25:48.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.772+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/754903130 shutdown_connections 2024-06-09T22:25:48.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.772+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/754903130 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb2c40 0x5602a1bb3020 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.772+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/754903130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bad3d0 0x5602a1bad810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.772+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/754903130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb3560 0x5602a1bace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.772+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/754903130 >> 172.21.15.104:0/754903130 conn(0x5602a1ad7dc0 msgr2=0x5602a1ba8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:48.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.772+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/754903130 shutdown_connections 2024-06-09T22:25:48.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.772+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/754903130 wait complete. 2024-06-09T22:25:48.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.772+0000 7f4fa1fceb80 1 -- start start 2024-06-09T22:25:48.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.772+0000 7f4fa1fceb80 1 Processor -- start 2024-06-09T22:25:48.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4fa1fceb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 0x5602a1bb9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4fa1fceb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb2c40 0x5602a1bb6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4fa1fceb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bb3560 0x5602a1bb6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4fa1fceb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5602a1bb9f10 con 0x5602a1bad3d0 2024-06-09T22:25:48.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4fa1fceb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5602a1a9ecf0 con 0x5602a1bb3560 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4fa1fceb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5602a1bba090 con 0x5602a1bb2c40 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4fa1fceb80 10 client.0 _ll_register_callbacks cb 0x5602a1a98a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4f954e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 0x5602a1bb9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4f954e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 0x5602a1bb9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55240/0 (socket says 172.21.15.104:55240) 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4f954e7700 1 -- 172.21.15.104:0/4180763840 learned_addr learned my addr 172.21.15.104:0/4180763840 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4f95ce8700 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bb3560 0x5602a1bb6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4f94ce6700 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb2c40 0x5602a1bb6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4f954e7700 1 -- 172.21.15.104:0/4180763840 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb2c40 msgr2=0x5602a1bb6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4f954e7700 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb2c40 0x5602a1bb6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4f954e7700 1 -- 172.21.15.104:0/4180763840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bb3560 msgr2=0x5602a1bb6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4f954e7700 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bb3560 0x5602a1bb6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.773+0000 7f4f954e7700 1 -- 172.21.15.104:0/4180763840 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602a1a4d9d0 con 0x5602a1bad3d0 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4f954e7700 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 0x5602a1bb9200 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c0249f0 tx=0x7f4f8c00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9b1be9d0219c3e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:48.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4f78ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4f78ff9700 1 -- 172.21.15.104:0/4180763840 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c00fa80 con 0x5602a1bad3d0 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4f78ff9700 1 -- 172.21.15.104:0/4180763840 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f4f8c02c050 con 0x5602a1bad3d0 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/4180763840 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f8c00fd50 con 0x5602a1bad3d0 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4f78ff9700 1 -- 172.21.15.104:0/4180763840 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4f8c024af0 con 0x5602a1bad3d0 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4fa1fceb80 10 client.16821 fetch_fsmap learned FSMap version 32 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/4180763840 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5602a1bc4770 con 0x5602a1bad3d0 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4f78ff9700 1 -- 172.21.15.104:0/4180763840 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f8c00fd50 con 0x5602a1bad3d0 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4f78ff9700 1 -- 172.21.15.104:0/4180763840 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4f8c037be0 con 0x5602a1bad3d0 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4fa1fceb80 10 client.16821 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4fa1fceb80 10 client.16821 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4fa1fceb80 20 client.16821 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5602a1bc6c40 0x5602a1bc9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4fa1fceb80 4 client.16821 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.774+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/4180763840 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602a1b0be10 con 0x5602a1bc6c40 2024-06-09T22:25:48.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.775+0000 7f4f94ce6700 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5602a1bc6c40 0x5602a1bc9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.775+0000 7f4f94ce6700 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5602a1bc6c40 0x5602a1bc9030 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51ba46ed72256395 server_cookie=f903b56b60ec4523 in_seq=0 out_seq=0 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.775+0000 7f4f78ff9700 10 client.16821 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4f78ff9700 1 -- 172.21.15.104:0/4180763840 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5602a1b0be10 con 0x5602a1bc6c40 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4f78ff9700 10 client.16821 handle_command_reply: tid=0 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4fa1fceb80 1 client.16821 shutdown 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/4180763840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 msgr2=0x5602a1bb9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 0x5602a1bb9200 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c0249f0 tx=0x7f4f8c00bfd0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/4180763840 shutdown_connections 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5602a1bc6c40 0x5602a1bc9030 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 0x5602a1bb9200 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bb3560 0x5602a1bb6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/4180763840 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb2c40 0x5602a1bb6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/4180763840 >> 172.21.15.104:0/4180763840 conn(0x5602a1ad7dc0 msgr2=0x5602a1ba7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4f78ff9700 0 client.16821 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/4180763840 shutdown_connections 2024-06-09T22:25:48.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/4180763840 wait complete. 2024-06-09T22:25:48.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.776+0000 7f4fa1fceb80 20 client.16821 trim_cache size 0 max 16384 2024-06-09T22:25:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.796+0000 7f4fa1fceb80 1 Processor -- start 2024-06-09T22:25:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.796+0000 7f4fa1fceb80 1 -- start start 2024-06-09T22:25:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.796+0000 7f4fa1fceb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 0x5602a1bace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.796+0000 7f4fa1fceb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bb2c40 0x5602a1bb53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.796+0000 7f4fa1fceb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb3560 0x5602a1bce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.796+0000 7f4fa1fceb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901b0770 con 0x5602a1bad3d0 2024-06-09T22:25:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.796+0000 7f4fa1fceb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901b22b0 con 0x5602a1bb2c40 2024-06-09T22:25:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.796+0000 7f4fa1fceb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901b0e40 con 0x5602a1bb3560 2024-06-09T22:25:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.796+0000 7f4f954e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 0x5602a1bace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.796+0000 7f4f954e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 0x5602a1bace90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55250/0 (socket says 172.21.15.104:55250) 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.796+0000 7f4f954e7700 1 -- 172.21.15.104:0/547305946 learned_addr learned my addr 172.21.15.104:0/547305946 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.797+0000 7f4f954e7700 1 -- 172.21.15.104:0/547305946 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb3560 msgr2=0x5602a1bce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.797+0000 7f4f95ce8700 1 --2- 172.21.15.104:0/547305946 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb3560 0x5602a1bce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.797+0000 7f4f94ce6700 1 --2- 172.21.15.104:0/547305946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bb2c40 0x5602a1bb53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.797+0000 7f4f954e7700 1 --2- 172.21.15.104:0/547305946 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb3560 0x5602a1bce5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.797+0000 7f4f954e7700 1 -- 172.21.15.104:0/547305946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bb2c40 msgr2=0x5602a1bb53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.797+0000 7f4f954e7700 1 --2- 172.21.15.104:0/547305946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bb2c40 0x5602a1bb53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.797+0000 7f4f954e7700 1 -- 172.21.15.104:0/547305946 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602a1ba8eb0 con 0x5602a1bad3d0 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.797+0000 7f4f94ce6700 1 --2- 172.21.15.104:0/547305946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bb2c40 0x5602a1bb53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.797+0000 7f4f954e7700 1 --2- 172.21.15.104:0/547305946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 0x5602a1bace90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c040400 tx=0x7f4f8c024a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77666a41ebfc3fcc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.797+0000 7f4f7affd700 1 -- 172.21.15.104:0/547305946 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c024c90 con 0x5602a1bad3d0 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.797+0000 7f4f7affd700 1 -- 172.21.15.104:0/547305946 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f8c0405f0 con 0x5602a1bad3d0 2024-06-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.797+0000 7f4f7affd700 1 -- 172.21.15.104:0/547305946 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c037400 con 0x5602a1bad3d0 2024-06-09T22:25:48.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/547305946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 msgr2=0x5602a1bace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:48.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/547305946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 0x5602a1bace90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c040400 tx=0x7f4f8c024a40 comp rx=0 tx=0).stop 2024-06-09T22:25:48.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/547305946 shutdown_connections 2024-06-09T22:25:48.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/547305946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bad3d0 0x5602a1bace90 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/547305946 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb3560 0x5602a1bce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/547305946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bb2c40 0x5602a1bb53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/547305946 >> 172.21.15.104:0/547305946 conn(0x5602a1ad7dc0 msgr2=0x5602a1bb4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/547305946 shutdown_connections 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/547305946 wait complete. 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 -- start start 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 Processor -- start 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bad3d0 0x5602a1bbf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb2c40 0x5602a1bbbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb3560 0x5602a1bbc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901b1be0 con 0x5602a1bb3560 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901b1510 con 0x5602a1bad3d0 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.798+0000 7f4fa1fceb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901ae6f0 con 0x5602a1bb2c40 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4fa1fceb80 10 client.0 _ll_register_callbacks cb 0x5602a1a98a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:48.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f95ce8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb3560 0x5602a1bbc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f95ce8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb3560 0x5602a1bbc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55254/0 (socket says 172.21.15.104:55254) 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f95ce8700 1 -- 172.21.15.104:0/2433003474 learned_addr learned my addr 172.21.15.104:0/2433003474 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f94ce6700 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb2c40 0x5602a1bbbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f954e7700 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bad3d0 0x5602a1bbf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f95ce8700 1 -- 172.21.15.104:0/2433003474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb2c40 msgr2=0x5602a1bbbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f95ce8700 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb2c40 0x5602a1bbbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f95ce8700 1 -- 172.21.15.104:0/2433003474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bad3d0 msgr2=0x5602a1bbf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f95ce8700 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bad3d0 0x5602a1bbf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f95ce8700 1 -- 172.21.15.104:0/2433003474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602a1bbc880 con 0x5602a1bb3560 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f954e7700 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bad3d0 0x5602a1bbf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f95ce8700 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb3560 0x5602a1bbc240 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4f880cc490 tx=0x7f4f88015d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82af9d11a44e31a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f78ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f78ff9700 1 -- 172.21.15.104:0/2433003474 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8809f370 con 0x5602a1bb3560 2024-06-09T22:25:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f78ff9700 1 -- 172.21.15.104:0/2433003474 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f4f8809b460 con 0x5602a1bb3560 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/2433003474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f8c0405f0 con 0x5602a1bb3560 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4f78ff9700 1 -- 172.21.15.104:0/2433003474 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4f880a3e70 con 0x5602a1bb3560 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4fa1fceb80 10 client.16827 fetch_fsmap learned FSMap version 32 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.799+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/2433003474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5602a1c323f0 con 0x5602a1bb3560 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.800+0000 7f4f78ff9700 1 -- 172.21.15.104:0/2433003474 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f8c0405f0 con 0x5602a1bb3560 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.800+0000 7f4f78ff9700 1 -- 172.21.15.104:0/2433003474 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4f880ccb80 con 0x5602a1bb3560 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.800+0000 7f4fa1fceb80 10 client.16827 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.800+0000 7f4fa1fceb80 10 client.16827 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.800+0000 7f4fa1fceb80 20 client.16827 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.800+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5602a1c32af0 0x5602a1c34f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.800+0000 7f4fa1fceb80 4 client.16827 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "4224fc39", "release": true, "await": true}] 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.800+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/2433003474 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "quiesce db", "set_id": "4224fc39", "release": true, "await": true}) -- 0x5602a1b06390 con 0x5602a1c32af0 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.800+0000 7f4f954e7700 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5602a1c32af0 0x5602a1c34f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:48.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.800+0000 7f4f78ff9700 10 client.16827 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:48.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.800+0000 7f4f954e7700 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5602a1c32af0 0x5602a1c34f10 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=665bf6bea5de16f server_cookie=a1fbd46ab08099a1 in_seq=0 out_seq=0 2024-06-09T22:25:48.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4f78ff9700 1 -- 172.21.15.104:0/2433003474 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5602a1b06390 con 0x5602a1c32af0 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4f78ff9700 10 client.16827 handle_command_reply: tid=0 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4fa1fceb80 1 client.16827 shutdown 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/2433003474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb3560 msgr2=0x5602a1bbc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb3560 0x5602a1bbc240 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4f880cc490 tx=0x7f4f88015d40 comp rx=0 tx=0).stop 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/2433003474 shutdown_connections 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5602a1c32af0 0x5602a1c34f10 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4f78ff9700 0 client.16827 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5602a1bb3560 0x5602a1bbc240 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5602a1bad3d0 0x5602a1bbf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4fa1fceb80 1 --2- 172.21.15.104:0/2433003474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602a1bb2c40 0x5602a1bbbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/2433003474 >> 172.21.15.104:0/2433003474 conn(0x5602a1ad7dc0 msgr2=0x5602a1bb85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.814+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/2433003474 shutdown_connections 2024-06-09T22:25:48.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4fa1fceb80 1 -- 172.21.15.104:0/2433003474 wait complete. 2024-06-09T22:25:48.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4fa1fceb80 20 client.16827 trim_cache size 0 max 16384 2024-06-09T22:25:48.817 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:25:48.817 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 32, 2024-06-09T22:25:48.817 INFO:teuthology.orchestra.run.smithi104.stdout: "leader": 24460, 2024-06-09T22:25:48.817 INFO:teuthology.orchestra.run.smithi104.stdout: "set_version": 12, 2024-06-09T22:25:48.817 INFO:teuthology.orchestra.run.smithi104.stdout: "sets": { 2024-06-09T22:25:48.817 INFO:teuthology.orchestra.run.smithi104.stdout: "4224fc39": { 2024-06-09T22:25:48.817 INFO:teuthology.orchestra.run.smithi104.stdout: "version": 12, 2024-06-09T22:25:48.817 INFO:teuthology.orchestra.run.smithi104.stdout: "age_ref": 0.0, 2024-06-09T22:25:48.817 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RELEASED", 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: "timeout": 90.0, 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: "expiration": 138.2, 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: "members": { 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: "file:/": { 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: "excluded": false, 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RELEASED", 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:25:48.818 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:25:48.819 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:25:48.819 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:25:48.819 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:25:48.819 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:25:48.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4f9774b700 1 -- 172.21.15.104:0/582141608 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4f7c0a2d90 msgr2=0x7f4f7c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:48.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4f9774b700 1 --2- 172.21.15.104:0/582141608 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4f7c0a2d90 0x7f4f7c0a5250 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00b5c0 tx=0x7f4f8c005960 comp rx=0 tx=0).stop 2024-06-09T22:25:48.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4f9774b700 1 -- 172.21.15.104:0/582141608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f90107480 msgr2=0x7f4f90103ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:48.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4f9774b700 1 --2- 172.21.15.104:0/582141608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f90107480 0x7f4f90103ac0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4f8800b9e0 tx=0x7f4f8800bcf0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4f9774b700 1 -- 172.21.15.104:0/582141608 shutdown_connections 2024-06-09T22:25:48.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4f9774b700 1 --2- 172.21.15.104:0/582141608 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4f7c0a2d90 0x7f4f7c0a5250 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4f9774b700 1 --2- 172.21.15.104:0/582141608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f90107480 0x7f4f90103ac0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4f9774b700 1 --2- 172.21.15.104:0/582141608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f90101b10 0x7f4f90103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4f9774b700 1 --2- 172.21.15.104:0/582141608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f901024d0 0x7f4f90103580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:48.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4f9774b700 1 -- 172.21.15.104:0/582141608 >> 172.21.15.104:0/582141608 conn(0x7f4f90077990 msgr2=0x7f4f900ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:48.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4f9774b700 1 -- 172.21.15.104:0/582141608 shutdown_connections 2024-06-09T22:25:48.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:25:48.815+0000 7f4f9774b700 1 -- 172.21.15.104:0/582141608 wait complete. 2024-06-09T22:25:49.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:48 smithi153 ceph-mon[36599]: pgmap v148: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 514 KiB/s wr, 32 op/s 2024-06-09T22:25:49.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:48 smithi104 ceph-mon[32389]: pgmap v148: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 514 KiB/s wr, 32 op/s 2024-06-09T22:25:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.259+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/1083096828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0ac1019b0 msgr2=0x7ff0ac101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.259+0000 7ff0b3c5d640 1 --2- 172.21.15.104:0/1083096828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0ac1019b0 0x7ff0ac101e30 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009930 tx=0x7ff0a802a420 comp rx=0 tx=0).stop 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.259+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/1083096828 shutdown_connections 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.259+0000 7ff0b3c5d640 1 --2- 172.21.15.104:0/1083096828 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0ac102370 0x7ff0ac10e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.259+0000 7ff0b3c5d640 1 --2- 172.21.15.104:0/1083096828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0ac1019b0 0x7ff0ac101e30 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.259+0000 7ff0b3c5d640 1 --2- 172.21.15.104:0/1083096828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0ac1079c0 0x7ff0ac107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.259+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/1083096828 >> 172.21.15.104:0/1083096828 conn(0x7ff0ac0fd460 msgr2=0x7ff0ac0ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.260+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/1083096828 shutdown_connections 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.260+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/1083096828 wait complete. 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.260+0000 7ff0b3c5d640 1 Processor -- start 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.260+0000 7ff0b3c5d640 1 -- start start 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.260+0000 7ff0b3c5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0ac1019b0 0x7ff0ac198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.260+0000 7ff0b3c5d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0ac102370 0x7ff0ac1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.260+0000 7ff0b3c5d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0ac1079c0 0x7ff0ac19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.260+0000 7ff0b3c5d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac111dd0 con 0x7ff0ac1079c0 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.260+0000 7ff0b3c5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac111c50 con 0x7ff0ac1019b0 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.260+0000 7ff0b3c5d640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac111f50 con 0x7ff0ac102370 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b21fa640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0ac1079c0 0x7ff0ac19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b19f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0ac1019b0 0x7ff0ac198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b19f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0ac1019b0 0x7ff0ac198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:53180/0 (socket says 172.21.15.104:53180) 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b19f9640 1 -- 172.21.15.104:0/382569577 learned_addr learned my addr 172.21.15.104:0/382569577 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b11f8640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0ac102370 0x7ff0ac1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b21fa640 1 -- 172.21.15.104:0/382569577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0ac102370 msgr2=0x7ff0ac1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b21fa640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0ac102370 0x7ff0ac1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b21fa640 1 -- 172.21.15.104:0/382569577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0ac1019b0 msgr2=0x7ff0ac198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b21fa640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0ac1019b0 0x7ff0ac198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b21fa640 1 -- 172.21.15.104:0/382569577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ac19d210 con 0x7ff0ac1079c0 2024-06-09T22:25:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b19f9640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0ac1019b0 0x7ff0ac198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b11f8640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0ac102370 0x7ff0ac1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b21fa640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0ac1079c0 0x7ff0ac19cae0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009fd0 tx=0x7ff0a400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0a2ffd640 1 -- 172.21.15.104:0/382569577 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a400ce20 con 0x7ff0ac1079c0 2024-06-09T22:25:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/382569577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0ac19d500 con 0x7ff0ac1079c0 2024-06-09T22:25:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/382569577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0ac1a4d90 con 0x7ff0ac1079c0 2024-06-09T22:25:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0a2ffd640 1 -- 172.21.15.104:0/382569577 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0a4010430 con 0x7ff0ac1079c0 2024-06-09T22:25:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.261+0000 7ff0a2ffd640 1 -- 172.21.15.104:0/382569577 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a400f620 con 0x7ff0ac1079c0 2024-06-09T22:25:49.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.264+0000 7ff0b696f740 1 Processor -- start 2024-06-09T22:25:49.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.264+0000 7ff0a2ffd640 1 -- 172.21.15.104:0/382569577 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ff0a400f7c0 con 0x7ff0ac1079c0 2024-06-09T22:25:49.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.264+0000 7ff0b696f740 1 -- start start 2024-06-09T22:25:49.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.264+0000 7ff0b696f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3f170 0x56252cf3f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b696f740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3fa90 0x56252cf49e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b696f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf4a350 0x56252cf4c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b696f740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56252cea34f0 con 0x56252cf3fa90 2024-06-09T22:25:49.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b696f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56252ce41870 con 0x56252cf3f170 2024-06-09T22:25:49.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b696f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56252ce9f4b0 con 0x56252cf4a350 2024-06-09T22:25:49.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b19f9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3fa90 0x56252cf49e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b11f8640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf4a350 0x56252cf4c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b19f9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3fa90 0x56252cf49e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55268/0 (socket says 172.21.15.104:55268) 2024-06-09T22:25:49.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b19f9640 1 -- 172.21.15.104:0/941609721 learned_addr learned my addr 172.21.15.104:0/941609721 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:49.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b11f8640 1 --2- 172.21.15.104:0/941609721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3f170 0x56252cf3f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b19f9640 1 -- 172.21.15.104:0/941609721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf4a350 msgr2=0x56252cf4c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:49.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b19f9640 1 --2- 172.21.15.104:0/941609721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf4a350 0x56252cf4c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b19f9640 1 -- 172.21.15.104:0/941609721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3f170 msgr2=0x56252cf3f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:49.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b19f9640 1 --2- 172.21.15.104:0/941609721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3f170 0x56252cf3f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b19f9640 1 -- 172.21.15.104:0/941609721 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56252ce5d9b0 con 0x56252cf3fa90 2024-06-09T22:25:49.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0a2ffd640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff0980a2da0 0x7ff0980a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b19f9640 1 --2- 172.21.15.104:0/941609721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3fa90 0x56252cf49e10 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0115c0 tx=0x7ff09c011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c297826cc78c90f4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:49.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0b11f8640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff0980a2da0 0x7ff0980a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff0a2ffd640 1 -- 172.21.15.104:0/382569577 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff0a40cfa10 con 0x7ff0ac1079c0 2024-06-09T22:25:49.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.265+0000 7ff097fff640 1 -- 172.21.15.104:0/941609721 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c00f5e0 con 0x56252cf3fa90 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.266+0000 7ff097fff640 1 -- 172.21.15.104:0/941609721 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff09c00fc60 con 0x56252cf3fa90 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.266+0000 7ff097fff640 1 -- 172.21.15.104:0/941609721 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c019cf0 con 0x56252cf3fa90 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.266+0000 7ff0b696f740 1 -- 172.21.15.104:0/941609721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3fa90 msgr2=0x56252cf49e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.266+0000 7ff0b696f740 1 --2- 172.21.15.104:0/941609721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3fa90 0x56252cf49e10 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0115c0 tx=0x7ff09c011a80 comp rx=0 tx=0).stop 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.266+0000 7ff0b11f8640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff0980a2da0 0x7ff0980a5260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009900 tx=0x7ff0a80058d0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.266+0000 7ff0b696f740 1 -- 172.21.15.104:0/941609721 shutdown_connections 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.266+0000 7ff0b696f740 1 --2- 172.21.15.104:0/941609721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf4a350 0x56252cf4c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.266+0000 7ff0b696f740 1 --2- 172.21.15.104:0/941609721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3fa90 0x56252cf49e10 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.266+0000 7ff0b696f740 1 --2- 172.21.15.104:0/941609721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3f170 0x56252cf3f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.266+0000 7ff0b696f740 1 -- 172.21.15.104:0/941609721 >> 172.21.15.104:0/941609721 conn(0x56252cf3b680 msgr2=0x56252cf3ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.266+0000 7ff0b696f740 1 -- 172.21.15.104:0/941609721 shutdown_connections 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b696f740 1 -- 172.21.15.104:0/941609721 wait complete. 2024-06-09T22:25:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b696f740 1 -- start start 2024-06-09T22:25:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b696f740 1 Processor -- start 2024-06-09T22:25:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b696f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3f170 0x56252cf47280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b696f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3fa90 0x56252cf477c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b696f740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 0x56252cf47d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b696f740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56252cf4cf80 con 0x56252cf4a350 2024-06-09T22:25:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b696f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56252ce2cae0 con 0x56252cf3fa90 2024-06-09T22:25:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b696f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56252cf4d100 con 0x56252cf3f170 2024-06-09T22:25:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b696f740 10 client.0 _ll_register_callbacks cb 0x56252ce659b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b19f9640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3f170 0x56252cf47280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b21fa640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 0x56252cf47d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b19f9640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3f170 0x56252cf47280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:54162/0 (socket says 172.21.15.104:54162) 2024-06-09T22:25:49.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b21fa640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 0x56252cf47d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55272/0 (socket says 172.21.15.104:55272) 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b19f9640 1 -- 172.21.15.104:0/1707917717 learned_addr learned my addr 172.21.15.104:0/1707917717 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.267+0000 7ff0b11f8640 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3fa90 0x56252cf477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0b19f9640 1 -- 172.21.15.104:0/1707917717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3fa90 msgr2=0x56252cf477c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0b19f9640 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3fa90 0x56252cf477c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0b19f9640 1 -- 172.21.15.104:0/1707917717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 msgr2=0x56252cf47d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0b19f9640 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 0x56252cf47d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0b19f9640 1 -- 172.21.15.104:0/1707917717 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56252cbb7210 con 0x56252cf3f170 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0b21fa640 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 0x56252cf47d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0b19f9640 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3f170 0x56252cf47280 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0198d0 tx=0x7ff09c009550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeb71fe43643a44f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0957fa640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0957fa640 1 -- 172.21.15.104:0/1707917717 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c00f5e0 con 0x56252cf3f170 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0b696f740 1 -- 172.21.15.104:0/1707917717 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff09c00fc60 con 0x56252cf3f170 2024-06-09T22:25:49.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0957fa640 1 -- 172.21.15.104:0/1707917717 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff09c00f780 con 0x56252cf3f170 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0957fa640 1 -- 172.21.15.104:0/1707917717 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff09c033050 con 0x56252cf3f170 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0b696f740 10 client.26242 fetch_fsmap learned FSMap version 32 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.268+0000 7ff0b696f740 1 -- 172.21.15.104:0/1707917717 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56252cc83620 con 0x56252cf3f170 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.269+0000 7ff0957fa640 1 -- 172.21.15.104:0/1707917717 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff09c00fc60 con 0x56252cf3f170 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.269+0000 7ff0957fa640 1 -- 172.21.15.104:0/1707917717 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff09c028020 con 0x56252cf3f170 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.269+0000 7ff0b696f740 10 client.26242 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.269+0000 7ff0b696f740 10 client.26242 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.269+0000 7ff0b696f740 20 client.26242 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.269+0000 7ff0b696f740 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56252cf5a340 0x56252cf5c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.269+0000 7ff0b696f740 4 client.26242 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.269+0000 7ff0b696f740 1 -- 172.21.15.104:0/1707917717 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56252ceb6f10 con 0x56252cf5a340 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.269+0000 7ff0b11f8640 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56252cf5a340 0x56252cf5c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.276 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.270+0000 7ff0957fa640 10 client.26242 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:49.277 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.270+0000 7ff0b11f8640 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56252cf5a340 0x56252cf5c730 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1902a3cc949559eb server_cookie=86941fc0b31304bb in_seq=0 out_seq=0 2024-06-09T22:25:49.278 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.270+0000 7ff0957fa640 1 -- 172.21.15.104:0/1707917717 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56252ceb6f10 con 0x56252cf5a340 2024-06-09T22:25:49.278 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.270+0000 7ff0957fa640 10 client.26242 handle_command_reply: tid=0 2024-06-09T22:25:49.278 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.270+0000 7ff0b696f740 1 client.26242 shutdown 2024-06-09T22:25:49.278 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.271+0000 7ff0b696f740 1 -- 172.21.15.104:0/1707917717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3f170 msgr2=0x56252cf47280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:49.278 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.271+0000 7ff0b696f740 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3f170 0x56252cf47280 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0198d0 tx=0x7ff09c009550 comp rx=0 tx=0).stop 2024-06-09T22:25:49.278 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.271+0000 7ff0b696f740 1 -- 172.21.15.104:0/1707917717 shutdown_connections 2024-06-09T22:25:49.278 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.271+0000 7ff0b696f740 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56252cf5a340 0x56252cf5c730 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.278 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.271+0000 7ff0b696f740 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 0x56252cf47d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.278 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.271+0000 7ff0b696f740 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3fa90 0x56252cf477c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.278 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.271+0000 7ff0b696f740 1 --2- 172.21.15.104:0/1707917717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3f170 0x56252cf47280 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.278 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.271+0000 7ff0957fa640 0 client.26242 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:49.278 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.271+0000 7ff0b696f740 1 -- 172.21.15.104:0/1707917717 >> 172.21.15.104:0/1707917717 conn(0x56252cf3b680 msgr2=0x56252cf41400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.271+0000 7ff0b696f740 1 -- 172.21.15.104:0/1707917717 shutdown_connections 2024-06-09T22:25:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.271+0000 7ff0b696f740 1 -- 172.21.15.104:0/1707917717 wait complete. 2024-06-09T22:25:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.271+0000 7ff0b696f740 20 client.26242 trim_cache size 0 max 16384 2024-06-09T22:25:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b696f740 1 Processor -- start 2024-06-09T22:25:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b696f740 1 -- start start 2024-06-09T22:25:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b696f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3f170 0x56252cf41aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b696f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3fa90 0x56252cf3a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b696f740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 0x56252cf3aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b696f740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff09000fe90 con 0x56252cf4a350 2024-06-09T22:25:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b696f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff0900119d0 con 0x56252cf3f170 2024-06-09T22:25:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b696f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff090010560 con 0x56252cf3fa90 2024-06-09T22:25:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b21fa640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 0x56252cf3aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b21fa640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 0x56252cf3aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55282/0 (socket says 172.21.15.104:55282) 2024-06-09T22:25:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b21fa640 1 -- 172.21.15.104:0/3226187765 learned_addr learned my addr 172.21.15.104:0/3226187765 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b11f8640 1 --2- 172.21.15.104:0/3226187765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3fa90 0x56252cf3a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b19f9640 1 --2- 172.21.15.104:0/3226187765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3f170 0x56252cf41aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b21fa640 1 -- 172.21.15.104:0/3226187765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3fa90 msgr2=0x56252cf3a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b21fa640 1 --2- 172.21.15.104:0/3226187765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3fa90 0x56252cf3a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b21fa640 1 -- 172.21.15.104:0/3226187765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3f170 msgr2=0x56252cf41aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b21fa640 1 --2- 172.21.15.104:0/3226187765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3f170 0x56252cf41aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.286+0000 7ff0b21fa640 1 -- 172.21.15.104:0/3226187765 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56252cf4ce60 con 0x56252cf4a350 2024-06-09T22:25:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.287+0000 7ff0b21fa640 1 --2- 172.21.15.104:0/3226187765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 0x56252cf3aba0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ff0a40cce50 tx=0x7ff0a409bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f97ffc3a9d305c8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.287+0000 7ff0977fe640 1 -- 172.21.15.104:0/3226187765 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a409b520 con 0x56252cf4a350 2024-06-09T22:25:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.287+0000 7ff0977fe640 1 -- 172.21.15.104:0/3226187765 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0a4098230 con 0x56252cf4a350 2024-06-09T22:25:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.287+0000 7ff0977fe640 1 -- 172.21.15.104:0/3226187765 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a400fab0 con 0x56252cf4a350 2024-06-09T22:25:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 -- 172.21.15.104:0/3226187765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 msgr2=0x56252cf3aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 --2- 172.21.15.104:0/3226187765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 0x56252cf3aba0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ff0a40cce50 tx=0x7ff0a409bd40 comp rx=0 tx=0).stop 2024-06-09T22:25:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 -- 172.21.15.104:0/3226187765 shutdown_connections 2024-06-09T22:25:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 --2- 172.21.15.104:0/3226187765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf4a350 0x56252cf3aba0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 --2- 172.21.15.104:0/3226187765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3fa90 0x56252cf3a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 --2- 172.21.15.104:0/3226187765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf3f170 0x56252cf41aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 -- 172.21.15.104:0/3226187765 >> 172.21.15.104:0/3226187765 conn(0x56252cf3b680 msgr2=0x56252cf39f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 -- 172.21.15.104:0/3226187765 shutdown_connections 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 -- 172.21.15.104:0/3226187765 wait complete. 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 -- start start 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 Processor -- start 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3f170 0x56252cf52940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3fa90 0x56252cf4dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf4a350 0x56252cf4e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff090011300 con 0x56252cf3f170 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.288+0000 7ff0b696f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff090010c30 con 0x56252cf4a350 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b696f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff09000de50 con 0x56252cf3fa90 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b696f740 10 client.0 _ll_register_callbacks cb 0x7ff0a8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b19f9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3f170 0x56252cf52940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b19f9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3f170 0x56252cf52940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55290/0 (socket says 172.21.15.104:55290) 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b11f8640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3fa90 0x56252cf4dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b19f9640 1 -- 172.21.15.104:0/435123018 learned_addr learned my addr 172.21.15.104:0/435123018 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b21fa640 1 --2- 172.21.15.104:0/435123018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf4a350 0x56252cf4e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b19f9640 1 -- 172.21.15.104:0/435123018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3fa90 msgr2=0x56252cf4dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b19f9640 1 --2- 172.21.15.104:0/435123018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3fa90 0x56252cf4dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b19f9640 1 -- 172.21.15.104:0/435123018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf4a350 msgr2=0x56252cf4e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b19f9640 1 --2- 172.21.15.104:0/435123018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf4a350 0x56252cf4e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b19f9640 1 -- 172.21.15.104:0/435123018 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56252cf4e860 con 0x56252cf3f170 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b19f9640 1 --2- 172.21.15.104:0/435123018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3f170 0x56252cf52940 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ff09c004910 tx=0x7ff09c004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609da3b69859619a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0957fa640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0957fa640 1 -- 172.21.15.104:0/435123018 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c011230 con 0x56252cf3f170 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0957fa640 1 -- 172.21.15.104:0/435123018 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff09c0113d0 con 0x56252cf3f170 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b696f740 1 -- 172.21.15.104:0/435123018 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0a4098230 con 0x56252cf3f170 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0957fa640 1 -- 172.21.15.104:0/435123018 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff09c004030 con 0x56252cf3f170 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b696f740 10 client.16845 fetch_fsmap learned FSMap version 32 2024-06-09T22:25:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0b696f740 1 -- 172.21.15.104:0/435123018 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56252cfc4e20 con 0x56252cf3f170 2024-06-09T22:25:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.289+0000 7ff0957fa640 1 -- 172.21.15.104:0/435123018 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0a4098230 con 0x56252cf3f170 2024-06-09T22:25:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.290+0000 7ff0957fa640 1 -- 172.21.15.104:0/435123018 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff09c028040 con 0x56252cf3f170 2024-06-09T22:25:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.290+0000 7ff0b696f740 10 client.16845 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:25:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.290+0000 7ff0b696f740 10 client.16845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:25:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.290+0000 7ff0b696f740 20 client.16845 populate_metadata read hostname 'smithi104' 2024-06-09T22:25:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.290+0000 7ff0b696f740 1 --2- 172.21.15.104:0/435123018 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56252cfc5480 0x56252cfc78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:25:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.290+0000 7ff0b696f740 4 client.16845 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:25:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.290+0000 7ff0b696f740 1 -- 172.21.15.104:0/435123018 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56252ceb0c20 con 0x56252cfc5480 2024-06-09T22:25:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.290+0000 7ff0b11f8640 1 --2- 172.21.15.104:0/435123018 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56252cfc5480 0x56252cfc78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:25:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.290+0000 7ff0b11f8640 1 --2- 172.21.15.104:0/435123018 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56252cfc5480 0x56252cfc78a0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b257cbee7abec291 server_cookie=b3b2d272f36473c2 in_seq=0 out_seq=0 2024-06-09T22:25:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.290+0000 7ff0957fa640 10 client.16845 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:49.300 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0957fa640 1 -- 172.21.15.104:0/435123018 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56252ceb0c20 con 0x56252cfc5480 2024-06-09T22:25:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0957fa640 10 client.16845 handle_command_reply: tid=0 2024-06-09T22:25:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0b696f740 1 client.16845 shutdown 2024-06-09T22:25:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0b696f740 1 -- 172.21.15.104:0/435123018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3f170 msgr2=0x56252cf52940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0b696f740 1 --2- 172.21.15.104:0/435123018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3f170 0x56252cf52940 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ff09c004910 tx=0x7ff09c004990 comp rx=0 tx=0).stop 2024-06-09T22:25:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0b696f740 1 -- 172.21.15.104:0/435123018 shutdown_connections 2024-06-09T22:25:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0b696f740 1 --2- 172.21.15.104:0/435123018 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56252cfc5480 0x56252cfc78a0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0b696f740 1 --2- 172.21.15.104:0/435123018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56252cf4a350 0x56252cf4e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0b696f740 1 --2- 172.21.15.104:0/435123018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56252cf3fa90 0x56252cf4dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0b696f740 1 --2- 172.21.15.104:0/435123018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56252cf3f170 0x56252cf52940 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0957fa640 0 client.16845 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:25:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0b696f740 1 -- 172.21.15.104:0/435123018 >> 172.21.15.104:0/435123018 conn(0x56252cf3b680 msgr2=0x56252cf3a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0b696f740 1 -- 172.21.15.104:0/435123018 shutdown_connections 2024-06-09T22:25:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0b696f740 1 -- 172.21.15.104:0/435123018 wait complete. 2024-06-09T22:25:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.299+0000 7ff0b696f740 20 client.16845 trim_cache size 0 max 16384 2024-06-09T22:25:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:25:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:25:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:25:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.300+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/382569577 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff0980a2da0 msgr2=0x7ff0980a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.300+0000 7ff0b3c5d640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff0980a2da0 0x7ff0980a5260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009900 tx=0x7ff0a80058d0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.300+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/382569577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0ac1079c0 msgr2=0x7ff0ac19cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:25:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.300+0000 7ff0b3c5d640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0ac1079c0 0x7ff0ac19cae0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009fd0 tx=0x7ff0a400ef10 comp rx=0 tx=0).stop 2024-06-09T22:25:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.300+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/382569577 shutdown_connections 2024-06-09T22:25:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.300+0000 7ff0b3c5d640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff0980a2da0 0x7ff0980a5260 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.300+0000 7ff0b3c5d640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0ac1079c0 0x7ff0ac19cae0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.300+0000 7ff0b3c5d640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0ac102370 0x7ff0ac1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.300+0000 7ff0b3c5d640 1 --2- 172.21.15.104:0/382569577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0ac1019b0 0x7ff0ac198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:25:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.300+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/382569577 >> 172.21.15.104:0/382569577 conn(0x7ff0ac0fd460 msgr2=0x7ff0ac0fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:25:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.300+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/382569577 shutdown_connections 2024-06-09T22:25:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:25:49.300+0000 7ff0b3c5d640 1 -- 172.21.15.104:0/382569577 wait complete. 2024-06-09T22:25:49.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:48 smithi194 ceph-mon[36856]: pgmap v148: 129 pgs: 129 active+clean; 73 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 514 KiB/s wr, 32 op/s 2024-06-09T22:25:49.725 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '4224fc39', total seconds elapsed: 24.3 2024-06-09T22:25:49.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 99012 2.67 MB/sec execute 55 sec latency 15266.805 ms 2024-06-09T22:25:50.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 100351 2.74 MB/sec execute 56 sec latency 16.501 ms 2024-06-09T22:25:51.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:50 smithi153 ceph-mon[36599]: pgmap v149: 129 pgs: 129 active+clean; 72 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 765 KiB/s wr, 47 op/s 2024-06-09T22:25:51.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:50 smithi104 ceph-mon[32389]: pgmap v149: 129 pgs: 129 active+clean; 72 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 765 KiB/s wr, 47 op/s 2024-06-09T22:25:51.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:50 smithi194 ceph-mon[36856]: pgmap v149: 129 pgs: 129 active+clean; 72 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 765 KiB/s wr, 47 op/s 2024-06-09T22:25:51.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 103155 3.01 MB/sec execute 57 sec latency 16.638 ms 2024-06-09T22:25:52.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 105246 3.08 MB/sec execute 58 sec latency 15.366 ms 2024-06-09T22:25:53.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:52 smithi153 ceph-mon[36599]: pgmap v150: 129 pgs: 129 active+clean; 72 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 507 KiB/s wr, 29 op/s 2024-06-09T22:25:53.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:52 smithi104 ceph-mon[32389]: pgmap v150: 129 pgs: 129 active+clean; 72 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 507 KiB/s wr, 29 op/s 2024-06-09T22:25:53.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:52 smithi194 ceph-mon[36856]: pgmap v150: 129 pgs: 129 active+clean; 72 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 507 KiB/s wr, 29 op/s 2024-06-09T22:25:53.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 106983 3.24 MB/sec execute 59 sec latency 19.978 ms 2024-06-09T22:25:54.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 109367 3.39 MB/sec execute 60 sec latency 27.802 ms 2024-06-09T22:25:55.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:54 smithi153 ceph-mon[36599]: pgmap v151: 129 pgs: 129 active+clean; 79 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-06-09T22:25:55.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:54 smithi104 ceph-mon[32389]: pgmap v151: 129 pgs: 129 active+clean; 79 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-06-09T22:25:55.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:54 smithi194 ceph-mon[36856]: pgmap v151: 129 pgs: 129 active+clean; 79 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-06-09T22:25:55.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 110874 3.45 MB/sec execute 61 sec latency 19.986 ms 2024-06-09T22:25:56.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 113757 3.69 MB/sec execute 62 sec latency 20.663 ms 2024-06-09T22:25:57.007 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:56 smithi104 ceph-mon[32389]: pgmap v152: 129 pgs: 129 active+clean; 79 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-06-09T22:25:57.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:56 smithi153 ceph-mon[36599]: pgmap v152: 129 pgs: 129 active+clean; 79 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-06-09T22:25:57.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:56 smithi194 ceph-mon[36856]: pgmap v152: 129 pgs: 129 active+clean; 79 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-06-09T22:25:57.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 115235 3.72 MB/sec execute 63 sec latency 22.896 ms 2024-06-09T22:25:58.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 117514 3.88 MB/sec execute 64 sec latency 13.772 ms 2024-06-09T22:25:59.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:25:58 smithi153 ceph-mon[36599]: pgmap v153: 129 pgs: 129 active+clean; 79 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-06-09T22:25:59.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:25:58 smithi104 ceph-mon[32389]: pgmap v153: 129 pgs: 129 active+clean; 79 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-06-09T22:25:59.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:25:58 smithi194 ceph-mon[36856]: pgmap v153: 129 pgs: 129 active+clean; 79 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-06-09T22:25:59.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 119652 3.94 MB/sec execute 65 sec latency 15.882 ms 2024-06-09T22:26:00.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 121322 4.05 MB/sec execute 66 sec latency 17.165 ms 2024-06-09T22:26:01.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:00 smithi153 ceph-mon[36599]: pgmap v154: 129 pgs: 129 active+clean; 86 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.2 MiB/s wr, 219 op/s 2024-06-09T22:26:01.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:00 smithi104 ceph-mon[32389]: pgmap v154: 129 pgs: 129 active+clean; 86 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.2 MiB/s wr, 219 op/s 2024-06-09T22:26:01.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:00 smithi194 ceph-mon[36856]: pgmap v154: 129 pgs: 129 active+clean; 86 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.2 MiB/s wr, 219 op/s 2024-06-09T22:26:01.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 121813 4.02 MB/sec execute 67 sec latency 92.263 ms 2024-06-09T22:26:02.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.392+0000 7ff1942a3640 1 -- 172.21.15.104:0/295807304 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff18c101100 msgr2=0x7ff18c10ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:02.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.392+0000 7ff1942a3640 1 --2- 172.21.15.104:0/295807304 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff18c101100 0x7ff18c10ddc0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009900 tx=0x7ff17c02a410 comp rx=0 tx=0).stop 2024-06-09T22:26:02.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.393+0000 7ff1942a3640 1 -- 172.21.15.104:0/295807304 shutdown_connections 2024-06-09T22:26:02.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.393+0000 7ff1942a3640 1 --2- 172.21.15.104:0/295807304 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff18c10e300 0x7ff18c1106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.393+0000 7ff1942a3640 1 --2- 172.21.15.104:0/295807304 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff18c101100 0x7ff18c10ddc0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.393+0000 7ff1942a3640 1 --2- 172.21.15.104:0/295807304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c100750 0x7ff18c100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.393+0000 7ff1942a3640 1 -- 172.21.15.104:0/295807304 >> 172.21.15.104:0/295807304 conn(0x7ff18c0fc370 msgr2=0x7ff18c0fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:02.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.393+0000 7ff1942a3640 1 -- 172.21.15.104:0/295807304 shutdown_connections 2024-06-09T22:26:02.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.393+0000 7ff1942a3640 1 -- 172.21.15.104:0/295807304 wait complete. 2024-06-09T22:26:02.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.393+0000 7ff1942a3640 1 Processor -- start 2024-06-09T22:26:02.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.393+0000 7ff1942a3640 1 -- start start 2024-06-09T22:26:02.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff1942a3640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff18c100750 0x7ff18c19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff1942a3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101100 0x7ff18c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff1942a3640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff18c10e300 0x7ff18c19ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff1942a3640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c113fa0 con 0x7ff18c10e300 2024-06-09T22:26:02.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff1942a3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c113e20 con 0x7ff18c101100 2024-06-09T22:26:02.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff1942a3640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c114120 con 0x7ff18c100750 2024-06-09T22:26:02.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff192840640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff18c10e300 0x7ff18c19ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff19203f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff18c100750 0x7ff18c19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff19183e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101100 0x7ff18c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff192840640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff18c10e300 0x7ff18c19ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49696/0 (socket says 172.21.15.104:49696) 2024-06-09T22:26:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff19183e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101100 0x7ff18c19a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:49864/0 (socket says 172.21.15.104:49864) 2024-06-09T22:26:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff192840640 1 -- 172.21.15.104:0/3105510995 learned_addr learned my addr 172.21.15.104:0/3105510995 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff19203f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff18c100750 0x7ff18c19a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:52160/0 (socket says 172.21.15.104:52160) 2024-06-09T22:26:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff192840640 1 -- 172.21.15.104:0/3105510995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff18c100750 msgr2=0x7ff18c19a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff192840640 1 --2- 172.21.15.104:0/3105510995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff18c100750 0x7ff18c19a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff192840640 1 -- 172.21.15.104:0/3105510995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101100 msgr2=0x7ff18c19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff192840640 1 --2- 172.21.15.104:0/3105510995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101100 0x7ff18c19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff192840640 1 -- 172.21.15.104:0/3105510995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff18c19f3f0 con 0x7ff18c10e300 2024-06-09T22:26:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff192840640 1 --2- 172.21.15.104:0/3105510995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff18c10e300 0x7ff18c19ecc0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff18400eb10 tx=0x7ff18400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.394+0000 7ff1837fe640 1 -- 172.21.15.104:0/3105510995 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff184004b20 con 0x7ff18c10e300 2024-06-09T22:26:02.399 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.395+0000 7ff1837fe640 1 -- 172.21.15.104:0/3105510995 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff184004cc0 con 0x7ff18c10e300 2024-06-09T22:26:02.399 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.395+0000 7ff1942a3640 1 -- 172.21.15.104:0/3105510995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff18c19f6e0 con 0x7ff18c10e300 2024-06-09T22:26:02.399 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.395+0000 7ff1837fe640 1 -- 172.21.15.104:0/3105510995 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff184010790 con 0x7ff18c10e300 2024-06-09T22:26:02.399 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.395+0000 7ff1942a3640 1 -- 172.21.15.104:0/3105510995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff18c1a6f70 con 0x7ff18c10e300 2024-06-09T22:26:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.397+0000 7ff1837fe640 1 -- 172.21.15.104:0/3105510995 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ff184005020 con 0x7ff18c10e300 2024-06-09T22:26:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.397+0000 7ff196fa8740 1 Processor -- start 2024-06-09T22:26:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.397+0000 7ff196fa8740 1 -- start start 2024-06-09T22:26:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff196fa8740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0170 0x558c655b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff196fa8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655b0a90 0x558c655bae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff196fa8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655bb350 0x558c655bd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff196fa8740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x558c655144f0 con 0x558c655b0170 2024-06-09T22:26:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff196fa8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558c654b2870 con 0x558c655bb350 2024-06-09T22:26:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff196fa8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558c655104b0 con 0x558c655b0a90 2024-06-09T22:26:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff1837fe640 1 --2- 172.21.15.104:0/3105510995 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff1780a2da0 0x7ff1780a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff19183e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0170 0x558c655b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff19203f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655b0a90 0x558c655bae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff19203f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655b0a90 0x558c655bae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:52168/0 (socket says 172.21.15.104:52168) 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff19203f640 1 -- 172.21.15.104:0/2609007928 learned_addr learned my addr 172.21.15.104:0/2609007928 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff19183e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0170 0x558c655b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49712/0 (socket says 172.21.15.104:49712) 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff192840640 1 --2- 172.21.15.104:0/3105510995 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff1780a2da0 0x7ff1780a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff19183e640 1 --2- 172.21.15.104:0/2609007928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655bb350 0x558c655bd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff19203f640 1 -- 172.21.15.104:0/2609007928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655bb350 msgr2=0x558c655bd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff19203f640 1 --2- 172.21.15.104:0/2609007928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655bb350 0x558c655bd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff19203f640 1 -- 172.21.15.104:0/2609007928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0170 msgr2=0x558c655b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff19203f640 1 --2- 172.21.15.104:0/2609007928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0170 0x558c655b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff19203f640 1 -- 172.21.15.104:0/2609007928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c654ce9b0 con 0x558c655b0a90 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff1837fe640 1 -- 172.21.15.104:0/3105510995 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff1840cfe70 con 0x7ff18c10e300 2024-06-09T22:26:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.398+0000 7ff19203f640 1 --2- 172.21.15.104:0/2609007928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655b0a90 0x558c655bae10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff18800bcd0 tx=0x7ff1880107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70d126b823471527 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.399+0000 7ff180bfe640 1 -- 172.21.15.104:0/2609007928 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188005220 con 0x558c655b0a90 2024-06-09T22:26:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.399+0000 7ff180bfe640 1 -- 172.21.15.104:0/2609007928 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff188004c80 con 0x558c655b0a90 2024-06-09T22:26:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.399+0000 7ff180bfe640 1 -- 172.21.15.104:0/2609007928 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188011780 con 0x558c655b0a90 2024-06-09T22:26:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.399+0000 7ff196fa8740 1 -- 172.21.15.104:0/2609007928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655b0a90 msgr2=0x558c655bae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.399+0000 7ff192840640 1 --2- 172.21.15.104:0/3105510995 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff1780a2da0 0x7ff1780a5260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff184098480 tx=0x7ff184005970 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.399+0000 7ff196fa8740 1 --2- 172.21.15.104:0/2609007928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655b0a90 0x558c655bae10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff18800bcd0 tx=0x7ff1880107a0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.399+0000 7ff196fa8740 1 -- 172.21.15.104:0/2609007928 shutdown_connections 2024-06-09T22:26:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.399+0000 7ff196fa8740 1 --2- 172.21.15.104:0/2609007928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655bb350 0x558c655bd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.399+0000 7ff196fa8740 1 --2- 172.21.15.104:0/2609007928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655b0a90 0x558c655bae10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.399+0000 7ff196fa8740 1 --2- 172.21.15.104:0/2609007928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0170 0x558c655b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff196fa8740 1 -- 172.21.15.104:0/2609007928 >> 172.21.15.104:0/2609007928 conn(0x558c655ac680 msgr2=0x558c655aca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff196fa8740 1 -- 172.21.15.104:0/2609007928 shutdown_connections 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff196fa8740 1 -- 172.21.15.104:0/2609007928 wait complete. 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff196fa8740 1 -- start start 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff196fa8740 1 Processor -- start 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff196fa8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 0x558c655b8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff196fa8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655b0a90 0x558c655b8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff196fa8740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655bb350 0x558c655b8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff196fa8740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x558c655bdf80 con 0x558c655bb350 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff196fa8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558c6549dae0 con 0x558c655b0170 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff196fa8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558c655be100 con 0x558c655b0a90 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff196fa8740 10 client.0 _ll_register_callbacks cb 0x558c654d69b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff19183e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655bb350 0x558c655b8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff19183e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655bb350 0x558c655b8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49714/0 (socket says 172.21.15.104:49714) 2024-06-09T22:26:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff19203f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655b0a90 0x558c655b8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.400+0000 7ff19183e640 1 -- 172.21.15.104:0/3711423796 learned_addr learned my addr 172.21.15.104:0/3711423796 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff19183e640 1 --2- 172.21.15.104:0/3711423796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 0x558c655b8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff19183e640 1 -- 172.21.15.104:0/3711423796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655b0a90 msgr2=0x558c655b8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff19183e640 1 --2- 172.21.15.104:0/3711423796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655b0a90 0x558c655b8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff19183e640 1 -- 172.21.15.104:0/3711423796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 msgr2=0x558c655b8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff19183e640 1 --2- 172.21.15.104:0/3711423796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 0x558c655b8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff19183e640 1 -- 172.21.15.104:0/3711423796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c65228210 con 0x558c655bb350 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff19183e640 1 --2- 172.21.15.104:0/3711423796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655bb350 0x558c655b8da0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009900 tx=0x7ff17c0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b967e893440f8082 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff175ffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff175ffb640 1 -- 172.21.15.104:0/3711423796 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c044070 con 0x558c655bb350 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff196fa8740 1 -- 172.21.15.104:0/3711423796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff188004c80 con 0x558c655bb350 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff175ffb640 1 -- 172.21.15.104:0/3711423796 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff17c004230 con 0x558c655bb350 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff175ffb640 1 -- 172.21.15.104:0/3711423796 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff17c035040 con 0x558c655bb350 2024-06-09T22:26:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff196fa8740 10 client.16857 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff196fa8740 1 -- 172.21.15.104:0/3711423796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558c652f4620 con 0x558c655bb350 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.401+0000 7ff175ffb640 1 -- 172.21.15.104:0/3711423796 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff188004c80 con 0x558c655bb350 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.402+0000 7ff175ffb640 1 -- 172.21.15.104:0/3711423796 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff17c0051e0 con 0x558c655bb350 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.402+0000 7ff196fa8740 10 client.16857 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.402+0000 7ff196fa8740 10 client.16857 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.402+0000 7ff196fa8740 20 client.16857 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.402+0000 7ff196fa8740 1 --2- 172.21.15.104:0/3711423796 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558c655cb900 0x558c655cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.402+0000 7ff196fa8740 4 client.16857 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.402+0000 7ff196fa8740 1 -- 172.21.15.104:0/3711423796 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c65527f10 con 0x558c655cb900 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.402+0000 7ff19203f640 1 --2- 172.21.15.104:0/3711423796 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558c655cb900 0x558c655cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.402+0000 7ff175ffb640 10 client.16857 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.402+0000 7ff19203f640 1 --2- 172.21.15.104:0/3711423796 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558c655cb900 0x558c655cdd80 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a24be089bdd7f7e9 server_cookie=92d555069c2f4989 in_seq=0 out_seq=0 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff175ffb640 1 -- 172.21.15.104:0/3711423796 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x558c65527f10 con 0x558c655cb900 2024-06-09T22:26:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff175ffb640 10 client.16857 handle_command_reply: tid=0 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff196fa8740 1 client.16857 shutdown 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff196fa8740 1 -- 172.21.15.104:0/3711423796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655bb350 msgr2=0x558c655b8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff196fa8740 1 --2- 172.21.15.104:0/3711423796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655bb350 0x558c655b8da0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009900 tx=0x7ff17c0058d0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff196fa8740 1 -- 172.21.15.104:0/3711423796 shutdown_connections 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff196fa8740 1 --2- 172.21.15.104:0/3711423796 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558c655cb900 0x558c655cdd80 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff196fa8740 1 --2- 172.21.15.104:0/3711423796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655bb350 0x558c655b8da0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff196fa8740 1 --2- 172.21.15.104:0/3711423796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655b0a90 0x558c655b8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff175ffb640 0 client.16857 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff196fa8740 1 --2- 172.21.15.104:0/3711423796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 0x558c655b8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff196fa8740 1 -- 172.21.15.104:0/3711423796 >> 172.21.15.104:0/3711423796 conn(0x558c655ac680 msgr2=0x558c655b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff196fa8740 1 -- 172.21.15.104:0/3711423796 shutdown_connections 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff196fa8740 1 -- 172.21.15.104:0/3711423796 wait complete. 2024-06-09T22:26:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.403+0000 7ff196fa8740 20 client.16857 trim_cache size 0 max 16384 2024-06-09T22:26:02.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff196fa8740 1 Processor -- start 2024-06-09T22:26:02.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff196fa8740 1 -- start start 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff196fa8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 0x558c655b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff196fa8740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 0x558c655d3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff196fa8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655bb350 0x558c655d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff196fa8740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c1b0ff0 con 0x558c655b0a90 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff196fa8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c1b2b30 con 0x558c655b0170 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff196fa8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c1b16c0 con 0x558c655bb350 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff19203f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 0x558c655d3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff19203f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 0x558c655d3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49730/0 (socket says 172.21.15.104:49730) 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff19203f640 1 -- 172.21.15.104:0/3961706701 learned_addr learned my addr 172.21.15.104:0/3961706701 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff19183e640 1 --2- 172.21.15.104:0/3961706701 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655bb350 0x558c655d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff19183e640 1 --2- 172.21.15.104:0/3961706701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 0x558c655b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff19203f640 1 -- 172.21.15.104:0/3961706701 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655bb350 msgr2=0x558c655d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff19203f640 1 --2- 172.21.15.104:0/3961706701 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655bb350 0x558c655d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff19203f640 1 -- 172.21.15.104:0/3961706701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 msgr2=0x558c655b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff19203f640 1 --2- 172.21.15.104:0/3961706701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 0x558c655b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff19203f640 1 -- 172.21.15.104:0/3961706701 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c655b08c0 con 0x558c655b0a90 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.421+0000 7ff19203f640 1 --2- 172.21.15.104:0/3961706701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 0x558c655d3690 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff18800bcd0 tx=0x7ff18800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495a379dde8b3f83 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.422+0000 7ff177fff640 1 -- 172.21.15.104:0/3961706701 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188005500 con 0x558c655b0a90 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.422+0000 7ff177fff640 1 -- 172.21.15.104:0/3961706701 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff188004b40 con 0x558c655b0a90 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.422+0000 7ff177fff640 1 -- 172.21.15.104:0/3961706701 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188029a20 con 0x558c655b0a90 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.422+0000 7ff196fa8740 1 -- 172.21.15.104:0/3961706701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 msgr2=0x558c655d3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.422+0000 7ff196fa8740 1 --2- 172.21.15.104:0/3961706701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 0x558c655d3690 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff18800bcd0 tx=0x7ff18800b410 comp rx=0 tx=0).stop 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.422+0000 7ff196fa8740 1 -- 172.21.15.104:0/3961706701 shutdown_connections 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.422+0000 7ff196fa8740 1 --2- 172.21.15.104:0/3961706701 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655bb350 0x558c655d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.422+0000 7ff196fa8740 1 --2- 172.21.15.104:0/3961706701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 0x558c655d3690 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.422+0000 7ff196fa8740 1 --2- 172.21.15.104:0/3961706701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 0x558c655b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.422+0000 7ff196fa8740 1 -- 172.21.15.104:0/3961706701 >> 172.21.15.104:0/3961706701 conn(0x558c655ac680 msgr2=0x558c655aca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.422+0000 7ff196fa8740 1 -- 172.21.15.104:0/3961706701 shutdown_connections 2024-06-09T22:26:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.422+0000 7ff196fa8740 1 -- 172.21.15.104:0/3961706701 wait complete. 2024-06-09T22:26:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff196fa8740 1 -- start start 2024-06-09T22:26:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff196fa8740 1 Processor -- start 2024-06-09T22:26:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff196fa8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 0x558c655be910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff196fa8740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 0x558c655bee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff196fa8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655bb350 0x558c65634650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff196fa8740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c1b2460 con 0x558c655b0a90 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff196fa8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c1b1d90 con 0x558c655b0170 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff196fa8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c1aefb0 con 0x558c655bb350 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff196fa8740 10 client.0 _ll_register_callbacks cb 0x7ff17c0318b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff19203f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 0x558c655bee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff19203f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 0x558c655bee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49746/0 (socket says 172.21.15.104:49746) 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff19203f640 1 -- 172.21.15.104:0/573186587 learned_addr learned my addr 172.21.15.104:0/573186587 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff19183e640 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655bb350 0x558c65634650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.423+0000 7ff19183e640 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 0x558c655be910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff19203f640 1 -- 172.21.15.104:0/573186587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655bb350 msgr2=0x558c65634650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff19203f640 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655bb350 0x558c65634650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff19203f640 1 -- 172.21.15.104:0/573186587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 msgr2=0x558c655be910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff19203f640 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 0x558c655be910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff19203f640 1 -- 172.21.15.104:0/573186587 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c65634bc0 con 0x558c655b0a90 2024-06-09T22:26:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff19183e640 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655bb350 0x558c65634650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff19203f640 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 0x558c655bee50 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x558c655ad1f0 tx=0x7ff188005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa95ec41274b6e7 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff175ffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff175ffb640 1 -- 172.21.15.104:0/573186587 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188004860 con 0x558c655b0a90 2024-06-09T22:26:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff175ffb640 1 -- 172.21.15.104:0/573186587 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff188005300 con 0x558c655b0a90 2024-06-09T22:26:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff196fa8740 1 -- 172.21.15.104:0/573186587 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff188004b40 con 0x558c655b0a90 2024-06-09T22:26:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff175ffb640 1 -- 172.21.15.104:0/573186587 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff188042050 con 0x558c655b0a90 2024-06-09T22:26:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff196fa8740 10 client.16863 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff196fa8740 1 -- 172.21.15.104:0/573186587 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558c65635ed0 con 0x558c655b0a90 2024-06-09T22:26:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff175ffb640 1 -- 172.21.15.104:0/573186587 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff188004b40 con 0x558c655b0a90 2024-06-09T22:26:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff175ffb640 1 -- 172.21.15.104:0/573186587 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff188005540 con 0x558c655b0a90 2024-06-09T22:26:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff196fa8740 10 client.16863 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff196fa8740 10 client.16863 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff196fa8740 20 client.16863 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff196fa8740 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558c65636410 0x558c65638830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff196fa8740 4 client.16863 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:26:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.424+0000 7ff196fa8740 1 -- 172.21.15.104:0/573186587 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558c65521c20 con 0x558c65636410 2024-06-09T22:26:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.425+0000 7ff19183e640 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558c65636410 0x558c65638830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.425+0000 7ff19183e640 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558c65636410 0x558c65638830 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e6b177297f607b1 server_cookie=85108085688b897a in_seq=0 out_seq=0 2024-06-09T22:26:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.425+0000 7ff175ffb640 10 client.16863 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff175ffb640 1 -- 172.21.15.104:0/573186587 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x558c65521c20 con 0x558c65636410 2024-06-09T22:26:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff175ffb640 10 client.16863 handle_command_reply: tid=0 2024-06-09T22:26:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff196fa8740 1 client.16863 shutdown 2024-06-09T22:26:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff196fa8740 1 -- 172.21.15.104:0/573186587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 msgr2=0x558c655bee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff196fa8740 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 0x558c655bee50 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x558c655ad1f0 tx=0x7ff188005c60 comp rx=0 tx=0).stop 2024-06-09T22:26:02.436 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff196fa8740 1 -- 172.21.15.104:0/573186587 shutdown_connections 2024-06-09T22:26:02.436 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff196fa8740 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558c65636410 0x558c65638830 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.436 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff196fa8740 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558c655bb350 0x558c65634650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.436 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff175ffb640 0 client.16863 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:02.436 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff196fa8740 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558c655b0a90 0x558c655bee50 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.436 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff196fa8740 1 --2- 172.21.15.104:0/573186587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c655b0170 0x558c655be910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.436 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff196fa8740 1 -- 172.21.15.104:0/573186587 >> 172.21.15.104:0/573186587 conn(0x558c655ac680 msgr2=0x558c655b9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:02.436 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff196fa8740 1 -- 172.21.15.104:0/573186587 shutdown_connections 2024-06-09T22:26:02.436 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff196fa8740 1 -- 172.21.15.104:0/573186587 wait complete. 2024-06-09T22:26:02.436 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.434+0000 7ff196fa8740 20 client.16863 trim_cache size 0 max 16384 2024-06-09T22:26:02.436 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:26:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:26:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:26:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.435+0000 7ff1942a3640 1 -- 172.21.15.104:0/3105510995 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff1780a2da0 msgr2=0x7ff1780a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.435+0000 7ff1942a3640 1 --2- 172.21.15.104:0/3105510995 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff1780a2da0 0x7ff1780a5260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff184098480 tx=0x7ff184005970 comp rx=0 tx=0).stop 2024-06-09T22:26:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.435+0000 7ff1942a3640 1 -- 172.21.15.104:0/3105510995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff18c10e300 msgr2=0x7ff18c19ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.435+0000 7ff1942a3640 1 --2- 172.21.15.104:0/3105510995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff18c10e300 0x7ff18c19ecc0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff18400eb10 tx=0x7ff18400c2d0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.435+0000 7ff1942a3640 1 -- 172.21.15.104:0/3105510995 shutdown_connections 2024-06-09T22:26:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.435+0000 7ff1942a3640 1 --2- 172.21.15.104:0/3105510995 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff1780a2da0 0x7ff1780a5260 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.435+0000 7ff1942a3640 1 --2- 172.21.15.104:0/3105510995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff18c10e300 0x7ff18c19ecc0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.435+0000 7ff1942a3640 1 --2- 172.21.15.104:0/3105510995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101100 0x7ff18c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.435+0000 7ff1942a3640 1 --2- 172.21.15.104:0/3105510995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff18c100750 0x7ff18c19a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.435+0000 7ff1942a3640 1 -- 172.21.15.104:0/3105510995 >> 172.21.15.104:0/3105510995 conn(0x7ff18c0fc370 msgr2=0x7ff18c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.435+0000 7ff1942a3640 1 -- 172.21.15.104:0/3105510995 shutdown_connections 2024-06-09T22:26:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:02.435+0000 7ff1942a3640 1 -- 172.21.15.104:0/3105510995 wait complete. 2024-06-09T22:26:02.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 124426 4.21 MB/sec execute 68 sec latency 11.245 ms 2024-06-09T22:26:03.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:02 smithi153 ceph-mon[36599]: pgmap v155: 129 pgs: 129 active+clean; 86 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-06-09T22:26:03.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:02 smithi104 ceph-mon[32389]: pgmap v155: 129 pgs: 129 active+clean; 86 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-06-09T22:26:03.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:02 smithi194 ceph-mon[36856]: pgmap v155: 129 pgs: 129 active+clean; 86 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-06-09T22:26:03.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 126278 4.23 MB/sec execute 69 sec latency 25.803 ms 2024-06-09T22:26:04.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 128110 4.37 MB/sec execute 70 sec latency 32.159 ms 2024-06-09T22:26:05.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:04 smithi153 ceph-mon[36599]: pgmap v156: 129 pgs: 129 active+clean; 92 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.7 MiB/s wr, 322 op/s 2024-06-09T22:26:05.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:04 smithi104 ceph-mon[32389]: pgmap v156: 129 pgs: 129 active+clean; 92 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.7 MiB/s wr, 322 op/s 2024-06-09T22:26:05.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:04 smithi194 ceph-mon[36856]: pgmap v156: 129 pgs: 129 active+clean; 92 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.7 MiB/s wr, 322 op/s 2024-06-09T22:26:05.728 INFO:tasks.workunit.client.0.smithi104.stdout: 1 130163 4.40 MB/sec execute 71 sec latency 33.235 ms 2024-06-09T22:26:06.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 131834 4.51 MB/sec execute 72 sec latency 10.914 ms 2024-06-09T22:26:07.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:06 smithi153 ceph-mon[36599]: pgmap v157: 129 pgs: 129 active+clean; 92 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 220 op/s 2024-06-09T22:26:07.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:06 smithi104 ceph-mon[32389]: pgmap v157: 129 pgs: 129 active+clean; 92 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 220 op/s 2024-06-09T22:26:07.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:06 smithi194 ceph-mon[36856]: pgmap v157: 129 pgs: 129 active+clean; 92 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 220 op/s 2024-06-09T22:26:07.528 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:26:07.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 133768 4.55 MB/sec execute 73 sec latency 16.312 ms 2024-06-09T22:26:07.788 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:08.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 135487 4.65 MB/sec execute 74 sec latency 10.917 ms 2024-06-09T22:26:09.229 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:08 smithi104 ceph-mon[32389]: pgmap v158: 129 pgs: 129 active+clean; 92 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 220 op/s 2024-06-09T22:26:09.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:08 smithi153 ceph-mon[36599]: pgmap v158: 129 pgs: 129 active+clean; 92 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 220 op/s 2024-06-09T22:26:09.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:08 smithi194 ceph-mon[36856]: pgmap v158: 129 pgs: 129 active+clean; 92 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 220 op/s 2024-06-09T22:26:09.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 137299 4.69 MB/sec execute 75 sec latency 61.562 ms 2024-06-09T22:26:09.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.857+0000 7f933e791700 1 -- 172.21.15.104:0/4090713974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93381071a0 msgr2=0x7f9338107620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:09.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.857+0000 7f933e791700 1 --2- 172.21.15.104:0/4090713974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93381071a0 0x7f9338107620 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f932c0099c0 tx=0x7f932c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.857+0000 7f933e791700 1 -- 172.21.15.104:0/4090713974 shutdown_connections 2024-06-09T22:26:09.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.857+0000 7f933e791700 1 --2- 172.21.15.104:0/4090713974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9338107b60 0x7f933810e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.857+0000 7f933e791700 1 --2- 172.21.15.104:0/4090713974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9338105330 0x7f9338105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.857+0000 7f933e791700 1 --2- 172.21.15.104:0/4090713974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93381071a0 0x7f9338107620 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.857+0000 7f933e791700 1 -- 172.21.15.104:0/4090713974 >> 172.21.15.104:0/4090713974 conn(0x7f93380766b0 msgr2=0x7f9338076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:09.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.857+0000 7f933e791700 1 -- 172.21.15.104:0/4090713974 shutdown_connections 2024-06-09T22:26:09.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.857+0000 7f933e791700 1 -- 172.21.15.104:0/4090713974 wait complete. 2024-06-09T22:26:09.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.857+0000 7f933e791700 1 Processor -- start 2024-06-09T22:26:09.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.857+0000 7f933e791700 1 -- start start 2024-06-09T22:26:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f933e791700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9338105330 0x7f933819e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f933e791700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93381071a0 0x7f933819ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f933e791700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9338107b60 0x7f9338198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f933e791700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9338111750 con 0x7f9338105330 2024-06-09T22:26:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f933e791700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f93381115d0 con 0x7f9338107b60 2024-06-09T22:26:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f933e791700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93381118d0 con 0x7f93381071a0 2024-06-09T22:26:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f9337fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9338105330 0x7f933819e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f93377fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93381071a0 0x7f933819ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f9337fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9338105330 0x7f933819e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41082/0 (socket says 172.21.15.104:41082) 2024-06-09T22:26:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f9337fff700 1 -- 172.21.15.104:0/3884650879 learned_addr learned my addr 172.21.15.104:0/3884650879 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f933cd2e700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9338107b60 0x7f9338198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f93377fe700 1 -- 172.21.15.104:0/3884650879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9338107b60 msgr2=0x7f9338198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:09.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f93377fe700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9338107b60 0x7f9338198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f93377fe700 1 -- 172.21.15.104:0/3884650879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9338105330 msgr2=0x7f933819e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:09.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f93377fe700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9338105330 0x7f933819e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f93377fe700 1 -- 172.21.15.104:0/3884650879 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9338199230 con 0x7f93381071a0 2024-06-09T22:26:09.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.858+0000 7f933cd2e700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9338107b60 0x7f9338198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:26:09.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.859+0000 7f93377fe700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93381071a0 0x7f933819ee50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f932c00b580 tx=0x7f932c003e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:09.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.859+0000 7f93357fa700 1 -- 172.21.15.104:0/3884650879 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c009e60 con 0x7f93381071a0 2024-06-09T22:26:09.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.859+0000 7f933e791700 1 -- 172.21.15.104:0/3884650879 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93381994c0 con 0x7f93381071a0 2024-06-09T22:26:09.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.859+0000 7f933e791700 1 -- 172.21.15.104:0/3884650879 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93381a57e0 con 0x7f93381071a0 2024-06-09T22:26:09.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.859+0000 7f93357fa700 1 -- 172.21.15.104:0/3884650879 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f932c003680 con 0x7f93381071a0 2024-06-09T22:26:09.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.859+0000 7f93357fa700 1 -- 172.21.15.104:0/3884650879 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c00fda0 con 0x7f93381071a0 2024-06-09T22:26:09.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.862+0000 7f93357fa700 1 -- 172.21.15.104:0/3884650879 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f932c00f460 con 0x7f93381071a0 2024-06-09T22:26:09.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.862+0000 7f93357fa700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f93200a2e10 0x7f93200a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.862+0000 7f93357fa700 1 -- 172.21.15.104:0/3884650879 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f932c0d04a0 con 0x7f93381071a0 2024-06-09T22:26:09.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.862+0000 7f9337fff700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f93200a2e10 0x7f93200a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.863+0000 7f9337fff700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f93200a2e10 0x7f93200a52d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9328009d60 tx=0x7f9328009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.863+0000 7f9349014b80 1 Processor -- start 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.863+0000 7f9349014b80 1 -- start start 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.863+0000 7f9349014b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd759af60 0x560cd759b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.863+0000 7f9349014b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd7595260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.863+0000 7f9349014b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 0x560cd7595bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.863+0000 7f9349014b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x560cd7487f60 con 0x560cd75957a0 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.863+0000 7f9349014b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560cd74e6850 con 0x560cd759b880 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.863+0000 7f9349014b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560cd744e460 con 0x560cd759af60 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f9337fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd759af60 0x560cd759b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f933cd2e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 0x560cd7595bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f9337fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd759af60 0x560cd759b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:41712/0 (socket says 172.21.15.104:41712) 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f9337fff700 1 -- 172.21.15.104:0/3425613546 learned_addr learned my addr 172.21.15.104:0/3425613546 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f93377fe700 1 --2- 172.21.15.104:0/3425613546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd7595260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f933cd2e700 1 -- 172.21.15.104:0/3425613546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd759af60 msgr2=0x560cd759b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f933cd2e700 1 --2- 172.21.15.104:0/3425613546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd759af60 0x560cd759b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f933cd2e700 1 -- 172.21.15.104:0/3425613546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 msgr2=0x560cd7595260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f933cd2e700 1 --2- 172.21.15.104:0/3425613546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd7595260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f933cd2e700 1 -- 172.21.15.104:0/3425613546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cd74358d0 con 0x560cd75957a0 2024-06-09T22:26:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f933cd2e700 1 --2- 172.21.15.104:0/3425613546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 0x560cd7595bc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9330009d10 tx=0x7f93300077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506d260d9a07417f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f931e7fc700 1 -- 172.21.15.104:0/3425613546 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9330009e60 con 0x560cd75957a0 2024-06-09T22:26:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f931e7fc700 1 -- 172.21.15.104:0/3425613546 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9330019070 con 0x560cd75957a0 2024-06-09T22:26:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.864+0000 7f931e7fc700 1 -- 172.21.15.104:0/3425613546 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93300139d0 con 0x560cd75957a0 2024-06-09T22:26:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.865+0000 7f9349014b80 1 -- 172.21.15.104:0/3425613546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 msgr2=0x560cd7595bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.865+0000 7f9349014b80 1 --2- 172.21.15.104:0/3425613546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 0x560cd7595bc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9330009d10 tx=0x7f93300077e0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.865+0000 7f9349014b80 1 -- 172.21.15.104:0/3425613546 shutdown_connections 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.865+0000 7f9349014b80 1 --2- 172.21.15.104:0/3425613546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 0x560cd7595bc0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.865+0000 7f9349014b80 1 --2- 172.21.15.104:0/3425613546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd759af60 0x560cd759b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.865+0000 7f9349014b80 1 --2- 172.21.15.104:0/3425613546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd7595260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.865+0000 7f9349014b80 1 -- 172.21.15.104:0/3425613546 >> 172.21.15.104:0/3425613546 conn(0x560cd74c2080 msgr2=0x560cd7590220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.865+0000 7f9349014b80 1 -- 172.21.15.104:0/3425613546 shutdown_connections 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.865+0000 7f9349014b80 1 -- 172.21.15.104:0/3425613546 wait complete. 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f9349014b80 1 -- start start 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f9349014b80 1 Processor -- start 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f9349014b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 0x560cd75a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f9349014b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759af60 0x560cd759e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f9349014b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd759b880 0x560cd759ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f9349014b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x560cd7486fc0 con 0x560cd75957a0 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f9349014b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560cd74e2810 con 0x560cd759af60 2024-06-09T22:26:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f9349014b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560cd75a2260 con 0x560cd759b880 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f9349014b80 10 client.0 _ll_register_callbacks cb 0x560cd747c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f9337fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 0x560cd75a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f933cd2e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd759b880 0x560cd759ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f93377fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759af60 0x560cd759e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f9337fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 0x560cd75a15f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41104/0 (socket says 172.21.15.104:41104) 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.866+0000 7f9337fff700 1 -- 172.21.15.104:0/3976630810 learned_addr learned my addr 172.21.15.104:0/3976630810 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f933cd2e700 1 -- 172.21.15.104:0/3976630810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759af60 msgr2=0x560cd759e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f933cd2e700 1 --2- 172.21.15.104:0/3976630810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759af60 0x560cd759e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f933cd2e700 1 -- 172.21.15.104:0/3976630810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 msgr2=0x560cd75a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f933cd2e700 1 --2- 172.21.15.104:0/3976630810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 0x560cd75a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f933cd2e700 1 -- 172.21.15.104:0/3976630810 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cd74ac750 con 0x560cd759b880 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f9337fff700 1 --2- 172.21.15.104:0/3976630810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 0x560cd75a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f933cd2e700 1 --2- 172.21.15.104:0/3976630810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd759b880 0x560cd759ee30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9330009d10 tx=0x7f93300137e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efa81dfd4df52bbe server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:09.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f93177fe700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:26:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f93177fe700 1 -- 172.21.15.104:0/3976630810 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9330009e60 con 0x560cd759b880 2024-06-09T22:26:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f9349014b80 1 -- 172.21.15.104:0/3976630810 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9330019070 con 0x560cd759b880 2024-06-09T22:26:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f93177fe700 1 -- 172.21.15.104:0/3976630810 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f9330013990 con 0x560cd759b880 2024-06-09T22:26:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f93177fe700 1 -- 172.21.15.104:0/3976630810 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f933002f050 con 0x560cd759b880 2024-06-09T22:26:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f9349014b80 10 client.26278 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.867+0000 7f9349014b80 1 -- 172.21.15.104:0/3976630810 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560cd75ac9e0 con 0x560cd759b880 2024-06-09T22:26:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.868+0000 7f93177fe700 1 -- 172.21.15.104:0/3976630810 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9330019070 con 0x560cd759b880 2024-06-09T22:26:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.868+0000 7f93177fe700 1 -- 172.21.15.104:0/3976630810 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9330004400 con 0x560cd759b880 2024-06-09T22:26:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.868+0000 7f9349014b80 10 client.26278 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.868+0000 7f9349014b80 10 client.26278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.868+0000 7f9349014b80 20 client.26278 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.868+0000 7f9349014b80 1 --2- 172.21.15.104:0/3976630810 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560cd75aef10 0x560cd75b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.868+0000 7f9349014b80 4 client.26278 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.868+0000 7f9349014b80 1 -- 172.21.15.104:0/3976630810 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cd74f3d50 con 0x560cd75aef10 2024-06-09T22:26:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.868+0000 7f9337fff700 1 --2- 172.21.15.104:0/3976630810 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560cd75aef10 0x560cd75b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.868+0000 7f93177fe700 10 client.26278 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.868+0000 7f9337fff700 1 --2- 172.21.15.104:0/3976630810 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560cd75aef10 0x560cd75b1300 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3698b6abedd87b0 server_cookie=57777b606cad6bb6 in_seq=0 out_seq=0 2024-06-09T22:26:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f93177fe700 1 -- 172.21.15.104:0/3976630810 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x560cd74f3d50 con 0x560cd75aef10 2024-06-09T22:26:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f93177fe700 10 client.26278 handle_command_reply: tid=0 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f9349014b80 1 client.26278 shutdown 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f9349014b80 1 -- 172.21.15.104:0/3976630810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd759b880 msgr2=0x560cd759ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f9349014b80 1 --2- 172.21.15.104:0/3976630810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd759b880 0x560cd759ee30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9330009d10 tx=0x7f93300137e0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f9349014b80 1 -- 172.21.15.104:0/3976630810 shutdown_connections 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f9349014b80 1 --2- 172.21.15.104:0/3976630810 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560cd75aef10 0x560cd75b1300 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f9349014b80 1 --2- 172.21.15.104:0/3976630810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd75957a0 0x560cd75a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f9349014b80 1 --2- 172.21.15.104:0/3976630810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd759b880 0x560cd759ee30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f9349014b80 1 --2- 172.21.15.104:0/3976630810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759af60 0x560cd759e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f9349014b80 1 -- 172.21.15.104:0/3976630810 >> 172.21.15.104:0/3976630810 conn(0x560cd74c2080 msgr2=0x560cd758f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f93177fe700 0 client.26278 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.869+0000 7f9349014b80 1 -- 172.21.15.104:0/3976630810 shutdown_connections 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.870+0000 7f9349014b80 1 -- 172.21.15.104:0/3976630810 wait complete. 2024-06-09T22:26:09.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.870+0000 7f9349014b80 20 client.26278 trim_cache size 0 max 16384 2024-06-09T22:26:09.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f9349014b80 1 Processor -- start 2024-06-09T22:26:09.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f9349014b80 1 -- start start 2024-06-09T22:26:09.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f9349014b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd75957a0 0x560cd7595260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f9349014b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 0x560cd7599050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f9349014b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd759c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f9349014b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f93381ae4c0 con 0x560cd759af60 2024-06-09T22:26:09.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f9349014b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f93381b0000 con 0x560cd759b880 2024-06-09T22:26:09.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f9349014b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93381aeb90 con 0x560cd75957a0 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f93377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 0x560cd7599050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f93377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 0x560cd7599050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41114/0 (socket says 172.21.15.104:41114) 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f93377fe700 1 -- 172.21.15.104:0/373511827 learned_addr learned my addr 172.21.15.104:0/373511827 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f933cd2e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd759c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f9337fff700 1 --2- 172.21.15.104:0/373511827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd75957a0 0x560cd7595260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f933cd2e700 1 -- 172.21.15.104:0/373511827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd75957a0 msgr2=0x560cd7595260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f933cd2e700 1 --2- 172.21.15.104:0/373511827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd75957a0 0x560cd7595260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f933cd2e700 1 -- 172.21.15.104:0/373511827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 msgr2=0x560cd7599050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f933cd2e700 1 --2- 172.21.15.104:0/373511827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 0x560cd7599050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f933cd2e700 1 -- 172.21.15.104:0/373511827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cd759f7f0 con 0x560cd759b880 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f93377fe700 1 --2- 172.21.15.104:0/373511827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 0x560cd7599050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.892+0000 7f933cd2e700 1 --2- 172.21.15.104:0/373511827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd759c4f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9330004970 tx=0x7f93300077e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1719c644de21a4a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.893+0000 7f931dffb700 1 -- 172.21.15.104:0/373511827 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9330009e60 con 0x560cd759b880 2024-06-09T22:26:09.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.893+0000 7f931dffb700 1 -- 172.21.15.104:0/373511827 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9330019070 con 0x560cd759b880 2024-06-09T22:26:09.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.893+0000 7f931dffb700 1 -- 172.21.15.104:0/373511827 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9330029e10 con 0x560cd759b880 2024-06-09T22:26:09.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.893+0000 7f9349014b80 1 -- 172.21.15.104:0/373511827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 msgr2=0x560cd759c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:09.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.893+0000 7f9349014b80 1 --2- 172.21.15.104:0/373511827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd759c4f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9330004970 tx=0x7f93300077e0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.893+0000 7f9349014b80 1 -- 172.21.15.104:0/373511827 shutdown_connections 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.893+0000 7f9349014b80 1 --2- 172.21.15.104:0/373511827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd759c4f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.893+0000 7f9349014b80 1 --2- 172.21.15.104:0/373511827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd75957a0 0x560cd7595260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.893+0000 7f9349014b80 1 --2- 172.21.15.104:0/373511827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 0x560cd7599050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.893+0000 7f9349014b80 1 -- 172.21.15.104:0/373511827 >> 172.21.15.104:0/373511827 conn(0x560cd74c2080 msgr2=0x560cd759d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.893+0000 7f9349014b80 1 -- 172.21.15.104:0/373511827 shutdown_connections 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.893+0000 7f9349014b80 1 -- 172.21.15.104:0/373511827 wait complete. 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f9349014b80 1 -- start start 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f9349014b80 1 Processor -- start 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f9349014b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd75957a0 0x560cd75a7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f9349014b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 0x560cd75a7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f9349014b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd75b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f9349014b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f93381af930 con 0x560cd759af60 2024-06-09T22:26:09.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f9349014b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f93381af260 con 0x560cd759b880 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f9349014b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93381ac440 con 0x560cd75957a0 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f9349014b80 10 client.0 _ll_register_callbacks cb 0x560cd747c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f93377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 0x560cd75a7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f9337fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd75957a0 0x560cd75a7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f93377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 0x560cd75a7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41128/0 (socket says 172.21.15.104:41128) 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f9337fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd75957a0 0x560cd75a7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:41726/0 (socket says 172.21.15.104:41726) 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.894+0000 7f93377fe700 1 -- 172.21.15.104:0/1777981799 learned_addr learned my addr 172.21.15.104:0/1777981799 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f933cd2e700 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd75b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f9337fff700 1 -- 172.21.15.104:0/1777981799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 msgr2=0x560cd75b58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f9337fff700 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd75b58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f9337fff700 1 -- 172.21.15.104:0/1777981799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 msgr2=0x560cd75a7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f9337fff700 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 0x560cd75a7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f9337fff700 1 -- 172.21.15.104:0/1777981799 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cd75b7340 con 0x560cd75957a0 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f93377fe700 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 0x560cd75a7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f933cd2e700 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd75b58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f9337fff700 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd75957a0 0x560cd75a7520 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f93280098a0 tx=0x7f93280047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b0406531e685511 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f93177fe700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f93177fe700 1 -- 172.21.15.104:0/1777981799 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328037060 con 0x560cd75957a0 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f9349014b80 1 -- 172.21.15.104:0/1777981799 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9330019070 con 0x560cd75957a0 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f93177fe700 1 -- 172.21.15.104:0/1777981799 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f932801da80 con 0x560cd75957a0 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f93177fe700 1 -- 172.21.15.104:0/1777981799 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f932803e050 con 0x560cd75957a0 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f9349014b80 10 client.26284 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.895+0000 7f9349014b80 1 -- 172.21.15.104:0/1777981799 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560cd761d1a0 con 0x560cd75957a0 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.896+0000 7f93177fe700 1 -- 172.21.15.104:0/1777981799 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9330019070 con 0x560cd75957a0 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.896+0000 7f93177fe700 1 -- 172.21.15.104:0/1777981799 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9328028410 con 0x560cd75957a0 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.896+0000 7f9349014b80 10 client.26284 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.896+0000 7f9349014b80 10 client.26284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.896+0000 7f9349014b80 20 client.26284 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.896+0000 7f9349014b80 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560cd761d810 0x560cd761fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.896+0000 7f9349014b80 4 client.26284 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.896+0000 7f9349014b80 1 -- 172.21.15.104:0/1777981799 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560cd74ee2d0 con 0x560cd761d810 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.896+0000 7f93377fe700 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560cd761d810 0x560cd761fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.896+0000 7f93177fe700 10 client.26284 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.896+0000 7f93377fe700 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560cd761d810 0x560cd761fc30 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3102ed22aa615b server_cookie=10ae63f78f90dece in_seq=0 out_seq=0 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f93177fe700 1 -- 172.21.15.104:0/1777981799 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560cd74ee2d0 con 0x560cd761d810 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f93177fe700 10 client.26284 handle_command_reply: tid=0 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f9349014b80 1 client.26284 shutdown 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f9349014b80 1 -- 172.21.15.104:0/1777981799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd75957a0 msgr2=0x560cd75a7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f9349014b80 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd75957a0 0x560cd75a7520 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f93280098a0 tx=0x7f93280047c0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f9349014b80 1 -- 172.21.15.104:0/1777981799 shutdown_connections 2024-06-09T22:26:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f9349014b80 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x560cd761d810 0x560cd761fc30 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f9349014b80 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cd759b880 0x560cd75b58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f9349014b80 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560cd75957a0 0x560cd75a7520 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f9349014b80 1 --2- 172.21.15.104:0/1777981799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560cd759af60 0x560cd75a7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f93177fe700 0 client.26284 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f9349014b80 1 -- 172.21.15.104:0/1777981799 >> 172.21.15.104:0/1777981799 conn(0x560cd74c2080 msgr2=0x560cd759d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f9349014b80 1 -- 172.21.15.104:0/1777981799 shutdown_connections 2024-06-09T22:26:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f9349014b80 1 -- 172.21.15.104:0/1777981799 wait complete. 2024-06-09T22:26:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.897+0000 7f9349014b80 20 client.26284 trim_cache size 0 max 16384 2024-06-09T22:26:09.904 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:26:09.905 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "no active scrubs running", 2024-06-09T22:26:09.905 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": {} 2024-06-09T22:26:09.905 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:26:09.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.898+0000 7f933e791700 1 -- 172.21.15.104:0/3884650879 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f93200a2e10 msgr2=0x7f93200a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:09.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.898+0000 7f933e791700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f93200a2e10 0x7f93200a52d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9328009d60 tx=0x7f9328009260 comp rx=0 tx=0).stop 2024-06-09T22:26:09.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.898+0000 7f933e791700 1 -- 172.21.15.104:0/3884650879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93381071a0 msgr2=0x7f933819ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.898+0000 7f933e791700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93381071a0 0x7f933819ee50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f932c00b580 tx=0x7f932c003e00 comp rx=0 tx=0).stop 2024-06-09T22:26:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.898+0000 7f933e791700 1 -- 172.21.15.104:0/3884650879 shutdown_connections 2024-06-09T22:26:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.898+0000 7f933e791700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f93200a2e10 0x7f93200a52d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.898+0000 7f933e791700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9338105330 0x7f933819e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.898+0000 7f933e791700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9338107b60 0x7f9338198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.898+0000 7f933e791700 1 --2- 172.21.15.104:0/3884650879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93381071a0 0x7f933819ee50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.898+0000 7f933e791700 1 -- 172.21.15.104:0/3884650879 >> 172.21.15.104:0/3884650879 conn(0x7f93380766b0 msgr2=0x7f9338100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.898+0000 7f933e791700 1 -- 172.21.15.104:0/3884650879 shutdown_connections 2024-06-09T22:26:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:09.898+0000 7f933e791700 1 -- 172.21.15.104:0/3884650879 wait complete. 2024-06-09T22:26:10.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 139044 4.78 MB/sec execute 76 sec latency 15.492 ms 2024-06-09T22:26:10.878 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-09T22:26:10.879 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:26:11.142 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:11.169 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:10 smithi104 ceph-mon[32389]: pgmap v159: 129 pgs: 129 active+clean; 99 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 2.7 MiB/s wr, 318 op/s 2024-06-09T22:26:11.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:10 smithi153 ceph-mon[36599]: pgmap v159: 129 pgs: 129 active+clean; 99 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 2.7 MiB/s wr, 318 op/s 2024-06-09T22:26:11.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:10 smithi194 ceph-mon[36856]: pgmap v159: 129 pgs: 129 active+clean; 99 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 2.7 MiB/s wr, 318 op/s 2024-06-09T22:26:11.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 141135 4.86 MB/sec execute 77 sec latency 16.054 ms 2024-06-09T22:26:12.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 142530 4.90 MB/sec execute 78 sec latency 29.991 ms 2024-06-09T22:26:13.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:12 smithi104 ceph-mon[32389]: pgmap v160: 129 pgs: 129 active+clean; 99 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 216 op/s 2024-06-09T22:26:13.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:12 smithi153 ceph-mon[36599]: pgmap v160: 129 pgs: 129 active+clean; 99 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 216 op/s 2024-06-09T22:26:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.282+0000 7f916dc86700 1 -- 172.21.15.104:0/1963679510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9168100b50 msgr2=0x7f9168100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.282+0000 7f916dc86700 1 --2- 172.21.15.104:0/1963679510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9168100b50 0x7f9168100fd0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f915c009990 tx=0x7f915c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:26:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.282+0000 7f916dc86700 1 -- 172.21.15.104:0/1963679510 shutdown_connections 2024-06-09T22:26:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.282+0000 7f916dc86700 1 --2- 172.21.15.104:0/1963679510 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9168101510 0x7f916810d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.282+0000 7f916dc86700 1 --2- 172.21.15.104:0/1963679510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9168106b90 0x7f9168106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.282+0000 7f916dc86700 1 --2- 172.21.15.104:0/1963679510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9168100b50 0x7f9168100fd0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.282+0000 7f916dc86700 1 -- 172.21.15.104:0/1963679510 >> 172.21.15.104:0/1963679510 conn(0x7f9168077a00 msgr2=0x7f91680fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.282+0000 7f916dc86700 1 -- 172.21.15.104:0/1963679510 shutdown_connections 2024-06-09T22:26:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.282+0000 7f916dc86700 1 -- 172.21.15.104:0/1963679510 wait complete. 2024-06-09T22:26:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f916dc86700 1 Processor -- start 2024-06-09T22:26:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f916dc86700 1 -- start start 2024-06-09T22:26:13.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f916dc86700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9168100b50 0x7f9168199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:13.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f916dc86700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9168101510 0x7f9168199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:13.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f916dc86700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9168106b90 0x7f91681a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:13.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f916dc86700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f91681116a0 con 0x7f9168106b90 2024-06-09T22:26:13.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f916dc86700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9168111520 con 0x7f9168100b50 2024-06-09T22:26:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f916dc86700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9168111820 con 0x7f9168101510 2024-06-09T22:26:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f9167fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9168106b90 0x7f91681a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f9166ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9168101510 0x7f9168199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f9167fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9168106b90 0x7f91681a4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41134/0 (socket says 172.21.15.104:41134) 2024-06-09T22:26:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f9167fff700 1 -- 172.21.15.104:0/2711972192 learned_addr learned my addr 172.21.15.104:0/2711972192 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.283+0000 7f91677fe700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9168100b50 0x7f9168199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.284+0000 7f9167fff700 1 -- 172.21.15.104:0/2711972192 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9168101510 msgr2=0x7f9168199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.284+0000 7f9167fff700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9168101510 0x7f9168199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.284+0000 7f9167fff700 1 -- 172.21.15.104:0/2711972192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9168100b50 msgr2=0x7f9168199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:13.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.284+0000 7f9167fff700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9168100b50 0x7f9168199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.284+0000 7f9167fff700 1 -- 172.21.15.104:0/2711972192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91681a5040 con 0x7f9168106b90 2024-06-09T22:26:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.284+0000 7f91677fe700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9168100b50 0x7f9168199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:26:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.284+0000 7f9167fff700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9168106b90 0x7f91681a4b00 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f916000ef30 tx=0x7f916000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.284+0000 7f9164ff9700 1 -- 172.21.15.104:0/2711972192 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91600092e0 con 0x7f9168106b90 2024-06-09T22:26:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.284+0000 7f9164ff9700 1 -- 172.21.15.104:0/2711972192 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9160004bb0 con 0x7f9168106b90 2024-06-09T22:26:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.284+0000 7f916dc86700 1 -- 172.21.15.104:0/2711972192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91681a5210 con 0x7f9168106b90 2024-06-09T22:26:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.284+0000 7f9164ff9700 1 -- 172.21.15.104:0/2711972192 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160004280 con 0x7f9168106b90 2024-06-09T22:26:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.284+0000 7f916dc86700 1 -- 172.21.15.104:0/2711972192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91681a5710 con 0x7f9168106b90 2024-06-09T22:26:13.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.285+0000 7f916dc86700 1 -- 172.21.15.104:0/2711972192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9168102200 con 0x7f9168106b90 2024-06-09T22:26:13.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.287+0000 7f9164ff9700 1 -- 172.21.15.104:0/2711972192 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f916001b030 con 0x7f9168106b90 2024-06-09T22:26:13.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.287+0000 7f9164ff9700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f91500a2e10 0x7f91500a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:13.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.288+0000 7f91677fe700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f91500a2e10 0x7f91500a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:13.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.288+0000 7f9164ff9700 1 -- 172.21.15.104:0/2711972192 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f91600cf290 con 0x7f9168106b90 2024-06-09T22:26:13.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.288+0000 7f91677fe700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f91500a2e10 0x7f91500a52d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9158009ca0 tx=0x7f9158009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.292+0000 7f9164ff9700 1 -- 172.21.15.104:0/2711972192 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9160022020 con 0x7f9168106b90 2024-06-09T22:26:13.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:12 smithi194 ceph-mon[36856]: pgmap v160: 129 pgs: 129 active+clean; 99 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 216 op/s 2024-06-09T22:26:13.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.557+0000 7f916dc86700 1 -- 172.21.15.104:0/2711972192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f916802dca0 con 0x7f9168106b90 2024-06-09T22:26:13.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.558+0000 7f9164ff9700 1 -- 172.21.15.104:0/2711972192 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7f9160010040 con 0x7f9168106b90 2024-06-09T22:26:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:26:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:26:13.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.561+0000 7f916dc86700 1 -- 172.21.15.104:0/2711972192 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f91500a2e10 msgr2=0x7f91500a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:13.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.561+0000 7f916dc86700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f91500a2e10 0x7f91500a52d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9158009ca0 tx=0x7f9158009260 comp rx=0 tx=0).stop 2024-06-09T22:26:13.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.561+0000 7f916dc86700 1 -- 172.21.15.104:0/2711972192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9168106b90 msgr2=0x7f91681a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:13.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.561+0000 7f916dc86700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9168106b90 0x7f91681a4b00 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f916000ef30 tx=0x7f916000c3a0 comp rx=0 tx=0).stop 2024-06-09T22:26:13.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.561+0000 7f916dc86700 1 -- 172.21.15.104:0/2711972192 shutdown_connections 2024-06-09T22:26:13.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.561+0000 7f916dc86700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f91500a2e10 0x7f91500a52d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:13.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.561+0000 7f916dc86700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9168106b90 0x7f91681a4b00 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:13.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.561+0000 7f916dc86700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9168100b50 0x7f9168199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:13.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.561+0000 7f916dc86700 1 --2- 172.21.15.104:0/2711972192 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9168101510 0x7f9168199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:13.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.561+0000 7f916dc86700 1 -- 172.21.15.104:0/2711972192 >> 172.21.15.104:0/2711972192 conn(0x7f9168077a00 msgr2=0x7f9168103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:13.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.561+0000 7f916dc86700 1 -- 172.21.15.104:0/2711972192 shutdown_connections 2024-06-09T22:26:13.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:13.561+0000 7f916dc86700 1 -- 172.21.15.104:0/2711972192 wait complete. 2024-06-09T22:26:13.562 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:26:13.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 144464 4.94 MB/sec execute 79 sec latency 54.336 ms 2024-06-09T22:26:14.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:13 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2711972192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:26:14.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:13 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2711972192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:26:14.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:13 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2711972192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:26:14.381 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-09T22:26:14.630 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:14.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 146181 5.02 MB/sec execute 80 sec latency 14.203 ms 2024-06-09T22:26:15.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:14 smithi104 ceph-mon[32389]: pgmap v161: 129 pgs: 129 active+clean; 106 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.4 MiB/s wr, 291 op/s 2024-06-09T22:26:15.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:14 smithi153 ceph-mon[36599]: pgmap v161: 129 pgs: 129 active+clean; 106 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.4 MiB/s wr, 291 op/s 2024-06-09T22:26:15.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:14 smithi194 ceph-mon[36856]: pgmap v161: 129 pgs: 129 active+clean; 106 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.4 MiB/s wr, 291 op/s 2024-06-09T22:26:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.529+0000 7f6eac4df640 1 -- 172.21.15.104:0/2216800590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4103550 msgr2=0x7f6ea410ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.529+0000 7f6eac4df640 1 --2- 172.21.15.104:0/2216800590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4103550 0x7f6ea410ba30 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6e94009900 tx=0x7f6e9402a510 comp rx=0 tx=0).stop 2024-06-09T22:26:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.530+0000 7f6eac4df640 1 -- 172.21.15.104:0/2216800590 shutdown_connections 2024-06-09T22:26:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.530+0000 7f6eac4df640 1 --2- 172.21.15.104:0/2216800590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ea410bf70 0x7f6ea410e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.530+0000 7f6eac4df640 1 --2- 172.21.15.104:0/2216800590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4103550 0x7f6ea410ba30 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.530+0000 7f6eac4df640 1 --2- 172.21.15.104:0/2216800590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ea4102c30 0x7f6ea4103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.530+0000 7f6eac4df640 1 -- 172.21.15.104:0/2216800590 >> 172.21.15.104:0/2216800590 conn(0x7f6ea4077a20 msgr2=0x7f6ea4077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.530+0000 7f6eac4df640 1 -- 172.21.15.104:0/2216800590 shutdown_connections 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.530+0000 7f6eac4df640 1 -- 172.21.15.104:0/2216800590 wait complete. 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.530+0000 7f6eac4df640 1 Processor -- start 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.530+0000 7f6eac4df640 1 -- start start 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eac4df640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4102c30 0x7f6ea41981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eac4df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ea4103550 0x7f6ea4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eac4df640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ea410bf70 0x7f6ea419cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eac4df640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4111d70 con 0x7f6ea4102c30 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eac4df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4111bf0 con 0x7f6ea4103550 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eac4df640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4111ef0 con 0x7f6ea410bf70 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eaa27b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4102c30 0x7f6ea41981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eaa27b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4102c30 0x7f6ea41981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41164/0 (socket says 172.21.15.104:41164) 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eaa27b640 1 -- 172.21.15.104:0/3218210715 learned_addr learned my addr 172.21.15.104:0/3218210715 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6ea9a7a640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ea4103550 0x7f6ea4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ea410bf70 0x7f6ea419cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eaa27b640 1 -- 172.21.15.104:0/3218210715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ea410bf70 msgr2=0x7f6ea419cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eaa27b640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ea410bf70 0x7f6ea419cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eaa27b640 1 -- 172.21.15.104:0/3218210715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ea4103550 msgr2=0x7f6ea4198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eaa27b640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ea4103550 0x7f6ea4198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eaa27b640 1 -- 172.21.15.104:0/3218210715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea419d1e0 con 0x7f6ea4102c30 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eaa27b640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4102c30 0x7f6ea41981f0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000e9f0 tx=0x7f6ea000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6e9b7fe640 1 -- 172.21.15.104:0/3218210715 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea000cd80 con 0x7f6ea4102c30 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6e9b7fe640 1 -- 172.21.15.104:0/3218210715 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea0010910 con 0x7f6ea4102c30 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eac4df640 1 -- 172.21.15.104:0/3218210715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea419d4d0 con 0x7f6ea4102c30 2024-06-09T22:26:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6e9b7fe640 1 -- 172.21.15.104:0/3218210715 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0018990 con 0x7f6ea4102c30 2024-06-09T22:26:15.535 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.531+0000 7f6eac4df640 1 -- 172.21.15.104:0/3218210715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea41a4d60 con 0x7f6ea4102c30 2024-06-09T22:26:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaf1e0740 1 Processor -- start 2024-06-09T22:26:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaf1e0740 1 -- start start 2024-06-09T22:26:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6e9b7fe640 1 -- 172.21.15.104:0/3218210715 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f6ea0010ac0 con 0x7f6ea4102c30 2024-06-09T22:26:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaf1e0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f36d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaf1e0740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f377e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaf1e0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f37a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaf1e0740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55790f2d14f0 con 0x55790f36da90 2024-06-09T22:26:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaf1e0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55790f26f870 con 0x55790f378350 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaf1e0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55790f2cd4b0 con 0x55790f36d170 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaaa7c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f36d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaa27b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f377e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaaa7c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f36d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:41784/0 (socket says 172.21.15.104:41784) 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaaa7c640 1 -- 172.21.15.104:0/4044990495 learned_addr learned my addr 172.21.15.104:0/4044990495 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6ea9a7a640 1 --2- 172.21.15.104:0/4044990495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f37a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaaa7c640 1 -- 172.21.15.104:0/4044990495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 msgr2=0x55790f37a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/4044990495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f37a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.535+0000 7f6eaaa7c640 1 -- 172.21.15.104:0/4044990495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 msgr2=0x55790f377e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.536+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/4044990495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f377e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.536+0000 7f6eaaa7c640 1 -- 172.21.15.104:0/4044990495 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55790f28b9b0 con 0x55790f36d170 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.536+0000 7f6e9b7fe640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6e900a2da0 0x7f6e900a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.536+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/4044990495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f36d550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009f50 tx=0x7f6e9c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5bf1088aeb523c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.536+0000 7f6eaa27b640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6e900a2da0 0x7f6e900a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.536+0000 7f6e98bfe640 1 -- 172.21.15.104:0/4044990495 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c004d20 con 0x55790f36d170 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.536+0000 7f6e98bfe640 1 -- 172.21.15.104:0/4044990495 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c004050 con 0x55790f36d170 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.536+0000 7f6e9b7fe640 1 -- 172.21.15.104:0/3218210715 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6ea00cfab0 con 0x7f6ea4102c30 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.536+0000 7f6e98bfe640 1 -- 172.21.15.104:0/4044990495 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c005280 con 0x55790f36d170 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.537+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4044990495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 msgr2=0x55790f36d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.537+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4044990495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f36d550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009f50 tx=0x7f6e9c0118d0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.537+0000 7f6eaa27b640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6e900a2da0 0x7f6e900a5260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55790f36e710 tx=0x7f6ea00d35a0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.537+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4044990495 shutdown_connections 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.537+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4044990495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f37a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.537+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4044990495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f377e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.537+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4044990495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f36d550 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.537+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4044990495 >> 172.21.15.104:0/4044990495 conn(0x55790f369680 msgr2=0x55790f369a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.537+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4044990495 shutdown_connections 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.537+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4044990495 wait complete. 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.537+0000 7f6eaf1e0740 1 -- start start 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.537+0000 7f6eaf1e0740 1 Processor -- start 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6eaf1e0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f375310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6eaf1e0740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f375850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6eaf1e0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f375d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6eaf1e0740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55790f37af80 con 0x55790f36da90 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6eaf1e0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55790f25aae0 con 0x55790f378350 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6eaf1e0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55790f37b100 con 0x55790f36d170 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6eaf1e0740 10 client.0 _ll_register_callbacks cb 0x55790f2939b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6ea9a7a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f375850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6ea9a7a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f375850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41180/0 (socket says 172.21.15.104:41180) 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6ea9a7a640 1 -- 172.21.15.104:0/4211397303 learned_addr learned my addr 172.21.15.104:0/4211397303 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f375310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f375d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6ea9a7a640 1 -- 172.21.15.104:0/4211397303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 msgr2=0x55790f375310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6ea9a7a640 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f375310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6ea9a7a640 1 -- 172.21.15.104:0/4211397303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 msgr2=0x55790f375d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6ea9a7a640 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f375d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6ea9a7a640 1 -- 172.21.15.104:0/4211397303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55790efe5210 con 0x55790f36da90 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f375d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6ea9a7a640 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f375850 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f6e9402aa30 tx=0x7f6e940317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546c254d3e5ace19 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6e8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6e8dffb640 1 -- 172.21.15.104:0/4211397303 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94035040 con 0x55790f36da90 2024-06-09T22:26:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6e8dffb640 1 -- 172.21.15.104:0/4211397303 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6e94004240 con 0x55790f36da90 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.538+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4211397303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e9c004050 con 0x55790f36da90 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.539+0000 7f6e8dffb640 1 -- 172.21.15.104:0/4211397303 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6e9404f050 con 0x55790f36da90 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.539+0000 7f6eaf1e0740 10 client.16899 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.539+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4211397303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55790f0b1620 con 0x55790f36da90 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.539+0000 7f6e8dffb640 1 -- 172.21.15.104:0/4211397303 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c004050 con 0x55790f36da90 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.539+0000 7f6e8dffb640 1 -- 172.21.15.104:0/4211397303 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6e94004a10 con 0x55790f36da90 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.539+0000 7f6eaf1e0740 10 client.16899 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.539+0000 7f6eaf1e0740 10 client.16899 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.539+0000 7f6eaf1e0740 20 client.16899 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.539+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55790f388550 0x55790f38a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.539+0000 7f6eaf1e0740 4 client.16899 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.539+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4211397303 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55790f2e4f10 con 0x55790f388550 2024-06-09T22:26:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.539+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55790f388550 0x55790f38a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.546 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.540+0000 7f6e8dffb640 10 client.16899 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:15.546 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.540+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55790f388550 0x55790f38a960 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=790a3c5b39db100 server_cookie=11e0180ad714e82b in_seq=0 out_seq=0 2024-06-09T22:26:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.540+0000 7f6e8dffb640 1 -- 172.21.15.104:0/4211397303 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55790f2e4f10 con 0x55790f388550 2024-06-09T22:26:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.540+0000 7f6e8dffb640 10 client.16899 handle_command_reply: tid=0 2024-06-09T22:26:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.540+0000 7f6eaf1e0740 1 client.16899 shutdown 2024-06-09T22:26:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.540+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4211397303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 msgr2=0x55790f375850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.540+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f375850 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f6e9402aa30 tx=0x7f6e940317e0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.541+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4211397303 shutdown_connections 2024-06-09T22:26:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.541+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55790f388550 0x55790f38a960 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.541+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f375d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.541+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f375850 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.541+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4211397303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f375310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.541+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4211397303 >> 172.21.15.104:0/4211397303 conn(0x55790f369680 msgr2=0x55790f36f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.541+0000 7f6e8dffb640 0 client.16899 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.541+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4211397303 shutdown_connections 2024-06-09T22:26:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.541+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4211397303 wait complete. 2024-06-09T22:26:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.541+0000 7f6eaf1e0740 20 client.16899 trim_cache size 0 max 16384 2024-06-09T22:26:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.556+0000 7f6eaf1e0740 1 Processor -- start 2024-06-09T22:26:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.556+0000 7f6eaf1e0740 1 -- start start 2024-06-09T22:26:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaf1e0740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36d170 0x55790f36fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaf1e0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f36da90 0x55790f390270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaf1e0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f378350 0x55790f3907b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaf1e0740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41aede0 con 0x55790f36d170 2024-06-09T22:26:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaf1e0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41b0920 con 0x55790f36da90 2024-06-09T22:26:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaf1e0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41af4b0 con 0x55790f378350 2024-06-09T22:26:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaaa7c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36d170 0x55790f36fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaaa7c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36d170 0x55790f36fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41196/0 (socket says 172.21.15.104:41196) 2024-06-09T22:26:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaaa7c640 1 -- 172.21.15.104:0/4203366624 learned_addr learned my addr 172.21.15.104:0/4203366624 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6ea9a7a640 1 --2- 172.21.15.104:0/4203366624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f36da90 0x55790f390270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/4203366624 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f378350 0x55790f3907b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaaa7c640 1 -- 172.21.15.104:0/4203366624 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f378350 msgr2=0x55790f3907b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/4203366624 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f378350 0x55790f3907b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaaa7c640 1 -- 172.21.15.104:0/4203366624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f36da90 msgr2=0x55790f390270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/4203366624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f36da90 0x55790f390270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaaa7c640 1 -- 172.21.15.104:0/4203366624 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55790f363ae0 con 0x55790f36d170 2024-06-09T22:26:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/4203366624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36d170 0x55790f36fb30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c01b040 tx=0x7f6e9c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c29a32952bd4d3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6e8ffff640 1 -- 172.21.15.104:0/4203366624 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c030040 con 0x55790f36d170 2024-06-09T22:26:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6e8ffff640 1 -- 172.21.15.104:0/4203366624 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c004530 con 0x55790f36d170 2024-06-09T22:26:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.557+0000 7f6e8ffff640 1 -- 172.21.15.104:0/4203366624 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c027d10 con 0x55790f36d170 2024-06-09T22:26:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.558+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4203366624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36d170 msgr2=0x55790f36fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.558+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4203366624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36d170 0x55790f36fb30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c01b040 tx=0x7f6e9c0118d0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.558+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4203366624 shutdown_connections 2024-06-09T22:26:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.558+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4203366624 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f378350 0x55790f3907b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.558+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4203366624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f36da90 0x55790f390270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.558+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/4203366624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36d170 0x55790f36fb30 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.558+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4203366624 >> 172.21.15.104:0/4203366624 conn(0x55790f369680 msgr2=0x55790f3705d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.558+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4203366624 shutdown_connections 2024-06-09T22:26:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.558+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/4203366624 wait complete. 2024-06-09T22:26:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.558+0000 7f6eaf1e0740 1 -- start start 2024-06-09T22:26:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.558+0000 7f6eaf1e0740 1 Processor -- start 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6eaf1e0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f37b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6eaf1e0740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f37be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6eaf1e0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f3f1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6eaf1e0740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41b0250 con 0x55790f36da90 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6eaf1e0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41afb80 con 0x55790f378350 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6eaf1e0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41acda0 con 0x55790f36d170 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6eaf1e0740 10 client.0 _ll_register_callbacks cb 0x7f6e94009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6ea9a7a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f37be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6ea9a7a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f37be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41208/0 (socket says 172.21.15.104:41208) 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6ea9a7a640 1 -- 172.21.15.104:0/2451747582 learned_addr learned my addr 172.21.15.104:0/2451747582 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f37b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f3f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6ea9a7a640 1 -- 172.21.15.104:0/2451747582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 msgr2=0x55790f37b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6ea9a7a640 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f37b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6ea9a7a640 1 -- 172.21.15.104:0/2451747582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 msgr2=0x55790f3f1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6ea9a7a640 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f3f1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6ea9a7a640 1 -- 172.21.15.104:0/2451747582 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55790f3f1840 con 0x55790f36da90 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6ea9a7a640 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f37be30 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x55790f374b70 tx=0x7f6e94005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf6e9d6cf0ab69d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6e8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6e8dffb640 1 -- 172.21.15.104:0/2451747582 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94035070 con 0x55790f36da90 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6e8dffb640 1 -- 172.21.15.104:0/2451747582 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f6e940048d0 con 0x55790f36da90 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.559+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/2451747582 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e9c004530 con 0x55790f36da90 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.560+0000 7f6e8dffb640 1 -- 172.21.15.104:0/2451747582 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6e94061050 con 0x55790f36da90 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.560+0000 7f6eaf1e0740 10 client.16905 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.560+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/2451747582 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55790f3f2b80 con 0x55790f36da90 2024-06-09T22:26:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.560+0000 7f6e8dffb640 1 -- 172.21.15.104:0/2451747582 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c004530 con 0x55790f36da90 2024-06-09T22:26:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.560+0000 7f6e8dffb640 1 -- 172.21.15.104:0/2451747582 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6e94046480 con 0x55790f36da90 2024-06-09T22:26:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.560+0000 7f6eaf1e0740 10 client.16905 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.560+0000 7f6eaf1e0740 10 client.16905 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.560+0000 7f6eaf1e0740 20 client.16905 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.560+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55790f3f3320 0x55790f3f5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.560+0000 7f6eaf1e0740 4 client.16905 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:26:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.560+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/2451747582 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55790f2dec20 con 0x55790f3f3320 2024-06-09T22:26:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.560+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55790f3f3320 0x55790f3f5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.561+0000 7f6eaaa7c640 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55790f3f3320 0x55790f3f5740 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=417404a249e98690 server_cookie=c4d6544793184d90 in_seq=0 out_seq=0 2024-06-09T22:26:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.561+0000 7f6e8dffb640 10 client.16905 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.568+0000 7f6e8dffb640 1 -- 172.21.15.104:0/2451747582 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55790f2dec20 con 0x55790f3f3320 2024-06-09T22:26:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.568+0000 7f6e8dffb640 10 client.16905 handle_command_reply: tid=0 2024-06-09T22:26:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.568+0000 7f6eaf1e0740 1 client.16905 shutdown 2024-06-09T22:26:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/2451747582 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 msgr2=0x55790f37be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f37be30 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x55790f374b70 tx=0x7f6e94005e20 comp rx=0 tx=0).stop 2024-06-09T22:26:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/2451747582 shutdown_connections 2024-06-09T22:26:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55790f3f3320 0x55790f3f5740 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55790f378350 0x55790f3f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55790f36da90 0x55790f37be30 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6e8dffb640 0 client.16905 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eaf1e0740 1 --2- 172.21.15.104:0/2451747582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790f36d170 0x55790f37b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/2451747582 >> 172.21.15.104:0/2451747582 conn(0x55790f369680 msgr2=0x55790f3752b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/2451747582 shutdown_connections 2024-06-09T22:26:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eaf1e0740 1 -- 172.21.15.104:0/2451747582 wait complete. 2024-06-09T22:26:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eaf1e0740 20 client.16905 trim_cache size 0 max 16384 2024-06-09T22:26:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:26:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:26:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:26:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eac4df640 1 -- 172.21.15.104:0/3218210715 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6e900a2da0 msgr2=0x7f6e900a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eac4df640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6e900a2da0 0x7f6e900a5260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55790f36e710 tx=0x7f6ea00d35a0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eac4df640 1 -- 172.21.15.104:0/3218210715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4102c30 msgr2=0x7f6ea41981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eac4df640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4102c30 0x7f6ea41981f0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000e9f0 tx=0x7f6ea000eeb0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eac4df640 1 -- 172.21.15.104:0/3218210715 shutdown_connections 2024-06-09T22:26:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eac4df640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6e900a2da0 0x7f6e900a5260 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eac4df640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ea410bf70 0x7f6ea419cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eac4df640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ea4103550 0x7f6ea4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eac4df640 1 --2- 172.21.15.104:0/3218210715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4102c30 0x7f6ea41981f0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.569+0000 7f6eac4df640 1 -- 172.21.15.104:0/3218210715 >> 172.21.15.104:0/3218210715 conn(0x7f6ea4077a20 msgr2=0x7f6ea40fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.570+0000 7f6eac4df640 1 -- 172.21.15.104:0/3218210715 shutdown_connections 2024-06-09T22:26:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:15.570+0000 7f6eac4df640 1 -- 172.21.15.104:0/3218210715 wait complete. 2024-06-09T22:26:15.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 148388 5.11 MB/sec execute 81 sec latency 16.164 ms 2024-06-09T22:26:16.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.562+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/428854829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcd406b3a0 msgr2=0x7fbcd410f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:16.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.562+0000 7fbcdc2f8700 1 --2- 172.21.15.104:0/428854829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcd406b3a0 0x7fbcd410f310 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00b0a0 tx=0x7fbccc00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.562+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/428854829 shutdown_connections 2024-06-09T22:26:16.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.562+0000 7fbcdc2f8700 1 --2- 172.21.15.104:0/428854829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcd406b3a0 0x7fbcd410f310 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 --2- 172.21.15.104:0/428854829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd41084d0 0x7fbcd41088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 --2- 172.21.15.104:0/428854829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd406aa00 0x7fbcd406ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/428854829 >> 172.21.15.104:0/428854829 conn(0x7fbcd4078330 msgr2=0x7fbcd4078740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:16.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/428854829 shutdown_connections 2024-06-09T22:26:16.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/428854829 wait complete. 2024-06-09T22:26:16.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 Processor -- start 2024-06-09T22:26:16.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 -- start start 2024-06-09T22:26:16.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd406aa00 0x7fbcd4101450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcd406b3a0 0x7fbcd4101990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd41084d0 0x7fbcd4101ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd4112750 con 0x7fbcd406b3a0 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd41125d0 con 0x7fbcd41084d0 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.563+0000 7fbcdc2f8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd41128d0 con 0x7fbcd406aa00 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcd9893700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcd406b3a0 0x7fbcd4101990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcda895700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd41084d0 0x7fbcd4101ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcd9893700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcd406b3a0 0x7fbcd4101990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46044/0 (socket says 172.21.15.104:46044) 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcd9893700 1 -- 172.21.15.104:0/3715641146 learned_addr learned my addr 172.21.15.104:0/3715641146 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcda094700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd406aa00 0x7fbcd4101450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcda895700 1 -- 172.21.15.104:0/3715641146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd406aa00 msgr2=0x7fbcd4101450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcda895700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd406aa00 0x7fbcd4101450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcda895700 1 -- 172.21.15.104:0/3715641146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcd406b3a0 msgr2=0x7fbcd4101990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:16.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcda895700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcd406b3a0 0x7fbcd4101990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcda895700 1 -- 172.21.15.104:0/3715641146 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd41a70f0 con 0x7fbcd41084d0 2024-06-09T22:26:16.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcda094700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd406aa00 0x7fbcd4101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:16.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcd9893700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcd406b3a0 0x7fbcd4101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:16.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbcda895700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd41084d0 0x7fbcd4101ed0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00b0a0 tx=0x7fbccc004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:16.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.564+0000 7fbccb7fe700 1 -- 172.21.15.104:0/3715641146 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc021070 con 0x7fbcd41084d0 2024-06-09T22:26:16.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.565+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/3715641146 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd41a7380 con 0x7fbcd41084d0 2024-06-09T22:26:16.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.565+0000 7fbccb7fe700 1 -- 172.21.15.104:0/3715641146 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbccc007b80 con 0x7fbcd41084d0 2024-06-09T22:26:16.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.565+0000 7fbccb7fe700 1 -- 172.21.15.104:0/3715641146 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc005040 con 0x7fbcd41084d0 2024-06-09T22:26:16.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.565+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/3715641146 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd41a7830 con 0x7fbcd41084d0 2024-06-09T22:26:16.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.567+0000 7fbccb7fe700 1 -- 172.21.15.104:0/3715641146 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fbccc024070 con 0x7fbcd41084d0 2024-06-09T22:26:16.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.568+0000 7fbccb7fe700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbcc00a2d90 0x7fbcc00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.568+0000 7fbcda094700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbcc00a2d90 0x7fbcc00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.568+0000 7fbccb7fe700 1 -- 172.21.15.104:0/3715641146 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fbccc00f020 con 0x7fbcd41084d0 2024-06-09T22:26:16.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.569+0000 7fbcda094700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbcc00a2d90 0x7fbcc00a5250 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009fd0 tx=0x7fbcd0009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:16.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.569+0000 7fbce6b7bb80 1 Processor -- start 2024-06-09T22:26:16.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.569+0000 7fbce6b7bb80 1 -- start start 2024-06-09T22:26:16.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.569+0000 7fbce6b7bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3af60 0x56323fb3b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.569+0000 7fbce6b7bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3b880 0x56323fb35260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.569+0000 7fbce6b7bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb35bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.569+0000 7fbce6b7bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56323fa27f60 con 0x56323fb3af60 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.569+0000 7fbce6b7bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56323fa86850 con 0x56323fb357a0 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.569+0000 7fbce6b7bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56323f9ee460 con 0x56323fb3b880 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.569+0000 7fbcda094700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3af60 0x56323fb3b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.569+0000 7fbcd9893700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3b880 0x56323fb35260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.569+0000 7fbcda094700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3af60 0x56323fb3b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46058/0 (socket says 172.21.15.104:46058) 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.570+0000 7fbcda094700 1 -- 172.21.15.104:0/3942508183 learned_addr learned my addr 172.21.15.104:0/3942508183 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.570+0000 7fbcda895700 1 --2- 172.21.15.104:0/3942508183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb35bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.570+0000 7fbcda094700 1 -- 172.21.15.104:0/3942508183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3b880 msgr2=0x56323fb35260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.570+0000 7fbcda094700 1 --2- 172.21.15.104:0/3942508183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3b880 0x56323fb35260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.570+0000 7fbcda094700 1 -- 172.21.15.104:0/3942508183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 msgr2=0x56323fb35bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.570+0000 7fbcda094700 1 --2- 172.21.15.104:0/3942508183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb35bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.570+0000 7fbcda094700 1 -- 172.21.15.104:0/3942508183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56323f9d58d0 con 0x56323fb3af60 2024-06-09T22:26:16.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.570+0000 7fbcd9893700 1 --2- 172.21.15.104:0/3942508183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3b880 0x56323fb35260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:16.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.570+0000 7fbcda094700 1 --2- 172.21.15.104:0/3942508183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3af60 0x56323fb3b340 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000e720 tx=0x7fbcd0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b744f2f3c243dfb6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:16.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.570+0000 7fbcc8d70700 1 -- 172.21.15.104:0/3942508183 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0026600 con 0x56323fb3af60 2024-06-09T22:26:16.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.570+0000 7fbcc8d70700 1 -- 172.21.15.104:0/3942508183 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcd0026c80 con 0x56323fb3af60 2024-06-09T22:26:16.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.570+0000 7fbcc8d70700 1 -- 172.21.15.104:0/3942508183 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd001ed20 con 0x56323fb3af60 2024-06-09T22:26:16.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.571+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/3942508183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3af60 msgr2=0x56323fb3b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:16.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.571+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/3942508183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3af60 0x56323fb3b340 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000e720 tx=0x7fbcd0014000 comp rx=0 tx=0).stop 2024-06-09T22:26:16.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.571+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/3942508183 shutdown_connections 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.571+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/3942508183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3af60 0x56323fb3b340 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.571+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/3942508183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb35bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.571+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/3942508183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3b880 0x56323fb35260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.571+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/3942508183 >> 172.21.15.104:0/3942508183 conn(0x56323fa62080 msgr2=0x56323fb30220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.571+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/3942508183 shutdown_connections 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.571+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/3942508183 wait complete. 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbce6b7bb80 1 -- start start 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbce6b7bb80 1 Processor -- start 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbce6b7bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb357a0 0x56323fb41610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbce6b7bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb3af60 0x56323fb3e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbce6b7bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3b880 0x56323fb3ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbce6b7bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56323fa26fc0 con 0x56323fb357a0 2024-06-09T22:26:16.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbce6b7bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56323fa82810 con 0x56323fb3af60 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbce6b7bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56323fb42260 con 0x56323fb3b880 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbce6b7bb80 10 client.0 _ll_register_callbacks cb 0x56323fa1c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbcda094700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb357a0 0x56323fb41610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbcda094700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb357a0 0x56323fb41610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46060/0 (socket says 172.21.15.104:46060) 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbcda094700 1 -- 172.21.15.104:0/679070413 learned_addr learned my addr 172.21.15.104:0/679070413 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbcd9893700 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb3af60 0x56323fb3e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbcda895700 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3b880 0x56323fb3ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbcd9893700 1 -- 172.21.15.104:0/679070413 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3b880 msgr2=0x56323fb3ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbcd9893700 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3b880 0x56323fb3ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbcd9893700 1 -- 172.21.15.104:0/679070413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb357a0 msgr2=0x56323fb41610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbcd9893700 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb357a0 0x56323fb41610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbcd9893700 1 -- 172.21.15.104:0/679070413 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56323fa4c750 con 0x56323fb3af60 2024-06-09T22:26:16.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.572+0000 7fbcda094700 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb357a0 0x56323fb41610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbcd9893700 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb3af60 0x56323fb3e910 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbcd40788a0 tx=0x7fbcc4009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b412bf5725579a0c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbcbdffb700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbcbdffb700 1 -- 172.21.15.104:0/679070413 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc400f970 con 0x56323fb3af60 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/679070413 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcd0026c80 con 0x56323fb3af60 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbcbdffb700 1 -- 172.21.15.104:0/679070413 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fbcc400fb10 con 0x56323fb3af60 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbcbdffb700 1 -- 172.21.15.104:0/679070413 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbcc400fd10 con 0x56323fb3af60 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbce6b7bb80 10 client.26237 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/679070413 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56323fb4ca70 con 0x56323fb3af60 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbcbdffb700 1 -- 172.21.15.104:0/679070413 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcd0026c80 con 0x56323fb3af60 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbcbdffb700 1 -- 172.21.15.104:0/679070413 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fbcc4005420 con 0x56323fb3af60 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbce6b7bb80 10 client.26237 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbce6b7bb80 10 client.26237 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:26:16.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbce6b7bb80 20 client.26237 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:16.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.573+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x56323fb4f0b0 0x56323fb514a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.574+0000 7fbce6b7bb80 4 client.26237 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:16.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.574+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/679070413 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56323fa93d50 con 0x56323fb4f0b0 2024-06-09T22:26:16.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.574+0000 7fbcda094700 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x56323fb4f0b0 0x56323fb514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.574+0000 7fbcda094700 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x56323fb4f0b0 0x56323fb514a0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9eb5e75cf5fbe639 server_cookie=96ab6485c1fe06d4 in_seq=0 out_seq=0 2024-06-09T22:26:16.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.574+0000 7fbcbdffb700 10 client.26237 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:26:16.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbcbdffb700 1 -- 172.21.15.104:0/679070413 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56323fa93d50 con 0x56323fb4f0b0 2024-06-09T22:26:16.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbcbdffb700 10 client.26237 handle_command_reply: tid=0 2024-06-09T22:26:16.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbce6b7bb80 1 client.26237 shutdown 2024-06-09T22:26:16.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/679070413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb3af60 msgr2=0x56323fb3e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:16.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb3af60 0x56323fb3e910 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbcd40788a0 tx=0x7fbcc4009f30 comp rx=0 tx=0).stop 2024-06-09T22:26:16.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/679070413 shutdown_connections 2024-06-09T22:26:16.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb357a0 0x56323fb41610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x56323fb4f0b0 0x56323fb514a0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3b880 0x56323fb3ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/679070413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb3af60 0x56323fb3e910 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/679070413 >> 172.21.15.104:0/679070413 conn(0x56323fa62080 msgr2=0x56323fb2f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:16.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbcbdffb700 0 client.26237 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:26:16.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/679070413 shutdown_connections 2024-06-09T22:26:16.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/679070413 wait complete. 2024-06-09T22:26:16.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.575+0000 7fbce6b7bb80 20 client.26237 trim_cache size 0 max 16384 2024-06-09T22:26:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.590+0000 7fbce6b7bb80 1 Processor -- start 2024-06-09T22:26:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.590+0000 7fbce6b7bb80 1 -- start start 2024-06-09T22:26:16.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.590+0000 7fbce6b7bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb3efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.590+0000 7fbce6b7bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 0x56323fb3d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.590+0000 7fbce6b7bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 0x56323fb2fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.590+0000 7fbce6b7bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd41b06c0 con 0x56323fb3b880 2024-06-09T22:26:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.590+0000 7fbce6b7bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd41b2200 con 0x56323fb357a0 2024-06-09T22:26:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.590+0000 7fbce6b7bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd41b0d90 con 0x56323fb3af60 2024-06-09T22:26:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcda895700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 0x56323fb2fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcd9893700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 0x56323fb3d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcd9893700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 0x56323fb3d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:51188/0 (socket says 172.21.15.104:51188) 2024-06-09T22:26:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcda895700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 0x56323fb2fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46074/0 (socket says 172.21.15.104:46074) 2024-06-09T22:26:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcd9893700 1 -- 172.21.15.104:0/1400000555 learned_addr learned my addr 172.21.15.104:0/1400000555 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcda094700 1 --2- 172.21.15.104:0/1400000555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb3efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcd9893700 1 -- 172.21.15.104:0/1400000555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 msgr2=0x56323fb3efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcd9893700 1 --2- 172.21.15.104:0/1400000555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb3efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcd9893700 1 -- 172.21.15.104:0/1400000555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 msgr2=0x56323fb2fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcd9893700 1 --2- 172.21.15.104:0/1400000555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 0x56323fb2fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcd9893700 1 -- 172.21.15.104:0/1400000555 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56323fb3cec0 con 0x56323fb3af60 2024-06-09T22:26:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcda895700 1 --2- 172.21.15.104:0/1400000555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 0x56323fb2fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcd9893700 1 --2- 172.21.15.104:0/1400000555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 0x56323fb3d8b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbcc401c560 tx=0x7fbcc4009cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d28bb9d05c8546a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcbffff700 1 -- 172.21.15.104:0/1400000555 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc400f970 con 0x56323fb3af60 2024-06-09T22:26:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcbffff700 1 -- 172.21.15.104:0/1400000555 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcc400fb10 con 0x56323fb3af60 2024-06-09T22:26:16.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.591+0000 7fbcbffff700 1 -- 172.21.15.104:0/1400000555 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc40107d0 con 0x56323fb3af60 2024-06-09T22:26:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.592+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/1400000555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 msgr2=0x56323fb3d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.592+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/1400000555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 0x56323fb3d8b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbcc401c560 tx=0x7fbcc4009cc0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.592+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/1400000555 shutdown_connections 2024-06-09T22:26:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.592+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/1400000555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 0x56323fb2fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.592+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/1400000555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb3efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.592+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/1400000555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 0x56323fb3d8b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.592+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/1400000555 >> 172.21.15.104:0/1400000555 conn(0x56323fa62080 msgr2=0x56323fb3e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.592+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/1400000555 shutdown_connections 2024-06-09T22:26:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.592+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/1400000555 wait complete. 2024-06-09T22:26:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbce6b7bb80 1 -- start start 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbce6b7bb80 1 Processor -- start 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbce6b7bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb47520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbce6b7bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 0x56323fb47a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbce6b7bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 0x56323fb55ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbce6b7bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd41b1b30 con 0x56323fb3b880 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbce6b7bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd41b1460 con 0x56323fb357a0 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbce6b7bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd41ae640 con 0x56323fb3af60 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbce6b7bb80 10 client.0 _ll_register_callbacks cb 0x56323fa1c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcda094700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb47520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcd9893700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 0x56323fb47a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcd9893700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 0x56323fb47a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:51196/0 (socket says 172.21.15.104:51196) 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcda094700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb47520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:33682/0 (socket says 172.21.15.104:33682) 2024-06-09T22:26:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcd9893700 1 -- 172.21.15.104:0/4081010080 learned_addr learned my addr 172.21.15.104:0/4081010080 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcda895700 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 0x56323fb55ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcd9893700 1 -- 172.21.15.104:0/4081010080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 msgr2=0x56323fb47520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcd9893700 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb47520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcd9893700 1 -- 172.21.15.104:0/4081010080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 msgr2=0x56323fb55ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcd9893700 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 0x56323fb55ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcd9893700 1 -- 172.21.15.104:0/4081010080 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56323fb56410 con 0x56323fb3af60 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcda094700 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb47520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.593+0000 7fbcda895700 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 0x56323fb55ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbcd9893700 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 0x56323fb47a60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4005790 tx=0x7fbcc401cd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2af4ee4dc0fc844 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbcbdffb700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbcbdffb700 1 -- 172.21.15.104:0/4081010080 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc400f970 con 0x56323fb3af60 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/4081010080 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcc400fb10 con 0x56323fb3af60 2024-06-09T22:26:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbcbdffb700 1 -- 172.21.15.104:0/4081010080 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fbcc4010640 con 0x56323fb3af60 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbcbdffb700 1 -- 172.21.15.104:0/4081010080 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbcc4010840 con 0x56323fb3af60 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbce6b7bb80 10 client.26326 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/4081010080 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56323fbb9510 con 0x56323fb3af60 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbcbdffb700 1 -- 172.21.15.104:0/4081010080 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcc400fb10 con 0x56323fb3af60 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbcbdffb700 1 -- 172.21.15.104:0/4081010080 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fbcc4029810 con 0x56323fb3af60 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbce6b7bb80 10 client.26326 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbce6b7bb80 10 client.26326 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbce6b7bb80 20 client.26326 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x56323fbb9a60 0x56323fbbbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbce6b7bb80 4 client.26326 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.594+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/4081010080 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56323fa8e2d0 con 0x56323fbb9a60 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbcda094700 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x56323fbb9a60 0x56323fbbbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbcda094700 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x56323fbb9a60 0x56323fbbbe80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e67f851f61ff1687 server_cookie=bef7da6f086f03cf in_seq=0 out_seq=0 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbcbdffb700 10 client.26326 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbcbdffb700 1 -- 172.21.15.104:0/4081010080 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56323fa8e2d0 con 0x56323fbb9a60 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbcbdffb700 10 client.26326 handle_command_reply: tid=0 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbce6b7bb80 1 client.26326 shutdown 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/4081010080 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 msgr2=0x56323fb47a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 0x56323fb47a60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4005790 tx=0x7fbcc401cd40 comp rx=0 tx=0).stop 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/4081010080 shutdown_connections 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56323fb3b880 0x56323fb55ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x56323fbb9a60 0x56323fbbbe80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56323fb357a0 0x56323fb47520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbcbdffb700 0 client.26326 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbce6b7bb80 1 --2- 172.21.15.104:0/4081010080 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56323fb3af60 0x56323fb47a60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.595+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/4081010080 >> 172.21.15.104:0/4081010080 conn(0x56323fa62080 msgr2=0x56323fb3bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:16.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/4081010080 shutdown_connections 2024-06-09T22:26:16.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbce6b7bb80 1 -- 172.21.15.104:0/4081010080 wait complete. 2024-06-09T22:26:16.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbce6b7bb80 20 client.26326 trim_cache size 0 max 16384 2024-06-09T22:26:16.603 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:26:16.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/3715641146 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbcc00a2d90 msgr2=0x7fbcc00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:16.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbcdc2f8700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbcc00a2d90 0x7fbcc00a5250 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009fd0 tx=0x7fbcd0009260 comp rx=0 tx=0).stop 2024-06-09T22:26:16.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/3715641146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd41084d0 msgr2=0x7fbcd4101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:16.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbcdc2f8700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd41084d0 0x7fbcd4101ed0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00b0a0 tx=0x7fbccc004d00 comp rx=0 tx=0).stop 2024-06-09T22:26:16.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/3715641146 shutdown_connections 2024-06-09T22:26:16.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbcdc2f8700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbcc00a2d90 0x7fbcc00a5250 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbcdc2f8700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd41084d0 0x7fbcd4101ed0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbcdc2f8700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcd406aa00 0x7fbcd4101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbcdc2f8700 1 --2- 172.21.15.104:0/3715641146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcd406b3a0 0x7fbcd4101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:16.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/3715641146 >> 172.21.15.104:0/3715641146 conn(0x7fbcd4078330 msgr2=0x7fbcd40ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:16.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/3715641146 shutdown_connections 2024-06-09T22:26:16.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:16.596+0000 7fbcdc2f8700 1 -- 172.21.15.104:0/3715641146 wait complete. 2024-06-09T22:26:16.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 149733 5.13 MB/sec execute 82 sec latency 65.698 ms 2024-06-09T22:26:17.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:16 smithi153 ceph-mon[36599]: pgmap v162: 129 pgs: 129 active+clean; 106 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-06-09T22:26:17.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:16 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:26:17.257 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:16 smithi104 ceph-mon[32389]: pgmap v162: 129 pgs: 129 active+clean; 106 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-06-09T22:26:17.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:16 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:26:17.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:16 smithi194 ceph-mon[36856]: pgmap v162: 129 pgs: 129 active+clean; 106 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-06-09T22:26:17.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:16 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:26:17.433 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-09T22:26:17.686 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:17.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 151974 5.22 MB/sec execute 83 sec latency 18.461 ms 2024-06-09T22:26:18.086 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:17 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:26:18.087 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:17 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:26:18.087 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:17 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:26:18.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:17 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:26:18.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:17 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:26:18.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:17 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:26:18.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:17 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:26:18.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:17 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:26:18.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:17 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:26:18.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 153464 5.24 MB/sec execute 84 sec latency 14.110 ms 2024-06-09T22:26:19.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:19 smithi104 ceph-mon[32389]: pgmap v163: 129 pgs: 129 active+clean; 106 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-06-09T22:26:19.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:19 smithi194 ceph-mon[36856]: pgmap v163: 129 pgs: 129 active+clean; 106 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-06-09T22:26:19.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:19 smithi153 ceph-mon[36599]: pgmap v163: 129 pgs: 129 active+clean; 106 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-06-09T22:26:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.580+0000 7f4528b06700 1 -- 172.21.15.104:0/4154374488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4524108510 msgr2=0x7f45241088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.580+0000 7f4528b06700 1 --2- 172.21.15.104:0/4154374488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4524108510 0x7f45241088f0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f4514009990 tx=0x7f4514009ca0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.580+0000 7f4528b06700 1 -- 172.21.15.104:0/4154374488 shutdown_connections 2024-06-09T22:26:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.580+0000 7f4528b06700 1 --2- 172.21.15.104:0/4154374488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4524108510 0x7f45241088f0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.580+0000 7f4528b06700 1 --2- 172.21.15.104:0/4154374488 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4524101440 0x7f452410f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.580+0000 7f4528b06700 1 --2- 172.21.15.104:0/4154374488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4524100a80 0x7f4524100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.580+0000 7f4528b06700 1 -- 172.21.15.104:0/4154374488 >> 172.21.15.104:0/4154374488 conn(0x7f4524077a00 msgr2=0x7f45240fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.580+0000 7f4528b06700 1 -- 172.21.15.104:0/4154374488 shutdown_connections 2024-06-09T22:26:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.580+0000 7f4528b06700 1 -- 172.21.15.104:0/4154374488 wait complete. 2024-06-09T22:26:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f4528b06700 1 Processor -- start 2024-06-09T22:26:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f4528b06700 1 -- start start 2024-06-09T22:26:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f4528b06700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4524100a80 0x7f452419add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f4528b06700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4524101440 0x7f452419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f4528b06700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4524108510 0x7f452419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f4528b06700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4524112790 con 0x7f4524101440 2024-06-09T22:26:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f4528b06700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4524112610 con 0x7f4524108510 2024-06-09T22:26:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f4528b06700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4524112910 con 0x7f4524100a80 2024-06-09T22:26:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f452259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4524100a80 0x7f452419add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f4521d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4524101440 0x7f452419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f452259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4524100a80 0x7f452419add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:51216/0 (socket says 172.21.15.104:51216) 2024-06-09T22:26:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f4521d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4524101440 0x7f452419b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46110/0 (socket says 172.21.15.104:46110) 2024-06-09T22:26:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f452259c700 1 -- 172.21.15.104:0/456141971 learned_addr learned my addr 172.21.15.104:0/456141971 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.581+0000 7f4522d9d700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4524108510 0x7f452419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.582+0000 7f452259c700 1 -- 172.21.15.104:0/456141971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4524108510 msgr2=0x7f452419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.582+0000 7f452259c700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4524108510 0x7f452419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.582+0000 7f452259c700 1 -- 172.21.15.104:0/456141971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4524101440 msgr2=0x7f452419b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.582+0000 7f452259c700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4524101440 0x7f452419b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.582+0000 7f452259c700 1 -- 172.21.15.104:0/456141971 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452419fde0 con 0x7f4524100a80 2024-06-09T22:26:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.582+0000 7f452259c700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4524100a80 0x7f452419add0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4514000c00 tx=0x7f451400bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.582+0000 7f45137fe700 1 -- 172.21.15.104:0/456141971 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4514027070 con 0x7f4524100a80 2024-06-09T22:26:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.582+0000 7f45137fe700 1 -- 172.21.15.104:0/456141971 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4514003d70 con 0x7f4524100a80 2024-06-09T22:26:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.582+0000 7f45137fe700 1 -- 172.21.15.104:0/456141971 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451400f460 con 0x7f4524100a80 2024-06-09T22:26:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.582+0000 7f4528b06700 1 -- 172.21.15.104:0/456141971 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45241a0070 con 0x7f4524100a80 2024-06-09T22:26:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.582+0000 7f4528b06700 1 -- 172.21.15.104:0/456141971 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45241a0440 con 0x7f4524100a80 2024-06-09T22:26:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.584+0000 7f45137fe700 1 -- 172.21.15.104:0/456141971 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f451402a070 con 0x7f4524100a80 2024-06-09T22:26:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.585+0000 7f45137fe700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f450c0a2e10 0x7f450c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.585+0000 7f4521d9b700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f450c0a2e10 0x7f450c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.585+0000 7f45137fe700 1 -- 172.21.15.104:0/456141971 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f45140d0510 con 0x7f4524100a80 2024-06-09T22:26:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.586+0000 7f4521d9b700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f450c0a2e10 0x7f450c0a52d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f452410eff0 tx=0x7f4518009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.586+0000 7f4533389b80 1 Processor -- start 2024-06-09T22:26:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f4533389b80 1 -- start start 2024-06-09T22:26:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f4533389b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f87f60 0x55f104f88340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f4533389b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f88880 0x55f104f82260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f4533389b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f827a0 0x55f104f82bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f4533389b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55f104e74f60 con 0x55f104f827a0 2024-06-09T22:26:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f4533389b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f104ed3850 con 0x55f104f87f60 2024-06-09T22:26:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f4533389b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f104e3b460 con 0x55f104f88880 2024-06-09T22:26:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f452259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f87f60 0x55f104f88340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f4521d9b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f88880 0x55f104f82260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f452259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f87f60 0x55f104f88340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:33700/0 (socket says 172.21.15.104:33700) 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f452259c700 1 -- 172.21.15.104:0/1390338274 learned_addr learned my addr 172.21.15.104:0/1390338274 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f4522d9d700 1 --2- 172.21.15.104:0/1390338274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f827a0 0x55f104f82bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f452259c700 1 -- 172.21.15.104:0/1390338274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f88880 msgr2=0x55f104f82260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f452259c700 1 --2- 172.21.15.104:0/1390338274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f88880 0x55f104f82260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f452259c700 1 -- 172.21.15.104:0/1390338274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f827a0 msgr2=0x55f104f82bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f452259c700 1 --2- 172.21.15.104:0/1390338274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f827a0 0x55f104f82bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.587+0000 7f452259c700 1 -- 172.21.15.104:0/1390338274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f104e228d0 con 0x55f104f87f60 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.588+0000 7f452259c700 1 --2- 172.21.15.104:0/1390338274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f87f60 0x55f104f88340 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f45140a4e00 tx=0x7f451409c1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e581d84749389713 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.588+0000 7f4510ff9700 1 -- 172.21.15.104:0/1390338274 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451409c750 con 0x55f104f87f60 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.588+0000 7f4510ff9700 1 -- 172.21.15.104:0/1390338274 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45140a0b00 con 0x55f104f87f60 2024-06-09T22:26:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.588+0000 7f4510ff9700 1 -- 172.21.15.104:0/1390338274 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4514022700 con 0x55f104f87f60 2024-06-09T22:26:19.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.589+0000 7f4533389b80 1 -- 172.21.15.104:0/1390338274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f87f60 msgr2=0x55f104f88340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.589+0000 7f4533389b80 1 --2- 172.21.15.104:0/1390338274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f87f60 0x55f104f88340 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f45140a4e00 tx=0x7f451409c1a0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.589+0000 7f4533389b80 1 -- 172.21.15.104:0/1390338274 shutdown_connections 2024-06-09T22:26:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.589+0000 7f4533389b80 1 --2- 172.21.15.104:0/1390338274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f827a0 0x55f104f82bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.589+0000 7f4533389b80 1 --2- 172.21.15.104:0/1390338274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f87f60 0x55f104f88340 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.589+0000 7f4533389b80 1 --2- 172.21.15.104:0/1390338274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f88880 0x55f104f82260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.589+0000 7f4533389b80 1 -- 172.21.15.104:0/1390338274 >> 172.21.15.104:0/1390338274 conn(0x55f104eaf080 msgr2=0x55f104f7d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.589+0000 7f4533389b80 1 -- 172.21.15.104:0/1390338274 shutdown_connections 2024-06-09T22:26:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.589+0000 7f4533389b80 1 -- 172.21.15.104:0/1390338274 wait complete. 2024-06-09T22:26:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.589+0000 7f4533389b80 1 -- start start 2024-06-09T22:26:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.589+0000 7f4533389b80 1 Processor -- start 2024-06-09T22:26:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.589+0000 7f4533389b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f827a0 0x55f104f8e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4533389b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f87f60 0x55f104f8b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4533389b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f88880 0x55f104f8beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4533389b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55f104e73fc0 con 0x55f104f88880 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4533389b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f104ecf810 con 0x55f104f87f60 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4533389b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f104f8f010 con 0x55f104f827a0 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f452259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f827a0 0x55f104f8e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4522d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f88880 0x55f104f8beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f452259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f827a0 0x55f104f8e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:51240/0 (socket says 172.21.15.104:51240) 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f452259c700 1 -- 172.21.15.104:0/2521570680 learned_addr learned my addr 172.21.15.104:0/2521570680 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4533389b80 10 client.0 _ll_register_callbacks cb 0x55f104e695b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4521d9b700 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f87f60 0x55f104f8b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f452259c700 1 -- 172.21.15.104:0/2521570680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f87f60 msgr2=0x55f104f8b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f452259c700 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f87f60 0x55f104f8b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f452259c700 1 -- 172.21.15.104:0/2521570680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f88880 msgr2=0x55f104f8beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f452259c700 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f88880 0x55f104f8beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f452259c700 1 -- 172.21.15.104:0/2521570680 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f104e99750 con 0x55f104f827a0 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4522d9d700 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f88880 0x55f104f8beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f452259c700 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f827a0 0x55f104f8e670 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f451400b2e0 tx=0x7f451400b490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41add270eeefaffc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4509ffb700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4509ffb700 1 -- 172.21.15.104:0/2521570680 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451409c750 con 0x55f104f827a0 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4533389b80 1 -- 172.21.15.104:0/2521570680 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45140a0b00 con 0x55f104f827a0 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.590+0000 7f4509ffb700 1 -- 172.21.15.104:0/2521570680 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f45140094b0 con 0x55f104f827a0 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.591+0000 7f4509ffb700 1 -- 172.21.15.104:0/2521570680 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f45140d0180 con 0x55f104f827a0 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.591+0000 7f4533389b80 10 client.26338 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.591+0000 7f4533389b80 1 -- 172.21.15.104:0/2521570680 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f104f99740 con 0x55f104f827a0 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.591+0000 7f4509ffb700 1 -- 172.21.15.104:0/2521570680 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45140a0b00 con 0x55f104f827a0 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.591+0000 7f4509ffb700 1 -- 172.21.15.104:0/2521570680 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f45140a4ea0 con 0x55f104f827a0 2024-06-09T22:26:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.591+0000 7f4533389b80 10 client.26338 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.591+0000 7f4533389b80 10 client.26338 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.591+0000 7f4533389b80 20 client.26338 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.591+0000 7f4533389b80 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55f104f9bc10 0x55f104f9e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.591+0000 7f4533389b80 4 client.26338 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.591+0000 7f4533389b80 1 -- 172.21.15.104:0/2521570680 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f104ee0d50 con 0x55f104f9bc10 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.591+0000 7f4521d9b700 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55f104f9bc10 0x55f104f9e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.592+0000 7f4509ffb700 10 client.26338 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.592+0000 7f4521d9b700 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55f104f9bc10 0x55f104f9e000 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c5eba91f4dad1b77 server_cookie=ad1dab10b91fa308 in_seq=0 out_seq=0 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.592+0000 7f4509ffb700 1 -- 172.21.15.104:0/2521570680 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55f104ee0d50 con 0x55f104f9bc10 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.592+0000 7f4509ffb700 10 client.26338 handle_command_reply: tid=0 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.592+0000 7f4533389b80 1 client.26338 shutdown 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.593+0000 7f4533389b80 1 -- 172.21.15.104:0/2521570680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f827a0 msgr2=0x55f104f8e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.593+0000 7f4533389b80 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f827a0 0x55f104f8e670 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f451400b2e0 tx=0x7f451400b490 comp rx=0 tx=0).stop 2024-06-09T22:26:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.593+0000 7f4533389b80 1 -- 172.21.15.104:0/2521570680 shutdown_connections 2024-06-09T22:26:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.593+0000 7f4533389b80 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55f104f9bc10 0x55f104f9e000 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.593+0000 7f4533389b80 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f88880 0x55f104f8beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.593+0000 7f4533389b80 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f827a0 0x55f104f8e670 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.593+0000 7f4533389b80 1 --2- 172.21.15.104:0/2521570680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f87f60 0x55f104f8b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.593+0000 7f4509ffb700 0 client.26338 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:26:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.593+0000 7f4533389b80 1 -- 172.21.15.104:0/2521570680 >> 172.21.15.104:0/2521570680 conn(0x55f104eaf080 msgr2=0x55f104f7c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.593+0000 7f4533389b80 1 -- 172.21.15.104:0/2521570680 shutdown_connections 2024-06-09T22:26:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.593+0000 7f4533389b80 1 -- 172.21.15.104:0/2521570680 wait complete. 2024-06-09T22:26:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.593+0000 7f4533389b80 20 client.26338 trim_cache size 0 max 16384 2024-06-09T22:26:19.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f4533389b80 1 Processor -- start 2024-06-09T22:26:19.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f4533389b80 1 -- start start 2024-06-09T22:26:19.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f4533389b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f827a0 0x55f104f82260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f4533389b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f87f60 0x55f104f894f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f4533389b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f88880 0x55f104f7ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f4533389b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f45241b0600 con 0x55f104f87f60 2024-06-09T22:26:19.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f4533389b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45241b2140 con 0x55f104f827a0 2024-06-09T22:26:19.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f4533389b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f45241b0cd0 con 0x55f104f88880 2024-06-09T22:26:19.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f4521d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f87f60 0x55f104f894f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f452259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f827a0 0x55f104f82260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f452259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f827a0 0x55f104f82260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:33710/0 (socket says 172.21.15.104:33710) 2024-06-09T22:26:19.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f4521d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f87f60 0x55f104f894f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46138/0 (socket says 172.21.15.104:46138) 2024-06-09T22:26:19.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f452259c700 1 -- 172.21.15.104:0/1690693441 learned_addr learned my addr 172.21.15.104:0/1690693441 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f4522d9d700 1 --2- 172.21.15.104:0/1690693441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f88880 0x55f104f7ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.610+0000 7f4521d9b700 1 -- 172.21.15.104:0/1690693441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f88880 msgr2=0x55f104f7ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.611+0000 7f4521d9b700 1 --2- 172.21.15.104:0/1690693441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f88880 0x55f104f7ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.611+0000 7f4521d9b700 1 -- 172.21.15.104:0/1690693441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f827a0 msgr2=0x55f104f82260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.611+0000 7f4521d9b700 1 --2- 172.21.15.104:0/1690693441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f827a0 0x55f104f82260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.611+0000 7f4521d9b700 1 -- 172.21.15.104:0/1690693441 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f104f98980 con 0x55f104f87f60 2024-06-09T22:26:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.611+0000 7f452259c700 1 --2- 172.21.15.104:0/1690693441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f827a0 0x55f104f82260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.611+0000 7f4522d9d700 1 --2- 172.21.15.104:0/1690693441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f88880 0x55f104f7ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:26:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.611+0000 7f4521d9b700 1 --2- 172.21.15.104:0/1690693441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f87f60 0x55f104f894f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x55f104f9e760 tx=0x7f4518014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94549268a7602f66 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:19.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.611+0000 7f450bfff700 1 -- 172.21.15.104:0/1690693441 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518003940 con 0x55f104f87f60 2024-06-09T22:26:19.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.611+0000 7f450bfff700 1 -- 172.21.15.104:0/1690693441 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f451801d410 con 0x55f104f87f60 2024-06-09T22:26:19.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.611+0000 7f450bfff700 1 -- 172.21.15.104:0/1690693441 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518029650 con 0x55f104f87f60 2024-06-09T22:26:19.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.612+0000 7f4533389b80 1 -- 172.21.15.104:0/1690693441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f87f60 msgr2=0x55f104f894f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:19.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.612+0000 7f4533389b80 1 --2- 172.21.15.104:0/1690693441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f87f60 0x55f104f894f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x55f104f9e760 tx=0x7f4518014000 comp rx=0 tx=0).stop 2024-06-09T22:26:19.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.612+0000 7f4533389b80 1 -- 172.21.15.104:0/1690693441 shutdown_connections 2024-06-09T22:26:19.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.612+0000 7f4533389b80 1 --2- 172.21.15.104:0/1690693441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f88880 0x55f104f7ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.612+0000 7f4533389b80 1 --2- 172.21.15.104:0/1690693441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f827a0 0x55f104f82260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.612+0000 7f4533389b80 1 --2- 172.21.15.104:0/1690693441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f87f60 0x55f104f894f0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.612+0000 7f4533389b80 1 -- 172.21.15.104:0/1690693441 >> 172.21.15.104:0/1690693441 conn(0x55f104eaf080 msgr2=0x55f104f970e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:19.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.612+0000 7f4533389b80 1 -- 172.21.15.104:0/1690693441 shutdown_connections 2024-06-09T22:26:19.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.612+0000 7f4533389b80 1 -- 172.21.15.104:0/1690693441 wait complete. 2024-06-09T22:26:19.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4533389b80 1 -- start start 2024-06-09T22:26:19.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4533389b80 1 Processor -- start 2024-06-09T22:26:19.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4533389b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f827a0 0x55f104f94290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4533389b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f87f60 0x55f104f947d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4533389b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f88880 0x55f104f917f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4533389b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f45241b1a70 con 0x55f104f827a0 2024-06-09T22:26:19.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4533389b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45241b13a0 con 0x55f104f88880 2024-06-09T22:26:19.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4533389b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f45241ae580 con 0x55f104f87f60 2024-06-09T22:26:19.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4533389b80 10 client.0 _ll_register_callbacks cb 0x55f104e695b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:19.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4521d9b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f87f60 0x55f104f947d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4521d9b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f87f60 0x55f104f947d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:51256/0 (socket says 172.21.15.104:51256) 2024-06-09T22:26:19.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f452259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f827a0 0x55f104f94290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4521d9b700 1 -- 172.21.15.104:0/2046591511 learned_addr learned my addr 172.21.15.104:0/2046591511 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.613+0000 7f4522d9d700 1 --2- 172.21.15.104:0/2046591511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f88880 0x55f104f917f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4521d9b700 1 -- 172.21.15.104:0/2046591511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f88880 msgr2=0x55f104f917f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4521d9b700 1 --2- 172.21.15.104:0/2046591511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f88880 0x55f104f917f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4521d9b700 1 -- 172.21.15.104:0/2046591511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f827a0 msgr2=0x55f104f94290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4521d9b700 1 --2- 172.21.15.104:0/2046591511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f827a0 0x55f104f94290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4521d9b700 1 -- 172.21.15.104:0/2046591511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f104f92030 con 0x55f104f87f60 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4521d9b700 1 --2- 172.21.15.104:0/2046591511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f87f60 0x55f104f947d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x55f104f9e760 tx=0x7f451801d5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45c4cde460407e11 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4509ffb700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4509ffb700 1 -- 172.21.15.104:0/2046591511 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518040060 con 0x55f104f87f60 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4509ffb700 1 -- 172.21.15.104:0/2046591511 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f451801daa0 con 0x55f104f87f60 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4533389b80 1 -- 172.21.15.104:0/2046591511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f451801d410 con 0x55f104f87f60 2024-06-09T22:26:19.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4509ffb700 1 -- 172.21.15.104:0/2046591511 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4518047050 con 0x55f104f87f60 2024-06-09T22:26:19.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4533389b80 10 client.26344 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:19.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4533389b80 1 -- 172.21.15.104:0/2046591511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f105006270 con 0x55f104f87f60 2024-06-09T22:26:19.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.614+0000 7f4509ffb700 1 -- 172.21.15.104:0/2046591511 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f451801d410 con 0x55f104f87f60 2024-06-09T22:26:19.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.615+0000 7f4509ffb700 1 -- 172.21.15.104:0/2046591511 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f451803a030 con 0x55f104f87f60 2024-06-09T22:26:19.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.615+0000 7f4533389b80 10 client.26344 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:19.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.615+0000 7f4533389b80 10 client.26344 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:26:19.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.615+0000 7f4533389b80 20 client.26344 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:19.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.615+0000 7f4533389b80 1 --2- 172.21.15.104:0/2046591511 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55f105006950 0x55f105008d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:19.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.615+0000 7f4533389b80 4 client.26344 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:26:19.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.615+0000 7f4533389b80 1 -- 172.21.15.104:0/2046591511 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f104edb2d0 con 0x55f105006950 2024-06-09T22:26:19.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.615+0000 7f452259c700 1 --2- 172.21.15.104:0/2046591511 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55f105006950 0x55f105008d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:19.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.615+0000 7f452259c700 1 --2- 172.21.15.104:0/2046591511 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55f105006950 0x55f105008d70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c06e2b13ffe705a8 server_cookie=6b8df4ddbb57d718 in_seq=0 out_seq=0 2024-06-09T22:26:19.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.615+0000 7f4509ffb700 10 client.26344 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:26:19.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4509ffb700 1 -- 172.21.15.104:0/2046591511 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f104edb2d0 con 0x55f105006950 2024-06-09T22:26:19.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4509ffb700 10 client.26344 handle_command_reply: tid=0 2024-06-09T22:26:19.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4533389b80 1 client.26344 shutdown 2024-06-09T22:26:19.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4533389b80 1 -- 172.21.15.104:0/2046591511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f87f60 msgr2=0x55f104f947d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:19.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4533389b80 1 --2- 172.21.15.104:0/2046591511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f87f60 0x55f104f947d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x55f104f9e760 tx=0x7f451801d5c0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4533389b80 1 -- 172.21.15.104:0/2046591511 shutdown_connections 2024-06-09T22:26:19.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4533389b80 1 --2- 172.21.15.104:0/2046591511 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55f105006950 0x55f105008d70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4509ffb700 0 client.26344 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:26:19.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4533389b80 1 --2- 172.21.15.104:0/2046591511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f104f827a0 0x55f104f94290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4533389b80 1 --2- 172.21.15.104:0/2046591511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f104f88880 0x55f104f917f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4533389b80 1 --2- 172.21.15.104:0/2046591511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f104f87f60 0x55f104f947d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4533389b80 1 -- 172.21.15.104:0/2046591511 >> 172.21.15.104:0/2046591511 conn(0x55f104eaf080 msgr2=0x55f104f81250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:19.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4533389b80 1 -- 172.21.15.104:0/2046591511 shutdown_connections 2024-06-09T22:26:19.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.616+0000 7f4533389b80 1 -- 172.21.15.104:0/2046591511 wait complete. 2024-06-09T22:26:19.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4533389b80 20 client.26344 trim_cache size 0 max 16384 2024-06-09T22:26:19.623 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:26:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4528b06700 1 -- 172.21.15.104:0/456141971 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f450c0a2e10 msgr2=0x7f450c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4528b06700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f450c0a2e10 0x7f450c0a52d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f452410eff0 tx=0x7f4518009260 comp rx=0 tx=0).stop 2024-06-09T22:26:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4528b06700 1 -- 172.21.15.104:0/456141971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4524100a80 msgr2=0x7f452419add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4528b06700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4524100a80 0x7f452419add0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4514000c00 tx=0x7f451400bab0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4528b06700 1 -- 172.21.15.104:0/456141971 shutdown_connections 2024-06-09T22:26:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4528b06700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f450c0a2e10 0x7f450c0a52d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4528b06700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4524108510 0x7f452419f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4528b06700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4524100a80 0x7f452419add0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4528b06700 1 --2- 172.21.15.104:0/456141971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4524101440 0x7f452419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:19.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4528b06700 1 -- 172.21.15.104:0/456141971 >> 172.21.15.104:0/456141971 conn(0x7f4524077a00 msgr2=0x7f45240fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:19.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4528b06700 1 -- 172.21.15.104:0/456141971 shutdown_connections 2024-06-09T22:26:19.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:19.617+0000 7f4528b06700 1 -- 172.21.15.104:0/456141971 wait complete. 2024-06-09T22:26:19.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 155717 5.32 MB/sec execute 85 sec latency 16.349 ms 2024-06-09T22:26:20.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-09T22:26:20.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 157040 5.34 MB/sec execute 86 sec latency 23.177 ms 2024-06-09T22:26:20.846 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:21.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:21 smithi104 ceph-mon[32389]: pgmap v164: 129 pgs: 129 active+clean; 113 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 257 op/s 2024-06-09T22:26:21.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:21 smithi194 ceph-mon[36856]: pgmap v164: 129 pgs: 129 active+clean; 113 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 257 op/s 2024-06-09T22:26:21.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:21 smithi153 ceph-mon[36599]: pgmap v164: 129 pgs: 129 active+clean; 113 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 257 op/s 2024-06-09T22:26:21.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 159109 5.42 MB/sec execute 87 sec latency 21.085 ms 2024-06-09T22:26:22.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.623+0000 7fd853a74700 1 -- 172.21.15.104:0/3834068779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd84c101d50 msgr2=0x7fd84c10e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:22.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.623+0000 7fd853a74700 1 --2- 172.21.15.104:0/3834068779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd84c101d50 0x7fd84c10e240 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fd84400b0a0 tx=0x7fd84400b3b0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.623+0000 7fd853a74700 1 -- 172.21.15.104:0/3834068779 shutdown_connections 2024-06-09T22:26:22.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.623+0000 7fd853a74700 1 --2- 172.21.15.104:0/3834068779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd84c101d50 0x7fd84c10e240 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.623+0000 7fd853a74700 1 --2- 172.21.15.104:0/3834068779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c107400 0x7fd84c1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.623+0000 7fd853a74700 1 --2- 172.21.15.104:0/3834068779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd84c101390 0x7fd84c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.623+0000 7fd853a74700 1 -- 172.21.15.104:0/3834068779 >> 172.21.15.104:0/3834068779 conn(0x7fd84c078340 msgr2=0x7fd84c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:22.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.623+0000 7fd853a74700 1 -- 172.21.15.104:0/3834068779 shutdown_connections 2024-06-09T22:26:22.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.623+0000 7fd853a74700 1 -- 172.21.15.104:0/3834068779 wait complete. 2024-06-09T22:26:22.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.624+0000 7fd853a74700 1 Processor -- start 2024-06-09T22:26:22.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.624+0000 7fd853a74700 1 -- start start 2024-06-09T22:26:22.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.624+0000 7fd853a74700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd84c101390 0x7fd84c19e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.624+0000 7fd853a74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c101d50 0x7fd84c19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.624+0000 7fd853a74700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd84c107400 0x7fd84c1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.624+0000 7fd853a74700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c111690 con 0x7fd84c101390 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.624+0000 7fd853a74700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c111510 con 0x7fd84c107400 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.624+0000 7fd853a74700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c111810 con 0x7fd84c101d50 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.624+0000 7fd851810700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd84c101390 0x7fd84c19e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.624+0000 7fd851810700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd84c101390 0x7fd84c19e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46164/0 (socket says 172.21.15.104:46164) 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.624+0000 7fd85100f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c101d50 0x7fd84c19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.624+0000 7fd851810700 1 -- 172.21.15.104:0/1589423125 learned_addr learned my addr 172.21.15.104:0/1589423125 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd852011700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd84c107400 0x7fd84c1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd85100f700 1 -- 172.21.15.104:0/1589423125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd84c107400 msgr2=0x7fd84c1989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd85100f700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd84c107400 0x7fd84c1989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd85100f700 1 -- 172.21.15.104:0/1589423125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd84c101390 msgr2=0x7fd84c19e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:22.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd85100f700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd84c101390 0x7fd84c19e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd85100f700 1 -- 172.21.15.104:0/1589423125 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd84c199140 con 0x7fd84c101d50 2024-06-09T22:26:22.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd851810700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd84c101390 0x7fd84c19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:22.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd85100f700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c101d50 0x7fd84c19ed10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00c970 tx=0x7fd83c00cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:22.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd842ffd700 1 -- 172.21.15.104:0/1589423125 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c014070 con 0x7fd84c101d50 2024-06-09T22:26:22.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd853a74700 1 -- 172.21.15.104:0/1589423125 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd84c199430 con 0x7fd84c101d50 2024-06-09T22:26:22.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd842ffd700 1 -- 172.21.15.104:0/1589423125 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd83c004440 con 0x7fd84c101d50 2024-06-09T22:26:22.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd842ffd700 1 -- 172.21.15.104:0/1589423125 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c007b60 con 0x7fd84c101d50 2024-06-09T22:26:22.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.625+0000 7fd853a74700 1 -- 172.21.15.104:0/1589423125 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd84c1a56a0 con 0x7fd84c101d50 2024-06-09T22:26:22.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.628+0000 7fd842ffd700 1 -- 172.21.15.104:0/1589423125 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fd83c017070 con 0x7fd84c101d50 2024-06-09T22:26:22.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.628+0000 7fd842ffd700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd8380a2d90 0x7fd8380a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.628+0000 7fd851810700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd8380a2d90 0x7fd8380a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.628+0000 7fd842ffd700 1 -- 172.21.15.104:0/1589423125 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fd83c0cf3f0 con 0x7fd84c101d50 2024-06-09T22:26:22.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.629+0000 7fd85e2f7b80 1 Processor -- start 2024-06-09T22:26:22.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.629+0000 7fd851810700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd8380a2d90 0x7fd8380a5250 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd84c107c80 tx=0x7fd848009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:22.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.629+0000 7fd85e2f7b80 1 -- start start 2024-06-09T22:26:22.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.629+0000 7fd85e2f7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa74ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.629+0000 7fd85e2f7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa7459e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.629+0000 7fd85e2f7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa745f20 0x5557fa746340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.629+0000 7fd85e2f7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5557fa697810 con 0x5557fa74bf60 2024-06-09T22:26:22.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.629+0000 7fd85e2f7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5557fa63f720 con 0x5557fa745f20 2024-06-09T22:26:22.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.629+0000 7fd85e2f7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5557fa6937d0 con 0x5557fa74b640 2024-06-09T22:26:22.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.629+0000 7fd85100f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa7459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd851810700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa74ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd85100f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa7459e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46178/0 (socket says 172.21.15.104:46178) 2024-06-09T22:26:22.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd85100f700 1 -- 172.21.15.104:0/2282738779 learned_addr learned my addr 172.21.15.104:0/2282738779 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:22.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd852011700 1 --2- 172.21.15.104:0/2282738779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa745f20 0x5557fa746340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd85100f700 1 -- 172.21.15.104:0/2282738779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 msgr2=0x5557fa74ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:22.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd85100f700 1 --2- 172.21.15.104:0/2282738779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa74ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd85100f700 1 -- 172.21.15.104:0/2282738779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa745f20 msgr2=0x5557fa746340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:22.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd85100f700 1 --2- 172.21.15.104:0/2282738779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa745f20 0x5557fa746340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd85100f700 1 -- 172.21.15.104:0/2282738779 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557fa635f50 con 0x5557fa74bf60 2024-06-09T22:26:22.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd85100f700 1 --2- 172.21.15.104:0/2282738779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa7459e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd83c097f70 tx=0x7fd83c09af50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3793468c6dd5494 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:22.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd837fff700 1 -- 172.21.15.104:0/2282738779 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c09bca0 con 0x5557fa74bf60 2024-06-09T22:26:22.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd837fff700 1 -- 172.21.15.104:0/2282738779 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd83c09b5a0 con 0x5557fa74bf60 2024-06-09T22:26:22.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.630+0000 7fd837fff700 1 -- 172.21.15.104:0/2282738779 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c00e3b0 con 0x5557fa74bf60 2024-06-09T22:26:22.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.631+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2282738779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 msgr2=0x5557fa7459e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:22.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.631+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2282738779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa7459e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd83c097f70 tx=0x7fd83c09af50 comp rx=0 tx=0).stop 2024-06-09T22:26:22.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.631+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2282738779 shutdown_connections 2024-06-09T22:26:22.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.631+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2282738779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa745f20 0x5557fa746340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.631+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2282738779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa74ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.631+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2282738779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa7459e0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.631+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2282738779 >> 172.21.15.104:0/2282738779 conn(0x5557fa670cc0 msgr2=0x5557fa740850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:22.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.631+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2282738779 shutdown_connections 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.631+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2282738779 wait complete. 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.631+0000 7fd85e2f7b80 1 -- start start 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.632+0000 7fd85e2f7b80 1 Processor -- start 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.632+0000 7fd85e2f7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa745f20 0x5557fa751d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.632+0000 7fd85e2f7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa74b640 0x5557fa74f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.632+0000 7fd85e2f7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa74f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.632+0000 7fd85e2f7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5557fa7529d0 con 0x5557fa74bf60 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.632+0000 7fd852011700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa74f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.632+0000 7fd85e2f7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5557fa637bf0 con 0x5557fa74b640 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.632+0000 7fd85e2f7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5557fa752b50 con 0x5557fa745f20 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.632+0000 7fd85100f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa74b640 0x5557fa74f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.632+0000 7fd852011700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa74f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46186/0 (socket says 172.21.15.104:46186) 2024-06-09T22:26:22.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.632+0000 7fd852011700 1 -- 172.21.15.104:0/2227667526 learned_addr learned my addr 172.21.15.104:0/2227667526 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:22.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd85e2f7b80 10 client.0 _ll_register_callbacks cb 0x5557fa631920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:22.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd851810700 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa745f20 0x5557fa751d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd85100f700 1 -- 172.21.15.104:0/2227667526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa745f20 msgr2=0x5557fa751d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:22.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd85100f700 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa745f20 0x5557fa751d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd85100f700 1 -- 172.21.15.104:0/2227667526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 msgr2=0x5557fa74f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:22.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd85100f700 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa74f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd85100f700 1 -- 172.21.15.104:0/2227667526 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557fa5e68d0 con 0x5557fa74b640 2024-06-09T22:26:22.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd851810700 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa745f20 0x5557fa751d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:26:22.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd85100f700 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa74b640 0x5557fa74f0e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0a3ce0 tx=0x7fd83c09f9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9ab9e4edda0eb11 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:22.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd8357fa700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:26:22.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd8357fa700 1 -- 172.21.15.104:0/2227667526 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c09bca0 con 0x5557fa74b640 2024-06-09T22:26:22.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2227667526 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd83c09b5a0 con 0x5557fa74b640 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd8357fa700 1 -- 172.21.15.104:0/2227667526 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fd83c0998a0 con 0x5557fa74b640 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.633+0000 7fd8357fa700 1 -- 172.21.15.104:0/2227667526 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd83c00e3b0 con 0x5557fa74b640 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.634+0000 7fd85e2f7b80 10 client.26270 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.634+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2227667526 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5557fa75d3a0 con 0x5557fa74b640 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.634+0000 7fd8357fa700 1 -- 172.21.15.104:0/2227667526 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd83c09b5a0 con 0x5557fa74b640 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.634+0000 7fd8357fa700 1 -- 172.21.15.104:0/2227667526 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd83c09a9f0 con 0x5557fa74b640 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.634+0000 7fd85e2f7b80 10 client.26270 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.634+0000 7fd85e2f7b80 10 client.26270 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.634+0000 7fd85e2f7b80 20 client.26270 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.634+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5557fa75f870 0x5557fa761c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.634+0000 7fd85e2f7b80 4 client.26270 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.634+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2227667526 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557fa6a4d10 con 0x5557fa75f870 2024-06-09T22:26:22.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.634+0000 7fd851810700 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5557fa75f870 0x5557fa761c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.635+0000 7fd851810700 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5557fa75f870 0x5557fa761c60 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4708ef469b078d57 server_cookie=248e80ca7d1982d5 in_seq=0 out_seq=0 2024-06-09T22:26:22.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.635+0000 7fd8357fa700 10 client.26270 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:26:22.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.635+0000 7fd8357fa700 1 -- 172.21.15.104:0/2227667526 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5557fa6a4d10 con 0x5557fa75f870 2024-06-09T22:26:22.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.635+0000 7fd8357fa700 10 client.26270 handle_command_reply: tid=0 2024-06-09T22:26:22.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.635+0000 7fd85e2f7b80 1 client.26270 shutdown 2024-06-09T22:26:22.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.636+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2227667526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa74b640 msgr2=0x5557fa74f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:22.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.636+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa74b640 0x5557fa74f0e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0a3ce0 tx=0x7fd83c09f9d0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.636+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2227667526 shutdown_connections 2024-06-09T22:26:22.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.636+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5557fa75f870 0x5557fa761c60 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.636+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa74f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.636+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa745f20 0x5557fa751d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.636+0000 7fd8357fa700 0 client.26270 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:26:22.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.636+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2227667526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa74b640 0x5557fa74f0e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.636+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2227667526 >> 172.21.15.104:0/2227667526 conn(0x5557fa670cc0 msgr2=0x5557fa7405f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:22.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.636+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2227667526 shutdown_connections 2024-06-09T22:26:22.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.636+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2227667526 wait complete. 2024-06-09T22:26:22.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.636+0000 7fd85e2f7b80 20 client.26270 trim_cache size 0 max 16384 2024-06-09T22:26:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd85e2f7b80 1 Processor -- start 2024-06-09T22:26:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd85e2f7b80 1 -- start start 2024-06-09T22:26:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd85e2f7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa745f20 0x5557fa7459e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd85e2f7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa74cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd85e2f7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa73a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd85e2f7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1ae590 con 0x5557fa74bf60 2024-06-09T22:26:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd85e2f7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1b00d0 con 0x5557fa745f20 2024-06-09T22:26:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd85e2f7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1aec60 con 0x5557fa74b640 2024-06-09T22:26:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd85100f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa74cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd851810700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa745f20 0x5557fa7459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd85100f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa74cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:51304/0 (socket says 172.21.15.104:51304) 2024-06-09T22:26:22.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd85100f700 1 -- 172.21.15.104:0/987470097 learned_addr learned my addr 172.21.15.104:0/987470097 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:22.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.654+0000 7fd852011700 1 --2- 172.21.15.104:0/987470097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa73a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.655+0000 7fd85100f700 1 -- 172.21.15.104:0/987470097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa745f20 msgr2=0x5557fa7459e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:22.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.655+0000 7fd85100f700 1 --2- 172.21.15.104:0/987470097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa745f20 0x5557fa7459e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.655+0000 7fd85100f700 1 -- 172.21.15.104:0/987470097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 msgr2=0x5557fa73a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:22.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.655+0000 7fd85100f700 1 --2- 172.21.15.104:0/987470097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa73a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.655+0000 7fd85100f700 1 -- 172.21.15.104:0/987470097 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557fa74bc70 con 0x5557fa74b640 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.655+0000 7fd85100f700 1 --2- 172.21.15.104:0/987470097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa74cc90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd83c099850 tx=0x7fd83c0972f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d645e9fbb6f50320 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.655+0000 7fd8377fe700 1 -- 172.21.15.104:0/987470097 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c09b5a0 con 0x5557fa74b640 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.655+0000 7fd8377fe700 1 -- 172.21.15.104:0/987470097 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd83c0a4210 con 0x5557fa74b640 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.655+0000 7fd8377fe700 1 -- 172.21.15.104:0/987470097 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c097480 con 0x5557fa74b640 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.656+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/987470097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 msgr2=0x5557fa74cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.656+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/987470097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa74cc90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd83c099850 tx=0x7fd83c0972f0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.657+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/987470097 shutdown_connections 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.657+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/987470097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa74bf60 0x5557fa73a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.657+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/987470097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa745f20 0x5557fa7459e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.657+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/987470097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa74cc90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.657+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/987470097 >> 172.21.15.104:0/987470097 conn(0x5557fa670cc0 msgr2=0x5557fa739e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.657+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/987470097 shutdown_connections 2024-06-09T22:26:22.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.657+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/987470097 wait complete. 2024-06-09T22:26:22.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.657+0000 7fd85e2f7b80 1 -- start start 2024-06-09T22:26:22.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.657+0000 7fd85e2f7b80 1 Processor -- start 2024-06-09T22:26:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd85e2f7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa745f20 0x5557fa7624b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd85e2f7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa7629f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd85e2f7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa74bf60 0x5557fa764a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd85e2f7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1afa00 con 0x5557fa745f20 2024-06-09T22:26:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd85e2f7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1af330 con 0x5557fa74bf60 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd85e2f7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1ac510 con 0x5557fa74b640 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd85e2f7b80 10 client.0 _ll_register_callbacks cb 0x5557fa631920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd851810700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa745f20 0x5557fa7624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd851810700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa745f20 0x5557fa7624b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:46200/0 (socket says 172.21.15.104:46200) 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd851810700 1 -- 172.21.15.104:0/2674889509 learned_addr learned my addr 172.21.15.104:0/2674889509 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd85100f700 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa7629f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd852011700 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa74bf60 0x5557fa764a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd851810700 1 -- 172.21.15.104:0/2674889509 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 msgr2=0x5557fa7629f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd851810700 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa7629f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd851810700 1 -- 172.21.15.104:0/2674889509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa74bf60 msgr2=0x5557fa764a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd851810700 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa74bf60 0x5557fa764a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd851810700 1 -- 172.21.15.104:0/2674889509 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557fa7650e0 con 0x5557fa745f20 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd851810700 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa745f20 0x5557fa7624b0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x5557fa752ba0 tx=0x7fd848013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a14ddb8b7a9b8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd8357fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.658+0000 7fd8357fa700 1 -- 172.21.15.104:0/2674889509 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84802c070 con 0x5557fa745f20 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2674889509 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd83c0a4210 con 0x5557fa745f20 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd8357fa700 1 -- 172.21.15.104:0/2674889509 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fd84801f700 con 0x5557fa745f20 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd8357fa700 1 -- 172.21.15.104:0/2674889509 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd84801f900 con 0x5557fa745f20 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd85e2f7b80 10 client.16959 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2674889509 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5557fa7c9d70 con 0x5557fa745f20 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd8357fa700 1 -- 172.21.15.104:0/2674889509 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd83c0a4210 con 0x5557fa745f20 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd8357fa700 1 -- 172.21.15.104:0/2674889509 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd848030ca0 con 0x5557fa745f20 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd85e2f7b80 10 client.16959 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd85e2f7b80 10 client.16959 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd85e2f7b80 20 client.16959 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5557fa7ca490 0x5557fa7cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd85e2f7b80 4 client.16959 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2674889509 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5557fa69f290 con 0x5557fa7ca490 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.659+0000 7fd85100f700 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5557fa7ca490 0x5557fa7cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.660+0000 7fd8357fa700 10 client.16959 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.660+0000 7fd85100f700 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5557fa7ca490 0x5557fa7cc8b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=36d46f7c0e756962 server_cookie=b86bbc2c7071f4b9 in_seq=0 out_seq=0 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.661+0000 7fd8357fa700 1 -- 172.21.15.104:0/2674889509 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5557fa69f290 con 0x5557fa7ca490 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.661+0000 7fd8357fa700 10 client.16959 handle_command_reply: tid=0 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.661+0000 7fd85e2f7b80 1 client.16959 shutdown 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.662+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2674889509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa745f20 msgr2=0x5557fa7624b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.662+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa745f20 0x5557fa7624b0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x5557fa752ba0 tx=0x7fd848013040 comp rx=0 tx=0).stop 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.662+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2674889509 shutdown_connections 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.662+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5557fa7ca490 0x5557fa7cc8b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.662+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5557fa745f20 0x5557fa7624b0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.662+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557fa74bf60 0x5557fa764a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.662+0000 7fd8357fa700 0 client.16959 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:26:22.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.662+0000 7fd85e2f7b80 1 --2- 172.21.15.104:0/2674889509 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5557fa74b640 0x5557fa7629f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.662+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2674889509 >> 172.21.15.104:0/2674889509 conn(0x5557fa670cc0 msgr2=0x5557fa74cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:22.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.662+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2674889509 shutdown_connections 2024-06-09T22:26:22.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.662+0000 7fd85e2f7b80 1 -- 172.21.15.104:0/2674889509 wait complete. 2024-06-09T22:26:22.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.662+0000 7fd85e2f7b80 20 client.16959 trim_cache size 0 max 16384 2024-06-09T22:26:22.670 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:26:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.663+0000 7fd853a74700 1 -- 172.21.15.104:0/1589423125 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd8380a2d90 msgr2=0x7fd8380a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.663+0000 7fd853a74700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd8380a2d90 0x7fd8380a5250 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd84c107c80 tx=0x7fd848009260 comp rx=0 tx=0).stop 2024-06-09T22:26:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.663+0000 7fd853a74700 1 -- 172.21.15.104:0/1589423125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c101d50 msgr2=0x7fd84c19ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.663+0000 7fd853a74700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c101d50 0x7fd84c19ed10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00c970 tx=0x7fd83c00cc80 comp rx=0 tx=0).stop 2024-06-09T22:26:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.663+0000 7fd853a74700 1 -- 172.21.15.104:0/1589423125 shutdown_connections 2024-06-09T22:26:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.663+0000 7fd853a74700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd8380a2d90 0x7fd8380a5250 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.663+0000 7fd853a74700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd84c101390 0x7fd84c19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.663+0000 7fd853a74700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd84c107400 0x7fd84c1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.663+0000 7fd853a74700 1 --2- 172.21.15.104:0/1589423125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c101d50 0x7fd84c19ed10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:22.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.663+0000 7fd853a74700 1 -- 172.21.15.104:0/1589423125 >> 172.21.15.104:0/1589423125 conn(0x7fd84c078340 msgr2=0x7fd84c06a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:22.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.663+0000 7fd853a74700 1 -- 172.21.15.104:0/1589423125 shutdown_connections 2024-06-09T22:26:22.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:22.663+0000 7fd853a74700 1 -- 172.21.15.104:0/1589423125 wait complete. 2024-06-09T22:26:22.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 160431 5.43 MB/sec execute 88 sec latency 60.681 ms 2024-06-09T22:26:23.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:23 smithi104 ceph-mon[32389]: pgmap v165: 129 pgs: 129 active+clean; 113 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 159 op/s 2024-06-09T22:26:23.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:23 smithi194 ceph-mon[36856]: pgmap v165: 129 pgs: 129 active+clean; 113 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 159 op/s 2024-06-09T22:26:23.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:23 smithi153 ceph-mon[36599]: pgmap v165: 129 pgs: 129 active+clean; 113 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 159 op/s 2024-06-09T22:26:23.525 INFO:tasks.quiescer.fs.[cephfs]:Total duration (58.9) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-06-09T22:26:23.526 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.45 2024-06-09T22:26:23.526 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:26:23.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-09T22:26:23.729 INFO:tasks.workunit.client.0.smithi104.stdout: 1 162651 5.51 MB/sec execute 89 sec latency 16.008 ms 2024-06-09T22:26:23.787 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:24.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 164164 5.53 MB/sec execute 90 sec latency 25.986 ms 2024-06-09T22:26:25.020 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:25.233 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:25 smithi104 ceph-mon[32389]: pgmap v166: 129 pgs: 129 active+clean; 120 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 246 op/s 2024-06-09T22:26:25.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:25 smithi194 ceph-mon[36856]: pgmap v166: 129 pgs: 129 active+clean; 120 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 246 op/s 2024-06-09T22:26:25.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:25 smithi153 ceph-mon[36599]: pgmap v166: 129 pgs: 129 active+clean; 120 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 246 op/s 2024-06-09T22:26:25.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 166154 5.60 MB/sec execute 91 sec latency 33.338 ms 2024-06-09T22:26:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.237+0000 7f4ac02a3700 1 -- 172.21.15.104:0/2569035215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab8101390 msgr2=0x7f4ab8101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.237+0000 7f4ac02a3700 1 --2- 172.21.15.104:0/2569035215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab8101390 0x7f4ab8101810 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009990 tx=0x7f4aa8009ca0 comp rx=0 tx=0).stop 2024-06-09T22:26:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.237+0000 7f4ac02a3700 1 -- 172.21.15.104:0/2569035215 shutdown_connections 2024-06-09T22:26:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.237+0000 7f4ac02a3700 1 --2- 172.21.15.104:0/2569035215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab8101d50 0x7f4ab810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.237+0000 7f4ac02a3700 1 --2- 172.21.15.104:0/2569035215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab8107400 0x7f4ab81077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.237+0000 7f4ac02a3700 1 --2- 172.21.15.104:0/2569035215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab8101390 0x7f4ab8101810 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:26.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.237+0000 7f4ac02a3700 1 -- 172.21.15.104:0/2569035215 >> 172.21.15.104:0/2569035215 conn(0x7f4ab8078340 msgr2=0x7f4ab8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:26.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.237+0000 7f4ac02a3700 1 -- 172.21.15.104:0/2569035215 shutdown_connections 2024-06-09T22:26:26.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.237+0000 7f4ac02a3700 1 -- 172.21.15.104:0/2569035215 wait complete. 2024-06-09T22:26:26.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4ac02a3700 1 Processor -- start 2024-06-09T22:26:26.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4ac02a3700 1 -- start start 2024-06-09T22:26:26.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4ac02a3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab8101390 0x7f4ab8199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:26.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4ac02a3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab8101d50 0x7f4ab8199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:26.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4ac02a3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab8107400 0x7f4ab81a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:26.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4ac02a3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab81116f0 con 0x7f4ab8107400 2024-06-09T22:26:26.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4ac02a3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab8111570 con 0x7f4ab8101d50 2024-06-09T22:26:26.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4ac02a3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab8111870 con 0x7f4ab8101390 2024-06-09T22:26:26.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4abe840700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab8107400 0x7f4ab81a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:26.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4abe840700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab8107400 0x7f4ab81a4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41238/0 (socket says 172.21.15.104:41238) 2024-06-09T22:26:26.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4abe840700 1 -- 172.21.15.104:0/3971235890 learned_addr learned my addr 172.21.15.104:0/3971235890 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:26.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4abe03f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab8101390 0x7f4ab8199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:26.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4abd83e700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab8101d50 0x7f4ab8199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:26.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4abe840700 1 -- 172.21.15.104:0/3971235890 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab8101390 msgr2=0x7f4ab8199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:26.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4abe840700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab8101390 0x7f4ab8199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:26.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4abe840700 1 -- 172.21.15.104:0/3971235890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab8101d50 msgr2=0x7f4ab8199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:26.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4abe840700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab8101d50 0x7f4ab8199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:26.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.238+0000 7f4abe840700 1 -- 172.21.15.104:0/3971235890 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab81a5050 con 0x7f4ab8107400 2024-06-09T22:26:26.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.239+0000 7f4abe840700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab8107400 0x7f4ab81a4b10 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0007ee0 tx=0x7f4ab000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:26.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.239+0000 7f4aaf7fe700 1 -- 172.21.15.104:0/3971235890 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0004bb0 con 0x7f4ab8107400 2024-06-09T22:26:26.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.239+0000 7f4aaf7fe700 1 -- 172.21.15.104:0/3971235890 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ab0004d50 con 0x7f4ab8107400 2024-06-09T22:26:26.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.239+0000 7f4aaf7fe700 1 -- 172.21.15.104:0/3971235890 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0005960 con 0x7f4ab8107400 2024-06-09T22:26:26.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.239+0000 7f4ac02a3700 1 -- 172.21.15.104:0/3971235890 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab81a5280 con 0x7f4ab8107400 2024-06-09T22:26:26.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.239+0000 7f4ac02a3700 1 -- 172.21.15.104:0/3971235890 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab81a5740 con 0x7f4ab8107400 2024-06-09T22:26:26.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.240+0000 7f4ac02a3700 1 -- 172.21.15.104:0/3971235890 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ab8109970 con 0x7f4ab8107400 2024-06-09T22:26:26.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.241+0000 7f4aaf7fe700 1 -- 172.21.15.104:0/3971235890 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f4ab001a020 con 0x7f4ab8107400 2024-06-09T22:26:26.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.241+0000 7f4aaf7fe700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4aa40a2d90 0x7f4aa40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:26.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.242+0000 7f4abe03f700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4aa40a2d90 0x7f4aa40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:26.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.242+0000 7f4aaf7fe700 1 -- 172.21.15.104:0/3971235890 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f4ab00cf5b0 con 0x7f4ab8107400 2024-06-09T22:26:26.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.242+0000 7f4abe03f700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4aa40a2d90 0x7f4aa40a5250 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009a10 tx=0x7f4ab4009190 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:26.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.245+0000 7f4aaf7fe700 1 -- 172.21.15.104:0/3971235890 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4ab0010070 con 0x7f4ab8107400 2024-06-09T22:26:26.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.499+0000 7f4ac02a3700 1 -- 172.21.15.104:0/3971235890 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ab806a780 con 0x7f4ab8107400 2024-06-09T22:26:26.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.500+0000 7f4aaf7fe700 1 -- 172.21.15.104:0/3971235890 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7f4ab009be60 con 0x7f4ab8107400 2024-06-09T22:26:26.502 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:26:26.502 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:26:26.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.503+0000 7f4ac02a3700 1 -- 172.21.15.104:0/3971235890 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4aa40a2d90 msgr2=0x7f4aa40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:26.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.503+0000 7f4ac02a3700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4aa40a2d90 0x7f4aa40a5250 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009a10 tx=0x7f4ab4009190 comp rx=0 tx=0).stop 2024-06-09T22:26:26.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.503+0000 7f4ac02a3700 1 -- 172.21.15.104:0/3971235890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab8107400 msgr2=0x7f4ab81a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:26.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.503+0000 7f4ac02a3700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab8107400 0x7f4ab81a4b10 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0007ee0 tx=0x7f4ab000d3b0 comp rx=0 tx=0).stop 2024-06-09T22:26:26.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.503+0000 7f4ac02a3700 1 -- 172.21.15.104:0/3971235890 shutdown_connections 2024-06-09T22:26:26.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.503+0000 7f4ac02a3700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4aa40a2d90 0x7f4aa40a5250 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:26.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.503+0000 7f4ac02a3700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab8107400 0x7f4ab81a4b10 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:26.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.503+0000 7f4ac02a3700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab8101390 0x7f4ab8199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:26.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.503+0000 7f4ac02a3700 1 --2- 172.21.15.104:0/3971235890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab8101d50 0x7f4ab8199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:26.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.503+0000 7f4ac02a3700 1 -- 172.21.15.104:0/3971235890 >> 172.21.15.104:0/3971235890 conn(0x7f4ab8078340 msgr2=0x7f4ab80ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:26.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.503+0000 7f4ac02a3700 1 -- 172.21.15.104:0/3971235890 shutdown_connections 2024-06-09T22:26:26.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:26.503+0000 7f4ac02a3700 1 -- 172.21.15.104:0/3971235890 wait complete. 2024-06-09T22:26:26.505 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:26:26.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 167472 5.61 MB/sec execute 92 sec latency 55.228 ms 2024-06-09T22:26:27.185 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:27 smithi104 ceph-mon[32389]: pgmap v167: 129 pgs: 129 active+clean; 120 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-06-09T22:26:27.185 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:27 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3971235890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:26:27.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:27 smithi194 ceph-mon[36856]: pgmap v167: 129 pgs: 129 active+clean; 120 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-06-09T22:26:27.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:27 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3971235890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:26:27.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:27 smithi153 ceph-mon[36599]: pgmap v167: 129 pgs: 129 active+clean; 120 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-06-09T22:26:27.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:27 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3971235890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:26:27.608 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24460 quiesce db / --timeout 90 --expiration 149.45 --await-for 45.0' 2024-06-09T22:26:27.608 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24460 quiesce db / --timeout 90 --expiration 149.45 --await-for 45.0 2024-06-09T22:26:27.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 169651 5.68 MB/sec execute 93 sec latency 19.695 ms 2024-06-09T22:26:28.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.115+0000 7ffa47dab700 1 -- 172.21.15.104:0/245359478 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa401013c0 msgr2=0x7ffa40101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.115+0000 7ffa47dab700 1 --2- 172.21.15.104:0/245359478 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa401013c0 0x7ffa40101840 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009990 tx=0x7ffa30009ca0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.115+0000 7ffa47dab700 1 -- 172.21.15.104:0/245359478 shutdown_connections 2024-06-09T22:26:28.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.115+0000 7ffa47dab700 1 --2- 172.21.15.104:0/245359478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa40101d80 0x7ffa4010e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.115+0000 7ffa47dab700 1 --2- 172.21.15.104:0/245359478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa40107430 0x7ffa40107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.115+0000 7ffa47dab700 1 --2- 172.21.15.104:0/245359478 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa401013c0 0x7ffa40101840 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.115+0000 7ffa47dab700 1 -- 172.21.15.104:0/245359478 >> 172.21.15.104:0/245359478 conn(0x7ffa400783d0 msgr2=0x7ffa400787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.116+0000 7ffa47dab700 1 -- 172.21.15.104:0/245359478 shutdown_connections 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.116+0000 7ffa47dab700 1 -- 172.21.15.104:0/245359478 wait complete. 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.116+0000 7ffa47dab700 1 Processor -- start 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.116+0000 7ffa47dab700 1 -- start start 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa47dab700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa401013c0 0x7ffa4019ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa47dab700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa40101d80 0x7ffa4019b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa47dab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa40107430 0x7ffa4019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa47dab700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ffa401116a0 con 0x7ffa40107430 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa47dab700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa40111520 con 0x7ffa401013c0 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa47dab700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffa40111820 con 0x7ffa40101d80 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa46348700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa40107430 0x7ffa4019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa46348700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa40107430 0x7ffa4019f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41260/0 (socket says 172.21.15.104:41260) 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa46348700 1 -- 172.21.15.104:0/1920200937 learned_addr learned my addr 172.21.15.104:0/1920200937 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa45346700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa40101d80 0x7ffa4019b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa45b47700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa401013c0 0x7ffa4019ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa46348700 1 -- 172.21.15.104:0/1920200937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa40101d80 msgr2=0x7ffa4019b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa46348700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa40101d80 0x7ffa4019b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa46348700 1 -- 172.21.15.104:0/1920200937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa401013c0 msgr2=0x7ffa4019ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa46348700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa401013c0 0x7ffa4019ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa46348700 1 -- 172.21.15.104:0/1920200937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4019fe50 con 0x7ffa40107430 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa45b47700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa401013c0 0x7ffa4019ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa45346700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa40101d80 0x7ffa4019b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.117+0000 7ffa46348700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa40107430 0x7ffa4019f710 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ffa38007ee0 tx=0x7ffa3800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.118+0000 7ffa36ffd700 1 -- 172.21.15.104:0/1920200937 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3800dbd0 con 0x7ffa40107430 2024-06-09T22:26:28.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.118+0000 7ffa36ffd700 1 -- 172.21.15.104:0/1920200937 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3800dd70 con 0x7ffa40107430 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.118+0000 7ffa47dab700 1 -- 172.21.15.104:0/1920200937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa401a0140 con 0x7ffa40107430 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.118+0000 7ffa36ffd700 1 -- 172.21.15.104:0/1920200937 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38005a60 con 0x7ffa40107430 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.118+0000 7ffa47dab700 1 -- 172.21.15.104:0/1920200937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa401a79c0 con 0x7ffa40107430 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.120+0000 7ffa36ffd700 1 -- 172.21.15.104:0/1920200937 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ffa3801a020 con 0x7ffa40107430 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.120+0000 7ffa36ffd700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa2c0a2d90 0x7ffa2c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.121+0000 7ffa45b47700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa2c0a2d90 0x7ffa2c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.121+0000 7ffa36ffd700 1 -- 172.21.15.104:0/1920200937 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ffa380ceef0 con 0x7ffa40107430 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.121+0000 7ffa45b47700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa2c0a2d90 0x7ffa2c0a5250 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009c90 tx=0x7ffa3c009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.121+0000 7ffa5262eb80 1 Processor -- start 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.121+0000 7ffa5262eb80 1 -- start start 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.121+0000 7ffa5262eb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 0x55e1271c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa5262eb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa5262eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271c3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa5262eb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e127114810 con 0x55e1271c8f60 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa5262eb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e1270bc720 con 0x55e1271c8640 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa5262eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e1271107d0 con 0x55e1271c2f20 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa45b47700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 0x55e1271c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa45346700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa45b47700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 0x55e1271c8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:43240/0 (socket says 172.21.15.104:43240) 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa45b47700 1 -- 172.21.15.104:0/1419308358 learned_addr learned my addr 172.21.15.104:0/1419308358 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa46348700 1 --2- 172.21.15.104:0/1419308358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa45b47700 1 -- 172.21.15.104:0/1419308358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 msgr2=0x55e1271c3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa45b47700 1 --2- 172.21.15.104:0/1419308358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271c3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa45b47700 1 -- 172.21.15.104:0/1419308358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 msgr2=0x55e1271c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa45b47700 1 --2- 172.21.15.104:0/1419308358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271c29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa45b47700 1 -- 172.21.15.104:0/1419308358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1270b2f50 con 0x55e1271c8640 2024-06-09T22:26:28.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa45b47700 1 --2- 172.21.15.104:0/1419308358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 0x55e1271c8a20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c010f70 tx=0x7ffa3c013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eccf2b0c76ed9c28 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:28.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa2bfff700 1 -- 172.21.15.104:0/1419308358 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c011d30 con 0x55e1271c8640 2024-06-09T22:26:28.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa2bfff700 1 -- 172.21.15.104:0/1419308358 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3c01e900 con 0x55e1271c8640 2024-06-09T22:26:28.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.122+0000 7ffa2bfff700 1 -- 172.21.15.104:0/1419308358 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c01c820 con 0x55e1271c8640 2024-06-09T22:26:28.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.123+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1419308358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 msgr2=0x55e1271c8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.123+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/1419308358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 0x55e1271c8a20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c010f70 tx=0x7ffa3c013000 comp rx=0 tx=0).stop 2024-06-09T22:26:28.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.123+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1419308358 shutdown_connections 2024-06-09T22:26:28.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.123+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/1419308358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271c3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.123+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/1419308358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 0x55e1271c8a20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.123+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/1419308358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.123+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1419308358 >> 172.21.15.104:0/1419308358 conn(0x55e1270edcc0 msgr2=0x55e1271bd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.123+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1419308358 shutdown_connections 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.123+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1419308358 wait complete. 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa5262eb80 1 -- start start 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa5262eb80 1 Processor -- start 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa5262eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa5262eb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8640 0x55e1271cc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa5262eb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8f60 0x55e1271cc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa5262eb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e1271cf9d0 con 0x55e1271c8640 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa5262eb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e1270b4bf0 con 0x55e1271c8f60 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa5262eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e1271cfb50 con 0x55e1271c2f20 2024-06-09T22:26:28.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa5262eb80 10 client.0 _ll_register_callbacks cb 0x55e1270ae920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa45346700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8640 0x55e1271cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa45346700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8640 0x55e1271cc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41286/0 (socket says 172.21.15.104:41286) 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa45b47700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa46348700 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8f60 0x55e1271cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.124+0000 7ffa45346700 1 -- 172.21.15.104:0/1207356228 learned_addr learned my addr 172.21.15.104:0/1207356228 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa45346700 1 -- 172.21.15.104:0/1207356228 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 msgr2=0x55e1271ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa45346700 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa45346700 1 -- 172.21.15.104:0/1207356228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8f60 msgr2=0x55e1271cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa45346700 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8f60 0x55e1271cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa45346700 1 -- 172.21.15.104:0/1207356228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1270638d0 con 0x55e1271c8640 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa46348700 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8f60 0x55e1271cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa45b47700 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:28.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa45346700 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8640 0x55e1271cc010 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009990 tx=0x7ffa300096f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a95e331e297a5f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa297fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa297fa700 1 -- 172.21.15.104:0/1207356228 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa30003880 con 0x55e1271c8640 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1207356228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa3c01e900 con 0x55e1271c8640 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa297fa700 1 -- 172.21.15.104:0/1207356228 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ffa30003a20 con 0x55e1271c8640 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa297fa700 1 -- 172.21.15.104:0/1207356228 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa30032050 con 0x55e1271c8640 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa5262eb80 10 client.16977 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.125+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1207356228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e1271da310 con 0x55e1271c8640 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.126+0000 7ffa297fa700 1 -- 172.21.15.104:0/1207356228 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3c01e900 con 0x55e1271c8640 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.126+0000 7ffa297fa700 1 -- 172.21.15.104:0/1207356228 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ffa30024450 con 0x55e1271c8640 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.126+0000 7ffa5262eb80 10 client.16977 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.126+0000 7ffa5262eb80 10 client.16977 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.126+0000 7ffa5262eb80 20 client.16977 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.126+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55e1271dc8a0 0x55e1271dec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.126+0000 7ffa5262eb80 4 client.16977 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.126+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1207356228 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e127121d10 con 0x55e1271dc8a0 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.126+0000 7ffa45b47700 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55e1271dc8a0 0x55e1271dec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.126+0000 7ffa45b47700 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55e1271dc8a0 0x55e1271dec90 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9accf9f02328bb5a server_cookie=468e435b31985846 in_seq=0 out_seq=0 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.126+0000 7ffa297fa700 10 client.16977 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.127+0000 7ffa297fa700 1 -- 172.21.15.104:0/1207356228 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55e127121d10 con 0x55e1271dc8a0 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.127+0000 7ffa297fa700 10 client.16977 handle_command_reply: tid=0 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.127+0000 7ffa5262eb80 1 client.16977 shutdown 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.127+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1207356228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8640 msgr2=0x55e1271cc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.127+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8640 0x55e1271cc010 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009990 tx=0x7ffa300096f0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.127+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1207356228 shutdown_connections 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.127+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55e1271dc8a0 0x55e1271dec90 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.127+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8f60 0x55e1271cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.127+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.127+0000 7ffa297fa700 0 client.16977 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:26:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.127+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/1207356228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8640 0x55e1271cc010 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.127+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1207356228 >> 172.21.15.104:0/1207356228 conn(0x55e1270edcc0 msgr2=0x55e1271c19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.128+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1207356228 shutdown_connections 2024-06-09T22:26:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.128+0000 7ffa5262eb80 1 -- 172.21.15.104:0/1207356228 wait complete. 2024-06-09T22:26:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.128+0000 7ffa5262eb80 20 client.16977 trim_cache size 0 max 16384 2024-06-09T22:26:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa5262eb80 1 Processor -- start 2024-06-09T22:26:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa5262eb80 1 -- start start 2024-06-09T22:26:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa5262eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa5262eb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 0x55e1271e45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa5262eb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271e4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa5262eb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ffa401b06b0 con 0x55e1271c8f60 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa5262eb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa401b21f0 con 0x55e1271c8640 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa5262eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffa401b0d80 con 0x55e1271c2f20 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa46348700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271e4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa45346700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 0x55e1271e45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa46348700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271e4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41290/0 (socket says 172.21.15.104:41290) 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa46348700 1 -- 172.21.15.104:0/2471559025 learned_addr learned my addr 172.21.15.104:0/2471559025 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa45b47700 1 --2- 172.21.15.104:0/2471559025 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa45346700 1 -- 172.21.15.104:0/2471559025 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 msgr2=0x55e1271c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa45346700 1 --2- 172.21.15.104:0/2471559025 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271c29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa45346700 1 -- 172.21.15.104:0/2471559025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 msgr2=0x55e1271e4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa45346700 1 --2- 172.21.15.104:0/2471559025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271e4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.144+0000 7ffa45346700 1 -- 172.21.15.104:0/2471559025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1271ccef0 con 0x55e1271c8640 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.145+0000 7ffa45346700 1 --2- 172.21.15.104:0/2471559025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 0x55e1271e45a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000b820 tx=0x7ffa30009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9941d54cfd307801 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.145+0000 7ffa2b7fe700 1 -- 172.21.15.104:0/2471559025 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3003a020 con 0x55e1271c8640 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.145+0000 7ffa2b7fe700 1 -- 172.21.15.104:0/2471559025 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa30024ce0 con 0x55e1271c8640 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.145+0000 7ffa2b7fe700 1 -- 172.21.15.104:0/2471559025 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3000f6c0 con 0x55e1271c8640 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.145+0000 7ffa5262eb80 1 -- 172.21.15.104:0/2471559025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 msgr2=0x55e1271e45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.145+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/2471559025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 0x55e1271e45a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000b820 tx=0x7ffa30009290 comp rx=0 tx=0).stop 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 -- 172.21.15.104:0/2471559025 shutdown_connections 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/2471559025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271e4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/2471559025 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c2f20 0x55e1271c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/2471559025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c8640 0x55e1271e45a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 -- 172.21.15.104:0/2471559025 >> 172.21.15.104:0/2471559025 conn(0x55e1270edcc0 msgr2=0x55e1271cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 -- 172.21.15.104:0/2471559025 shutdown_connections 2024-06-09T22:26:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 -- 172.21.15.104:0/2471559025 wait complete. 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 -- start start 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 Processor -- start 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c2f20 0x55e1271d5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c8640 0x55e1271d5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271e3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ffa401b1b20 con 0x55e1271c8f60 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa401b1450 con 0x55e1271c2f20 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffa401ae630 con 0x55e1271c8640 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.146+0000 7ffa5262eb80 10 client.0 _ll_register_callbacks cb 0x55e1270ae920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa46348700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271e3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa46348700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271e3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41296/0 (socket says 172.21.15.104:41296) 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa45b47700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c2f20 0x55e1271d5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa46348700 1 -- 172.21.15.104:0/3979531930 learned_addr learned my addr 172.21.15.104:0/3979531930 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa45346700 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c8640 0x55e1271d5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa46348700 1 -- 172.21.15.104:0/3979531930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c8640 msgr2=0x55e1271d5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa46348700 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c8640 0x55e1271d5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa46348700 1 -- 172.21.15.104:0/3979531930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c2f20 msgr2=0x55e1271d5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa46348700 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c2f20 0x55e1271d5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa46348700 1 -- 172.21.15.104:0/3979531930 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1271e4070 con 0x55e1271c8f60 2024-06-09T22:26:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa45b47700 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c2f20 0x55e1271d5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa46348700 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271e3990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800dfd0 tx=0x7ffa3809b170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a5735a8dddcea server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa297fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa297fa700 1 -- 172.21.15.104:0/3979531930 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3809f0f0 con 0x55e1271c8f60 2024-06-09T22:26:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa297fa700 1 -- 172.21.15.104:0/3979531930 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ffa38022250 con 0x55e1271c8f60 2024-06-09T22:26:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa5262eb80 1 -- 172.21.15.104:0/3979531930 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa30024ce0 con 0x55e1271c8f60 2024-06-09T22:26:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa297fa700 1 -- 172.21.15.104:0/3979531930 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa38022450 con 0x55e1271c8f60 2024-06-09T22:26:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa5262eb80 10 client.16983 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.147+0000 7ffa5262eb80 1 -- 172.21.15.104:0/3979531930 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e127246f50 con 0x55e1271c8f60 2024-06-09T22:26:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.148+0000 7ffa297fa700 1 -- 172.21.15.104:0/3979531930 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa30024ce0 con 0x55e1271c8f60 2024-06-09T22:26:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.148+0000 7ffa297fa700 1 -- 172.21.15.104:0/3979531930 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ffa380b6060 con 0x55e1271c8f60 2024-06-09T22:26:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.148+0000 7ffa5262eb80 10 client.16983 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.148+0000 7ffa5262eb80 10 client.16983 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:26:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.148+0000 7ffa5262eb80 20 client.16983 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.148+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55e127247630 0x55e127249a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.148+0000 7ffa5262eb80 4 client.16983 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:26:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.148+0000 7ffa5262eb80 1 -- 172.21.15.104:0/3979531930 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e12711c290 con 0x55e127247630 2024-06-09T22:26:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.148+0000 7ffa45b47700 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55e127247630 0x55e127249a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.148+0000 7ffa45b47700 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55e127247630 0x55e127249a70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=94d09171bebb0d7 server_cookie=60a4870320a73b3e in_seq=0 out_seq=0 2024-06-09T22:26:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.148+0000 7ffa297fa700 10 client.16983 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:26:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa297fa700 1 -- 172.21.15.104:0/3979531930 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e12711c290 con 0x55e127247630 2024-06-09T22:26:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa297fa700 10 client.16983 handle_command_reply: tid=0 2024-06-09T22:26:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa5262eb80 1 client.16983 shutdown 2024-06-09T22:26:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa5262eb80 1 -- 172.21.15.104:0/3979531930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 msgr2=0x55e1271e3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271e3990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800dfd0 tx=0x7ffa3809b170 comp rx=0 tx=0).stop 2024-06-09T22:26:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa5262eb80 1 -- 172.21.15.104:0/3979531930 shutdown_connections 2024-06-09T22:26:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55e127247630 0x55e127249a70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1271c8f60 0x55e1271e3990 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1271c2f20 0x55e1271d5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa297fa700 0 client.16983 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:26:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa5262eb80 1 --2- 172.21.15.104:0/3979531930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1271c8640 0x55e1271d5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa5262eb80 1 -- 172.21.15.104:0/3979531930 >> 172.21.15.104:0/3979531930 conn(0x55e1270edcc0 msgr2=0x55e1271b6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa5262eb80 1 -- 172.21.15.104:0/3979531930 shutdown_connections 2024-06-09T22:26:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa5262eb80 1 -- 172.21.15.104:0/3979531930 wait complete. 2024-06-09T22:26:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.149+0000 7ffa5262eb80 20 client.16983 trim_cache size 0 max 16384 2024-06-09T22:26:28.209 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:26:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.150+0000 7ffa47dab700 1 -- 172.21.15.104:0/1920200937 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa2c0a2d90 msgr2=0x7ffa2c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.150+0000 7ffa47dab700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa2c0a2d90 0x7ffa2c0a5250 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009c90 tx=0x7ffa3c009260 comp rx=0 tx=0).stop 2024-06-09T22:26:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.150+0000 7ffa47dab700 1 -- 172.21.15.104:0/1920200937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa40107430 msgr2=0x7ffa4019f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.150+0000 7ffa47dab700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa40107430 0x7ffa4019f710 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ffa38007ee0 tx=0x7ffa3800d3b0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.150+0000 7ffa47dab700 1 -- 172.21.15.104:0/1920200937 shutdown_connections 2024-06-09T22:26:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.150+0000 7ffa47dab700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ffa2c0a2d90 0x7ffa2c0a5250 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.150+0000 7ffa47dab700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffa40107430 0x7ffa4019f710 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.150+0000 7ffa47dab700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa401013c0 0x7ffa4019ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.150+0000 7ffa47dab700 1 --2- 172.21.15.104:0/1920200937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa40101d80 0x7ffa4019b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.150+0000 7ffa47dab700 1 -- 172.21.15.104:0/1920200937 >> 172.21.15.104:0/1920200937 conn(0x7ffa400783d0 msgr2=0x7ffa4006ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.150+0000 7ffa47dab700 1 -- 172.21.15.104:0/1920200937 shutdown_connections 2024-06-09T22:26:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:28.150+0000 7ffa47dab700 1 -- 172.21.15.104:0/1920200937 wait complete. 2024-06-09T22:26:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.665+0000 7f7c50e1b640 1 -- 172.21.15.104:0/1306426297 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c4c102c30 msgr2=0x7f7c4c103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.665+0000 7f7c50e1b640 1 --2- 172.21.15.104:0/1306426297 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c4c102c30 0x7f7c4c103010 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009900 tx=0x7f7c3c02a510 comp rx=0 tx=0).stop 2024-06-09T22:26:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.665+0000 7f7c50e1b640 1 -- 172.21.15.104:0/1306426297 shutdown_connections 2024-06-09T22:26:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.665+0000 7f7c50e1b640 1 --2- 172.21.15.104:0/1306426297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c4c10bf70 0x7f7c4c10e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.665+0000 7f7c50e1b640 1 --2- 172.21.15.104:0/1306426297 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c4c103550 0x7f7c4c10ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.665+0000 7f7c50e1b640 1 --2- 172.21.15.104:0/1306426297 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c4c102c30 0x7f7c4c103010 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.665+0000 7f7c50e1b640 1 -- 172.21.15.104:0/1306426297 >> 172.21.15.104:0/1306426297 conn(0x7f7c4c077a20 msgr2=0x7f7c4c077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.665+0000 7f7c50e1b640 1 -- 172.21.15.104:0/1306426297 shutdown_connections 2024-06-09T22:26:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.665+0000 7f7c50e1b640 1 -- 172.21.15.104:0/1306426297 wait complete. 2024-06-09T22:26:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.666+0000 7f7c50e1b640 1 Processor -- start 2024-06-09T22:26:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.666+0000 7f7c50e1b640 1 -- start start 2024-06-09T22:26:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c50e1b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c4c102c30 0x7f7c4c198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c50e1b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c4c103550 0x7f7c4c198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c50e1b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c4c10bf70 0x7f7c4c19cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c50e1b640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c111dd0 con 0x7f7c4c102c30 2024-06-09T22:26:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c50e1b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c111c50 con 0x7f7c4c103550 2024-06-09T22:26:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c50e1b640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c111f50 con 0x7f7c4c10bf70 2024-06-09T22:26:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c4c102c30 0x7f7c4c198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c4c102c30 0x7f7c4c198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41326/0 (socket says 172.21.15.104:41326) 2024-06-09T22:26:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c4a59c640 1 -- 172.21.15.104:0/3296639166 learned_addr learned my addr 172.21.15.104:0/3296639166 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c4c103550 0x7f7c4c198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c4ad9d640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c4c10bf70 0x7f7c4c19cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c4ad9d640 1 -- 172.21.15.104:0/3296639166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c4c103550 msgr2=0x7f7c4c198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c4ad9d640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c4c103550 0x7f7c4c198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c4ad9d640 1 -- 172.21.15.104:0/3296639166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c4c102c30 msgr2=0x7f7c4c198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c4ad9d640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c4c102c30 0x7f7c4c198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c4ad9d640 1 -- 172.21.15.104:0/3296639166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c19d2b0 con 0x7f7c4c10bf70 2024-06-09T22:26:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c4c103550 0x7f7c4c198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c4a59c640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c4c102c30 0x7f7c4c198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.667+0000 7f7c4ad9d640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c4c10bf70 0x7f7c4c19cb80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400be20 tx=0x7f7c4400bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.668+0000 7f7c3b7fe640 1 -- 172.21.15.104:0/3296639166 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44019070 con 0x7f7c4c10bf70 2024-06-09T22:26:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.668+0000 7f7c3b7fe640 1 -- 172.21.15.104:0/3296639166 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c44014920 con 0x7f7c4c10bf70 2024-06-09T22:26:28.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.668+0000 7f7c50e1b640 1 -- 172.21.15.104:0/3296639166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4c19d5a0 con 0x7f7c4c10bf70 2024-06-09T22:26:28.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.668+0000 7f7c3b7fe640 1 -- 172.21.15.104:0/3296639166 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44012920 con 0x7f7c4c10bf70 2024-06-09T22:26:28.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.668+0000 7f7c50e1b640 1 -- 172.21.15.104:0/3296639166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4c1a4e30 con 0x7f7c4c10bf70 2024-06-09T22:26:28.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.671+0000 7f7c3b7fe640 1 -- 172.21.15.104:0/3296639166 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7c4401c070 con 0x7f7c4c10bf70 2024-06-09T22:26:28.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.671+0000 7f7c53bfd740 1 Processor -- start 2024-06-09T22:26:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.672+0000 7f7c53bfd740 1 -- start start 2024-06-09T22:26:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.672+0000 7f7c53bfd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a4773550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.672+0000 7f7c53bfd740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a477de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.672+0000 7f7c53bfd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a4780740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.672+0000 7f7c53bfd740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b6a46d74f0 con 0x55b6a4773a90 2024-06-09T22:26:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.672+0000 7f7c53bfd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b6a4675870 con 0x55b6a477e350 2024-06-09T22:26:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.672+0000 7f7c53bfd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b6a46d34b0 con 0x55b6a4773170 2024-06-09T22:26:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.672+0000 7f7c3b7fe640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7c340a2e20 0x7f7c340a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.672+0000 7f7c49d9b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a4773550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.672+0000 7f7c49d9b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a4773550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:60086/0 (socket says 172.21.15.104:60086) 2024-06-09T22:26:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.672+0000 7f7c49d9b640 1 -- 172.21.15.104:0/1857506546 learned_addr learned my addr 172.21.15.104:0/1857506546 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.672+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/1857506546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a4780740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c3b7fe640 1 -- 172.21.15.104:0/3296639166 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7c440ceea0 con 0x7f7c4c10bf70 2024-06-09T22:26:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c4a59c640 1 --2- 172.21.15.104:0/1857506546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a477de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c49d9b640 1 -- 172.21.15.104:0/1857506546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 msgr2=0x55b6a4780740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c4ad9d640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7c340a2e20 0x7f7c340a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/1857506546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a4780740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c49d9b640 1 -- 172.21.15.104:0/1857506546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 msgr2=0x55b6a477de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/1857506546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a477de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c49d9b640 1 -- 172.21.15.104:0/1857506546 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6a46919b0 con 0x55b6a4773170 2024-06-09T22:26:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/1857506546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a4773550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009fd0 tx=0x7f7c40009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca8b5d67dd1a224 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c4ad9d640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7c340a2e20 0x7f7c340a52e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400bdf0 tx=0x7f7c4400e040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c38ff9640 1 -- 172.21.15.104:0/1857506546 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4000f910 con 0x55b6a4773170 2024-06-09T22:26:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c38ff9640 1 -- 172.21.15.104:0/1857506546 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4000e3b0 con 0x55b6a4773170 2024-06-09T22:26:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.673+0000 7f7c38ff9640 1 -- 172.21.15.104:0/1857506546 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4000d5a0 con 0x55b6a4773170 2024-06-09T22:26:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.674+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1857506546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 msgr2=0x55b6a4773550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.674+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/1857506546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a4773550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009fd0 tx=0x7f7c40009c30 comp rx=0 tx=0).stop 2024-06-09T22:26:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.674+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1857506546 shutdown_connections 2024-06-09T22:26:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.674+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/1857506546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a4780740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.674+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/1857506546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a477de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.674+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/1857506546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a4773550 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.674+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1857506546 >> 172.21.15.104:0/1857506546 conn(0x55b6a476f680 msgr2=0x55b6a476fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.674+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1857506546 shutdown_connections 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.675+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1857506546 wait complete. 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.675+0000 7f7c53bfd740 1 -- start start 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.675+0000 7f7c53bfd740 1 Processor -- start 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.675+0000 7f7c53bfd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a47774e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.675+0000 7f7c53bfd740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a4777a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.675+0000 7f7c53bfd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a477bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.675+0000 7f7c53bfd740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b6a4780ec0 con 0x55b6a4773a90 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.675+0000 7f7c53bfd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b6a4660ae0 con 0x55b6a477e350 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.675+0000 7f7c53bfd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b6a4781040 con 0x55b6a4773170 2024-06-09T22:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.675+0000 7f7c53bfd740 10 client.0 _ll_register_callbacks cb 0x55b6a46999b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a4777a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c49d9b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a47774e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c49d9b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a47774e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:60094/0 (socket says 172.21.15.104:60094) 2024-06-09T22:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a4777a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41342/0 (socket says 172.21.15.104:41342) 2024-06-09T22:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c49d9b640 1 -- 172.21.15.104:0/1842658576 learned_addr learned my addr 172.21.15.104:0/1842658576 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/1842658576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a477bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c49d9b640 1 -- 172.21.15.104:0/1842658576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 msgr2=0x55b6a477bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/1842658576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a477bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c49d9b640 1 -- 172.21.15.104:0/1842658576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 msgr2=0x55b6a4777a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/1842658576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a4777a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c49d9b640 1 -- 172.21.15.104:0/1842658576 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6a43eb210 con 0x55b6a4773170 2024-06-09T22:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/1842658576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a47774e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000ee70 tx=0x7f7c4001a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc7871531cd251bf server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1842658576 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4000fde0 con 0x55b6a4773170 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1842658576 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7c4002c480 con 0x55b6a4773170 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1842658576 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c4000e3b0 con 0x55b6a4773170 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1842658576 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7c40035050 con 0x55b6a4773170 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c53bfd740 10 client.26395 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.676+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1842658576 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b6a44b7620 con 0x55b6a4773170 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.677+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1842658576 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4000e3b0 con 0x55b6a4773170 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.677+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1842658576 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7c4002b500 con 0x55b6a4773170 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.677+0000 7f7c53bfd740 10 client.26395 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.677+0000 7f7c53bfd740 10 client.26395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.677+0000 7f7c53bfd740 20 client.26395 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.677+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/1842658576 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b6a478e4a0 0x55b6a4790890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.677+0000 7f7c53bfd740 4 client.26395 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.677+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1842658576 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6a46eaf10 con 0x55b6a478e4a0 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.677+0000 7f7c4a59c640 1 --2- 172.21.15.104:0/1842658576 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b6a478e4a0 0x55b6a4790890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.677+0000 7f7c4a59c640 1 --2- 172.21.15.104:0/1842658576 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b6a478e4a0 0x55b6a4790890 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b30d6f79a1fcbb4 server_cookie=889c4f5c79cd9de9 in_seq=0 out_seq=0 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.677+0000 7f7c31ffb640 10 client.26395 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1842658576 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b6a46eaf10 con 0x55b6a478e4a0 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c31ffb640 10 client.26395 handle_command_reply: tid=0 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c53bfd740 1 client.26395 shutdown 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1842658576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 msgr2=0x55b6a47774e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/1842658576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a47774e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000ee70 tx=0x7f7c4001a040 comp rx=0 tx=0).stop 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1842658576 shutdown_connections 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/1842658576 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b6a478e4a0 0x55b6a4790890 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/1842658576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a477bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/1842658576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a4777a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.685 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/1842658576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a47774e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.685 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c31ffb640 0 client.26395 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:28.685 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1842658576 >> 172.21.15.104:0/1842658576 conn(0x55b6a476f680 msgr2=0x55b6a4775400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:28.685 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1842658576 shutdown_connections 2024-06-09T22:26:28.685 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c53bfd740 1 -- 172.21.15.104:0/1842658576 wait complete. 2024-06-09T22:26:28.685 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.679+0000 7f7c53bfd740 20 client.26395 trim_cache size 0 max 16384 2024-06-09T22:26:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c53bfd740 1 Processor -- start 2024-06-09T22:26:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c53bfd740 1 -- start start 2024-06-09T22:26:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c53bfd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a477bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c53bfd740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a4777a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c53bfd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a4795ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c53bfd740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1acc90 con 0x55b6a4773a90 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c53bfd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1ae7d0 con 0x55b6a477e350 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c53bfd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1ad360 con 0x55b6a4773170 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a4777a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c49d9b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a477bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a4777a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41358/0 (socket says 172.21.15.104:41358) 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c4a59c640 1 -- 172.21.15.104:0/3227046184 learned_addr learned my addr 172.21.15.104:0/3227046184 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/3227046184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a4795ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c4a59c640 1 -- 172.21.15.104:0/3227046184 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 msgr2=0x55b6a477bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c4a59c640 1 --2- 172.21.15.104:0/3227046184 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a477bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.696+0000 7f7c4a59c640 1 -- 172.21.15.104:0/3227046184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 msgr2=0x55b6a4795ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.697+0000 7f7c4a59c640 1 --2- 172.21.15.104:0/3227046184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a4795ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.697+0000 7f7c4a59c640 1 -- 172.21.15.104:0/3227046184 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6a4776760 con 0x55b6a4773a90 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.697+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/3227046184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a4795ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.697+0000 7f7c4a59c640 1 --2- 172.21.15.104:0/3227046184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a4777a30 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c000c00 tx=0x7f7c3c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a803d6ff08235c3e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:28.701 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.697+0000 7f7c33fff640 1 -- 172.21.15.104:0/3227046184 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c051020 con 0x55b6a4773a90 2024-06-09T22:26:28.701 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.697+0000 7f7c33fff640 1 -- 172.21.15.104:0/3227046184 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c3c005520 con 0x55b6a4773a90 2024-06-09T22:26:28.701 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.697+0000 7f7c33fff640 1 -- 172.21.15.104:0/3227046184 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c035040 con 0x55b6a4773a90 2024-06-09T22:26:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.698+0000 7f7c53bfd740 1 -- 172.21.15.104:0/3227046184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 msgr2=0x55b6a4777a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.698+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/3227046184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a4777a30 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c000c00 tx=0x7f7c3c005a20 comp rx=0 tx=0).stop 2024-06-09T22:26:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.698+0000 7f7c53bfd740 1 -- 172.21.15.104:0/3227046184 shutdown_connections 2024-06-09T22:26:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.698+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/3227046184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a477e350 0x55b6a4795ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.698+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/3227046184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773a90 0x55b6a4777a30 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.698+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/3227046184 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a4773170 0x55b6a477bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.698+0000 7f7c53bfd740 1 -- 172.21.15.104:0/3227046184 >> 172.21.15.104:0/3227046184 conn(0x55b6a476f680 msgr2=0x55b6a476e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.698+0000 7f7c53bfd740 1 -- 172.21.15.104:0/3227046184 shutdown_connections 2024-06-09T22:26:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.698+0000 7f7c53bfd740 1 -- 172.21.15.104:0/3227046184 wait complete. 2024-06-09T22:26:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.698+0000 7f7c53bfd740 1 -- start start 2024-06-09T22:26:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.698+0000 7f7c53bfd740 1 Processor -- start 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c53bfd740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773170 0x55b6a478a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c53bfd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a4773a90 0x55b6a47833c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c53bfd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a477e350 0x55b6a4783900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c53bfd740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1ae100 con 0x55b6a4773170 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c53bfd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1ada30 con 0x55b6a4773a90 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c53bfd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1aac50 con 0x55b6a477e350 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c53bfd740 10 client.0 _ll_register_callbacks cb 0x7f7c400116c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c49d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773170 0x55b6a478a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c49d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773170 0x55b6a478a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41364/0 (socket says 172.21.15.104:41364) 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c49d9b640 1 -- 172.21.15.104:0/93456904 learned_addr learned my addr 172.21.15.104:0/93456904 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c4a59c640 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a4773a90 0x55b6a47833c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a477e350 0x55b6a4783900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c49d9b640 1 -- 172.21.15.104:0/93456904 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a477e350 msgr2=0x55b6a4783900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a477e350 0x55b6a4783900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c49d9b640 1 -- 172.21.15.104:0/93456904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a4773a90 msgr2=0x55b6a47833c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a4773a90 0x55b6a47833c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c49d9b640 1 -- 172.21.15.104:0/93456904 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6a4783f40 con 0x55b6a4773170 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c49d9b640 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773170 0x55b6a478a790 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009fd0 tx=0x7f7c40011770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db53505b88377e69 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c31ffb640 1 -- 172.21.15.104:0/93456904 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40032040 con 0x55b6a4773170 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c31ffb640 1 -- 172.21.15.104:0/93456904 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7c4000fbe0 con 0x55b6a4773170 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.699+0000 7f7c53bfd740 1 -- 172.21.15.104:0/93456904 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c3c005520 con 0x55b6a4773170 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c31ffb640 1 -- 172.21.15.104:0/93456904 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7c4000fde0 con 0x55b6a4773170 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c53bfd740 10 client.17001 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c53bfd740 1 -- 172.21.15.104:0/93456904 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b6a47f8ce0 con 0x55b6a4773170 2024-06-09T22:26:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c31ffb640 1 -- 172.21.15.104:0/93456904 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c3c005520 con 0x55b6a4773170 2024-06-09T22:26:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c31ffb640 1 -- 172.21.15.104:0/93456904 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7c40005200 con 0x55b6a4773170 2024-06-09T22:26:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c53bfd740 10 client.17001 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c53bfd740 10 client.17001 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c53bfd740 20 client.17001 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b6a47f9210 0x55b6a47fb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c53bfd740 4 client.17001 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:26:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c53bfd740 1 -- 172.21.15.104:0/93456904 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b6a46e4c20 con 0x55b6a47f9210 2024-06-09T22:26:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c4a59c640 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b6a47f9210 0x55b6a47fb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c31ffb640 10 client.17001 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.700+0000 7f7c4a59c640 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b6a47f9210 0x55b6a47fb630 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8af45eb7b71da19e server_cookie=f6ad0bd45b16f817 in_seq=0 out_seq=0 2024-06-09T22:26:28.710 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c31ffb640 1 -- 172.21.15.104:0/93456904 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b6a46e4c20 con 0x55b6a47f9210 2024-06-09T22:26:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c31ffb640 10 client.17001 handle_command_reply: tid=0 2024-06-09T22:26:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c53bfd740 1 client.17001 shutdown 2024-06-09T22:26:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c53bfd740 1 -- 172.21.15.104:0/93456904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773170 msgr2=0x55b6a478a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773170 0x55b6a478a790 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009fd0 tx=0x7f7c40011770 comp rx=0 tx=0).stop 2024-06-09T22:26:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c53bfd740 1 -- 172.21.15.104:0/93456904 shutdown_connections 2024-06-09T22:26:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b6a47f9210 0x55b6a47fb630 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b6a477e350 0x55b6a4783900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b6a4773a90 0x55b6a47833c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c53bfd740 1 --2- 172.21.15.104:0/93456904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6a4773170 0x55b6a478a790 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c53bfd740 1 -- 172.21.15.104:0/93456904 >> 172.21.15.104:0/93456904 conn(0x55b6a476f680 msgr2=0x55b6a4774b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c31ffb640 0 client.17001 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c53bfd740 1 -- 172.21.15.104:0/93456904 shutdown_connections 2024-06-09T22:26:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c53bfd740 1 -- 172.21.15.104:0/93456904 wait complete. 2024-06-09T22:26:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.709+0000 7f7c53bfd740 20 client.17001 trim_cache size 0 max 16384 2024-06-09T22:26:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:26:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:26:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:26:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.710+0000 7f7c50e1b640 1 -- 172.21.15.104:0/3296639166 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7c340a2e20 msgr2=0x7f7c340a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.710+0000 7f7c50e1b640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7c340a2e20 0x7f7c340a52e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400bdf0 tx=0x7f7c4400e040 comp rx=0 tx=0).stop 2024-06-09T22:26:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.710+0000 7f7c50e1b640 1 -- 172.21.15.104:0/3296639166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c4c10bf70 msgr2=0x7f7c4c19cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.710+0000 7f7c50e1b640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c4c10bf70 0x7f7c4c19cb80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400be20 tx=0x7f7c4400bf10 comp rx=0 tx=0).stop 2024-06-09T22:26:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.710+0000 7f7c50e1b640 1 -- 172.21.15.104:0/3296639166 shutdown_connections 2024-06-09T22:26:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.710+0000 7f7c50e1b640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7c340a2e20 0x7f7c340a52e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.710+0000 7f7c50e1b640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c4c10bf70 0x7f7c4c19cb80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.710+0000 7f7c50e1b640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c4c103550 0x7f7c4c198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.710+0000 7f7c50e1b640 1 --2- 172.21.15.104:0/3296639166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c4c102c30 0x7f7c4c198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.710+0000 7f7c50e1b640 1 -- 172.21.15.104:0/3296639166 >> 172.21.15.104:0/3296639166 conn(0x7f7c4c077a20 msgr2=0x7f7c4c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:28.714 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.710+0000 7f7c50e1b640 1 -- 172.21.15.104:0/3296639166 shutdown_connections 2024-06-09T22:26:28.714 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:28.710+0000 7f7c50e1b640 1 -- 172.21.15.104:0/3296639166 wait complete. 2024-06-09T22:26:28.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 171062 5.70 MB/sec execute 94 sec latency 15.653 ms 2024-06-09T22:26:29.073 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:29.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:29 smithi194 ceph-mon[36856]: pgmap v168: 129 pgs: 129 active+clean; 120 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-06-09T22:26:29.329 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:29 smithi104 ceph-mon[32389]: pgmap v168: 129 pgs: 129 active+clean; 120 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-06-09T22:26:29.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:29 smithi153 ceph-mon[36599]: pgmap v168: 129 pgs: 129 active+clean; 120 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-06-09T22:26:29.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 173203 5.76 MB/sec execute 95 sec latency 16.271 ms 2024-06-09T22:26:29.878 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-09T22:26:30.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 174544 5.78 MB/sec execute 96 sec latency 45.424 ms 2024-06-09T22:26:31.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:31 smithi104 ceph-mon[32389]: pgmap v169: 129 pgs: 129 active+clean; 125 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.0 MiB/s wr, 273 op/s 2024-06-09T22:26:31.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:31 smithi194 ceph-mon[36856]: pgmap v169: 129 pgs: 129 active+clean; 125 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.0 MiB/s wr, 273 op/s 2024-06-09T22:26:31.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:31 smithi153 ceph-mon[36599]: pgmap v169: 129 pgs: 129 active+clean; 125 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.0 MiB/s wr, 273 op/s 2024-06-09T22:26:31.617 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:31.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.628+0000 7f2d03fc3700 1 -- 172.21.15.104:0/2673218960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc101fa0 msgr2=0x7f2cfc105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:31.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.628+0000 7f2d03fc3700 1 --2- 172.21.15.104:0/2673218960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc101fa0 0x7f2cfc105fe0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0099c0 tx=0x7f2cec009cd0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.628+0000 7f2d03fc3700 1 -- 172.21.15.104:0/2673218960 shutdown_connections 2024-06-09T22:26:31.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.628+0000 7f2d03fc3700 1 --2- 172.21.15.104:0/2673218960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cfc1066c0 0x7f2cfc10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.628+0000 7f2d03fc3700 1 --2- 172.21.15.104:0/2673218960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cfc1015f0 0x7f2cfc1019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.628+0000 7f2d03fc3700 1 --2- 172.21.15.104:0/2673218960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc101fa0 0x7f2cfc105fe0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.628+0000 7f2d03fc3700 1 -- 172.21.15.104:0/2673218960 >> 172.21.15.104:0/2673218960 conn(0x7f2cfc078330 msgr2=0x7f2cfc078740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:31.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.628+0000 7f2d03fc3700 1 -- 172.21.15.104:0/2673218960 shutdown_connections 2024-06-09T22:26:31.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.628+0000 7f2d03fc3700 1 -- 172.21.15.104:0/2673218960 wait complete. 2024-06-09T22:26:31.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.629+0000 7f2d03fc3700 1 Processor -- start 2024-06-09T22:26:31.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.629+0000 7f2d03fc3700 1 -- start start 2024-06-09T22:26:31.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.629+0000 7f2d03fc3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cfc1015f0 0x7f2cfc1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.629+0000 7f2d03fc3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cfc101fa0 0x7f2cfc199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.629+0000 7f2d03fc3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc1066c0 0x7f2cfc1a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.629+0000 7f2d03fc3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2cfc111680 con 0x7f2cfc1066c0 2024-06-09T22:26:31.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.629+0000 7f2d03fc3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2cfc111500 con 0x7f2cfc101fa0 2024-06-09T22:26:31.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.629+0000 7f2d03fc3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2cfc111800 con 0x7f2cfc1015f0 2024-06-09T22:26:31.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.629+0000 7f2d01d5f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cfc1015f0 0x7f2cfc1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.629+0000 7f2d01d5f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cfc1015f0 0x7f2cfc1999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:60130/0 (socket says 172.21.15.104:60130) 2024-06-09T22:26:31.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.629+0000 7f2d01d5f700 1 -- 172.21.15.104:0/648230130 learned_addr learned my addr 172.21.15.104:0/648230130 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:31.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.630+0000 7f2d02560700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc1066c0 0x7f2cfc1a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.630+0000 7f2d0155e700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cfc101fa0 0x7f2cfc199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.630+0000 7f2d01d5f700 1 -- 172.21.15.104:0/648230130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cfc101fa0 msgr2=0x7f2cfc199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:31.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.630+0000 7f2d01d5f700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cfc101fa0 0x7f2cfc199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.630+0000 7f2d01d5f700 1 -- 172.21.15.104:0/648230130 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc1066c0 msgr2=0x7f2cfc1a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:31.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.630+0000 7f2d01d5f700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc1066c0 0x7f2cfc1a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.630+0000 7f2d01d5f700 1 -- 172.21.15.104:0/648230130 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cfc1a4fa0 con 0x7f2cfc1015f0 2024-06-09T22:26:31.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.630+0000 7f2d01d5f700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cfc1015f0 0x7f2cfc1999e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800e9a0 tx=0x7f2cf800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:31.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.630+0000 7f2cf2ffd700 1 -- 172.21.15.104:0/648230130 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf800ca10 con 0x7f2cfc1015f0 2024-06-09T22:26:31.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.630+0000 7f2cf2ffd700 1 -- 172.21.15.104:0/648230130 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cf800cbb0 con 0x7f2cfc1015f0 2024-06-09T22:26:31.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.630+0000 7f2cf2ffd700 1 -- 172.21.15.104:0/648230130 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8018680 con 0x7f2cfc1015f0 2024-06-09T22:26:31.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.630+0000 7f2d03fc3700 1 -- 172.21.15.104:0/648230130 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cfc1a51d0 con 0x7f2cfc1015f0 2024-06-09T22:26:31.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.631+0000 7f2d03fc3700 1 -- 172.21.15.104:0/648230130 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cfc1a55d0 con 0x7f2cfc1015f0 2024-06-09T22:26:31.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.633+0000 7f2cf2ffd700 1 -- 172.21.15.104:0/648230130 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f2cf8018820 con 0x7f2cfc1015f0 2024-06-09T22:26:31.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.634+0000 7f2cf2ffd700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2ce80a2eb0 0x7f2ce80a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.634+0000 7f2d0155e700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2ce80a2eb0 0x7f2ce80a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.634+0000 7f2cf2ffd700 1 -- 172.21.15.104:0/648230130 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f2cf80cf980 con 0x7f2cfc1015f0 2024-06-09T22:26:31.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.635+0000 7f2d0155e700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2ce80a2eb0 0x7f2ce80a5370 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2cec000c00 tx=0x7f2cec005fb0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:31.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.635+0000 7f2d0e846b80 1 Processor -- start 2024-06-09T22:26:31.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.635+0000 7f2d0e846b80 1 -- start start 2024-06-09T22:26:31.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d0e846b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 0x561b389b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d0e846b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b7600 0x561b389b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d0e846b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b1470 0x561b389b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d0e846b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561b389029b0 con 0x561b389b7600 2024-06-09T22:26:31.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d0e846b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561b388aa8c0 con 0x561b389b6ce0 2024-06-09T22:26:31.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d0e846b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561b388fe970 con 0x561b389b1470 2024-06-09T22:26:31.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d01d5f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 0x561b389b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d0155e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b7600 0x561b389b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d01d5f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 0x561b389b70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:43350/0 (socket says 172.21.15.104:43350) 2024-06-09T22:26:31.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d01d5f700 1 -- 172.21.15.104:0/1683981043 learned_addr learned my addr 172.21.15.104:0/1683981043 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:31.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d02560700 1 --2- 172.21.15.104:0/1683981043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b1470 0x561b389b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d02560700 1 -- 172.21.15.104:0/1683981043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 msgr2=0x561b389b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:31.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d02560700 1 --2- 172.21.15.104:0/1683981043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 0x561b389b70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d02560700 1 -- 172.21.15.104:0/1683981043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b7600 msgr2=0x561b389b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:31.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d02560700 1 --2- 172.21.15.104:0/1683981043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b7600 0x561b389b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d02560700 1 -- 172.21.15.104:0/1683981043 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b388a10f0 con 0x561b389b1470 2024-06-09T22:26:31.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.636+0000 7f2d0155e700 1 --2- 172.21.15.104:0/1683981043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b7600 0x561b389b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:31.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.637+0000 7f2d02560700 1 --2- 172.21.15.104:0/1683981043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b1470 0x561b389b18b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4011580 tx=0x7f2cf4009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d381dbca972bc72 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:31.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.637+0000 7f2ce7fff700 1 -- 172.21.15.104:0/1683981043 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4011e80 con 0x561b389b1470 2024-06-09T22:26:31.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.637+0000 7f2ce7fff700 1 -- 172.21.15.104:0/1683981043 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cf4004bb0 con 0x561b389b1470 2024-06-09T22:26:31.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.637+0000 7f2ce7fff700 1 -- 172.21.15.104:0/1683981043 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4004840 con 0x561b389b1470 2024-06-09T22:26:31.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.638+0000 7f2d0e846b80 1 -- 172.21.15.104:0/1683981043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b1470 msgr2=0x561b389b18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:31.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.638+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/1683981043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b1470 0x561b389b18b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4011580 tx=0x7f2cf4009920 comp rx=0 tx=0).stop 2024-06-09T22:26:31.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.638+0000 7f2d0e846b80 1 -- 172.21.15.104:0/1683981043 shutdown_connections 2024-06-09T22:26:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.638+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/1683981043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b1470 0x561b389b18b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.638+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/1683981043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 0x561b389b70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.638+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/1683981043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b7600 0x561b389b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.638+0000 7f2d0e846b80 1 -- 172.21.15.104:0/1683981043 >> 172.21.15.104:0/1683981043 conn(0x561b388dbe60 msgr2=0x561b389ac1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.638+0000 7f2d0e846b80 1 -- 172.21.15.104:0/1683981043 shutdown_connections 2024-06-09T22:26:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.638+0000 7f2d0e846b80 1 -- 172.21.15.104:0/1683981043 wait complete. 2024-06-09T22:26:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.638+0000 7f2d0e846b80 1 -- start start 2024-06-09T22:26:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d0e846b80 1 Processor -- start 2024-06-09T22:26:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d0e846b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389bcef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d0e846b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 0x561b389bd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d0e846b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b7600 0x561b389c0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d0e846b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561b389c2480 con 0x561b389b1470 2024-06-09T22:26:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d0e846b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561b388a2d90 con 0x561b389b6ce0 2024-06-09T22:26:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d0e846b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561b389c2600 con 0x561b389b7600 2024-06-09T22:26:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d0e846b80 10 client.0 _ll_register_callbacks cb 0x561b3889cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d0155e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 0x561b389bd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d01d5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389bcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d0155e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 0x561b389bd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:43358/0 (socket says 172.21.15.104:43358) 2024-06-09T22:26:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d0155e700 1 -- 172.21.15.104:0/546930935 learned_addr learned my addr 172.21.15.104:0/546930935 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d01d5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389bcef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41396/0 (socket says 172.21.15.104:41396) 2024-06-09T22:26:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.639+0000 7f2d02560700 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b7600 0x561b389c0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2d0155e700 1 -- 172.21.15.104:0/546930935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b7600 msgr2=0x561b389c0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2d0155e700 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b7600 0x561b389c0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2d0155e700 1 -- 172.21.15.104:0/546930935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 msgr2=0x561b389bcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2d0155e700 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389bcef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2d0155e700 1 -- 172.21.15.104:0/546930935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b38851a70 con 0x561b389b6ce0 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2d01d5f700 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389bcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2d02560700 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b7600 0x561b389c0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2d0155e700 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 0x561b389bd430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0099c0 tx=0x7f2cec009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e70d6918a6e729 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2ce57fa700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2ce57fa700 1 -- 172.21.15.104:0/546930935 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cec024ce0 con 0x561b389b6ce0 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2ce57fa700 1 -- 172.21.15.104:0/546930935 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f2cec030460 con 0x561b389b6ce0 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2d0e846b80 1 -- 172.21.15.104:0/546930935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cf4004bb0 con 0x561b389b6ce0 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2ce57fa700 1 -- 172.21.15.104:0/546930935 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2cec03e050 con 0x561b389b6ce0 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2d0e846b80 10 client.26324 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.640+0000 7f2d0e846b80 1 -- 172.21.15.104:0/546930935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561b389d11c0 con 0x561b389b6ce0 2024-06-09T22:26:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.641+0000 7f2ce57fa700 1 -- 172.21.15.104:0/546930935 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cf4004bb0 con 0x561b389b6ce0 2024-06-09T22:26:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.641+0000 7f2ce57fa700 1 -- 172.21.15.104:0/546930935 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2cec00f840 con 0x561b389b6ce0 2024-06-09T22:26:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.641+0000 7f2d0e846b80 10 client.26324 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.641+0000 7f2d0e846b80 10 client.26324 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:26:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.641+0000 7f2d0e846b80 20 client.26324 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.641+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561b389d3c00 0x561b389d6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.641+0000 7f2d0e846b80 4 client.26324 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.641+0000 7f2d0e846b80 1 -- 172.21.15.104:0/546930935 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b3890feb0 con 0x561b389d3c00 2024-06-09T22:26:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.641+0000 7f2d01d5f700 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561b389d3c00 0x561b389d6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.642+0000 7f2ce57fa700 10 client.26324 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.642+0000 7f2d01d5f700 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561b389d3c00 0x561b389d6040 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbf33c75ec433dcd server_cookie=4fdf452dd6f1e1d3 in_seq=0 out_seq=0 2024-06-09T22:26:31.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2ce57fa700 1 -- 172.21.15.104:0/546930935 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x561b3890feb0 con 0x561b389d3c00 2024-06-09T22:26:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2ce57fa700 10 client.26324 handle_command_reply: tid=0 2024-06-09T22:26:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2d0e846b80 1 client.26324 shutdown 2024-06-09T22:26:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2d0e846b80 1 -- 172.21.15.104:0/546930935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 msgr2=0x561b389bd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 0x561b389bd430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0099c0 tx=0x7f2cec009310 comp rx=0 tx=0).stop 2024-06-09T22:26:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2d0e846b80 1 -- 172.21.15.104:0/546930935 shutdown_connections 2024-06-09T22:26:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561b389d3c00 0x561b389d6040 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389bcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b7600 0x561b389c0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/546930935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b6ce0 0x561b389bd430 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2d0e846b80 1 -- 172.21.15.104:0/546930935 >> 172.21.15.104:0/546930935 conn(0x561b388dbe60 msgr2=0x561b389abdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2ce57fa700 0 client.26324 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2d0e846b80 1 -- 172.21.15.104:0/546930935 shutdown_connections 2024-06-09T22:26:31.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2d0e846b80 1 -- 172.21.15.104:0/546930935 wait complete. 2024-06-09T22:26:31.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.643+0000 7f2d0e846b80 20 client.26324 trim_cache size 0 max 16384 2024-06-09T22:26:31.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d0e846b80 1 Processor -- start 2024-06-09T22:26:31.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d0e846b80 1 -- start start 2024-06-09T22:26:31.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d0e846b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d0e846b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b6ce0 0x561b389ab8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d0e846b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b7600 0x561b389abe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d0e846b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2cfc1ac1c0 con 0x561b389b1470 2024-06-09T22:26:31.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d0e846b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2cfc1add00 con 0x561b389b7600 2024-06-09T22:26:31.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d0e846b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2cfc1ac890 con 0x561b389b6ce0 2024-06-09T22:26:31.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d01d5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d02560700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b7600 0x561b389abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d01d5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389b0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41410/0 (socket says 172.21.15.104:41410) 2024-06-09T22:26:31.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d01d5f700 1 -- 172.21.15.104:0/3347192307 learned_addr learned my addr 172.21.15.104:0/3347192307 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:31.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d0155e700 1 --2- 172.21.15.104:0/3347192307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b6ce0 0x561b389ab8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d01d5f700 1 -- 172.21.15.104:0/3347192307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b6ce0 msgr2=0x561b389ab8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:31.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d01d5f700 1 --2- 172.21.15.104:0/3347192307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b6ce0 0x561b389ab8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d01d5f700 1 -- 172.21.15.104:0/3347192307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b7600 msgr2=0x561b389abe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:31.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d01d5f700 1 --2- 172.21.15.104:0/3347192307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b7600 0x561b389abe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d01d5f700 1 -- 172.21.15.104:0/3347192307 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b389ace70 con 0x561b389b1470 2024-06-09T22:26:31.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.669+0000 7f2d02560700 1 --2- 172.21.15.104:0/3347192307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b7600 0x561b389abe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:31.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.670+0000 7f2d01d5f700 1 --2- 172.21.15.104:0/3347192307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389b0f30 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f2cf809c800 tx=0x7f2cf800c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef1b2bea057d6cb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:31.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.670+0000 7f2ce77fe700 1 -- 172.21.15.104:0/3347192307 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf80ba070 con 0x561b389b1470 2024-06-09T22:26:31.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.670+0000 7f2ce77fe700 1 -- 172.21.15.104:0/3347192307 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cf80a5770 con 0x561b389b1470 2024-06-09T22:26:31.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.670+0000 7f2ce77fe700 1 -- 172.21.15.104:0/3347192307 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf809c1b0 con 0x561b389b1470 2024-06-09T22:26:31.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.670+0000 7f2d0e846b80 1 -- 172.21.15.104:0/3347192307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 msgr2=0x561b389b0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:31.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.670+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/3347192307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389b0f30 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f2cf809c800 tx=0x7f2cf800c480 comp rx=0 tx=0).stop 2024-06-09T22:26:31.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.671+0000 7f2d0e846b80 1 -- 172.21.15.104:0/3347192307 shutdown_connections 2024-06-09T22:26:31.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.671+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/3347192307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389b0f30 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.671+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/3347192307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b7600 0x561b389abe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.671+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/3347192307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b6ce0 0x561b389ab8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.671+0000 7f2d0e846b80 1 -- 172.21.15.104:0/3347192307 >> 172.21.15.104:0/3347192307 conn(0x561b388dbe60 msgr2=0x561b389bcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:31.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.671+0000 7f2d0e846b80 1 -- 172.21.15.104:0/3347192307 shutdown_connections 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.671+0000 7f2d0e846b80 1 -- 172.21.15.104:0/3347192307 wait complete. 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.671+0000 7f2d0e846b80 1 -- start start 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.671+0000 7f2d0e846b80 1 Processor -- start 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.671+0000 7f2d0e846b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389d0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.671+0000 7f2d0e846b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b6ce0 0x561b389d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d01d5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0e846b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b7600 0x561b389cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d01d5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389d0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41416/0 (socket says 172.21.15.104:41416) 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d01d5f700 1 -- 172.21.15.104:0/484295165 learned_addr learned my addr 172.21.15.104:0/484295165 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0e846b80 1 -- 172.21.15.104:0/484295165 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2cfc1ad630 con 0x561b389b1470 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0e846b80 1 -- 172.21.15.104:0/484295165 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2cfc1acf60 con 0x561b389b7600 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0e846b80 1 -- 172.21.15.104:0/484295165 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2cfc1aa140 con 0x561b389b6ce0 2024-06-09T22:26:31.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0155e700 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b6ce0 0x561b389d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0e846b80 10 client.0 _ll_register_callbacks cb 0x561b3889cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:31.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d02560700 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b7600 0x561b389cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0155e700 1 -- 172.21.15.104:0/484295165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b7600 msgr2=0x561b389cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:31.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0155e700 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b7600 0x561b389cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0155e700 1 -- 172.21.15.104:0/484295165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 msgr2=0x561b389d0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:31.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0155e700 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389d0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0155e700 1 -- 172.21.15.104:0/484295165 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b389cdb70 con 0x561b389b6ce0 2024-06-09T22:26:31.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d02560700 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b7600 0x561b389cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:26:31.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d01d5f700 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389d0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:31.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0155e700 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b6ce0 0x561b389d0c70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0098e0 tx=0x7f2cec024d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d83cd499a26a5bb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:31.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2ce57fa700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:26:31.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2ce57fa700 1 -- 172.21.15.104:0/484295165 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cec00f870 con 0x561b389b6ce0 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2d0e846b80 1 -- 172.21.15.104:0/484295165 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cf80a5770 con 0x561b389b6ce0 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.672+0000 7f2ce57fa700 1 -- 172.21.15.104:0/484295165 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f2cec00fa10 con 0x561b389b6ce0 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.673+0000 7f2ce57fa700 1 -- 172.21.15.104:0/484295165 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2cec051050 con 0x561b389b6ce0 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.673+0000 7f2d0e846b80 10 client.26419 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.673+0000 7f2d0e846b80 1 -- 172.21.15.104:0/484295165 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561b38a3f530 con 0x561b389b6ce0 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.673+0000 7f2ce57fa700 1 -- 172.21.15.104:0/484295165 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cf80a5770 con 0x561b389b6ce0 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.673+0000 7f2ce57fa700 1 -- 172.21.15.104:0/484295165 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2cec04c030 con 0x561b389b6ce0 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.673+0000 7f2d0e846b80 10 client.26419 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.673+0000 7f2d0e846b80 10 client.26419 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.673+0000 7f2d0e846b80 20 client.26419 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.673+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561b38a3fca0 0x561b38a420c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.673+0000 7f2d0e846b80 4 client.26419 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.45, "await_for": 45.0}] 2024-06-09T22:26:31.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.673+0000 7f2d0e846b80 1 -- 172.21.15.104:0/484295165 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.45, "await_for": 45.0}) -- 0x561b3890a430 con 0x561b38a3fca0 2024-06-09T22:26:31.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.673+0000 7f2d01d5f700 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561b38a3fca0 0x561b38a420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:31.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.674+0000 7f2ce57fa700 10 client.26419 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:31.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.674+0000 7f2d01d5f700 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561b38a3fca0 0x561b38a420c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dfb8067dd68b678 server_cookie=e867f69d6143a28e in_seq=0 out_seq=0 2024-06-09T22:26:31.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.79 MB/sec execute 97 sec latency 37.154 ms 2024-06-09T22:26:31.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.752+0000 7f2ce57fa700 1 -- 172.21.15.104:0/484295165 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x561b3890a430 con 0x561b38a3fca0 2024-06-09T22:26:31.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.752+0000 7f2ce57fa700 10 client.26419 handle_command_reply: tid=0 2024-06-09T22:26:31.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.753+0000 7f2d0e846b80 1 client.26419 shutdown 2024-06-09T22:26:31.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.754+0000 7f2d0e846b80 1 -- 172.21.15.104:0/484295165 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b6ce0 msgr2=0x561b389d0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:31.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.754+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b6ce0 0x561b389d0c70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0098e0 tx=0x7f2cec024d90 comp rx=0 tx=0).stop 2024-06-09T22:26:31.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.754+0000 7f2d0e846b80 1 -- 172.21.15.104:0/484295165 shutdown_connections 2024-06-09T22:26:31.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.754+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561b38a3fca0 0x561b38a420c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.754+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561b389b1470 0x561b389d0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.754+0000 7f2ce57fa700 0 client.26419 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:31.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.754+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b389b7600 0x561b389cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.754+0000 7f2d0e846b80 1 --2- 172.21.15.104:0/484295165 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561b389b6ce0 0x561b389d0c70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.754+0000 7f2d0e846b80 1 -- 172.21.15.104:0/484295165 >> 172.21.15.104:0/484295165 conn(0x561b388dbe60 msgr2=0x561b389ab8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:31.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.754+0000 7f2d0e846b80 1 -- 172.21.15.104:0/484295165 shutdown_connections 2024-06-09T22:26:31.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.755+0000 7f2d0e846b80 1 -- 172.21.15.104:0/484295165 wait complete. 2024-06-09T22:26:31.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.755+0000 7f2d0e846b80 20 client.26419 trim_cache size 0 max 16384 2024-06-09T22:26:31.757 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:26:31.757 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 32, 2024-06-09T22:26:31.758 INFO:teuthology.orchestra.run.smithi104.stdout: "leader": 24460, 2024-06-09T22:26:31.758 INFO:teuthology.orchestra.run.smithi104.stdout: "set_version": 14, 2024-06-09T22:26:31.758 INFO:teuthology.orchestra.run.smithi104.stdout: "sets": { 2024-06-09T22:26:31.758 INFO:teuthology.orchestra.run.smithi104.stdout: "b7cbcb6e": { 2024-06-09T22:26:31.758 INFO:teuthology.orchestra.run.smithi104.stdout: "version": 14, 2024-06-09T22:26:31.758 INFO:teuthology.orchestra.run.smithi104.stdout: "age_ref": 0.0, 2024-06-09T22:26:31.758 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:26:31.758 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "QUIESCED", 2024-06-09T22:26:31.758 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:26:31.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:26:31.759 INFO:teuthology.orchestra.run.smithi104.stdout: "timeout": 90.0, 2024-06-09T22:26:31.759 INFO:teuthology.orchestra.run.smithi104.stdout: "expiration": 149.4, 2024-06-09T22:26:31.759 INFO:teuthology.orchestra.run.smithi104.stdout: "members": { 2024-06-09T22:26:31.759 INFO:teuthology.orchestra.run.smithi104.stdout: "file:/": { 2024-06-09T22:26:31.759 INFO:teuthology.orchestra.run.smithi104.stdout: "excluded": false, 2024-06-09T22:26:31.759 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:26:31.759 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "QUIESCED", 2024-06-09T22:26:31.759 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:26:31.759 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:26:31.759 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:26:31.759 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:26:31.760 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:26:31.760 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:26:31.760 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:26:31.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.756+0000 7f2d03fc3700 1 -- 172.21.15.104:0/648230130 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2ce80a2eb0 msgr2=0x7f2ce80a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:31.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.756+0000 7f2d03fc3700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2ce80a2eb0 0x7f2ce80a5370 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2cec000c00 tx=0x7f2cec005fb0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.756+0000 7f2d03fc3700 1 -- 172.21.15.104:0/648230130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cfc1015f0 msgr2=0x7f2cfc1999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:31.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.756+0000 7f2d03fc3700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cfc1015f0 0x7f2cfc1999e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800e9a0 tx=0x7f2cf800ecb0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.756+0000 7f2d03fc3700 1 -- 172.21.15.104:0/648230130 shutdown_connections 2024-06-09T22:26:31.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.756+0000 7f2d03fc3700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2ce80a2eb0 0x7f2ce80a5370 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2cec000c00 tx=0x7f2cec005fb0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.756+0000 7f2d03fc3700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cfc1066c0 0x7f2cfc1a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.756+0000 7f2d03fc3700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cfc1015f0 0x7f2cfc1999e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.756+0000 7f2d03fc3700 1 --2- 172.21.15.104:0/648230130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cfc101fa0 0x7f2cfc199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:31.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.756+0000 7f2d03fc3700 1 -- 172.21.15.104:0/648230130 >> 172.21.15.104:0/648230130 conn(0x7f2cfc078330 msgr2=0x7f2cfc0ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:31.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.758+0000 7f2d03fc3700 1 -- 172.21.15.104:0/648230130 shutdown_connections 2024-06-09T22:26:31.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:31.758+0000 7f2d03fc3700 1 -- 172.21.15.104:0/648230130 wait complete. 2024-06-09T22:26:32.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.73 MB/sec execute 98 sec latency 1027.726 ms 2024-06-09T22:26:33.215 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:33 smithi104 ceph-mon[32389]: pgmap v170: 129 pgs: 129 active+clean; 125 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 189 op/s 2024-06-09T22:26:33.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:33 smithi194 ceph-mon[36856]: pgmap v170: 129 pgs: 129 active+clean; 125 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 189 op/s 2024-06-09T22:26:33.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:33 smithi153 ceph-mon[36599]: pgmap v170: 129 pgs: 129 active+clean; 125 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 189 op/s 2024-06-09T22:26:33.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.67 MB/sec execute 99 sec latency 2027.775 ms 2024-06-09T22:26:33.971 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b7cbcb6e', quiesce took 10.4 seconds. Will release after: 19.049999999999997 2024-06-09T22:26:34.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.61 MB/sec execute 100 sec latency 3027.904 ms 2024-06-09T22:26:34.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.866+0000 7f787c9d3700 1 -- 172.21.15.104:0/2624179247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7878101390 msgr2=0x7f7878101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:34.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.866+0000 7f787c9d3700 1 --2- 172.21.15.104:0/2624179247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7878101390 0x7f7878101810 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7860009990 tx=0x7f7860009ca0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.866+0000 7f787c9d3700 1 -- 172.21.15.104:0/2624179247 shutdown_connections 2024-06-09T22:26:34.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.866+0000 7f787c9d3700 1 --2- 172.21.15.104:0/2624179247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7878101d50 0x7f787810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.866+0000 7f787c9d3700 1 --2- 172.21.15.104:0/2624179247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7878107400 0x7f78781077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.866+0000 7f787c9d3700 1 --2- 172.21.15.104:0/2624179247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7878101390 0x7f7878101810 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.866+0000 7f787c9d3700 1 -- 172.21.15.104:0/2624179247 >> 172.21.15.104:0/2624179247 conn(0x7f7878078340 msgr2=0x7f7878078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:34.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.867+0000 7f787c9d3700 1 -- 172.21.15.104:0/2624179247 shutdown_connections 2024-06-09T22:26:34.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.867+0000 7f787c9d3700 1 -- 172.21.15.104:0/2624179247 wait complete. 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.867+0000 7f787c9d3700 1 Processor -- start 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.867+0000 7f787c9d3700 1 -- start start 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f787c9d3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7878101390 0x7f787819d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f787c9d3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7878101d50 0x7f787819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f787c9d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7878107400 0x7f78781979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f787c9d3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7878111690 con 0x7f7878107400 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f787c9d3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7878111510 con 0x7f7878101390 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f787c9d3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7878111810 con 0x7f7878101d50 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f7876d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7878107400 0x7f78781979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f7876d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7878107400 0x7f78781979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41436/0 (socket says 172.21.15.104:41436) 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f7876d9d700 1 -- 172.21.15.104:0/247596578 learned_addr learned my addr 172.21.15.104:0/247596578 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f7875d9b700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7878101d50 0x7f787819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f787659c700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7878101390 0x7f787819d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f7876d9d700 1 -- 172.21.15.104:0/247596578 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7878101d50 msgr2=0x7f787819dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:34.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f7876d9d700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7878101d50 0x7f787819dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f7876d9d700 1 -- 172.21.15.104:0/247596578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7878101390 msgr2=0x7f787819d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:34.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f7876d9d700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7878101390 0x7f787819d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f7876d9d700 1 -- 172.21.15.104:0/247596578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7878198110 con 0x7f7878107400 2024-06-09T22:26:34.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f7876d9d700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7878107400 0x7f78781979b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f786800ada0 tx=0x7f786800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:34.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f786f7fe700 1 -- 172.21.15.104:0/247596578 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7868017020 con 0x7f7878107400 2024-06-09T22:26:34.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f786f7fe700 1 -- 172.21.15.104:0/247596578 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786800cdb0 con 0x7f7878107400 2024-06-09T22:26:34.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f787c9d3700 1 -- 172.21.15.104:0/247596578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7878198400 con 0x7f7878107400 2024-06-09T22:26:34.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f786f7fe700 1 -- 172.21.15.104:0/247596578 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78680053e0 con 0x7f7878107400 2024-06-09T22:26:34.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.868+0000 7f787c9d3700 1 -- 172.21.15.104:0/247596578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78781a46c0 con 0x7f7878107400 2024-06-09T22:26:34.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.871+0000 7f786f7fe700 1 -- 172.21.15.104:0/247596578 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f786800c8d0 con 0x7f7878107400 2024-06-09T22:26:34.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.871+0000 7f786f7fe700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f78640a2d90 0x7f78640a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.871+0000 7f787659c700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f78640a2d90 0x7f78640a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.871+0000 7f786f7fe700 1 -- 172.21.15.104:0/247596578 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f78680cef10 con 0x7f7878107400 2024-06-09T22:26:34.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.872+0000 7f787659c700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f78640a2d90 0x7f78640a5250 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7878198af0 tx=0x7f787000b3c0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:34.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.872+0000 7f7887256b80 1 Processor -- start 2024-06-09T22:26:34.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.872+0000 7f7887256b80 1 -- start start 2024-06-09T22:26:34.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.872+0000 7f7887256b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 0x5601f7b8fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.872+0000 7f7887256b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 0x5601f7b899e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.872+0000 7f7887256b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b89f20 0x5601f7b8a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.872+0000 7f7887256b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5601f7adb810 con 0x5601f7b8ff60 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.872+0000 7f7887256b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5601f7a83720 con 0x5601f7b8f640 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.872+0000 7f7887256b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5601f7ad77d0 con 0x5601f7b89f20 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f7875d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 0x5601f7b899e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f7875d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 0x5601f7b899e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41438/0 (socket says 172.21.15.104:41438) 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f7875d9b700 1 -- 172.21.15.104:0/2300885259 learned_addr learned my addr 172.21.15.104:0/2300885259 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f7876d9d700 1 --2- 172.21.15.104:0/2300885259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b89f20 0x5601f7b8a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f787659c700 1 --2- 172.21.15.104:0/2300885259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 0x5601f7b8fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f7875d9b700 1 -- 172.21.15.104:0/2300885259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b89f20 msgr2=0x5601f7b8a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f7875d9b700 1 --2- 172.21.15.104:0/2300885259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b89f20 0x5601f7b8a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f7875d9b700 1 -- 172.21.15.104:0/2300885259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 msgr2=0x5601f7b8fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f7875d9b700 1 --2- 172.21.15.104:0/2300885259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 0x5601f7b8fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f7875d9b700 1 -- 172.21.15.104:0/2300885259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601f7a79f50 con 0x5601f7b8ff60 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f7875d9b700 1 --2- 172.21.15.104:0/2300885259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 0x5601f7b899e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7860009990 tx=0x7f7860005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af90658da3c101b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:34.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f786cff9700 1 -- 172.21.15.104:0/2300885259 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7860028070 con 0x5601f7b8ff60 2024-06-09T22:26:34.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f786cff9700 1 -- 172.21.15.104:0/2300885259 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78600055b0 con 0x5601f7b8ff60 2024-06-09T22:26:34.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.873+0000 7f786cff9700 1 -- 172.21.15.104:0/2300885259 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786000bc10 con 0x5601f7b8ff60 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.874+0000 7f7887256b80 1 -- 172.21.15.104:0/2300885259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 msgr2=0x5601f7b899e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.874+0000 7f7887256b80 1 --2- 172.21.15.104:0/2300885259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 0x5601f7b899e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7860009990 tx=0x7f7860005c70 comp rx=0 tx=0).stop 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.874+0000 7f7887256b80 1 -- 172.21.15.104:0/2300885259 shutdown_connections 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.874+0000 7f7887256b80 1 --2- 172.21.15.104:0/2300885259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b89f20 0x5601f7b8a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.874+0000 7f7887256b80 1 --2- 172.21.15.104:0/2300885259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 0x5601f7b8fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.874+0000 7f7887256b80 1 --2- 172.21.15.104:0/2300885259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 0x5601f7b899e0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.874+0000 7f7887256b80 1 -- 172.21.15.104:0/2300885259 >> 172.21.15.104:0/2300885259 conn(0x5601f7ab4cc0 msgr2=0x5601f7b84850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.874+0000 7f7887256b80 1 -- 172.21.15.104:0/2300885259 shutdown_connections 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.874+0000 7f7887256b80 1 -- 172.21.15.104:0/2300885259 wait complete. 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7887256b80 1 -- start start 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7887256b80 1 Processor -- start 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7887256b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b89f20 0x5601f7b95cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7887256b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8f640 0x5601f7b92fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7887256b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b8ff60 0x5601f7b93510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7887256b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5601f7b96970 con 0x5601f7b8f640 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7887256b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5601f7a7bbf0 con 0x5601f7b89f20 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7887256b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5601f7b96af0 con 0x5601f7b8ff60 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7887256b80 10 client.0 _ll_register_callbacks cb 0x5601f7a75920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7875d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8f640 0x5601f7b92fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7875d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8f640 0x5601f7b92fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41448/0 (socket says 172.21.15.104:41448) 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7875d9b700 1 -- 172.21.15.104:0/1990428598 learned_addr learned my addr 172.21.15.104:0/1990428598 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7876d9d700 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b8ff60 0x5601f7b93510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f787659c700 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b89f20 0x5601f7b95cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7875d9b700 1 -- 172.21.15.104:0/1990428598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b8ff60 msgr2=0x5601f7b93510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7875d9b700 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b8ff60 0x5601f7b93510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7875d9b700 1 -- 172.21.15.104:0/1990428598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b89f20 msgr2=0x5601f7b95cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:34.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.875+0000 7f7875d9b700 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b89f20 0x5601f7b95cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f7875d9b700 1 -- 172.21.15.104:0/1990428598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601f7a2a8d0 con 0x5601f7b8f640 2024-06-09T22:26:34.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f787659c700 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b89f20 0x5601f7b95cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:34.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f7875d9b700 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8f640 0x5601f7b92fd0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f786000b810 tx=0x7f7860005380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297a60d21cec208c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:34.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f785dffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:34.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f785dffb700 1 -- 172.21.15.104:0/1990428598 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7860028070 con 0x5601f7b8f640 2024-06-09T22:26:34.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f785dffb700 1 -- 172.21.15.104:0/1990428598 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f786000bc80 con 0x5601f7b8f640 2024-06-09T22:26:34.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f7887256b80 1 -- 172.21.15.104:0/1990428598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78600055b0 con 0x5601f7b8f640 2024-06-09T22:26:34.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f785dffb700 1 -- 172.21.15.104:0/1990428598 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f786000be80 con 0x5601f7b8f640 2024-06-09T22:26:34.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f7887256b80 10 client.17031 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:34.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f7887256b80 1 -- 172.21.15.104:0/1990428598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5601f7ba1270 con 0x5601f7b8f640 2024-06-09T22:26:34.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f785dffb700 1 -- 172.21.15.104:0/1990428598 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78600055b0 con 0x5601f7b8f640 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f785dffb700 1 -- 172.21.15.104:0/1990428598 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7860035030 con 0x5601f7b8f640 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f7887256b80 10 client.17031 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.876+0000 7f7887256b80 10 client.17031 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.877+0000 7f7887256b80 20 client.17031 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.877+0000 7f7887256b80 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5601f7ba3740 0x5601f7ba5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.877+0000 7f7887256b80 4 client.17031 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.877+0000 7f7887256b80 1 -- 172.21.15.104:0/1990428598 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601f7ae8d10 con 0x5601f7ba3740 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.877+0000 7f787659c700 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5601f7ba3740 0x5601f7ba5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.877+0000 7f787659c700 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5601f7ba3740 0x5601f7ba5b30 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20bec3a11540946d server_cookie=77f60b9b9511b0be in_seq=0 out_seq=0 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.877+0000 7f785dffb700 10 client.17031 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f785dffb700 1 -- 172.21.15.104:0/1990428598 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5601f7ae8d10 con 0x5601f7ba3740 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f785dffb700 10 client.17031 handle_command_reply: tid=0 2024-06-09T22:26:34.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f7887256b80 1 client.17031 shutdown 2024-06-09T22:26:34.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f7887256b80 1 -- 172.21.15.104:0/1990428598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8f640 msgr2=0x5601f7b92fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:34.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f7887256b80 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8f640 0x5601f7b92fd0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f786000b810 tx=0x7f7860005380 comp rx=0 tx=0).stop 2024-06-09T22:26:34.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f7887256b80 1 -- 172.21.15.104:0/1990428598 shutdown_connections 2024-06-09T22:26:34.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f7887256b80 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5601f7ba3740 0x5601f7ba5b30 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f7887256b80 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b8ff60 0x5601f7b93510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f7887256b80 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b89f20 0x5601f7b95cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f7887256b80 1 --2- 172.21.15.104:0/1990428598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8f640 0x5601f7b92fd0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f7887256b80 1 -- 172.21.15.104:0/1990428598 >> 172.21.15.104:0/1990428598 conn(0x5601f7ab4cc0 msgr2=0x5601f7b889d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:34.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f785dffb700 0 client.17031 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:34.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f7887256b80 1 -- 172.21.15.104:0/1990428598 shutdown_connections 2024-06-09T22:26:34.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f7887256b80 1 -- 172.21.15.104:0/1990428598 wait complete. 2024-06-09T22:26:34.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.878+0000 7f7887256b80 20 client.17031 trim_cache size 0 max 16384 2024-06-09T22:26:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.893+0000 7f7887256b80 1 Processor -- start 2024-06-09T22:26:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.893+0000 7f7887256b80 1 -- start start 2024-06-09T22:26:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.893+0000 7f7887256b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b89f20 0x5601f7b899e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f7887256b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 0x5601f7b936a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f7887256b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b8ff60 0x5601f7b90400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f7887256b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f78781ad310 con 0x5601f7b89f20 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f7887256b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78781aee50 con 0x5601f7b8f640 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f7887256b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f78781ad9e0 con 0x5601f7b8ff60 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f787659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b89f20 0x5601f7b899e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f787659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b89f20 0x5601f7b899e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41464/0 (socket says 172.21.15.104:41464) 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f787659c700 1 -- 172.21.15.104:0/3932434064 learned_addr learned my addr 172.21.15.104:0/3932434064 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f787659c700 1 -- 172.21.15.104:0/3932434064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b8ff60 msgr2=0x5601f7b90400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f787659c700 1 --2- 172.21.15.104:0/3932434064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b8ff60 0x5601f7b90400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f787659c700 1 -- 172.21.15.104:0/3932434064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 msgr2=0x5601f7b936a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f7875d9b700 1 --2- 172.21.15.104:0/3932434064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 0x5601f7b936a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f787659c700 1 --2- 172.21.15.104:0/3932434064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 0x5601f7b936a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f787659c700 1 -- 172.21.15.104:0/3932434064 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601f7b8fd70 con 0x5601f7b89f20 2024-06-09T22:26:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f787659c700 1 --2- 172.21.15.104:0/3932434064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b89f20 0x5601f7b899e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f78700037c0 tx=0x7f7870013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744b9001c88a661d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f785ffff700 1 -- 172.21.15.104:0/3932434064 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7870004830 con 0x5601f7b89f20 2024-06-09T22:26:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f785ffff700 1 -- 172.21.15.104:0/3932434064 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78700207d0 con 0x5601f7b89f20 2024-06-09T22:26:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.894+0000 7f785ffff700 1 -- 172.21.15.104:0/3932434064 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7870005240 con 0x5601f7b89f20 2024-06-09T22:26:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 -- 172.21.15.104:0/3932434064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b89f20 msgr2=0x5601f7b899e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 --2- 172.21.15.104:0/3932434064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b89f20 0x5601f7b899e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f78700037c0 tx=0x7f7870013000 comp rx=0 tx=0).stop 2024-06-09T22:26:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 -- 172.21.15.104:0/3932434064 shutdown_connections 2024-06-09T22:26:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 --2- 172.21.15.104:0/3932434064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b89f20 0x5601f7b899e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 --2- 172.21.15.104:0/3932434064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b8ff60 0x5601f7b90400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 --2- 172.21.15.104:0/3932434064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 0x5601f7b936a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 -- 172.21.15.104:0/3932434064 >> 172.21.15.104:0/3932434064 conn(0x5601f7ab4cc0 msgr2=0x5601f7b92f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 -- 172.21.15.104:0/3932434064 shutdown_connections 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 -- 172.21.15.104:0/3932434064 wait complete. 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 -- start start 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 Processor -- start 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b89f20 0x5601f7ba5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 0x5601f7ba5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.895+0000 7f7887256b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 0x5601f7c0c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7887256b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f78781ae780 con 0x5601f7b8ff60 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7887256b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78781ae0b0 con 0x5601f7b8f640 2024-06-09T22:26:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7887256b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f78781ab290 con 0x5601f7b89f20 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7887256b80 10 client.0 _ll_register_callbacks cb 0x5601f7a75920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7876d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 0x5601f7c0c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7876d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 0x5601f7c0c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41478/0 (socket says 172.21.15.104:41478) 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7876d9d700 1 -- 172.21.15.104:0/1300009468 learned_addr learned my addr 172.21.15.104:0/1300009468 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f787659c700 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b89f20 0x5601f7ba5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7875d9b700 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 0x5601f7ba5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7876d9d700 1 -- 172.21.15.104:0/1300009468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b89f20 msgr2=0x5601f7ba5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7876d9d700 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b89f20 0x5601f7ba5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7876d9d700 1 -- 172.21.15.104:0/1300009468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 msgr2=0x5601f7ba5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7876d9d700 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 0x5601f7ba5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7876d9d700 1 -- 172.21.15.104:0/1300009468 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601f7ba6630 con 0x5601f7b8ff60 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7876d9d700 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 0x5601f7c0c070 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f786800c680 tx=0x7f7868097410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93083f3438bc850a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f785dffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f785dffb700 1 -- 172.21.15.104:0/1300009468 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786809b150 con 0x5601f7b8ff60 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f785dffb700 1 -- 172.21.15.104:0/1300009468 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f78680cec50 con 0x5601f7b8ff60 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7887256b80 1 -- 172.21.15.104:0/1300009468 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78700207d0 con 0x5601f7b8ff60 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f785dffb700 1 -- 172.21.15.104:0/1300009468 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f786809f500 con 0x5601f7b8ff60 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7887256b80 10 client.17037 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.896+0000 7f7887256b80 1 -- 172.21.15.104:0/1300009468 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5601f7c0d680 con 0x5601f7b8ff60 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.897+0000 7f785dffb700 1 -- 172.21.15.104:0/1300009468 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78700207d0 con 0x5601f7b8ff60 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.897+0000 7f785dffb700 1 -- 172.21.15.104:0/1300009468 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7868012070 con 0x5601f7b8ff60 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.897+0000 7f7887256b80 10 client.17037 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.897+0000 7f7887256b80 10 client.17037 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.897+0000 7f7887256b80 20 client.17037 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.897+0000 7f7887256b80 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5601f7c0dd10 0x5601f7c10130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.897+0000 7f7887256b80 4 client.17037 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.897+0000 7f7887256b80 1 -- 172.21.15.104:0/1300009468 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5601f7ae3290 con 0x5601f7c0dd10 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.897+0000 7f787659c700 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5601f7c0dd10 0x5601f7c10130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.897+0000 7f785dffb700 10 client.17037 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.897+0000 7f787659c700 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5601f7c0dd10 0x5601f7c10130 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b3e1a53476dcb3d server_cookie=67f4ad64c1763f6d in_seq=0 out_seq=0 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f785dffb700 1 -- 172.21.15.104:0/1300009468 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5601f7ae3290 con 0x5601f7c0dd10 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f785dffb700 10 client.17037 handle_command_reply: tid=0 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f7887256b80 1 client.17037 shutdown 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f7887256b80 1 -- 172.21.15.104:0/1300009468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 msgr2=0x5601f7c0c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f7887256b80 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 0x5601f7c0c070 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f786800c680 tx=0x7f7868097410 comp rx=0 tx=0).stop 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f7887256b80 1 -- 172.21.15.104:0/1300009468 shutdown_connections 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f7887256b80 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5601f7c0dd10 0x5601f7c10130 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f785dffb700 0 client.17037 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f7887256b80 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f7b8ff60 0x5601f7c0c070 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f7887256b80 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f7b89f20 0x5601f7ba5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f7887256b80 1 --2- 172.21.15.104:0/1300009468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5601f7b8f640 0x5601f7ba5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f7887256b80 1 -- 172.21.15.104:0/1300009468 >> 172.21.15.104:0/1300009468 conn(0x5601f7ab4cc0 msgr2=0x5601f7b7de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f7887256b80 1 -- 172.21.15.104:0/1300009468 shutdown_connections 2024-06-09T22:26:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f7887256b80 1 -- 172.21.15.104:0/1300009468 wait complete. 2024-06-09T22:26:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f7887256b80 20 client.17037 trim_cache size 0 max 16384 2024-06-09T22:26:34.905 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:26:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f787c9d3700 1 -- 172.21.15.104:0/247596578 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f78640a2d90 msgr2=0x7f78640a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f787c9d3700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f78640a2d90 0x7f78640a5250 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7878198af0 tx=0x7f787000b3c0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f787c9d3700 1 -- 172.21.15.104:0/247596578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7878107400 msgr2=0x7f78781979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.898+0000 7f787c9d3700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7878107400 0x7f78781979b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f786800ada0 tx=0x7f786800c330 comp rx=0 tx=0).stop 2024-06-09T22:26:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.899+0000 7f787c9d3700 1 -- 172.21.15.104:0/247596578 shutdown_connections 2024-06-09T22:26:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.899+0000 7f787c9d3700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f78640a2d90 0x7f78640a5250 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.899+0000 7f787c9d3700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7878107400 0x7f78781979b0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.899+0000 7f787c9d3700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7878101390 0x7f787819d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.899+0000 7f787c9d3700 1 --2- 172.21.15.104:0/247596578 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7878101d50 0x7f787819dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.899+0000 7f787c9d3700 1 -- 172.21.15.104:0/247596578 >> 172.21.15.104:0/247596578 conn(0x7f7878078340 msgr2=0x7f78780fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:34.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.899+0000 7f787c9d3700 1 -- 172.21.15.104:0/247596578 shutdown_connections 2024-06-09T22:26:34.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:34.899+0000 7f787c9d3700 1 -- 172.21.15.104:0/247596578 wait complete. 2024-06-09T22:26:35.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:35 smithi194 ceph-mon[36856]: pgmap v171: 129 pgs: 129 active+clean; 128 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 269 op/s 2024-06-09T22:26:35.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:35 smithi153 ceph-mon[36599]: pgmap v171: 129 pgs: 129 active+clean; 128 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 269 op/s 2024-06-09T22:26:35.508 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:35 smithi104 ceph-mon[32389]: pgmap v171: 129 pgs: 129 active+clean; 128 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 269 op/s 2024-06-09T22:26:35.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.56 MB/sec execute 101 sec latency 4027.956 ms 2024-06-09T22:26:36.730 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.50 MB/sec execute 102 sec latency 5028.067 ms 2024-06-09T22:26:36.843 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-09T22:26:36.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-09T22:26:37.088 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:37.210 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:37 smithi104 ceph-mon[32389]: pgmap v172: 129 pgs: 129 active+clean; 128 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 182 op/s 2024-06-09T22:26:37.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:37 smithi194 ceph-mon[36856]: pgmap v172: 129 pgs: 129 active+clean; 128 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 182 op/s 2024-06-09T22:26:37.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:37 smithi153 ceph-mon[36599]: pgmap v172: 129 pgs: 129 active+clean; 128 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 182 op/s 2024-06-09T22:26:37.731 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.45 MB/sec execute 103 sec latency 6028.144 ms 2024-06-09T22:26:38.731 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.40 MB/sec execute 104 sec latency 7028.205 ms 2024-06-09T22:26:39.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.948+0000 7fa952358700 1 -- 172.21.15.104:0/2977857694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa94c102740 msgr2=0x7fa94c102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:39.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.948+0000 7fa952358700 1 --2- 172.21.15.104:0/2977857694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa94c102740 0x7fa94c102b20 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fa93c0099c0 tx=0x7fa93c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.948+0000 7fa952358700 1 -- 172.21.15.104:0/2977857694 shutdown_connections 2024-06-09T22:26:39.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.948+0000 7fa952358700 1 --2- 172.21.15.104:0/2977857694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa94c102740 0x7fa94c102b20 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.948+0000 7fa952358700 1 --2- 172.21.15.104:0/2977857694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa94c107840 0x7fa94c10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.948+0000 7fa952358700 1 --2- 172.21.15.104:0/2977857694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa94c1030f0 0x7fa94c107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.948+0000 7fa952358700 1 -- 172.21.15.104:0/2977857694 >> 172.21.15.104:0/2977857694 conn(0x7fa94c0795f0 msgr2=0x7fa94c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:39.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.948+0000 7fa952358700 1 -- 172.21.15.104:0/2977857694 shutdown_connections 2024-06-09T22:26:39.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.948+0000 7fa952358700 1 -- 172.21.15.104:0/2977857694 wait complete. 2024-06-09T22:26:39.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.948+0000 7fa952358700 1 Processor -- start 2024-06-09T22:26:39.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa952358700 1 -- start start 2024-06-09T22:26:39.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa952358700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa94c102740 0x7fa94c19e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa952358700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa94c1030f0 0x7fa94c19ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa952358700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa94c107840 0x7fa94c198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa952358700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c112780 con 0x7fa94c102740 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa952358700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c112600 con 0x7fa94c107840 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa952358700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c112900 con 0x7fa94c1030f0 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa94bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa94c102740 0x7fa94c19e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa94bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa94c102740 0x7fa94c19e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58350/0 (socket says 172.21.15.104:58350) 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa94bfff700 1 -- 172.21.15.104:0/4277248056 learned_addr learned my addr 172.21.15.104:0/4277248056 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa9508f5700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa94c107840 0x7fa94c198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa94c1030f0 0x7fa94c19ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa94bfff700 1 -- 172.21.15.104:0/4277248056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa94c1030f0 msgr2=0x7fa94c19ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa94bfff700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa94c1030f0 0x7fa94c19ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa94bfff700 1 -- 172.21.15.104:0/4277248056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa94c107840 msgr2=0x7fa94c198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:39.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa94bfff700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa94c107840 0x7fa94c198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.949+0000 7fa94bfff700 1 -- 172.21.15.104:0/4277248056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c199160 con 0x7fa94c102740 2024-06-09T22:26:39.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.950+0000 7fa94bfff700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa94c102740 0x7fa94c19e840 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fa93c006010 tx=0x7fa93c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:39.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.950+0000 7fa9497fa700 1 -- 172.21.15.104:0/4277248056 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c026070 con 0x7fa94c102740 2024-06-09T22:26:39.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.950+0000 7fa9497fa700 1 -- 172.21.15.104:0/4277248056 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa93c004cd0 con 0x7fa94c102740 2024-06-09T22:26:39.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.950+0000 7fa952358700 1 -- 172.21.15.104:0/4277248056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c1993f0 con 0x7fa94c102740 2024-06-09T22:26:39.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.950+0000 7fa9497fa700 1 -- 172.21.15.104:0/4277248056 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c003ba0 con 0x7fa94c102740 2024-06-09T22:26:39.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.950+0000 7fa952358700 1 -- 172.21.15.104:0/4277248056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c1a5710 con 0x7fa94c102740 2024-06-09T22:26:39.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.953+0000 7fa9497fa700 1 -- 172.21.15.104:0/4277248056 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa93c029070 con 0x7fa94c102740 2024-06-09T22:26:39.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.953+0000 7fa9497fa700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa9340a2e10 0x7fa9340a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.953+0000 7fa9497fa700 1 -- 172.21.15.104:0/4277248056 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fa93c0d0060 con 0x7fa94c102740 2024-06-09T22:26:39.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.953+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa9340a2e10 0x7fa9340a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa9340a2e10 0x7fa9340a52d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa94c101f60 tx=0x7fa94000b3c0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:39.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa95cbdbb80 1 Processor -- start 2024-06-09T22:26:39.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa95cbdbb80 1 -- start start 2024-06-09T22:26:39.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa95cbdbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db13f0 0x55cfa8db17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa95cbdbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db1d10 0x55cfa8dab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa95cbdbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8dac070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa95cbdbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55cfa8c99fe0 con 0x55cfa8db1d10 2024-06-09T22:26:39.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa95cbdbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cfa8cf88d0 con 0x55cfa8dabc30 2024-06-09T22:26:39.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa95cbdbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cfa8c604e0 con 0x55cfa8db13f0 2024-06-09T22:26:39.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa94b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db1d10 0x55cfa8dab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa94b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db1d10 0x55cfa8dab6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58354/0 (socket says 172.21.15.104:58354) 2024-06-09T22:26:39.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa94b7fe700 1 -- 172.21.15.104:0/3666030116 learned_addr learned my addr 172.21.15.104:0/3666030116 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa9508f5700 1 --2- 172.21.15.104:0/3666030116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8dac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.954+0000 7fa94bfff700 1 --2- 172.21.15.104:0/3666030116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db13f0 0x55cfa8db17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.955+0000 7fa94b7fe700 1 -- 172.21.15.104:0/3666030116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db13f0 msgr2=0x55cfa8db17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.955+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/3666030116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db13f0 0x55cfa8db17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.955+0000 7fa94b7fe700 1 -- 172.21.15.104:0/3666030116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 msgr2=0x55cfa8dac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.955+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/3666030116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8dac070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.955+0000 7fa94b7fe700 1 -- 172.21.15.104:0/3666030116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfa8c47950 con 0x55cfa8db1d10 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.955+0000 7fa9508f5700 1 --2- 172.21.15.104:0/3666030116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8dac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.955+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/3666030116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db1d10 0x55cfa8dab6f0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa94000b670 tx=0x7fa940013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5bf864ecb1cb8d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.955+0000 7fa9327fc700 1 -- 172.21.15.104:0/3666030116 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940011ad0 con 0x55cfa8db1d10 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.955+0000 7fa9327fc700 1 -- 172.21.15.104:0/3666030116 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa940003e90 con 0x55cfa8db1d10 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.955+0000 7fa9327fc700 1 -- 172.21.15.104:0/3666030116 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940005560 con 0x55cfa8db1d10 2024-06-09T22:26:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.956+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/3666030116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db1d10 msgr2=0x55cfa8dab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:39.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.956+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/3666030116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db1d10 0x55cfa8dab6f0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa94000b670 tx=0x7fa940013000 comp rx=0 tx=0).stop 2024-06-09T22:26:39.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.956+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/3666030116 shutdown_connections 2024-06-09T22:26:39.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.956+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/3666030116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8dac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.956+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/3666030116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db13f0 0x55cfa8db17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.956+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/3666030116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db1d10 0x55cfa8dab6f0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.956+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/3666030116 >> 172.21.15.104:0/3666030116 conn(0x55cfa8cd4100 msgr2=0x55cfa8da22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:39.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.956+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/3666030116 shutdown_connections 2024-06-09T22:26:39.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.956+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/3666030116 wait complete. 2024-06-09T22:26:39.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.956+0000 7fa95cbdbb80 1 -- start start 2024-06-09T22:26:39.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.956+0000 7fa95cbdbb80 1 Processor -- start 2024-06-09T22:26:39.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa95cbdbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8dabc30 0x55cfa8db7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa95cbdbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db13f0 0x55cfa8db4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa95cbdbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8db1d10 0x55cfa8db5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa95cbdbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55cfa8c99040 con 0x55cfa8dabc30 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa95cbdbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cfa8cf4890 con 0x55cfa8db1d10 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa95cbdbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cfa8db8700 con 0x55cfa8db13f0 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa95cbdbb80 10 client.0 _ll_register_callbacks cb 0x55cfa8c8e630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa94bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8dabc30 0x55cfa8db7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa94bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8dabc30 0x55cfa8db7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58364/0 (socket says 172.21.15.104:58364) 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa94bfff700 1 -- 172.21.15.104:0/903707968 learned_addr learned my addr 172.21.15.104:0/903707968 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db13f0 0x55cfa8db4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa9508f5700 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8db1d10 0x55cfa8db5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa94bfff700 1 -- 172.21.15.104:0/903707968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db13f0 msgr2=0x55cfa8db4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa94bfff700 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db13f0 0x55cfa8db4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa94bfff700 1 -- 172.21.15.104:0/903707968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8db1d10 msgr2=0x55cfa8db5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa94bfff700 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8db1d10 0x55cfa8db5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa94bfff700 1 -- 172.21.15.104:0/903707968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfa8cbe7d0 con 0x55cfa8dabc30 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa9508f5700 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8db1d10 0x55cfa8db5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa94bfff700 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8dabc30 0x55cfa8db7a50 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa93c0cd4a0 tx=0x7fa93c09bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ed6d755fa5801b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa92b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa92b7fe700 1 -- 172.21.15.104:0/903707968 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c0b5060 con 0x55cfa8dabc30 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.957+0000 7fa92b7fe700 1 -- 172.21.15.104:0/903707968 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fa93c09c700 con 0x55cfa8dabc30 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/903707968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa940003e90 con 0x55cfa8dabc30 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa92b7fe700 1 -- 172.21.15.104:0/903707968 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa93c0b5200 con 0x55cfa8dabc30 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa95cbdbb80 10 client.17049 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/903707968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cfa8dc2e80 con 0x55cfa8dabc30 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa92b7fe700 1 -- 172.21.15.104:0/903707968 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa940003e90 con 0x55cfa8dabc30 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa92b7fe700 1 -- 172.21.15.104:0/903707968 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa93c0a49f0 con 0x55cfa8dabc30 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa95cbdbb80 10 client.17049 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:39.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa95cbdbb80 10 client.17049 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa95cbdbb80 20 client.17049 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cfa8dc5440 0x55cfa8dc7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa95cbdbb80 4 client.17049 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/903707968 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfa8d05dd0 con 0x55cfa8dc5440 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.958+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cfa8dc5440 0x55cfa8dc7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.959+0000 7fa92b7fe700 10 client.17049 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.959+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cfa8dc5440 0x55cfa8dc7870 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70ae53c3c7433e7c server_cookie=b64e0c1ba2c45aa3 in_seq=0 out_seq=0 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa92b7fe700 1 -- 172.21.15.104:0/903707968 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55cfa8d05dd0 con 0x55cfa8dc5440 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa92b7fe700 10 client.17049 handle_command_reply: tid=0 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa95cbdbb80 1 client.17049 shutdown 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/903707968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8dabc30 msgr2=0x55cfa8db7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8dabc30 0x55cfa8db7a50 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa93c0cd4a0 tx=0x7fa93c09bc80 comp rx=0 tx=0).stop 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/903707968 shutdown_connections 2024-06-09T22:26:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cfa8dc5440 0x55cfa8dc7870 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8dabc30 0x55cfa8db7a50 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa92b7fe700 0 client.17049 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8db1d10 0x55cfa8db5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/903707968 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db13f0 0x55cfa8db4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/903707968 >> 172.21.15.104:0/903707968 conn(0x55cfa8cd4100 msgr2=0x55cfa8da1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/903707968 shutdown_connections 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/903707968 wait complete. 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.960+0000 7fa95cbdbb80 20 client.17049 trim_cache size 0 max 16384 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.980+0000 7fa95cbdbb80 1 Processor -- start 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.980+0000 7fa95cbdbb80 1 -- start start 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.980+0000 7fa95cbdbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8db7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.980+0000 7fa95cbdbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 0x55cfa8da1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.980+0000 7fa95cbdbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db1d10 0x55cfa8da2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.980+0000 7fa95cbdbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1ac190 con 0x55cfa8db13f0 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.980+0000 7fa95cbdbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1adcd0 con 0x55cfa8dabc30 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.980+0000 7fa95cbdbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1ac860 con 0x55cfa8db1d10 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa94b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 0x55cfa8da1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa94b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 0x55cfa8da1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58376/0 (socket says 172.21.15.104:58376) 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa94b7fe700 1 -- 172.21.15.104:0/823343653 learned_addr learned my addr 172.21.15.104:0/823343653 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa94bfff700 1 --2- 172.21.15.104:0/823343653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8db7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa9508f5700 1 --2- 172.21.15.104:0/823343653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db1d10 0x55cfa8da2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa94b7fe700 1 -- 172.21.15.104:0/823343653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db1d10 msgr2=0x55cfa8da2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/823343653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db1d10 0x55cfa8da2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa94b7fe700 1 -- 172.21.15.104:0/823343653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 msgr2=0x55cfa8db7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/823343653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8db7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa94b7fe700 1 -- 172.21.15.104:0/823343653 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfa8da3ab0 con 0x55cfa8db13f0 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa94bfff700 1 --2- 172.21.15.104:0/823343653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8db7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:26:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa9508f5700 1 --2- 172.21.15.104:0/823343653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db1d10 0x55cfa8da2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/823343653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 0x55cfa8da1f00 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa940011720 tx=0x7fa940004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40dba72a260a1944 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa931ffb700 1 -- 172.21.15.104:0/823343653 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940011c70 con 0x55cfa8db13f0 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa931ffb700 1 -- 172.21.15.104:0/823343653 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9400056b0 con 0x55cfa8db13f0 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.981+0000 7fa931ffb700 1 -- 172.21.15.104:0/823343653 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940033450 con 0x55cfa8db13f0 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.982+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/823343653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 msgr2=0x55cfa8da1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.982+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/823343653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 0x55cfa8da1f00 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa940011720 tx=0x7fa940004410 comp rx=0 tx=0).stop 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.982+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/823343653 shutdown_connections 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.982+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/823343653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db1d10 0x55cfa8da2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.982+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/823343653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8db7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.982+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/823343653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 0x55cfa8da1f00 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.982+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/823343653 >> 172.21.15.104:0/823343653 conn(0x55cfa8cd4100 msgr2=0x55cfa8db4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.982+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/823343653 shutdown_connections 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.982+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/823343653 wait complete. 2024-06-09T22:26:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.982+0000 7fa95cbdbb80 1 -- start start 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.982+0000 7fa95cbdbb80 1 Processor -- start 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa95cbdbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8dba960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa95cbdbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 0x55cfa8dbaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa95cbdbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db1d10 0x55cfa8dbb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa95cbdbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1ad600 con 0x55cfa8db13f0 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa95cbdbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1acf30 con 0x55cfa8dabc30 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa95cbdbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1aa110 con 0x55cfa8db1d10 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa95cbdbb80 10 client.0 _ll_register_callbacks cb 0x55cfa8c8e630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa94b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 0x55cfa8dbaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa94b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 0x55cfa8dbaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58390/0 (socket says 172.21.15.104:58390) 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa94b7fe700 1 -- 172.21.15.104:0/1265327567 learned_addr learned my addr 172.21.15.104:0/1265327567 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa94bfff700 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8dba960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa9508f5700 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db1d10 0x55cfa8dbb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa94b7fe700 1 -- 172.21.15.104:0/1265327567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db1d10 msgr2=0x55cfa8dbb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db1d10 0x55cfa8dbb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa94b7fe700 1 -- 172.21.15.104:0/1265327567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 msgr2=0x55cfa8dba960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8dba960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa94b7fe700 1 -- 172.21.15.104:0/1265327567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfa8e30380 con 0x55cfa8db13f0 2024-06-09T22:26:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa94b7fe700 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 0x55cfa8dbaea0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa94000b600 tx=0x7fa940011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1319818b92ac375 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa92b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa92b7fe700 1 -- 172.21.15.104:0/1265327567 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940004da0 con 0x55cfa8db13f0 2024-06-09T22:26:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa92b7fe700 1 -- 172.21.15.104:0/1265327567 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fa940004f40 con 0x55cfa8db13f0 2024-06-09T22:26:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.983+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/1265327567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9400056b0 con 0x55cfa8db13f0 2024-06-09T22:26:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.984+0000 7fa92b7fe700 1 -- 172.21.15.104:0/1265327567 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa940005140 con 0x55cfa8db13f0 2024-06-09T22:26:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.984+0000 7fa95cbdbb80 10 client.17055 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.984+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/1265327567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cfa8e315f0 con 0x55cfa8db13f0 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.984+0000 7fa92b7fe700 1 -- 172.21.15.104:0/1265327567 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9400056b0 con 0x55cfa8db13f0 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.984+0000 7fa92b7fe700 1 -- 172.21.15.104:0/1265327567 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa940026440 con 0x55cfa8db13f0 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.984+0000 7fa95cbdbb80 10 client.17055 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.984+0000 7fa95cbdbb80 10 client.17055 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.984+0000 7fa95cbdbb80 20 client.17055 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.984+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cfa8e31aa0 0x55cfa8e33ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.984+0000 7fa95cbdbb80 4 client.17055 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.984+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/1265327567 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cfa8d00350 con 0x55cfa8e31aa0 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.984+0000 7fa94bfff700 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cfa8e31aa0 0x55cfa8e33ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.985+0000 7fa94bfff700 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cfa8e31aa0 0x55cfa8e33ec0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee03cb233e0f6765 server_cookie=3c92138dc23c127a in_seq=0 out_seq=0 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.985+0000 7fa92b7fe700 10 client.17055 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:39.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa92b7fe700 1 -- 172.21.15.104:0/1265327567 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cfa8d00350 con 0x55cfa8e31aa0 2024-06-09T22:26:39.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa92b7fe700 10 client.17055 handle_command_reply: tid=0 2024-06-09T22:26:39.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa95cbdbb80 1 client.17055 shutdown 2024-06-09T22:26:39.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/1265327567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 msgr2=0x55cfa8dbaea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:39.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 0x55cfa8dbaea0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa94000b600 tx=0x7fa940011cf0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/1265327567 shutdown_connections 2024-06-09T22:26:39.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cfa8e31aa0 0x55cfa8e33ec0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cfa8db1d10 0x55cfa8dbb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa92b7fe700 0 client.17055 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:39.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cfa8dabc30 0x55cfa8dba960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa95cbdbb80 1 --2- 172.21.15.104:0/1265327567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cfa8db13f0 0x55cfa8dbaea0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/1265327567 >> 172.21.15.104:0/1265327567 conn(0x55cfa8cd4100 msgr2=0x55cfa8dc10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:39.185 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:26:39.185 INFO:teuthology.orchestra.run.smithi104.stdout: "return_code": 0, 2024-06-09T22:26:39.185 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_tag": "ffc707ed-535e-42bc-b9f5-912730ee01d7", 2024-06-09T22:26:39.185 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": "asynchronous" 2024-06-09T22:26:39.185 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:26:39.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/1265327567 shutdown_connections 2024-06-09T22:26:39.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.986+0000 7fa95cbdbb80 1 -- 172.21.15.104:0/1265327567 wait complete. 2024-06-09T22:26:39.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa95cbdbb80 20 client.17055 trim_cache size 0 max 16384 2024-06-09T22:26:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa952358700 1 -- 172.21.15.104:0/4277248056 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa9340a2e10 msgr2=0x7fa9340a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa952358700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa9340a2e10 0x7fa9340a52d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa94c101f60 tx=0x7fa94000b3c0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa952358700 1 -- 172.21.15.104:0/4277248056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa94c102740 msgr2=0x7fa94c19e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa952358700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa94c102740 0x7fa94c19e840 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fa93c006010 tx=0x7fa93c009f80 comp rx=0 tx=0).stop 2024-06-09T22:26:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa952358700 1 -- 172.21.15.104:0/4277248056 shutdown_connections 2024-06-09T22:26:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa952358700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa9340a2e10 0x7fa9340a52d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa952358700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa94c102740 0x7fa94c19e840 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa952358700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa94c107840 0x7fa94c198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa952358700 1 --2- 172.21.15.104:0/4277248056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa94c1030f0 0x7fa94c19ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa952358700 1 -- 172.21.15.104:0/4277248056 >> 172.21.15.104:0/4277248056 conn(0x7fa94c0795f0 msgr2=0x7fa94c108990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa952358700 1 -- 172.21.15.104:0/4277248056 shutdown_connections 2024-06-09T22:26:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:38.987+0000 7fa952358700 1 -- 172.21.15.104:0/4277248056 wait complete. 2024-06-09T22:26:39.187 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:39 smithi104 ceph-mon[32389]: pgmap v173: 129 pgs: 129 active+clean; 128 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 182 op/s 2024-06-09T22:26:39.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:39 smithi153 ceph-mon[36599]: pgmap v173: 129 pgs: 129 active+clean; 128 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 182 op/s 2024-06-09T22:26:39.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:39 smithi194 ceph-mon[36856]: pgmap v173: 129 pgs: 129 active+clean; 128 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 182 op/s 2024-06-09T22:26:39.731 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.35 MB/sec execute 105 sec latency 8028.241 ms 2024-06-09T22:26:40.015 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:26:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:40.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:40 smithi104 ceph-mon[32389]: scrub queued for path: / 2024-06-09T22:26:40.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:40 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:40 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:26:40.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:40 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:26:40.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:40 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:26:40.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:40 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:40 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:26:40.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:40 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:40 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:40 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:40 smithi194 ceph-mon[36856]: scrub queued for path: / 2024-06-09T22:26:40.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:40 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:40 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:26:40.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:40 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:26:40.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:40 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:26:40.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:40 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:40 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:26:40.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:40 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:40 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:40 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:40 smithi153 ceph-mon[36599]: scrub queued for path: / 2024-06-09T22:26:40.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:40 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:40 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:26:40.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:40 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:26:40.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:40 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:26:40.495 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:40 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.495 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:40 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:26:40.495 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:40 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.495 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:40 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.495 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:40 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:26:40.731 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.30 MB/sec execute 106 sec latency 9028.338 ms 2024-06-09T22:26:41.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:41 smithi194 ceph-mon[36856]: pgmap v174: 129 pgs: 129 active+clean; 130 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-06-09T22:26:41.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:41 smithi153 ceph-mon[36599]: pgmap v174: 129 pgs: 129 active+clean; 130 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-06-09T22:26:41.509 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:41 smithi104 ceph-mon[32389]: pgmap v174: 129 pgs: 129 active+clean; 130 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-06-09T22:26:41.731 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.25 MB/sec execute 107 sec latency 10028.391 ms 2024-06-09T22:26:41.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.830+0000 7fb1f7fff640 1 -- 172.21.15.104:0/4148387647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f8100080 msgr2=0x7fb1f8100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.830+0000 7fb1f7fff640 1 --2- 172.21.15.104:0/4148387647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f8100080 0x7fb1f8100460 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800aa10 tx=0x7fb1e802c820 comp rx=0 tx=0).stop 2024-06-09T22:26:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.830+0000 7fb1f7fff640 1 -- 172.21.15.104:0/4148387647 shutdown_connections 2024-06-09T22:26:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.830+0000 7fb1f7fff640 1 --2- 172.21.15.104:0/4148387647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1f8105150 0x7fb1f810cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.830+0000 7fb1f7fff640 1 --2- 172.21.15.104:0/4148387647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f8100a30 0x7fb1f8104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.830+0000 7fb1f7fff640 1 --2- 172.21.15.104:0/4148387647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f8100080 0x7fb1f8100460 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.830+0000 7fb1f7fff640 1 -- 172.21.15.104:0/4148387647 >> 172.21.15.104:0/4148387647 conn(0x7fb1f8076c20 msgr2=0x7fb1f80fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.830+0000 7fb1f7fff640 1 -- 172.21.15.104:0/4148387647 shutdown_connections 2024-06-09T22:26:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.830+0000 7fb1f7fff640 1 -- 172.21.15.104:0/4148387647 wait complete. 2024-06-09T22:26:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.831+0000 7fb1f7fff640 1 Processor -- start 2024-06-09T22:26:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.831+0000 7fb1f7fff640 1 -- start start 2024-06-09T22:26:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.831+0000 7fb1f7fff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f8100080 0x7fb1f810b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.831+0000 7fb1f7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1f8105150 0x7fb1f810bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.831+0000 7fb1f7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f81a4150 0x7fb1f81a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.831+0000 7fb1f7fff640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f810ef80 con 0x7fb1f8105150 2024-06-09T22:26:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.831+0000 7fb1f7fff640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f810ee00 con 0x7fb1f81a4150 2024-06-09T22:26:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.831+0000 7fb1f7fff640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f810f100 con 0x7fb1f8100080 2024-06-09T22:26:41.834 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.831+0000 7fb1f67fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1f8105150 0x7fb1f810bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.834 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.832+0000 7fb1f67fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1f8105150 0x7fb1f810bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58408/0 (socket says 172.21.15.104:58408) 2024-06-09T22:26:41.834 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.832+0000 7fb1f77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f81a4150 0x7fb1f81a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.834 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.832+0000 7fb1f67fc640 1 -- 172.21.15.104:0/1243198635 learned_addr learned my addr 172.21.15.104:0/1243198635 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:41.834 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.832+0000 7fb1f6ffd640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f8100080 0x7fb1f810b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.832+0000 7fb1f67fc640 1 -- 172.21.15.104:0/1243198635 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f8100080 msgr2=0x7fb1f810b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.832+0000 7fb1f67fc640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f8100080 0x7fb1f810b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.832+0000 7fb1f67fc640 1 -- 172.21.15.104:0/1243198635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f81a4150 msgr2=0x7fb1f81a4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.832+0000 7fb1f67fc640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f81a4150 0x7fb1f81a4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.832+0000 7fb1f67fc640 1 -- 172.21.15.104:0/1243198635 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f81a4c40 con 0x7fb1f8105150 2024-06-09T22:26:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.833+0000 7fb1f67fc640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1f8105150 0x7fb1f810bd80 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000e9c0 tx=0x7fb1e000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.833+0000 7fb1effff640 1 -- 172.21.15.104:0/1243198635 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e000cd80 con 0x7fb1f8105150 2024-06-09T22:26:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.833+0000 7fb1effff640 1 -- 172.21.15.104:0/1243198635 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1e0005500 con 0x7fb1f8105150 2024-06-09T22:26:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.833+0000 7fb1effff640 1 -- 172.21.15.104:0/1243198635 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0010690 con 0x7fb1f8105150 2024-06-09T22:26:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.833+0000 7fb1f7fff640 1 -- 172.21.15.104:0/1243198635 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1f81a4f30 con 0x7fb1f8105150 2024-06-09T22:26:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.833+0000 7fb1f7fff640 1 -- 172.21.15.104:0/1243198635 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1f81ab530 con 0x7fb1f8105150 2024-06-09T22:26:41.838 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.837+0000 7fb2001db740 1 Processor -- start 2024-06-09T22:26:41.838 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.837+0000 7fb2001db740 1 -- start start 2024-06-09T22:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.837+0000 7fb2001db740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e987740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.837+0000 7fb2001db740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e987c80 0x56097e992000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.837+0000 7fb2001db740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e992540 0x56097e994930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.837+0000 7fb2001db740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56097e8eb4f0 con 0x56097e987c80 2024-06-09T22:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.837+0000 7fb2001db740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56097e889870 con 0x56097e987360 2024-06-09T22:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.837+0000 7fb2001db740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56097e8e74b0 con 0x56097e992540 2024-06-09T22:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.838+0000 7fb1f77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e987740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.838+0000 7fb1f77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e987740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:44954/0 (socket says 172.21.15.104:44954) 2024-06-09T22:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.838+0000 7fb1f77fe640 1 -- 172.21.15.104:0/3604844072 learned_addr learned my addr 172.21.15.104:0/3604844072 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.838+0000 7fb1f6ffd640 1 --2- 172.21.15.104:0/3604844072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e987c80 0x56097e992000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.838+0000 7fb1effff640 1 -- 172.21.15.104:0/1243198635 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fb1e00046c0 con 0x7fb1f8105150 2024-06-09T22:26:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.838+0000 7fb1f67fc640 1 --2- 172.21.15.104:0/3604844072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e992540 0x56097e994930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.838+0000 7fb1f6ffd640 1 -- 172.21.15.104:0/3604844072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e992540 msgr2=0x56097e994930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.838+0000 7fb1f6ffd640 1 --2- 172.21.15.104:0/3604844072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e992540 0x56097e994930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.838+0000 7fb1f6ffd640 1 -- 172.21.15.104:0/3604844072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 msgr2=0x56097e987740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.838+0000 7fb1f6ffd640 1 --2- 172.21.15.104:0/3604844072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e987740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.838+0000 7fb1f6ffd640 1 -- 172.21.15.104:0/3604844072 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56097e8a59b0 con 0x56097e987c80 2024-06-09T22:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.839+0000 7fb1f6ffd640 1 --2- 172.21.15.104:0/3604844072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e987c80 0x56097e992000 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb1e802cd40 tx=0x7fb1e8005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab3b33c8cac1ece server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.839+0000 7fb1ed7fa640 1 -- 172.21.15.104:0/3604844072 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8044020 con 0x56097e987c80 2024-06-09T22:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.839+0000 7fb1ed7fa640 1 -- 172.21.15.104:0/3604844072 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1e8032e60 con 0x56097e987c80 2024-06-09T22:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.839+0000 7fb1ed7fa640 1 -- 172.21.15.104:0/3604844072 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8032ae0 con 0x56097e987c80 2024-06-09T22:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.840+0000 7fb1effff640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1e40a2ef0 0x7fb1e40a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.840+0000 7fb1effff640 1 -- 172.21.15.104:0/1243198635 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb1e0010430 con 0x7fb1f8105150 2024-06-09T22:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.840+0000 7fb1f77fe640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1e40a2ef0 0x7fb1e40a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.840+0000 7fb2001db740 1 -- 172.21.15.104:0/3604844072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e987c80 msgr2=0x56097e992000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.840+0000 7fb2001db740 1 --2- 172.21.15.104:0/3604844072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e987c80 0x56097e992000 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb1e802cd40 tx=0x7fb1e8005de0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.840+0000 7fb2001db740 1 -- 172.21.15.104:0/3604844072 shutdown_connections 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.840+0000 7fb2001db740 1 --2- 172.21.15.104:0/3604844072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e992540 0x56097e994930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.841+0000 7fb2001db740 1 --2- 172.21.15.104:0/3604844072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e987c80 0x56097e992000 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.841+0000 7fb2001db740 1 --2- 172.21.15.104:0/3604844072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e987740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.841+0000 7fb2001db740 1 -- 172.21.15.104:0/3604844072 >> 172.21.15.104:0/3604844072 conn(0x56097e983870 msgr2=0x56097e983c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.841+0000 7fb2001db740 1 -- 172.21.15.104:0/3604844072 shutdown_connections 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.841+0000 7fb2001db740 1 -- 172.21.15.104:0/3604844072 wait complete. 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.841+0000 7fb1f77fe640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1e40a2ef0 0x7fb1e40a53b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0011630 tx=0x7fb1f0009500 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.841+0000 7fb2001db740 1 -- start start 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb2001db740 1 Processor -- start 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb2001db740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e98f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb2001db740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e987c80 0x56097e98f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb2001db740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e992540 0x56097e98ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb2001db740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56097e995110 con 0x56097e987c80 2024-06-09T22:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb2001db740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56097e874ae0 con 0x56097e987360 2024-06-09T22:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb2001db740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56097e995290 con 0x56097e992540 2024-06-09T22:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb2001db740 10 client.0 _ll_register_callbacks cb 0x56097e8ad9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb1f6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e98f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb1f6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e98f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:44960/0 (socket says 172.21.15.104:44960) 2024-06-09T22:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb1f6ffd640 1 -- 172.21.15.104:0/2474622134 learned_addr learned my addr 172.21.15.104:0/2474622134 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb1f67fc640 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e987c80 0x56097e98f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb1f77fe640 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e992540 0x56097e98ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb1f6ffd640 1 -- 172.21.15.104:0/2474622134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e992540 msgr2=0x56097e98ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb1f6ffd640 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e992540 0x56097e98ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb1f6ffd640 1 -- 172.21.15.104:0/2474622134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e987c80 msgr2=0x56097e98f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb1f6ffd640 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e987c80 0x56097e98f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb1f6ffd640 1 -- 172.21.15.104:0/2474622134 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56097e5ff210 con 0x56097e987360 2024-06-09T22:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.842+0000 7fb1f67fc640 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e987c80 0x56097e98f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.843+0000 7fb1f6ffd640 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e98f4a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800f270 tx=0x7fb1e800f2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d7eb5ca5da89289 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.843+0000 7fb1de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.843+0000 7fb1de7fc640 1 -- 172.21.15.104:0/2474622134 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8044030 con 0x56097e987360 2024-06-09T22:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.843+0000 7fb2001db740 1 -- 172.21.15.104:0/2474622134 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1e8032e60 con 0x56097e987360 2024-06-09T22:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.843+0000 7fb1de7fc640 1 -- 172.21.15.104:0/2474622134 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb1e8004250 con 0x56097e987360 2024-06-09T22:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.843+0000 7fb1de7fc640 1 -- 172.21.15.104:0/2474622134 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1e805a050 con 0x56097e987360 2024-06-09T22:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.843+0000 7fb2001db740 10 client.26369 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.843+0000 7fb2001db740 1 -- 172.21.15.104:0/2474622134 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56097e6cb620 con 0x56097e987360 2024-06-09T22:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.843+0000 7fb1de7fc640 1 -- 172.21.15.104:0/2474622134 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1e8032e60 con 0x56097e987360 2024-06-09T22:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.844+0000 7fb1de7fc640 1 -- 172.21.15.104:0/2474622134 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb1e800b040 con 0x56097e987360 2024-06-09T22:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.844+0000 7fb2001db740 10 client.26369 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.844+0000 7fb2001db740 10 client.26369 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.844+0000 7fb2001db740 20 client.26369 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.844+0000 7fb2001db740 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56097e9a2660 0x56097e9a4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.844+0000 7fb2001db740 4 client.26369 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.844+0000 7fb2001db740 1 -- 172.21.15.104:0/2474622134 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56097e8fef10 con 0x56097e9a2660 2024-06-09T22:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.844+0000 7fb1f67fc640 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56097e9a2660 0x56097e9a4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.845+0000 7fb1de7fc640 10 client.26369 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.845+0000 7fb1f67fc640 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56097e9a2660 0x56097e9a4af0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18713ffe847c15ca server_cookie=c5f4217f25b90740 in_seq=0 out_seq=0 2024-06-09T22:26:41.851 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb1de7fc640 1 -- 172.21.15.104:0/2474622134 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56097e8fef10 con 0x56097e9a2660 2024-06-09T22:26:41.851 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb1de7fc640 10 client.26369 handle_command_reply: tid=0 2024-06-09T22:26:41.851 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb2001db740 1 client.26369 shutdown 2024-06-09T22:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb2001db740 1 -- 172.21.15.104:0/2474622134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 msgr2=0x56097e98f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb2001db740 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e98f4a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800f270 tx=0x7fb1e800f2a0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb2001db740 1 -- 172.21.15.104:0/2474622134 shutdown_connections 2024-06-09T22:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb2001db740 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56097e9a2660 0x56097e9a4af0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb2001db740 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e992540 0x56097e98ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb2001db740 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e987c80 0x56097e98f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb1de7fc640 0 client.26369 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb2001db740 1 --2- 172.21.15.104:0/2474622134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e98f4a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb2001db740 1 -- 172.21.15.104:0/2474622134 >> 172.21.15.104:0/2474622134 conn(0x56097e983870 msgr2=0x56097e9895f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb2001db740 1 -- 172.21.15.104:0/2474622134 shutdown_connections 2024-06-09T22:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb2001db740 1 -- 172.21.15.104:0/2474622134 wait complete. 2024-06-09T22:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.846+0000 7fb2001db740 20 client.26369 trim_cache size 0 max 16384 2024-06-09T22:26:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.862+0000 7fb2001db740 1 Processor -- start 2024-06-09T22:26:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.862+0000 7fb2001db740 1 -- start start 2024-06-09T22:26:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb2001db740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987360 0x56097e9aa400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb2001db740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987c80 0x56097e9aa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb2001db740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 0x56097e990cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb2001db740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f81b5210 con 0x56097e992540 2024-06-09T22:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb2001db740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f81b6d50 con 0x56097e987c80 2024-06-09T22:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb2001db740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f81b58e0 con 0x56097e987360 2024-06-09T22:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb1f6ffd640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987360 0x56097e9aa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb1f6ffd640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987360 0x56097e9aa400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:43938/0 (socket says 172.21.15.104:43938) 2024-06-09T22:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb1f6ffd640 1 -- 172.21.15.104:0/3350873670 learned_addr learned my addr 172.21.15.104:0/3350873670 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb1f77fe640 1 --2- 172.21.15.104:0/3350873670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 0x56097e990cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb1f67fc640 1 --2- 172.21.15.104:0/3350873670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987c80 0x56097e9aa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb1f77fe640 1 -- 172.21.15.104:0/3350873670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987360 msgr2=0x56097e9aa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb1f77fe640 1 --2- 172.21.15.104:0/3350873670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987360 0x56097e9aa400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb1f77fe640 1 -- 172.21.15.104:0/3350873670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987c80 msgr2=0x56097e9aa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb1f77fe640 1 --2- 172.21.15.104:0/3350873670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987c80 0x56097e9aa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb1f77fe640 1 -- 172.21.15.104:0/3350873670 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56097e987ab0 con 0x56097e992540 2024-06-09T22:26:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.863+0000 7fb1f67fc640 1 --2- 172.21.15.104:0/3350873670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987c80 0x56097e9aa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.864+0000 7fb1f77fe640 1 --2- 172.21.15.104:0/3350873670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 0x56097e990cd0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0005fd0 tx=0x7fb1f0011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f65e9e75a27c756 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.864+0000 7fb1ecff9640 1 -- 172.21.15.104:0/3350873670 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0004510 con 0x56097e992540 2024-06-09T22:26:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.864+0000 7fb1ecff9640 1 -- 172.21.15.104:0/3350873670 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1f00046b0 con 0x56097e992540 2024-06-09T22:26:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.864+0000 7fb1ecff9640 1 -- 172.21.15.104:0/3350873670 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f001e520 con 0x56097e992540 2024-06-09T22:26:41.871 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.864+0000 7fb2001db740 1 -- 172.21.15.104:0/3350873670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 msgr2=0x56097e990cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:41.871 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.864+0000 7fb2001db740 1 --2- 172.21.15.104:0/3350873670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 0x56097e990cd0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0005fd0 tx=0x7fb1f0011f90 comp rx=0 tx=0).stop 2024-06-09T22:26:41.871 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 -- 172.21.15.104:0/3350873670 shutdown_connections 2024-06-09T22:26:41.871 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 --2- 172.21.15.104:0/3350873670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 0x56097e990cd0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.871 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 --2- 172.21.15.104:0/3350873670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987c80 0x56097e9aa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 --2- 172.21.15.104:0/3350873670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987360 0x56097e9aa400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 -- 172.21.15.104:0/3350873670 >> 172.21.15.104:0/3350873670 conn(0x56097e983870 msgr2=0x56097e98a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 -- 172.21.15.104:0/3350873670 shutdown_connections 2024-06-09T22:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 -- 172.21.15.104:0/3350873670 wait complete. 2024-06-09T22:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 -- start start 2024-06-09T22:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 Processor -- start 2024-06-09T22:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e9955c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987c80 0x56097e995b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 0x56097ea0b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f81b6680 con 0x56097e992540 2024-06-09T22:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f81b5fb0 con 0x56097e987360 2024-06-09T22:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f81b31d0 con 0x56097e987c80 2024-06-09T22:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb2001db740 10 client.0 _ll_register_callbacks cb 0x7fb1f0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb1f77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 0x56097ea0b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb1f67fc640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987c80 0x56097e995b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb1f77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 0x56097ea0b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58448/0 (socket says 172.21.15.104:58448) 2024-06-09T22:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb1f67fc640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987c80 0x56097e995b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:43948/0 (socket says 172.21.15.104:43948) 2024-06-09T22:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb1f77fe640 1 -- 172.21.15.104:0/1120066303 learned_addr learned my addr 172.21.15.104:0/1120066303 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.865+0000 7fb1f6ffd640 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e9955c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb1f77fe640 1 -- 172.21.15.104:0/1120066303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987c80 msgr2=0x56097e995b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb1f77fe640 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987c80 0x56097e995b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb1f77fe640 1 -- 172.21.15.104:0/1120066303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 msgr2=0x56097e9955c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb1f77fe640 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e9955c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb1f77fe640 1 -- 172.21.15.104:0/1120066303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56097e996390 con 0x56097e992540 2024-06-09T22:26:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb1f67fc640 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987c80 0x56097e995b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb1f77fe640 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 0x56097ea0b3c0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0004f10 tx=0x7fb1f0004f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a28f99067913a2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb1de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb1de7fc640 1 -- 172.21.15.104:0/1120066303 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0036060 con 0x56097e992540 2024-06-09T22:26:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb1de7fc640 1 -- 172.21.15.104:0/1120066303 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb1f000f770 con 0x56097e992540 2024-06-09T22:26:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb2001db740 1 -- 172.21.15.104:0/1120066303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1f00046b0 con 0x56097e992540 2024-06-09T22:26:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb1de7fc640 1 -- 172.21.15.104:0/1120066303 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1f0004510 con 0x56097e992540 2024-06-09T22:26:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb2001db740 10 client.17073 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.866+0000 7fb2001db740 1 -- 172.21.15.104:0/1120066303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56097ea0c9e0 con 0x56097e992540 2024-06-09T22:26:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.867+0000 7fb1de7fc640 1 -- 172.21.15.104:0/1120066303 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1f00046b0 con 0x56097e992540 2024-06-09T22:26:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.867+0000 7fb1de7fc640 1 -- 172.21.15.104:0/1120066303 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb1f0031440 con 0x56097e992540 2024-06-09T22:26:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.867+0000 7fb2001db740 10 client.17073 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.867+0000 7fb2001db740 10 client.17073 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.867+0000 7fb2001db740 20 client.17073 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.867+0000 7fb2001db740 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56097ea0d0d0 0x56097ea0f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.867+0000 7fb2001db740 4 client.17073 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.867+0000 7fb2001db740 1 -- 172.21.15.104:0/1120066303 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56097e8f8c20 con 0x56097ea0d0d0 2024-06-09T22:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.867+0000 7fb1f6ffd640 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56097ea0d0d0 0x56097ea0f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.868+0000 7fb1f6ffd640 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56097ea0d0d0 0x56097ea0f4f0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f2a633c73096914 server_cookie=1d64405ecbeb9843 in_seq=0 out_seq=0 2024-06-09T22:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.868+0000 7fb1de7fc640 10 client.17073 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.875+0000 7fb1de7fc640 1 -- 172.21.15.104:0/1120066303 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56097e8f8c20 con 0x56097ea0d0d0 2024-06-09T22:26:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.875+0000 7fb1de7fc640 10 client.17073 handle_command_reply: tid=0 2024-06-09T22:26:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.875+0000 7fb2001db740 1 client.17073 shutdown 2024-06-09T22:26:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb2001db740 1 -- 172.21.15.104:0/1120066303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 msgr2=0x56097ea0b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb2001db740 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 0x56097ea0b3c0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0004f10 tx=0x7fb1f0004f40 comp rx=0 tx=0).stop 2024-06-09T22:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb2001db740 1 -- 172.21.15.104:0/1120066303 shutdown_connections 2024-06-09T22:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb2001db740 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56097ea0d0d0 0x56097ea0f4f0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb2001db740 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56097e992540 0x56097ea0b3c0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb1de7fc640 0 client.17073 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb2001db740 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56097e987c80 0x56097e995b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb2001db740 1 --2- 172.21.15.104:0/1120066303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56097e987360 0x56097e9955c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb2001db740 1 -- 172.21.15.104:0/1120066303 >> 172.21.15.104:0/1120066303 conn(0x56097e983870 msgr2=0x56097e982140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb2001db740 1 -- 172.21.15.104:0/1120066303 shutdown_connections 2024-06-09T22:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb2001db740 1 -- 172.21.15.104:0/1120066303 wait complete. 2024-06-09T22:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb2001db740 20 client.17073 trim_cache size 0 max 16384 2024-06-09T22:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:26:41.882 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb1f7fff640 1 -- 172.21.15.104:0/1243198635 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1e40a2ef0 msgr2=0x7fb1e40a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:41.882 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb1f7fff640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1e40a2ef0 0x7fb1e40a53b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0011630 tx=0x7fb1f0009500 comp rx=0 tx=0).stop 2024-06-09T22:26:41.882 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb1f7fff640 1 -- 172.21.15.104:0/1243198635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1f8105150 msgr2=0x7fb1f810bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:41.882 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb1f7fff640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1f8105150 0x7fb1f810bd80 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000e9c0 tx=0x7fb1e000ee80 comp rx=0 tx=0).stop 2024-06-09T22:26:41.882 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb1f7fff640 1 -- 172.21.15.104:0/1243198635 shutdown_connections 2024-06-09T22:26:41.882 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb1f7fff640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb1e40a2ef0 0x7fb1e40a53b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.883 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb1f7fff640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f81a4150 0x7fb1f81a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.883 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb1f7fff640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1f8105150 0x7fb1f810bd80 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.883 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb1f7fff640 1 --2- 172.21.15.104:0/1243198635 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1f8100080 0x7fb1f810b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:41.883 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.876+0000 7fb1f7fff640 1 -- 172.21.15.104:0/1243198635 >> 172.21.15.104:0/1243198635 conn(0x7fb1f8076c20 msgr2=0x7fb1f81105b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:41.883 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.877+0000 7fb1f7fff640 1 -- 172.21.15.104:0/1243198635 shutdown_connections 2024-06-09T22:26:41.883 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:41.877+0000 7fb1f7fff640 1 -- 172.21.15.104:0/1243198635 wait complete. 2024-06-09T22:26:42.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.149+0000 7fc646504700 1 -- 172.21.15.104:0/2470276539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc640101380 msgr2=0x7fc640101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:42.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.149+0000 7fc646504700 1 --2- 172.21.15.104:0/2470276539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc640101380 0x7fc640101800 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fc628009990 tx=0x7fc628009ca0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.149+0000 7fc646504700 1 -- 172.21.15.104:0/2470276539 shutdown_connections 2024-06-09T22:26:42.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.149+0000 7fc646504700 1 --2- 172.21.15.104:0/2470276539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640101d40 0x7fc64010e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.149+0000 7fc646504700 1 --2- 172.21.15.104:0/2470276539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6401073f0 0x7fc6401077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.149+0000 7fc646504700 1 --2- 172.21.15.104:0/2470276539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc640101380 0x7fc640101800 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.149+0000 7fc646504700 1 -- 172.21.15.104:0/2470276539 >> 172.21.15.104:0/2470276539 conn(0x7fc640078300 msgr2=0x7fc640078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:42.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.149+0000 7fc646504700 1 -- 172.21.15.104:0/2470276539 shutdown_connections 2024-06-09T22:26:42.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.149+0000 7fc646504700 1 -- 172.21.15.104:0/2470276539 wait complete. 2024-06-09T22:26:42.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc646504700 1 Processor -- start 2024-06-09T22:26:42.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc646504700 1 -- start start 2024-06-09T22:26:42.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc646504700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc640101380 0x7fc64019e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc646504700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640101d40 0x7fc64019ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc646504700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6401073f0 0x7fc640198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc646504700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc640111660 con 0x7fc640101380 2024-06-09T22:26:42.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc646504700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401114e0 con 0x7fc640101d40 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc646504700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401117e0 con 0x7fc6401073f0 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc63ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc640101380 0x7fc64019e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc63ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc640101380 0x7fc64019e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58468/0 (socket says 172.21.15.104:58468) 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc63ffff700 1 -- 172.21.15.104:0/3431150676 learned_addr learned my addr 172.21.15.104:0/3431150676 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc644aa1700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6401073f0 0x7fc640198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc637fff700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640101d40 0x7fc64019ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc63ffff700 1 -- 172.21.15.104:0/3431150676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6401073f0 msgr2=0x7fc640198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc63ffff700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6401073f0 0x7fc640198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc63ffff700 1 -- 172.21.15.104:0/3431150676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640101d40 msgr2=0x7fc64019ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc63ffff700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640101d40 0x7fc64019ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc63ffff700 1 -- 172.21.15.104:0/3431150676 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc640199090 con 0x7fc640101380 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc637fff700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640101d40 0x7fc64019ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:26:42.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.150+0000 7fc63ffff700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc640101380 0x7fc64019e770 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c9b0 tx=0x7fc63000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:42.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.151+0000 7fc63dffb700 1 -- 172.21.15.104:0/3431150676 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc630007b00 con 0x7fc640101380 2024-06-09T22:26:42.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.151+0000 7fc646504700 1 -- 172.21.15.104:0/3431150676 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc640199380 con 0x7fc640101380 2024-06-09T22:26:42.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.151+0000 7fc63dffb700 1 -- 172.21.15.104:0/3431150676 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc630007ca0 con 0x7fc640101380 2024-06-09T22:26:42.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.151+0000 7fc63dffb700 1 -- 172.21.15.104:0/3431150676 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc630018730 con 0x7fc640101380 2024-06-09T22:26:42.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.151+0000 7fc646504700 1 -- 172.21.15.104:0/3431150676 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6401a5640 con 0x7fc640101380 2024-06-09T22:26:42.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.153+0000 7fc63dffb700 1 -- 172.21.15.104:0/3431150676 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc6300188d0 con 0x7fc640101380 2024-06-09T22:26:42.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.154+0000 7fc63dffb700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc62c0a2d90 0x7fc62c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.154+0000 7fc637fff700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc62c0a2d90 0x7fc62c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.154+0000 7fc63dffb700 1 -- 172.21.15.104:0/3431150676 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fc6300cf750 con 0x7fc640101380 2024-06-09T22:26:42.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.154+0000 7fc637fff700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc62c0a2d90 0x7fc62c0a5250 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc628006010 tx=0x7fc62800b540 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:42.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.154+0000 7fc650d87b80 1 Processor -- start 2024-06-09T22:26:42.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.154+0000 7fc650d87b80 1 -- start start 2024-06-09T22:26:42.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.154+0000 7fc650d87b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 0x55bb8e73ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.154+0000 7fc650d87b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e73bf60 0x55bb8e7359e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.154+0000 7fc650d87b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 0x55bb8e736340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc650d87b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55bb8e687810 con 0x55bb8e735f20 2024-06-09T22:26:42.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc650d87b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bb8e62f720 con 0x55bb8e73bf60 2024-06-09T22:26:42.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc650d87b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bb8e6837d0 con 0x55bb8e73b640 2024-06-09T22:26:42.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc644aa1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 0x55bb8e736340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc644aa1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 0x55bb8e736340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58470/0 (socket says 172.21.15.104:58470) 2024-06-09T22:26:42.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc644aa1700 1 -- 172.21.15.104:0/3394098013 learned_addr learned my addr 172.21.15.104:0/3394098013 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:42.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc63ffff700 1 --2- 172.21.15.104:0/3394098013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 0x55bb8e73ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc637fff700 1 --2- 172.21.15.104:0/3394098013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e73bf60 0x55bb8e7359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc644aa1700 1 -- 172.21.15.104:0/3394098013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 msgr2=0x55bb8e73ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:42.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc644aa1700 1 --2- 172.21.15.104:0/3394098013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 0x55bb8e73ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc644aa1700 1 -- 172.21.15.104:0/3394098013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e73bf60 msgr2=0x55bb8e7359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:42.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc644aa1700 1 --2- 172.21.15.104:0/3394098013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e73bf60 0x55bb8e7359e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc644aa1700 1 -- 172.21.15.104:0/3394098013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb8e625f50 con 0x55bb8e735f20 2024-06-09T22:26:42.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc637fff700 1 --2- 172.21.15.104:0/3394098013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e73bf60 0x55bb8e7359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:26:42.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc644aa1700 1 --2- 172.21.15.104:0/3394098013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 0x55bb8e736340 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fc63800f5a0 tx=0x7fc63800f8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0a3b39b746ece7 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:42.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc6367fc700 1 -- 172.21.15.104:0/3394098013 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc638018030 con 0x55bb8e735f20 2024-06-09T22:26:42.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc6367fc700 1 -- 172.21.15.104:0/3394098013 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc638004bb0 con 0x55bb8e735f20 2024-06-09T22:26:42.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.155+0000 7fc6367fc700 1 -- 172.21.15.104:0/3394098013 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc638005020 con 0x55bb8e735f20 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.156+0000 7fc650d87b80 1 -- 172.21.15.104:0/3394098013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 msgr2=0x55bb8e736340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.156+0000 7fc650d87b80 1 --2- 172.21.15.104:0/3394098013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 0x55bb8e736340 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fc63800f5a0 tx=0x7fc63800f8b0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.156+0000 7fc650d87b80 1 -- 172.21.15.104:0/3394098013 shutdown_connections 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.156+0000 7fc650d87b80 1 --2- 172.21.15.104:0/3394098013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 0x55bb8e736340 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.156+0000 7fc650d87b80 1 --2- 172.21.15.104:0/3394098013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 0x55bb8e73ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.156+0000 7fc650d87b80 1 --2- 172.21.15.104:0/3394098013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e73bf60 0x55bb8e7359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.156+0000 7fc650d87b80 1 -- 172.21.15.104:0/3394098013 >> 172.21.15.104:0/3394098013 conn(0x55bb8e660cc0 msgr2=0x55bb8e730850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.156+0000 7fc650d87b80 1 -- 172.21.15.104:0/3394098013 shutdown_connections 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.156+0000 7fc650d87b80 1 -- 172.21.15.104:0/3394098013 wait complete. 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc650d87b80 1 -- start start 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc650d87b80 1 Processor -- start 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc650d87b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e735f20 0x55bb8e741cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc650d87b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 0x55bb8e73efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc650d87b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73bf60 0x55bb8e73f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc650d87b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55bb8e7429d0 con 0x55bb8e73bf60 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc650d87b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bb8e627bf0 con 0x55bb8e735f20 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc650d87b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bb8e742b50 con 0x55bb8e73b640 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc650d87b80 10 client.0 _ll_register_callbacks cb 0x55bb8e621920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc644aa1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73bf60 0x55bb8e73f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc644aa1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73bf60 0x55bb8e73f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58476/0 (socket says 172.21.15.104:58476) 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc644aa1700 1 -- 172.21.15.104:0/1303787173 learned_addr learned my addr 172.21.15.104:0/1303787173 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc63ffff700 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e735f20 0x55bb8e741cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.157+0000 7fc637fff700 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 0x55bb8e73efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc644aa1700 1 -- 172.21.15.104:0/1303787173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 msgr2=0x55bb8e73efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc644aa1700 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 0x55bb8e73efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc644aa1700 1 -- 172.21.15.104:0/1303787173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e735f20 msgr2=0x55bb8e741cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc644aa1700 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e735f20 0x55bb8e741cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc644aa1700 1 -- 172.21.15.104:0/1303787173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb8e5d68d0 con 0x55bb8e73bf60 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc63ffff700 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e735f20 0x55bb8e741cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc644aa1700 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73bf60 0x55bb8e73f4f0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc638014d70 tx=0x7fc638014720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495ac04ea4a4d27 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc61f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc61f7fe700 1 -- 172.21.15.104:0/1303787173 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6380149c0 con 0x55bb8e73bf60 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc61f7fe700 1 -- 172.21.15.104:0/1303787173 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fc638004250 con 0x55bb8e73bf60 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc650d87b80 1 -- 172.21.15.104:0/1303787173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc638004bb0 con 0x55bb8e73bf60 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc61f7fe700 1 -- 172.21.15.104:0/1303787173 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc638004450 con 0x55bb8e73bf60 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc650d87b80 10 client.17085 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.158+0000 7fc650d87b80 1 -- 172.21.15.104:0/1303787173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bb8e74d220 con 0x55bb8e73bf60 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.159+0000 7fc61f7fe700 1 -- 172.21.15.104:0/1303787173 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6380045f0 con 0x55bb8e73bf60 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.159+0000 7fc61f7fe700 1 -- 172.21.15.104:0/1303787173 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc63801db80 con 0x55bb8e73bf60 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.159+0000 7fc650d87b80 10 client.17085 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:42.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.159+0000 7fc650d87b80 10 client.17085 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.159+0000 7fc650d87b80 20 client.17085 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.159+0000 7fc650d87b80 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55bb8e74f6f0 0x55bb8e751ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.159+0000 7fc650d87b80 4 client.17085 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.159+0000 7fc650d87b80 1 -- 172.21.15.104:0/1303787173 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc638004bb0 con 0x55bb8e74f6f0 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.159+0000 7fc63ffff700 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55bb8e74f6f0 0x55bb8e751ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.159+0000 7fc63ffff700 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55bb8e74f6f0 0x55bb8e751ae0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32df7e4292373e34 server_cookie=efb49838b41cb1d8 in_seq=0 out_seq=0 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.159+0000 7fc61f7fe700 10 client.17085 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.160+0000 7fc61f7fe700 1 -- 172.21.15.104:0/1303787173 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x7fc638004bb0 con 0x55bb8e74f6f0 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.160+0000 7fc61f7fe700 10 client.17085 handle_command_reply: tid=0 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.160+0000 7fc650d87b80 1 client.17085 shutdown 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.160+0000 7fc650d87b80 1 -- 172.21.15.104:0/1303787173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73bf60 msgr2=0x55bb8e73f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.160+0000 7fc650d87b80 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73bf60 0x55bb8e73f4f0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc638014d70 tx=0x7fc638014720 comp rx=0 tx=0).stop 2024-06-09T22:26:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.160+0000 7fc650d87b80 1 -- 172.21.15.104:0/1303787173 shutdown_connections 2024-06-09T22:26:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.160+0000 7fc650d87b80 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55bb8e74f6f0 0x55bb8e751ae0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.160+0000 7fc650d87b80 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73bf60 0x55bb8e73f4f0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.160+0000 7fc650d87b80 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e735f20 0x55bb8e741cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.160+0000 7fc650d87b80 1 --2- 172.21.15.104:0/1303787173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 0x55bb8e73efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.160+0000 7fc61f7fe700 0 client.17085 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.160+0000 7fc650d87b80 1 -- 172.21.15.104:0/1303787173 >> 172.21.15.104:0/1303787173 conn(0x55bb8e660cc0 msgr2=0x55bb8e7349d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.161+0000 7fc650d87b80 1 -- 172.21.15.104:0/1303787173 shutdown_connections 2024-06-09T22:26:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.161+0000 7fc650d87b80 1 -- 172.21.15.104:0/1303787173 wait complete. 2024-06-09T22:26:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.161+0000 7fc650d87b80 20 client.17085 trim_cache size 0 max 16384 2024-06-09T22:26:42.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.196+0000 7fc650d87b80 1 Processor -- start 2024-06-09T22:26:42.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.196+0000 7fc650d87b80 1 -- start start 2024-06-09T22:26:42.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.196+0000 7fc650d87b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e735f20 0x55bb8e7359e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.196+0000 7fc650d87b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73b640 0x55bb8e7309f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.196+0000 7fc650d87b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73bf60 0x55bb8e73ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.196+0000 7fc650d87b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401ac2e0 con 0x55bb8e73b640 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.196+0000 7fc650d87b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401ade20 con 0x55bb8e735f20 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.196+0000 7fc650d87b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401ac9b0 con 0x55bb8e73bf60 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.196+0000 7fc637fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73b640 0x55bb8e7309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.196+0000 7fc637fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73b640 0x55bb8e7309f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58490/0 (socket says 172.21.15.104:58490) 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.196+0000 7fc637fff700 1 -- 172.21.15.104:0/1324477683 learned_addr learned my addr 172.21.15.104:0/1324477683 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.196+0000 7fc63ffff700 1 --2- 172.21.15.104:0/1324477683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e735f20 0x55bb8e7359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.197+0000 7fc644aa1700 1 --2- 172.21.15.104:0/1324477683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73bf60 0x55bb8e73ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.197+0000 7fc63ffff700 1 -- 172.21.15.104:0/1324477683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73bf60 msgr2=0x55bb8e73ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.197+0000 7fc63ffff700 1 --2- 172.21.15.104:0/1324477683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73bf60 0x55bb8e73ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.197+0000 7fc63ffff700 1 -- 172.21.15.104:0/1324477683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73b640 msgr2=0x55bb8e7309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:42.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.197+0000 7fc63ffff700 1 --2- 172.21.15.104:0/1324477683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73b640 0x55bb8e7309f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.197+0000 7fc63ffff700 1 -- 172.21.15.104:0/1324477683 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb8e73cf90 con 0x55bb8e735f20 2024-06-09T22:26:42.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.197+0000 7fc63ffff700 1 --2- 172.21.15.104:0/1324477683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e735f20 0x55bb8e7359e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc6300ccb30 tx=0x7fc6300b1040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32d8bcc79b415f9f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:42.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.197+0000 7fc635ffb700 1 -- 172.21.15.104:0/1324477683 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6300b8070 con 0x55bb8e735f20 2024-06-09T22:26:42.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.197+0000 7fc635ffb700 1 -- 172.21.15.104:0/1324477683 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6300ba300 con 0x55bb8e735f20 2024-06-09T22:26:42.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.197+0000 7fc635ffb700 1 -- 172.21.15.104:0/1324477683 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6300cf360 con 0x55bb8e735f20 2024-06-09T22:26:42.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.198+0000 7fc650d87b80 1 -- 172.21.15.104:0/1324477683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e735f20 msgr2=0x55bb8e7359e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:42.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.198+0000 7fc650d87b80 1 --2- 172.21.15.104:0/1324477683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e735f20 0x55bb8e7359e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc6300ccb30 tx=0x7fc6300b1040 comp rx=0 tx=0).stop 2024-06-09T22:26:42.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.198+0000 7fc650d87b80 1 -- 172.21.15.104:0/1324477683 shutdown_connections 2024-06-09T22:26:42.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.198+0000 7fc650d87b80 1 --2- 172.21.15.104:0/1324477683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73bf60 0x55bb8e73ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.198+0000 7fc650d87b80 1 --2- 172.21.15.104:0/1324477683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e735f20 0x55bb8e7359e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.198+0000 7fc650d87b80 1 --2- 172.21.15.104:0/1324477683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e73b640 0x55bb8e7309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.198+0000 7fc650d87b80 1 -- 172.21.15.104:0/1324477683 >> 172.21.15.104:0/1324477683 conn(0x55bb8e660cc0 msgr2=0x55bb8e73c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.198+0000 7fc650d87b80 1 -- 172.21.15.104:0/1324477683 shutdown_connections 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.198+0000 7fc650d87b80 1 -- 172.21.15.104:0/1324477683 wait complete. 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.198+0000 7fc650d87b80 1 -- start start 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc650d87b80 1 Processor -- start 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc650d87b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 0x55bb8e748470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc650d87b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 0x55bb8e7568b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc650d87b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e73bf60 0x55bb8e756df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc650d87b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401ad750 con 0x55bb8e735f20 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc63ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 0x55bb8e748470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc63ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 0x55bb8e748470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58498/0 (socket says 172.21.15.104:58498) 2024-06-09T22:26:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc644aa1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e73bf60 0x55bb8e756df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc63ffff700 1 -- 172.21.15.104:0/3409514903 learned_addr learned my addr 172.21.15.104:0/3409514903 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:42.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc650d87b80 1 -- 172.21.15.104:0/3409514903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401ad080 con 0x55bb8e73bf60 2024-06-09T22:26:42.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc650d87b80 1 -- 172.21.15.104:0/3409514903 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401aa260 con 0x55bb8e73b640 2024-06-09T22:26:42.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc650d87b80 10 client.0 _ll_register_callbacks cb 0x55bb8e621920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:42.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc637fff700 1 --2- 172.21.15.104:0/3409514903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 0x55bb8e7568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc63ffff700 1 -- 172.21.15.104:0/3409514903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 msgr2=0x55bb8e7568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:42.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc63ffff700 1 --2- 172.21.15.104:0/3409514903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 0x55bb8e7568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc63ffff700 1 -- 172.21.15.104:0/3409514903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e73bf60 msgr2=0x55bb8e756df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:42.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc63ffff700 1 --2- 172.21.15.104:0/3409514903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e73bf60 0x55bb8e756df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.199+0000 7fc63ffff700 1 -- 172.21.15.104:0/3409514903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb8e7574a0 con 0x55bb8e735f20 2024-06-09T22:26:42.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc63ffff700 1 --2- 172.21.15.104:0/3409514903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 0x55bb8e748470 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc63009b890 tx=0x7fc63009b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2feb5bffca1d4c74 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:42.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc61f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:42.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc61f7fe700 1 -- 172.21.15.104:0/3409514903 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6300b8070 con 0x55bb8e735f20 2024-06-09T22:26:42.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc61f7fe700 1 -- 172.21.15.104:0/3409514903 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fc6300cf360 con 0x55bb8e735f20 2024-06-09T22:26:42.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc650d87b80 1 -- 172.21.15.104:0/3409514903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bb8e694d10 con 0x55bb8e735f20 2024-06-09T22:26:42.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc61f7fe700 1 -- 172.21.15.104:0/3409514903 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc6300cf560 con 0x55bb8e735f20 2024-06-09T22:26:42.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc650d87b80 10 client.17091 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:42.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc650d87b80 1 -- 172.21.15.104:0/3409514903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bb8e7b9f10 con 0x55bb8e735f20 2024-06-09T22:26:42.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc61f7fe700 1 -- 172.21.15.104:0/3409514903 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bb8e694d10 con 0x55bb8e735f20 2024-06-09T22:26:42.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc61f7fe700 1 -- 172.21.15.104:0/3409514903 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc6300cf040 con 0x55bb8e735f20 2024-06-09T22:26:42.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc650d87b80 10 client.17091 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:42.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc650d87b80 10 client.17091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:42.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc650d87b80 20 client.17091 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:42.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc650d87b80 1 --2- 172.21.15.104:0/3409514903 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55bb8e7ba330 0x55bb8e7bc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:42.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc650d87b80 4 client.17091 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:26:42.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.200+0000 7fc650d87b80 1 -- 172.21.15.104:0/3409514903 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bb8e68f290 con 0x55bb8e7ba330 2024-06-09T22:26:42.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.201+0000 7fc637fff700 1 --2- 172.21.15.104:0/3409514903 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55bb8e7ba330 0x55bb8e7bc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:42.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.201+0000 7fc637fff700 1 --2- 172.21.15.104:0/3409514903 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55bb8e7ba330 0x55bb8e7bc720 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6ffcdbfec3e161d server_cookie=4fd01e9e11d50c23 in_seq=0 out_seq=0 2024-06-09T22:26:42.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.201+0000 7fc61f7fe700 10 client.17091 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:42.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.201+0000 7fc61f7fe700 1 -- 172.21.15.104:0/3409514903 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bb8e68f290 con 0x55bb8e7ba330 2024-06-09T22:26:42.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.201+0000 7fc61f7fe700 10 client.17091 handle_command_reply: tid=0 2024-06-09T22:26:42.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.201+0000 7fc650d87b80 1 client.17091 shutdown 2024-06-09T22:26:42.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.201+0000 7fc650d87b80 1 -- 172.21.15.104:0/3409514903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 msgr2=0x55bb8e748470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:42.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.201+0000 7fc650d87b80 1 --2- 172.21.15.104:0/3409514903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 0x55bb8e748470 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc63009b890 tx=0x7fc63009b2e0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.201+0000 7fc650d87b80 1 -- 172.21.15.104:0/3409514903 shutdown_connections 2024-06-09T22:26:42.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.201+0000 7fc650d87b80 1 --2- 172.21.15.104:0/3409514903 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55bb8e7ba330 0x55bb8e7bc720 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.202+0000 7fc650d87b80 1 --2- 172.21.15.104:0/3409514903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bb8e735f20 0x55bb8e748470 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.202+0000 7fc61f7fe700 0 client.17091 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:42.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.202+0000 7fc650d87b80 1 --2- 172.21.15.104:0/3409514903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb8e73bf60 0x55bb8e756df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.202+0000 7fc650d87b80 1 --2- 172.21.15.104:0/3409514903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb8e73b640 0x55bb8e7568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.202+0000 7fc650d87b80 1 -- 172.21.15.104:0/3409514903 >> 172.21.15.104:0/3409514903 conn(0x55bb8e660cc0 msgr2=0x55bb8e729e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:42.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.202+0000 7fc650d87b80 1 -- 172.21.15.104:0/3409514903 shutdown_connections 2024-06-09T22:26:42.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.202+0000 7fc650d87b80 1 -- 172.21.15.104:0/3409514903 wait complete. 2024-06-09T22:26:42.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.202+0000 7fc650d87b80 20 client.17091 trim_cache size 0 max 16384 2024-06-09T22:26:42.211 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:26:42.211 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-09T22:26:42.211 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": { 2024-06-09T22:26:42.211 INFO:teuthology.orchestra.run.smithi104.stdout: "ffc707ed-535e-42bc-b9f5-912730ee01d7": { 2024-06-09T22:26:42.211 INFO:teuthology.orchestra.run.smithi104.stdout: "path": "/", 2024-06-09T22:26:42.211 INFO:teuthology.orchestra.run.smithi104.stdout: "tag": "ffc707ed-535e-42bc-b9f5-912730ee01d7", 2024-06-09T22:26:42.211 INFO:teuthology.orchestra.run.smithi104.stdout: "options": "recursive,force" 2024-06-09T22:26:42.211 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:26:42.211 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:26:42.211 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:26:42.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.202+0000 7fc646504700 1 -- 172.21.15.104:0/3431150676 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc62c0a2d90 msgr2=0x7fc62c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:42.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.202+0000 7fc646504700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc62c0a2d90 0x7fc62c0a5250 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc628006010 tx=0x7fc62800b540 comp rx=0 tx=0).stop 2024-06-09T22:26:42.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.202+0000 7fc646504700 1 -- 172.21.15.104:0/3431150676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc640101380 msgr2=0x7fc64019e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:42.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.202+0000 7fc646504700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc640101380 0x7fc64019e770 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c9b0 tx=0x7fc63000ccc0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.203+0000 7fc646504700 1 -- 172.21.15.104:0/3431150676 shutdown_connections 2024-06-09T22:26:42.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.203+0000 7fc646504700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc62c0a2d90 0x7fc62c0a5250 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.203+0000 7fc646504700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc640101380 0x7fc64019e770 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.203+0000 7fc646504700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6401073f0 0x7fc640198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.203+0000 7fc646504700 1 --2- 172.21.15.104:0/3431150676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640101d40 0x7fc64019ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:42.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.203+0000 7fc646504700 1 -- 172.21.15.104:0/3431150676 >> 172.21.15.104:0/3431150676 conn(0x7fc640078300 msgr2=0x7fc64006ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:42.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.203+0000 7fc646504700 1 -- 172.21.15.104:0/3431150676 shutdown_connections 2024-06-09T22:26:42.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:42.203+0000 7fc646504700 1 -- 172.21.15.104:0/3431150676 wait complete. 2024-06-09T22:26:42.731 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.20 MB/sec execute 108 sec latency 11028.422 ms 2024-06-09T22:26:43.079 INFO:tasks.cephfs.filesystem:scrub status for tag:ffc707ed-535e-42bc-b9f5-912730ee01d7 - {'path': '/', 'tag': 'ffc707ed-535e-42bc-b9f5-912730ee01d7', 'options': 'recursive,force'} 2024-06-09T22:26:43.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:43 smithi194 ceph-mon[36856]: pgmap v175: 129 pgs: 129 active+clean; 130 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-06-09T22:26:43.400 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:43 smithi104 ceph-mon[32389]: pgmap v175: 129 pgs: 129 active+clean; 130 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-06-09T22:26:43.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:43 smithi153 ceph-mon[36599]: pgmap v175: 129 pgs: 129 active+clean; 130 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-06-09T22:26:43.731 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.15 MB/sec execute 109 sec latency 12028.722 ms 2024-06-09T22:26:44.731 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.10 MB/sec execute 110 sec latency 13028.803 ms 2024-06-09T22:26:45.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:45 smithi194 ceph-mon[36856]: pgmap v176: 129 pgs: 129 active+clean; 130 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.4 MiB/s wr, 142 op/s 2024-06-09T22:26:45.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:45 smithi153 ceph-mon[36599]: pgmap v176: 129 pgs: 129 active+clean; 130 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.4 MiB/s wr, 142 op/s 2024-06-09T22:26:45.508 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:45 smithi104 ceph-mon[32389]: pgmap v176: 129 pgs: 129 active+clean; 130 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.4 MiB/s wr, 142 op/s 2024-06-09T22:26:45.731 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.06 MB/sec execute 111 sec latency 14028.875 ms 2024-06-09T22:26:46.731 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 5.01 MB/sec execute 112 sec latency 15028.923 ms 2024-06-09T22:26:47.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:47 smithi194 ceph-mon[36856]: pgmap v177: 129 pgs: 129 active+clean; 130 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-06-09T22:26:47.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:47 smithi153 ceph-mon[36599]: pgmap v177: 129 pgs: 129 active+clean; 130 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-06-09T22:26:47.508 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:47 smithi104 ceph-mon[32389]: pgmap v177: 129 pgs: 129 active+clean; 130 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-06-09T22:26:47.731 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 4.97 MB/sec execute 113 sec latency 16028.985 ms 2024-06-09T22:26:48.731 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 4.92 MB/sec execute 114 sec latency 17029.051 ms 2024-06-09T22:26:49.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:49 smithi194 ceph-mon[36856]: pgmap v178: 129 pgs: 129 active+clean; 130 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-06-09T22:26:49.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:49 smithi153 ceph-mon[36599]: pgmap v178: 129 pgs: 129 active+clean; 130 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-06-09T22:26:49.508 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:49 smithi104 ceph-mon[32389]: pgmap v178: 129 pgs: 129 active+clean; 130 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-06-09T22:26:49.732 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 4.88 MB/sec execute 115 sec latency 18029.141 ms 2024-06-09T22:26:50.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:50 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:26:50.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:50 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:26:50.508 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:50 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:26:50.732 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 4.84 MB/sec execute 116 sec latency 19029.209 ms 2024-06-09T22:26:51.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:51 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:26:51.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:51 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:26:51.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:51 smithi194 ceph-mon[36856]: pgmap v179: 129 pgs: 129 active+clean; 128 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-06-09T22:26:51.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:51 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:26:51.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:51 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:26:51.494 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:51 smithi153 ceph-mon[36599]: pgmap v179: 129 pgs: 129 active+clean; 128 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-06-09T22:26:51.508 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:51 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:26:51.508 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:51 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:26:51.508 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:51 smithi104 ceph-mon[32389]: pgmap v179: 129 pgs: 129 active+clean; 128 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-06-09T22:26:51.732 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 4.80 MB/sec execute 117 sec latency 20029.312 ms 2024-06-09T22:26:52.732 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 4.76 MB/sec execute 118 sec latency 21029.429 ms 2024-06-09T22:26:53.022 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:26:53.275 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:53.724 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:53 smithi104 ceph-mon[32389]: pgmap v180: 129 pgs: 129 active+clean; 128 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-06-09T22:26:53.732 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 4.72 MB/sec execute 119 sec latency 22029.497 ms 2024-06-09T22:26:53.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:53 smithi194 ceph-mon[36856]: pgmap v180: 129 pgs: 129 active+clean; 128 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-06-09T22:26:53.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:53 smithi153 ceph-mon[36599]: pgmap v180: 129 pgs: 129 active+clean; 128 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-06-09T22:26:54.732 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 4.68 MB/sec execute 120 sec latency 23029.534 ms 2024-06-09T22:26:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.968+0000 7f37853e7640 1 -- 172.21.15.104:0/2768444387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3780102370 msgr2=0x7f378010e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.968+0000 7f37853e7640 1 --2- 172.21.15.104:0/2768444387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3780102370 0x7f378010e880 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f3778009cd0 tx=0x7f377802a8e0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.968+0000 7f37853e7640 1 -- 172.21.15.104:0/2768444387 shutdown_connections 2024-06-09T22:26:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.968+0000 7f37853e7640 1 --2- 172.21.15.104:0/2768444387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3780102370 0x7f378010e880 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.968+0000 7f37853e7640 1 --2- 172.21.15.104:0/2768444387 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37801019b0 0x7f3780101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.968+0000 7f37853e7640 1 --2- 172.21.15.104:0/2768444387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37801079c0 0x7f3780107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.968+0000 7f37853e7640 1 -- 172.21.15.104:0/2768444387 >> 172.21.15.104:0/2768444387 conn(0x7f3780078e30 msgr2=0x7f3780079240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.968+0000 7f37853e7640 1 -- 172.21.15.104:0/2768444387 shutdown_connections 2024-06-09T22:26:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.968+0000 7f37853e7640 1 -- 172.21.15.104:0/2768444387 wait complete. 2024-06-09T22:26:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f37853e7640 1 Processor -- start 2024-06-09T22:26:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f37853e7640 1 -- start start 2024-06-09T22:26:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f37853e7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37801019b0 0x7f37801981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f37853e7640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3780102370 0x7f3780198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f37853e7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37801079c0 0x7f378019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f37853e7640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3780111da0 con 0x7f37801019b0 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f37853e7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3780111c20 con 0x7f37801079c0 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f37853e7640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3780111f20 con 0x7f3780102370 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f377effd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37801019b0 0x7f37801981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f377effd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37801019b0 0x7f37801981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56788/0 (socket says 172.21.15.104:56788) 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f377effd640 1 -- 172.21.15.104:0/3616548059 learned_addr learned my addr 172.21.15.104:0/3616548059 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f377f7fe640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37801079c0 0x7f378019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f3777fff640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3780102370 0x7f3780198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f377effd640 1 -- 172.21.15.104:0/3616548059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3780102370 msgr2=0x7f3780198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f377effd640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3780102370 0x7f3780198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.973 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f377effd640 1 -- 172.21.15.104:0/3616548059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37801079c0 msgr2=0x7f378019ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:54.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f377effd640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37801079c0 0x7f378019ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f377effd640 1 -- 172.21.15.104:0/3616548059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378019d160 con 0x7f37801019b0 2024-06-09T22:26:54.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.969+0000 7f377effd640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37801019b0 0x7f37801981c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f377000d950 tx=0x7f377000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:54.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.970+0000 7f377cff9640 1 -- 172.21.15.104:0/3616548059 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770017030 con 0x7f37801019b0 2024-06-09T22:26:54.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.970+0000 7f377cff9640 1 -- 172.21.15.104:0/3616548059 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3770004020 con 0x7f37801019b0 2024-06-09T22:26:54.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.970+0000 7f37853e7640 1 -- 172.21.15.104:0/3616548059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f378019d450 con 0x7f37801019b0 2024-06-09T22:26:54.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.970+0000 7f377cff9640 1 -- 172.21.15.104:0/3616548059 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37700047a0 con 0x7f37801019b0 2024-06-09T22:26:54.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.970+0000 7f37853e7640 1 -- 172.21.15.104:0/3616548059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37801a4ce0 con 0x7f37801019b0 2024-06-09T22:26:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.972+0000 7f377cff9640 1 -- 172.21.15.104:0/3616548059 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f37700041d0 con 0x7f37801019b0 2024-06-09T22:26:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.972+0000 7f37881d4740 1 Processor -- start 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.972+0000 7f37881d4740 1 -- start start 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f37881d4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466170 0x55a06d466550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377cff9640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f376c0a2e20 0x7f376c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f37881d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466a90 0x55a06d470e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f37881d4740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d473740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f37881d4740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a06d3ca4f0 con 0x55a06d471350 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f37881d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a06d368870 con 0x55a06d466a90 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f37881d4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a06d3c64b0 con 0x55a06d466170 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377effd640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f376c0a2e20 0x7f376c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d473740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d473740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56796/0 (socket says 172.21.15.104:56796) 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377f7fe640 1 -- 172.21.15.104:0/141013174 learned_addr learned my addr 172.21.15.104:0/141013174 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f3777fff640 1 --2- 172.21.15.104:0/141013174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466a90 0x55a06d470e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377cff9640 1 -- 172.21.15.104:0/3616548059 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f37700cf090 con 0x7f37801019b0 2024-06-09T22:26:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377f7fe640 1 --2- 172.21.15.104:0/141013174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466170 0x55a06d466550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377f7fe640 1 -- 172.21.15.104:0/141013174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466170 msgr2=0x55a06d466550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377f7fe640 1 --2- 172.21.15.104:0/141013174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466170 0x55a06d466550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377f7fe640 1 -- 172.21.15.104:0/141013174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466a90 msgr2=0x55a06d470e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377f7fe640 1 --2- 172.21.15.104:0/141013174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466a90 0x55a06d470e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377f7fe640 1 -- 172.21.15.104:0/141013174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a06d3849b0 con 0x55a06d471350 2024-06-09T22:26:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377effd640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f376c0a2e20 0x7f376c0a52e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f377009b1d0 tx=0x7f3770021000 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f377f7fe640 1 --2- 172.21.15.104:0/141013174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d473740 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f3778004ab0 tx=0x7f37780093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1a6143f366aca8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.973+0000 7f37757fa640 1 -- 172.21.15.104:0/141013174 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778031690 con 0x55a06d471350 2024-06-09T22:26:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.974+0000 7f37757fa640 1 -- 172.21.15.104:0/141013174 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3778036030 con 0x55a06d471350 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.974+0000 7f37757fa640 1 -- 172.21.15.104:0/141013174 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778004390 con 0x55a06d471350 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.974+0000 7f37881d4740 1 -- 172.21.15.104:0/141013174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 msgr2=0x55a06d473740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.974+0000 7f37881d4740 1 --2- 172.21.15.104:0/141013174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d473740 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f3778004ab0 tx=0x7f37780093a0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.974+0000 7f37881d4740 1 -- 172.21.15.104:0/141013174 shutdown_connections 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.974+0000 7f37881d4740 1 --2- 172.21.15.104:0/141013174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d473740 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.974+0000 7f37881d4740 1 --2- 172.21.15.104:0/141013174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466a90 0x55a06d470e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.974+0000 7f37881d4740 1 --2- 172.21.15.104:0/141013174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466170 0x55a06d466550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.974+0000 7f37881d4740 1 -- 172.21.15.104:0/141013174 >> 172.21.15.104:0/141013174 conn(0x55a06d462680 msgr2=0x55a06d462a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.974+0000 7f37881d4740 1 -- 172.21.15.104:0/141013174 shutdown_connections 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.974+0000 7f37881d4740 1 -- 172.21.15.104:0/141013174 wait complete. 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f37881d4740 1 -- start start 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f37881d4740 1 Processor -- start 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f37881d4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466170 0x55a06d46e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f37881d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466a90 0x55a06d46e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f37881d4740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d46ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f37881d4740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a06d473f20 con 0x55a06d471350 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f37881d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a06d353ae0 con 0x55a06d466a90 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f37881d4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a06d4740a0 con 0x55a06d466170 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f37881d4740 10 client.0 _ll_register_callbacks cb 0x55a06d38c9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f377f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d46ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f377f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d46ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56802/0 (socket says 172.21.15.104:56802) 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f377f7fe640 1 -- 172.21.15.104:0/39532947 learned_addr learned my addr 172.21.15.104:0/39532947 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f3777fff640 1 --2- 172.21.15.104:0/39532947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466a90 0x55a06d46e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f377f7fe640 1 -- 172.21.15.104:0/39532947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466170 msgr2=0x55a06d46e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f377f7fe640 1 --2- 172.21.15.104:0/39532947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466170 0x55a06d46e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f377f7fe640 1 -- 172.21.15.104:0/39532947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466a90 msgr2=0x55a06d46e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f377f7fe640 1 --2- 172.21.15.104:0/39532947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466a90 0x55a06d46e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f377f7fe640 1 -- 172.21.15.104:0/39532947 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a06d0de210 con 0x55a06d471350 2024-06-09T22:26:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f377f7fe640 1 --2- 172.21.15.104:0/39532947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d46ed10 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f3778009a60 tx=0x7f3778005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6679c8d870fc8971 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.975+0000 7f37667fc640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37667fc640 1 -- 172.21.15.104:0/39532947 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778003970 con 0x55a06d471350 2024-06-09T22:26:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37667fc640 1 -- 172.21.15.104:0/39532947 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f3778003b10 con 0x55a06d471350 2024-06-09T22:26:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37881d4740 1 -- 172.21.15.104:0/39532947 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3778036030 con 0x55a06d471350 2024-06-09T22:26:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37667fc640 1 -- 172.21.15.104:0/39532947 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f377805b050 con 0x55a06d471350 2024-06-09T22:26:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37881d4740 10 client.17103 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37881d4740 1 -- 172.21.15.104:0/39532947 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a06d1aa620 con 0x55a06d471350 2024-06-09T22:26:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37667fc640 1 -- 172.21.15.104:0/39532947 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3778036030 con 0x55a06d471350 2024-06-09T22:26:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37667fc640 1 -- 172.21.15.104:0/39532947 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3778030070 con 0x55a06d471350 2024-06-09T22:26:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37881d4740 10 client.17103 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37881d4740 10 client.17103 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37881d4740 20 client.17103 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37881d4740 1 --2- 172.21.15.104:0/39532947 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a06d481320 0x55a06d483710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37881d4740 4 client.17103 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f37881d4740 1 -- 172.21.15.104:0/39532947 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a06d3ddf10 con 0x55a06d481320 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.976+0000 7f3777fff640 1 --2- 172.21.15.104:0/39532947 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a06d481320 0x55a06d483710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.977+0000 7f37667fc640 10 client.17103 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.977+0000 7f3777fff640 1 --2- 172.21.15.104:0/39532947 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a06d481320 0x55a06d483710 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=215df9e4bc0c176 server_cookie=dcac13e457ef2065 in_seq=0 out_seq=0 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37667fc640 1 -- 172.21.15.104:0/39532947 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a06d3ddf10 con 0x55a06d481320 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37667fc640 10 client.17103 handle_command_reply: tid=0 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37881d4740 1 client.17103 shutdown 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37881d4740 1 -- 172.21.15.104:0/39532947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 msgr2=0x55a06d46ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37881d4740 1 --2- 172.21.15.104:0/39532947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d46ed10 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f3778009a60 tx=0x7f3778005ef0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37881d4740 1 -- 172.21.15.104:0/39532947 shutdown_connections 2024-06-09T22:26:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37881d4740 1 --2- 172.21.15.104:0/39532947 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a06d481320 0x55a06d483710 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37881d4740 1 --2- 172.21.15.104:0/39532947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d46ed10 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37881d4740 1 --2- 172.21.15.104:0/39532947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466a90 0x55a06d46e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37667fc640 0 client.17103 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37881d4740 1 --2- 172.21.15.104:0/39532947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466170 0x55a06d46e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37881d4740 1 -- 172.21.15.104:0/39532947 >> 172.21.15.104:0/39532947 conn(0x55a06d462680 msgr2=0x55a06d4683b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37881d4740 1 -- 172.21.15.104:0/39532947 shutdown_connections 2024-06-09T22:26:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37881d4740 1 -- 172.21.15.104:0/39532947 wait complete. 2024-06-09T22:26:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.978+0000 7f37881d4740 20 client.17103 trim_cache size 0 max 16384 2024-06-09T22:26:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.994+0000 7f37881d4740 1 Processor -- start 2024-06-09T22:26:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.994+0000 7f37881d4740 1 -- start start 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.994+0000 7f37881d4740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d466170 0x55a06d468b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.994+0000 7f37881d4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466a90 0x55a06d486730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.994+0000 7f37881d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d471350 0x55a06d486c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.994+0000 7f37881d4740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f37801aecc0 con 0x55a06d466170 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.994+0000 7f37881d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f37801b0800 con 0x55a06d471350 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.994+0000 7f37881d4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f37801af390 con 0x55a06d466a90 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.994+0000 7f377f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d466170 0x55a06d468b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.994+0000 7f377f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d466170 0x55a06d468b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56804/0 (socket says 172.21.15.104:56804) 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.994+0000 7f377f7fe640 1 -- 172.21.15.104:0/3482407244 learned_addr learned my addr 172.21.15.104:0/3482407244 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.995+0000 7f3777fff640 1 --2- 172.21.15.104:0/3482407244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466a90 0x55a06d486730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.995+0000 7f377f7fe640 1 -- 172.21.15.104:0/3482407244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466a90 msgr2=0x55a06d486730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.995+0000 7f377f7fe640 1 --2- 172.21.15.104:0/3482407244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466a90 0x55a06d486730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.995+0000 7f377f7fe640 1 -- 172.21.15.104:0/3482407244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d471350 msgr2=0x55a06d486c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:54.998 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.995+0000 7f377f7fe640 1 --2- 172.21.15.104:0/3482407244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d471350 0x55a06d486c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:54.998 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.995+0000 7f377f7fe640 1 -- 172.21.15.104:0/3482407244 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a06d473f70 con 0x55a06d466170 2024-06-09T22:26:54.998 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.995+0000 7f377f7fe640 1 --2- 172.21.15.104:0/3482407244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d466170 0x55a06d468b30 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3778049500 tx=0x7f3778004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785d2d1b7905e028 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:54.998 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.995+0000 7f3774ff9640 1 -- 172.21.15.104:0/3482407244 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778031430 con 0x55a06d466170 2024-06-09T22:26:54.998 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.995+0000 7f3774ff9640 1 -- 172.21.15.104:0/3482407244 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3778036040 con 0x55a06d466170 2024-06-09T22:26:54.998 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.995+0000 7f3774ff9640 1 -- 172.21.15.104:0/3482407244 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778031b90 con 0x55a06d466170 2024-06-09T22:26:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 -- 172.21.15.104:0/3482407244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d466170 msgr2=0x55a06d468b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 --2- 172.21.15.104:0/3482407244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d466170 0x55a06d468b30 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3778049500 tx=0x7f3778004990 comp rx=0 tx=0).stop 2024-06-09T22:26:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 -- 172.21.15.104:0/3482407244 shutdown_connections 2024-06-09T22:26:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 --2- 172.21.15.104:0/3482407244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d471350 0x55a06d486c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 --2- 172.21.15.104:0/3482407244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466a90 0x55a06d486730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 --2- 172.21.15.104:0/3482407244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d466170 0x55a06d468b30 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 -- 172.21.15.104:0/3482407244 >> 172.21.15.104:0/3482407244 conn(0x55a06d462680 msgr2=0x55a06d4617e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 -- 172.21.15.104:0/3482407244 shutdown_connections 2024-06-09T22:26:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 -- 172.21.15.104:0/3482407244 wait complete. 2024-06-09T22:26:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 -- start start 2024-06-09T22:26:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 Processor -- start 2024-06-09T22:26:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466170 0x55a06d486b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466a90 0x55a06d487040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d484090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f37801b0130 con 0x55a06d471350 2024-06-09T22:26:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.996+0000 7f37881d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f37801afa60 con 0x55a06d466170 2024-06-09T22:26:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f37881d4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f37801acc80 con 0x55a06d466a90 2024-06-09T22:26:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f37881d4740 10 client.0 _ll_register_callbacks cb 0x7f37680092a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f377f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d484090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f377f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d484090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56806/0 (socket says 172.21.15.104:56806) 2024-06-09T22:26:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f377f7fe640 1 -- 172.21.15.104:0/3424197901 learned_addr learned my addr 172.21.15.104:0/3424197901 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f3777fff640 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466a90 0x55a06d487040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f377f7fe640 1 -- 172.21.15.104:0/3424197901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466a90 msgr2=0x55a06d487040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f377f7fe640 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466a90 0x55a06d487040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f377f7fe640 1 -- 172.21.15.104:0/3424197901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466170 msgr2=0x55a06d486b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f377f7fe640 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466170 0x55a06d486b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f377f7fe640 1 -- 172.21.15.104:0/3424197901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a06d484860 con 0x55a06d471350 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f3777fff640 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466a90 0x55a06d487040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f377f7fe640 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d484090 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3778049590 tx=0x7f3778005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52459f64e8b21d2f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f37667fc640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f37667fc640 1 -- 172.21.15.104:0/3424197901 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778031930 con 0x55a06d471350 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f37667fc640 1 -- 172.21.15.104:0/3424197901 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f3778031ad0 con 0x55a06d471350 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f37881d4740 1 -- 172.21.15.104:0/3424197901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3778036040 con 0x55a06d471350 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f37667fc640 1 -- 172.21.15.104:0/3424197901 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3778031cd0 con 0x55a06d471350 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f37881d4740 10 client.17109 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.997+0000 7f37881d4740 1 -- 172.21.15.104:0/3424197901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a06d4eb8d0 con 0x55a06d471350 2024-06-09T22:26:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.998+0000 7f37667fc640 1 -- 172.21.15.104:0/3424197901 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3778036040 con 0x55a06d471350 2024-06-09T22:26:55.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.998+0000 7f37667fc640 1 -- 172.21.15.104:0/3424197901 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3778049620 con 0x55a06d471350 2024-06-09T22:26:55.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.998+0000 7f37881d4740 10 client.17109 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:55.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.998+0000 7f37881d4740 10 client.17109 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:26:55.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.998+0000 7f37881d4740 20 client.17109 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:55.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.998+0000 7f37881d4740 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a06d4ebda0 0x55a06d4ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:55.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.998+0000 7f37881d4740 4 client.17109 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:26:55.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.998+0000 7f37881d4740 1 -- 172.21.15.104:0/3424197901 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a06d3d7c20 con 0x55a06d4ebda0 2024-06-09T22:26:55.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.998+0000 7f3777fff640 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a06d4ebda0 0x55a06d4ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:55.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.999+0000 7f37667fc640 10 client.17109 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:55.007 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:54.999+0000 7f3777fff640 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a06d4ebda0 0x55a06d4ee1c0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95af92511ca2237f server_cookie=5de5f9dcd357b7cc in_seq=0 out_seq=0 2024-06-09T22:26:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.008+0000 7f37667fc640 1 -- 172.21.15.104:0/3424197901 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a06d3d7c20 con 0x55a06d4ebda0 2024-06-09T22:26:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.008+0000 7f37667fc640 10 client.17109 handle_command_reply: tid=0 2024-06-09T22:26:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.008+0000 7f37881d4740 1 client.17109 shutdown 2024-06-09T22:26:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.008+0000 7f37881d4740 1 -- 172.21.15.104:0/3424197901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 msgr2=0x55a06d484090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:55.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.008+0000 7f37881d4740 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d484090 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3778049590 tx=0x7f3778005cf0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37881d4740 1 -- 172.21.15.104:0/3424197901 shutdown_connections 2024-06-09T22:26:55.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37881d4740 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a06d4ebda0 0x55a06d4ee1c0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37881d4740 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a06d471350 0x55a06d484090 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37881d4740 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a06d466a90 0x55a06d487040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37667fc640 0 client.17109 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:55.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37881d4740 1 --2- 172.21.15.104:0/3424197901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a06d466170 0x55a06d486b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37881d4740 1 -- 172.21.15.104:0/3424197901 >> 172.21.15.104:0/3424197901 conn(0x55a06d462680 msgr2=0x55a06d460f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:55.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37881d4740 1 -- 172.21.15.104:0/3424197901 shutdown_connections 2024-06-09T22:26:55.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37881d4740 1 -- 172.21.15.104:0/3424197901 wait complete. 2024-06-09T22:26:55.011 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37881d4740 20 client.17109 trim_cache size 0 max 16384 2024-06-09T22:26:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:26:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:26:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:26:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37853e7640 1 -- 172.21.15.104:0/3616548059 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f376c0a2e20 msgr2=0x7f376c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37853e7640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f376c0a2e20 0x7f376c0a52e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f377009b1d0 tx=0x7f3770021000 comp rx=0 tx=0).stop 2024-06-09T22:26:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37853e7640 1 -- 172.21.15.104:0/3616548059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37801019b0 msgr2=0x7f37801981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37853e7640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37801019b0 0x7f37801981c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f377000d950 tx=0x7f377000de10 comp rx=0 tx=0).stop 2024-06-09T22:26:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37853e7640 1 -- 172.21.15.104:0/3616548059 shutdown_connections 2024-06-09T22:26:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37853e7640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f376c0a2e20 0x7f376c0a52e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37853e7640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37801079c0 0x7f378019ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37853e7640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3780102370 0x7f3780198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37853e7640 1 --2- 172.21.15.104:0/3616548059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37801019b0 0x7f37801981c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37853e7640 1 -- 172.21.15.104:0/3616548059 >> 172.21.15.104:0/3616548059 conn(0x7f3780078e30 msgr2=0x7f37800ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.009+0000 7f37853e7640 1 -- 172.21.15.104:0/3616548059 shutdown_connections 2024-06-09T22:26:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:26:55.010+0000 7f37853e7640 1 -- 172.21.15.104:0/3616548059 wait complete. 2024-06-09T22:26:55.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.087+0000 7fa9ed497700 1 -- 172.21.15.104:0/3856059077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806abc0 msgr2=0x7fa9e806b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.087+0000 7fa9ed497700 1 --2- 172.21.15.104:0/3856059077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806abc0 0x7fa9e806b040 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009990 tx=0x7fa9d8009ca0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.087+0000 7fa9ed497700 1 -- 172.21.15.104:0/3856059077 shutdown_connections 2024-06-09T22:26:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.087+0000 7fa9ed497700 1 --2- 172.21.15.104:0/3856059077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9e806b580 0x7fa9e810f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.087+0000 7fa9ed497700 1 --2- 172.21.15.104:0/3856059077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e8108500 0x7fa9e81088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.087+0000 7fa9ed497700 1 --2- 172.21.15.104:0/3856059077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806abc0 0x7fa9e806b040 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.087+0000 7fa9ed497700 1 -- 172.21.15.104:0/3856059077 >> 172.21.15.104:0/3856059077 conn(0x7fa9e8078340 msgr2=0x7fa9e8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.087+0000 7fa9ed497700 1 -- 172.21.15.104:0/3856059077 shutdown_connections 2024-06-09T22:26:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.087+0000 7fa9ed497700 1 -- 172.21.15.104:0/3856059077 wait complete. 2024-06-09T22:26:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.087+0000 7fa9ed497700 1 Processor -- start 2024-06-09T22:26:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9ed497700 1 -- start start 2024-06-09T22:26:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9ed497700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806abc0 0x7fa9e819e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9ed497700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9e806b580 0x7fa9e819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9ed497700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e8108500 0x7fa9e8198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9ed497700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81127e0 con 0x7fa9e806abc0 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9ed497700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8112660 con 0x7fa9e8108500 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9ed497700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8112960 con 0x7fa9e806b580 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9e6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806abc0 0x7fa9e819e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9e6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806abc0 0x7fa9e819e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56830/0 (socket says 172.21.15.104:56830) 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9e6ffd700 1 -- 172.21.15.104:0/2651766211 learned_addr learned my addr 172.21.15.104:0/2651766211 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9e67fc700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9e806b580 0x7fa9e819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9e6ffd700 1 -- 172.21.15.104:0/2651766211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9e806b580 msgr2=0x7fa9e819ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9e6ffd700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9e806b580 0x7fa9e819ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9e6ffd700 1 -- 172.21.15.104:0/2651766211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e8108500 msgr2=0x7fa9e8198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9e77fe700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e8108500 0x7fa9e8198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9e6ffd700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e8108500 0x7fa9e8198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9e6ffd700 1 -- 172.21.15.104:0/2651766211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e8199240 con 0x7fa9e806abc0 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9e77fe700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e8108500 0x7fa9e8198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9e6ffd700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806abc0 0x7fa9e819e8d0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000c920 tx=0x7fa9d000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.088+0000 7fa9dffff700 1 -- 172.21.15.104:0/2651766211 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0007b00 con 0x7fa9e806abc0 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.089+0000 7fa9ed497700 1 -- 172.21.15.104:0/2651766211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e8199530 con 0x7fa9e806abc0 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.089+0000 7fa9dffff700 1 -- 172.21.15.104:0/2651766211 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d0004d10 con 0x7fa9e806abc0 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.089+0000 7fa9dffff700 1 -- 172.21.15.104:0/2651766211 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d00056e0 con 0x7fa9e806abc0 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.089+0000 7fa9ed497700 1 -- 172.21.15.104:0/2651766211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e81a57a0 con 0x7fa9e806abc0 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.090+0000 7fa9ed497700 1 -- 172.21.15.104:0/2651766211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e8069180 con 0x7fa9e806abc0 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.091+0000 7fa9dffff700 1 -- 172.21.15.104:0/2651766211 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa9d0007ca0 con 0x7fa9e806abc0 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.092+0000 7fa9dffff700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa9d40a2d90 0x7fa9d40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.092+0000 7fa9e67fc700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa9d40a2d90 0x7fa9d40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:55.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.092+0000 7fa9dffff700 1 -- 172.21.15.104:0/2651766211 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fa9d00cf430 con 0x7fa9e806abc0 2024-06-09T22:26:55.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.092+0000 7fa9e67fc700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa9d40a2d90 0x7fa9d40a5250 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8005cd0 tx=0x7fa9d8005c40 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:55.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.096+0000 7fa9dffff700 1 -- 172.21.15.104:0/2651766211 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa9d0014030 con 0x7fa9e806abc0 2024-06-09T22:26:55.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.340+0000 7fa9ed497700 1 -- 172.21.15.104:0/2651766211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9e802dca0 con 0x7fa9e806abc0 2024-06-09T22:26:55.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.340+0000 7fa9dffff700 1 -- 172.21.15.104:0/2651766211 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7fa9d009bce0 con 0x7fa9e806abc0 2024-06-09T22:26:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:26:55.342 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:26:55.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.342+0000 7fa9ed497700 1 -- 172.21.15.104:0/2651766211 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa9d40a2d90 msgr2=0x7fa9d40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:55.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.342+0000 7fa9ed497700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa9d40a2d90 0x7fa9d40a5250 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8005cd0 tx=0x7fa9d8005c40 comp rx=0 tx=0).stop 2024-06-09T22:26:55.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.342+0000 7fa9ed497700 1 -- 172.21.15.104:0/2651766211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806abc0 msgr2=0x7fa9e819e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:55.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.342+0000 7fa9ed497700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806abc0 0x7fa9e819e8d0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000c920 tx=0x7fa9d000cce0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.342+0000 7fa9ed497700 1 -- 172.21.15.104:0/2651766211 shutdown_connections 2024-06-09T22:26:55.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.342+0000 7fa9ed497700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa9d40a2d90 0x7fa9d40a5250 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.342+0000 7fa9ed497700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806abc0 0x7fa9e819e8d0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.343+0000 7fa9ed497700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e8108500 0x7fa9e8198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.343+0000 7fa9ed497700 1 --2- 172.21.15.104:0/2651766211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9e806b580 0x7fa9e819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:55.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.343+0000 7fa9ed497700 1 -- 172.21.15.104:0/2651766211 >> 172.21.15.104:0/2651766211 conn(0x7fa9e8078340 msgr2=0x7fa9e81000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:55.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.343+0000 7fa9ed497700 1 -- 172.21.15.104:0/2651766211 shutdown_connections 2024-06-09T22:26:55.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:55.343+0000 7fa9ed497700 1 -- 172.21.15.104:0/2651766211 wait complete. 2024-06-09T22:26:55.345 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:26:55.711 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:55 smithi104 ceph-mon[32389]: pgmap v181: 129 pgs: 129 active+clean; 128 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 2.0 MiB/s wr, 94 op/s 2024-06-09T22:26:55.711 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:55 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2651766211' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:26:55.732 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 4.64 MB/sec execute 121 sec latency 24029.562 ms 2024-06-09T22:26:55.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:55 smithi194 ceph-mon[36856]: pgmap v181: 129 pgs: 129 active+clean; 128 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 2.0 MiB/s wr, 94 op/s 2024-06-09T22:26:55.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:55 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2651766211' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:26:55.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:55 smithi153 ceph-mon[36599]: pgmap v181: 129 pgs: 129 active+clean; 128 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 2.0 MiB/s wr, 94 op/s 2024-06-09T22:26:55.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:55 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2651766211' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:26:56.081 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24460 quiesce db --set-id b7cbcb6e --release --await' 2024-06-09T22:26:56.081 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24460 quiesce db --set-id b7cbcb6e --release --await 2024-06-09T22:26:56.319 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:56.732 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 4.60 MB/sec execute 122 sec latency 25029.602 ms 2024-06-09T22:26:57.732 INFO:tasks.workunit.client.0.smithi104.stdout: 1 176278 4.56 MB/sec execute 123 sec latency 26029.643 ms 2024-06-09T22:26:57.772 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:57 smithi104 ceph-mon[32389]: pgmap v182: 129 pgs: 129 active+clean; 128 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 54 op/s 2024-06-09T22:26:57.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:57 smithi194 ceph-mon[36856]: pgmap v182: 129 pgs: 129 active+clean; 128 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 54 op/s 2024-06-09T22:26:57.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:57 smithi153 ceph-mon[36599]: pgmap v182: 129 pgs: 129 active+clean; 128 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 54 op/s 2024-06-09T22:26:58.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.149+0000 7f36e9e52700 1 -- 172.21.15.104:0/2976645615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e406b5a0 msgr2=0x7f36e410f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:58.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.149+0000 7f36e9e52700 1 --2- 172.21.15.104:0/2976645615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e406b5a0 0x7f36e410f320 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00b0a0 tx=0x7f36dc00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.149+0000 7f36e9e52700 1 -- 172.21.15.104:0/2976645615 shutdown_connections 2024-06-09T22:26:58.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.149+0000 7f36e9e52700 1 --2- 172.21.15.104:0/2976645615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e406b5a0 0x7f36e410f320 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.149+0000 7f36e9e52700 1 --2- 172.21.15.104:0/2976645615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36e4108500 0x7f36e41088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.149+0000 7f36e9e52700 1 --2- 172.21.15.104:0/2976645615 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36e406abe0 0x7f36e406b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.149+0000 7f36e9e52700 1 -- 172.21.15.104:0/2976645615 >> 172.21.15.104:0/2976645615 conn(0x7f36e4078360 msgr2=0x7f36e4078770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:58.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.149+0000 7f36e9e52700 1 -- 172.21.15.104:0/2976645615 shutdown_connections 2024-06-09T22:26:58.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.149+0000 7f36e9e52700 1 -- 172.21.15.104:0/2976645615 wait complete. 2024-06-09T22:26:58.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e9e52700 1 Processor -- start 2024-06-09T22:26:58.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e9e52700 1 -- start start 2024-06-09T22:26:58.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e9e52700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36e406abe0 0x7f36e419e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e9e52700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e406b5a0 0x7f36e419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e9e52700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36e4108500 0x7f36e4198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e9e52700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41127d0 con 0x7f36e406b5a0 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e9e52700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f36e4112650 con 0x7f36e406abe0 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e9e52700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f36e4112950 con 0x7f36e4108500 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e406b5a0 0x7f36e419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e406b5a0 0x7f36e419ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33212/0 (socket says 172.21.15.104:33212) 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e2ffd700 1 -- 172.21.15.104:0/3159586657 learned_addr learned my addr 172.21.15.104:0/3159586657 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e37fe700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36e406abe0 0x7f36e419e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e3fff700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36e4108500 0x7f36e4198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e2ffd700 1 -- 172.21.15.104:0/3159586657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36e4108500 msgr2=0x7f36e4198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.150+0000 7f36e2ffd700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36e4108500 0x7f36e4198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.151+0000 7f36e2ffd700 1 -- 172.21.15.104:0/3159586657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36e406abe0 msgr2=0x7f36e419e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:58.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.151+0000 7f36e2ffd700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36e406abe0 0x7f36e419e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.151+0000 7f36e2ffd700 1 -- 172.21.15.104:0/3159586657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e4199250 con 0x7f36e406b5a0 2024-06-09T22:26:58.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.151+0000 7f36e2ffd700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e406b5a0 0x7f36e419ee20 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f36d800d980 tx=0x7f36d800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:58.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.151+0000 7f36e0ff9700 1 -- 172.21.15.104:0/3159586657 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8017070 con 0x7f36e406b5a0 2024-06-09T22:26:58.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.151+0000 7f36e9e52700 1 -- 172.21.15.104:0/3159586657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e4199540 con 0x7f36e406b5a0 2024-06-09T22:26:58.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.151+0000 7f36e0ff9700 1 -- 172.21.15.104:0/3159586657 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36d8005020 con 0x7f36e406b5a0 2024-06-09T22:26:58.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.151+0000 7f36e0ff9700 1 -- 172.21.15.104:0/3159586657 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8009bc0 con 0x7f36e406b5a0 2024-06-09T22:26:58.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.151+0000 7f36e9e52700 1 -- 172.21.15.104:0/3159586657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e41a57b0 con 0x7f36e406b5a0 2024-06-09T22:26:58.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.153+0000 7f36e0ff9700 1 -- 172.21.15.104:0/3159586657 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f36d80051d0 con 0x7f36e406b5a0 2024-06-09T22:26:58.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.154+0000 7f36e0ff9700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f36cc0a2d90 0x7f36cc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.154+0000 7f36e37fe700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f36cc0a2d90 0x7f36cc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.154+0000 7f36e0ff9700 1 -- 172.21.15.104:0/3159586657 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f36d80cf6f0 con 0x7f36e406b5a0 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.154+0000 7f36e37fe700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f36cc0a2d90 0x7f36cc0a5250 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f36e40ffa40 tx=0x7f36d400b410 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36f46d5b80 1 Processor -- start 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36f46d5b80 1 -- start start 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36f46d5b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc909c40 0x55efbc90a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36f46d5b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc90a560 0x55efbc903e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36f46d5b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc9043d0 0x55efbc904810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36f46d5b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55efbc855910 con 0x55efbc9043d0 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36f46d5b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55efbc7fd820 con 0x55efbc909c40 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36f46d5b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55efbc8518d0 con 0x55efbc90a560 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36e3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc9043d0 0x55efbc904810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36e3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc9043d0 0x55efbc904810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33222/0 (socket says 172.21.15.104:33222) 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36e3fff700 1 -- 172.21.15.104:0/261340047 learned_addr learned my addr 172.21.15.104:0/261340047 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36e37fe700 1 --2- 172.21.15.104:0/261340047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc909c40 0x55efbc90a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36e2ffd700 1 --2- 172.21.15.104:0/261340047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc90a560 0x55efbc903e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36e3fff700 1 -- 172.21.15.104:0/261340047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc90a560 msgr2=0x55efbc903e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36e3fff700 1 --2- 172.21.15.104:0/261340047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc90a560 0x55efbc903e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36e3fff700 1 -- 172.21.15.104:0/261340047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc909c40 msgr2=0x55efbc90a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36e3fff700 1 --2- 172.21.15.104:0/261340047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc909c40 0x55efbc90a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36e3fff700 1 -- 172.21.15.104:0/261340047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efbc7f4050 con 0x55efbc9043d0 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36e37fe700 1 --2- 172.21.15.104:0/261340047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc909c40 0x55efbc90a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.155+0000 7f36e3fff700 1 --2- 172.21.15.104:0/261340047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc9043d0 0x55efbc904810 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00b070 tx=0x7f36dc009df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b50c0d5b8059d88 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.156+0000 7f36c9ffb700 1 -- 172.21.15.104:0/261340047 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc022070 con 0x55efbc9043d0 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.156+0000 7f36c9ffb700 1 -- 172.21.15.104:0/261340047 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36dc0074e0 con 0x55efbc9043d0 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.156+0000 7f36c9ffb700 1 -- 172.21.15.104:0/261340047 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc01c6a0 con 0x55efbc9043d0 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.156+0000 7f36f46d5b80 1 -- 172.21.15.104:0/261340047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc9043d0 msgr2=0x55efbc904810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.156+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/261340047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc9043d0 0x55efbc904810 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00b070 tx=0x7f36dc009df0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.156+0000 7f36f46d5b80 1 -- 172.21.15.104:0/261340047 shutdown_connections 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.156+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/261340047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc9043d0 0x55efbc904810 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.156+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/261340047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc909c40 0x55efbc90a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.156+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/261340047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc90a560 0x55efbc903e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.156+0000 7f36f46d5b80 1 -- 172.21.15.104:0/261340047 >> 172.21.15.104:0/261340047 conn(0x55efbc82edc0 msgr2=0x55efbc8ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:58.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.157+0000 7f36f46d5b80 1 -- 172.21.15.104:0/261340047 shutdown_connections 2024-06-09T22:26:58.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.157+0000 7f36f46d5b80 1 -- 172.21.15.104:0/261340047 wait complete. 2024-06-09T22:26:58.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.157+0000 7f36f46d5b80 1 -- start start 2024-06-09T22:26:58.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.157+0000 7f36f46d5b80 1 Processor -- start 2024-06-09T22:26:58.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.157+0000 7f36f46d5b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 0x55efbc910240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.157+0000 7f36f46d5b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc909c40 0x55efbc90d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.157+0000 7f36f46d5b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc90a560 0x55efbc90da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.157+0000 7f36f46d5b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55efbc910f70 con 0x55efbc909c40 2024-06-09T22:26:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.157+0000 7f36f46d5b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55efbc7f5cf0 con 0x55efbc9043d0 2024-06-09T22:26:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.157+0000 7f36f46d5b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55efbc9110f0 con 0x55efbc90a560 2024-06-09T22:26:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.157+0000 7f36f46d5b80 10 client.0 _ll_register_callbacks cb 0x55efbc7efa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc909c40 0x55efbc90d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc90a560 0x55efbc90da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc909c40 0x55efbc90d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33236/0 (socket says 172.21.15.104:33236) 2024-06-09T22:26:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc90a560 0x55efbc90da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:41792/0 (socket says 172.21.15.104:41792) 2024-06-09T22:26:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e2ffd700 1 -- 172.21.15.104:0/3565455958 learned_addr learned my addr 172.21.15.104:0/3565455958 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e37fe700 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 0x55efbc910240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e2ffd700 1 -- 172.21.15.104:0/3565455958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc90a560 msgr2=0x55efbc90da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:58.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e2ffd700 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc90a560 0x55efbc90da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e2ffd700 1 -- 172.21.15.104:0/3565455958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 msgr2=0x55efbc910240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:58.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e2ffd700 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 0x55efbc910240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e2ffd700 1 -- 172.21.15.104:0/3565455958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efbc7a49d0 con 0x55efbc909c40 2024-06-09T22:26:58.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e3fff700 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc90a560 0x55efbc90da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:58.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36e2ffd700 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc909c40 0x55efbc90d540 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f36d80ccb30 tx=0x7f36d809c090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87a69b2d77b9f1a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:58.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36c2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:58.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36c2ffd700 1 -- 172.21.15.104:0/3565455958 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d809fc50 con 0x55efbc909c40 2024-06-09T22:26:58.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36c2ffd700 1 -- 172.21.15.104:0/3565455958 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f36d80860b0 con 0x55efbc909c40 2024-06-09T22:26:58.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36f46d5b80 1 -- 172.21.15.104:0/3565455958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36dc0074e0 con 0x55efbc909c40 2024-06-09T22:26:58.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36c2ffd700 1 -- 172.21.15.104:0/3565455958 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f36d80a4510 con 0x55efbc909c40 2024-06-09T22:26:58.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36f46d5b80 10 client.17127 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:58.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.158+0000 7f36f46d5b80 1 -- 172.21.15.104:0/3565455958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55efbc91b8c0 con 0x55efbc909c40 2024-06-09T22:26:58.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.159+0000 7f36c2ffd700 1 -- 172.21.15.104:0/3565455958 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36dc0074e0 con 0x55efbc909c40 2024-06-09T22:26:58.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.159+0000 7f36c2ffd700 1 -- 172.21.15.104:0/3565455958 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f36d809bb00 con 0x55efbc909c40 2024-06-09T22:26:58.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.159+0000 7f36f46d5b80 10 client.17127 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:58.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.159+0000 7f36f46d5b80 10 client.17127 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:26:58.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.159+0000 7f36f46d5b80 20 client.17127 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:58.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.159+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55efbc91df00 0x55efbc9202f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.159+0000 7f36f46d5b80 4 client.17127 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.159+0000 7f36f46d5b80 1 -- 172.21.15.104:0/3565455958 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efbc862e10 con 0x55efbc91df00 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.159+0000 7f36e37fe700 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55efbc91df00 0x55efbc9202f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.160+0000 7f36c2ffd700 10 client.17127 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.160+0000 7f36e37fe700 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55efbc91df00 0x55efbc9202f0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdff4729ad383249 server_cookie=a12edb01119f83d0 in_seq=0 out_seq=0 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.160+0000 7f36c2ffd700 1 -- 172.21.15.104:0/3565455958 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55efbc862e10 con 0x55efbc91df00 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.160+0000 7f36c2ffd700 10 client.17127 handle_command_reply: tid=0 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.160+0000 7f36f46d5b80 1 client.17127 shutdown 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.161+0000 7f36f46d5b80 1 -- 172.21.15.104:0/3565455958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc909c40 msgr2=0x55efbc90d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.161+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc909c40 0x55efbc90d540 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f36d80ccb30 tx=0x7f36d809c090 comp rx=0 tx=0).stop 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.161+0000 7f36f46d5b80 1 -- 172.21.15.104:0/3565455958 shutdown_connections 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.161+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55efbc91df00 0x55efbc9202f0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.161+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc90a560 0x55efbc90da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.161+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 0x55efbc910240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.161+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/3565455958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc909c40 0x55efbc90d540 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.161+0000 7f36f46d5b80 1 -- 172.21.15.104:0/3565455958 >> 172.21.15.104:0/3565455958 conn(0x55efbc82edc0 msgr2=0x55efbc8fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.161+0000 7f36c2ffd700 0 client.17127 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.161+0000 7f36f46d5b80 1 -- 172.21.15.104:0/3565455958 shutdown_connections 2024-06-09T22:26:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.161+0000 7f36f46d5b80 1 -- 172.21.15.104:0/3565455958 wait complete. 2024-06-09T22:26:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.161+0000 7f36f46d5b80 20 client.17127 trim_cache size 0 max 16384 2024-06-09T22:26:58.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36f46d5b80 1 Processor -- start 2024-06-09T22:26:58.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36f46d5b80 1 -- start start 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36f46d5b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 0x55efbc903e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36f46d5b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc909c40 0x55efbc8feea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36f46d5b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 0x55efbc90f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36f46d5b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41ae390 con 0x55efbc90a560 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36f46d5b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41afed0 con 0x55efbc9043d0 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36f46d5b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41aea60 con 0x55efbc909c40 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36e3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 0x55efbc90f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36e3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 0x55efbc90f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33248/0 (socket says 172.21.15.104:33248) 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36e2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc909c40 0x55efbc8feea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36e3fff700 1 -- 172.21.15.104:0/1776195973 learned_addr learned my addr 172.21.15.104:0/1776195973 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.181+0000 7f36e37fe700 1 --2- 172.21.15.104:0/1776195973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 0x55efbc903e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.182+0000 7f36e3fff700 1 -- 172.21.15.104:0/1776195973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc909c40 msgr2=0x55efbc8feea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.182+0000 7f36e3fff700 1 --2- 172.21.15.104:0/1776195973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc909c40 0x55efbc8feea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.182+0000 7f36e3fff700 1 -- 172.21.15.104:0/1776195973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 msgr2=0x55efbc903e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:58.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.182+0000 7f36e3fff700 1 --2- 172.21.15.104:0/1776195973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 0x55efbc903e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.182+0000 7f36e3fff700 1 -- 172.21.15.104:0/1776195973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efbc90fe80 con 0x55efbc90a560 2024-06-09T22:26:58.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.182+0000 7f36e3fff700 1 --2- 172.21.15.104:0/1776195973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 0x55efbc90f480 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f36dc007ea0 tx=0x7f36dc004c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b380948ab0da04 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:58.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.182+0000 7f36c97fa700 1 -- 172.21.15.104:0/1776195973 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc02a070 con 0x55efbc90a560 2024-06-09T22:26:58.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.182+0000 7f36c97fa700 1 -- 172.21.15.104:0/1776195973 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36dc0079c0 con 0x55efbc90a560 2024-06-09T22:26:58.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.182+0000 7f36c97fa700 1 -- 172.21.15.104:0/1776195973 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc022040 con 0x55efbc90a560 2024-06-09T22:26:58.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 -- 172.21.15.104:0/1776195973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 msgr2=0x55efbc90f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:58.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/1776195973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 0x55efbc90f480 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f36dc007ea0 tx=0x7f36dc004c00 comp rx=0 tx=0).stop 2024-06-09T22:26:58.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 -- 172.21.15.104:0/1776195973 shutdown_connections 2024-06-09T22:26:58.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/1776195973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 0x55efbc90f480 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/1776195973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 0x55efbc903e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/1776195973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc909c40 0x55efbc8feea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 -- 172.21.15.104:0/1776195973 >> 172.21.15.104:0/1776195973 conn(0x55efbc82edc0 msgr2=0x55efbc90ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:58.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 -- 172.21.15.104:0/1776195973 shutdown_connections 2024-06-09T22:26:58.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 -- 172.21.15.104:0/1776195973 wait complete. 2024-06-09T22:26:58.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 -- start start 2024-06-09T22:26:58.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 Processor -- start 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 0x55efbc9165c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc909c40 0x55efbc916b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 0x55efbc9133b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41af800 con 0x55efbc90a560 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41af130 con 0x55efbc9043d0 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.183+0000 7f36f46d5b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41ac310 con 0x55efbc909c40 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36f46d5b80 10 client.0 _ll_register_callbacks cb 0x55efbc7efa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36e3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 0x55efbc9133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36e3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 0x55efbc9133b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33264/0 (socket says 172.21.15.104:33264) 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36e3fff700 1 -- 172.21.15.104:0/2037908069 learned_addr learned my addr 172.21.15.104:0/2037908069 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36e2ffd700 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc909c40 0x55efbc916b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36e37fe700 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 0x55efbc9165c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36e3fff700 1 -- 172.21.15.104:0/2037908069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc909c40 msgr2=0x55efbc916b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36e3fff700 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc909c40 0x55efbc916b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36e3fff700 1 -- 172.21.15.104:0/2037908069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 msgr2=0x55efbc9165c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:26:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36e3fff700 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 0x55efbc9165c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36e3fff700 1 -- 172.21.15.104:0/2037908069 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efbc913bf0 con 0x55efbc90a560 2024-06-09T22:26:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36e2ffd700 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc909c40 0x55efbc916b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:26:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36e3fff700 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 0x55efbc9133b0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009a20 tx=0x7f36dc007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c3d005be1101f7 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:26:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36c2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:26:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36c2ffd700 1 -- 172.21.15.104:0/2037908069 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc022040 con 0x55efbc90a560 2024-06-09T22:26:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36c2ffd700 1 -- 172.21.15.104:0/2037908069 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f36dc02fac0 con 0x55efbc90a560 2024-06-09T22:26:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36f46d5b80 1 -- 172.21.15.104:0/2037908069 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36dc0079c0 con 0x55efbc90a560 2024-06-09T22:26:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36c2ffd700 1 -- 172.21.15.104:0/2037908069 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f36dc02a040 con 0x55efbc90a560 2024-06-09T22:26:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36f46d5b80 10 client.17133 fetch_fsmap learned FSMap version 32 2024-06-09T22:26:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.184+0000 7f36f46d5b80 1 -- 172.21.15.104:0/2037908069 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55efbc989600 con 0x55efbc90a560 2024-06-09T22:26:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.185+0000 7f36c2ffd700 1 -- 172.21.15.104:0/2037908069 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36dc0079c0 con 0x55efbc90a560 2024-06-09T22:26:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.185+0000 7f36c2ffd700 1 -- 172.21.15.104:0/2037908069 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f36dc011070 con 0x55efbc90a560 2024-06-09T22:26:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.185+0000 7f36f46d5b80 10 client.17133 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:26:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.185+0000 7f36f46d5b80 10 client.17133 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:26:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.185+0000 7f36f46d5b80 20 client.17133 populate_metadata read hostname 'smithi104' 2024-06-09T22:26:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.185+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55efbc989830 0x55efbc98bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:26:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.185+0000 7f36f46d5b80 4 client.17133 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b7cbcb6e", "release": true, "await": true}] 2024-06-09T22:26:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.185+0000 7f36f46d5b80 1 -- 172.21.15.104:0/2037908069 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b7cbcb6e", "release": true, "await": true}) -- 0x55efbc85d390 con 0x55efbc989830 2024-06-09T22:26:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.185+0000 7f36e37fe700 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55efbc989830 0x55efbc98bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:26:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.186+0000 7f36e37fe700 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55efbc989830 0x55efbc98bc20 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f554f648e49a59e3 server_cookie=463c7224bbe2fab1 in_seq=0 out_seq=0 2024-06-09T22:26:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.186+0000 7f36c2ffd700 10 client.17133 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36c2ffd700 1 -- 172.21.15.104:0/2037908069 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55efbc85d390 con 0x55efbc989830 2024-06-09T22:26:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36c2ffd700 10 client.17133 handle_command_reply: tid=0 2024-06-09T22:26:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36f46d5b80 1 client.17133 shutdown 2024-06-09T22:26:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36f46d5b80 1 -- 172.21.15.104:0/2037908069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 msgr2=0x55efbc9133b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 0x55efbc9133b0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009a20 tx=0x7f36dc007710 comp rx=0 tx=0).stop 2024-06-09T22:26:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36f46d5b80 1 -- 172.21.15.104:0/2037908069 shutdown_connections 2024-06-09T22:26:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55efbc989830 0x55efbc98bc20 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efbc90a560 0x55efbc9133b0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efbc9043d0 0x55efbc9165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36f46d5b80 1 --2- 172.21.15.104:0/2037908069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55efbc909c40 0x55efbc916b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36f46d5b80 1 -- 172.21.15.104:0/2037908069 >> 172.21.15.104:0/2037908069 conn(0x55efbc82edc0 msgr2=0x55efbc90ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:58.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36c2ffd700 0 client.17133 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:26:58.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36f46d5b80 1 -- 172.21.15.104:0/2037908069 shutdown_connections 2024-06-09T22:26:58.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36f46d5b80 1 -- 172.21.15.104:0/2037908069 wait complete. 2024-06-09T22:26:58.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.198+0000 7f36f46d5b80 20 client.17133 trim_cache size 0 max 16384 2024-06-09T22:26:58.201 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:26:58.201 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 32, 2024-06-09T22:26:58.201 INFO:teuthology.orchestra.run.smithi104.stdout: "leader": 24460, 2024-06-09T22:26:58.201 INFO:teuthology.orchestra.run.smithi104.stdout: "set_version": 16, 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: "sets": { 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: "b7cbcb6e": { 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: "version": 16, 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: "age_ref": 0.0, 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RELEASED", 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: "timeout": 90.0, 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: "expiration": 149.4, 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: "members": { 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: "file:/": { 2024-06-09T22:26:58.202 INFO:teuthology.orchestra.run.smithi104.stdout: "excluded": false, 2024-06-09T22:26:58.203 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:26:58.203 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RELEASED", 2024-06-09T22:26:58.203 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:26:58.203 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:26:58.203 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:26:58.203 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:26:58.203 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:26:58.203 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:26:58.203 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:26:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.199+0000 7f36e9e52700 1 -- 172.21.15.104:0/3159586657 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f36cc0a2d90 msgr2=0x7f36cc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.199+0000 7f36e9e52700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f36cc0a2d90 0x7f36cc0a5250 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f36e40ffa40 tx=0x7f36d400b410 comp rx=0 tx=0).stop 2024-06-09T22:26:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.199+0000 7f36e9e52700 1 -- 172.21.15.104:0/3159586657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e406b5a0 msgr2=0x7f36e419ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:26:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.199+0000 7f36e9e52700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e406b5a0 0x7f36e419ee20 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f36d800d980 tx=0x7f36d800dc90 comp rx=0 tx=0).stop 2024-06-09T22:26:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.199+0000 7f36e9e52700 1 -- 172.21.15.104:0/3159586657 shutdown_connections 2024-06-09T22:26:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.199+0000 7f36e9e52700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f36cc0a2d90 0x7f36cc0a5250 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.199+0000 7f36e9e52700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36e4108500 0x7f36e4198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.199+0000 7f36e9e52700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36e406abe0 0x7f36e419e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.199+0000 7f36e9e52700 1 --2- 172.21.15.104:0/3159586657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36e406b5a0 0x7f36e419ee20 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:26:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.199+0000 7f36e9e52700 1 -- 172.21.15.104:0/3159586657 >> 172.21.15.104:0/3159586657 conn(0x7f36e4078360 msgr2=0x7f36e4100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:26:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.199+0000 7f36e9e52700 1 -- 172.21.15.104:0/3159586657 shutdown_connections 2024-06-09T22:26:58.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:26:58.199+0000 7f36e9e52700 1 -- 172.21.15.104:0/3159586657 wait complete. 2024-06-09T22:26:58.732 INFO:tasks.workunit.client.0.smithi104.stdout: 1 177588 4.62 MB/sec execute 124 sec latency 26498.547 ms 2024-06-09T22:26:58.934 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b7cbcb6e', total seconds elapsed: 35.4 2024-06-09T22:26:58.934 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.45 2024-06-09T22:26:58.934 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:26:59.188 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:26:59.584 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:26:59 smithi104 ceph-mon[32389]: pgmap v183: 129 pgs: 129 active+clean; 128 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 54 op/s 2024-06-09T22:26:59.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 178790 4.63 MB/sec execute 125 sec latency 20.664 ms 2024-06-09T22:26:59.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:26:59 smithi194 ceph-mon[36856]: pgmap v183: 129 pgs: 129 active+clean; 128 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 54 op/s 2024-06-09T22:26:59.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:26:59 smithi153 ceph-mon[36599]: pgmap v183: 129 pgs: 129 active+clean; 128 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 54 op/s 2024-06-09T22:27:00.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 181029 4.69 MB/sec execute 126 sec latency 29.310 ms 2024-06-09T22:27:01.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.354+0000 7fe51407a700 1 -- 172.21.15.104:0/239386621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe50c100a80 msgr2=0x7fe50c100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:01.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.354+0000 7fe51407a700 1 --2- 172.21.15.104:0/239386621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe50c100a80 0x7fe50c100f00 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009990 tx=0x7fe4fc009ca0 comp rx=0 tx=0).stop 2024-06-09T22:27:01.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.354+0000 7fe51407a700 1 -- 172.21.15.104:0/239386621 shutdown_connections 2024-06-09T22:27:01.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.354+0000 7fe51407a700 1 --2- 172.21.15.104:0/239386621 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe50c101440 0x7fe50c10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:01.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.354+0000 7fe51407a700 1 --2- 172.21.15.104:0/239386621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50c108510 0x7fe50c1088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:01.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.354+0000 7fe51407a700 1 --2- 172.21.15.104:0/239386621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe50c100a80 0x7fe50c100f00 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:01.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.354+0000 7fe51407a700 1 -- 172.21.15.104:0/239386621 >> 172.21.15.104:0/239386621 conn(0x7fe50c077a00 msgr2=0x7fe50c0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:01.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.354+0000 7fe51407a700 1 -- 172.21.15.104:0/239386621 shutdown_connections 2024-06-09T22:27:01.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.354+0000 7fe51407a700 1 -- 172.21.15.104:0/239386621 wait complete. 2024-06-09T22:27:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe51407a700 1 Processor -- start 2024-06-09T22:27:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe51407a700 1 -- start start 2024-06-09T22:27:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe51407a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe50c100a80 0x7fe50c19e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe51407a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50c101440 0x7fe50c19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe51407a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe50c108510 0x7fe50c198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe51407a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c112830 con 0x7fe50c100a80 2024-06-09T22:27:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe51407a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1126b0 con 0x7fe50c101440 2024-06-09T22:27:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe51407a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1129b0 con 0x7fe50c108510 2024-06-09T22:27:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe511615700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50c101440 0x7fe50c19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe512617700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe50c108510 0x7fe50c198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe511615700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50c101440 0x7fe50c19edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:49990/0 (socket says 172.21.15.104:49990) 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe511615700 1 -- 172.21.15.104:0/240214727 learned_addr learned my addr 172.21.15.104:0/240214727 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe511e16700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe50c100a80 0x7fe50c19e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe511615700 1 -- 172.21.15.104:0/240214727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe50c108510 msgr2=0x7fe50c198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe511615700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe50c108510 0x7fe50c198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.355+0000 7fe511615700 1 -- 172.21.15.104:0/240214727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe50c100a80 msgr2=0x7fe50c19e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.356+0000 7fe511615700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe50c100a80 0x7fe50c19e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.356+0000 7fe511615700 1 -- 172.21.15.104:0/240214727 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50c1991d0 con 0x7fe50c101440 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.356+0000 7fe511e16700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe50c100a80 0x7fe50c19e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.356+0000 7fe511615700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50c101440 0x7fe50c19edf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc005910 tx=0x7fe4fc00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.356+0000 7fe502ffd700 1 -- 172.21.15.104:0/240214727 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc026070 con 0x7fe50c101440 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.356+0000 7fe502ffd700 1 -- 172.21.15.104:0/240214727 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4fc003d70 con 0x7fe50c101440 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.356+0000 7fe51407a700 1 -- 172.21.15.104:0/240214727 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe50c199460 con 0x7fe50c101440 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.356+0000 7fe502ffd700 1 -- 172.21.15.104:0/240214727 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc00fe40 con 0x7fe50c101440 2024-06-09T22:27:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.356+0000 7fe51407a700 1 -- 172.21.15.104:0/240214727 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe50c101b90 con 0x7fe50c101440 2024-06-09T22:27:01.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.358+0000 7fe51407a700 1 -- 172.21.15.104:0/240214727 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe50c199fa0 con 0x7fe50c101440 2024-06-09T22:27:01.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.359+0000 7fe502ffd700 1 -- 172.21.15.104:0/240214727 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fe4fc029070 con 0x7fe50c101440 2024-06-09T22:27:01.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.359+0000 7fe502ffd700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe4f80a2e50 0x7fe4f80a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:01.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.359+0000 7fe511e16700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe4f80a2e50 0x7fe4f80a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:01.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.359+0000 7fe502ffd700 1 -- 172.21.15.104:0/240214727 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fe4fc0d06c0 con 0x7fe50c101440 2024-06-09T22:27:01.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.360+0000 7fe511e16700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe4f80a2e50 0x7fe4f80a5310 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe50c199bb0 tx=0x7fe508009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:01.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.363+0000 7fe502ffd700 1 -- 172.21.15.104:0/240214727 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe4fc013070 con 0x7fe50c101440 2024-06-09T22:27:01.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.629+0000 7fe51407a700 1 -- 172.21.15.104:0/240214727 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe50c02dca0 con 0x7fe50c101440 2024-06-09T22:27:01.632 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:01 smithi104 ceph-mon[32389]: pgmap v184: 129 pgs: 129 active+clean; 127 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 2.1 MiB/s wr, 93 op/s 2024-06-09T22:27:01.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.630+0000 7fe502ffd700 1 -- 172.21.15.104:0/240214727 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7fe4fc09d000 con 0x7fe50c101440 2024-06-09T22:27:01.632 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:27:01.632 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:27:01.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.633+0000 7fe51407a700 1 -- 172.21.15.104:0/240214727 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe4f80a2e50 msgr2=0x7fe4f80a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:01.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.633+0000 7fe51407a700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe4f80a2e50 0x7fe4f80a5310 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe50c199bb0 tx=0x7fe508009260 comp rx=0 tx=0).stop 2024-06-09T22:27:01.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.633+0000 7fe51407a700 1 -- 172.21.15.104:0/240214727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50c101440 msgr2=0x7fe50c19edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:01.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.633+0000 7fe51407a700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50c101440 0x7fe50c19edf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc005910 tx=0x7fe4fc00bab0 comp rx=0 tx=0).stop 2024-06-09T22:27:01.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.633+0000 7fe51407a700 1 -- 172.21.15.104:0/240214727 shutdown_connections 2024-06-09T22:27:01.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.633+0000 7fe51407a700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe4f80a2e50 0x7fe4f80a5310 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:01.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.633+0000 7fe51407a700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe50c100a80 0x7fe50c19e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:01.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.633+0000 7fe51407a700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe50c108510 0x7fe50c198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:01.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.633+0000 7fe51407a700 1 --2- 172.21.15.104:0/240214727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50c101440 0x7fe50c19edf0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:01.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.633+0000 7fe51407a700 1 -- 172.21.15.104:0/240214727 >> 172.21.15.104:0/240214727 conn(0x7fe50c077a00 msgr2=0x7fe50c0ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:01.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.633+0000 7fe51407a700 1 -- 172.21.15.104:0/240214727 shutdown_connections 2024-06-09T22:27:01.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:01.633+0000 7fe51407a700 1 -- 172.21.15.104:0/240214727 wait complete. 2024-06-09T22:27:01.636 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:27:01.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 181921 4.67 MB/sec execute 127 sec latency 77.504 ms 2024-06-09T22:27:01.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:01 smithi194 ceph-mon[36856]: pgmap v184: 129 pgs: 129 active+clean; 127 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 2.1 MiB/s wr, 93 op/s 2024-06-09T22:27:01.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:01 smithi153 ceph-mon[36599]: pgmap v184: 129 pgs: 129 active+clean; 127 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 2.1 MiB/s wr, 93 op/s 2024-06-09T22:27:02.421 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24460 quiesce db / --timeout 90 --expiration 149.45 --await-for 45.0' 2024-06-09T22:27:02.421 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24460 quiesce db / --timeout 90 --expiration 149.45 --await-for 45.0 2024-06-09T22:27:02.676 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:27:02.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 182471 4.67 MB/sec execute 128 sec latency 88.592 ms 2024-06-09T22:27:02.758 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:02 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/240214727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:27:02.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:02 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/240214727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:27:02.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:02 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/240214727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:27:03.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 184711 4.74 MB/sec execute 129 sec latency 14.528 ms 2024-06-09T22:27:03.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:03 smithi194 ceph-mon[36856]: pgmap v185: 129 pgs: 129 active+clean; 127 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 1.5 MiB/s wr, 65 op/s 2024-06-09T22:27:03.961 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:03 smithi104 ceph-mon[32389]: pgmap v185: 129 pgs: 129 active+clean; 127 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 1.5 MiB/s wr, 65 op/s 2024-06-09T22:27:03.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:03 smithi153 ceph-mon[36599]: pgmap v185: 129 pgs: 129 active+clean; 127 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 1.5 MiB/s wr, 65 op/s 2024-06-09T22:27:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.544+0000 7fa8dab09700 1 -- 172.21.15.104:0/2225280283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8d4108510 msgr2=0x7fa8d41088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.544+0000 7fa8dab09700 1 --2- 172.21.15.104:0/2225280283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8d4108510 0x7fa8d41088f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009990 tx=0x7fa8c8009ca0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.545+0000 7fa8dab09700 1 -- 172.21.15.104:0/2225280283 shutdown_connections 2024-06-09T22:27:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.545+0000 7fa8dab09700 1 --2- 172.21.15.104:0/2225280283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8d4108510 0x7fa8d41088f0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.545+0000 7fa8dab09700 1 --2- 172.21.15.104:0/2225280283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8d4101440 0x7fa8d410f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.545+0000 7fa8dab09700 1 --2- 172.21.15.104:0/2225280283 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8d4100a80 0x7fa8d4100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.545+0000 7fa8dab09700 1 -- 172.21.15.104:0/2225280283 >> 172.21.15.104:0/2225280283 conn(0x7fa8d4077a30 msgr2=0x7fa8d40fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.545+0000 7fa8dab09700 1 -- 172.21.15.104:0/2225280283 shutdown_connections 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.545+0000 7fa8dab09700 1 -- 172.21.15.104:0/2225280283 wait complete. 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.545+0000 7fa8dab09700 1 Processor -- start 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8dab09700 1 -- start start 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8dab09700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8d4100a80 0x7fa8d4101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8dab09700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8d4101440 0x7fa8d41021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8dab09700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8d4108510 0x7fa8d41026e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8dab09700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d4112730 con 0x7fa8d4101440 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8dab09700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d41125b0 con 0x7fa8d4100a80 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8dab09700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d41128b0 con 0x7fa8d4108510 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8d3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8d4101440 0x7fa8d41021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8d3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8d4101440 0x7fa8d41021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33316/0 (socket says 172.21.15.104:33316) 2024-06-09T22:27:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8d3fff700 1 -- 172.21.15.104:0/3504683042 learned_addr learned my addr 172.21.15.104:0/3504683042 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8d4100a80 0x7fa8d4101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8d90a6700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8d4108510 0x7fa8d41026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8d3fff700 1 -- 172.21.15.104:0/3504683042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8d4108510 msgr2=0x7fa8d41026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8d3fff700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8d4108510 0x7fa8d41026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8d3fff700 1 -- 172.21.15.104:0/3504683042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8d4100a80 msgr2=0x7fa8d4101c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8d3fff700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8d4100a80 0x7fa8d4101c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8d3fff700 1 -- 172.21.15.104:0/3504683042 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8d41a7050 con 0x7fa8d4101440 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8d90a6700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8d4108510 0x7fa8d41026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.546+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8d4100a80 0x7fa8d4101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.547+0000 7fa8d3fff700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8d4101440 0x7fa8d41021a0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa8c000e9a0 tx=0x7fa8c000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.547+0000 7fa8d1ffb700 1 -- 172.21.15.104:0/3504683042 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c000ca10 con 0x7fa8d4101440 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.547+0000 7fa8dab09700 1 -- 172.21.15.104:0/3504683042 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8d41a7340 con 0x7fa8d4101440 2024-06-09T22:27:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.547+0000 7fa8d1ffb700 1 -- 172.21.15.104:0/3504683042 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8c000cbb0 con 0x7fa8d4101440 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.547+0000 7fa8d1ffb700 1 -- 172.21.15.104:0/3504683042 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0018680 con 0x7fa8d4101440 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.547+0000 7fa8dab09700 1 -- 172.21.15.104:0/3504683042 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8d41a7880 con 0x7fa8d4101440 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.549+0000 7fa8d1ffb700 1 -- 172.21.15.104:0/3504683042 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa8c0010430 con 0x7fa8d4101440 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.550+0000 7fa8d1ffb700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa8c40a2e30 0x7fa8c40a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.550+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa8c40a2e30 0x7fa8c40a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.550+0000 7fa8d1ffb700 1 -- 172.21.15.104:0/3504683042 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fa8c00cfa00 con 0x7fa8d4101440 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.550+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa8c40a2e30 0x7fa8c40a52f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800b2e0 tx=0x7fa8c8005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8e538cb80 1 Processor -- start 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8e538cb80 1 -- start start 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8e538cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac79ce0 0x5648cac7a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8e538cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac7a600 0x5648cac73f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8e538cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac748b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8e538cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5648cabc59b0 con 0x5648cac7a600 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8e538cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5648cab6d8c0 con 0x5648cac79ce0 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8e538cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5648cabc1970 con 0x5648cac74470 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8d3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac7a600 0x5648cac73f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8d3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac7a600 0x5648cac73f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33328/0 (socket says 172.21.15.104:33328) 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8d3fff700 1 -- 172.21.15.104:0/2728545993 learned_addr learned my addr 172.21.15.104:0/2728545993 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/2728545993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac79ce0 0x5648cac7a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8d90a6700 1 --2- 172.21.15.104:0/2728545993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac748b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8d3fff700 1 -- 172.21.15.104:0/2728545993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 msgr2=0x5648cac748b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8d3fff700 1 --2- 172.21.15.104:0/2728545993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac748b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8d3fff700 1 -- 172.21.15.104:0/2728545993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac79ce0 msgr2=0x5648cac7a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8d3fff700 1 --2- 172.21.15.104:0/2728545993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac79ce0 0x5648cac7a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.551+0000 7fa8d3fff700 1 -- 172.21.15.104:0/2728545993 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648cab640f0 con 0x5648cac7a600 2024-06-09T22:27:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.552+0000 7fa8d3fff700 1 --2- 172.21.15.104:0/2728545993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac7a600 0x5648cac73f30 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0098010 tx=0x7fa8c009c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541c627cd69bcc02 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:04.709 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:27:04.709 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 32, 2024-06-09T22:27:04.709 INFO:teuthology.orchestra.run.smithi104.stdout: "leader": 24460, 2024-06-09T22:27:04.709 INFO:teuthology.orchestra.run.smithi104.stdout: "set_version": 18, 2024-06-09T22:27:04.709 INFO:teuthology.orchestra.run.smithi104.stdout: "sets": { 2024-06-09T22:27:04.709 INFO:teuthology.orchestra.run.smithi104.stdout: "6813a49": { 2024-06-09T22:27:04.709 INFO:teuthology.orchestra.run.smithi104.stdout: "version": 18, 2024-06-09T22:27:04.709 INFO:teuthology.orchestra.run.smithi104.stdout: "age_ref": 0.0, 2024-06-09T22:27:04.709 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:27:04.709 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "QUIESCED", 2024-06-09T22:27:04.709 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: "timeout": 90.0, 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: "expiration": 149.4, 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: "members": { 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: "file:/": { 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: "excluded": false, 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "QUIESCED", 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:27:04.710 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:27:04.711 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:27:04.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.552+0000 7fa8beffd700 1 -- 172.21.15.104:0/2728545993 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c00a4820 con 0x5648cac7a600 2024-06-09T22:27:04.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.552+0000 7fa8beffd700 1 -- 172.21.15.104:0/2728545993 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8c0099eb0 con 0x5648cac7a600 2024-06-09T22:27:04.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.552+0000 7fa8beffd700 1 -- 172.21.15.104:0/2728545993 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0010980 con 0x5648cac7a600 2024-06-09T22:27:04.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.552+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2728545993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac7a600 msgr2=0x5648cac73f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:04.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.552+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/2728545993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac7a600 0x5648cac73f30 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0098010 tx=0x7fa8c009c330 comp rx=0 tx=0).stop 2024-06-09T22:27:04.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.553+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2728545993 shutdown_connections 2024-06-09T22:27:04.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.553+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/2728545993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac748b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.553+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/2728545993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac79ce0 0x5648cac7a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.553+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/2728545993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac7a600 0x5648cac73f30 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.553+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2728545993 >> 172.21.15.104:0/2728545993 conn(0x5648cab9ee60 msgr2=0x5648cac6f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.553+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2728545993 shutdown_connections 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.553+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2728545993 wait complete. 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.553+0000 7fa8e538cb80 1 -- start start 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.553+0000 7fa8e538cb80 1 Processor -- start 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.553+0000 7fa8e538cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac80380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8e538cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 0x5648cac7d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8e538cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac7a600 0x5648cac7dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8e538cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5648cac81070 con 0x5648cac79ce0 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d88a5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac80380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 0x5648cac7d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 0x5648cac7d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33336/0 (socket says 172.21.15.104:33336) 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d88a5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac80380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:41872/0 (socket says 172.21.15.104:41872) 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d3fff700 1 -- 172.21.15.104:0/4075743384 learned_addr learned my addr 172.21.15.104:0/4075743384 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d90a6700 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac7a600 0x5648cac7dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8e538cb80 1 -- 172.21.15.104:0/4075743384 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5648cab65d90 con 0x5648cac7a600 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8e538cb80 1 -- 172.21.15.104:0/4075743384 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5648cac811f0 con 0x5648cac74470 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8e538cb80 10 client.0 _ll_register_callbacks cb 0x5648cab5fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d90a6700 1 -- 172.21.15.104:0/4075743384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 msgr2=0x5648cac80380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d90a6700 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac80380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d90a6700 1 -- 172.21.15.104:0/4075743384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 msgr2=0x5648cac7d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d90a6700 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 0x5648cac7d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d90a6700 1 -- 172.21.15.104:0/4075743384 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648cab14a70 con 0x5648cac7a600 2024-06-09T22:27:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d3fff700 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 0x5648cac7d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8d90a6700 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac7a600 0x5648cac7dbc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc004630 tx=0x7fa8cc004660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b3f58dbaac955bb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.554+0000 7fa8b3fff700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8b3fff700 1 -- 172.21.15.104:0/4075743384 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc01a070 con 0x5648cac7a600 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8b3fff700 1 -- 172.21.15.104:0/4075743384 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fa8cc004db0 con 0x5648cac7a600 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8e538cb80 1 -- 172.21.15.104:0/4075743384 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8c0099eb0 con 0x5648cac7a600 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8b3fff700 1 -- 172.21.15.104:0/4075743384 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa8cc025050 con 0x5648cac7a600 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8e538cb80 10 client.26435 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8e538cb80 1 -- 172.21.15.104:0/4075743384 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5648cac8b9f0 con 0x5648cac7a600 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8b3fff700 1 -- 172.21.15.104:0/4075743384 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8c0099eb0 con 0x5648cac7a600 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8b3fff700 1 -- 172.21.15.104:0/4075743384 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa8cc0169c0 con 0x5648cac7a600 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8e538cb80 10 client.26435 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8e538cb80 10 client.26435 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8e538cb80 20 client.26435 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5648cac8df80 0x5648cac903a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8e538cb80 4 client.26435 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8e538cb80 1 -- 172.21.15.104:0/4075743384 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648cabd2eb0 con 0x5648cac8df80 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.555+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5648cac8df80 0x5648cac903a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.556+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5648cac8df80 0x5648cac903a0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c5fef224b1b2cd3 server_cookie=e4863f7c414e8a63 in_seq=0 out_seq=0 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.556+0000 7fa8b3fff700 10 client.26435 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.556+0000 7fa8b3fff700 1 -- 172.21.15.104:0/4075743384 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5648cabd2eb0 con 0x5648cac8df80 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.556+0000 7fa8b3fff700 10 client.26435 handle_command_reply: tid=0 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.556+0000 7fa8e538cb80 1 client.26435 shutdown 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.557+0000 7fa8e538cb80 1 -- 172.21.15.104:0/4075743384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac7a600 msgr2=0x5648cac7dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.557+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac7a600 0x5648cac7dbc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc004630 tx=0x7fa8cc004660 comp rx=0 tx=0).stop 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.557+0000 7fa8e538cb80 1 -- 172.21.15.104:0/4075743384 shutdown_connections 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.557+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5648cac8df80 0x5648cac903a0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.557+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac7a600 0x5648cac7dbc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.557+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac80380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.557+0000 7fa8b3fff700 0 client.26435 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.557+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/4075743384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 0x5648cac7d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.557+0000 7fa8e538cb80 1 -- 172.21.15.104:0/4075743384 >> 172.21.15.104:0/4075743384 conn(0x5648cab9ee60 msgr2=0x5648cac72f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.557+0000 7fa8e538cb80 1 -- 172.21.15.104:0/4075743384 shutdown_connections 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.557+0000 7fa8e538cb80 1 -- 172.21.15.104:0/4075743384 wait complete. 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.557+0000 7fa8e538cb80 20 client.26435 trim_cache size 0 max 16384 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8e538cb80 1 Processor -- start 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8e538cb80 1 -- start start 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8e538cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac74470 0x5648cac7c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8e538cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac79ce0 0x5648cac7b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8e538cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac7a600 0x5648cac95c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8e538cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d41b0610 con 0x5648cac74470 2024-06-09T22:27:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8e538cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d41b2150 con 0x5648cac79ce0 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8e538cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d41b0ce0 con 0x5648cac7a600 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8d88a5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac74470 0x5648cac7c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8d88a5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac74470 0x5648cac7c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33352/0 (socket says 172.21.15.104:33352) 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8d88a5700 1 -- 172.21.15.104:0/1780637064 learned_addr learned my addr 172.21.15.104:0/1780637064 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8d90a6700 1 --2- 172.21.15.104:0/1780637064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac7a600 0x5648cac95c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8d3fff700 1 --2- 172.21.15.104:0/1780637064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac79ce0 0x5648cac7b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8d88a5700 1 -- 172.21.15.104:0/1780637064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac7a600 msgr2=0x5648cac95c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/1780637064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac7a600 0x5648cac95c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8d88a5700 1 -- 172.21.15.104:0/1780637064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac79ce0 msgr2=0x5648cac7b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/1780637064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac79ce0 0x5648cac7b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8d88a5700 1 -- 172.21.15.104:0/1780637064 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648cac90a20 con 0x5648cac74470 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.579+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/1780637064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac74470 0x5648cac7c600 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009990 tx=0x7fa8c8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89acd832be9aaedc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.580+0000 7fa8be7fc700 1 -- 172.21.15.104:0/1780637064 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8005240 con 0x5648cac74470 2024-06-09T22:27:04.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.580+0000 7fa8be7fc700 1 -- 172.21.15.104:0/1780637064 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8c8004e20 con 0x5648cac74470 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.580+0000 7fa8be7fc700 1 -- 172.21.15.104:0/1780637064 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c800f460 con 0x5648cac74470 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.580+0000 7fa8e538cb80 1 -- 172.21.15.104:0/1780637064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac74470 msgr2=0x5648cac7c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.580+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/1780637064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac74470 0x5648cac7c600 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009990 tx=0x7fa8c8020000 comp rx=0 tx=0).stop 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.580+0000 7fa8e538cb80 1 -- 172.21.15.104:0/1780637064 shutdown_connections 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.580+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/1780637064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac74470 0x5648cac7c600 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.580+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/1780637064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac7a600 0x5648cac95c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.580+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/1780637064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac79ce0 0x5648cac7b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.580+0000 7fa8e538cb80 1 -- 172.21.15.104:0/1780637064 >> 172.21.15.104:0/1780637064 conn(0x5648cab9ee60 msgr2=0x5648cac7aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.580+0000 7fa8e538cb80 1 -- 172.21.15.104:0/1780637064 shutdown_connections 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.580+0000 7fa8e538cb80 1 -- 172.21.15.104:0/1780637064 wait complete. 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8e538cb80 1 -- start start 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8e538cb80 1 Processor -- start 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8e538cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac86480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8e538cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 0x5648cac869c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8e538cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac7a600 0x5648cac83450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8e538cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d41b1a80 con 0x5648cac79ce0 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8e538cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d41b13b0 con 0x5648cac7a600 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8e538cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d41ae590 con 0x5648cac74470 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8e538cb80 10 client.0 _ll_register_callbacks cb 0x5648cab5fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8d3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 0x5648cac869c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8d3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 0x5648cac869c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33362/0 (socket says 172.21.15.104:33362) 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8d3fff700 1 -- 172.21.15.104:0/2505900276 learned_addr learned my addr 172.21.15.104:0/2505900276 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac86480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.581+0000 7fa8d90a6700 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac7a600 0x5648cac83450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8d3fff700 1 -- 172.21.15.104:0/2505900276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 msgr2=0x5648cac86480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8d3fff700 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac86480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8d3fff700 1 -- 172.21.15.104:0/2505900276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac7a600 msgr2=0x5648cac83450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8d3fff700 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac7a600 0x5648cac83450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8d3fff700 1 -- 172.21.15.104:0/2505900276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648cac83990 con 0x5648cac79ce0 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8d3fff700 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 0x5648cac869c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa8c009bb40 tx=0x7fa8c0010c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b6be11f619f354 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8b3fff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8b3fff700 1 -- 172.21.15.104:0/2505900276 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c00b8070 con 0x5648cac79ce0 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2505900276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8c8004e20 con 0x5648cac79ce0 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8b3fff700 1 -- 172.21.15.104:0/2505900276 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fa8c00ba2f0 con 0x5648cac79ce0 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8b3fff700 1 -- 172.21.15.104:0/2505900276 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa8c00b2030 con 0x5648cac79ce0 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8e538cb80 10 client.17157 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2505900276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5648cacf9440 con 0x5648cac79ce0 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8b3fff700 1 -- 172.21.15.104:0/2505900276 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8c8004e20 con 0x5648cac79ce0 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8b3fff700 1 -- 172.21.15.104:0/2505900276 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa8c009b000 con 0x5648cac79ce0 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8e538cb80 10 client.17157 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8e538cb80 10 client.17157 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8e538cb80 20 client.17157 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5648cacf9780 0x5648cacfbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8e538cb80 4 client.17157 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.45, "await_for": 45.0}] 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.582+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2505900276 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.45, "await_for": 45.0}) -- 0x5648cabcd430 con 0x5648cacf9780 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.583+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5648cacf9780 0x5648cacfbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.583+0000 7fa8b3fff700 10 client.17157 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.583+0000 7fa8d88a5700 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5648cacf9780 0x5648cacfbb70 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56dce6d025f8c978 server_cookie=4ab5f2ec33e84f73 in_seq=0 out_seq=0 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8b3fff700 1 -- 172.21.15.104:0/2505900276 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5648cabcd430 con 0x5648cacf9780 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8b3fff700 10 client.17157 handle_command_reply: tid=0 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8e538cb80 1 client.17157 shutdown 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2505900276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 msgr2=0x5648cac869c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 0x5648cac869c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa8c009bb40 tx=0x7fa8c0010c10 comp rx=0 tx=0).stop 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2505900276 shutdown_connections 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5648cacf9780 0x5648cacfbb70 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648cac7a600 0x5648cac83450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5648cac74470 0x5648cac86480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8b3fff700 0 client.17157 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8e538cb80 1 --2- 172.21.15.104:0/2505900276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5648cac79ce0 0x5648cac869c0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2505900276 >> 172.21.15.104:0/2505900276 conn(0x5648cab9ee60 msgr2=0x5648cac6e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2505900276 shutdown_connections 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.674+0000 7fa8e538cb80 1 -- 172.21.15.104:0/2505900276 wait complete. 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8e538cb80 20 client.17157 trim_cache size 0 max 16384 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8dab09700 1 -- 172.21.15.104:0/3504683042 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa8c40a2e30 msgr2=0x7fa8c40a52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8dab09700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa8c40a2e30 0x7fa8c40a52f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800b2e0 tx=0x7fa8c8005960 comp rx=0 tx=0).stop 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8dab09700 1 -- 172.21.15.104:0/3504683042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8d4101440 msgr2=0x7fa8d41021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8dab09700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8d4101440 0x7fa8d41021a0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa8c000e9a0 tx=0x7fa8c000ecb0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8dab09700 1 -- 172.21.15.104:0/3504683042 shutdown_connections 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8dab09700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa8c40a2e30 0x7fa8c40a52f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8dab09700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8d4108510 0x7fa8d41026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8dab09700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8d4100a80 0x7fa8d4101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8dab09700 1 --2- 172.21.15.104:0/3504683042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8d4101440 0x7fa8d41021a0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:04.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8dab09700 1 -- 172.21.15.104:0/3504683042 >> 172.21.15.104:0/3504683042 conn(0x7fa8d4077a30 msgr2=0x7fa8d40fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:04.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8dab09700 1 -- 172.21.15.104:0/3504683042 shutdown_connections 2024-06-09T22:27:04.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:04.675+0000 7fa8dab09700 1 -- 172.21.15.104:0/3504683042 wait complete. 2024-06-09T22:27:04.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.74 MB/sec execute 130 sec latency 92.684 ms 2024-06-09T22:27:05.660 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '6813a49', quiesce took 6.7 seconds. Will release after: 22.75 2024-06-09T22:27:05.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.71 MB/sec execute 131 sec latency 1035.501 ms 2024-06-09T22:27:05.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:05 smithi194 ceph-mon[36856]: pgmap v186: 129 pgs: 129 active+clean; 129 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 189 op/s 2024-06-09T22:27:05.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:05 smithi153 ceph-mon[36599]: pgmap v186: 129 pgs: 129 active+clean; 129 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 189 op/s 2024-06-09T22:27:06.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:05 smithi104 ceph-mon[32389]: pgmap v186: 129 pgs: 129 active+clean; 129 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 189 op/s 2024-06-09T22:27:06.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.67 MB/sec execute 132 sec latency 2035.581 ms 2024-06-09T22:27:07.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.64 MB/sec execute 133 sec latency 3035.637 ms 2024-06-09T22:27:07.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:07 smithi194 ceph-mon[36856]: pgmap v187: 129 pgs: 129 active+clean; 129 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-06-09T22:27:07.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:07 smithi153 ceph-mon[36599]: pgmap v187: 129 pgs: 129 active+clean; 129 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-06-09T22:27:08.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:07 smithi104 ceph-mon[32389]: pgmap v187: 129 pgs: 129 active+clean; 129 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-06-09T22:27:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.107+0000 7ff331f77640 1 -- 172.21.15.104:0/380057306 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32c105850 msgr2=0x7ff32c105c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.107+0000 7ff331f77640 1 --2- 172.21.15.104:0/380057306 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32c105850 0x7ff32c105c30 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009900 tx=0x7ff31c02a410 comp rx=0 tx=0).stop 2024-06-09T22:27:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.107+0000 7ff331f77640 1 -- 172.21.15.104:0/380057306 shutdown_connections 2024-06-09T22:27:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.107+0000 7ff331f77640 1 --2- 172.21.15.104:0/380057306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff32c1015b0 0x7ff32c10d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.107+0000 7ff331f77640 1 --2- 172.21.15.104:0/380057306 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff32c100bf0 0x7ff32c101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.107+0000 7ff331f77640 1 --2- 172.21.15.104:0/380057306 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32c105850 0x7ff32c105c30 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.107+0000 7ff331f77640 1 -- 172.21.15.104:0/380057306 >> 172.21.15.104:0/380057306 conn(0x7ff32c0766a0 msgr2=0x7ff32c076ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.107+0000 7ff331f77640 1 -- 172.21.15.104:0/380057306 shutdown_connections 2024-06-09T22:27:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.107+0000 7ff331f77640 1 -- 172.21.15.104:0/380057306 wait complete. 2024-06-09T22:27:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff331f77640 1 Processor -- start 2024-06-09T22:27:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff331f77640 1 -- start start 2024-06-09T22:27:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff331f77640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32c100bf0 0x7ff32c198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff331f77640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff32c1015b0 0x7ff32c199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff331f77640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff32c105850 0x7ff32c1a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff331f77640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c110c90 con 0x7ff32c100bf0 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff331f77640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c110b10 con 0x7ff32c1015b0 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff331f77640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c110e10 con 0x7ff32c105850 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff32b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32c100bf0 0x7ff32c198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff32b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32c100bf0 0x7ff32c198fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50242/0 (socket says 172.21.15.104:50242) 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff32b7fe640 1 -- 172.21.15.104:0/2952102233 learned_addr learned my addr 172.21.15.104:0/2952102233 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff32affd640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff32c1015b0 0x7ff32c199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.108+0000 7ff32bfff640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff32c105850 0x7ff32c1a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.109+0000 7ff32b7fe640 1 -- 172.21.15.104:0/2952102233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff32c105850 msgr2=0x7ff32c1a4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.109+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff32c105850 0x7ff32c1a4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.109+0000 7ff32b7fe640 1 -- 172.21.15.104:0/2952102233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff32c1015b0 msgr2=0x7ff32c199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.109+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff32c1015b0 0x7ff32c199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.109+0000 7ff32b7fe640 1 -- 172.21.15.104:0/2952102233 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32c1a4590 con 0x7ff32c100bf0 2024-06-09T22:27:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.109+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32c100bf0 0x7ff32c198fd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7ff31c006010 tx=0x7ff31c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.109+0000 7ff328ff9640 1 -- 172.21.15.104:0/2952102233 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c03f070 con 0x7ff32c100bf0 2024-06-09T22:27:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.109+0000 7ff331f77640 1 -- 172.21.15.104:0/2952102233 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff32c1a4760 con 0x7ff32c100bf0 2024-06-09T22:27:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.109+0000 7ff328ff9640 1 -- 172.21.15.104:0/2952102233 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff31c031c30 con 0x7ff32c100bf0 2024-06-09T22:27:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.109+0000 7ff328ff9640 1 -- 172.21.15.104:0/2952102233 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c035030 con 0x7ff32c100bf0 2024-06-09T22:27:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.109+0000 7ff331f77640 1 -- 172.21.15.104:0/2952102233 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff32c1a4bc0 con 0x7ff32c100bf0 2024-06-09T22:27:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.111+0000 7ff328ff9640 1 -- 172.21.15.104:0/2952102233 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ff31c042070 con 0x7ff32c100bf0 2024-06-09T22:27:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff334d63740 1 Processor -- start 2024-06-09T22:27:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff334d63740 1 -- start start 2024-06-09T22:27:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff334d63740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19e170 0x557f2d19e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff334d63740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d19ea90 0x557f2d1a8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff328ff9640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff3140a2da0 0x7ff3140a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff334d63740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d1a9350 0x557f2d1ab740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff334d63740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557f2d1024f0 con 0x557f2d19e170 2024-06-09T22:27:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff334d63740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557f2d0a0870 con 0x557f2d1a9350 2024-06-09T22:27:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff334d63740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557f2d0fe4b0 con 0x557f2d19ea90 2024-06-09T22:27:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff32bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19e170 0x557f2d19e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff32bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19e170 0x557f2d19e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50252/0 (socket says 172.21.15.104:50252) 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff32affd640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff3140a2da0 0x7ff3140a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff32bfff640 1 -- 172.21.15.104:0/4161451463 learned_addr learned my addr 172.21.15.104:0/4161451463 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/4161451463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d19ea90 0x557f2d1a8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff32bfff640 1 --2- 172.21.15.104:0/4161451463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d1a9350 0x557f2d1ab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.112+0000 7ff328ff9640 1 -- 172.21.15.104:0/2952102233 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff31c0eff70 con 0x7ff32c100bf0 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.113+0000 7ff32bfff640 1 -- 172.21.15.104:0/4161451463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d19ea90 msgr2=0x557f2d1a8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.113+0000 7ff32bfff640 1 --2- 172.21.15.104:0/4161451463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d19ea90 0x557f2d1a8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.113+0000 7ff32bfff640 1 -- 172.21.15.104:0/4161451463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d1a9350 msgr2=0x557f2d1ab740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.113+0000 7ff32bfff640 1 --2- 172.21.15.104:0/4161451463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d1a9350 0x557f2d1ab740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.113+0000 7ff32bfff640 1 -- 172.21.15.104:0/4161451463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f2d0bc9b0 con 0x557f2d19e170 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.113+0000 7ff32affd640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff3140a2da0 0x7ff3140a5260 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff32c06ad10 tx=0x7ff32000b390 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.113+0000 7ff32bfff640 1 --2- 172.21.15.104:0/4161451463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19e170 0x557f2d19e550 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7ff324005c20 tx=0x7ff3240099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df38dbe39ef8b15e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.113+0000 7ff311ffb640 1 -- 172.21.15.104:0/4161451463 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3240150f0 con 0x557f2d19e170 2024-06-09T22:27:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.113+0000 7ff311ffb640 1 -- 172.21.15.104:0/4161451463 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff324015290 con 0x557f2d19e170 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.113+0000 7ff311ffb640 1 -- 172.21.15.104:0/4161451463 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3240054c0 con 0x557f2d19e170 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.114+0000 7ff334d63740 1 -- 172.21.15.104:0/4161451463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19e170 msgr2=0x557f2d19e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.114+0000 7ff334d63740 1 --2- 172.21.15.104:0/4161451463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19e170 0x557f2d19e550 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7ff324005c20 tx=0x7ff3240099f0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.114+0000 7ff334d63740 1 -- 172.21.15.104:0/4161451463 shutdown_connections 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.114+0000 7ff334d63740 1 --2- 172.21.15.104:0/4161451463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d1a9350 0x557f2d1ab740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.114+0000 7ff334d63740 1 --2- 172.21.15.104:0/4161451463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d19ea90 0x557f2d1a8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.114+0000 7ff334d63740 1 --2- 172.21.15.104:0/4161451463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19e170 0x557f2d19e550 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.114+0000 7ff334d63740 1 -- 172.21.15.104:0/4161451463 >> 172.21.15.104:0/4161451463 conn(0x557f2d19a680 msgr2=0x557f2d19aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.114+0000 7ff334d63740 1 -- 172.21.15.104:0/4161451463 shutdown_connections 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.114+0000 7ff334d63740 1 -- 172.21.15.104:0/4161451463 wait complete. 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.114+0000 7ff334d63740 1 -- start start 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.114+0000 7ff334d63740 1 Processor -- start 2024-06-09T22:27:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff334d63740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d19e170 0x557f2d1a6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff334d63740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d1a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff334d63740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d1a9350 0x557f2d1a6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff334d63740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557f2d1abf20 con 0x557f2d19ea90 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff334d63740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557f2d08bae0 con 0x557f2d19e170 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff334d63740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557f2d1ac0a0 con 0x557f2d1a9350 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff334d63740 10 client.0 _ll_register_callbacks cb 0x557f2d0c49b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff32affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d1a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff32affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d1a67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50258/0 (socket says 172.21.15.104:50258) 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff32affd640 1 -- 172.21.15.104:0/2501877648 learned_addr learned my addr 172.21.15.104:0/2501877648 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff32bfff640 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d1a9350 0x557f2d1a6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d19e170 0x557f2d1a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff32affd640 1 -- 172.21.15.104:0/2501877648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d1a9350 msgr2=0x557f2d1a6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff32affd640 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d1a9350 0x557f2d1a6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff32affd640 1 -- 172.21.15.104:0/2501877648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d19e170 msgr2=0x557f2d1a6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff32affd640 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d19e170 0x557f2d1a6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff32affd640 1 -- 172.21.15.104:0/2501877648 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f2ce16210 con 0x557f2d19ea90 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d19e170 0x557f2d1a6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff32affd640 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d1a67b0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff320003ea0 tx=0x7ff320004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e806929c6efa78d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff30affd640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff30affd640 1 -- 172.21.15.104:0/2501877648 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff320005220 con 0x557f2d19ea90 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff30affd640 1 -- 172.21.15.104:0/2501877648 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff3200053c0 con 0x557f2d19ea90 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff334d63740 1 -- 172.21.15.104:0/2501877648 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff324015290 con 0x557f2d19ea90 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.115+0000 7ff30affd640 1 -- 172.21.15.104:0/2501877648 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff32002e050 con 0x557f2d19ea90 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.116+0000 7ff334d63740 10 client.17169 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.116+0000 7ff334d63740 1 -- 172.21.15.104:0/2501877648 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557f2cee2620 con 0x557f2d19ea90 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.116+0000 7ff30affd640 1 -- 172.21.15.104:0/2501877648 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff324015290 con 0x557f2d19ea90 2024-06-09T22:27:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.116+0000 7ff30affd640 1 -- 172.21.15.104:0/2501877648 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff32001e810 con 0x557f2d19ea90 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.116+0000 7ff334d63740 10 client.17169 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.116+0000 7ff334d63740 10 client.17169 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.116+0000 7ff334d63740 20 client.17169 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.116+0000 7ff334d63740 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f2d1b9470 0x557f2d1bb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.116+0000 7ff334d63740 4 client.17169 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.116+0000 7ff334d63740 1 -- 172.21.15.104:0/2501877648 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f2d115f10 con 0x557f2d1b9470 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.116+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f2d1b9470 0x557f2d1bb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.117+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f2d1b9470 0x557f2d1bb8e0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8993ae57d6e98474 server_cookie=34087ef3e36c4734 in_seq=0 out_seq=0 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.117+0000 7ff30affd640 10 client.17169 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.117+0000 7ff30affd640 1 -- 172.21.15.104:0/2501877648 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x557f2d115f10 con 0x557f2d1b9470 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.117+0000 7ff30affd640 10 client.17169 handle_command_reply: tid=0 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff334d63740 1 client.17169 shutdown 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff334d63740 1 -- 172.21.15.104:0/2501877648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 msgr2=0x557f2d1a67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff334d63740 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d1a67b0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff320003ea0 tx=0x7ff320004730 comp rx=0 tx=0).stop 2024-06-09T22:27:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff334d63740 1 -- 172.21.15.104:0/2501877648 shutdown_connections 2024-06-09T22:27:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff334d63740 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f2d1b9470 0x557f2d1bb8e0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff334d63740 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d1a9350 0x557f2d1a6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff334d63740 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d1a67b0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff334d63740 1 --2- 172.21.15.104:0/2501877648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d19e170 0x557f2d1a6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff30affd640 0 client.17169 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff334d63740 1 -- 172.21.15.104:0/2501877648 >> 172.21.15.104:0/2501877648 conn(0x557f2d19a680 msgr2=0x557f2d1a03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff334d63740 1 -- 172.21.15.104:0/2501877648 shutdown_connections 2024-06-09T22:27:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff334d63740 1 -- 172.21.15.104:0/2501877648 wait complete. 2024-06-09T22:27:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.118+0000 7ff334d63740 20 client.17169 trim_cache size 0 max 16384 2024-06-09T22:27:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.133+0000 7ff334d63740 1 Processor -- start 2024-06-09T22:27:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.133+0000 7ff334d63740 1 -- start start 2024-06-09T22:27:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.133+0000 7ff334d63740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d19e170 0x557f2d1a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.133+0000 7ff334d63740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d1a81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.133+0000 7ff334d63740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d1a9350 0x557f2d1b16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.133+0000 7ff334d63740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1aed60 con 0x557f2d19ea90 2024-06-09T22:27:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.133+0000 7ff334d63740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1b08a0 con 0x557f2d1a9350 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.133+0000 7ff334d63740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1af430 con 0x557f2d19e170 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.133+0000 7ff32affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d1a81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.133+0000 7ff32affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d1a81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50262/0 (socket says 172.21.15.104:50262) 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.133+0000 7ff32affd640 1 -- 172.21.15.104:0/2143045625 learned_addr learned my addr 172.21.15.104:0/2143045625 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.134+0000 7ff32bfff640 1 --2- 172.21.15.104:0/2143045625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d1a9350 0x557f2d1b16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.134+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/2143045625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d19e170 0x557f2d1a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.134+0000 7ff32affd640 1 -- 172.21.15.104:0/2143045625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d19e170 msgr2=0x557f2d1a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.134+0000 7ff32affd640 1 --2- 172.21.15.104:0/2143045625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d19e170 0x557f2d1a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.134+0000 7ff32affd640 1 -- 172.21.15.104:0/2143045625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d1a9350 msgr2=0x557f2d1b16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.134+0000 7ff32affd640 1 --2- 172.21.15.104:0/2143045625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d1a9350 0x557f2d1b16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.134+0000 7ff32affd640 1 -- 172.21.15.104:0/2143045625 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f2d19f100 con 0x557f2d19ea90 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.134+0000 7ff32bfff640 1 --2- 172.21.15.104:0/2143045625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d1a9350 0x557f2d1b16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.134+0000 7ff32affd640 1 --2- 172.21.15.104:0/2143045625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d1a81b0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff32000bf60 tx=0x7ff320004650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36fff8eee260fdcd server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.134+0000 7ff3117fa640 1 -- 172.21.15.104:0/2143045625 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff320005240 con 0x557f2d19ea90 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.134+0000 7ff3117fa640 1 -- 172.21.15.104:0/2143045625 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff32001e900 con 0x557f2d19ea90 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.134+0000 7ff3117fa640 1 -- 172.21.15.104:0/2143045625 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3200049e0 con 0x557f2d19ea90 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 -- 172.21.15.104:0/2143045625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 msgr2=0x557f2d1a81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 --2- 172.21.15.104:0/2143045625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d1a81b0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff32000bf60 tx=0x7ff320004650 comp rx=0 tx=0).stop 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 -- 172.21.15.104:0/2143045625 shutdown_connections 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 --2- 172.21.15.104:0/2143045625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d1a9350 0x557f2d1b16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 --2- 172.21.15.104:0/2143045625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d1a81b0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 --2- 172.21.15.104:0/2143045625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d19e170 0x557f2d1a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 -- 172.21.15.104:0/2143045625 >> 172.21.15.104:0/2143045625 conn(0x557f2d19a680 msgr2=0x557f2d1a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 -- 172.21.15.104:0/2143045625 shutdown_connections 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 -- 172.21.15.104:0/2143045625 wait complete. 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 -- start start 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 Processor -- start 2024-06-09T22:27:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d19e170 0x557f2d21c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d21c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.135+0000 7ff334d63740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d1a9350 0x557f2d227200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff334d63740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1b01d0 con 0x557f2d19ea90 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff334d63740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1afb00 con 0x557f2d19e170 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff334d63740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1acd20 con 0x557f2d1a9350 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff334d63740 10 client.0 _ll_register_callbacks cb 0x7ff320004360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff32affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d21c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff32affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d21c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50264/0 (socket says 172.21.15.104:50264) 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff32affd640 1 -- 172.21.15.104:0/3629173041 learned_addr learned my addr 172.21.15.104:0/3629173041 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d19e170 0x557f2d21c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff32bfff640 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d1a9350 0x557f2d227200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff32affd640 1 -- 172.21.15.104:0/3629173041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d1a9350 msgr2=0x557f2d227200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff32affd640 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d1a9350 0x557f2d227200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff32affd640 1 -- 172.21.15.104:0/3629173041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d19e170 msgr2=0x557f2d21c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff32affd640 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d19e170 0x557f2d21c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff32affd640 1 -- 172.21.15.104:0/3629173041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f2d21cfb0 con 0x557f2d19ea90 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d19e170 0x557f2d21c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff32affd640 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d21c6c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7ff32000bfd0 tx=0x7ff32001e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a1cbef7823eb4d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff30affd640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff30affd640 1 -- 172.21.15.104:0/3629173041 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32002b040 con 0x557f2d19ea90 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff30affd640 1 -- 172.21.15.104:0/3629173041 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7ff32003b840 con 0x557f2d19ea90 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff334d63740 1 -- 172.21.15.104:0/3629173041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff32001e900 con 0x557f2d19ea90 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff30affd640 1 -- 172.21.15.104:0/3629173041 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff32004b050 con 0x557f2d19ea90 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff334d63740 10 client.17175 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.136+0000 7ff334d63740 1 -- 172.21.15.104:0/3629173041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557f2d228850 con 0x557f2d19ea90 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.137+0000 7ff30affd640 1 -- 172.21.15.104:0/3629173041 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff32001e900 con 0x557f2d19ea90 2024-06-09T22:27:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.137+0000 7ff30affd640 1 -- 172.21.15.104:0/3629173041 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff320032990 con 0x557f2d19ea90 2024-06-09T22:27:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.137+0000 7ff334d63740 10 client.17175 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.137+0000 7ff334d63740 10 client.17175 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.137+0000 7ff334d63740 20 client.17175 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.137+0000 7ff334d63740 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f2d228e60 0x557f2d22b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.137+0000 7ff334d63740 4 client.17175 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:27:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.137+0000 7ff334d63740 1 -- 172.21.15.104:0/3629173041 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557f2d10fc20 con 0x557f2d228e60 2024-06-09T22:27:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.137+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f2d228e60 0x557f2d22b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.137+0000 7ff32b7fe640 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f2d228e60 0x557f2d22b280 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83587b983ed4d435 server_cookie=98f44349350c782f in_seq=0 out_seq=0 2024-06-09T22:27:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.137+0000 7ff30affd640 10 client.17175 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.148+0000 7ff30affd640 1 -- 172.21.15.104:0/3629173041 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x557f2d10fc20 con 0x557f2d228e60 2024-06-09T22:27:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.148+0000 7ff30affd640 10 client.17175 handle_command_reply: tid=0 2024-06-09T22:27:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.148+0000 7ff334d63740 1 client.17175 shutdown 2024-06-09T22:27:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.148+0000 7ff334d63740 1 -- 172.21.15.104:0/3629173041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 msgr2=0x557f2d21c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.148+0000 7ff334d63740 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d21c6c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7ff32000bfd0 tx=0x7ff32001e6d0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.148+0000 7ff334d63740 1 -- 172.21.15.104:0/3629173041 shutdown_connections 2024-06-09T22:27:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.148+0000 7ff334d63740 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f2d228e60 0x557f2d22b280 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.148+0000 7ff334d63740 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f2d1a9350 0x557f2d227200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.148+0000 7ff30affd640 0 client.17175 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.148+0000 7ff334d63740 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f2d19ea90 0x557f2d21c6c0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.148+0000 7ff334d63740 1 --2- 172.21.15.104:0/3629173041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f2d19e170 0x557f2d21c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.148+0000 7ff334d63740 1 -- 172.21.15.104:0/3629173041 >> 172.21.15.104:0/3629173041 conn(0x557f2d19a680 msgr2=0x557f2d1a8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff334d63740 1 -- 172.21.15.104:0/3629173041 shutdown_connections 2024-06-09T22:27:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff334d63740 1 -- 172.21.15.104:0/3629173041 wait complete. 2024-06-09T22:27:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff334d63740 20 client.17175 trim_cache size 0 max 16384 2024-06-09T22:27:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:27:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:27:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:27:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff331f77640 1 -- 172.21.15.104:0/2952102233 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff3140a2da0 msgr2=0x7ff3140a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff331f77640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff3140a2da0 0x7ff3140a5260 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff32c06ad10 tx=0x7ff32000b390 comp rx=0 tx=0).stop 2024-06-09T22:27:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff331f77640 1 -- 172.21.15.104:0/2952102233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32c100bf0 msgr2=0x7ff32c198fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff331f77640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32c100bf0 0x7ff32c198fd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7ff31c006010 tx=0x7ff31c003c10 comp rx=0 tx=0).stop 2024-06-09T22:27:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff331f77640 1 -- 172.21.15.104:0/2952102233 shutdown_connections 2024-06-09T22:27:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff331f77640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7ff3140a2da0 0x7ff3140a5260 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff331f77640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff32c105850 0x7ff32c1a4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff331f77640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff32c1015b0 0x7ff32c199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff331f77640 1 --2- 172.21.15.104:0/2952102233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32c100bf0 0x7ff32c198fd0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff331f77640 1 -- 172.21.15.104:0/2952102233 >> 172.21.15.104:0/2952102233 conn(0x7ff32c0766a0 msgr2=0x7ff32c0fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff331f77640 1 -- 172.21.15.104:0/2952102233 shutdown_connections 2024-06-09T22:27:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:08.149+0000 7ff331f77640 1 -- 172.21.15.104:0/2952102233 wait complete. 2024-06-09T22:27:08.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.60 MB/sec execute 134 sec latency 4035.693 ms 2024-06-09T22:27:09.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.57 MB/sec execute 135 sec latency 5035.732 ms 2024-06-09T22:27:09.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:09 smithi153 ceph-mon[36599]: pgmap v188: 129 pgs: 129 active+clean; 129 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-06-09T22:27:09.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:09 smithi194 ceph-mon[36856]: pgmap v188: 129 pgs: 129 active+clean; 129 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-06-09T22:27:10.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:09 smithi104 ceph-mon[32389]: pgmap v188: 129 pgs: 129 active+clean; 129 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-06-09T22:27:10.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.53 MB/sec execute 136 sec latency 6035.819 ms 2024-06-09T22:27:11.733 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.50 MB/sec execute 137 sec latency 7035.869 ms 2024-06-09T22:27:11.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:11 smithi194 ceph-mon[36856]: pgmap v189: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 196 op/s 2024-06-09T22:27:11.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:11 smithi153 ceph-mon[36599]: pgmap v189: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 196 op/s 2024-06-09T22:27:12.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:11 smithi104 ceph-mon[32389]: pgmap v189: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 196 op/s 2024-06-09T22:27:12.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.47 MB/sec execute 138 sec latency 8035.978 ms 2024-06-09T22:27:13.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:27:13.332 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:27:13.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.44 MB/sec execute 139 sec latency 9036.047 ms 2024-06-09T22:27:13.758 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:13 smithi104 ceph-mon[32389]: pgmap v190: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-06-09T22:27:13.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:13 smithi194 ceph-mon[36856]: pgmap v190: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-06-09T22:27:13.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:13 smithi153 ceph-mon[36599]: pgmap v190: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-06-09T22:27:14.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.40 MB/sec execute 140 sec latency 10036.162 ms 2024-06-09T22:27:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.358+0000 7fb81757d700 1 -- 172.21.15.104:0/2937351811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb810107570 msgr2=0x7fb810107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.358+0000 7fb81757d700 1 --2- 172.21.15.104:0/2937351811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb810107570 0x7fb810107950 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009990 tx=0x7fb80c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.358+0000 7fb81757d700 1 -- 172.21.15.104:0/2937351811 shutdown_connections 2024-06-09T22:27:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.358+0000 7fb81757d700 1 --2- 172.21.15.104:0/2937351811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb810107570 0x7fb810107950 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.358+0000 7fb81757d700 1 --2- 172.21.15.104:0/2937351811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb810101810 0x7fb81010f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.358+0000 7fb81757d700 1 --2- 172.21.15.104:0/2937351811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb810100e50 0x7fb8101012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.358+0000 7fb81757d700 1 -- 172.21.15.104:0/2937351811 >> 172.21.15.104:0/2937351811 conn(0x7fb8100766b0 msgr2=0x7fb810076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.358+0000 7fb81757d700 1 -- 172.21.15.104:0/2937351811 shutdown_connections 2024-06-09T22:27:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.358+0000 7fb81757d700 1 -- 172.21.15.104:0/2937351811 wait complete. 2024-06-09T22:27:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb81757d700 1 Processor -- start 2024-06-09T22:27:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb81757d700 1 -- start start 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb81757d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb810100e50 0x7fb81019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb81757d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb810101810 0x7fb81019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb81757d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb810107570 0x7fb810198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb81757d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb8101127b0 con 0x7fb810100e50 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb81757d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb810112630 con 0x7fb810107570 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb81757d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb810112930 con 0x7fb810101810 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb815319700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb810100e50 0x7fb81019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb815319700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb810100e50 0x7fb81019e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50278/0 (socket says 172.21.15.104:50278) 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb815319700 1 -- 172.21.15.104:0/2661077560 learned_addr learned my addr 172.21.15.104:0/2661077560 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb814b18700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb810101810 0x7fb81019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb815b1a700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb810107570 0x7fb810198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb815319700 1 -- 172.21.15.104:0/2661077560 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb810101810 msgr2=0x7fb81019edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.359+0000 7fb815319700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb810101810 0x7fb81019edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.360+0000 7fb815319700 1 -- 172.21.15.104:0/2661077560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb810107570 msgr2=0x7fb810198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.360+0000 7fb815319700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb810107570 0x7fb810198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.360+0000 7fb815319700 1 -- 172.21.15.104:0/2661077560 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb810199190 con 0x7fb810100e50 2024-06-09T22:27:15.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.360+0000 7fb815319700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb810100e50 0x7fb81019e870 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb80c000c00 tx=0x7fb80c00bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:15.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.360+0000 7fb8067fc700 1 -- 172.21.15.104:0/2661077560 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c027070 con 0x7fb810100e50 2024-06-09T22:27:15.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.360+0000 7fb81757d700 1 -- 172.21.15.104:0/2661077560 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb810199420 con 0x7fb810100e50 2024-06-09T22:27:15.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.360+0000 7fb8067fc700 1 -- 172.21.15.104:0/2661077560 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb80c022470 con 0x7fb810100e50 2024-06-09T22:27:15.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.360+0000 7fb8067fc700 1 -- 172.21.15.104:0/2661077560 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c00f6c0 con 0x7fb810100e50 2024-06-09T22:27:15.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.360+0000 7fb81757d700 1 -- 172.21.15.104:0/2661077560 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8101a5740 con 0x7fb810100e50 2024-06-09T22:27:15.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.363+0000 7fb8067fc700 1 -- 172.21.15.104:0/2661077560 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fb80c02a070 con 0x7fb810100e50 2024-06-09T22:27:15.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.363+0000 7fb8067fc700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb7fc0a2d90 0x7fb7fc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.364+0000 7fb8067fc700 1 -- 172.21.15.104:0/2661077560 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb80c0d07f0 con 0x7fb810100e50 2024-06-09T22:27:15.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.364+0000 7fb814b18700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb7fc0a2d90 0x7fb7fc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.364+0000 7fb814b18700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb7fc0a2d90 0x7fb7fc0a5250 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb800005950 tx=0x7fb8000058e0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.364+0000 7fb821e00b80 1 Processor -- start 2024-06-09T22:27:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.364+0000 7fb821e00b80 1 -- start start 2024-06-09T22:27:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb821e00b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733dfd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb821e00b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733dfdc90 0x55b733df7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb821e00b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 0x55b733df7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb821e00b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b733ce5f60 con 0x55b733df7bb0 2024-06-09T22:27:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb821e00b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b733d44850 con 0x55b733dfd370 2024-06-09T22:27:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb821e00b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b733cac460 con 0x55b733dfdc90 2024-06-09T22:27:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb815b1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 0x55b733df7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb815b1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 0x55b733df7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50290/0 (socket says 172.21.15.104:50290) 2024-06-09T22:27:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb815b1a700 1 -- 172.21.15.104:0/2838901860 learned_addr learned my addr 172.21.15.104:0/2838901860 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb814b18700 1 --2- 172.21.15.104:0/2838901860 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733dfdc90 0x55b733df7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb815319700 1 --2- 172.21.15.104:0/2838901860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733dfd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb815319700 1 -- 172.21.15.104:0/2838901860 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733dfdc90 msgr2=0x55b733df7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb815319700 1 --2- 172.21.15.104:0/2838901860 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733dfdc90 0x55b733df7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb815319700 1 -- 172.21.15.104:0/2838901860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 msgr2=0x55b733df7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb815319700 1 --2- 172.21.15.104:0/2838901860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 0x55b733df7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.365+0000 7fb815319700 1 -- 172.21.15.104:0/2838901860 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b733c938d0 con 0x55b733dfd370 2024-06-09T22:27:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.366+0000 7fb815319700 1 --2- 172.21.15.104:0/2838901860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733dfd750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0cdbd0 tx=0x7fb80c09c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fb0e920506fb91d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:15.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.366+0000 7fb7fb7fe700 1 -- 172.21.15.104:0/2838901860 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c09ca30 con 0x55b733dfd370 2024-06-09T22:27:15.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.366+0000 7fb7fb7fe700 1 -- 172.21.15.104:0/2838901860 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb80c0a0de0 con 0x55b733dfd370 2024-06-09T22:27:15.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.366+0000 7fb7fb7fe700 1 -- 172.21.15.104:0/2838901860 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c0a30c0 con 0x55b733dfd370 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.366+0000 7fb821e00b80 1 -- 172.21.15.104:0/2838901860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 msgr2=0x55b733dfd750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.366+0000 7fb821e00b80 1 --2- 172.21.15.104:0/2838901860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733dfd750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0cdbd0 tx=0x7fb80c09c5f0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 -- 172.21.15.104:0/2838901860 shutdown_connections 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 --2- 172.21.15.104:0/2838901860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 0x55b733df7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 --2- 172.21.15.104:0/2838901860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733dfd750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 --2- 172.21.15.104:0/2838901860 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733dfdc90 0x55b733df7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 -- 172.21.15.104:0/2838901860 >> 172.21.15.104:0/2838901860 conn(0x55b733d20080 msgr2=0x55b733dee220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 -- 172.21.15.104:0/2838901860 shutdown_connections 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 -- 172.21.15.104:0/2838901860 wait complete. 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 -- start start 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 Processor -- start 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733df7bb0 0x55b733e03960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733e00d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 0x55b733e01240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b733ce4fc0 con 0x55b733dfdc90 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b733d40810 con 0x55b733dfd370 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.367+0000 7fb821e00b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b733e04430 con 0x55b733df7bb0 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb821e00b80 10 client.0 _ll_register_callbacks cb 0x55b733cda5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb815b1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 0x55b733e01240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb815b1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 0x55b733e01240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50300/0 (socket says 172.21.15.104:50300) 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb815b1a700 1 -- 172.21.15.104:0/85610217 learned_addr learned my addr 172.21.15.104:0/85610217 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb815319700 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733df7bb0 0x55b733e03960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb814b18700 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733e00d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb815b1a700 1 -- 172.21.15.104:0/85610217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733df7bb0 msgr2=0x55b733e03960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb815b1a700 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733df7bb0 0x55b733e03960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb815b1a700 1 -- 172.21.15.104:0/85610217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 msgr2=0x55b733e00d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:15.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb815b1a700 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733e00d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb815b1a700 1 -- 172.21.15.104:0/85610217 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b733d0a750 con 0x55b733dfdc90 2024-06-09T22:27:15.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb814b18700 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733e00d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:27:15.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb815b1a700 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 0x55b733e01240 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb808009550 tx=0x7fb808007b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffc46ef673940f3e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:15.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb7f8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:15.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb7f8ff9700 1 -- 172.21.15.104:0/85610217 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80801a030 con 0x55b733dfdc90 2024-06-09T22:27:15.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb821e00b80 1 -- 172.21.15.104:0/85610217 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb80c0a0de0 con 0x55b733dfdc90 2024-06-09T22:27:15.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb7f8ff9700 1 -- 172.21.15.104:0/85610217 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb808003a00 con 0x55b733dfdc90 2024-06-09T22:27:15.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb7f8ff9700 1 -- 172.21.15.104:0/85610217 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb808026050 con 0x55b733dfdc90 2024-06-09T22:27:15.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb821e00b80 10 client.17187 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:15.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.368+0000 7fb821e00b80 1 -- 172.21.15.104:0/85610217 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b733e0eb50 con 0x55b733dfdc90 2024-06-09T22:27:15.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.369+0000 7fb7f8ff9700 1 -- 172.21.15.104:0/85610217 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb80c0a0de0 con 0x55b733dfdc90 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.369+0000 7fb7f8ff9700 1 -- 172.21.15.104:0/85610217 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb808004190 con 0x55b733dfdc90 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.369+0000 7fb821e00b80 10 client.17187 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.369+0000 7fb821e00b80 10 client.17187 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.369+0000 7fb821e00b80 20 client.17187 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.369+0000 7fb821e00b80 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b733e11180 0x55b733e13590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.369+0000 7fb821e00b80 4 client.17187 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.369+0000 7fb821e00b80 1 -- 172.21.15.104:0/85610217 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b733d51d50 con 0x55b733e11180 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.369+0000 7fb815319700 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b733e11180 0x55b733e13590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.370+0000 7fb815319700 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b733e11180 0x55b733e13590 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4643a106eb2007f server_cookie=6f6064a2855df8f6 in_seq=0 out_seq=0 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.370+0000 7fb7f8ff9700 10 client.17187 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.370+0000 7fb7f8ff9700 1 -- 172.21.15.104:0/85610217 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b733d51d50 con 0x55b733e11180 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.370+0000 7fb7f8ff9700 10 client.17187 handle_command_reply: tid=0 2024-06-09T22:27:15.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.370+0000 7fb821e00b80 1 client.17187 shutdown 2024-06-09T22:27:15.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.371+0000 7fb821e00b80 1 -- 172.21.15.104:0/85610217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 msgr2=0x55b733e01240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:15.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.371+0000 7fb821e00b80 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 0x55b733e01240 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb808009550 tx=0x7fb808007b30 comp rx=0 tx=0).stop 2024-06-09T22:27:15.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.371+0000 7fb821e00b80 1 -- 172.21.15.104:0/85610217 shutdown_connections 2024-06-09T22:27:15.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.371+0000 7fb821e00b80 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b733e11180 0x55b733e13590 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.371+0000 7fb821e00b80 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 0x55b733e01240 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.371+0000 7fb821e00b80 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733df7bb0 0x55b733e03960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.371+0000 7fb7f8ff9700 0 client.17187 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:15.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.371+0000 7fb821e00b80 1 --2- 172.21.15.104:0/85610217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733e00d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.371+0000 7fb821e00b80 1 -- 172.21.15.104:0/85610217 >> 172.21.15.104:0/85610217 conn(0x55b733d20080 msgr2=0x55b733ded700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:15.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.371+0000 7fb821e00b80 1 -- 172.21.15.104:0/85610217 shutdown_connections 2024-06-09T22:27:15.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.371+0000 7fb821e00b80 1 -- 172.21.15.104:0/85610217 wait complete. 2024-06-09T22:27:15.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.371+0000 7fb821e00b80 20 client.17187 trim_cache size 0 max 16384 2024-06-09T22:27:15.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.392+0000 7fb821e00b80 1 Processor -- start 2024-06-09T22:27:15.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.392+0000 7fb821e00b80 1 -- start start 2024-06-09T22:27:15.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.392+0000 7fb821e00b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733df7bb0 0x55b733dfb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.392+0000 7fb821e00b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733dfe780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.392+0000 7fb821e00b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 0x55b733dede80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.392+0000 7fb821e00b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb8101ac210 con 0x55b733dfdc90 2024-06-09T22:27:15.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.392+0000 7fb821e00b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb8101add50 con 0x55b733dfd370 2024-06-09T22:27:15.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.392+0000 7fb821e00b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb8101ac8e0 con 0x55b733df7bb0 2024-06-09T22:27:15.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.392+0000 7fb815b1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 0x55b733dede80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.392+0000 7fb815b1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 0x55b733dede80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50308/0 (socket says 172.21.15.104:50308) 2024-06-09T22:27:15.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.392+0000 7fb815b1a700 1 -- 172.21.15.104:0/2861856420 learned_addr learned my addr 172.21.15.104:0/2861856420 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:15.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.392+0000 7fb815319700 1 --2- 172.21.15.104:0/2861856420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733df7bb0 0x55b733dfb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.393+0000 7fb814b18700 1 --2- 172.21.15.104:0/2861856420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733dfe780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.393+0000 7fb815b1a700 1 -- 172.21.15.104:0/2861856420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733df7bb0 msgr2=0x55b733dfb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:15.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.393+0000 7fb815b1a700 1 --2- 172.21.15.104:0/2861856420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733df7bb0 0x55b733dfb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.393+0000 7fb815b1a700 1 -- 172.21.15.104:0/2861856420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 msgr2=0x55b733dfe780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:15.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.393+0000 7fb815b1a700 1 --2- 172.21.15.104:0/2861856420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733dfe780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.393+0000 7fb815b1a700 1 -- 172.21.15.104:0/2861856420 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b733dfecc0 con 0x55b733dfdc90 2024-06-09T22:27:15.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.393+0000 7fb815b1a700 1 --2- 172.21.15.104:0/2861856420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 0x55b733dede80 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb8080049e0 tx=0x7fb8080021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e1f7855cfb9bc3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:15.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.393+0000 7fb7faffd700 1 -- 172.21.15.104:0/2861856420 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80801a040 con 0x55b733dfdc90 2024-06-09T22:27:15.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.393+0000 7fb7faffd700 1 -- 172.21.15.104:0/2861856420 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb808004c90 con 0x55b733dfdc90 2024-06-09T22:27:15.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.393+0000 7fb7faffd700 1 -- 172.21.15.104:0/2861856420 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb808003cf0 con 0x55b733dfdc90 2024-06-09T22:27:15.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.394+0000 7fb821e00b80 1 -- 172.21.15.104:0/2861856420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 msgr2=0x55b733dede80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:15.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.394+0000 7fb821e00b80 1 --2- 172.21.15.104:0/2861856420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 0x55b733dede80 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb8080049e0 tx=0x7fb8080021d0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.394+0000 7fb821e00b80 1 -- 172.21.15.104:0/2861856420 shutdown_connections 2024-06-09T22:27:15.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.394+0000 7fb821e00b80 1 --2- 172.21.15.104:0/2861856420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733dfdc90 0x55b733dede80 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.394+0000 7fb821e00b80 1 --2- 172.21.15.104:0/2861856420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733df7bb0 0x55b733dfb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.394+0000 7fb821e00b80 1 --2- 172.21.15.104:0/2861856420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfd370 0x55b733dfe780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.394+0000 7fb821e00b80 1 -- 172.21.15.104:0/2861856420 >> 172.21.15.104:0/2861856420 conn(0x55b733d20080 msgr2=0x55b733dfe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:15.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.394+0000 7fb821e00b80 1 -- 172.21.15.104:0/2861856420 shutdown_connections 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.394+0000 7fb821e00b80 1 -- 172.21.15.104:0/2861856420 wait complete. 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.394+0000 7fb821e00b80 1 -- start start 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.394+0000 7fb821e00b80 1 Processor -- start 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb821e00b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 0x55b733e09e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb821e00b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733dfd370 0x55b733e18890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb821e00b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfdc90 0x55b733e18dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb821e00b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb8101ad680 con 0x55b733df7bb0 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb821e00b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb8101acfb0 con 0x55b733dfdc90 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb821e00b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb8101aa190 con 0x55b733dfd370 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb821e00b80 10 client.0 _ll_register_callbacks cb 0x55b733cda5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb815319700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 0x55b733e09e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb815319700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 0x55b733e09e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50322/0 (socket says 172.21.15.104:50322) 2024-06-09T22:27:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb815319700 1 -- 172.21.15.104:0/1715439348 learned_addr learned my addr 172.21.15.104:0/1715439348 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb815b1a700 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfdc90 0x55b733e18dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb814b18700 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733dfd370 0x55b733e18890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb815319700 1 -- 172.21.15.104:0/1715439348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733dfd370 msgr2=0x55b733e18890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb815319700 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733dfd370 0x55b733e18890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb815319700 1 -- 172.21.15.104:0/1715439348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfdc90 msgr2=0x55b733e18dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb815319700 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfdc90 0x55b733e18dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb815319700 1 -- 172.21.15.104:0/1715439348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b733e19640 con 0x55b733df7bb0 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb815b1a700 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfdc90 0x55b733e18dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.395+0000 7fb814b18700 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733dfd370 0x55b733e18890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb815319700 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 0x55b733e09e10 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb80c098710 tx=0x7fb80c0a3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3ddf3eb1233ab1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb7f8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb7f8ff9700 1 -- 172.21.15.104:0/1715439348 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c09c570 con 0x55b733df7bb0 2024-06-09T22:27:15.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb7f8ff9700 1 -- 172.21.15.104:0/1715439348 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fb80c0bb200 con 0x55b733df7bb0 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb821e00b80 1 -- 172.21.15.104:0/1715439348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb808004c90 con 0x55b733df7bb0 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb7f8ff9700 1 -- 172.21.15.104:0/1715439348 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb80c0a0de0 con 0x55b733df7bb0 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb821e00b80 10 client.17193 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb821e00b80 1 -- 172.21.15.104:0/1715439348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b733e7bf00 con 0x55b733df7bb0 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb7f8ff9700 1 -- 172.21.15.104:0/1715439348 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb808004c90 con 0x55b733df7bb0 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb7f8ff9700 1 -- 172.21.15.104:0/1715439348 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb80c0d04b0 con 0x55b733df7bb0 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb821e00b80 10 client.17193 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb821e00b80 10 client.17193 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb821e00b80 20 client.17193 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb821e00b80 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b733e11180 0x55b733e7e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb821e00b80 4 client.17193 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.396+0000 7fb821e00b80 1 -- 172.21.15.104:0/1715439348 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b733d4c2d0 con 0x55b733e11180 2024-06-09T22:27:15.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb814b18700 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b733e11180 0x55b733e7e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb7f8ff9700 10 client.17193 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb814b18700 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b733e11180 0x55b733e7e480 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7193d8f67cc49780 server_cookie=d26f4988c6cfc4f2 in_seq=0 out_seq=0 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb7f8ff9700 1 -- 172.21.15.104:0/1715439348 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b733d4c2d0 con 0x55b733e11180 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb7f8ff9700 10 client.17193 handle_command_reply: tid=0 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb821e00b80 1 client.17193 shutdown 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb821e00b80 1 -- 172.21.15.104:0/1715439348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 msgr2=0x55b733e09e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb821e00b80 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 0x55b733e09e10 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb80c098710 tx=0x7fb80c0a3040 comp rx=0 tx=0).stop 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb821e00b80 1 -- 172.21.15.104:0/1715439348 shutdown_connections 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb821e00b80 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b733e11180 0x55b733e7e480 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb821e00b80 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b733df7bb0 0x55b733e09e10 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb821e00b80 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b733dfdc90 0x55b733e18dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb821e00b80 1 --2- 172.21.15.104:0/1715439348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b733dfd370 0x55b733e18890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb821e00b80 1 -- 172.21.15.104:0/1715439348 >> 172.21.15.104:0/1715439348 conn(0x55b733d20080 msgr2=0x55b733df2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:15.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.397+0000 7fb7f8ff9700 0 client.17193 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:15.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb821e00b80 1 -- 172.21.15.104:0/1715439348 shutdown_connections 2024-06-09T22:27:15.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb821e00b80 1 -- 172.21.15.104:0/1715439348 wait complete. 2024-06-09T22:27:15.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb821e00b80 20 client.17193 trim_cache size 0 max 16384 2024-06-09T22:27:15.405 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:27:15.405 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "no active scrubs running", 2024-06-09T22:27:15.405 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": {} 2024-06-09T22:27:15.405 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:27:15.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb81757d700 1 -- 172.21.15.104:0/2661077560 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb7fc0a2d90 msgr2=0x7fb7fc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:15.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb81757d700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb7fc0a2d90 0x7fb7fc0a5250 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb800005950 tx=0x7fb8000058e0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb81757d700 1 -- 172.21.15.104:0/2661077560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb810100e50 msgr2=0x7fb81019e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:15.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb81757d700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb810100e50 0x7fb81019e870 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb80c000c00 tx=0x7fb80c00bb60 comp rx=0 tx=0).stop 2024-06-09T22:27:15.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb81757d700 1 -- 172.21.15.104:0/2661077560 shutdown_connections 2024-06-09T22:27:15.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb81757d700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb7fc0a2d90 0x7fb7fc0a5250 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb81757d700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb810100e50 0x7fb81019e870 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb81757d700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb810107570 0x7fb810198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb81757d700 1 --2- 172.21.15.104:0/2661077560 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb810101810 0x7fb81019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:15.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.398+0000 7fb81757d700 1 -- 172.21.15.104:0/2661077560 >> 172.21.15.104:0/2661077560 conn(0x7fb8100766b0 msgr2=0x7fb8100ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:15.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.399+0000 7fb81757d700 1 -- 172.21.15.104:0/2661077560 shutdown_connections 2024-06-09T22:27:15.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:15.399+0000 7fb81757d700 1 -- 172.21.15.104:0/2661077560 wait complete. 2024-06-09T22:27:15.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.37 MB/sec execute 141 sec latency 11036.210 ms 2024-06-09T22:27:15.758 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:15 smithi104 ceph-mon[32389]: pgmap v191: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-06-09T22:27:15.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:15 smithi194 ceph-mon[36856]: pgmap v191: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-06-09T22:27:15.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:15 smithi153 ceph-mon[36599]: pgmap v191: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-06-09T22:27:16.167 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-09T22:27:16.167 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:27:16.412 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:27:16.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.34 MB/sec execute 142 sec latency 12036.248 ms 2024-06-09T22:27:17.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.31 MB/sec execute 143 sec latency 13036.329 ms 2024-06-09T22:27:17.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:17 smithi194 ceph-mon[36856]: pgmap v192: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 218 KiB/s wr, 39 op/s 2024-06-09T22:27:17.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:17 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:27:17.926 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:17 smithi104 ceph-mon[32389]: pgmap v192: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 218 KiB/s wr, 39 op/s 2024-06-09T22:27:17.926 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:17 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:27:17.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:17 smithi153 ceph-mon[36599]: pgmap v192: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 218 KiB/s wr, 39 op/s 2024-06-09T22:27:17.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:17 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:27:18.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.361+0000 7f781aaf3700 1 -- 172.21.15.104:0/2025542246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7814101d30 msgr2=0x7f781410e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:18.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.361+0000 7f781aaf3700 1 --2- 172.21.15.104:0/2025542246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7814101d30 0x7f781410e1d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f780c00b0a0 tx=0x7f780c00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:27:18.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.361+0000 7f781aaf3700 1 -- 172.21.15.104:0/2025542246 shutdown_connections 2024-06-09T22:27:18.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.361+0000 7f781aaf3700 1 --2- 172.21.15.104:0/2025542246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7814101d30 0x7f781410e1d0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:18.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.361+0000 7f781aaf3700 1 --2- 172.21.15.104:0/2025542246 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7814107420 0x7f7814107800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:18.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.361+0000 7f781aaf3700 1 --2- 172.21.15.104:0/2025542246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7814101370 0x7f78141017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:18.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.361+0000 7f781aaf3700 1 -- 172.21.15.104:0/2025542246 >> 172.21.15.104:0/2025542246 conn(0x7f78140783a0 msgr2=0x7f78140787b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:18.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.361+0000 7f781aaf3700 1 -- 172.21.15.104:0/2025542246 shutdown_connections 2024-06-09T22:27:18.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.361+0000 7f781aaf3700 1 -- 172.21.15.104:0/2025542246 wait complete. 2024-06-09T22:27:18.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.362+0000 7f781aaf3700 1 Processor -- start 2024-06-09T22:27:18.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.362+0000 7f781aaf3700 1 -- start start 2024-06-09T22:27:18.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.362+0000 7f781aaf3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7814101370 0x7f78141999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:18.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.362+0000 7f781aaf3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7814101d30 0x7f7814199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:18.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781aaf3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7814107420 0x7f78141a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:18.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781aaf3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f78141116a0 con 0x7f7814101370 2024-06-09T22:27:18.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781aaf3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7814111520 con 0x7f7814107420 2024-06-09T22:27:18.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781aaf3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7814111820 con 0x7f7814101d30 2024-06-09T22:27:18.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781888f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7814101370 0x7f78141999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:18.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781888f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7814101370 0x7f78141999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42788/0 (socket says 172.21.15.104:42788) 2024-06-09T22:27:18.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781888f700 1 -- 172.21.15.104:0/1830575263 learned_addr learned my addr 172.21.15.104:0/1830575263 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:18.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f7819090700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7814107420 0x7f78141a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:18.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f7813fff700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7814101d30 0x7f7814199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:18.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781888f700 1 -- 172.21.15.104:0/1830575263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7814101d30 msgr2=0x7f7814199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:18.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781888f700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7814101d30 0x7f7814199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:18.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781888f700 1 -- 172.21.15.104:0/1830575263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7814107420 msgr2=0x7f78141a4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:18.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781888f700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7814107420 0x7f78141a4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:18.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781888f700 1 -- 172.21.15.104:0/1830575263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78141a4f60 con 0x7f7814101370 2024-06-09T22:27:18.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f7819090700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7814107420 0x7f78141a4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:27:18.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781888f700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7814101370 0x7f78141999a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f780800d980 tx=0x7f780800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:18.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f7811ffb700 1 -- 172.21.15.104:0/1830575263 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7808017070 con 0x7f7814101370 2024-06-09T22:27:18.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f781aaf3700 1 -- 172.21.15.104:0/1830575263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78141a5130 con 0x7f7814101370 2024-06-09T22:27:18.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f7811ffb700 1 -- 172.21.15.104:0/1830575263 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7808004d10 con 0x7f7814101370 2024-06-09T22:27:18.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.363+0000 7f7811ffb700 1 -- 172.21.15.104:0/1830575263 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78080055c0 con 0x7f7814101370 2024-06-09T22:27:18.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.364+0000 7f781aaf3700 1 -- 172.21.15.104:0/1830575263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78141a5640 con 0x7f7814101370 2024-06-09T22:27:18.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.365+0000 7f781aaf3700 1 -- 172.21.15.104:0/1830575263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7814069180 con 0x7f7814101370 2024-06-09T22:27:18.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.366+0000 7f7811ffb700 1 -- 172.21.15.104:0/1830575263 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7808004750 con 0x7f7814101370 2024-06-09T22:27:18.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.366+0000 7f7811ffb700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f78040a2d90 0x7f78040a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:18.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.366+0000 7f7811ffb700 1 -- 172.21.15.104:0/1830575263 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f78080cf6f0 con 0x7f7814101370 2024-06-09T22:27:18.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.366+0000 7f7813fff700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f78040a2d90 0x7f78040a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:18.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.367+0000 7f7813fff700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f78040a2d90 0x7f78040a5250 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7800005ea0 tx=0x7f7800005e30 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:18.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.371+0000 7f7811ffb700 1 -- 172.21.15.104:0/1830575263 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7808020020 con 0x7f7814101370 2024-06-09T22:27:18.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.614+0000 7f781aaf3700 1 -- 172.21.15.104:0/1830575263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f781402de60 con 0x7f7814101370 2024-06-09T22:27:18.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.614+0000 7f7811ffb700 1 -- 172.21.15.104:0/1830575263 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7f780809bfa0 con 0x7f7814101370 2024-06-09T22:27:18.616 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:27:18.616 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:27:18.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.616+0000 7f781aaf3700 1 -- 172.21.15.104:0/1830575263 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f78040a2d90 msgr2=0x7f78040a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:18.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.616+0000 7f781aaf3700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f78040a2d90 0x7f78040a5250 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7800005ea0 tx=0x7f7800005e30 comp rx=0 tx=0).stop 2024-06-09T22:27:18.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.616+0000 7f781aaf3700 1 -- 172.21.15.104:0/1830575263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7814101370 msgr2=0x7f78141999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:18.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.616+0000 7f781aaf3700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7814101370 0x7f78141999a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f780800d980 tx=0x7f780800dd40 comp rx=0 tx=0).stop 2024-06-09T22:27:18.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.617+0000 7f781aaf3700 1 -- 172.21.15.104:0/1830575263 shutdown_connections 2024-06-09T22:27:18.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.617+0000 7f781aaf3700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f78040a2d90 0x7f78040a5250 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:18.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.617+0000 7f781aaf3700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7814101370 0x7f78141999a0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:18.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.617+0000 7f781aaf3700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7814107420 0x7f78141a4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:18.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.617+0000 7f781aaf3700 1 --2- 172.21.15.104:0/1830575263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7814101d30 0x7f7814199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:18.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.617+0000 7f781aaf3700 1 -- 172.21.15.104:0/1830575263 >> 172.21.15.104:0/1830575263 conn(0x7f78140783a0 msgr2=0x7f7814100030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:18.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.617+0000 7f781aaf3700 1 -- 172.21.15.104:0/1830575263 shutdown_connections 2024-06-09T22:27:18.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:18.617+0000 7f781aaf3700 1 -- 172.21.15.104:0/1830575263 wait complete. 2024-06-09T22:27:18.620 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:27:18.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.28 MB/sec execute 144 sec latency 14036.378 ms 2024-06-09T22:27:18.990 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:18 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:27:18.990 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:18 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:27:18.990 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:18 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:27:18.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:18 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:27:18.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:18 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:27:18.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:18 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:27:19.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:18 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:27:19.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:18 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:27:19.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:18 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:27:19.423 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-09T22:27:19.666 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:27:19.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.25 MB/sec execute 145 sec latency 15036.419 ms 2024-06-09T22:27:19.758 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:19 smithi104 ceph-mon[32389]: pgmap v193: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 218 KiB/s wr, 39 op/s 2024-06-09T22:27:19.758 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:19 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1830575263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:27:19.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:19 smithi194 ceph-mon[36856]: pgmap v193: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 218 KiB/s wr, 39 op/s 2024-06-09T22:27:19.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:19 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1830575263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:27:19.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:19 smithi153 ceph-mon[36599]: pgmap v193: 129 pgs: 129 active+clean; 131 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 218 KiB/s wr, 39 op/s 2024-06-09T22:27:19.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:19 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1830575263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:27:20.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.22 MB/sec execute 146 sec latency 16036.492 ms 2024-06-09T22:27:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.256+0000 7f83f4df9640 1 -- 172.21.15.104:0/450859535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83f0100280 msgr2=0x7f83f0100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.256+0000 7f83f4df9640 1 --2- 172.21.15.104:0/450859535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83f0100280 0x7f83f0100660 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f83e000aa10 tx=0x7f83e002b720 comp rx=0 tx=0).stop 2024-06-09T22:27:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.256+0000 7f83f4df9640 1 -- 172.21.15.104:0/450859535 shutdown_connections 2024-06-09T22:27:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.256+0000 7f83f4df9640 1 --2- 172.21.15.104:0/450859535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f0105350 0x7f83f010cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.256+0000 7f83f4df9640 1 --2- 172.21.15.104:0/450859535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f0100c30 0x7f83f0104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.256+0000 7f83f4df9640 1 --2- 172.21.15.104:0/450859535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83f0100280 0x7f83f0100660 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.256+0000 7f83f4df9640 1 -- 172.21.15.104:0/450859535 >> 172.21.15.104:0/450859535 conn(0x7f83f0076dc0 msgr2=0x7f83f00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.256+0000 7f83f4df9640 1 -- 172.21.15.104:0/450859535 shutdown_connections 2024-06-09T22:27:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.256+0000 7f83f4df9640 1 -- 172.21.15.104:0/450859535 wait complete. 2024-06-09T22:27:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.256+0000 7f83f4df9640 1 Processor -- start 2024-06-09T22:27:21.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83f4df9640 1 -- start start 2024-06-09T22:27:21.260 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83f4df9640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f0100280 0x7f83f01982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83f4df9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f0105350 0x7f83f01987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83f4df9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83f019cb10 0x7f83f019cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83f4df9640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f83f010fb80 con 0x7f83f019cb10 2024-06-09T22:27:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83f4df9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f83f010fa00 con 0x7f83f0105350 2024-06-09T22:27:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83f4df9640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f83f010fd00 con 0x7f83f0100280 2024-06-09T22:27:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83eed9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83f019cb10 0x7f83f019cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83eed9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83f019cb10 0x7f83f019cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42802/0 (socket says 172.21.15.104:42802) 2024-06-09T22:27:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83eed9d640 1 -- 172.21.15.104:0/1497155196 learned_addr learned my addr 172.21.15.104:0/1497155196 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f0105350 0x7f83f01987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83ee59c640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f0100280 0x7f83f01982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83eed9d640 1 -- 172.21.15.104:0/1497155196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f0100280 msgr2=0x7f83f01982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83eed9d640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f0100280 0x7f83f01982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83eed9d640 1 -- 172.21.15.104:0/1497155196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f0105350 msgr2=0x7f83f01987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83eed9d640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f0105350 0x7f83f01987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.257+0000 7f83eed9d640 1 -- 172.21.15.104:0/1497155196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f019d600 con 0x7f83f019cb10 2024-06-09T22:27:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.258+0000 7f83eed9d640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83f019cb10 0x7f83f019cfc0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f83e800eaa0 tx=0x7f83e800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.258+0000 7f83e7fff640 1 -- 172.21.15.104:0/1497155196 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e800cdb0 con 0x7f83f019cb10 2024-06-09T22:27:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.258+0000 7f83f4df9640 1 -- 172.21.15.104:0/1497155196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83f019d8f0 con 0x7f83f019cb10 2024-06-09T22:27:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.258+0000 7f83e7fff640 1 -- 172.21.15.104:0/1497155196 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83e8005020 con 0x7f83f019cb10 2024-06-09T22:27:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.258+0000 7f83e7fff640 1 -- 172.21.15.104:0/1497155196 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e80116b0 con 0x7f83f019cb10 2024-06-09T22:27:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.258+0000 7f83f4df9640 1 -- 172.21.15.104:0/1497155196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83f01a5090 con 0x7f83f019cb10 2024-06-09T22:27:21.268 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.260+0000 7f83e7fff640 1 -- 172.21.15.104:0/1497155196 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f83e8005560 con 0x7f83f019cb10 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.260+0000 7f83f7be8740 1 Processor -- start 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83f7be8740 1 -- start start 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83f7be8740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717170 0x55b697717550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83f7be8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697717a90 0x55b697721e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83f7be8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697722350 0x55b697724740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83f7be8740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b69767b4f0 con 0x55b697717170 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83f7be8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b697619870 con 0x55b697722350 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83f7be8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b6976774b0 con 0x55b697717a90 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83e5d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717170 0x55b697717550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83e5d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717170 0x55b697717550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42812/0 (socket says 172.21.15.104:42812) 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83e5d9b640 1 -- 172.21.15.104:0/816129672 learned_addr learned my addr 172.21.15.104:0/816129672 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83e7fff640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f83dc0a2da0 0x7f83dc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83ee59c640 1 --2- 172.21.15.104:0/816129672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697717a90 0x55b697721e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83eed9d640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f83dc0a2da0 0x7f83dc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/816129672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697722350 0x55b697724740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83e5d9b640 1 -- 172.21.15.104:0/816129672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697717a90 msgr2=0x55b697721e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/816129672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697717a90 0x55b697721e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83e5d9b640 1 -- 172.21.15.104:0/816129672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697722350 msgr2=0x55b697724740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/816129672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697722350 0x55b697724740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83e5d9b640 1 -- 172.21.15.104:0/816129672 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6976359b0 con 0x55b697717170 2024-06-09T22:27:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.261+0000 7f83e7fff640 1 -- 172.21.15.104:0/1497155196 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f83e80cf960 con 0x7f83f019cb10 2024-06-09T22:27:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.262+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/816129672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717170 0x55b697717550 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f83d800bbc0 tx=0x7f83d8004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0580ecd49450c5 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.262+0000 7f83eed9d640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f83dc0a2da0 0x7f83dc0a5260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f83e8066780 tx=0x7f83e806ebf0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.262+0000 7f83e4d99640 1 -- 172.21.15.104:0/816129672 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d8005020 con 0x55b697717170 2024-06-09T22:27:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.262+0000 7f83e4d99640 1 -- 172.21.15.104:0/816129672 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83d80056a0 con 0x55b697717170 2024-06-09T22:27:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.262+0000 7f83e4d99640 1 -- 172.21.15.104:0/816129672 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d8010660 con 0x55b697717170 2024-06-09T22:27:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.262+0000 7f83f7be8740 1 -- 172.21.15.104:0/816129672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717170 msgr2=0x55b697717550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.262+0000 7f83f7be8740 1 --2- 172.21.15.104:0/816129672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717170 0x55b697717550 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f83d800bbc0 tx=0x7f83d8004610 comp rx=0 tx=0).stop 2024-06-09T22:27:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.262+0000 7f83f7be8740 1 -- 172.21.15.104:0/816129672 shutdown_connections 2024-06-09T22:27:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.262+0000 7f83f7be8740 1 --2- 172.21.15.104:0/816129672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697722350 0x55b697724740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.262+0000 7f83f7be8740 1 --2- 172.21.15.104:0/816129672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697717a90 0x55b697721e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.262+0000 7f83f7be8740 1 --2- 172.21.15.104:0/816129672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717170 0x55b697717550 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.262+0000 7f83f7be8740 1 -- 172.21.15.104:0/816129672 >> 172.21.15.104:0/816129672 conn(0x55b697713680 msgr2=0x55b697713a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83f7be8740 1 -- 172.21.15.104:0/816129672 shutdown_connections 2024-06-09T22:27:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83f7be8740 1 -- 172.21.15.104:0/816129672 wait complete. 2024-06-09T22:27:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83f7be8740 1 -- start start 2024-06-09T22:27:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83f7be8740 1 Processor -- start 2024-06-09T22:27:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83f7be8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717170 0x55b69771f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83f7be8740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 0x55b69771f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83f7be8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697722350 0x55b69771fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83f7be8740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b697724f20 con 0x55b697717a90 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83f7be8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b697604ae0 con 0x55b697717170 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83f7be8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b6977250a0 con 0x55b697722350 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83f7be8740 10 client.0 _ll_register_callbacks cb 0x55b69763d9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83ee59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 0x55b69771f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83ee59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 0x55b69771f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42822/0 (socket says 172.21.15.104:42822) 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83ee59c640 1 -- 172.21.15.104:0/447607676 learned_addr learned my addr 172.21.15.104:0/447607676 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697722350 0x55b69771fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.263+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717170 0x55b69771f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83ee59c640 1 -- 172.21.15.104:0/447607676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697722350 msgr2=0x55b69771fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83ee59c640 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697722350 0x55b69771fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83ee59c640 1 -- 172.21.15.104:0/447607676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717170 msgr2=0x55b69771f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83ee59c640 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717170 0x55b69771f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83ee59c640 1 -- 172.21.15.104:0/447607676 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b69738f210 con 0x55b697717a90 2024-06-09T22:27:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83ee59c640 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 0x55b69771f770 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f83e0000c00 tx=0x7f83e0030a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6d5be877a4ea8b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83d5ffb640 1 -- 172.21.15.104:0/447607676 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e0042070 con 0x55b697717a90 2024-06-09T22:27:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83d5ffb640 1 -- 172.21.15.104:0/447607676 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f83e0004450 con 0x55b697717a90 2024-06-09T22:27:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83f7be8740 1 -- 172.21.15.104:0/447607676 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83d80056a0 con 0x55b697717a90 2024-06-09T22:27:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83d5ffb640 1 -- 172.21.15.104:0/447607676 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f83e004e050 con 0x55b697717a90 2024-06-09T22:27:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83f7be8740 10 client.17211 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83f7be8740 1 -- 172.21.15.104:0/447607676 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b69745b620 con 0x55b697717a90 2024-06-09T22:27:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83d5ffb640 1 -- 172.21.15.104:0/447607676 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83d80056a0 con 0x55b697717a90 2024-06-09T22:27:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83d5ffb640 1 -- 172.21.15.104:0/447607676 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f83e003b040 con 0x55b697717a90 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83f7be8740 10 client.17211 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83f7be8740 10 client.17211 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83f7be8740 20 client.17211 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83f7be8740 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b697732330 0x55b697734720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83f7be8740 4 client.17211 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.264+0000 7f83f7be8740 1 -- 172.21.15.104:0/447607676 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b69768ef10 con 0x55b697732330 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.265+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b697732330 0x55b697734720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.265+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b697732330 0x55b697734720 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd2c817db526ec5 server_cookie=19442fce322cd8c3 in_seq=0 out_seq=0 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.265+0000 7f83d5ffb640 10 client.17211 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83d5ffb640 1 -- 172.21.15.104:0/447607676 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b69768ef10 con 0x55b697732330 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83d5ffb640 10 client.17211 handle_command_reply: tid=0 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83f7be8740 1 client.17211 shutdown 2024-06-09T22:27:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83f7be8740 1 -- 172.21.15.104:0/447607676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 msgr2=0x55b69771f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83f7be8740 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 0x55b69771f770 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f83e0000c00 tx=0x7f83e0030a30 comp rx=0 tx=0).stop 2024-06-09T22:27:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83f7be8740 1 -- 172.21.15.104:0/447607676 shutdown_connections 2024-06-09T22:27:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83f7be8740 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b697732330 0x55b697734720 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83f7be8740 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697722350 0x55b69771fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83f7be8740 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 0x55b69771f770 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83d5ffb640 0 client.17211 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83f7be8740 1 --2- 172.21.15.104:0/447607676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717170 0x55b69771f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83f7be8740 1 -- 172.21.15.104:0/447607676 >> 172.21.15.104:0/447607676 conn(0x55b697713680 msgr2=0x55b6977193b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83f7be8740 1 -- 172.21.15.104:0/447607676 shutdown_connections 2024-06-09T22:27:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83f7be8740 1 -- 172.21.15.104:0/447607676 wait complete. 2024-06-09T22:27:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.266+0000 7f83f7be8740 20 client.17211 trim_cache size 0 max 16384 2024-06-09T22:27:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83f7be8740 1 Processor -- start 2024-06-09T22:27:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83f7be8740 1 -- start start 2024-06-09T22:27:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83f7be8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697717170 0x55b697719b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83f7be8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717a90 0x55b697721170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83f7be8740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697722350 0x55b69773a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83f7be8740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f83f01aed80 con 0x55b697722350 2024-06-09T22:27:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83f7be8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f83f01b08c0 con 0x55b697717a90 2024-06-09T22:27:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83f7be8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f83f01af450 con 0x55b697717170 2024-06-09T22:27:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83e5d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697722350 0x55b69773a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83e5d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697722350 0x55b69773a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42834/0 (socket says 172.21.15.104:42834) 2024-06-09T22:27:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83e5d9b640 1 -- 172.21.15.104:0/1567103428 learned_addr learned my addr 172.21.15.104:0/1567103428 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83ee59c640 1 --2- 172.21.15.104:0/1567103428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717a90 0x55b697721170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/1567103428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697717170 0x55b697719b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83e5d9b640 1 -- 172.21.15.104:0/1567103428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697717170 msgr2=0x55b697719b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/1567103428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697717170 0x55b697719b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83e5d9b640 1 -- 172.21.15.104:0/1567103428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717a90 msgr2=0x55b697721170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/1567103428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717a90 0x55b697721170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.281+0000 7f83e5d9b640 1 -- 172.21.15.104:0/1567103428 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b697734da0 con 0x55b697722350 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/1567103428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697722350 0x55b69773a030 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f83d800bbc0 tx=0x7f83d800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cab0969ef42a7c2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83d7fff640 1 -- 172.21.15.104:0/1567103428 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d8004120 con 0x55b697722350 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83d7fff640 1 -- 172.21.15.104:0/1567103428 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83d8025020 con 0x55b697722350 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83d7fff640 1 -- 172.21.15.104:0/1567103428 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d802e350 con 0x55b697722350 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83f7be8740 1 -- 172.21.15.104:0/1567103428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697722350 msgr2=0x55b69773a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83f7be8740 1 --2- 172.21.15.104:0/1567103428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697722350 0x55b69773a030 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f83d800bbc0 tx=0x7f83d800b990 comp rx=0 tx=0).stop 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83f7be8740 1 -- 172.21.15.104:0/1567103428 shutdown_connections 2024-06-09T22:27:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83f7be8740 1 --2- 172.21.15.104:0/1567103428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697722350 0x55b69773a030 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83f7be8740 1 --2- 172.21.15.104:0/1567103428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717a90 0x55b697721170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83f7be8740 1 --2- 172.21.15.104:0/1567103428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697717170 0x55b697719b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83f7be8740 1 -- 172.21.15.104:0/1567103428 >> 172.21.15.104:0/1567103428 conn(0x55b697713680 msgr2=0x55b697720a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83f7be8740 1 -- 172.21.15.104:0/1567103428 shutdown_connections 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.282+0000 7f83f7be8740 1 -- 172.21.15.104:0/1567103428 wait complete. 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83f7be8740 1 -- start start 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83f7be8740 1 Processor -- start 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83f7be8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717170 0x55b69772a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83f7be8740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 0x55b69772aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83f7be8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697722350 0x55b69771bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83f7be8740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f83f01b01f0 con 0x55b697717a90 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83f7be8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f83f01afb20 con 0x55b697717170 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83f7be8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f83f01acd40 con 0x55b697722350 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83f7be8740 10 client.0 _ll_register_callbacks cb 0x7f83e000a750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83ee59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 0x55b69772aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83ee59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 0x55b69772aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42850/0 (socket says 172.21.15.104:42850) 2024-06-09T22:27:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83ee59c640 1 -- 172.21.15.104:0/4033182190 learned_addr learned my addr 172.21.15.104:0/4033182190 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83e5d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717170 0x55b69772a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/4033182190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697722350 0x55b69771bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83ee59c640 1 -- 172.21.15.104:0/4033182190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697722350 msgr2=0x55b69771bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83ee59c640 1 --2- 172.21.15.104:0/4033182190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697722350 0x55b69771bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83ee59c640 1 -- 172.21.15.104:0/4033182190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717170 msgr2=0x55b69772a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83ee59c640 1 --2- 172.21.15.104:0/4033182190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717170 0x55b69772a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83ee59c640 1 -- 172.21.15.104:0/4033182190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b69771c2f0 con 0x55b697717a90 2024-06-09T22:27:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83ee59c640 1 --2- 172.21.15.104:0/4033182190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 0x55b69772aa20 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f83e0031040 tx=0x7f83e0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa73730b7967679 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.283+0000 7f83d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83d5ffb640 1 -- 172.21.15.104:0/4033182190 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e0042030 con 0x55b697717a90 2024-06-09T22:27:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83d5ffb640 1 -- 172.21.15.104:0/4033182190 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f83e0003a70 con 0x55b697717a90 2024-06-09T22:27:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83f7be8740 1 -- 172.21.15.104:0/4033182190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83d8025020 con 0x55b697717a90 2024-06-09T22:27:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83d5ffb640 1 -- 172.21.15.104:0/4033182190 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f83e005f050 con 0x55b697717a90 2024-06-09T22:27:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83f7be8740 10 client.17217 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83f7be8740 1 -- 172.21.15.104:0/4033182190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b69779ca50 con 0x55b697717a90 2024-06-09T22:27:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83d5ffb640 1 -- 172.21.15.104:0/4033182190 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83d8025020 con 0x55b697717a90 2024-06-09T22:27:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83d5ffb640 1 -- 172.21.15.104:0/4033182190 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f83e005a030 con 0x55b697717a90 2024-06-09T22:27:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83f7be8740 10 client.17217 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83f7be8740 10 client.17217 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83f7be8740 20 client.17217 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83f7be8740 1 --2- 172.21.15.104:0/4033182190 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b69779d250 0x55b69779f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83f7be8740 4 client.17217 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:27:21.292 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83f7be8740 1 -- 172.21.15.104:0/4033182190 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b697688c20 con 0x55b69779d250 2024-06-09T22:27:21.292 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.284+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/4033182190 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b69779d250 0x55b69779f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.292 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.285+0000 7f83e5d9b640 1 --2- 172.21.15.104:0/4033182190 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b69779d250 0x55b69779f670 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ee48047dbb626f server_cookie=9afbffe840ce5d3a in_seq=0 out_seq=0 2024-06-09T22:27:21.292 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.285+0000 7f83d5ffb640 10 client.17217 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83d5ffb640 1 -- 172.21.15.104:0/4033182190 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b697688c20 con 0x55b69779d250 2024-06-09T22:27:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83d5ffb640 10 client.17217 handle_command_reply: tid=0 2024-06-09T22:27:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83f7be8740 1 client.17217 shutdown 2024-06-09T22:27:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83f7be8740 1 -- 172.21.15.104:0/4033182190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 msgr2=0x55b69772aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83f7be8740 1 --2- 172.21.15.104:0/4033182190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 0x55b69772aa20 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f83e0031040 tx=0x7f83e0005fd0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83f7be8740 1 -- 172.21.15.104:0/4033182190 shutdown_connections 2024-06-09T22:27:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83f7be8740 1 --2- 172.21.15.104:0/4033182190 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b69779d250 0x55b69779f670 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83f7be8740 1 --2- 172.21.15.104:0/4033182190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b697722350 0x55b69771bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83d5ffb640 0 client.17217 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83f7be8740 1 --2- 172.21.15.104:0/4033182190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b697717a90 0x55b69772aa20 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83f7be8740 1 --2- 172.21.15.104:0/4033182190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b697717170 0x55b69772a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83f7be8740 1 -- 172.21.15.104:0/4033182190 >> 172.21.15.104:0/4033182190 conn(0x55b697713680 msgr2=0x55b69771f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83f7be8740 1 -- 172.21.15.104:0/4033182190 shutdown_connections 2024-06-09T22:27:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83f7be8740 1 -- 172.21.15.104:0/4033182190 wait complete. 2024-06-09T22:27:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.294+0000 7f83f7be8740 20 client.17217 trim_cache size 0 max 16384 2024-06-09T22:27:21.298 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:27:21.298 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:27:21.298 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:27:21.298 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.295+0000 7f83f4df9640 1 -- 172.21.15.104:0/1497155196 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f83dc0a2da0 msgr2=0x7f83dc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.295+0000 7f83f4df9640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f83dc0a2da0 0x7f83dc0a5260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f83e8066780 tx=0x7f83e806ebf0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.295+0000 7f83f4df9640 1 -- 172.21.15.104:0/1497155196 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83f019cb10 msgr2=0x7f83f019cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.295+0000 7f83f4df9640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83f019cb10 0x7f83f019cfc0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f83e800eaa0 tx=0x7f83e800ef60 comp rx=0 tx=0).stop 2024-06-09T22:27:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.295+0000 7f83f4df9640 1 -- 172.21.15.104:0/1497155196 shutdown_connections 2024-06-09T22:27:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.295+0000 7f83f4df9640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f83dc0a2da0 0x7f83dc0a5260 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.295+0000 7f83f4df9640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83f019cb10 0x7f83f019cfc0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.295+0000 7f83f4df9640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83f0105350 0x7f83f01987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.295+0000 7f83f4df9640 1 --2- 172.21.15.104:0/1497155196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83f0100280 0x7f83f01982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.295+0000 7f83f4df9640 1 -- 172.21.15.104:0/1497155196 >> 172.21.15.104:0/1497155196 conn(0x7f83f0076dc0 msgr2=0x7f83f0103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.295+0000 7f83f4df9640 1 -- 172.21.15.104:0/1497155196 shutdown_connections 2024-06-09T22:27:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:21.295+0000 7f83f4df9640 1 -- 172.21.15.104:0/1497155196 wait complete. 2024-06-09T22:27:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.399+0000 7f1fbb44a700 1 -- 172.21.15.104:0/1233293355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fb4101650 msgr2=0x7f1fb4101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.399+0000 7f1fbb44a700 1 --2- 172.21.15.104:0/1233293355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fb4101650 0x7f1fb4101a30 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40099c0 tx=0x7f1fa4009cd0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.399+0000 7f1fbb44a700 1 -- 172.21.15.104:0/1233293355 shutdown_connections 2024-06-09T22:27:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.399+0000 7f1fbb44a700 1 --2- 172.21.15.104:0/1233293355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fb4101650 0x7f1fb4101a30 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.399+0000 7f1fbb44a700 1 --2- 172.21.15.104:0/1233293355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb41066a0 0x7f1fb410e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.399+0000 7f1fbb44a700 1 --2- 172.21.15.104:0/1233293355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb4102000 0x7f1fb4105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.399+0000 7f1fbb44a700 1 -- 172.21.15.104:0/1233293355 >> 172.21.15.104:0/1233293355 conn(0x7f1fb40773e0 msgr2=0x7f1fb40777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.399+0000 7f1fbb44a700 1 -- 172.21.15.104:0/1233293355 shutdown_connections 2024-06-09T22:27:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.399+0000 7f1fbb44a700 1 -- 172.21.15.104:0/1233293355 wait complete. 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.400+0000 7f1fbb44a700 1 Processor -- start 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.400+0000 7f1fbb44a700 1 -- start start 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.400+0000 7f1fbb44a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fb4101650 0x7f1fb4199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.400+0000 7f1fbb44a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb4102000 0x7f1fb419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.400+0000 7f1fbb44a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb41066a0 0x7f1fb41a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.400+0000 7f1fbb44a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41116b0 con 0x7f1fb4101650 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.400+0000 7f1fbb44a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb4111530 con 0x7f1fb41066a0 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.400+0000 7f1fbb44a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb4111830 con 0x7f1fb4102000 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.400+0000 7f1fb91e6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fb4101650 0x7f1fb4199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.400+0000 7f1fb91e6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fb4101650 0x7f1fb4199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42872/0 (socket says 172.21.15.104:42872) 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.400+0000 7f1fb91e6700 1 -- 172.21.15.104:0/3337425705 learned_addr learned my addr 172.21.15.104:0/3337425705 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1fb99e7700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb41066a0 0x7f1fb41a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb4102000 0x7f1fb419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1fb91e6700 1 -- 172.21.15.104:0/3337425705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb4102000 msgr2=0x7f1fb419a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb4102000 0x7f1fb419a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1fb91e6700 1 -- 172.21.15.104:0/3337425705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb41066a0 msgr2=0x7f1fb41a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb41066a0 0x7f1fb41a4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1fb91e6700 1 -- 172.21.15.104:0/3337425705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb41a50f0 con 0x7f1fb4101650 2024-06-09T22:27:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1fb99e7700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb41066a0 0x7f1fb41a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:27:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb4102000 0x7f1fb419a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:27:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fb4101650 0x7f1fb4199b30 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40059d0 tx=0x7f1fa400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1faa7fc700 1 -- 172.21.15.104:0/3337425705 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4027070 con 0x7f1fb4101650 2024-06-09T22:27:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1faa7fc700 1 -- 172.21.15.104:0/3337425705 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa4003680 con 0x7f1fb4101650 2024-06-09T22:27:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1fbb44a700 1 -- 172.21.15.104:0/3337425705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb41a52c0 con 0x7f1fb4101650 2024-06-09T22:27:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1faa7fc700 1 -- 172.21.15.104:0/3337425705 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4017650 con 0x7f1fb4101650 2024-06-09T22:27:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.401+0000 7f1fbb44a700 1 -- 172.21.15.104:0/3337425705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb41a5710 con 0x7f1fb4101650 2024-06-09T22:27:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.403+0000 7f1faa7fc700 1 -- 172.21.15.104:0/3337425705 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f1fa402a070 con 0x7f1fb4101650 2024-06-09T22:27:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.404+0000 7f1faa7fc700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1fa00a2d90 0x7f1fa00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.405+0000 7f1faa7fc700 1 -- 172.21.15.104:0/3337425705 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1fa40d04d0 con 0x7f1fb4101650 2024-06-09T22:27:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.405+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1fa00a2d90 0x7f1fa00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.405+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1fa00a2d90 0x7f1fa00a5250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4107bb0 tx=0x7f1fb0009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.405+0000 7f1fc5ccdb80 1 Processor -- start 2024-06-09T22:27:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.405+0000 7f1fc5ccdb80 1 -- start start 2024-06-09T22:27:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fc5ccdb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 0x55914d964340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fc5ccdb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d964880 0x55914d95e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fc5ccdb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d95e7a0 0x55914d95ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fc5ccdb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55914d850f60 con 0x55914d963f60 2024-06-09T22:27:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fc5ccdb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55914d8af850 con 0x55914d95e7a0 2024-06-09T22:27:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fc5ccdb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55914d817460 con 0x55914d964880 2024-06-09T22:27:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb91e6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 0x55914d964340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb91e6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 0x55914d964340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42876/0 (socket says 172.21.15.104:42876) 2024-06-09T22:27:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb91e6700 1 -- 172.21.15.104:0/880986608 learned_addr learned my addr 172.21.15.104:0/880986608 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb99e7700 1 --2- 172.21.15.104:0/880986608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d95e7a0 0x55914d95ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/880986608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d964880 0x55914d95e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb91e6700 1 -- 172.21.15.104:0/880986608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d964880 msgr2=0x55914d95e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/880986608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d964880 0x55914d95e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb91e6700 1 -- 172.21.15.104:0/880986608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d95e7a0 msgr2=0x55914d95ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/880986608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d95e7a0 0x55914d95ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb91e6700 1 -- 172.21.15.104:0/880986608 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55914d7fe8d0 con 0x55914d963f60 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb99e7700 1 --2- 172.21.15.104:0/880986608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d95e7a0 0x55914d95ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/880986608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d964880 0x55914d95e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/880986608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 0x55914d964340 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40a4d80 tx=0x7f1fa4098a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef4dc06df5fe27d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1f9f7fe700 1 -- 172.21.15.104:0/880986608 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa409c030 con 0x55914d963f60 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1f9f7fe700 1 -- 172.21.15.104:0/880986608 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa409cdd0 con 0x55914d963f60 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.406+0000 7f1f9f7fe700 1 -- 172.21.15.104:0/880986608 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa409c930 con 0x55914d963f60 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.407+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/880986608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 msgr2=0x55914d964340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.407+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/880986608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 0x55914d964340 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40a4d80 tx=0x7f1fa4098a90 comp rx=0 tx=0).stop 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.407+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/880986608 shutdown_connections 2024-06-09T22:27:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.407+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/880986608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 0x55914d964340 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.407+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/880986608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d95e7a0 0x55914d95ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.407+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/880986608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d964880 0x55914d95e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.407+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/880986608 >> 172.21.15.104:0/880986608 conn(0x55914d88b080 msgr2=0x55914d959220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.407+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/880986608 shutdown_connections 2024-06-09T22:27:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.407+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/880986608 wait complete. 2024-06-09T22:27:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fc5ccdb80 1 -- start start 2024-06-09T22:27:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fc5ccdb80 1 Processor -- start 2024-06-09T22:27:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fc5ccdb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d95e7a0 0x55914d96a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fc5ccdb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 0x55914d9678c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fc5ccdb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d964880 0x55914d967e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fc5ccdb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55914d84ffc0 con 0x55914d963f60 2024-06-09T22:27:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fc5ccdb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55914d8ab810 con 0x55914d964880 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fc5ccdb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55914d96b010 con 0x55914d95e7a0 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fc5ccdb80 10 client.0 _ll_register_callbacks cb 0x55914d8455b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fb89e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 0x55914d9678c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fb89e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 0x55914d9678c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42888/0 (socket says 172.21.15.104:42888) 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fb89e5700 1 -- 172.21.15.104:0/448555499 learned_addr learned my addr 172.21.15.104:0/448555499 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d95e7a0 0x55914d96a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.408+0000 7f1fb99e7700 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d964880 0x55914d967e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1fb89e5700 1 -- 172.21.15.104:0/448555499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d95e7a0 msgr2=0x55914d96a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d95e7a0 0x55914d96a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1fb89e5700 1 -- 172.21.15.104:0/448555499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d964880 msgr2=0x55914d967e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d964880 0x55914d967e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1fb89e5700 1 -- 172.21.15.104:0/448555499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55914d875750 con 0x55914d963f60 2024-06-09T22:27:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 0x55914d9678c0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f1fb00044e0 tx=0x7f1fb00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b966f6ca480fc167 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:21.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1f9cff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:21.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1f9cff9700 1 -- 172.21.15.104:0/448555499 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0024070 con 0x55914d963f60 2024-06-09T22:27:21.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1f9cff9700 1 -- 172.21.15.104:0/448555499 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1fb0012660 con 0x55914d963f60 2024-06-09T22:27:21.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/448555499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fa409cdd0 con 0x55914d963f60 2024-06-09T22:27:21.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1f9cff9700 1 -- 172.21.15.104:0/448555499 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1fb0030050 con 0x55914d963f60 2024-06-09T22:27:21.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1fc5ccdb80 10 client.17229 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:21.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/448555499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55914d9756b0 con 0x55914d963f60 2024-06-09T22:27:21.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.409+0000 7f1f9cff9700 1 -- 172.21.15.104:0/448555499 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa409cdd0 con 0x55914d963f60 2024-06-09T22:27:21.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.410+0000 7f1f9cff9700 1 -- 172.21.15.104:0/448555499 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1fb0020920 con 0x55914d963f60 2024-06-09T22:27:21.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.410+0000 7f1fc5ccdb80 10 client.17229 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:21.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.410+0000 7f1fc5ccdb80 10 client.17229 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:27:21.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.410+0000 7f1fc5ccdb80 20 client.17229 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:21.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.410+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55914d977ce0 0x55914d97a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.410+0000 7f1fc5ccdb80 4 client.17229 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:21.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.410+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/448555499 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55914d8bcd50 con 0x55914d977ce0 2024-06-09T22:27:21.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.410+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55914d977ce0 0x55914d97a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.411+0000 7f1f9cff9700 10 client.17229 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:27:21.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.411+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55914d977ce0 0x55914d97a0d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d5b7217afd150a43 server_cookie=afba31f513659143 in_seq=0 out_seq=0 2024-06-09T22:27:21.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.411+0000 7f1f9cff9700 1 -- 172.21.15.104:0/448555499 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55914d8bcd50 con 0x55914d977ce0 2024-06-09T22:27:21.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.411+0000 7f1f9cff9700 10 client.17229 handle_command_reply: tid=0 2024-06-09T22:27:21.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.411+0000 7f1fc5ccdb80 1 client.17229 shutdown 2024-06-09T22:27:21.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.411+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/448555499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 msgr2=0x55914d9678c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.411+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 0x55914d9678c0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f1fb00044e0 tx=0x7f1fb00047c0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.411+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/448555499 shutdown_connections 2024-06-09T22:27:21.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.411+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55914d977ce0 0x55914d97a0d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.411+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d964880 0x55914d967e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.412+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d95e7a0 0x55914d96a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.412+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/448555499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d963f60 0x55914d9678c0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.412+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/448555499 >> 172.21.15.104:0/448555499 conn(0x55914d88b080 msgr2=0x55914d958700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:21.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.411+0000 7f1f9cff9700 0 client.17229 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:27:21.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.412+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/448555499 shutdown_connections 2024-06-09T22:27:21.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.412+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/448555499 wait complete. 2024-06-09T22:27:21.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.412+0000 7f1fc5ccdb80 20 client.17229 trim_cache size 0 max 16384 2024-06-09T22:27:21.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fc5ccdb80 1 Processor -- start 2024-06-09T22:27:21.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fc5ccdb80 1 -- start start 2024-06-09T22:27:21.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fc5ccdb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d95e7a0 0x55914d965430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fc5ccdb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d963f60 0x55914d958e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fc5ccdb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d964880 0x55914d9593c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fc5ccdb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41ae480 con 0x55914d964880 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fc5ccdb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41affc0 con 0x55914d95e7a0 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fc5ccdb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41aeb50 con 0x55914d963f60 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fb99e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d964880 0x55914d9593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fb99e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d964880 0x55914d9593c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42890/0 (socket says 172.21.15.104:42890) 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fb99e7700 1 -- 172.21.15.104:0/1692637341 learned_addr learned my addr 172.21.15.104:0/1692637341 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/1692637341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d95e7a0 0x55914d965430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/1692637341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d963f60 0x55914d958e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fb99e7700 1 -- 172.21.15.104:0/1692637341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d963f60 msgr2=0x55914d958e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fb99e7700 1 --2- 172.21.15.104:0/1692637341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d963f60 0x55914d958e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fb99e7700 1 -- 172.21.15.104:0/1692637341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d95e7a0 msgr2=0x55914d965430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fb99e7700 1 --2- 172.21.15.104:0/1692637341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d95e7a0 0x55914d965430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fb99e7700 1 -- 172.21.15.104:0/1692637341 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55914d95e260 con 0x55914d964880 2024-06-09T22:27:21.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/1692637341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d963f60 0x55914d958e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:27:21.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.429+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/1692637341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d95e7a0 0x55914d965430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:27:21.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.430+0000 7f1fb99e7700 1 --2- 172.21.15.104:0/1692637341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d964880 0x55914d9593c0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1fac009bd0 tx=0x7f1fac009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e5affc4b762932 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:21.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.430+0000 7f1f9effd700 1 -- 172.21.15.104:0/1692637341 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac01a020 con 0x55914d964880 2024-06-09T22:27:21.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.430+0000 7f1f9effd700 1 -- 172.21.15.104:0/1692637341 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fac016400 con 0x55914d964880 2024-06-09T22:27:21.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.430+0000 7f1f9effd700 1 -- 172.21.15.104:0/1692637341 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac0155c0 con 0x55914d964880 2024-06-09T22:27:21.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.430+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/1692637341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d964880 msgr2=0x55914d9593c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.430+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/1692637341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d964880 0x55914d9593c0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1fac009bd0 tx=0x7f1fac009040 comp rx=0 tx=0).stop 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.430+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/1692637341 shutdown_connections 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.430+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/1692637341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d964880 0x55914d9593c0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.430+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/1692637341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d95e7a0 0x55914d965430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.430+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/1692637341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d963f60 0x55914d958e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.430+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/1692637341 >> 172.21.15.104:0/1692637341 conn(0x55914d88b080 msgr2=0x55914d964d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.430+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/1692637341 shutdown_connections 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/1692637341 wait complete. 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fc5ccdb80 1 -- start start 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fc5ccdb80 1 Processor -- start 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fc5ccdb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d95e7a0 0x55914d970a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fc5ccdb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d963f60 0x55914d97f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fc5ccdb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d964880 0x55914d97f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fc5ccdb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41af8f0 con 0x55914d95e7a0 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fc5ccdb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41af220 con 0x55914d964880 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fc5ccdb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41ac400 con 0x55914d963f60 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fc5ccdb80 10 client.0 _ll_register_callbacks cb 0x55914d8455b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fb91e6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d95e7a0 0x55914d970a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fb91e6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d95e7a0 0x55914d970a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42902/0 (socket says 172.21.15.104:42902) 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.431+0000 7f1fb91e6700 1 -- 172.21.15.104:0/4068276101 learned_addr learned my addr 172.21.15.104:0/4068276101 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d963f60 0x55914d97f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1fb99e7700 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d964880 0x55914d97f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1fb91e6700 1 -- 172.21.15.104:0/4068276101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d963f60 msgr2=0x55914d97f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d963f60 0x55914d97f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1fb91e6700 1 -- 172.21.15.104:0/4068276101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d964880 msgr2=0x55914d97f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d964880 0x55914d97f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1fb91e6700 1 -- 172.21.15.104:0/4068276101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55914d9e1960 con 0x55914d95e7a0 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d963f60 0x55914d97f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1fb91e6700 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d95e7a0 0x55914d970a80 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4000c00 tx=0x7f1fa4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2164ad32abbb8da6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1f9cff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1f9cff9700 1 -- 172.21.15.104:0/4068276101 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa40b4070 con 0x55914d95e7a0 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1f9cff9700 1 -- 172.21.15.104:0/4068276101 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1fa40d0170 con 0x55914d95e7a0 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/4068276101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fac016400 con 0x55914d95e7a0 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1f9cff9700 1 -- 172.21.15.104:0/4068276101 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1fa409c030 con 0x55914d95e7a0 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1fc5ccdb80 10 client.17235 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/4068276101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55914d9e29f0 con 0x55914d95e7a0 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1f9cff9700 1 -- 172.21.15.104:0/4068276101 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fac016400 con 0x55914d95e7a0 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.432+0000 7f1f9cff9700 1 -- 172.21.15.104:0/4068276101 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1fa40a4e20 con 0x55914d95e7a0 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.433+0000 7f1fc5ccdb80 10 client.17235 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.433+0000 7f1fc5ccdb80 10 client.17235 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:27:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.433+0000 7f1fc5ccdb80 20 client.17235 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.433+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55914d9e2e60 0x55914d9e5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.433+0000 7f1fc5ccdb80 4 client.17235 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:27:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.433+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/4068276101 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55914d8b72d0 con 0x55914d9e2e60 2024-06-09T22:27:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.433+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55914d9e2e60 0x55914d9e5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.433+0000 7f1f9cff9700 10 client.17235 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:27:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.433+0000 7f1fb89e5700 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55914d9e2e60 0x55914d9e5250 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ffceb7b039c4430d server_cookie=2c2e770b0c59bacc in_seq=0 out_seq=0 2024-06-09T22:27:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1f9cff9700 1 -- 172.21.15.104:0/4068276101 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55914d8b72d0 con 0x55914d9e2e60 2024-06-09T22:27:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1f9cff9700 10 client.17235 handle_command_reply: tid=0 2024-06-09T22:27:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1fc5ccdb80 1 client.17235 shutdown 2024-06-09T22:27:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/4068276101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d95e7a0 msgr2=0x55914d970a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d95e7a0 0x55914d970a80 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4000c00 tx=0x7f1fa4005fd0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/4068276101 shutdown_connections 2024-06-09T22:27:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55914d95e7a0 0x55914d970a80 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55914d9e2e60 0x55914d9e5250 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55914d964880 0x55914d97f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1fc5ccdb80 1 --2- 172.21.15.104:0/4068276101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55914d963f60 0x55914d97f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/4068276101 >> 172.21.15.104:0/4068276101 conn(0x55914d88b080 msgr2=0x55914d9690a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1f9cff9700 0 client.17235 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:27:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/4068276101 shutdown_connections 2024-06-09T22:27:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1fc5ccdb80 1 -- 172.21.15.104:0/4068276101 wait complete. 2024-06-09T22:27:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.434+0000 7f1fc5ccdb80 20 client.17235 trim_cache size 0 max 16384 2024-06-09T22:27:21.441 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:27:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.435+0000 7f1fbb44a700 1 -- 172.21.15.104:0/3337425705 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1fa00a2d90 msgr2=0x7f1fa00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.435+0000 7f1fbb44a700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1fa00a2d90 0x7f1fa00a5250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4107bb0 tx=0x7f1fb0009260 comp rx=0 tx=0).stop 2024-06-09T22:27:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.435+0000 7f1fbb44a700 1 -- 172.21.15.104:0/3337425705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fb4101650 msgr2=0x7f1fb4199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.435+0000 7f1fbb44a700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fb4101650 0x7f1fb4199b30 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40059d0 tx=0x7f1fa400fab0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.435+0000 7f1fbb44a700 1 -- 172.21.15.104:0/3337425705 shutdown_connections 2024-06-09T22:27:21.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.435+0000 7f1fbb44a700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1fa00a2d90 0x7f1fa00a5250 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.435+0000 7f1fbb44a700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fb4101650 0x7f1fb4199b30 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.435+0000 7f1fbb44a700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb41066a0 0x7f1fb41a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.435+0000 7f1fbb44a700 1 --2- 172.21.15.104:0/3337425705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb4102000 0x7f1fb419a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:21.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.435+0000 7f1fbb44a700 1 -- 172.21.15.104:0/3337425705 >> 172.21.15.104:0/3337425705 conn(0x7f1fb40773e0 msgr2=0x7f1fb40ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:21.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.435+0000 7f1fbb44a700 1 -- 172.21.15.104:0/3337425705 shutdown_connections 2024-06-09T22:27:21.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:21.435+0000 7f1fbb44a700 1 -- 172.21.15.104:0/3337425705 wait complete. 2024-06-09T22:27:21.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.19 MB/sec execute 147 sec latency 17036.531 ms 2024-06-09T22:27:21.758 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:21 smithi104 ceph-mon[32389]: pgmap v194: 129 pgs: 129 active+clean; 133 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 423 KiB/s wr, 46 op/s 2024-06-09T22:27:21.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:21 smithi153 ceph-mon[36599]: pgmap v194: 129 pgs: 129 active+clean; 133 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 423 KiB/s wr, 46 op/s 2024-06-09T22:27:22.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:21 smithi194 ceph-mon[36856]: pgmap v194: 129 pgs: 129 active+clean; 133 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 423 KiB/s wr, 46 op/s 2024-06-09T22:27:22.204 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-09T22:27:22.444 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:27:22.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.17 MB/sec execute 148 sec latency 18036.579 ms 2024-06-09T22:27:23.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.14 MB/sec execute 149 sec latency 19036.633 ms 2024-06-09T22:27:23.881 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:23 smithi104 ceph-mon[32389]: pgmap v195: 129 pgs: 129 active+clean; 133 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 209 KiB/s wr, 13 op/s 2024-06-09T22:27:23.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:23 smithi153 ceph-mon[36599]: pgmap v195: 129 pgs: 129 active+clean; 133 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 209 KiB/s wr, 13 op/s 2024-06-09T22:27:24.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:23 smithi194 ceph-mon[36856]: pgmap v195: 129 pgs: 129 active+clean; 133 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 209 KiB/s wr, 13 op/s 2024-06-09T22:27:24.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.232+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3533681858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d18107400 msgr2=0x7f7d181077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.232+0000 7f7d1ce2b700 1 --2- 172.21.15.104:0/3533681858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d18107400 0x7f7d181077e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009990 tx=0x7f7d08009ca0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.233+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3533681858 shutdown_connections 2024-06-09T22:27:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.233+0000 7f7d1ce2b700 1 --2- 172.21.15.104:0/3533681858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d18107400 0x7f7d181077e0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.233+0000 7f7d1ce2b700 1 --2- 172.21.15.104:0/3533681858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d18101d50 0x7f7d1810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.233+0000 7f7d1ce2b700 1 --2- 172.21.15.104:0/3533681858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d18101390 0x7f7d18101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.233+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3533681858 >> 172.21.15.104:0/3533681858 conn(0x7f7d18078340 msgr2=0x7f7d18078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.233+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3533681858 shutdown_connections 2024-06-09T22:27:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.233+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3533681858 wait complete. 2024-06-09T22:27:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.233+0000 7f7d1ce2b700 1 Processor -- start 2024-06-09T22:27:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.233+0000 7f7d1ce2b700 1 -- start start 2024-06-09T22:27:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d1ce2b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d18101390 0x7f7d1819e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d1ce2b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d18101d50 0x7f7d1819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d1ce2b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d18107400 0x7f7d18198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d1ce2b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181116f0 con 0x7f7d18107400 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d1ce2b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7d18111570 con 0x7f7d18101d50 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d1ce2b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7d18111870 con 0x7f7d18101390 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d16d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d18107400 0x7f7d18198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d16d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d18107400 0x7f7d18198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42930/0 (socket says 172.21.15.104:42930) 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d16d9d700 1 -- 172.21.15.104:0/3545045785 learned_addr learned my addr 172.21.15.104:0/3545045785 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d1659c700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d18101390 0x7f7d1819e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d15d9b700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d18101d50 0x7f7d1819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d16d9d700 1 -- 172.21.15.104:0/3545045785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d18101390 msgr2=0x7f7d1819e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d16d9d700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d18101390 0x7f7d1819e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d16d9d700 1 -- 172.21.15.104:0/3545045785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d18101d50 msgr2=0x7f7d1819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d16d9d700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d18101d50 0x7f7d1819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d16d9d700 1 -- 172.21.15.104:0/3545045785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d18199190 con 0x7f7d18107400 2024-06-09T22:27:24.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.234+0000 7f7d16d9d700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d18107400 0x7f7d18198a30 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000b970 tx=0x7f7d1000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:24.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.235+0000 7f7d077fe700 1 -- 172.21.15.104:0/3545045785 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1000d860 con 0x7f7d18107400 2024-06-09T22:27:24.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.235+0000 7f7d077fe700 1 -- 172.21.15.104:0/3545045785 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d1000e430 con 0x7f7d18107400 2024-06-09T22:27:24.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.235+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3545045785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d18199480 con 0x7f7d18107400 2024-06-09T22:27:24.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.235+0000 7f7d077fe700 1 -- 172.21.15.104:0/3545045785 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10017660 con 0x7f7d18107400 2024-06-09T22:27:24.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.235+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3545045785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d181a5740 con 0x7f7d18107400 2024-06-09T22:27:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.237+0000 7f7d077fe700 1 -- 172.21.15.104:0/3545045785 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7d1000e5e0 con 0x7f7d18107400 2024-06-09T22:27:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.237+0000 7f7d077fe700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7d000a2d90 0x7f7d000a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.238+0000 7f7d1659c700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7d000a2d90 0x7f7d000a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.238+0000 7f7d077fe700 1 -- 172.21.15.104:0/3545045785 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7d100cf350 con 0x7f7d18107400 2024-06-09T22:27:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.238+0000 7f7d1659c700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7d000a2d90 0x7f7d000a5250 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800b5c0 tx=0x7f7d08005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.238+0000 7f7d276aeb80 1 Processor -- start 2024-06-09T22:27:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d276aeb80 1 -- start start 2024-06-09T22:27:24.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d276aeb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece80a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d276aeb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 0x5632ece7a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d276aeb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 0x5632ece7b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d276aeb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5632ecdcc810 con 0x5632ece80640 2024-06-09T22:27:24.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d276aeb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5632ecd74720 con 0x5632ece7af20 2024-06-09T22:27:24.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d276aeb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5632ecdc87d0 con 0x5632ece80f60 2024-06-09T22:27:24.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d1659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece80a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d1659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece80a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42944/0 (socket says 172.21.15.104:42944) 2024-06-09T22:27:24.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d1659c700 1 -- 172.21.15.104:0/269974684 learned_addr learned my addr 172.21.15.104:0/269974684 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:24.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d16d9d700 1 --2- 172.21.15.104:0/269974684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 0x5632ece7b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d15d9b700 1 --2- 172.21.15.104:0/269974684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 0x5632ece7a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d1659c700 1 -- 172.21.15.104:0/269974684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 msgr2=0x5632ece7a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:24.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d1659c700 1 --2- 172.21.15.104:0/269974684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 0x5632ece7a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d1659c700 1 -- 172.21.15.104:0/269974684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 msgr2=0x5632ece7b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:24.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d1659c700 1 --2- 172.21.15.104:0/269974684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 0x5632ece7b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d1659c700 1 -- 172.21.15.104:0/269974684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632ecd6af50 con 0x5632ece80640 2024-06-09T22:27:24.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.239+0000 7f7d16d9d700 1 --2- 172.21.15.104:0/269974684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 0x5632ece7b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:24.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.240+0000 7f7d1659c700 1 --2- 172.21.15.104:0/269974684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece80a20 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009990 tx=0x7f7d08020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690fc862f21c49a1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:24.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.240+0000 7f7d04ff9700 1 -- 172.21.15.104:0/269974684 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0800f4e0 con 0x5632ece80640 2024-06-09T22:27:24.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.240+0000 7f7d04ff9700 1 -- 172.21.15.104:0/269974684 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d0800fb60 con 0x5632ece80640 2024-06-09T22:27:24.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.240+0000 7f7d04ff9700 1 -- 172.21.15.104:0/269974684 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08024cf0 con 0x5632ece80640 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 -- 172.21.15.104:0/269974684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 msgr2=0x5632ece80a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/269974684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece80a20 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009990 tx=0x7f7d08020000 comp rx=0 tx=0).stop 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 -- 172.21.15.104:0/269974684 shutdown_connections 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/269974684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece80a20 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/269974684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 0x5632ece7b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/269974684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 0x5632ece7a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 -- 172.21.15.104:0/269974684 >> 172.21.15.104:0/269974684 conn(0x5632ecda5cc0 msgr2=0x5632ece75850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 -- 172.21.15.104:0/269974684 shutdown_connections 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 -- 172.21.15.104:0/269974684 wait complete. 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 -- start start 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 Processor -- start 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 0x5632ece86cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece83fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 0x5632ece84500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5632ece87910 con 0x5632ece80640 2024-06-09T22:27:24.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5632ecd6cbf0 con 0x5632ece7af20 2024-06-09T22:27:24.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5632ece87a90 con 0x5632ece80f60 2024-06-09T22:27:24.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.241+0000 7f7d276aeb80 10 client.0 _ll_register_callbacks cb 0x5632ecd66920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:24.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d15d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece83fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d15d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece83fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42954/0 (socket says 172.21.15.104:42954) 2024-06-09T22:27:24.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d15d9b700 1 -- 172.21.15.104:0/511597163 learned_addr learned my addr 172.21.15.104:0/511597163 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:24.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d16d9d700 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 0x5632ece84500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d1659c700 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 0x5632ece86cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d15d9b700 1 -- 172.21.15.104:0/511597163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 msgr2=0x5632ece84500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d15d9b700 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 0x5632ece84500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d15d9b700 1 -- 172.21.15.104:0/511597163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 msgr2=0x5632ece86cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d15d9b700 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 0x5632ece86cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d15d9b700 1 -- 172.21.15.104:0/511597163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632ecd1b8d0 con 0x5632ece80640 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d15d9b700 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece83fc0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c007f20 tx=0x7f7d0c012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca2d04557a48616 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7cfdffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7cfdffb700 1 -- 172.21.15.104:0/511597163 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c004460 con 0x5632ece80640 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7cfdffb700 1 -- 172.21.15.104:0/511597163 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7d0c003680 con 0x5632ece80640 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d276aeb80 1 -- 172.21.15.104:0/511597163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d0800fb60 con 0x5632ece80640 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7cfdffb700 1 -- 172.21.15.104:0/511597163 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7d0c024050 con 0x5632ece80640 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d276aeb80 10 client.17247 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.242+0000 7f7d276aeb80 1 -- 172.21.15.104:0/511597163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5632ece92270 con 0x5632ece80640 2024-06-09T22:27:24.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.243+0000 7f7cfdffb700 1 -- 172.21.15.104:0/511597163 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d0800fb60 con 0x5632ece80640 2024-06-09T22:27:24.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.243+0000 7f7cfdffb700 1 -- 172.21.15.104:0/511597163 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7d0c019020 con 0x5632ece80640 2024-06-09T22:27:24.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.243+0000 7f7d276aeb80 10 client.17247 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:24.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.243+0000 7f7d276aeb80 10 client.17247 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:27:24.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.243+0000 7f7d276aeb80 20 client.17247 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:24.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.243+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5632ece948b0 0x5632ece96ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.243+0000 7f7d276aeb80 4 client.17247 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:24.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.243+0000 7f7d276aeb80 1 -- 172.21.15.104:0/511597163 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632ecdd9d10 con 0x5632ece948b0 2024-06-09T22:27:24.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.243+0000 7f7d1659c700 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5632ece948b0 0x5632ece96ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.244+0000 7f7d1659c700 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5632ece948b0 0x5632ece96ca0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=88f628c18e052477 server_cookie=e93fdfb29bf2c27d in_seq=0 out_seq=0 2024-06-09T22:27:24.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.244+0000 7f7cfdffb700 10 client.17247 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:27:24.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.244+0000 7f7cfdffb700 1 -- 172.21.15.104:0/511597163 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5632ecdd9d10 con 0x5632ece948b0 2024-06-09T22:27:24.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.244+0000 7f7cfdffb700 10 client.17247 handle_command_reply: tid=0 2024-06-09T22:27:24.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.244+0000 7f7d276aeb80 1 client.17247 shutdown 2024-06-09T22:27:24.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.245+0000 7f7d276aeb80 1 -- 172.21.15.104:0/511597163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 msgr2=0x5632ece83fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:24.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.245+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece83fc0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c007f20 tx=0x7f7d0c012730 comp rx=0 tx=0).stop 2024-06-09T22:27:24.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.245+0000 7f7d276aeb80 1 -- 172.21.15.104:0/511597163 shutdown_connections 2024-06-09T22:27:24.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.245+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 0x5632ece84500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.245+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 0x5632ece86cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.245+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5632ece948b0 0x5632ece96ca0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.245+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/511597163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece83fc0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.245+0000 7f7cfdffb700 0 client.17247 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:27:24.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.245+0000 7f7d276aeb80 1 -- 172.21.15.104:0/511597163 >> 172.21.15.104:0/511597163 conn(0x5632ecda5cc0 msgr2=0x5632ece799d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:24.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.245+0000 7f7d276aeb80 1 -- 172.21.15.104:0/511597163 shutdown_connections 2024-06-09T22:27:24.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.245+0000 7f7d276aeb80 1 -- 172.21.15.104:0/511597163 wait complete. 2024-06-09T22:27:24.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.245+0000 7f7d276aeb80 20 client.17247 trim_cache size 0 max 16384 2024-06-09T22:27:24.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d276aeb80 1 Processor -- start 2024-06-09T22:27:24.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d276aeb80 1 -- start start 2024-06-09T22:27:24.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d276aeb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece7af20 0x5632ece7a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d276aeb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80640 0x5632ece81340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d276aeb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece80f60 0x5632ece81880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d276aeb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181ae460 con 0x5632ece7af20 2024-06-09T22:27:24.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d276aeb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181affa0 con 0x5632ece80f60 2024-06-09T22:27:24.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d276aeb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181aeb30 con 0x5632ece80640 2024-06-09T22:27:24.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d1659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece7af20 0x5632ece7a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d1659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece7af20 0x5632ece7a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42968/0 (socket says 172.21.15.104:42968) 2024-06-09T22:27:24.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d1659c700 1 -- 172.21.15.104:0/3493154266 learned_addr learned my addr 172.21.15.104:0/3493154266 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:24.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d15d9b700 1 --2- 172.21.15.104:0/3493154266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80640 0x5632ece81340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d1659c700 1 -- 172.21.15.104:0/3493154266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80640 msgr2=0x5632ece81340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:24.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d1659c700 1 --2- 172.21.15.104:0/3493154266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80640 0x5632ece81340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d1659c700 1 -- 172.21.15.104:0/3493154266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece80f60 msgr2=0x5632ece81880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d16d9d700 1 --2- 172.21.15.104:0/3493154266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece80f60 0x5632ece81880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d1659c700 1 --2- 172.21.15.104:0/3493154266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece80f60 0x5632ece81880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d1659c700 1 -- 172.21.15.104:0/3493154266 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632ece76520 con 0x5632ece7af20 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.260+0000 7f7d16d9d700 1 --2- 172.21.15.104:0/3493154266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece80f60 0x5632ece81880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7d1659c700 1 --2- 172.21.15.104:0/3493154266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece7af20 0x5632ece7a9e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7d08005f50 tx=0x7f7d0801a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda2ca6e9ea92764 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7cfffff700 1 -- 172.21.15.104:0/3493154266 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08024740 con 0x5632ece7af20 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7cfffff700 1 -- 172.21.15.104:0/3493154266 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d08024dc0 con 0x5632ece7af20 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7cfffff700 1 -- 172.21.15.104:0/3493154266 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0802f400 con 0x5632ece7af20 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7d276aeb80 1 -- 172.21.15.104:0/3493154266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece7af20 msgr2=0x5632ece7a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/3493154266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece7af20 0x5632ece7a9e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7d08005f50 tx=0x7f7d0801a040 comp rx=0 tx=0).stop 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7d276aeb80 1 -- 172.21.15.104:0/3493154266 shutdown_connections 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/3493154266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece7af20 0x5632ece7a9e0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/3493154266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece80f60 0x5632ece81880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/3493154266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80640 0x5632ece81340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7d276aeb80 1 -- 172.21.15.104:0/3493154266 >> 172.21.15.104:0/3493154266 conn(0x5632ecda5cc0 msgr2=0x5632ece83f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7d276aeb80 1 -- 172.21.15.104:0/3493154266 shutdown_connections 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.261+0000 7f7d276aeb80 1 -- 172.21.15.104:0/3493154266 wait complete. 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d276aeb80 1 -- start start 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d276aeb80 1 Processor -- start 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d276aeb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 0x5632ece8cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d276aeb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece8d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d276aeb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 0x5632ece9b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d276aeb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181af8d0 con 0x5632ece80640 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d276aeb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181af200 con 0x5632ece7af20 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d276aeb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181ac3e0 con 0x5632ece80f60 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d276aeb80 10 client.0 _ll_register_callbacks cb 0x5632ecd66920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d15d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece8d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d15d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece8d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42972/0 (socket says 172.21.15.104:42972) 2024-06-09T22:27:24.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d15d9b700 1 -- 172.21.15.104:0/748714910 learned_addr learned my addr 172.21.15.104:0/748714910 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:24.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d16d9d700 1 --2- 172.21.15.104:0/748714910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 0x5632ece9b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d15d9b700 1 -- 172.21.15.104:0/748714910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 msgr2=0x5632ece9b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:24.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d15d9b700 1 --2- 172.21.15.104:0/748714910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 0x5632ece9b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d15d9b700 1 -- 172.21.15.104:0/748714910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 msgr2=0x5632ece8cf30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-09T22:27:24.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d15d9b700 1 --2- 172.21.15.104:0/748714910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 0x5632ece8cf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.262+0000 7f7d15d9b700 1 -- 172.21.15.104:0/748714910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632ece9bfe0 con 0x5632ece80640 2024-06-09T22:27:24.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7d15d9b700 1 --2- 172.21.15.104:0/748714910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece8d470 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c007c40 tx=0x7f7d0c012b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97c9ce44250c0f2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:24.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7cfdffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:24.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7cfdffb700 1 -- 172.21.15.104:0/748714910 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c005690 con 0x5632ece80640 2024-06-09T22:27:24.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7d276aeb80 1 -- 172.21.15.104:0/748714910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d08024dc0 con 0x5632ece80640 2024-06-09T22:27:24.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7cfdffb700 1 -- 172.21.15.104:0/748714910 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7d0c01b440 con 0x5632ece80640 2024-06-09T22:27:24.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7cfdffb700 1 -- 172.21.15.104:0/748714910 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7d0c01b640 con 0x5632ece80640 2024-06-09T22:27:24.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7d276aeb80 10 client.17253 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:24.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7d276aeb80 1 -- 172.21.15.104:0/748714910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5632ecefee10 con 0x5632ece80640 2024-06-09T22:27:24.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7cfdffb700 1 -- 172.21.15.104:0/748714910 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d08024dc0 con 0x5632ece80640 2024-06-09T22:27:24.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7cfdffb700 1 -- 172.21.15.104:0/748714910 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7d0c019040 con 0x5632ece80640 2024-06-09T22:27:24.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7d276aeb80 10 client.17253 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:24.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7d276aeb80 10 client.17253 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:27:24.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7d276aeb80 20 client.17253 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:24.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/748714910 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5632eceff510 0x5632ecf01930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:24.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7d276aeb80 4 client.17253 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:27:24.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.263+0000 7f7d276aeb80 1 -- 172.21.15.104:0/748714910 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5632ecdd4290 con 0x5632eceff510 2024-06-09T22:27:24.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7d1659c700 1 --2- 172.21.15.104:0/748714910 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5632eceff510 0x5632ecf01930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:24.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7d1659c700 1 --2- 172.21.15.104:0/748714910 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5632eceff510 0x5632ecf01930 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=47f16fd55805a2f6 server_cookie=fe3adf069d28e3f0 in_seq=0 out_seq=0 2024-06-09T22:27:24.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7cfdffb700 10 client.17253 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:27:24.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7cfdffb700 1 -- 172.21.15.104:0/748714910 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5632ecdd4290 con 0x5632eceff510 2024-06-09T22:27:24.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7cfdffb700 10 client.17253 handle_command_reply: tid=0 2024-06-09T22:27:24.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7d276aeb80 1 client.17253 shutdown 2024-06-09T22:27:24.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7d276aeb80 1 -- 172.21.15.104:0/748714910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 msgr2=0x5632ece8d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:24.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/748714910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece8d470 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c007c40 tx=0x7f7d0c012b30 comp rx=0 tx=0).stop 2024-06-09T22:27:24.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7d276aeb80 1 -- 172.21.15.104:0/748714910 shutdown_connections 2024-06-09T22:27:24.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/748714910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632ece80f60 0x5632ece9b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/748714910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5632ece7af20 0x5632ece8cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/748714910 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5632eceff510 0x5632ecf01930 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7d276aeb80 1 --2- 172.21.15.104:0/748714910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5632ece80640 0x5632ece8d470 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7d276aeb80 1 -- 172.21.15.104:0/748714910 >> 172.21.15.104:0/748714910 conn(0x5632ecda5cc0 msgr2=0x5632ece81340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:24.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7cfdffb700 0 client.17253 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:27:24.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.264+0000 7f7d276aeb80 1 -- 172.21.15.104:0/748714910 shutdown_connections 2024-06-09T22:27:24.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d276aeb80 1 -- 172.21.15.104:0/748714910 wait complete. 2024-06-09T22:27:24.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d276aeb80 20 client.17253 trim_cache size 0 max 16384 2024-06-09T22:27:24.273 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:27:24.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3545045785 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7d000a2d90 msgr2=0x7f7d000a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:24.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d1ce2b700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7d000a2d90 0x7f7d000a5250 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800b5c0 tx=0x7f7d08005960 comp rx=0 tx=0).stop 2024-06-09T22:27:24.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3545045785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d18107400 msgr2=0x7f7d18198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:24.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d1ce2b700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d18107400 0x7f7d18198a30 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000b970 tx=0x7f7d1000bc80 comp rx=0 tx=0).stop 2024-06-09T22:27:24.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3545045785 shutdown_connections 2024-06-09T22:27:24.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d1ce2b700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7d000a2d90 0x7f7d000a5250 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d1ce2b700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d18107400 0x7f7d18198a30 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d1ce2b700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d18101390 0x7f7d1819e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d1ce2b700 1 --2- 172.21.15.104:0/3545045785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d18101d50 0x7f7d1819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:24.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3545045785 >> 172.21.15.104:0/3545045785 conn(0x7f7d18078340 msgr2=0x7f7d181005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:24.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3545045785 shutdown_connections 2024-06-09T22:27:24.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:24.265+0000 7f7d1ce2b700 1 -- 172.21.15.104:0/3545045785 wait complete. 2024-06-09T22:27:24.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.11 MB/sec execute 150 sec latency 20036.687 ms 2024-06-09T22:27:25.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-09T22:27:25.326 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:27:25.701 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:25 smithi104 ceph-mon[32389]: pgmap v196: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 232 KiB/s wr, 23 op/s 2024-06-09T22:27:25.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.08 MB/sec execute 151 sec latency 21036.742 ms 2024-06-09T22:27:25.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:25 smithi153 ceph-mon[36599]: pgmap v196: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 232 KiB/s wr, 23 op/s 2024-06-09T22:27:26.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:25 smithi194 ceph-mon[36856]: pgmap v196: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 232 KiB/s wr, 23 op/s 2024-06-09T22:27:26.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.06 MB/sec execute 152 sec latency 22036.793 ms 2024-06-09T22:27:27.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.300+0000 7f7a5aaca700 1 -- 172.21.15.104:0/1371408033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a54102e80 msgr2=0x7f7a5410f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:27.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.300+0000 7f7a5aaca700 1 --2- 172.21.15.104:0/1371408033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a54102e80 0x7f7a5410f330 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00b0a0 tx=0x7f7a4c00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.300+0000 7f7a5aaca700 1 -- 172.21.15.104:0/1371408033 shutdown_connections 2024-06-09T22:27:27.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.300+0000 7f7a5aaca700 1 --2- 172.21.15.104:0/1371408033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a54102e80 0x7f7a5410f330 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.300+0000 7f7a5aaca700 1 --2- 172.21.15.104:0/1371408033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7a541084f0 0x7f7a541088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.300+0000 7f7a5aaca700 1 --2- 172.21.15.104:0/1371408033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a541024e0 0x7f7a54102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.300+0000 7f7a5aaca700 1 -- 172.21.15.104:0/1371408033 >> 172.21.15.104:0/1371408033 conn(0x7f7a54079530 msgr2=0x7f7a54079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:27.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.301+0000 7f7a5aaca700 1 -- 172.21.15.104:0/1371408033 shutdown_connections 2024-06-09T22:27:27.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.301+0000 7f7a5aaca700 1 -- 172.21.15.104:0/1371408033 wait complete. 2024-06-09T22:27:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.301+0000 7f7a5aaca700 1 Processor -- start 2024-06-09T22:27:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.301+0000 7f7a5aaca700 1 -- start start 2024-06-09T22:27:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a5aaca700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a541024e0 0x7f7a5419e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a5aaca700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7a54102e80 0x7f7a5419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a5aaca700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a541084f0 0x7f7a541989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a5aaca700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7a541127f0 con 0x7f7a541084f0 2024-06-09T22:27:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a5aaca700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a54112670 con 0x7f7a541024e0 2024-06-09T22:27:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a5aaca700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7a54112970 con 0x7f7a54102e80 2024-06-09T22:27:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a59067700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a541084f0 0x7f7a541989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a59067700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a541084f0 0x7f7a541989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49066/0 (socket says 172.21.15.104:49066) 2024-06-09T22:27:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a59067700 1 -- 172.21.15.104:0/668644513 learned_addr learned my addr 172.21.15.104:0/668644513 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a53fff700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7a54102e80 0x7f7a5419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a58866700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a541024e0 0x7f7a5419e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a59067700 1 -- 172.21.15.104:0/668644513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7a54102e80 msgr2=0x7f7a5419ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a59067700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7a54102e80 0x7f7a5419ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a59067700 1 -- 172.21.15.104:0/668644513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a541024e0 msgr2=0x7f7a5419e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a59067700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a541024e0 0x7f7a5419e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a59067700 1 -- 172.21.15.104:0/668644513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a54199110 con 0x7f7a541084f0 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a59067700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a541084f0 0x7f7a541989b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c0062a0 tx=0x7f7a4c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a51ffb700 1 -- 172.21.15.104:0/668644513 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c021070 con 0x7f7a541084f0 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a51ffb700 1 -- 172.21.15.104:0/668644513 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c007db0 con 0x7f7a541084f0 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a5aaca700 1 -- 172.21.15.104:0/668644513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a541993a0 con 0x7f7a541084f0 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.302+0000 7f7a51ffb700 1 -- 172.21.15.104:0/668644513 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c005460 con 0x7f7a541084f0 2024-06-09T22:27:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.303+0000 7f7a5aaca700 1 -- 172.21.15.104:0/668644513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a541a56c0 con 0x7f7a541084f0 2024-06-09T22:27:27.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.305+0000 7f7a51ffb700 1 -- 172.21.15.104:0/668644513 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7a4c024070 con 0x7f7a541084f0 2024-06-09T22:27:27.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.305+0000 7f7a51ffb700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7a440a2d90 0x7f7a440a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.305+0000 7f7a58866700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7a440a2d90 0x7f7a440a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.305+0000 7f7a51ffb700 1 -- 172.21.15.104:0/668644513 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7a4c00f020 con 0x7f7a541084f0 2024-06-09T22:27:27.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.306+0000 7f7a58866700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7a440a2d90 0x7f7a440a5250 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7a54199af0 tx=0x7f7a48005d70 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:27.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.306+0000 7f7a6534db80 1 Processor -- start 2024-06-09T22:27:27.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.306+0000 7f7a6534db80 1 -- start start 2024-06-09T22:27:27.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.306+0000 7f7a6534db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 0x55a9d51723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.306+0000 7f7a6534db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d51728f0 0x55a9d51766f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.306+0000 7f7a6534db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 0x55a9d5177090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.306+0000 7f7a6534db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a9d50bf810 con 0x55a9d5176c30 2024-06-09T22:27:27.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.306+0000 7f7a6534db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a9d5067720 con 0x55a9d5171fd0 2024-06-09T22:27:27.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.306+0000 7f7a6534db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a9d50bb7d0 con 0x55a9d51728f0 2024-06-09T22:27:27.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.306+0000 7f7a59067700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 0x55a9d5177090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a59067700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 0x55a9d5177090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49082/0 (socket says 172.21.15.104:49082) 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a59067700 1 -- 172.21.15.104:0/1877312844 learned_addr learned my addr 172.21.15.104:0/1877312844 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a58866700 1 --2- 172.21.15.104:0/1877312844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 0x55a9d51723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a53fff700 1 --2- 172.21.15.104:0/1877312844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d51728f0 0x55a9d51766f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a59067700 1 -- 172.21.15.104:0/1877312844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d51728f0 msgr2=0x55a9d51766f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a59067700 1 --2- 172.21.15.104:0/1877312844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d51728f0 0x55a9d51766f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a59067700 1 -- 172.21.15.104:0/1877312844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 msgr2=0x55a9d51723b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a59067700 1 --2- 172.21.15.104:0/1877312844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 0x55a9d51723b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a59067700 1 -- 172.21.15.104:0/1877312844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9d505df50 con 0x55a9d5176c30 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a59067700 1 --2- 172.21.15.104:0/1877312844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 0x55a9d5177090 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c0a5840 tx=0x7f7a4c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480ffd5f83486c9f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a3effd700 1 -- 172.21.15.104:0/1877312844 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c0a5d70 con 0x55a9d5176c30 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a3effd700 1 -- 172.21.15.104:0/1877312844 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c0074e0 con 0x55a9d5176c30 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.307+0000 7f7a3effd700 1 -- 172.21.15.104:0/1877312844 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c0a1170 con 0x55a9d5176c30 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.308+0000 7f7a6534db80 1 -- 172.21.15.104:0/1877312844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 msgr2=0x55a9d5177090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:27.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.308+0000 7f7a6534db80 1 --2- 172.21.15.104:0/1877312844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 0x55a9d5177090 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c0a5840 tx=0x7f7a4c007f60 comp rx=0 tx=0).stop 2024-06-09T22:27:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.308+0000 7f7a6534db80 1 -- 172.21.15.104:0/1877312844 shutdown_connections 2024-06-09T22:27:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.308+0000 7f7a6534db80 1 --2- 172.21.15.104:0/1877312844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 0x55a9d5177090 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.308+0000 7f7a6534db80 1 --2- 172.21.15.104:0/1877312844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 0x55a9d51723b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.308+0000 7f7a6534db80 1 --2- 172.21.15.104:0/1877312844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d51728f0 0x55a9d51766f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.308+0000 7f7a6534db80 1 -- 172.21.15.104:0/1877312844 >> 172.21.15.104:0/1877312844 conn(0x55a9d5098cc0 msgr2=0x55a9d5168850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.308+0000 7f7a6534db80 1 -- 172.21.15.104:0/1877312844 shutdown_connections 2024-06-09T22:27:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.308+0000 7f7a6534db80 1 -- 172.21.15.104:0/1877312844 wait complete. 2024-06-09T22:27:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.308+0000 7f7a6534db80 1 -- start start 2024-06-09T22:27:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a6534db80 1 Processor -- start 2024-06-09T22:27:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a6534db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5171fd0 0x55a9d517e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a6534db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d51728f0 0x55a9d517b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a6534db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 0x55a9d517baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a6534db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a9d517eaf0 con 0x55a9d5176c30 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a6534db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a9d505fbf0 con 0x55a9d51728f0 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a6534db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a9d517ec70 con 0x55a9d5171fd0 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a59067700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 0x55a9d517baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a6534db80 10 client.0 _ll_register_callbacks cb 0x55a9d5059920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a59067700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 0x55a9d517baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49092/0 (socket says 172.21.15.104:49092) 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a59067700 1 -- 172.21.15.104:0/424173724 learned_addr learned my addr 172.21.15.104:0/424173724 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a58866700 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5171fd0 0x55a9d517e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.309+0000 7f7a53fff700 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d51728f0 0x55a9d517b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a59067700 1 -- 172.21.15.104:0/424173724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5171fd0 msgr2=0x55a9d517e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a59067700 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5171fd0 0x55a9d517e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a59067700 1 -- 172.21.15.104:0/424173724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d51728f0 msgr2=0x55a9d517b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a59067700 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d51728f0 0x55a9d517b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a59067700 1 -- 172.21.15.104:0/424173724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9d500e8d0 con 0x55a9d5176c30 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a59067700 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 0x55a9d517baa0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c098e70 tx=0x7f7a4c099430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54535682855d2651 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a33fff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a33fff700 1 -- 172.21.15.104:0/424173724 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c0b5030 con 0x55a9d5176c30 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a33fff700 1 -- 172.21.15.104:0/424173724 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7a4c09b490 con 0x55a9d5176c30 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a6534db80 1 -- 172.21.15.104:0/424173724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a4c0074e0 con 0x55a9d5176c30 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a33fff700 1 -- 172.21.15.104:0/424173724 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7a4c0a5d70 con 0x55a9d5176c30 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a6534db80 10 client.17265 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.310+0000 7f7a6534db80 1 -- 172.21.15.104:0/424173724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a9d517eaf0 con 0x55a9d5176c30 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.311+0000 7f7a33fff700 1 -- 172.21.15.104:0/424173724 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c0074e0 con 0x55a9d5176c30 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.311+0000 7f7a33fff700 1 -- 172.21.15.104:0/424173724 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7a4c011070 con 0x55a9d5176c30 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.311+0000 7f7a6534db80 10 client.17265 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.311+0000 7f7a6534db80 10 client.17265 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:27:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.311+0000 7f7a6534db80 20 client.17265 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.311+0000 7f7a6534db80 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a9d518b850 0x55a9d518dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.311+0000 7f7a6534db80 4 client.17265 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.311+0000 7f7a6534db80 1 -- 172.21.15.104:0/424173724 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9d50ccd10 con 0x55a9d518b850 2024-06-09T22:27:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.311+0000 7f7a58866700 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a9d518b850 0x55a9d518dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.312+0000 7f7a33fff700 10 client.17265 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:27:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.312+0000 7f7a58866700 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a9d518b850 0x55a9d518dc40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1fa0e7b87b3b32 server_cookie=f3c599b794cb21ba in_seq=0 out_seq=0 2024-06-09T22:27:27.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.312+0000 7f7a33fff700 1 -- 172.21.15.104:0/424173724 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a9d50ccd10 con 0x55a9d518b850 2024-06-09T22:27:27.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.312+0000 7f7a33fff700 10 client.17265 handle_command_reply: tid=0 2024-06-09T22:27:27.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.312+0000 7f7a6534db80 1 client.17265 shutdown 2024-06-09T22:27:27.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.312+0000 7f7a6534db80 1 -- 172.21.15.104:0/424173724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 msgr2=0x55a9d517baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:27.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.312+0000 7f7a6534db80 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 0x55a9d517baa0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c098e70 tx=0x7f7a4c099430 comp rx=0 tx=0).stop 2024-06-09T22:27:27.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.312+0000 7f7a6534db80 1 -- 172.21.15.104:0/424173724 shutdown_connections 2024-06-09T22:27:27.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.312+0000 7f7a6534db80 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a9d518b850 0x55a9d518dc40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.313+0000 7f7a6534db80 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d5176c30 0x55a9d517baa0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.313+0000 7f7a33fff700 0 client.17265 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:27:27.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.313+0000 7f7a6534db80 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5171fd0 0x55a9d517e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.313+0000 7f7a6534db80 1 --2- 172.21.15.104:0/424173724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d51728f0 0x55a9d517b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.313+0000 7f7a6534db80 1 -- 172.21.15.104:0/424173724 >> 172.21.15.104:0/424173724 conn(0x55a9d5098cc0 msgr2=0x55a9d516c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:27.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.313+0000 7f7a6534db80 1 -- 172.21.15.104:0/424173724 shutdown_connections 2024-06-09T22:27:27.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.313+0000 7f7a6534db80 1 -- 172.21.15.104:0/424173724 wait complete. 2024-06-09T22:27:27.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.313+0000 7f7a6534db80 20 client.17265 trim_cache size 0 max 16384 2024-06-09T22:27:27.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a6534db80 1 Processor -- start 2024-06-09T22:27:27.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a6534db80 1 -- start start 2024-06-09T22:27:27.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a6534db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 0x55a9d5176290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a6534db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 0x55a9d5193530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a6534db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5176c30 0x55a9d5193a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a6534db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7a541ac1c0 con 0x55a9d51728f0 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a6534db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a541add00 con 0x55a9d5171fd0 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a6534db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7a541ac890 con 0x55a9d5176c30 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a53fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 0x55a9d5193530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a53fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 0x55a9d5193530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49102/0 (socket says 172.21.15.104:49102) 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a53fff700 1 -- 172.21.15.104:0/1742065772 learned_addr learned my addr 172.21.15.104:0/1742065772 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a58866700 1 --2- 172.21.15.104:0/1742065772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 0x55a9d5176290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a59067700 1 --2- 172.21.15.104:0/1742065772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5176c30 0x55a9d5193a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a53fff700 1 -- 172.21.15.104:0/1742065772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5176c30 msgr2=0x55a9d5193a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a53fff700 1 --2- 172.21.15.104:0/1742065772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5176c30 0x55a9d5193a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a53fff700 1 -- 172.21.15.104:0/1742065772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 msgr2=0x55a9d5176290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a53fff700 1 --2- 172.21.15.104:0/1742065772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 0x55a9d5176290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.328+0000 7f7a53fff700 1 -- 172.21.15.104:0/1742065772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9d518e250 con 0x55a9d51728f0 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a53fff700 1 --2- 172.21.15.104:0/1742065772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 0x55a9d5193530 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000ab30 tx=0x7f7a4000ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94b0bc696bb59d8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:27.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a3e7fc700 1 -- 172.21.15.104:0/1742065772 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a400129e0 con 0x55a9d51728f0 2024-06-09T22:27:27.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a3e7fc700 1 -- 172.21.15.104:0/1742065772 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a40012b80 con 0x55a9d51728f0 2024-06-09T22:27:27.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a3e7fc700 1 -- 172.21.15.104:0/1742065772 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a40014670 con 0x55a9d51728f0 2024-06-09T22:27:27.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a6534db80 1 -- 172.21.15.104:0/1742065772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 msgr2=0x55a9d5193530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:27.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a6534db80 1 --2- 172.21.15.104:0/1742065772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 0x55a9d5193530 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000ab30 tx=0x7f7a4000ab60 comp rx=0 tx=0).stop 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a6534db80 1 -- 172.21.15.104:0/1742065772 shutdown_connections 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a6534db80 1 --2- 172.21.15.104:0/1742065772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5176c30 0x55a9d5193a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a6534db80 1 --2- 172.21.15.104:0/1742065772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 0x55a9d5176290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a6534db80 1 --2- 172.21.15.104:0/1742065772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 0x55a9d5193530 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a6534db80 1 -- 172.21.15.104:0/1742065772 >> 172.21.15.104:0/1742065772 conn(0x55a9d5098cc0 msgr2=0x55a9d5175b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a6534db80 1 -- 172.21.15.104:0/1742065772 shutdown_connections 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.329+0000 7f7a6534db80 1 -- 172.21.15.104:0/1742065772 wait complete. 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a6534db80 1 -- start start 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a6534db80 1 Processor -- start 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a6534db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 0x55a9d51844c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a6534db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 0x55a9d517f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a6534db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5176c30 0x55a9d517fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a6534db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7a541ad630 con 0x55a9d51728f0 2024-06-09T22:27:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a6534db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a541acf60 con 0x55a9d5171fd0 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a6534db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7a541aa140 con 0x55a9d5176c30 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a6534db80 10 client.0 _ll_register_callbacks cb 0x55a9d5059920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a53fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 0x55a9d517f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a53fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 0x55a9d517f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49104/0 (socket says 172.21.15.104:49104) 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a53fff700 1 -- 172.21.15.104:0/820048404 learned_addr learned my addr 172.21.15.104:0/820048404 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.330+0000 7f7a59067700 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5176c30 0x55a9d517fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a58866700 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 0x55a9d51844c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a53fff700 1 -- 172.21.15.104:0/820048404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5176c30 msgr2=0x55a9d517fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a53fff700 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5176c30 0x55a9d517fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a53fff700 1 -- 172.21.15.104:0/820048404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 msgr2=0x55a9d51844c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a53fff700 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 0x55a9d51844c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a53fff700 1 -- 172.21.15.104:0/820048404 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9d5180440 con 0x55a9d51728f0 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a53fff700 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 0x55a9d517f840 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f7a4001e040 tx=0x7f7a40012860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f84b420848a5d1e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:27.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a33fff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a33fff700 1 -- 172.21.15.104:0/820048404 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4002b070 con 0x55a9d51728f0 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a6534db80 1 -- 172.21.15.104:0/820048404 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a40012b80 con 0x55a9d51728f0 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a33fff700 1 -- 172.21.15.104:0/820048404 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7a4001c8f0 con 0x55a9d51728f0 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a33fff700 1 -- 172.21.15.104:0/820048404 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7a40037050 con 0x55a9d51728f0 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a6534db80 10 client.17271 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a6534db80 1 -- 172.21.15.104:0/820048404 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a9d51f63d0 con 0x55a9d51728f0 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.331+0000 7f7a33fff700 1 -- 172.21.15.104:0/820048404 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a40012b80 con 0x55a9d51728f0 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.332+0000 7f7a33fff700 1 -- 172.21.15.104:0/820048404 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7a40025530 con 0x55a9d51728f0 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.332+0000 7f7a6534db80 10 client.17271 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.332+0000 7f7a6534db80 10 client.17271 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.332+0000 7f7a6534db80 20 client.17271 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.332+0000 7f7a6534db80 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a9d518b850 0x55a9d51f8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:27.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.332+0000 7f7a6534db80 4 client.17271 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.332+0000 7f7a6534db80 1 -- 172.21.15.104:0/820048404 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a9d50c7290 con 0x55a9d518b850 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.332+0000 7f7a58866700 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a9d518b850 0x55a9d51f8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.332+0000 7f7a33fff700 10 client.17271 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.332+0000 7f7a58866700 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a9d518b850 0x55a9d51f8a10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=20db2ab7a636e48b server_cookie=c42966696362af0d in_seq=0 out_seq=0 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.332+0000 7f7a33fff700 1 -- 172.21.15.104:0/820048404 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a9d50c7290 con 0x55a9d518b850 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.332+0000 7f7a33fff700 10 client.17271 handle_command_reply: tid=0 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a6534db80 1 client.17271 shutdown 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a6534db80 1 -- 172.21.15.104:0/820048404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 msgr2=0x55a9d517f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a6534db80 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 0x55a9d517f840 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f7a4001e040 tx=0x7f7a40012860 comp rx=0 tx=0).stop 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a6534db80 1 -- 172.21.15.104:0/820048404 shutdown_connections 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a6534db80 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55a9d518b850 0x55a9d51f8a10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a33fff700 0 client.17271 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a6534db80 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a9d5176c30 0x55a9d517fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a6534db80 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a9d5171fd0 0x55a9d51844c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a6534db80 1 --2- 172.21.15.104:0/820048404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a9d51728f0 0x55a9d517f840 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a6534db80 1 -- 172.21.15.104:0/820048404 >> 172.21.15.104:0/820048404 conn(0x55a9d5098cc0 msgr2=0x55a9d5176410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:27.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a6534db80 1 -- 172.21.15.104:0/820048404 shutdown_connections 2024-06-09T22:27:27.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a6534db80 1 -- 172.21.15.104:0/820048404 wait complete. 2024-06-09T22:27:27.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a6534db80 20 client.17271 trim_cache size 0 max 16384 2024-06-09T22:27:27.340 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:27:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.333+0000 7f7a5aaca700 1 -- 172.21.15.104:0/668644513 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7a440a2d90 msgr2=0x7f7a440a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.334+0000 7f7a5aaca700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7a440a2d90 0x7f7a440a5250 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7a54199af0 tx=0x7f7a48005d70 comp rx=0 tx=0).stop 2024-06-09T22:27:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.334+0000 7f7a5aaca700 1 -- 172.21.15.104:0/668644513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a541084f0 msgr2=0x7f7a541989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.334+0000 7f7a5aaca700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a541084f0 0x7f7a541989b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c0062a0 tx=0x7f7a4c00bf10 comp rx=0 tx=0).stop 2024-06-09T22:27:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.334+0000 7f7a5aaca700 1 -- 172.21.15.104:0/668644513 shutdown_connections 2024-06-09T22:27:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.334+0000 7f7a5aaca700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7a440a2d90 0x7f7a440a5250 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.334+0000 7f7a5aaca700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a541084f0 0x7f7a541989b0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.334+0000 7f7a5aaca700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a541024e0 0x7f7a5419e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.334+0000 7f7a5aaca700 1 --2- 172.21.15.104:0/668644513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7a54102e80 0x7f7a5419ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.334+0000 7f7a5aaca700 1 -- 172.21.15.104:0/668644513 >> 172.21.15.104:0/668644513 conn(0x7f7a54079530 msgr2=0x7f7a541004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.334+0000 7f7a5aaca700 1 -- 172.21.15.104:0/668644513 shutdown_connections 2024-06-09T22:27:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:27.334+0000 7f7a5aaca700 1 -- 172.21.15.104:0/668644513 wait complete. 2024-06-09T22:27:27.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.03 MB/sec execute 153 sec latency 23036.862 ms 2024-06-09T22:27:27.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:27 smithi153 ceph-mon[36599]: pgmap v197: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 227 KiB/s wr, 16 op/s 2024-06-09T22:27:28.003 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:27 smithi104 ceph-mon[32389]: pgmap v197: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 227 KiB/s wr, 16 op/s 2024-06-09T22:27:28.031 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-09T22:27:28.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:27 smithi194 ceph-mon[36856]: pgmap v197: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 227 KiB/s wr, 16 op/s 2024-06-09T22:27:28.271 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:27:28.411 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:27:28.734 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 4.00 MB/sec execute 154 sec latency 24036.914 ms 2024-06-09T22:27:29.600 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:27:29.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 3.98 MB/sec execute 155 sec latency 25037.002 ms 2024-06-09T22:27:29.848 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:29 smithi104 ceph-mon[32389]: pgmap v198: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 227 KiB/s wr, 16 op/s 2024-06-09T22:27:29.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:29 smithi153 ceph-mon[36599]: pgmap v198: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 227 KiB/s wr, 16 op/s 2024-06-09T22:27:30.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:29 smithi194 ceph-mon[36856]: pgmap v198: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 227 KiB/s wr, 16 op/s 2024-06-09T22:27:30.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 3.95 MB/sec execute 156 sec latency 26037.083 ms 2024-06-09T22:27:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.807+0000 7fdfcbf47700 1 -- 172.21.15.104:0/2011634016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfc4105330 msgr2=0x7fdfc4105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.807+0000 7fdfcbf47700 1 --2- 172.21.15.104:0/2011634016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfc4105330 0x7fdfc4105710 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00099c0 tx=0x7fdfc0009cd0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.807+0000 7fdfcbf47700 1 -- 172.21.15.104:0/2011634016 shutdown_connections 2024-06-09T22:27:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.807+0000 7fdfcbf47700 1 --2- 172.21.15.104:0/2011634016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfc4105330 0x7fdfc4105710 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.807+0000 7fdfcbf47700 1 --2- 172.21.15.104:0/2011634016 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc4107b60 0x7fdfc410e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.807+0000 7fdfcbf47700 1 --2- 172.21.15.104:0/2011634016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfc41071a0 0x7fdfc4107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.807+0000 7fdfcbf47700 1 -- 172.21.15.104:0/2011634016 >> 172.21.15.104:0/2011634016 conn(0x7fdfc40766b0 msgr2=0x7fdfc4076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.807+0000 7fdfcbf47700 1 -- 172.21.15.104:0/2011634016 shutdown_connections 2024-06-09T22:27:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.807+0000 7fdfcbf47700 1 -- 172.21.15.104:0/2011634016 wait complete. 2024-06-09T22:27:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfcbf47700 1 Processor -- start 2024-06-09T22:27:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfcbf47700 1 -- start start 2024-06-09T22:27:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfcbf47700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfc4105330 0x7fdfc4199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfcbf47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfc41071a0 0x7fdfc4199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfcbf47700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc4107b60 0x7fdfc41a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfcbf47700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc4111750 con 0x7fdfc41071a0 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfcbf47700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41115d0 con 0x7fdfc4105330 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfcbf47700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41118d0 con 0x7fdfc4107b60 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfc94e2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfc41071a0 0x7fdfc4199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfc94e2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfc41071a0 0x7fdfc4199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49118/0 (socket says 172.21.15.104:49118) 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfc94e2700 1 -- 172.21.15.104:0/3563312662 learned_addr learned my addr 172.21.15.104:0/3563312662 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfca4e4700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc4107b60 0x7fdfc41a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfc4105330 0x7fdfc4199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfc94e2700 1 -- 172.21.15.104:0/3563312662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc4107b60 msgr2=0x7fdfc41a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc4107b60 0x7fdfc41a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfc94e2700 1 -- 172.21.15.104:0/3563312662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfc4105330 msgr2=0x7fdfc4199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfc4105330 0x7fdfc4199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.808+0000 7fdfc94e2700 1 -- 172.21.15.104:0/3563312662 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc41a4fd0 con 0x7fdfc41071a0 2024-06-09T22:27:30.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.809+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfc41071a0 0x7fdfc4199f50 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400c920 tx=0x7fdfb400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:30.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.809+0000 7fdfbaffd700 1 -- 172.21.15.104:0/3563312662 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4004a30 con 0x7fdfc41071a0 2024-06-09T22:27:30.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.809+0000 7fdfcbf47700 1 -- 172.21.15.104:0/3563312662 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc41a5200 con 0x7fdfc41071a0 2024-06-09T22:27:30.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.809+0000 7fdfbaffd700 1 -- 172.21.15.104:0/3563312662 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfb4005500 con 0x7fdfc41071a0 2024-06-09T22:27:30.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.809+0000 7fdfbaffd700 1 -- 172.21.15.104:0/3563312662 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4007d50 con 0x7fdfc41071a0 2024-06-09T22:27:30.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.809+0000 7fdfcbf47700 1 -- 172.21.15.104:0/3563312662 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc41a5710 con 0x7fdfc41071a0 2024-06-09T22:27:30.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.811+0000 7fdfbaffd700 1 -- 172.21.15.104:0/3563312662 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fdfb4004d60 con 0x7fdfc41071a0 2024-06-09T22:27:30.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.812+0000 7fdfbaffd700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fdfb00a2d90 0x7fdfb00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.812+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fdfb00a2d90 0x7fdfb00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.812+0000 7fdfbaffd700 1 -- 172.21.15.104:0/3563312662 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fdfb40cf650 con 0x7fdfc41071a0 2024-06-09T22:27:30.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.812+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fdfb00a2d90 0x7fdfb00a5250 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0000c00 tx=0x7fdfc0005fb0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:30.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfd67cab80 1 Processor -- start 2024-06-09T22:27:30.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfd67cab80 1 -- start start 2024-06-09T22:27:30.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfd67cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 0x55be6ebaca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfd67cab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6ebacf60 0x55be6eba69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfd67cab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6eba6f20 0x55be6eba7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfd67cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55be6eaf8810 con 0x55be6ebac640 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfd67cab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55be6eaa0720 con 0x55be6ebacf60 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfd67cab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55be6eaf47d0 con 0x55be6eba6f20 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfc9ce3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 0x55be6ebaca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfc9ce3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 0x55be6ebaca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49128/0 (socket says 172.21.15.104:49128) 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfc9ce3700 1 -- 172.21.15.104:0/3666730609 learned_addr learned my addr 172.21.15.104:0/3666730609 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfca4e4700 1 --2- 172.21.15.104:0/3666730609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6eba6f20 0x55be6eba7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/3666730609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6ebacf60 0x55be6eba69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfc9ce3700 1 -- 172.21.15.104:0/3666730609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6eba6f20 msgr2=0x55be6eba7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/3666730609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6eba6f20 0x55be6eba7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfc9ce3700 1 -- 172.21.15.104:0/3666730609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6ebacf60 msgr2=0x55be6eba69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:30.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/3666730609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6ebacf60 0x55be6eba69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfc9ce3700 1 -- 172.21.15.104:0/3666730609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be6ea96f50 con 0x55be6ebac640 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.813+0000 7fdfca4e4700 1 --2- 172.21.15.104:0/3666730609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6eba6f20 0x55be6eba7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.814+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/3666730609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 0x55be6ebaca20 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0015000 tx=0x7fdfc0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6e30a664c2b740 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.814+0000 7fdfaffff700 1 -- 172.21.15.104:0/3666730609 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc000be70 con 0x55be6ebac640 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.814+0000 7fdfaffff700 1 -- 172.21.15.104:0/3666730609 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc000fd50 con 0x55be6ebac640 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.814+0000 7fdfaffff700 1 -- 172.21.15.104:0/3666730609 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc002f400 con 0x55be6ebac640 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.815+0000 7fdfd67cab80 1 -- 172.21.15.104:0/3666730609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 msgr2=0x55be6ebaca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.815+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/3666730609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 0x55be6ebaca20 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0015000 tx=0x7fdfc0020000 comp rx=0 tx=0).stop 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.815+0000 7fdfd67cab80 1 -- 172.21.15.104:0/3666730609 shutdown_connections 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.815+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/3666730609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 0x55be6ebaca20 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.815+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/3666730609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6eba6f20 0x55be6eba7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.815+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/3666730609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6ebacf60 0x55be6eba69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.815+0000 7fdfd67cab80 1 -- 172.21.15.104:0/3666730609 >> 172.21.15.104:0/3666730609 conn(0x55be6ead1cc0 msgr2=0x55be6eba1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:30.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.815+0000 7fdfd67cab80 1 -- 172.21.15.104:0/3666730609 shutdown_connections 2024-06-09T22:27:30.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.815+0000 7fdfd67cab80 1 -- 172.21.15.104:0/3666730609 wait complete. 2024-06-09T22:27:30.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.815+0000 7fdfd67cab80 1 -- start start 2024-06-09T22:27:30.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.815+0000 7fdfd67cab80 1 Processor -- start 2024-06-09T22:27:30.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfd67cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6eba6f20 0x55be6ebb2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfd67cab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebac640 0x55be6ebaffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfd67cab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6ebacf60 0x55be6ebb04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfd67cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55be6ebb3910 con 0x55be6eba6f20 2024-06-09T22:27:30.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfd67cab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55be6ea98bf0 con 0x55be6ebacf60 2024-06-09T22:27:30.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfd67cab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55be6ebb3a90 con 0x55be6ebac640 2024-06-09T22:27:30.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfd67cab80 10 client.0 _ll_register_callbacks cb 0x55be6ea92920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:30.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfc9ce3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6eba6f20 0x55be6ebb2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfc9ce3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6eba6f20 0x55be6ebb2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49140/0 (socket says 172.21.15.104:49140) 2024-06-09T22:27:30.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfc9ce3700 1 -- 172.21.15.104:0/2562837303 learned_addr learned my addr 172.21.15.104:0/2562837303 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:30.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebac640 0x55be6ebaffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfc9ce3700 1 -- 172.21.15.104:0/2562837303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebac640 msgr2=0x55be6ebaffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:30.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebac640 0x55be6ebaffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfc9ce3700 1 -- 172.21.15.104:0/2562837303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6ebacf60 msgr2=0x55be6ebb04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:30.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfca4e4700 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6ebacf60 0x55be6ebb04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6ebacf60 0x55be6ebb04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfc9ce3700 1 -- 172.21.15.104:0/2562837303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be6ea478d0 con 0x55be6eba6f20 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6eba6f20 0x55be6ebb2ca0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000b600 tx=0x7fdfc000fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a090744c554d5b7 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfad7fa700 1 -- 172.21.15.104:0/2562837303 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc003c060 con 0x55be6eba6f20 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfad7fa700 1 -- 172.21.15.104:0/2562837303 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fdfc0024620 con 0x55be6eba6f20 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfd67cab80 1 -- 172.21.15.104:0/2562837303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc000fd50 con 0x55be6eba6f20 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.816+0000 7fdfad7fa700 1 -- 172.21.15.104:0/2562837303 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdfc0043050 con 0x55be6eba6f20 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.817+0000 7fdfd67cab80 10 client.17283 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.817+0000 7fdfd67cab80 1 -- 172.21.15.104:0/2562837303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55be6ebbe1f0 con 0x55be6eba6f20 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.817+0000 7fdfad7fa700 1 -- 172.21.15.104:0/2562837303 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc000fd50 con 0x55be6eba6f20 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.817+0000 7fdfad7fa700 1 -- 172.21.15.104:0/2562837303 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdfc002f400 con 0x55be6eba6f20 2024-06-09T22:27:30.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.817+0000 7fdfd67cab80 10 client.17283 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:30.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.817+0000 7fdfd67cab80 10 client.17283 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:27:30.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.817+0000 7fdfd67cab80 20 client.17283 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:30.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.817+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55be6ebc06c0 0x55be6ebc2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.817+0000 7fdfd67cab80 4 client.17283 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:30.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.817+0000 7fdfd67cab80 1 -- 172.21.15.104:0/2562837303 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be6eb05d10 con 0x55be6ebc06c0 2024-06-09T22:27:30.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.817+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55be6ebc06c0 0x55be6ebc2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.818+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55be6ebc06c0 0x55be6ebc2ab0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d15ba7ed6b707a64 server_cookie=1422b8a796876d39 in_seq=0 out_seq=0 2024-06-09T22:27:30.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.818+0000 7fdfad7fa700 10 client.17283 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:27:30.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.818+0000 7fdfad7fa700 1 -- 172.21.15.104:0/2562837303 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55be6eb05d10 con 0x55be6ebc06c0 2024-06-09T22:27:30.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.818+0000 7fdfad7fa700 10 client.17283 handle_command_reply: tid=0 2024-06-09T22:27:30.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfd67cab80 1 client.17283 shutdown 2024-06-09T22:27:30.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfd67cab80 1 -- 172.21.15.104:0/2562837303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6eba6f20 msgr2=0x55be6ebb2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:30.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6eba6f20 0x55be6ebb2ca0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000b600 tx=0x7fdfc000fc00 comp rx=0 tx=0).stop 2024-06-09T22:27:30.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfd67cab80 1 -- 172.21.15.104:0/2562837303 shutdown_connections 2024-06-09T22:27:30.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55be6ebc06c0 0x55be6ebc2ab0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6eba6f20 0x55be6ebb2ca0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6ebacf60 0x55be6ebb04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/2562837303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebac640 0x55be6ebaffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfad7fa700 0 client.17283 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:27:30.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfd67cab80 1 -- 172.21.15.104:0/2562837303 >> 172.21.15.104:0/2562837303 conn(0x55be6ead1cc0 msgr2=0x55be6eba59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:30.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfd67cab80 1 -- 172.21.15.104:0/2562837303 shutdown_connections 2024-06-09T22:27:30.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfd67cab80 1 -- 172.21.15.104:0/2562837303 wait complete. 2024-06-09T22:27:30.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.819+0000 7fdfd67cab80 20 client.17283 trim_cache size 0 max 16384 2024-06-09T22:27:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfd67cab80 1 Processor -- start 2024-06-09T22:27:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfd67cab80 1 -- start start 2024-06-09T22:27:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfd67cab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6eba6f20 0x55be6eba69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfd67cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 0x55be6eb9ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfd67cab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebacf60 0x55be6eb9b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfd67cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41ae3f0 con 0x55be6ebac640 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfd67cab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41aff30 con 0x55be6eba6f20 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfd67cab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41aeac0 con 0x55be6ebacf60 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfc94e2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 0x55be6eb9ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfc94e2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 0x55be6eb9ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49154/0 (socket says 172.21.15.104:49154) 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfc94e2700 1 -- 172.21.15.104:0/1621578768 learned_addr learned my addr 172.21.15.104:0/1621578768 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfc94e2700 1 -- 172.21.15.104:0/1621578768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebacf60 msgr2=0x55be6eb9b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/1621578768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebacf60 0x55be6eb9b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfc94e2700 1 -- 172.21.15.104:0/1621578768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6eba6f20 msgr2=0x55be6eba69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/1621578768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6eba6f20 0x55be6eba69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/1621578768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6eba6f20 0x55be6eba69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfc94e2700 1 -- 172.21.15.104:0/1621578768 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be6ebb3990 con 0x55be6ebac640 2024-06-09T22:27:30.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.834+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/1621578768 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 0x55be6eb9ae00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400c8f0 tx=0x7fdfb4086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4025f95279905019 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:30.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.835+0000 7fdfaf7fe700 1 -- 172.21.15.104:0/1621578768 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb40b3070 con 0x55be6ebac640 2024-06-09T22:27:30.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.835+0000 7fdfaf7fe700 1 -- 172.21.15.104:0/1621578768 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfb40ce040 con 0x55be6ebac640 2024-06-09T22:27:30.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.835+0000 7fdfaf7fe700 1 -- 172.21.15.104:0/1621578768 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb401d040 con 0x55be6ebac640 2024-06-09T22:27:30.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.835+0000 7fdfd67cab80 1 -- 172.21.15.104:0/1621578768 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 msgr2=0x55be6eb9ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:30.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.835+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/1621578768 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 0x55be6eb9ae00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400c8f0 tx=0x7fdfb4086040 comp rx=0 tx=0).stop 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.835+0000 7fdfd67cab80 1 -- 172.21.15.104:0/1621578768 shutdown_connections 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.835+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/1621578768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebacf60 0x55be6eb9b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.835+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/1621578768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6eba6f20 0x55be6eba69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.835+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/1621578768 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebac640 0x55be6eb9ae00 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.835+0000 7fdfd67cab80 1 -- 172.21.15.104:0/1621578768 >> 172.21.15.104:0/1621578768 conn(0x55be6ead1cc0 msgr2=0x55be6ebaa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.835+0000 7fdfd67cab80 1 -- 172.21.15.104:0/1621578768 shutdown_connections 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.835+0000 7fdfd67cab80 1 -- 172.21.15.104:0/1621578768 wait complete. 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfd67cab80 1 -- start start 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfd67cab80 1 Processor -- start 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfd67cab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6eba6f20 0x55be6ebc3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfd67cab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebac640 0x55be6ebc3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfd67cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebacf60 0x55be6ec28ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfd67cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41af860 con 0x55be6ebacf60 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfd67cab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41af190 con 0x55be6eba6f20 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfd67cab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41ac370 con 0x55be6ebac640 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfd67cab80 10 client.0 _ll_register_callbacks cb 0x55be6ea92920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfca4e4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebacf60 0x55be6ec28ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfc9ce3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6eba6f20 0x55be6ebc3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfc9ce3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6eba6f20 0x55be6ebc3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:38756/0 (socket says 172.21.15.104:38756) 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfc94e2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebac640 0x55be6ebc3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfc9ce3700 1 -- 172.21.15.104:0/316982534 learned_addr learned my addr 172.21.15.104:0/316982534 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.836+0000 7fdfca4e4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebacf60 0x55be6ec28ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49162/0 (socket says 172.21.15.104:49162) 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfc9ce3700 1 -- 172.21.15.104:0/316982534 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebac640 msgr2=0x55be6ebc3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebac640 0x55be6ebc3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfc9ce3700 1 -- 172.21.15.104:0/316982534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebacf60 msgr2=0x55be6ec28ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebacf60 0x55be6ec28ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfc9ce3700 1 -- 172.21.15.104:0/316982534 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be6ec295b0 con 0x55be6eba6f20 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebac640 0x55be6ebc3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfca4e4700 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebacf60 0x55be6ec28ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfc9ce3700 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6eba6f20 0x55be6ebc3020 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000bab0 tx=0x7fdfc00094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ceb70c516c46b61 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfad7fa700 1 -- 172.21.15.104:0/316982534 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc003c070 con 0x55be6eba6f20 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfd67cab80 1 -- 172.21.15.104:0/316982534 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfb40ce040 con 0x55be6eba6f20 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfad7fa700 1 -- 172.21.15.104:0/316982534 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fdfc002f9f0 con 0x55be6eba6f20 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfad7fa700 1 -- 172.21.15.104:0/316982534 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdfc002fbf0 con 0x55be6eba6f20 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfd67cab80 10 client.26540 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.837+0000 7fdfd67cab80 1 -- 172.21.15.104:0/316982534 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55be6ec2a890 con 0x55be6eba6f20 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.838+0000 7fdfad7fa700 1 -- 172.21.15.104:0/316982534 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfb40ce040 con 0x55be6eba6f20 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.838+0000 7fdfad7fa700 1 -- 172.21.15.104:0/316982534 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdfc0037ae0 con 0x55be6eba6f20 2024-06-09T22:27:30.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.838+0000 7fdfd67cab80 10 client.26540 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.838+0000 7fdfd67cab80 10 client.26540 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.838+0000 7fdfd67cab80 20 client.26540 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.838+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55be6ec2ae80 0x55be6ec2d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.838+0000 7fdfd67cab80 4 client.26540 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.838+0000 7fdfd67cab80 1 -- 172.21.15.104:0/316982534 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55be6eb00290 con 0x55be6ec2ae80 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.838+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55be6ec2ae80 0x55be6ec2d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfad7fa700 10 client.26540 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfc94e2700 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55be6ec2ae80 0x55be6ec2d2a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8a000f926bfed0f2 server_cookie=f4ce55b359836da2 in_seq=0 out_seq=0 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfad7fa700 1 -- 172.21.15.104:0/316982534 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55be6eb00290 con 0x55be6ec2ae80 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfad7fa700 10 client.26540 handle_command_reply: tid=0 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfd67cab80 1 client.26540 shutdown 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfd67cab80 1 -- 172.21.15.104:0/316982534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6eba6f20 msgr2=0x55be6ebc3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:30.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6eba6f20 0x55be6ebc3020 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000bab0 tx=0x7fdfc00094b0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfd67cab80 1 -- 172.21.15.104:0/316982534 shutdown_connections 2024-06-09T22:27:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55be6ec2ae80 0x55be6ec2d2a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55be6ebacf60 0x55be6ec28ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55be6eba6f20 0x55be6ebc3020 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfd67cab80 1 --2- 172.21.15.104:0/316982534 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be6ebac640 0x55be6ebc3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfd67cab80 1 -- 172.21.15.104:0/316982534 >> 172.21.15.104:0/316982534 conn(0x55be6ead1cc0 msgr2=0x55be6eb9ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfad7fa700 0 client.26540 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:27:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfd67cab80 1 -- 172.21.15.104:0/316982534 shutdown_connections 2024-06-09T22:27:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfd67cab80 1 -- 172.21.15.104:0/316982534 wait complete. 2024-06-09T22:27:30.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.839+0000 7fdfd67cab80 20 client.26540 trim_cache size 0 max 16384 2024-06-09T22:27:30.846 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:27:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.840+0000 7fdfcbf47700 1 -- 172.21.15.104:0/3563312662 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fdfb00a2d90 msgr2=0x7fdfb00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.840+0000 7fdfcbf47700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fdfb00a2d90 0x7fdfb00a5250 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0000c00 tx=0x7fdfc0005fb0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.840+0000 7fdfcbf47700 1 -- 172.21.15.104:0/3563312662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfc41071a0 msgr2=0x7fdfc4199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.840+0000 7fdfcbf47700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfc41071a0 0x7fdfc4199f50 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400c920 tx=0x7fdfb400cc30 comp rx=0 tx=0).stop 2024-06-09T22:27:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.840+0000 7fdfcbf47700 1 -- 172.21.15.104:0/3563312662 shutdown_connections 2024-06-09T22:27:30.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.840+0000 7fdfcbf47700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fdfb00a2d90 0x7fdfb00a5250 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.840+0000 7fdfcbf47700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc4107b60 0x7fdfc41a4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.840+0000 7fdfcbf47700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfc4105330 0x7fdfc4199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.840+0000 7fdfcbf47700 1 --2- 172.21.15.104:0/3563312662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfc41071a0 0x7fdfc4199f50 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.840+0000 7fdfcbf47700 1 -- 172.21.15.104:0/3563312662 >> 172.21.15.104:0/3563312662 conn(0x7fdfc40766b0 msgr2=0x7fdfc40fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.840+0000 7fdfcbf47700 1 -- 172.21.15.104:0/3563312662 shutdown_connections 2024-06-09T22:27:30.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:30.840+0000 7fdfcbf47700 1 -- 172.21.15.104:0/3563312662 wait complete. 2024-06-09T22:27:31.727 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:31 smithi104 ceph-mon[32389]: pgmap v199: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 231 KiB/s wr, 21 op/s 2024-06-09T22:27:31.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 3.93 MB/sec execute 157 sec latency 27037.154 ms 2024-06-09T22:27:31.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.963+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/1537553464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f481013c0 msgr2=0x7f7f48101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:31.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.963+0000 7f7f4fc8c700 1 --2- 172.21.15.104:0/1537553464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f481013c0 0x7f7f48101840 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009990 tx=0x7f7f38009ca0 comp rx=0 tx=0).stop 2024-06-09T22:27:31.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.963+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/1537553464 shutdown_connections 2024-06-09T22:27:31.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.963+0000 7f7f4fc8c700 1 --2- 172.21.15.104:0/1537553464 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f48101d80 0x7f7f4810e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:31.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.963+0000 7f7f4fc8c700 1 --2- 172.21.15.104:0/1537553464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f48107430 0x7f7f48107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:31.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.963+0000 7f7f4fc8c700 1 --2- 172.21.15.104:0/1537553464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f481013c0 0x7f7f48101840 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:31.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.963+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/1537553464 >> 172.21.15.104:0/1537553464 conn(0x7f7f480783d0 msgr2=0x7f7f480787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:31.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.963+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/1537553464 shutdown_connections 2024-06-09T22:27:31.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.963+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/1537553464 wait complete. 2024-06-09T22:27:31.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4fc8c700 1 Processor -- start 2024-06-09T22:27:31.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4fc8c700 1 -- start start 2024-06-09T22:27:31.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4fc8c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f481013c0 0x7f7f48199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:31.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4fc8c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f48101d80 0x7f7f4819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:31.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4fc8c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f48107430 0x7f7f481a4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:31.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4fc8c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7f481116d0 con 0x7f7f48107430 2024-06-09T22:27:31.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4fc8c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f48111550 con 0x7f7f48101d80 2024-06-09T22:27:31.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4fc8c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f48111850 con 0x7f7f481013c0 2024-06-09T22:27:31.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4d227700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f48101d80 0x7f7f4819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:31.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4d227700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f48101d80 0x7f7f4819a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:38776/0 (socket says 172.21.15.104:38776) 2024-06-09T22:27:31.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4da28700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f481013c0 0x7f7f48199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:31.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4d227700 1 -- 172.21.15.104:0/2199404423 learned_addr learned my addr 172.21.15.104:0/2199404423 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.964+0000 7f7f4e229700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f48107430 0x7f7f481a4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.965+0000 7f7f4d227700 1 -- 172.21.15.104:0/2199404423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f481013c0 msgr2=0x7f7f48199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.965+0000 7f7f4d227700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f481013c0 0x7f7f48199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.965+0000 7f7f4d227700 1 -- 172.21.15.104:0/2199404423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f48107430 msgr2=0x7f7f481a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.965+0000 7f7f4d227700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f48107430 0x7f7f481a4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.965+0000 7f7f4d227700 1 -- 172.21.15.104:0/2199404423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f481a50d0 con 0x7f7f48101d80 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.965+0000 7f7f4e229700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f48107430 0x7f7f481a4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.965+0000 7f7f4d227700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f48101d80 0x7f7f4819a050 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7f38000c00 tx=0x7f7f38003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.965+0000 7f7f3effd700 1 -- 172.21.15.104:0/2199404423 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38027070 con 0x7f7f48101d80 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.965+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/2199404423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f48197900 con 0x7f7f48101d80 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.965+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/2199404423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f48197e30 con 0x7f7f48101d80 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.965+0000 7f7f3effd700 1 -- 172.21.15.104:0/2199404423 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f38004050 con 0x7f7f48101d80 2024-06-09T22:27:31.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.965+0000 7f7f3effd700 1 -- 172.21.15.104:0/2199404423 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3800f460 con 0x7f7f48101d80 2024-06-09T22:27:31.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.966+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/2199404423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f2c005330 con 0x7f7f48101d80 2024-06-09T22:27:31.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.967+0000 7f7f3effd700 1 -- 172.21.15.104:0/2199404423 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7f3802a070 con 0x7f7f48101d80 2024-06-09T22:27:31.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.968+0000 7f7f3effd700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f340a2d90 0x7f7f340a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:31.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.968+0000 7f7f4da28700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f340a2d90 0x7f7f340a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:31.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.968+0000 7f7f3effd700 1 -- 172.21.15.104:0/2199404423 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7f380d0660 con 0x7f7f48101d80 2024-06-09T22:27:31.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.968+0000 7f7f4da28700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f340a2d90 0x7f7f340a5250 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7f440071c0 tx=0x7f7f44010040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:31.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:31.971+0000 7f7f3effd700 1 -- 172.21.15.104:0/2199404423 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7f38013070 con 0x7f7f48101d80 2024-06-09T22:27:31.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:31 smithi153 ceph-mon[36599]: pgmap v199: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 231 KiB/s wr, 21 op/s 2024-06-09T22:27:32.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:31 smithi194 ceph-mon[36856]: pgmap v199: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 231 KiB/s wr, 21 op/s 2024-06-09T22:27:32.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.217+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/2199404423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f2c005620 con 0x7f7f48101d80 2024-06-09T22:27:32.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.218+0000 7f7f3effd700 1 -- 172.21.15.104:0/2199404423 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7f7f3809cfa0 con 0x7f7f48101d80 2024-06-09T22:27:32.220 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:27:32.220 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:27:32.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.220+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/2199404423 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f340a2d90 msgr2=0x7f7f340a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:32.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.220+0000 7f7f4fc8c700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f340a2d90 0x7f7f340a5250 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7f440071c0 tx=0x7f7f44010040 comp rx=0 tx=0).stop 2024-06-09T22:27:32.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.220+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/2199404423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f48101d80 msgr2=0x7f7f4819a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:32.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.220+0000 7f7f4fc8c700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f48101d80 0x7f7f4819a050 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7f38000c00 tx=0x7f7f38003680 comp rx=0 tx=0).stop 2024-06-09T22:27:32.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.220+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/2199404423 shutdown_connections 2024-06-09T22:27:32.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.220+0000 7f7f4fc8c700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f340a2d90 0x7f7f340a5250 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:32.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.220+0000 7f7f4fc8c700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f48107430 0x7f7f481a4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.220+0000 7f7f4fc8c700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f481013c0 0x7f7f48199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.220+0000 7f7f4fc8c700 1 --2- 172.21.15.104:0/2199404423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f48101d80 0x7f7f4819a050 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.220+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/2199404423 >> 172.21.15.104:0/2199404423 conn(0x7f7f480783d0 msgr2=0x7f7f48100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.220+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/2199404423 shutdown_connections 2024-06-09T22:27:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:32.220+0000 7f7f4fc8c700 1 -- 172.21.15.104:0/2199404423 wait complete. 2024-06-09T22:27:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:27:32.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 3.90 MB/sec execute 158 sec latency 28037.265 ms 2024-06-09T22:27:32.987 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:32 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2199404423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:27:32.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:32 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2199404423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:27:33.015 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24460 quiesce db --set-id 6813a49 --release --await' 2024-06-09T22:27:33.015 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24460 quiesce db --set-id 6813a49 --release --await 2024-06-09T22:27:33.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:32 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2199404423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:27:33.477 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-09T22:27:33.588 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:27:33.710 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:33 smithi104 ceph-mon[32389]: pgmap v200: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 14 op/s 2024-06-09T22:27:33.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 3.88 MB/sec execute 159 sec latency 29037.310 ms 2024-06-09T22:27:33.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:33 smithi153 ceph-mon[36599]: pgmap v200: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 14 op/s 2024-06-09T22:27:34.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:33 smithi194 ceph-mon[36856]: pgmap v200: 129 pgs: 129 active+clean; 134 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 14 op/s 2024-06-09T22:27:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.387+0000 7f473255a640 1 -- 172.21.15.104:0/2692986949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f472c10c000 msgr2=0x7f472c10e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.387+0000 7f473255a640 1 --2- 172.21.15.104:0/2692986949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f472c10c000 0x7f472c10e430 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f4724009ca0 tx=0x7f472402a950 comp rx=0 tx=0).stop 2024-06-09T22:27:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.387+0000 7f473255a640 1 -- 172.21.15.104:0/2692986949 shutdown_connections 2024-06-09T22:27:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.387+0000 7f473255a640 1 --2- 172.21.15.104:0/2692986949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f472c10c000 0x7f472c10e430 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.387+0000 7f473255a640 1 --2- 172.21.15.104:0/2692986949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f472c103550 0x7f472c10ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.387+0000 7f473255a640 1 --2- 172.21.15.104:0/2692986949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f472c102c30 0x7f472c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.387+0000 7f473255a640 1 -- 172.21.15.104:0/2692986949 >> 172.21.15.104:0/2692986949 conn(0x7f472c0779f0 msgr2=0x7f472c077e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.387+0000 7f473255a640 1 -- 172.21.15.104:0/2692986949 shutdown_connections 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.387+0000 7f473255a640 1 -- 172.21.15.104:0/2692986949 wait complete. 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f473255a640 1 Processor -- start 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f473255a640 1 -- start start 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f473255a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f472c102c30 0x7f472c1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f473255a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f472c103550 0x7f472c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f473255a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f472c10c000 0x7f472c19ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f473255a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f472c111d90 con 0x7f472c10c000 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f473255a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f472c111c10 con 0x7f472c102c30 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f473255a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f472c111f10 con 0x7f472c103550 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f4730af7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f472c10c000 0x7f472c19ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f4730af7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f472c10c000 0x7f472c19ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49196/0 (socket says 172.21.15.104:49196) 2024-06-09T22:27:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f4730af7640 1 -- 172.21.15.104:0/4197269623 learned_addr learned my addr 172.21.15.104:0/4197269623 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f472bfff640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f472c102c30 0x7f472c1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f472b7fe640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f472c103550 0x7f472c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f4730af7640 1 -- 172.21.15.104:0/4197269623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f472c103550 msgr2=0x7f472c198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f4730af7640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f472c103550 0x7f472c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f4730af7640 1 -- 172.21.15.104:0/4197269623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f472c102c30 msgr2=0x7f472c1981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f4730af7640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f472c102c30 0x7f472c1981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f4730af7640 1 -- 172.21.15.104:0/4197269623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f472c19d1c0 con 0x7f472c10c000 2024-06-09T22:27:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.388+0000 7f472bfff640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f472c102c30 0x7f472c1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:27:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.389+0000 7f4730af7640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f472c10c000 0x7f472c19ca90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f472402ae70 tx=0x7f472402f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.389+0000 7f47297fa640 1 -- 172.21.15.104:0/4197269623 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4724034040 con 0x7f472c10c000 2024-06-09T22:27:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.389+0000 7f473255a640 1 -- 172.21.15.104:0/4197269623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f472c19d450 con 0x7f472c10c000 2024-06-09T22:27:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.389+0000 7f47297fa640 1 -- 172.21.15.104:0/4197269623 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4724030920 con 0x7f472c10c000 2024-06-09T22:27:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.389+0000 7f47297fa640 1 -- 172.21.15.104:0/4197269623 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4724045480 con 0x7f472c10c000 2024-06-09T22:27:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.389+0000 7f473255a640 1 -- 172.21.15.104:0/4197269623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f472c1a4d40 con 0x7f472c10c000 2024-06-09T22:27:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.391+0000 7f47297fa640 1 -- 172.21.15.104:0/4197269623 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f4724037020 con 0x7f472c10c000 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.391+0000 7f4735264740 1 Processor -- start 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.391+0000 7f4735264740 1 -- start start 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f47297fa640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f47180a2da0 0x7f47180a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f4735264740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68c170 0x564f9e68c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f4735264740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e68ca90 0x564f9e696e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f4735264740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e697350 0x564f9e699740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f4735264740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564f9e5f04f0 con 0x564f9e68ca90 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f4735264740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564f9e58e870 con 0x564f9e68c170 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f4735264740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564f9e5ec4b0 con 0x564f9e697350 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f472b7fe640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f47180a2da0 0x7f47180a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f472b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e68ca90 0x564f9e696e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f47297fa640 1 -- 172.21.15.104:0/4197269623 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f47240f06d0 con 0x7f472c10c000 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f4730af7640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e697350 0x564f9e699740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f472b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e68ca90 0x564f9e696e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49200/0 (socket says 172.21.15.104:49200) 2024-06-09T22:27:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f472b7fe640 1 -- 172.21.15.104:0/2711431704 learned_addr learned my addr 172.21.15.104:0/2711431704 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f472bfff640 1 --2- 172.21.15.104:0/2711431704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68c170 0x564f9e68c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f472b7fe640 1 -- 172.21.15.104:0/2711431704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e697350 msgr2=0x564f9e699740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f472b7fe640 1 --2- 172.21.15.104:0/2711431704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e697350 0x564f9e699740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f472b7fe640 1 -- 172.21.15.104:0/2711431704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68c170 msgr2=0x564f9e68c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f472b7fe640 1 --2- 172.21.15.104:0/2711431704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68c170 0x564f9e68c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f472b7fe640 1 -- 172.21.15.104:0/2711431704 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f9e5aa9b0 con 0x564f9e68ca90 2024-06-09T22:27:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f4730af7640 1 --2- 172.21.15.104:0/2711431704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e697350 0x564f9e699740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f472b7fe640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f47180a2da0 0x7f47180a5260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4714013780 tx=0x7f471400b9c0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f472b7fe640 1 --2- 172.21.15.104:0/2711431704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e68ca90 0x564f9e696e10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f4714013fa0 tx=0x7f471401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9841911e9038f609 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.392+0000 7f47227fc640 1 -- 172.21.15.104:0/2711431704 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471400e730 con 0x564f9e68ca90 2024-06-09T22:27:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.393+0000 7f47227fc640 1 -- 172.21.15.104:0/2711431704 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f471400edb0 con 0x564f9e68ca90 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.393+0000 7f47227fc640 1 -- 172.21.15.104:0/2711431704 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471401fd00 con 0x564f9e68ca90 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.393+0000 7f4735264740 1 -- 172.21.15.104:0/2711431704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e68ca90 msgr2=0x564f9e696e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.393+0000 7f4735264740 1 --2- 172.21.15.104:0/2711431704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e68ca90 0x564f9e696e10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f4714013fa0 tx=0x7f471401a000 comp rx=0 tx=0).stop 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.393+0000 7f4735264740 1 -- 172.21.15.104:0/2711431704 shutdown_connections 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.393+0000 7f4735264740 1 --2- 172.21.15.104:0/2711431704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e697350 0x564f9e699740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.393+0000 7f4735264740 1 --2- 172.21.15.104:0/2711431704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e68ca90 0x564f9e696e10 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.393+0000 7f4735264740 1 --2- 172.21.15.104:0/2711431704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68c170 0x564f9e68c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.393+0000 7f4735264740 1 -- 172.21.15.104:0/2711431704 >> 172.21.15.104:0/2711431704 conn(0x564f9e688680 msgr2=0x564f9e688a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.393+0000 7f4735264740 1 -- 172.21.15.104:0/2711431704 shutdown_connections 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.393+0000 7f4735264740 1 -- 172.21.15.104:0/2711431704 wait complete. 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4735264740 1 -- start start 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4735264740 1 Processor -- start 2024-06-09T22:27:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4735264740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 0x564f9e6942e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4735264740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 0x564f9e694820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4735264740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e694d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4735264740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x564f9e699f20 con 0x564f9e697350 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4735264740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564f9e579ae0 con 0x564f9e68ca90 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4735264740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564f9e69a0a0 con 0x564f9e68c170 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4735264740 10 client.0 _ll_register_callbacks cb 0x564f9e5b29b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4730af7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e694d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4730af7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e694d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49214/0 (socket says 172.21.15.104:49214) 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4730af7640 1 -- 172.21.15.104:0/3864476976 learned_addr learned my addr 172.21.15.104:0/3864476976 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f472bfff640 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 0x564f9e6942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f472b7fe640 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 0x564f9e694820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4730af7640 1 -- 172.21.15.104:0/3864476976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 msgr2=0x564f9e6942e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4730af7640 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 0x564f9e6942e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4730af7640 1 -- 172.21.15.104:0/3864476976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 msgr2=0x564f9e694820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4730af7640 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 0x564f9e694820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.394+0000 7f4730af7640 1 -- 172.21.15.104:0/3864476976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f9e304210 con 0x564f9e697350 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f4730af7640 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e694d60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4724004310 tx=0x7f47240bc590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e533971daa9a8a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f470b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f470b7fe640 1 -- 172.21.15.104:0/3864476976 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472402c040 con 0x564f9e697350 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f470b7fe640 1 -- 172.21.15.104:0/3864476976 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f47240f0340 con 0x564f9e697350 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f4735264740 1 -- 172.21.15.104:0/3864476976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f471400edb0 con 0x564f9e697350 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f470b7fe640 1 -- 172.21.15.104:0/3864476976 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f47240bd090 con 0x564f9e697350 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f4735264740 10 client.17307 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f4735264740 1 -- 172.21.15.104:0/3864476976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564f9e3d0620 con 0x564f9e697350 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f470b7fe640 1 -- 172.21.15.104:0/3864476976 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f471400edb0 con 0x564f9e697350 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f470b7fe640 1 -- 172.21.15.104:0/3864476976 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f472404d030 con 0x564f9e697350 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f4735264740 10 client.17307 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f4735264740 10 client.17307 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f4735264740 20 client.17307 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f4735264740 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564f9e6a7450 0x564f9e6a98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f4735264740 4 client.17307 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f4735264740 1 -- 172.21.15.104:0/3864476976 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f9e603f10 con 0x564f9e6a7450 2024-06-09T22:27:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.395+0000 7f472bfff640 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564f9e6a7450 0x564f9e6a98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.396+0000 7f472bfff640 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564f9e6a7450 0x564f9e6a98b0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e4b019ee3849ec0 server_cookie=165de48e83195549 in_seq=0 out_seq=0 2024-06-09T22:27:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.396+0000 7f470b7fe640 10 client.17307 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f470b7fe640 1 -- 172.21.15.104:0/3864476976 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x564f9e603f10 con 0x564f9e6a7450 2024-06-09T22:27:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f470b7fe640 10 client.17307 handle_command_reply: tid=0 2024-06-09T22:27:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f4735264740 1 client.17307 shutdown 2024-06-09T22:27:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f4735264740 1 -- 172.21.15.104:0/3864476976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 msgr2=0x564f9e694d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f4735264740 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e694d60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4724004310 tx=0x7f47240bc590 comp rx=0 tx=0).stop 2024-06-09T22:27:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f4735264740 1 -- 172.21.15.104:0/3864476976 shutdown_connections 2024-06-09T22:27:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f4735264740 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564f9e6a7450 0x564f9e6a98b0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f4735264740 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e694d60 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f4735264740 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 0x564f9e694820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f470b7fe640 0 client.17307 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f4735264740 1 --2- 172.21.15.104:0/3864476976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 0x564f9e6942e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f4735264740 1 -- 172.21.15.104:0/3864476976 >> 172.21.15.104:0/3864476976 conn(0x564f9e688680 msgr2=0x564f9e68e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f4735264740 1 -- 172.21.15.104:0/3864476976 shutdown_connections 2024-06-09T22:27:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f4735264740 1 -- 172.21.15.104:0/3864476976 wait complete. 2024-06-09T22:27:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.397+0000 7f4735264740 20 client.17307 trim_cache size 0 max 16384 2024-06-09T22:27:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4735264740 1 Processor -- start 2024-06-09T22:27:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4735264740 1 -- start start 2024-06-09T22:27:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4735264740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 0x564f9e687660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4735264740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 0x564f9e6af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4735264740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e6af700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4735264740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f472c1aec10 con 0x564f9e697350 2024-06-09T22:27:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4735264740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f472c1b0750 con 0x564f9e68ca90 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4735264740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f472c1af2e0 con 0x564f9e68c170 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4730af7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e6af700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4730af7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e6af700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49216/0 (socket says 172.21.15.104:49216) 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4730af7640 1 -- 172.21.15.104:0/1473204201 learned_addr learned my addr 172.21.15.104:0/1473204201 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f472b7fe640 1 --2- 172.21.15.104:0/1473204201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 0x564f9e6af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4730af7640 1 -- 172.21.15.104:0/1473204201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 msgr2=0x564f9e687660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4730af7640 1 --2- 172.21.15.104:0/1473204201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 0x564f9e687660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4730af7640 1 -- 172.21.15.104:0/1473204201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 msgr2=0x564f9e6af1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4730af7640 1 --2- 172.21.15.104:0/1473204201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 0x564f9e6af1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4730af7640 1 -- 172.21.15.104:0/1473204201 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f9e6a9f30 con 0x564f9e697350 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.412+0000 7f4730af7640 1 --2- 172.21.15.104:0/1473204201 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e6af700 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f47240bc690 tx=0x7f47240baea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bfa6c335bc35864 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.413+0000 7f4721ffb640 1 -- 172.21.15.104:0/1473204201 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47240b8c30 con 0x564f9e697350 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.413+0000 7f4721ffb640 1 -- 172.21.15.104:0/1473204201 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4724004040 con 0x564f9e697350 2024-06-09T22:27:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.413+0000 7f4721ffb640 1 -- 172.21.15.104:0/1473204201 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47240bcbf0 con 0x564f9e697350 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.413+0000 7f4735264740 1 -- 172.21.15.104:0/1473204201 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 msgr2=0x564f9e6af700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.413+0000 7f4735264740 1 --2- 172.21.15.104:0/1473204201 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e6af700 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f47240bc690 tx=0x7f47240baea0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.413+0000 7f4735264740 1 -- 172.21.15.104:0/1473204201 shutdown_connections 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.413+0000 7f4735264740 1 --2- 172.21.15.104:0/1473204201 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e6af700 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.413+0000 7f4735264740 1 --2- 172.21.15.104:0/1473204201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 0x564f9e6af1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.413+0000 7f4735264740 1 --2- 172.21.15.104:0/1473204201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 0x564f9e687660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.413+0000 7f4735264740 1 -- 172.21.15.104:0/1473204201 >> 172.21.15.104:0/1473204201 conn(0x564f9e688680 msgr2=0x564f9e686f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.413+0000 7f4735264740 1 -- 172.21.15.104:0/1473204201 shutdown_connections 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.413+0000 7f4735264740 1 -- 172.21.15.104:0/1473204201 wait complete. 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4735264740 1 -- start start 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4735264740 1 Processor -- start 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4735264740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 0x564f9e69f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4735264740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 0x564f9e69c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4735264740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e69c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4735264740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f472c1b0080 con 0x564f9e697350 2024-06-09T22:27:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4735264740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f472c1af9b0 con 0x564f9e68ca90 2024-06-09T22:27:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4735264740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f472c1acbd0 con 0x564f9e68c170 2024-06-09T22:27:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4735264740 10 client.0 _ll_register_callbacks cb 0x7f4724005f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4730af7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e69c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4730af7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e69c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49226/0 (socket says 172.21.15.104:49226) 2024-06-09T22:27:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4730af7640 1 -- 172.21.15.104:0/1479327453 learned_addr learned my addr 172.21.15.104:0/1479327453 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f472b7fe640 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 0x564f9e69c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f472bfff640 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 0x564f9e69f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4730af7640 1 -- 172.21.15.104:0/1479327453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 msgr2=0x564f9e69f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4730af7640 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 0x564f9e69f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4730af7640 1 -- 172.21.15.104:0/1479327453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 msgr2=0x564f9e69c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4730af7640 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 0x564f9e69c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.414+0000 7f4730af7640 1 -- 172.21.15.104:0/1479327453 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f9e68d540 con 0x564f9e697350 2024-06-09T22:27:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f4730af7640 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e69c800 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f4724004af0 tx=0x7f47240095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b31623401b053f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f470b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f470b7fe640 1 -- 172.21.15.104:0/1479327453 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472408c070 con 0x564f9e697350 2024-06-09T22:27:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f470b7fe640 1 -- 172.21.15.104:0/1479327453 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f47240edca0 con 0x564f9e697350 2024-06-09T22:27:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f4735264740 1 -- 172.21.15.104:0/1479327453 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4724004040 con 0x564f9e697350 2024-06-09T22:27:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f470b7fe640 1 -- 172.21.15.104:0/1479327453 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f47240b8c30 con 0x564f9e697350 2024-06-09T22:27:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f4735264740 10 client.17313 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f4735264740 1 -- 172.21.15.104:0/1479327453 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564f9e68d540 con 0x564f9e697350 2024-06-09T22:27:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f470b7fe640 1 -- 172.21.15.104:0/1479327453 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4724004040 con 0x564f9e697350 2024-06-09T22:27:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f470b7fe640 1 -- 172.21.15.104:0/1479327453 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f472404d070 con 0x564f9e697350 2024-06-09T22:27:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f4735264740 10 client.17313 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f4735264740 10 client.17313 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f4735264740 20 client.17313 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f4735264740 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564f9e712fd0 0x564f9e7153c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f4735264740 4 client.17313 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:27:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f4735264740 1 -- 172.21.15.104:0/1479327453 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564f9e5fdc20 con 0x564f9e712fd0 2024-06-09T22:27:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.415+0000 7f472bfff640 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564f9e712fd0 0x564f9e7153c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.416+0000 7f470b7fe640 10 client.17313 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.416+0000 7f472bfff640 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564f9e712fd0 0x564f9e7153c0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dae2ce175512040 server_cookie=5e4655103ddd6f92 in_seq=0 out_seq=0 2024-06-09T22:27:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.425+0000 7f470b7fe640 1 -- 172.21.15.104:0/1479327453 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x564f9e5fdc20 con 0x564f9e712fd0 2024-06-09T22:27:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.425+0000 7f470b7fe640 10 client.17313 handle_command_reply: tid=0 2024-06-09T22:27:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.425+0000 7f4735264740 1 client.17313 shutdown 2024-06-09T22:27:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f4735264740 1 -- 172.21.15.104:0/1479327453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 msgr2=0x564f9e69c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f4735264740 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e69c800 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f4724004af0 tx=0x7f47240095c0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f4735264740 1 -- 172.21.15.104:0/1479327453 shutdown_connections 2024-06-09T22:27:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f4735264740 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x564f9e712fd0 0x564f9e7153c0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f4735264740 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x564f9e697350 0x564f9e69c800 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f470b7fe640 0 client.17313 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f4735264740 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f9e68ca90 0x564f9e69c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f4735264740 1 --2- 172.21.15.104:0/1479327453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564f9e68c170 0x564f9e69f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f4735264740 1 -- 172.21.15.104:0/1479327453 >> 172.21.15.104:0/1479327453 conn(0x564f9e688680 msgr2=0x564f9e686f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f4735264740 1 -- 172.21.15.104:0/1479327453 shutdown_connections 2024-06-09T22:27:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f4735264740 1 -- 172.21.15.104:0/1479327453 wait complete. 2024-06-09T22:27:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f4735264740 20 client.17313 trim_cache size 0 max 16384 2024-06-09T22:27:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:27:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:27:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:27:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f473255a640 1 -- 172.21.15.104:0/4197269623 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f47180a2da0 msgr2=0x7f47180a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f473255a640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f47180a2da0 0x7f47180a5260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4714013780 tx=0x7f471400b9c0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f473255a640 1 -- 172.21.15.104:0/4197269623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f472c10c000 msgr2=0x7f472c19ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.426+0000 7f473255a640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f472c10c000 0x7f472c19ca90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f472402ae70 tx=0x7f472402f5b0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.427+0000 7f473255a640 1 -- 172.21.15.104:0/4197269623 shutdown_connections 2024-06-09T22:27:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.427+0000 7f473255a640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f47180a2da0 0x7f47180a5260 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.427+0000 7f473255a640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f472c10c000 0x7f472c19ca90 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.427+0000 7f473255a640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f472c103550 0x7f472c198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.427+0000 7f473255a640 1 --2- 172.21.15.104:0/4197269623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f472c102c30 0x7f472c1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.427+0000 7f473255a640 1 -- 172.21.15.104:0/4197269623 >> 172.21.15.104:0/4197269623 conn(0x7f472c0779f0 msgr2=0x7f472c0fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.427+0000 7f473255a640 1 -- 172.21.15.104:0/4197269623 shutdown_connections 2024-06-09T22:27:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:34.427+0000 7f473255a640 1 -- 172.21.15.104:0/4197269623 wait complete. 2024-06-09T22:27:34.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 3.85 MB/sec execute 160 sec latency 30037.368 ms 2024-06-09T22:27:34.877 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:27:35.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 3.83 MB/sec execute 161 sec latency 31037.418 ms 2024-06-09T22:27:35.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:35 smithi153 ceph-mon[36599]: pgmap v201: 129 pgs: 129 active+clean; 140 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 600 KiB/s wr, 29 op/s 2024-06-09T22:27:36.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:35 smithi104 ceph-mon[32389]: pgmap v201: 129 pgs: 129 active+clean; 140 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 600 KiB/s wr, 29 op/s 2024-06-09T22:27:36.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:35 smithi194 ceph-mon[36856]: pgmap v201: 129 pgs: 129 active+clean; 140 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 600 KiB/s wr, 29 op/s 2024-06-09T22:27:36.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 185859 3.81 MB/sec execute 162 sec latency 32037.444 ms 2024-06-09T22:27:36.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.947+0000 7f17660a8700 1 -- 172.21.15.104:0/15591237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1760101390 msgr2=0x7f1760101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:36.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.947+0000 7f17660a8700 1 --2- 172.21.15.104:0/15591237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1760101390 0x7f1760101810 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f1748009990 tx=0x7f1748009ca0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.947+0000 7f17660a8700 1 -- 172.21.15.104:0/15591237 shutdown_connections 2024-06-09T22:27:36.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.947+0000 7f17660a8700 1 --2- 172.21.15.104:0/15591237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1760101d50 0x7f176010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.947+0000 7f17660a8700 1 --2- 172.21.15.104:0/15591237 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1760107400 0x7f17601077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.947+0000 7f17660a8700 1 --2- 172.21.15.104:0/15591237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1760101390 0x7f1760101810 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.947+0000 7f17660a8700 1 -- 172.21.15.104:0/15591237 >> 172.21.15.104:0/15591237 conn(0x7f1760078340 msgr2=0x7f1760078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.947+0000 7f17660a8700 1 -- 172.21.15.104:0/15591237 shutdown_connections 2024-06-09T22:27:36.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.947+0000 7f17660a8700 1 -- 172.21.15.104:0/15591237 wait complete. 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.948+0000 7f17660a8700 1 Processor -- start 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.948+0000 7f17660a8700 1 -- start start 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.948+0000 7f17660a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1760101390 0x7f176019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.948+0000 7f17660a8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1760101d50 0x7f176019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.948+0000 7f17660a8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1760107400 0x7f1760198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.948+0000 7f17660a8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1760112790 con 0x7f1760101390 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.948+0000 7f17660a8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1760112610 con 0x7f1760107400 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.948+0000 7f17660a8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1760112910 con 0x7f1760101d50 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1760101390 0x7f176019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1760101390 0x7f176019e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55074/0 (socket says 172.21.15.104:55074) 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175f7fe700 1 -- 172.21.15.104:0/168211568 learned_addr learned my addr 172.21.15.104:0/168211568 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175effd700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1760101d50 0x7f176019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175ffff700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1760107400 0x7f1760198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175f7fe700 1 -- 172.21.15.104:0/168211568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1760101d50 msgr2=0x7f176019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:36.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175f7fe700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1760101d50 0x7f176019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175f7fe700 1 -- 172.21.15.104:0/168211568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1760107400 msgr2=0x7f1760198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:36.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175f7fe700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1760107400 0x7f1760198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175f7fe700 1 -- 172.21.15.104:0/168211568 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17601991d0 con 0x7f1760101390 2024-06-09T22:27:36.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175f7fe700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1760101390 0x7f176019e860 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f17500060b0 tx=0x7f175000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:36.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175cff9700 1 -- 172.21.15.104:0/168211568 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175001a070 con 0x7f1760101390 2024-06-09T22:27:36.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f17660a8700 1 -- 172.21.15.104:0/168211568 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17601994c0 con 0x7f1760101390 2024-06-09T22:27:36.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175cff9700 1 -- 172.21.15.104:0/168211568 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f175000dd20 con 0x7f1760101390 2024-06-09T22:27:36.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.949+0000 7f175cff9700 1 -- 172.21.15.104:0/168211568 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1750014720 con 0x7f1760101390 2024-06-09T22:27:36.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.950+0000 7f17660a8700 1 -- 172.21.15.104:0/168211568 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17601a5730 con 0x7f1760101390 2024-06-09T22:27:36.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.952+0000 7f175cff9700 1 -- 172.21.15.104:0/168211568 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f175001d070 con 0x7f1760101390 2024-06-09T22:27:36.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.952+0000 7f175cff9700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f174c0a2d90 0x7f174c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.952+0000 7f175effd700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f174c0a2d90 0x7f174c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.953+0000 7f175cff9700 1 -- 172.21.15.104:0/168211568 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f17500ceeb0 con 0x7f1760101390 2024-06-09T22:27:36.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.953+0000 7f175effd700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f174c0a2d90 0x7f174c0a5250 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f17480098e0 tx=0x7f1748005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:36.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.953+0000 7f177092bb80 1 Processor -- start 2024-06-09T22:27:36.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.953+0000 7f177092bb80 1 -- start start 2024-06-09T22:27:36.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.953+0000 7f177092bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2256b050 0x558b2256b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.953+0000 7f177092bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 0x558b225652a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.953+0000 7f177092bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b225657e0 0x558b22565c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.953+0000 7f177092bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x558b224b2910 con 0x558b2256b970 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.953+0000 7f177092bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558b2245a820 con 0x558b2256b050 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.953+0000 7f177092bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558b224ae8d0 con 0x558b225657e0 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.953+0000 7f175effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 0x558b225652a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f175effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 0x558b225652a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55076/0 (socket says 172.21.15.104:55076) 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f175effd700 1 -- 172.21.15.104:0/2508522705 learned_addr learned my addr 172.21.15.104:0/2508522705 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f175f7fe700 1 --2- 172.21.15.104:0/2508522705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2256b050 0x558b2256b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f175ffff700 1 --2- 172.21.15.104:0/2508522705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b225657e0 0x558b22565c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f175ffff700 1 -- 172.21.15.104:0/2508522705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2256b050 msgr2=0x558b2256b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f175ffff700 1 --2- 172.21.15.104:0/2508522705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2256b050 0x558b2256b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f175ffff700 1 -- 172.21.15.104:0/2508522705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 msgr2=0x558b225652a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f175ffff700 1 --2- 172.21.15.104:0/2508522705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 0x558b225652a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f175ffff700 1 -- 172.21.15.104:0/2508522705 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b22451050 con 0x558b225657e0 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f175effd700 1 --2- 172.21.15.104:0/2508522705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 0x558b225652a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:36.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f175ffff700 1 --2- 172.21.15.104:0/2508522705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b225657e0 0x558b22565c20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1758004630 tx=0x7f175800b970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e834beb81cedc1e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:36.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f1755ffb700 1 -- 172.21.15.104:0/2508522705 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1758004a30 con 0x558b225657e0 2024-06-09T22:27:36.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f1755ffb700 1 -- 172.21.15.104:0/2508522705 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1758015070 con 0x558b225657e0 2024-06-09T22:27:36.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.954+0000 7f1755ffb700 1 -- 172.21.15.104:0/2508522705 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1758003a40 con 0x558b225657e0 2024-06-09T22:27:36.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.955+0000 7f177092bb80 1 -- 172.21.15.104:0/2508522705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b225657e0 msgr2=0x558b22565c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:36.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.955+0000 7f177092bb80 1 --2- 172.21.15.104:0/2508522705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b225657e0 0x558b22565c20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1758004630 tx=0x7f175800b970 comp rx=0 tx=0).stop 2024-06-09T22:27:36.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.955+0000 7f177092bb80 1 -- 172.21.15.104:0/2508522705 shutdown_connections 2024-06-09T22:27:36.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.955+0000 7f177092bb80 1 --2- 172.21.15.104:0/2508522705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b225657e0 0x558b22565c20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.955+0000 7f177092bb80 1 --2- 172.21.15.104:0/2508522705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2256b050 0x558b2256b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.955+0000 7f177092bb80 1 --2- 172.21.15.104:0/2508522705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 0x558b225652a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.955+0000 7f177092bb80 1 -- 172.21.15.104:0/2508522705 >> 172.21.15.104:0/2508522705 conn(0x558b2248bdc0 msgr2=0x558b2255c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:36.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.955+0000 7f177092bb80 1 -- 172.21.15.104:0/2508522705 shutdown_connections 2024-06-09T22:27:36.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.955+0000 7f177092bb80 1 -- 172.21.15.104:0/2508522705 wait complete. 2024-06-09T22:27:36.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.955+0000 7f177092bb80 1 -- start start 2024-06-09T22:27:36.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f177092bb80 1 Processor -- start 2024-06-09T22:27:36.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f177092bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b225657e0 0x558b22571670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f177092bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 0x558b2256e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f177092bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b2256b970 0x558b2256eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f177092bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x558b225723e0 con 0x558b2256b050 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f177092bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558b22452cf0 con 0x558b225657e0 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f177092bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558b22572560 con 0x558b2256b970 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f177092bb80 10 client.0 _ll_register_callbacks cb 0x558b2244ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f175effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 0x558b2256e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f175f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b225657e0 0x558b22571670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f175effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 0x558b2256e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55090/0 (socket says 172.21.15.104:55090) 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f175effd700 1 -- 172.21.15.104:0/3078251907 learned_addr learned my addr 172.21.15.104:0/3078251907 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f175ffff700 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b2256b970 0x558b2256eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f175effd700 1 -- 172.21.15.104:0/3078251907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b2256b970 msgr2=0x558b2256eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f175effd700 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b2256b970 0x558b2256eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f175effd700 1 -- 172.21.15.104:0/3078251907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b225657e0 msgr2=0x558b22571670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f175effd700 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b225657e0 0x558b22571670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f175effd700 1 -- 172.21.15.104:0/3078251907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b224019d0 con 0x558b2256b050 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f175f7fe700 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b225657e0 0x558b22571670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.956+0000 7f175effd700 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 0x558b2256e970 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f17480059d0 tx=0x7f174801a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f1222d0433a384 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f173effd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f173effd700 1 -- 172.21.15.104:0/3078251907 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1748024610 con 0x558b2256b050 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f173effd700 1 -- 172.21.15.104:0/3078251907 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f17480247b0 con 0x558b2256b050 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f177092bb80 1 -- 172.21.15.104:0/3078251907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1758015070 con 0x558b2256b050 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f173effd700 1 -- 172.21.15.104:0/3078251907 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f174803e050 con 0x558b2256b050 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f177092bb80 10 client.17325 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f177092bb80 1 -- 172.21.15.104:0/3078251907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558b2257cc70 con 0x558b2256b050 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f173effd700 1 -- 172.21.15.104:0/3078251907 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1758015070 con 0x558b2256b050 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f173effd700 1 -- 172.21.15.104:0/3078251907 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1748030b30 con 0x558b2256b050 2024-06-09T22:27:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f177092bb80 10 client.17325 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f177092bb80 10 client.17325 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f177092bb80 20 client.17325 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f177092bb80 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558b2257f2b0 0x558b225816a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f177092bb80 4 client.17325 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.957+0000 7f177092bb80 1 -- 172.21.15.104:0/3078251907 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b224bfe10 con 0x558b2257f2b0 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.958+0000 7f175f7fe700 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558b2257f2b0 0x558b225816a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.958+0000 7f175f7fe700 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558b2257f2b0 0x558b225816a0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9a0a2ebfbd13b87 server_cookie=323c125742c0ca9f in_seq=0 out_seq=0 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.958+0000 7f173effd700 10 client.17325 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f173effd700 1 -- 172.21.15.104:0/3078251907 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x558b224bfe10 con 0x558b2257f2b0 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f173effd700 10 client.17325 handle_command_reply: tid=0 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f177092bb80 1 client.17325 shutdown 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f177092bb80 1 -- 172.21.15.104:0/3078251907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 msgr2=0x558b2256e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f177092bb80 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 0x558b2256e970 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f17480059d0 tx=0x7f174801a050 comp rx=0 tx=0).stop 2024-06-09T22:27:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f177092bb80 1 -- 172.21.15.104:0/3078251907 shutdown_connections 2024-06-09T22:27:36.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f177092bb80 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558b2257f2b0 0x558b225816a0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f173effd700 0 client.17325 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:36.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f177092bb80 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b2256b970 0x558b2256eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f177092bb80 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b225657e0 0x558b22571670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f177092bb80 1 --2- 172.21.15.104:0/3078251907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 0x558b2256e970 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f177092bb80 1 -- 172.21.15.104:0/3078251907 >> 172.21.15.104:0/3078251907 conn(0x558b2248bdc0 msgr2=0x558b2255bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:36.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f177092bb80 1 -- 172.21.15.104:0/3078251907 shutdown_connections 2024-06-09T22:27:36.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f177092bb80 1 -- 172.21.15.104:0/3078251907 wait complete. 2024-06-09T22:27:36.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.959+0000 7f177092bb80 20 client.17325 trim_cache size 0 max 16384 2024-06-09T22:27:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.979+0000 7f177092bb80 1 Processor -- start 2024-06-09T22:27:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.979+0000 7f177092bb80 1 -- start start 2024-06-09T22:27:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.979+0000 7f177092bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b225657e0 0x558b225652a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.979+0000 7f177092bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 0x558b225816a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.979+0000 7f177092bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b2256b970 0x558b2255bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.979+0000 7f177092bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f17601ac2e0 con 0x558b2256b050 2024-06-09T22:27:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.979+0000 7f177092bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17601ade20 con 0x558b225657e0 2024-06-09T22:27:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.979+0000 7f177092bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f17601ac9b0 con 0x558b2256b970 2024-06-09T22:27:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.979+0000 7f175effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 0x558b225816a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.979+0000 7f175effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 0x558b225816a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55092/0 (socket says 172.21.15.104:55092) 2024-06-09T22:27:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.979+0000 7f175effd700 1 -- 172.21.15.104:0/902077786 learned_addr learned my addr 172.21.15.104:0/902077786 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.980+0000 7f175f7fe700 1 --2- 172.21.15.104:0/902077786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b225657e0 0x558b225652a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.980+0000 7f175effd700 1 -- 172.21.15.104:0/902077786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b2256b970 msgr2=0x558b2255bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.980+0000 7f175effd700 1 --2- 172.21.15.104:0/902077786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b2256b970 0x558b2255bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.980+0000 7f175effd700 1 -- 172.21.15.104:0/902077786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b225657e0 msgr2=0x558b225652a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.980+0000 7f175effd700 1 --2- 172.21.15.104:0/902077786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b225657e0 0x558b225652a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.980+0000 7f175effd700 1 -- 172.21.15.104:0/902077786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b2256cee0 con 0x558b2256b050 2024-06-09T22:27:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.980+0000 7f175f7fe700 1 --2- 172.21.15.104:0/902077786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b225657e0 0x558b225652a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:27:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.980+0000 7f175effd700 1 --2- 172.21.15.104:0/902077786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 0x558b225816a0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f1748000c00 tx=0x7f174800bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69dc4985d34e45eb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.980+0000 7f17557fa700 1 -- 172.21.15.104:0/902077786 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1748048070 con 0x558b2256b050 2024-06-09T22:27:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.980+0000 7f17557fa700 1 -- 172.21.15.104:0/902077786 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1748035070 con 0x558b2256b050 2024-06-09T22:27:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.980+0000 7f17557fa700 1 -- 172.21.15.104:0/902077786 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1748030b40 con 0x558b2256b050 2024-06-09T22:27:36.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.981+0000 7f177092bb80 1 -- 172.21.15.104:0/902077786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 msgr2=0x558b225816a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:36.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.981+0000 7f177092bb80 1 --2- 172.21.15.104:0/902077786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 0x558b225816a0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f1748000c00 tx=0x7f174800bf70 comp rx=0 tx=0).stop 2024-06-09T22:27:36.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.981+0000 7f177092bb80 1 -- 172.21.15.104:0/902077786 shutdown_connections 2024-06-09T22:27:36.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.981+0000 7f177092bb80 1 --2- 172.21.15.104:0/902077786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b2256b970 0x558b2255bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.981+0000 7f177092bb80 1 --2- 172.21.15.104:0/902077786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b225657e0 0x558b225652a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.981+0000 7f177092bb80 1 --2- 172.21.15.104:0/902077786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b050 0x558b225816a0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.981+0000 7f177092bb80 1 -- 172.21.15.104:0/902077786 >> 172.21.15.104:0/902077786 conn(0x558b2248bdc0 msgr2=0x558b2255b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:36.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.981+0000 7f177092bb80 1 -- 172.21.15.104:0/902077786 shutdown_connections 2024-06-09T22:27:36.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.981+0000 7f177092bb80 1 -- 172.21.15.104:0/902077786 wait complete. 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.981+0000 7f177092bb80 1 -- start start 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.981+0000 7f177092bb80 1 Processor -- start 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f177092bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b225657e0 0x558b22586bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f177092bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2256b050 0x558b22587130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f177092bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 0x558b22587670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f177092bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f17601ad750 con 0x558b2256b970 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f177092bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17601ad080 con 0x558b2256b050 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f177092bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f17601aa260 con 0x558b225657e0 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f177092bb80 10 client.0 _ll_register_callbacks cb 0x558b2244ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f175ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 0x558b22587670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f175ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 0x558b22587670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55106/0 (socket says 172.21.15.104:55106) 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f175ffff700 1 -- 172.21.15.104:0/3391862415 learned_addr learned my addr 172.21.15.104:0/3391862415 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f175effd700 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2256b050 0x558b22587130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f175f7fe700 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b225657e0 0x558b22586bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f175ffff700 1 -- 172.21.15.104:0/3391862415 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b225657e0 msgr2=0x558b22586bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f175ffff700 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b225657e0 0x558b22586bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f175ffff700 1 -- 172.21.15.104:0/3391862415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2256b050 msgr2=0x558b22587130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f175ffff700 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2256b050 0x558b22587130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f175ffff700 1 -- 172.21.15.104:0/3391862415 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b225e8ff0 con 0x558b2256b970 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f175ffff700 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 0x558b22587670 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f175800bcc0 tx=0x7f1758004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f1c0d0ef8e6d90 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f173effd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f173effd700 1 -- 172.21.15.104:0/3391862415 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1758005190 con 0x558b2256b970 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f173effd700 1 -- 172.21.15.104:0/3391862415 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f1758005330 con 0x558b2256b970 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.982+0000 7f177092bb80 1 -- 172.21.15.104:0/3391862415 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1748035070 con 0x558b2256b970 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.983+0000 7f173effd700 1 -- 172.21.15.104:0/3391862415 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1758031050 con 0x558b2256b970 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.983+0000 7f177092bb80 10 client.17331 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.983+0000 7f177092bb80 1 -- 172.21.15.104:0/3391862415 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558b225ea260 con 0x558b2256b970 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.983+0000 7f173effd700 1 -- 172.21.15.104:0/3391862415 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1748035070 con 0x558b2256b970 2024-06-09T22:27:36.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.983+0000 7f173effd700 1 -- 172.21.15.104:0/3391862415 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f175802c030 con 0x558b2256b970 2024-06-09T22:27:36.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.983+0000 7f177092bb80 10 client.17331 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:36.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.983+0000 7f177092bb80 10 client.17331 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:27:36.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.983+0000 7f177092bb80 20 client.17331 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:36.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.983+0000 7f177092bb80 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558b2257f2b0 0x558b225ec6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:36.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.983+0000 7f177092bb80 4 client.17331 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6813a49", "release": true, "await": true}] 2024-06-09T22:27:36.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.983+0000 7f177092bb80 1 -- 172.21.15.104:0/3391862415 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6813a49", "release": true, "await": true}) -- 0x558b224ba390 con 0x558b2257f2b0 2024-06-09T22:27:36.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.983+0000 7f175f7fe700 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558b2257f2b0 0x558b225ec6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:36.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.984+0000 7f175f7fe700 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558b2257f2b0 0x558b225ec6d0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4944d86ae8a6a1af server_cookie=2477a67e44c94320 in_seq=0 out_seq=0 2024-06-09T22:27:36.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.984+0000 7f173effd700 10 client.17331 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:36.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f173effd700 1 -- 172.21.15.104:0/3391862415 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x558b224ba390 con 0x558b2257f2b0 2024-06-09T22:27:36.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f173effd700 10 client.17331 handle_command_reply: tid=0 2024-06-09T22:27:36.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f177092bb80 1 client.17331 shutdown 2024-06-09T22:27:36.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f177092bb80 1 -- 172.21.15.104:0/3391862415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 msgr2=0x558b22587670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:36.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f177092bb80 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 0x558b22587670 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f175800bcc0 tx=0x7f1758004940 comp rx=0 tx=0).stop 2024-06-09T22:27:36.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f177092bb80 1 -- 172.21.15.104:0/3391862415 shutdown_connections 2024-06-09T22:27:36.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f177092bb80 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x558b2257f2b0 0x558b225ec6d0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f177092bb80 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558b2256b970 0x558b22587670 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f177092bb80 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b225657e0 0x558b22586bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f173effd700 0 client.17331 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:36.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f177092bb80 1 --2- 172.21.15.104:0/3391862415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2256b050 0x558b22587130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:36.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f177092bb80 1 -- 172.21.15.104:0/3391862415 >> 172.21.15.104:0/3391862415 conn(0x558b2248bdc0 msgr2=0x558b2256bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:36.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f177092bb80 1 -- 172.21.15.104:0/3391862415 shutdown_connections 2024-06-09T22:27:36.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f177092bb80 1 -- 172.21.15.104:0/3391862415 wait complete. 2024-06-09T22:27:36.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f177092bb80 20 client.17331 trim_cache size 0 max 16384 2024-06-09T22:27:36.998 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 32, 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "leader": 24460, 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "set_version": 20, 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "sets": { 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "6813a49": { 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "version": 20, 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "age_ref": 0.0, 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RELEASED", 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "timeout": 90.0, 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "expiration": 149.4, 2024-06-09T22:27:36.999 INFO:teuthology.orchestra.run.smithi104.stdout: "members": { 2024-06-09T22:27:37.000 INFO:teuthology.orchestra.run.smithi104.stdout: "file:/": { 2024-06-09T22:27:37.000 INFO:teuthology.orchestra.run.smithi104.stdout: "excluded": false, 2024-06-09T22:27:37.000 INFO:teuthology.orchestra.run.smithi104.stdout: "state": { 2024-06-09T22:27:37.000 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RELEASED", 2024-06-09T22:27:37.000 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0 2024-06-09T22:27:37.000 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:27:37.000 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:27:37.000 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:27:37.000 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:27:37.000 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:27:37.000 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:27:37.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f17660a8700 1 -- 172.21.15.104:0/168211568 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f174c0a2d90 msgr2=0x7f174c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:37.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f17660a8700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f174c0a2d90 0x7f174c0a5250 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f17480098e0 tx=0x7f1748005960 comp rx=0 tx=0).stop 2024-06-09T22:27:37.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f17660a8700 1 -- 172.21.15.104:0/168211568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1760101390 msgr2=0x7f176019e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:37.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f17660a8700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1760101390 0x7f176019e860 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f17500060b0 tx=0x7f175000d3b0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f17660a8700 1 -- 172.21.15.104:0/168211568 shutdown_connections 2024-06-09T22:27:37.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f17660a8700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f174c0a2d90 0x7f174c0a5250 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f17660a8700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1760101390 0x7f176019e860 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f17660a8700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1760107400 0x7f1760198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f17660a8700 1 --2- 172.21.15.104:0/168211568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1760101d50 0x7f176019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f17660a8700 1 -- 172.21.15.104:0/168211568 >> 172.21.15.104:0/168211568 conn(0x7f1760078340 msgr2=0x7f17600fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:37.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f17660a8700 1 -- 172.21.15.104:0/168211568 shutdown_connections 2024-06-09T22:27:37.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:36.996+0000 7f17660a8700 1 -- 172.21.15.104:0/168211568 wait complete. 2024-06-09T22:27:37.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.145+0000 7f7aa348a700 1 -- 172.21.15.104:0/429156299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c108530 msgr2=0x7f7a9c108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:37.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.145+0000 7f7aa348a700 1 --2- 172.21.15.104:0/429156299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c108530 0x7f7a9c108910 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009990 tx=0x7f7a98009ca0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.145+0000 7f7aa348a700 1 -- 172.21.15.104:0/429156299 shutdown_connections 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.145+0000 7f7aa348a700 1 --2- 172.21.15.104:0/429156299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c108530 0x7f7a9c108910 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.145+0000 7f7aa348a700 1 --2- 172.21.15.104:0/429156299 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7a9c06b520 0x7f7a9c10f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.145+0000 7f7aa348a700 1 --2- 172.21.15.104:0/429156299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a9c06ab60 0x7f7a9c06afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.145+0000 7f7aa348a700 1 -- 172.21.15.104:0/429156299 >> 172.21.15.104:0/429156299 conn(0x7f7a9c0783d0 msgr2=0x7f7a9c0787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.145+0000 7f7aa348a700 1 -- 172.21.15.104:0/429156299 shutdown_connections 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.145+0000 7f7aa348a700 1 -- 172.21.15.104:0/429156299 wait complete. 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.145+0000 7f7aa348a700 1 Processor -- start 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.145+0000 7f7aa348a700 1 -- start start 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa348a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a9c06ab60 0x7f7a9c19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa348a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c06b520 0x7f7a9c19b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa348a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7a9c108530 0x7f7a9c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa348a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1127b0 con 0x7f7a9c06b520 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa348a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c112630 con 0x7f7a9c06ab60 2024-06-09T22:27:37.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa348a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c112930 con 0x7f7a9c108530 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa0a25700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c06b520 0x7f7a9c19b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa1226700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a9c06ab60 0x7f7a9c19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa0a25700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c06b520 0x7f7a9c19b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55114/0 (socket says 172.21.15.104:55114) 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa0a25700 1 -- 172.21.15.104:0/3913397455 learned_addr learned my addr 172.21.15.104:0/3913397455 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa1a27700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7a9c108530 0x7f7a9c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa1226700 1 -- 172.21.15.104:0/3913397455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7a9c108530 msgr2=0x7f7a9c19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa1226700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7a9c108530 0x7f7a9c19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa1226700 1 -- 172.21.15.104:0/3913397455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c06b520 msgr2=0x7f7a9c19b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa1226700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c06b520 0x7f7a9c19b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa1226700 1 -- 172.21.15.104:0/3913397455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a9c19fe20 con 0x7f7a9c06ab60 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa0a25700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c06b520 0x7f7a9c19b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.146+0000 7f7aa1226700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a9c06ab60 0x7f7a9c19ae10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7a980059d0 tx=0x7f7a9800bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.147+0000 7f7a927fc700 1 -- 172.21.15.104:0/3913397455 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a98027070 con 0x7f7a9c06ab60 2024-06-09T22:27:37.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.147+0000 7f7aa348a700 1 -- 172.21.15.104:0/3913397455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a9c1a00b0 con 0x7f7a9c06ab60 2024-06-09T22:27:37.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.147+0000 7f7a927fc700 1 -- 172.21.15.104:0/3913397455 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a98003d70 con 0x7f7a9c06ab60 2024-06-09T22:27:37.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.147+0000 7f7a927fc700 1 -- 172.21.15.104:0/3913397455 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a98022e30 con 0x7f7a9c06ab60 2024-06-09T22:27:37.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.147+0000 7f7aa348a700 1 -- 172.21.15.104:0/3913397455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a9c1a7990 con 0x7f7a9c06ab60 2024-06-09T22:27:37.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.149+0000 7f7a927fc700 1 -- 172.21.15.104:0/3913397455 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7a9802a070 con 0x7f7a9c06ab60 2024-06-09T22:27:37.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.149+0000 7f7a927fc700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7a880a2d90 0x7f7a880a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.150+0000 7f7aa0a25700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7a880a2d90 0x7f7a880a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.150+0000 7f7a927fc700 1 -- 172.21.15.104:0/3913397455 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7a980d0470 con 0x7f7a9c06ab60 2024-06-09T22:27:37.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.150+0000 7f7aa0a25700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7a880a2d90 0x7f7a880a5250 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c103460 tx=0x7f7a8c009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:37.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.150+0000 7f7aadd0db80 1 Processor -- start 2024-06-09T22:27:37.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aadd0db80 1 -- start start 2024-06-09T22:27:37.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aadd0db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 0x55cb42e99a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aadd0db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99f60 0x55cb42e939e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aadd0db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e93f20 0x55cb42e94340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aadd0db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55cb42de5810 con 0x55cb42e99640 2024-06-09T22:27:37.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aadd0db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cb42d8d720 con 0x55cb42e99f60 2024-06-09T22:27:37.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aadd0db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cb42de17d0 con 0x55cb42e93f20 2024-06-09T22:27:37.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aa1226700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 0x55cb42e99a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aa1226700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 0x55cb42e99a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55122/0 (socket says 172.21.15.104:55122) 2024-06-09T22:27:37.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aa0a25700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99f60 0x55cb42e939e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aa1226700 1 -- 172.21.15.104:0/1196251377 learned_addr learned my addr 172.21.15.104:0/1196251377 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:37.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aa1a27700 1 --2- 172.21.15.104:0/1196251377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e93f20 0x55cb42e94340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aa1226700 1 -- 172.21.15.104:0/1196251377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e93f20 msgr2=0x55cb42e94340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:37.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aa1226700 1 --2- 172.21.15.104:0/1196251377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e93f20 0x55cb42e94340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aa1226700 1 -- 172.21.15.104:0/1196251377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99f60 msgr2=0x55cb42e939e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:37.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aa1226700 1 --2- 172.21.15.104:0/1196251377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99f60 0x55cb42e939e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aa1226700 1 -- 172.21.15.104:0/1196251377 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb42d83f50 con 0x55cb42e99640 2024-06-09T22:27:37.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aa0a25700 1 --2- 172.21.15.104:0/1196251377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99f60 0x55cb42e939e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:37.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.151+0000 7f7aa1226700 1 --2- 172.21.15.104:0/1196251377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 0x55cb42e99a20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7a980cd950 tx=0x7f7a98022850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca0d8327522b1e4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:37.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.152+0000 7f7a877fe700 1 -- 172.21.15.104:0/1196251377 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a980095d0 con 0x55cb42e99640 2024-06-09T22:27:37.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.152+0000 7f7a877fe700 1 -- 172.21.15.104:0/1196251377 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a9809ce50 con 0x55cb42e99640 2024-06-09T22:27:37.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.152+0000 7f7a877fe700 1 -- 172.21.15.104:0/1196251377 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9809cc10 con 0x55cb42e99640 2024-06-09T22:27:37.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.153+0000 7f7aadd0db80 1 -- 172.21.15.104:0/1196251377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 msgr2=0x55cb42e99a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:37.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.153+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/1196251377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 0x55cb42e99a20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7a980cd950 tx=0x7f7a98022850 comp rx=0 tx=0).stop 2024-06-09T22:27:37.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.153+0000 7f7aadd0db80 1 -- 172.21.15.104:0/1196251377 shutdown_connections 2024-06-09T22:27:37.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.153+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/1196251377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 0x55cb42e99a20 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.153+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/1196251377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e93f20 0x55cb42e94340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.153+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/1196251377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99f60 0x55cb42e939e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.153+0000 7f7aadd0db80 1 -- 172.21.15.104:0/1196251377 >> 172.21.15.104:0/1196251377 conn(0x55cb42dbecc0 msgr2=0x55cb42e8e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:37.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.153+0000 7f7aadd0db80 1 -- 172.21.15.104:0/1196251377 shutdown_connections 2024-06-09T22:27:37.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.153+0000 7f7aadd0db80 1 -- 172.21.15.104:0/1196251377 wait complete. 2024-06-09T22:27:37.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.153+0000 7f7aadd0db80 1 -- start start 2024-06-09T22:27:37.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.153+0000 7f7aadd0db80 1 Processor -- start 2024-06-09T22:27:37.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aadd0db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e93f20 0x55cb42e9fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aadd0db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99640 0x55cb42e9d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aadd0db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e99f60 0x55cb42e9d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aadd0db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55cb42ea06f0 con 0x55cb42e93f20 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aadd0db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cb42d85bf0 con 0x55cb42e99640 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aadd0db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cb42ea0870 con 0x55cb42e99f60 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aadd0db80 10 client.0 _ll_register_callbacks cb 0x55cb42d7f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aa1226700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e93f20 0x55cb42e9fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aa1226700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e93f20 0x55cb42e9fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55124/0 (socket says 172.21.15.104:55124) 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aa1226700 1 -- 172.21.15.104:0/3410813787 learned_addr learned my addr 172.21.15.104:0/3410813787 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aa1a27700 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e99f60 0x55cb42e9d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aa0a25700 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99640 0x55cb42e9d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aa1226700 1 -- 172.21.15.104:0/3410813787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e99f60 msgr2=0x55cb42e9d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aa1226700 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e99f60 0x55cb42e9d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aa1226700 1 -- 172.21.15.104:0/3410813787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99640 msgr2=0x55cb42e9d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aa1226700 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99640 0x55cb42e9d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.154+0000 7f7aa1226700 1 -- 172.21.15.104:0/3410813787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb42d348d0 con 0x55cb42e93f20 2024-06-09T22:27:37.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.155+0000 7f7aa1226700 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e93f20 0x55cb42e9fd90 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f7a98099120 tx=0x7f7a98005ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd59d5c8f499855 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.155+0000 7f7a84ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.155+0000 7f7a84ff9700 1 -- 172.21.15.104:0/3410813787 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a980095d0 con 0x55cb42e93f20 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.155+0000 7f7a84ff9700 1 -- 172.21.15.104:0/3410813787 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7a980d0170 con 0x55cb42e93f20 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.155+0000 7f7aadd0db80 1 -- 172.21.15.104:0/3410813787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a9809ce50 con 0x55cb42e93f20 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.155+0000 7f7a84ff9700 1 -- 172.21.15.104:0/3410813787 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7a9809c750 con 0x55cb42e93f20 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.155+0000 7f7aadd0db80 10 client.17343 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.155+0000 7f7aadd0db80 1 -- 172.21.15.104:0/3410813787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cb42ea06f0 con 0x55cb42e93f20 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.155+0000 7f7a84ff9700 1 -- 172.21.15.104:0/3410813787 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a9809ce50 con 0x55cb42e93f20 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.156+0000 7f7a84ff9700 1 -- 172.21.15.104:0/3410813787 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7a980a4ea0 con 0x55cb42e93f20 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.156+0000 7f7aadd0db80 10 client.17343 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.156+0000 7f7aadd0db80 10 client.17343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.156+0000 7f7aadd0db80 20 client.17343 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.156+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cb42ead570 0x55cb42eaf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.156+0000 7f7aadd0db80 4 client.17343 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.156+0000 7f7aadd0db80 1 -- 172.21.15.104:0/3410813787 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb42df2d10 con 0x55cb42ead570 2024-06-09T22:27:37.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.156+0000 7f7aa0a25700 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cb42ead570 0x55cb42eaf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.156+0000 7f7aa0a25700 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cb42ead570 0x55cb42eaf9b0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7957e7d70f48876 server_cookie=4da086d295660692 in_seq=0 out_seq=0 2024-06-09T22:27:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.156+0000 7f7a84ff9700 10 client.17343 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7a84ff9700 1 -- 172.21.15.104:0/3410813787 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55cb42df2d10 con 0x55cb42ead570 2024-06-09T22:27:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7a84ff9700 10 client.17343 handle_command_reply: tid=0 2024-06-09T22:27:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7aadd0db80 1 client.17343 shutdown 2024-06-09T22:27:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7aadd0db80 1 -- 172.21.15.104:0/3410813787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e93f20 msgr2=0x55cb42e9fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e93f20 0x55cb42e9fd90 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f7a98099120 tx=0x7f7a98005ad0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7aadd0db80 1 -- 172.21.15.104:0/3410813787 shutdown_connections 2024-06-09T22:27:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cb42ead570 0x55cb42eaf9b0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e93f20 0x55cb42e9fd90 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e99f60 0x55cb42e9d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/3410813787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99640 0x55cb42e9d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7aadd0db80 1 -- 172.21.15.104:0/3410813787 >> 172.21.15.104:0/3410813787 conn(0x55cb42dbecc0 msgr2=0x55cb42e8e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:37.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7a84ff9700 0 client.17343 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:37.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7aadd0db80 1 -- 172.21.15.104:0/3410813787 shutdown_connections 2024-06-09T22:27:37.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7aadd0db80 1 -- 172.21.15.104:0/3410813787 wait complete. 2024-06-09T22:27:37.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.157+0000 7f7aadd0db80 20 client.17343 trim_cache size 0 max 16384 2024-06-09T22:27:37.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.173+0000 7f7aadd0db80 1 Processor -- start 2024-06-09T22:27:37.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.173+0000 7f7aadd0db80 1 -- start start 2024-06-09T22:27:37.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.173+0000 7f7aadd0db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e93f20 0x55cb42e939e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.173+0000 7f7aadd0db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 0x55cb42e97800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.173+0000 7f7aadd0db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99f60 0x55cb42ea9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.173+0000 7f7aadd0db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1b06d0 con 0x55cb42e99640 2024-06-09T22:27:37.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.173+0000 7f7aadd0db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1b2210 con 0x55cb42e99f60 2024-06-09T22:27:37.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.173+0000 7f7aadd0db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1b0da0 con 0x55cb42e93f20 2024-06-09T22:27:37.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.173+0000 7f7aa0a25700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 0x55cb42e97800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.173+0000 7f7aa0a25700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 0x55cb42e97800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55132/0 (socket says 172.21.15.104:55132) 2024-06-09T22:27:37.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.173+0000 7f7aa0a25700 1 -- 172.21.15.104:0/4213761907 learned_addr learned my addr 172.21.15.104:0/4213761907 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:37.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.173+0000 7f7aa1226700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e93f20 0x55cb42e939e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.174+0000 7f7aa1a27700 1 --2- 172.21.15.104:0/4213761907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99f60 0x55cb42ea9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.174+0000 7f7aa0a25700 1 -- 172.21.15.104:0/4213761907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e93f20 msgr2=0x55cb42e939e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:37.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.174+0000 7f7aa0a25700 1 --2- 172.21.15.104:0/4213761907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e93f20 0x55cb42e939e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.174+0000 7f7aa0a25700 1 -- 172.21.15.104:0/4213761907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99f60 msgr2=0x55cb42ea9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:37.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.174+0000 7f7aa0a25700 1 --2- 172.21.15.104:0/4213761907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99f60 0x55cb42ea9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.174+0000 7f7aa0a25700 1 -- 172.21.15.104:0/4213761907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb42e9b600 con 0x55cb42e99640 2024-06-09T22:27:37.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.174+0000 7f7aa0a25700 1 --2- 172.21.15.104:0/4213761907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 0x55cb42e97800 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c013010 tx=0x7f7a8c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22bf57cbf575535 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:37.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.174+0000 7f7a86ffd700 1 -- 172.21.15.104:0/4213761907 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c004830 con 0x55cb42e99640 2024-06-09T22:27:37.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.174+0000 7f7a86ffd700 1 -- 172.21.15.104:0/4213761907 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a8c005020 con 0x55cb42e99640 2024-06-09T22:27:37.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.174+0000 7f7a86ffd700 1 -- 172.21.15.104:0/4213761907 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c01d730 con 0x55cb42e99640 2024-06-09T22:27:37.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.175+0000 7f7aadd0db80 1 -- 172.21.15.104:0/4213761907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 msgr2=0x55cb42e97800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:37.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.175+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/4213761907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 0x55cb42e97800 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c013010 tx=0x7f7a8c014000 comp rx=0 tx=0).stop 2024-06-09T22:27:37.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.175+0000 7f7aadd0db80 1 -- 172.21.15.104:0/4213761907 shutdown_connections 2024-06-09T22:27:37.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.175+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/4213761907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e99f60 0x55cb42ea9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.175+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/4213761907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e93f20 0x55cb42e939e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.175+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/4213761907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99640 0x55cb42e97800 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.175+0000 7f7aadd0db80 1 -- 172.21.15.104:0/4213761907 >> 172.21.15.104:0/4213761907 conn(0x55cb42dbecc0 msgr2=0x55cb42e9a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.175+0000 7f7aadd0db80 1 -- 172.21.15.104:0/4213761907 shutdown_connections 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.175+0000 7f7aadd0db80 1 -- 172.21.15.104:0/4213761907 wait complete. 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.175+0000 7f7aadd0db80 1 -- start start 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.175+0000 7f7aadd0db80 1 Processor -- start 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aadd0db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e93f20 0x55cb42eb58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aadd0db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e99640 0x55cb42ea5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aadd0db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99f60 0x55cb42ea6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aadd0db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1b1b40 con 0x55cb42e99f60 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aadd0db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1b1470 con 0x55cb42e93f20 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aadd0db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1ae650 con 0x55cb42e99640 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aadd0db80 10 client.0 _ll_register_callbacks cb 0x55cb42d7f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aa0a25700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e99640 0x55cb42ea5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aa1226700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e93f20 0x55cb42eb58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aa1226700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e93f20 0x55cb42eb58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:47798/0 (socket says 172.21.15.104:47798) 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aa1226700 1 -- 172.21.15.104:0/2319785056 learned_addr learned my addr 172.21.15.104:0/2319785056 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aa1a27700 1 --2- 172.21.15.104:0/2319785056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99f60 0x55cb42ea6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aa1226700 1 -- 172.21.15.104:0/2319785056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e99640 msgr2=0x55cb42ea5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aa1226700 1 --2- 172.21.15.104:0/2319785056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e99640 0x55cb42ea5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aa1226700 1 -- 172.21.15.104:0/2319785056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99f60 msgr2=0x55cb42ea6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aa1226700 1 --2- 172.21.15.104:0/2319785056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99f60 0x55cb42ea6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.176+0000 7f7aa1226700 1 -- 172.21.15.104:0/2319785056 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb42ea6710 con 0x55cb42e93f20 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7aa1226700 1 --2- 172.21.15.104:0/2319785056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e93f20 0x55cb42eb58c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7a9809c450 tx=0x7f7a98022850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68628fbb9c0c3204 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7a84ff9700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7a84ff9700 1 -- 172.21.15.104:0/2319785056 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9809c290 con 0x55cb42e93f20 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7a84ff9700 1 -- 172.21.15.104:0/2319785056 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f7a98003890 con 0x55cb42e93f20 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7aadd0db80 1 -- 172.21.15.104:0/2319785056 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a8c005020 con 0x55cb42e93f20 2024-06-09T22:27:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7a84ff9700 1 -- 172.21.15.104:0/2319785056 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7a980095d0 con 0x55cb42e93f20 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7aadd0db80 10 client.26588 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7aadd0db80 1 -- 172.21.15.104:0/2319785056 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cb42ea6710 con 0x55cb42e93f20 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7a84ff9700 1 -- 172.21.15.104:0/2319785056 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a8c005020 con 0x55cb42e93f20 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7a84ff9700 1 -- 172.21.15.104:0/2319785056 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7a98082050 con 0x55cb42e93f20 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7aadd0db80 10 client.26588 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7aadd0db80 10 client.26588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.177+0000 7f7aadd0db80 20 client.26588 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.178+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/2319785056 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cb42f175d0 0x55cb42f199c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.178+0000 7f7aadd0db80 4 client.26588 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.178+0000 7f7aadd0db80 1 -- 172.21.15.104:0/2319785056 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cb42ded290 con 0x55cb42f175d0 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.178+0000 7f7aa0a25700 1 --2- 172.21.15.104:0/2319785056 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cb42f175d0 0x55cb42f199c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.178+0000 7f7a84ff9700 10 client.26588 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.178+0000 7f7aa0a25700 1 --2- 172.21.15.104:0/2319785056 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cb42f175d0 0x55cb42f199c0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8879608b0f7f60b6 server_cookie=95ee4dc348afea59 in_seq=0 out_seq=0 2024-06-09T22:27:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.179+0000 7f7a84ff9700 1 -- 172.21.15.104:0/2319785056 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cb42ded290 con 0x55cb42f175d0 2024-06-09T22:27:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.179+0000 7f7a84ff9700 10 client.26588 handle_command_reply: tid=0 2024-06-09T22:27:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.179+0000 7f7aadd0db80 1 client.26588 shutdown 2024-06-09T22:27:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.179+0000 7f7aadd0db80 1 -- 172.21.15.104:0/2319785056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e93f20 msgr2=0x55cb42eb58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.179+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/2319785056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e93f20 0x55cb42eb58c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7a9809c450 tx=0x7f7a98022850 comp rx=0 tx=0).stop 2024-06-09T22:27:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.179+0000 7f7aadd0db80 1 -- 172.21.15.104:0/2319785056 shutdown_connections 2024-06-09T22:27:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.179+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/2319785056 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cb42f175d0 0x55cb42f199c0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.179+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/2319785056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cb42e99f60 0x55cb42ea6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.179+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/2319785056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb42e93f20 0x55cb42eb58c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.179+0000 7f7a84ff9700 0 client.26588 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.179+0000 7f7aadd0db80 1 --2- 172.21.15.104:0/2319785056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cb42e99640 0x55cb42ea5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.179+0000 7f7aadd0db80 1 -- 172.21.15.104:0/2319785056 >> 172.21.15.104:0/2319785056 conn(0x55cb42dbecc0 msgr2=0x55cb42e9a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:37.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.180+0000 7f7aadd0db80 1 -- 172.21.15.104:0/2319785056 shutdown_connections 2024-06-09T22:27:37.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.180+0000 7f7aadd0db80 1 -- 172.21.15.104:0/2319785056 wait complete. 2024-06-09T22:27:37.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.180+0000 7f7aadd0db80 20 client.26588 trim_cache size 0 max 16384 2024-06-09T22:27:37.186 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:27:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.181+0000 7f7aa348a700 1 -- 172.21.15.104:0/3913397455 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7a880a2d90 msgr2=0x7f7a880a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.181+0000 7f7aa348a700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7a880a2d90 0x7f7a880a5250 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c103460 tx=0x7f7a8c009260 comp rx=0 tx=0).stop 2024-06-09T22:27:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.181+0000 7f7aa348a700 1 -- 172.21.15.104:0/3913397455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a9c06ab60 msgr2=0x7f7a9c19ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.181+0000 7f7aa348a700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a9c06ab60 0x7f7a9c19ae10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7a980059d0 tx=0x7f7a9800bab0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.181+0000 7f7aa348a700 1 -- 172.21.15.104:0/3913397455 shutdown_connections 2024-06-09T22:27:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.181+0000 7f7aa348a700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7a880a2d90 0x7f7a880a5250 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.181+0000 7f7aa348a700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7a9c108530 0x7f7a9c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.181+0000 7f7aa348a700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a9c06ab60 0x7f7a9c19ae10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.181+0000 7f7aa348a700 1 --2- 172.21.15.104:0/3913397455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c06b520 0x7f7a9c19b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.181+0000 7f7aa348a700 1 -- 172.21.15.104:0/3913397455 >> 172.21.15.104:0/3913397455 conn(0x7f7a9c0783d0 msgr2=0x7f7a9c10d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.181+0000 7f7aa348a700 1 -- 172.21.15.104:0/3913397455 shutdown_connections 2024-06-09T22:27:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:37.181+0000 7f7aa348a700 1 -- 172.21.15.104:0/3913397455 wait complete. 2024-06-09T22:27:37.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 187229 3.82 MB/sec execute 163 sec latency 32300.821 ms 2024-06-09T22:27:37.932 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:37 smithi104 ceph-mon[32389]: pgmap v202: 129 pgs: 129 active+clean; 140 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 577 KiB/s wr, 19 op/s 2024-06-09T22:27:37.960 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '6813a49', total seconds elapsed: 39.0 2024-06-09T22:27:37.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:37 smithi153 ceph-mon[36599]: pgmap v202: 129 pgs: 129 active+clean; 140 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 577 KiB/s wr, 19 op/s 2024-06-09T22:27:38.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:37 smithi194 ceph-mon[36856]: pgmap v202: 129 pgs: 129 active+clean; 140 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 577 KiB/s wr, 19 op/s 2024-06-09T22:27:38.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 188795 3.85 MB/sec execute 164 sec latency 16.128 ms 2024-06-09T22:27:39.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 190929 3.90 MB/sec execute 165 sec latency 16.449 ms 2024-06-09T22:27:39.792 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-09T22:27:39.792 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-09T22:27:39.862 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:39 smithi194 ceph-mon[36856]: pgmap v203: 129 pgs: 129 active+clean; 140 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 577 KiB/s wr, 19 op/s 2024-06-09T22:27:39.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:39 smithi153 ceph-mon[36599]: pgmap v203: 129 pgs: 129 active+clean; 140 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 577 KiB/s wr, 19 op/s 2024-06-09T22:27:40.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:39 smithi104 ceph-mon[32389]: pgmap v203: 129 pgs: 129 active+clean; 140 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 577 KiB/s wr, 19 op/s 2024-06-09T22:27:40.045 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:27:40.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 192454 3.92 MB/sec execute 166 sec latency 14.642 ms 2024-06-09T22:27:41.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 194569 3.97 MB/sec execute 167 sec latency 21.509 ms 2024-06-09T22:27:41.872 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:41 smithi104 ceph-mon[32389]: pgmap v204: 129 pgs: 129 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 827 KiB/s wr, 57 op/s 2024-06-09T22:27:41.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:41 smithi153 ceph-mon[36599]: pgmap v204: 129 pgs: 129 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 827 KiB/s wr, 57 op/s 2024-06-09T22:27:42.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:41 smithi194 ceph-mon[36856]: pgmap v204: 129 pgs: 129 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 827 KiB/s wr, 57 op/s 2024-06-09T22:27:42.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.258+0000 7f00c1a8c700 1 -- 172.21.15.104:0/2063995804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00bc103150 msgr2=0x7f00bc107140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:42.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.258+0000 7f00c1a8c700 1 --2- 172.21.15.104:0/2063995804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00bc103150 0x7f00bc107140 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f00b00099c0 tx=0x7f00b0009cd0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.258+0000 7f00c1a8c700 1 -- 172.21.15.104:0/2063995804 shutdown_connections 2024-06-09T22:27:42.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.258+0000 7f00c1a8c700 1 --2- 172.21.15.104:0/2063995804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00bc107870 0x7f00bc10f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.258+0000 7f00c1a8c700 1 --2- 172.21.15.104:0/2063995804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00bc1027a0 0x7f00bc102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.258+0000 7f00c1a8c700 1 --2- 172.21.15.104:0/2063995804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00bc103150 0x7f00bc107140 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.258+0000 7f00c1a8c700 1 -- 172.21.15.104:0/2063995804 >> 172.21.15.104:0/2063995804 conn(0x7f00bc079520 msgr2=0x7f00bc079930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:42.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.258+0000 7f00c1a8c700 1 -- 172.21.15.104:0/2063995804 shutdown_connections 2024-06-09T22:27:42.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.258+0000 7f00c1a8c700 1 -- 172.21.15.104:0/2063995804 wait complete. 2024-06-09T22:27:42.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.258+0000 7f00c1a8c700 1 Processor -- start 2024-06-09T22:27:42.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00c1a8c700 1 -- start start 2024-06-09T22:27:42.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00c1a8c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00bc1027a0 0x7f00bc19e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00c1a8c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00bc103150 0x7f00bc19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00c1a8c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00bc107870 0x7f00bc1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00c1a8c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1127a0 con 0x7f00bc103150 2024-06-09T22:27:42.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00c1a8c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc112620 con 0x7f00bc107870 2024-06-09T22:27:42.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00c1a8c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc112920 con 0x7f00bc1027a0 2024-06-09T22:27:42.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00baffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00bc103150 0x7f00bc19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00bb7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00bc1027a0 0x7f00bc19e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00bbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00bc107870 0x7f00bc1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00bb7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00bc1027a0 0x7f00bc19e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:54822/0 (socket says 172.21.15.104:54822) 2024-06-09T22:27:42.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00baffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00bc103150 0x7f00bc19ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55166/0 (socket says 172.21.15.104:55166) 2024-06-09T22:27:42.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00baffd700 1 -- 172.21.15.104:0/4202592680 learned_addr learned my addr 172.21.15.104:0/4202592680 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00baffd700 1 -- 172.21.15.104:0/4202592680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00bc1027a0 msgr2=0x7f00bc19e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00baffd700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00bc1027a0 0x7f00bc19e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00baffd700 1 -- 172.21.15.104:0/4202592680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00bc107870 msgr2=0x7f00bc1989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00baffd700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00bc107870 0x7f00bc1989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00baffd700 1 -- 172.21.15.104:0/4202592680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00bc199120 con 0x7f00bc103150 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00bbfff700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00bc107870 0x7f00bc1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.259+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00bc1027a0 0x7f00bc19e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.260+0000 7f00baffd700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00bc103150 0x7f00bc19ed40 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f00b0006010 tx=0x7f00b000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.260+0000 7f00b8ff9700 1 -- 172.21.15.104:0/4202592680 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b0003680 con 0x7f00bc103150 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.260+0000 7f00c1a8c700 1 -- 172.21.15.104:0/4202592680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00bc1993b0 con 0x7f00bc103150 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.260+0000 7f00b8ff9700 1 -- 172.21.15.104:0/4202592680 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00b00050d0 con 0x7f00bc103150 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.260+0000 7f00c1a8c700 1 -- 172.21.15.104:0/4202592680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00bc1a56d0 con 0x7f00bc103150 2024-06-09T22:27:42.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.260+0000 7f00b8ff9700 1 -- 172.21.15.104:0/4202592680 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b0003db0 con 0x7f00bc103150 2024-06-09T22:27:42.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.262+0000 7f00b8ff9700 1 -- 172.21.15.104:0/4202592680 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f00b0005280 con 0x7f00bc103150 2024-06-09T22:27:42.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.263+0000 7f00b8ff9700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f00a40a2f30 0x7f00a40a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.263+0000 7f00b8ff9700 1 -- 172.21.15.104:0/4202592680 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f00b00d0450 con 0x7f00bc103150 2024-06-09T22:27:42.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.263+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f00a40a2f30 0x7f00a40a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.263+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f00a40a2f30 0x7f00a40a53f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f00bc199b00 tx=0x7f00ac009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:42.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00cc30fb80 1 Processor -- start 2024-06-09T22:27:42.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00cc30fb80 1 -- start start 2024-06-09T22:27:42.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00cc30fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f746c0 0x55cc26f74aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00cc30fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 0x55cc26f6ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00cc30fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 0x55cc26f6f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00cc30fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55cc26ec0890 con 0x55cc26f74fe0 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00cc30fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cc26e687a0 con 0x55cc26f746c0 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00cc30fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cc26ebc850 con 0x55cc26f6efa0 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00baffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 0x55cc26f6ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00baffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 0x55cc26f6ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55180/0 (socket says 172.21.15.104:55180) 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00baffd700 1 -- 172.21.15.104:0/222145276 learned_addr learned my addr 172.21.15.104:0/222145276 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/222145276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f746c0 0x55cc26f74aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.264+0000 7f00bbfff700 1 --2- 172.21.15.104:0/222145276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 0x55cc26f6f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.265+0000 7f00bbfff700 1 -- 172.21.15.104:0/222145276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f746c0 msgr2=0x55cc26f74aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.265+0000 7f00bbfff700 1 --2- 172.21.15.104:0/222145276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f746c0 0x55cc26f74aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.265+0000 7f00bbfff700 1 -- 172.21.15.104:0/222145276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 msgr2=0x55cc26f6ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.265+0000 7f00bbfff700 1 --2- 172.21.15.104:0/222145276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 0x55cc26f6ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.265+0000 7f00bbfff700 1 -- 172.21.15.104:0/222145276 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc26e5efd0 con 0x55cc26f6efa0 2024-06-09T22:27:42.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.265+0000 7f00bbfff700 1 --2- 172.21.15.104:0/222145276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 0x55cc26f6f3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f00b4003db0 tx=0x7f00b4009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48208ea50268fb47 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:42.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.265+0000 7f00a1ffb700 1 -- 172.21.15.104:0/222145276 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4019030 con 0x55cc26f6efa0 2024-06-09T22:27:42.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.265+0000 7f00a1ffb700 1 -- 172.21.15.104:0/222145276 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00b4004bb0 con 0x55cc26f6efa0 2024-06-09T22:27:42.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.265+0000 7f00a1ffb700 1 -- 172.21.15.104:0/222145276 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4007d40 con 0x55cc26f6efa0 2024-06-09T22:27:42.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.266+0000 7f00cc30fb80 1 -- 172.21.15.104:0/222145276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 msgr2=0x55cc26f6f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:42.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.266+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/222145276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 0x55cc26f6f3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f00b4003db0 tx=0x7f00b4009920 comp rx=0 tx=0).stop 2024-06-09T22:27:42.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.266+0000 7f00cc30fb80 1 -- 172.21.15.104:0/222145276 shutdown_connections 2024-06-09T22:27:42.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.266+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/222145276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 0x55cc26f6f3c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.266+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/222145276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f746c0 0x55cc26f74aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.266+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/222145276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 0x55cc26f6ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.266+0000 7f00cc30fb80 1 -- 172.21.15.104:0/222145276 >> 172.21.15.104:0/222145276 conn(0x55cc26e99d40 msgr2=0x55cc26f698d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.266+0000 7f00cc30fb80 1 -- 172.21.15.104:0/222145276 shutdown_connections 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.266+0000 7f00cc30fb80 1 -- 172.21.15.104:0/222145276 wait complete. 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.266+0000 7f00cc30fb80 1 -- start start 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00cc30fb80 1 Processor -- start 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00cc30fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 0x55cc26f7ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00cc30fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f746c0 0x55cc26f77fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00cc30fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f74fe0 0x55cc26f78520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00cc30fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55cc26f7fe60 con 0x55cc26f746c0 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00cc30fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cc26e60c70 con 0x55cc26f74fe0 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00cc30fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cc26f7ffe0 con 0x55cc26f6efa0 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00baffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f746c0 0x55cc26f77fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00bb7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 0x55cc26f7ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00bbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f74fe0 0x55cc26f78520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00bbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f74fe0 0x55cc26f78520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:47840/0 (socket says 172.21.15.104:47840) 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00bb7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 0x55cc26f7ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:54838/0 (socket says 172.21.15.104:54838) 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00baffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f746c0 0x55cc26f77fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55192/0 (socket says 172.21.15.104:55192) 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00bbfff700 1 -- 172.21.15.104:0/574688267 learned_addr learned my addr 172.21.15.104:0/574688267 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00cc30fb80 10 client.0 _ll_register_callbacks cb 0x55cc26e5a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00baffd700 1 -- 172.21.15.104:0/574688267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 msgr2=0x55cc26f7ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00baffd700 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 0x55cc26f7ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00baffd700 1 -- 172.21.15.104:0/574688267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f74fe0 msgr2=0x55cc26f78520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00baffd700 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f74fe0 0x55cc26f78520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00baffd700 1 -- 172.21.15.104:0/574688267 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc26e0f950 con 0x55cc26f746c0 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 0x55cc26f7ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.267+0000 7f00bbfff700 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f74fe0 0x55cc26f78520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.268+0000 7f00baffd700 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f746c0 0x55cc26f77fe0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f00b00cd890 tx=0x7f00b009c250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ad57de5014a2ca server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.268+0000 7f009affd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.268+0000 7f009affd700 1 -- 172.21.15.104:0/574688267 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b009c690 con 0x55cc26f746c0 2024-06-09T22:27:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.268+0000 7f009affd700 1 -- 172.21.15.104:0/574688267 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f00b00163a0 con 0x55cc26f746c0 2024-06-09T22:27:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.268+0000 7f00cc30fb80 1 -- 172.21.15.104:0/574688267 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00b4004bb0 con 0x55cc26f746c0 2024-06-09T22:27:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.268+0000 7f009affd700 1 -- 172.21.15.104:0/574688267 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f00b00165a0 con 0x55cc26f746c0 2024-06-09T22:27:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.268+0000 7f00cc30fb80 10 client.17361 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.268+0000 7f00cc30fb80 1 -- 172.21.15.104:0/574688267 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cc26f8a6d0 con 0x55cc26f746c0 2024-06-09T22:27:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.268+0000 7f009affd700 1 -- 172.21.15.104:0/574688267 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00b00a0a40 con 0x55cc26f746c0 2024-06-09T22:27:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.268+0000 7f009affd700 1 -- 172.21.15.104:0/574688267 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f00b0085bd0 con 0x55cc26f746c0 2024-06-09T22:27:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.268+0000 7f00cc30fb80 10 client.17361 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.268+0000 7f00cc30fb80 10 client.17361 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.269+0000 7f00cc30fb80 20 client.17361 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.269+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cc26f8cba0 0x55cc26f8ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.269+0000 7f00cc30fb80 4 client.17361 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.269+0000 7f00cc30fb80 1 -- 172.21.15.104:0/574688267 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f00b4004bb0 con 0x55cc26f8cba0 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.269+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cc26f8cba0 0x55cc26f8ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.269+0000 7f009affd700 10 client.17361 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.269+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cc26f8cba0 0x55cc26f8ef90 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a08deb03084b29bf server_cookie=ddcc8127c3384190 in_seq=0 out_seq=0 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f009affd700 1 -- 172.21.15.104:0/574688267 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x7f00b4004bb0 con 0x55cc26f8cba0 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f009affd700 10 client.17361 handle_command_reply: tid=0 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f00cc30fb80 1 client.17361 shutdown 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f00cc30fb80 1 -- 172.21.15.104:0/574688267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f746c0 msgr2=0x55cc26f77fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f746c0 0x55cc26f77fe0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f00b00cd890 tx=0x7f00b009c250 comp rx=0 tx=0).stop 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f00cc30fb80 1 -- 172.21.15.104:0/574688267 shutdown_connections 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cc26f8cba0 0x55cc26f8ef90 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f74fe0 0x55cc26f78520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f6efa0 0x55cc26f7ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f009affd700 0 client.17361 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/574688267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f746c0 0x55cc26f77fe0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f00cc30fb80 1 -- 172.21.15.104:0/574688267 >> 172.21.15.104:0/574688267 conn(0x55cc26e99d40 msgr2=0x55cc26f6da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f00cc30fb80 1 -- 172.21.15.104:0/574688267 shutdown_connections 2024-06-09T22:27:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f00cc30fb80 1 -- 172.21.15.104:0/574688267 wait complete. 2024-06-09T22:27:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.270+0000 7f00cc30fb80 20 client.17361 trim_cache size 0 max 16384 2024-06-09T22:27:42.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00cc30fb80 1 Processor -- start 2024-06-09T22:27:42.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00cc30fb80 1 -- start start 2024-06-09T22:27:42.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00cc30fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f6efa0 0x55cc26f6ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00cc30fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f746c0 0x55cc26f75ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00cc30fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f74fe0 0x55cc26f94880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00cc30fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1ac250 con 0x55cc26f6efa0 2024-06-09T22:27:42.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00cc30fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1add90 con 0x55cc26f746c0 2024-06-09T22:27:42.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00cc30fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1ac920 con 0x55cc26f74fe0 2024-06-09T22:27:42.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00bb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f6efa0 0x55cc26f6ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00bb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f6efa0 0x55cc26f6ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55200/0 (socket says 172.21.15.104:55200) 2024-06-09T22:27:42.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00bb7fe700 1 -- 172.21.15.104:0/2987726434 learned_addr learned my addr 172.21.15.104:0/2987726434 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:42.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00baffd700 1 --2- 172.21.15.104:0/2987726434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f746c0 0x55cc26f75ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00bbfff700 1 --2- 172.21.15.104:0/2987726434 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f74fe0 0x55cc26f94880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00bb7fe700 1 -- 172.21.15.104:0/2987726434 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f74fe0 msgr2=0x55cc26f94880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:42.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/2987726434 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f74fe0 0x55cc26f94880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00bb7fe700 1 -- 172.21.15.104:0/2987726434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f746c0 msgr2=0x55cc26f75ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:42.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/2987726434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f746c0 0x55cc26f75ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00bb7fe700 1 -- 172.21.15.104:0/2987726434 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc26f77db0 con 0x55cc26f6efa0 2024-06-09T22:27:42.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.293+0000 7f00baffd700 1 --2- 172.21.15.104:0/2987726434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f746c0 0x55cc26f75ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:27:42.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.294+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/2987726434 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f6efa0 0x55cc26f6ea60 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f00ac013010 tx=0x7f00ac014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25537ee2844bf81 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:42.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.294+0000 7f00a17fa700 1 -- 172.21.15.104:0/2987726434 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac02a470 con 0x55cc26f6efa0 2024-06-09T22:27:42.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.294+0000 7f00a17fa700 1 -- 172.21.15.104:0/2987726434 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00ac02aaf0 con 0x55cc26f6efa0 2024-06-09T22:27:42.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.294+0000 7f00a17fa700 1 -- 172.21.15.104:0/2987726434 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac01ecc0 con 0x55cc26f6efa0 2024-06-09T22:27:42.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.295+0000 7f00cc30fb80 1 -- 172.21.15.104:0/2987726434 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f6efa0 msgr2=0x55cc26f6ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:42.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.295+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/2987726434 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f6efa0 0x55cc26f6ea60 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f00ac013010 tx=0x7f00ac014000 comp rx=0 tx=0).stop 2024-06-09T22:27:42.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.295+0000 7f00cc30fb80 1 -- 172.21.15.104:0/2987726434 shutdown_connections 2024-06-09T22:27:42.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.295+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/2987726434 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f6efa0 0x55cc26f6ea60 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.295+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/2987726434 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f74fe0 0x55cc26f94880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.295+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/2987726434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f746c0 0x55cc26f75ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.295+0000 7f00cc30fb80 1 -- 172.21.15.104:0/2987726434 >> 172.21.15.104:0/2987726434 conn(0x55cc26e99d40 msgr2=0x55cc26f753c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.295+0000 7f00cc30fb80 1 -- 172.21.15.104:0/2987726434 shutdown_connections 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.295+0000 7f00cc30fb80 1 -- 172.21.15.104:0/2987726434 wait complete. 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.295+0000 7f00cc30fb80 1 -- start start 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.295+0000 7f00cc30fb80 1 Processor -- start 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00cc30fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f6efa0 0x55cc26f858e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00cc30fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f746c0 0x55cc26f82270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00cc30fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 0x55cc26f827b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00cc30fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1ad6c0 con 0x55cc26f74fe0 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00cc30fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1acff0 con 0x55cc26f6efa0 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00cc30fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc1aa1d0 con 0x55cc26f746c0 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00cc30fb80 10 client.0 _ll_register_callbacks cb 0x55cc26e5a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00bbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 0x55cc26f827b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00bb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f6efa0 0x55cc26f858e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00baffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f746c0 0x55cc26f82270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00bb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f6efa0 0x55cc26f858e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:47854/0 (socket says 172.21.15.104:47854) 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00baffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f746c0 0x55cc26f82270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:54856/0 (socket says 172.21.15.104:54856) 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00bbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 0x55cc26f827b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55216/0 (socket says 172.21.15.104:55216) 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00bb7fe700 1 -- 172.21.15.104:0/1034075943 learned_addr learned my addr 172.21.15.104:0/1034075943 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00bb7fe700 1 -- 172.21.15.104:0/1034075943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f746c0 msgr2=0x55cc26f82270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/1034075943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f746c0 0x55cc26f82270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00bb7fe700 1 -- 172.21.15.104:0/1034075943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 msgr2=0x55cc26f827b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/1034075943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 0x55cc26f827b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00bb7fe700 1 -- 172.21.15.104:0/1034075943 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc26f82ee0 con 0x55cc26f6efa0 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00bbfff700 1 --2- 172.21.15.104:0/1034075943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 0x55cc26f827b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f00bb7fe700 1 --2- 172.21.15.104:0/1034075943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f6efa0 0x55cc26f858e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f00ac0094f0 tx=0x7f00ac02ade0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8afdafa77e39d708 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.296+0000 7f009affd700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:27:42.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f009affd700 1 -- 172.21.15.104:0/1034075943 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac03b030 con 0x55cc26f6efa0 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f00cc30fb80 1 -- 172.21.15.104:0/1034075943 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00ac02aaf0 con 0x55cc26f6efa0 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f009affd700 1 -- 172.21.15.104:0/1034075943 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f00ac009020 con 0x55cc26f6efa0 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f009affd700 1 -- 172.21.15.104:0/1034075943 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f00ac045050 con 0x55cc26f6efa0 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f00cc30fb80 10 client.26606 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f00cc30fb80 1 -- 172.21.15.104:0/1034075943 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cc26ff87f0 con 0x55cc26f6efa0 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f009affd700 1 -- 172.21.15.104:0/1034075943 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00ac02aaf0 con 0x55cc26f6efa0 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f009affd700 1 -- 172.21.15.104:0/1034075943 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f00ac032070 con 0x55cc26f6efa0 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f00cc30fb80 10 client.26606 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f00cc30fb80 10 client.26606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f00cc30fb80 20 client.26606 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/1034075943 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cc26ff8ed0 0x55cc26ffb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f00cc30fb80 4 client.26606 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-09T22:27:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f00cc30fb80 1 -- 172.21.15.104:0/1034075943 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cc26ecdd90 con 0x55cc26ff8ed0 2024-06-09T22:27:42.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.297+0000 7f00baffd700 1 --2- 172.21.15.104:0/1034075943 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cc26ff8ed0 0x55cc26ffb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:42.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.298+0000 7f00baffd700 1 --2- 172.21.15.104:0/1034075943 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cc26ff8ed0 0x55cc26ffb2f0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1374d4e7e8b898ac server_cookie=7e661eb2cb3d8af in_seq=0 out_seq=0 2024-06-09T22:27:42.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.298+0000 7f009affd700 10 client.26606 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.299+0000 7f009affd700 1 -- 172.21.15.104:0/1034075943 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cc26ecdd90 con 0x55cc26ff8ed0 2024-06-09T22:27:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.299+0000 7f009affd700 10 client.26606 handle_command_reply: tid=0 2024-06-09T22:27:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.299+0000 7f00cc30fb80 1 client.26606 shutdown 2024-06-09T22:27:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.299+0000 7f00cc30fb80 1 -- 172.21.15.104:0/1034075943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f6efa0 msgr2=0x55cc26f858e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.299+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/1034075943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f6efa0 0x55cc26f858e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f00ac0094f0 tx=0x7f00ac02ade0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.299+0000 7f00cc30fb80 1 -- 172.21.15.104:0/1034075943 shutdown_connections 2024-06-09T22:27:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.299+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/1034075943 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55cc26ff8ed0 0x55cc26ffb2f0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.299+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/1034075943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cc26f74fe0 0x55cc26f827b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.299+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/1034075943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cc26f6efa0 0x55cc26f858e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.299+0000 7f009affd700 0 client.26606 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:42.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.299+0000 7f00cc30fb80 1 --2- 172.21.15.104:0/1034075943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc26f746c0 0x55cc26f82270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.299+0000 7f00cc30fb80 1 -- 172.21.15.104:0/1034075943 >> 172.21.15.104:0/1034075943 conn(0x55cc26e99d40 msgr2=0x55cc26f62e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:42.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00cc30fb80 1 -- 172.21.15.104:0/1034075943 shutdown_connections 2024-06-09T22:27:42.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00cc30fb80 1 -- 172.21.15.104:0/1034075943 wait complete. 2024-06-09T22:27:42.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00cc30fb80 20 client.26606 trim_cache size 0 max 16384 2024-06-09T22:27:42.305 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:27:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: "return_code": 0, 2024-06-09T22:27:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_tag": "d8fec2b9-0ca9-4b43-beda-f632b817a17c", 2024-06-09T22:27:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": "asynchronous" 2024-06-09T22:27:42.306 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:27:42.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00c1a8c700 1 -- 172.21.15.104:0/4202592680 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f00a40a2f30 msgr2=0x7f00a40a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:42.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00c1a8c700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f00a40a2f30 0x7f00a40a53f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f00bc199b00 tx=0x7f00ac009260 comp rx=0 tx=0).stop 2024-06-09T22:27:42.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00c1a8c700 1 -- 172.21.15.104:0/4202592680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00bc103150 msgr2=0x7f00bc19ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:42.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00c1a8c700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00bc103150 0x7f00bc19ed40 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f00b0006010 tx=0x7f00b000f690 comp rx=0 tx=0).stop 2024-06-09T22:27:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00c1a8c700 1 -- 172.21.15.104:0/4202592680 shutdown_connections 2024-06-09T22:27:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00c1a8c700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f00a40a2f30 0x7f00a40a53f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00c1a8c700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00bc107870 0x7f00bc1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00c1a8c700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00bc1027a0 0x7f00bc19e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00c1a8c700 1 --2- 172.21.15.104:0/4202592680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00bc103150 0x7f00bc19ed40 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00c1a8c700 1 -- 172.21.15.104:0/4202592680 >> 172.21.15.104:0/4202592680 conn(0x7f00bc079520 msgr2=0x7f00bc101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00c1a8c700 1 -- 172.21.15.104:0/4202592680 shutdown_connections 2024-06-09T22:27:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:42.300+0000 7f00c1a8c700 1 -- 172.21.15.104:0/4202592680 wait complete. 2024-06-09T22:27:42.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 196004 3.99 MB/sec execute 168 sec latency 15.584 ms 2024-06-09T22:27:43.115 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:27:43.368 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:27:43.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 198268 4.04 MB/sec execute 169 sec latency 16.192 ms 2024-06-09T22:27:43.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:43 smithi104 ceph-mon[32389]: pgmap v205: 129 pgs: 129 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 823 KiB/s wr, 52 op/s 2024-06-09T22:27:43.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:43 smithi104 ceph-mon[32389]: scrub queued for path: / 2024-06-09T22:27:43.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:43 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:43.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:43 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:27:43.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:43 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:27:43.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:43 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:27:43.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:43 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:43.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:43 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:27:43.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:43 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:43.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:43 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:43.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:43 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:43.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:43 smithi153 ceph-mon[36599]: pgmap v205: 129 pgs: 129 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 823 KiB/s wr, 52 op/s 2024-06-09T22:27:43.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:43 smithi153 ceph-mon[36599]: scrub queued for path: / 2024-06-09T22:27:43.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:43 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:43.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:43 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:27:43.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:43 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:27:43.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:43 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:27:43.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:43 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:43.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:43 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:27:43.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:43 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:43.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:43 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:43.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:43 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:44.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:43 smithi194 ceph-mon[36856]: pgmap v205: 129 pgs: 129 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 823 KiB/s wr, 52 op/s 2024-06-09T22:27:44.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:43 smithi194 ceph-mon[36856]: scrub queued for path: / 2024-06-09T22:27:44.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:43 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:44.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:43 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:27:44.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:43 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:27:44.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:43 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:27:44.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:43 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:44.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:43 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:27:44.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:43 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:44.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:43 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:44.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:43 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:27:44.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 199588 4.05 MB/sec execute 170 sec latency 32.898 ms 2024-06-09T22:27:45.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.164+0000 7f547ae58700 1 -- 172.21.15.104:0/3067531540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5474101690 msgr2=0x7f5474101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:45.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.164+0000 7f547ae58700 1 --2- 172.21.15.104:0/3067531540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5474101690 0x7f5474101b10 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f54640099c0 tx=0x7f5464009cd0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.164+0000 7f547ae58700 1 -- 172.21.15.104:0/3067531540 shutdown_connections 2024-06-09T22:27:45.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.164+0000 7f547ae58700 1 --2- 172.21.15.104:0/3067531540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5474102050 0x7f547410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.164+0000 7f547ae58700 1 --2- 172.21.15.104:0/3067531540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5474107590 0x7f5474107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.164+0000 7f547ae58700 1 --2- 172.21.15.104:0/3067531540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5474101690 0x7f5474101b10 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.164+0000 7f547ae58700 1 -- 172.21.15.104:0/3067531540 >> 172.21.15.104:0/3067531540 conn(0x7f5474078150 msgr2=0x7f5474078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.164+0000 7f547ae58700 1 -- 172.21.15.104:0/3067531540 shutdown_connections 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.164+0000 7f547ae58700 1 -- 172.21.15.104:0/3067531540 wait complete. 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f547ae58700 1 Processor -- start 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f547ae58700 1 -- start start 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f547ae58700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5474101690 0x7f5474197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f547ae58700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5474102050 0x7f5474197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f547ae58700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5474107590 0x7f54741982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f547ae58700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5474111710 con 0x7f5474101690 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f547ae58700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5474111590 con 0x7f5474107590 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f547ae58700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5474111890 con 0x7f5474102050 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f5478bf4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5474101690 0x7f5474197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f5478bf4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5474101690 0x7f5474197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55244/0 (socket says 172.21.15.104:55244) 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f5478bf4700 1 -- 172.21.15.104:0/3441783140 learned_addr learned my addr 172.21.15.104:0/3441783140 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:45.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f546bfff700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5474102050 0x7f5474197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f54793f5700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5474107590 0x7f54741982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.165+0000 7f5478bf4700 1 -- 172.21.15.104:0/3441783140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5474102050 msgr2=0x7f5474197d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.166+0000 7f5478bf4700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5474102050 0x7f5474197d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.166+0000 7f5478bf4700 1 -- 172.21.15.104:0/3441783140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5474107590 msgr2=0x7f54741982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.166+0000 7f5478bf4700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5474107590 0x7f54741982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.166+0000 7f5478bf4700 1 -- 172.21.15.104:0/3441783140 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54741a70b0 con 0x7f5474101690 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.166+0000 7f546bfff700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5474102050 0x7f5474197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.166+0000 7f54793f5700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5474107590 0x7f54741982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.166+0000 7f5478bf4700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5474101690 0x7f5474197840 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f547000e9c0 tx=0x7f547000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.166+0000 7f5469ffb700 1 -- 172.21.15.104:0/3441783140 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547000cbd0 con 0x7f5474101690 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.166+0000 7f5469ffb700 1 -- 172.21.15.104:0/3441783140 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f547000cd70 con 0x7f5474101690 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.166+0000 7f547ae58700 1 -- 172.21.15.104:0/3441783140 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54741a73a0 con 0x7f5474101690 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.166+0000 7f5469ffb700 1 -- 172.21.15.104:0/3441783140 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5470010430 con 0x7f5474101690 2024-06-09T22:27:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.166+0000 7f547ae58700 1 -- 172.21.15.104:0/3441783140 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54741a7890 con 0x7f5474101690 2024-06-09T22:27:45.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.168+0000 7f5469ffb700 1 -- 172.21.15.104:0/3441783140 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5470004750 con 0x7f5474101690 2024-06-09T22:27:45.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.169+0000 7f5469ffb700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f545c0a2d90 0x7f545c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.169+0000 7f5469ffb700 1 -- 172.21.15.104:0/3441783140 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f54700cfa60 con 0x7f5474101690 2024-06-09T22:27:45.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.169+0000 7f546bfff700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f545c0a2d90 0x7f545c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.169+0000 7f546bfff700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f545c0a2d90 0x7f545c0a5250 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f54741999a0 tx=0x7f5464005fb0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f54856dbb80 1 Processor -- start 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f54856dbb80 1 -- start start 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f54856dbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 0x561cc2456a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f54856dbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2456f60 0x561cc24509e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f54856dbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2450f20 0x561cc2451340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f54856dbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561cc23a2810 con 0x561cc2450f20 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f54856dbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561cc234a720 con 0x561cc2456640 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f54856dbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561cc239e7d0 con 0x561cc2456f60 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f5478bf4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 0x561cc2456a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f54793f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2450f20 0x561cc2451340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f5478bf4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 0x561cc2456a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:47888/0 (socket says 172.21.15.104:47888) 2024-06-09T22:27:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f546bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2456f60 0x561cc24509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f5478bf4700 1 -- 172.21.15.104:0/4065220314 learned_addr learned my addr 172.21.15.104:0/4065220314 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:45.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f5478bf4700 1 -- 172.21.15.104:0/4065220314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2456f60 msgr2=0x561cc24509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:45.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f5478bf4700 1 --2- 172.21.15.104:0/4065220314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2456f60 0x561cc24509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f5478bf4700 1 -- 172.21.15.104:0/4065220314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2450f20 msgr2=0x561cc2451340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:45.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f5478bf4700 1 --2- 172.21.15.104:0/4065220314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2450f20 0x561cc2451340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f5478bf4700 1 -- 172.21.15.104:0/4065220314 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cc2340f50 con 0x561cc2456640 2024-06-09T22:27:45.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.170+0000 7f54793f5700 1 --2- 172.21.15.104:0/4065220314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2450f20 0x561cc2451340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:45.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.171+0000 7f5478bf4700 1 --2- 172.21.15.104:0/4065220314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 0x561cc2456a20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f54700985e0 tx=0x7f547009b750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff020de043b5bc1b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:45.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.171+0000 7f545affd700 1 -- 172.21.15.104:0/4065220314 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547009bc10 con 0x561cc2456640 2024-06-09T22:27:45.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.171+0000 7f545affd700 1 -- 172.21.15.104:0/4065220314 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f547009ffc0 con 0x561cc2456640 2024-06-09T22:27:45.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.171+0000 7f545affd700 1 -- 172.21.15.104:0/4065220314 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5470004a40 con 0x561cc2456640 2024-06-09T22:27:45.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.171+0000 7f54856dbb80 1 -- 172.21.15.104:0/4065220314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 msgr2=0x561cc2456a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.172+0000 7f54856dbb80 1 --2- 172.21.15.104:0/4065220314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 0x561cc2456a20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f54700985e0 tx=0x7f547009b750 comp rx=0 tx=0).stop 2024-06-09T22:27:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.172+0000 7f54856dbb80 1 -- 172.21.15.104:0/4065220314 shutdown_connections 2024-06-09T22:27:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.172+0000 7f54856dbb80 1 --2- 172.21.15.104:0/4065220314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2450f20 0x561cc2451340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.172+0000 7f54856dbb80 1 --2- 172.21.15.104:0/4065220314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 0x561cc2456a20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.172+0000 7f54856dbb80 1 --2- 172.21.15.104:0/4065220314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2456f60 0x561cc24509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.172+0000 7f54856dbb80 1 -- 172.21.15.104:0/4065220314 >> 172.21.15.104:0/4065220314 conn(0x561cc237bcc0 msgr2=0x561cc244b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.172+0000 7f54856dbb80 1 -- 172.21.15.104:0/4065220314 shutdown_connections 2024-06-09T22:27:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.172+0000 7f54856dbb80 1 -- 172.21.15.104:0/4065220314 wait complete. 2024-06-09T22:27:45.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.172+0000 7f54856dbb80 1 -- start start 2024-06-09T22:27:45.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.172+0000 7f54856dbb80 1 Processor -- start 2024-06-09T22:27:45.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.172+0000 7f54856dbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2450f20 0x561cc245cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f54856dbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 0x561cc245a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f54856dbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2456f60 0x561cc245a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f54856dbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561cc245d9d0 con 0x561cc2450f20 2024-06-09T22:27:45.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f54856dbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561cc2342bf0 con 0x561cc2456640 2024-06-09T22:27:45.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f54856dbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561cc245db50 con 0x561cc2456f60 2024-06-09T22:27:45.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f54856dbb80 10 client.0 _ll_register_callbacks cb 0x561cc233c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:45.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f546bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 0x561cc245a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f546bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 0x561cc245a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:47902/0 (socket says 172.21.15.104:47902) 2024-06-09T22:27:45.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f5478bf4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2450f20 0x561cc245cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f546bfff700 1 -- 172.21.15.104:0/1331723797 learned_addr learned my addr 172.21.15.104:0/1331723797 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:45.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f54793f5700 1 --2- 172.21.15.104:0/1331723797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2456f60 0x561cc245a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f546bfff700 1 -- 172.21.15.104:0/1331723797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2456f60 msgr2=0x561cc245a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:45.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f546bfff700 1 --2- 172.21.15.104:0/1331723797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2456f60 0x561cc245a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f546bfff700 1 -- 172.21.15.104:0/1331723797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2450f20 msgr2=0x561cc245cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:45.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f546bfff700 1 --2- 172.21.15.104:0/1331723797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2450f20 0x561cc245cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f546bfff700 1 -- 172.21.15.104:0/1331723797 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cc22f18d0 con 0x561cc2456640 2024-06-09T22:27:45.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f546bfff700 1 --2- 172.21.15.104:0/1331723797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 0x561cc245a0b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f54640099c0 tx=0x7f5464009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91edad9ece8eee31 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:45.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.173+0000 7f5453fff700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:27:45.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f5453fff700 1 -- 172.21.15.104:0/1331723797 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546400f460 con 0x561cc2456640 2024-06-09T22:27:45.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f54856dbb80 1 -- 172.21.15.104:0/1331723797 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f547009ffc0 con 0x561cc2456640 2024-06-09T22:27:45.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f5453fff700 1 -- 172.21.15.104:0/1331723797 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f546400f600 con 0x561cc2456640 2024-06-09T22:27:45.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f5453fff700 1 -- 172.21.15.104:0/1331723797 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f546403d050 con 0x561cc2456640 2024-06-09T22:27:45.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f54856dbb80 10 client.26618 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:45.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f54856dbb80 1 -- 172.21.15.104:0/1331723797 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561cc2468390 con 0x561cc2456640 2024-06-09T22:27:45.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f5453fff700 1 -- 172.21.15.104:0/1331723797 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f547009ffc0 con 0x561cc2456640 2024-06-09T22:27:45.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f5453fff700 1 -- 172.21.15.104:0/1331723797 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5464024960 con 0x561cc2456640 2024-06-09T22:27:45.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f54856dbb80 10 client.26618 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:45.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f54856dbb80 10 client.26618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:45.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f54856dbb80 20 client.26618 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:45.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f54856dbb80 1 --2- 172.21.15.104:0/1331723797 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561cc246a9d0 0x561cc246cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f54856dbb80 4 client.26618 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:45.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f54856dbb80 1 -- 172.21.15.104:0/1331723797 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cc23afd10 con 0x561cc246a9d0 2024-06-09T22:27:45.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.174+0000 7f5478bf4700 1 --2- 172.21.15.104:0/1331723797 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561cc246a9d0 0x561cc246cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.175+0000 7f5453fff700 10 client.26618 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:45.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.175+0000 7f5478bf4700 1 --2- 172.21.15.104:0/1331723797 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561cc246a9d0 0x561cc246cdc0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36628f1f27bb38b1 server_cookie=c346079ac8259af3 in_seq=0 out_seq=0 2024-06-09T22:27:45.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.175+0000 7f5453fff700 1 -- 172.21.15.104:0/1331723797 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x561cc23afd10 con 0x561cc246a9d0 2024-06-09T22:27:45.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.175+0000 7f5453fff700 10 client.26618 handle_command_reply: tid=0 2024-06-09T22:27:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f54856dbb80 1 client.26618 shutdown 2024-06-09T22:27:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f54856dbb80 1 -- 172.21.15.104:0/1331723797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 msgr2=0x561cc245a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f54856dbb80 1 --2- 172.21.15.104:0/1331723797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 0x561cc245a0b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f54640099c0 tx=0x7f5464009cd0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f54856dbb80 1 -- 172.21.15.104:0/1331723797 shutdown_connections 2024-06-09T22:27:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f54856dbb80 1 --2- 172.21.15.104:0/1331723797 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561cc246a9d0 0x561cc246cdc0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f54856dbb80 1 --2- 172.21.15.104:0/1331723797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2450f20 0x561cc245cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f54856dbb80 1 --2- 172.21.15.104:0/1331723797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2456f60 0x561cc245a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f5453fff700 0 client.26618 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f54856dbb80 1 --2- 172.21.15.104:0/1331723797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456640 0x561cc245a0b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f54856dbb80 1 -- 172.21.15.104:0/1331723797 >> 172.21.15.104:0/1331723797 conn(0x561cc237bcc0 msgr2=0x561cc244f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f54856dbb80 1 -- 172.21.15.104:0/1331723797 shutdown_connections 2024-06-09T22:27:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f54856dbb80 1 -- 172.21.15.104:0/1331723797 wait complete. 2024-06-09T22:27:45.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.176+0000 7f54856dbb80 20 client.26618 trim_cache size 0 max 16384 2024-06-09T22:27:45.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f54856dbb80 1 Processor -- start 2024-06-09T22:27:45.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f54856dbb80 1 -- start start 2024-06-09T22:27:45.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f54856dbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2450f20 0x561cc24509e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f54856dbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 0x561cc2466c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f54856dbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456f60 0x561cc2459010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f54856dbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f54741b0640 con 0x561cc2456640 2024-06-09T22:27:45.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f54856dbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54741b2180 con 0x561cc2456f60 2024-06-09T22:27:45.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f54856dbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f54741b0d10 con 0x561cc2450f20 2024-06-09T22:27:45.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f546bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 0x561cc2466c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f546bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 0x561cc2466c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55260/0 (socket says 172.21.15.104:55260) 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f546bfff700 1 -- 172.21.15.104:0/4046092923 learned_addr learned my addr 172.21.15.104:0/4046092923 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f5478bf4700 1 --2- 172.21.15.104:0/4046092923 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2450f20 0x561cc24509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.198+0000 7f54793f5700 1 --2- 172.21.15.104:0/4046092923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456f60 0x561cc2459010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.199+0000 7f546bfff700 1 -- 172.21.15.104:0/4046092923 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2450f20 msgr2=0x561cc24509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.199+0000 7f546bfff700 1 --2- 172.21.15.104:0/4046092923 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2450f20 0x561cc24509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.199+0000 7f546bfff700 1 -- 172.21.15.104:0/4046092923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456f60 msgr2=0x561cc2459010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.199+0000 7f546bfff700 1 --2- 172.21.15.104:0/4046092923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456f60 0x561cc2459010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.199+0000 7f546bfff700 1 -- 172.21.15.104:0/4046092923 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cc2458620 con 0x561cc2456640 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.199+0000 7f54793f5700 1 --2- 172.21.15.104:0/4046092923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456f60 0x561cc2459010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.199+0000 7f546bfff700 1 --2- 172.21.15.104:0/4046092923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 0x561cc2466c70 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f54640098e0 tx=0x7f54640059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15a4cfa7982399d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.199+0000 7f545a7fc700 1 -- 172.21.15.104:0/4046092923 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54640243e0 con 0x561cc2456640 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.199+0000 7f545a7fc700 1 -- 172.21.15.104:0/4046092923 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5464034070 con 0x561cc2456640 2024-06-09T22:27:45.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.199+0000 7f545a7fc700 1 -- 172.21.15.104:0/4046092923 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464041b60 con 0x561cc2456640 2024-06-09T22:27:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.200+0000 7f54856dbb80 1 -- 172.21.15.104:0/4046092923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 msgr2=0x561cc2466c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.200+0000 7f54856dbb80 1 --2- 172.21.15.104:0/4046092923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 0x561cc2466c70 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f54640098e0 tx=0x7f54640059d0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.200+0000 7f54856dbb80 1 -- 172.21.15.104:0/4046092923 shutdown_connections 2024-06-09T22:27:45.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.200+0000 7f54856dbb80 1 --2- 172.21.15.104:0/4046092923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456f60 0x561cc2459010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.200+0000 7f54856dbb80 1 --2- 172.21.15.104:0/4046092923 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2450f20 0x561cc24509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.200+0000 7f54856dbb80 1 --2- 172.21.15.104:0/4046092923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 0x561cc2466c70 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.200+0000 7f54856dbb80 1 -- 172.21.15.104:0/4046092923 >> 172.21.15.104:0/4046092923 conn(0x561cc237bcc0 msgr2=0x561cc2444e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:45.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.200+0000 7f54856dbb80 1 -- 172.21.15.104:0/4046092923 shutdown_connections 2024-06-09T22:27:45.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.200+0000 7f54856dbb80 1 -- 172.21.15.104:0/4046092923 wait complete. 2024-06-09T22:27:45.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.200+0000 7f54856dbb80 1 -- start start 2024-06-09T22:27:45.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.200+0000 7f54856dbb80 1 Processor -- start 2024-06-09T22:27:45.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f54856dbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2450f20 0x561cc2462de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f54856dbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 0x561cc2463320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f54856dbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456f60 0x561cc245fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f54856dbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f54741b1ab0 con 0x561cc2456640 2024-06-09T22:27:45.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f54856dbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54741b13e0 con 0x561cc2456f60 2024-06-09T22:27:45.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f54856dbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f54741ae5c0 con 0x561cc2450f20 2024-06-09T22:27:45.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f54856dbb80 10 client.0 _ll_register_callbacks cb 0x561cc233c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:45.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f546bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 0x561cc2463320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f5478bf4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2450f20 0x561cc2462de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f546bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 0x561cc2463320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55262/0 (socket says 172.21.15.104:55262) 2024-06-09T22:27:45.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f5478bf4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2450f20 0x561cc2462de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:54932/0 (socket says 172.21.15.104:54932) 2024-06-09T22:27:45.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f546bfff700 1 -- 172.21.15.104:0/2487656966 learned_addr learned my addr 172.21.15.104:0/2487656966 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:45.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f54793f5700 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456f60 0x561cc245fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f546bfff700 1 -- 172.21.15.104:0/2487656966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2450f20 msgr2=0x561cc2462de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f546bfff700 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2450f20 0x561cc2462de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f546bfff700 1 -- 172.21.15.104:0/2487656966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456f60 msgr2=0x561cc245fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f546bfff700 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456f60 0x561cc245fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f546bfff700 1 -- 172.21.15.104:0/2487656966 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cc24603f0 con 0x561cc2456640 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f5478bf4700 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2450f20 0x561cc2462de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f546bfff700 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 0x561cc2463320 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5464000c00 tx=0x7f5464009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f6d59532a80695 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f5453fff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f5453fff700 1 -- 172.21.15.104:0/2487656966 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546404f030 con 0x561cc2456640 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f54856dbb80 1 -- 172.21.15.104:0/2487656966 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5464034070 con 0x561cc2456640 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.201+0000 7f5453fff700 1 -- 172.21.15.104:0/2487656966 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7f54640417a0 con 0x561cc2456640 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.202+0000 7f5453fff700 1 -- 172.21.15.104:0/2487656966 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5464059050 con 0x561cc2456640 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.202+0000 7f54856dbb80 10 client.17385 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.202+0000 7f54856dbb80 1 -- 172.21.15.104:0/2487656966 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561cc24d5e90 con 0x561cc2456640 2024-06-09T22:27:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.202+0000 7f5453fff700 1 -- 172.21.15.104:0/2487656966 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5464034070 con 0x561cc2456640 2024-06-09T22:27:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.202+0000 7f5453fff700 1 -- 172.21.15.104:0/2487656966 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f546404a450 con 0x561cc2456640 2024-06-09T22:27:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.202+0000 7f54856dbb80 10 client.17385 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.202+0000 7f54856dbb80 10 client.17385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.202+0000 7f54856dbb80 20 client.17385 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.202+0000 7f54856dbb80 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561cc246a9d0 0x561cc24d84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.202+0000 7f54856dbb80 4 client.17385 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:27:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.202+0000 7f54856dbb80 1 -- 172.21.15.104:0/2487656966 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561cc23aa290 con 0x561cc246a9d0 2024-06-09T22:27:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.202+0000 7f5478bf4700 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561cc246a9d0 0x561cc24d84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f5478bf4700 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561cc246a9d0 0x561cc24d84c0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9a69bd7cbeb0e00 server_cookie=aa0cdbb73c28e384 in_seq=0 out_seq=0 2024-06-09T22:27:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f5453fff700 10 client.17385 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f5453fff700 1 -- 172.21.15.104:0/2487656966 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561cc23aa290 con 0x561cc246a9d0 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f5453fff700 10 client.17385 handle_command_reply: tid=0 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f54856dbb80 1 client.17385 shutdown 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f54856dbb80 1 -- 172.21.15.104:0/2487656966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 msgr2=0x561cc2463320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f54856dbb80 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 0x561cc2463320 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5464000c00 tx=0x7f5464009cd0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f54856dbb80 1 -- 172.21.15.104:0/2487656966 shutdown_connections 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f54856dbb80 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x561cc246a9d0 0x561cc24d84c0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f54856dbb80 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cc2456f60 0x561cc245fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f54856dbb80 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561cc2450f20 0x561cc2462de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f54856dbb80 1 --2- 172.21.15.104:0/2487656966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561cc2456640 0x561cc2463320 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f54856dbb80 1 -- 172.21.15.104:0/2487656966 >> 172.21.15.104:0/2487656966 conn(0x561cc237bcc0 msgr2=0x561cc2457340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f5453fff700 0 client.17385 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f54856dbb80 1 -- 172.21.15.104:0/2487656966 shutdown_connections 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f54856dbb80 1 -- 172.21.15.104:0/2487656966 wait complete. 2024-06-09T22:27:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.203+0000 7f54856dbb80 20 client.17385 trim_cache size 0 max 16384 2024-06-09T22:27:45.212 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:27:45.212 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-09T22:27:45.212 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": { 2024-06-09T22:27:45.212 INFO:teuthology.orchestra.run.smithi104.stdout: "d8fec2b9-0ca9-4b43-beda-f632b817a17c": { 2024-06-09T22:27:45.212 INFO:teuthology.orchestra.run.smithi104.stdout: "path": "/", 2024-06-09T22:27:45.212 INFO:teuthology.orchestra.run.smithi104.stdout: "tag": "d8fec2b9-0ca9-4b43-beda-f632b817a17c", 2024-06-09T22:27:45.212 INFO:teuthology.orchestra.run.smithi104.stdout: "options": "recursive,force" 2024-06-09T22:27:45.212 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:27:45.213 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:27:45.213 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:27:45.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.204+0000 7f547ae58700 1 -- 172.21.15.104:0/3441783140 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f545c0a2d90 msgr2=0x7f545c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:45.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.204+0000 7f547ae58700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f545c0a2d90 0x7f545c0a5250 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f54741999a0 tx=0x7f5464005fb0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.204+0000 7f547ae58700 1 -- 172.21.15.104:0/3441783140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5474101690 msgr2=0x7f5474197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:45.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.204+0000 7f547ae58700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5474101690 0x7f5474197840 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f547000e9c0 tx=0x7f547000ecd0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.204+0000 7f547ae58700 1 -- 172.21.15.104:0/3441783140 shutdown_connections 2024-06-09T22:27:45.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.204+0000 7f547ae58700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f545c0a2d90 0x7f545c0a5250 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.204+0000 7f547ae58700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5474101690 0x7f5474197840 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.204+0000 7f547ae58700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5474107590 0x7f54741982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.204+0000 7f547ae58700 1 --2- 172.21.15.104:0/3441783140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5474102050 0x7f5474197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:45.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.204+0000 7f547ae58700 1 -- 172.21.15.104:0/3441783140 >> 172.21.15.104:0/3441783140 conn(0x7f5474078150 msgr2=0x7f54741001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:45.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.204+0000 7f547ae58700 1 -- 172.21.15.104:0/3441783140 shutdown_connections 2024-06-09T22:27:45.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:27:45.204+0000 7f547ae58700 1 -- 172.21.15.104:0/3441783140 wait complete. 2024-06-09T22:27:45.735 INFO:tasks.workunit.client.0.smithi104.stdout: 1 201845 4.10 MB/sec execute 171 sec latency 17.924 ms 2024-06-09T22:27:45.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:45 smithi153 ceph-mon[36599]: pgmap v206: 129 pgs: 129 active+clean; 144 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-06-09T22:27:46.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:45 smithi104 ceph-mon[32389]: pgmap v206: 129 pgs: 129 active+clean; 144 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-06-09T22:27:46.061 INFO:tasks.cephfs.filesystem:scrub status for tag:d8fec2b9-0ca9-4b43-beda-f632b817a17c - {'path': '/', 'tag': 'd8fec2b9-0ca9-4b43-beda-f632b817a17c', 'options': 'recursive,force'} 2024-06-09T22:27:46.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:45 smithi194 ceph-mon[36856]: pgmap v206: 129 pgs: 129 active+clean; 144 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-06-09T22:27:46.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 203175 4.12 MB/sec execute 172 sec latency 14.330 ms 2024-06-09T22:27:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.522+0000 7feb61e76640 1 -- 172.21.15.104:0/2181917179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb5c103550 msgr2=0x7feb5c10ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.522+0000 7feb61e76640 1 --2- 172.21.15.104:0/2181917179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb5c103550 0x7feb5c10ba30 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7feb50009860 tx=0x7feb5002a460 comp rx=0 tx=0).stop 2024-06-09T22:27:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.523+0000 7feb61e76640 1 -- 172.21.15.104:0/2181917179 shutdown_connections 2024-06-09T22:27:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.523+0000 7feb61e76640 1 --2- 172.21.15.104:0/2181917179 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb5c10bf70 0x7feb5c10e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.523+0000 7feb61e76640 1 --2- 172.21.15.104:0/2181917179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb5c103550 0x7feb5c10ba30 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.523+0000 7feb61e76640 1 --2- 172.21.15.104:0/2181917179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb5c102c30 0x7feb5c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.523+0000 7feb61e76640 1 -- 172.21.15.104:0/2181917179 >> 172.21.15.104:0/2181917179 conn(0x7feb5c077a20 msgr2=0x7feb5c077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.523+0000 7feb61e76640 1 -- 172.21.15.104:0/2181917179 shutdown_connections 2024-06-09T22:27:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.523+0000 7feb61e76640 1 -- 172.21.15.104:0/2181917179 wait complete. 2024-06-09T22:27:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb61e76640 1 Processor -- start 2024-06-09T22:27:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb61e76640 1 -- start start 2024-06-09T22:27:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb61e76640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb5c102c30 0x7feb5c1982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb61e76640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb5c103550 0x7feb5c198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb61e76640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb5c10bf70 0x7feb5c19cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb61e76640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feb5c111d70 con 0x7feb5c103550 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb61e76640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb5c111bf0 con 0x7feb5c10bf70 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb61e76640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feb5c111ef0 con 0x7feb5c102c30 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb5affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb5c103550 0x7feb5c198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb5c102c30 0x7feb5c1982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb5bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb5c10bf70 0x7feb5c19cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb5c102c30 0x7feb5c1982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:33612/0 (socket says 172.21.15.104:33612) 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.524+0000 7feb5b7fe640 1 -- 172.21.15.104:0/1089936119 learned_addr learned my addr 172.21.15.104:0/1089936119 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.525+0000 7feb5b7fe640 1 -- 172.21.15.104:0/1089936119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb5c10bf70 msgr2=0x7feb5c19cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.525+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb5c10bf70 0x7feb5c19cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.525+0000 7feb5b7fe640 1 -- 172.21.15.104:0/1089936119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb5c103550 msgr2=0x7feb5c198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.525+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb5c103550 0x7feb5c198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.525+0000 7feb5b7fe640 1 -- 172.21.15.104:0/1089936119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb5c19d270 con 0x7feb5c102c30 2024-06-09T22:27:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.525+0000 7feb5bfff640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb5c10bf70 0x7feb5c19cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.525+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb5c102c30 0x7feb5c1982d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feb4c00e9c0 tx=0x7feb4c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.525+0000 7feb58ff9640 1 -- 172.21.15.104:0/1089936119 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c00cde0 con 0x7feb5c102c30 2024-06-09T22:27:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.525+0000 7feb58ff9640 1 -- 172.21.15.104:0/1089936119 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb4c0035f0 con 0x7feb5c102c30 2024-06-09T22:27:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.525+0000 7feb61e76640 1 -- 172.21.15.104:0/1089936119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb5c19d560 con 0x7feb5c102c30 2024-06-09T22:27:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.525+0000 7feb58ff9640 1 -- 172.21.15.104:0/1089936119 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c010690 con 0x7feb5c102c30 2024-06-09T22:27:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.525+0000 7feb61e76640 1 -- 172.21.15.104:0/1089936119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb5c1a4df0 con 0x7feb5c102c30 2024-06-09T22:27:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.527+0000 7feb58ff9640 1 -- 172.21.15.104:0/1089936119 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7feb4c0037a0 con 0x7feb5c102c30 2024-06-09T22:27:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.528+0000 7feb58ff9640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7feb440a2da0 0x7feb440a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.528+0000 7feb5affd640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7feb440a2da0 0x7feb440a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.528+0000 7feb58ff9640 1 -- 172.21.15.104:0/1089936119 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7feb4c0cfaa0 con 0x7feb5c102c30 2024-06-09T22:27:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.528+0000 7feb64b84740 1 Processor -- start 2024-06-09T22:27:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.528+0000 7feb64b84740 1 -- start start 2024-06-09T22:27:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb64b84740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 0x55c0848ba550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb64b84740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848baa90 0x55c0848c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb64b84740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848c5350 0x55c0848c7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb5affd640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7feb440a2da0 0x7feb440a5260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feb5c10ddb0 tx=0x7feb500094d0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb64b84740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55c08481e4f0 con 0x55c0848ba170 2024-06-09T22:27:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb64b84740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c0847bc870 con 0x55c0848baa90 2024-06-09T22:27:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb64b84740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c08481a4b0 con 0x55c0848c5350 2024-06-09T22:27:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 0x55c0848ba550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 0x55c0848ba550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50380/0 (socket says 172.21.15.104:50380) 2024-06-09T22:27:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb5b7fe640 1 -- 172.21.15.104:0/213787652 learned_addr learned my addr 172.21.15.104:0/213787652 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb5affd640 1 --2- 172.21.15.104:0/213787652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848baa90 0x55c0848c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb5bfff640 1 --2- 172.21.15.104:0/213787652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848c5350 0x55c0848c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb5b7fe640 1 -- 172.21.15.104:0/213787652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848c5350 msgr2=0x55c0848c7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/213787652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848c5350 0x55c0848c7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb5b7fe640 1 -- 172.21.15.104:0/213787652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848baa90 msgr2=0x55c0848c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/213787652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848baa90 0x55c0848c4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb5b7fe640 1 -- 172.21.15.104:0/213787652 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0847d89b0 con 0x55c0848ba170 2024-06-09T22:27:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.529+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/213787652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 0x55c0848ba550 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7feb4c00c8e0 tx=0x7feb4c09b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fcc092579aa2dca server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.530+0000 7feb41ffb640 1 -- 172.21.15.104:0/213787652 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c0b1070 con 0x55c0848ba170 2024-06-09T22:27:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.530+0000 7feb41ffb640 1 -- 172.21.15.104:0/213787652 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb4c0b4e50 con 0x55c0848ba170 2024-06-09T22:27:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.530+0000 7feb41ffb640 1 -- 172.21.15.104:0/213787652 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c0982c0 con 0x55c0848ba170 2024-06-09T22:27:47.536 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.531+0000 7feb64b84740 1 -- 172.21.15.104:0/213787652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 msgr2=0x55c0848ba550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.531+0000 7feb64b84740 1 --2- 172.21.15.104:0/213787652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 0x55c0848ba550 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7feb4c00c8e0 tx=0x7feb4c09b890 comp rx=0 tx=0).stop 2024-06-09T22:27:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.531+0000 7feb64b84740 1 -- 172.21.15.104:0/213787652 shutdown_connections 2024-06-09T22:27:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.531+0000 7feb64b84740 1 --2- 172.21.15.104:0/213787652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848c5350 0x55c0848c7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.531+0000 7feb64b84740 1 --2- 172.21.15.104:0/213787652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848baa90 0x55c0848c4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.531+0000 7feb64b84740 1 --2- 172.21.15.104:0/213787652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 0x55c0848ba550 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.531+0000 7feb64b84740 1 -- 172.21.15.104:0/213787652 >> 172.21.15.104:0/213787652 conn(0x55c0848b6680 msgr2=0x55c0848b6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.531+0000 7feb64b84740 1 -- 172.21.15.104:0/213787652 shutdown_connections 2024-06-09T22:27:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.531+0000 7feb64b84740 1 -- 172.21.15.104:0/213787652 wait complete. 2024-06-09T22:27:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.531+0000 7feb64b84740 1 -- start start 2024-06-09T22:27:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb64b84740 1 Processor -- start 2024-06-09T22:27:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb64b84740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 0x55c0848c23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb64b84740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848baa90 0x55c0848c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb64b84740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848c5350 0x55c0848c2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb64b84740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55c0848c7f80 con 0x55c0848ba170 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb64b84740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c0847a7ae0 con 0x55c0848c5350 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb64b84740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c0848c8100 con 0x55c0848baa90 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb64b84740 10 client.0 _ll_register_callbacks cb 0x55c0847e09b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 0x55c0848c23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 0x55c0848c23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50388/0 (socket says 172.21.15.104:50388) 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb5b7fe640 1 -- 172.21.15.104:0/246373116 learned_addr learned my addr 172.21.15.104:0/246373116 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb5affd640 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848baa90 0x55c0848c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb5bfff640 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848c5350 0x55c0848c2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb5b7fe640 1 -- 172.21.15.104:0/246373116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848baa90 msgr2=0x55c0848c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:47.538 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848baa90 0x55c0848c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb5b7fe640 1 -- 172.21.15.104:0/246373116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848c5350 msgr2=0x55c0848c2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848c5350 0x55c0848c2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.532+0000 7feb5b7fe640 1 -- 172.21.15.104:0/246373116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c084532210 con 0x55c0848ba170 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.533+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 0x55c0848c23e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0a4390 tx=0x7feb4c009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3198b1cff9218c4c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.533+0000 7feb3affd640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.533+0000 7feb3affd640 1 -- 172.21.15.104:0/246373116 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c0b1070 con 0x55c0848ba170 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.533+0000 7feb3affd640 1 -- 172.21.15.104:0/246373116 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7feb4c09b0a0 con 0x55c0848ba170 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.533+0000 7feb64b84740 1 -- 172.21.15.104:0/246373116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb4c0b4e50 con 0x55c0848ba170 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.533+0000 7feb3affd640 1 -- 172.21.15.104:0/246373116 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feb4c09bc50 con 0x55c0848ba170 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.533+0000 7feb64b84740 10 client.17397 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.533+0000 7feb64b84740 1 -- 172.21.15.104:0/246373116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c0845fe620 con 0x55c0848ba170 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.533+0000 7feb3affd640 1 -- 172.21.15.104:0/246373116 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb4c0b4e50 con 0x55c0848ba170 2024-06-09T22:27:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.534+0000 7feb3affd640 1 -- 172.21.15.104:0/246373116 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7feb4c0979a0 con 0x55c0848ba170 2024-06-09T22:27:47.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.534+0000 7feb64b84740 10 client.17397 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:47.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.534+0000 7feb64b84740 10 client.17397 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:47.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.534+0000 7feb64b84740 20 client.17397 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:47.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.534+0000 7feb64b84740 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c0848d5470 0x55c0848d7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.534+0000 7feb64b84740 4 client.17397 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:27:47.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.534+0000 7feb64b84740 1 -- 172.21.15.104:0/246373116 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c084831f10 con 0x55c0848d5470 2024-06-09T22:27:47.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.535+0000 7feb5affd640 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c0848d5470 0x55c0848d7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.535+0000 7feb3affd640 10 client.17397 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:47.540 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.535+0000 7feb5affd640 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c0848d5470 0x55c0848d7860 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3897bb3b00ac0d67 server_cookie=20fe236815951934 in_seq=0 out_seq=0 2024-06-09T22:27:47.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb3affd640 1 -- 172.21.15.104:0/246373116 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55c084831f10 con 0x55c0848d5470 2024-06-09T22:27:47.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb3affd640 10 client.17397 handle_command_reply: tid=0 2024-06-09T22:27:47.541 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb64b84740 1 client.17397 shutdown 2024-06-09T22:27:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb64b84740 1 -- 172.21.15.104:0/246373116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 msgr2=0x55c0848c23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb64b84740 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 0x55c0848c23e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0a4390 tx=0x7feb4c009620 comp rx=0 tx=0).stop 2024-06-09T22:27:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb64b84740 1 -- 172.21.15.104:0/246373116 shutdown_connections 2024-06-09T22:27:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb64b84740 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c0848d5470 0x55c0848d7860 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb64b84740 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848c5350 0x55c0848c2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb64b84740 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848baa90 0x55c0848c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb64b84740 1 --2- 172.21.15.104:0/246373116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848ba170 0x55c0848c23e0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb3affd640 0 client.17397 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb64b84740 1 -- 172.21.15.104:0/246373116 >> 172.21.15.104:0/246373116 conn(0x55c0848b6680 msgr2=0x55c0848bc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb64b84740 1 -- 172.21.15.104:0/246373116 shutdown_connections 2024-06-09T22:27:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb64b84740 1 -- 172.21.15.104:0/246373116 wait complete. 2024-06-09T22:27:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.536+0000 7feb64b84740 20 client.17397 trim_cache size 0 max 16384 2024-06-09T22:27:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.552+0000 7feb64b84740 1 Processor -- start 2024-06-09T22:27:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.552+0000 7feb64b84740 1 -- start start 2024-06-09T22:27:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.552+0000 7feb64b84740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848b6680 0x55c0848b1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.552+0000 7feb64b84740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848ba170 0x55c0848c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.552+0000 7feb64b84740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848baa90 0x55c0848c4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.552+0000 7feb64b84740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feb5c1aedc0 con 0x55c0848b6680 2024-06-09T22:27:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.552+0000 7feb64b84740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb5c1b0900 con 0x55c0848ba170 2024-06-09T22:27:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.552+0000 7feb64b84740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feb5c1af490 con 0x55c0848baa90 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.552+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848b6680 0x55c0848b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848b6680 0x55c0848b1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50404/0 (socket says 172.21.15.104:50404) 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb5b7fe640 1 -- 172.21.15.104:0/2446673385 learned_addr learned my addr 172.21.15.104:0/2446673385 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb5affd640 1 --2- 172.21.15.104:0/2446673385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848ba170 0x55c0848c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb5bfff640 1 --2- 172.21.15.104:0/2446673385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848baa90 0x55c0848c4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb5b7fe640 1 -- 172.21.15.104:0/2446673385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848baa90 msgr2=0x55c0848c4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/2446673385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848baa90 0x55c0848c4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb5b7fe640 1 -- 172.21.15.104:0/2446673385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848ba170 msgr2=0x55c0848c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/2446673385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848ba170 0x55c0848c1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb5b7fe640 1 -- 172.21.15.104:0/2446673385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0848b9f30 con 0x55c0848b6680 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/2446673385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848b6680 0x55c0848b1560 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0ba5b0 tx=0x7feb4c0cd100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb611cdc5bdb013a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb417fa640 1 -- 172.21.15.104:0/2446673385 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c0b1070 con 0x55c0848b6680 2024-06-09T22:27:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb417fa640 1 -- 172.21.15.104:0/2446673385 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb4c0b4e50 con 0x55c0848b6680 2024-06-09T22:27:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.553+0000 7feb417fa640 1 -- 172.21.15.104:0/2446673385 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c0a48c0 con 0x55c0848b6680 2024-06-09T22:27:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.554+0000 7feb64b84740 1 -- 172.21.15.104:0/2446673385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848b6680 msgr2=0x55c0848b1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.554+0000 7feb64b84740 1 --2- 172.21.15.104:0/2446673385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848b6680 0x55c0848b1560 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0ba5b0 tx=0x7feb4c0cd100 comp rx=0 tx=0).stop 2024-06-09T22:27:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.554+0000 7feb64b84740 1 -- 172.21.15.104:0/2446673385 shutdown_connections 2024-06-09T22:27:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.554+0000 7feb64b84740 1 --2- 172.21.15.104:0/2446673385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848baa90 0x55c0848c4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.554+0000 7feb64b84740 1 --2- 172.21.15.104:0/2446673385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848ba170 0x55c0848c1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.554+0000 7feb64b84740 1 --2- 172.21.15.104:0/2446673385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848b6680 0x55c0848b1560 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.554+0000 7feb64b84740 1 -- 172.21.15.104:0/2446673385 >> 172.21.15.104:0/2446673385 conn(0x55c0848bddb0 msgr2=0x55c0848c3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.554+0000 7feb64b84740 1 -- 172.21.15.104:0/2446673385 shutdown_connections 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.554+0000 7feb64b84740 1 -- 172.21.15.104:0/2446673385 wait complete. 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.554+0000 7feb64b84740 1 -- start start 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.554+0000 7feb64b84740 1 Processor -- start 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb64b84740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848b6680 0x55c0848daa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb64b84740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848ba170 0x55c0848d7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb64b84740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848baa90 0x55c0848d8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb64b84740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feb5c1b0230 con 0x55c0848baa90 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb64b84740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb5c1afb60 con 0x55c0848ba170 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb64b84740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feb5c1acd80 con 0x55c0848b6680 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb64b84740 10 client.0 _ll_register_callbacks cb 0x7feb540093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:27:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848b6680 0x55c0848daa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848baa90 0x55c0848d8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848b6680 0x55c0848daa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:33656/0 (socket says 172.21.15.104:33656) 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848baa90 0x55c0848d8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50414/0 (socket says 172.21.15.104:50414) 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5b7fe640 1 -- 172.21.15.104:0/1767130708 learned_addr learned my addr 172.21.15.104:0/1767130708 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5affd640 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848ba170 0x55c0848d7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5b7fe640 1 -- 172.21.15.104:0/1767130708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848ba170 msgr2=0x55c0848d7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848ba170 0x55c0848d7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5b7fe640 1 -- 172.21.15.104:0/1767130708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848baa90 msgr2=0x55c0848d8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848baa90 0x55c0848d8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5b7fe640 1 -- 172.21.15.104:0/1767130708 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0848d8980 con 0x55c0848b6680 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5bfff640 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848baa90 0x55c0848d8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb5b7fe640 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848b6680 0x55c0848daa70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7feb4c097df0 tx=0x7feb4c009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2cdb911f52f8065 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:27:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.555+0000 7feb3affd640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:27:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb3affd640 1 -- 172.21.15.104:0/1767130708 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c0b1070 con 0x55c0848b6680 2024-06-09T22:27:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb3affd640 1 -- 172.21.15.104:0/1767130708 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7feb4c09b2b0 con 0x55c0848b6680 2024-06-09T22:27:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb64b84740 1 -- 172.21.15.104:0/1767130708 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb4c0b4e50 con 0x55c0848b6680 2024-06-09T22:27:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb3affd640 1 -- 172.21.15.104:0/1767130708 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feb4c09b790 con 0x55c0848b6680 2024-06-09T22:27:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb64b84740 10 client.26734 fetch_fsmap learned FSMap version 32 2024-06-09T22:27:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb64b84740 1 -- 172.21.15.104:0/1767130708 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c0848d8980 con 0x55c0848b6680 2024-06-09T22:27:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb3affd640 1 -- 172.21.15.104:0/1767130708 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb4c0b4e50 con 0x55c0848b6680 2024-06-09T22:27:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb3affd640 1 -- 172.21.15.104:0/1767130708 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7feb4c0ac070 con 0x55c0848b6680 2024-06-09T22:27:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb64b84740 10 client.26734 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:27:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb64b84740 10 client.26734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:27:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb64b84740 20 client.26734 populate_metadata read hostname 'smithi104' 2024-06-09T22:27:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb64b84740 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c0848c5350 0x55c084941e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:27:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb64b84740 4 client.26734 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:27:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb64b84740 1 -- 172.21.15.104:0/1767130708 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c08482bc20 con 0x55c0848c5350 2024-06-09T22:27:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.556+0000 7feb5affd640 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c0848c5350 0x55c084941e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:27:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.557+0000 7feb3affd640 10 client.26734 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.557+0000 7feb5affd640 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c0848c5350 0x55c084941e90 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=337b7b7211ddbd0f server_cookie=40338271eaa511c2 in_seq=0 out_seq=0 2024-06-09T22:27:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb3affd640 1 -- 172.21.15.104:0/1767130708 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55c08482bc20 con 0x55c0848c5350 2024-06-09T22:27:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb3affd640 10 client.26734 handle_command_reply: tid=0 2024-06-09T22:27:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb64b84740 1 client.26734 shutdown 2024-06-09T22:27:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb64b84740 1 -- 172.21.15.104:0/1767130708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848b6680 msgr2=0x55c0848daa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb64b84740 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848b6680 0x55c0848daa70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7feb4c097df0 tx=0x7feb4c009620 comp rx=0 tx=0).stop 2024-06-09T22:27:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb64b84740 1 -- 172.21.15.104:0/1767130708 shutdown_connections 2024-06-09T22:27:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb64b84740 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c0848c5350 0x55c084941e90 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb64b84740 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c0848baa90 0x55c0848d8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb64b84740 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c0848ba170 0x55c0848d7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb64b84740 1 --2- 172.21.15.104:0/1767130708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0848b6680 0x55c0848daa70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb3affd640 0 client.26734 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:27:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb64b84740 1 -- 172.21.15.104:0/1767130708 >> 172.21.15.104:0/1767130708 conn(0x55c0848bddb0 msgr2=0x55c0848d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb64b84740 1 -- 172.21.15.104:0/1767130708 shutdown_connections 2024-06-09T22:27:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb64b84740 1 -- 172.21.15.104:0/1767130708 wait complete. 2024-06-09T22:27:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.565+0000 7feb64b84740 20 client.26734 trim_cache size 0 max 16384 2024-06-09T22:27:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:27:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:27:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:27:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.566+0000 7feb61e76640 1 -- 172.21.15.104:0/1089936119 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7feb440a2da0 msgr2=0x7feb440a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.566+0000 7feb61e76640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7feb440a2da0 0x7feb440a5260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feb5c10ddb0 tx=0x7feb500094d0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.566+0000 7feb61e76640 1 -- 172.21.15.104:0/1089936119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb5c102c30 msgr2=0x7feb5c1982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:27:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.566+0000 7feb61e76640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb5c102c30 0x7feb5c1982d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feb4c00e9c0 tx=0x7feb4c00ee80 comp rx=0 tx=0).stop 2024-06-09T22:27:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.566+0000 7feb61e76640 1 -- 172.21.15.104:0/1089936119 shutdown_connections 2024-06-09T22:27:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.566+0000 7feb61e76640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7feb440a2da0 0x7feb440a5260 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.566+0000 7feb61e76640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb5c10bf70 0x7feb5c19cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.566+0000 7feb61e76640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb5c103550 0x7feb5c198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.566+0000 7feb61e76640 1 --2- 172.21.15.104:0/1089936119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb5c102c30 0x7feb5c1982d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:27:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.566+0000 7feb61e76640 1 -- 172.21.15.104:0/1089936119 >> 172.21.15.104:0/1089936119 conn(0x7feb5c077a20 msgr2=0x7feb5c0fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:27:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.566+0000 7feb61e76640 1 -- 172.21.15.104:0/1089936119 shutdown_connections 2024-06-09T22:27:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:27:47.566+0000 7feb61e76640 1 -- 172.21.15.104:0/1089936119 wait complete. 2024-06-09T22:27:47.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 205543 4.17 MB/sec execute 173 sec latency 17.134 ms 2024-06-09T22:27:47.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:47 smithi153 ceph-mon[36599]: pgmap v207: 129 pgs: 129 active+clean; 144 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1022 KiB/s wr, 127 op/s 2024-06-09T22:27:48.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:47 smithi104 ceph-mon[32389]: pgmap v207: 129 pgs: 129 active+clean; 144 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1022 KiB/s wr, 127 op/s 2024-06-09T22:27:48.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:47 smithi194 ceph-mon[36856]: pgmap v207: 129 pgs: 129 active+clean; 144 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1022 KiB/s wr, 127 op/s 2024-06-09T22:27:48.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 206632 4.18 MB/sec execute 174 sec latency 73.069 ms 2024-06-09T22:27:49.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 208721 4.22 MB/sec execute 175 sec latency 18.545 ms 2024-06-09T22:27:49.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:49 smithi153 ceph-mon[36599]: pgmap v208: 129 pgs: 129 active+clean; 144 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1022 KiB/s wr, 127 op/s 2024-06-09T22:27:50.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:49 smithi104 ceph-mon[32389]: pgmap v208: 129 pgs: 129 active+clean; 144 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1022 KiB/s wr, 127 op/s 2024-06-09T22:27:50.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:49 smithi194 ceph-mon[36856]: pgmap v208: 129 pgs: 129 active+clean; 144 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1022 KiB/s wr, 127 op/s 2024-06-09T22:27:50.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 209485 4.23 MB/sec execute 176 sec latency 709.927 ms 2024-06-09T22:27:51.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 209485 4.21 MB/sec execute 177 sec latency 1710.051 ms 2024-06-09T22:27:51.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:51 smithi153 ceph-mon[36599]: pgmap v209: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-06-09T22:27:52.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:51 smithi104 ceph-mon[32389]: pgmap v209: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-06-09T22:27:52.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:51 smithi194 ceph-mon[36856]: pgmap v209: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-06-09T22:27:52.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 209485 4.19 MB/sec execute 178 sec latency 2710.148 ms 2024-06-09T22:27:53.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 209485 4.16 MB/sec execute 179 sec latency 3710.208 ms 2024-06-09T22:27:53.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:53 smithi153 ceph-mon[36599]: pgmap v210: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-06-09T22:27:54.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:53 smithi104 ceph-mon[32389]: pgmap v210: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-06-09T22:27:54.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:53 smithi194 ceph-mon[36856]: pgmap v210: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-06-09T22:27:54.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 209485 4.14 MB/sec execute 180 sec latency 4710.335 ms 2024-06-09T22:27:55.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 210150 4.12 MB/sec execute 181 sec latency 5006.780 ms 2024-06-09T22:27:55.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:55 smithi153 ceph-mon[36599]: pgmap v211: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.7 MiB/s wr, 200 op/s 2024-06-09T22:27:55.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:55 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:27:55.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:55 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:27:55.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:55 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:27:56.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:55 smithi104 ceph-mon[32389]: pgmap v211: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.7 MiB/s wr, 200 op/s 2024-06-09T22:27:56.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:55 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:27:56.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:55 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:27:56.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:55 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:27:56.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:55 smithi194 ceph-mon[36856]: pgmap v211: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.7 MiB/s wr, 200 op/s 2024-06-09T22:27:56.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:55 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:27:56.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:55 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:27:56.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:55 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:27:56.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 212472 4.17 MB/sec execute 182 sec latency 18.485 ms 2024-06-09T22:27:57.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 213792 4.18 MB/sec execute 183 sec latency 15.259 ms 2024-06-09T22:27:57.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:57 smithi153 ceph-mon[36599]: pgmap v212: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 929 KiB/s wr, 110 op/s 2024-06-09T22:27:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:57 smithi104 ceph-mon[32389]: pgmap v212: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 929 KiB/s wr, 110 op/s 2024-06-09T22:27:58.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:57 smithi194 ceph-mon[36856]: pgmap v212: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 929 KiB/s wr, 110 op/s 2024-06-09T22:27:58.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 216365 4.25 MB/sec execute 184 sec latency 20.011 ms 2024-06-09T22:27:59.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 217586 4.25 MB/sec execute 185 sec latency 14.349 ms 2024-06-09T22:27:59.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:27:59 smithi153 ceph-mon[36599]: pgmap v213: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 929 KiB/s wr, 110 op/s 2024-06-09T22:28:00.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:27:59 smithi104 ceph-mon[32389]: pgmap v213: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 929 KiB/s wr, 110 op/s 2024-06-09T22:28:00.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:27:59 smithi194 ceph-mon[36856]: pgmap v213: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 929 KiB/s wr, 110 op/s 2024-06-09T22:28:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.656+0000 7fd53214f640 1 -- 172.21.15.104:0/270937170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd52c10bf70 msgr2=0x7fd52c10e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.656+0000 7fd53214f640 1 --2- 172.21.15.104:0/270937170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd52c10bf70 0x7fd52c10e410 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fd524009ca0 tx=0x7fd52402a9e0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.656+0000 7fd53214f640 1 -- 172.21.15.104:0/270937170 shutdown_connections 2024-06-09T22:28:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.656+0000 7fd53214f640 1 --2- 172.21.15.104:0/270937170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd52c10bf70 0x7fd52c10e410 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.656+0000 7fd53214f640 1 --2- 172.21.15.104:0/270937170 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd52c103550 0x7fd52c10ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.656+0000 7fd53214f640 1 --2- 172.21.15.104:0/270937170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c102c30 0x7fd52c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.656+0000 7fd53214f640 1 -- 172.21.15.104:0/270937170 >> 172.21.15.104:0/270937170 conn(0x7fd52c077a20 msgr2=0x7fd52c077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.656+0000 7fd53214f640 1 -- 172.21.15.104:0/270937170 shutdown_connections 2024-06-09T22:28:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.657+0000 7fd53214f640 1 -- 172.21.15.104:0/270937170 wait complete. 2024-06-09T22:28:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.657+0000 7fd53214f640 1 Processor -- start 2024-06-09T22:28:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.657+0000 7fd53214f640 1 -- start start 2024-06-09T22:28:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.657+0000 7fd53214f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c102c30 0x7fd52c1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.657+0000 7fd53214f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd52c103550 0x7fd52c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.657+0000 7fd53214f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd52c10bf70 0x7fd52c19ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.657+0000 7fd53214f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c111dc0 con 0x7fd52c103550 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.657+0000 7fd53214f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c111c40 con 0x7fd52c102c30 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.657+0000 7fd53214f640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c111f40 con 0x7fd52c10bf70 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd52affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd52c103550 0x7fd52c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd52affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd52c103550 0x7fd52c1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55986/0 (socket says 172.21.15.104:55986) 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd52affd640 1 -- 172.21.15.104:0/3934217095 learned_addr learned my addr 172.21.15.104:0/3934217095 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd52b7fe640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c102c30 0x7fd52c1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd52bfff640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd52c10bf70 0x7fd52c19ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd52affd640 1 -- 172.21.15.104:0/3934217095 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd52c10bf70 msgr2=0x7fd52c19ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd52affd640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd52c10bf70 0x7fd52c19ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd52affd640 1 -- 172.21.15.104:0/3934217095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c102c30 msgr2=0x7fd52c1981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd52affd640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c102c30 0x7fd52c1981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd52affd640 1 -- 172.21.15.104:0/3934217095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52c19d190 con 0x7fd52c103550 2024-06-09T22:28:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd52affd640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd52c103550 0x7fd52c1986e0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fd52000e9f0 tx=0x7fd52000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd528ff9640 1 -- 172.21.15.104:0/3934217095 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52000cd80 con 0x7fd52c103550 2024-06-09T22:28:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd528ff9640 1 -- 172.21.15.104:0/3934217095 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd520005500 con 0x7fd52c103550 2024-06-09T22:28:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd53214f640 1 -- 172.21.15.104:0/3934217095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52c19d480 con 0x7fd52c103550 2024-06-09T22:28:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd528ff9640 1 -- 172.21.15.104:0/3934217095 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd520010690 con 0x7fd52c103550 2024-06-09T22:28:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.658+0000 7fd53214f640 1 -- 172.21.15.104:0/3934217095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd52c1a4d10 con 0x7fd52c103550 2024-06-09T22:28:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.661+0000 7fd528ff9640 1 -- 172.21.15.104:0/3934217095 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fd5200046c0 con 0x7fd52c103550 2024-06-09T22:28:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.661+0000 7fd534e5c740 1 Processor -- start 2024-06-09T22:28:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.661+0000 7fd534e5c740 1 -- start start 2024-06-09T22:28:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.661+0000 7fd534e5c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a7054550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.661+0000 7fd534e5c740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a705ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.661+0000 7fd534e5c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a705f350 0x55a3a7061740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.661+0000 7fd534e5c740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a3a6fb84f0 con 0x55a3a7054a90 2024-06-09T22:28:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.661+0000 7fd534e5c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a3a6f56870 con 0x55a3a705f350 2024-06-09T22:28:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.661+0000 7fd534e5c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a3a6fb44b0 con 0x55a3a7054170 2024-06-09T22:28:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a705ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52bfff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a7054550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a705ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56002/0 (socket says 172.21.15.104:56002) 2024-06-09T22:28:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52b7fe640 1 -- 172.21.15.104:0/3985983771 learned_addr learned my addr 172.21.15.104:0/3985983771 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52bfff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a7054550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:54626/0 (socket says 172.21.15.104:54626) 2024-06-09T22:28:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52affd640 1 --2- 172.21.15.104:0/3985983771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a705f350 0x55a3a7061740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd528ff9640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd5140a2da0 0x7fd5140a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52b7fe640 1 -- 172.21.15.104:0/3985983771 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 msgr2=0x55a3a7054550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52b7fe640 1 --2- 172.21.15.104:0/3985983771 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a7054550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52b7fe640 1 -- 172.21.15.104:0/3985983771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a705f350 msgr2=0x55a3a7061740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52b7fe640 1 --2- 172.21.15.104:0/3985983771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a705f350 0x55a3a7061740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52b7fe640 1 -- 172.21.15.104:0/3985983771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3a6f729b0 con 0x55a3a7054a90 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52bfff640 1 --2- 172.21.15.104:0/3985983771 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a7054550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52affd640 1 --2- 172.21.15.104:0/3985983771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a705f350 0x55a3a7061740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52bfff640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd5140a2da0 0x7fd5140a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd528ff9640 1 -- 172.21.15.104:0/3934217095 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fd5200cfb70 con 0x7fd52c103550 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52b7fe640 1 --2- 172.21.15.104:0/3985983771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a705ee10 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009cf0 tx=0x7fd51c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc7e42191871f0b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd511ffb640 1 -- 172.21.15.104:0/3985983771 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c004230 con 0x55a3a7054a90 2024-06-09T22:28:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd52bfff640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd5140a2da0 0x7fd5140a5260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd524006010 tx=0x7fd5240095c0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd511ffb640 1 -- 172.21.15.104:0/3985983771 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd51c0043d0 con 0x55a3a7054a90 2024-06-09T22:28:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.662+0000 7fd511ffb640 1 -- 172.21.15.104:0/3985983771 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c0198c0 con 0x55a3a7054a90 2024-06-09T22:28:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.663+0000 7fd534e5c740 1 -- 172.21.15.104:0/3985983771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 msgr2=0x55a3a705ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.663+0000 7fd534e5c740 1 --2- 172.21.15.104:0/3985983771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a705ee10 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009cf0 tx=0x7fd51c011810 comp rx=0 tx=0).stop 2024-06-09T22:28:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.663+0000 7fd534e5c740 1 -- 172.21.15.104:0/3985983771 shutdown_connections 2024-06-09T22:28:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.663+0000 7fd534e5c740 1 --2- 172.21.15.104:0/3985983771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a705f350 0x55a3a7061740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.663+0000 7fd534e5c740 1 --2- 172.21.15.104:0/3985983771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a705ee10 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.663+0000 7fd534e5c740 1 --2- 172.21.15.104:0/3985983771 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a7054550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.663+0000 7fd534e5c740 1 -- 172.21.15.104:0/3985983771 >> 172.21.15.104:0/3985983771 conn(0x55a3a7050680 msgr2=0x55a3a7050a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.663+0000 7fd534e5c740 1 -- 172.21.15.104:0/3985983771 shutdown_connections 2024-06-09T22:28:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.663+0000 7fd534e5c740 1 -- 172.21.15.104:0/3985983771 wait complete. 2024-06-09T22:28:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd534e5c740 1 -- start start 2024-06-09T22:28:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd534e5c740 1 Processor -- start 2024-06-09T22:28:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd534e5c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a705c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd534e5c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054a90 0x55a3a705c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd534e5c740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a705f350 0x55a3a705ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd534e5c740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a3a7061f80 con 0x55a3a705f350 2024-06-09T22:28:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd534e5c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a3a6f41ae0 con 0x55a3a7054a90 2024-06-09T22:28:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd534e5c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a3a7062100 con 0x55a3a7054170 2024-06-09T22:28:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd534e5c740 10 client.0 _ll_register_callbacks cb 0x55a3a6f7a9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd52bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a705f350 0x55a3a705ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd52b7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a705c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd52bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a705f350 0x55a3a705ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56014/0 (socket says 172.21.15.104:56014) 2024-06-09T22:28:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd52bfff640 1 -- 172.21.15.104:0/513184166 learned_addr learned my addr 172.21.15.104:0/513184166 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd52b7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a705c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:54638/0 (socket says 172.21.15.104:54638) 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.664+0000 7fd52affd640 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054a90 0x55a3a705c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd52b7fe640 1 -- 172.21.15.104:0/513184166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054a90 msgr2=0x55a3a705c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd52b7fe640 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054a90 0x55a3a705c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd52b7fe640 1 -- 172.21.15.104:0/513184166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a705f350 msgr2=0x55a3a705ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd52b7fe640 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a705f350 0x55a3a705ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd52b7fe640 1 -- 172.21.15.104:0/513184166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3a6ccc210 con 0x55a3a7054170 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd52bfff640 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a705f350 0x55a3a705ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd52affd640 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054a90 0x55a3a705c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd52b7fe640 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a705c240 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009cf0 tx=0x7fd51c0045f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e66ac57d3e14776 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd50affd640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd50affd640 1 -- 172.21.15.104:0/513184166 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c027070 con 0x55a3a7054170 2024-06-09T22:28:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd534e5c740 1 -- 172.21.15.104:0/513184166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd51c0043d0 con 0x55a3a7054170 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd50affd640 1 -- 172.21.15.104:0/513184166 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fd51c0037f0 con 0x55a3a7054170 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd50affd640 1 -- 172.21.15.104:0/513184166 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd51c033050 con 0x55a3a7054170 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd534e5c740 10 client.26746 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd534e5c740 1 -- 172.21.15.104:0/513184166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a3a6d98620 con 0x55a3a7054170 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.665+0000 7fd50affd640 1 -- 172.21.15.104:0/513184166 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd51c0043d0 con 0x55a3a7054170 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.666+0000 7fd50affd640 1 -- 172.21.15.104:0/513184166 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd51c004ba0 con 0x55a3a7054170 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.666+0000 7fd534e5c740 10 client.26746 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.666+0000 7fd534e5c740 10 client.26746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.666+0000 7fd534e5c740 20 client.26746 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.666+0000 7fd534e5c740 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3a706f3e0 0x55a3a70717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.666+0000 7fd534e5c740 4 client.26746 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.666+0000 7fd534e5c740 1 -- 172.21.15.104:0/513184166 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3a6fcbf10 con 0x55a3a706f3e0 2024-06-09T22:28:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.666+0000 7fd52affd640 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3a706f3e0 0x55a3a70717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.666+0000 7fd50affd640 10 client.26746 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.666+0000 7fd52affd640 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3a706f3e0 0x55a3a70717d0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9aa8834af01ee15 server_cookie=c862700ecbcf2db9 in_seq=0 out_seq=0 2024-06-09T22:28:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd50affd640 1 -- 172.21.15.104:0/513184166 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a3a6fcbf10 con 0x55a3a706f3e0 2024-06-09T22:28:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd50affd640 10 client.26746 handle_command_reply: tid=0 2024-06-09T22:28:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd534e5c740 1 client.26746 shutdown 2024-06-09T22:28:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd534e5c740 1 -- 172.21.15.104:0/513184166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 msgr2=0x55a3a705c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd534e5c740 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a705c240 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009cf0 tx=0x7fd51c0045f0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd534e5c740 1 -- 172.21.15.104:0/513184166 shutdown_connections 2024-06-09T22:28:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd534e5c740 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3a706f3e0 0x55a3a70717d0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd534e5c740 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a705f350 0x55a3a705ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd534e5c740 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054a90 0x55a3a705c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd534e5c740 1 --2- 172.21.15.104:0/513184166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a705c240 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd50affd640 0 client.26746 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd534e5c740 1 -- 172.21.15.104:0/513184166 >> 172.21.15.104:0/513184166 conn(0x55a3a7050680 msgr2=0x55a3a70563b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd534e5c740 1 -- 172.21.15.104:0/513184166 shutdown_connections 2024-06-09T22:28:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd534e5c740 1 -- 172.21.15.104:0/513184166 wait complete. 2024-06-09T22:28:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.667+0000 7fd534e5c740 20 client.26746 trim_cache size 0 max 16384 2024-06-09T22:28:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd534e5c740 1 Processor -- start 2024-06-09T22:28:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd534e5c740 1 -- start start 2024-06-09T22:28:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd534e5c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a705bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd534e5c740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a7056af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd534e5c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a705f350 0x55a3a705c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd534e5c740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1acbd0 con 0x55a3a7054a90 2024-06-09T22:28:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd534e5c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1ae710 con 0x55a3a705f350 2024-06-09T22:28:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd534e5c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1ad2a0 con 0x55a3a7054170 2024-06-09T22:28:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd52affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a7056af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd52b7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a705bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd52affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a7056af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56030/0 (socket says 172.21.15.104:56030) 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd52b7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a705bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:54648/0 (socket says 172.21.15.104:54648) 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd52affd640 1 -- 172.21.15.104:0/3879605662 learned_addr learned my addr 172.21.15.104:0/3879605662 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.684+0000 7fd52bfff640 1 --2- 172.21.15.104:0/3879605662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a705f350 0x55a3a705c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.685+0000 7fd52affd640 1 -- 172.21.15.104:0/3879605662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 msgr2=0x55a3a705bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.685+0000 7fd52affd640 1 --2- 172.21.15.104:0/3879605662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a705bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.685+0000 7fd52affd640 1 -- 172.21.15.104:0/3879605662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a705f350 msgr2=0x55a3a705c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.685+0000 7fd52affd640 1 --2- 172.21.15.104:0/3879605662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a705f350 0x55a3a705c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.685+0000 7fd52affd640 1 -- 172.21.15.104:0/3879605662 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3a706b1e0 con 0x55a3a7054a90 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.685+0000 7fd52b7fe640 1 --2- 172.21.15.104:0/3879605662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a705bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.685+0000 7fd52affd640 1 --2- 172.21.15.104:0/3879605662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a7056af0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd52009ba20 tx=0x7fd520004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be40c72024a2013d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.685+0000 7fd5117fa640 1 -- 172.21.15.104:0/3879605662 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52009baf0 con 0x55a3a7054a90 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.685+0000 7fd5117fa640 1 -- 172.21.15.104:0/3879605662 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd52009c420 con 0x55a3a7054a90 2024-06-09T22:28:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.685+0000 7fd5117fa640 1 -- 172.21.15.104:0/3879605662 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52009b680 con 0x55a3a7054a90 2024-06-09T22:28:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 -- 172.21.15.104:0/3879605662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 msgr2=0x55a3a7056af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 --2- 172.21.15.104:0/3879605662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a7056af0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd52009ba20 tx=0x7fd520004a40 comp rx=0 tx=0).stop 2024-06-09T22:28:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 -- 172.21.15.104:0/3879605662 shutdown_connections 2024-06-09T22:28:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 --2- 172.21.15.104:0/3879605662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a705f350 0x55a3a705c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 --2- 172.21.15.104:0/3879605662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a7056af0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 --2- 172.21.15.104:0/3879605662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a7054170 0x55a3a705bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 -- 172.21.15.104:0/3879605662 >> 172.21.15.104:0/3879605662 conn(0x55a3a7050680 msgr2=0x55a3a705c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 -- 172.21.15.104:0/3879605662 shutdown_connections 2024-06-09T22:28:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 -- 172.21.15.104:0/3879605662 wait complete. 2024-06-09T22:28:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 -- start start 2024-06-09T22:28:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 Processor -- start 2024-06-09T22:28:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054170 0x55a3a7062490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a70629d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a705f350 0x55a3a70d7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1ae040 con 0x55a3a7054a90 2024-06-09T22:28:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1ad970 con 0x55a3a7054170 2024-06-09T22:28:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1aab90 con 0x55a3a705f350 2024-06-09T22:28:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.686+0000 7fd534e5c740 10 client.0 _ll_register_callbacks cb 0x7fd5200d3530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd52affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a70629d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd52b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054170 0x55a3a7062490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd52b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054170 0x55a3a7062490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:51720/0 (socket says 172.21.15.104:51720) 2024-06-09T22:28:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd52b7fe640 1 -- 172.21.15.104:0/4061117804 learned_addr learned my addr 172.21.15.104:0/4061117804 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd52bfff640 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a705f350 0x55a3a70d7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd52b7fe640 1 -- 172.21.15.104:0/4061117804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a705f350 msgr2=0x55a3a70d7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd52b7fe640 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a705f350 0x55a3a70d7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd52b7fe640 1 -- 172.21.15.104:0/4061117804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 msgr2=0x55a3a70629d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd52b7fe640 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a70629d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd52b7fe640 1 -- 172.21.15.104:0/4061117804 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3a7063260 con 0x55a3a7054170 2024-06-09T22:28:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd52affd640 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a70629d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd52b7fe640 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054170 0x55a3a7062490 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0194b0 tx=0x7fd51c009f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1368eeca4e86fb99 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd50affd640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd50affd640 1 -- 172.21.15.104:0/4061117804 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c03c060 con 0x55a3a7054170 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd50affd640 1 -- 172.21.15.104:0/4061117804 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9408+0+0 (secure 0 0 0) 0x7fd51c0035f0 con 0x55a3a7054170 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd534e5c740 1 -- 172.21.15.104:0/4061117804 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd52009c420 con 0x55a3a7054170 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd50affd640 1 -- 172.21.15.104:0/4061117804 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd51c043050 con 0x55a3a7054170 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd534e5c740 10 client.26657 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.687+0000 7fd534e5c740 1 -- 172.21.15.104:0/4061117804 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a3a70d94f0 con 0x55a3a7054170 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.688+0000 7fd50affd640 1 -- 172.21.15.104:0/4061117804 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd52009c420 con 0x55a3a7054170 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.688+0000 7fd50affd640 1 -- 172.21.15.104:0/4061117804 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd51c004660 con 0x55a3a7054170 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.688+0000 7fd534e5c740 10 client.26657 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.688+0000 7fd534e5c740 10 client.26657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.688+0000 7fd534e5c740 20 client.26657 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.688+0000 7fd534e5c740 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3a70d9b50 0x55a3a70dbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.688+0000 7fd534e5c740 4 client.26657 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:28:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.688+0000 7fd534e5c740 1 -- 172.21.15.104:0/4061117804 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a3a6fc5c20 con 0x55a3a70d9b50 2024-06-09T22:28:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.688+0000 7fd52affd640 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3a70d9b50 0x55a3a70dbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.688+0000 7fd52affd640 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3a70d9b50 0x55a3a70dbf70 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9eb405487ec00eca server_cookie=2c987ae7eaf8f01b in_seq=0 out_seq=0 2024-06-09T22:28:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.688+0000 7fd50affd640 10 client.26657 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd50affd640 1 -- 172.21.15.104:0/4061117804 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a3a6fc5c20 con 0x55a3a70d9b50 2024-06-09T22:28:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd50affd640 10 client.26657 handle_command_reply: tid=0 2024-06-09T22:28:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd534e5c740 1 client.26657 shutdown 2024-06-09T22:28:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd534e5c740 1 -- 172.21.15.104:0/4061117804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054170 msgr2=0x55a3a7062490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd534e5c740 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054170 0x55a3a7062490 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0194b0 tx=0x7fd51c009f10 comp rx=0 tx=0).stop 2024-06-09T22:28:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd534e5c740 1 -- 172.21.15.104:0/4061117804 shutdown_connections 2024-06-09T22:28:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd534e5c740 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3a70d9b50 0x55a3a70dbf70 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd534e5c740 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3a705f350 0x55a3a70d7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd534e5c740 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3a7054a90 0x55a3a70629d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd50affd640 0 client.26657 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd534e5c740 1 --2- 172.21.15.104:0/4061117804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3a7054170 0x55a3a7062490 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd534e5c740 1 -- 172.21.15.104:0/4061117804 >> 172.21.15.104:0/4061117804 conn(0x55a3a7050680 msgr2=0x55a3a706aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd534e5c740 1 -- 172.21.15.104:0/4061117804 shutdown_connections 2024-06-09T22:28:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd534e5c740 1 -- 172.21.15.104:0/4061117804 wait complete. 2024-06-09T22:28:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.697+0000 7fd534e5c740 20 client.26657 trim_cache size 0 max 16384 2024-06-09T22:28:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:28:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:28:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:28:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.698+0000 7fd53214f640 1 -- 172.21.15.104:0/3934217095 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd5140a2da0 msgr2=0x7fd5140a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.698+0000 7fd53214f640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd5140a2da0 0x7fd5140a5260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd524006010 tx=0x7fd5240095c0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.698+0000 7fd53214f640 1 -- 172.21.15.104:0/3934217095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd52c103550 msgr2=0x7fd52c1986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.698+0000 7fd53214f640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd52c103550 0x7fd52c1986e0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fd52000e9f0 tx=0x7fd52000edf0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.698+0000 7fd53214f640 1 -- 172.21.15.104:0/3934217095 shutdown_connections 2024-06-09T22:28:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.698+0000 7fd53214f640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd5140a2da0 0x7fd5140a5260 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.698+0000 7fd53214f640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd52c10bf70 0x7fd52c19ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.698+0000 7fd53214f640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd52c103550 0x7fd52c1986e0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.698+0000 7fd53214f640 1 --2- 172.21.15.104:0/3934217095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c102c30 0x7fd52c1981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.698+0000 7fd53214f640 1 -- 172.21.15.104:0/3934217095 >> 172.21.15.104:0/3934217095 conn(0x7fd52c077a20 msgr2=0x7fd52c0fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.698+0000 7fd53214f640 1 -- 172.21.15.104:0/3934217095 shutdown_connections 2024-06-09T22:28:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:00.698+0000 7fd53214f640 1 -- 172.21.15.104:0/3934217095 wait complete. 2024-06-09T22:28:00.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 219933 4.31 MB/sec execute 186 sec latency 29.571 ms 2024-06-09T22:28:01.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 220973 4.30 MB/sec execute 187 sec latency 24.428 ms 2024-06-09T22:28:01.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:01 smithi153 ceph-mon[36599]: pgmap v214: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.4 MiB/s wr, 168 op/s 2024-06-09T22:28:02.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:01 smithi104 ceph-mon[32389]: pgmap v214: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.4 MiB/s wr, 168 op/s 2024-06-09T22:28:02.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:01 smithi194 ceph-mon[36856]: pgmap v214: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.4 MiB/s wr, 168 op/s 2024-06-09T22:28:02.736 INFO:tasks.workunit.client.0.smithi104.stdout: 1 221574 4.30 MB/sec execute 188 sec latency 61.137 ms 2024-06-09T22:28:03.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 223739 4.35 MB/sec execute 189 sec latency 20.674 ms 2024-06-09T22:28:03.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:03 smithi153 ceph-mon[36599]: pgmap v215: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 719 KiB/s wr, 85 op/s 2024-06-09T22:28:04.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:03 smithi104 ceph-mon[32389]: pgmap v215: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 719 KiB/s wr, 85 op/s 2024-06-09T22:28:04.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:03 smithi194 ceph-mon[36856]: pgmap v215: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 719 KiB/s wr, 85 op/s 2024-06-09T22:28:04.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 225043 4.35 MB/sec execute 190 sec latency 18.028 ms 2024-06-09T22:28:05.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 227215 4.40 MB/sec execute 191 sec latency 24.047 ms 2024-06-09T22:28:05.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:05 smithi153 ceph-mon[36599]: pgmap v216: 129 pgs: 129 active+clean; 165 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-06-09T22:28:06.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:05 smithi104 ceph-mon[32389]: pgmap v216: 129 pgs: 129 active+clean; 165 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-06-09T22:28:06.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:05 smithi194 ceph-mon[36856]: pgmap v216: 129 pgs: 129 active+clean; 165 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-06-09T22:28:06.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 228491 4.41 MB/sec execute 192 sec latency 18.411 ms 2024-06-09T22:28:06.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:06 smithi153 ceph-mon[36599]: osdmap e101: 12 total, 12 up, 12 in 2024-06-09T22:28:06.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:06 smithi153 ceph-mon[36599]: pgmap v218: 129 pgs: 129 active+clean; 165 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2024-06-09T22:28:07.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:06 smithi104 ceph-mon[32389]: osdmap e101: 12 total, 12 up, 12 in 2024-06-09T22:28:07.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:06 smithi104 ceph-mon[32389]: pgmap v218: 129 pgs: 129 active+clean; 165 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2024-06-09T22:28:07.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:06 smithi194 ceph-mon[36856]: osdmap e101: 12 total, 12 up, 12 in 2024-06-09T22:28:07.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:06 smithi194 ceph-mon[36856]: pgmap v218: 129 pgs: 129 active+clean; 165 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2024-06-09T22:28:07.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 230688 4.45 MB/sec execute 193 sec latency 20.601 ms 2024-06-09T22:28:08.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 232005 4.46 MB/sec execute 194 sec latency 20.731 ms 2024-06-09T22:28:09.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:08 smithi194 ceph-mon[36856]: pgmap v219: 129 pgs: 129 active+clean; 165 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2024-06-09T22:28:09.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:08 smithi153 ceph-mon[36599]: pgmap v219: 129 pgs: 129 active+clean; 165 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2024-06-09T22:28:09.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:08 smithi104 ceph-mon[32389]: pgmap v219: 129 pgs: 129 active+clean; 165 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2024-06-09T22:28:09.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 234287 4.50 MB/sec execute 195 sec latency 26.558 ms 2024-06-09T22:28:10.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 235595 4.51 MB/sec execute 196 sec latency 16.075 ms 2024-06-09T22:28:11.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:10 smithi153 ceph-mon[36599]: pgmap v220: 129 pgs: 129 active+clean; 177 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 229 op/s 2024-06-09T22:28:11.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:10 smithi194 ceph-mon[36856]: pgmap v220: 129 pgs: 129 active+clean; 177 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 229 op/s 2024-06-09T22:28:11.258 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:10 smithi104 ceph-mon[32389]: pgmap v220: 129 pgs: 129 active+clean; 177 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 229 op/s 2024-06-09T22:28:11.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 237891 4.56 MB/sec execute 197 sec latency 26.925 ms 2024-06-09T22:28:12.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 239149 4.56 MB/sec execute 198 sec latency 16.786 ms 2024-06-09T22:28:13.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 240705 4.58 MB/sec execute 199 sec latency 231.920 ms 2024-06-09T22:28:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.792+0000 7f499cb48640 1 -- 172.21.15.104:0/3333527811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f499810bf70 msgr2=0x7f499810e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.792+0000 7f499cb48640 1 --2- 172.21.15.104:0/3333527811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f499810bf70 0x7f499810e410 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f4990009ca0 tx=0x7f499002a9e0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.792+0000 7f499cb48640 1 -- 172.21.15.104:0/3333527811 shutdown_connections 2024-06-09T22:28:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.792+0000 7f499cb48640 1 --2- 172.21.15.104:0/3333527811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f499810bf70 0x7f499810e410 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.792+0000 7f499cb48640 1 --2- 172.21.15.104:0/3333527811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4998103550 0x7f499810ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.792+0000 7f499cb48640 1 --2- 172.21.15.104:0/3333527811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4998102c30 0x7f4998103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.792+0000 7f499cb48640 1 -- 172.21.15.104:0/3333527811 >> 172.21.15.104:0/3333527811 conn(0x7f4998077a20 msgr2=0x7f4998077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.792+0000 7f499cb48640 1 -- 172.21.15.104:0/3333527811 shutdown_connections 2024-06-09T22:28:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.792+0000 7f499cb48640 1 -- 172.21.15.104:0/3333527811 wait complete. 2024-06-09T22:28:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f499cb48640 1 Processor -- start 2024-06-09T22:28:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f499cb48640 1 -- start start 2024-06-09T22:28:13.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f499cb48640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4998102c30 0x7f49981981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f499cb48640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4998103550 0x7f49981986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f499cb48640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f499810bf70 0x7f499819ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f499cb48640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4998111d70 con 0x7f499810bf70 2024-06-09T22:28:13.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f499cb48640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4998111bf0 con 0x7f4998103550 2024-06-09T22:28:13.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f499cb48640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4998111ef0 con 0x7f4998102c30 2024-06-09T22:28:13.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f4996d9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f499810bf70 0x7f499819ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f4995d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4998103550 0x7f49981986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.797 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f499659c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4998102c30 0x7f49981981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f4995d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4998103550 0x7f49981986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:57130/0 (socket says 172.21.15.104:57130) 2024-06-09T22:28:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f499659c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4998102c30 0x7f49981981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:45130/0 (socket says 172.21.15.104:45130) 2024-06-09T22:28:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f4995d9b640 1 -- 172.21.15.104:0/225065007 learned_addr learned my addr 172.21.15.104:0/225065007 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f4996d9d640 1 -- 172.21.15.104:0/225065007 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4998102c30 msgr2=0x7f49981981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f4996d9d640 1 --2- 172.21.15.104:0/225065007 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4998102c30 0x7f49981981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f4996d9d640 1 -- 172.21.15.104:0/225065007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4998103550 msgr2=0x7f49981986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.793+0000 7f4996d9d640 1 --2- 172.21.15.104:0/225065007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4998103550 0x7f49981986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.794+0000 7f4996d9d640 1 -- 172.21.15.104:0/225065007 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499819d1a0 con 0x7f499810bf70 2024-06-09T22:28:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.794+0000 7f4996d9d640 1 --2- 172.21.15.104:0/225065007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f499810bf70 0x7f499819ca70 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f4990005cc0 tx=0x7f4990003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.794+0000 7f49877fe640 1 -- 172.21.15.104:0/225065007 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4990047070 con 0x7f499810bf70 2024-06-09T22:28:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.794+0000 7f499cb48640 1 -- 172.21.15.104:0/225065007 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f499819d430 con 0x7f499810bf70 2024-06-09T22:28:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.794+0000 7f49877fe640 1 -- 172.21.15.104:0/225065007 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f499002fb20 con 0x7f499810bf70 2024-06-09T22:28:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.794+0000 7f49877fe640 1 -- 172.21.15.104:0/225065007 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4990033050 con 0x7f499810bf70 2024-06-09T22:28:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.794+0000 7f499cb48640 1 -- 172.21.15.104:0/225065007 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49981a4d20 con 0x7f499810bf70 2024-06-09T22:28:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.796+0000 7f49877fe640 1 -- 172.21.15.104:0/225065007 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f499004a070 con 0x7f499810bf70 2024-06-09T22:28:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.797+0000 7f499f84c740 1 Processor -- start 2024-06-09T22:28:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.797+0000 7f499f84c740 1 -- start start 2024-06-09T22:28:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.797+0000 7f499f84c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377df550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.797+0000 7f499f84c740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377e9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.797+0000 7f499f84c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x5593377ec740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.797+0000 7f499f84c740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5593377434f0 con 0x5593377dfa90 2024-06-09T22:28:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.797+0000 7f499f84c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5593376e1870 con 0x5593377ea350 2024-06-09T22:28:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.797+0000 7f499f84c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55933773f4b0 con 0x5593377df170 2024-06-09T22:28:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.797+0000 7f499659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377e9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.797+0000 7f4995d9b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377df550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f499659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377e9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60902/0 (socket says 172.21.15.104:60902) 2024-06-09T22:28:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f499659c640 1 -- 172.21.15.104:0/3348408342 learned_addr learned my addr 172.21.15.104:0/3348408342 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f49877fe640 1 --2- 172.21.15.104:0/225065007 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f49800a2da0 0x7f49800a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f4995d9b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377df550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:45136/0 (socket says 172.21.15.104:45136) 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f4995d9b640 1 --2- 172.21.15.104:0/3348408342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x5593377ec740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f4996d9d640 1 --2- 172.21.15.104:0/225065007 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f49800a2da0 0x7f49800a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f499659c640 1 -- 172.21.15.104:0/3348408342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 msgr2=0x5593377df550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f499659c640 1 --2- 172.21.15.104:0/3348408342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377df550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f499659c640 1 -- 172.21.15.104:0/3348408342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 msgr2=0x5593377ec740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f499659c640 1 --2- 172.21.15.104:0/3348408342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x5593377ec740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f499659c640 1 -- 172.21.15.104:0/3348408342 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593376fd9b0 con 0x5593377dfa90 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f49877fe640 1 -- 172.21.15.104:0/225065007 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f49900f0300 con 0x7f499810bf70 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f4995d9b640 1 --2- 172.21.15.104:0/3348408342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377df550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f499659c640 1 --2- 172.21.15.104:0/3348408342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377e9e10 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f498800ac20 tx=0x7f498800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28fae7d63d2a3331 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f4984ff9640 1 -- 172.21.15.104:0/3348408342 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498801d070 con 0x5593377dfa90 2024-06-09T22:28:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f4996d9d640 1 --2- 172.21.15.104:0/225065007 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f49800a2da0 0x7f49800a5260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f49900edc80 tx=0x7f49900093d0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f4984ff9640 1 -- 172.21.15.104:0/3348408342 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4988018b20 con 0x5593377dfa90 2024-06-09T22:28:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.798+0000 7f4984ff9640 1 -- 172.21.15.104:0/3348408342 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4988010b90 con 0x5593377dfa90 2024-06-09T22:28:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.799+0000 7f499f84c740 1 -- 172.21.15.104:0/3348408342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 msgr2=0x5593377e9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.799+0000 7f499f84c740 1 --2- 172.21.15.104:0/3348408342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377e9e10 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f498800ac20 tx=0x7f498800f790 comp rx=0 tx=0).stop 2024-06-09T22:28:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.799+0000 7f499f84c740 1 -- 172.21.15.104:0/3348408342 shutdown_connections 2024-06-09T22:28:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.799+0000 7f499f84c740 1 --2- 172.21.15.104:0/3348408342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x5593377ec740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.799+0000 7f499f84c740 1 --2- 172.21.15.104:0/3348408342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377e9e10 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.799+0000 7f499f84c740 1 --2- 172.21.15.104:0/3348408342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377df550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.799+0000 7f499f84c740 1 -- 172.21.15.104:0/3348408342 >> 172.21.15.104:0/3348408342 conn(0x5593377db680 msgr2=0x5593377dba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.799+0000 7f499f84c740 1 -- 172.21.15.104:0/3348408342 shutdown_connections 2024-06-09T22:28:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.799+0000 7f499f84c740 1 -- 172.21.15.104:0/3348408342 wait complete. 2024-06-09T22:28:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f499f84c740 1 -- start start 2024-06-09T22:28:13.805 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f499f84c740 1 Processor -- start 2024-06-09T22:28:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f499f84c740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377df170 0x5593377e72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f499f84c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377dfa90 0x5593377e77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f499f84c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x5593377e7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f499f84c740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5593377ecf80 con 0x5593377df170 2024-06-09T22:28:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f499f84c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5593376ccae0 con 0x5593377ea350 2024-06-09T22:28:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f499f84c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5593377ed100 con 0x5593377dfa90 2024-06-09T22:28:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f499f84c740 10 client.0 _ll_register_callbacks cb 0x5593377059b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f4995d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377df170 0x5593377e72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f4995d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377df170 0x5593377e72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60908/0 (socket says 172.21.15.104:60908) 2024-06-09T22:28:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f4995d9b640 1 -- 172.21.15.104:0/1067085052 learned_addr learned my addr 172.21.15.104:0/1067085052 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f499659c640 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377dfa90 0x5593377e77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.800+0000 7f4995d9b640 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x5593377e7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f4995d9b640 1 -- 172.21.15.104:0/1067085052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377dfa90 msgr2=0x5593377e77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f4995d9b640 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377dfa90 0x5593377e77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f4995d9b640 1 -- 172.21.15.104:0/1067085052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 msgr2=0x5593377e7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f4995d9b640 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x5593377e7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f4995d9b640 1 -- 172.21.15.104:0/1067085052 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559337457210 con 0x5593377df170 2024-06-09T22:28:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f499659c640 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377dfa90 0x5593377e77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f4995d9b640 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377df170 0x5593377e72b0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f498c011bf0 tx=0x7f498c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d366db71be0a2e4a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f497dffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f497dffb640 1 -- 172.21.15.104:0/1067085052 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c00fd10 con 0x5593377df170 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f497dffb640 1 -- 172.21.15.104:0/1067085052 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f498c005020 con 0x5593377df170 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f499f84c740 1 -- 172.21.15.104:0/1067085052 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4988018b20 con 0x5593377df170 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f497dffb640 1 -- 172.21.15.104:0/1067085052 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f498c005220 con 0x5593377df170 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f499f84c740 10 client.17433 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.801+0000 7f499f84c740 1 -- 172.21.15.104:0/1067085052 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559337523620 con 0x5593377df170 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.802+0000 7f497dffb640 1 -- 172.21.15.104:0/1067085052 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4988018b20 con 0x5593377df170 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.802+0000 7f497dffb640 1 -- 172.21.15.104:0/1067085052 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f498c022470 con 0x5593377df170 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.802+0000 7f499f84c740 10 client.17433 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.802+0000 7f499f84c740 10 client.17433 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.802+0000 7f499f84c740 20 client.17433 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.802+0000 7f499f84c740 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5593377fa560 0x5593377fc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.802+0000 7f499f84c740 4 client.17433 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.802+0000 7f499f84c740 1 -- 172.21.15.104:0/1067085052 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559337756f10 con 0x5593377fa560 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.802+0000 7f499659c640 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5593377fa560 0x5593377fc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.802+0000 7f497dffb640 10 client.17433 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.802+0000 7f499659c640 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5593377fa560 0x5593377fc950 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=743d7e2911573254 server_cookie=9e1615e37f982bf6 in_seq=0 out_seq=0 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.803+0000 7f497dffb640 1 -- 172.21.15.104:0/1067085052 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x559337756f10 con 0x5593377fa560 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.803+0000 7f497dffb640 10 client.17433 handle_command_reply: tid=0 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.803+0000 7f499f84c740 1 client.17433 shutdown 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.804+0000 7f499f84c740 1 -- 172.21.15.104:0/1067085052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377df170 msgr2=0x5593377e72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.804+0000 7f499f84c740 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377df170 0x5593377e72b0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f498c011bf0 tx=0x7f498c004730 comp rx=0 tx=0).stop 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.804+0000 7f499f84c740 1 -- 172.21.15.104:0/1067085052 shutdown_connections 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.804+0000 7f499f84c740 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5593377fa560 0x5593377fc950 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.804+0000 7f499f84c740 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x5593377e7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.804+0000 7f499f84c740 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377dfa90 0x5593377e77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.804+0000 7f499f84c740 1 --2- 172.21.15.104:0/1067085052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377df170 0x5593377e72b0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.804+0000 7f499f84c740 1 -- 172.21.15.104:0/1067085052 >> 172.21.15.104:0/1067085052 conn(0x5593377db680 msgr2=0x5593377e13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.804+0000 7f497dffb640 0 client.17433 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.804+0000 7f499f84c740 1 -- 172.21.15.104:0/1067085052 shutdown_connections 2024-06-09T22:28:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.804+0000 7f499f84c740 1 -- 172.21.15.104:0/1067085052 wait complete. 2024-06-09T22:28:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.804+0000 7f499f84c740 20 client.17433 trim_cache size 0 max 16384 2024-06-09T22:28:13.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:13 smithi194 ceph-mon[36856]: pgmap v221: 129 pgs: 129 active+clean; 177 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 229 op/s 2024-06-09T22:28:13.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:13 smithi194 ceph-mon[36856]: osdmap e102: 12 total, 12 up, 12 in 2024-06-09T22:28:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.819+0000 7f499f84c740 1 Processor -- start 2024-06-09T22:28:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.819+0000 7f499f84c740 1 -- start start 2024-06-09T22:28:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.819+0000 7f499f84c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377e1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.819+0000 7f499f84c740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377da660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.819+0000 7f499f84c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x559337802240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.819+0000 7f499f84c740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f49981aee60 con 0x5593377dfa90 2024-06-09T22:28:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.819+0000 7f499f84c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f49981b09a0 con 0x5593377ea350 2024-06-09T22:28:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.819+0000 7f499f84c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f49981af530 con 0x5593377df170 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f499659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377da660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f499659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377da660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60912/0 (socket says 172.21.15.104:60912) 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f499659c640 1 -- 172.21.15.104:0/2179165783 learned_addr learned my addr 172.21.15.104:0/2179165783 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f4995d9b640 1 --2- 172.21.15.104:0/2179165783 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377e1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f4995d9b640 1 --2- 172.21.15.104:0/2179165783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x559337802240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f499659c640 1 -- 172.21.15.104:0/2179165783 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 msgr2=0x5593377e1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f499659c640 1 --2- 172.21.15.104:0/2179165783 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377e1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f499659c640 1 -- 172.21.15.104:0/2179165783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 msgr2=0x559337802240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f499659c640 1 --2- 172.21.15.104:0/2179165783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x559337802240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f499659c640 1 -- 172.21.15.104:0/2179165783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593377db4b0 con 0x5593377dfa90 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f499659c640 1 --2- 172.21.15.104:0/2179165783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377da660 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f498801b040 tx=0x7f498800a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf44a1e6a8edef2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f497ffff640 1 -- 172.21.15.104:0/2179165783 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498801d040 con 0x5593377dfa90 2024-06-09T22:28:13.827 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f497ffff640 1 -- 172.21.15.104:0/2179165783 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4988005040 con 0x5593377dfa90 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.820+0000 7f497ffff640 1 -- 172.21.15.104:0/2179165783 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49880109e0 con 0x5593377dfa90 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.821+0000 7f499f84c740 1 -- 172.21.15.104:0/2179165783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 msgr2=0x5593377da660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.821+0000 7f499f84c740 1 --2- 172.21.15.104:0/2179165783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377da660 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f498801b040 tx=0x7f498800a2d0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.821+0000 7f499f84c740 1 -- 172.21.15.104:0/2179165783 shutdown_connections 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.821+0000 7f499f84c740 1 --2- 172.21.15.104:0/2179165783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x559337802240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.821+0000 7f499f84c740 1 --2- 172.21.15.104:0/2179165783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377da660 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.821+0000 7f499f84c740 1 --2- 172.21.15.104:0/2179165783 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377e1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.821+0000 7f499f84c740 1 -- 172.21.15.104:0/2179165783 >> 172.21.15.104:0/2179165783 conn(0x5593377db680 msgr2=0x5593377d9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.821+0000 7f499f84c740 1 -- 172.21.15.104:0/2179165783 shutdown_connections 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.821+0000 7f499f84c740 1 -- 172.21.15.104:0/2179165783 wait complete. 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.821+0000 7f499f84c740 1 -- start start 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.821+0000 7f499f84c740 1 Processor -- start 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499f84c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377ed940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499f84c740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377ede80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499f84c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x559337863200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499f84c740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f49981b02d0 con 0x5593377dfa90 2024-06-09T22:28:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499f84c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f49981afc00 con 0x5593377ea350 2024-06-09T22:28:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499f84c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f49981ace60 con 0x5593377df170 2024-06-09T22:28:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499f84c740 10 client.0 _ll_register_callbacks cb 0x7f498c011a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377ede80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377ede80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60924/0 (socket says 172.21.15.104:60924) 2024-06-09T22:28:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499659c640 1 -- 172.21.15.104:0/2772121371 learned_addr learned my addr 172.21.15.104:0/2772121371 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f4995d9b640 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377ed940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499659c640 1 -- 172.21.15.104:0/2772121371 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 msgr2=0x5593377ed940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499659c640 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377ed940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499659c640 1 -- 172.21.15.104:0/2772121371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 msgr2=0x559337863200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f4995d9b640 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x559337863200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499659c640 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x559337863200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499659c640 1 -- 172.21.15.104:0/2772121371 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559337863790 con 0x5593377dfa90 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499659c640 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377ede80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x5593377def30 tx=0x7f498800fe30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926f9dd7f20b8a30 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f497dffb640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f497dffb640 1 -- 172.21.15.104:0/2772121371 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4988034070 con 0x5593377dfa90 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f497dffb640 1 -- 172.21.15.104:0/2772121371 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f49880053f0 con 0x5593377dfa90 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.822+0000 7f499f84c740 1 -- 172.21.15.104:0/2772121371 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4988005040 con 0x5593377dfa90 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f497dffb640 1 -- 172.21.15.104:0/2772121371 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4988040050 con 0x5593377dfa90 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f499f84c740 10 client.17439 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f499f84c740 1 -- 172.21.15.104:0/2772121371 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559337864a40 con 0x5593377dfa90 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f497dffb640 1 -- 172.21.15.104:0/2772121371 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4988005040 con 0x5593377dfa90 2024-06-09T22:28:13.830 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f497dffb640 1 -- 172.21.15.104:0/2772121371 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f49880107a0 con 0x5593377dfa90 2024-06-09T22:28:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f499f84c740 10 client.17439 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f499f84c740 10 client.17439 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f499f84c740 20 client.17439 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f499f84c740 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559337864f60 0x559337867380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f499f84c740 4 client.17439 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:28:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f499f84c740 1 -- 172.21.15.104:0/2772121371 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559337750c20 con 0x559337864f60 2024-06-09T22:28:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f4995d9b640 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559337864f60 0x559337867380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f497dffb640 10 client.17439 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.823+0000 7f4995d9b640 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559337864f60 0x559337867380 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bada85abd5b547a server_cookie=d7042e434ae5570e in_seq=0 out_seq=0 2024-06-09T22:28:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.832+0000 7f497dffb640 1 -- 172.21.15.104:0/2772121371 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x559337750c20 con 0x559337864f60 2024-06-09T22:28:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.832+0000 7f497dffb640 10 client.17439 handle_command_reply: tid=0 2024-06-09T22:28:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.832+0000 7f499f84c740 1 client.17439 shutdown 2024-06-09T22:28:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.832+0000 7f499f84c740 1 -- 172.21.15.104:0/2772121371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 msgr2=0x5593377ede80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.832+0000 7f499f84c740 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377ede80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x5593377def30 tx=0x7f498800fe30 comp rx=0 tx=0).stop 2024-06-09T22:28:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499f84c740 1 -- 172.21.15.104:0/2772121371 shutdown_connections 2024-06-09T22:28:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499f84c740 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559337864f60 0x559337867380 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499f84c740 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593377ea350 0x559337863200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f497dffb640 0 client.17439 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499f84c740 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5593377dfa90 0x5593377ede80 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499f84c740 1 --2- 172.21.15.104:0/2772121371 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5593377df170 0x5593377ed940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499f84c740 1 -- 172.21.15.104:0/2772121371 >> 172.21.15.104:0/2772121371 conn(0x5593377db680 msgr2=0x5593377e7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499f84c740 1 -- 172.21.15.104:0/2772121371 shutdown_connections 2024-06-09T22:28:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499f84c740 1 -- 172.21.15.104:0/2772121371 wait complete. 2024-06-09T22:28:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499f84c740 20 client.17439 trim_cache size 0 max 16384 2024-06-09T22:28:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:28:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:28:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:28:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499cb48640 1 -- 172.21.15.104:0/225065007 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f49800a2da0 msgr2=0x7f49800a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499cb48640 1 --2- 172.21.15.104:0/225065007 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f49800a2da0 0x7f49800a5260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f49900edc80 tx=0x7f49900093d0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499cb48640 1 -- 172.21.15.104:0/225065007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f499810bf70 msgr2=0x7f499819ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499cb48640 1 --2- 172.21.15.104:0/225065007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f499810bf70 0x7f499819ca70 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f4990005cc0 tx=0x7f4990003ba0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499cb48640 1 -- 172.21.15.104:0/225065007 shutdown_connections 2024-06-09T22:28:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499cb48640 1 --2- 172.21.15.104:0/225065007 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f49800a2da0 0x7f49800a5260 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499cb48640 1 --2- 172.21.15.104:0/225065007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f499810bf70 0x7f499819ca70 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499cb48640 1 --2- 172.21.15.104:0/225065007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4998103550 0x7f49981986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499cb48640 1 --2- 172.21.15.104:0/225065007 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4998102c30 0x7f49981981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.833+0000 7f499cb48640 1 -- 172.21.15.104:0/225065007 >> 172.21.15.104:0/225065007 conn(0x7f4998077a20 msgr2=0x7f49980fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.834+0000 7f499cb48640 1 -- 172.21.15.104:0/225065007 shutdown_connections 2024-06-09T22:28:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:13.834+0000 7f499cb48640 1 -- 172.21.15.104:0/225065007 wait complete. 2024-06-09T22:28:13.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:13 smithi153 ceph-mon[36599]: pgmap v221: 129 pgs: 129 active+clean; 177 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 229 op/s 2024-06-09T22:28:13.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:13 smithi153 ceph-mon[36599]: osdmap e102: 12 total, 12 up, 12 in 2024-06-09T22:28:14.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:13 smithi104 ceph-mon[32389]: pgmap v221: 129 pgs: 129 active+clean; 177 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 229 op/s 2024-06-09T22:28:14.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:13 smithi104 ceph-mon[32389]: osdmap e102: 12 total, 12 up, 12 in 2024-06-09T22:28:14.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 242041 4.59 MB/sec execute 200 sec latency 26.137 ms 2024-06-09T22:28:15.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 244220 4.63 MB/sec execute 201 sec latency 23.412 ms 2024-06-09T22:28:15.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:15 smithi194 ceph-mon[36856]: pgmap v223: 129 pgs: 129 active+clean; 188 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.5 MiB/s wr, 253 op/s 2024-06-09T22:28:15.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:15 smithi153 ceph-mon[36599]: pgmap v223: 129 pgs: 129 active+clean; 188 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.5 MiB/s wr, 253 op/s 2024-06-09T22:28:16.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:15 smithi104 ceph-mon[32389]: pgmap v223: 129 pgs: 129 active+clean; 188 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.5 MiB/s wr, 253 op/s 2024-06-09T22:28:16.062 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:28:16.321 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:28:16.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 245587 4.64 MB/sec execute 202 sec latency 33.248 ms 2024-06-09T22:28:17.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 247754 4.68 MB/sec execute 203 sec latency 15.575 ms 2024-06-09T22:28:17.758 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:17 smithi104 ceph-mon[32389]: pgmap v224: 129 pgs: 129 active+clean; 188 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 205 op/s 2024-06-09T22:28:17.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:17 smithi194 ceph-mon[36856]: pgmap v224: 129 pgs: 129 active+clean; 188 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 205 op/s 2024-06-09T22:28:17.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:17 smithi153 ceph-mon[36599]: pgmap v224: 129 pgs: 129 active+clean; 188 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 205 op/s 2024-06-09T22:28:18.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.168+0000 7f4b19547700 1 -- 172.21.15.104:0/1049649323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b14106370 msgr2=0x7f4b14106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:18.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.168+0000 7f4b19547700 1 --2- 172.21.15.104:0/1049649323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b14106370 0x7f4b14106750 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009990 tx=0x7f4b04009ca0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.168+0000 7f4b19547700 1 -- 172.21.15.104:0/1049649323 shutdown_connections 2024-06-09T22:28:18.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.168+0000 7f4b19547700 1 --2- 172.21.15.104:0/1049649323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b14106370 0x7f4b14106750 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.168+0000 7f4b19547700 1 --2- 172.21.15.104:0/1049649323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b14102550 0x7f4b1410e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.168+0000 7f4b19547700 1 --2- 172.21.15.104:0/1049649323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b14101b90 0x7f4b14102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.168+0000 7f4b19547700 1 -- 172.21.15.104:0/1049649323 >> 172.21.15.104:0/1049649323 conn(0x7f4b14077a00 msgr2=0x7f4b140fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:18.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.168+0000 7f4b19547700 1 -- 172.21.15.104:0/1049649323 shutdown_connections 2024-06-09T22:28:18.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.168+0000 7f4b19547700 1 -- 172.21.15.104:0/1049649323 wait complete. 2024-06-09T22:28:18.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.168+0000 7f4b19547700 1 Processor -- start 2024-06-09T22:28:18.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b19547700 1 -- start start 2024-06-09T22:28:18.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b19547700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b14101b90 0x7f4b1419d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b19547700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b14102550 0x7f4b1419dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b19547700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b14106370 0x7f4b14197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b19547700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4b14111640 con 0x7f4b14106370 2024-06-09T22:28:18.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b19547700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4b141114c0 con 0x7f4b14101b90 2024-06-09T22:28:18.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b19547700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4b141117c0 con 0x7f4b14102550 2024-06-09T22:28:18.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b137fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b14106370 0x7f4b14197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b127fc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b14102550 0x7f4b1419dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b12ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b14101b90 0x7f4b1419d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b127fc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b14102550 0x7f4b1419dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:49486/0 (socket says 172.21.15.104:49486) 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b12ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b14101b90 0x7f4b1419d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:34382/0 (socket says 172.21.15.104:34382) 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b127fc700 1 -- 172.21.15.104:0/828934836 learned_addr learned my addr 172.21.15.104:0/828934836 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b127fc700 1 -- 172.21.15.104:0/828934836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b14101b90 msgr2=0x7f4b1419d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b127fc700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b14101b90 0x7f4b1419d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b127fc700 1 -- 172.21.15.104:0/828934836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b14106370 msgr2=0x7f4b14197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b127fc700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b14106370 0x7f4b14197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b127fc700 1 -- 172.21.15.104:0/828934836 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b14198060 con 0x7f4b14102550 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b137fe700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b14106370 0x7f4b14197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.169+0000 7f4b12ffd700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b14101b90 0x7f4b1419d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.170+0000 7f4b127fc700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b14102550 0x7f4b1419dbe0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800eab0 tx=0x7f4b0800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.170+0000 7f4afbfff700 1 -- 172.21.15.104:0/828934836 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0801c070 con 0x7f4b14102550 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.170+0000 7f4b19547700 1 -- 172.21.15.104:0/828934836 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b14198350 con 0x7f4b14102550 2024-06-09T22:28:18.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.170+0000 7f4b19547700 1 -- 172.21.15.104:0/828934836 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b14100a80 con 0x7f4b14102550 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.170+0000 7f4afbfff700 1 -- 172.21.15.104:0/828934836 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b08017410 con 0x7f4b14102550 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.170+0000 7f4afbfff700 1 -- 172.21.15.104:0/828934836 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b08026060 con 0x7f4b14102550 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.172+0000 7f4afbfff700 1 -- 172.21.15.104:0/828934836 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f4b0801f070 con 0x7f4b14102550 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.173+0000 7f4afbfff700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4afc0a2d90 0x7f4afc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.173+0000 7f4b12ffd700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4afc0a2d90 0x7f4afc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.173+0000 7f4afbfff700 1 -- 172.21.15.104:0/828934836 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f4b080d0b40 con 0x7f4b14102550 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b23dcab80 1 Processor -- start 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b12ffd700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4afc0a2d90 0x7f4afc0a5250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400b2e0 tx=0x7f4b04005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b23dcab80 1 -- start start 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b23dcab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2b640 0x557265a2ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b23dcab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2bf60 0x557265a259e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b23dcab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a25f20 0x557265a26340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b23dcab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557265977810 con 0x557265a2b640 2024-06-09T22:28:18.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b23dcab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55726591f720 con 0x557265a25f20 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b23dcab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5572659737d0 con 0x557265a2bf60 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b12ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2b640 0x557265a2ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b12ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2b640 0x557265a2ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48714/0 (socket says 172.21.15.104:48714) 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b137fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a25f20 0x557265a26340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b127fc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2bf60 0x557265a259e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b12ffd700 1 -- 172.21.15.104:0/792849269 learned_addr learned my addr 172.21.15.104:0/792849269 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b12ffd700 1 -- 172.21.15.104:0/792849269 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2bf60 msgr2=0x557265a259e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b12ffd700 1 --2- 172.21.15.104:0/792849269 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2bf60 0x557265a259e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b12ffd700 1 -- 172.21.15.104:0/792849269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a25f20 msgr2=0x557265a26340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b12ffd700 1 --2- 172.21.15.104:0/792849269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a25f20 0x557265a26340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b12ffd700 1 -- 172.21.15.104:0/792849269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557265915f50 con 0x557265a2b640 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.174+0000 7f4b137fe700 1 --2- 172.21.15.104:0/792849269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a25f20 0x557265a26340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:18.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.175+0000 7f4b12ffd700 1 --2- 172.21.15.104:0/792849269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2b640 0x557265a2ba20 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009990 tx=0x7f4b04020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94fda724de13c68 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.175+0000 7f4af97fa700 1 -- 172.21.15.104:0/792849269 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b04004830 con 0x557265a2b640 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.175+0000 7f4af97fa700 1 -- 172.21.15.104:0/792849269 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b04005020 con 0x557265a2b640 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.175+0000 7f4af97fa700 1 -- 172.21.15.104:0/792849269 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b04003b60 con 0x557265a2b640 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 -- 172.21.15.104:0/792849269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2b640 msgr2=0x557265a2ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/792849269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2b640 0x557265a2ba20 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009990 tx=0x7f4b04020000 comp rx=0 tx=0).stop 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 -- 172.21.15.104:0/792849269 shutdown_connections 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/792849269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2b640 0x557265a2ba20 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/792849269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a25f20 0x557265a26340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/792849269 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2bf60 0x557265a259e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 -- 172.21.15.104:0/792849269 >> 172.21.15.104:0/792849269 conn(0x557265950cc0 msgr2=0x557265a20850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 -- 172.21.15.104:0/792849269 shutdown_connections 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 -- 172.21.15.104:0/792849269 wait complete. 2024-06-09T22:28:18.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 -- start start 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 Processor -- start 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a25f20 0x557265a31cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2b640 0x557265a2efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a2bf60 0x557265a2f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.176+0000 7f4b23dcab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557265a32970 con 0x557265a25f20 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b23dcab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557265917bf0 con 0x557265a2bf60 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b23dcab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557265a32af0 con 0x557265a2b640 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b23dcab80 10 client.0 _ll_register_callbacks cb 0x557265911920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b12ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a25f20 0x557265a31cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b127fc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2b640 0x557265a2efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b127fc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2b640 0x557265a2efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:49508/0 (socket says 172.21.15.104:49508) 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b127fc700 1 -- 172.21.15.104:0/3010161004 learned_addr learned my addr 172.21.15.104:0/3010161004 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b137fe700 1 --2- 172.21.15.104:0/3010161004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a2bf60 0x557265a2f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b127fc700 1 -- 172.21.15.104:0/3010161004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a2bf60 msgr2=0x557265a2f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b127fc700 1 --2- 172.21.15.104:0/3010161004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a2bf60 0x557265a2f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b127fc700 1 -- 172.21.15.104:0/3010161004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a25f20 msgr2=0x557265a31cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b127fc700 1 --2- 172.21.15.104:0/3010161004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a25f20 0x557265a31cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b127fc700 1 -- 172.21.15.104:0/3010161004 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572658c68d0 con 0x557265a2b640 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b127fc700 1 --2- 172.21.15.104:0/3010161004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2b640 0x557265a2efd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800b5a0 tx=0x7f4b0809c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=902f56e5e7292fed server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4af6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4af6ff5700 1 -- 172.21.15.104:0/3010161004 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0809cc90 con 0x557265a2b640 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3010161004 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b04005020 con 0x557265a2b640 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4af6ff5700 1 -- 172.21.15.104:0/3010161004 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f4b080a5680 con 0x557265a2b640 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4af6ff5700 1 -- 172.21.15.104:0/3010161004 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4b080a1040 con 0x557265a2b640 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b23dcab80 10 client.26782 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.177+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3010161004 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557265a3d380 con 0x557265a2b640 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.178+0000 7f4af6ff5700 1 -- 172.21.15.104:0/3010161004 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b04005020 con 0x557265a2b640 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.178+0000 7f4af6ff5700 1 -- 172.21.15.104:0/3010161004 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4b0807bb40 con 0x557265a2b640 2024-06-09T22:28:18.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.178+0000 7f4b23dcab80 10 client.26782 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.178+0000 7f4b23dcab80 10 client.26782 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.178+0000 7f4b23dcab80 20 client.26782 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.178+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/3010161004 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557265a3f910 0x557265a41d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.178+0000 7f4b23dcab80 4 client.26782 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.178+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3010161004 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557265984d10 con 0x557265a3f910 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.178+0000 7f4b12ffd700 1 --2- 172.21.15.104:0/3010161004 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557265a3f910 0x557265a41d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.179+0000 7f4af6ff5700 10 client.26782 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.179+0000 7f4b12ffd700 1 --2- 172.21.15.104:0/3010161004 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557265a3f910 0x557265a41d20 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa4f66a7dc7e6073 server_cookie=f9ef1e789bba8c7c in_seq=0 out_seq=0 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4af6ff5700 1 -- 172.21.15.104:0/3010161004 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x557265984d10 con 0x557265a3f910 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4af6ff5700 10 client.26782 handle_command_reply: tid=0 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4b23dcab80 1 client.26782 shutdown 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3010161004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2b640 msgr2=0x557265a2efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/3010161004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2b640 0x557265a2efd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800b5a0 tx=0x7f4b0809c670 comp rx=0 tx=0).stop 2024-06-09T22:28:18.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3010161004 shutdown_connections 2024-06-09T22:28:18.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/3010161004 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557265a3f910 0x557265a41d20 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/3010161004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a25f20 0x557265a31cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/3010161004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a2bf60 0x557265a2f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/3010161004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2b640 0x557265a2efd0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4af6ff5700 0 client.26782 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:18.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3010161004 >> 172.21.15.104:0/3010161004 conn(0x557265950cc0 msgr2=0x557265a249d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:18.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3010161004 shutdown_connections 2024-06-09T22:28:18.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3010161004 wait complete. 2024-06-09T22:28:18.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.180+0000 7f4b23dcab80 20 client.26782 trim_cache size 0 max 16384 2024-06-09T22:28:18.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.210+0000 7f4b23dcab80 1 Processor -- start 2024-06-09T22:28:18.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.212+0000 7f4b23dcab80 1 -- start start 2024-06-09T22:28:18.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.212+0000 7f4b23dcab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a25f20 0x557265a259e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.212+0000 7f4b23dcab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a2b640 0x557265a2c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.212+0000 7f4b23dcab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2bf60 0x557265a2c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.212+0000 7f4b23dcab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0000edc0 con 0x557265a25f20 2024-06-09T22:28:18.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.212+0000 7f4b23dcab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4b00010900 con 0x557265a2b640 2024-06-09T22:28:18.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.212+0000 7f4b23dcab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0000f490 con 0x557265a2bf60 2024-06-09T22:28:18.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.213+0000 7f4b127fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a2b640 0x557265a2c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.213+0000 7f4b127fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a2b640 0x557265a2c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:34412/0 (socket says 172.21.15.104:34412) 2024-06-09T22:28:18.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.213+0000 7f4b127fc700 1 -- 172.21.15.104:0/3715772670 learned_addr learned my addr 172.21.15.104:0/3715772670 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:18.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.215+0000 7f4b127fc700 1 -- 172.21.15.104:0/3715772670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2bf60 msgr2=0x557265a2c940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:28:18.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.215+0000 7f4b127fc700 1 --2- 172.21.15.104:0/3715772670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2bf60 0x557265a2c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.215+0000 7f4b127fc700 1 -- 172.21.15.104:0/3715772670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a25f20 msgr2=0x557265a259e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:28:18.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.215+0000 7f4b127fc700 1 --2- 172.21.15.104:0/3715772670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a25f20 0x557265a259e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.215+0000 7f4b127fc700 1 -- 172.21.15.104:0/3715772670 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557265a2d560 con 0x557265a2b640 2024-06-09T22:28:18.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.216+0000 7f4b127fc700 1 --2- 172.21.15.104:0/3715772670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a2b640 0x557265a2c400 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4b0809cc20 tx=0x7f4b0800ef50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2809d11818d6654e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:18.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.217+0000 7f4af8ff9700 1 -- 172.21.15.104:0/3715772670 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b080a5900 con 0x557265a2b640 2024-06-09T22:28:18.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.217+0000 7f4af8ff9700 1 -- 172.21.15.104:0/3715772670 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b0807bab0 con 0x557265a2b640 2024-06-09T22:28:18.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.218+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3715772670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a2b640 msgr2=0x557265a2c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:18.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.218+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/3715772670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a2b640 0x557265a2c400 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4b0809cc20 tx=0x7f4b0800ef50 comp rx=0 tx=0).stop 2024-06-09T22:28:18.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.223+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3715772670 shutdown_connections 2024-06-09T22:28:18.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.223+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/3715772670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a25f20 0x557265a259e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.223+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/3715772670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2bf60 0x557265a2c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.223+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/3715772670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a2b640 0x557265a2c400 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.223+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3715772670 >> 172.21.15.104:0/3715772670 conn(0x557265950cc0 msgr2=0x557265a2d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.227+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3715772670 shutdown_connections 2024-06-09T22:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.227+0000 7f4b23dcab80 1 -- 172.21.15.104:0/3715772670 wait complete. 2024-06-09T22:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.227+0000 7f4b23dcab80 1 -- start start 2024-06-09T22:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.227+0000 7f4b23dcab80 1 Processor -- start 2024-06-09T22:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.228+0000 7f4b23dcab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a25f20 0x557265a37e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.228+0000 7f4b23dcab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2b640 0x557265a38350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.228+0000 7f4b23dcab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2bf60 0x557265a34de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.228+0000 7f4b23dcab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4b00010230 con 0x557265a2bf60 2024-06-09T22:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.228+0000 7f4b23dcab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0000fb60 con 0x557265a25f20 2024-06-09T22:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.228+0000 7f4b23dcab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0000cd80 con 0x557265a2b640 2024-06-09T22:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.228+0000 7f4b23dcab80 10 client.0 _ll_register_callbacks cb 0x557265911920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.229+0000 7f4b12ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a25f20 0x557265a37e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.229+0000 7f4b137fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2bf60 0x557265a34de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.229+0000 7f4b127fc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2b640 0x557265a38350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.229+0000 7f4b137fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2bf60 0x557265a34de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48722/0 (socket says 172.21.15.104:48722) 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.229+0000 7f4b137fe700 1 -- 172.21.15.104:0/2067587931 learned_addr learned my addr 172.21.15.104:0/2067587931 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.229+0000 7f4b137fe700 1 -- 172.21.15.104:0/2067587931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2b640 msgr2=0x557265a38350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.229+0000 7f4b137fe700 1 --2- 172.21.15.104:0/2067587931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2b640 0x557265a38350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.229+0000 7f4b137fe700 1 -- 172.21.15.104:0/2067587931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a25f20 msgr2=0x557265a37e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.229+0000 7f4b137fe700 1 --2- 172.21.15.104:0/2067587931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a25f20 0x557265a37e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.229+0000 7f4b137fe700 1 -- 172.21.15.104:0/2067587931 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557265a35320 con 0x557265a2bf60 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.229+0000 7f4b137fe700 1 --2- 172.21.15.104:0/2067587931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2bf60 0x557265a34de0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00f000 tx=0x7f4b0c0177a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb8bad78b02a6f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.229+0000 7f4af6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.230+0000 7f4af6ff5700 1 -- 172.21.15.104:0/2067587931 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c004db0 con 0x557265a2bf60 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.230+0000 7f4af6ff5700 1 -- 172.21.15.104:0/2067587931 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f4b0c0076b0 con 0x557265a2bf60 2024-06-09T22:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.230+0000 7f4b23dcab80 1 -- 172.21.15.104:0/2067587931 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55726597f290 con 0x557265a2bf60 2024-06-09T22:28:18.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.231+0000 7f4af6ff5700 1 -- 172.21.15.104:0/2067587931 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4b0c0078b0 con 0x557265a2bf60 2024-06-09T22:28:18.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.231+0000 7f4b23dcab80 10 client.17454 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:18.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.231+0000 7f4b23dcab80 1 -- 172.21.15.104:0/2067587931 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557265aa2650 con 0x557265a2bf60 2024-06-09T22:28:18.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.232+0000 7f4af6ff5700 1 -- 172.21.15.104:0/2067587931 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55726597f290 con 0x557265a2bf60 2024-06-09T22:28:18.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.232+0000 7f4af6ff5700 1 -- 172.21.15.104:0/2067587931 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4b0c01fcd0 con 0x557265a2bf60 2024-06-09T22:28:18.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.232+0000 7f4b23dcab80 10 client.17454 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.232+0000 7f4b23dcab80 10 client.17454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.232+0000 7f4b23dcab80 20 client.17454 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.232+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/2067587931 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557265aa2d90 0x557265aa51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.232+0000 7f4b23dcab80 4 client.17454 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.232+0000 7f4b23dcab80 1 -- 172.21.15.104:0/2067587931 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5572657f9250 con 0x557265aa2d90 2024-06-09T22:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.234+0000 7f4b12ffd700 1 --2- 172.21.15.104:0/2067587931 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557265aa2d90 0x557265aa51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:18.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.238+0000 7f4b12ffd700 1 --2- 172.21.15.104:0/2067587931 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557265aa2d90 0x557265aa51b0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=305f6c725014a584 server_cookie=8f9885c80e8e62a6 in_seq=0 out_seq=0 2024-06-09T22:28:18.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.238+0000 7f4af6ff5700 10 client.17454 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:18.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.239+0000 7f4af6ff5700 1 -- 172.21.15.104:0/2067587931 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5572657f9250 con 0x557265aa2d90 2024-06-09T22:28:18.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.239+0000 7f4af6ff5700 10 client.17454 handle_command_reply: tid=0 2024-06-09T22:28:18.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.239+0000 7f4b23dcab80 1 client.17454 shutdown 2024-06-09T22:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.242+0000 7f4b23dcab80 1 -- 172.21.15.104:0/2067587931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2bf60 msgr2=0x557265a34de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.242+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/2067587931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2bf60 0x557265a34de0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00f000 tx=0x7f4b0c0177a0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.243+0000 7f4b23dcab80 1 -- 172.21.15.104:0/2067587931 shutdown_connections 2024-06-09T22:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.243+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/2067587931 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557265aa2d90 0x557265aa51b0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.243+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/2067587931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557265a2bf60 0x557265a34de0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.243+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/2067587931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557265a25f20 0x557265a37e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.243+0000 7f4b23dcab80 1 --2- 172.21.15.104:0/2067587931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557265a2b640 0x557265a38350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.243+0000 7f4af6ff5700 0 client.17454 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.243+0000 7f4b23dcab80 1 -- 172.21.15.104:0/2067587931 >> 172.21.15.104:0/2067587931 conn(0x557265950cc0 msgr2=0x557265a19e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.243+0000 7f4b23dcab80 1 -- 172.21.15.104:0/2067587931 shutdown_connections 2024-06-09T22:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.243+0000 7f4b23dcab80 1 -- 172.21.15.104:0/2067587931 wait complete. 2024-06-09T22:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.243+0000 7f4b23dcab80 20 client.17454 trim_cache size 0 max 16384 2024-06-09T22:28:18.247 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:28:18.247 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "no active scrubs running", 2024-06-09T22:28:18.247 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": {} 2024-06-09T22:28:18.247 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.247+0000 7f4b19547700 1 -- 172.21.15.104:0/828934836 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4afc0a2d90 msgr2=0x7f4afc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.247+0000 7f4b19547700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4afc0a2d90 0x7f4afc0a5250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400b2e0 tx=0x7f4b04005960 comp rx=0 tx=0).stop 2024-06-09T22:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.247+0000 7f4b19547700 1 -- 172.21.15.104:0/828934836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b14102550 msgr2=0x7f4b1419dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.247+0000 7f4b19547700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b14102550 0x7f4b1419dbe0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800eab0 tx=0x7f4b0800edc0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.247+0000 7f4b19547700 1 -- 172.21.15.104:0/828934836 shutdown_connections 2024-06-09T22:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.247+0000 7f4b19547700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4afc0a2d90 0x7f4afc0a5250 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.247+0000 7f4b19547700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b14106370 0x7f4b14197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.247+0000 7f4b19547700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b14101b90 0x7f4b1419d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.247+0000 7f4b19547700 1 --2- 172.21.15.104:0/828934836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b14102550 0x7f4b1419dbe0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.247+0000 7f4b19547700 1 -- 172.21.15.104:0/828934836 >> 172.21.15.104:0/828934836 conn(0x7f4b14077a00 msgr2=0x7f4b141042e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.248+0000 7f4b19547700 1 -- 172.21.15.104:0/828934836 shutdown_connections 2024-06-09T22:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:18.248+0000 7f4b19547700 1 -- 172.21.15.104:0/828934836 wait complete. 2024-06-09T22:28:18.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 248908 4.68 MB/sec execute 204 sec latency 57.319 ms 2024-06-09T22:28:18.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:18 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:28:18.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:18 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:28:19.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:18 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:28:19.300 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-09T22:28:19.301 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:28:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:28:19.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 250764 4.70 MB/sec execute 205 sec latency 16.992 ms 2024-06-09T22:28:19.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:19 smithi194 ceph-mon[36856]: pgmap v225: 129 pgs: 129 active+clean; 188 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 205 op/s 2024-06-09T22:28:19.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:19 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:28:19.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:19 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:28:19.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:19 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:28:19.813 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:19 smithi104 ceph-mon[32389]: pgmap v225: 129 pgs: 129 active+clean; 188 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 205 op/s 2024-06-09T22:28:19.813 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:19 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:28:19.813 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:19 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:28:19.813 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:19 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:28:19.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:19 smithi153 ceph-mon[36599]: pgmap v225: 129 pgs: 129 active+clean; 188 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 205 op/s 2024-06-09T22:28:19.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:19 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:28:19.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:19 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:28:19.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:19 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:28:20.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 252441 4.73 MB/sec execute 206 sec latency 10.036 ms 2024-06-09T22:28:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.434+0000 7facc1a76700 1 -- 172.21.15.104:0/2042004107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facbc1027b0 msgr2=0x7facbc102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.434+0000 7facc1a76700 1 --2- 172.21.15.104:0/2042004107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facbc1027b0 0x7facbc102c30 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7facb0009990 tx=0x7facb0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:28:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.434+0000 7facc1a76700 1 -- 172.21.15.104:0/2042004107 shutdown_connections 2024-06-09T22:28:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.434+0000 7facc1a76700 1 --2- 172.21.15.104:0/2042004107 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facbc103170 0x7facbc10f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.434+0000 7facc1a76700 1 --2- 172.21.15.104:0/2042004107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facbc108650 0x7facbc108a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.434+0000 7facc1a76700 1 --2- 172.21.15.104:0/2042004107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facbc1027b0 0x7facbc102c30 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.434+0000 7facc1a76700 1 -- 172.21.15.104:0/2042004107 >> 172.21.15.104:0/2042004107 conn(0x7facbc07a420 msgr2=0x7facbc07a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.434+0000 7facc1a76700 1 -- 172.21.15.104:0/2042004107 shutdown_connections 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.434+0000 7facc1a76700 1 -- 172.21.15.104:0/2042004107 wait complete. 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.435+0000 7facc1a76700 1 Processor -- start 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.435+0000 7facc1a76700 1 -- start start 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facc1a76700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facbc1027b0 0x7facbc19e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facc1a76700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facbc103170 0x7facbc19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facc1a76700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facbc108650 0x7facbc198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facc1a76700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7facbc112880 con 0x7facbc108650 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facc1a76700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7facbc112700 con 0x7facbc1027b0 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facc1a76700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7facbc112a00 con 0x7facbc103170 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facbbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facbc108650 0x7facbc198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facbaffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facbc103170 0x7facbc19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facbbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facbc108650 0x7facbc198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48736/0 (socket says 172.21.15.104:48736) 2024-06-09T22:28:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facbaffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facbc103170 0x7facbc19ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:49524/0 (socket says 172.21.15.104:49524) 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facbbfff700 1 -- 172.21.15.104:0/3564144633 learned_addr learned my addr 172.21.15.104:0/3564144633 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facbb7fe700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facbc1027b0 0x7facbc19e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facbbfff700 1 -- 172.21.15.104:0/3564144633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facbc103170 msgr2=0x7facbc19ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facbbfff700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facbc103170 0x7facbc19ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facbbfff700 1 -- 172.21.15.104:0/3564144633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facbc1027b0 msgr2=0x7facbc19e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facbbfff700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facbc1027b0 0x7facbc19e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facbbfff700 1 -- 172.21.15.104:0/3564144633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facbc1991e0 con 0x7facbc108650 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facbbfff700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facbc108650 0x7facbc198a80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7facac00efc0 tx=0x7facac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facb8ff9700 1 -- 172.21.15.104:0/3564144633 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac0092e0 con 0x7facbc108650 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facc1a76700 1 -- 172.21.15.104:0/3564144633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facbc199470 con 0x7facbc108650 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facc1a76700 1 -- 172.21.15.104:0/3564144633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facbc1a5790 con 0x7facbc108650 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.436+0000 7facb8ff9700 1 -- 172.21.15.104:0/3564144633 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facac004bb0 con 0x7facbc108650 2024-06-09T22:28:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.437+0000 7facb8ff9700 1 -- 172.21.15.104:0/3564144633 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac004020 con 0x7facbc108650 2024-06-09T22:28:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.438+0000 7facc1a76700 1 -- 172.21.15.104:0/3564144633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faca8005330 con 0x7facbc108650 2024-06-09T22:28:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.438+0000 7facb8ff9700 1 -- 172.21.15.104:0/3564144633 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7facac01b030 con 0x7facbc108650 2024-06-09T22:28:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.439+0000 7facb8ff9700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faca40a2d90 0x7faca40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:21.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.439+0000 7facbb7fe700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faca40a2d90 0x7faca40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:21.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.440+0000 7facb8ff9700 1 -- 172.21.15.104:0/3564144633 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7facac0cf030 con 0x7facbc108650 2024-06-09T22:28:21.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.440+0000 7facbb7fe700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faca40a2d90 0x7faca40a5250 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7facbc199bc0 tx=0x7facb4009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:21.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.444+0000 7facb8ff9700 1 -- 172.21.15.104:0/3564144633 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7facac022020 con 0x7facbc108650 2024-06-09T22:28:21.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.705+0000 7facc1a76700 1 -- 172.21.15.104:0/3564144633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faca80058f0 con 0x7facbc108650 2024-06-09T22:28:21.707 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:21 smithi104 ceph-mon[32389]: pgmap v226: 129 pgs: 129 active+clean; 200 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 229 op/s 2024-06-09T22:28:21.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.705+0000 7facb8ff9700 1 -- 172.21.15.104:0/3564144633 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7facac010040 con 0x7facbc108650 2024-06-09T22:28:21.708 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:28:21.708 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:28:21.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.708+0000 7facc1a76700 1 -- 172.21.15.104:0/3564144633 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faca40a2d90 msgr2=0x7faca40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:21.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.708+0000 7facc1a76700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faca40a2d90 0x7faca40a5250 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7facbc199bc0 tx=0x7facb4009260 comp rx=0 tx=0).stop 2024-06-09T22:28:21.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.708+0000 7facc1a76700 1 -- 172.21.15.104:0/3564144633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facbc108650 msgr2=0x7facbc198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:21.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.708+0000 7facc1a76700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facbc108650 0x7facbc198a80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7facac00efc0 tx=0x7facac00c3a0 comp rx=0 tx=0).stop 2024-06-09T22:28:21.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.708+0000 7facc1a76700 1 -- 172.21.15.104:0/3564144633 shutdown_connections 2024-06-09T22:28:21.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.708+0000 7facc1a76700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7faca40a2d90 0x7faca40a5250 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:21.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.708+0000 7facc1a76700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facbc108650 0x7facbc198a80 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:21.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.708+0000 7facc1a76700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facbc1027b0 0x7facbc19e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:21.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.708+0000 7facc1a76700 1 --2- 172.21.15.104:0/3564144633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7facbc103170 0x7facbc19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:21.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.708+0000 7facc1a76700 1 -- 172.21.15.104:0/3564144633 >> 172.21.15.104:0/3564144633 conn(0x7facbc07a420 msgr2=0x7facbc101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:21.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.708+0000 7facc1a76700 1 -- 172.21.15.104:0/3564144633 shutdown_connections 2024-06-09T22:28:21.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:21.708+0000 7facc1a76700 1 -- 172.21.15.104:0/3564144633 wait complete. 2024-06-09T22:28:21.711 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:28:21.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 254309 4.74 MB/sec execute 207 sec latency 32.169 ms 2024-06-09T22:28:21.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:21 smithi194 ceph-mon[36856]: pgmap v226: 129 pgs: 129 active+clean; 200 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 229 op/s 2024-06-09T22:28:21.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:21 smithi153 ceph-mon[36599]: pgmap v226: 129 pgs: 129 active+clean; 200 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 229 op/s 2024-06-09T22:28:22.477 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-09T22:28:22.726 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:28:22.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 255987 4.78 MB/sec execute 208 sec latency 11.462 ms 2024-06-09T22:28:22.758 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:22 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3564144633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:28:22.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:22 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3564144633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:28:22.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:22 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3564144633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:28:23.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 257776 4.79 MB/sec execute 209 sec latency 51.092 ms 2024-06-09T22:28:23.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:23 smithi194 ceph-mon[36856]: pgmap v227: 129 pgs: 129 active+clean; 200 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 229 op/s 2024-06-09T22:28:23.897 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:23 smithi104 ceph-mon[32389]: pgmap v227: 129 pgs: 129 active+clean; 200 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 229 op/s 2024-06-09T22:28:23.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:23 smithi153 ceph-mon[36599]: pgmap v227: 129 pgs: 129 active+clean; 200 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 229 op/s 2024-06-09T22:28:24.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 259433 4.82 MB/sec execute 210 sec latency 21.921 ms 2024-06-09T22:28:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 -- 172.21.15.104:0/2271507248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a70107840 msgr2=0x7f3a7010f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 --2- 172.21.15.104:0/2271507248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a70107840 0x7f3a7010f3c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800b0d0 tx=0x7f3a6800b3e0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 -- 172.21.15.104:0/2271507248 shutdown_connections 2024-06-09T22:28:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 --2- 172.21.15.104:0/2271507248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a70107840 0x7f3a7010f3c0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 --2- 172.21.15.104:0/2271507248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a70102740 0x7f3a70102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 --2- 172.21.15.104:0/2271507248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a701030f0 0x7f3a70107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 -- 172.21.15.104:0/2271507248 >> 172.21.15.104:0/2271507248 conn(0x7f3a700795f0 msgr2=0x7f3a70079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 -- 172.21.15.104:0/2271507248 shutdown_connections 2024-06-09T22:28:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 -- 172.21.15.104:0/2271507248 wait complete. 2024-06-09T22:28:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 Processor -- start 2024-06-09T22:28:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 -- start start 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a70102740 0x7f3a7019e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.828+0000 7f3a7805b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a701030f0 0x7f3a7019ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a7805b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a70107840 0x7f3a70198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a7805b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3a70112780 con 0x7f3a70102740 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a7805b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a70112600 con 0x7f3a70107840 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a7805b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3a70112900 con 0x7f3a701030f0 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a75df7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a70102740 0x7f3a7019e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a75df7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a70102740 0x7f3a7019e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48762/0 (socket says 172.21.15.104:48762) 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a75df7700 1 -- 172.21.15.104:0/2605208349 learned_addr learned my addr 172.21.15.104:0/2605208349 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a755f6700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a701030f0 0x7f3a7019ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a765f8700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a70107840 0x7f3a70198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a75df7700 1 -- 172.21.15.104:0/2605208349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a701030f0 msgr2=0x7f3a7019ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a75df7700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a701030f0 0x7f3a7019ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a75df7700 1 -- 172.21.15.104:0/2605208349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a70107840 msgr2=0x7f3a70198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:24.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a75df7700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a70107840 0x7f3a70198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a75df7700 1 -- 172.21.15.104:0/2605208349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a701990e0 con 0x7f3a70102740 2024-06-09T22:28:24.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a765f8700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a70107840 0x7f3a70198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:28:24.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a75df7700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a70102740 0x7f3a7019e7c0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000e9c0 tx=0x7f3a6000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:24.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a66ffd700 1 -- 172.21.15.104:0/2605208349 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6000cbd0 con 0x7f3a70102740 2024-06-09T22:28:24.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a66ffd700 1 -- 172.21.15.104:0/2605208349 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6000cd70 con 0x7f3a70102740 2024-06-09T22:28:24.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a7805b700 1 -- 172.21.15.104:0/2605208349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a701993d0 con 0x7f3a70102740 2024-06-09T22:28:24.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a66ffd700 1 -- 172.21.15.104:0/2605208349 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a60010430 con 0x7f3a70102740 2024-06-09T22:28:24.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.829+0000 7f3a7805b700 1 -- 172.21.15.104:0/2605208349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a701a5690 con 0x7f3a70102740 2024-06-09T22:28:24.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.832+0000 7f3a66ffd700 1 -- 172.21.15.104:0/2605208349 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f3a60004750 con 0x7f3a70102740 2024-06-09T22:28:24.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.833+0000 7f3a66ffd700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f3a5c0a2d90 0x7f3a5c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.833+0000 7f3a755f6700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f3a5c0a2d90 0x7f3a5c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.833+0000 7f3a66ffd700 1 -- 172.21.15.104:0/2605208349 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f3a600cfb30 con 0x7f3a70102740 2024-06-09T22:28:24.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.833+0000 7f3a755f6700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f3a5c0a2d90 0x7f3a5c0a5250 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3a70102030 tx=0x7f3a6c0073d0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:24.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.833+0000 7f3a828deb80 1 Processor -- start 2024-06-09T22:28:24.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.833+0000 7f3a828deb80 1 -- start start 2024-06-09T22:28:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a828deb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 0x55e1443afa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a828deb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a828deb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443a9f20 0x55e1443aa340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a828deb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e1442fb810 con 0x55e1443a9f20 2024-06-09T22:28:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a828deb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e1442a3720 con 0x55e1443aff60 2024-06-09T22:28:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a828deb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e1442f77d0 con 0x55e1443af640 2024-06-09T22:28:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a75df7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 0x55e1443afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a765f8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443a9f20 0x55e1443aa340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a75df7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 0x55e1443afa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:49546/0 (socket says 172.21.15.104:49546) 2024-06-09T22:28:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a75df7700 1 -- 172.21.15.104:0/1358626817 learned_addr learned my addr 172.21.15.104:0/1358626817 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a755f6700 1 --2- 172.21.15.104:0/1358626817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a75df7700 1 -- 172.21.15.104:0/1358626817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 msgr2=0x55e1443a99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a75df7700 1 --2- 172.21.15.104:0/1358626817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443a99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a75df7700 1 -- 172.21.15.104:0/1358626817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443a9f20 msgr2=0x55e1443aa340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a75df7700 1 --2- 172.21.15.104:0/1358626817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443a9f20 0x55e1443aa340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a75df7700 1 -- 172.21.15.104:0/1358626817 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e144299f50 con 0x55e1443af640 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a765f8700 1 --2- 172.21.15.104:0/1358626817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443a9f20 0x55e1443aa340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.834+0000 7f3a75df7700 1 --2- 172.21.15.104:0/1358626817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 0x55e1443afa20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3a600a4360 tx=0x7f3a6009baf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e48cf2bab0f755fb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.835+0000 7f3a5bfff700 1 -- 172.21.15.104:0/1358626817 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6009bc20 con 0x55e1443af640 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.835+0000 7f3a5bfff700 1 -- 172.21.15.104:0/1358626817 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6009ffd0 con 0x55e1443af640 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.835+0000 7f3a5bfff700 1 -- 172.21.15.104:0/1358626817 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a60004a40 con 0x55e1443af640 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.835+0000 7f3a828deb80 1 -- 172.21.15.104:0/1358626817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 msgr2=0x55e1443afa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.835+0000 7f3a828deb80 1 --2- 172.21.15.104:0/1358626817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 0x55e1443afa20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3a600a4360 tx=0x7f3a6009baf0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.835+0000 7f3a828deb80 1 -- 172.21.15.104:0/1358626817 shutdown_connections 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.835+0000 7f3a828deb80 1 --2- 172.21.15.104:0/1358626817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443a9f20 0x55e1443aa340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.835+0000 7f3a828deb80 1 --2- 172.21.15.104:0/1358626817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 0x55e1443afa20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.835+0000 7f3a828deb80 1 --2- 172.21.15.104:0/1358626817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443a99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.835+0000 7f3a828deb80 1 -- 172.21.15.104:0/1358626817 >> 172.21.15.104:0/1358626817 conn(0x55e1442d4cc0 msgr2=0x55e1443a4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.835+0000 7f3a828deb80 1 -- 172.21.15.104:0/1358626817 shutdown_connections 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.835+0000 7f3a828deb80 1 -- 172.21.15.104:0/1358626817 wait complete. 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a828deb80 1 -- start start 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a828deb80 1 Processor -- start 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a828deb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443a9f20 0x55e1443b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a828deb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 0x55e1443b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a828deb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443aff60 0x55e1443b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a828deb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e1443bade0 con 0x55e1443aff60 2024-06-09T22:28:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a828deb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e14429bbf0 con 0x55e1443a9f20 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a828deb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e1443baf60 con 0x55e1443af640 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a828deb80 10 client.0 _ll_register_callbacks cb 0x55e144295920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a75df7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443a9f20 0x55e1443b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a755f6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 0x55e1443b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a765f8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443aff60 0x55e1443b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a75df7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443a9f20 0x55e1443b5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:34470/0 (socket says 172.21.15.104:34470) 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a755f6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 0x55e1443b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:49560/0 (socket says 172.21.15.104:49560) 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.836+0000 7f3a75df7700 1 -- 172.21.15.104:0/480226608 learned_addr learned my addr 172.21.15.104:0/480226608 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a75df7700 1 -- 172.21.15.104:0/480226608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 msgr2=0x55e1443b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a75df7700 1 --2- 172.21.15.104:0/480226608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 0x55e1443b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a75df7700 1 -- 172.21.15.104:0/480226608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443aff60 msgr2=0x55e1443b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a75df7700 1 --2- 172.21.15.104:0/480226608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443aff60 0x55e1443b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a75df7700 1 -- 172.21.15.104:0/480226608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e14424a8d0 con 0x55e1443a9f20 2024-06-09T22:28:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a75df7700 1 --2- 172.21.15.104:0/480226608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443a9f20 0x55e1443b5d80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3a600cd3d0 tx=0x7f3a6000cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=376038593ea6d279 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a597fa700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a597fa700 1 -- 172.21.15.104:0/480226608 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a600ba050 con 0x55e1443a9f20 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a828deb80 1 -- 172.21.15.104:0/480226608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a6009ffd0 con 0x55e1443a9f20 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a597fa700 1 -- 172.21.15.104:0/480226608 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f3a60098290 con 0x55e1443a9f20 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a597fa700 1 -- 172.21.15.104:0/480226608 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3a6009bc20 con 0x55e1443a9f20 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a828deb80 10 client.26702 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a828deb80 1 -- 172.21.15.104:0/480226608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e1443c5790 con 0x55e1443a9f20 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.837+0000 7f3a597fa700 1 -- 172.21.15.104:0/480226608 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6009ffd0 con 0x55e1443a9f20 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.838+0000 7f3a597fa700 1 -- 172.21.15.104:0/480226608 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3a6009b070 con 0x55e1443a9f20 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.838+0000 7f3a828deb80 10 client.26702 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.838+0000 7f3a828deb80 10 client.26702 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.838+0000 7f3a828deb80 20 client.26702 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.838+0000 7f3a828deb80 1 --2- 172.21.15.104:0/480226608 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55e1443c7c60 0x55e1443ca050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.838+0000 7f3a828deb80 4 client.26702 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:24.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.838+0000 7f3a828deb80 1 -- 172.21.15.104:0/480226608 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e144308d10 con 0x55e1443c7c60 2024-06-09T22:28:24.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.838+0000 7f3a755f6700 1 --2- 172.21.15.104:0/480226608 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55e1443c7c60 0x55e1443ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.838+0000 7f3a755f6700 1 --2- 172.21.15.104:0/480226608 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55e1443c7c60 0x55e1443ca050 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ab979a06ada69694 server_cookie=e6553885b092657d in_seq=0 out_seq=0 2024-06-09T22:28:24.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.838+0000 7f3a597fa700 10 client.26702 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:28:24.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a597fa700 1 -- 172.21.15.104:0/480226608 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55e144308d10 con 0x55e1443c7c60 2024-06-09T22:28:24.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a597fa700 10 client.26702 handle_command_reply: tid=0 2024-06-09T22:28:24.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a828deb80 1 client.26702 shutdown 2024-06-09T22:28:24.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a828deb80 1 -- 172.21.15.104:0/480226608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443a9f20 msgr2=0x55e1443b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:24.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a828deb80 1 --2- 172.21.15.104:0/480226608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443a9f20 0x55e1443b5d80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3a600cd3d0 tx=0x7f3a6000cf90 comp rx=0 tx=0).stop 2024-06-09T22:28:24.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a828deb80 1 -- 172.21.15.104:0/480226608 shutdown_connections 2024-06-09T22:28:24.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a828deb80 1 --2- 172.21.15.104:0/480226608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443aff60 0x55e1443b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a828deb80 1 --2- 172.21.15.104:0/480226608 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55e1443c7c60 0x55e1443ca050 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a828deb80 1 --2- 172.21.15.104:0/480226608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443a9f20 0x55e1443b5d80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a828deb80 1 --2- 172.21.15.104:0/480226608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443af640 0x55e1443b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a828deb80 1 -- 172.21.15.104:0/480226608 >> 172.21.15.104:0/480226608 conn(0x55e1442d4cc0 msgr2=0x55e1443a89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:24.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a597fa700 0 client.26702 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:28:24.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a828deb80 1 -- 172.21.15.104:0/480226608 shutdown_connections 2024-06-09T22:28:24.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a828deb80 1 -- 172.21.15.104:0/480226608 wait complete. 2024-06-09T22:28:24.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.839+0000 7f3a828deb80 20 client.26702 trim_cache size 0 max 16384 2024-06-09T22:28:24.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a828deb80 1 Processor -- start 2024-06-09T22:28:24.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a828deb80 1 -- start start 2024-06-09T22:28:24.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a828deb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443a9f20 0x55e1443a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a828deb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 0x55e14439e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a828deb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443ad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a828deb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3a701ac410 con 0x55e1443af640 2024-06-09T22:28:24.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a828deb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a701adf50 con 0x55e1443aff60 2024-06-09T22:28:24.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a828deb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3a701acae0 con 0x55e1443a9f20 2024-06-09T22:28:24.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a755f6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 0x55e14439e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a755f6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 0x55e14439e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48792/0 (socket says 172.21.15.104:48792) 2024-06-09T22:28:24.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a755f6700 1 -- 172.21.15.104:0/2145117794 learned_addr learned my addr 172.21.15.104:0/2145117794 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:24.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a75df7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443a9f20 0x55e1443a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.855+0000 7f3a765f8700 1 --2- 172.21.15.104:0/2145117794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.856+0000 7f3a755f6700 1 -- 172.21.15.104:0/2145117794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443a9f20 msgr2=0x55e1443a99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:24.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.856+0000 7f3a755f6700 1 --2- 172.21.15.104:0/2145117794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443a9f20 0x55e1443a99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.856+0000 7f3a755f6700 1 -- 172.21.15.104:0/2145117794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 msgr2=0x55e1443ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:24.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.856+0000 7f3a755f6700 1 --2- 172.21.15.104:0/2145117794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.856+0000 7f3a755f6700 1 -- 172.21.15.104:0/2145117794 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1443afa20 con 0x55e1443af640 2024-06-09T22:28:24.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.856+0000 7f3a765f8700 1 --2- 172.21.15.104:0/2145117794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:24.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.856+0000 7f3a755f6700 1 --2- 172.21.15.104:0/2145117794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 0x55e14439e690 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0079c0 tx=0x7f3a6c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6ea347938b27d6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:24.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.856+0000 7f3a5b7fe700 1 -- 172.21.15.104:0/2145117794 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c02e070 con 0x55e1443af640 2024-06-09T22:28:24.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.856+0000 7f3a5b7fe700 1 -- 172.21.15.104:0/2145117794 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6c015070 con 0x55e1443af640 2024-06-09T22:28:24.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.856+0000 7f3a5b7fe700 1 -- 172.21.15.104:0/2145117794 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c029a10 con 0x55e1443af640 2024-06-09T22:28:24.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.857+0000 7f3a828deb80 1 -- 172.21.15.104:0/2145117794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 msgr2=0x55e14439e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.857+0000 7f3a828deb80 1 --2- 172.21.15.104:0/2145117794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 0x55e14439e690 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0079c0 tx=0x7f3a6c016000 comp rx=0 tx=0).stop 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.857+0000 7f3a828deb80 1 -- 172.21.15.104:0/2145117794 shutdown_connections 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.857+0000 7f3a828deb80 1 --2- 172.21.15.104:0/2145117794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.857+0000 7f3a828deb80 1 --2- 172.21.15.104:0/2145117794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443a9f20 0x55e1443a99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.857+0000 7f3a828deb80 1 --2- 172.21.15.104:0/2145117794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 0x55e14439e690 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.857+0000 7f3a828deb80 1 -- 172.21.15.104:0/2145117794 >> 172.21.15.104:0/2145117794 conn(0x55e1442d4cc0 msgr2=0x55e1443b0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.857+0000 7f3a828deb80 1 -- 172.21.15.104:0/2145117794 shutdown_connections 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.857+0000 7f3a828deb80 1 -- 172.21.15.104:0/2145117794 wait complete. 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.857+0000 7f3a828deb80 1 -- start start 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.857+0000 7f3a828deb80 1 Processor -- start 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a828deb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443a9f20 0x55e1443ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a828deb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 0x55e1443cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a828deb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443ca250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a828deb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3a701ad880 con 0x55e1443af640 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a828deb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a701ad1b0 con 0x55e1443aff60 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a828deb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3a701aa3d0 con 0x55e1443a9f20 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a828deb80 10 client.0 _ll_register_callbacks cb 0x55e144295920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a755f6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 0x55e1443cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a75df7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443a9f20 0x55e1443ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a755f6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 0x55e1443cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48806/0 (socket says 172.21.15.104:48806) 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a75df7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443a9f20 0x55e1443ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:49584/0 (socket says 172.21.15.104:49584) 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a755f6700 1 -- 172.21.15.104:0/1272237305 learned_addr learned my addr 172.21.15.104:0/1272237305 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a765f8700 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a755f6700 1 -- 172.21.15.104:0/1272237305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443a9f20 msgr2=0x55e1443ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a755f6700 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443a9f20 0x55e1443ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a755f6700 1 -- 172.21.15.104:0/1272237305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 msgr2=0x55e1443ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a755f6700 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a755f6700 1 -- 172.21.15.104:0/1272237305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1443caa60 con 0x55e1443af640 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a75df7700 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443a9f20 0x55e1443ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a755f6700 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 0x55e1443cd2f0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c010f10 tx=0x7f3a6c004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca752a187aa4c11 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a597fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a597fa700 1 -- 172.21.15.104:0/1272237305 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c037070 con 0x55e1443af640 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a597fa700 1 -- 172.21.15.104:0/1272237305 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f3a6c029610 con 0x55e1443af640 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.858+0000 7f3a828deb80 1 -- 172.21.15.104:0/1272237305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a6c015070 con 0x55e1443af640 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.859+0000 7f3a597fa700 1 -- 172.21.15.104:0/1272237305 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3a6c02e040 con 0x55e1443af640 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.859+0000 7f3a828deb80 10 client.17478 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.859+0000 7f3a828deb80 1 -- 172.21.15.104:0/1272237305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e144432100 con 0x55e1443af640 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.859+0000 7f3a597fa700 1 -- 172.21.15.104:0/1272237305 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6c015070 con 0x55e1443af640 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.859+0000 7f3a597fa700 1 -- 172.21.15.104:0/1272237305 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3a6c011440 con 0x55e1443af640 2024-06-09T22:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.859+0000 7f3a828deb80 10 client.17478 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.859+0000 7f3a828deb80 10 client.17478 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.859+0000 7f3a828deb80 20 client.17478 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.859+0000 7f3a828deb80 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55e144432700 0x55e144434b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.859+0000 7f3a828deb80 4 client.17478 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.859+0000 7f3a828deb80 1 -- 172.21.15.104:0/1272237305 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e144303290 con 0x55e144432700 2024-06-09T22:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.859+0000 7f3a75df7700 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55e144432700 0x55e144434b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a75df7700 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55e144432700 0x55e144434b20 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=21acddc53175967 server_cookie=dc03d0718cc7c77a in_seq=0 out_seq=0 2024-06-09T22:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a597fa700 10 client.17478 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a597fa700 1 -- 172.21.15.104:0/1272237305 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e144303290 con 0x55e144432700 2024-06-09T22:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a597fa700 10 client.17478 handle_command_reply: tid=0 2024-06-09T22:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a828deb80 1 client.17478 shutdown 2024-06-09T22:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a828deb80 1 -- 172.21.15.104:0/1272237305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 msgr2=0x55e1443cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:24.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a828deb80 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 0x55e1443cd2f0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c010f10 tx=0x7f3a6c004610 comp rx=0 tx=0).stop 2024-06-09T22:28:24.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a828deb80 1 -- 172.21.15.104:0/1272237305 shutdown_connections 2024-06-09T22:28:24.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a828deb80 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55e144432700 0x55e144434b20 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a828deb80 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e1443aff60 0x55e1443ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a828deb80 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1443a9f20 0x55e1443ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a597fa700 0 client.17478 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:28:24.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a828deb80 1 --2- 172.21.15.104:0/1272237305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e1443af640 0x55e1443cd2f0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a828deb80 1 -- 172.21.15.104:0/1272237305 >> 172.21.15.104:0/1272237305 conn(0x55e1442d4cc0 msgr2=0x55e14439e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:24.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a828deb80 1 -- 172.21.15.104:0/1272237305 shutdown_connections 2024-06-09T22:28:24.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a828deb80 1 -- 172.21.15.104:0/1272237305 wait complete. 2024-06-09T22:28:24.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.860+0000 7f3a828deb80 20 client.17478 trim_cache size 0 max 16384 2024-06-09T22:28:24.868 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:28:24.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.861+0000 7f3a7805b700 1 -- 172.21.15.104:0/2605208349 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f3a5c0a2d90 msgr2=0x7f3a5c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:24.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.861+0000 7f3a7805b700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f3a5c0a2d90 0x7f3a5c0a5250 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3a70102030 tx=0x7f3a6c0073d0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.861+0000 7f3a7805b700 1 -- 172.21.15.104:0/2605208349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a70102740 msgr2=0x7f3a7019e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:24.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.861+0000 7f3a7805b700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a70102740 0x7f3a7019e7c0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000e9c0 tx=0x7f3a6000ecd0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.861+0000 7f3a7805b700 1 -- 172.21.15.104:0/2605208349 shutdown_connections 2024-06-09T22:28:24.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.861+0000 7f3a7805b700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f3a5c0a2d90 0x7f3a5c0a5250 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.861+0000 7f3a7805b700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a70102740 0x7f3a7019e7c0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.861+0000 7f3a7805b700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a70107840 0x7f3a70198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.861+0000 7f3a7805b700 1 --2- 172.21.15.104:0/2605208349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a701030f0 0x7f3a7019ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:24.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.861+0000 7f3a7805b700 1 -- 172.21.15.104:0/2605208349 >> 172.21.15.104:0/2605208349 conn(0x7f3a700795f0 msgr2=0x7f3a701006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:24.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.861+0000 7f3a7805b700 1 -- 172.21.15.104:0/2605208349 shutdown_connections 2024-06-09T22:28:24.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:24.861+0000 7f3a7805b700 1 -- 172.21.15.104:0/2605208349 wait complete. 2024-06-09T22:28:25.664 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-09T22:28:25.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 260745 4.83 MB/sec execute 211 sec latency 28.431 ms 2024-06-09T22:28:25.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:25 smithi194 ceph-mon[36856]: pgmap v228: 129 pgs: 129 active+clean; 201 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 307 op/s 2024-06-09T22:28:25.922 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:28:25.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:25 smithi153 ceph-mon[36599]: pgmap v228: 129 pgs: 129 active+clean; 201 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 307 op/s 2024-06-09T22:28:26.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:25 smithi104 ceph-mon[32389]: pgmap v228: 129 pgs: 129 active+clean; 201 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 307 op/s 2024-06-09T22:28:26.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 262882 4.87 MB/sec execute 212 sec latency 20.537 ms 2024-06-09T22:28:26.942 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.940+0000 7f6c625c1640 1 -- 172.21.15.104:0/2250927722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c5c1012c0 msgr2=0x7f6c5c10d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:26.942 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.940+0000 7f6c625c1640 1 --2- 172.21.15.104:0/2250927722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c5c1012c0 0x7f6c5c10d780 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009cd0 tx=0x7f6c5402a8e0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.942 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.940+0000 7f6c625c1640 1 -- 172.21.15.104:0/2250927722 shutdown_connections 2024-06-09T22:28:26.942 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.940+0000 7f6c625c1640 1 --2- 172.21.15.104:0/2250927722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c5c1012c0 0x7f6c5c10d780 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.942 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.940+0000 7f6c625c1640 1 --2- 172.21.15.104:0/2250927722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c5c100900 0x7f6c5c100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.940+0000 7f6c625c1640 1 --2- 172.21.15.104:0/2250927722 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c1068c0 0x7f6c5c106ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.940+0000 7f6c625c1640 1 -- 172.21.15.104:0/2250927722 >> 172.21.15.104:0/2250927722 conn(0x7f6c5c077ab0 msgr2=0x7f6c5c077ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:26.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.940+0000 7f6c625c1640 1 -- 172.21.15.104:0/2250927722 shutdown_connections 2024-06-09T22:28:26.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.940+0000 7f6c625c1640 1 -- 172.21.15.104:0/2250927722 wait complete. 2024-06-09T22:28:26.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.940+0000 7f6c625c1640 1 Processor -- start 2024-06-09T22:28:26.943 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.940+0000 7f6c625c1640 1 -- start start 2024-06-09T22:28:26.944 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c625c1640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c5c100900 0x7f6c5c198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c625c1640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c1012c0 0x7f6c5c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c625c1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c5c1068c0 0x7f6c5c19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c625c1640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c110cd0 con 0x7f6c5c100900 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c625c1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c110b50 con 0x7f6c5c1068c0 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c625c1640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c110e50 con 0x7f6c5c1012c0 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c5bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c5c100900 0x7f6c5c198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c5bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c5c100900 0x7f6c5c198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58318/0 (socket says 172.21.15.104:58318) 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c5bfff640 1 -- 172.21.15.104:0/136341505 learned_addr learned my addr 172.21.15.104:0/136341505 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c5b7fe640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c1012c0 0x7f6c5c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c5c1068c0 0x7f6c5c19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c5bfff640 1 -- 172.21.15.104:0/136341505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c1012c0 msgr2=0x7f6c5c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c5bfff640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c1012c0 0x7f6c5c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c5bfff640 1 -- 172.21.15.104:0/136341505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c5c1068c0 msgr2=0x7f6c5c19cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:26.945 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c5bfff640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c5c1068c0 0x7f6c5c19cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c5bfff640 1 -- 172.21.15.104:0/136341505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c5c19d210 con 0x7f6c5c100900 2024-06-09T22:28:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c5bfff640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c5c100900 0x7f6c5c198220 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00e9f0 tx=0x7f6c4c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c597fa640 1 -- 172.21.15.104:0/136341505 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c00cd80 con 0x7f6c5c100900 2024-06-09T22:28:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c625c1640 1 -- 172.21.15.104:0/136341505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c5c19d500 con 0x7f6c5c100900 2024-06-09T22:28:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c597fa640 1 -- 172.21.15.104:0/136341505 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c4c005500 con 0x7f6c5c100900 2024-06-09T22:28:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.941+0000 7f6c597fa640 1 -- 172.21.15.104:0/136341505 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c010690 con 0x7f6c5c100900 2024-06-09T22:28:26.946 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.942+0000 7f6c625c1640 1 -- 172.21.15.104:0/136341505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c5c1a4d90 con 0x7f6c5c100900 2024-06-09T22:28:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c597fa640 1 -- 172.21.15.104:0/136341505 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f6c4c0046c0 con 0x7f6c5c100900 2024-06-09T22:28:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c653af740 1 Processor -- start 2024-06-09T22:28:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c653af740 1 -- start start 2024-06-09T22:28:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c653af740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d16170 0x55a350d16550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c653af740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16a90 0x55a350d20e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c653af740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d21350 0x55a350d23740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c653af740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a350c7a4f0 con 0x55a350d16170 2024-06-09T22:28:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c653af740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a350c18870 con 0x55a350d21350 2024-06-09T22:28:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c653af740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a350c764b0 con 0x55a350d16a90 2024-06-09T22:28:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c597fa640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6c440a2da0 0x7f6c440a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c60b5e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d16170 0x55a350d16550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c5b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d21350 0x55a350d23740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c5bfff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16a90 0x55a350d20e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c60b5e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d16170 0x55a350d16550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58326/0 (socket says 172.21.15.104:58326) 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c5b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d21350 0x55a350d23740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:51458/0 (socket says 172.21.15.104:51458) 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c60b5e640 1 -- 172.21.15.104:0/516781880 learned_addr learned my addr 172.21.15.104:0/516781880 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.944+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6c440a2da0 0x7f6c440a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c597fa640 1 -- 172.21.15.104:0/136341505 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f6c4c0cfbc0 con 0x7f6c5c100900 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c5b7fe640 1 -- 172.21.15.104:0/516781880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16a90 msgr2=0x55a350d20e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c5b7fe640 1 --2- 172.21.15.104:0/516781880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16a90 0x55a350d20e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c5b7fe640 1 -- 172.21.15.104:0/516781880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d16170 msgr2=0x55a350d16550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c5b7fe640 1 --2- 172.21.15.104:0/516781880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d16170 0x55a350d16550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c5b7fe640 1 -- 172.21.15.104:0/516781880 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a350c349b0 con 0x55a350d21350 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c5bfff640 1 --2- 172.21.15.104:0/516781880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16a90 0x55a350d20e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/516781880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d16170 0x55a350d16550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6c440a2da0 0x7f6c440a5260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009430 tx=0x7f6c54004a40 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c5b7fe640 1 --2- 172.21.15.104:0/516781880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d21350 0x55a350d23740 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6c50009bf0 tx=0x7f6c50011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12fdd842630adfdb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c427fc640 1 -- 172.21.15.104:0/516781880 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c50004050 con 0x55a350d21350 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c427fc640 1 -- 172.21.15.104:0/516781880 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c50005500 con 0x55a350d21350 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.945+0000 7f6c427fc640 1 -- 172.21.15.104:0/516781880 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c50004e00 con 0x55a350d21350 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.946+0000 7f6c653af740 1 -- 172.21.15.104:0/516781880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d21350 msgr2=0x55a350d23740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.946+0000 7f6c653af740 1 --2- 172.21.15.104:0/516781880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d21350 0x55a350d23740 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6c50009bf0 tx=0x7f6c50011810 comp rx=0 tx=0).stop 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.946+0000 7f6c653af740 1 -- 172.21.15.104:0/516781880 shutdown_connections 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.946+0000 7f6c653af740 1 --2- 172.21.15.104:0/516781880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d21350 0x55a350d23740 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.946+0000 7f6c653af740 1 --2- 172.21.15.104:0/516781880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16a90 0x55a350d20e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.946+0000 7f6c653af740 1 --2- 172.21.15.104:0/516781880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d16170 0x55a350d16550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.946+0000 7f6c653af740 1 -- 172.21.15.104:0/516781880 >> 172.21.15.104:0/516781880 conn(0x55a350d12680 msgr2=0x55a350d12a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.946+0000 7f6c653af740 1 -- 172.21.15.104:0/516781880 shutdown_connections 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.946+0000 7f6c653af740 1 -- 172.21.15.104:0/516781880 wait complete. 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.946+0000 7f6c653af740 1 -- start start 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.946+0000 7f6c653af740 1 Processor -- start 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c653af740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d1e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c653af740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 0x55a350d1e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c653af740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d1ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c653af740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a350d23f80 con 0x55a350d21350 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c653af740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a350c03ae0 con 0x55a350d16a90 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c653af740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a350d24100 con 0x55a350d16170 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c653af740 10 client.0 _ll_register_callbacks cb 0x55a350c3c9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c60b5e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d1ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c5bfff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d1e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c60b5e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d1ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58338/0 (socket says 172.21.15.104:58338) 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c5b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 0x55a350d1e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c60b5e640 1 -- 172.21.15.104:0/876867109 learned_addr learned my addr 172.21.15.104:0/876867109 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c60b5e640 1 -- 172.21.15.104:0/876867109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 msgr2=0x55a350d1e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/876867109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d1e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c60b5e640 1 -- 172.21.15.104:0/876867109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 msgr2=0x55a350d1e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/876867109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 0x55a350d1e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c60b5e640 1 -- 172.21.15.104:0/876867109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a35098e210 con 0x55a350d21350 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c5bfff640 1 --2- 172.21.15.104:0/876867109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d1e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/876867109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d1ec80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f6c54031ea0 tx=0x7f6c540095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ee4f565d7fa07c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c3b7fe640 1 -- 172.21.15.104:0/876867109 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c540046d0 con 0x55a350d21350 2024-06-09T22:28:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c3b7fe640 1 -- 172.21.15.104:0/876867109 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f6c540305a0 con 0x55a350d21350 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.947+0000 7f6c653af740 1 -- 172.21.15.104:0/876867109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c50005500 con 0x55a350d21350 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.948+0000 7f6c3b7fe640 1 -- 172.21.15.104:0/876867109 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6c540307a0 con 0x55a350d21350 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.948+0000 7f6c653af740 10 client.17490 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.948+0000 7f6c653af740 1 -- 172.21.15.104:0/876867109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a350a5a620 con 0x55a350d21350 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.948+0000 7f6c3b7fe640 1 -- 172.21.15.104:0/876867109 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c50005500 con 0x55a350d21350 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.948+0000 7f6c3b7fe640 1 -- 172.21.15.104:0/876867109 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6c54048040 con 0x55a350d21350 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.948+0000 7f6c653af740 10 client.17490 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.948+0000 7f6c653af740 10 client.17490 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.948+0000 7f6c653af740 20 client.17490 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.948+0000 7f6c653af740 1 --2- 172.21.15.104:0/876867109 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a350d313f0 0x55a350d337e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.948+0000 7f6c653af740 4 client.17490 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.948+0000 7f6c653af740 1 -- 172.21.15.104:0/876867109 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a350c8df10 con 0x55a350d313f0 2024-06-09T22:28:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.948+0000 7f6c5bfff640 1 --2- 172.21.15.104:0/876867109 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a350d313f0 0x55a350d337e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.949+0000 7f6c3b7fe640 10 client.17490 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.949+0000 7f6c5bfff640 1 --2- 172.21.15.104:0/876867109 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a350d313f0 0x55a350d337e0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceedcff6022c63db server_cookie=932715cc09761374 in_seq=0 out_seq=0 2024-06-09T22:28:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.949+0000 7f6c3b7fe640 1 -- 172.21.15.104:0/876867109 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a350c8df10 con 0x55a350d313f0 2024-06-09T22:28:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.949+0000 7f6c3b7fe640 10 client.17490 handle_command_reply: tid=0 2024-06-09T22:28:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.949+0000 7f6c653af740 1 client.17490 shutdown 2024-06-09T22:28:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.950+0000 7f6c653af740 1 -- 172.21.15.104:0/876867109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 msgr2=0x55a350d1ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.950+0000 7f6c653af740 1 --2- 172.21.15.104:0/876867109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d1ec80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f6c54031ea0 tx=0x7f6c540095c0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.950+0000 7f6c653af740 1 -- 172.21.15.104:0/876867109 shutdown_connections 2024-06-09T22:28:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.950+0000 7f6c653af740 1 --2- 172.21.15.104:0/876867109 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a350d313f0 0x55a350d337e0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.950+0000 7f6c653af740 1 --2- 172.21.15.104:0/876867109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d1ec80 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.950+0000 7f6c653af740 1 --2- 172.21.15.104:0/876867109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 0x55a350d1e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.950+0000 7f6c3b7fe640 0 client.17490 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:26.959 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.950+0000 7f6c653af740 1 --2- 172.21.15.104:0/876867109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d1e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.959 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.950+0000 7f6c653af740 1 -- 172.21.15.104:0/876867109 >> 172.21.15.104:0/876867109 conn(0x55a350d12680 msgr2=0x55a350d18400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:26.959 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.950+0000 7f6c653af740 1 -- 172.21.15.104:0/876867109 shutdown_connections 2024-06-09T22:28:26.959 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.950+0000 7f6c653af740 1 -- 172.21.15.104:0/876867109 wait complete. 2024-06-09T22:28:26.959 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.950+0000 7f6c653af740 20 client.17490 trim_cache size 0 max 16384 2024-06-09T22:28:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.965+0000 7f6c653af740 1 Processor -- start 2024-06-09T22:28:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.965+0000 7f6c653af740 1 -- start start 2024-06-09T22:28:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.965+0000 7f6c653af740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d18b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.965+0000 7f6c653af740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 0x55a350d1d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.965+0000 7f6c653af740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d29730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.965+0000 7f6c653af740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c1aeec0 con 0x55a350d21350 2024-06-09T22:28:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.965+0000 7f6c653af740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c1b0a00 con 0x55a350d16a90 2024-06-09T22:28:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.965+0000 7f6c653af740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c1af590 con 0x55a350d16170 2024-06-09T22:28:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.965+0000 7f6c60b5e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d29730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c5bfff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d18b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c60b5e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d29730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58352/0 (socket says 172.21.15.104:58352) 2024-06-09T22:28:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c60b5e640 1 -- 172.21.15.104:0/1236283326 learned_addr learned my addr 172.21.15.104:0/1236283326 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c5b7fe640 1 --2- 172.21.15.104:0/1236283326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 0x55a350d1d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c60b5e640 1 -- 172.21.15.104:0/1236283326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 msgr2=0x55a350d18b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/1236283326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d18b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c60b5e640 1 -- 172.21.15.104:0/1236283326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 msgr2=0x55a350d1d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/1236283326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 0x55a350d1d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c60b5e640 1 -- 172.21.15.104:0/1236283326 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a350d2d1e0 con 0x55a350d21350 2024-06-09T22:28:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/1236283326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d29730 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f6c54031730 tx=0x7f6c5402a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abacf2cba03f0e9c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c41ffb640 1 -- 172.21.15.104:0/1236283326 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c54065070 con 0x55a350d21350 2024-06-09T22:28:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c41ffb640 1 -- 172.21.15.104:0/1236283326 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c54036070 con 0x55a350d21350 2024-06-09T22:28:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.966+0000 7f6c41ffb640 1 -- 172.21.15.104:0/1236283326 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c54054b20 con 0x55a350d21350 2024-06-09T22:28:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.967+0000 7f6c653af740 1 -- 172.21.15.104:0/1236283326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 msgr2=0x55a350d29730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.967+0000 7f6c653af740 1 --2- 172.21.15.104:0/1236283326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d29730 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f6c54031730 tx=0x7f6c5402a8e0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.967+0000 7f6c653af740 1 -- 172.21.15.104:0/1236283326 shutdown_connections 2024-06-09T22:28:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.967+0000 7f6c653af740 1 --2- 172.21.15.104:0/1236283326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d29730 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.967+0000 7f6c653af740 1 --2- 172.21.15.104:0/1236283326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 0x55a350d1d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.967+0000 7f6c653af740 1 --2- 172.21.15.104:0/1236283326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d18b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.967+0000 7f6c653af740 1 -- 172.21.15.104:0/1236283326 >> 172.21.15.104:0/1236283326 conn(0x55a350d12680 msgr2=0x55a350d1cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.967+0000 7f6c653af740 1 -- 172.21.15.104:0/1236283326 shutdown_connections 2024-06-09T22:28:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.967+0000 7f6c653af740 1 -- 172.21.15.104:0/1236283326 wait complete. 2024-06-09T22:28:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.967+0000 7f6c653af740 1 -- start start 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.967+0000 7f6c653af740 1 Processor -- start 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c653af740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d38700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c653af740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 0x55a350d38c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c653af740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d9e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c653af740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c1b0330 con 0x55a350d21350 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c653af740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c1afc60 con 0x55a350d16a90 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c653af740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c1acec0 con 0x55a350d16170 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c653af740 10 client.0 _ll_register_callbacks cb 0x7f6c4c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c60b5e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d9e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c60b5e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d9e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58368/0 (socket says 172.21.15.104:58368) 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c60b5e640 1 -- 172.21.15.104:0/618696032 learned_addr learned my addr 172.21.15.104:0/618696032 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c5bfff640 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d38700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c5b7fe640 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 0x55a350d38c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c60b5e640 1 -- 172.21.15.104:0/618696032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 msgr2=0x55a350d38700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d38700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c60b5e640 1 -- 172.21.15.104:0/618696032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 msgr2=0x55a350d38c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 0x55a350d38c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c60b5e640 1 -- 172.21.15.104:0/618696032 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a350d9e630 con 0x55a350d21350 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c60b5e640 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d9e0f0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f6c54054a20 tx=0x7f6c540099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e566bc29e20608d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c3b7fe640 1 -- 172.21.15.104:0/618696032 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c54065070 con 0x55a350d21350 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c3b7fe640 1 -- 172.21.15.104:0/618696032 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f6c54054b20 con 0x55a350d21350 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.968+0000 7f6c653af740 1 -- 172.21.15.104:0/618696032 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c54036070 con 0x55a350d21350 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c3b7fe640 1 -- 172.21.15.104:0/618696032 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6c54071050 con 0x55a350d21350 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c653af740 10 client.17496 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c653af740 1 -- 172.21.15.104:0/618696032 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a350d9f840 con 0x55a350d21350 2024-06-09T22:28:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c3b7fe640 1 -- 172.21.15.104:0/618696032 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c54036070 con 0x55a350d21350 2024-06-09T22:28:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c3b7fe640 1 -- 172.21.15.104:0/618696032 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6c54030070 con 0x55a350d21350 2024-06-09T22:28:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c653af740 10 client.17496 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c653af740 10 client.17496 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c653af740 20 client.17496 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c653af740 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a350d9fea0 0x55a350da22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c653af740 4 client.17496 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:28:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c653af740 1 -- 172.21.15.104:0/618696032 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a350c87c20 con 0x55a350d9fea0 2024-06-09T22:28:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c5bfff640 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a350d9fea0 0x55a350da22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c5bfff640 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a350d9fea0 0x55a350da22c0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4acfca4374925559 server_cookie=d54ddfd0820dcf0e in_seq=0 out_seq=0 2024-06-09T22:28:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.969+0000 7f6c3b7fe640 10 client.17496 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c3b7fe640 1 -- 172.21.15.104:0/618696032 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a350c87c20 con 0x55a350d9fea0 2024-06-09T22:28:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c3b7fe640 10 client.17496 handle_command_reply: tid=0 2024-06-09T22:28:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c653af740 1 client.17496 shutdown 2024-06-09T22:28:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c653af740 1 -- 172.21.15.104:0/618696032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 msgr2=0x55a350d9e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c653af740 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d9e0f0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f6c54054a20 tx=0x7f6c540099a0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c653af740 1 -- 172.21.15.104:0/618696032 shutdown_connections 2024-06-09T22:28:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c653af740 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a350d9fea0 0x55a350da22c0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c653af740 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a350d21350 0x55a350d9e0f0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c653af740 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a350d16a90 0x55a350d38c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c3b7fe640 0 client.17496 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c653af740 1 --2- 172.21.15.104:0/618696032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a350d16170 0x55a350d38700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c653af740 1 -- 172.21.15.104:0/618696032 >> 172.21.15.104:0/618696032 conn(0x55a350d12680 msgr2=0x55a350d1d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c653af740 1 -- 172.21.15.104:0/618696032 shutdown_connections 2024-06-09T22:28:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c653af740 1 -- 172.21.15.104:0/618696032 wait complete. 2024-06-09T22:28:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.978+0000 7f6c653af740 20 client.17496 trim_cache size 0 max 16384 2024-06-09T22:28:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:28:26.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:28:26.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:28:26.982 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.979+0000 7f6c625c1640 1 -- 172.21.15.104:0/136341505 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6c440a2da0 msgr2=0x7f6c440a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.979+0000 7f6c625c1640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6c440a2da0 0x7f6c440a5260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009430 tx=0x7f6c54004a40 comp rx=0 tx=0).stop 2024-06-09T22:28:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.979+0000 7f6c625c1640 1 -- 172.21.15.104:0/136341505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c5c100900 msgr2=0x7f6c5c198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.979+0000 7f6c625c1640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c5c100900 0x7f6c5c198220 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00e9f0 tx=0x7f6c4c00edf0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.979+0000 7f6c625c1640 1 -- 172.21.15.104:0/136341505 shutdown_connections 2024-06-09T22:28:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.979+0000 7f6c625c1640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6c440a2da0 0x7f6c440a5260 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.979+0000 7f6c625c1640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c5c1068c0 0x7f6c5c19cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.979+0000 7f6c625c1640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c1012c0 0x7f6c5c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.979+0000 7f6c625c1640 1 --2- 172.21.15.104:0/136341505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c5c100900 0x7f6c5c198220 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.979+0000 7f6c625c1640 1 -- 172.21.15.104:0/136341505 >> 172.21.15.104:0/136341505 conn(0x7f6c5c077ab0 msgr2=0x7f6c5c06a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.979+0000 7f6c625c1640 1 -- 172.21.15.104:0/136341505 shutdown_connections 2024-06-09T22:28:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:26.979+0000 7f6c625c1640 1 -- 172.21.15.104:0/136341505 wait complete. 2024-06-09T22:28:27.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 264288 4.87 MB/sec execute 213 sec latency 52.460 ms 2024-06-09T22:28:27.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.737+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/3953928049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fb0108500 msgr2=0x7f7fb01088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:27.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.737+0000 7f7fb4a2d700 1 --2- 172.21.15.104:0/3953928049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fb0108500 0x7f7fb01088e0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0009990 tx=0x7f7fa0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.737+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/3953928049 shutdown_connections 2024-06-09T22:28:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.737+0000 7f7fb4a2d700 1 --2- 172.21.15.104:0/3953928049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fb0108500 0x7f7fb01088e0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.737+0000 7f7fb4a2d700 1 --2- 172.21.15.104:0/3953928049 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fb006ab30 0x7f7fb010f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.737+0000 7f7fb4a2d700 1 --2- 172.21.15.104:0/3953928049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb006a170 0x7f7fb006a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.737+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/3953928049 >> 172.21.15.104:0/3953928049 conn(0x7f7fb0078340 msgr2=0x7f7fb0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.737+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/3953928049 shutdown_connections 2024-06-09T22:28:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.737+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/3953928049 wait complete. 2024-06-09T22:28:27.741 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:27 smithi104 ceph-mon[32389]: pgmap v229: 129 pgs: 129 active+clean; 201 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 199 op/s 2024-06-09T22:28:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.738+0000 7f7fb4a2d700 1 Processor -- start 2024-06-09T22:28:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.738+0000 7f7fb4a2d700 1 -- start start 2024-06-09T22:28:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fb4a2d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb006a170 0x7f7fb019e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fb4a2d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fb006ab30 0x7f7fb019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fb4a2d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fb0108500 0x7f7fb0198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fb4a2d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb0112760 con 0x7f7fb006ab30 2024-06-09T22:28:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fb4a2d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01125e0 con 0x7f7fb006a170 2024-06-09T22:28:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fb4a2d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01128e0 con 0x7f7fb0108500 2024-06-09T22:28:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fae59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb006a170 0x7f7fb019e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7faed9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fb0108500 0x7f7fb0198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fae59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb006a170 0x7f7fb019e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:51502/0 (socket says 172.21.15.104:51502) 2024-06-09T22:28:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7faed9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fb0108500 0x7f7fb0198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:55798/0 (socket says 172.21.15.104:55798) 2024-06-09T22:28:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fae59c700 1 -- 172.21.15.104:0/2863369355 learned_addr learned my addr 172.21.15.104:0/2863369355 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:27.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fb006ab30 0x7f7fb019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fae59c700 1 -- 172.21.15.104:0/2863369355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fb0108500 msgr2=0x7f7fb0198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:27.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fae59c700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fb0108500 0x7f7fb0198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fae59c700 1 -- 172.21.15.104:0/2863369355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fb006ab30 msgr2=0x7f7fb019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:27.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fae59c700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fb006ab30 0x7f7fb019ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fae59c700 1 -- 172.21.15.104:0/2863369355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb01991a0 con 0x7f7fb006a170 2024-06-09T22:28:27.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.739+0000 7f7fae59c700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb006a170 0x7f7fb019e830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0000c00 tx=0x7f7fa000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:27.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.740+0000 7f7f9f7fe700 1 -- 172.21.15.104:0/2863369355 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0027070 con 0x7f7fb006a170 2024-06-09T22:28:27.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.740+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/2863369355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb0199430 con 0x7f7fb006a170 2024-06-09T22:28:27.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.740+0000 7f7f9f7fe700 1 -- 172.21.15.104:0/2863369355 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fa0003d70 con 0x7f7fb006a170 2024-06-09T22:28:27.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.740+0000 7f7f9f7fe700 1 -- 172.21.15.104:0/2863369355 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa000f460 con 0x7f7fb006a170 2024-06-09T22:28:27.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.740+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/2863369355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb01a5700 con 0x7f7fb006a170 2024-06-09T22:28:27.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.742+0000 7f7f9f7fe700 1 -- 172.21.15.104:0/2863369355 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7fa002a070 con 0x7f7fb006a170 2024-06-09T22:28:27.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.743+0000 7f7f9f7fe700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f980a2d90 0x7f7f980a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.743+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f980a2d90 0x7f7f980a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.743+0000 7f7f9f7fe700 1 -- 172.21.15.104:0/2863369355 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f7fa00d0650 con 0x7f7fb006a170 2024-06-09T22:28:27.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.743+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f980a2d90 0x7f7f980a5250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00788b0 tx=0x7f7fa4009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:27.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7fbf2b0b80 1 Processor -- start 2024-06-09T22:28:27.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7fbf2b0b80 1 -- start start 2024-06-09T22:28:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7fbf2b0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f1fff60 0x55dc9f200340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7fbf2b0b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f200880 0x55dc9f1fa260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7fbf2b0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f1fabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7fbf2b0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55dc9f0ecf60 con 0x55dc9f1fa7a0 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7fbf2b0b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dc9f14b850 con 0x55dc9f200880 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7fbf2b0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dc9f0b3460 con 0x55dc9f1fff60 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7fae59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f1fff60 0x55dc9f200340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7faed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f1fabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7fadd9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f200880 0x55dc9f1fa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7fae59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f1fff60 0x55dc9f200340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:55814/0 (socket says 172.21.15.104:55814) 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7faed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f1fabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58384/0 (socket says 172.21.15.104:58384) 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.744+0000 7f7fae59c700 1 -- 172.21.15.104:0/1272276210 learned_addr learned my addr 172.21.15.104:0/1272276210 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.745+0000 7f7fadd9b700 1 -- 172.21.15.104:0/1272276210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f1fff60 msgr2=0x55dc9f200340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.745+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/1272276210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f1fff60 0x55dc9f200340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.745+0000 7f7fadd9b700 1 -- 172.21.15.104:0/1272276210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 msgr2=0x55dc9f1fabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.745+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/1272276210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f1fabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.745+0000 7f7fadd9b700 1 -- 172.21.15.104:0/1272276210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc9f09a8d0 con 0x55dc9f200880 2024-06-09T22:28:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.745+0000 7f7fae59c700 1 --2- 172.21.15.104:0/1272276210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f1fff60 0x55dc9f200340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.745+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/1272276210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f200880 0x55dc9f1fa260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4011c90 tx=0x7f7fa4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be05090a01178afe server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.745+0000 7f7f9cff9700 1 -- 172.21.15.104:0/1272276210 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4004230 con 0x55dc9f200880 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.745+0000 7f7f9cff9700 1 -- 172.21.15.104:0/1272276210 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fa400ed20 con 0x55dc9f200880 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.745+0000 7f7f9cff9700 1 -- 172.21.15.104:0/1272276210 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4026470 con 0x55dc9f200880 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/1272276210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f200880 msgr2=0x55dc9f1fa260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/1272276210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f200880 0x55dc9f1fa260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4011c90 tx=0x7f7fa4014000 comp rx=0 tx=0).stop 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/1272276210 shutdown_connections 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/1272276210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f1fabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/1272276210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f1fff60 0x55dc9f200340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/1272276210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f200880 0x55dc9f1fa260 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/1272276210 >> 172.21.15.104:0/1272276210 conn(0x55dc9f127080 msgr2=0x55dc9f1f5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/1272276210 shutdown_connections 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/1272276210 wait complete. 2024-06-09T22:28:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 -- start start 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 Processor -- start 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f206600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fff60 0x55dc9f203950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f200880 0x55dc9f203e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55dc9f0ebfc0 con 0x55dc9f1fa7a0 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dc9f147810 con 0x55dc9f1fff60 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.746+0000 7f7fbf2b0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dc9f207260 con 0x55dc9f200880 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fbf2b0b80 10 client.0 _ll_register_callbacks cb 0x55dc9f0e15b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fae59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f206600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fae59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f206600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58394/0 (socket says 172.21.15.104:58394) 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fae59c700 1 -- 172.21.15.104:0/2146734711 learned_addr learned my addr 172.21.15.104:0/2146734711 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fff60 0x55dc9f203950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7faed9d700 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f200880 0x55dc9f203e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fae59c700 1 -- 172.21.15.104:0/2146734711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f200880 msgr2=0x55dc9f203e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fae59c700 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f200880 0x55dc9f203e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fae59c700 1 -- 172.21.15.104:0/2146734711 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fff60 msgr2=0x55dc9f203950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fae59c700 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fff60 0x55dc9f203950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fae59c700 1 -- 172.21.15.104:0/2146734711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc9f111750 con 0x55dc9f1fa7a0 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fff60 0x55dc9f203950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fae59c700 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f206600 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0006010 tx=0x7f7fa009c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501090709eb4aabe server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7f95ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7f95ffb700 1 -- 172.21.15.104:0/2146734711 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa009c7d0 con 0x55dc9f1fa7a0 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7f95ffb700 1 -- 172.21.15.104:0/2146734711 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f7fa0022700 con 0x55dc9f1fa7a0 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/2146734711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fa400ed20 con 0x55dc9f1fa7a0 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7f95ffb700 1 -- 172.21.15.104:0/2146734711 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7fa00a0b80 con 0x55dc9f1fa7a0 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fbf2b0b80 10 client.17508 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.747+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/2146734711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dc9f2119f0 con 0x55dc9f1fa7a0 2024-06-09T22:28:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.748+0000 7f7f95ffb700 1 -- 172.21.15.104:0/2146734711 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fa400ed20 con 0x55dc9f1fa7a0 2024-06-09T22:28:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.748+0000 7f7f95ffb700 1 -- 172.21.15.104:0/2146734711 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7fa009ca30 con 0x55dc9f1fa7a0 2024-06-09T22:28:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.748+0000 7f7fbf2b0b80 10 client.17508 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.748+0000 7f7fbf2b0b80 10 client.17508 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:28:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.748+0000 7f7fbf2b0b80 20 client.17508 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.748+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55dc9f213f10 0x55dc9f216320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.748+0000 7f7fbf2b0b80 4 client.17508 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.748+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/2146734711 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc9f158d50 con 0x55dc9f213f10 2024-06-09T22:28:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.748+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55dc9f213f10 0x55dc9f216320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.748+0000 7f7f95ffb700 10 client.17508 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:28:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.748+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55dc9f213f10 0x55dc9f216320 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e59445ea3ca6b144 server_cookie=2fcbee5b2bec2ba1 in_seq=0 out_seq=0 2024-06-09T22:28:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7f95ffb700 1 -- 172.21.15.104:0/2146734711 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55dc9f158d50 con 0x55dc9f213f10 2024-06-09T22:28:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7f95ffb700 10 client.17508 handle_command_reply: tid=0 2024-06-09T22:28:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7fbf2b0b80 1 client.17508 shutdown 2024-06-09T22:28:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/2146734711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 msgr2=0x55dc9f206600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f206600 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0006010 tx=0x7f7fa009c620 comp rx=0 tx=0).stop 2024-06-09T22:28:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/2146734711 shutdown_connections 2024-06-09T22:28:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55dc9f213f10 0x55dc9f216320 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f206600 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f200880 0x55dc9f203e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/2146734711 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fff60 0x55dc9f203950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/2146734711 >> 172.21.15.104:0/2146734711 conn(0x55dc9f127080 msgr2=0x55dc9f1f4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7f95ffb700 0 client.17508 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:28:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/2146734711 shutdown_connections 2024-06-09T22:28:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.749+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/2146734711 wait complete. 2024-06-09T22:28:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.750+0000 7f7fbf2b0b80 20 client.17508 trim_cache size 0 max 16384 2024-06-09T22:28:27.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.765+0000 7f7fbf2b0b80 1 Processor -- start 2024-06-09T22:28:27.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.765+0000 7f7fbf2b0b80 1 -- start start 2024-06-09T22:28:27.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.765+0000 7f7fbf2b0b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f1fa260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.765+0000 7f7fbf2b0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f1fff60 0x55dc9f201530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7fbf2b0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f200880 0x55dc9f1fe050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7fbf2b0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01b0740 con 0x55dc9f200880 2024-06-09T22:28:27.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7fbf2b0b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01b2280 con 0x55dc9f1fa7a0 2024-06-09T22:28:27.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7fbf2b0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01b0e10 con 0x55dc9f1fff60 2024-06-09T22:28:27.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7faed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f200880 0x55dc9f1fe050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7fadd9b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f1fff60 0x55dc9f201530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7fae59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f1fa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7faed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f200880 0x55dc9f1fe050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58398/0 (socket says 172.21.15.104:58398) 2024-06-09T22:28:27.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7faed9d700 1 -- 172.21.15.104:0/3384094658 learned_addr learned my addr 172.21.15.104:0/3384094658 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:27.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7faed9d700 1 -- 172.21.15.104:0/3384094658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f1fff60 msgr2=0x55dc9f201530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:27.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7faed9d700 1 --2- 172.21.15.104:0/3384094658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f1fff60 0x55dc9f201530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7faed9d700 1 -- 172.21.15.104:0/3384094658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fa7a0 msgr2=0x55dc9f1fa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:27.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7faed9d700 1 --2- 172.21.15.104:0/3384094658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f1fa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7faed9d700 1 -- 172.21.15.104:0/3384094658 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc9f1f1810 con 0x55dc9f200880 2024-06-09T22:28:27.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7fae59c700 1 --2- 172.21.15.104:0/3384094658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f1fa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:28:27.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7faed9d700 1 --2- 172.21.15.104:0/3384094658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f200880 0x55dc9f1fe050 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009ee0 tx=0x7f7fa8009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cea10ca62424269 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:27.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7f97fff700 1 -- 172.21.15.104:0/3384094658 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8007dc0 con 0x55dc9f200880 2024-06-09T22:28:27.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7f97fff700 1 -- 172.21.15.104:0/3384094658 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fa801b070 con 0x55dc9f200880 2024-06-09T22:28:27.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.766+0000 7f7f97fff700 1 -- 172.21.15.104:0/3384094658 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8005280 con 0x55dc9f200880 2024-06-09T22:28:27.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.767+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/3384094658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f200880 msgr2=0x55dc9f1fe050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:27.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.767+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/3384094658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f200880 0x55dc9f1fe050 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009ee0 tx=0x7f7fa8009f10 comp rx=0 tx=0).stop 2024-06-09T22:28:27.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.767+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/3384094658 shutdown_connections 2024-06-09T22:28:27.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.767+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/3384094658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f200880 0x55dc9f1fe050 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.767+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/3384094658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f1fa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.767+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/3384094658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f1fff60 0x55dc9f201530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.767+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/3384094658 >> 172.21.15.104:0/3384094658 conn(0x55dc9f127080 msgr2=0x55dc9f1feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.767+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/3384094658 shutdown_connections 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.767+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/3384094658 wait complete. 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.767+0000 7f7fbf2b0b80 1 -- start start 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fbf2b0b80 1 Processor -- start 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fbf2b0b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f20c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fbf2b0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fff60 0x55dc9f20ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fbf2b0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f200880 0x55dc9f21ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fbf2b0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01b1bb0 con 0x55dc9f1fff60 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fbf2b0b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01b14e0 con 0x55dc9f1fa7a0 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fbf2b0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01ae700 con 0x55dc9f200880 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fbf2b0b80 10 client.0 _ll_register_callbacks cb 0x55dc9f0e15b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fadd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fff60 0x55dc9f20ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fadd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fff60 0x55dc9f20ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58400/0 (socket says 172.21.15.104:58400) 2024-06-09T22:28:27.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fadd9b700 1 -- 172.21.15.104:0/659053338 learned_addr learned my addr 172.21.15.104:0/659053338 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:27.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7faed9d700 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f200880 0x55dc9f21ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fae59c700 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f20c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fadd9b700 1 -- 172.21.15.104:0/659053338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f200880 msgr2=0x55dc9f21ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:27.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f200880 0x55dc9f21ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fadd9b700 1 -- 172.21.15.104:0/659053338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fa7a0 msgr2=0x55dc9f20c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:27.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f20c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fadd9b700 1 -- 172.21.15.104:0/659053338 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc9f21b170 con 0x55dc9f1fff60 2024-06-09T22:28:27.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7fadd9b700 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fff60 0x55dc9f20ca60 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4003a70 tx=0x7f7fa4005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ef9169a6ab55d2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:27.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.768+0000 7f7f95ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:27.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7f95ffb700 1 -- 172.21.15.104:0/659053338 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4003cc0 con 0x55dc9f1fff60 2024-06-09T22:28:27.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7f95ffb700 1 -- 172.21.15.104:0/659053338 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f7fa40050c0 con 0x55dc9f1fff60 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/659053338 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fa801b070 con 0x55dc9f1fff60 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7f95ffb700 1 -- 172.21.15.104:0/659053338 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7fa403d050 con 0x55dc9f1fff60 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7fbf2b0b80 10 client.17514 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/659053338 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dc9f27e3d0 con 0x55dc9f1fff60 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7f95ffb700 1 -- 172.21.15.104:0/659053338 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fa801b070 con 0x55dc9f1fff60 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7f95ffb700 1 -- 172.21.15.104:0/659053338 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7fa0087050 con 0x55dc9f1fff60 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7fbf2b0b80 10 client.17514 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7fbf2b0b80 10 client.17514 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7fbf2b0b80 20 client.17514 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55dc9f27e830 0x55dc9f280c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7fbf2b0b80 4 client.17514 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.769+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/659053338 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dc9f1532d0 con 0x55dc9f27e830 2024-06-09T22:28:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7fae59c700 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55dc9f27e830 0x55dc9f280c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7f95ffb700 10 client.17514 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7fae59c700 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55dc9f27e830 0x55dc9f280c50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=294a3d096cb22ee6 server_cookie=b605b5eb09c09ed4 in_seq=0 out_seq=0 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7f95ffb700 1 -- 172.21.15.104:0/659053338 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dc9f1532d0 con 0x55dc9f27e830 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7f95ffb700 10 client.17514 handle_command_reply: tid=0 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7fbf2b0b80 1 client.17514 shutdown 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/659053338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fff60 msgr2=0x55dc9f20ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fff60 0x55dc9f20ca60 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4003a70 tx=0x7f7fa4005d30 comp rx=0 tx=0).stop 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/659053338 shutdown_connections 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dc9f200880 0x55dc9f21ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc9f1fa7a0 0x55dc9f20c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x55dc9f27e830 0x55dc9f280c50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7f95ffb700 0 client.17514 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7fbf2b0b80 1 --2- 172.21.15.104:0/659053338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55dc9f1fff60 0x55dc9f20ca60 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.770+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/659053338 >> 172.21.15.104:0/659053338 conn(0x55dc9f127080 msgr2=0x55dc9f201370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:27.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/659053338 shutdown_connections 2024-06-09T22:28:27.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fbf2b0b80 1 -- 172.21.15.104:0/659053338 wait complete. 2024-06-09T22:28:27.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fbf2b0b80 20 client.17514 trim_cache size 0 max 16384 2024-06-09T22:28:27.779 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:28:27.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/2863369355 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f980a2d90 msgr2=0x7f7f980a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:27.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fb4a2d700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f980a2d90 0x7f7f980a5250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00788b0 tx=0x7f7fa4009260 comp rx=0 tx=0).stop 2024-06-09T22:28:27.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/2863369355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb006a170 msgr2=0x7f7fb019e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:27.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fb4a2d700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb006a170 0x7f7fb019e830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0000c00 tx=0x7f7fa000bab0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/2863369355 shutdown_connections 2024-06-09T22:28:27.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fb4a2d700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f980a2d90 0x7f7f980a5250 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fb4a2d700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fb0108500 0x7f7fb0198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fb4a2d700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb006a170 0x7f7fb019e830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fb4a2d700 1 --2- 172.21.15.104:0/2863369355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fb006ab30 0x7f7fb019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:27.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/2863369355 >> 172.21.15.104:0/2863369355 conn(0x7f7fb0078340 msgr2=0x7f7fb00ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:27.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/2863369355 shutdown_connections 2024-06-09T22:28:27.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:27.771+0000 7f7fb4a2d700 1 -- 172.21.15.104:0/2863369355 wait complete. 2024-06-09T22:28:27.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:27 smithi194 ceph-mon[36856]: pgmap v229: 129 pgs: 129 active+clean; 201 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 199 op/s 2024-06-09T22:28:27.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:27 smithi153 ceph-mon[36599]: pgmap v229: 129 pgs: 129 active+clean; 201 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 199 op/s 2024-06-09T22:28:28.590 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-09T22:28:28.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 266360 4.91 MB/sec execute 214 sec latency 31.480 ms 2024-06-09T22:28:28.840 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:28:29.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 267719 4.91 MB/sec execute 215 sec latency 24.131 ms 2024-06-09T22:28:29.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:29 smithi194 ceph-mon[36856]: pgmap v230: 129 pgs: 129 active+clean; 201 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 199 op/s 2024-06-09T22:28:29.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:29 smithi153 ceph-mon[36599]: pgmap v230: 129 pgs: 129 active+clean; 201 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 199 op/s 2024-06-09T22:28:30.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:29 smithi104 ceph-mon[32389]: pgmap v230: 129 pgs: 129 active+clean; 201 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 199 op/s 2024-06-09T22:28:30.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.670+0000 7f5c49b47700 1 -- 172.21.15.104:0/1111161848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c44108510 msgr2=0x7f5c441088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:30.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.670+0000 7f5c49b47700 1 --2- 172.21.15.104:0/1111161848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c44108510 0x7f5c441088f0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009990 tx=0x7f5c34009ca0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.670+0000 7f5c49b47700 1 -- 172.21.15.104:0/1111161848 shutdown_connections 2024-06-09T22:28:30.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.670+0000 7f5c49b47700 1 --2- 172.21.15.104:0/1111161848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c44108510 0x7f5c441088f0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.670+0000 7f5c49b47700 1 --2- 172.21.15.104:0/1111161848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c44101440 0x7f5c4410f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.670+0000 7f5c49b47700 1 --2- 172.21.15.104:0/1111161848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c44100a80 0x7f5c44100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.670+0000 7f5c49b47700 1 -- 172.21.15.104:0/1111161848 >> 172.21.15.104:0/1111161848 conn(0x7f5c44077a00 msgr2=0x7f5c440fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:30.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.670+0000 7f5c49b47700 1 -- 172.21.15.104:0/1111161848 shutdown_connections 2024-06-09T22:28:30.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.670+0000 7f5c49b47700 1 -- 172.21.15.104:0/1111161848 wait complete. 2024-06-09T22:28:30.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.671+0000 7f5c49b47700 1 Processor -- start 2024-06-09T22:28:30.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.671+0000 7f5c49b47700 1 -- start start 2024-06-09T22:28:30.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.671+0000 7f5c49b47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c44100a80 0x7f5c44101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.671+0000 7f5c49b47700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c44101440 0x7f5c441021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.671+0000 7f5c49b47700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c44108510 0x7f5c441026e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.671+0000 7f5c49b47700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5c44112770 con 0x7f5c44100a80 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c49b47700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5c441125f0 con 0x7f5c44108510 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c49b47700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c441128f0 con 0x7f5c44101440 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c44100a80 0x7f5c44101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c44100a80 0x7f5c44101c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58418/0 (socket says 172.21.15.104:58418) 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c437fe700 1 -- 172.21.15.104:0/3034959654 learned_addr learned my addr 172.21.15.104:0/3034959654 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c42ffd700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c44101440 0x7f5c441021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c43fff700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c44108510 0x7f5c441026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c437fe700 1 -- 172.21.15.104:0/3034959654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c44101440 msgr2=0x7f5c441021a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c437fe700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c44101440 0x7f5c441021a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c437fe700 1 -- 172.21.15.104:0/3034959654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c44108510 msgr2=0x7f5c441026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:30.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c437fe700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c44108510 0x7f5c441026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c437fe700 1 -- 172.21.15.104:0/3034959654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c441a7120 con 0x7f5c44100a80 2024-06-09T22:28:30.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c43fff700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c44108510 0x7f5c441026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:30.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c437fe700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c44100a80 0x7f5c44101c60 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400bae0 tx=0x7f5c3400bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:30.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c40ff9700 1 -- 172.21.15.104:0/3034959654 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c34026070 con 0x7f5c44100a80 2024-06-09T22:28:30.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c49b47700 1 -- 172.21.15.104:0/3034959654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c441a73b0 con 0x7f5c44100a80 2024-06-09T22:28:30.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.672+0000 7f5c40ff9700 1 -- 172.21.15.104:0/3034959654 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c34003d70 con 0x7f5c44100a80 2024-06-09T22:28:30.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.673+0000 7f5c40ff9700 1 -- 172.21.15.104:0/3034959654 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c34021410 con 0x7f5c44100a80 2024-06-09T22:28:30.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.673+0000 7f5c49b47700 1 -- 172.21.15.104:0/3034959654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c441a78e0 con 0x7f5c44100a80 2024-06-09T22:28:30.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.675+0000 7f5c40ff9700 1 -- 172.21.15.104:0/3034959654 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5c34029070 con 0x7f5c44100a80 2024-06-09T22:28:30.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.676+0000 7f5c40ff9700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5c2c0a2d90 0x7f5c2c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.676+0000 7f5c40ff9700 1 -- 172.21.15.104:0/3034959654 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f5c340d04f0 con 0x7f5c44100a80 2024-06-09T22:28:30.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.676+0000 7f5c42ffd700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5c2c0a2d90 0x7f5c2c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.676+0000 7f5c42ffd700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5c2c0a2d90 0x7f5c2c0a5250 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5c44107b30 tx=0x7f5c38009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.676+0000 7f5c543cab80 1 Processor -- start 2024-06-09T22:28:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.676+0000 7f5c543cab80 1 -- start start 2024-06-09T22:28:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c543cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 0x5611b18dda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c543cab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18ddf60 0x5611b18d79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c543cab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18d7f20 0x5611b18d8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c543cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5611b1829810 con 0x5611b18dd640 2024-06-09T22:28:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c543cab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5611b17d1720 con 0x5611b18ddf60 2024-06-09T22:28:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c543cab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5611b18257d0 con 0x5611b18d7f20 2024-06-09T22:28:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 0x5611b18dda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 0x5611b18dda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58434/0 (socket says 172.21.15.104:58434) 2024-06-09T22:28:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c42ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18ddf60 0x5611b18d79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c437fe700 1 -- 172.21.15.104:0/2712317901 learned_addr learned my addr 172.21.15.104:0/2712317901 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c43fff700 1 --2- 172.21.15.104:0/2712317901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18d7f20 0x5611b18d8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c437fe700 1 -- 172.21.15.104:0/2712317901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18d7f20 msgr2=0x5611b18d8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c437fe700 1 --2- 172.21.15.104:0/2712317901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18d7f20 0x5611b18d8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c437fe700 1 -- 172.21.15.104:0/2712317901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18ddf60 msgr2=0x5611b18d79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c437fe700 1 --2- 172.21.15.104:0/2712317901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18ddf60 0x5611b18d79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c437fe700 1 -- 172.21.15.104:0/2712317901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611b17c7f50 con 0x5611b18dd640 2024-06-09T22:28:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c437fe700 1 --2- 172.21.15.104:0/2712317901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 0x5611b18dda20 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f5c340990e0 tx=0x7f5c34005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61954451971d2066 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c29ffb700 1 -- 172.21.15.104:0/2712317901 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3409c070 con 0x5611b18dd640 2024-06-09T22:28:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c29ffb700 1 -- 172.21.15.104:0/2712317901 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c3409ce10 con 0x5611b18dd640 2024-06-09T22:28:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.677+0000 7f5c29ffb700 1 -- 172.21.15.104:0/2712317901 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c34087110 con 0x5611b18dd640 2024-06-09T22:28:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.678+0000 7f5c543cab80 1 -- 172.21.15.104:0/2712317901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 msgr2=0x5611b18dda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.678+0000 7f5c543cab80 1 --2- 172.21.15.104:0/2712317901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 0x5611b18dda20 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f5c340990e0 tx=0x7f5c34005a00 comp rx=0 tx=0).stop 2024-06-09T22:28:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.678+0000 7f5c543cab80 1 -- 172.21.15.104:0/2712317901 shutdown_connections 2024-06-09T22:28:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.678+0000 7f5c543cab80 1 --2- 172.21.15.104:0/2712317901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 0x5611b18dda20 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.678+0000 7f5c543cab80 1 --2- 172.21.15.104:0/2712317901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18d7f20 0x5611b18d8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.678+0000 7f5c543cab80 1 --2- 172.21.15.104:0/2712317901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18ddf60 0x5611b18d79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.678+0000 7f5c543cab80 1 -- 172.21.15.104:0/2712317901 >> 172.21.15.104:0/2712317901 conn(0x5611b1802cc0 msgr2=0x5611b18d2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.678+0000 7f5c543cab80 1 -- 172.21.15.104:0/2712317901 shutdown_connections 2024-06-09T22:28:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.678+0000 7f5c543cab80 1 -- 172.21.15.104:0/2712317901 wait complete. 2024-06-09T22:28:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c543cab80 1 -- start start 2024-06-09T22:28:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c543cab80 1 Processor -- start 2024-06-09T22:28:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c543cab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18d7f20 0x5611b18e3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c543cab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18dd640 0x5611b18e1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c543cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18ddf60 0x5611b18e15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c543cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5611b18e46f0 con 0x5611b18ddf60 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c543cab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5611b17c9bf0 con 0x5611b18dd640 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c543cab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5611b18e4870 con 0x5611b18d7f20 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c543cab80 10 client.0 _ll_register_callbacks cb 0x5611b17c3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c437fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18d7f20 0x5611b18e3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c437fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18d7f20 0x5611b18e3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:55864/0 (socket says 172.21.15.104:55864) 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c437fe700 1 -- 172.21.15.104:0/314351682 learned_addr learned my addr 172.21.15.104:0/314351682 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c42ffd700 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18dd640 0x5611b18e1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.679+0000 7f5c43fff700 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18ddf60 0x5611b18e15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c437fe700 1 -- 172.21.15.104:0/314351682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18dd640 msgr2=0x5611b18e1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c437fe700 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18dd640 0x5611b18e1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c437fe700 1 -- 172.21.15.104:0/314351682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18ddf60 msgr2=0x5611b18e15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c437fe700 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18ddf60 0x5611b18e15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c437fe700 1 -- 172.21.15.104:0/314351682 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611b17788d0 con 0x5611b18d7f20 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c42ffd700 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18dd640 0x5611b18e1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c43fff700 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18ddf60 0x5611b18e15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c437fe700 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18d7f20 0x5611b18e3d60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5c340cd960 tx=0x7f5c34003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b822ca57079bcc37 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c22ffd700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c22ffd700 1 -- 172.21.15.104:0/314351682 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c34087110 con 0x5611b18d7f20 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c543cab80 1 -- 172.21.15.104:0/314351682 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c3409ce10 con 0x5611b18d7f20 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c22ffd700 1 -- 172.21.15.104:0/314351682 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f5c3409cb80 con 0x5611b18d7f20 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c22ffd700 1 -- 172.21.15.104:0/314351682 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5c3409c070 con 0x5611b18d7f20 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c543cab80 10 client.26854 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c543cab80 1 -- 172.21.15.104:0/314351682 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5611b18e46f0 con 0x5611b18d7f20 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c22ffd700 1 -- 172.21.15.104:0/314351682 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c3409ce10 con 0x5611b18d7f20 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.680+0000 7f5c22ffd700 1 -- 172.21.15.104:0/314351682 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5c340b0ba0 con 0x5611b18d7f20 2024-06-09T22:28:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.681+0000 7f5c543cab80 10 client.26854 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.681+0000 7f5c543cab80 10 client.26854 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.681+0000 7f5c543cab80 20 client.26854 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.681+0000 7f5c543cab80 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5611b18f13f0 0x5611b18f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.681+0000 7f5c543cab80 4 client.26854 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.681+0000 7f5c543cab80 1 -- 172.21.15.104:0/314351682 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611b1836d10 con 0x5611b18f13f0 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.681+0000 7f5c42ffd700 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5611b18f13f0 0x5611b18f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.681+0000 7f5c22ffd700 10 client.26854 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.681+0000 7f5c42ffd700 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5611b18f13f0 0x5611b18f37e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1c88c1f83aca9a41 server_cookie=a4c444210f9b4abd in_seq=0 out_seq=0 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c22ffd700 1 -- 172.21.15.104:0/314351682 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5611b1836d10 con 0x5611b18f13f0 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c22ffd700 10 client.26854 handle_command_reply: tid=0 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c543cab80 1 client.26854 shutdown 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c543cab80 1 -- 172.21.15.104:0/314351682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18d7f20 msgr2=0x5611b18e3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c543cab80 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18d7f20 0x5611b18e3d60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5c340cd960 tx=0x7f5c34003f90 comp rx=0 tx=0).stop 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c543cab80 1 -- 172.21.15.104:0/314351682 shutdown_connections 2024-06-09T22:28:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c543cab80 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5611b18f13f0 0x5611b18f37e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c543cab80 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18ddf60 0x5611b18e15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c543cab80 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18d7f20 0x5611b18e3d60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c543cab80 1 --2- 172.21.15.104:0/314351682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18dd640 0x5611b18e1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c22ffd700 0 client.26854 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:28:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c543cab80 1 -- 172.21.15.104:0/314351682 >> 172.21.15.104:0/314351682 conn(0x5611b1802cc0 msgr2=0x5611b18d25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c543cab80 1 -- 172.21.15.104:0/314351682 shutdown_connections 2024-06-09T22:28:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.682+0000 7f5c543cab80 1 -- 172.21.15.104:0/314351682 wait complete. 2024-06-09T22:28:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.683+0000 7f5c543cab80 20 client.26854 trim_cache size 0 max 16384 2024-06-09T22:28:30.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.698+0000 7f5c543cab80 1 Processor -- start 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c543cab80 1 -- start start 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c543cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18d7f20 0x5611b18d79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c543cab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18dd640 0x5611b18ed990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c543cab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18ddf60 0x5611b18dea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c543cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5c441b07c0 con 0x5611b18d7f20 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c543cab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5c441b2300 con 0x5611b18ddf60 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c543cab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c441b0e90 con 0x5611b18dd640 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18d7f20 0x5611b18d79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18d7f20 0x5611b18d79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58450/0 (socket says 172.21.15.104:58450) 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c437fe700 1 -- 172.21.15.104:0/819664700 learned_addr learned my addr 172.21.15.104:0/819664700 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c43fff700 1 --2- 172.21.15.104:0/819664700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18ddf60 0x5611b18dea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c42ffd700 1 --2- 172.21.15.104:0/819664700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18dd640 0x5611b18ed990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c437fe700 1 -- 172.21.15.104:0/819664700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18dd640 msgr2=0x5611b18ed990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:30.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c437fe700 1 --2- 172.21.15.104:0/819664700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18dd640 0x5611b18ed990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c437fe700 1 -- 172.21.15.104:0/819664700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18ddf60 msgr2=0x5611b18dea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:30.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c437fe700 1 --2- 172.21.15.104:0/819664700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18ddf60 0x5611b18dea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c437fe700 1 -- 172.21.15.104:0/819664700 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611b18df5d0 con 0x5611b18d7f20 2024-06-09T22:28:30.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c42ffd700 1 --2- 172.21.15.104:0/819664700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18dd640 0x5611b18ed990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:28:30.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c43fff700 1 --2- 172.21.15.104:0/819664700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18ddf60 0x5611b18dea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:28:30.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.699+0000 7f5c437fe700 1 --2- 172.21.15.104:0/819664700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18d7f20 0x5611b18d79e0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f5c34098b70 tx=0x7f5c34005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f30db1b8477767 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:30.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.700+0000 7f5c297fa700 1 -- 172.21.15.104:0/819664700 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3409ce10 con 0x5611b18d7f20 2024-06-09T22:28:30.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.700+0000 7f5c297fa700 1 -- 172.21.15.104:0/819664700 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c3409c710 con 0x5611b18d7f20 2024-06-09T22:28:30.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.700+0000 7f5c297fa700 1 -- 172.21.15.104:0/819664700 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c340a52f0 con 0x5611b18d7f20 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.700+0000 7f5c543cab80 1 -- 172.21.15.104:0/819664700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18d7f20 msgr2=0x5611b18d79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.700+0000 7f5c543cab80 1 --2- 172.21.15.104:0/819664700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18d7f20 0x5611b18d79e0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f5c34098b70 tx=0x7f5c34005a00 comp rx=0 tx=0).stop 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.700+0000 7f5c543cab80 1 -- 172.21.15.104:0/819664700 shutdown_connections 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.700+0000 7f5c543cab80 1 --2- 172.21.15.104:0/819664700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18d7f20 0x5611b18d79e0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.700+0000 7f5c543cab80 1 --2- 172.21.15.104:0/819664700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18ddf60 0x5611b18dea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.700+0000 7f5c543cab80 1 --2- 172.21.15.104:0/819664700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18dd640 0x5611b18ed990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.700+0000 7f5c543cab80 1 -- 172.21.15.104:0/819664700 >> 172.21.15.104:0/819664700 conn(0x5611b1802cc0 msgr2=0x5611b18de340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.700+0000 7f5c543cab80 1 -- 172.21.15.104:0/819664700 shutdown_connections 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.700+0000 7f5c543cab80 1 -- 172.21.15.104:0/819664700 wait complete. 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c543cab80 1 -- start start 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c543cab80 1 Processor -- start 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c543cab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18d7f20 0x5611b18e9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c543cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 0x5611b18ea040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c543cab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18ddf60 0x5611b18f8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c543cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5c441b1c30 con 0x5611b18dd640 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c543cab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5c441b1560 con 0x5611b18d7f20 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c543cab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c441ae780 con 0x5611b18ddf60 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c543cab80 10 client.0 _ll_register_callbacks cb 0x5611b17c3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c437fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18d7f20 0x5611b18e9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c42ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 0x5611b18ea040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c437fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18d7f20 0x5611b18e9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:51598/0 (socket says 172.21.15.104:51598) 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c42ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 0x5611b18ea040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58462/0 (socket says 172.21.15.104:58462) 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.701+0000 7f5c437fe700 1 -- 172.21.15.104:0/3105600654 learned_addr learned my addr 172.21.15.104:0/3105600654 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c43fff700 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18ddf60 0x5611b18f8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c437fe700 1 -- 172.21.15.104:0/3105600654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18ddf60 msgr2=0x5611b18f8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c437fe700 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18ddf60 0x5611b18f8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c437fe700 1 -- 172.21.15.104:0/3105600654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 msgr2=0x5611b18ea040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:30.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c437fe700 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 0x5611b18ea040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c437fe700 1 -- 172.21.15.104:0/3105600654 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611b18f86c0 con 0x5611b18d7f20 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c42ffd700 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 0x5611b18ea040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c43fff700 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18ddf60 0x5611b18f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c437fe700 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18d7f20 0x5611b18e9b00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5c3409a9c0 tx=0x7f5c340cd980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2942d4d520739ce server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c22ffd700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c22ffd700 1 -- 172.21.15.104:0/3105600654 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3409ce10 con 0x5611b18d7f20 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c543cab80 1 -- 172.21.15.104:0/3105600654 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c3409c710 con 0x5611b18d7f20 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c22ffd700 1 -- 172.21.15.104:0/3105600654 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f5c34003890 con 0x5611b18d7f20 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c22ffd700 1 -- 172.21.15.104:0/3105600654 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5c340a52f0 con 0x5611b18d7f20 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c543cab80 10 client.26762 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c543cab80 1 -- 172.21.15.104:0/3105600654 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5611b18f86c0 con 0x5611b18d7f20 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.702+0000 7f5c22ffd700 1 -- 172.21.15.104:0/3105600654 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c3409c710 con 0x5611b18d7f20 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c22ffd700 1 -- 172.21.15.104:0/3105600654 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5c3409b5d0 con 0x5611b18d7f20 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c543cab80 10 client.26762 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c543cab80 10 client.26762 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c543cab80 20 client.26762 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c543cab80 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5611b195bb00 0x5611b195def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c543cab80 4 client.26762 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c543cab80 1 -- 172.21.15.104:0/3105600654 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5611b1831290 con 0x5611b195bb00 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c42ffd700 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5611b195bb00 0x5611b195def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c42ffd700 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5611b195bb00 0x5611b195def0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4e1b062267f4c430 server_cookie=db19418b41dac60e in_seq=0 out_seq=0 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c22ffd700 10 client.26762 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c22ffd700 1 -- 172.21.15.104:0/3105600654 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5611b1831290 con 0x5611b195bb00 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c22ffd700 10 client.26762 handle_command_reply: tid=0 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c543cab80 1 client.26762 shutdown 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c543cab80 1 -- 172.21.15.104:0/3105600654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18d7f20 msgr2=0x5611b18e9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.703+0000 7f5c543cab80 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18d7f20 0x5611b18e9b00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5c3409a9c0 tx=0x7f5c340cd980 comp rx=0 tx=0).stop 2024-06-09T22:28:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c543cab80 1 -- 172.21.15.104:0/3105600654 shutdown_connections 2024-06-09T22:28:30.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c543cab80 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5611b195bb00 0x5611b195def0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c543cab80 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5611b18ddf60 0x5611b18f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c543cab80 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5611b18d7f20 0x5611b18e9b00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c543cab80 1 --2- 172.21.15.104:0/3105600654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611b18dd640 0x5611b18ea040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c22ffd700 0 client.26762 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:28:30.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c543cab80 1 -- 172.21.15.104:0/3105600654 >> 172.21.15.104:0/3105600654 conn(0x5611b1802cc0 msgr2=0x5611b18de340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:30.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c543cab80 1 -- 172.21.15.104:0/3105600654 shutdown_connections 2024-06-09T22:28:30.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c543cab80 1 -- 172.21.15.104:0/3105600654 wait complete. 2024-06-09T22:28:30.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c543cab80 20 client.26762 trim_cache size 0 max 16384 2024-06-09T22:28:30.711 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:28:30.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c49b47700 1 -- 172.21.15.104:0/3034959654 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5c2c0a2d90 msgr2=0x7f5c2c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:30.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c49b47700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5c2c0a2d90 0x7f5c2c0a5250 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5c44107b30 tx=0x7f5c38009260 comp rx=0 tx=0).stop 2024-06-09T22:28:30.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c49b47700 1 -- 172.21.15.104:0/3034959654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c44100a80 msgr2=0x7f5c44101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:30.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c49b47700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c44100a80 0x7f5c44101c60 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400bae0 tx=0x7f5c3400bb10 comp rx=0 tx=0).stop 2024-06-09T22:28:30.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c49b47700 1 -- 172.21.15.104:0/3034959654 shutdown_connections 2024-06-09T22:28:30.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c49b47700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f5c2c0a2d90 0x7f5c2c0a5250 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c49b47700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c44100a80 0x7f5c44101c60 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c49b47700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c44108510 0x7f5c441026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c49b47700 1 --2- 172.21.15.104:0/3034959654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c44101440 0x7f5c441021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:30.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c49b47700 1 -- 172.21.15.104:0/3034959654 >> 172.21.15.104:0/3034959654 conn(0x7f5c44077a00 msgr2=0x7f5c440ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:30.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c49b47700 1 -- 172.21.15.104:0/3034959654 shutdown_connections 2024-06-09T22:28:30.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:30.704+0000 7f5c49b47700 1 -- 172.21.15.104:0/3034959654 wait complete. 2024-06-09T22:28:30.737 INFO:tasks.workunit.client.0.smithi104.stdout: 1 269851 4.95 MB/sec execute 216 sec latency 52.529 ms 2024-06-09T22:28:31.494 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-09T22:28:31.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 271095 4.96 MB/sec execute 217 sec latency 20.450 ms 2024-06-09T22:28:31.759 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:28:31.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:31 smithi194 ceph-mon[36856]: pgmap v231: 129 pgs: 129 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.8 MiB/s wr, 291 op/s 2024-06-09T22:28:31.881 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:31 smithi104 ceph-mon[32389]: pgmap v231: 129 pgs: 129 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.8 MiB/s wr, 291 op/s 2024-06-09T22:28:31.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:31 smithi153 ceph-mon[36599]: pgmap v231: 129 pgs: 129 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.8 MiB/s wr, 291 op/s 2024-06-09T22:28:32.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 273366 5.00 MB/sec execute 218 sec latency 16.771 ms 2024-06-09T22:28:33.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 274586 5.00 MB/sec execute 219 sec latency 46.768 ms 2024-06-09T22:28:33.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.751+0000 7f7432f34700 1 -- 172.21.15.104:0/1792760740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f742c107430 msgr2=0x7f742c107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:33.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.751+0000 7f7432f34700 1 --2- 172.21.15.104:0/1792760740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f742c107430 0x7f742c107810 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f7420009990 tx=0x7f7420009ca0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.751+0000 7f7432f34700 1 -- 172.21.15.104:0/1792760740 shutdown_connections 2024-06-09T22:28:33.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.751+0000 7f7432f34700 1 --2- 172.21.15.104:0/1792760740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f742c107430 0x7f742c107810 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.751+0000 7f7432f34700 1 --2- 172.21.15.104:0/1792760740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742c101d80 0x7f742c10e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.751+0000 7f7432f34700 1 --2- 172.21.15.104:0/1792760740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f742c1013c0 0x7f742c101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.751+0000 7f7432f34700 1 -- 172.21.15.104:0/1792760740 >> 172.21.15.104:0/1792760740 conn(0x7f742c0783d0 msgr2=0x7f742c0787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.751+0000 7f7432f34700 1 -- 172.21.15.104:0/1792760740 shutdown_connections 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.751+0000 7f7432f34700 1 -- 172.21.15.104:0/1792760740 wait complete. 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f7432f34700 1 Processor -- start 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f7432f34700 1 -- start start 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f7432f34700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742c1013c0 0x7f742c19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f7432f34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f742c101d80 0x7f742c19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f7432f34700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f742c107430 0x7f742c198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f7432f34700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f742c1116a0 con 0x7f742c101d80 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f7432f34700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f742c111520 con 0x7f742c1013c0 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f7432f34700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f742c111820 con 0x7f742c107430 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f742bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f742c101d80 0x7f742c19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f742bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f742c101d80 0x7f742c19ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58480/0 (socket says 172.21.15.104:58480) 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f742bfff700 1 -- 172.21.15.104:0/2742484667 learned_addr learned my addr 172.21.15.104:0/2742484667 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f7430cd0700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742c1013c0 0x7f742c19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f74314d1700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f742c107430 0x7f742c198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f742bfff700 1 -- 172.21.15.104:0/2742484667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f742c107430 msgr2=0x7f742c198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f742bfff700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f742c107430 0x7f742c198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f742bfff700 1 -- 172.21.15.104:0/2742484667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742c1013c0 msgr2=0x7f742c19e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f742bfff700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742c1013c0 0x7f742c19e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.752+0000 7f742bfff700 1 -- 172.21.15.104:0/2742484667 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742c1991c0 con 0x7f742c101d80 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.753+0000 7f742bfff700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f742c101d80 0x7f742c19ede0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f741800d980 tx=0x7f741800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.753+0000 7f7429ffb700 1 -- 172.21.15.104:0/2742484667 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7418016070 con 0x7f742c101d80 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.753+0000 7f7432f34700 1 -- 172.21.15.104:0/2742484667 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742c1994b0 con 0x7f742c101d80 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.753+0000 7f7429ffb700 1 -- 172.21.15.104:0/2742484667 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7418004d10 con 0x7f742c101d80 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.753+0000 7f7429ffb700 1 -- 172.21.15.104:0/2742484667 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7418005670 con 0x7f742c101d80 2024-06-09T22:28:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.753+0000 7f7432f34700 1 -- 172.21.15.104:0/2742484667 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f742c1a5770 con 0x7f742c101d80 2024-06-09T22:28:33.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.755+0000 7f7429ffb700 1 -- 172.21.15.104:0/2742484667 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7418004750 con 0x7f742c101d80 2024-06-09T22:28:33.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.756+0000 7f7429ffb700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f741c0a2d90 0x7f741c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.756+0000 7f7430cd0700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f741c0a2d90 0x7f741c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.756+0000 7f7429ffb700 1 -- 172.21.15.104:0/2742484667 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f74180cf5e0 con 0x7f742c101d80 2024-06-09T22:28:33.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.756+0000 7f7430cd0700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f741c0a2d90 0x7f741c0a5250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f742000b5c0 tx=0x7f7420005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:33.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f743d7b7b80 1 Processor -- start 2024-06-09T22:28:33.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f743d7b7b80 1 -- start start 2024-06-09T22:28:33.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f743d7b7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 0x55b019a0b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f743d7b7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a05260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f743d7b7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a057a0 0x55b019a05bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f743d7b7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b0198f7f60 con 0x55b019a0b880 2024-06-09T22:28:33.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f743d7b7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b019956850 con 0x55b019a057a0 2024-06-09T22:28:33.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f743d7b7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b0198be460 con 0x55b019a0af60 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f742bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a05260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f7430cd0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 0x55b019a0b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f7430cd0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 0x55b019a0b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:55900/0 (socket says 172.21.15.104:55900) 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f7430cd0700 1 -- 172.21.15.104:0/3885268078 learned_addr learned my addr 172.21.15.104:0/3885268078 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.757+0000 7f74314d1700 1 --2- 172.21.15.104:0/3885268078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a057a0 0x55b019a05bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.758+0000 7f7430cd0700 1 -- 172.21.15.104:0/3885268078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a057a0 msgr2=0x55b019a05bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.758+0000 7f7430cd0700 1 --2- 172.21.15.104:0/3885268078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a057a0 0x55b019a05bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.758+0000 7f7430cd0700 1 -- 172.21.15.104:0/3885268078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 msgr2=0x55b019a05260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.758+0000 7f7430cd0700 1 --2- 172.21.15.104:0/3885268078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a05260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.758+0000 7f7430cd0700 1 -- 172.21.15.104:0/3885268078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0198a58d0 con 0x55b019a0af60 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.758+0000 7f742bfff700 1 --2- 172.21.15.104:0/3885268078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a05260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.758+0000 7f7430cd0700 1 --2- 172.21.15.104:0/3885268078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 0x55b019a0b340 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7420015000 tx=0x7f7420020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60e6134dbe63e88a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.758+0000 7f7416ffd700 1 -- 172.21.15.104:0/3885268078 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f742000f460 con 0x55b019a0af60 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.758+0000 7f7416ffd700 1 -- 172.21.15.104:0/3885268078 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f742000fae0 con 0x55b019a0af60 2024-06-09T22:28:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.758+0000 7f7416ffd700 1 -- 172.21.15.104:0/3885268078 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7420024c50 con 0x55b019a0af60 2024-06-09T22:28:33.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.759+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3885268078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 msgr2=0x55b019a0b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:33.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.759+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3885268078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 0x55b019a0b340 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7420015000 tx=0x7f7420020000 comp rx=0 tx=0).stop 2024-06-09T22:28:33.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.759+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3885268078 shutdown_connections 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.759+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3885268078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a057a0 0x55b019a05bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.759+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3885268078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 0x55b019a0b340 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.759+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3885268078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a05260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.759+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3885268078 >> 172.21.15.104:0/3885268078 conn(0x55b019932080 msgr2=0x55b019a00220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.759+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3885268078 shutdown_connections 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.759+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3885268078 wait complete. 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.759+0000 7f743d7b7b80 1 -- start start 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.759+0000 7f743d7b7b80 1 Processor -- start 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f743d7b7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a057a0 0x55b019a115c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f743d7b7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a0af60 0x55b019a0e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f743d7b7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a0ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f743d7b7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b0198f6fc0 con 0x55b019a0b880 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f743d7b7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b019952810 con 0x55b019a0af60 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f743d7b7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b019a121a0 con 0x55b019a057a0 2024-06-09T22:28:33.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f743d7b7b80 10 client.0 _ll_register_callbacks cb 0x55b0198ec5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f74314d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a0ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f7430cd0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a057a0 0x55b019a115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f74314d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a0ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58504/0 (socket says 172.21.15.104:58504) 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f74314d1700 1 -- 172.21.15.104:0/3917017951 learned_addr learned my addr 172.21.15.104:0/3917017951 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f742bfff700 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a0af60 0x55b019a0e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f74314d1700 1 -- 172.21.15.104:0/3917017951 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a057a0 msgr2=0x55b019a115c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f74314d1700 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a057a0 0x55b019a115c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f74314d1700 1 -- 172.21.15.104:0/3917017951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a0af60 msgr2=0x55b019a0e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f74314d1700 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a0af60 0x55b019a0e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f74314d1700 1 -- 172.21.15.104:0/3917017951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b01991c750 con 0x55b019a0b880 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f7430cd0700 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a057a0 0x55b019a115c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f74314d1700 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a0ee00 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f74240042f0 tx=0x7f7424004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928ba391d1136011 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.760+0000 7f740bfff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:33.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f740bfff700 1 -- 172.21.15.104:0/3917017951 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f742401a070 con 0x55b019a0b880 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f740bfff700 1 -- 172.21.15.104:0/3917017951 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f7424007700 con 0x55b019a0b880 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3917017951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f742000fae0 con 0x55b019a0b880 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f740bfff700 1 -- 172.21.15.104:0/3917017951 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7424026050 con 0x55b019a0b880 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f743d7b7b80 10 client.17544 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3917017951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b019a1c9b0 con 0x55b019a0b880 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f740bfff700 1 -- 172.21.15.104:0/3917017951 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f742000fae0 con 0x55b019a0b880 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f740bfff700 1 -- 172.21.15.104:0/3917017951 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7424016a90 con 0x55b019a0b880 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f743d7b7b80 10 client.17544 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f743d7b7b80 10 client.17544 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f743d7b7b80 20 client.17544 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55b019a1ef90 0x55b019a21380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f743d7b7b80 4 client.17544 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.761+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3917017951 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b019963d50 con 0x55b019a1ef90 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.762+0000 7f7430cd0700 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55b019a1ef90 0x55b019a21380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.762+0000 7f7430cd0700 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55b019a1ef90 0x55b019a21380 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6948bb1ea6c8668e server_cookie=44b7ffad8506fdbd in_seq=0 out_seq=0 2024-06-09T22:28:33.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.762+0000 7f740bfff700 10 client.17544 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:28:33.768 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:33 smithi104 ceph-mon[32389]: pgmap v232: 129 pgs: 129 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f740bfff700 1 -- 172.21.15.104:0/3917017951 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b019963d50 con 0x55b019a1ef90 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f740bfff700 10 client.17544 handle_command_reply: tid=0 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f743d7b7b80 1 client.17544 shutdown 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3917017951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 msgr2=0x55b019a0ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a0ee00 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f74240042f0 tx=0x7f7424004320 comp rx=0 tx=0).stop 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3917017951 shutdown_connections 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55b019a1ef90 0x55b019a21380 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f740bfff700 0 client.17544 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a0ee00 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a057a0 0x55b019a115c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3917017951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a0af60 0x55b019a0e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3917017951 >> 172.21.15.104:0/3917017951 conn(0x55b019932080 msgr2=0x55b0199ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3917017951 shutdown_connections 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3917017951 wait complete. 2024-06-09T22:28:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.763+0000 7f743d7b7b80 20 client.17544 trim_cache size 0 max 16384 2024-06-09T22:28:33.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f743d7b7b80 1 Processor -- start 2024-06-09T22:28:33.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f743d7b7b80 1 -- start start 2024-06-09T22:28:33.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f743d7b7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a057a0 0x55b019a05260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f743d7b7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a0af60 0x55b019a0c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f743d7b7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0b880 0x55b019a0e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f743d7b7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f742c1ae570 con 0x55b019a057a0 2024-06-09T22:28:33.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f743d7b7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f742c1b00b0 con 0x55b019a0af60 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f743d7b7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f742c1aec40 con 0x55b019a0b880 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f7430cd0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a057a0 0x55b019a05260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f7430cd0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a057a0 0x55b019a05260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58514/0 (socket says 172.21.15.104:58514) 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f7430cd0700 1 -- 172.21.15.104:0/3001909903 learned_addr learned my addr 172.21.15.104:0/3001909903 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f742bfff700 1 --2- 172.21.15.104:0/3001909903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a0af60 0x55b019a0c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.780+0000 7f74314d1700 1 --2- 172.21.15.104:0/3001909903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0b880 0x55b019a0e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.781+0000 7f7430cd0700 1 -- 172.21.15.104:0/3001909903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0b880 msgr2=0x55b019a0e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.781+0000 7f7430cd0700 1 --2- 172.21.15.104:0/3001909903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0b880 0x55b019a0e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.781+0000 7f7430cd0700 1 -- 172.21.15.104:0/3001909903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a0af60 msgr2=0x55b019a0c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.781+0000 7f7430cd0700 1 --2- 172.21.15.104:0/3001909903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a0af60 0x55b019a0c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.781+0000 7f7430cd0700 1 -- 172.21.15.104:0/3001909903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b019a0f7a0 con 0x55b019a057a0 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.781+0000 7f742bfff700 1 --2- 172.21.15.104:0/3001909903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a0af60 0x55b019a0c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.781+0000 7f7430cd0700 1 --2- 172.21.15.104:0/3001909903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a057a0 0x55b019a05260 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f7420031040 tx=0x7f742000f5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ffd8142786f996 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:33.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.781+0000 7f74167fc700 1 -- 172.21.15.104:0/3001909903 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f742000f760 con 0x55b019a057a0 2024-06-09T22:28:33.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.781+0000 7f74167fc700 1 -- 172.21.15.104:0/3001909903 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f742000f900 con 0x55b019a057a0 2024-06-09T22:28:33.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.781+0000 7f74167fc700 1 -- 172.21.15.104:0/3001909903 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7420038750 con 0x55b019a057a0 2024-06-09T22:28:33.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.782+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3001909903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a057a0 msgr2=0x55b019a05260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:33.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.782+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3001909903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a057a0 0x55b019a05260 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f7420031040 tx=0x7f742000f5f0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.782+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3001909903 shutdown_connections 2024-06-09T22:28:33.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.782+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3001909903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a057a0 0x55b019a05260 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.782+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3001909903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0b880 0x55b019a0e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.782+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/3001909903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a0af60 0x55b019a0c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.782+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3001909903 >> 172.21.15.104:0/3001909903 conn(0x55b019932080 msgr2=0x55b019a0bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:33.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.782+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3001909903 shutdown_connections 2024-06-09T22:28:33.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.782+0000 7f743d7b7b80 1 -- 172.21.15.104:0/3001909903 wait complete. 2024-06-09T22:28:33.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.782+0000 7f743d7b7b80 1 -- start start 2024-06-09T22:28:33.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.782+0000 7f743d7b7b80 1 Processor -- start 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f743d7b7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a057a0 0x55b019a17840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f743d7b7b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 0x55b019a17d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f743d7b7b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a26410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f743d7b7b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f742c1af9e0 con 0x55b019a0b880 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f743d7b7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f742c1af310 con 0x55b019a057a0 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f743d7b7b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f742c1ac530 con 0x55b019a0af60 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f743d7b7b80 10 client.0 _ll_register_callbacks cb 0x55b0198ec5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f742bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 0x55b019a17d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f74314d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a26410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f742bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 0x55b019a17d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:55920/0 (socket says 172.21.15.104:55920) 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f742bfff700 1 -- 172.21.15.104:0/2030586020 learned_addr learned my addr 172.21.15.104:0/2030586020 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f7430cd0700 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a057a0 0x55b019a17840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f742bfff700 1 -- 172.21.15.104:0/2030586020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a057a0 msgr2=0x55b019a17840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f742bfff700 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a057a0 0x55b019a17840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f742bfff700 1 -- 172.21.15.104:0/2030586020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 msgr2=0x55b019a26410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f742bfff700 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a26410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f742bfff700 1 -- 172.21.15.104:0/2030586020 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b019a26c50 con 0x55b019a0af60 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f7430cd0700 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a057a0 0x55b019a17840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f74314d1700 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a26410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f742bfff700 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 0x55b019a17d80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7418097b90 tx=0x7f7418007510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f7f4df17fb0e755 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f740bfff700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f740bfff700 1 -- 172.21.15.104:0/2030586020 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7418004a40 con 0x55b019a0af60 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f743d7b7b80 1 -- 172.21.15.104:0/2030586020 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f742000f900 con 0x55b019a0af60 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.783+0000 7f740bfff700 1 -- 172.21.15.104:0/2030586020 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f7418004340 con 0x55b019a0af60 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.784+0000 7f740bfff700 1 -- 172.21.15.104:0/2030586020 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f74180ba080 con 0x55b019a0af60 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.784+0000 7f743d7b7b80 10 client.26878 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.784+0000 7f743d7b7b80 1 -- 172.21.15.104:0/2030586020 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b019a8dde0 con 0x55b019a0af60 2024-06-09T22:28:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.784+0000 7f740bfff700 1 -- 172.21.15.104:0/2030586020 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f742000f900 con 0x55b019a0af60 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.784+0000 7f740bfff700 1 -- 172.21.15.104:0/2030586020 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f74180a3f10 con 0x55b019a0af60 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.784+0000 7f743d7b7b80 10 client.26878 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.784+0000 7f743d7b7b80 10 client.26878 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.784+0000 7f743d7b7b80 20 client.26878 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.784+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55b019a1ef90 0x55b019a904e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.784+0000 7f743d7b7b80 4 client.26878 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.784+0000 7f743d7b7b80 1 -- 172.21.15.104:0/2030586020 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b01995e2d0 con 0x55b019a1ef90 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.784+0000 7f7430cd0700 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55b019a1ef90 0x55b019a904e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f740bfff700 10 client.26878 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f7430cd0700 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55b019a1ef90 0x55b019a904e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=263bed4fb7c2f1ff server_cookie=aa22c15d7701793c in_seq=0 out_seq=0 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f740bfff700 1 -- 172.21.15.104:0/2030586020 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b01995e2d0 con 0x55b019a1ef90 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f740bfff700 10 client.26878 handle_command_reply: tid=0 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f743d7b7b80 1 client.26878 shutdown 2024-06-09T22:28:33.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f743d7b7b80 1 -- 172.21.15.104:0/2030586020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 msgr2=0x55b019a17d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:33.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 0x55b019a17d80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7418097b90 tx=0x7f7418007510 comp rx=0 tx=0).stop 2024-06-09T22:28:33.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f743d7b7b80 1 -- 172.21.15.104:0/2030586020 shutdown_connections 2024-06-09T22:28:33.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55b019a1ef90 0x55b019a904e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b019a0b880 0x55b019a26410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b019a057a0 0x55b019a17840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f740bfff700 0 client.26878 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:28:33.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f743d7b7b80 1 --2- 172.21.15.104:0/2030586020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b019a0af60 0x55b019a17d80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f743d7b7b80 1 -- 172.21.15.104:0/2030586020 >> 172.21.15.104:0/2030586020 conn(0x55b019932080 msgr2=0x55b019a099e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:33.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f743d7b7b80 1 -- 172.21.15.104:0/2030586020 shutdown_connections 2024-06-09T22:28:33.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.785+0000 7f743d7b7b80 1 -- 172.21.15.104:0/2030586020 wait complete. 2024-06-09T22:28:33.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f743d7b7b80 20 client.26878 trim_cache size 0 max 16384 2024-06-09T22:28:33.794 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:28:33.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f7432f34700 1 -- 172.21.15.104:0/2742484667 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f741c0a2d90 msgr2=0x7f741c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:33.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f7432f34700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f741c0a2d90 0x7f741c0a5250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f742000b5c0 tx=0x7f7420005960 comp rx=0 tx=0).stop 2024-06-09T22:28:33.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f7432f34700 1 -- 172.21.15.104:0/2742484667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f742c101d80 msgr2=0x7f742c19ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:33.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f7432f34700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f742c101d80 0x7f742c19ede0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f741800d980 tx=0x7f741800dc90 comp rx=0 tx=0).stop 2024-06-09T22:28:33.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f7432f34700 1 -- 172.21.15.104:0/2742484667 shutdown_connections 2024-06-09T22:28:33.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f7432f34700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f741c0a2d90 0x7f741c0a5250 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f7432f34700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f742c107430 0x7f742c198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f7432f34700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742c1013c0 0x7f742c19e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f7432f34700 1 --2- 172.21.15.104:0/2742484667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f742c101d80 0x7f742c19ede0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:33.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f7432f34700 1 -- 172.21.15.104:0/2742484667 >> 172.21.15.104:0/2742484667 conn(0x7f742c0783d0 msgr2=0x7f742c100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:33.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f7432f34700 1 -- 172.21.15.104:0/2742484667 shutdown_connections 2024-06-09T22:28:33.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:33.786+0000 7f7432f34700 1 -- 172.21.15.104:0/2742484667 wait complete. 2024-06-09T22:28:33.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:33 smithi153 ceph-mon[36599]: pgmap v232: 129 pgs: 129 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-06-09T22:28:34.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:33 smithi194 ceph-mon[36856]: pgmap v232: 129 pgs: 129 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-06-09T22:28:34.710 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-09T22:28:34.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 276805 5.03 MB/sec execute 220 sec latency 43.351 ms 2024-06-09T22:28:34.968 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:28:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 278128 5.04 MB/sec execute 221 sec latency 16.850 ms 2024-06-09T22:28:35.758 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:35 smithi104 ceph-mon[32389]: pgmap v233: 129 pgs: 129 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 277 op/s 2024-06-09T22:28:35.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:35 smithi153 ceph-mon[36599]: pgmap v233: 129 pgs: 129 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 277 op/s 2024-06-09T22:28:36.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:35 smithi194 ceph-mon[36856]: pgmap v233: 129 pgs: 129 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 277 op/s 2024-06-09T22:28:36.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 280347 5.07 MB/sec execute 222 sec latency 58.337 ms 2024-06-09T22:28:36.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.816+0000 7f19430f8700 1 -- 172.21.15.104:0/22625544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f193c113090 msgr2=0x7f193c115530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:36.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.816+0000 7f19430f8700 1 --2- 172.21.15.104:0/22625544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f193c113090 0x7f193c115530 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f193400b0a0 tx=0x7f193400b3b0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.816+0000 7f19430f8700 1 -- 172.21.15.104:0/22625544 shutdown_connections 2024-06-09T22:28:36.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.816+0000 7f19430f8700 1 --2- 172.21.15.104:0/22625544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f193c113090 0x7f193c115530 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.816+0000 7f19430f8700 1 --2- 172.21.15.104:0/22625544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f193c100ce0 0x7f193c1010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.816+0000 7f19430f8700 1 --2- 172.21.15.104:0/22625544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f193c101690 0x7f193c1129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.816+0000 7f19430f8700 1 -- 172.21.15.104:0/22625544 >> 172.21.15.104:0/22625544 conn(0x7f193c077a00 msgr2=0x7f193c0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:36.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.816+0000 7f19430f8700 1 -- 172.21.15.104:0/22625544 shutdown_connections 2024-06-09T22:28:36.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.816+0000 7f19430f8700 1 -- 172.21.15.104:0/22625544 wait complete. 2024-06-09T22:28:36.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f19430f8700 1 Processor -- start 2024-06-09T22:28:36.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f19430f8700 1 -- start start 2024-06-09T22:28:36.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f19430f8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f193c100ce0 0x7f193c19f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f19430f8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f193c101690 0x7f193c19f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f19430f8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f193c113090 0x7f193c1a3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f19430f8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f193c118da0 con 0x7f193c100ce0 2024-06-09T22:28:36.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f19430f8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f193c118c20 con 0x7f193c101690 2024-06-09T22:28:36.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f19430f8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f193c118f20 con 0x7f193c113090 2024-06-09T22:28:36.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f1940e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f193c100ce0 0x7f193c19f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f1940e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f193c100ce0 0x7f193c19f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33592/0 (socket says 172.21.15.104:33592) 2024-06-09T22:28:36.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f1940e94700 1 -- 172.21.15.104:0/1521175590 learned_addr learned my addr 172.21.15.104:0/1521175590 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:36.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f193bfff700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f193c101690 0x7f193c19f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f1941695700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f193c113090 0x7f193c1a3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f1940e94700 1 -- 172.21.15.104:0/1521175590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f193c113090 msgr2=0x7f193c1a3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:36.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f1940e94700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f193c113090 0x7f193c1a3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f1940e94700 1 -- 172.21.15.104:0/1521175590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f193c101690 msgr2=0x7f193c19f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:36.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f1940e94700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f193c101690 0x7f193c19f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f1940e94700 1 -- 172.21.15.104:0/1521175590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193c1a4220 con 0x7f193c100ce0 2024-06-09T22:28:36.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f1941695700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f193c113090 0x7f193c1a3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:28:36.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.817+0000 7f1940e94700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f193c100ce0 0x7f193c19f210 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f193000e9a0 tx=0x7f193000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:36.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.818+0000 7f1939ffb700 1 -- 172.21.15.104:0/1521175590 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193000cbd0 con 0x7f193c100ce0 2024-06-09T22:28:36.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.818+0000 7f19430f8700 1 -- 172.21.15.104:0/1521175590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f193c1a4510 con 0x7f193c100ce0 2024-06-09T22:28:36.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.818+0000 7f1939ffb700 1 -- 172.21.15.104:0/1521175590 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f193000cd70 con 0x7f193c100ce0 2024-06-09T22:28:36.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.818+0000 7f1939ffb700 1 -- 172.21.15.104:0/1521175590 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930018750 con 0x7f193c100ce0 2024-06-09T22:28:36.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.818+0000 7f19430f8700 1 -- 172.21.15.104:0/1521175590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f193c1abd90 con 0x7f193c100ce0 2024-06-09T22:28:36.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.820+0000 7f1939ffb700 1 -- 172.21.15.104:0/1521175590 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f1930010430 con 0x7f193c100ce0 2024-06-09T22:28:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.820+0000 7f1939ffb700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f192c0a2d90 0x7f192c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.821+0000 7f193bfff700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f192c0a2d90 0x7f192c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.821+0000 7f1939ffb700 1 -- 172.21.15.104:0/1521175590 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f19300cfb10 con 0x7f193c100ce0 2024-06-09T22:28:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.821+0000 7f193bfff700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f192c0a2d90 0x7f192c0a5250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f193c112ae0 tx=0x7f1928009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.821+0000 7f194d97bb80 1 Processor -- start 2024-06-09T22:28:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.821+0000 7f194d97bb80 1 -- start start 2024-06-09T22:28:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f194d97bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c9f60 0x55c9373ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f194d97bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373ca880 0x55c9373c4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f194d97bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373c47a0 0x55c9373c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f194d97bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55c9372b6f60 con 0x55c9373c9f60 2024-06-09T22:28:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f194d97bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c937315850 con 0x55c9373c47a0 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f194d97bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c93727d460 con 0x55c9373ca880 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f1940e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c9f60 0x55c9373ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f1941695700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373c47a0 0x55c9373c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f193bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373ca880 0x55c9373c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f1940e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c9f60 0x55c9373ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33596/0 (socket says 172.21.15.104:33596) 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f1940e94700 1 -- 172.21.15.104:0/946093059 learned_addr learned my addr 172.21.15.104:0/946093059 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f1941695700 1 -- 172.21.15.104:0/946093059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373ca880 msgr2=0x55c9373c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f1941695700 1 --2- 172.21.15.104:0/946093059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373ca880 0x55c9373c4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f1941695700 1 -- 172.21.15.104:0/946093059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c9f60 msgr2=0x55c9373ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f1941695700 1 --2- 172.21.15.104:0/946093059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c9f60 0x55c9373ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f1941695700 1 -- 172.21.15.104:0/946093059 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9372648d0 con 0x55c9373c47a0 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f193bfff700 1 --2- 172.21.15.104:0/946093059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373ca880 0x55c9373c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.822+0000 7f1941695700 1 --2- 172.21.15.104:0/946093059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373c47a0 0x55c9373c4bc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1934009fd0 tx=0x7f19340092e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1238d1c4d7c4c64a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:36.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.823+0000 7f1926ffd700 1 -- 172.21.15.104:0/946093059 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934022070 con 0x55c9373c47a0 2024-06-09T22:28:36.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.823+0000 7f1926ffd700 1 -- 172.21.15.104:0/946093059 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f193400bcb0 con 0x55c9373c47a0 2024-06-09T22:28:36.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.823+0000 7f1926ffd700 1 -- 172.21.15.104:0/946093059 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193401c440 con 0x55c9373c47a0 2024-06-09T22:28:36.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.823+0000 7f194d97bb80 1 -- 172.21.15.104:0/946093059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373c47a0 msgr2=0x55c9373c4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.823+0000 7f194d97bb80 1 --2- 172.21.15.104:0/946093059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373c47a0 0x55c9373c4bc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1934009fd0 tx=0x7f19340092e0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.823+0000 7f194d97bb80 1 -- 172.21.15.104:0/946093059 shutdown_connections 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.823+0000 7f194d97bb80 1 --2- 172.21.15.104:0/946093059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c9f60 0x55c9373ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.823+0000 7f194d97bb80 1 --2- 172.21.15.104:0/946093059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373c47a0 0x55c9373c4bc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.823+0000 7f194d97bb80 1 --2- 172.21.15.104:0/946093059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373ca880 0x55c9373c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.823+0000 7f194d97bb80 1 -- 172.21.15.104:0/946093059 >> 172.21.15.104:0/946093059 conn(0x55c9372f1080 msgr2=0x55c9373bf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.823+0000 7f194d97bb80 1 -- 172.21.15.104:0/946093059 shutdown_connections 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.823+0000 7f194d97bb80 1 -- 172.21.15.104:0/946093059 wait complete. 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f194d97bb80 1 -- start start 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f194d97bb80 1 Processor -- start 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f194d97bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f194d97bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f194d97bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f194d97bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55c9372b5fc0 con 0x55c9373c47a0 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f194d97bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c937311810 con 0x55c9373ca880 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f194d97bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c9373d1200 con 0x55c9373c9f60 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f194d97bb80 10 client.0 _ll_register_callbacks cb 0x55c9372ab5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f1940e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f193bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f193bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:40236/0 (socket says 172.21.15.104:40236) 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f1941695700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.824+0000 7f193bfff700 1 -- 172.21.15.104:0/4115811893 learned_addr learned my addr 172.21.15.104:0/4115811893 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f193bfff700 1 -- 172.21.15.104:0/4115811893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 msgr2=0x55c9373cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f193bfff700 1 --2- 172.21.15.104:0/4115811893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f193bfff700 1 -- 172.21.15.104:0/4115811893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 msgr2=0x55c9373d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f193bfff700 1 --2- 172.21.15.104:0/4115811893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f193bfff700 1 -- 172.21.15.104:0/4115811893 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9372db750 con 0x55c9373c9f60 2024-06-09T22:28:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f1941695700 1 --2- 172.21.15.104:0/4115811893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f193bfff700 1 --2- 172.21.15.104:0/4115811893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373cd8d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f192801d7b0 tx=0x7f19280047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=938efb9fd219ff27 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f191bfff700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f191bfff700 1 -- 172.21.15.104:0/4115811893 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1928004020 con 0x55c9373c9f60 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f194d97bb80 1 -- 172.21.15.104:0/4115811893 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f193400bcb0 con 0x55c9373c9f60 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f191bfff700 1 -- 172.21.15.104:0/4115811893 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f1928009040 con 0x55c9373c9f60 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f191bfff700 1 -- 172.21.15.104:0/4115811893 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1928030050 con 0x55c9373c9f60 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f194d97bb80 10 client.26890 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.825+0000 7f194d97bb80 1 -- 172.21.15.104:0/4115811893 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c9373dba00 con 0x55c9373c9f60 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.826+0000 7f191bfff700 1 -- 172.21.15.104:0/4115811893 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f193400bcb0 con 0x55c9373c9f60 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.826+0000 7f191bfff700 1 -- 172.21.15.104:0/4115811893 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f192801c690 con 0x55c9373c9f60 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.826+0000 7f194d97bb80 10 client.26890 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.826+0000 7f194d97bb80 10 client.26890 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.826+0000 7f194d97bb80 20 client.26890 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.826+0000 7f194d97bb80 1 --2- 172.21.15.104:0/4115811893 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c9373de0e0 0x55c9373e0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.826+0000 7f194d97bb80 4 client.26890 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.826+0000 7f194d97bb80 1 -- 172.21.15.104:0/4115811893 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c937322d50 con 0x55c9373de0e0 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.826+0000 7f1940e94700 1 --2- 172.21.15.104:0/4115811893 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c9373de0e0 0x55c9373e0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f1940e94700 1 --2- 172.21.15.104:0/4115811893 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c9373de0e0 0x55c9373e0500 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bc7fab97d1d1cec server_cookie=14312b3a7f55c4c6 in_seq=0 out_seq=0 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f191bfff700 10 client.26890 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f191bfff700 1 -- 172.21.15.104:0/4115811893 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55c937322d50 con 0x55c9373de0e0 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f191bfff700 10 client.26890 handle_command_reply: tid=0 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f194d97bb80 1 client.26890 shutdown 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f194d97bb80 1 -- 172.21.15.104:0/4115811893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 msgr2=0x55c9373cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f194d97bb80 1 --2- 172.21.15.104:0/4115811893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373cd8d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f192801d7b0 tx=0x7f19280047c0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f194d97bb80 1 -- 172.21.15.104:0/4115811893 shutdown_connections 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f194d97bb80 1 --2- 172.21.15.104:0/4115811893 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c9373de0e0 0x55c9373e0500 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f194d97bb80 1 --2- 172.21.15.104:0/4115811893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f194d97bb80 1 --2- 172.21.15.104:0/4115811893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f194d97bb80 1 --2- 172.21.15.104:0/4115811893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373cd8d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f191bfff700 0 client.26890 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:36.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.827+0000 7f194d97bb80 1 -- 172.21.15.104:0/4115811893 >> 172.21.15.104:0/4115811893 conn(0x55c9372f1080 msgr2=0x55c9373be700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:36.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.828+0000 7f194d97bb80 1 -- 172.21.15.104:0/4115811893 shutdown_connections 2024-06-09T22:28:36.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.828+0000 7f194d97bb80 1 -- 172.21.15.104:0/4115811893 wait complete. 2024-06-09T22:28:36.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.828+0000 7f194d97bb80 20 client.26890 trim_cache size 0 max 16384 2024-06-09T22:28:36.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.843+0000 7f194d97bb80 1 Processor -- start 2024-06-09T22:28:36.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.843+0000 7f194d97bb80 1 -- start start 2024-06-09T22:28:36.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.843+0000 7f194d97bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373c4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.843+0000 7f194d97bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.843+0000 7f194d97bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.843+0000 7f194d97bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f193c1b4b30 con 0x55c9373c47a0 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.843+0000 7f194d97bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f193c1b6670 con 0x55c9373ca880 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.843+0000 7f194d97bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f193c1b5200 con 0x55c9373c9f60 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f1940e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f193bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f1940e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33620/0 (socket says 172.21.15.104:33620) 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f1940e94700 1 -- 172.21.15.104:0/2513274832 learned_addr learned my addr 172.21.15.104:0/2513274832 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f1941695700 1 --2- 172.21.15.104:0/2513274832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f1940e94700 1 -- 172.21.15.104:0/2513274832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 msgr2=0x55c9373c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f1940e94700 1 --2- 172.21.15.104:0/2513274832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f1940e94700 1 -- 172.21.15.104:0/2513274832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 msgr2=0x55c9373cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f1940e94700 1 --2- 172.21.15.104:0/2513274832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f1940e94700 1 -- 172.21.15.104:0/2513274832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9373e0b60 con 0x55c9373c47a0 2024-06-09T22:28:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f1941695700 1 --2- 172.21.15.104:0/2513274832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:36.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f1940e94700 1 --2- 172.21.15.104:0/2513274832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373c4260 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f19300ccf40 tx=0x7f193001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c17f4fc215cbae server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:36.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f19267fc700 1 -- 172.21.15.104:0/2513274832 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19300b9070 con 0x55c9373c47a0 2024-06-09T22:28:36.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f19267fc700 1 -- 172.21.15.104:0/2513274832 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19300b1070 con 0x55c9373c47a0 2024-06-09T22:28:36.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.844+0000 7f19267fc700 1 -- 172.21.15.104:0/2513274832 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19300b4e60 con 0x55c9373c47a0 2024-06-09T22:28:36.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.845+0000 7f194d97bb80 1 -- 172.21.15.104:0/2513274832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 msgr2=0x55c9373c4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:36.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.845+0000 7f194d97bb80 1 --2- 172.21.15.104:0/2513274832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373c4260 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f19300ccf40 tx=0x7f193001a040 comp rx=0 tx=0).stop 2024-06-09T22:28:36.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.845+0000 7f194d97bb80 1 -- 172.21.15.104:0/2513274832 shutdown_connections 2024-06-09T22:28:36.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.845+0000 7f194d97bb80 1 --2- 172.21.15.104:0/2513274832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373c4260 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.845+0000 7f194d97bb80 1 --2- 172.21.15.104:0/2513274832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.845+0000 7f194d97bb80 1 --2- 172.21.15.104:0/2513274832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.845+0000 7f194d97bb80 1 -- 172.21.15.104:0/2513274832 >> 172.21.15.104:0/2513274832 conn(0x55c9372f1080 msgr2=0x55c9373cf100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:36.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.845+0000 7f194d97bb80 1 -- 172.21.15.104:0/2513274832 shutdown_connections 2024-06-09T22:28:36.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.845+0000 7f194d97bb80 1 -- 172.21.15.104:0/2513274832 wait complete. 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.845+0000 7f194d97bb80 1 -- start start 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.845+0000 7f194d97bb80 1 Processor -- start 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f194d97bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373d6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f194d97bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373d6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f194d97bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f194d97bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f193c1b5fa0 con 0x55c9373c47a0 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f194d97bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f193c1b58d0 con 0x55c9373ca880 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f194d97bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f193c1b2af0 con 0x55c9373c9f60 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f194d97bb80 10 client.0 _ll_register_callbacks cb 0x55c9372ab5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f1940e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373d6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f193bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373d6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f1940e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373d6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33630/0 (socket says 172.21.15.104:33630) 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f193bfff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373d6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:40252/0 (socket says 172.21.15.104:40252) 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f1940e94700 1 -- 172.21.15.104:0/519120038 learned_addr learned my addr 172.21.15.104:0/519120038 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f1941695700 1 --2- 172.21.15.104:0/519120038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f1940e94700 1 -- 172.21.15.104:0/519120038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 msgr2=0x55c9373d6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f1940e94700 1 --2- 172.21.15.104:0/519120038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373d6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f1940e94700 1 -- 172.21.15.104:0/519120038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 msgr2=0x55c9373e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f1940e94700 1 --2- 172.21.15.104:0/519120038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373e4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f1940e94700 1 -- 172.21.15.104:0/519120038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9373e52c0 con 0x55c9373c47a0 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f1940e94700 1 --2- 172.21.15.104:0/519120038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373d6520 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f19300098c0 tx=0x7f193009bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d5caf724ab6018 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f191bfff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.846+0000 7f191bfff700 1 -- 172.21.15.104:0/519120038 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19300b9030 con 0x55c9373c47a0 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f191bfff700 1 -- 172.21.15.104:0/519120038 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f1930099f60 con 0x55c9373c47a0 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f194d97bb80 1 -- 172.21.15.104:0/519120038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19300b1070 con 0x55c9373c47a0 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f191bfff700 1 -- 172.21.15.104:0/519120038 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f19300b4e60 con 0x55c9373c47a0 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f194d97bb80 10 client.17568 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f194d97bb80 1 -- 172.21.15.104:0/519120038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c9374484a0 con 0x55c9373c47a0 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f191bfff700 1 -- 172.21.15.104:0/519120038 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19300b1070 con 0x55c9373c47a0 2024-06-09T22:28:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f191bfff700 1 -- 172.21.15.104:0/519120038 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f19300a4440 con 0x55c9373c47a0 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f194d97bb80 10 client.17568 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f194d97bb80 10 client.17568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f194d97bb80 20 client.17568 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f194d97bb80 1 --2- 172.21.15.104:0/519120038 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c9374489f0 0x55c93744ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f194d97bb80 4 client.17568 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f194d97bb80 1 -- 172.21.15.104:0/519120038 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c93731d2d0 con 0x55c9374489f0 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.847+0000 7f193bfff700 1 --2- 172.21.15.104:0/519120038 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c9374489f0 0x55c93744ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f193bfff700 1 --2- 172.21.15.104:0/519120038 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c9374489f0 0x55c93744ae10 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a67245e016cc8aac server_cookie=63ec17afb25f55f8 in_seq=0 out_seq=0 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f191bfff700 10 client.17568 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f191bfff700 1 -- 172.21.15.104:0/519120038 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c93731d2d0 con 0x55c9374489f0 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f191bfff700 10 client.17568 handle_command_reply: tid=0 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f194d97bb80 1 client.17568 shutdown 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f194d97bb80 1 -- 172.21.15.104:0/519120038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 msgr2=0x55c9373d6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f194d97bb80 1 --2- 172.21.15.104:0/519120038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373d6520 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f19300098c0 tx=0x7f193009bc60 comp rx=0 tx=0).stop 2024-06-09T22:28:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f194d97bb80 1 -- 172.21.15.104:0/519120038 shutdown_connections 2024-06-09T22:28:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f194d97bb80 1 --2- 172.21.15.104:0/519120038 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55c9374489f0 0x55c93744ae10 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f194d97bb80 1 --2- 172.21.15.104:0/519120038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c9373c47a0 0x55c9373d6520 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f191bfff700 0 client.17568 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f194d97bb80 1 --2- 172.21.15.104:0/519120038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9373ca880 0x55c9373e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f194d97bb80 1 --2- 172.21.15.104:0/519120038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c9373c9f60 0x55c9373d6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f194d97bb80 1 -- 172.21.15.104:0/519120038 >> 172.21.15.104:0/519120038 conn(0x55c9372f1080 msgr2=0x55c9373cb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f194d97bb80 1 -- 172.21.15.104:0/519120038 shutdown_connections 2024-06-09T22:28:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.848+0000 7f194d97bb80 1 -- 172.21.15.104:0/519120038 wait complete. 2024-06-09T22:28:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f194d97bb80 20 client.17568 trim_cache size 0 max 16384 2024-06-09T22:28:36.855 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:28:36.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f19430f8700 1 -- 172.21.15.104:0/1521175590 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f192c0a2d90 msgr2=0x7f192c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:36.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f19430f8700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f192c0a2d90 0x7f192c0a5250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f193c112ae0 tx=0x7f1928009260 comp rx=0 tx=0).stop 2024-06-09T22:28:36.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f19430f8700 1 -- 172.21.15.104:0/1521175590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f193c100ce0 msgr2=0x7f193c19f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:36.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f19430f8700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f193c100ce0 0x7f193c19f210 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f193000e9a0 tx=0x7f193000ecb0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f19430f8700 1 -- 172.21.15.104:0/1521175590 shutdown_connections 2024-06-09T22:28:36.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f19430f8700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f192c0a2d90 0x7f192c0a5250 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f19430f8700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f193c100ce0 0x7f193c19f210 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f19430f8700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f193c113090 0x7f193c1a3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f19430f8700 1 --2- 172.21.15.104:0/1521175590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f193c101690 0x7f193c19f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f19430f8700 1 -- 172.21.15.104:0/1521175590 >> 172.21.15.104:0/1521175590 conn(0x7f193c077a00 msgr2=0x7f193c111890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f19430f8700 1 -- 172.21.15.104:0/1521175590 shutdown_connections 2024-06-09T22:28:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:36.849+0000 7f19430f8700 1 -- 172.21.15.104:0/1521175590 wait complete. 2024-06-09T22:28:37.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 281645 5.08 MB/sec execute 223 sec latency 21.442 ms 2024-06-09T22:28:37.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:37 smithi153 ceph-mon[36599]: pgmap v234: 129 pgs: 129 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 182 op/s 2024-06-09T22:28:38.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:37 smithi104 ceph-mon[32389]: pgmap v234: 129 pgs: 129 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 182 op/s 2024-06-09T22:28:38.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:37 smithi194 ceph-mon[36856]: pgmap v234: 129 pgs: 129 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 182 op/s 2024-06-09T22:28:38.594 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-09T22:28:38.594 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-09T22:28:38.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 283767 5.11 MB/sec execute 224 sec latency 23.301 ms 2024-06-09T22:28:38.864 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:28:39.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 284787 5.10 MB/sec execute 225 sec latency 33.991 ms 2024-06-09T22:28:39.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:39 smithi153 ceph-mon[36599]: pgmap v235: 129 pgs: 129 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 182 op/s 2024-06-09T22:28:40.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:39 smithi104 ceph-mon[32389]: pgmap v235: 129 pgs: 129 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 182 op/s 2024-06-09T22:28:40.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:39 smithi194 ceph-mon[36856]: pgmap v235: 129 pgs: 129 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 182 op/s 2024-06-09T22:28:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.071+0000 7fb3dee2c640 1 -- 172.21.15.104:0/2249979415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3d8106890 msgr2=0x7fb3d8106c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.071+0000 7fb3dee2c640 1 --2- 172.21.15.104:0/2249979415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3d8106890 0x7fb3d8106c70 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8009930 tx=0x7fb3c802a420 comp rx=0 tx=0).stop 2024-06-09T22:28:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.071+0000 7fb3dee2c640 1 -- 172.21.15.104:0/2249979415 shutdown_connections 2024-06-09T22:28:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.071+0000 7fb3dee2c640 1 --2- 172.21.15.104:0/2249979415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8101290 0x7fb3d810d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.071+0000 7fb3dee2c640 1 --2- 172.21.15.104:0/2249979415 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3d81008d0 0x7fb3d8100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.071+0000 7fb3dee2c640 1 --2- 172.21.15.104:0/2249979415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3d8106890 0x7fb3d8106c70 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.071+0000 7fb3dee2c640 1 -- 172.21.15.104:0/2249979415 >> 172.21.15.104:0/2249979415 conn(0x7fb3d8077a20 msgr2=0x7fb3d8077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.071+0000 7fb3dee2c640 1 -- 172.21.15.104:0/2249979415 shutdown_connections 2024-06-09T22:28:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.071+0000 7fb3dee2c640 1 -- 172.21.15.104:0/2249979415 wait complete. 2024-06-09T22:28:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3dee2c640 1 Processor -- start 2024-06-09T22:28:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3dee2c640 1 -- start start 2024-06-09T22:28:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3dee2c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3d81008d0 0x7fb3d8197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3dee2c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8101290 0x7fb3d8197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3dee2c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3d8106890 0x7fb3d819ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3dee2c640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d8110cf0 con 0x7fb3d8106890 2024-06-09T22:28:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3dee2c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d8110b70 con 0x7fb3d8101290 2024-06-09T22:28:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3dee2c640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d8110e70 con 0x7fb3d81008d0 2024-06-09T22:28:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3dd3c9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3d8106890 0x7fb3d819ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3dd3c9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3d8106890 0x7fb3d819ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33644/0 (socket says 172.21.15.104:33644) 2024-06-09T22:28:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3dd3c9640 1 -- 172.21.15.104:0/1055721930 learned_addr learned my addr 172.21.15.104:0/1055721930 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3dcbc8640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3d81008d0 0x7fb3d8197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.072+0000 7fb3d7fff640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8101290 0x7fb3d8197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.073+0000 7fb3d7fff640 1 -- 172.21.15.104:0/1055721930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3d81008d0 msgr2=0x7fb3d8197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.073+0000 7fb3d7fff640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3d81008d0 0x7fb3d8197150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.073+0000 7fb3d7fff640 1 -- 172.21.15.104:0/1055721930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3d8106890 msgr2=0x7fb3d819ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.073+0000 7fb3d7fff640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3d8106890 0x7fb3d819ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.073+0000 7fb3d7fff640 1 -- 172.21.15.104:0/1055721930 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d819c140 con 0x7fb3d8101290 2024-06-09T22:28:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.073+0000 7fb3dd3c9640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3d8106890 0x7fb3d819ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.073+0000 7fb3d7fff640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8101290 0x7fb3d8197690 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009fd0 tx=0x7fb3cc00ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.073+0000 7fb3d5ffb640 1 -- 172.21.15.104:0/1055721930 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc004b20 con 0x7fb3d8101290 2024-06-09T22:28:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.073+0000 7fb3dee2c640 1 -- 172.21.15.104:0/1055721930 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d819c430 con 0x7fb3d8101290 2024-06-09T22:28:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.073+0000 7fb3d5ffb640 1 -- 172.21.15.104:0/1055721930 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3cc004cc0 con 0x7fb3d8101290 2024-06-09T22:28:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.073+0000 7fb3d5ffb640 1 -- 172.21.15.104:0/1055721930 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc0107b0 con 0x7fb3d8101290 2024-06-09T22:28:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.073+0000 7fb3dee2c640 1 -- 172.21.15.104:0/1055721930 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d81a3cc0 con 0x7fb3d8101290 2024-06-09T22:28:40.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.076+0000 7fb3d5ffb640 1 -- 172.21.15.104:0/1055721930 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fb3cc005020 con 0x7fb3d8101290 2024-06-09T22:28:40.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.076+0000 7fb3e1c0e740 1 Processor -- start 2024-06-09T22:28:40.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.076+0000 7fb3e1c0e740 1 -- start start 2024-06-09T22:28:40.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.076+0000 7fb3e1c0e740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 0x55e88f7fe760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.076+0000 7fb3e1c0e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7feca0 0x55e88f809020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.076+0000 7fb3e1c0e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f809560 0x55e88f80b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.076+0000 7fb3e1c0e740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e88f7604f0 con 0x55e88f7fe380 2024-06-09T22:28:40.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.076+0000 7fb3e1c0e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e88f6fe870 con 0x55e88f809560 2024-06-09T22:28:40.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.076+0000 7fb3e1c0e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e88f75c4b0 con 0x55e88f7feca0 2024-06-09T22:28:40.085 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dd3c9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 0x55e88f7fe760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dcbc8640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7feca0 0x55e88f809020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3d5ffb640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb3c00a2da0 0x7fb3c00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dd3c9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 0x55e88f7fe760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33656/0 (socket says 172.21.15.104:33656) 2024-06-09T22:28:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dd3c9640 1 -- 172.21.15.104:0/3594618942 learned_addr learned my addr 172.21.15.104:0/3594618942 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3d7fff640 1 --2- 172.21.15.104:0/3594618942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f809560 0x55e88f80b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dd3c9640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb3c00a2da0 0x7fb3c00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3d5ffb640 1 -- 172.21.15.104:0/1055721930 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7fb3cc0cff80 con 0x7fb3d8101290 2024-06-09T22:28:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dcbc8640 1 -- 172.21.15.104:0/3594618942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f809560 msgr2=0x55e88f80b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dcbc8640 1 --2- 172.21.15.104:0/3594618942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f809560 0x55e88f80b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dcbc8640 1 -- 172.21.15.104:0/3594618942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 msgr2=0x55e88f7fe760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.086 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dcbc8640 1 --2- 172.21.15.104:0/3594618942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 0x55e88f7fe760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dcbc8640 1 -- 172.21.15.104:0/3594618942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e88f71a9b0 con 0x55e88f7feca0 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dd3c9640 1 --2- 172.21.15.104:0/3594618942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 0x55e88f7fe760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dcbc8640 1 --2- 172.21.15.104:0/3594618942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7feca0 0x55e88f809020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb3c802a940 tx=0x7fb3c8005de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=815af233ee3cad9d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3dd3c9640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb3c00a2da0 0x7fb3c00a5260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0005d00 tx=0x7fb3d0004730 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3beffd640 1 -- 172.21.15.104:0/3594618942 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8035040 con 0x55e88f7feca0 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3beffd640 1 -- 172.21.15.104:0/3594618942 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3c8005520 con 0x55e88f7feca0 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.077+0000 7fb3beffd640 1 -- 172.21.15.104:0/3594618942 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80035f0 con 0x55e88f7feca0 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.078+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3594618942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7feca0 msgr2=0x55e88f809020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.078+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/3594618942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7feca0 0x55e88f809020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb3c802a940 tx=0x7fb3c8005de0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.078+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3594618942 shutdown_connections 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.078+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/3594618942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f809560 0x55e88f80b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.078+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/3594618942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7feca0 0x55e88f809020 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.078+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/3594618942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 0x55e88f7fe760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.078+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3594618942 >> 172.21.15.104:0/3594618942 conn(0x55e88f7fa890 msgr2=0x55e88f7fac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:40.087 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.078+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3594618942 shutdown_connections 2024-06-09T22:28:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.078+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3594618942 wait complete. 2024-06-09T22:28:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.078+0000 7fb3e1c0e740 1 -- start start 2024-06-09T22:28:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.078+0000 7fb3e1c0e740 1 Processor -- start 2024-06-09T22:28:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3e1c0e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7fe380 0x55e88f8064e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3e1c0e740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7feca0 0x55e88f806a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3e1c0e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f809560 0x55e88f806f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3e1c0e740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55e88f80c130 con 0x55e88f7feca0 2024-06-09T22:28:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3e1c0e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e88f6e9ae0 con 0x55e88f809560 2024-06-09T22:28:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3e1c0e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e88f80c2b0 con 0x55e88f7fe380 2024-06-09T22:28:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3e1c0e740 10 client.0 _ll_register_callbacks cb 0x55e88f7229b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7feca0 0x55e88f806a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.088 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3dcbc8640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7fe380 0x55e88f8064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7feca0 0x55e88f806a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33668/0 (socket says 172.21.15.104:33668) 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3d7fff640 1 -- 172.21.15.104:0/3767800323 learned_addr learned my addr 172.21.15.104:0/3767800323 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3dd3c9640 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f809560 0x55e88f806f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3dcbc8640 1 -- 172.21.15.104:0/3767800323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f809560 msgr2=0x55e88f806f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3dcbc8640 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f809560 0x55e88f806f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3dcbc8640 1 -- 172.21.15.104:0/3767800323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7feca0 msgr2=0x55e88f806a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3dcbc8640 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7feca0 0x55e88f806a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3dcbc8640 1 -- 172.21.15.104:0/3767800323 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e88f474210 con 0x55e88f7fe380 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3d7fff640 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7feca0 0x55e88f806a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3dcbc8640 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7fe380 0x55e88f8064e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8031550 tx=0x7fb3c80048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ceece34a08cbf622 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.079+0000 7fb3b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3b7fff640 1 -- 172.21.15.104:0/3767800323 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8052060 con 0x55e88f7fe380 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3767800323 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3c8005520 con 0x55e88f7fe380 2024-06-09T22:28:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3b7fff640 1 -- 172.21.15.104:0/3767800323 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7fb3c8004390 con 0x55e88f7fe380 2024-06-09T22:28:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3b7fff640 1 -- 172.21.15.104:0/3767800323 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb3c8035070 con 0x55e88f7fe380 2024-06-09T22:28:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3e1c0e740 10 client.26908 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3767800323 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e88f540620 con 0x55e88f7fe380 2024-06-09T22:28:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3b7fff640 1 -- 172.21.15.104:0/3767800323 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3c8005520 con 0x55e88f7fe380 2024-06-09T22:28:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3b7fff640 1 -- 172.21.15.104:0/3767800323 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb3c80038a0 con 0x55e88f7fe380 2024-06-09T22:28:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3e1c0e740 10 client.26908 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3e1c0e740 10 client.26908 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3e1c0e740 20 client.26908 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e88f8194f0 0x55e88f81b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3e1c0e740 4 client.26908 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.080+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3767800323 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e88f773f10 con 0x55e88f8194f0 2024-06-09T22:28:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.081+0000 7fb3d7fff640 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e88f8194f0 0x55e88f81b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.081+0000 7fb3b7fff640 10 client.26908 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.081+0000 7fb3d7fff640 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e88f8194f0 0x55e88f81b8e0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4c2f00e2133e85d server_cookie=eca222035e48e12 in_seq=0 out_seq=0 2024-06-09T22:28:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3b7fff640 1 -- 172.21.15.104:0/3767800323 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55e88f773f10 con 0x55e88f8194f0 2024-06-09T22:28:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3b7fff640 10 client.26908 handle_command_reply: tid=0 2024-06-09T22:28:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3e1c0e740 1 client.26908 shutdown 2024-06-09T22:28:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3767800323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7fe380 msgr2=0x55e88f8064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7fe380 0x55e88f8064e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8031550 tx=0x7fb3c80048f0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3767800323 shutdown_connections 2024-06-09T22:28:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e88f8194f0 0x55e88f81b8e0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f809560 0x55e88f806f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3b7fff640 0 client.26908 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7feca0 0x55e88f806a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.092 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/3767800323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7fe380 0x55e88f8064e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.092 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3767800323 >> 172.21.15.104:0/3767800323 conn(0x55e88f7fa890 msgr2=0x55e88f7f77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:40.092 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3767800323 shutdown_connections 2024-06-09T22:28:40.092 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/3767800323 wait complete. 2024-06-09T22:28:40.092 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.082+0000 7fb3e1c0e740 20 client.26908 trim_cache size 0 max 16384 2024-06-09T22:28:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.098+0000 7fb3e1c0e740 1 Processor -- start 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.098+0000 7fb3e1c0e740 1 -- start start 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3e1c0e740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 0x55e88f820e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3e1c0e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f7feca0 0x55e88f8213c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3e1c0e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f809560 0x55e88f7f6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3e1c0e740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81acc00 con 0x55e88f7fe380 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3e1c0e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81ae740 con 0x55e88f7feca0 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3e1c0e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81ad2d0 con 0x55e88f809560 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3dcbc8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 0x55e88f820e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f7feca0 0x55e88f8213c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3dcbc8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 0x55e88f820e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33676/0 (socket says 172.21.15.104:33676) 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3dcbc8640 1 -- 172.21.15.104:0/1603413417 learned_addr learned my addr 172.21.15.104:0/1603413417 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3d7fff640 1 --2- 172.21.15.104:0/1603413417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f7feca0 0x55e88f8213c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:53184/0 (socket says 172.21.15.104:53184) 2024-06-09T22:28:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3dd3c9640 1 --2- 172.21.15.104:0/1603413417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f809560 0x55e88f7f6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3dcbc8640 1 -- 172.21.15.104:0/1603413417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f809560 msgr2=0x55e88f7f6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3dcbc8640 1 --2- 172.21.15.104:0/1603413417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f809560 0x55e88f7f6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3dcbc8640 1 -- 172.21.15.104:0/1603413417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f7feca0 msgr2=0x55e88f8213c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3dcbc8640 1 --2- 172.21.15.104:0/1603413417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f7feca0 0x55e88f8213c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3dcbc8640 1 -- 172.21.15.104:0/1603413417 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e88f81bc50 con 0x55e88f7fe380 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3d7fff640 1 --2- 172.21.15.104:0/1603413417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f7feca0 0x55e88f8213c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3dcbc8640 1 --2- 172.21.15.104:0/1603413417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 0x55e88f820e80 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8006010 tx=0x7fb3c8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cbc238854a5e13 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3be7fc640 1 -- 172.21.15.104:0/1603413417 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c805b070 con 0x55e88f7fe380 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3be7fc640 1 -- 172.21.15.104:0/1603413417 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3c8003f00 con 0x55e88f7fe380 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.099+0000 7fb3be7fc640 1 -- 172.21.15.104:0/1603413417 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c803ca90 con 0x55e88f7fe380 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.100+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/1603413417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 msgr2=0x55e88f820e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.100+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/1603413417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 0x55e88f820e80 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8006010 tx=0x7fb3c8000bc0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.100+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/1603413417 shutdown_connections 2024-06-09T22:28:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.100+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/1603413417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f809560 0x55e88f7f6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.100+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/1603413417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f7feca0 0x55e88f8213c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.100+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/1603413417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f7fe380 0x55e88f820e80 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.100+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/1603413417 >> 172.21.15.104:0/1603413417 conn(0x55e88f7fa890 msgr2=0x55e88f801740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.100+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/1603413417 shutdown_connections 2024-06-09T22:28:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.100+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/1603413417 wait complete. 2024-06-09T22:28:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.100+0000 7fb3e1c0e740 1 -- start start 2024-06-09T22:28:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3e1c0e740 1 Processor -- start 2024-06-09T22:28:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3e1c0e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f7fe380 0x55e88f80cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3e1c0e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7feca0 0x55e88f80d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3e1c0e740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f809560 0x55e88f87be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3e1c0e740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81ae070 con 0x55e88f809560 2024-06-09T22:28:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3e1c0e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81ad9a0 con 0x55e88f7fe380 2024-06-09T22:28:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3e1c0e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81aac00 con 0x55e88f7feca0 2024-06-09T22:28:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3e1c0e740 10 client.0 _ll_register_callbacks cb 0x7fb3d0009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3dd3c9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f809560 0x55e88f87be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3dd3c9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f809560 0x55e88f87be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33688/0 (socket says 172.21.15.104:33688) 2024-06-09T22:28:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3dd3c9640 1 -- 172.21.15.104:0/4264150178 learned_addr learned my addr 172.21.15.104:0/4264150178 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3dcbc8640 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f7fe380 0x55e88f80cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3d7fff640 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7feca0 0x55e88f80d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3dd3c9640 1 -- 172.21.15.104:0/4264150178 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7feca0 msgr2=0x55e88f80d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3dd3c9640 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7feca0 0x55e88f80d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3dd3c9640 1 -- 172.21.15.104:0/4264150178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f7fe380 msgr2=0x55e88f80cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3dd3c9640 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f7fe380 0x55e88f80cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.101+0000 7fb3dd3c9640 1 -- 172.21.15.104:0/4264150178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e88f87c400 con 0x55e88f809560 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3dd3c9640 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f809560 0x55e88f87be10 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0022bb0 tx=0x7fb3d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cbc063fe58de3cf server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3b7fff640 1 -- 172.21.15.104:0/4264150178 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0013d00 con 0x55e88f809560 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3b7fff640 1 -- 172.21.15.104:0/4264150178 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7fb3d0010930 con 0x55e88f809560 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/4264150178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e88f76dc20 con 0x55e88f809560 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3b7fff640 1 -- 172.21.15.104:0/4264150178 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb3d0036050 con 0x55e88f809560 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3e1c0e740 10 client.17586 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/4264150178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e88f8837b0 con 0x55e88f809560 2024-06-09T22:28:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3b7fff640 1 -- 172.21.15.104:0/4264150178 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e88f76dc20 con 0x55e88f809560 2024-06-09T22:28:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3b7fff640 1 -- 172.21.15.104:0/4264150178 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb3d001f040 con 0x55e88f809560 2024-06-09T22:28:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3e1c0e740 10 client.17586 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3e1c0e740 10 client.17586 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3e1c0e740 20 client.17586 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e88f883f60 0x55e88f886380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3e1c0e740 4 client.17586 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:28:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/4264150178 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e88f767f70 con 0x55e88f883f60 2024-06-09T22:28:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.102+0000 7fb3dcbc8640 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e88f883f60 0x55e88f886380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.103+0000 7fb3dcbc8640 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e88f883f60 0x55e88f886380 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c10a32d60503490 server_cookie=38ad1618f457eba9 in_seq=0 out_seq=0 2024-06-09T22:28:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.103+0000 7fb3b7fff640 10 client.17586 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3b7fff640 1 -- 172.21.15.104:0/4264150178 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55e88f767f70 con 0x55e88f883f60 2024-06-09T22:28:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3b7fff640 10 client.17586 handle_command_reply: tid=0 2024-06-09T22:28:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3e1c0e740 1 client.17586 shutdown 2024-06-09T22:28:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/4264150178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f809560 msgr2=0x55e88f87be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f809560 0x55e88f87be10 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0022bb0 tx=0x7fb3d00046c0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/4264150178 shutdown_connections 2024-06-09T22:28:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55e88f883f60 0x55e88f886380 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e88f809560 0x55e88f87be10 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e88f7feca0 0x55e88f80d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3e1c0e740 1 --2- 172.21.15.104:0/4264150178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e88f7fe380 0x55e88f80cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3b7fff640 0 client.17586 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/4264150178 >> 172.21.15.104:0/4264150178 conn(0x55e88f7fa890 msgr2=0x55e88f7f6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/4264150178 shutdown_connections 2024-06-09T22:28:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3e1c0e740 1 -- 172.21.15.104:0/4264150178 wait complete. 2024-06-09T22:28:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.112+0000 7fb3e1c0e740 20 client.17586 trim_cache size 0 max 16384 2024-06-09T22:28:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:28:40.116 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:28:40.116 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:28:40.116 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.113+0000 7fb3dee2c640 1 -- 172.21.15.104:0/1055721930 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb3c00a2da0 msgr2=0x7fb3c00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.113+0000 7fb3dee2c640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb3c00a2da0 0x7fb3c00a5260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0005d00 tx=0x7fb3d0004730 comp rx=0 tx=0).stop 2024-06-09T22:28:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.113+0000 7fb3dee2c640 1 -- 172.21.15.104:0/1055721930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8101290 msgr2=0x7fb3d8197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.113+0000 7fb3dee2c640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8101290 0x7fb3d8197690 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009fd0 tx=0x7fb3cc00ef90 comp rx=0 tx=0).stop 2024-06-09T22:28:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.113+0000 7fb3dee2c640 1 -- 172.21.15.104:0/1055721930 shutdown_connections 2024-06-09T22:28:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.113+0000 7fb3dee2c640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb3c00a2da0 0x7fb3c00a5260 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.113+0000 7fb3dee2c640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3d8106890 0x7fb3d819ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.113+0000 7fb3dee2c640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d8101290 0x7fb3d8197690 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.113+0000 7fb3dee2c640 1 --2- 172.21.15.104:0/1055721930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3d81008d0 0x7fb3d8197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.113+0000 7fb3dee2c640 1 -- 172.21.15.104:0/1055721930 >> 172.21.15.104:0/1055721930 conn(0x7fb3d8077a20 msgr2=0x7fb3d8103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.113+0000 7fb3dee2c640 1 -- 172.21.15.104:0/1055721930 shutdown_connections 2024-06-09T22:28:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:40.113+0000 7fb3dee2c640 1 -- 172.21.15.104:0/1055721930 wait complete. 2024-06-09T22:28:40.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 286969 5.13 MB/sec execute 226 sec latency 62.066 ms 2024-06-09T22:28:40.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.787+0000 7f994b6d9700 1 -- 172.21.15.104:0/940502405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9944101d50 msgr2=0x7f994410e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.787+0000 7f994b6d9700 1 --2- 172.21.15.104:0/940502405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9944101d50 0x7f994410e240 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f9934009990 tx=0x7f9934009ca0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.787+0000 7f994b6d9700 1 -- 172.21.15.104:0/940502405 shutdown_connections 2024-06-09T22:28:40.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.787+0000 7f994b6d9700 1 --2- 172.21.15.104:0/940502405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9944101d50 0x7f994410e240 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.787+0000 7f994b6d9700 1 --2- 172.21.15.104:0/940502405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9944107400 0x7f99441077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.787+0000 7f994b6d9700 1 --2- 172.21.15.104:0/940502405 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9944101390 0x7f9944101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.787+0000 7f994b6d9700 1 -- 172.21.15.104:0/940502405 >> 172.21.15.104:0/940502405 conn(0x7f9944078340 msgr2=0x7f9944078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.787+0000 7f994b6d9700 1 -- 172.21.15.104:0/940502405 shutdown_connections 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.787+0000 7f994b6d9700 1 -- 172.21.15.104:0/940502405 wait complete. 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.788+0000 7f994b6d9700 1 Processor -- start 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.788+0000 7f994b6d9700 1 -- start start 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.788+0000 7f994b6d9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9944101390 0x7f9944199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.788+0000 7f994b6d9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9944101d50 0x7f9944199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.788+0000 7f994b6d9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9944107400 0x7f99441a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.788+0000 7f994b6d9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9944111670 con 0x7f9944101390 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.788+0000 7f994b6d9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99441114f0 con 0x7f9944107400 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.788+0000 7f994b6d9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f99441117f0 con 0x7f9944101d50 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.788+0000 7f9949475700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9944101390 0x7f9944199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f9949475700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9944101390 0x7f9944199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33712/0 (socket says 172.21.15.104:33712) 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f9949475700 1 -- 172.21.15.104:0/2643362360 learned_addr learned my addr 172.21.15.104:0/2643362360 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f9948c74700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9944101d50 0x7f9944199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f9949c76700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9944107400 0x7f99441a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f9949475700 1 -- 172.21.15.104:0/2643362360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9944101d50 msgr2=0x7f9944199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f9949475700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9944101d50 0x7f9944199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f9949475700 1 -- 172.21.15.104:0/2643362360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9944107400 msgr2=0x7f99441a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f9949475700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9944107400 0x7f99441a4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f9949475700 1 -- 172.21.15.104:0/2643362360 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99441a4f50 con 0x7f9944101390 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f9949475700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9944101390 0x7f9944199990 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f993c0060b0 tx=0x7f993c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f99427fc700 1 -- 172.21.15.104:0/2643362360 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c01a070 con 0x7f9944101390 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f99427fc700 1 -- 172.21.15.104:0/2643362360 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f993c00dd20 con 0x7f9944101390 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f994b6d9700 1 -- 172.21.15.104:0/2643362360 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99441a51e0 con 0x7f9944101390 2024-06-09T22:28:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f994b6d9700 1 -- 172.21.15.104:0/2643362360 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99441a56f0 con 0x7f9944101390 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.789+0000 7f99427fc700 1 -- 172.21.15.104:0/2643362360 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c0146c0 con 0x7f9944101390 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.791+0000 7f99427fc700 1 -- 172.21.15.104:0/2643362360 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f993c01d070 con 0x7f9944101390 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.792+0000 7f99427fc700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f992c0a2d90 0x7f992c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.792+0000 7f99427fc700 1 -- 172.21.15.104:0/2643362360 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f993c0cef50 con 0x7f9944101390 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.792+0000 7f9948c74700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f992c0a2d90 0x7f992c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.793+0000 7f9948c74700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f992c0a2d90 0x7f992c0a5250 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f99440788d0 tx=0x7f9938009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.793+0000 7f9955f5cb80 1 Processor -- start 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.793+0000 7f9955f5cb80 1 -- start start 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.793+0000 7f9955f5cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f025276c0 0x557f02527aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.793+0000 7f9955f5cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02527fe0 0x557f02521a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.793+0000 7f9955f5cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02521fa0 0x557f025223c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.793+0000 7f9955f5cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557f02473890 con 0x557f02527fe0 2024-06-09T22:28:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.793+0000 7f9955f5cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557f0241b7a0 con 0x557f025276c0 2024-06-09T22:28:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.793+0000 7f9955f5cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557f0246f850 con 0x557f02521fa0 2024-06-09T22:28:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.793+0000 7f9948c74700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02527fe0 0x557f02521a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.793+0000 7f9948c74700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02527fe0 0x557f02521a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33720/0 (socket says 172.21.15.104:33720) 2024-06-09T22:28:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f9948c74700 1 -- 172.21.15.104:0/3824955958 learned_addr learned my addr 172.21.15.104:0/3824955958 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f9949c76700 1 --2- 172.21.15.104:0/3824955958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02521fa0 0x557f025223c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f9949475700 1 --2- 172.21.15.104:0/3824955958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f025276c0 0x557f02527aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f9948c74700 1 -- 172.21.15.104:0/3824955958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02521fa0 msgr2=0x557f025223c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f9948c74700 1 --2- 172.21.15.104:0/3824955958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02521fa0 0x557f025223c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f9948c74700 1 -- 172.21.15.104:0/3824955958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f025276c0 msgr2=0x557f02527aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f9948c74700 1 --2- 172.21.15.104:0/3824955958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f025276c0 0x557f02527aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f9948c74700 1 -- 172.21.15.104:0/3824955958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f02411fd0 con 0x557f02527fe0 2024-06-09T22:28:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f9949475700 1 --2- 172.21.15.104:0/3824955958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f025276c0 0x557f02527aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f9949c76700 1 --2- 172.21.15.104:0/3824955958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02521fa0 0x557f025223c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f9948c74700 1 --2- 172.21.15.104:0/3824955958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02527fe0 0x557f02521a60 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9938011d20 tx=0x7f9938014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286fae07d656d171 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f992b7fe700 1 -- 172.21.15.104:0/3824955958 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993800eb20 con 0x557f02527fe0 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f992b7fe700 1 -- 172.21.15.104:0/3824955958 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f993800ecc0 con 0x557f02527fe0 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.794+0000 7f992b7fe700 1 -- 172.21.15.104:0/3824955958 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993801d770 con 0x557f02527fe0 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.795+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3824955958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02527fe0 msgr2=0x557f02521a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.795+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/3824955958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02527fe0 0x557f02521a60 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9938011d20 tx=0x7f9938014000 comp rx=0 tx=0).stop 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.795+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3824955958 shutdown_connections 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.795+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/3824955958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02521fa0 0x557f025223c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.795+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/3824955958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f025276c0 0x557f02527aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.795+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/3824955958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02527fe0 0x557f02521a60 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.795+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3824955958 >> 172.21.15.104:0/3824955958 conn(0x557f0244cd40 msgr2=0x557f0251c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.795+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3824955958 shutdown_connections 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.795+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3824955958 wait complete. 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.795+0000 7f9955f5cb80 1 -- start start 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.795+0000 7f9955f5cb80 1 Processor -- start 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9955f5cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02521fa0 0x557f0252dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9955f5cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f025276c0 0x557f0252b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9955f5cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f02527fe0 0x557f0252b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9955f5cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557f0252ea50 con 0x557f025276c0 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9955f5cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557f02413c70 con 0x557f02527fe0 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9955f5cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557f0252ebd0 con 0x557f02521fa0 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9955f5cb80 10 client.0 _ll_register_callbacks cb 0x557f0240d9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9948c74700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f025276c0 0x557f0252b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9949475700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02521fa0 0x557f0252dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9948c74700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f025276c0 0x557f0252b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33728/0 (socket says 172.21.15.104:33728) 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9948c74700 1 -- 172.21.15.104:0/4207217066 learned_addr learned my addr 172.21.15.104:0/4207217066 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9949c76700 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f02527fe0 0x557f0252b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9948c74700 1 -- 172.21.15.104:0/4207217066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02521fa0 msgr2=0x557f0252dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9948c74700 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02521fa0 0x557f0252dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9948c74700 1 -- 172.21.15.104:0/4207217066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f02527fe0 msgr2=0x557f0252b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9948c74700 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f02527fe0 0x557f0252b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9948c74700 1 -- 172.21.15.104:0/4207217066 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f023c2950 con 0x557f025276c0 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9949475700 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02521fa0 0x557f0252dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.796+0000 7f9949c76700 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f02527fe0 0x557f0252b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9948c74700 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f025276c0 0x557f0252b0d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f993801d010 tx=0x7f993800ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c80cc0de5a6e768 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9928ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9928ff9700 1 -- 172.21.15.104:0/4207217066 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9938005310 con 0x557f025276c0 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9928ff9700 1 -- 172.21.15.104:0/4207217066 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f99380054b0 con 0x557f025276c0 2024-06-09T22:28:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9955f5cb80 1 -- 172.21.15.104:0/4207217066 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f993800ecc0 con 0x557f025276c0 2024-06-09T22:28:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9928ff9700 1 -- 172.21.15.104:0/4207217066 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f993803b050 con 0x557f025276c0 2024-06-09T22:28:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9955f5cb80 10 client.17598 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9955f5cb80 1 -- 172.21.15.104:0/4207217066 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557f02539320 con 0x557f025276c0 2024-06-09T22:28:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9928ff9700 1 -- 172.21.15.104:0/4207217066 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f993800ecc0 con 0x557f025276c0 2024-06-09T22:28:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9928ff9700 1 -- 172.21.15.104:0/4207217066 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9938036030 con 0x557f025276c0 2024-06-09T22:28:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9955f5cb80 10 client.17598 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9955f5cb80 10 client.17598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9955f5cb80 20 client.17598 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f0253b8a0 0x557f0253dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9955f5cb80 4 client.17598 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9955f5cb80 1 -- 172.21.15.104:0/4207217066 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f02480d90 con 0x557f0253b8a0 2024-06-09T22:28:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.797+0000 7f9949475700 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f0253b8a0 0x557f0253dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.798+0000 7f9949475700 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f0253b8a0 0x557f0253dc90 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf14b11c72a02ca4 server_cookie=ccfa58955bb57e76 in_seq=0 out_seq=0 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.798+0000 7f9928ff9700 10 client.17598 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.798+0000 7f9928ff9700 1 -- 172.21.15.104:0/4207217066 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x557f02480d90 con 0x557f0253b8a0 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.798+0000 7f9928ff9700 10 client.17598 handle_command_reply: tid=0 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9955f5cb80 1 client.17598 shutdown 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9955f5cb80 1 -- 172.21.15.104:0/4207217066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f025276c0 msgr2=0x557f0252b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f025276c0 0x557f0252b0d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f993801d010 tx=0x7f993800ef00 comp rx=0 tx=0).stop 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9955f5cb80 1 -- 172.21.15.104:0/4207217066 shutdown_connections 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f0253b8a0 0x557f0253dc90 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f02527fe0 0x557f0252b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02521fa0 0x557f0252dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/4207217066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f025276c0 0x557f0252b0d0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9928ff9700 0 client.17598 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9955f5cb80 1 -- 172.21.15.104:0/4207217066 >> 172.21.15.104:0/4207217066 conn(0x557f0244cd40 msgr2=0x557f02520a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9955f5cb80 1 -- 172.21.15.104:0/4207217066 shutdown_connections 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9955f5cb80 1 -- 172.21.15.104:0/4207217066 wait complete. 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.799+0000 7f9955f5cb80 20 client.17598 trim_cache size 0 max 16384 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9955f5cb80 1 Processor -- start 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9955f5cb80 1 -- start start 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9955f5cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 0x557f02521a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9955f5cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f025276c0 0x557f02528d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9955f5cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f02527fe0 0x557f02516590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9955f5cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f99441ae6a0 con 0x557f02521fa0 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9955f5cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99441b01e0 con 0x557f02527fe0 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9955f5cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f99441aed70 con 0x557f025276c0 2024-06-09T22:28:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9949475700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 0x557f02521a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9949475700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 0x557f02521a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33744/0 (socket says 172.21.15.104:33744) 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9949475700 1 -- 172.21.15.104:0/2175031322 learned_addr learned my addr 172.21.15.104:0/2175031322 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9948c74700 1 --2- 172.21.15.104:0/2175031322 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f025276c0 0x557f02528d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9949c76700 1 --2- 172.21.15.104:0/2175031322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f02527fe0 0x557f02516590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9949475700 1 -- 172.21.15.104:0/2175031322 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f025276c0 msgr2=0x557f02528d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9949475700 1 --2- 172.21.15.104:0/2175031322 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f025276c0 0x557f02528d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9949475700 1 -- 172.21.15.104:0/2175031322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f02527fe0 msgr2=0x557f02516590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9949475700 1 --2- 172.21.15.104:0/2175031322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f02527fe0 0x557f02516590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9949475700 1 -- 172.21.15.104:0/2175031322 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f0252ebc0 con 0x557f02521fa0 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.820+0000 7f9949c76700 1 --2- 172.21.15.104:0/2175031322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f02527fe0 0x557f02516590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.821+0000 7f9949475700 1 --2- 172.21.15.104:0/2175031322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 0x557f02521a60 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f993c09ada0 tx=0x7f993c015de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52186cfe3b0e3a5 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.821+0000 7f992affd700 1 -- 172.21.15.104:0/2175031322 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c09f450 con 0x557f02521fa0 2024-06-09T22:28:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.821+0000 7f992affd700 1 -- 172.21.15.104:0/2175031322 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f993c0a3d10 con 0x557f02521fa0 2024-06-09T22:28:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.821+0000 7f992affd700 1 -- 172.21.15.104:0/2175031322 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c09b300 con 0x557f02521fa0 2024-06-09T22:28:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.821+0000 7f9955f5cb80 1 -- 172.21.15.104:0/2175031322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 msgr2=0x557f02521a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.821+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/2175031322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 0x557f02521a60 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f993c09ada0 tx=0x7f993c015de0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.821+0000 7f9955f5cb80 1 -- 172.21.15.104:0/2175031322 shutdown_connections 2024-06-09T22:28:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.821+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/2175031322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 0x557f02521a60 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.821+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/2175031322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f02527fe0 0x557f02516590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.821+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/2175031322 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f025276c0 0x557f02528d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.821+0000 7f9955f5cb80 1 -- 172.21.15.104:0/2175031322 >> 172.21.15.104:0/2175031322 conn(0x557f0244cd40 msgr2=0x557f02515e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9955f5cb80 1 -- 172.21.15.104:0/2175031322 shutdown_connections 2024-06-09T22:28:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9955f5cb80 1 -- 172.21.15.104:0/2175031322 wait complete. 2024-06-09T22:28:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9955f5cb80 1 -- start start 2024-06-09T22:28:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9955f5cb80 1 Processor -- start 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9955f5cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 0x557f02533ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9955f5cb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f025276c0 0x557f02534400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9955f5cb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02527fe0 0x557f02530e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9955f5cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f99441afb10 con 0x557f02521fa0 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9955f5cb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99441af440 con 0x557f025276c0 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9955f5cb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f99441ac660 con 0x557f02527fe0 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9955f5cb80 10 client.0 _ll_register_callbacks cb 0x557f0240d9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9949475700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 0x557f02533ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9949475700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 0x557f02533ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33756/0 (socket says 172.21.15.104:33756) 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9949475700 1 -- 172.21.15.104:0/3290890966 learned_addr learned my addr 172.21.15.104:0/3290890966 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.822+0000 7f9949c76700 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02527fe0 0x557f02530e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9948c74700 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f025276c0 0x557f02534400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9949475700 1 -- 172.21.15.104:0/3290890966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02527fe0 msgr2=0x557f02530e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9949475700 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02527fe0 0x557f02530e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9949475700 1 -- 172.21.15.104:0/3290890966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f025276c0 msgr2=0x557f02534400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9949475700 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f025276c0 0x557f02534400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9949475700 1 -- 172.21.15.104:0/3290890966 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f02534a80 con 0x557f02521fa0 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9949475700 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 0x557f02533ec0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f993c0a37e0 tx=0x7f993c09b1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62c5314d75791b8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9928ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9928ff9700 1 -- 172.21.15.104:0/3290890966 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c0159e0 con 0x557f02521fa0 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9928ff9700 1 -- 172.21.15.104:0/3290890966 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f993c09f0f0 con 0x557f02521fa0 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3290890966 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f993c0a3d10 con 0x557f02521fa0 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9928ff9700 1 -- 172.21.15.104:0/3290890966 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f993c09b300 con 0x557f02521fa0 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9955f5cb80 10 client.17604 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3290890966 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557f025a6840 con 0x557f02521fa0 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9928ff9700 1 -- 172.21.15.104:0/3290890966 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f993c0a3d10 con 0x557f02521fa0 2024-06-09T22:28:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9928ff9700 1 -- 172.21.15.104:0/3290890966 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f993c0b6030 con 0x557f02521fa0 2024-06-09T22:28:40.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9955f5cb80 10 client.17604 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:40.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9955f5cb80 10 client.17604 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:40.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.823+0000 7f9955f5cb80 20 client.17604 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:40.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.824+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f025a6e50 0x557f025a9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:40.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.824+0000 7f9955f5cb80 4 client.17604 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-09T22:28:40.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.824+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3290890966 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557f0247b310 con 0x557f025a6e50 2024-06-09T22:28:40.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.824+0000 7f9948c74700 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f025a6e50 0x557f025a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:40.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.824+0000 7f9948c74700 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f025a6e50 0x557f025a9270 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b74ca201332213f server_cookie=15104e216eb21171 in_seq=0 out_seq=0 2024-06-09T22:28:40.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.824+0000 7f9928ff9700 10 client.17604 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:40.905 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:28:40.905 INFO:teuthology.orchestra.run.smithi104.stdout: "return_code": 0, 2024-06-09T22:28:40.906 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_tag": "9f5ec0f2-1077-4839-80dc-903eafb4a32c", 2024-06-09T22:28:40.906 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": "asynchronous" 2024-06-09T22:28:40.906 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:28:40.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.825+0000 7f9928ff9700 1 -- 172.21.15.104:0/3290890966 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557f0247b310 con 0x557f025a6e50 2024-06-09T22:28:40.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.825+0000 7f9928ff9700 10 client.17604 handle_command_reply: tid=0 2024-06-09T22:28:40.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.825+0000 7f9955f5cb80 1 client.17604 shutdown 2024-06-09T22:28:40.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.825+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3290890966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 msgr2=0x557f02533ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.825+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 0x557f02533ec0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f993c0a37e0 tx=0x7f993c09b1f0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3290890966 shutdown_connections 2024-06-09T22:28:40.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x557f025a6e50 0x557f025a9270 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557f02521fa0 0x557f02533ec0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f02527fe0 0x557f02530e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f9955f5cb80 1 --2- 172.21.15.104:0/3290890966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f025276c0 0x557f02534400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3290890966 >> 172.21.15.104:0/3290890966 conn(0x557f0244cd40 msgr2=0x557f02528b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f9928ff9700 0 client.17604 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3290890966 shutdown_connections 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f9955f5cb80 1 -- 172.21.15.104:0/3290890966 wait complete. 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f9955f5cb80 20 client.17604 trim_cache size 0 max 16384 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f994b6d9700 1 -- 172.21.15.104:0/2643362360 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f992c0a2d90 msgr2=0x7f992c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f994b6d9700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f992c0a2d90 0x7f992c0a5250 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f99440788d0 tx=0x7f9938009260 comp rx=0 tx=0).stop 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f994b6d9700 1 -- 172.21.15.104:0/2643362360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9944101390 msgr2=0x7f9944199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f994b6d9700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9944101390 0x7f9944199990 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f993c0060b0 tx=0x7f993c00d3b0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f994b6d9700 1 -- 172.21.15.104:0/2643362360 shutdown_connections 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f994b6d9700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f992c0a2d90 0x7f992c0a5250 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f994b6d9700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9944101390 0x7f9944199990 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.826+0000 7f994b6d9700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9944107400 0x7f99441a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.827+0000 7f994b6d9700 1 --2- 172.21.15.104:0/2643362360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9944101d50 0x7f9944199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:40.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.827+0000 7f994b6d9700 1 -- 172.21.15.104:0/2643362360 >> 172.21.15.104:0/2643362360 conn(0x7f9944078340 msgr2=0x7f99440ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:40.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.827+0000 7f994b6d9700 1 -- 172.21.15.104:0/2643362360 shutdown_connections 2024-06-09T22:28:40.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:40.827+0000 7f994b6d9700 1 -- 172.21.15.104:0/2643362360 wait complete. 2024-06-09T22:28:41.630 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:28:41.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 288113 5.14 MB/sec execute 227 sec latency 20.511 ms 2024-06-09T22:28:41.862 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:41 smithi194 ceph-mon[36856]: pgmap v236: 129 pgs: 129 active+clean; 222 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.1 MiB/s wr, 269 op/s 2024-06-09T22:28:41.863 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:41 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:28:41.863 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:41 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:28:41.863 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:41 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:41.863 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:41 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:28:41.863 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:41 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:41.863 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:41 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:28:41.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:41 smithi153 ceph-mon[36599]: pgmap v236: 129 pgs: 129 active+clean; 222 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.1 MiB/s wr, 269 op/s 2024-06-09T22:28:41.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:41 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:28:41.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:41 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:28:41.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:41 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:41.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:41 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:28:41.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:41 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:41.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:41 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:42.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:41 smithi104 ceph-mon[32389]: pgmap v236: 129 pgs: 129 active+clean; 222 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.1 MiB/s wr, 269 op/s 2024-06-09T22:28:42.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:41 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:28:42.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:41 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:28:42.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:41 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:42.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:41 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:28:42.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:41 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:42.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:41 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:42.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 290247 5.17 MB/sec execute 228 sec latency 21.724 ms 2024-06-09T22:28:42.758 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:42 smithi104 ceph-mon[32389]: scrub queued for path: / 2024-06-09T22:28:42.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:42 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:42.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:42 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:28:42.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:42 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:42.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:42 smithi153 ceph-mon[36599]: scrub queued for path: / 2024-06-09T22:28:42.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:42 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:42.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:42 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:28:42.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:42 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:43.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:42 smithi194 ceph-mon[36856]: scrub queued for path: / 2024-06-09T22:28:43.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:42 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:43.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:42 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:28:43.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:42 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:28:43.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.682+0000 7f580808a700 1 -- 172.21.15.104:0/4072744123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5800101440 msgr2=0x7f580010f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:43.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.682+0000 7f580808a700 1 --2- 172.21.15.104:0/4072744123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5800101440 0x7f580010f330 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f57f800b0a0 tx=0x7f57f800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.682+0000 7f580808a700 1 -- 172.21.15.104:0/4072744123 shutdown_connections 2024-06-09T22:28:43.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.682+0000 7f580808a700 1 --2- 172.21.15.104:0/4072744123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5800101440 0x7f580010f330 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.682+0000 7f580808a700 1 --2- 172.21.15.104:0/4072744123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5800108510 0x7f58001088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.682+0000 7f580808a700 1 --2- 172.21.15.104:0/4072744123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5800100a80 0x7f5800100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.682+0000 7f580808a700 1 -- 172.21.15.104:0/4072744123 >> 172.21.15.104:0/4072744123 conn(0x7f5800077a00 msgr2=0x7f58000fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.682+0000 7f580808a700 1 -- 172.21.15.104:0/4072744123 shutdown_connections 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.682+0000 7f580808a700 1 -- 172.21.15.104:0/4072744123 wait complete. 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f580808a700 1 Processor -- start 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f580808a700 1 -- start start 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f580808a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5800100a80 0x7f580019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f580808a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5800101440 0x7f580019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f580808a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5800108510 0x7f5800198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f580808a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f58001127f0 con 0x7f5800101440 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f580808a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5800112670 con 0x7f5800108510 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f580808a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5800112970 con 0x7f5800100a80 2024-06-09T22:28:43.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f5805625700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5800101440 0x7f580019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f5805625700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5800101440 0x7f580019eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33772/0 (socket says 172.21.15.104:33772) 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f5805625700 1 -- 172.21.15.104:0/1597107440 learned_addr learned my addr 172.21.15.104:0/1597107440 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f5805e26700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5800100a80 0x7f580019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.683+0000 7f5806627700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5800108510 0x7f5800198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.684+0000 7f5805625700 1 -- 172.21.15.104:0/1597107440 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5800100a80 msgr2=0x7f580019e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.684+0000 7f5805625700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5800100a80 0x7f580019e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.684+0000 7f5805625700 1 -- 172.21.15.104:0/1597107440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5800108510 msgr2=0x7f5800198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.684+0000 7f5805625700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5800108510 0x7f5800198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.684+0000 7f5805625700 1 -- 172.21.15.104:0/1597107440 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58001991d0 con 0x7f5800101440 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.684+0000 7f5805625700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5800101440 0x7f580019eda0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00e9a0 tx=0x7f57fc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.684+0000 7f57f2ffd700 1 -- 172.21.15.104:0/1597107440 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc00ca10 con 0x7f5800101440 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.684+0000 7f580808a700 1 -- 172.21.15.104:0/1597107440 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58001994c0 con 0x7f5800101440 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.684+0000 7f57f2ffd700 1 -- 172.21.15.104:0/1597107440 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57fc00cbb0 con 0x7f5800101440 2024-06-09T22:28:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.684+0000 7f57f2ffd700 1 -- 172.21.15.104:0/1597107440 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc018680 con 0x7f5800101440 2024-06-09T22:28:43.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.684+0000 7f580808a700 1 -- 172.21.15.104:0/1597107440 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58001a5730 con 0x7f5800101440 2024-06-09T22:28:43.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.687+0000 7f57f2ffd700 1 -- 172.21.15.104:0/1597107440 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f57fc010430 con 0x7f5800101440 2024-06-09T22:28:43.689 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:43 smithi104 ceph-mon[32389]: pgmap v237: 129 pgs: 129 active+clean; 222 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 177 op/s 2024-06-09T22:28:43.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.687+0000 7f57f2ffd700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f57ec0a2d90 0x7f57ec0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.688+0000 7f5805e26700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f57ec0a2d90 0x7f57ec0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.688+0000 7f57f2ffd700 1 -- 172.21.15.104:0/1597107440 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f57fc0cfa60 con 0x7f5800101440 2024-06-09T22:28:43.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.688+0000 7f5805e26700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f57ec0a2d90 0x7f57ec0a5250 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f58001075a0 tx=0x7f57f4009190 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:43.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f581290db80 1 Processor -- start 2024-06-09T22:28:43.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f581290db80 1 -- start start 2024-06-09T22:28:43.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f581290db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c94b4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f581290db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94ae670 0x55a3c94aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f581290db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94af030 0x55a3c94bb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f581290db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a3c939cf60 con 0x55a3c94b4470 2024-06-09T22:28:43.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f581290db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a3c93fb850 con 0x55a3c94ae670 2024-06-09T22:28:43.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f581290db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a3c9363460 con 0x55a3c94af030 2024-06-09T22:28:43.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f5805e26700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c94b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f5806627700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94af030 0x55a3c94bb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f5805e26700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c94b4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33774/0 (socket says 172.21.15.104:33774) 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f5805e26700 1 -- 172.21.15.104:0/2304579129 learned_addr learned my addr 172.21.15.104:0/2304579129 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.689+0000 7f5805625700 1 --2- 172.21.15.104:0/2304579129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94ae670 0x55a3c94aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.690+0000 7f5805e26700 1 -- 172.21.15.104:0/2304579129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94af030 msgr2=0x55a3c94bb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.690+0000 7f5805e26700 1 --2- 172.21.15.104:0/2304579129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94af030 0x55a3c94bb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.690+0000 7f5805e26700 1 -- 172.21.15.104:0/2304579129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94ae670 msgr2=0x55a3c94aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.690+0000 7f5805e26700 1 --2- 172.21.15.104:0/2304579129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94ae670 0x55a3c94aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.690+0000 7f5805e26700 1 -- 172.21.15.104:0/2304579129 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3c934a8d0 con 0x55a3c94b4470 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.690+0000 7f5805e26700 1 --2- 172.21.15.104:0/2304579129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c94b4850 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f57f4009fd0 tx=0x7f57f4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de61c9c0d2858963 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.690+0000 7f57ebfff700 1 -- 172.21.15.104:0/2304579129 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f401b040 con 0x55a3c94b4470 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.690+0000 7f57ebfff700 1 -- 172.21.15.104:0/2304579129 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57f4024070 con 0x55a3c94b4470 2024-06-09T22:28:43.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.690+0000 7f57ebfff700 1 -- 172.21.15.104:0/2304579129 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f4013740 con 0x55a3c94b4470 2024-06-09T22:28:43.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.691+0000 7f581290db80 1 -- 172.21.15.104:0/2304579129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 msgr2=0x55a3c94b4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:43.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.691+0000 7f581290db80 1 --2- 172.21.15.104:0/2304579129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c94b4850 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f57f4009fd0 tx=0x7f57f4011000 comp rx=0 tx=0).stop 2024-06-09T22:28:43.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.691+0000 7f581290db80 1 -- 172.21.15.104:0/2304579129 shutdown_connections 2024-06-09T22:28:43.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.691+0000 7f581290db80 1 --2- 172.21.15.104:0/2304579129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c94b4850 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.691+0000 7f581290db80 1 --2- 172.21.15.104:0/2304579129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94af030 0x55a3c94bb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.691+0000 7f581290db80 1 --2- 172.21.15.104:0/2304579129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94ae670 0x55a3c94aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.691+0000 7f581290db80 1 -- 172.21.15.104:0/2304579129 >> 172.21.15.104:0/2304579129 conn(0x55a3c93d7080 msgr2=0x55a3c94a5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.691+0000 7f581290db80 1 -- 172.21.15.104:0/2304579129 shutdown_connections 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.691+0000 7f581290db80 1 -- 172.21.15.104:0/2304579129 wait complete. 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f581290db80 1 -- start start 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f581290db80 1 Processor -- start 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f581290db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 0x55a3c94bae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f581290db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94af030 0x55a3c94b8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f581290db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c94b8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f581290db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55a3c939bfc0 con 0x55a3c94b4470 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f581290db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a3c93f7810 con 0x55a3c94af030 2024-06-09T22:28:43.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f5805e26700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 0x55a3c94bae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f581290db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a3c94bb750 con 0x55a3c94ae670 2024-06-09T22:28:43.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f5805e26700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 0x55a3c94bae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:40376/0 (socket says 172.21.15.104:40376) 2024-06-09T22:28:43.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f5806627700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c94b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f5805e26700 1 -- 172.21.15.104:0/3776053882 learned_addr learned my addr 172.21.15.104:0/3776053882 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:43.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.692+0000 7f581290db80 10 client.0 _ll_register_callbacks cb 0x55a3c93915b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:43.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f5805625700 1 --2- 172.21.15.104:0/3776053882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94af030 0x55a3c94b8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f5805e26700 1 -- 172.21.15.104:0/3776053882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94af030 msgr2=0x55a3c94b8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:43.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f5805e26700 1 --2- 172.21.15.104:0/3776053882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94af030 0x55a3c94b8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f5805e26700 1 -- 172.21.15.104:0/3776053882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 msgr2=0x55a3c94b8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:43.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f5805e26700 1 --2- 172.21.15.104:0/3776053882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c94b8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f5805e26700 1 -- 172.21.15.104:0/3776053882 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3c93c1750 con 0x55a3c94ae670 2024-06-09T22:28:43.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f5805e26700 1 --2- 172.21.15.104:0/3776053882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 0x55a3c94bae40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f57f4017010 tx=0x7f57f40159a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=158d8b82a94e72fb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f57e97fa700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f57e97fa700 1 -- 172.21.15.104:0/3776053882 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f402f070 con 0x55a3c94ae670 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f581290db80 1 -- 172.21.15.104:0/3776053882 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57f4024070 con 0x55a3c94ae670 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f57e97fa700 1 -- 172.21.15.104:0/3776053882 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f57f4028830 con 0x55a3c94ae670 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f57e97fa700 1 -- 172.21.15.104:0/3776053882 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f57f403b050 con 0x55a3c94ae670 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f581290db80 10 client.26944 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.693+0000 7f581290db80 1 -- 172.21.15.104:0/3776053882 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a3c94c5d20 con 0x55a3c94ae670 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.694+0000 7f57e97fa700 1 -- 172.21.15.104:0/3776053882 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57f4024070 con 0x55a3c94ae670 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.694+0000 7f57e97fa700 1 -- 172.21.15.104:0/3776053882 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f57f401e030 con 0x55a3c94ae670 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.694+0000 7f581290db80 10 client.26944 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.694+0000 7f581290db80 10 client.26944 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.694+0000 7f581290db80 20 client.26944 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.694+0000 7f581290db80 1 --2- 172.21.15.104:0/3776053882 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3c94c8250 0x55a3c94ca640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.694+0000 7f581290db80 4 client.26944 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:43.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.694+0000 7f581290db80 1 -- 172.21.15.104:0/3776053882 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3c9408d50 con 0x55a3c94c8250 2024-06-09T22:28:43.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.694+0000 7f5805625700 1 --2- 172.21.15.104:0/3776053882 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3c94c8250 0x55a3c94ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.694+0000 7f57e97fa700 10 client.26944 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:43.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.694+0000 7f5805625700 1 --2- 172.21.15.104:0/3776053882 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3c94c8250 0x55a3c94ca640 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63744055bc880779 server_cookie=e4c076771c44e02d in_seq=0 out_seq=0 2024-06-09T22:28:43.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f57e97fa700 1 -- 172.21.15.104:0/3776053882 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a3c9408d50 con 0x55a3c94c8250 2024-06-09T22:28:43.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f57e97fa700 10 client.26944 handle_command_reply: tid=0 2024-06-09T22:28:43.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f581290db80 1 client.26944 shutdown 2024-06-09T22:28:43.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f581290db80 1 -- 172.21.15.104:0/3776053882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 msgr2=0x55a3c94bae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:43.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f581290db80 1 --2- 172.21.15.104:0/3776053882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 0x55a3c94bae40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f57f4017010 tx=0x7f57f40159a0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f581290db80 1 -- 172.21.15.104:0/3776053882 shutdown_connections 2024-06-09T22:28:43.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f581290db80 1 --2- 172.21.15.104:0/3776053882 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3c94c8250 0x55a3c94ca640 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f581290db80 1 --2- 172.21.15.104:0/3776053882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c94b8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f581290db80 1 --2- 172.21.15.104:0/3776053882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 0x55a3c94bae40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f57e97fa700 0 client.26944 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:43.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f581290db80 1 --2- 172.21.15.104:0/3776053882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94af030 0x55a3c94b8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f581290db80 1 -- 172.21.15.104:0/3776053882 >> 172.21.15.104:0/3776053882 conn(0x55a3c93d7080 msgr2=0x55a3c94a4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:43.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f581290db80 1 -- 172.21.15.104:0/3776053882 shutdown_connections 2024-06-09T22:28:43.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.695+0000 7f581290db80 1 -- 172.21.15.104:0/3776053882 wait complete. 2024-06-09T22:28:43.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.696+0000 7f581290db80 20 client.26944 trim_cache size 0 max 16384 2024-06-09T22:28:43.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f581290db80 1 Processor -- start 2024-06-09T22:28:43.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f581290db80 1 -- start start 2024-06-09T22:28:43.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f581290db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94ae670 0x55a3c94b3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f581290db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94af030 0x55a3c94a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f581290db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94b4470 0x55a3c94a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f581290db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f58001ac3c0 con 0x55a3c94ae670 2024-06-09T22:28:43.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f581290db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58001adf00 con 0x55a3c94b4470 2024-06-09T22:28:43.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f581290db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f58001aca90 con 0x55a3c94af030 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f5805e26700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94ae670 0x55a3c94b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f5805e26700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94ae670 0x55a3c94b3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33800/0 (socket says 172.21.15.104:33800) 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f5805e26700 1 -- 172.21.15.104:0/3437657567 learned_addr learned my addr 172.21.15.104:0/3437657567 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f5806627700 1 --2- 172.21.15.104:0/3437657567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94b4470 0x55a3c94a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f5805625700 1 --2- 172.21.15.104:0/3437657567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94af030 0x55a3c94a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f5806627700 1 -- 172.21.15.104:0/3437657567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94af030 msgr2=0x55a3c94a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f5806627700 1 --2- 172.21.15.104:0/3437657567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94af030 0x55a3c94a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f5806627700 1 -- 172.21.15.104:0/3437657567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94ae670 msgr2=0x55a3c94b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f5806627700 1 --2- 172.21.15.104:0/3437657567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94ae670 0x55a3c94b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f5806627700 1 -- 172.21.15.104:0/3437657567 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3c94a98c0 con 0x55a3c94b4470 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.717+0000 7f5805e26700 1 --2- 172.21.15.104:0/3437657567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94ae670 0x55a3c94b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.718+0000 7f5806627700 1 --2- 172.21.15.104:0/3437657567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94b4470 0x55a3c94a53c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f57f8009fd0 tx=0x7f57f800bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d71ce691f85ac958 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:43.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.718+0000 7f57eb7fe700 1 -- 172.21.15.104:0/3437657567 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f8003a00 con 0x55a3c94b4470 2024-06-09T22:28:43.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.718+0000 7f57eb7fe700 1 -- 172.21.15.104:0/3437657567 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57f8003ba0 con 0x55a3c94b4470 2024-06-09T22:28:43.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.718+0000 7f57eb7fe700 1 -- 172.21.15.104:0/3437657567 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f80276e0 con 0x55a3c94b4470 2024-06-09T22:28:43.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.718+0000 7f581290db80 1 -- 172.21.15.104:0/3437657567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94b4470 msgr2=0x55a3c94a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:43.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.718+0000 7f581290db80 1 --2- 172.21.15.104:0/3437657567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94b4470 0x55a3c94a53c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f57f8009fd0 tx=0x7f57f800bab0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.718+0000 7f581290db80 1 -- 172.21.15.104:0/3437657567 shutdown_connections 2024-06-09T22:28:43.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.718+0000 7f581290db80 1 --2- 172.21.15.104:0/3437657567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94ae670 0x55a3c94b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.718+0000 7f581290db80 1 --2- 172.21.15.104:0/3437657567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94b4470 0x55a3c94a53c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.718+0000 7f581290db80 1 --2- 172.21.15.104:0/3437657567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94af030 0x55a3c94a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.718+0000 7f581290db80 1 -- 172.21.15.104:0/3437657567 >> 172.21.15.104:0/3437657567 conn(0x55a3c93d7080 msgr2=0x55a3c94b3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.719+0000 7f581290db80 1 -- 172.21.15.104:0/3437657567 shutdown_connections 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.719+0000 7f581290db80 1 -- 172.21.15.104:0/3437657567 wait complete. 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.719+0000 7f581290db80 1 -- start start 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.719+0000 7f581290db80 1 Processor -- start 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.719+0000 7f581290db80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 0x55a3c94bcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.719+0000 7f581290db80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94af030 0x55a3c94bd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.719+0000 7f581290db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c9531c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.719+0000 7f581290db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f58001ad830 con 0x55a3c94b4470 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.719+0000 7f581290db80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58001ad160 con 0x55a3c94af030 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.719+0000 7f581290db80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f58001aa380 con 0x55a3c94ae670 2024-06-09T22:28:43.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.719+0000 7f581290db80 10 client.0 _ll_register_callbacks cb 0x55a3c93915b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:43.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.719+0000 7f5806627700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c9531c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f5806627700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c9531c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33804/0 (socket says 172.21.15.104:33804) 2024-06-09T22:28:43.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f5806627700 1 -- 172.21.15.104:0/2619912373 learned_addr learned my addr 172.21.15.104:0/2619912373 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:43.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f5805e26700 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 0x55a3c94bcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f5805625700 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94af030 0x55a3c94bd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f5806627700 1 -- 172.21.15.104:0/2619912373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 msgr2=0x55a3c94bcd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:43.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f5806627700 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 0x55a3c94bcd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f5806627700 1 -- 172.21.15.104:0/2619912373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94af030 msgr2=0x55a3c94bd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:43.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f5806627700 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94af030 0x55a3c94bd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f5806627700 1 -- 172.21.15.104:0/2619912373 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3c95322b0 con 0x55a3c94b4470 2024-06-09T22:28:43.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f5805e26700 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 0x55a3c94bcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:28:43.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f5806627700 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c9531c70 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f57f8009a20 tx=0x7f57f8003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4594936f2465b90e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f57e97fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f57e97fa700 1 -- 172.21.15.104:0/2619912373 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f8007e30 con 0x55a3c94b4470 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f57e97fa700 1 -- 172.21.15.104:0/2619912373 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f57f80078b0 con 0x55a3c94b4470 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f581290db80 1 -- 172.21.15.104:0/2619912373 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57f8003ba0 con 0x55a3c94b4470 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f57e97fa700 1 -- 172.21.15.104:0/2619912373 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f57f8038050 con 0x55a3c94b4470 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f581290db80 10 client.17622 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.720+0000 7f581290db80 1 -- 172.21.15.104:0/2619912373 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a3c95335a0 con 0x55a3c94b4470 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.721+0000 7f57e97fa700 1 -- 172.21.15.104:0/2619912373 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57f8003ba0 con 0x55a3c94b4470 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.721+0000 7f57e97fa700 1 -- 172.21.15.104:0/2619912373 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f57f8011070 con 0x55a3c94b4470 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.721+0000 7f581290db80 10 client.17622 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.721+0000 7f581290db80 10 client.17622 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.721+0000 7f581290db80 20 client.17622 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.721+0000 7f581290db80 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3c9533b40 0x55a3c9535f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:43.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.721+0000 7f581290db80 4 client.17622 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.721+0000 7f581290db80 1 -- 172.21.15.104:0/2619912373 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a3c94032d0 con 0x55a3c9533b40 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.721+0000 7f5805e26700 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3c9533b40 0x55a3c9535f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.721+0000 7f5805e26700 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3c9533b40 0x55a3c9535f60 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd2caa4e8a133c05 server_cookie=86982e72c7c45a33 in_seq=0 out_seq=0 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.721+0000 7f57e97fa700 10 client.17622 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f57e97fa700 1 -- 172.21.15.104:0/2619912373 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a3c94032d0 con 0x55a3c9533b40 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f57e97fa700 10 client.17622 handle_command_reply: tid=0 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f581290db80 1 client.17622 shutdown 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f581290db80 1 -- 172.21.15.104:0/2619912373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 msgr2=0x55a3c9531c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f581290db80 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c9531c70 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f57f8009a20 tx=0x7f57f8003b20 comp rx=0 tx=0).stop 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f581290db80 1 -- 172.21.15.104:0/2619912373 shutdown_connections 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f581290db80 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55a3c9533b40 0x55a3c9535f60 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f581290db80 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a3c94b4470 0x55a3c9531c70 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f581290db80 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a3c94ae670 0x55a3c94bcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f581290db80 1 --2- 172.21.15.104:0/2619912373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a3c94af030 0x55a3c94bd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f57e97fa700 0 client.17622 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:43.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f581290db80 1 -- 172.21.15.104:0/2619912373 >> 172.21.15.104:0/2619912373 conn(0x55a3c93d7080 msgr2=0x55a3c94b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:43.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f581290db80 1 -- 172.21.15.104:0/2619912373 shutdown_connections 2024-06-09T22:28:43.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f581290db80 1 -- 172.21.15.104:0/2619912373 wait complete. 2024-06-09T22:28:43.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f581290db80 20 client.17622 trim_cache size 0 max 16384 2024-06-09T22:28:43.731 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:28:43.731 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-09T22:28:43.731 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": { 2024-06-09T22:28:43.731 INFO:teuthology.orchestra.run.smithi104.stdout: "9f5ec0f2-1077-4839-80dc-903eafb4a32c": { 2024-06-09T22:28:43.731 INFO:teuthology.orchestra.run.smithi104.stdout: "path": "/", 2024-06-09T22:28:43.731 INFO:teuthology.orchestra.run.smithi104.stdout: "tag": "9f5ec0f2-1077-4839-80dc-903eafb4a32c", 2024-06-09T22:28:43.731 INFO:teuthology.orchestra.run.smithi104.stdout: "options": "recursive,force" 2024-06-09T22:28:43.731 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:28:43.731 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:28:43.731 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:28:43.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f580808a700 1 -- 172.21.15.104:0/1597107440 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f57ec0a2d90 msgr2=0x7f57ec0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:43.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f580808a700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f57ec0a2d90 0x7f57ec0a5250 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f58001075a0 tx=0x7f57f4009190 comp rx=0 tx=0).stop 2024-06-09T22:28:43.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f580808a700 1 -- 172.21.15.104:0/1597107440 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5800101440 msgr2=0x7f580019eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:43.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.722+0000 7f580808a700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5800101440 0x7f580019eda0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00e9a0 tx=0x7f57fc00ecb0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.723+0000 7f580808a700 1 -- 172.21.15.104:0/1597107440 shutdown_connections 2024-06-09T22:28:43.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.723+0000 7f580808a700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f57ec0a2d90 0x7f57ec0a5250 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.723+0000 7f580808a700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5800108510 0x7f5800198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.723+0000 7f580808a700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5800100a80 0x7f580019e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.723+0000 7f580808a700 1 --2- 172.21.15.104:0/1597107440 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5800101440 0x7f580019eda0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:43.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.723+0000 7f580808a700 1 -- 172.21.15.104:0/1597107440 >> 172.21.15.104:0/1597107440 conn(0x7f5800077a00 msgr2=0x7f58000ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:43.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.723+0000 7f580808a700 1 -- 172.21.15.104:0/1597107440 shutdown_connections 2024-06-09T22:28:43.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:28:43.723+0000 7f580808a700 1 -- 172.21.15.104:0/1597107440 wait complete. 2024-06-09T22:28:43.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 291542 5.18 MB/sec execute 229 sec latency 54.506 ms 2024-06-09T22:28:43.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:43 smithi153 ceph-mon[36599]: pgmap v237: 129 pgs: 129 active+clean; 222 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 177 op/s 2024-06-09T22:28:44.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:43 smithi194 ceph-mon[36856]: pgmap v237: 129 pgs: 129 active+clean; 222 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 177 op/s 2024-06-09T22:28:44.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 293264 5.18 MB/sec execute 230 sec latency 64.806 ms 2024-06-09T22:28:44.742 INFO:tasks.cephfs.filesystem:scrub status for tag:9f5ec0f2-1077-4839-80dc-903eafb4a32c - {'path': '/', 'tag': '9f5ec0f2-1077-4839-80dc-903eafb4a32c', 'options': 'recursive,force'} 2024-06-09T22:28:45.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 294884 5.21 MB/sec execute 231 sec latency 33.007 ms 2024-06-09T22:28:45.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:45 smithi153 ceph-mon[36599]: pgmap v238: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.3 MiB/s wr, 279 op/s 2024-06-09T22:28:46.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:45 smithi104 ceph-mon[32389]: pgmap v238: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.3 MiB/s wr, 279 op/s 2024-06-09T22:28:46.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:45 smithi194 ceph-mon[36856]: pgmap v238: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.3 MiB/s wr, 279 op/s 2024-06-09T22:28:46.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 296760 5.22 MB/sec execute 232 sec latency 17.894 ms 2024-06-09T22:28:47.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 298431 5.25 MB/sec execute 233 sec latency 10.232 ms 2024-06-09T22:28:47.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:47 smithi153 ceph-mon[36599]: pgmap v239: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 188 op/s 2024-06-09T22:28:48.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:47 smithi104 ceph-mon[32389]: pgmap v239: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 188 op/s 2024-06-09T22:28:48.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:47 smithi194 ceph-mon[36856]: pgmap v239: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 188 op/s 2024-06-09T22:28:48.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 300174 5.25 MB/sec execute 234 sec latency 41.206 ms 2024-06-09T22:28:49.738 INFO:tasks.workunit.client.0.smithi104.stdout: 1 301891 5.29 MB/sec execute 235 sec latency 66.664 ms 2024-06-09T22:28:49.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:49 smithi153 ceph-mon[36599]: pgmap v240: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 188 op/s 2024-06-09T22:28:50.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:49 smithi104 ceph-mon[32389]: pgmap v240: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 188 op/s 2024-06-09T22:28:50.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:49 smithi194 ceph-mon[36856]: pgmap v240: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 188 op/s 2024-06-09T22:28:50.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 303226 5.29 MB/sec execute 236 sec latency 23.885 ms 2024-06-09T22:28:51.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 305368 5.32 MB/sec execute 237 sec latency 17.240 ms 2024-06-09T22:28:51.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:51 smithi153 ceph-mon[36599]: pgmap v241: 129 pgs: 129 active+clean; 235 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 287 op/s 2024-06-09T22:28:52.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:51 smithi104 ceph-mon[32389]: pgmap v241: 129 pgs: 129 active+clean; 235 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 287 op/s 2024-06-09T22:28:52.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:51 smithi194 ceph-mon[36856]: pgmap v241: 129 pgs: 129 active+clean; 235 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 287 op/s 2024-06-09T22:28:52.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 306757 5.32 MB/sec execute 238 sec latency 19.473 ms 2024-06-09T22:28:53.209 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.205+0000 7f32832da640 1 -- 172.21.15.104:0/476963771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f327c1008d0 msgr2=0x7f327c100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:53.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.205+0000 7f32832da640 1 --2- 172.21.15.104:0/476963771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f327c1008d0 0x7f327c100d50 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f326c009930 tx=0x7f326c02a420 comp rx=0 tx=0).stop 2024-06-09T22:28:53.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.205+0000 7f32832da640 1 -- 172.21.15.104:0/476963771 shutdown_connections 2024-06-09T22:28:53.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.205+0000 7f32832da640 1 --2- 172.21.15.104:0/476963771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f327c101290 0x7f327c10d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.205+0000 7f32832da640 1 --2- 172.21.15.104:0/476963771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f327c1008d0 0x7f327c100d50 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.205+0000 7f32832da640 1 --2- 172.21.15.104:0/476963771 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f327c106890 0x7f327c106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.205+0000 7f32832da640 1 -- 172.21.15.104:0/476963771 >> 172.21.15.104:0/476963771 conn(0x7f327c077a20 msgr2=0x7f327c077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:53.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.205+0000 7f32832da640 1 -- 172.21.15.104:0/476963771 shutdown_connections 2024-06-09T22:28:53.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.205+0000 7f32832da640 1 -- 172.21.15.104:0/476963771 wait complete. 2024-06-09T22:28:53.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f32832da640 1 Processor -- start 2024-06-09T22:28:53.210 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f32832da640 1 -- start start 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f32832da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f327c1008d0 0x7f327c199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f32832da640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f327c101290 0x7f327c1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f32832da640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f327c106890 0x7f327c1a4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f32832da640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f327c110cf0 con 0x7f327c106890 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f32832da640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f327c110b70 con 0x7f327c1008d0 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f32832da640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f327c110e70 con 0x7f327c101290 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f3281877640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f327c106890 0x7f327c1a4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f3281076640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f327c1008d0 0x7f327c199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f3280875640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f327c101290 0x7f327c1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f3281877640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f327c106890 0x7f327c1a4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49784/0 (socket says 172.21.15.104:49784) 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.206+0000 7f3281877640 1 -- 172.21.15.104:0/349222058 learned_addr learned my addr 172.21.15.104:0/349222058 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f3281877640 1 -- 172.21.15.104:0/349222058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f327c101290 msgr2=0x7f327c1995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f3281877640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f327c101290 0x7f327c1995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f3281877640 1 -- 172.21.15.104:0/349222058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f327c1008d0 msgr2=0x7f327c199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f3281877640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f327c1008d0 0x7f327c199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f3281877640 1 -- 172.21.15.104:0/349222058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f327c1a4650 con 0x7f327c106890 2024-06-09T22:28:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f3280875640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f327c101290 0x7f327c1995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f3281076640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f327c1008d0 0x7f327c199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:28:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f3281877640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f327c106890 0x7f327c1a4110 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f327400eb10 tx=0x7f327400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f32727fc640 1 -- 172.21.15.104:0/349222058 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274004b20 con 0x7f327c106890 2024-06-09T22:28:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f32727fc640 1 -- 172.21.15.104:0/349222058 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3274004cc0 con 0x7f327c106890 2024-06-09T22:28:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f32727fc640 1 -- 172.21.15.104:0/349222058 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274010760 con 0x7f327c106890 2024-06-09T22:28:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f32832da640 1 -- 172.21.15.104:0/349222058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f327c1a4820 con 0x7f327c106890 2024-06-09T22:28:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.207+0000 7f32832da640 1 -- 172.21.15.104:0/349222058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f327c1a4cf0 con 0x7f327c106890 2024-06-09T22:28:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.209+0000 7f32727fc640 1 -- 172.21.15.104:0/349222058 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f3274005020 con 0x7f327c106890 2024-06-09T22:28:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3285fed740 1 Processor -- start 2024-06-09T22:28:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3285fed740 1 -- start start 2024-06-09T22:28:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3285fed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecd170 0x55fd29ecd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3285fed740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecda90 0x55fd29ed7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3285fed740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29eda740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3285fed740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fd29e314f0 con 0x55fd29ed8350 2024-06-09T22:28:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3285fed740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fd29dcf870 con 0x55fd29ecda90 2024-06-09T22:28:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3285fed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fd29e2d4b0 con 0x55fd29ecd170 2024-06-09T22:28:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3280875640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecd170 0x55fd29ecd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3280875640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecd170 0x55fd29ecd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:46370/0 (socket says 172.21.15.104:46370) 2024-06-09T22:28:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3281076640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecda90 0x55fd29ed7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3280875640 1 -- 172.21.15.104:0/977141555 learned_addr learned my addr 172.21.15.104:0/977141555 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f3280875640 1 --2- 172.21.15.104:0/977141555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29eda740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.210+0000 7f32727fc640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f32680a2da0 0x7f32680a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.211+0000 7f3281076640 1 -- 172.21.15.104:0/977141555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecd170 msgr2=0x55fd29ecd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.211+0000 7f3281076640 1 --2- 172.21.15.104:0/977141555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecd170 0x55fd29ecd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.211+0000 7f3281877640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f32680a2da0 0x7f32680a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.211+0000 7f3281076640 1 -- 172.21.15.104:0/977141555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 msgr2=0x55fd29eda740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.211+0000 7f3281076640 1 --2- 172.21.15.104:0/977141555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29eda740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.211+0000 7f3281076640 1 -- 172.21.15.104:0/977141555 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd29deb9b0 con 0x55fd29ecda90 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.211+0000 7f3281076640 1 --2- 172.21.15.104:0/977141555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecda90 0x55fd29ed7e10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3278009fb0 tx=0x7f32780118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39026cc4b85e046 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.211+0000 7f32727fc640 1 -- 172.21.15.104:0/349222058 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f32740cfdf0 con 0x7f327c106890 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.211+0000 7f3281877640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f32680a2da0 0x7f32680a5260 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f327409bc20 tx=0x7f3274004e70 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.211+0000 7f32677fe640 1 -- 172.21.15.104:0/977141555 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3278004310 con 0x55fd29ecda90 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.211+0000 7f32677fe640 1 -- 172.21.15.104:0/977141555 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32780044b0 con 0x55fd29ecda90 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.211+0000 7f32677fe640 1 -- 172.21.15.104:0/977141555 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3278004310 con 0x55fd29ecda90 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.212+0000 7f3285fed740 1 -- 172.21.15.104:0/977141555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecda90 msgr2=0x55fd29ed7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.212+0000 7f3285fed740 1 --2- 172.21.15.104:0/977141555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecda90 0x55fd29ed7e10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3278009fb0 tx=0x7f32780118d0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.212+0000 7f3285fed740 1 -- 172.21.15.104:0/977141555 shutdown_connections 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.212+0000 7f3285fed740 1 --2- 172.21.15.104:0/977141555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29eda740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.212+0000 7f3285fed740 1 --2- 172.21.15.104:0/977141555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecda90 0x55fd29ed7e10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.212+0000 7f3285fed740 1 --2- 172.21.15.104:0/977141555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecd170 0x55fd29ecd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.212+0000 7f3285fed740 1 -- 172.21.15.104:0/977141555 >> 172.21.15.104:0/977141555 conn(0x55fd29ec9680 msgr2=0x55fd29ec9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.212+0000 7f3285fed740 1 -- 172.21.15.104:0/977141555 shutdown_connections 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.212+0000 7f3285fed740 1 -- 172.21.15.104:0/977141555 wait complete. 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.212+0000 7f3285fed740 1 -- start start 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.212+0000 7f3285fed740 1 Processor -- start 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3285fed740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ecd170 0x55fd29ed5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3285fed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecda90 0x55fd29ed57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3285fed740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ed8350 0x55fd29ed5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3285fed740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55fd29edaf80 con 0x55fd29ecd170 2024-06-09T22:28:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3285fed740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fd29dbaae0 con 0x55fd29ed8350 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3285fed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fd29edb100 con 0x55fd29ecda90 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3285fed740 10 client.0 _ll_register_callbacks cb 0x55fd29df39b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3280875640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ecd170 0x55fd29ed5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3280875640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ecd170 0x55fd29ed5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49798/0 (socket says 172.21.15.104:49798) 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3280875640 1 -- 172.21.15.104:0/2066521046 learned_addr learned my addr 172.21.15.104:0/2066521046 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3281076640 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecda90 0x55fd29ed57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3280875640 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ed8350 0x55fd29ed5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3280875640 1 -- 172.21.15.104:0/2066521046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecda90 msgr2=0x55fd29ed57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3280875640 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecda90 0x55fd29ed57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3280875640 1 -- 172.21.15.104:0/2066521046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ed8350 msgr2=0x55fd29ed5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3280875640 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ed8350 0x55fd29ed5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3280875640 1 -- 172.21.15.104:0/2066521046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd29b45210 con 0x55fd29ecd170 2024-06-09T22:28:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3280875640 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ecd170 0x55fd29ed5290 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f326c009230 tx=0x7f326c02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1aa2f1f518b66 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.213+0000 7f3264ff9640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3264ff9640 1 -- 172.21.15.104:0/2066521046 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c035040 con 0x55fd29ecd170 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3264ff9640 1 -- 172.21.15.104:0/2066521046 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f326c004970 con 0x55fd29ecd170 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3285fed740 1 -- 172.21.15.104:0/2066521046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32780044b0 con 0x55fd29ecd170 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3264ff9640 1 -- 172.21.15.104:0/2066521046 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f326c004b70 con 0x55fd29ecd170 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3285fed740 10 client.17634 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3285fed740 1 -- 172.21.15.104:0/2066521046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fd29c11620 con 0x55fd29ecd170 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3264ff9640 1 -- 172.21.15.104:0/2066521046 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32780044b0 con 0x55fd29ecd170 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3264ff9640 1 -- 172.21.15.104:0/2066521046 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f326c0035f0 con 0x55fd29ecd170 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3285fed740 10 client.17634 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3285fed740 10 client.17634 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3285fed740 20 client.17634 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3285fed740 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fd29ee8520 0x55fd29eea910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3285fed740 4 client.17634 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:28:53.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3285fed740 1 -- 172.21.15.104:0/2066521046 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd29e44f10 con 0x55fd29ee8520 2024-06-09T22:28:53.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.214+0000 7f3281076640 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fd29ee8520 0x55fd29eea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.215+0000 7f3264ff9640 10 client.17634 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:53.223 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.215+0000 7f3281076640 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fd29ee8520 0x55fd29eea910 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bd69a4cbcffd90 server_cookie=26c911d21a3e47c9 in_seq=0 out_seq=0 2024-06-09T22:28:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3264ff9640 1 -- 172.21.15.104:0/2066521046 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55fd29e44f10 con 0x55fd29ee8520 2024-06-09T22:28:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3264ff9640 10 client.17634 handle_command_reply: tid=0 2024-06-09T22:28:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3285fed740 1 client.17634 shutdown 2024-06-09T22:28:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3285fed740 1 -- 172.21.15.104:0/2066521046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ecd170 msgr2=0x55fd29ed5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3285fed740 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ecd170 0x55fd29ed5290 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f326c009230 tx=0x7f326c02a420 comp rx=0 tx=0).stop 2024-06-09T22:28:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3285fed740 1 -- 172.21.15.104:0/2066521046 shutdown_connections 2024-06-09T22:28:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3285fed740 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fd29ee8520 0x55fd29eea910 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3285fed740 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ed8350 0x55fd29ed5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3264ff9640 0 client.17634 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3285fed740 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecda90 0x55fd29ed57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3285fed740 1 --2- 172.21.15.104:0/2066521046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ecd170 0x55fd29ed5290 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3285fed740 1 -- 172.21.15.104:0/2066521046 >> 172.21.15.104:0/2066521046 conn(0x55fd29ec9680 msgr2=0x55fd29ecf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3285fed740 1 -- 172.21.15.104:0/2066521046 shutdown_connections 2024-06-09T22:28:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3285fed740 1 -- 172.21.15.104:0/2066521046 wait complete. 2024-06-09T22:28:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.216+0000 7f3285fed740 20 client.17634 trim_cache size 0 max 16384 2024-06-09T22:28:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.231+0000 7f3285fed740 1 Processor -- start 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.231+0000 7f3285fed740 1 -- start start 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3285fed740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecd170 0x55fd29ef0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3285fed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecda90 0x55fd29ef0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3285fed740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29ed24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3285fed740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f327c1aee80 con 0x55fd29ed8350 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3285fed740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f327c1b09c0 con 0x55fd29ecd170 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3285fed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f327c1af550 con 0x55fd29ecda90 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3280875640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29ed24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3281076640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecda90 0x55fd29ef0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3280875640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29ed24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49814/0 (socket says 172.21.15.104:49814) 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3280875640 1 -- 172.21.15.104:0/2152112792 learned_addr learned my addr 172.21.15.104:0/2152112792 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3280875640 1 --2- 172.21.15.104:0/2152112792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecd170 0x55fd29ef0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3280875640 1 -- 172.21.15.104:0/2152112792 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecda90 msgr2=0x55fd29ef0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3280875640 1 --2- 172.21.15.104:0/2152112792 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecda90 0x55fd29ef0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3280875640 1 -- 172.21.15.104:0/2152112792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecd170 msgr2=0x55fd29ef0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3280875640 1 --2- 172.21.15.104:0/2152112792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecd170 0x55fd29ef0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3280875640 1 -- 172.21.15.104:0/2152112792 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd29eeac80 con 0x55fd29ed8350 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3280875640 1 --2- 172.21.15.104:0/2152112792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29ed24f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f326c0058c0 tx=0x7f326c031510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeca8f152dc8e225 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3266ffd640 1 -- 172.21.15.104:0/2152112792 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c004d10 con 0x55fd29ed8350 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3266ffd640 1 -- 172.21.15.104:0/2152112792 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f326c041070 con 0x55fd29ed8350 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.232+0000 7f3266ffd640 1 -- 172.21.15.104:0/2152112792 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c031cf0 con 0x55fd29ed8350 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.233+0000 7f3285fed740 1 -- 172.21.15.104:0/2152112792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 msgr2=0x55fd29ed24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.233+0000 7f3285fed740 1 --2- 172.21.15.104:0/2152112792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29ed24f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f326c0058c0 tx=0x7f326c031510 comp rx=0 tx=0).stop 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.233+0000 7f3285fed740 1 -- 172.21.15.104:0/2152112792 shutdown_connections 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.233+0000 7f3285fed740 1 --2- 172.21.15.104:0/2152112792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29ed24f0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.233+0000 7f3285fed740 1 --2- 172.21.15.104:0/2152112792 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecda90 0x55fd29ef0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.233+0000 7f3285fed740 1 --2- 172.21.15.104:0/2152112792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecd170 0x55fd29ef0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.233+0000 7f3285fed740 1 -- 172.21.15.104:0/2152112792 >> 172.21.15.104:0/2152112792 conn(0x55fd29ec9680 msgr2=0x55fd29ed04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.233+0000 7f3285fed740 1 -- 172.21.15.104:0/2152112792 shutdown_connections 2024-06-09T22:28:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.233+0000 7f3285fed740 1 -- 172.21.15.104:0/2152112792 wait complete. 2024-06-09T22:28:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.233+0000 7f3285fed740 1 -- start start 2024-06-09T22:28:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3285fed740 1 Processor -- start 2024-06-09T22:28:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3285fed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecd170 0x55fd29ee0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3285fed740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecda90 0x55fd29ee0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3285fed740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29edbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3285fed740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f327c1b02f0 con 0x55fd29ed8350 2024-06-09T22:28:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3285fed740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f327c1afc20 con 0x55fd29ecda90 2024-06-09T22:28:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3285fed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f327c1ace80 con 0x55fd29ecd170 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3285fed740 10 client.0 _ll_register_callbacks cb 0x7f326c005280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3280875640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29edbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3281076640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecda90 0x55fd29ee0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3280875640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29edbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49818/0 (socket says 172.21.15.104:49818) 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3281076640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecda90 0x55fd29ee0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:39388/0 (socket says 172.21.15.104:39388) 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3280875640 1 -- 172.21.15.104:0/2240167559 learned_addr learned my addr 172.21.15.104:0/2240167559 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3280875640 1 --2- 172.21.15.104:0/2240167559 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecd170 0x55fd29ee0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3280875640 1 -- 172.21.15.104:0/2240167559 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecd170 msgr2=0x55fd29ee0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3280875640 1 --2- 172.21.15.104:0/2240167559 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecd170 0x55fd29ee0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3280875640 1 -- 172.21.15.104:0/2240167559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecda90 msgr2=0x55fd29ee0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3280875640 1 --2- 172.21.15.104:0/2240167559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecda90 0x55fd29ee0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.234+0000 7f3280875640 1 -- 172.21.15.104:0/2240167559 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd29edc4d0 con 0x55fd29ed8350 2024-06-09T22:28:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3280875640 1 --2- 172.21.15.104:0/2240167559 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29edbcd0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f326c009850 tx=0x7f326c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879765fd2af648c2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3264ff9640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3264ff9640 1 -- 172.21.15.104:0/2240167559 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c0315b0 con 0x55fd29ed8350 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3264ff9640 1 -- 172.21.15.104:0/2240167559 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f326c031750 con 0x55fd29ed8350 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3285fed740 1 -- 172.21.15.104:0/2240167559 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f326c041070 con 0x55fd29ed8350 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3264ff9640 1 -- 172.21.15.104:0/2240167559 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f326c031950 con 0x55fd29ed8350 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3285fed740 10 client.17640 fetch_fsmap learned FSMap version 32 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3285fed740 1 -- 172.21.15.104:0/2240167559 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fd29f57240 con 0x55fd29ed8350 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3264ff9640 1 -- 172.21.15.104:0/2240167559 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f326c041070 con 0x55fd29ed8350 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3264ff9640 1 -- 172.21.15.104:0/2240167559 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f326c059430 con 0x55fd29ed8350 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3285fed740 10 client.17640 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3285fed740 10 client.17640 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3285fed740 20 client.17640 populate_metadata read hostname 'smithi104' 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3285fed740 1 --2- 172.21.15.104:0/2240167559 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fd29f577f0 0x55fd29f59be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:28:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3285fed740 4 client.17640 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:28:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3285fed740 1 -- 172.21.15.104:0/2240167559 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fd29e3ec20 con 0x55fd29f577f0 2024-06-09T22:28:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.235+0000 7f3281076640 1 --2- 172.21.15.104:0/2240167559 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fd29f577f0 0x55fd29f59be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:28:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.236+0000 7f3264ff9640 10 client.17640 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.236+0000 7f3281076640 1 --2- 172.21.15.104:0/2240167559 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fd29f577f0 0x55fd29f59be0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ccf357c9c463567 server_cookie=f38d550939623d35 in_seq=0 out_seq=0 2024-06-09T22:28:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.245+0000 7f3264ff9640 1 -- 172.21.15.104:0/2240167559 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55fd29e3ec20 con 0x55fd29f577f0 2024-06-09T22:28:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3264ff9640 10 client.17640 handle_command_reply: tid=0 2024-06-09T22:28:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3285fed740 1 client.17640 shutdown 2024-06-09T22:28:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3285fed740 1 -- 172.21.15.104:0/2240167559 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 msgr2=0x55fd29edbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3285fed740 1 --2- 172.21.15.104:0/2240167559 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29edbcd0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f326c009850 tx=0x7f326c005740 comp rx=0 tx=0).stop 2024-06-09T22:28:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3285fed740 1 -- 172.21.15.104:0/2240167559 shutdown_connections 2024-06-09T22:28:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3285fed740 1 --2- 172.21.15.104:0/2240167559 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55fd29f577f0 0x55fd29f59be0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3285fed740 1 --2- 172.21.15.104:0/2240167559 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fd29ed8350 0x55fd29edbcd0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3285fed740 1 --2- 172.21.15.104:0/2240167559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd29ecda90 0x55fd29ee0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3285fed740 1 --2- 172.21.15.104:0/2240167559 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fd29ecd170 0x55fd29ee0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3264ff9640 0 client.17640 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:28:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3285fed740 1 -- 172.21.15.104:0/2240167559 >> 172.21.15.104:0/2240167559 conn(0x55fd29ec9680 msgr2=0x55fd29eceb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3285fed740 1 -- 172.21.15.104:0/2240167559 shutdown_connections 2024-06-09T22:28:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3285fed740 1 -- 172.21.15.104:0/2240167559 wait complete. 2024-06-09T22:28:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f3285fed740 20 client.17640 trim_cache size 0 max 16384 2024-06-09T22:28:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:28:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:28:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:28:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f32832da640 1 -- 172.21.15.104:0/349222058 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f32680a2da0 msgr2=0x7f32680a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.246+0000 7f32832da640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f32680a2da0 0x7f32680a5260 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f327409bc20 tx=0x7f3274004e70 comp rx=0 tx=0).stop 2024-06-09T22:28:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.247+0000 7f32832da640 1 -- 172.21.15.104:0/349222058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f327c106890 msgr2=0x7f327c1a4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:28:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.247+0000 7f32832da640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f327c106890 0x7f327c1a4110 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f327400eb10 tx=0x7f327400c2d0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.247+0000 7f32832da640 1 -- 172.21.15.104:0/349222058 shutdown_connections 2024-06-09T22:28:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.247+0000 7f32832da640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f32680a2da0 0x7f32680a5260 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.247+0000 7f32832da640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f327c106890 0x7f327c1a4110 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.247+0000 7f32832da640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f327c101290 0x7f327c1995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.247+0000 7f32832da640 1 --2- 172.21.15.104:0/349222058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f327c1008d0 0x7f327c199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:28:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.247+0000 7f32832da640 1 -- 172.21.15.104:0/349222058 >> 172.21.15.104:0/349222058 conn(0x7f327c077a20 msgr2=0x7f327c103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:28:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.247+0000 7f32832da640 1 -- 172.21.15.104:0/349222058 shutdown_connections 2024-06-09T22:28:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:28:53.247+0000 7f32832da640 1 -- 172.21.15.104:0/349222058 wait complete. 2024-06-09T22:28:53.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 308904 5.36 MB/sec execute 239 sec latency 17.311 ms 2024-06-09T22:28:53.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:53 smithi153 ceph-mon[36599]: pgmap v242: 129 pgs: 129 active+clean; 235 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-06-09T22:28:54.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:53 smithi104 ceph-mon[32389]: pgmap v242: 129 pgs: 129 active+clean; 235 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-06-09T22:28:54.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:53 smithi194 ceph-mon[36856]: pgmap v242: 129 pgs: 129 active+clean; 235 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-06-09T22:28:54.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 310119 5.36 MB/sec execute 240 sec latency 76.258 ms 2024-06-09T22:28:55.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 312372 5.39 MB/sec execute 241 sec latency 38.025 ms 2024-06-09T22:28:55.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:55 smithi153 ceph-mon[36599]: pgmap v243: 129 pgs: 129 active+clean; 242 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 299 op/s 2024-06-09T22:28:55.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:55 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:28:55.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:55 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:28:55.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:55 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:28:56.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:55 smithi104 ceph-mon[32389]: pgmap v243: 129 pgs: 129 active+clean; 242 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 299 op/s 2024-06-09T22:28:56.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:55 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:28:56.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:55 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:28:56.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:55 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:28:56.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:55 smithi194 ceph-mon[36856]: pgmap v243: 129 pgs: 129 active+clean; 242 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 299 op/s 2024-06-09T22:28:56.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:55 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:28:56.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:55 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:28:56.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:55 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:28:56.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 313674 5.39 MB/sec execute 242 sec latency 17.345 ms 2024-06-09T22:28:57.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 315964 5.43 MB/sec execute 243 sec latency 19.876 ms 2024-06-09T22:28:57.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:57 smithi153 ceph-mon[36599]: pgmap v244: 129 pgs: 129 active+clean; 242 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 197 op/s 2024-06-09T22:28:58.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:57 smithi104 ceph-mon[32389]: pgmap v244: 129 pgs: 129 active+clean; 242 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 197 op/s 2024-06-09T22:28:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:57 smithi194 ceph-mon[36856]: pgmap v244: 129 pgs: 129 active+clean; 242 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 197 op/s 2024-06-09T22:28:58.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 317326 5.43 MB/sec execute 244 sec latency 18.121 ms 2024-06-09T22:28:59.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 319475 5.46 MB/sec execute 245 sec latency 50.420 ms 2024-06-09T22:28:59.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:28:59 smithi153 ceph-mon[36599]: pgmap v245: 129 pgs: 129 active+clean; 242 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 197 op/s 2024-06-09T22:29:00.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:28:59 smithi104 ceph-mon[32389]: pgmap v245: 129 pgs: 129 active+clean; 242 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 197 op/s 2024-06-09T22:29:00.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:28:59 smithi194 ceph-mon[36856]: pgmap v245: 129 pgs: 129 active+clean; 242 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 197 op/s 2024-06-09T22:29:00.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 320739 5.46 MB/sec execute 246 sec latency 32.772 ms 2024-06-09T22:29:01.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 321536 5.44 MB/sec execute 247 sec latency 54.030 ms 2024-06-09T22:29:01.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:01 smithi153 ceph-mon[36599]: pgmap v246: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 298 op/s 2024-06-09T22:29:02.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:01 smithi104 ceph-mon[32389]: pgmap v246: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 298 op/s 2024-06-09T22:29:02.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:01 smithi194 ceph-mon[36856]: pgmap v246: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 298 op/s 2024-06-09T22:29:02.739 INFO:tasks.workunit.client.0.smithi104.stdout: 1 323206 5.47 MB/sec execute 248 sec latency 51.046 ms 2024-06-09T22:29:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:29:03.467 INFO:tasks.workunit.client.0.smithi104.stdout:[324320] write failed on handle 13652 (Resource temporarily unavailable) 2024-06-09T22:29:03.467 INFO:tasks.workunit.client.0.smithi104.stdout:Child failed with status 1 2024-06-09T22:29:03.467 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-06-09T22:29:03.468 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-06-09T22:29:03.756 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:03 smithi104 ceph-mon[32389]: pgmap v247: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 199 op/s 2024-06-09T22:29:03.757 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi104 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7b7a5d375cf4e4afab96b9d4d229b699ae8304bb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-06-09T22:29:03.979 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=97aa6dc78f564db98a050a8253e0ba2a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi104 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7b7a5d375cf4e4afab96b9d4d229b699ae8304bb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-06-09T22:29:03.981 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-06-09T22:29:03.992 INFO:tasks.quiescer:joining Quiescers 2024-06-09T22:29:03.992 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-06-09T22:29:03.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:29:03.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:03 smithi153 ceph-mon[36599]: pgmap v247: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 199 op/s 2024-06-09T22:29:04.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:03 smithi194 ceph-mon[36856]: pgmap v247: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 199 op/s 2024-06-09T22:29:04.244 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:29:05.697 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:05 smithi104 ceph-mon[32389]: pgmap v248: 129 pgs: 129 active+clean; 255 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 320 op/s 2024-06-09T22:29:05.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:05 smithi153 ceph-mon[36599]: pgmap v248: 129 pgs: 129 active+clean; 255 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 320 op/s 2024-06-09T22:29:06.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:05 smithi194 ceph-mon[36856]: pgmap v248: 129 pgs: 129 active+clean; 255 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 320 op/s 2024-06-09T22:29:06.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.264+0000 7f9ca7746700 1 -- 172.21.15.104:0/2009027977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca0107430 msgr2=0x7f9ca0107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.264+0000 7f9ca7746700 1 --2- 172.21.15.104:0/2009027977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca0107430 0x7f9ca0107810 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009990 tx=0x7f9c9c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.264+0000 7f9ca7746700 1 -- 172.21.15.104:0/2009027977 shutdown_connections 2024-06-09T22:29:06.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.264+0000 7f9ca7746700 1 --2- 172.21.15.104:0/2009027977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca0107430 0x7f9ca0107810 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.264+0000 7f9ca7746700 1 --2- 172.21.15.104:0/2009027977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ca0101d80 0x7f9ca010e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.264+0000 7f9ca7746700 1 --2- 172.21.15.104:0/2009027977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca01013c0 0x7f9ca0101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.264+0000 7f9ca7746700 1 -- 172.21.15.104:0/2009027977 >> 172.21.15.104:0/2009027977 conn(0x7f9ca00783d0 msgr2=0x7f9ca00787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:06.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.264+0000 7f9ca7746700 1 -- 172.21.15.104:0/2009027977 shutdown_connections 2024-06-09T22:29:06.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.264+0000 7f9ca7746700 1 -- 172.21.15.104:0/2009027977 wait complete. 2024-06-09T22:29:06.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.265+0000 7f9ca7746700 1 Processor -- start 2024-06-09T22:29:06.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.265+0000 7f9ca7746700 1 -- start start 2024-06-09T22:29:06.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.265+0000 7f9ca7746700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca01013c0 0x7f9ca019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.265+0000 7f9ca7746700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ca0101d80 0x7f9ca019ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.265+0000 7f9ca7746700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca0107430 0x7f9ca0198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.265+0000 7f9ca7746700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01116a0 con 0x7f9ca01013c0 2024-06-09T22:29:06.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.265+0000 7f9ca7746700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0111520 con 0x7f9ca0107430 2024-06-09T22:29:06.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.265+0000 7f9ca7746700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0111820 con 0x7f9ca0101d80 2024-06-09T22:29:06.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.265+0000 7f9ca54e2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca01013c0 0x7f9ca019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.265+0000 7f9ca54e2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca01013c0 0x7f9ca019e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58574/0 (socket says 172.21.15.104:58574) 2024-06-09T22:29:06.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.265+0000 7f9ca54e2700 1 -- 172.21.15.104:0/3373017241 learned_addr learned my addr 172.21.15.104:0/3373017241 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:06.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.265+0000 7f9ca4ce1700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ca0101d80 0x7f9ca019ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.266+0000 7f9ca5ce3700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca0107430 0x7f9ca0198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.266+0000 7f9ca54e2700 1 -- 172.21.15.104:0/3373017241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ca0101d80 msgr2=0x7f9ca019ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.266+0000 7f9ca54e2700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ca0101d80 0x7f9ca019ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.266+0000 7f9ca54e2700 1 -- 172.21.15.104:0/3373017241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca0107430 msgr2=0x7f9ca0198d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.266+0000 7f9ca54e2700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca0107430 0x7f9ca0198d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.266+0000 7f9ca54e2700 1 -- 172.21.15.104:0/3373017241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca0199640 con 0x7f9ca01013c0 2024-06-09T22:29:06.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.266+0000 7f9ca54e2700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca01013c0 0x7f9ca019e8f0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c000c00 tx=0x7f9c9c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:06.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.266+0000 7f9c967fc700 1 -- 172.21.15.104:0/3373017241 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c027070 con 0x7f9ca01013c0 2024-06-09T22:29:06.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.266+0000 7f9ca7746700 1 -- 172.21.15.104:0/3373017241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca01998d0 con 0x7f9ca01013c0 2024-06-09T22:29:06.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.266+0000 7f9ca7746700 1 -- 172.21.15.104:0/3373017241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca01a5890 con 0x7f9ca01013c0 2024-06-09T22:29:06.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.266+0000 7f9c967fc700 1 -- 172.21.15.104:0/3373017241 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c9c003780 con 0x7f9ca01013c0 2024-06-09T22:29:06.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.266+0000 7f9c967fc700 1 -- 172.21.15.104:0/3373017241 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c022db0 con 0x7f9ca01013c0 2024-06-09T22:29:06.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.267+0000 7f9ca7746700 1 -- 172.21.15.104:0/3373017241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c84005330 con 0x7f9ca01013c0 2024-06-09T22:29:06.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.269+0000 7f9c967fc700 1 -- 172.21.15.104:0/3373017241 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f9c9c02a070 con 0x7f9ca01013c0 2024-06-09T22:29:06.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.269+0000 7f9c967fc700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9c8c0a2e10 0x7f9c8c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.269+0000 7f9ca4ce1700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9c8c0a2e10 0x7f9c8c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.269+0000 7f9c967fc700 1 -- 172.21.15.104:0/3373017241 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9584+0+0 (secure 0 0 0) 0x7f9c9c0d0390 con 0x7f9ca01013c0 2024-06-09T22:29:06.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.270+0000 7f9ca4ce1700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9c8c0a2e10 0x7f9c8c0a52d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9ca019a170 tx=0x7f9c9000a380 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:06.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.278+0000 7f9c967fc700 1 -- 172.21.15.104:0/3373017241 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9c9c013070 con 0x7f9ca01013c0 2024-06-09T22:29:06.346 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.341+0000 7fc5cad34640 1 -- 172.21.15.104:0/419050253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c41014b0 msgr2=0x7fc5c410d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.346 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.341+0000 7fc5cad34640 1 --2- 172.21.15.104:0/419050253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c41014b0 0x7fc5c410d6e0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc009ca0 tx=0x7fc5bc02a8c0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.347 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 -- 172.21.15.104:0/419050253 shutdown_connections 2024-06-09T22:29:06.347 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 --2- 172.21.15.104:0/419050253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c41014b0 0x7fc5c410d6e0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.347 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 --2- 172.21.15.104:0/419050253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5c4100b10 0x7fc5c4100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.347 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 --2- 172.21.15.104:0/419050253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c4106940 0x7fc5c4106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.347 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 -- 172.21.15.104:0/419050253 >> 172.21.15.104:0/419050253 conn(0x7fc5c4077780 msgr2=0x7fc5c4077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:06.347 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 -- 172.21.15.104:0/419050253 shutdown_connections 2024-06-09T22:29:06.347 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 -- 172.21.15.104:0/419050253 wait complete. 2024-06-09T22:29:06.347 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 Processor -- start 2024-06-09T22:29:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 -- start start 2024-06-09T22:29:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c4100b10 0x7fc5c4198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c41014b0 0x7fc5c4199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5c4106940 0x7fc5c41a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c4110cc0 con 0x7fc5c41014b0 2024-06-09T22:29:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c4110b40 con 0x7fc5c4100b10 2024-06-09T22:29:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.342+0000 7fc5cad34640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c4110e40 con 0x7fc5c4106940 2024-06-09T22:29:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c41014b0 0x7fc5c4199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c41014b0 0x7fc5c4199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58592/0 (socket says 172.21.15.104:58592) 2024-06-09T22:29:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c3fff640 1 -- 172.21.15.104:0/1981621772 learned_addr learned my addr 172.21.15.104:0/1981621772 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c4100b10 0x7fc5c4198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c92d1640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5c4106940 0x7fc5c41a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c3fff640 1 -- 172.21.15.104:0/1981621772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5c4106940 msgr2=0x7fc5c41a4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c3fff640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5c4106940 0x7fc5c41a4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c3fff640 1 -- 172.21.15.104:0/1981621772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c4100b10 msgr2=0x7fc5c4198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c3fff640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c4100b10 0x7fc5c4198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c3fff640 1 -- 172.21.15.104:0/1981621772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c41a45a0 con 0x7fc5c41014b0 2024-06-09T22:29:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c3fff640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c41014b0 0x7fc5c4199520 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000ea60 tx=0x7fc5b000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c1ffb640 1 -- 172.21.15.104:0/1981621772 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000cde0 con 0x7fc5c41014b0 2024-06-09T22:29:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c1ffb640 1 -- 172.21.15.104:0/1981621772 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b00035f0 con 0x7fc5c41014b0 2024-06-09T22:29:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5cad34640 1 -- 172.21.15.104:0/1981621772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c41a4770 con 0x7fc5c41014b0 2024-06-09T22:29:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5c1ffb640 1 -- 172.21.15.104:0/1981621772 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0010670 con 0x7fc5c41014b0 2024-06-09T22:29:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.343+0000 7fc5cad34640 1 -- 172.21.15.104:0/1981621772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c41a4c90 con 0x7fc5c41014b0 2024-06-09T22:29:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.345+0000 7fc5c1ffb640 1 -- 172.21.15.104:0/1981621772 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc5b00037a0 con 0x7fc5c41014b0 2024-06-09T22:29:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.346+0000 7fc5c1ffb640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc5b40a2da0 0x7fc5b40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.346+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc5b40a2da0 0x7fc5b40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.346+0000 7fc5c1ffb640 1 -- 172.21.15.104:0/1981621772 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9584+0+0 (secure 0 0 0) 0x7fc5b00cfb20 con 0x7fc5c41014b0 2024-06-09T22:29:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.347+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc5b40a2da0 0x7fc5b40a5260 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009b50 tx=0x7fc5b8005f90 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.347+0000 7fc5cda43740 1 Processor -- start 2024-06-09T22:29:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.347+0000 7fc5cda43740 1 -- start start 2024-06-09T22:29:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.347+0000 7fc5cda43740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 0x559415279550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.347+0000 7fc5cda43740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415279a90 0x559415283e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.347+0000 7fc5cda43740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415284350 0x559415286740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.347+0000 7fc5cda43740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5594151dd4f0 con 0x559415279170 2024-06-09T22:29:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.347+0000 7fc5cda43740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55941517b870 con 0x559415284350 2024-06-09T22:29:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.347+0000 7fc5cda43740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5594151d94b0 con 0x559415279a90 2024-06-09T22:29:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.347+0000 7fc5c8ad0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 0x559415279550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.347+0000 7fc5c8ad0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 0x559415279550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58596/0 (socket says 172.21.15.104:58596) 2024-06-09T22:29:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.347+0000 7fc5c8ad0640 1 -- 172.21.15.104:0/2388247282 learned_addr learned my addr 172.21.15.104:0/2388247282 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.348+0000 7fc5c3fff640 1 --2- 172.21.15.104:0/2388247282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415279a90 0x559415283e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.348+0000 7fc5c92d1640 1 --2- 172.21.15.104:0/2388247282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415284350 0x559415286740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.348+0000 7fc5c8ad0640 1 -- 172.21.15.104:0/2388247282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415279a90 msgr2=0x559415283e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.348+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/2388247282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415279a90 0x559415283e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.348+0000 7fc5c8ad0640 1 -- 172.21.15.104:0/2388247282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415284350 msgr2=0x559415286740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.348+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/2388247282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415284350 0x559415286740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.348+0000 7fc5c8ad0640 1 -- 172.21.15.104:0/2388247282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594151979b0 con 0x559415279170 2024-06-09T22:29:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.348+0000 7fc5c92d1640 1 --2- 172.21.15.104:0/2388247282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415284350 0x559415286740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:29:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.348+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/2388247282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 0x559415279550 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8005bc0 tx=0x7fc5b8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622b40d782b0b1bc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.348+0000 7fc5aeffd640 1 -- 172.21.15.104:0/2388247282 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b801e420 con 0x559415279170 2024-06-09T22:29:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.348+0000 7fc5aeffd640 1 -- 172.21.15.104:0/2388247282 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b801eaa0 con 0x559415279170 2024-06-09T22:29:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.348+0000 7fc5aeffd640 1 -- 172.21.15.104:0/2388247282 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8026c20 con 0x559415279170 2024-06-09T22:29:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.349+0000 7fc5cda43740 1 -- 172.21.15.104:0/2388247282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 msgr2=0x559415279550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.349+0000 7fc5cda43740 1 --2- 172.21.15.104:0/2388247282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 0x559415279550 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8005bc0 tx=0x7fc5b8014000 comp rx=0 tx=0).stop 2024-06-09T22:29:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.349+0000 7fc5cda43740 1 -- 172.21.15.104:0/2388247282 shutdown_connections 2024-06-09T22:29:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.349+0000 7fc5cda43740 1 --2- 172.21.15.104:0/2388247282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415284350 0x559415286740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.349+0000 7fc5cda43740 1 --2- 172.21.15.104:0/2388247282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415279a90 0x559415283e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.349+0000 7fc5cda43740 1 --2- 172.21.15.104:0/2388247282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 0x559415279550 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.349+0000 7fc5cda43740 1 -- 172.21.15.104:0/2388247282 >> 172.21.15.104:0/2388247282 conn(0x559415275680 msgr2=0x559415275a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.349+0000 7fc5cda43740 1 -- 172.21.15.104:0/2388247282 shutdown_connections 2024-06-09T22:29:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.349+0000 7fc5cda43740 1 -- 172.21.15.104:0/2388247282 wait complete. 2024-06-09T22:29:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.349+0000 7fc5cda43740 1 -- start start 2024-06-09T22:29:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.349+0000 7fc5cda43740 1 Processor -- start 2024-06-09T22:29:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5cda43740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415279170 0x559415281310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5cda43740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 0x559415281850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5cda43740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415284350 0x559415281d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5cda43740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559415286f80 con 0x559415279a90 2024-06-09T22:29:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5cda43740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559415166ae0 con 0x559415279170 2024-06-09T22:29:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5cda43740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559415287100 con 0x559415284350 2024-06-09T22:29:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5cda43740 10 client.0 _ll_register_callbacks cb 0x55941519f9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 0x559415281850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 0x559415281850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58604/0 (socket says 172.21.15.104:58604) 2024-06-09T22:29:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c3fff640 1 -- 172.21.15.104:0/4192274211 learned_addr learned my addr 172.21.15.104:0/4192274211 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c92d1640 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415284350 0x559415281d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415279170 0x559415281310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c3fff640 1 -- 172.21.15.104:0/4192274211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415284350 msgr2=0x559415281d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c3fff640 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415284350 0x559415281d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c3fff640 1 -- 172.21.15.104:0/4192274211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415279170 msgr2=0x559415281310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c3fff640 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415279170 0x559415281310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c3fff640 1 -- 172.21.15.104:0/4192274211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559414ef1210 con 0x559415279a90 2024-06-09T22:29:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415279170 0x559415281310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:29:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c92d1640 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415284350 0x559415281d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5c3fff640 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 0x559415281850 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00985f0 tx=0x7fc5b009c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47a8b6ba0c7ba03 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5a3fff640 1 -- 172.21.15.104:0/4192274211 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0098290 con 0x559415279a90 2024-06-09T22:29:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5a3fff640 1 -- 172.21.15.104:0/4192274211 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9584+0+0 (secure 0 0 0) 0x7fc5b009b070 con 0x559415279a90 2024-06-09T22:29:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5cda43740 1 -- 172.21.15.104:0/4192274211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5b801eaa0 con 0x559415279a90 2024-06-09T22:29:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5a3fff640 1 -- 172.21.15.104:0/4192274211 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc5b0098430 con 0x559415279a90 2024-06-09T22:29:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5cda43740 10 client.17658 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.350+0000 7fc5cda43740 1 -- 172.21.15.104:0/4192274211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559414fbd620 con 0x559415279a90 2024-06-09T22:29:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.351+0000 7fc5a3fff640 1 -- 172.21.15.104:0/4192274211 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b801eaa0 con 0x559415279a90 2024-06-09T22:29:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.351+0000 7fc5a3fff640 1 -- 172.21.15.104:0/4192274211 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc5b00ba2b0 con 0x559415279a90 2024-06-09T22:29:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.351+0000 7fc5cda43740 10 client.17658 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.351+0000 7fc5cda43740 10 client.17658 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:29:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.351+0000 7fc5cda43740 20 client.17658 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.351+0000 7fc5cda43740 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559415294470 0x559415296880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.351+0000 7fc5cda43740 4 client.17658 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:29:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.351+0000 7fc5cda43740 1 -- 172.21.15.104:0/4192274211 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594151f0f10 con 0x559415294470 2024-06-09T22:29:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.351+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559415294470 0x559415296880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.351+0000 7fc5a3fff640 10 client.17658 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.351+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559415294470 0x559415296880 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac3fbfcfe997a000 server_cookie=7b339911843ffa2d in_seq=0 out_seq=0 2024-06-09T22:29:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.352+0000 7fc5a3fff640 1 -- 172.21.15.104:0/4192274211 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5594151f0f10 con 0x559415294470 2024-06-09T22:29:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.352+0000 7fc5a3fff640 10 client.17658 handle_command_reply: tid=0 2024-06-09T22:29:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.352+0000 7fc5cda43740 1 client.17658 shutdown 2024-06-09T22:29:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.352+0000 7fc5cda43740 1 -- 172.21.15.104:0/4192274211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 msgr2=0x559415281850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.352+0000 7fc5cda43740 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 0x559415281850 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00985f0 tx=0x7fc5b009c460 comp rx=0 tx=0).stop 2024-06-09T22:29:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.352+0000 7fc5cda43740 1 -- 172.21.15.104:0/4192274211 shutdown_connections 2024-06-09T22:29:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.352+0000 7fc5cda43740 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x559415294470 0x559415296880 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.352+0000 7fc5cda43740 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415284350 0x559415281d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.352+0000 7fc5a3fff640 0 client.17658 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.352+0000 7fc5cda43740 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 0x559415281850 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.352+0000 7fc5cda43740 1 --2- 172.21.15.104:0/4192274211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415279170 0x559415281310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.352+0000 7fc5cda43740 1 -- 172.21.15.104:0/4192274211 >> 172.21.15.104:0/4192274211 conn(0x559415275680 msgr2=0x55941527b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.353+0000 7fc5cda43740 1 -- 172.21.15.104:0/4192274211 shutdown_connections 2024-06-09T22:29:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.353+0000 7fc5cda43740 1 -- 172.21.15.104:0/4192274211 wait complete. 2024-06-09T22:29:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.353+0000 7fc5cda43740 20 client.17658 trim_cache size 0 max 16384 2024-06-09T22:29:06.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.383+0000 7fc5cda43740 1 Processor -- start 2024-06-09T22:29:06.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5cda43740 1 -- start start 2024-06-09T22:29:06.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5cda43740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 0x55941527bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5cda43740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415279a90 0x5594152819e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5cda43740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415284350 0x55941529c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5cda43740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c41aefe0 con 0x559415279170 2024-06-09T22:29:06.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5cda43740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c41b0b20 con 0x559415284350 2024-06-09T22:29:06.388 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5cda43740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c41af6b0 con 0x559415279a90 2024-06-09T22:29:06.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5c8ad0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 0x55941527bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5c8ad0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 0x55941527bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58614/0 (socket says 172.21.15.104:58614) 2024-06-09T22:29:06.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5c8ad0640 1 -- 172.21.15.104:0/1858260220 learned_addr learned my addr 172.21.15.104:0/1858260220 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:06.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5c92d1640 1 --2- 172.21.15.104:0/1858260220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415284350 0x55941529c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5c3fff640 1 --2- 172.21.15.104:0/1858260220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415279a90 0x5594152819e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5c8ad0640 1 -- 172.21.15.104:0/1858260220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415279a90 msgr2=0x5594152819e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:06.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/1858260220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415279a90 0x5594152819e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.389 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5c8ad0640 1 -- 172.21.15.104:0/1858260220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415284350 msgr2=0x55941529c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:06.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/1858260220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415284350 0x55941529c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5c8ad0640 1 -- 172.21.15.104:0/1858260220 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559415278f30 con 0x559415279170 2024-06-09T22:29:06.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5c3fff640 1 --2- 172.21.15.104:0/1858260220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415279a90 0x5594152819e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:29:06.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.384+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/1858260220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 0x55941527bb30 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80118d0 tx=0x7fc5b80267c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd790f623819143 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:06.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.385+0000 7fc5ae7fc640 1 -- 172.21.15.104:0/1858260220 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8026970 con 0x559415279170 2024-06-09T22:29:06.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.385+0000 7fc5ae7fc640 1 -- 172.21.15.104:0/1858260220 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b8005050 con 0x559415279170 2024-06-09T22:29:06.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.385+0000 7fc5ae7fc640 1 -- 172.21.15.104:0/1858260220 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b801e8b0 con 0x559415279170 2024-06-09T22:29:06.390 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.385+0000 7fc5cda43740 1 -- 172.21.15.104:0/1858260220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 msgr2=0x55941527bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.385+0000 7fc5cda43740 1 --2- 172.21.15.104:0/1858260220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 0x55941527bb30 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80118d0 tx=0x7fc5b80267c0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.385+0000 7fc5cda43740 1 -- 172.21.15.104:0/1858260220 shutdown_connections 2024-06-09T22:29:06.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.385+0000 7fc5cda43740 1 --2- 172.21.15.104:0/1858260220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415284350 0x55941529c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.385+0000 7fc5cda43740 1 --2- 172.21.15.104:0/1858260220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415279a90 0x5594152819e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.385+0000 7fc5cda43740 1 --2- 172.21.15.104:0/1858260220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279170 0x55941527bb30 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.385+0000 7fc5cda43740 1 -- 172.21.15.104:0/1858260220 >> 172.21.15.104:0/1858260220 conn(0x559415275680 msgr2=0x5594152812d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:06.391 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.385+0000 7fc5cda43740 1 -- 172.21.15.104:0/1858260220 shutdown_connections 2024-06-09T22:29:06.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.385+0000 7fc5cda43740 1 -- 172.21.15.104:0/1858260220 wait complete. 2024-06-09T22:29:06.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5cda43740 1 -- start start 2024-06-09T22:29:06.392 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5cda43740 1 Processor -- start 2024-06-09T22:29:06.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5cda43740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415279170 0x559415287910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5cda43740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 0x559415287e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.394 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5cda43740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415284350 0x5594152fd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5cda43740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c41b0450 con 0x559415279a90 2024-06-09T22:29:06.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5cda43740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c41afd80 con 0x559415279170 2024-06-09T22:29:06.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5cda43740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c41acfc0 con 0x559415284350 2024-06-09T22:29:06.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5cda43740 10 client.0 _ll_register_callbacks cb 0x7fc5b0003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:06.395 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5c3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 0x559415287e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5c3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 0x559415287e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58622/0 (socket says 172.21.15.104:58622) 2024-06-09T22:29:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5c3fff640 1 -- 172.21.15.104:0/3172449072 learned_addr learned my addr 172.21.15.104:0/3172449072 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5c92d1640 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415284350 0x5594152fd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.386+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415279170 0x559415287910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5c3fff640 1 -- 172.21.15.104:0/3172449072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415284350 msgr2=0x5594152fd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5c3fff640 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415284350 0x5594152fd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5c3fff640 1 -- 172.21.15.104:0/3172449072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415279170 msgr2=0x559415287910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5c3fff640 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415279170 0x559415287910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5c3fff640 1 -- 172.21.15.104:0/3172449072 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594152fd6e0 con 0x559415279a90 2024-06-09T22:29:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5c3fff640 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 0x559415287e50 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00a4360 tx=0x7fc5b000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75e579bc81df6f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5a3fff640 1 -- 172.21.15.104:0/3172449072 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0081060 con 0x559415279a90 2024-06-09T22:29:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5a3fff640 1 -- 172.21.15.104:0/3172449072 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9584+0+0 (secure 0 0 0) 0x7fc5b0097970 con 0x559415279a90 2024-06-09T22:29:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5cda43740 1 -- 172.21.15.104:0/3172449072 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5b8005050 con 0x559415279a90 2024-06-09T22:29:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5a3fff640 1 -- 172.21.15.104:0/3172449072 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc5b0098290 con 0x559415279a90 2024-06-09T22:29:06.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5cda43740 10 client.17664 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:06.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5cda43740 1 -- 172.21.15.104:0/3172449072 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5594152fe930 con 0x559415279a90 2024-06-09T22:29:06.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5a3fff640 1 -- 172.21.15.104:0/3172449072 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b8005050 con 0x559415279a90 2024-06-09T22:29:06.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5a3fff640 1 -- 172.21.15.104:0/3172449072 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc5b009b070 con 0x559415279a90 2024-06-09T22:29:06.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5cda43740 10 client.17664 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:06.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5cda43740 10 client.17664 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-06-09T22:29:06.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5cda43740 20 client.17664 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:06.398 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5cda43740 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5594152ff0b0 0x5594153014d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5cda43740 4 client.17664 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-06-09T22:29:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.387+0000 7fc5cda43740 1 -- 172.21.15.104:0/3172449072 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5594151eac20 con 0x5594152ff0b0 2024-06-09T22:29:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.388+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5594152ff0b0 0x5594153014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.388+0000 7fc5c8ad0640 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5594152ff0b0 0x5594153014d0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce486f39b1ddf734 server_cookie=e3f9aedfaaaca716 in_seq=0 out_seq=0 2024-06-09T22:29:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.388+0000 7fc5a3fff640 10 client.17664 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:06.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.398+0000 7fc5a3fff640 1 -- 172.21.15.104:0/3172449072 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5594151eac20 con 0x5594152ff0b0 2024-06-09T22:29:06.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.398+0000 7fc5a3fff640 10 client.17664 handle_command_reply: tid=0 2024-06-09T22:29:06.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.398+0000 7fc5cda43740 1 client.17664 shutdown 2024-06-09T22:29:06.401 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cda43740 1 -- 172.21.15.104:0/3172449072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 msgr2=0x559415287e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cda43740 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 0x559415287e50 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00a4360 tx=0x7fc5b000ef90 comp rx=0 tx=0).stop 2024-06-09T22:29:06.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cda43740 1 -- 172.21.15.104:0/3172449072 shutdown_connections 2024-06-09T22:29:06.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cda43740 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5594152ff0b0 0x5594153014d0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cda43740 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559415284350 0x5594152fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cda43740 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559415279a90 0x559415287e50 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5a3fff640 0 client.17664 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:06.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cda43740 1 --2- 172.21.15.104:0/3172449072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559415279170 0x559415287910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.402 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cda43740 1 -- 172.21.15.104:0/3172449072 >> 172.21.15.104:0/3172449072 conn(0x559415275680 msgr2=0x55941528fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:06.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cda43740 1 -- 172.21.15.104:0/3172449072 shutdown_connections 2024-06-09T22:29:06.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cda43740 1 -- 172.21.15.104:0/3172449072 wait complete. 2024-06-09T22:29:06.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cda43740 20 client.17664 trim_cache size 0 max 16384 2024-06-09T22:29:06.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:{ 2024-06-09T22:29:06.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout: "return_code": 0 2024-06-09T22:29:06.403 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stdout:} 2024-06-09T22:29:06.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cad34640 1 -- 172.21.15.104:0/1981621772 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc5b40a2da0 msgr2=0x7fc5b40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.404 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cad34640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc5b40a2da0 0x7fc5b40a5260 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009b50 tx=0x7fc5b8005f90 comp rx=0 tx=0).stop 2024-06-09T22:29:06.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cad34640 1 -- 172.21.15.104:0/1981621772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c41014b0 msgr2=0x7fc5c4199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.399+0000 7fc5cad34640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c41014b0 0x7fc5c4199520 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000ea60 tx=0x7fc5b000ef20 comp rx=0 tx=0).stop 2024-06-09T22:29:06.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.400+0000 7fc5cad34640 1 -- 172.21.15.104:0/1981621772 shutdown_connections 2024-06-09T22:29:06.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.400+0000 7fc5cad34640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc5b40a2da0 0x7fc5b40a5260 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.400+0000 7fc5cad34640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc5c4106940 0x7fc5c41a4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.400+0000 7fc5cad34640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5c41014b0 0x7fc5c4199520 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.400+0000 7fc5cad34640 1 --2- 172.21.15.104:0/1981621772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c4100b10 0x7fc5c4198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.405 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.400+0000 7fc5cad34640 1 -- 172.21.15.104:0/1981621772 >> 172.21.15.104:0/1981621772 conn(0x7fc5c4077780 msgr2=0x7fc5c40ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:06.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.400+0000 7fc5cad34640 1 -- 172.21.15.104:0/1981621772 shutdown_connections 2024-06-09T22:29:06.406 INFO:teuthology.task.background_exec.ubuntu@smithi104.front.sepia.ceph.com.smithi104.stderr:2024-06-09T22:29:06.400+0000 7fc5cad34640 1 -- 172.21.15.104:0/1981621772 wait complete. 2024-06-09T22:29:06.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.550+0000 7f9ca7746700 1 -- 172.21.15.104:0/3373017241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c840058f0 con 0x7f9ca01013c0 2024-06-09T22:29:06.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.551+0000 7f9c967fc700 1 -- 172.21.15.104:0/3373017241 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7f9c9c09cc20 con 0x7f9ca01013c0 2024-06-09T22:29:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:29:06.553 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:29:06.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.553+0000 7f9ca7746700 1 -- 172.21.15.104:0/3373017241 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9c8c0a2e10 msgr2=0x7f9c8c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.553+0000 7f9ca7746700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9c8c0a2e10 0x7f9c8c0a52d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9ca019a170 tx=0x7f9c9000a380 comp rx=0 tx=0).stop 2024-06-09T22:29:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.553+0000 7f9ca7746700 1 -- 172.21.15.104:0/3373017241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca01013c0 msgr2=0x7f9ca019e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.553+0000 7f9ca7746700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca01013c0 0x7f9ca019e8f0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c000c00 tx=0x7f9c9c0047b0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.553+0000 7f9ca7746700 1 -- 172.21.15.104:0/3373017241 shutdown_connections 2024-06-09T22:29:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.553+0000 7f9ca7746700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9c8c0a2e10 0x7f9c8c0a52d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.553+0000 7f9ca7746700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca01013c0 0x7f9ca019e8f0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.553+0000 7f9ca7746700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca0107430 0x7f9ca0198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.553+0000 7f9ca7746700 1 --2- 172.21.15.104:0/3373017241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ca0101d80 0x7f9ca019ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:06.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.553+0000 7f9ca7746700 1 -- 172.21.15.104:0/3373017241 >> 172.21.15.104:0/3373017241 conn(0x7f9ca00783d0 msgr2=0x7f9ca00ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:06.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.553+0000 7f9ca7746700 1 -- 172.21.15.104:0/3373017241 shutdown_connections 2024-06-09T22:29:06.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:06.553+0000 7f9ca7746700 1 -- 172.21.15.104:0/3373017241 wait complete. 2024-06-09T22:29:06.557 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:29:06.967 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:06 smithi104 ceph-mon[32389]: osdmap e103: 12 total, 12 up, 12 in 2024-06-09T22:29:06.967 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:06 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3373017241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:29:06.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:06 smithi153 ceph-mon[36599]: osdmap e103: 12 total, 12 up, 12 in 2024-06-09T22:29:06.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:06 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3373017241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:29:07.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:06 smithi194 ceph-mon[36856]: osdmap e103: 12 total, 12 up, 12 in 2024-06-09T22:29:07.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:06 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3373017241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:29:07.403 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24460 quiesce db --cancel --all' 2024-06-09T22:29:07.403 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24460 quiesce db --cancel --all 2024-06-09T22:29:07.643 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:29:07.904 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:07 smithi104 ceph-mon[32389]: pgmap v250: 129 pgs: 129 active+clean; 255 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 266 op/s 2024-06-09T22:29:07.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:07 smithi153 ceph-mon[36599]: pgmap v250: 129 pgs: 129 active+clean; 255 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 266 op/s 2024-06-09T22:29:08.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:07 smithi194 ceph-mon[36856]: pgmap v250: 129 pgs: 129 active+clean; 255 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 266 op/s 2024-06-09T22:29:09.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.508+0000 7f821e8a7700 1 -- 172.21.15.104:0/93552318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f821806ac20 msgr2=0x7f821806b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:09.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.508+0000 7f821e8a7700 1 --2- 172.21.15.104:0/93552318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f821806ac20 0x7f821806b0a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f8200009990 tx=0x7f8200009ca0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.508+0000 7f821e8a7700 1 -- 172.21.15.104:0/93552318 shutdown_connections 2024-06-09T22:29:09.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.508+0000 7f821e8a7700 1 --2- 172.21.15.104:0/93552318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f821806b5e0 0x7f821810f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.508+0000 7f821e8a7700 1 --2- 172.21.15.104:0/93552318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8218108530 0x7f8218108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.508+0000 7f821e8a7700 1 --2- 172.21.15.104:0/93552318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f821806ac20 0x7f821806b0a0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.508+0000 7f821e8a7700 1 -- 172.21.15.104:0/93552318 >> 172.21.15.104:0/93552318 conn(0x7f82180782c0 msgr2=0x7f82180786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.508+0000 7f821e8a7700 1 -- 172.21.15.104:0/93552318 shutdown_connections 2024-06-09T22:29:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.508+0000 7f821e8a7700 1 -- 172.21.15.104:0/93552318 wait complete. 2024-06-09T22:29:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f821e8a7700 1 Processor -- start 2024-06-09T22:29:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f821e8a7700 1 -- start start 2024-06-09T22:29:09.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f821e8a7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f821806ac20 0x7f821819e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f821e8a7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f821806b5e0 0x7f821819ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f821e8a7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8218108530 0x7f8218198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f821e8a7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8218112780 con 0x7f8218108530 2024-06-09T22:29:09.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f821e8a7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8218112600 con 0x7f821806ac20 2024-06-09T22:29:09.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f821e8a7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8218112900 con 0x7f821806b5e0 2024-06-09T22:29:09.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f821ce44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8218108530 0x7f8218198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f8217fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f821806ac20 0x7f821819e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f821ce44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8218108530 0x7f8218198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58646/0 (socket says 172.21.15.104:58646) 2024-06-09T22:29:09.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f8217fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f821806ac20 0x7f821819e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:39654/0 (socket says 172.21.15.104:39654) 2024-06-09T22:29:09.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f821ce44700 1 -- 172.21.15.104:0/3878372003 learned_addr learned my addr 172.21.15.104:0/3878372003 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:09.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.509+0000 7f820ffff700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f821806b5e0 0x7f821819ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.510+0000 7f821ce44700 1 -- 172.21.15.104:0/3878372003 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f821806b5e0 msgr2=0x7f821819ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:09.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.510+0000 7f821ce44700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f821806b5e0 0x7f821819ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.510+0000 7f821ce44700 1 -- 172.21.15.104:0/3878372003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f821806ac20 msgr2=0x7f821819e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:09.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.510+0000 7f821ce44700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f821806ac20 0x7f821819e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.510+0000 7f821ce44700 1 -- 172.21.15.104:0/3878372003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82181991c0 con 0x7f8218108530 2024-06-09T22:29:09.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.510+0000 7f821ce44700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8218108530 0x7f8218198a60 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f8210007ee0 tx=0x7f821000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:09.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.510+0000 7f8215ffb700 1 -- 172.21.15.104:0/3878372003 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8210004bb0 con 0x7f8218108530 2024-06-09T22:29:09.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.510+0000 7f8215ffb700 1 -- 172.21.15.104:0/3878372003 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8210004d50 con 0x7f8218108530 2024-06-09T22:29:09.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.510+0000 7f821e8a7700 1 -- 172.21.15.104:0/3878372003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82181994b0 con 0x7f8218108530 2024-06-09T22:29:09.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.510+0000 7f8215ffb700 1 -- 172.21.15.104:0/3878372003 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8210005960 con 0x7f8218108530 2024-06-09T22:29:09.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.510+0000 7f821e8a7700 1 -- 172.21.15.104:0/3878372003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82181a5770 con 0x7f8218108530 2024-06-09T22:29:09.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.513+0000 7f8215ffb700 1 -- 172.21.15.104:0/3878372003 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f821001a020 con 0x7f8218108530 2024-06-09T22:29:09.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f8215ffb700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f82040a2d90 0x7f82040a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f8217fff700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f82040a2d90 0x7f82040a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f8215ffb700 1 -- 172.21.15.104:0/3878372003 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9584+0+0 (secure 0 0 0) 0x7f82100cefb0 con 0x7f8218108530 2024-06-09T22:29:09.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f822912ab80 1 Processor -- start 2024-06-09T22:29:09.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f8217fff700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f82040a2d90 0x7f82040a5250 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8218199ba0 tx=0x7f820800a380 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:09.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f822912ab80 1 -- start start 2024-06-09T22:29:09.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f822912ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 0x556a77a85f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f822912ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a864d0 0x556a77a7fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f822912ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a80340 0x556a77a80780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f822912ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x556a779d1880 con 0x556a77a85bb0 2024-06-09T22:29:09.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f822912ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556a77979790 con 0x556a77a80340 2024-06-09T22:29:09.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f822912ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556a779cd840 con 0x556a77a864d0 2024-06-09T22:29:09.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.514+0000 7f8217fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 0x556a77a85f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f8217fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 0x556a77a85f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58660/0 (socket says 172.21.15.104:58660) 2024-06-09T22:29:09.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f8217fff700 1 -- 172.21.15.104:0/1478830121 learned_addr learned my addr 172.21.15.104:0/1478830121 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:09.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f820ffff700 1 --2- 172.21.15.104:0/1478830121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a864d0 0x556a77a7fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f821ce44700 1 --2- 172.21.15.104:0/1478830121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a80340 0x556a77a80780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f8217fff700 1 -- 172.21.15.104:0/1478830121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a864d0 msgr2=0x556a77a7fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:09.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f8217fff700 1 --2- 172.21.15.104:0/1478830121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a864d0 0x556a77a7fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f8217fff700 1 -- 172.21.15.104:0/1478830121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a80340 msgr2=0x556a77a80780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:09.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f8217fff700 1 --2- 172.21.15.104:0/1478830121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a80340 0x556a77a80780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f8217fff700 1 -- 172.21.15.104:0/1478830121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a7796ffc0 con 0x556a77a85bb0 2024-06-09T22:29:09.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f821ce44700 1 --2- 172.21.15.104:0/1478830121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a80340 0x556a77a80780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:09.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f8217fff700 1 --2- 172.21.15.104:0/1478830121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 0x556a77a85f90 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f8208004530 tx=0x7f8208012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b661e0e6d07f0a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:09.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f820e7fc700 1 -- 172.21.15.104:0/1478830121 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8208016070 con 0x556a77a85bb0 2024-06-09T22:29:09.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f820e7fc700 1 -- 172.21.15.104:0/1478830121 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8208014070 con 0x556a77a85bb0 2024-06-09T22:29:09.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.515+0000 7f820e7fc700 1 -- 172.21.15.104:0/1478830121 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8208010e10 con 0x556a77a85bb0 2024-06-09T22:29:09.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.516+0000 7f822912ab80 1 -- 172.21.15.104:0/1478830121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 msgr2=0x556a77a85f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:09.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.516+0000 7f822912ab80 1 --2- 172.21.15.104:0/1478830121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 0x556a77a85f90 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f8208004530 tx=0x7f8208012000 comp rx=0 tx=0).stop 2024-06-09T22:29:09.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.516+0000 7f822912ab80 1 -- 172.21.15.104:0/1478830121 shutdown_connections 2024-06-09T22:29:09.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.516+0000 7f822912ab80 1 --2- 172.21.15.104:0/1478830121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 0x556a77a85f90 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.516+0000 7f822912ab80 1 --2- 172.21.15.104:0/1478830121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a80340 0x556a77a80780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.516+0000 7f822912ab80 1 --2- 172.21.15.104:0/1478830121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a864d0 0x556a77a7fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.516+0000 7f822912ab80 1 -- 172.21.15.104:0/1478830121 >> 172.21.15.104:0/1478830121 conn(0x556a779aad30 msgr2=0x556a77a7b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:09.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.516+0000 7f822912ab80 1 -- 172.21.15.104:0/1478830121 shutdown_connections 2024-06-09T22:29:09.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.516+0000 7f822912ab80 1 -- 172.21.15.104:0/1478830121 wait complete. 2024-06-09T22:29:09.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.516+0000 7f822912ab80 1 -- start start 2024-06-09T22:29:09.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f822912ab80 1 Processor -- start 2024-06-09T22:29:09.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f822912ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a80340 0x556a77a8c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f822912ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 0x556a77a894d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f820ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 0x556a77a894d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f822912ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a864d0 0x556a77a89a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f820ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 0x556a77a894d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58668/0 (socket says 172.21.15.104:58668) 2024-06-09T22:29:09.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f8217fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a80340 0x556a77a8c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f820ffff700 1 -- 172.21.15.104:0/1012432173 learned_addr learned my addr 172.21.15.104:0/1012432173 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:09.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f822912ab80 1 -- 172.21.15.104:0/1012432173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x556a77a8cf40 con 0x556a77a85bb0 2024-06-09T22:29:09.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f822912ab80 1 -- 172.21.15.104:0/1012432173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556a77971c60 con 0x556a77a864d0 2024-06-09T22:29:09.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f822912ab80 1 -- 172.21.15.104:0/1012432173 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556a77a8d0c0 con 0x556a77a80340 2024-06-09T22:29:09.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f822912ab80 10 client.0 _ll_register_callbacks cb 0x556a7796b990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:09.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f821ce44700 1 --2- 172.21.15.104:0/1012432173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a864d0 0x556a77a89a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f820ffff700 1 -- 172.21.15.104:0/1012432173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a80340 msgr2=0x556a77a8c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:09.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f820ffff700 1 --2- 172.21.15.104:0/1012432173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a80340 0x556a77a8c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f820ffff700 1 -- 172.21.15.104:0/1012432173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a864d0 msgr2=0x556a77a89a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:09.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f820ffff700 1 --2- 172.21.15.104:0/1012432173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a864d0 0x556a77a89a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.517+0000 7f820ffff700 1 -- 172.21.15.104:0/1012432173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a77920940 con 0x556a77a85bb0 2024-06-09T22:29:09.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f820ffff700 1 --2- 172.21.15.104:0/1012432173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 0x556a77a894d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f8200000c00 tx=0x7f820000bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8dd7b1226038a33 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:09.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f81f77fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:09.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f81f77fe700 1 -- 172.21.15.104:0/1012432173 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200029060 con 0x556a77a85bb0 2024-06-09T22:29:09.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f81f77fe700 1 -- 172.21.15.104:0/1012432173 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9584+0+0 (secure 0 0 0) 0x7f8200003b60 con 0x556a77a85bb0 2024-06-09T22:29:09.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f822912ab80 1 -- 172.21.15.104:0/1012432173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8208014070 con 0x556a77a85bb0 2024-06-09T22:29:09.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f81f77fe700 1 -- 172.21.15.104:0/1012432173 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8200030050 con 0x556a77a85bb0 2024-06-09T22:29:09.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f822912ab80 10 client.17676 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:09.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f822912ab80 1 -- 172.21.15.104:0/1012432173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556a77a977b0 con 0x556a77a85bb0 2024-06-09T22:29:09.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f81f77fe700 1 -- 172.21.15.104:0/1012432173 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8208014070 con 0x556a77a85bb0 2024-06-09T22:29:09.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f81f77fe700 1 -- 172.21.15.104:0/1012432173 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f820000f790 con 0x556a77a85bb0 2024-06-09T22:29:09.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f822912ab80 10 client.17676 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:09.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f822912ab80 10 client.17676 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:29:09.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f822912ab80 20 client.17676 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:09.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f822912ab80 1 --2- 172.21.15.104:0/1012432173 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a77a99df0 0x556a77a9c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f822912ab80 4 client.17676 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:29:09.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.518+0000 7f822912ab80 1 -- 172.21.15.104:0/1012432173 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a779ded80 con 0x556a77a99df0 2024-06-09T22:29:09.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.519+0000 7f8217fff700 1 --2- 172.21.15.104:0/1012432173 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a77a99df0 0x556a77a9c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.519+0000 7f81f77fe700 10 client.17676 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:09.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.519+0000 7f8217fff700 1 --2- 172.21.15.104:0/1012432173 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a77a99df0 0x556a77a9c1e0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ac45c8e38211065 server_cookie=521c821952d202b2 in_seq=0 out_seq=0 2024-06-09T22:29:09.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f81f77fe700 1 -- 172.21.15.104:0/1012432173 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x556a779ded80 con 0x556a77a99df0 2024-06-09T22:29:09.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f81f77fe700 10 client.17676 handle_command_reply: tid=0 2024-06-09T22:29:09.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f822912ab80 1 client.17676 shutdown 2024-06-09T22:29:09.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f822912ab80 1 -- 172.21.15.104:0/1012432173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 msgr2=0x556a77a894d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:09.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f822912ab80 1 --2- 172.21.15.104:0/1012432173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 0x556a77a894d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f8200000c00 tx=0x7f820000bfb0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f822912ab80 1 -- 172.21.15.104:0/1012432173 shutdown_connections 2024-06-09T22:29:09.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f822912ab80 1 --2- 172.21.15.104:0/1012432173 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a77a99df0 0x556a77a9c1e0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f822912ab80 1 --2- 172.21.15.104:0/1012432173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a864d0 0x556a77a89a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f822912ab80 1 --2- 172.21.15.104:0/1012432173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a80340 0x556a77a8c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f81f77fe700 0 client.17676 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:09.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f822912ab80 1 --2- 172.21.15.104:0/1012432173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a85bb0 0x556a77a894d0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f822912ab80 1 -- 172.21.15.104:0/1012432173 >> 172.21.15.104:0/1012432173 conn(0x556a779aad30 msgr2=0x556a77a7ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:09.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f822912ab80 1 -- 172.21.15.104:0/1012432173 shutdown_connections 2024-06-09T22:29:09.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f822912ab80 1 -- 172.21.15.104:0/1012432173 wait complete. 2024-06-09T22:29:09.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.520+0000 7f822912ab80 20 client.17676 trim_cache size 0 max 16384 2024-06-09T22:29:09.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f822912ab80 1 Processor -- start 2024-06-09T22:29:09.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f822912ab80 1 -- start start 2024-06-09T22:29:09.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f822912ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 0x556a77a7fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f822912ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a85bb0 0x556a77a89b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f822912ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a864d0 0x556a77a86fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f822912ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f82181ae690 con 0x556a77a80340 2024-06-09T22:29:09.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f822912ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82181b01d0 con 0x556a77a864d0 2024-06-09T22:29:09.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f822912ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f82181aed60 con 0x556a77a85bb0 2024-06-09T22:29:09.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f8217fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 0x556a77a7fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f8217fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 0x556a77a7fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58682/0 (socket says 172.21.15.104:58682) 2024-06-09T22:29:09.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f8217fff700 1 -- 172.21.15.104:0/4179438663 learned_addr learned my addr 172.21.15.104:0/4179438663 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:09.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f820ffff700 1 --2- 172.21.15.104:0/4179438663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a85bb0 0x556a77a89b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f821ce44700 1 --2- 172.21.15.104:0/4179438663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a864d0 0x556a77a86fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f8217fff700 1 -- 172.21.15.104:0/4179438663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a85bb0 msgr2=0x556a77a89b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:09.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f8217fff700 1 --2- 172.21.15.104:0/4179438663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a85bb0 0x556a77a89b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f8217fff700 1 -- 172.21.15.104:0/4179438663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a864d0 msgr2=0x556a77a86fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:09.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f8217fff700 1 --2- 172.21.15.104:0/4179438663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a864d0 0x556a77a86fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.540+0000 7f8217fff700 1 -- 172.21.15.104:0/4179438663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a77a75790 con 0x556a77a80340 2024-06-09T22:29:09.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.541+0000 7f8217fff700 1 --2- 172.21.15.104:0/4179438663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 0x556a77a7fe00 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f820800a4a0 tx=0x7f820800a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea2a988b7c9a1df server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:09.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.541+0000 7f820dffb700 1 -- 172.21.15.104:0/4179438663 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8208036070 con 0x556a77a80340 2024-06-09T22:29:09.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.541+0000 7f820dffb700 1 -- 172.21.15.104:0/4179438663 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8208014050 con 0x556a77a80340 2024-06-09T22:29:09.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.541+0000 7f820dffb700 1 -- 172.21.15.104:0/4179438663 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8208016040 con 0x556a77a80340 2024-06-09T22:29:09.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.541+0000 7f822912ab80 1 -- 172.21.15.104:0/4179438663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 msgr2=0x556a77a7fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:09.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.541+0000 7f822912ab80 1 --2- 172.21.15.104:0/4179438663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 0x556a77a7fe00 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f820800a4a0 tx=0x7f820800a410 comp rx=0 tx=0).stop 2024-06-09T22:29:09.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.541+0000 7f822912ab80 1 -- 172.21.15.104:0/4179438663 shutdown_connections 2024-06-09T22:29:09.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.541+0000 7f822912ab80 1 --2- 172.21.15.104:0/4179438663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 0x556a77a7fe00 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.541+0000 7f822912ab80 1 --2- 172.21.15.104:0/4179438663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a864d0 0x556a77a86fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.541+0000 7f822912ab80 1 --2- 172.21.15.104:0/4179438663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a85bb0 0x556a77a89b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.541+0000 7f822912ab80 1 -- 172.21.15.104:0/4179438663 >> 172.21.15.104:0/4179438663 conn(0x556a779aad30 msgr2=0x556a77a868b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:09.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.542+0000 7f822912ab80 1 -- 172.21.15.104:0/4179438663 shutdown_connections 2024-06-09T22:29:09.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.542+0000 7f822912ab80 1 -- 172.21.15.104:0/4179438663 wait complete. 2024-06-09T22:29:09.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.542+0000 7f822912ab80 1 -- start start 2024-06-09T22:29:09.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.542+0000 7f822912ab80 1 Processor -- start 2024-06-09T22:29:09.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.542+0000 7f822912ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 0x556a77a925d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.542+0000 7f822912ab80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a85bb0 0x556a77a92b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.542+0000 7f822912ab80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a864d0 0x556a77a8f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.542+0000 7f822912ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f82181afb00 con 0x556a77a80340 2024-06-09T22:29:09.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.542+0000 7f822912ab80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82181af430 con 0x556a77a85bb0 2024-06-09T22:29:09.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.542+0000 7f822912ab80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f82181ac630 con 0x556a77a864d0 2024-06-09T22:29:09.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f822912ab80 10 client.0 _ll_register_callbacks cb 0x556a7796b990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:09.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f8217fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 0x556a77a925d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f8217fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 0x556a77a925d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58694/0 (socket says 172.21.15.104:58694) 2024-06-09T22:29:09.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f8217fff700 1 -- 172.21.15.104:0/3281510948 learned_addr learned my addr 172.21.15.104:0/3281510948 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:09.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f820ffff700 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a85bb0 0x556a77a92b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f821ce44700 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a864d0 0x556a77a8f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f8217fff700 1 -- 172.21.15.104:0/3281510948 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a864d0 msgr2=0x556a77a8f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:09.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f8217fff700 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a864d0 0x556a77a8f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f8217fff700 1 -- 172.21.15.104:0/3281510948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a85bb0 msgr2=0x556a77a92b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:09.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f8217fff700 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a85bb0 0x556a77a92b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f8217fff700 1 -- 172.21.15.104:0/3281510948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a77a8fb90 con 0x556a77a80340 2024-06-09T22:29:09.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f8217fff700 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 0x556a77a925d0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f820800aa70 tx=0x7f8208005ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f60847335ae2832 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:09.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f81f77fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:09.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f81f77fe700 1 -- 172.21.15.104:0/3281510948 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820803f060 con 0x556a77a80340 2024-06-09T22:29:09.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f81f77fe700 1 -- 172.21.15.104:0/3281510948 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9584+0+0 (secure 0 0 0) 0x7f8208005040 con 0x556a77a80340 2024-06-09T22:29:09.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f822912ab80 1 -- 172.21.15.104:0/3281510948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8208014050 con 0x556a77a80340 2024-06-09T22:29:09.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f81f77fe700 1 -- 172.21.15.104:0/3281510948 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8208046050 con 0x556a77a80340 2024-06-09T22:29:09.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f822912ab80 10 client.17682 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:09.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.543+0000 7f822912ab80 1 -- 172.21.15.104:0/3281510948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556a77b055d0 con 0x556a77a80340 2024-06-09T22:29:09.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.544+0000 7f81f77fe700 1 -- 172.21.15.104:0/3281510948 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8208014050 con 0x556a77a80340 2024-06-09T22:29:09.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.544+0000 7f81f77fe700 1 -- 172.21.15.104:0/3281510948 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f82080045b0 con 0x556a77a80340 2024-06-09T22:29:09.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.544+0000 7f822912ab80 10 client.17682 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:09.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.544+0000 7f822912ab80 10 client.17682 resolve_mds: validated gid 24460 aka daemon mds.i 2024-06-09T22:29:09.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.544+0000 7f822912ab80 20 client.17682 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:09.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.544+0000 7f822912ab80 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a77a99df0 0x556a77b07b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:09.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.544+0000 7f822912ab80 4 client.17682 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-06-09T22:29:09.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.544+0000 7f822912ab80 1 -- 172.21.15.104:0/3281510948 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x556a779d9300 con 0x556a77a99df0 2024-06-09T22:29:09.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.544+0000 7f820ffff700 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a77a99df0 0x556a77b07b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:09.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.544+0000 7f820ffff700 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a77a99df0 0x556a77b07b10 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af6418aaeb916387 server_cookie=1148788794a0810c in_seq=0 out_seq=0 2024-06-09T22:29:09.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.544+0000 7f81f77fe700 10 client.17682 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:09.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f81f77fe700 1 -- 172.21.15.104:0/3281510948 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x556a779d9300 con 0x556a77a99df0 2024-06-09T22:29:09.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f81f77fe700 10 client.17682 handle_command_reply: tid=0 2024-06-09T22:29:09.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f822912ab80 1 client.17682 shutdown 2024-06-09T22:29:09.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f822912ab80 1 -- 172.21.15.104:0/3281510948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 msgr2=0x556a77a925d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:09.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f822912ab80 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 0x556a77a925d0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f820800aa70 tx=0x7f8208005ba0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f822912ab80 1 -- 172.21.15.104:0/3281510948 shutdown_connections 2024-06-09T22:29:09.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f822912ab80 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x556a77a99df0 0x556a77b07b10 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f822912ab80 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556a77a80340 0x556a77a925d0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f822912ab80 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a77a864d0 0x556a77a8f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f81f77fe700 0 client.17682 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:09.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f822912ab80 1 --2- 172.21.15.104:0/3281510948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a77a85bb0 0x556a77a92b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f822912ab80 1 -- 172.21.15.104:0/3281510948 >> 172.21.15.104:0/3281510948 conn(0x556a779aad30 msgr2=0x556a77a7a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:09.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f822912ab80 1 -- 172.21.15.104:0/3281510948 shutdown_connections 2024-06-09T22:29:09.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f822912ab80 1 -- 172.21.15.104:0/3281510948 wait complete. 2024-06-09T22:29:09.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.545+0000 7f822912ab80 20 client.17682 trim_cache size 0 max 16384 2024-06-09T22:29:09.558 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:29:09.558 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 32, 2024-06-09T22:29:09.558 INFO:teuthology.orchestra.run.smithi104.stdout: "leader": 24460, 2024-06-09T22:29:09.558 INFO:teuthology.orchestra.run.smithi104.stdout: "set_version": 20, 2024-06-09T22:29:09.558 INFO:teuthology.orchestra.run.smithi104.stdout: "sets": {} 2024-06-09T22:29:09.559 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:29:09.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.546+0000 7f821e8a7700 1 -- 172.21.15.104:0/3878372003 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f82040a2d90 msgr2=0x7f82040a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:09.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.546+0000 7f821e8a7700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f82040a2d90 0x7f82040a5250 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8218199ba0 tx=0x7f820800a380 comp rx=0 tx=0).stop 2024-06-09T22:29:09.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.546+0000 7f821e8a7700 1 -- 172.21.15.104:0/3878372003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8218108530 msgr2=0x7f8218198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:09.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.546+0000 7f821e8a7700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8218108530 0x7f8218198a60 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f8210007ee0 tx=0x7f821000d3b0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.546+0000 7f821e8a7700 1 -- 172.21.15.104:0/3878372003 shutdown_connections 2024-06-09T22:29:09.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.546+0000 7f821e8a7700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f82040a2d90 0x7f82040a5250 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.546+0000 7f821e8a7700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8218108530 0x7f8218198a60 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.546+0000 7f821e8a7700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f821806ac20 0x7f821819e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.546+0000 7f821e8a7700 1 --2- 172.21.15.104:0/3878372003 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f821806b5e0 0x7f821819ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:09.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.546+0000 7f821e8a7700 1 -- 172.21.15.104:0/3878372003 >> 172.21.15.104:0/3878372003 conn(0x7f82180782c0 msgr2=0x7f82180fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:09.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.546+0000 7f821e8a7700 1 -- 172.21.15.104:0/3878372003 shutdown_connections 2024-06-09T22:29:09.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:09.546+0000 7f821e8a7700 1 -- 172.21.15.104:0/3878372003 wait complete. 2024-06-09T22:29:09.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:09 smithi153 ceph-mon[36599]: pgmap v251: 129 pgs: 129 active+clean; 255 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 266 op/s 2024-06-09T22:29:10.008 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:09 smithi104 ceph-mon[32389]: pgmap v251: 129 pgs: 129 active+clean; 255 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 266 op/s 2024-06-09T22:29:10.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:09 smithi194 ceph-mon[36856]: pgmap v251: 129 pgs: 129 active+clean; 255 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 266 op/s 2024-06-09T22:29:10.834 INFO:tasks.quiescer:done joining Quiescers 2024-06-09T22:29:10.834 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-06-09T22:29:10.846 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi104.front.sepia.ceph.com 2024-06-09T22:29:10.855 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-06-09T22:29:10.867 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-06-09T22:29:10.867 INFO:teuthology.task.exec:Executing custom commands... 2024-06-09T22:29:10.868 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi104.front.sepia.ceph.com 2024-06-09T22:29:10.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-06-09T22:29:10.927 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-06-09T22:29:10.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:10 smithi153 ceph-mon[36599]: pgmap v252: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 185 op/s 2024-06-09T22:29:11.000 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:10 smithi104 ceph-mon[32389]: pgmap v252: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 185 op/s 2024-06-09T22:29:11.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:10 smithi194 ceph-mon[36856]: pgmap v252: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 185 op/s 2024-06-09T22:29:13.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:13 smithi194 ceph-mon[36856]: pgmap v253: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 185 op/s 2024-06-09T22:29:13.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:13 smithi194 ceph-mon[36856]: osdmap e104: 12 total, 12 up, 12 in 2024-06-09T22:29:13.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:13 smithi153 ceph-mon[36599]: pgmap v253: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 185 op/s 2024-06-09T22:29:13.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:13 smithi153 ceph-mon[36599]: osdmap e104: 12 total, 12 up, 12 in 2024-06-09T22:29:14.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:13 smithi104 ceph-mon[32389]: pgmap v253: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 185 op/s 2024-06-09T22:29:14.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:13 smithi104 ceph-mon[32389]: osdmap e104: 12 total, 12 up, 12 in 2024-06-09T22:29:14.743 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:29:15.006 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:29:15.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:15 smithi104 ceph-mon[32389]: pgmap v255: 129 pgs: 129 active+clean; 252 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-06-09T22:29:15.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:15 smithi194 ceph-mon[36856]: pgmap v255: 129 pgs: 129 active+clean; 252 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-06-09T22:29:15.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:15 smithi153 ceph-mon[36599]: pgmap v255: 129 pgs: 129 active+clean; 252 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-06-09T22:29:17.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.149+0000 7fd1e1935700 1 -- 172.21.15.104:0/2233629081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1dc108500 msgr2=0x7fd1dc1088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:17.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.149+0000 7fd1e1935700 1 --2- 172.21.15.104:0/2233629081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1dc108500 0x7fd1dc1088e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009990 tx=0x7fd1cc009ca0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.149+0000 7fd1e1935700 1 -- 172.21.15.104:0/2233629081 shutdown_connections 2024-06-09T22:29:17.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.149+0000 7fd1e1935700 1 --2- 172.21.15.104:0/2233629081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1dc108500 0x7fd1dc1088e0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.149+0000 7fd1e1935700 1 --2- 172.21.15.104:0/2233629081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1dc06ab30 0x7fd1dc10f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.149+0000 7fd1e1935700 1 --2- 172.21.15.104:0/2233629081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1dc06a170 0x7fd1dc06a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.149+0000 7fd1e1935700 1 -- 172.21.15.104:0/2233629081 >> 172.21.15.104:0/2233629081 conn(0x7fd1dc078340 msgr2=0x7fd1dc078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:17.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.149+0000 7fd1e1935700 1 -- 172.21.15.104:0/2233629081 shutdown_connections 2024-06-09T22:29:17.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.149+0000 7fd1e1935700 1 -- 172.21.15.104:0/2233629081 wait complete. 2024-06-09T22:29:17.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.149+0000 7fd1e1935700 1 Processor -- start 2024-06-09T22:29:17.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.149+0000 7fd1e1935700 1 -- start start 2024-06-09T22:29:17.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1e1935700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1dc06a170 0x7fd1dc19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1e1935700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1dc06ab30 0x7fd1dc19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1e1935700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1dc108500 0x7fd1dc198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1e1935700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc112830 con 0x7fd1dc108500 2024-06-09T22:29:17.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1e1935700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1126b0 con 0x7fd1dc06ab30 2024-06-09T22:29:17.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1e1935700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1129b0 con 0x7fd1dc06a170 2024-06-09T22:29:17.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1dc108500 0x7fd1dc198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1dc108500 0x7fd1dc198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38178/0 (socket says 172.21.15.104:38178) 2024-06-09T22:29:17.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1db7fe700 1 -- 172.21.15.104:0/1937009480 learned_addr learned my addr 172.21.15.104:0/1937009480 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:17.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1daffd700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1dc06a170 0x7fd1dc19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1d27fc700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1dc06ab30 0x7fd1dc19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1db7fe700 1 -- 172.21.15.104:0/1937009480 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1dc06a170 msgr2=0x7fd1dc19e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:17.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1db7fe700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1dc06a170 0x7fd1dc19e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1db7fe700 1 -- 172.21.15.104:0/1937009480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1dc06ab30 msgr2=0x7fd1dc19ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:17.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1db7fe700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1dc06ab30 0x7fd1dc19ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.150+0000 7fd1db7fe700 1 -- 172.21.15.104:0/1937009480 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1dc199250 con 0x7fd1dc108500 2024-06-09T22:29:17.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.151+0000 7fd1db7fe700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1dc108500 0x7fd1dc198af0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4007ee0 tx=0x7fd1d400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:17.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.151+0000 7fd1e0933700 1 -- 172.21.15.104:0/1937009480 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4004bb0 con 0x7fd1dc108500 2024-06-09T22:29:17.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.151+0000 7fd1e0933700 1 -- 172.21.15.104:0/1937009480 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1d4004d50 con 0x7fd1dc108500 2024-06-09T22:29:17.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.151+0000 7fd1e1935700 1 -- 172.21.15.104:0/1937009480 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1dc199540 con 0x7fd1dc108500 2024-06-09T22:29:17.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.151+0000 7fd1e0933700 1 -- 172.21.15.104:0/1937009480 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4005960 con 0x7fd1dc108500 2024-06-09T22:29:17.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.151+0000 7fd1e1935700 1 -- 172.21.15.104:0/1937009480 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1dc1a5800 con 0x7fd1dc108500 2024-06-09T22:29:17.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.153+0000 7fd1e0933700 1 -- 172.21.15.104:0/1937009480 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fd1d401a020 con 0x7fd1dc108500 2024-06-09T22:29:17.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.154+0000 7fd1e0933700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd1c80a2e10 0x7fd1c80a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.154+0000 7fd1daffd700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd1c80a2e10 0x7fd1c80a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.154+0000 7fd1e0933700 1 -- 172.21.15.104:0/1937009480 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fd1d40cec90 con 0x7fd1dc108500 2024-06-09T22:29:17.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.154+0000 7fd1daffd700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd1c80a2e10 0x7fd1c80a52d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00b5c0 tx=0x7fd1cc005fd0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:17.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1ec1b8b80 1 Processor -- start 2024-06-09T22:29:17.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1ec1b8b80 1 -- start start 2024-06-09T22:29:17.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1ec1b8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f25181f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1ec1b8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25181f880 0x55f251819260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1ec1b8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2518197a0 0x55f251819bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1ec1b8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55f25170bf60 con 0x55f25181ef60 2024-06-09T22:29:17.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1ec1b8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f25176a850 con 0x55f2518197a0 2024-06-09T22:29:17.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1ec1b8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f2516d2460 con 0x55f25181f880 2024-06-09T22:29:17.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1daffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f25181f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1daffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f25181f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38184/0 (socket says 172.21.15.104:38184) 2024-06-09T22:29:17.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1daffd700 1 -- 172.21.15.104:0/1157366364 learned_addr learned my addr 172.21.15.104:0/1157366364 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:17.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1db7fe700 1 --2- 172.21.15.104:0/1157366364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2518197a0 0x55f251819bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1d27fc700 1 --2- 172.21.15.104:0/1157366364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25181f880 0x55f251819260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1daffd700 1 -- 172.21.15.104:0/1157366364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25181f880 msgr2=0x55f251819260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:17.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1daffd700 1 --2- 172.21.15.104:0/1157366364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25181f880 0x55f251819260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1daffd700 1 -- 172.21.15.104:0/1157366364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2518197a0 msgr2=0x55f251819bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:17.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1daffd700 1 --2- 172.21.15.104:0/1157366364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2518197a0 0x55f251819bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.155+0000 7fd1daffd700 1 -- 172.21.15.104:0/1157366364 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2516b98d0 con 0x55f25181ef60 2024-06-09T22:29:17.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.156+0000 7fd1daffd700 1 --2- 172.21.15.104:0/1157366364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f25181f340 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00ba90 tx=0x7fd1cc020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365d29b163476f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:17.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.156+0000 7fd1d1ffb700 1 -- 172.21.15.104:0/1157366364 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc00bda0 con 0x55f25181ef60 2024-06-09T22:29:17.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.156+0000 7fd1d1ffb700 1 -- 172.21.15.104:0/1157366364 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1cc00fd50 con 0x55f25181ef60 2024-06-09T22:29:17.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.156+0000 7fd1d1ffb700 1 -- 172.21.15.104:0/1157366364 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc024d10 con 0x55f25181ef60 2024-06-09T22:29:17.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.156+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/1157366364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 msgr2=0x55f25181f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:17.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.156+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/1157366364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f25181f340 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00ba90 tx=0x7fd1cc020000 comp rx=0 tx=0).stop 2024-06-09T22:29:17.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.157+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/1157366364 shutdown_connections 2024-06-09T22:29:17.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.157+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/1157366364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f25181f340 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.157+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/1157366364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2518197a0 0x55f251819bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.157+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/1157366364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25181f880 0x55f251819260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.157+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/1157366364 >> 172.21.15.104:0/1157366364 conn(0x55f251746080 msgr2=0x55f251814220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:17.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.157+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/1157366364 shutdown_connections 2024-06-09T22:29:17.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.157+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/1157366364 wait complete. 2024-06-09T22:29:17.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.157+0000 7fd1ec1b8b80 1 -- start start 2024-06-09T22:29:17.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.157+0000 7fd1ec1b8b80 1 Processor -- start 2024-06-09T22:29:17.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1ec1b8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f2518197a0 0x55f251825580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1ec1b8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f251822880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1ec1b8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25181f880 0x55f251822dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1ec1b8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55f25170afc0 con 0x55f25181ef60 2024-06-09T22:29:17.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1ec1b8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f251766810 con 0x55f25181f880 2024-06-09T22:29:17.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1ec1b8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f2518261a0 con 0x55f2518197a0 2024-06-09T22:29:17.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1ec1b8b80 10 client.0 _ll_register_callbacks cb 0x55f2517005b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:17.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1d27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f251822880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1d27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f251822880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38198/0 (socket says 172.21.15.104:38198) 2024-06-09T22:29:17.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1d27fc700 1 -- 172.21.15.104:0/3939090971 learned_addr learned my addr 172.21.15.104:0/3939090971 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:17.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1daffd700 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f2518197a0 0x55f251825580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1db7fe700 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25181f880 0x55f251822dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1d27fc700 1 -- 172.21.15.104:0/3939090971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f2518197a0 msgr2=0x55f251825580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:17.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1d27fc700 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f2518197a0 0x55f251825580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1d27fc700 1 -- 172.21.15.104:0/3939090971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25181f880 msgr2=0x55f251822dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:17.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1d27fc700 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25181f880 0x55f251822dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1d27fc700 1 -- 172.21.15.104:0/3939090971 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f251730750 con 0x55f25181ef60 2024-06-09T22:29:17.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1d27fc700 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f251822880 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4011260 tx=0x7fd1c4011570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9b24ec7ebe61e4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:17.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1baffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:17.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1baffd700 1 -- 172.21.15.104:0/3939090971 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4019070 con 0x55f25181ef60 2024-06-09T22:29:17.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/3939090971 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1cc00fd50 con 0x55f25181ef60 2024-06-09T22:29:17.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.158+0000 7fd1baffd700 1 -- 172.21.15.104:0/3939090971 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fd1c4015460 con 0x55f25181ef60 2024-06-09T22:29:17.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.159+0000 7fd1baffd700 1 -- 172.21.15.104:0/3939090971 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd1c4025050 con 0x55f25181ef60 2024-06-09T22:29:17.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.159+0000 7fd1ec1b8b80 10 client.17694 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:17.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.159+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/3939090971 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f251830990 con 0x55f25181ef60 2024-06-09T22:29:17.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.159+0000 7fd1baffd700 1 -- 172.21.15.104:0/3939090971 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1cc00fd50 con 0x55f25181ef60 2024-06-09T22:29:17.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.159+0000 7fd1baffd700 1 -- 172.21.15.104:0/3939090971 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd1c4014850 con 0x55f25181ef60 2024-06-09T22:29:17.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.159+0000 7fd1ec1b8b80 10 client.17694 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:17.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.159+0000 7fd1ec1b8b80 10 client.17694 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:29:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.159+0000 7fd1ec1b8b80 20 client.17694 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.159+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55f251832f70 0x55f251835390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.159+0000 7fd1ec1b8b80 4 client.17694 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:29:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.159+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/3939090971 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f251777d50 con 0x55f251832f70 2024-06-09T22:29:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.159+0000 7fd1daffd700 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55f251832f70 0x55f251835390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.160+0000 7fd1baffd700 10 client.17694 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.160+0000 7fd1daffd700 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55f251832f70 0x55f251835390 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f48691df8bb1b30 server_cookie=1033a39d1ba6f4a7 in_seq=0 out_seq=0 2024-06-09T22:29:17.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.160+0000 7fd1baffd700 1 -- 172.21.15.104:0/3939090971 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55f251777d50 con 0x55f251832f70 2024-06-09T22:29:17.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.160+0000 7fd1baffd700 10 client.17694 handle_command_reply: tid=0 2024-06-09T22:29:17.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.160+0000 7fd1ec1b8b80 1 client.17694 shutdown 2024-06-09T22:29:17.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.161+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/3939090971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 msgr2=0x55f251822880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.161+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f251822880 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4011260 tx=0x7fd1c4011570 comp rx=0 tx=0).stop 2024-06-09T22:29:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.161+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/3939090971 shutdown_connections 2024-06-09T22:29:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.161+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55f251832f70 0x55f251835390 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.161+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25181f880 0x55f251822dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.161+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f2518197a0 0x55f251825580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.161+0000 7fd1baffd700 0 client.17694 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.161+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/3939090971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f251822880 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.161+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/3939090971 >> 172.21.15.104:0/3939090971 conn(0x55f251746080 msgr2=0x55f251813700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.161+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/3939090971 shutdown_connections 2024-06-09T22:29:17.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.161+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/3939090971 wait complete. 2024-06-09T22:29:17.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.161+0000 7fd1ec1b8b80 20 client.17694 trim_cache size 0 max 16384 2024-06-09T22:29:17.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.181+0000 7fd1ec1b8b80 1 Processor -- start 2024-06-09T22:29:17.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.181+0000 7fd1ec1b8b80 1 -- start start 2024-06-09T22:29:17.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.181+0000 7fd1ec1b8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f2518197a0 0x55f2518247c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.181+0000 7fd1ec1b8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25181ef60 0x55f25182b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.181+0000 7fd1ec1b8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181f880 0x55f25182bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.181+0000 7fd1ec1b8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1ae610 con 0x55f25181f880 2024-06-09T22:29:17.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.181+0000 7fd1ec1b8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1b0150 con 0x55f25181ef60 2024-06-09T22:29:17.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.181+0000 7fd1ec1b8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1aece0 con 0x55f2518197a0 2024-06-09T22:29:17.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.181+0000 7fd1db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181f880 0x55f25182bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.181+0000 7fd1db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181f880 0x55f25182bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38202/0 (socket says 172.21.15.104:38202) 2024-06-09T22:29:17.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.181+0000 7fd1db7fe700 1 -- 172.21.15.104:0/66969978 learned_addr learned my addr 172.21.15.104:0/66969978 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:17.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.181+0000 7fd1daffd700 1 --2- 172.21.15.104:0/66969978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f2518197a0 0x55f2518247c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.182+0000 7fd1db7fe700 1 -- 172.21.15.104:0/66969978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f2518197a0 msgr2=0x55f2518247c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:17.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.182+0000 7fd1db7fe700 1 --2- 172.21.15.104:0/66969978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f2518197a0 0x55f2518247c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.182+0000 7fd1d27fc700 1 --2- 172.21.15.104:0/66969978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25181ef60 0x55f25182b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.182+0000 7fd1db7fe700 1 -- 172.21.15.104:0/66969978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25181ef60 msgr2=0x55f25182b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:17.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.182+0000 7fd1db7fe700 1 --2- 172.21.15.104:0/66969978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25181ef60 0x55f25182b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.182+0000 7fd1db7fe700 1 -- 172.21.15.104:0/66969978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f251820860 con 0x55f25181f880 2024-06-09T22:29:17.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.182+0000 7fd1db7fe700 1 --2- 172.21.15.104:0/66969978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181f880 0x55f25182bdb0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4097240 tx=0x7fd1d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703e581e591ecfb2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:17.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.182+0000 7fd1d17fa700 1 -- 172.21.15.104:0/66969978 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d409ade0 con 0x55f25181f880 2024-06-09T22:29:17.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.182+0000 7fd1d17fa700 1 -- 172.21.15.104:0/66969978 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1d409f190 con 0x55f25181f880 2024-06-09T22:29:17.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.182+0000 7fd1d17fa700 1 -- 172.21.15.104:0/66969978 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d409b040 con 0x55f25181f880 2024-06-09T22:29:17.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.183+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/66969978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181f880 msgr2=0x55f25182bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:17.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.183+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/66969978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181f880 0x55f25182bdb0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4097240 tx=0x7fd1d400b040 comp rx=0 tx=0).stop 2024-06-09T22:29:17.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.183+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/66969978 shutdown_connections 2024-06-09T22:29:17.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.183+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/66969978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181f880 0x55f25182bdb0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.183+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/66969978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f2518197a0 0x55f2518247c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.183+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/66969978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f25181ef60 0x55f25182b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.183+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/66969978 >> 172.21.15.104:0/66969978 conn(0x55f251746080 msgr2=0x55f2518240b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:17.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.183+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/66969978 shutdown_connections 2024-06-09T22:29:17.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.183+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/66969978 wait complete. 2024-06-09T22:29:17.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.183+0000 7fd1ec1b8b80 1 -- start start 2024-06-09T22:29:17.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.183+0000 7fd1ec1b8b80 1 Processor -- start 2024-06-09T22:29:17.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1ec1b8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2518197a0 0x55f25183b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1ec1b8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f251839c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1ec1b8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25181f880 0x55f25183a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1ec1b8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1afa80 con 0x55f25181ef60 2024-06-09T22:29:17.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1ec1b8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1af3b0 con 0x55f2518197a0 2024-06-09T22:29:17.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1ec1b8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc1ac5d0 con 0x55f25181f880 2024-06-09T22:29:17.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1d27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f251839c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1ec1b8b80 10 client.0 _ll_register_callbacks cb 0x55f2517005b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:17.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1d27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f251839c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38216/0 (socket says 172.21.15.104:38216) 2024-06-09T22:29:17.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1d27fc700 1 -- 172.21.15.104:0/2631976963 learned_addr learned my addr 172.21.15.104:0/2631976963 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:17.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1daffd700 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2518197a0 0x55f25183b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1db7fe700 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25181f880 0x55f25183a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1d27fc700 1 -- 172.21.15.104:0/2631976963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25181f880 msgr2=0x55f25183a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:17.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1d27fc700 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25181f880 0x55f25183a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1d27fc700 1 -- 172.21.15.104:0/2631976963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2518197a0 msgr2=0x55f25183b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:17.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1d27fc700 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2518197a0 0x55f25183b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1d27fc700 1 -- 172.21.15.104:0/2631976963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f25183a750 con 0x55f25181ef60 2024-06-09T22:29:17.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1daffd700 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2518197a0 0x55f25183b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:17.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1d27fc700 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f251839c50 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4008010 tx=0x7fd1c40118c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc45e6d8b15a5fa server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:17.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1baffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:17.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1baffd700 1 -- 172.21.15.104:0/2631976963 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4019070 con 0x55f25181ef60 2024-06-09T22:29:17.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1baffd700 1 -- 172.21.15.104:0/2631976963 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fd1c4005a50 con 0x55f25181ef60 2024-06-09T22:29:17.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.184+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/2631976963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1d409f190 con 0x55f25181ef60 2024-06-09T22:29:17.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.185+0000 7fd1baffd700 1 -- 172.21.15.104:0/2631976963 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd1c4035050 con 0x55f25181ef60 2024-06-09T22:29:17.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.185+0000 7fd1ec1b8b80 10 client.17700 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:17.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.185+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/2631976963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f2518a2a30 con 0x55f25181ef60 2024-06-09T22:29:17.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.185+0000 7fd1baffd700 1 -- 172.21.15.104:0/2631976963 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1d409f190 con 0x55f25181ef60 2024-06-09T22:29:17.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.185+0000 7fd1baffd700 1 -- 172.21.15.104:0/2631976963 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd1c401d890 con 0x55f25181ef60 2024-06-09T22:29:17.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.185+0000 7fd1ec1b8b80 10 client.17700 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:17.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.185+0000 7fd1ec1b8b80 10 client.17700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:29:17.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.185+0000 7fd1ec1b8b80 20 client.17700 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:17.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.185+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55f2518a30e0 0x55f2518a5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:17.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.185+0000 7fd1ec1b8b80 4 client.17700 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:29:17.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.185+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/2631976963 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f2517722d0 con 0x55f2518a30e0 2024-06-09T22:29:17.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.185+0000 7fd1daffd700 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55f2518a30e0 0x55f2518a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:17.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1baffd700 10 client.17700 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:17.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1daffd700 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55f2518a30e0 0x55f2518a5500 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a412c74f0981bff server_cookie=7f87fce8f3bc2169 in_seq=0 out_seq=0 2024-06-09T22:29:17.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1baffd700 1 -- 172.21.15.104:0/2631976963 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f2517722d0 con 0x55f2518a30e0 2024-06-09T22:29:17.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1baffd700 10 client.17700 handle_command_reply: tid=0 2024-06-09T22:29:17.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1ec1b8b80 1 client.17700 shutdown 2024-06-09T22:29:17.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/2631976963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 msgr2=0x55f251839c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:17.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f251839c50 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4008010 tx=0x7fd1c40118c0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/2631976963 shutdown_connections 2024-06-09T22:29:17.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55f2518a30e0 0x55f2518a5500 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25181f880 0x55f25183a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2518197a0 0x55f25183b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1ec1b8b80 1 --2- 172.21.15.104:0/2631976963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f25181ef60 0x55f251839c50 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1baffd700 0 client.17700 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:17.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/2631976963 >> 172.21.15.104:0/2631976963 conn(0x55f251746080 msgr2=0x55f251824940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:17.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/2631976963 shutdown_connections 2024-06-09T22:29:17.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1ec1b8b80 1 -- 172.21.15.104:0/2631976963 wait complete. 2024-06-09T22:29:17.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.186+0000 7fd1ec1b8b80 20 client.17700 trim_cache size 0 max 16384 2024-06-09T22:29:17.198 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:29:17.198 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "no active scrubs running", 2024-06-09T22:29:17.199 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": {} 2024-06-09T22:29:17.199 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:29:17.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.187+0000 7fd1e1935700 1 -- 172.21.15.104:0/1937009480 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd1c80a2e10 msgr2=0x7fd1c80a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:17.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.187+0000 7fd1e1935700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd1c80a2e10 0x7fd1c80a52d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00b5c0 tx=0x7fd1cc005fd0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.187+0000 7fd1e1935700 1 -- 172.21.15.104:0/1937009480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1dc108500 msgr2=0x7fd1dc198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:17.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.187+0000 7fd1e1935700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1dc108500 0x7fd1dc198af0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4007ee0 tx=0x7fd1d400d3b0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.187+0000 7fd1e1935700 1 -- 172.21.15.104:0/1937009480 shutdown_connections 2024-06-09T22:29:17.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.187+0000 7fd1e1935700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd1c80a2e10 0x7fd1c80a52d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.187+0000 7fd1e1935700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1dc108500 0x7fd1dc198af0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.187+0000 7fd1e1935700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1dc06a170 0x7fd1dc19e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.187+0000 7fd1e1935700 1 --2- 172.21.15.104:0/1937009480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1dc06ab30 0x7fd1dc19ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:17.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.187+0000 7fd1e1935700 1 -- 172.21.15.104:0/1937009480 >> 172.21.15.104:0/1937009480 conn(0x7fd1dc078340 msgr2=0x7fd1dc100030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:17.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.187+0000 7fd1e1935700 1 -- 172.21.15.104:0/1937009480 shutdown_connections 2024-06-09T22:29:17.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:17.187+0000 7fd1e1935700 1 -- 172.21.15.104:0/1937009480 wait complete. 2024-06-09T22:29:17.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:17 smithi104 ceph-mon[32389]: pgmap v256: 129 pgs: 129 active+clean; 252 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-06-09T22:29:17.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:17 smithi194 ceph-mon[36856]: pgmap v256: 129 pgs: 129 active+clean; 252 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-06-09T22:29:17.898 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-09T22:29:17.898 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:29:17.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:17 smithi153 ceph-mon[36599]: pgmap v256: 129 pgs: 129 active+clean; 252 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-06-09T22:29:18.137 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:29:19.676 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:19 smithi104 ceph-mon[32389]: pgmap v257: 129 pgs: 129 active+clean; 252 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-06-09T22:29:19.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.933+0000 7fb468fd4700 1 -- 172.21.15.104:0/839840700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb464101d50 msgr2=0x7fb46410e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:19.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.933+0000 7fb468fd4700 1 --2- 172.21.15.104:0/839840700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb464101d50 0x7fb46410e240 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00b0a0 tx=0x7fb45c00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:29:19.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.933+0000 7fb468fd4700 1 -- 172.21.15.104:0/839840700 shutdown_connections 2024-06-09T22:29:19.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.933+0000 7fb468fd4700 1 --2- 172.21.15.104:0/839840700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb464101d50 0x7fb46410e240 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:19.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.933+0000 7fb468fd4700 1 --2- 172.21.15.104:0/839840700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb464107400 0x7fb4641077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.933+0000 7fb468fd4700 1 --2- 172.21.15.104:0/839840700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb464101390 0x7fb464101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.933+0000 7fb468fd4700 1 -- 172.21.15.104:0/839840700 >> 172.21.15.104:0/839840700 conn(0x7fb464078340 msgr2=0x7fb464078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.933+0000 7fb468fd4700 1 -- 172.21.15.104:0/839840700 shutdown_connections 2024-06-09T22:29:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.933+0000 7fb468fd4700 1 -- 172.21.15.104:0/839840700 wait complete. 2024-06-09T22:29:19.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb468fd4700 1 Processor -- start 2024-06-09T22:29:19.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb468fd4700 1 -- start start 2024-06-09T22:29:19.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb468fd4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb464101390 0x7fb4641999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:19.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb468fd4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb464101d50 0x7fb464199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:19.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb468fd4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb464107400 0x7fb4641a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:19.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb468fd4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb464111690 con 0x7fb464101d50 2024-06-09T22:29:19.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb468fd4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb464111510 con 0x7fb464107400 2024-06-09T22:29:19.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb468fd4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb464111810 con 0x7fb464101390 2024-06-09T22:29:19.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb461d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb464101d50 0x7fb464199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:19.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb46259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb464101390 0x7fb4641999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:19.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb461d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb464101d50 0x7fb464199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38224/0 (socket says 172.21.15.104:38224) 2024-06-09T22:29:19.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb46259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb464101390 0x7fb4641999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:57190/0 (socket says 172.21.15.104:57190) 2024-06-09T22:29:19.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb461d9b700 1 -- 172.21.15.104:0/3306626421 learned_addr learned my addr 172.21.15.104:0/3306626421 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:19.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.934+0000 7fb462d9d700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb464107400 0x7fb4641a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:19.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.935+0000 7fb46259c700 1 -- 172.21.15.104:0/3306626421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb464107400 msgr2=0x7fb4641a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:19.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.935+0000 7fb46259c700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb464107400 0x7fb4641a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:19.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.935+0000 7fb46259c700 1 -- 172.21.15.104:0/3306626421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb464101d50 msgr2=0x7fb464199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:19.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.935+0000 7fb46259c700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb464101d50 0x7fb464199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:19.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.935+0000 7fb46259c700 1 -- 172.21.15.104:0/3306626421 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4641a4f70 con 0x7fb464101390 2024-06-09T22:29:19.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.935+0000 7fb461d9b700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb464101d50 0x7fb464199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:29:19.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.935+0000 7fb46259c700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb464101390 0x7fb4641999b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb45400e9f0 tx=0x7fb45400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:19.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.935+0000 7fb4537fe700 1 -- 172.21.15.104:0/3306626421 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb454004220 con 0x7fb464101390 2024-06-09T22:29:19.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.935+0000 7fb4537fe700 1 -- 172.21.15.104:0/3306626421 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb454004d10 con 0x7fb464101390 2024-06-09T22:29:19.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.935+0000 7fb468fd4700 1 -- 172.21.15.104:0/3306626421 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4641a5200 con 0x7fb464101390 2024-06-09T22:29:19.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.935+0000 7fb4537fe700 1 -- 172.21.15.104:0/3306626421 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb454005020 con 0x7fb464101390 2024-06-09T22:29:19.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.935+0000 7fb468fd4700 1 -- 172.21.15.104:0/3306626421 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4641a56c0 con 0x7fb464101390 2024-06-09T22:29:19.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.936+0000 7fb468fd4700 1 -- 172.21.15.104:0/3306626421 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb464069180 con 0x7fb464101390 2024-06-09T22:29:19.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.937+0000 7fb4537fe700 1 -- 172.21.15.104:0/3306626421 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fb454042b60 con 0x7fb464101390 2024-06-09T22:29:19.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.938+0000 7fb4537fe700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb44c0a2d90 0x7fb44c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:19.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.938+0000 7fb461d9b700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb44c0a2d90 0x7fb44c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:19.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.938+0000 7fb4537fe700 1 -- 172.21.15.104:0/3306626421 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fb4540cf650 con 0x7fb464101390 2024-06-09T22:29:19.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.938+0000 7fb461d9b700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb44c0a2d90 0x7fb44c0a5250 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb4640788d0 tx=0x7fb458009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:19.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:19.941+0000 7fb4537fe700 1 -- 172.21.15.104:0/3306626421 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb45400f3c0 con 0x7fb464101390 2024-06-09T22:29:19.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:19 smithi153 ceph-mon[36599]: pgmap v257: 129 pgs: 129 active+clean; 252 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-06-09T22:29:20.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:19 smithi194 ceph-mon[36856]: pgmap v257: 129 pgs: 129 active+clean; 252 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-06-09T22:29:20.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.202+0000 7fb468fd4700 1 -- 172.21.15.104:0/3306626421 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb46402dca0 con 0x7fb464101390 2024-06-09T22:29:20.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.203+0000 7fb4537fe700 1 -- 172.21.15.104:0/3306626421 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7fb45409be40 con 0x7fb464101390 2024-06-09T22:29:20.206 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:29:20.206 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:29:20.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.205+0000 7fb468fd4700 1 -- 172.21.15.104:0/3306626421 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb44c0a2d90 msgr2=0x7fb44c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:20.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.205+0000 7fb468fd4700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb44c0a2d90 0x7fb44c0a5250 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb4640788d0 tx=0x7fb458009260 comp rx=0 tx=0).stop 2024-06-09T22:29:20.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.205+0000 7fb468fd4700 1 -- 172.21.15.104:0/3306626421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb464101390 msgr2=0x7fb4641999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:20.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.205+0000 7fb468fd4700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb464101390 0x7fb4641999b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb45400e9f0 tx=0x7fb45400ed00 comp rx=0 tx=0).stop 2024-06-09T22:29:20.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.205+0000 7fb468fd4700 1 -- 172.21.15.104:0/3306626421 shutdown_connections 2024-06-09T22:29:20.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.205+0000 7fb468fd4700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fb44c0a2d90 0x7fb44c0a5250 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:20.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.205+0000 7fb468fd4700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb464107400 0x7fb4641a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:20.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.205+0000 7fb468fd4700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb464101390 0x7fb4641999b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:20.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.205+0000 7fb468fd4700 1 --2- 172.21.15.104:0/3306626421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb464101d50 0x7fb464199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:20.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.206+0000 7fb468fd4700 1 -- 172.21.15.104:0/3306626421 >> 172.21.15.104:0/3306626421 conn(0x7fb464078340 msgr2=0x7fb4640ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:20.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.206+0000 7fb468fd4700 1 -- 172.21.15.104:0/3306626421 shutdown_connections 2024-06-09T22:29:20.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:20.206+0000 7fb468fd4700 1 -- 172.21.15.104:0/3306626421 wait complete. 2024-06-09T22:29:20.209 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:29:20.987 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:20 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:29:20.988 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:20 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3306626421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:29:20.988 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:20 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:29:20.988 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:20 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:29:20.988 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:20 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:29:20.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:20 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:29:20.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:20 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3306626421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:29:20.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:20 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:29:20.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:20 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:29:20.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:20 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:29:21.030 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-09T22:29:21.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:20 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:29:21.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:20 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3306626421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:29:21.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:20 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:29:21.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:20 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:29:21.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:20 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:29:21.270 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:29:21.665 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:21 smithi104 ceph-mon[32389]: pgmap v258: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 1.6 MiB/s wr, 69 op/s 2024-06-09T22:29:21.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:21 smithi153 ceph-mon[36599]: pgmap v258: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 1.6 MiB/s wr, 69 op/s 2024-06-09T22:29:22.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:21 smithi194 ceph-mon[36856]: pgmap v258: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 1.6 MiB/s wr, 69 op/s 2024-06-09T22:29:23.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.082+0000 7fc33de93700 1 -- 172.21.15.104:0/2599077818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc338106b90 msgr2=0x7fc338106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:23.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.082+0000 7fc33de93700 1 --2- 172.21.15.104:0/2599077818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc338106b90 0x7fc338106f70 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fc328009990 tx=0x7fc328009ca0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.083+0000 7fc33de93700 1 -- 172.21.15.104:0/2599077818 shutdown_connections 2024-06-09T22:29:23.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.083+0000 7fc33de93700 1 --2- 172.21.15.104:0/2599077818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc338106b90 0x7fc338106f70 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.083+0000 7fc33de93700 1 --2- 172.21.15.104:0/2599077818 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc338101510 0x7fc33810d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.083+0000 7fc33de93700 1 --2- 172.21.15.104:0/2599077818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc338100b50 0x7fc338100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.083+0000 7fc33de93700 1 -- 172.21.15.104:0/2599077818 >> 172.21.15.104:0/2599077818 conn(0x7fc338077a00 msgr2=0x7fc3380fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:23.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.083+0000 7fc33de93700 1 -- 172.21.15.104:0/2599077818 shutdown_connections 2024-06-09T22:29:23.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.083+0000 7fc33de93700 1 -- 172.21.15.104:0/2599077818 wait complete. 2024-06-09T22:29:23.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.083+0000 7fc33de93700 1 Processor -- start 2024-06-09T22:29:23.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.083+0000 7fc33de93700 1 -- start start 2024-06-09T22:29:23.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc33de93700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc338100b50 0x7fc33819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc33de93700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc338101510 0x7fc33819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc33de93700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc338106b90 0x7fc3381989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc33de93700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381116b0 con 0x7fc338101510 2024-06-09T22:29:23.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc33de93700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc338111530 con 0x7fc338100b50 2024-06-09T22:29:23.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc33de93700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc338111830 con 0x7fc338106b90 2024-06-09T22:29:23.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc336ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc338101510 0x7fc33819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc336ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc338101510 0x7fc33819ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38244/0 (socket says 172.21.15.104:38244) 2024-06-09T22:29:23.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc336ffd700 1 -- 172.21.15.104:0/3797170680 learned_addr learned my addr 172.21.15.104:0/3797170680 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:23.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc337fff700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc338106b90 0x7fc3381989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc3377fe700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc338100b50 0x7fc33819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc336ffd700 1 -- 172.21.15.104:0/3797170680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc338106b90 msgr2=0x7fc3381989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:23.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc336ffd700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc338106b90 0x7fc3381989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc336ffd700 1 -- 172.21.15.104:0/3797170680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc338100b50 msgr2=0x7fc33819e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:23.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc336ffd700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc338100b50 0x7fc33819e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc336ffd700 1 -- 172.21.15.104:0/3797170680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc338199150 con 0x7fc338101510 2024-06-09T22:29:23.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc337fff700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc338106b90 0x7fc3381989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:23.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc3377fe700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc338100b50 0x7fc33819e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:23.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc336ffd700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc338101510 0x7fc33819ed70 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00d950 tx=0x7fc32c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:23.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc334ff9700 1 -- 172.21.15.104:0/3797170680 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c016070 con 0x7fc338101510 2024-06-09T22:29:23.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.084+0000 7fc33de93700 1 -- 172.21.15.104:0/3797170680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc338199440 con 0x7fc338101510 2024-06-09T22:29:23.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.085+0000 7fc334ff9700 1 -- 172.21.15.104:0/3797170680 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc32c009b40 con 0x7fc338101510 2024-06-09T22:29:23.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.085+0000 7fc334ff9700 1 -- 172.21.15.104:0/3797170680 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c00f620 con 0x7fc338101510 2024-06-09T22:29:23.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.085+0000 7fc33de93700 1 -- 172.21.15.104:0/3797170680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3381a5700 con 0x7fc338101510 2024-06-09T22:29:23.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.087+0000 7fc334ff9700 1 -- 172.21.15.104:0/3797170680 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc32c019070 con 0x7fc338101510 2024-06-09T22:29:23.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.087+0000 7fc334ff9700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc3200a2d90 0x7fc3200a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.088+0000 7fc3377fe700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc3200a2d90 0x7fc3200a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.088+0000 7fc334ff9700 1 -- 172.21.15.104:0/3797170680 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fc32c0cf860 con 0x7fc338101510 2024-06-09T22:29:23.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.088+0000 7fc3377fe700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc3200a2d90 0x7fc3200a5250 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc328006010 tx=0x7fc32800b540 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:23.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.088+0000 7fc348716b80 1 Processor -- start 2024-06-09T22:29:23.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.088+0000 7fc348716b80 1 -- start start 2024-06-09T22:29:23.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc348716b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 0x55bda2551750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc348716b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda2551c90 0x55bda254b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc348716b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 0x55bda254bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc348716b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55bda2439f60 con 0x55bda254bbb0 2024-06-09T22:29:23.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc348716b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bda2498850 con 0x55bda2551c90 2024-06-09T22:29:23.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc348716b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bda2400460 con 0x55bda2551370 2024-06-09T22:29:23.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc337fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 0x55bda254bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc337fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 0x55bda254bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38248/0 (socket says 172.21.15.104:38248) 2024-06-09T22:29:23.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc337fff700 1 -- 172.21.15.104:0/562845182 learned_addr learned my addr 172.21.15.104:0/562845182 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:23.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc336ffd700 1 --2- 172.21.15.104:0/562845182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda2551c90 0x55bda254b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc3377fe700 1 --2- 172.21.15.104:0/562845182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 0x55bda2551750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc337fff700 1 -- 172.21.15.104:0/562845182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 msgr2=0x55bda2551750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:23.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc337fff700 1 --2- 172.21.15.104:0/562845182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 0x55bda2551750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc337fff700 1 -- 172.21.15.104:0/562845182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda2551c90 msgr2=0x55bda254b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:23.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc337fff700 1 --2- 172.21.15.104:0/562845182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda2551c90 0x55bda254b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc337fff700 1 -- 172.21.15.104:0/562845182 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bda23e78d0 con 0x55bda254bbb0 2024-06-09T22:29:23.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc336ffd700 1 --2- 172.21.15.104:0/562845182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda2551c90 0x55bda254b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:23.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc337fff700 1 --2- 172.21.15.104:0/562845182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 0x55bda254bfd0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc330009dd0 tx=0x7fc3300077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf122d6b288246d2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:23.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.089+0000 7fc31dffb700 1 -- 172.21.15.104:0/562845182 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330004bb0 con 0x55bda254bbb0 2024-06-09T22:29:23.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.090+0000 7fc31dffb700 1 -- 172.21.15.104:0/562845182 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc330004d50 con 0x55bda254bbb0 2024-06-09T22:29:23.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.090+0000 7fc31dffb700 1 -- 172.21.15.104:0/562845182 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330014670 con 0x55bda254bbb0 2024-06-09T22:29:23.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.090+0000 7fc348716b80 1 -- 172.21.15.104:0/562845182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 msgr2=0x55bda254bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:23.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.090+0000 7fc348716b80 1 --2- 172.21.15.104:0/562845182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 0x55bda254bfd0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc330009dd0 tx=0x7fc3300077e0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.090+0000 7fc348716b80 1 -- 172.21.15.104:0/562845182 shutdown_connections 2024-06-09T22:29:23.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.090+0000 7fc348716b80 1 --2- 172.21.15.104:0/562845182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 0x55bda254bfd0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.090+0000 7fc348716b80 1 --2- 172.21.15.104:0/562845182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 0x55bda2551750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.090+0000 7fc348716b80 1 --2- 172.21.15.104:0/562845182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda2551c90 0x55bda254b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.090+0000 7fc348716b80 1 -- 172.21.15.104:0/562845182 >> 172.21.15.104:0/562845182 conn(0x55bda2474080 msgr2=0x55bda2542220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:23.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.090+0000 7fc348716b80 1 -- 172.21.15.104:0/562845182 shutdown_connections 2024-06-09T22:29:23.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.090+0000 7fc348716b80 1 -- 172.21.15.104:0/562845182 wait complete. 2024-06-09T22:29:23.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.091+0000 7fc348716b80 1 -- start start 2024-06-09T22:29:23.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.091+0000 7fc348716b80 1 Processor -- start 2024-06-09T22:29:23.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc348716b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda254bbb0 0x55bda2557940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc348716b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 0x55bda2554c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc348716b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 0x55bda2555180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc348716b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55bda2438fc0 con 0x55bda2551c90 2024-06-09T22:29:23.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc348716b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bda2494810 con 0x55bda254bbb0 2024-06-09T22:29:23.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc348716b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bda2558670 con 0x55bda2551370 2024-06-09T22:29:23.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc348716b80 10 client.0 _ll_register_callbacks cb 0x55bda242e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:23.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc337fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 0x55bda2555180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc337fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 0x55bda2555180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38260/0 (socket says 172.21.15.104:38260) 2024-06-09T22:29:23.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc337fff700 1 -- 172.21.15.104:0/2819657858 learned_addr learned my addr 172.21.15.104:0/2819657858 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:23.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc3377fe700 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda254bbb0 0x55bda2557940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc336ffd700 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 0x55bda2554c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc337fff700 1 -- 172.21.15.104:0/2819657858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 msgr2=0x55bda2554c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:23.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc337fff700 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 0x55bda2554c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc337fff700 1 -- 172.21.15.104:0/2819657858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda254bbb0 msgr2=0x55bda2557940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:23.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc337fff700 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda254bbb0 0x55bda2557940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc337fff700 1 -- 172.21.15.104:0/2819657858 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bda245e750 con 0x55bda2551c90 2024-06-09T22:29:23.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc337fff700 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 0x55bda2555180 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc330005680 tx=0x7fc33000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3edbf037b780dbf7 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:23.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc316ffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:23.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc316ffd700 1 -- 172.21.15.104:0/2819657858 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330022070 con 0x55bda2551c90 2024-06-09T22:29:23.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc316ffd700 1 -- 172.21.15.104:0/2819657858 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fc33001e420 con 0x55bda2551c90 2024-06-09T22:29:23.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.092+0000 7fc348716b80 1 -- 172.21.15.104:0/2819657858 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc330004d50 con 0x55bda2551c90 2024-06-09T22:29:23.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.093+0000 7fc316ffd700 1 -- 172.21.15.104:0/2819657858 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc33002e050 con 0x55bda2551c90 2024-06-09T22:29:23.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.093+0000 7fc348716b80 10 client.17718 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:23.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.093+0000 7fc348716b80 1 -- 172.21.15.104:0/2819657858 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bda2562d70 con 0x55bda2551c90 2024-06-09T22:29:23.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.093+0000 7fc316ffd700 1 -- 172.21.15.104:0/2819657858 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc330004d50 con 0x55bda2551c90 2024-06-09T22:29:23.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.093+0000 7fc316ffd700 1 -- 172.21.15.104:0/2819657858 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc3300145d0 con 0x55bda2551c90 2024-06-09T22:29:23.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.093+0000 7fc348716b80 10 client.17718 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:23.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.093+0000 7fc348716b80 10 client.17718 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:29:23.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.093+0000 7fc348716b80 20 client.17718 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:23.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.093+0000 7fc348716b80 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55bda2565240 0x55bda2567630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.093+0000 7fc348716b80 4 client.17718 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:29:23.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.093+0000 7fc348716b80 1 -- 172.21.15.104:0/2819657858 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bda24a5d50 con 0x55bda2565240 2024-06-09T22:29:23.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.093+0000 7fc3377fe700 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55bda2565240 0x55bda2567630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.094+0000 7fc316ffd700 10 client.17718 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:29:23.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.094+0000 7fc3377fe700 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55bda2565240 0x55bda2567630 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=be808bb4d8a8d1eb server_cookie=2a3f6559a730c701 in_seq=0 out_seq=0 2024-06-09T22:29:23.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc316ffd700 1 -- 172.21.15.104:0/2819657858 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55bda24a5d50 con 0x55bda2565240 2024-06-09T22:29:23.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc316ffd700 10 client.17718 handle_command_reply: tid=0 2024-06-09T22:29:23.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc348716b80 1 client.17718 shutdown 2024-06-09T22:29:23.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc348716b80 1 -- 172.21.15.104:0/2819657858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 msgr2=0x55bda2555180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:23.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc348716b80 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 0x55bda2555180 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc330005680 tx=0x7fc33000bf40 comp rx=0 tx=0).stop 2024-06-09T22:29:23.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc348716b80 1 -- 172.21.15.104:0/2819657858 shutdown_connections 2024-06-09T22:29:23.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc348716b80 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 0x55bda2555180 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc348716b80 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55bda2565240 0x55bda2567630 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc348716b80 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda254bbb0 0x55bda2557940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc348716b80 1 --2- 172.21.15.104:0/2819657858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 0x55bda2554c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc348716b80 1 -- 172.21.15.104:0/2819657858 >> 172.21.15.104:0/2819657858 conn(0x55bda2474080 msgr2=0x55bda2541700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:23.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc316ffd700 0 client.17718 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:29:23.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc348716b80 1 -- 172.21.15.104:0/2819657858 shutdown_connections 2024-06-09T22:29:23.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc348716b80 1 -- 172.21.15.104:0/2819657858 wait complete. 2024-06-09T22:29:23.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.095+0000 7fc348716b80 20 client.17718 trim_cache size 0 max 16384 2024-06-09T22:29:23.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.110+0000 7fc348716b80 1 Processor -- start 2024-06-09T22:29:23.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.110+0000 7fc348716b80 1 -- start start 2024-06-09T22:29:23.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.110+0000 7fc348716b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda254bbb0 0x55bda254b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.110+0000 7fc348716b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 0x55bda2556b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.110+0000 7fc348716b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 0x55bda2541e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.110+0000 7fc348716b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381ac470 con 0x55bda2551c90 2024-06-09T22:29:23.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.110+0000 7fc348716b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381adfb0 con 0x55bda254bbb0 2024-06-09T22:29:23.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.110+0000 7fc348716b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381acb40 con 0x55bda2551370 2024-06-09T22:29:23.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.110+0000 7fc337fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 0x55bda2541e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.110+0000 7fc337fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 0x55bda2541e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38264/0 (socket says 172.21.15.104:38264) 2024-06-09T22:29:23.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.110+0000 7fc337fff700 1 -- 172.21.15.104:0/1556504456 learned_addr learned my addr 172.21.15.104:0/1556504456 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.111+0000 7fc336ffd700 1 --2- 172.21.15.104:0/1556504456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 0x55bda2556b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.111+0000 7fc337fff700 1 -- 172.21.15.104:0/1556504456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 msgr2=0x55bda2556b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.111+0000 7fc337fff700 1 --2- 172.21.15.104:0/1556504456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 0x55bda2556b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.111+0000 7fc337fff700 1 -- 172.21.15.104:0/1556504456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda254bbb0 msgr2=0x55bda254b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.111+0000 7fc3377fe700 1 --2- 172.21.15.104:0/1556504456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda254bbb0 0x55bda254b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.111+0000 7fc337fff700 1 --2- 172.21.15.104:0/1556504456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda254bbb0 0x55bda254b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.111+0000 7fc337fff700 1 -- 172.21.15.104:0/1556504456 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bda25554b0 con 0x55bda2551c90 2024-06-09T22:29:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.111+0000 7fc3377fe700 1 --2- 172.21.15.104:0/1556504456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda254bbb0 0x55bda254b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:29:23.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.111+0000 7fc337fff700 1 --2- 172.21.15.104:0/1556504456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 0x55bda2541e80 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc3300149a0 tx=0x7fc3300092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917db0cfdfd1ece8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:23.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.111+0000 7fc31d7fa700 1 -- 172.21.15.104:0/1556504456 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330022070 con 0x55bda2551c90 2024-06-09T22:29:23.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.111+0000 7fc31d7fa700 1 -- 172.21.15.104:0/1556504456 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc330006000 con 0x55bda2551c90 2024-06-09T22:29:23.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.111+0000 7fc31d7fa700 1 -- 172.21.15.104:0/1556504456 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330027440 con 0x55bda2551c90 2024-06-09T22:29:23.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.112+0000 7fc348716b80 1 -- 172.21.15.104:0/1556504456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 msgr2=0x55bda2541e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:23.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.112+0000 7fc348716b80 1 --2- 172.21.15.104:0/1556504456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 0x55bda2541e80 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc3300149a0 tx=0x7fc3300092d0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.112+0000 7fc348716b80 1 -- 172.21.15.104:0/1556504456 shutdown_connections 2024-06-09T22:29:23.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.112+0000 7fc348716b80 1 --2- 172.21.15.104:0/1556504456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda2551c90 0x55bda2541e80 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.112+0000 7fc348716b80 1 --2- 172.21.15.104:0/1556504456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda254bbb0 0x55bda254b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.112+0000 7fc348716b80 1 --2- 172.21.15.104:0/1556504456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551370 0x55bda2556b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.112+0000 7fc348716b80 1 -- 172.21.15.104:0/1556504456 >> 172.21.15.104:0/1556504456 conn(0x55bda2474080 msgr2=0x55bda2556470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:23.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.112+0000 7fc348716b80 1 -- 172.21.15.104:0/1556504456 shutdown_connections 2024-06-09T22:29:23.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.112+0000 7fc348716b80 1 -- 172.21.15.104:0/1556504456 wait complete. 2024-06-09T22:29:23.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.112+0000 7fc348716b80 1 -- start start 2024-06-09T22:29:23.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.112+0000 7fc348716b80 1 Processor -- start 2024-06-09T22:29:23.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc348716b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 0x55bda255de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc348716b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda2551370 0x55bda256c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc348716b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551c90 0x55bda256c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc348716b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381ad8e0 con 0x55bda254bbb0 2024-06-09T22:29:23.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc348716b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381ad210 con 0x55bda2551370 2024-06-09T22:29:23.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc348716b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381aa430 con 0x55bda2551c90 2024-06-09T22:29:23.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc3377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 0x55bda255de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc348716b80 10 client.0 _ll_register_callbacks cb 0x55bda242e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:23.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc3377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 0x55bda255de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38276/0 (socket says 172.21.15.104:38276) 2024-06-09T22:29:23.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc3377fe700 1 -- 172.21.15.104:0/3617753325 learned_addr learned my addr 172.21.15.104:0/3617753325 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:23.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc336ffd700 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda2551370 0x55bda256c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc337fff700 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551c90 0x55bda256c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc3377fe700 1 -- 172.21.15.104:0/3617753325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551c90 msgr2=0x55bda256c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:23.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc3377fe700 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551c90 0x55bda256c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc3377fe700 1 -- 172.21.15.104:0/3617753325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda2551370 msgr2=0x55bda256c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:23.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc3377fe700 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda2551370 0x55bda256c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc3377fe700 1 -- 172.21.15.104:0/3617753325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bda256ce90 con 0x55bda254bbb0 2024-06-09T22:29:23.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc3377fe700 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 0x55bda255de90 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fc328009620 tx=0x7fc32801a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77504b4693f89a4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:23.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc316ffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:23.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc316ffd700 1 -- 172.21.15.104:0/3617753325 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328003e30 con 0x55bda254bbb0 2024-06-09T22:29:23.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc316ffd700 1 -- 172.21.15.104:0/3617753325 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fc3280050b0 con 0x55bda254bbb0 2024-06-09T22:29:23.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc348716b80 1 -- 172.21.15.104:0/3617753325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc330006000 con 0x55bda254bbb0 2024-06-09T22:29:23.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.113+0000 7fc316ffd700 1 -- 172.21.15.104:0/3617753325 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc328045050 con 0x55bda254bbb0 2024-06-09T22:29:23.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.114+0000 7fc348716b80 10 client.17724 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:23.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.114+0000 7fc348716b80 1 -- 172.21.15.104:0/3617753325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bda25cf970 con 0x55bda254bbb0 2024-06-09T22:29:23.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.114+0000 7fc316ffd700 1 -- 172.21.15.104:0/3617753325 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc330006000 con 0x55bda254bbb0 2024-06-09T22:29:23.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.114+0000 7fc316ffd700 1 -- 172.21.15.104:0/3617753325 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc32803a020 con 0x55bda254bbb0 2024-06-09T22:29:23.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.114+0000 7fc348716b80 10 client.17724 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:23.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.114+0000 7fc348716b80 10 client.17724 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:29:23.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.114+0000 7fc348716b80 20 client.17724 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:23.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.114+0000 7fc348716b80 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55bda25d0030 0x55bda25d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:23.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.114+0000 7fc348716b80 4 client.17724 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:29:23.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.114+0000 7fc348716b80 1 -- 172.21.15.104:0/3617753325 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bda24a02d0 con 0x55bda25d0030 2024-06-09T22:29:23.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.114+0000 7fc336ffd700 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55bda25d0030 0x55bda25d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:23.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc336ffd700 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55bda25d0030 0x55bda25d2450 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a56108e6f59d7e9a server_cookie=2b7363691b3a2738 in_seq=0 out_seq=0 2024-06-09T22:29:23.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc316ffd700 10 client.17724 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:29:23.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc316ffd700 1 -- 172.21.15.104:0/3617753325 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bda24a02d0 con 0x55bda25d0030 2024-06-09T22:29:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc316ffd700 10 client.17724 handle_command_reply: tid=0 2024-06-09T22:29:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc348716b80 1 client.17724 shutdown 2024-06-09T22:29:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc348716b80 1 -- 172.21.15.104:0/3617753325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 msgr2=0x55bda255de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc348716b80 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 0x55bda255de90 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fc328009620 tx=0x7fc32801a050 comp rx=0 tx=0).stop 2024-06-09T22:29:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc348716b80 1 -- 172.21.15.104:0/3617753325 shutdown_connections 2024-06-09T22:29:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc348716b80 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bda254bbb0 0x55bda255de90 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc348716b80 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55bda25d0030 0x55bda25d2450 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc348716b80 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bda2551c90 0x55bda256c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc348716b80 1 --2- 172.21.15.104:0/3617753325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bda2551370 0x55bda256c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc348716b80 1 -- 172.21.15.104:0/3617753325 >> 172.21.15.104:0/3617753325 conn(0x55bda2474080 msgr2=0x55bda2556d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:23.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc316ffd700 0 client.17724 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:29:23.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc348716b80 1 -- 172.21.15.104:0/3617753325 shutdown_connections 2024-06-09T22:29:23.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc348716b80 1 -- 172.21.15.104:0/3617753325 wait complete. 2024-06-09T22:29:23.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.115+0000 7fc348716b80 20 client.17724 trim_cache size 0 max 16384 2024-06-09T22:29:23.130 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:29:23.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.116+0000 7fc33de93700 1 -- 172.21.15.104:0/3797170680 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc3200a2d90 msgr2=0x7fc3200a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:23.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.116+0000 7fc33de93700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc3200a2d90 0x7fc3200a5250 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc328006010 tx=0x7fc32800b540 comp rx=0 tx=0).stop 2024-06-09T22:29:23.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.116+0000 7fc33de93700 1 -- 172.21.15.104:0/3797170680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc338101510 msgr2=0x7fc33819ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:23.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.116+0000 7fc33de93700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc338101510 0x7fc33819ed70 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00d950 tx=0x7fc32c00dd10 comp rx=0 tx=0).stop 2024-06-09T22:29:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.116+0000 7fc33de93700 1 -- 172.21.15.104:0/3797170680 shutdown_connections 2024-06-09T22:29:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.116+0000 7fc33de93700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc3200a2d90 0x7fc3200a5250 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.116+0000 7fc33de93700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc338106b90 0x7fc3381989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.116+0000 7fc33de93700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc338100b50 0x7fc33819e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.116+0000 7fc33de93700 1 --2- 172.21.15.104:0/3797170680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc338101510 0x7fc33819ed70 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.116+0000 7fc33de93700 1 -- 172.21.15.104:0/3797170680 >> 172.21.15.104:0/3797170680 conn(0x7fc338077a00 msgr2=0x7fc33810e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.116+0000 7fc33de93700 1 -- 172.21.15.104:0/3797170680 shutdown_connections 2024-06-09T22:29:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:23.116+0000 7fc33de93700 1 -- 172.21.15.104:0/3797170680 wait complete. 2024-06-09T22:29:23.880 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:23 smithi104 ceph-mon[32389]: pgmap v259: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 1.6 MiB/s wr, 69 op/s 2024-06-09T22:29:23.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-09T22:29:23.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:23 smithi153 ceph-mon[36599]: pgmap v259: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 1.6 MiB/s wr, 69 op/s 2024-06-09T22:29:24.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:23 smithi194 ceph-mon[36856]: pgmap v259: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 1.6 MiB/s wr, 69 op/s 2024-06-09T22:29:24.145 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:29:25.667 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:25 smithi104 ceph-mon[32389]: pgmap v260: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 2.2 MiB/s wr, 94 op/s 2024-06-09T22:29:25.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.911+0000 7fea18d1f700 1 -- 172.21.15.104:0/2625707861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea14102770 msgr2=0x7fea14102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:25.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.911+0000 7fea18d1f700 1 --2- 172.21.15.104:0/2625707861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea14102770 0x7fea14102b50 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fea040099c0 tx=0x7fea04009cd0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.911+0000 7fea18d1f700 1 -- 172.21.15.104:0/2625707861 shutdown_connections 2024-06-09T22:29:25.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.911+0000 7fea18d1f700 1 --2- 172.21.15.104:0/2625707861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea14102770 0x7fea14102b50 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.911+0000 7fea18d1f700 1 --2- 172.21.15.104:0/2625707861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea14107840 0x7fea1410f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.911+0000 7fea18d1f700 1 --2- 172.21.15.104:0/2625707861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea14103120 0x7fea14107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.911+0000 7fea18d1f700 1 -- 172.21.15.104:0/2625707861 >> 172.21.15.104:0/2625707861 conn(0x7fea14079500 msgr2=0x7fea14079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:25.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.911+0000 7fea18d1f700 1 -- 172.21.15.104:0/2625707861 shutdown_connections 2024-06-09T22:29:25.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.911+0000 7fea18d1f700 1 -- 172.21.15.104:0/2625707861 wait complete. 2024-06-09T22:29:25.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea18d1f700 1 Processor -- start 2024-06-09T22:29:25.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea18d1f700 1 -- start start 2024-06-09T22:29:25.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea18d1f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea14102770 0x7fea14074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea18d1f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea14103120 0x7fea14074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea18d1f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea14107840 0x7fea14075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea18d1f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fea14112760 con 0x7fea14103120 2024-06-09T22:29:25.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea18d1f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea141125e0 con 0x7fea14102770 2024-06-09T22:29:25.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea18d1f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fea141128e0 con 0x7fea14107840 2024-06-09T22:29:25.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea11d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea14103120 0x7fea14074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea11d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea14103120 0x7fea14074cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38302/0 (socket says 172.21.15.104:38302) 2024-06-09T22:29:25.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea11d9b700 1 -- 172.21.15.104:0/1695923410 learned_addr learned my addr 172.21.15.104:0/1695923410 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:25.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea12d9d700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea14107840 0x7fea14075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.912+0000 7fea1259c700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea14102770 0x7fea14074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.913+0000 7fea11d9b700 1 -- 172.21.15.104:0/1695923410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea14107840 msgr2=0x7fea14075210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:25.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.913+0000 7fea11d9b700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea14107840 0x7fea14075210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.913+0000 7fea11d9b700 1 -- 172.21.15.104:0/1695923410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea14102770 msgr2=0x7fea14074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:25.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.913+0000 7fea11d9b700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea14102770 0x7fea14074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.913+0000 7fea11d9b700 1 -- 172.21.15.104:0/1695923410 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea141a70c0 con 0x7fea14103120 2024-06-09T22:29:25.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.913+0000 7fea11d9b700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea14103120 0x7fea14074cd0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fea0800d950 tx=0x7fea0800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:25.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.913+0000 7fea037fe700 1 -- 172.21.15.104:0/1695923410 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea08016070 con 0x7fea14103120 2024-06-09T22:29:25.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.913+0000 7fea18d1f700 1 -- 172.21.15.104:0/1695923410 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea141a73b0 con 0x7fea14103120 2024-06-09T22:29:25.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.913+0000 7fea037fe700 1 -- 172.21.15.104:0/1695923410 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea0800fd70 con 0x7fea14103120 2024-06-09T22:29:25.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.913+0000 7fea037fe700 1 -- 172.21.15.104:0/1695923410 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0800e610 con 0x7fea14103120 2024-06-09T22:29:25.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.913+0000 7fea18d1f700 1 -- 172.21.15.104:0/1695923410 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea141a7940 con 0x7fea14103120 2024-06-09T22:29:25.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.915+0000 7fea037fe700 1 -- 172.21.15.104:0/1695923410 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fea08019070 con 0x7fea14103120 2024-06-09T22:29:25.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.916+0000 7fea037fe700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe9fc0a2d90 0x7fe9fc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.916+0000 7fea1259c700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe9fc0a2d90 0x7fe9fc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.916+0000 7fea037fe700 1 -- 172.21.15.104:0/1695923410 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fea080cf2d0 con 0x7fea14103120 2024-06-09T22:29:25.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.917+0000 7fea1259c700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe9fc0a2d90 0x7fe9fc0a5250 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fea04000c00 tx=0x7fea04005fb0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:25.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.917+0000 7fea235a2b80 1 Processor -- start 2024-06-09T22:29:25.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.917+0000 7fea235a2b80 1 -- start start 2024-06-09T22:29:25.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.917+0000 7fea235a2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822640 0x556be7822a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.917+0000 7fea235a2b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be7822f60 0x556be781c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.917+0000 7fea235a2b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be781cf20 0x556be781d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.917+0000 7fea235a2b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x556be776e810 con 0x556be781cf20 2024-06-09T22:29:25.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.917+0000 7fea235a2b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556be7716720 con 0x556be7822f60 2024-06-09T22:29:25.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.917+0000 7fea235a2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556be776a7d0 con 0x556be7822640 2024-06-09T22:29:25.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea12d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be781cf20 0x556be781d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea12d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be781cf20 0x556be781d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38318/0 (socket says 172.21.15.104:38318) 2024-06-09T22:29:25.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea12d9d700 1 -- 172.21.15.104:0/1322087011 learned_addr learned my addr 172.21.15.104:0/1322087011 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:25.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea1259c700 1 --2- 172.21.15.104:0/1322087011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822640 0x556be7822a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea11d9b700 1 --2- 172.21.15.104:0/1322087011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be7822f60 0x556be781c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea12d9d700 1 -- 172.21.15.104:0/1322087011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822640 msgr2=0x556be7822a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:25.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea12d9d700 1 --2- 172.21.15.104:0/1322087011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822640 0x556be7822a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea12d9d700 1 -- 172.21.15.104:0/1322087011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be7822f60 msgr2=0x556be781c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:25.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea12d9d700 1 --2- 172.21.15.104:0/1322087011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be7822f60 0x556be781c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea12d9d700 1 -- 172.21.15.104:0/1322087011 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556be770cf50 con 0x556be781cf20 2024-06-09T22:29:25.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea12d9d700 1 --2- 172.21.15.104:0/1322087011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be781cf20 0x556be781d340 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fea0c003f30 tx=0x7fea0c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e560d7e0dc8654f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:25.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea00ff9700 1 -- 172.21.15.104:0/1322087011 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c004670 con 0x556be781cf20 2024-06-09T22:29:25.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea00ff9700 1 -- 172.21.15.104:0/1322087011 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea0c01a070 con 0x556be781cf20 2024-06-09T22:29:25.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.918+0000 7fea00ff9700 1 -- 172.21.15.104:0/1322087011 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c015770 con 0x556be781cf20 2024-06-09T22:29:25.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.919+0000 7fea235a2b80 1 -- 172.21.15.104:0/1322087011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be781cf20 msgr2=0x556be781d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:25.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.919+0000 7fea235a2b80 1 --2- 172.21.15.104:0/1322087011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be781cf20 0x556be781d340 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fea0c003f30 tx=0x7fea0c009c20 comp rx=0 tx=0).stop 2024-06-09T22:29:25.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.919+0000 7fea235a2b80 1 -- 172.21.15.104:0/1322087011 shutdown_connections 2024-06-09T22:29:25.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.919+0000 7fea235a2b80 1 --2- 172.21.15.104:0/1322087011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be781cf20 0x556be781d340 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.919+0000 7fea235a2b80 1 --2- 172.21.15.104:0/1322087011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822640 0x556be7822a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.919+0000 7fea235a2b80 1 --2- 172.21.15.104:0/1322087011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be7822f60 0x556be781c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.919+0000 7fea235a2b80 1 -- 172.21.15.104:0/1322087011 >> 172.21.15.104:0/1322087011 conn(0x556be7747cc0 msgr2=0x556be7817850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:25.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.919+0000 7fea235a2b80 1 -- 172.21.15.104:0/1322087011 shutdown_connections 2024-06-09T22:29:25.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.919+0000 7fea235a2b80 1 -- 172.21.15.104:0/1322087011 wait complete. 2024-06-09T22:29:25.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea235a2b80 1 -- start start 2024-06-09T22:29:25.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea235a2b80 1 Processor -- start 2024-06-09T22:29:25.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea235a2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be781cf20 0x556be7828c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea235a2b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 0x556be7825f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea235a2b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be7822f60 0x556be78264b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea235a2b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x556be78299d0 con 0x556be7822640 2024-06-09T22:29:25.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea235a2b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556be770ebf0 con 0x556be7822f60 2024-06-09T22:29:25.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea235a2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556be7829b50 con 0x556be781cf20 2024-06-09T22:29:25.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea235a2b80 10 client.0 _ll_register_callbacks cb 0x556be7708920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:25.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea11d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 0x556be7825f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea11d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 0x556be7825f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38328/0 (socket says 172.21.15.104:38328) 2024-06-09T22:29:25.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea11d9b700 1 -- 172.21.15.104:0/2169475724 learned_addr learned my addr 172.21.15.104:0/2169475724 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:25.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea1259c700 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be781cf20 0x556be7828c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.920+0000 7fea12d9d700 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be7822f60 0x556be78264b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea11d9b700 1 -- 172.21.15.104:0/2169475724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be781cf20 msgr2=0x556be7828c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:25.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea11d9b700 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be781cf20 0x556be7828c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea11d9b700 1 -- 172.21.15.104:0/2169475724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be7822f60 msgr2=0x556be78264b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:25.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea11d9b700 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be7822f60 0x556be78264b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea11d9b700 1 -- 172.21.15.104:0/2169475724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556be76bd8d0 con 0x556be7822640 2024-06-09T22:29:25.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea11d9b700 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 0x556be7825f70 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fea08097df0 tx=0x7fea0809b030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e10c040a35e54f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:25.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fe9f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:25.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fe9f9ffb700 1 -- 172.21.15.104:0/2169475724 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea08020040 con 0x556be7822640 2024-06-09T22:29:25.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fe9f9ffb700 1 -- 172.21.15.104:0/2169475724 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fea0809b680 con 0x556be7822640 2024-06-09T22:29:25.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea235a2b80 1 -- 172.21.15.104:0/2169475724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea0c01a070 con 0x556be7822640 2024-06-09T22:29:25.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fe9f9ffb700 1 -- 172.21.15.104:0/2169475724 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fea0809b420 con 0x556be7822640 2024-06-09T22:29:25.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea235a2b80 10 client.17736 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:25.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea235a2b80 1 -- 172.21.15.104:0/2169475724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556be78342f0 con 0x556be7822640 2024-06-09T22:29:25.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fe9f9ffb700 1 -- 172.21.15.104:0/2169475724 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea0c01a070 con 0x556be7822640 2024-06-09T22:29:25.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fe9f9ffb700 1 -- 172.21.15.104:0/2169475724 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fea080a3c00 con 0x556be7822640 2024-06-09T22:29:25.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea235a2b80 10 client.17736 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:25.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea235a2b80 10 client.17736 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:29:25.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea235a2b80 20 client.17736 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:25.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea235a2b80 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x556be7836930 0x556be7838d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea235a2b80 4 client.17736 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:29:25.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.921+0000 7fea235a2b80 1 -- 172.21.15.104:0/2169475724 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556be777bd10 con 0x556be7836930 2024-06-09T22:29:25.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.922+0000 7fea1259c700 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x556be7836930 0x556be7838d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.922+0000 7fe9f9ffb700 10 client.17736 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:29:25.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.922+0000 7fea1259c700 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x556be7836930 0x556be7838d20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7cc4754218ccb4f3 server_cookie=1876460118031a08 in_seq=0 out_seq=0 2024-06-09T22:29:25.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fe9f9ffb700 1 -- 172.21.15.104:0/2169475724 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x556be777bd10 con 0x556be7836930 2024-06-09T22:29:25.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fe9f9ffb700 10 client.17736 handle_command_reply: tid=0 2024-06-09T22:29:25.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fea235a2b80 1 client.17736 shutdown 2024-06-09T22:29:25.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fea235a2b80 1 -- 172.21.15.104:0/2169475724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 msgr2=0x556be7825f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:25.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fea235a2b80 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 0x556be7825f70 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fea08097df0 tx=0x7fea0809b030 comp rx=0 tx=0).stop 2024-06-09T22:29:25.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fea235a2b80 1 -- 172.21.15.104:0/2169475724 shutdown_connections 2024-06-09T22:29:25.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fea235a2b80 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be7822f60 0x556be78264b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fea235a2b80 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be781cf20 0x556be7828c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fea235a2b80 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x556be7836930 0x556be7838d20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fea235a2b80 1 --2- 172.21.15.104:0/2169475724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 0x556be7825f70 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fea235a2b80 1 -- 172.21.15.104:0/2169475724 >> 172.21.15.104:0/2169475724 conn(0x556be7747cc0 msgr2=0x556be781b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:25.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fe9f9ffb700 0 client.17736 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:29:25.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fea235a2b80 1 -- 172.21.15.104:0/2169475724 shutdown_connections 2024-06-09T22:29:25.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.923+0000 7fea235a2b80 1 -- 172.21.15.104:0/2169475724 wait complete. 2024-06-09T22:29:25.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.924+0000 7fea235a2b80 20 client.17736 trim_cache size 0 max 16384 2024-06-09T22:29:25.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.938+0000 7fea235a2b80 1 Processor -- start 2024-06-09T22:29:25.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.938+0000 7fea235a2b80 1 -- start start 2024-06-09T22:29:25.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.938+0000 7fea235a2b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be781cf20 0x556be781c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.938+0000 7fea235a2b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 0x556be7823400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.938+0000 7fea235a2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822f60 0x556be7823940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.938+0000 7fea235a2b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fea141b07a0 con 0x556be7822640 2024-06-09T22:29:25.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.938+0000 7fea235a2b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea141b22e0 con 0x556be781cf20 2024-06-09T22:29:25.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.938+0000 7fea235a2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fea141b0e70 con 0x556be7822f60 2024-06-09T22:29:25.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fea11d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 0x556be7823400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fea11d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 0x556be7823400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38330/0 (socket says 172.21.15.104:38330) 2024-06-09T22:29:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fea11d9b700 1 -- 172.21.15.104:0/1919186918 learned_addr learned my addr 172.21.15.104:0/1919186918 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fea1259c700 1 --2- 172.21.15.104:0/1919186918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be781cf20 0x556be781c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fea12d9d700 1 --2- 172.21.15.104:0/1919186918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822f60 0x556be7823940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fea11d9b700 1 -- 172.21.15.104:0/1919186918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822f60 msgr2=0x556be7823940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fea11d9b700 1 --2- 172.21.15.104:0/1919186918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822f60 0x556be7823940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fea11d9b700 1 -- 172.21.15.104:0/1919186918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be781cf20 msgr2=0x556be781c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fea11d9b700 1 --2- 172.21.15.104:0/1919186918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be781cf20 0x556be781c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fea11d9b700 1 -- 172.21.15.104:0/1919186918 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556be78244e0 con 0x556be7822640 2024-06-09T22:29:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fea1259c700 1 --2- 172.21.15.104:0/1919186918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be781cf20 0x556be781c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fea11d9b700 1 --2- 172.21.15.104:0/1919186918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 0x556be7823400 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fea080a3fa0 tx=0x7fea08005910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c624f4d23062bb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fe9fbfff700 1 -- 172.21.15.104:0/1919186918 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea08020050 con 0x556be7822640 2024-06-09T22:29:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fe9fbfff700 1 -- 172.21.15.104:0/1919186918 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea0809f7d0 con 0x556be7822640 2024-06-09T22:29:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.939+0000 7fe9fbfff700 1 -- 172.21.15.104:0/1919186918 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea080a3c00 con 0x556be7822640 2024-06-09T22:29:25.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 -- 172.21.15.104:0/1919186918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 msgr2=0x556be7823400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:25.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 --2- 172.21.15.104:0/1919186918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 0x556be7823400 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fea080a3fa0 tx=0x7fea08005910 comp rx=0 tx=0).stop 2024-06-09T22:29:25.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 -- 172.21.15.104:0/1919186918 shutdown_connections 2024-06-09T22:29:25.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 --2- 172.21.15.104:0/1919186918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822f60 0x556be7823940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 --2- 172.21.15.104:0/1919186918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be781cf20 0x556be781c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 --2- 172.21.15.104:0/1919186918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822640 0x556be7823400 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 -- 172.21.15.104:0/1919186918 >> 172.21.15.104:0/1919186918 conn(0x556be7747cc0 msgr2=0x556be7817380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:25.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 -- 172.21.15.104:0/1919186918 shutdown_connections 2024-06-09T22:29:25.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 -- 172.21.15.104:0/1919186918 wait complete. 2024-06-09T22:29:25.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 -- start start 2024-06-09T22:29:25.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 Processor -- start 2024-06-09T22:29:25.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be781cf20 0x556be782f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.940+0000 7fea235a2b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822640 0x556be782f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea235a2b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822f60 0x556be783dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea235a2b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fea141b1c10 con 0x556be7822f60 2024-06-09T22:29:25.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea235a2b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea141b1540 con 0x556be781cf20 2024-06-09T22:29:25.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea235a2b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fea141ae760 con 0x556be7822640 2024-06-09T22:29:25.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea235a2b80 10 client.0 _ll_register_callbacks cb 0x556be7825e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:25.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea12d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822f60 0x556be783dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea12d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822f60 0x556be783dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38334/0 (socket says 172.21.15.104:38334) 2024-06-09T22:29:25.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea12d9d700 1 -- 172.21.15.104:0/2086988167 learned_addr learned my addr 172.21.15.104:0/2086988167 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:25.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea1259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be781cf20 0x556be782f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea11d9b700 1 --2- 172.21.15.104:0/2086988167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822640 0x556be782f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea12d9d700 1 -- 172.21.15.104:0/2086988167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822640 msgr2=0x556be782f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea12d9d700 1 --2- 172.21.15.104:0/2086988167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822640 0x556be782f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea12d9d700 1 -- 172.21.15.104:0/2086988167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be781cf20 msgr2=0x556be782f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea12d9d700 1 --2- 172.21.15.104:0/2086988167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be781cf20 0x556be782f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea12d9d700 1 -- 172.21.15.104:0/2086988167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556be783e3a0 con 0x556be7822f60 2024-06-09T22:29:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea12d9d700 1 --2- 172.21.15.104:0/2086988167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822f60 0x556be783dba0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fea0c003f30 tx=0x7fea0c007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca206f6f8669a48d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fe9f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fe9f9ffb700 1 -- 172.21.15.104:0/2086988167 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c007e60 con 0x556be7822f60 2024-06-09T22:29:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fe9f9ffb700 1 -- 172.21.15.104:0/2086988167 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fea0c015890 con 0x556be7822f60 2024-06-09T22:29:25.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.941+0000 7fea235a2b80 1 -- 172.21.15.104:0/2086988167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea0809f7d0 con 0x556be7822f60 2024-06-09T22:29:25.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.942+0000 7fe9f9ffb700 1 -- 172.21.15.104:0/2086988167 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fea0c02f050 con 0x556be7822f60 2024-06-09T22:29:25.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.942+0000 7fea235a2b80 10 client.17742 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:25.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.942+0000 7fea235a2b80 1 -- 172.21.15.104:0/2086988167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556be78a10a0 con 0x556be7822f60 2024-06-09T22:29:25.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.942+0000 7fe9f9ffb700 1 -- 172.21.15.104:0/2086988167 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea0809f7d0 con 0x556be7822f60 2024-06-09T22:29:25.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.942+0000 7fe9f9ffb700 1 -- 172.21.15.104:0/2086988167 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fea0c028060 con 0x556be7822f60 2024-06-09T22:29:25.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.942+0000 7fea235a2b80 10 client.17742 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:25.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.942+0000 7fea235a2b80 10 client.17742 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:29:25.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.942+0000 7fea235a2b80 20 client.17742 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.942+0000 7fea235a2b80 1 --2- 172.21.15.104:0/2086988167 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x556be78a16c0 0x556be78a3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.942+0000 7fea235a2b80 4 client.17742 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:29:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.942+0000 7fea235a2b80 1 -- 172.21.15.104:0/2086988167 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556be7776290 con 0x556be78a16c0 2024-06-09T22:29:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.942+0000 7fea1259c700 1 --2- 172.21.15.104:0/2086988167 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x556be78a16c0 0x556be78a3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fe9f9ffb700 10 client.17742 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:29:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea1259c700 1 --2- 172.21.15.104:0/2086988167 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x556be78a16c0 0x556be78a3ae0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6c48080328ed912e server_cookie=d16c74c6091e7a8a in_seq=0 out_seq=0 2024-06-09T22:29:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fe9f9ffb700 1 -- 172.21.15.104:0/2086988167 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556be7776290 con 0x556be78a16c0 2024-06-09T22:29:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fe9f9ffb700 10 client.17742 handle_command_reply: tid=0 2024-06-09T22:29:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea235a2b80 1 client.17742 shutdown 2024-06-09T22:29:25.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea235a2b80 1 -- 172.21.15.104:0/2086988167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822f60 msgr2=0x556be783dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:25.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea235a2b80 1 --2- 172.21.15.104:0/2086988167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822f60 0x556be783dba0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fea0c003f30 tx=0x7fea0c007c50 comp rx=0 tx=0).stop 2024-06-09T22:29:25.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea235a2b80 1 -- 172.21.15.104:0/2086988167 shutdown_connections 2024-06-09T22:29:25.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea235a2b80 1 --2- 172.21.15.104:0/2086988167 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x556be78a16c0 0x556be78a3ae0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fe9f9ffb700 0 client.17742 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:29:25.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea235a2b80 1 --2- 172.21.15.104:0/2086988167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556be7822f60 0x556be783dba0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea235a2b80 1 --2- 172.21.15.104:0/2086988167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be781cf20 0x556be782f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea235a2b80 1 --2- 172.21.15.104:0/2086988167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556be7822640 0x556be782f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea235a2b80 1 -- 172.21.15.104:0/2086988167 >> 172.21.15.104:0/2086988167 conn(0x556be7747cc0 msgr2=0x556be7823400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:25.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea235a2b80 1 -- 172.21.15.104:0/2086988167 shutdown_connections 2024-06-09T22:29:25.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea235a2b80 1 -- 172.21.15.104:0/2086988167 wait complete. 2024-06-09T22:29:25.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.943+0000 7fea235a2b80 20 client.17742 trim_cache size 0 max 16384 2024-06-09T22:29:25.955 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:29:25.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.944+0000 7fea18d1f700 1 -- 172.21.15.104:0/1695923410 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe9fc0a2d90 msgr2=0x7fe9fc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:25.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.944+0000 7fea18d1f700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe9fc0a2d90 0x7fe9fc0a5250 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fea04000c00 tx=0x7fea04005fb0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.944+0000 7fea18d1f700 1 -- 172.21.15.104:0/1695923410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea14103120 msgr2=0x7fea14074cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:25.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.944+0000 7fea18d1f700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea14103120 0x7fea14074cd0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fea0800d950 tx=0x7fea0800dc60 comp rx=0 tx=0).stop 2024-06-09T22:29:25.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.944+0000 7fea18d1f700 1 -- 172.21.15.104:0/1695923410 shutdown_connections 2024-06-09T22:29:25.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.944+0000 7fea18d1f700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fe9fc0a2d90 0x7fe9fc0a5250 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.944+0000 7fea18d1f700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea14107840 0x7fea14075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.944+0000 7fea18d1f700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea14102770 0x7fea14074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.944+0000 7fea18d1f700 1 --2- 172.21.15.104:0/1695923410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea14103120 0x7fea14074cd0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:25.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.944+0000 7fea18d1f700 1 -- 172.21.15.104:0/1695923410 >> 172.21.15.104:0/1695923410 conn(0x7fea14079500 msgr2=0x7fea141005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:25.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.944+0000 7fea18d1f700 1 -- 172.21.15.104:0/1695923410 shutdown_connections 2024-06-09T22:29:25.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:25.944+0000 7fea18d1f700 1 -- 172.21.15.104:0/1695923410 wait complete. 2024-06-09T22:29:25.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:25 smithi153 ceph-mon[36599]: pgmap v260: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 2.2 MiB/s wr, 94 op/s 2024-06-09T22:29:26.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:25 smithi194 ceph-mon[36856]: pgmap v260: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 2.2 MiB/s wr, 94 op/s 2024-06-09T22:29:26.771 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-09T22:29:27.013 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:29:27.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:27 smithi104 ceph-mon[32389]: pgmap v261: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.3 MiB/s wr, 59 op/s 2024-06-09T22:29:27.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:27 smithi153 ceph-mon[36599]: pgmap v261: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.3 MiB/s wr, 59 op/s 2024-06-09T22:29:28.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:27 smithi194 ceph-mon[36856]: pgmap v261: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.3 MiB/s wr, 59 op/s 2024-06-09T22:29:28.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.738+0000 7f34e6370700 1 -- 172.21.15.104:0/3180272869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34e0108500 msgr2=0x7f34e01088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:28.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.738+0000 7f34e6370700 1 --2- 172.21.15.104:0/3180272869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34e0108500 0x7f34e01088e0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009990 tx=0x7f34d0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.738+0000 7f34e6370700 1 -- 172.21.15.104:0/3180272869 shutdown_connections 2024-06-09T22:29:28.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.738+0000 7f34e6370700 1 --2- 172.21.15.104:0/3180272869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34e0108500 0x7f34e01088e0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.738+0000 7f34e6370700 1 --2- 172.21.15.104:0/3180272869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34e006b5a0 0x7f34e010f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.738+0000 7f34e6370700 1 --2- 172.21.15.104:0/3180272869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e006abe0 0x7f34e006b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.738+0000 7f34e6370700 1 -- 172.21.15.104:0/3180272869 >> 172.21.15.104:0/3180272869 conn(0x7f34e0078360 msgr2=0x7f34e0078770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:28.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.738+0000 7f34e6370700 1 -- 172.21.15.104:0/3180272869 shutdown_connections 2024-06-09T22:29:28.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.738+0000 7f34e6370700 1 -- 172.21.15.104:0/3180272869 wait complete. 2024-06-09T22:29:28.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.738+0000 7f34e6370700 1 Processor -- start 2024-06-09T22:29:28.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.738+0000 7f34e6370700 1 -- start start 2024-06-09T22:29:28.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34e6370700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34e006abe0 0x7f34e0101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34e6370700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e006b5a0 0x7f34e01019d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34e6370700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34e0108500 0x7f34e0101f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34e6370700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f34e0112760 con 0x7f34e006abe0 2024-06-09T22:29:28.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34e6370700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f34e01125e0 con 0x7f34e0108500 2024-06-09T22:29:28.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34e6370700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f34e01128e0 con 0x7f34e006b5a0 2024-06-09T22:29:28.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34dffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34e006abe0 0x7f34e0101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34dffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34e006abe0 0x7f34e0101490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56348/0 (socket says 172.21.15.104:56348) 2024-06-09T22:29:28.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34dffff700 1 -- 172.21.15.104:0/2728148599 learned_addr learned my addr 172.21.15.104:0/2728148599 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:28.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34e490d700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34e0108500 0x7f34e0101f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34df7fe700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e006b5a0 0x7f34e01019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34dffff700 1 -- 172.21.15.104:0/2728148599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e006b5a0 msgr2=0x7f34e01019d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:28.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34dffff700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e006b5a0 0x7f34e01019d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34dffff700 1 -- 172.21.15.104:0/2728148599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34e0108500 msgr2=0x7f34e0101f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:28.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34dffff700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34e0108500 0x7f34e0101f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.739+0000 7f34dffff700 1 -- 172.21.15.104:0/2728148599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e01a7040 con 0x7f34e006abe0 2024-06-09T22:29:28.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.740+0000 7f34dffff700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34e006abe0 0x7f34e0101490 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f34d0000c00 tx=0x7f34d0003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:28.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.740+0000 7f34dd7fa700 1 -- 172.21.15.104:0/2728148599 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d0027070 con 0x7f34e006abe0 2024-06-09T22:29:28.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.740+0000 7f34e6370700 1 -- 172.21.15.104:0/2728148599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e01a72d0 con 0x7f34e006abe0 2024-06-09T22:29:28.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.740+0000 7f34dd7fa700 1 -- 172.21.15.104:0/2728148599 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34d0004050 con 0x7f34e006abe0 2024-06-09T22:29:28.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.740+0000 7f34dd7fa700 1 -- 172.21.15.104:0/2728148599 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d000f460 con 0x7f34e006abe0 2024-06-09T22:29:28.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.740+0000 7f34e6370700 1 -- 172.21.15.104:0/2728148599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e01a7720 con 0x7f34e006abe0 2024-06-09T22:29:28.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.742+0000 7f34dd7fa700 1 -- 172.21.15.104:0/2728148599 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f34d002a070 con 0x7f34e006abe0 2024-06-09T22:29:28.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.743+0000 7f34dd7fa700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f34c80a2d90 0x7f34c80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.743+0000 7f34df7fe700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f34c80a2d90 0x7f34c80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.743+0000 7f34dd7fa700 1 -- 172.21.15.104:0/2728148599 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f34d00d0800 con 0x7f34e006abe0 2024-06-09T22:29:28.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.743+0000 7f34df7fe700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f34c80a2d90 0x7f34c80a5250 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f34e0078860 tx=0x7f34d400a380 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:28.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.743+0000 7f34f0bf3b80 1 Processor -- start 2024-06-09T22:29:28.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.743+0000 7f34f0bf3b80 1 -- start start 2024-06-09T22:29:28.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34f0bf3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce3a5640 0x55d2ce3a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34f0bf3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5f60 0x55d2ce39f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34f0bf3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce39ff20 0x55d2ce3a0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34f0bf3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d2ce2f1810 con 0x55d2ce39ff20 2024-06-09T22:29:28.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34f0bf3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d2ce299720 con 0x55d2ce3a5640 2024-06-09T22:29:28.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34f0bf3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d2ce2ed7d0 con 0x55d2ce3a5f60 2024-06-09T22:29:28.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34e490d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce39ff20 0x55d2ce3a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34e490d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce39ff20 0x55d2ce3a0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56362/0 (socket says 172.21.15.104:56362) 2024-06-09T22:29:28.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34e490d700 1 -- 172.21.15.104:0/1080117346 learned_addr learned my addr 172.21.15.104:0/1080117346 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:28.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34dffff700 1 --2- 172.21.15.104:0/1080117346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce3a5640 0x55d2ce3a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34df7fe700 1 --2- 172.21.15.104:0/1080117346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5f60 0x55d2ce39f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34e490d700 1 -- 172.21.15.104:0/1080117346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5f60 msgr2=0x55d2ce39f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:28.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34e490d700 1 --2- 172.21.15.104:0/1080117346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5f60 0x55d2ce39f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34e490d700 1 -- 172.21.15.104:0/1080117346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce3a5640 msgr2=0x55d2ce3a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:28.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34e490d700 1 --2- 172.21.15.104:0/1080117346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce3a5640 0x55d2ce3a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34e490d700 1 -- 172.21.15.104:0/1080117346 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2ce28ff50 con 0x55d2ce39ff20 2024-06-09T22:29:28.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34dffff700 1 --2- 172.21.15.104:0/1080117346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce3a5640 0x55d2ce3a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:28.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.744+0000 7f34e490d700 1 --2- 172.21.15.104:0/1080117346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce39ff20 0x55d2ce3a0340 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f34d800f410 tx=0x7f34d800f720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc47965caea62e44 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:28.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.745+0000 7f34c67fc700 1 -- 172.21.15.104:0/1080117346 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8018030 con 0x55d2ce39ff20 2024-06-09T22:29:28.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.745+0000 7f34c67fc700 1 -- 172.21.15.104:0/1080117346 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34d800fe50 con 0x55d2ce39ff20 2024-06-09T22:29:28.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.745+0000 7f34c67fc700 1 -- 172.21.15.104:0/1080117346 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8005020 con 0x55d2ce39ff20 2024-06-09T22:29:28.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.745+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1080117346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce39ff20 msgr2=0x55d2ce3a0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:28.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.745+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/1080117346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce39ff20 0x55d2ce3a0340 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f34d800f410 tx=0x7f34d800f720 comp rx=0 tx=0).stop 2024-06-09T22:29:28.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.745+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1080117346 shutdown_connections 2024-06-09T22:29:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.745+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/1080117346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce39ff20 0x55d2ce3a0340 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.745+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/1080117346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce3a5640 0x55d2ce3a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.745+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/1080117346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5f60 0x55d2ce39f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.745+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1080117346 >> 172.21.15.104:0/1080117346 conn(0x55d2ce2cacc0 msgr2=0x55d2ce39a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.745+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1080117346 shutdown_connections 2024-06-09T22:29:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.745+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1080117346 wait complete. 2024-06-09T22:29:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34f0bf3b80 1 -- start start 2024-06-09T22:29:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34f0bf3b80 1 Processor -- start 2024-06-09T22:29:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34f0bf3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce39ff20 0x55d2ce3abc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34f0bf3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 0x55d2ce3a8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34f0bf3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5f60 0x55d2ce3a9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34f0bf3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d2ce3ac9d0 con 0x55d2ce3a5640 2024-06-09T22:29:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34f0bf3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d2ce291bf0 con 0x55d2ce39ff20 2024-06-09T22:29:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34f0bf3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d2ce3acb50 con 0x55d2ce3a5f60 2024-06-09T22:29:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34f0bf3b80 10 client.0 _ll_register_callbacks cb 0x55d2ce28b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34df7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 0x55d2ce3a8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34df7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 0x55d2ce3a8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56366/0 (socket says 172.21.15.104:56366) 2024-06-09T22:29:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34df7fe700 1 -- 172.21.15.104:0/1024712337 learned_addr learned my addr 172.21.15.104:0/1024712337 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.746+0000 7f34dffff700 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce39ff20 0x55d2ce3abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34e490d700 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5f60 0x55d2ce3a9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34df7fe700 1 -- 172.21.15.104:0/1024712337 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5f60 msgr2=0x55d2ce3a9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34df7fe700 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5f60 0x55d2ce3a9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34df7fe700 1 -- 172.21.15.104:0/1024712337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce39ff20 msgr2=0x55d2ce3abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34df7fe700 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce39ff20 0x55d2ce3abc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34df7fe700 1 -- 172.21.15.104:0/1024712337 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2ce2408d0 con 0x55d2ce3a5640 2024-06-09T22:29:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34dffff700 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce39ff20 0x55d2ce3abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:29:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34df7fe700 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 0x55d2ce3a8f40 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f34d4004690 tx=0x7f34d40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5bca1c58f519b32 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34bf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34bf7fe700 1 -- 172.21.15.104:0/1024712337 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4016070 con 0x55d2ce3a5640 2024-06-09T22:29:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34bf7fe700 1 -- 172.21.15.104:0/1024712337 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f34d4027670 con 0x55d2ce3a5640 2024-06-09T22:29:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1024712337 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34d800fe50 con 0x55d2ce3a5640 2024-06-09T22:29:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34bf7fe700 1 -- 172.21.15.104:0/1024712337 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f34d4030050 con 0x55d2ce3a5640 2024-06-09T22:29:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34f0bf3b80 10 client.17754 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:28.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.747+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1024712337 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d2ce3b7260 con 0x55d2ce3a5640 2024-06-09T22:29:28.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.748+0000 7f34bf7fe700 1 -- 172.21.15.104:0/1024712337 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34d800fe50 con 0x55d2ce3a5640 2024-06-09T22:29:28.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.748+0000 7f34bf7fe700 1 -- 172.21.15.104:0/1024712337 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f34d4019070 con 0x55d2ce3a5640 2024-06-09T22:29:28.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.748+0000 7f34f0bf3b80 10 client.17754 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:28.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.748+0000 7f34f0bf3b80 10 client.17754 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:29:28.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.748+0000 7f34f0bf3b80 20 client.17754 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:28.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.748+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d2ce3b98a0 0x55d2ce3bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.748+0000 7f34f0bf3b80 4 client.17754 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:29:28.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.748+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1024712337 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2ce2fed10 con 0x55d2ce3b98a0 2024-06-09T22:29:28.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.748+0000 7f34dffff700 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d2ce3b98a0 0x55d2ce3bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.748+0000 7f34dffff700 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d2ce3b98a0 0x55d2ce3bbc90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d42c0ca1df5c6e4 server_cookie=e44e809c2a84bd6a in_seq=0 out_seq=0 2024-06-09T22:29:28.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.748+0000 7f34bf7fe700 10 client.17754 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:29:28.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.749+0000 7f34bf7fe700 1 -- 172.21.15.104:0/1024712337 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d2ce2fed10 con 0x55d2ce3b98a0 2024-06-09T22:29:28.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.749+0000 7f34bf7fe700 10 client.17754 handle_command_reply: tid=0 2024-06-09T22:29:28.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.749+0000 7f34f0bf3b80 1 client.17754 shutdown 2024-06-09T22:29:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.749+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1024712337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 msgr2=0x55d2ce3a8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.749+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 0x55d2ce3a8f40 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f34d4004690 tx=0x7f34d40047c0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.749+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1024712337 shutdown_connections 2024-06-09T22:29:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.749+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d2ce3b98a0 0x55d2ce3bbc90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.749+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5f60 0x55d2ce3a9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.749+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce39ff20 0x55d2ce3abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.749+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/1024712337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 0x55d2ce3a8f40 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.749+0000 7f34bf7fe700 0 client.17754 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:29:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.749+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1024712337 >> 172.21.15.104:0/1024712337 conn(0x55d2ce2cacc0 msgr2=0x55d2ce39e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.750+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1024712337 shutdown_connections 2024-06-09T22:29:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.750+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/1024712337 wait complete. 2024-06-09T22:29:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.750+0000 7f34f0bf3b80 20 client.17754 trim_cache size 0 max 16384 2024-06-09T22:29:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34f0bf3b80 1 Processor -- start 2024-06-09T22:29:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34f0bf3b80 1 -- start start 2024-06-09T22:29:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34f0bf3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce39ff20 0x55d2ce39f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34f0bf3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5640 0x55d2ce39a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34f0bf3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5f60 0x55d2ce39a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34f0bf3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f34e01b05b0 con 0x55d2ce3a5f60 2024-06-09T22:29:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34f0bf3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f34e01b20f0 con 0x55d2ce39ff20 2024-06-09T22:29:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34f0bf3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f34e01b0c80 con 0x55d2ce3a5640 2024-06-09T22:29:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34e490d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5f60 0x55d2ce39a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34e490d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5f60 0x55d2ce39a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56372/0 (socket says 172.21.15.104:56372) 2024-06-09T22:29:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34e490d700 1 -- 172.21.15.104:0/2147606584 learned_addr learned my addr 172.21.15.104:0/2147606584 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34dffff700 1 --2- 172.21.15.104:0/2147606584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce39ff20 0x55d2ce39f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34df7fe700 1 --2- 172.21.15.104:0/2147606584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5640 0x55d2ce39a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34e490d700 1 -- 172.21.15.104:0/2147606584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5640 msgr2=0x55d2ce39a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34e490d700 1 --2- 172.21.15.104:0/2147606584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5640 0x55d2ce39a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34e490d700 1 -- 172.21.15.104:0/2147606584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce39ff20 msgr2=0x55d2ce39f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:28.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34e490d700 1 --2- 172.21.15.104:0/2147606584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce39ff20 0x55d2ce39f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34e490d700 1 -- 172.21.15.104:0/2147606584 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2ce3a74b0 con 0x55d2ce3a5f60 2024-06-09T22:29:28.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34dffff700 1 --2- 172.21.15.104:0/2147606584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce39ff20 0x55d2ce39f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:28.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.765+0000 7f34e490d700 1 --2- 172.21.15.104:0/2147606584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5f60 0x55d2ce39a8c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f34d800f330 tx=0x7f34d800a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a8f662d8ec49e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:28.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.766+0000 7f34c5ffb700 1 -- 172.21.15.104:0/2147606584 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8003d80 con 0x55d2ce3a5f60 2024-06-09T22:29:28.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.766+0000 7f34c5ffb700 1 -- 172.21.15.104:0/2147606584 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34d8021070 con 0x55d2ce3a5f60 2024-06-09T22:29:28.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.766+0000 7f34c5ffb700 1 -- 172.21.15.104:0/2147606584 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8005700 con 0x55d2ce3a5f60 2024-06-09T22:29:28.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.766+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/2147606584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5f60 msgr2=0x55d2ce39a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:28.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.766+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/2147606584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5f60 0x55d2ce39a8c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f34d800f330 tx=0x7f34d800a000 comp rx=0 tx=0).stop 2024-06-09T22:29:28.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.766+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/2147606584 shutdown_connections 2024-06-09T22:29:28.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.766+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/2147606584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5f60 0x55d2ce39a8c0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.766+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/2147606584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce39ff20 0x55d2ce39f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.766+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/2147606584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce3a5640 0x55d2ce39a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.766+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/2147606584 >> 172.21.15.104:0/2147606584 conn(0x55d2ce2cacc0 msgr2=0x55d2ce39bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.766+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/2147606584 shutdown_connections 2024-06-09T22:29:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.766+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/2147606584 wait complete. 2024-06-09T22:29:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34f0bf3b80 1 -- start start 2024-06-09T22:29:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34f0bf3b80 1 Processor -- start 2024-06-09T22:29:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34f0bf3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce39ff20 0x55d2ce3b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34f0bf3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 0x55d2ce3b2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34f0bf3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce3a5f60 0x55d2ce3c0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34f0bf3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f34e01b1a20 con 0x55d2ce3a5640 2024-06-09T22:29:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34f0bf3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f34e01b1350 con 0x55d2ce3a5f60 2024-06-09T22:29:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34f0bf3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f34e01ae570 con 0x55d2ce39ff20 2024-06-09T22:29:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34f0bf3b80 10 client.0 _ll_register_callbacks cb 0x55d2ce28b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34df7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 0x55d2ce3b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34df7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 0x55d2ce3b2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56386/0 (socket says 172.21.15.104:56386) 2024-06-09T22:29:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34df7fe700 1 -- 172.21.15.104:0/730754431 learned_addr learned my addr 172.21.15.104:0/730754431 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34dffff700 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce39ff20 0x55d2ce3b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.767+0000 7f34e490d700 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce3a5f60 0x55d2ce3c0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34df7fe700 1 -- 172.21.15.104:0/730754431 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce39ff20 msgr2=0x55d2ce3b1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34df7fe700 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce39ff20 0x55d2ce3b1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34df7fe700 1 -- 172.21.15.104:0/730754431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce3a5f60 msgr2=0x55d2ce3c0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34df7fe700 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce3a5f60 0x55d2ce3c0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34df7fe700 1 -- 172.21.15.104:0/730754431 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2ce3c2020 con 0x55d2ce3a5640 2024-06-09T22:29:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34dffff700 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce39ff20 0x55d2ce3b1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34e490d700 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce3a5f60 0x55d2ce3c0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:29:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34df7fe700 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 0x55d2ce3b2350 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x55d2ce3653e0 tx=0x7f34d401e640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987809c5ffb58d2c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34bf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34bf7fe700 1 -- 172.21.15.104:0/730754431 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4027880 con 0x55d2ce3a5640 2024-06-09T22:29:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34bf7fe700 1 -- 172.21.15.104:0/730754431 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f34d400a940 con 0x55d2ce3a5640 2024-06-09T22:29:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/730754431 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34d8021070 con 0x55d2ce3a5640 2024-06-09T22:29:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34bf7fe700 1 -- 172.21.15.104:0/730754431 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f34d4043050 con 0x55d2ce3a5640 2024-06-09T22:29:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34f0bf3b80 10 client.17760 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/730754431 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d2ce423900 con 0x55d2ce3a5640 2024-06-09T22:29:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34bf7fe700 1 -- 172.21.15.104:0/730754431 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34d8021070 con 0x55d2ce3a5640 2024-06-09T22:29:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.768+0000 7f34bf7fe700 1 -- 172.21.15.104:0/730754431 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f34d4019070 con 0x55d2ce3a5640 2024-06-09T22:29:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.769+0000 7f34f0bf3b80 10 client.17760 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.769+0000 7f34f0bf3b80 10 client.17760 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:29:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.769+0000 7f34f0bf3b80 20 client.17760 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.769+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d2ce4240c0 0x55d2ce4264e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.769+0000 7f34f0bf3b80 4 client.17760 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:29:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.769+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/730754431 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d2ce2f9290 con 0x55d2ce4240c0 2024-06-09T22:29:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.769+0000 7f34dffff700 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d2ce4240c0 0x55d2ce4264e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.769+0000 7f34dffff700 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d2ce4240c0 0x55d2ce4264e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3135910422b1acee server_cookie=4c5ca6347bd63d9a in_seq=0 out_seq=0 2024-06-09T22:29:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.769+0000 7f34bf7fe700 10 client.17760 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:29:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.769+0000 7f34bf7fe700 1 -- 172.21.15.104:0/730754431 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d2ce2f9290 con 0x55d2ce4240c0 2024-06-09T22:29:28.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.769+0000 7f34bf7fe700 10 client.17760 handle_command_reply: tid=0 2024-06-09T22:29:28.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.769+0000 7f34f0bf3b80 1 client.17760 shutdown 2024-06-09T22:29:28.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/730754431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 msgr2=0x55d2ce3b2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:28.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 0x55d2ce3b2350 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x55d2ce3653e0 tx=0x7f34d401e640 comp rx=0 tx=0).stop 2024-06-09T22:29:28.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/730754431 shutdown_connections 2024-06-09T22:29:28.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x55d2ce4240c0 0x55d2ce4264e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2ce3a5f60 0x55d2ce3c0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2ce39ff20 0x55d2ce3b1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34f0bf3b80 1 --2- 172.21.15.104:0/730754431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2ce3a5640 0x55d2ce3b2350 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/730754431 >> 172.21.15.104:0/730754431 conn(0x55d2ce2cacc0 msgr2=0x55d2ce3a8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:28.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34bf7fe700 0 client.17760 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:29:28.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/730754431 shutdown_connections 2024-06-09T22:29:28.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34f0bf3b80 1 -- 172.21.15.104:0/730754431 wait complete. 2024-06-09T22:29:28.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34f0bf3b80 20 client.17760 trim_cache size 0 max 16384 2024-06-09T22:29:28.781 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:29:28.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34e6370700 1 -- 172.21.15.104:0/2728148599 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f34c80a2d90 msgr2=0x7f34c80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:28.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34e6370700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f34c80a2d90 0x7f34c80a5250 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f34e0078860 tx=0x7f34d400a380 comp rx=0 tx=0).stop 2024-06-09T22:29:28.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34e6370700 1 -- 172.21.15.104:0/2728148599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34e006abe0 msgr2=0x7f34e0101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34e6370700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34e006abe0 0x7f34e0101490 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f34d0000c00 tx=0x7f34d0003680 comp rx=0 tx=0).stop 2024-06-09T22:29:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34e6370700 1 -- 172.21.15.104:0/2728148599 shutdown_connections 2024-06-09T22:29:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34e6370700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f34c80a2d90 0x7f34c80a5250 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34e6370700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34e006abe0 0x7f34e0101490 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34e6370700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34e0108500 0x7f34e0101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34e6370700 1 --2- 172.21.15.104:0/2728148599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e006b5a0 0x7f34e01019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.770+0000 7f34e6370700 1 -- 172.21.15.104:0/2728148599 >> 172.21.15.104:0/2728148599 conn(0x7f34e0078360 msgr2=0x7f34e00ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.771+0000 7f34e6370700 1 -- 172.21.15.104:0/2728148599 shutdown_connections 2024-06-09T22:29:28.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:28.771+0000 7f34e6370700 1 -- 172.21.15.104:0/2728148599 wait complete. 2024-06-09T22:29:29.570 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-09T22:29:29.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:29 smithi104 ceph-mon[32389]: pgmap v262: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.3 MiB/s wr, 59 op/s 2024-06-09T22:29:29.811 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:29:29.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:29 smithi153 ceph-mon[36599]: pgmap v262: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.3 MiB/s wr, 59 op/s 2024-06-09T22:29:30.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:29 smithi194 ceph-mon[36856]: pgmap v262: 129 pgs: 129 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.3 MiB/s wr, 59 op/s 2024-06-09T22:29:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.763+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3452843269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc784107400 msgr2=0x7fc7841077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.763+0000 7fc78c3bf700 1 --2- 172.21.15.104:0/3452843269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc784107400 0x7fc7841077e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc77400b0a0 tx=0x7fc77400b3b0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.763+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3452843269 shutdown_connections 2024-06-09T22:29:31.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.763+0000 7fc78c3bf700 1 --2- 172.21.15.104:0/3452843269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc784107400 0x7fc7841077e0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.763+0000 7fc78c3bf700 1 --2- 172.21.15.104:0/3452843269 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc784101d50 0x7fc78410e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.763+0000 7fc78c3bf700 1 --2- 172.21.15.104:0/3452843269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc784101390 0x7fc784101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.763+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3452843269 >> 172.21.15.104:0/3452843269 conn(0x7fc784078340 msgr2=0x7fc784078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:31.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.763+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3452843269 shutdown_connections 2024-06-09T22:29:31.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.763+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3452843269 wait complete. 2024-06-09T22:29:31.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.764+0000 7fc78c3bf700 1 Processor -- start 2024-06-09T22:29:31.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.764+0000 7fc78c3bf700 1 -- start start 2024-06-09T22:29:31.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.764+0000 7fc78c3bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc784101390 0x7fc78419d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.764+0000 7fc78c3bf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc784101d50 0x7fc78419dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.764+0000 7fc78c3bf700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc784107400 0x7fc784197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.764+0000 7fc78c3bf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc784111670 con 0x7fc784101390 2024-06-09T22:29:31.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.764+0000 7fc78c3bf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841114f0 con 0x7fc784101d50 2024-06-09T22:29:31.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.764+0000 7fc78c3bf700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841117f0 con 0x7fc784107400 2024-06-09T22:29:31.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.764+0000 7fc78a15b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc784101390 0x7fc78419d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.764+0000 7fc78a15b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc784101390 0x7fc78419d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56406/0 (socket says 172.21.15.104:56406) 2024-06-09T22:29:31.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.764+0000 7fc78a15b700 1 -- 172.21.15.104:0/3686916340 learned_addr learned my addr 172.21.15.104:0/3686916340 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:31.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc78a95c700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc784107400 0x7fc784197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc78995a700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc784101d50 0x7fc78419dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc78a15b700 1 -- 172.21.15.104:0/3686916340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc784107400 msgr2=0x7fc784197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:31.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc78a15b700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc784107400 0x7fc784197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc78a15b700 1 -- 172.21.15.104:0/3686916340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc784101d50 msgr2=0x7fc78419dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:31.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc78a15b700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc784101d50 0x7fc78419dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc78a15b700 1 -- 172.21.15.104:0/3686916340 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc784198090 con 0x7fc784101390 2024-06-09T22:29:31.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc78a95c700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc784107400 0x7fc784197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:31.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc78995a700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc784101d50 0x7fc78419dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:29:31.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc78a15b700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc784101390 0x7fc78419d770 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc77400b070 tx=0x7fc774004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:31.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc7837fe700 1 -- 172.21.15.104:0/3686916340 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774021070 con 0x7fc784101390 2024-06-09T22:29:31.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc7837fe700 1 -- 172.21.15.104:0/3686916340 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc77401d930 con 0x7fc784101390 2024-06-09T22:29:31.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3686916340 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc784198320 con 0x7fc784101390 2024-06-09T22:29:31.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc7837fe700 1 -- 172.21.15.104:0/3686916340 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774005440 con 0x7fc784101390 2024-06-09T22:29:31.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.765+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3686916340 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7841a4640 con 0x7fc784101390 2024-06-09T22:29:31.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.767+0000 7fc7837fe700 1 -- 172.21.15.104:0/3686916340 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc774024070 con 0x7fc784101390 2024-06-09T22:29:31.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.768+0000 7fc7837fe700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc76c0a2d90 0x7fc76c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.768+0000 7fc78995a700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc76c0a2d90 0x7fc76c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.768+0000 7fc7837fe700 1 -- 172.21.15.104:0/3686916340 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fc77400f020 con 0x7fc784101390 2024-06-09T22:29:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.768+0000 7fc78995a700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc76c0a2d90 0x7fc76c0a5250 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc778005d90 tx=0x7fc778005d00 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc796c42b80 1 Processor -- start 2024-06-09T22:29:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc796c42b80 1 -- start start 2024-06-09T22:29:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc796c42b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a42870 0x559214a42c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc796c42b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a43190 0x559214a3cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc796c42b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a3d130 0x559214a3d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc796c42b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55921498c810 con 0x559214a42870 2024-06-09T22:29:31.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc796c42b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559214934720 con 0x559214a3d130 2024-06-09T22:29:31.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc796c42b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5592149887d0 con 0x559214a43190 2024-06-09T22:29:31.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc78a15b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a42870 0x559214a42c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc78a15b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a42870 0x559214a42c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56408/0 (socket says 172.21.15.104:56408) 2024-06-09T22:29:31.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc78a15b700 1 -- 172.21.15.104:0/3943865204 learned_addr learned my addr 172.21.15.104:0/3943865204 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:31.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc78a95c700 1 --2- 172.21.15.104:0/3943865204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a3d130 0x559214a3d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc78995a700 1 --2- 172.21.15.104:0/3943865204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a43190 0x559214a3cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc78a15b700 1 -- 172.21.15.104:0/3943865204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a43190 msgr2=0x559214a3cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:31.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc78a15b700 1 --2- 172.21.15.104:0/3943865204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a43190 0x559214a3cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc78a15b700 1 -- 172.21.15.104:0/3943865204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a3d130 msgr2=0x559214a3d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:31.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc78a15b700 1 --2- 172.21.15.104:0/3943865204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a3d130 0x559214a3d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc78a15b700 1 -- 172.21.15.104:0/3943865204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55921492af50 con 0x559214a42870 2024-06-09T22:29:31.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.769+0000 7fc78995a700 1 --2- 172.21.15.104:0/3943865204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a43190 0x559214a3cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:29:31.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.770+0000 7fc78a15b700 1 --2- 172.21.15.104:0/3943865204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a42870 0x559214a42c50 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc774099750 tx=0x7fc77409d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd6c232b975b50f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:31.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.770+0000 7fc780d70700 1 -- 172.21.15.104:0/3943865204 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774097e60 con 0x559214a42870 2024-06-09T22:29:31.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.770+0000 7fc780d70700 1 -- 172.21.15.104:0/3943865204 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7740a5ed0 con 0x559214a42870 2024-06-09T22:29:31.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.770+0000 7fc780d70700 1 -- 172.21.15.104:0/3943865204 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7740bd0f0 con 0x559214a42870 2024-06-09T22:29:31.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 -- 172.21.15.104:0/3943865204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a42870 msgr2=0x559214a42c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:31.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 --2- 172.21.15.104:0/3943865204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a42870 0x559214a42c50 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc774099750 tx=0x7fc77409d9f0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 -- 172.21.15.104:0/3943865204 shutdown_connections 2024-06-09T22:29:31.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 --2- 172.21.15.104:0/3943865204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a42870 0x559214a42c50 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 --2- 172.21.15.104:0/3943865204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a3d130 0x559214a3d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 --2- 172.21.15.104:0/3943865204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a43190 0x559214a3cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 -- 172.21.15.104:0/3943865204 >> 172.21.15.104:0/3943865204 conn(0x559214965cc0 msgr2=0x559214a35850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:31.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 -- 172.21.15.104:0/3943865204 shutdown_connections 2024-06-09T22:29:31.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 -- 172.21.15.104:0/3943865204 wait complete. 2024-06-09T22:29:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 -- start start 2024-06-09T22:29:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 Processor -- start 2024-06-09T22:29:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 0x559214a48fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a462a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a43190 0x559214a467e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559214a498f0 con 0x559214a3d130 2024-06-09T22:29:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55921492cbf0 con 0x559214a43190 2024-06-09T22:29:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.771+0000 7fc796c42b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559214a49a70 con 0x559214a42870 2024-06-09T22:29:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc796c42b80 10 client.0 _ll_register_callbacks cb 0x559214926920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78a15b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 0x559214a48fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78a15b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 0x559214a48fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56414/0 (socket says 172.21.15.104:56414) 2024-06-09T22:29:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78a15b700 1 -- 172.21.15.104:0/1778375486 learned_addr learned my addr 172.21.15.104:0/1778375486 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78995a700 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78a95c700 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a43190 0x559214a467e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78a15b700 1 -- 172.21.15.104:0/1778375486 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 msgr2=0x559214a462a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78a15b700 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a462a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78a15b700 1 -- 172.21.15.104:0/1778375486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a43190 msgr2=0x559214a467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78a15b700 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a43190 0x559214a467e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78a15b700 1 -- 172.21.15.104:0/1778375486 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592148db8d0 con 0x559214a3d130 2024-06-09T22:29:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78995a700 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a462a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78a95c700 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a43190 0x559214a467e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc78a15b700 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 0x559214a48fa0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc77409d280 tx=0x7fc7740a1760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e3d31bcae31198 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc769ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc769ffb700 1 -- 172.21.15.104:0/1778375486 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774097e60 con 0x559214a3d130 2024-06-09T22:29:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc769ffb700 1 -- 172.21.15.104:0/1778375486 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fc77401dae0 con 0x559214a3d130 2024-06-09T22:29:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc796c42b80 1 -- 172.21.15.104:0/1778375486 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7740a5ed0 con 0x559214a3d130 2024-06-09T22:29:31.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc769ffb700 1 -- 172.21.15.104:0/1778375486 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc7740bd0f0 con 0x559214a3d130 2024-06-09T22:29:31.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc796c42b80 10 client.17772 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:31.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.772+0000 7fc796c42b80 1 -- 172.21.15.104:0/1778375486 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559214a498f0 con 0x559214a3d130 2024-06-09T22:29:31.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.773+0000 7fc769ffb700 1 -- 172.21.15.104:0/1778375486 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7740a5ed0 con 0x559214a3d130 2024-06-09T22:29:31.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.773+0000 7fc769ffb700 1 -- 172.21.15.104:0/1778375486 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc774011070 con 0x559214a3d130 2024-06-09T22:29:31.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.773+0000 7fc796c42b80 10 client.17772 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:31.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.773+0000 7fc796c42b80 10 client.17772 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:29:31.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.773+0000 7fc796c42b80 20 client.17772 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:31.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.773+0000 7fc796c42b80 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x559214a56510 0x559214a58920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.773+0000 7fc796c42b80 4 client.17772 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:29:31.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.773+0000 7fc796c42b80 1 -- 172.21.15.104:0/1778375486 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559214999d10 con 0x559214a56510 2024-06-09T22:29:31.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.773+0000 7fc78995a700 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x559214a56510 0x559214a58920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc78995a700 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x559214a56510 0x559214a58920 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e58b9c32cb72afb server_cookie=9ea4221a58e7d96e in_seq=0 out_seq=0 2024-06-09T22:29:31.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc769ffb700 10 client.17772 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:29:31.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc769ffb700 1 -- 172.21.15.104:0/1778375486 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x559214999d10 con 0x559214a56510 2024-06-09T22:29:31.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc769ffb700 10 client.17772 handle_command_reply: tid=0 2024-06-09T22:29:31.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc796c42b80 1 client.17772 shutdown 2024-06-09T22:29:31.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc796c42b80 1 -- 172.21.15.104:0/1778375486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 msgr2=0x559214a48fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:31.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc796c42b80 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 0x559214a48fa0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc77409d280 tx=0x7fc7740a1760 comp rx=0 tx=0).stop 2024-06-09T22:29:31.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc796c42b80 1 -- 172.21.15.104:0/1778375486 shutdown_connections 2024-06-09T22:29:31.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc796c42b80 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x559214a56510 0x559214a58920 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc796c42b80 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 0x559214a48fa0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc796c42b80 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a43190 0x559214a467e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc769ffb700 0 client.17772 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:29:31.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc796c42b80 1 --2- 172.21.15.104:0/1778375486 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a462a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.774+0000 7fc796c42b80 1 -- 172.21.15.104:0/1778375486 >> 172.21.15.104:0/1778375486 conn(0x559214965cc0 msgr2=0x559214a399d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:31.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.775+0000 7fc796c42b80 1 -- 172.21.15.104:0/1778375486 shutdown_connections 2024-06-09T22:29:31.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.775+0000 7fc796c42b80 1 -- 172.21.15.104:0/1778375486 wait complete. 2024-06-09T22:29:31.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.775+0000 7fc796c42b80 20 client.17772 trim_cache size 0 max 16384 2024-06-09T22:29:31.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.790+0000 7fc796c42b80 1 Processor -- start 2024-06-09T22:29:31.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.790+0000 7fc796c42b80 1 -- start start 2024-06-09T22:29:31.792 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:31 smithi104 ceph-mon[32389]: pgmap v263: 129 pgs: 129 active+clean; 250 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-06-09T22:29:31.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.790+0000 7fc796c42b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a3d130 0x559214a52b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.790+0000 7fc796c42b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a43f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.790+0000 7fc796c42b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a43190 0x559214a40920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.790+0000 7fc796c42b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841ac390 con 0x559214a43190 2024-06-09T22:29:31.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.790+0000 7fc796c42b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841aded0 con 0x559214a3d130 2024-06-09T22:29:31.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.790+0000 7fc796c42b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841aca60 con 0x559214a42870 2024-06-09T22:29:31.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.790+0000 7fc78a95c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a43190 0x559214a40920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.790+0000 7fc78a95c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a43190 0x559214a40920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56430/0 (socket says 172.21.15.104:56430) 2024-06-09T22:29:31.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.790+0000 7fc78a95c700 1 -- 172.21.15.104:0/1655968151 learned_addr learned my addr 172.21.15.104:0/1655968151 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:31.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.790+0000 7fc78a15b700 1 --2- 172.21.15.104:0/1655968151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a3d130 0x559214a52b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.791+0000 7fc78995a700 1 --2- 172.21.15.104:0/1655968151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a43f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.791+0000 7fc78a95c700 1 -- 172.21.15.104:0/1655968151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 msgr2=0x559214a43f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:31.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.791+0000 7fc78a95c700 1 --2- 172.21.15.104:0/1655968151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a43f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.791+0000 7fc78a95c700 1 -- 172.21.15.104:0/1655968151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a3d130 msgr2=0x559214a52b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:31.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.791+0000 7fc78a95c700 1 --2- 172.21.15.104:0/1655968151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a3d130 0x559214a52b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.791+0000 7fc78a95c700 1 -- 172.21.15.104:0/1655968151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559214a46b30 con 0x559214a43190 2024-06-09T22:29:31.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.791+0000 7fc78995a700 1 --2- 172.21.15.104:0/1655968151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a43f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:29:31.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.792+0000 7fc78a95c700 1 --2- 172.21.15.104:0/1655968151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a43190 0x559214a40920 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00dd80 tx=0x7fc77c012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1591e4efe4886a7 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:31.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.792+0000 7fc76bfff700 1 -- 172.21.15.104:0/1655968151 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c020070 con 0x559214a43190 2024-06-09T22:29:31.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.792+0000 7fc76bfff700 1 -- 172.21.15.104:0/1655968151 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc77c00de10 con 0x559214a43190 2024-06-09T22:29:31.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.792+0000 7fc76bfff700 1 -- 172.21.15.104:0/1655968151 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c010870 con 0x559214a43190 2024-06-09T22:29:31.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.793+0000 7fc796c42b80 1 -- 172.21.15.104:0/1655968151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a43190 msgr2=0x559214a40920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:31.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.793+0000 7fc796c42b80 1 --2- 172.21.15.104:0/1655968151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a43190 0x559214a40920 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00dd80 tx=0x7fc77c012730 comp rx=0 tx=0).stop 2024-06-09T22:29:31.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.793+0000 7fc796c42b80 1 -- 172.21.15.104:0/1655968151 shutdown_connections 2024-06-09T22:29:31.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.793+0000 7fc796c42b80 1 --2- 172.21.15.104:0/1655968151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a43190 0x559214a40920 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.793+0000 7fc796c42b80 1 --2- 172.21.15.104:0/1655968151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a3d130 0x559214a52b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.793+0000 7fc796c42b80 1 --2- 172.21.15.104:0/1655968151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a43f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.793+0000 7fc796c42b80 1 -- 172.21.15.104:0/1655968151 >> 172.21.15.104:0/1655968151 conn(0x559214965cc0 msgr2=0x559214a2ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:31.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.793+0000 7fc796c42b80 1 -- 172.21.15.104:0/1655968151 shutdown_connections 2024-06-09T22:29:31.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.793+0000 7fc796c42b80 1 -- 172.21.15.104:0/1655968151 wait complete. 2024-06-09T22:29:31.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.793+0000 7fc796c42b80 1 -- start start 2024-06-09T22:29:31.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.793+0000 7fc796c42b80 1 Processor -- start 2024-06-09T22:29:31.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc796c42b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 0x559214a3a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc796c42b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a3af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc796c42b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a43190 0x559214a4db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc796c42b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841ad800 con 0x559214a3d130 2024-06-09T22:29:31.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc796c42b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841ad130 con 0x559214a43190 2024-06-09T22:29:31.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc796c42b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841aa350 con 0x559214a42870 2024-06-09T22:29:31.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc796c42b80 10 client.0 _ll_register_callbacks cb 0x559214926920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:31.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc78a15b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 0x559214a3a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc78a15b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 0x559214a3a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56444/0 (socket says 172.21.15.104:56444) 2024-06-09T22:29:31.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc78a15b700 1 -- 172.21.15.104:0/2129941252 learned_addr learned my addr 172.21.15.104:0/2129941252 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:31.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc78a95c700 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a43190 0x559214a4db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc78a15b700 1 -- 172.21.15.104:0/2129941252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 msgr2=0x559214a3af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:31.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc78a15b700 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a3af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc78a15b700 1 -- 172.21.15.104:0/2129941252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a43190 msgr2=0x559214a4db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:31.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc78a15b700 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a43190 0x559214a4db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc78a15b700 1 -- 172.21.15.104:0/2129941252 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559214a4e070 con 0x559214a3d130 2024-06-09T22:29:31.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc78a95c700 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a43190 0x559214a4db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:29:31.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc78a15b700 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 0x559214a3a9e0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc77409d730 tx=0x7fc77409d1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0c3dc3cb274ea server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:31.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc769ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:31.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.794+0000 7fc769ffb700 1 -- 172.21.15.104:0/2129941252 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774097e60 con 0x559214a3d130 2024-06-09T22:29:31.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc769ffb700 1 -- 172.21.15.104:0/2129941252 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fc7740a5a40 con 0x559214a3d130 2024-06-09T22:29:31.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc796c42b80 1 -- 172.21.15.104:0/2129941252 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc77c00de10 con 0x559214a3d130 2024-06-09T22:29:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc769ffb700 1 -- 172.21.15.104:0/2129941252 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc7740a5ed0 con 0x559214a3d130 2024-06-09T22:29:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc796c42b80 10 client.17778 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc796c42b80 1 -- 172.21.15.104:0/2129941252 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559214a4e070 con 0x559214a3d130 2024-06-09T22:29:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc769ffb700 1 -- 172.21.15.104:0/2129941252 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7740a5c40 con 0x559214a3d130 2024-06-09T22:29:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc769ffb700 1 -- 172.21.15.104:0/2129941252 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc774011040 con 0x559214a3d130 2024-06-09T22:29:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc796c42b80 10 client.17778 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc796c42b80 10 client.17778 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:29:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc796c42b80 20 client.17778 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc796c42b80 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x559214ac7f50 0x559214aca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc796c42b80 4 client.17778 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:29:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.795+0000 7fc796c42b80 1 -- 172.21.15.104:0/2129941252 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fc77c00de10 con 0x559214ac7f50 2024-06-09T22:29:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc78995a700 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x559214ac7f50 0x559214aca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc769ffb700 10 client.17778 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:29:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc78995a700 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x559214ac7f50 0x559214aca340 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=af124657f54a1e35 server_cookie=1b5dd1e6eb883f7f in_seq=0 out_seq=0 2024-06-09T22:29:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc769ffb700 1 -- 172.21.15.104:0/2129941252 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fc77c00de10 con 0x559214ac7f50 2024-06-09T22:29:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc769ffb700 10 client.17778 handle_command_reply: tid=0 2024-06-09T22:29:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc796c42b80 1 client.17778 shutdown 2024-06-09T22:29:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc796c42b80 1 -- 172.21.15.104:0/2129941252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 msgr2=0x559214a3a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc796c42b80 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 0x559214a3a9e0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc77409d730 tx=0x7fc77409d1d0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc796c42b80 1 -- 172.21.15.104:0/2129941252 shutdown_connections 2024-06-09T22:29:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc796c42b80 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x559214ac7f50 0x559214aca340 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc769ffb700 0 client.17778 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:29:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc796c42b80 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559214a3d130 0x559214a3a9e0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc796c42b80 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559214a43190 0x559214a4db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc796c42b80 1 --2- 172.21.15.104:0/2129941252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559214a42870 0x559214a3af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.796+0000 7fc796c42b80 1 -- 172.21.15.104:0/2129941252 >> 172.21.15.104:0/2129941252 conn(0x559214965cc0 msgr2=0x559214a43e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.797+0000 7fc796c42b80 1 -- 172.21.15.104:0/2129941252 shutdown_connections 2024-06-09T22:29:31.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.797+0000 7fc796c42b80 1 -- 172.21.15.104:0/2129941252 wait complete. 2024-06-09T22:29:31.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.812+0000 7fc796c42b80 20 client.17778 trim_cache size 0 max 16384 2024-06-09T22:29:31.814 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:29:31.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.812+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3686916340 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc76c0a2d90 msgr2=0x7fc76c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:31.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.812+0000 7fc78c3bf700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc76c0a2d90 0x7fc76c0a5250 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc778005d90 tx=0x7fc778005d00 comp rx=0 tx=0).stop 2024-06-09T22:29:31.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.812+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3686916340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc784101390 msgr2=0x7fc78419d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:31.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.812+0000 7fc78c3bf700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc784101390 0x7fc78419d770 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc77400b070 tx=0x7fc774004ef0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.812+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3686916340 shutdown_connections 2024-06-09T22:29:31.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.812+0000 7fc78c3bf700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc76c0a2d90 0x7fc76c0a5250 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.812+0000 7fc78c3bf700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc784101390 0x7fc78419d770 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.812+0000 7fc78c3bf700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc784107400 0x7fc784197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.812+0000 7fc78c3bf700 1 --2- 172.21.15.104:0/3686916340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc784101d50 0x7fc78419dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:31.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.812+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3686916340 >> 172.21.15.104:0/3686916340 conn(0x7fc784078340 msgr2=0x7fc784100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:31.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.813+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3686916340 shutdown_connections 2024-06-09T22:29:31.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:31.813+0000 7fc78c3bf700 1 -- 172.21.15.104:0/3686916340 wait complete. 2024-06-09T22:29:31.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:31 smithi153 ceph-mon[36599]: pgmap v263: 129 pgs: 129 active+clean; 250 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-06-09T22:29:32.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:31 smithi194 ceph-mon[36856]: pgmap v263: 129 pgs: 129 active+clean; 250 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-06-09T22:29:32.595 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-09T22:29:32.835 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:29:33.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:33 smithi153 ceph-mon[36599]: pgmap v264: 129 pgs: 129 active+clean; 250 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.3 MiB/s wr, 56 op/s 2024-06-09T22:29:34.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:33 smithi104 ceph-mon[32389]: pgmap v264: 129 pgs: 129 active+clean; 250 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.3 MiB/s wr, 56 op/s 2024-06-09T22:29:34.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:33 smithi194 ceph-mon[36856]: pgmap v264: 129 pgs: 129 active+clean; 250 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.3 MiB/s wr, 56 op/s 2024-06-09T22:29:34.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.671+0000 7f5200320700 1 -- 172.21.15.104:0/2803528099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f8101d50 msgr2=0x7f51f810e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:34.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.671+0000 7f5200320700 1 --2- 172.21.15.104:0/2803528099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f8101d50 0x7f51f810e240 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009990 tx=0x7f51f0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.671+0000 7f5200320700 1 -- 172.21.15.104:0/2803528099 shutdown_connections 2024-06-09T22:29:34.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.671+0000 7f5200320700 1 --2- 172.21.15.104:0/2803528099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f8101d50 0x7f51f810e240 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.671+0000 7f5200320700 1 --2- 172.21.15.104:0/2803528099 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51f8107400 0x7f51f81077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.671+0000 7f5200320700 1 --2- 172.21.15.104:0/2803528099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f8101390 0x7f51f8101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.671+0000 7f5200320700 1 -- 172.21.15.104:0/2803528099 >> 172.21.15.104:0/2803528099 conn(0x7f51f8078340 msgr2=0x7f51f8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:34.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.671+0000 7f5200320700 1 -- 172.21.15.104:0/2803528099 shutdown_connections 2024-06-09T22:29:34.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.671+0000 7f5200320700 1 -- 172.21.15.104:0/2803528099 wait complete. 2024-06-09T22:29:34.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f5200320700 1 Processor -- start 2024-06-09T22:29:34.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f5200320700 1 -- start start 2024-06-09T22:29:34.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f5200320700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f8101390 0x7f51f8103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f5200320700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51f8101d50 0x7f51f81035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f5200320700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f8107400 0x7f51f8103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f5200320700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f51f81116f0 con 0x7f51f8107400 2024-06-09T22:29:34.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f5200320700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f51f8111570 con 0x7f51f8101390 2024-06-09T22:29:34.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f5200320700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f51f8111870 con 0x7f51f8101d50 2024-06-09T22:29:34.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f51fe8bd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f8107400 0x7f51f8103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f51fe8bd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f8107400 0x7f51f8103af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56466/0 (socket says 172.21.15.104:56466) 2024-06-09T22:29:34.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f51fe8bd700 1 -- 172.21.15.104:0/387731406 learned_addr learned my addr 172.21.15.104:0/387731406 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:34.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f8101390 0x7f51f8103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.672+0000 7f51fd8bb700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51f8101d50 0x7f51f81035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.673+0000 7f51fe8bd700 1 -- 172.21.15.104:0/387731406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51f8101d50 msgr2=0x7f51f81035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:34.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.673+0000 7f51fe8bd700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51f8101d50 0x7f51f81035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.673+0000 7f51fe8bd700 1 -- 172.21.15.104:0/387731406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f8101390 msgr2=0x7f51f8103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:34.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.673+0000 7f51fe8bd700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f8101390 0x7f51f8103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.673+0000 7f51fe8bd700 1 -- 172.21.15.104:0/387731406 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f81a7140 con 0x7f51f8107400 2024-06-09T22:29:34.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.673+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f8101390 0x7f51f8103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:34.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.673+0000 7f51fe8bd700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f8107400 0x7f51f8103af0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f51f00047b0 tx=0x7f51f0009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:34.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.673+0000 7f51f77fe700 1 -- 172.21.15.104:0/387731406 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f0004830 con 0x7f51f8107400 2024-06-09T22:29:34.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.673+0000 7f51f77fe700 1 -- 172.21.15.104:0/387731406 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f0005020 con 0x7f51f8107400 2024-06-09T22:29:34.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.673+0000 7f5200320700 1 -- 172.21.15.104:0/387731406 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f81a73d0 con 0x7f51f8107400 2024-06-09T22:29:34.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.673+0000 7f51f77fe700 1 -- 172.21.15.104:0/387731406 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f0003e60 con 0x7f51f8107400 2024-06-09T22:29:34.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.673+0000 7f5200320700 1 -- 172.21.15.104:0/387731406 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f81a7770 con 0x7f51f8107400 2024-06-09T22:29:34.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.675+0000 7f51f77fe700 1 -- 172.21.15.104:0/387731406 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f51f00051d0 con 0x7f51f8107400 2024-06-09T22:29:34.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.676+0000 7f51f77fe700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f51e00a2d90 0x7f51e00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.676+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f51e00a2d90 0x7f51e00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.676+0000 7f51f77fe700 1 -- 172.21.15.104:0/387731406 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f51f00d0720 con 0x7f51f8107400 2024-06-09T22:29:34.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.676+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f51e00a2d90 0x7f51e00a5250 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009fa0 tx=0x7f51e8009570 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:34.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.676+0000 7f520aba3b80 1 Processor -- start 2024-06-09T22:29:34.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f520aba3b80 1 -- start start 2024-06-09T22:29:34.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f520aba3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7655f60 0x55abb7656340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f520aba3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7656880 0x55abb7650260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f520aba3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb7650bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f520aba3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55abb7542f60 con 0x55abb76507a0 2024-06-09T22:29:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f520aba3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55abb75a1850 con 0x55abb7656880 2024-06-09T22:29:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f520aba3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55abb7509460 con 0x55abb7655f60 2024-06-09T22:29:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f51fe8bd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb7650bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f51fe8bd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb7650bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56470/0 (socket says 172.21.15.104:56470) 2024-06-09T22:29:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f51fe8bd700 1 -- 172.21.15.104:0/2113491722 learned_addr learned my addr 172.21.15.104:0/2113491722 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f51fd8bb700 1 --2- 172.21.15.104:0/2113491722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7656880 0x55abb7650260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/2113491722 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7655f60 0x55abb7656340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f51fe8bd700 1 -- 172.21.15.104:0/2113491722 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7655f60 msgr2=0x55abb7656340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f51fe8bd700 1 --2- 172.21.15.104:0/2113491722 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7655f60 0x55abb7656340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f51fe8bd700 1 -- 172.21.15.104:0/2113491722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7656880 msgr2=0x55abb7650260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f51fe8bd700 1 --2- 172.21.15.104:0/2113491722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7656880 0x55abb7650260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f51fe8bd700 1 -- 172.21.15.104:0/2113491722 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abb74f08d0 con 0x55abb76507a0 2024-06-09T22:29:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f51fd8bb700 1 --2- 172.21.15.104:0/2113491722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7656880 0x55abb7650260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.677+0000 7f51fe8bd700 1 --2- 172.21.15.104:0/2113491722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb7650bc0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f51f009c890 tx=0x7f51f009c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eba8879d5c34908 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.678+0000 7f51f4d70700 1 -- 172.21.15.104:0/2113491722 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f00a0cb0 con 0x55abb76507a0 2024-06-09T22:29:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.678+0000 7f51f4d70700 1 -- 172.21.15.104:0/2113491722 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f00a54e0 con 0x55abb76507a0 2024-06-09T22:29:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.678+0000 7f51f4d70700 1 -- 172.21.15.104:0/2113491722 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f00d0530 con 0x55abb76507a0 2024-06-09T22:29:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.678+0000 7f520aba3b80 1 -- 172.21.15.104:0/2113491722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 msgr2=0x55abb7650bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.678+0000 7f520aba3b80 1 --2- 172.21.15.104:0/2113491722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb7650bc0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f51f009c890 tx=0x7f51f009c350 comp rx=0 tx=0).stop 2024-06-09T22:29:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.678+0000 7f520aba3b80 1 -- 172.21.15.104:0/2113491722 shutdown_connections 2024-06-09T22:29:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.678+0000 7f520aba3b80 1 --2- 172.21.15.104:0/2113491722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb7650bc0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.678+0000 7f520aba3b80 1 --2- 172.21.15.104:0/2113491722 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7655f60 0x55abb7656340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.678+0000 7f520aba3b80 1 --2- 172.21.15.104:0/2113491722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7656880 0x55abb7650260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.678+0000 7f520aba3b80 1 -- 172.21.15.104:0/2113491722 >> 172.21.15.104:0/2113491722 conn(0x55abb757d080 msgr2=0x55abb764b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.679+0000 7f520aba3b80 1 -- 172.21.15.104:0/2113491722 shutdown_connections 2024-06-09T22:29:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.679+0000 7f520aba3b80 1 -- 172.21.15.104:0/2113491722 wait complete. 2024-06-09T22:29:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.679+0000 7f520aba3b80 1 -- start start 2024-06-09T22:29:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.679+0000 7f520aba3b80 1 Processor -- start 2024-06-09T22:29:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.679+0000 7f520aba3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb765c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.679+0000 7f520aba3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7655f60 0x55abb7659970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.679+0000 7f520aba3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb7659eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f520aba3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55abb7541fc0 con 0x55abb76507a0 2024-06-09T22:29:34.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f520aba3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55abb759d810 con 0x55abb7655f60 2024-06-09T22:29:34.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f520aba3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55abb765d010 con 0x55abb7656880 2024-06-09T22:29:34.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f520aba3b80 10 client.0 _ll_register_callbacks cb 0x55abb75375b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:34.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51fe0bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb765c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51fe0bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb765c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56486/0 (socket says 172.21.15.104:56486) 2024-06-09T22:29:34.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51fe0bc700 1 -- 172.21.15.104:0/2725389417 learned_addr learned my addr 172.21.15.104:0/2725389417 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51fd8bb700 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7655f60 0x55abb7659970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51fe8bd700 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb7659eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51fe0bc700 1 -- 172.21.15.104:0/2725389417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 msgr2=0x55abb7659eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb7659eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51fe0bc700 1 -- 172.21.15.104:0/2725389417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7655f60 msgr2=0x55abb7659970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7655f60 0x55abb7659970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51fe0bc700 1 -- 172.21.15.104:0/2725389417 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abb7567750 con 0x55abb76507a0 2024-06-09T22:29:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51fe8bd700 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb7659eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:29:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb765c5d0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f51e8017c00 tx=0x7f51e8004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d102403a920c7c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51ddffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51ddffb700 1 -- 172.21.15.104:0/2725389417 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e80167b0 con 0x55abb76507a0 2024-06-09T22:29:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f51ddffb700 1 -- 172.21.15.104:0/2725389417 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f51e8016950 con 0x55abb76507a0 2024-06-09T22:29:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.680+0000 7f520aba3b80 1 -- 172.21.15.104:0/2725389417 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51f00a54e0 con 0x55abb76507a0 2024-06-09T22:29:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.681+0000 7f51ddffb700 1 -- 172.21.15.104:0/2725389417 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f51e8030050 con 0x55abb76507a0 2024-06-09T22:29:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.681+0000 7f520aba3b80 10 client.17790 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.681+0000 7f520aba3b80 1 -- 172.21.15.104:0/2725389417 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55abb7667730 con 0x55abb76507a0 2024-06-09T22:29:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.681+0000 7f51ddffb700 1 -- 172.21.15.104:0/2725389417 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f00a54e0 con 0x55abb76507a0 2024-06-09T22:29:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.681+0000 7f51ddffb700 1 -- 172.21.15.104:0/2725389417 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f51e802b3b0 con 0x55abb76507a0 2024-06-09T22:29:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.681+0000 7f520aba3b80 10 client.17790 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.681+0000 7f520aba3b80 10 client.17790 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:29:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.681+0000 7f520aba3b80 20 client.17790 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.681+0000 7f520aba3b80 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55abb7669e10 0x55abb766c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.681+0000 7f520aba3b80 4 client.17790 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:29:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.681+0000 7f520aba3b80 1 -- 172.21.15.104:0/2725389417 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abb75aed50 con 0x55abb7669e10 2024-06-09T22:29:34.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.681+0000 7f51fd8bb700 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55abb7669e10 0x55abb766c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.682+0000 7f51fd8bb700 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55abb7669e10 0x55abb766c220 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f192b84fe8c570e server_cookie=74e91bf014afa38a in_seq=0 out_seq=0 2024-06-09T22:29:34.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.682+0000 7f51ddffb700 10 client.17790 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:34.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f51ddffb700 1 -- 172.21.15.104:0/2725389417 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55abb75aed50 con 0x55abb7669e10 2024-06-09T22:29:34.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f51ddffb700 10 client.17790 handle_command_reply: tid=0 2024-06-09T22:29:34.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f520aba3b80 1 client.17790 shutdown 2024-06-09T22:29:34.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f520aba3b80 1 -- 172.21.15.104:0/2725389417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 msgr2=0x55abb765c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:34.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f520aba3b80 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb765c5d0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f51e8017c00 tx=0x7f51e8004be0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f520aba3b80 1 -- 172.21.15.104:0/2725389417 shutdown_connections 2024-06-09T22:29:34.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f520aba3b80 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55abb7669e10 0x55abb766c220 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f520aba3b80 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb765c5d0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f520aba3b80 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb7659eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f51ddffb700 0 client.17790 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:34.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f520aba3b80 1 --2- 172.21.15.104:0/2725389417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7655f60 0x55abb7659970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f520aba3b80 1 -- 172.21.15.104:0/2725389417 >> 172.21.15.104:0/2725389417 conn(0x55abb757d080 msgr2=0x55abb764a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:34.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f520aba3b80 1 -- 172.21.15.104:0/2725389417 shutdown_connections 2024-06-09T22:29:34.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.683+0000 7f520aba3b80 1 -- 172.21.15.104:0/2725389417 wait complete. 2024-06-09T22:29:34.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.684+0000 7f520aba3b80 20 client.17790 trim_cache size 0 max 16384 2024-06-09T22:29:34.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.698+0000 7f520aba3b80 1 Processor -- start 2024-06-09T22:29:34.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.698+0000 7f520aba3b80 1 -- start start 2024-06-09T22:29:34.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f520aba3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb7650260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f520aba3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7655f60 0x55abb76572d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f520aba3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb764ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f520aba3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f51f81b06e0 con 0x55abb76507a0 2024-06-09T22:29:34.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f520aba3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f51f81b2220 con 0x55abb7655f60 2024-06-09T22:29:34.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f520aba3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f51f81b0db0 con 0x55abb7656880 2024-06-09T22:29:34.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51fe0bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb7650260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51fe0bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb7650260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56494/0 (socket says 172.21.15.104:56494) 2024-06-09T22:29:34.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51fe0bc700 1 -- 172.21.15.104:0/488681867 learned_addr learned my addr 172.21.15.104:0/488681867 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:34.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51fe8bd700 1 --2- 172.21.15.104:0/488681867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb764ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51fd8bb700 1 --2- 172.21.15.104:0/488681867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7655f60 0x55abb76572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51fe0bc700 1 -- 172.21.15.104:0/488681867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 msgr2=0x55abb764ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/488681867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb764ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51fe0bc700 1 -- 172.21.15.104:0/488681867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7655f60 msgr2=0x55abb76572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/488681867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7655f60 0x55abb76572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51fe0bc700 1 -- 172.21.15.104:0/488681867 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abb7657ee0 con 0x55abb76507a0 2024-06-09T22:29:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51fe8bd700 1 --2- 172.21.15.104:0/488681867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb764ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/488681867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb7650260 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f51e8007160 tx=0x7f51e800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5822a4dc3461de3e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51dffff700 1 -- 172.21.15.104:0/488681867 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e8039070 con 0x55abb76507a0 2024-06-09T22:29:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51dffff700 1 -- 172.21.15.104:0/488681867 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e8027070 con 0x55abb76507a0 2024-06-09T22:29:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.699+0000 7f51dffff700 1 -- 172.21.15.104:0/488681867 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e8034770 con 0x55abb76507a0 2024-06-09T22:29:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.700+0000 7f520aba3b80 1 -- 172.21.15.104:0/488681867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 msgr2=0x55abb7650260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.700+0000 7f520aba3b80 1 --2- 172.21.15.104:0/488681867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb7650260 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f51e8007160 tx=0x7f51e800ca40 comp rx=0 tx=0).stop 2024-06-09T22:29:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.700+0000 7f520aba3b80 1 -- 172.21.15.104:0/488681867 shutdown_connections 2024-06-09T22:29:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.700+0000 7f520aba3b80 1 --2- 172.21.15.104:0/488681867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb76507a0 0x55abb7650260 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.700+0000 7f520aba3b80 1 --2- 172.21.15.104:0/488681867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb764ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.700+0000 7f520aba3b80 1 --2- 172.21.15.104:0/488681867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb7655f60 0x55abb76572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.700+0000 7f520aba3b80 1 -- 172.21.15.104:0/488681867 >> 172.21.15.104:0/488681867 conn(0x55abb757d080 msgr2=0x55abb76589c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.700+0000 7f520aba3b80 1 -- 172.21.15.104:0/488681867 shutdown_connections 2024-06-09T22:29:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.700+0000 7f520aba3b80 1 -- 172.21.15.104:0/488681867 wait complete. 2024-06-09T22:29:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.700+0000 7f520aba3b80 1 -- start start 2024-06-09T22:29:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f520aba3b80 1 Processor -- start 2024-06-09T22:29:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f520aba3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb76507a0 0x55abb7662970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f520aba3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb7655f60 0x55abb76713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f520aba3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb7671930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f520aba3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f51f81b1b50 con 0x55abb7655f60 2024-06-09T22:29:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f520aba3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f51f81b1480 con 0x55abb76507a0 2024-06-09T22:29:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f520aba3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f51f81ae6a0 con 0x55abb7656880 2024-06-09T22:29:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f520aba3b80 10 client.0 _ll_register_callbacks cb 0x55abb75375b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f51fd8bb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb7655f60 0x55abb76713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f51fd8bb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb7655f60 0x55abb76713f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56502/0 (socket says 172.21.15.104:56502) 2024-06-09T22:29:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f51fd8bb700 1 -- 172.21.15.104:0/3135553700 learned_addr learned my addr 172.21.15.104:0/3135553700 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb76507a0 0x55abb7662970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f51fe8bd700 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb7671930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f51fd8bb700 1 -- 172.21.15.104:0/3135553700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 msgr2=0x55abb7671930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f51fd8bb700 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb7671930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f51fd8bb700 1 -- 172.21.15.104:0/3135553700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb76507a0 msgr2=0x55abb7662970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f51fd8bb700 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb76507a0 0x55abb7662970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f51fd8bb700 1 -- 172.21.15.104:0/3135553700 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abb7672160 con 0x55abb7655f60 2024-06-09T22:29:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f51fd8bb700 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb7655f60 0x55abb76713f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x55abb765af40 tx=0x7f51ec012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22e23dc9db3fdca server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.701+0000 7f51ddffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f51ddffb700 1 -- 172.21.15.104:0/3135553700 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec004230 con 0x55abb7655f60 2024-06-09T22:29:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f51ddffb700 1 -- 172.21.15.104:0/3135553700 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f51ec0043d0 con 0x55abb7655f60 2024-06-09T22:29:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f520aba3b80 1 -- 172.21.15.104:0/3135553700 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e8027070 con 0x55abb7655f60 2024-06-09T22:29:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f51ddffb700 1 -- 172.21.15.104:0/3135553700 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f51ec02c050 con 0x55abb7655f60 2024-06-09T22:29:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f520aba3b80 10 client.17796 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f520aba3b80 1 -- 172.21.15.104:0/3135553700 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55abb76d4af0 con 0x55abb7655f60 2024-06-09T22:29:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f51ddffb700 1 -- 172.21.15.104:0/3135553700 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e8027070 con 0x55abb7655f60 2024-06-09T22:29:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f51ddffb700 1 -- 172.21.15.104:0/3135553700 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f51ec0247b0 con 0x55abb7655f60 2024-06-09T22:29:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f520aba3b80 10 client.17796 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f520aba3b80 10 client.17796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:29:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f520aba3b80 20 client.17796 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f520aba3b80 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55abb76d5280 0x55abb76d76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f520aba3b80 4 client.17796 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:29:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f520aba3b80 1 -- 172.21.15.104:0/3135553700 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55abb75a92d0 con 0x55abb76d5280 2024-06-09T22:29:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.702+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55abb76d5280 0x55abb76d76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f51ddffb700 10 client.17796 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f51fe0bc700 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55abb76d5280 0x55abb76d76a0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1f993240872a7be server_cookie=30e4a56b22c054a in_seq=0 out_seq=0 2024-06-09T22:29:34.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f51ddffb700 1 -- 172.21.15.104:0/3135553700 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55abb75a92d0 con 0x55abb76d5280 2024-06-09T22:29:34.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f51ddffb700 10 client.17796 handle_command_reply: tid=0 2024-06-09T22:29:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f520aba3b80 1 client.17796 shutdown 2024-06-09T22:29:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f520aba3b80 1 -- 172.21.15.104:0/3135553700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb7655f60 msgr2=0x55abb76713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f520aba3b80 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb7655f60 0x55abb76713f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x55abb765af40 tx=0x7f51ec012730 comp rx=0 tx=0).stop 2024-06-09T22:29:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f520aba3b80 1 -- 172.21.15.104:0/3135553700 shutdown_connections 2024-06-09T22:29:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f520aba3b80 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55abb76d5280 0x55abb76d76a0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f520aba3b80 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb7656880 0x55abb7671930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f520aba3b80 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55abb76507a0 0x55abb7662970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f520aba3b80 1 --2- 172.21.15.104:0/3135553700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55abb7655f60 0x55abb76713f0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f520aba3b80 1 -- 172.21.15.104:0/3135553700 >> 172.21.15.104:0/3135553700 conn(0x55abb757d080 msgr2=0x55abb7658130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f51ddffb700 0 client.17796 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f520aba3b80 1 -- 172.21.15.104:0/3135553700 shutdown_connections 2024-06-09T22:29:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f520aba3b80 1 -- 172.21.15.104:0/3135553700 wait complete. 2024-06-09T22:29:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.703+0000 7f520aba3b80 20 client.17796 trim_cache size 0 max 16384 2024-06-09T22:29:34.716 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:29:34.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.704+0000 7f5200320700 1 -- 172.21.15.104:0/387731406 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f51e00a2d90 msgr2=0x7f51e00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:34.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.704+0000 7f5200320700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f51e00a2d90 0x7f51e00a5250 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009fa0 tx=0x7f51e8009570 comp rx=0 tx=0).stop 2024-06-09T22:29:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.704+0000 7f5200320700 1 -- 172.21.15.104:0/387731406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f8107400 msgr2=0x7f51f8103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.704+0000 7f5200320700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f8107400 0x7f51f8103af0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f51f00047b0 tx=0x7f51f0009ee0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.704+0000 7f5200320700 1 -- 172.21.15.104:0/387731406 shutdown_connections 2024-06-09T22:29:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.704+0000 7f5200320700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f51e00a2d90 0x7f51e00a5250 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.704+0000 7f5200320700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f8107400 0x7f51f8103af0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.704+0000 7f5200320700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f8101390 0x7f51f8103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.704+0000 7f5200320700 1 --2- 172.21.15.104:0/387731406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51f8101d50 0x7f51f81035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.704+0000 7f5200320700 1 -- 172.21.15.104:0/387731406 >> 172.21.15.104:0/387731406 conn(0x7f51f8078340 msgr2=0x7f51f81005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:34.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.704+0000 7f5200320700 1 -- 172.21.15.104:0/387731406 shutdown_connections 2024-06-09T22:29:34.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:34.704+0000 7f5200320700 1 -- 172.21.15.104:0/387731406 wait complete. 2024-06-09T22:29:35.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:35 smithi104 ceph-mon[32389]: pgmap v265: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-06-09T22:29:35.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:35 smithi153 ceph-mon[36599]: pgmap v265: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-06-09T22:29:36.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:35 smithi194 ceph-mon[36856]: pgmap v265: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-06-09T22:29:36.420 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-09T22:29:36.420 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-09T22:29:36.666 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:29:37.983 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:37 smithi104 ceph-mon[32389]: pgmap v266: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-06-09T22:29:37.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:37 smithi153 ceph-mon[36599]: pgmap v266: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-06-09T22:29:38.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:37 smithi194 ceph-mon[36856]: pgmap v266: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-06-09T22:29:38.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.550+0000 7fdb8c39c700 1 -- 172.21.15.104:0/548953834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb84101d50 msgr2=0x7fdb8410e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:38.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.550+0000 7fdb8c39c700 1 --2- 172.21.15.104:0/548953834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb84101d50 0x7fdb8410e240 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00b0a0 tx=0x7fdb7c00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.550+0000 7fdb8c39c700 1 -- 172.21.15.104:0/548953834 shutdown_connections 2024-06-09T22:29:38.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.550+0000 7fdb8c39c700 1 --2- 172.21.15.104:0/548953834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb84101d50 0x7fdb8410e240 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.550+0000 7fdb8c39c700 1 --2- 172.21.15.104:0/548953834 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdb84107400 0x7fdb841077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.550+0000 7fdb8c39c700 1 --2- 172.21.15.104:0/548953834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101390 0x7fdb84101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.550+0000 7fdb8c39c700 1 -- 172.21.15.104:0/548953834 >> 172.21.15.104:0/548953834 conn(0x7fdb84078340 msgr2=0x7fdb84078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.550+0000 7fdb8c39c700 1 -- 172.21.15.104:0/548953834 shutdown_connections 2024-06-09T22:29:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.550+0000 7fdb8c39c700 1 -- 172.21.15.104:0/548953834 wait complete. 2024-06-09T22:29:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb8c39c700 1 Processor -- start 2024-06-09T22:29:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb8c39c700 1 -- start start 2024-06-09T22:29:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb8c39c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101390 0x7fdb8419d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb8c39c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb84101d50 0x7fdb8419dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb8c39c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdb84107400 0x7fdb841978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb8c39c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84111690 con 0x7fdb84101d50 2024-06-09T22:29:38.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb8c39c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84111510 con 0x7fdb84101390 2024-06-09T22:29:38.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb8c39c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84111810 con 0x7fdb84107400 2024-06-09T22:29:38.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb89937700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb84101d50 0x7fdb8419dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb89937700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb84101d50 0x7fdb8419dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45230/0 (socket says 172.21.15.104:45230) 2024-06-09T22:29:38.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb89937700 1 -- 172.21.15.104:0/3028800435 learned_addr learned my addr 172.21.15.104:0/3028800435 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:38.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb8a138700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101390 0x7fdb8419d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.551+0000 7fdb8a939700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdb84107400 0x7fdb841978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.552+0000 7fdb89937700 1 -- 172.21.15.104:0/3028800435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdb84107400 msgr2=0x7fdb841978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:38.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.552+0000 7fdb89937700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdb84107400 0x7fdb841978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.552+0000 7fdb89937700 1 -- 172.21.15.104:0/3028800435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101390 msgr2=0x7fdb8419d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:38.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.552+0000 7fdb89937700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101390 0x7fdb8419d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.552+0000 7fdb89937700 1 -- 172.21.15.104:0/3028800435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb84198030 con 0x7fdb84101d50 2024-06-09T22:29:38.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.552+0000 7fdb8a138700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101390 0x7fdb8419d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:38.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.552+0000 7fdb89937700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb84101d50 0x7fdb8419dc50 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000c920 tx=0x7fdb8000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:38.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.552+0000 7fdb7b7fe700 1 -- 172.21.15.104:0/3028800435 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb80007b00 con 0x7fdb84101d50 2024-06-09T22:29:38.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.552+0000 7fdb7b7fe700 1 -- 172.21.15.104:0/3028800435 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb80004d10 con 0x7fdb84101d50 2024-06-09T22:29:38.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.552+0000 7fdb8c39c700 1 -- 172.21.15.104:0/3028800435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb84198320 con 0x7fdb84101d50 2024-06-09T22:29:38.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.552+0000 7fdb7b7fe700 1 -- 172.21.15.104:0/3028800435 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8000f450 con 0x7fdb84101d50 2024-06-09T22:29:38.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.552+0000 7fdb8c39c700 1 -- 172.21.15.104:0/3028800435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb8406a170 con 0x7fdb84101d50 2024-06-09T22:29:38.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.555+0000 7fdb7b7fe700 1 -- 172.21.15.104:0/3028800435 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fdb80004750 con 0x7fdb84101d50 2024-06-09T22:29:38.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb7b7fe700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fdb700a2d90 0x7fdb700a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb96c1fb80 1 Processor -- start 2024-06-09T22:29:38.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb8a138700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fdb700a2d90 0x7fdb700a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb7b7fe700 1 -- 172.21.15.104:0/3028800435 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fdb800cf870 con 0x7fdb84101d50 2024-06-09T22:29:38.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb96c1fb80 1 -- start start 2024-06-09T22:29:38.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb96c1fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bff6c0 0x55eea6bffaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb96c1fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 0x55eea6bf9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb96c1fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bf9fa0 0x55eea6bfa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb96c1fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55eea6b4b890 con 0x55eea6bfffe0 2024-06-09T22:29:38.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb96c1fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55eea6af37a0 con 0x55eea6bff6c0 2024-06-09T22:29:38.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb96c1fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55eea6b47850 con 0x55eea6bf9fa0 2024-06-09T22:29:38.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb8a138700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fdb700a2d90 0x7fdb700a5250 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdb74005e10 tx=0x7fdb74005da0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:38.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb89937700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 0x55eea6bf9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb89937700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 0x55eea6bf9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45246/0 (socket says 172.21.15.104:45246) 2024-06-09T22:29:38.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb89937700 1 -- 172.21.15.104:0/961624513 learned_addr learned my addr 172.21.15.104:0/961624513 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:38.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb8a138700 1 --2- 172.21.15.104:0/961624513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bff6c0 0x55eea6bffaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.556+0000 7fdb8a939700 1 --2- 172.21.15.104:0/961624513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bf9fa0 0x55eea6bfa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.557+0000 7fdb89937700 1 -- 172.21.15.104:0/961624513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bf9fa0 msgr2=0x55eea6bfa3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:38.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.557+0000 7fdb89937700 1 --2- 172.21.15.104:0/961624513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bf9fa0 0x55eea6bfa3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.557+0000 7fdb89937700 1 -- 172.21.15.104:0/961624513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bff6c0 msgr2=0x55eea6bffaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:38.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.557+0000 7fdb89937700 1 --2- 172.21.15.104:0/961624513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bff6c0 0x55eea6bffaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.557+0000 7fdb89937700 1 -- 172.21.15.104:0/961624513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eea6ae9fd0 con 0x55eea6bfffe0 2024-06-09T22:29:38.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.557+0000 7fdb89937700 1 --2- 172.21.15.104:0/961624513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 0x55eea6bf9a60 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fdb80097e20 tx=0x7fdb8009b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f43277d269f66f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:38.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.557+0000 7fdb78d70700 1 -- 172.21.15.104:0/961624513 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8001d040 con 0x55eea6bfffe0 2024-06-09T22:29:38.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.557+0000 7fdb78d70700 1 -- 172.21.15.104:0/961624513 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb8009fd70 con 0x55eea6bfffe0 2024-06-09T22:29:38.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.557+0000 7fdb78d70700 1 -- 172.21.15.104:0/961624513 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb80098030 con 0x55eea6bfffe0 2024-06-09T22:29:38.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.558+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/961624513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 msgr2=0x55eea6bf9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:38.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.558+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/961624513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 0x55eea6bf9a60 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fdb80097e20 tx=0x7fdb8009b580 comp rx=0 tx=0).stop 2024-06-09T22:29:38.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.558+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/961624513 shutdown_connections 2024-06-09T22:29:38.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.558+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/961624513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bf9fa0 0x55eea6bfa3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.558+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/961624513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bff6c0 0x55eea6bffaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.558+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/961624513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 0x55eea6bf9a60 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.558+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/961624513 >> 172.21.15.104:0/961624513 conn(0x55eea6b24d40 msgr2=0x55eea6bf48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:38.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.558+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/961624513 shutdown_connections 2024-06-09T22:29:38.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.558+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/961624513 wait complete. 2024-06-09T22:29:38.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.558+0000 7fdb96c1fb80 1 -- start start 2024-06-09T22:29:38.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.558+0000 7fdb96c1fb80 1 Processor -- start 2024-06-09T22:29:38.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb96c1fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bf9fa0 0x55eea6c05db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb96c1fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bff6c0 0x55eea6c030b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb96c1fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 0x55eea6c035f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb96c1fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55eea6c06a50 con 0x55eea6bfffe0 2024-06-09T22:29:38.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb96c1fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55eea6aebc70 con 0x55eea6bff6c0 2024-06-09T22:29:38.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb96c1fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55eea6c06bd0 con 0x55eea6bf9fa0 2024-06-09T22:29:38.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb96c1fb80 10 client.0 _ll_register_callbacks cb 0x55eea6ae59a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:38.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb8a939700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 0x55eea6c035f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb8a939700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 0x55eea6c035f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45256/0 (socket says 172.21.15.104:45256) 2024-06-09T22:29:38.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb8a939700 1 -- 172.21.15.104:0/2917126751 learned_addr learned my addr 172.21.15.104:0/2917126751 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:38.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb89937700 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bff6c0 0x55eea6c030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb8a138700 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bf9fa0 0x55eea6c05db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb8a939700 1 -- 172.21.15.104:0/2917126751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bf9fa0 msgr2=0x55eea6c05db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:38.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb8a939700 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bf9fa0 0x55eea6c05db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb8a939700 1 -- 172.21.15.104:0/2917126751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bff6c0 msgr2=0x55eea6c030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:38.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb8a939700 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bff6c0 0x55eea6c030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb8a939700 1 -- 172.21.15.104:0/2917126751 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eea6a9a950 con 0x55eea6bfffe0 2024-06-09T22:29:38.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb8a939700 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 0x55eea6c035f0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c015000 tx=0x7fdb7c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760da5929bad952f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:38.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb6dffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:38.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb6dffb700 1 -- 172.21.15.104:0/2917126751 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c0076e0 con 0x55eea6bfffe0 2024-06-09T22:29:38.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb6dffb700 1 -- 172.21.15.104:0/2917126751 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fdb7c007880 con 0x55eea6bfffe0 2024-06-09T22:29:38.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.559+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2917126751 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb8009fd70 con 0x55eea6bfffe0 2024-06-09T22:29:38.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.560+0000 7fdb6dffb700 1 -- 172.21.15.104:0/2917126751 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdb7c02f050 con 0x55eea6bfffe0 2024-06-09T22:29:38.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.560+0000 7fdb96c1fb80 10 client.17808 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:38.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.560+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2917126751 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55eea6c113d0 con 0x55eea6bfffe0 2024-06-09T22:29:38.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.560+0000 7fdb6dffb700 1 -- 172.21.15.104:0/2917126751 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb8009fd70 con 0x55eea6bfffe0 2024-06-09T22:29:38.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.560+0000 7fdb6dffb700 1 -- 172.21.15.104:0/2917126751 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdb7c011070 con 0x55eea6bfffe0 2024-06-09T22:29:38.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.560+0000 7fdb96c1fb80 10 client.17808 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:38.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.560+0000 7fdb96c1fb80 10 client.17808 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:29:38.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.560+0000 7fdb96c1fb80 20 client.17808 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:38.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.560+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55eea6c13a10 0x55eea6c15e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.560+0000 7fdb96c1fb80 4 client.17808 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:29:38.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.560+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2917126751 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eea6b58d90 con 0x55eea6c13a10 2024-06-09T22:29:38.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.560+0000 7fdb8a138700 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55eea6c13a10 0x55eea6c15e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.561+0000 7fdb8a138700 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55eea6c13a10 0x55eea6c15e00 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0559e95e8e9ad7d server_cookie=f78f9207873ddcbd in_seq=0 out_seq=0 2024-06-09T22:29:38.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.561+0000 7fdb6dffb700 10 client.17808 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:38.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.561+0000 7fdb6dffb700 1 -- 172.21.15.104:0/2917126751 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55eea6b58d90 con 0x55eea6c13a10 2024-06-09T22:29:38.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.561+0000 7fdb6dffb700 10 client.17808 handle_command_reply: tid=0 2024-06-09T22:29:38.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.561+0000 7fdb96c1fb80 1 client.17808 shutdown 2024-06-09T22:29:38.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.561+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2917126751 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 msgr2=0x55eea6c035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:38.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.561+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 0x55eea6c035f0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c015000 tx=0x7fdb7c004c80 comp rx=0 tx=0).stop 2024-06-09T22:29:38.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.562+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2917126751 shutdown_connections 2024-06-09T22:29:38.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.562+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55eea6c13a10 0x55eea6c15e00 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.562+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bfffe0 0x55eea6c035f0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.562+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bf9fa0 0x55eea6c05db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.562+0000 7fdb6dffb700 0 client.17808 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:38.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.562+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/2917126751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bff6c0 0x55eea6c030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.562+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2917126751 >> 172.21.15.104:0/2917126751 conn(0x55eea6b24d40 msgr2=0x55eea6bf4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:38.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.562+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2917126751 shutdown_connections 2024-06-09T22:29:38.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.562+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2917126751 wait complete. 2024-06-09T22:29:38.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.562+0000 7fdb96c1fb80 20 client.17808 trim_cache size 0 max 16384 2024-06-09T22:29:38.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb96c1fb80 1 Processor -- start 2024-06-09T22:29:38.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb96c1fb80 1 -- start start 2024-06-09T22:29:38.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb96c1fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 0x55eea6bf9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb96c1fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bff6c0 0x55eea6c00b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb96c1fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bfffe0 0x55eea6c1b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb96c1fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841b0810 con 0x55eea6bf9fa0 2024-06-09T22:29:38.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb96c1fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841b2350 con 0x55eea6bfffe0 2024-06-09T22:29:38.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb96c1fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841b0ee0 con 0x55eea6bff6c0 2024-06-09T22:29:38.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb8a138700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 0x55eea6bf9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb8a138700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 0x55eea6bf9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45270/0 (socket says 172.21.15.104:45270) 2024-06-09T22:29:38.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb8a138700 1 -- 172.21.15.104:0/2757086004 learned_addr learned my addr 172.21.15.104:0/2757086004 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:38.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb8a939700 1 --2- 172.21.15.104:0/2757086004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bfffe0 0x55eea6c1b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb89937700 1 --2- 172.21.15.104:0/2757086004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bff6c0 0x55eea6c00b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb8a138700 1 -- 172.21.15.104:0/2757086004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bff6c0 msgr2=0x55eea6c00b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:38.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb8a138700 1 --2- 172.21.15.104:0/2757086004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bff6c0 0x55eea6c00b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb8a138700 1 -- 172.21.15.104:0/2757086004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bfffe0 msgr2=0x55eea6c1b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:38.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.582+0000 7fdb8a138700 1 --2- 172.21.15.104:0/2757086004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bfffe0 0x55eea6c1b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb8a138700 1 -- 172.21.15.104:0/2757086004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eea6bff1b0 con 0x55eea6bf9fa0 2024-06-09T22:29:38.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb89937700 1 --2- 172.21.15.104:0/2757086004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bff6c0 0x55eea6c00b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:29:38.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb8a138700 1 --2- 172.21.15.104:0/2757086004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 0x55eea6bf9a60 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x55eea6c16620 tx=0x7fdb74015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3af81203fd2c0fe server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:38.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb6ffff700 1 -- 172.21.15.104:0/2757086004 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb74021bd0 con 0x55eea6bf9fa0 2024-06-09T22:29:38.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb6ffff700 1 -- 172.21.15.104:0/2757086004 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb74021d70 con 0x55eea6bf9fa0 2024-06-09T22:29:38.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb6ffff700 1 -- 172.21.15.104:0/2757086004 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb740293f0 con 0x55eea6bf9fa0 2024-06-09T22:29:38.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2757086004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 msgr2=0x55eea6bf9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:38.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/2757086004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 0x55eea6bf9a60 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x55eea6c16620 tx=0x7fdb74015000 comp rx=0 tx=0).stop 2024-06-09T22:29:38.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2757086004 shutdown_connections 2024-06-09T22:29:38.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/2757086004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 0x55eea6bf9a60 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/2757086004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bfffe0 0x55eea6c1b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/2757086004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bff6c0 0x55eea6c00b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.583+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2757086004 >> 172.21.15.104:0/2757086004 conn(0x55eea6b24d40 msgr2=0x55eea6c00480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:38.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2757086004 shutdown_connections 2024-06-09T22:29:38.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/2757086004 wait complete. 2024-06-09T22:29:38.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb96c1fb80 1 -- start start 2024-06-09T22:29:38.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb96c1fb80 1 Processor -- start 2024-06-09T22:29:38.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb96c1fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 0x55eea6c0c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb96c1fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bff6c0 0x55eea6c0c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb96c1fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bfffe0 0x55eea6c08e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb96c1fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841b1c80 con 0x55eea6bf9fa0 2024-06-09T22:29:38.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb96c1fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841b15b0 con 0x55eea6bfffe0 2024-06-09T22:29:38.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb96c1fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841ae7f0 con 0x55eea6bff6c0 2024-06-09T22:29:38.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb96c1fb80 10 client.0 _ll_register_callbacks cb 0x55eea6ae59a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:38.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb8a138700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 0x55eea6c0c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb8a138700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 0x55eea6c0c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45286/0 (socket says 172.21.15.104:45286) 2024-06-09T22:29:38.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb8a138700 1 -- 172.21.15.104:0/605915764 learned_addr learned my addr 172.21.15.104:0/605915764 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:38.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb89937700 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bff6c0 0x55eea6c0c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.584+0000 7fdb8a939700 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bfffe0 0x55eea6c08e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb8a138700 1 -- 172.21.15.104:0/605915764 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bff6c0 msgr2=0x55eea6c0c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:38.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb8a138700 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bff6c0 0x55eea6c0c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb8a138700 1 -- 172.21.15.104:0/605915764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bfffe0 msgr2=0x55eea6c08e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:38.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb8a138700 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bfffe0 0x55eea6c08e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb8a138700 1 -- 172.21.15.104:0/605915764 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eea6c094e0 con 0x55eea6bf9fa0 2024-06-09T22:29:38.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb8a939700 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bfffe0 0x55eea6c08e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:38.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb8a138700 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 0x55eea6c0c040 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fdb74010010 tx=0x7fdb7400b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3956b2acf07eeb1f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:38.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb6dffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:38.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb6dffb700 1 -- 172.21.15.104:0/605915764 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb740055f0 con 0x55eea6bf9fa0 2024-06-09T22:29:38.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb6dffb700 1 -- 172.21.15.104:0/605915764 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fdb74004750 con 0x55eea6bf9fa0 2024-06-09T22:29:38.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/605915764 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb74021d70 con 0x55eea6bf9fa0 2024-06-09T22:29:38.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb6dffb700 1 -- 172.21.15.104:0/605915764 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdb74004950 con 0x55eea6bf9fa0 2024-06-09T22:29:38.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb96c1fb80 10 client.17814 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:38.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/605915764 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55eea6c7efb0 con 0x55eea6bf9fa0 2024-06-09T22:29:38.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb6dffb700 1 -- 172.21.15.104:0/605915764 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb74021d70 con 0x55eea6bf9fa0 2024-06-09T22:29:38.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb6dffb700 1 -- 172.21.15.104:0/605915764 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdb74033440 con 0x55eea6bf9fa0 2024-06-09T22:29:38.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb96c1fb80 10 client.17814 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:38.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb96c1fb80 10 client.17814 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:29:38.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb96c1fb80 20 client.17814 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:38.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.585+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55eea6c7f630 0x55eea6c81a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:38.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.586+0000 7fdb96c1fb80 4 client.17814 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-09T22:29:38.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.586+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/605915764 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55eea6b53310 con 0x55eea6c7f630 2024-06-09T22:29:38.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.586+0000 7fdb89937700 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55eea6c7f630 0x55eea6c81a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:38.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.586+0000 7fdb89937700 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55eea6c7f630 0x55eea6c81a50 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80069ad463258c25 server_cookie=912d741b398fc143 in_seq=0 out_seq=0 2024-06-09T22:29:38.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.586+0000 7fdb6dffb700 10 client.17814 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:38.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.587+0000 7fdb6dffb700 1 -- 172.21.15.104:0/605915764 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55eea6b53310 con 0x55eea6c7f630 2024-06-09T22:29:38.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.587+0000 7fdb6dffb700 10 client.17814 handle_command_reply: tid=0 2024-06-09T22:29:38.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.587+0000 7fdb96c1fb80 1 client.17814 shutdown 2024-06-09T22:29:38.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.587+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/605915764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 msgr2=0x55eea6c0c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:38.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.587+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 0x55eea6c0c040 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fdb74010010 tx=0x7fdb7400b830 comp rx=0 tx=0).stop 2024-06-09T22:29:38.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/605915764 shutdown_connections 2024-06-09T22:29:38.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55eea6c7f630 0x55eea6c81a50 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55eea6bf9fa0 0x55eea6c0c040 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eea6bfffe0 0x55eea6c08e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb96c1fb80 1 --2- 172.21.15.104:0/605915764 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eea6bff6c0 0x55eea6c0c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/605915764 >> 172.21.15.104:0/605915764 conn(0x55eea6b24d40 msgr2=0x55eea6bede80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:38.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb6dffb700 0 client.17814 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:38.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/605915764 shutdown_connections 2024-06-09T22:29:38.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb96c1fb80 1 -- 172.21.15.104:0/605915764 wait complete. 2024-06-09T22:29:38.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb96c1fb80 20 client.17814 trim_cache size 0 max 16384 2024-06-09T22:29:38.601 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:29:38.601 INFO:teuthology.orchestra.run.smithi104.stdout: "return_code": 0, 2024-06-09T22:29:38.602 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_tag": "8be45a5b-eb33-48df-9e7d-b57a0309ad96", 2024-06-09T22:29:38.602 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": "asynchronous" 2024-06-09T22:29:38.602 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:29:38.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb8c39c700 1 -- 172.21.15.104:0/3028800435 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fdb700a2d90 msgr2=0x7fdb700a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:38.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb8c39c700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fdb700a2d90 0x7fdb700a5250 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdb74005e10 tx=0x7fdb74005da0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb8c39c700 1 -- 172.21.15.104:0/3028800435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb84101d50 msgr2=0x7fdb8419dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:38.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb8c39c700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb84101d50 0x7fdb8419dc50 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000c920 tx=0x7fdb8000cce0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb8c39c700 1 -- 172.21.15.104:0/3028800435 shutdown_connections 2024-06-09T22:29:38.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb8c39c700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fdb700a2d90 0x7fdb700a5250 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb8c39c700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdb84107400 0x7fdb841978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb8c39c700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101390 0x7fdb8419d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb8c39c700 1 --2- 172.21.15.104:0/3028800435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb84101d50 0x7fdb8419dc50 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:38.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.588+0000 7fdb8c39c700 1 -- 172.21.15.104:0/3028800435 >> 172.21.15.104:0/3028800435 conn(0x7fdb84078340 msgr2=0x7fdb840ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:38.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.589+0000 7fdb8c39c700 1 -- 172.21.15.104:0/3028800435 shutdown_connections 2024-06-09T22:29:38.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:38.589+0000 7fdb8c39c700 1 -- 172.21.15.104:0/3028800435 wait complete. 2024-06-09T22:29:39.303 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:29:39.542 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:29:39.845 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:39 smithi104 ceph-mon[32389]: pgmap v267: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-06-09T22:29:39.845 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:39 smithi104 ceph-mon[32389]: scrub queued for path: / 2024-06-09T22:29:39.846 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:39 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:39.846 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:39 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:29:39.846 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:39 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:29:39.846 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:39 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:29:39.846 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:39 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:39.846 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:39 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:29:39.846 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:39 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:39.847 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:39 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:39.847 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:39 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:39.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:39 smithi153 ceph-mon[36599]: pgmap v267: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-06-09T22:29:39.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:39 smithi153 ceph-mon[36599]: scrub queued for path: / 2024-06-09T22:29:39.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:39 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:39.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:39 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:29:39.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:39 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:29:39.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:39 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:29:39.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:39 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:39.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:39 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:29:39.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:39 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:39.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:39 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:39.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:39 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:40.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:39 smithi194 ceph-mon[36856]: pgmap v267: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-06-09T22:29:40.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:39 smithi194 ceph-mon[36856]: scrub queued for path: / 2024-06-09T22:29:40.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:39 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:40.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:39 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:29:40.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:39 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:29:40.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:39 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:29:40.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:39 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:40.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:39 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:29:40.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:39 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:40.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:39 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:40.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:39 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:29:41.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.275+0000 7f2cda2b8700 1 -- 172.21.15.104:0/3306970650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cd406ab10 msgr2=0x7f2cd410f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:41.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.275+0000 7f2cda2b8700 1 --2- 172.21.15.104:0/3306970650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cd406ab10 0x7f2cd410f320 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00b0d0 tx=0x7f2ccc00b3e0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.275+0000 7f2cda2b8700 1 -- 172.21.15.104:0/3306970650 shutdown_connections 2024-06-09T22:29:41.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.275+0000 7f2cda2b8700 1 --2- 172.21.15.104:0/3306970650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cd406ab10 0x7f2cd410f320 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.275+0000 7f2cda2b8700 1 --2- 172.21.15.104:0/3306970650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cd41084e0 0x7f2cd41088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.275+0000 7f2cda2b8700 1 --2- 172.21.15.104:0/3306970650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd406a170 0x7f2cd406a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.275+0000 7f2cda2b8700 1 -- 172.21.15.104:0/3306970650 >> 172.21.15.104:0/3306970650 conn(0x7f2cd40782c0 msgr2=0x7f2cd40786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:41.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.275+0000 7f2cda2b8700 1 -- 172.21.15.104:0/3306970650 shutdown_connections 2024-06-09T22:29:41.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.275+0000 7f2cda2b8700 1 -- 172.21.15.104:0/3306970650 wait complete. 2024-06-09T22:29:41.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.275+0000 7f2cda2b8700 1 Processor -- start 2024-06-09T22:29:41.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cda2b8700 1 -- start start 2024-06-09T22:29:41.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cda2b8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cd406a170 0x7f2cd4101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cda2b8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cd406ab10 0x7f2cd41019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cda2b8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd41084e0 0x7f2cd4101ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cda2b8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd4112750 con 0x7f2cd406a170 2024-06-09T22:29:41.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cda2b8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41125d0 con 0x7f2cd41084e0 2024-06-09T22:29:41.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cda2b8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41128d0 con 0x7f2cd406ab10 2024-06-09T22:29:41.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cd3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cd406a170 0x7f2cd4101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cd3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cd406a170 0x7f2cd4101460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45314/0 (socket says 172.21.15.104:45314) 2024-06-09T22:29:41.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cd3fff700 1 -- 172.21.15.104:0/1990105821 learned_addr learned my addr 172.21.15.104:0/1990105821 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:41.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cd406ab10 0x7f2cd41019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cd8855700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd41084e0 0x7f2cd4101ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cd3fff700 1 -- 172.21.15.104:0/1990105821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cd406ab10 msgr2=0x7f2cd41019a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:41.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cd3fff700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cd406ab10 0x7f2cd41019a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cd3fff700 1 -- 172.21.15.104:0/1990105821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd41084e0 msgr2=0x7f2cd4101ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:41.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cd3fff700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd41084e0 0x7f2cd4101ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.276+0000 7f2cd3fff700 1 -- 172.21.15.104:0/1990105821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd41a70f0 con 0x7f2cd406a170 2024-06-09T22:29:41.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.277+0000 7f2cd3fff700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cd406a170 0x7f2cd4101460 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400c970 tx=0x7f2cc400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:41.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.277+0000 7f2cd17fa700 1 -- 172.21.15.104:0/1990105821 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40078d0 con 0x7f2cd406a170 2024-06-09T22:29:41.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.277+0000 7f2cda2b8700 1 -- 172.21.15.104:0/1990105821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cd41a73e0 con 0x7f2cd406a170 2024-06-09T22:29:41.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.277+0000 7f2cd17fa700 1 -- 172.21.15.104:0/1990105821 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cc4018480 con 0x7f2cd406a170 2024-06-09T22:29:41.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.277+0000 7f2cd17fa700 1 -- 172.21.15.104:0/1990105821 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc400f6b0 con 0x7f2cd406a170 2024-06-09T22:29:41.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.277+0000 7f2cda2b8700 1 -- 172.21.15.104:0/1990105821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cd41a7920 con 0x7f2cd406a170 2024-06-09T22:29:41.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.279+0000 7f2cd17fa700 1 -- 172.21.15.104:0/1990105821 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f2cc4018630 con 0x7f2cd406a170 2024-06-09T22:29:41.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.280+0000 7f2cd17fa700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2cbc0a2d90 0x7f2cbc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.280+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2cbc0a2d90 0x7f2cbc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.280+0000 7f2cd17fa700 1 -- 172.21.15.104:0/1990105821 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f2cc40cf870 con 0x7f2cd406a170 2024-06-09T22:29:41.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.280+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2cbc0a2d90 0x7f2cbc0a5250 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40788c0 tx=0x7f2cc800b410 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:41.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.280+0000 7f2ce4b3bb80 1 Processor -- start 2024-06-09T22:29:41.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.280+0000 7f2ce4b3bb80 1 -- start start 2024-06-09T22:29:41.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2ce4b3bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9640 0x5631c75b9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2ce4b3bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 0x5631c75b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2ce4b3bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b3f20 0x5631c75b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2ce4b3bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5631c7505810 con 0x5631c75b9f60 2024-06-09T22:29:41.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2ce4b3bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5631c74ad720 con 0x5631c75b3f20 2024-06-09T22:29:41.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2ce4b3bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5631c75017d0 con 0x5631c75b9640 2024-06-09T22:29:41.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cd37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 0x5631c75b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cd37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 0x5631c75b39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45330/0 (socket says 172.21.15.104:45330) 2024-06-09T22:29:41.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cd37fe700 1 -- 172.21.15.104:0/3156740928 learned_addr learned my addr 172.21.15.104:0/3156740928 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:41.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cd3fff700 1 --2- 172.21.15.104:0/3156740928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9640 0x5631c75b9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cd37fe700 1 -- 172.21.15.104:0/3156740928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9640 msgr2=0x5631c75b9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:41.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/3156740928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9640 0x5631c75b9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cd37fe700 1 -- 172.21.15.104:0/3156740928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b3f20 msgr2=0x5631c75b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:41.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/3156740928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b3f20 0x5631c75b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cd37fe700 1 -- 172.21.15.104:0/3156740928 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631c74a3f50 con 0x5631c75b9f60 2024-06-09T22:29:41.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/3156740928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 0x5631c75b39e0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8010f40 tx=0x7f2cc8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2050f35ec6c0b0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:41.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cba7fc700 1 -- 172.21.15.104:0/3156740928 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc800be60 con 0x5631c75b9f60 2024-06-09T22:29:41.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cba7fc700 1 -- 172.21.15.104:0/3156740928 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cc8011db0 con 0x5631c75b9f60 2024-06-09T22:29:41.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.281+0000 7f2cba7fc700 1 -- 172.21.15.104:0/3156740928 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc801c750 con 0x5631c75b9f60 2024-06-09T22:29:41.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.282+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3156740928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 msgr2=0x5631c75b39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:41.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.282+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/3156740928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 0x5631c75b39e0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8010f40 tx=0x7f2cc8013000 comp rx=0 tx=0).stop 2024-06-09T22:29:41.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.282+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3156740928 shutdown_connections 2024-06-09T22:29:41.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.282+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/3156740928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b3f20 0x5631c75b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.282+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/3156740928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9640 0x5631c75b9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.282+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/3156740928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 0x5631c75b39e0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.282+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3156740928 >> 172.21.15.104:0/3156740928 conn(0x5631c74decc0 msgr2=0x5631c75ae850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:41.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.282+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3156740928 shutdown_connections 2024-06-09T22:29:41.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.282+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3156740928 wait complete. 2024-06-09T22:29:41.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2ce4b3bb80 1 -- start start 2024-06-09T22:29:41.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2ce4b3bb80 1 Processor -- start 2024-06-09T22:29:41.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2ce4b3bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b3f20 0x5631c75bfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2ce4b3bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9640 0x5631c75bcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2ce4b3bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b9f60 0x5631c75bd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2ce4b3bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5631c75c09d0 con 0x5631c75b3f20 2024-06-09T22:29:41.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2ce4b3bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5631c74a5bf0 con 0x5631c75b9f60 2024-06-09T22:29:41.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2ce4b3bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5631c75c0b50 con 0x5631c75b9640 2024-06-09T22:29:41.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2ce4b3bb80 10 client.0 _ll_register_callbacks cb 0x5631c749f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:41.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2cd3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b3f20 0x5631c75bfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2cd3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b3f20 0x5631c75bfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45332/0 (socket says 172.21.15.104:45332) 2024-06-09T22:29:41.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2cd3fff700 1 -- 172.21.15.104:0/3103068407 learned_addr learned my addr 172.21.15.104:0/3103068407 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:41.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9640 0x5631c75bcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.283+0000 7f2cd8855700 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b9f60 0x5631c75bd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2cd3fff700 1 -- 172.21.15.104:0/3103068407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9640 msgr2=0x5631c75bcfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:41.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2cd3fff700 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9640 0x5631c75bcfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2cd3fff700 1 -- 172.21.15.104:0/3103068407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b9f60 msgr2=0x5631c75bd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:41.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2cd3fff700 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b9f60 0x5631c75bd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2cd3fff700 1 -- 172.21.15.104:0/3103068407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631c74548d0 con 0x5631c75b3f20 2024-06-09T22:29:41.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2cd3fff700 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b3f20 0x5631c75bfcd0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400cfd0 tx=0x7f2cc409b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac57482c7b313fb server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:41.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2cb37fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:41.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2cb37fe700 1 -- 172.21.15.104:0/3103068407 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40b3020 con 0x5631c75b3f20 2024-06-09T22:29:41.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2cb37fe700 1 -- 172.21.15.104:0/3103068407 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f2cc40cf660 con 0x5631c75b3f20 2024-06-09T22:29:41.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3103068407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cc8011db0 con 0x5631c75b3f20 2024-06-09T22:29:41.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2cb37fe700 1 -- 172.21.15.104:0/3103068407 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2cc409fe00 con 0x5631c75b3f20 2024-06-09T22:29:41.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2ce4b3bb80 10 client.17826 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:41.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3103068407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5631c75cb300 con 0x5631c75b3f20 2024-06-09T22:29:41.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2cb37fe700 1 -- 172.21.15.104:0/3103068407 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cc8011db0 con 0x5631c75b3f20 2024-06-09T22:29:41.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2cb37fe700 1 -- 172.21.15.104:0/3103068407 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2cc40a20b0 con 0x5631c75b3f20 2024-06-09T22:29:41.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2ce4b3bb80 10 client.17826 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:41.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2ce4b3bb80 10 client.17826 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:29:41.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2ce4b3bb80 20 client.17826 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:41.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.284+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5631c75cd940 0x5631c75cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.285+0000 7f2ce4b3bb80 4 client.17826 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:29:41.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.285+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3103068407 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631c7512d10 con 0x5631c75cd940 2024-06-09T22:29:41.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.285+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5631c75cd940 0x5631c75cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.285+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5631c75cd940 0x5631c75cfd30 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5088f48d5cedf0d server_cookie=390bd5f80bea7f80 in_seq=0 out_seq=0 2024-06-09T22:29:41.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.285+0000 7f2cb37fe700 10 client.17826 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:41.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2cb37fe700 1 -- 172.21.15.104:0/3103068407 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5631c7512d10 con 0x5631c75cd940 2024-06-09T22:29:41.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2cb37fe700 10 client.17826 handle_command_reply: tid=0 2024-06-09T22:29:41.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2ce4b3bb80 1 client.17826 shutdown 2024-06-09T22:29:41.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3103068407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b3f20 msgr2=0x5631c75bfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:41.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b3f20 0x5631c75bfcd0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400cfd0 tx=0x7f2cc409b480 comp rx=0 tx=0).stop 2024-06-09T22:29:41.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3103068407 shutdown_connections 2024-06-09T22:29:41.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5631c75cd940 0x5631c75cfd30 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b3f20 0x5631c75bfcd0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b9f60 0x5631c75bd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2cb37fe700 0 client.17826 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:41.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/3103068407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9640 0x5631c75bcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3103068407 >> 172.21.15.104:0/3103068407 conn(0x5631c74decc0 msgr2=0x5631c75b29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:41.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3103068407 shutdown_connections 2024-06-09T22:29:41.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/3103068407 wait complete. 2024-06-09T22:29:41.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.286+0000 7f2ce4b3bb80 20 client.17826 trim_cache size 0 max 16384 2024-06-09T22:29:41.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2ce4b3bb80 1 Processor -- start 2024-06-09T22:29:41.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2ce4b3bb80 1 -- start start 2024-06-09T22:29:41.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2ce4b3bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b3f20 0x5631c75b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2ce4b3bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b9640 0x5631c75a8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2ce4b3bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 0x5631c75d5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2ce4b3bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41b07c0 con 0x5631c75b9f60 2024-06-09T22:29:41.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2ce4b3bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41b2300 con 0x5631c75b9640 2024-06-09T22:29:41.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2ce4b3bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41b0e90 con 0x5631c75b3f20 2024-06-09T22:29:41.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2cd8855700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 0x5631c75d5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2cd8855700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 0x5631c75d5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45334/0 (socket says 172.21.15.104:45334) 2024-06-09T22:29:41.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2cd3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b3f20 0x5631c75b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2cd8855700 1 -- 172.21.15.104:0/2427480453 learned_addr learned my addr 172.21.15.104:0/2427480453 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:41.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/2427480453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b9640 0x5631c75a8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2cd8855700 1 -- 172.21.15.104:0/2427480453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b3f20 msgr2=0x5631c75b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:41.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2cd8855700 1 --2- 172.21.15.104:0/2427480453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b3f20 0x5631c75b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2cd8855700 1 -- 172.21.15.104:0/2427480453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b9640 msgr2=0x5631c75a8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:41.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2cd8855700 1 --2- 172.21.15.104:0/2427480453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b9640 0x5631c75a8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2cd8855700 1 -- 172.21.15.104:0/2427480453 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631c75d02e0 con 0x5631c75b9f60 2024-06-09T22:29:41.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.307+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/2427480453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b9640 0x5631c75a8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:29:41.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.308+0000 7f2cd8855700 1 --2- 172.21.15.104:0/2427480453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 0x5631c75d5530 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc015000 tx=0x7f2ccc009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf356a3fbad299e3 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:41.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.308+0000 7f2cb9ffb700 1 -- 172.21.15.104:0/2427480453 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc003a00 con 0x5631c75b9f60 2024-06-09T22:29:41.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.308+0000 7f2cb9ffb700 1 -- 172.21.15.104:0/2427480453 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ccc021070 con 0x5631c75b9f60 2024-06-09T22:29:41.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.308+0000 7f2cb9ffb700 1 -- 172.21.15.104:0/2427480453 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc0266d0 con 0x5631c75b9f60 2024-06-09T22:29:41.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.308+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2427480453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 msgr2=0x5631c75d5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:41.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.308+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/2427480453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 0x5631c75d5530 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc015000 tx=0x7f2ccc009310 comp rx=0 tx=0).stop 2024-06-09T22:29:41.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.308+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2427480453 shutdown_connections 2024-06-09T22:29:41.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.308+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/2427480453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9f60 0x5631c75d5530 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.308+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/2427480453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b3f20 0x5631c75b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.308+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/2427480453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b9640 0x5631c75a8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.309+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2427480453 >> 172.21.15.104:0/2427480453 conn(0x5631c74decc0 msgr2=0x5631c75ba9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.309+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2427480453 shutdown_connections 2024-06-09T22:29:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.309+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2427480453 wait complete. 2024-06-09T22:29:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.309+0000 7f2ce4b3bb80 1 -- start start 2024-06-09T22:29:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.309+0000 7f2ce4b3bb80 1 Processor -- start 2024-06-09T22:29:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.309+0000 7f2ce4b3bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b3f20 0x5631c75c1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.309+0000 7f2ce4b3bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9640 0x5631c75c2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.309+0000 7f2ce4b3bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9f60 0x5631c75c2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.309+0000 7f2ce4b3bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41b1c30 con 0x5631c75b9640 2024-06-09T22:29:41.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.309+0000 7f2ce4b3bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41b1560 con 0x5631c75b3f20 2024-06-09T22:29:41.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.309+0000 7f2ce4b3bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41ae780 con 0x5631c75b9f60 2024-06-09T22:29:41.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.309+0000 7f2ce4b3bb80 10 client.0 _ll_register_callbacks cb 0x5631c749f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:29:41.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cd37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9640 0x5631c75c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cd37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9640 0x5631c75c2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45350/0 (socket says 172.21.15.104:45350) 2024-06-09T22:29:41.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cd37fe700 1 -- 172.21.15.104:0/2916518524 learned_addr learned my addr 172.21.15.104:0/2916518524 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:29:41.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cd3fff700 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b3f20 0x5631c75c1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cd8855700 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9f60 0x5631c75c2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cd37fe700 1 -- 172.21.15.104:0/2916518524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9f60 msgr2=0x5631c75c2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:41.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9f60 0x5631c75c2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cd37fe700 1 -- 172.21.15.104:0/2916518524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b3f20 msgr2=0x5631c75c1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:29:41.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b3f20 0x5631c75c1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cd37fe700 1 -- 172.21.15.104:0/2916518524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631c7637800 con 0x5631c75b9640 2024-06-09T22:29:41.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cd37fe700 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9640 0x5631c75c2410 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8010f10 tx=0x7f2cc800bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ad600ee4473664 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:29:41.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cb37fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:29:41.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cb37fe700 1 -- 172.21.15.104:0/2916518524 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8005510 con 0x5631c75b9640 2024-06-09T22:29:41.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2916518524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ccc021070 con 0x5631c75b9640 2024-06-09T22:29:41.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cb37fe700 1 -- 172.21.15.104:0/2916518524 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f2cc80056b0 con 0x5631c75b9640 2024-06-09T22:29:41.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2cb37fe700 1 -- 172.21.15.104:0/2916518524 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2cc8044050 con 0x5631c75b9640 2024-06-09T22:29:41.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2ce4b3bb80 10 client.17832 fetch_fsmap learned FSMap version 32 2024-06-09T22:29:41.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.310+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2916518524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5631c7638b10 con 0x5631c75b9640 2024-06-09T22:29:41.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.311+0000 7f2cb37fe700 1 -- 172.21.15.104:0/2916518524 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ccc021070 con 0x5631c75b9640 2024-06-09T22:29:41.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.311+0000 7f2cb37fe700 1 -- 172.21.15.104:0/2916518524 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2cc801e950 con 0x5631c75b9640 2024-06-09T22:29:41.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.311+0000 7f2ce4b3bb80 10 client.17832 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:29:41.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.311+0000 7f2ce4b3bb80 10 client.17832 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:29:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.311+0000 7f2ce4b3bb80 20 client.17832 populate_metadata read hostname 'smithi104' 2024-06-09T22:29:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.311+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5631c7638eb0 0x5631c763b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:29:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.311+0000 7f2ce4b3bb80 4 client.17832 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:29:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.311+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2916518524 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5631c750d290 con 0x5631c7638eb0 2024-06-09T22:29:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.311+0000 7f2cd3fff700 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5631c7638eb0 0x5631c763b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:29:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.311+0000 7f2cd3fff700 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5631c7638eb0 0x5631c763b2d0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=326ca23b2a6bcd13 server_cookie=4ee5687e32ca05f2 in_seq=0 out_seq=0 2024-06-09T22:29:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.311+0000 7f2cb37fe700 10 client.17832 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2cb37fe700 1 -- 172.21.15.104:0/2916518524 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5631c750d290 con 0x5631c7638eb0 2024-06-09T22:29:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2cb37fe700 10 client.17832 handle_command_reply: tid=0 2024-06-09T22:29:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2ce4b3bb80 1 client.17832 shutdown 2024-06-09T22:29:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2916518524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9640 msgr2=0x5631c75c2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9640 0x5631c75c2410 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8010f10 tx=0x7f2cc800bb20 comp rx=0 tx=0).stop 2024-06-09T22:29:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2916518524 shutdown_connections 2024-06-09T22:29:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5631c7638eb0 0x5631c763b2d0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5631c75b9f60 0x5631c75c2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c75b3f20 0x5631c75c1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2ce4b3bb80 1 --2- 172.21.15.104:0/2916518524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5631c75b9640 0x5631c75c2410 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2cb37fe700 0 client.17832 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:29:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2916518524 >> 172.21.15.104:0/2916518524 conn(0x5631c74decc0 msgr2=0x5631c75b8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2916518524 shutdown_connections 2024-06-09T22:29:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2ce4b3bb80 1 -- 172.21.15.104:0/2916518524 wait complete. 2024-06-09T22:29:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.312+0000 7f2ce4b3bb80 20 client.17832 trim_cache size 0 max 16384 2024-06-09T22:29:41.324 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:29:41.325 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-09T22:29:41.325 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": { 2024-06-09T22:29:41.325 INFO:teuthology.orchestra.run.smithi104.stdout: "8be45a5b-eb33-48df-9e7d-b57a0309ad96": { 2024-06-09T22:29:41.325 INFO:teuthology.orchestra.run.smithi104.stdout: "path": "/", 2024-06-09T22:29:41.325 INFO:teuthology.orchestra.run.smithi104.stdout: "tag": "8be45a5b-eb33-48df-9e7d-b57a0309ad96", 2024-06-09T22:29:41.325 INFO:teuthology.orchestra.run.smithi104.stdout: "options": "recursive,force" 2024-06-09T22:29:41.325 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:29:41.326 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:29:41.326 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:29:41.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.313+0000 7f2cda2b8700 1 -- 172.21.15.104:0/1990105821 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2cbc0a2d90 msgr2=0x7f2cbc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:41.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.313+0000 7f2cda2b8700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2cbc0a2d90 0x7f2cbc0a5250 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40788c0 tx=0x7f2cc800b410 comp rx=0 tx=0).stop 2024-06-09T22:29:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.313+0000 7f2cda2b8700 1 -- 172.21.15.104:0/1990105821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cd406a170 msgr2=0x7f2cd4101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:29:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.313+0000 7f2cda2b8700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cd406a170 0x7f2cd4101460 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400c970 tx=0x7f2cc400cc80 comp rx=0 tx=0).stop 2024-06-09T22:29:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.313+0000 7f2cda2b8700 1 -- 172.21.15.104:0/1990105821 shutdown_connections 2024-06-09T22:29:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.313+0000 7f2cda2b8700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2cbc0a2d90 0x7f2cbc0a5250 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.313+0000 7f2cda2b8700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cd406a170 0x7f2cd4101460 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.313+0000 7f2cda2b8700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd41084e0 0x7f2cd4101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.313+0000 7f2cda2b8700 1 --2- 172.21.15.104:0/1990105821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cd406ab10 0x7f2cd41019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:29:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.313+0000 7f2cda2b8700 1 -- 172.21.15.104:0/1990105821 >> 172.21.15.104:0/1990105821 conn(0x7f2cd40782c0 msgr2=0x7f2cd40ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:29:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.313+0000 7f2cda2b8700 1 -- 172.21.15.104:0/1990105821 shutdown_connections 2024-06-09T22:29:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:29:41.313+0000 7f2cda2b8700 1 -- 172.21.15.104:0/1990105821 wait complete. 2024-06-09T22:29:41.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:41 smithi153 ceph-mon[36599]: pgmap v268: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-06-09T22:29:42.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:41 smithi104 ceph-mon[32389]: pgmap v268: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-06-09T22:29:42.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:41 smithi194 ceph-mon[36856]: pgmap v268: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-06-09T22:29:42.133 INFO:tasks.cephfs.filesystem:scrub status for tag:8be45a5b-eb33-48df-9e7d-b57a0309ad96 - {'path': '/', 'tag': '8be45a5b-eb33-48df-9e7d-b57a0309ad96', 'options': 'recursive,force'} 2024-06-09T22:29:43.863 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:43 smithi194 ceph-mon[36856]: pgmap v269: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 1.0 MiB/s wr, 50 op/s 2024-06-09T22:29:43.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:43 smithi153 ceph-mon[36599]: pgmap v269: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 1.0 MiB/s wr, 50 op/s 2024-06-09T22:29:44.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:43 smithi104 ceph-mon[32389]: pgmap v269: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 1.0 MiB/s wr, 50 op/s 2024-06-09T22:29:45.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:45 smithi153 ceph-mon[36599]: pgmap v270: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.0 MiB/s wr, 55 op/s 2024-06-09T22:29:46.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:45 smithi104 ceph-mon[32389]: pgmap v270: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.0 MiB/s wr, 55 op/s 2024-06-09T22:29:46.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:45 smithi194 ceph-mon[36856]: pgmap v270: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.0 MiB/s wr, 55 op/s 2024-06-09T22:29:47.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:47 smithi153 ceph-mon[36599]: pgmap v271: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 300 KiB/s wr, 26 op/s 2024-06-09T22:29:48.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:47 smithi104 ceph-mon[32389]: pgmap v271: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 300 KiB/s wr, 26 op/s 2024-06-09T22:29:48.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:47 smithi194 ceph-mon[36856]: pgmap v271: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 300 KiB/s wr, 26 op/s 2024-06-09T22:29:49.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:49 smithi153 ceph-mon[36599]: pgmap v272: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 300 KiB/s wr, 26 op/s 2024-06-09T22:29:50.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:49 smithi104 ceph-mon[32389]: pgmap v272: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 300 KiB/s wr, 26 op/s 2024-06-09T22:29:50.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:49 smithi194 ceph-mon[36856]: pgmap v272: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 300 KiB/s wr, 26 op/s 2024-06-09T22:29:50.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:50 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:29:50.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:50 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:29:50.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:50 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:29:51.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:50 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:29:51.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:50 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:29:51.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:50 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:29:51.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:50 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:29:51.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:50 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:29:51.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:50 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:29:51.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:51 smithi153 ceph-mon[36599]: pgmap v273: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 308 KiB/s wr, 31 op/s 2024-06-09T22:29:52.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:51 smithi104 ceph-mon[32389]: pgmap v273: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 308 KiB/s wr, 31 op/s 2024-06-09T22:29:52.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:51 smithi194 ceph-mon[36856]: pgmap v273: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 308 KiB/s wr, 31 op/s 2024-06-09T22:29:53.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:53 smithi153 ceph-mon[36599]: pgmap v274: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-06-09T22:29:54.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:53 smithi104 ceph-mon[32389]: pgmap v274: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-06-09T22:29:54.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:53 smithi194 ceph-mon[36856]: pgmap v274: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-06-09T22:29:55.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:55 smithi153 ceph-mon[36599]: pgmap v275: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 16 KiB/s wr, 13 op/s 2024-06-09T22:29:56.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:55 smithi104 ceph-mon[32389]: pgmap v275: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 16 KiB/s wr, 13 op/s 2024-06-09T22:29:56.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:55 smithi194 ceph-mon[36856]: pgmap v275: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 16 KiB/s wr, 13 op/s 2024-06-09T22:29:57.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:57 smithi153 ceph-mon[36599]: pgmap v276: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-06-09T22:29:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:57 smithi104 ceph-mon[32389]: pgmap v276: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-06-09T22:29:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:57 smithi194 ceph-mon[36856]: pgmap v276: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-06-09T22:29:59.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:29:59 smithi153 ceph-mon[36599]: pgmap v277: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-06-09T22:30:00.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:29:59 smithi104 ceph-mon[32389]: pgmap v277: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-06-09T22:30:00.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:29:59 smithi194 ceph-mon[36856]: pgmap v277: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-06-09T22:30:00.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:00 smithi153 ceph-mon[36599]: overall HEALTH_OK 2024-06-09T22:30:01.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:00 smithi104 ceph-mon[32389]: overall HEALTH_OK 2024-06-09T22:30:01.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:00 smithi194 ceph-mon[36856]: overall HEALTH_OK 2024-06-09T22:30:01.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:01 smithi153 ceph-mon[36599]: pgmap v278: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-06-09T22:30:02.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:01 smithi104 ceph-mon[32389]: pgmap v278: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-06-09T22:30:02.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:01 smithi194 ceph-mon[36856]: pgmap v278: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-06-09T22:30:03.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:03 smithi153 ceph-mon[36599]: pgmap v279: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.0 KiB/s wr, 8 op/s 2024-06-09T22:30:04.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:03 smithi104 ceph-mon[32389]: pgmap v279: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.0 KiB/s wr, 8 op/s 2024-06-09T22:30:04.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:03 smithi194 ceph-mon[36856]: pgmap v279: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.0 KiB/s wr, 8 op/s 2024-06-09T22:30:04.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:04 smithi153 ceph-mon[36599]: pgmap v280: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-06-09T22:30:05.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:04 smithi104 ceph-mon[32389]: pgmap v280: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-06-09T22:30:05.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:04 smithi194 ceph-mon[36856]: pgmap v280: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-06-09T22:30:06.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:06 smithi153 ceph-mon[36599]: osdmap e105: 12 total, 12 up, 12 in 2024-06-09T22:30:06.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:06 smithi153 ceph-mon[36599]: pgmap v282: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 60 KiB/s wr, 31 op/s 2024-06-09T22:30:07.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:06 smithi104 ceph-mon[32389]: osdmap e105: 12 total, 12 up, 12 in 2024-06-09T22:30:07.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:06 smithi104 ceph-mon[32389]: pgmap v282: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 60 KiB/s wr, 31 op/s 2024-06-09T22:30:07.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:06 smithi194 ceph-mon[36856]: osdmap e105: 12 total, 12 up, 12 in 2024-06-09T22:30:07.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:06 smithi194 ceph-mon[36856]: pgmap v282: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 60 KiB/s wr, 31 op/s 2024-06-09T22:30:09.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:08 smithi194 ceph-mon[36856]: pgmap v283: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 60 KiB/s wr, 31 op/s 2024-06-09T22:30:09.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:08 smithi153 ceph-mon[36599]: pgmap v283: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 60 KiB/s wr, 31 op/s 2024-06-09T22:30:09.259 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:08 smithi104 ceph-mon[32389]: pgmap v283: 129 pgs: 129 active+clean; 249 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 60 KiB/s wr, 31 op/s 2024-06-09T22:30:11.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:10 smithi194 ceph-mon[36856]: pgmap v284: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 64 KiB/s wr, 31 op/s 2024-06-09T22:30:11.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:10 smithi153 ceph-mon[36599]: pgmap v284: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 64 KiB/s wr, 31 op/s 2024-06-09T22:30:11.259 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:10 smithi104 ceph-mon[32389]: pgmap v284: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 64 KiB/s wr, 31 op/s 2024-06-09T22:30:12.135 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:30:12.392 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:30:13.837 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:13 smithi104 ceph-mon[32389]: pgmap v285: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 64 KiB/s wr, 31 op/s 2024-06-09T22:30:13.837 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:13 smithi104 ceph-mon[32389]: osdmap e106: 12 total, 12 up, 12 in 2024-06-09T22:30:13.949 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:13 smithi153 ceph-mon[36599]: pgmap v285: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 64 KiB/s wr, 31 op/s 2024-06-09T22:30:13.949 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:13 smithi153 ceph-mon[36599]: osdmap e106: 12 total, 12 up, 12 in 2024-06-09T22:30:14.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:13 smithi194 ceph-mon[36856]: pgmap v285: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 64 KiB/s wr, 31 op/s 2024-06-09T22:30:14.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:13 smithi194 ceph-mon[36856]: osdmap e106: 12 total, 12 up, 12 in 2024-06-09T22:30:14.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.244+0000 7f04272be700 1 -- 172.21.15.104:0/155305403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0420101620 msgr2=0x7f0420101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:14.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.244+0000 7f04272be700 1 --2- 172.21.15.104:0/155305403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0420101620 0x7f0420101a00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f041c0099c0 tx=0x7f041c009cd0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.244+0000 7f04272be700 1 -- 172.21.15.104:0/155305403 shutdown_connections 2024-06-09T22:30:14.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.244+0000 7f04272be700 1 --2- 172.21.15.104:0/155305403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0420101620 0x7f0420101a00 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.244+0000 7f04272be700 1 --2- 172.21.15.104:0/155305403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04201066b0 0x7f042010e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.244+0000 7f04272be700 1 --2- 172.21.15.104:0/155305403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0420101fd0 0x7f0420105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.244+0000 7f04272be700 1 -- 172.21.15.104:0/155305403 >> 172.21.15.104:0/155305403 conn(0x7f0420078340 msgr2=0x7f0420078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.244+0000 7f04272be700 1 -- 172.21.15.104:0/155305403 shutdown_connections 2024-06-09T22:30:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.244+0000 7f04272be700 1 -- 172.21.15.104:0/155305403 wait complete. 2024-06-09T22:30:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f04272be700 1 Processor -- start 2024-06-09T22:30:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f04272be700 1 -- start start 2024-06-09T22:30:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f04272be700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0420101620 0x7f042019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f04272be700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0420101fd0 0x7f042019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f04272be700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04201066b0 0x7f0420198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f04272be700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0420111690 con 0x7f04201066b0 2024-06-09T22:30:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f04272be700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0420111510 con 0x7f0420101620 2024-06-09T22:30:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f04272be700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0420111810 con 0x7f0420101fd0 2024-06-09T22:30:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f042585b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04201066b0 0x7f0420198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f042585b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04201066b0 0x7f0420198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48740/0 (socket says 172.21.15.104:48740) 2024-06-09T22:30:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f042585b700 1 -- 172.21.15.104:0/3352724146 learned_addr learned my addr 172.21.15.104:0/3352724146 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f0424859700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0420101fd0 0x7f042019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.245+0000 7f042505a700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0420101620 0x7f042019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.246+0000 7f042585b700 1 -- 172.21.15.104:0/3352724146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0420101fd0 msgr2=0x7f042019edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.246+0000 7f042585b700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0420101fd0 0x7f042019edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.246+0000 7f042585b700 1 -- 172.21.15.104:0/3352724146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0420101620 msgr2=0x7f042019e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:14.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.246+0000 7f042585b700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0420101620 0x7f042019e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.246+0000 7f042585b700 1 -- 172.21.15.104:0/3352724146 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0420199190 con 0x7f04201066b0 2024-06-09T22:30:14.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.246+0000 7f042585b700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04201066b0 0x7f0420198a30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f0418007ee0 tx=0x7f041800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:14.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.246+0000 7f04167fc700 1 -- 172.21.15.104:0/3352724146 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0418004bb0 con 0x7f04201066b0 2024-06-09T22:30:14.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.246+0000 7f04272be700 1 -- 172.21.15.104:0/3352724146 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0420199480 con 0x7f04201066b0 2024-06-09T22:30:14.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.246+0000 7f04167fc700 1 -- 172.21.15.104:0/3352724146 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0418004d50 con 0x7f04201066b0 2024-06-09T22:30:14.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.246+0000 7f04167fc700 1 -- 172.21.15.104:0/3352724146 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0418005960 con 0x7f04201066b0 2024-06-09T22:30:14.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.246+0000 7f04272be700 1 -- 172.21.15.104:0/3352724146 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04201a5740 con 0x7f04201066b0 2024-06-09T22:30:14.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.248+0000 7f04167fc700 1 -- 172.21.15.104:0/3352724146 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f041801a020 con 0x7f04201066b0 2024-06-09T22:30:14.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.249+0000 7f04167fc700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f040c0a2d90 0x7f040c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.249+0000 7f042505a700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f040c0a2d90 0x7f040c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.249+0000 7f04167fc700 1 -- 172.21.15.104:0/3352724146 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f04180ceef0 con 0x7f04201066b0 2024-06-09T22:30:14.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.249+0000 7f042505a700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f040c0a2d90 0x7f040c0a5250 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f041c000c00 tx=0x7f041c005fb0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:14.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f0431b41b80 1 Processor -- start 2024-06-09T22:30:14.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f0431b41b80 1 -- start start 2024-06-09T22:30:14.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f0431b41b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 0x55b3193efa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f0431b41b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193eff60 0x55b3193e99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f0431b41b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b3193ea340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f0431b41b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b31933b810 con 0x55b3193ef640 2024-06-09T22:30:14.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f0431b41b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b3192e3720 con 0x55b3193e9f20 2024-06-09T22:30:14.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f0431b41b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b3193377d0 con 0x55b3193eff60 2024-06-09T22:30:14.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f042505a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 0x55b3193efa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f042505a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 0x55b3193efa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48750/0 (socket says 172.21.15.104:48750) 2024-06-09T22:30:14.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f042505a700 1 -- 172.21.15.104:0/3105360429 learned_addr learned my addr 172.21.15.104:0/3105360429 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:14.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f042585b700 1 --2- 172.21.15.104:0/3105360429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b3193ea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f0424859700 1 --2- 172.21.15.104:0/3105360429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193eff60 0x55b3193e99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f042505a700 1 -- 172.21.15.104:0/3105360429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193eff60 msgr2=0x55b3193e99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:14.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f042505a700 1 --2- 172.21.15.104:0/3105360429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193eff60 0x55b3193e99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f042505a700 1 -- 172.21.15.104:0/3105360429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 msgr2=0x55b3193ea340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:14.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f042505a700 1 --2- 172.21.15.104:0/3105360429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b3193ea340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.250+0000 7f042505a700 1 -- 172.21.15.104:0/3105360429 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3192d9f50 con 0x55b3193ef640 2024-06-09T22:30:14.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.251+0000 7f0424859700 1 --2- 172.21.15.104:0/3105360429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193eff60 0x55b3193e99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:30:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.251+0000 7f042505a700 1 --2- 172.21.15.104:0/3105360429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 0x55b3193efa20 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f041c015000 tx=0x7f041c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0267385183e3bea server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.251+0000 7f040b7fe700 1 -- 172.21.15.104:0/3105360429 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c00f460 con 0x55b3193ef640 2024-06-09T22:30:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.251+0000 7f040b7fe700 1 -- 172.21.15.104:0/3105360429 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f041c00fae0 con 0x55b3193ef640 2024-06-09T22:30:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.251+0000 7f040b7fe700 1 -- 172.21.15.104:0/3105360429 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c024c40 con 0x55b3193ef640 2024-06-09T22:30:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.252+0000 7f0431b41b80 1 -- 172.21.15.104:0/3105360429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 msgr2=0x55b3193efa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.252+0000 7f0431b41b80 1 --2- 172.21.15.104:0/3105360429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 0x55b3193efa20 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f041c015000 tx=0x7f041c020000 comp rx=0 tx=0).stop 2024-06-09T22:30:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.252+0000 7f0431b41b80 1 -- 172.21.15.104:0/3105360429 shutdown_connections 2024-06-09T22:30:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.252+0000 7f0431b41b80 1 --2- 172.21.15.104:0/3105360429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 0x55b3193efa20 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.252+0000 7f0431b41b80 1 --2- 172.21.15.104:0/3105360429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b3193ea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.252+0000 7f0431b41b80 1 --2- 172.21.15.104:0/3105360429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193eff60 0x55b3193e99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.252+0000 7f0431b41b80 1 -- 172.21.15.104:0/3105360429 >> 172.21.15.104:0/3105360429 conn(0x55b319314cc0 msgr2=0x55b3193e4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:14.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.252+0000 7f0431b41b80 1 -- 172.21.15.104:0/3105360429 shutdown_connections 2024-06-09T22:30:14.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.252+0000 7f0431b41b80 1 -- 172.21.15.104:0/3105360429 wait complete. 2024-06-09T22:30:14.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.252+0000 7f0431b41b80 1 -- start start 2024-06-09T22:30:14.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.252+0000 7f0431b41b80 1 Processor -- start 2024-06-09T22:30:14.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f0431b41b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b3193f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f0431b41b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193ef640 0x55b3193f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f0431b41b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 0x55b3193f3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f0431b41b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b3193f6910 con 0x55b3193eff60 2024-06-09T22:30:14.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f0431b41b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b3192dbbf0 con 0x55b3193e9f20 2024-06-09T22:30:14.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f0431b41b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b3193f6a90 con 0x55b3193ef640 2024-06-09T22:30:14.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f0431b41b80 10 client.0 _ll_register_callbacks cb 0x55b3192d5920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:14.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f042585b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 0x55b3193f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f042585b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 0x55b3193f3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48766/0 (socket says 172.21.15.104:48766) 2024-06-09T22:30:14.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f042505a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b3193f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f042585b700 1 -- 172.21.15.104:0/2009808690 learned_addr learned my addr 172.21.15.104:0/2009808690 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:14.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f0424859700 1 --2- 172.21.15.104:0/2009808690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193ef640 0x55b3193f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f042585b700 1 -- 172.21.15.104:0/2009808690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193ef640 msgr2=0x55b3193f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:14.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f042585b700 1 --2- 172.21.15.104:0/2009808690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193ef640 0x55b3193f2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f042585b700 1 -- 172.21.15.104:0/2009808690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 msgr2=0x55b3193f5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:14.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f042585b700 1 --2- 172.21.15.104:0/2009808690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b3193f5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f042585b700 1 -- 172.21.15.104:0/2009808690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b31928a8d0 con 0x55b3193eff60 2024-06-09T22:30:14.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f042585b700 1 --2- 172.21.15.104:0/2009808690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 0x55b3193f3500 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f041800d700 tx=0x7f041809af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743fdca9486244fc server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:14.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f0408ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:14.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f0408ff9700 1 -- 172.21.15.104:0/2009808690 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041809b040 con 0x55b3193eff60 2024-06-09T22:30:14.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f0408ff9700 1 -- 172.21.15.104:0/2009808690 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f04180cc4f0 con 0x55b3193eff60 2024-06-09T22:30:14.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.253+0000 7f0431b41b80 1 -- 172.21.15.104:0/2009808690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f041c00fae0 con 0x55b3193eff60 2024-06-09T22:30:14.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.254+0000 7f0408ff9700 1 -- 172.21.15.104:0/2009808690 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f041809f3f0 con 0x55b3193eff60 2024-06-09T22:30:14.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.254+0000 7f0431b41b80 10 client.17844 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:14.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.254+0000 7f0431b41b80 1 -- 172.21.15.104:0/2009808690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b319401290 con 0x55b3193eff60 2024-06-09T22:30:14.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.254+0000 7f0408ff9700 1 -- 172.21.15.104:0/2009808690 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f041c00fae0 con 0x55b3193eff60 2024-06-09T22:30:14.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.254+0000 7f0408ff9700 1 -- 172.21.15.104:0/2009808690 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f041809b2a0 con 0x55b3193eff60 2024-06-09T22:30:14.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.254+0000 7f0431b41b80 10 client.17844 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:14.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.254+0000 7f0431b41b80 10 client.17844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:30:14.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.254+0000 7f0431b41b80 20 client.17844 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:14.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.254+0000 7f0431b41b80 1 --2- 172.21.15.104:0/2009808690 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b3194038d0 0x55b319405cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.254+0000 7f0431b41b80 4 client.17844 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:30:14.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.254+0000 7f0431b41b80 1 -- 172.21.15.104:0/2009808690 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b319348d10 con 0x55b3194038d0 2024-06-09T22:30:14.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.254+0000 7f042505a700 1 --2- 172.21.15.104:0/2009808690 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b3194038d0 0x55b319405cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.255+0000 7f0408ff9700 10 client.17844 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:14.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.255+0000 7f042505a700 1 --2- 172.21.15.104:0/2009808690 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b3194038d0 0x55b319405cc0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c427abd50082a92 server_cookie=b1c8d960ab4dfaf2 in_seq=0 out_seq=0 2024-06-09T22:30:14.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.255+0000 7f0408ff9700 1 -- 172.21.15.104:0/2009808690 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b319348d10 con 0x55b3194038d0 2024-06-09T22:30:14.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.255+0000 7f0408ff9700 10 client.17844 handle_command_reply: tid=0 2024-06-09T22:30:14.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.255+0000 7f0431b41b80 1 client.17844 shutdown 2024-06-09T22:30:14.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.255+0000 7f0431b41b80 1 -- 172.21.15.104:0/2009808690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 msgr2=0x55b3193f3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:14.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.255+0000 7f0431b41b80 1 --2- 172.21.15.104:0/2009808690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 0x55b3193f3500 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f041800d700 tx=0x7f041809af20 comp rx=0 tx=0).stop 2024-06-09T22:30:14.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.256+0000 7f0431b41b80 1 -- 172.21.15.104:0/2009808690 shutdown_connections 2024-06-09T22:30:14.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.256+0000 7f0431b41b80 1 --2- 172.21.15.104:0/2009808690 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b3194038d0 0x55b319405cc0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.256+0000 7f0431b41b80 1 --2- 172.21.15.104:0/2009808690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 0x55b3193f3500 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.256+0000 7f0408ff9700 0 client.17844 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:14.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.256+0000 7f0431b41b80 1 --2- 172.21.15.104:0/2009808690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b3193f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.256+0000 7f0431b41b80 1 --2- 172.21.15.104:0/2009808690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193ef640 0x55b3193f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.256+0000 7f0431b41b80 1 -- 172.21.15.104:0/2009808690 >> 172.21.15.104:0/2009808690 conn(0x55b319314cc0 msgr2=0x55b3193e89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:14.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.256+0000 7f0431b41b80 1 -- 172.21.15.104:0/2009808690 shutdown_connections 2024-06-09T22:30:14.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.256+0000 7f0431b41b80 1 -- 172.21.15.104:0/2009808690 wait complete. 2024-06-09T22:30:14.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.256+0000 7f0431b41b80 20 client.17844 trim_cache size 0 max 16384 2024-06-09T22:30:14.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f0431b41b80 1 Processor -- start 2024-06-09T22:30:14.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f0431b41b80 1 -- start start 2024-06-09T22:30:14.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f0431b41b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b3193e99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f0431b41b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193ef640 0x55b3193e49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f0431b41b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 0x55b3193f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f0431b41b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f04201ae3d0 con 0x55b3193eff60 2024-06-09T22:30:14.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f0431b41b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f04201aff10 con 0x55b3193e9f20 2024-06-09T22:30:14.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f0431b41b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f04201aeaa0 con 0x55b3193ef640 2024-06-09T22:30:14.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f042585b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 0x55b3193f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f042585b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 0x55b3193f0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48778/0 (socket says 172.21.15.104:48778) 2024-06-09T22:30:14.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f042585b700 1 -- 172.21.15.104:0/2628940300 learned_addr learned my addr 172.21.15.104:0/2628940300 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:14.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f0424859700 1 --2- 172.21.15.104:0/2628940300 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193ef640 0x55b3193e49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f042505a700 1 --2- 172.21.15.104:0/2628940300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b3193e99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f042585b700 1 -- 172.21.15.104:0/2628940300 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193ef640 msgr2=0x55b3193e49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:14.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f042585b700 1 --2- 172.21.15.104:0/2628940300 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193ef640 0x55b3193e49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f042585b700 1 -- 172.21.15.104:0/2628940300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 msgr2=0x55b3193e99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:14.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f042585b700 1 --2- 172.21.15.104:0/2628940300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b3193e99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f042585b700 1 -- 172.21.15.104:0/2628940300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3194062a0 con 0x55b3193eff60 2024-06-09T22:30:14.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.276+0000 7f0424859700 1 --2- 172.21.15.104:0/2628940300 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193ef640 0x55b3193e49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:30:14.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f042585b700 1 --2- 172.21.15.104:0/2628940300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 0x55b3193f0bd0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f0418097a10 tx=0x7f04180cc340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894b1052190f0613 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:14.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f040affd700 1 -- 172.21.15.104:0/2628940300 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041809a9a0 con 0x55b3193eff60 2024-06-09T22:30:14.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f040affd700 1 -- 172.21.15.104:0/2628940300 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f041809b040 con 0x55b3193eff60 2024-06-09T22:30:14.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f040affd700 1 -- 172.21.15.104:0/2628940300 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0418022250 con 0x55b3193eff60 2024-06-09T22:30:14.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f0431b41b80 1 -- 172.21.15.104:0/2628940300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 msgr2=0x55b3193f0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:14.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f0431b41b80 1 --2- 172.21.15.104:0/2628940300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 0x55b3193f0bd0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f0418097a10 tx=0x7f04180cc340 comp rx=0 tx=0).stop 2024-06-09T22:30:14.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f0431b41b80 1 -- 172.21.15.104:0/2628940300 shutdown_connections 2024-06-09T22:30:14.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f0431b41b80 1 --2- 172.21.15.104:0/2628940300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193eff60 0x55b3193f0bd0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f0431b41b80 1 --2- 172.21.15.104:0/2628940300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b3193e99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f0431b41b80 1 --2- 172.21.15.104:0/2628940300 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193ef640 0x55b3193e49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f0431b41b80 1 -- 172.21.15.104:0/2628940300 >> 172.21.15.104:0/2628940300 conn(0x55b319314cc0 msgr2=0x55b3193dde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:14.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f0431b41b80 1 -- 172.21.15.104:0/2628940300 shutdown_connections 2024-06-09T22:30:14.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.277+0000 7f0431b41b80 1 -- 172.21.15.104:0/2628940300 wait complete. 2024-06-09T22:30:14.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f0431b41b80 1 -- start start 2024-06-09T22:30:14.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f0431b41b80 1 Processor -- start 2024-06-09T22:30:14.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f0431b41b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b31940b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f0431b41b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 0x55b31940b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f0431b41b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193eff60 0x55b31940bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f0431b41b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f04201af840 con 0x55b3193ef640 2024-06-09T22:30:14.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f0431b41b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f04201af170 con 0x55b3193e9f20 2024-06-09T22:30:14.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f0431b41b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f04201ac390 con 0x55b3193eff60 2024-06-09T22:30:14.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f0431b41b80 10 client.0 _ll_register_callbacks cb 0x55b3192d5920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:14.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f0424859700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 0x55b31940b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f0424859700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 0x55b31940b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48786/0 (socket says 172.21.15.104:48786) 2024-06-09T22:30:14.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f0424859700 1 -- 172.21.15.104:0/265794172 learned_addr learned my addr 172.21.15.104:0/265794172 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:14.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f042585b700 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193eff60 0x55b31940bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.278+0000 7f042505a700 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b31940b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0424859700 1 -- 172.21.15.104:0/265794172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193eff60 msgr2=0x55b31940bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:14.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0424859700 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193eff60 0x55b31940bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0424859700 1 -- 172.21.15.104:0/265794172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 msgr2=0x55b31940b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:14.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0424859700 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b31940b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0424859700 1 -- 172.21.15.104:0/265794172 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b31946d660 con 0x55b3193ef640 2024-06-09T22:30:14.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0424859700 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 0x55b31940b7a0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f0420101300 tx=0x7f0410007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7707263911d6fd server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:14.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0408ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:14.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0408ff9700 1 -- 172.21.15.104:0/265794172 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0410004050 con 0x55b3193ef640 2024-06-09T22:30:14.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0431b41b80 1 -- 172.21.15.104:0/265794172 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f041809b040 con 0x55b3193ef640 2024-06-09T22:30:14.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0408ff9700 1 -- 172.21.15.104:0/265794172 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f04100041f0 con 0x55b3193ef640 2024-06-09T22:30:14.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0408ff9700 1 -- 172.21.15.104:0/265794172 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0410026050 con 0x55b3193ef640 2024-06-09T22:30:14.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0431b41b80 10 client.17850 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:14.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0431b41b80 1 -- 172.21.15.104:0/265794172 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b31946e910 con 0x55b3193ef640 2024-06-09T22:30:14.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0408ff9700 1 -- 172.21.15.104:0/265794172 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f041809b040 con 0x55b3193ef640 2024-06-09T22:30:14.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0408ff9700 1 -- 172.21.15.104:0/265794172 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0410011600 con 0x55b3193ef640 2024-06-09T22:30:14.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0431b41b80 10 client.17850 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:14.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0431b41b80 10 client.17850 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:30:14.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0431b41b80 20 client.17850 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:14.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0431b41b80 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b3194038d0 0x55b3194710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:14.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0431b41b80 4 client.17850 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:30:14.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.279+0000 7f0431b41b80 1 -- 172.21.15.104:0/265794172 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b319343290 con 0x55b3194038d0 2024-06-09T22:30:14.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f042505a700 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b3194038d0 0x55b3194710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:14.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0408ff9700 10 client.17850 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:14.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f042505a700 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b3194038d0 0x55b3194710c0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfe04207389fbead server_cookie=88597dc7b53dc992 in_seq=0 out_seq=0 2024-06-09T22:30:14.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0408ff9700 1 -- 172.21.15.104:0/265794172 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b319343290 con 0x55b3194038d0 2024-06-09T22:30:14.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0408ff9700 10 client.17850 handle_command_reply: tid=0 2024-06-09T22:30:14.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0431b41b80 1 client.17850 shutdown 2024-06-09T22:30:14.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0431b41b80 1 -- 172.21.15.104:0/265794172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 msgr2=0x55b31940b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:14.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0431b41b80 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 0x55b31940b7a0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f0420101300 tx=0x7f0410007f90 comp rx=0 tx=0).stop 2024-06-09T22:30:14.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0431b41b80 1 -- 172.21.15.104:0/265794172 shutdown_connections 2024-06-09T22:30:14.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0431b41b80 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55b3194038d0 0x55b3194710c0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0431b41b80 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b3193eff60 0x55b31940bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0431b41b80 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b3193e9f20 0x55b31940b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0431b41b80 1 --2- 172.21.15.104:0/265794172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b3193ef640 0x55b31940b7a0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0431b41b80 1 -- 172.21.15.104:0/265794172 >> 172.21.15.104:0/265794172 conn(0x55b319314cc0 msgr2=0x55b3193de510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:14.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.280+0000 7f0408ff9700 0 client.17850 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:14.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f0431b41b80 1 -- 172.21.15.104:0/265794172 shutdown_connections 2024-06-09T22:30:14.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f0431b41b80 1 -- 172.21.15.104:0/265794172 wait complete. 2024-06-09T22:30:14.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f0431b41b80 20 client.17850 trim_cache size 0 max 16384 2024-06-09T22:30:14.294 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:30:14.294 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "no active scrubs running", 2024-06-09T22:30:14.294 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": {} 2024-06-09T22:30:14.294 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:30:14.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f04272be700 1 -- 172.21.15.104:0/3352724146 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f040c0a2d90 msgr2=0x7f040c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:14.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f04272be700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f040c0a2d90 0x7f040c0a5250 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f041c000c00 tx=0x7f041c005fb0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f04272be700 1 -- 172.21.15.104:0/3352724146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04201066b0 msgr2=0x7f0420198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:14.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f04272be700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04201066b0 0x7f0420198a30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f0418007ee0 tx=0x7f041800d3b0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f04272be700 1 -- 172.21.15.104:0/3352724146 shutdown_connections 2024-06-09T22:30:14.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f04272be700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f040c0a2d90 0x7f040c0a5250 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f04272be700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04201066b0 0x7f0420198a30 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f04272be700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0420101620 0x7f042019e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f04272be700 1 --2- 172.21.15.104:0/3352724146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0420101fd0 0x7f042019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:14.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f04272be700 1 -- 172.21.15.104:0/3352724146 >> 172.21.15.104:0/3352724146 conn(0x7f0420078340 msgr2=0x7f042006aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:14.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f04272be700 1 -- 172.21.15.104:0/3352724146 shutdown_connections 2024-06-09T22:30:14.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:14.281+0000 7f04272be700 1 -- 172.21.15.104:0/3352724146 wait complete. 2024-06-09T22:30:14.960 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-09T22:30:14.961 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:30:15.202 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:30:15.576 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:15 smithi104 ceph-mon[32389]: pgmap v287: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.0 KiB/s wr, 13 op/s 2024-06-09T22:30:15.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:15 smithi153 ceph-mon[36599]: pgmap v287: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.0 KiB/s wr, 13 op/s 2024-06-09T22:30:16.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:15 smithi194 ceph-mon[36856]: pgmap v287: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.0 KiB/s wr, 13 op/s 2024-06-09T22:30:17.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.019+0000 7fc4a20dc700 1 -- 172.21.15.104:0/861553371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c102740 msgr2=0x7fc49c102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:17.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.019+0000 7fc4a20dc700 1 --2- 172.21.15.104:0/861553371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c102740 0x7fc49c102b20 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00b0d0 tx=0x7fc48c00b3e0 comp rx=0 tx=0).stop 2024-06-09T22:30:17.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.019+0000 7fc4a20dc700 1 -- 172.21.15.104:0/861553371 shutdown_connections 2024-06-09T22:30:17.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.019+0000 7fc4a20dc700 1 --2- 172.21.15.104:0/861553371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c102740 0x7fc49c102b20 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:17.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.019+0000 7fc4a20dc700 1 --2- 172.21.15.104:0/861553371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc49c107840 0x7fc49c10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:17.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.019+0000 7fc4a20dc700 1 --2- 172.21.15.104:0/861553371 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc49c1030f0 0x7fc49c107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:17.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.019+0000 7fc4a20dc700 1 -- 172.21.15.104:0/861553371 >> 172.21.15.104:0/861553371 conn(0x7fc49c0795f0 msgr2=0x7fc49c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:17.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.019+0000 7fc4a20dc700 1 -- 172.21.15.104:0/861553371 shutdown_connections 2024-06-09T22:30:17.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.019+0000 7fc4a20dc700 1 -- 172.21.15.104:0/861553371 wait complete. 2024-06-09T22:30:17.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.019+0000 7fc4a20dc700 1 Processor -- start 2024-06-09T22:30:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.019+0000 7fc4a20dc700 1 -- start start 2024-06-09T22:30:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc4a20dc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc49c102740 0x7fc49c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc4a20dc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc49c1030f0 0x7fc49c074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc4a20dc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c107840 0x7fc49c075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc4a20dc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c1127d0 con 0x7fc49c107840 2024-06-09T22:30:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc4a20dc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c112650 con 0x7fc49c102740 2024-06-09T22:30:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc4a20dc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c112950 con 0x7fc49c1030f0 2024-06-09T22:30:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc49bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c107840 0x7fc49c075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:17.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc49bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c107840 0x7fc49c075210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47306/0 (socket says 172.21.15.104:47306) 2024-06-09T22:30:17.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc49bfff700 1 -- 172.21.15.104:0/1915706577 learned_addr learned my addr 172.21.15.104:0/1915706577 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:17.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc49affd700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc49c1030f0 0x7fc49c074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:17.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc49b7fe700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc49c102740 0x7fc49c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:17.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc49bfff700 1 -- 172.21.15.104:0/1915706577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc49c1030f0 msgr2=0x7fc49c074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:17.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc49bfff700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc49c1030f0 0x7fc49c074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:17.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc49bfff700 1 -- 172.21.15.104:0/1915706577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc49c102740 msgr2=0x7fc49c074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:17.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc49bfff700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc49c102740 0x7fc49c074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:17.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc49bfff700 1 -- 172.21.15.104:0/1915706577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49c1a71b0 con 0x7fc49c107840 2024-06-09T22:30:17.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc49bfff700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c107840 0x7fc49c075210 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc49400c970 tx=0x7fc49400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:17.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc498ff9700 1 -- 172.21.15.104:0/1915706577 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494004a30 con 0x7fc49c107840 2024-06-09T22:30:17.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc498ff9700 1 -- 172.21.15.104:0/1915706577 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc494005500 con 0x7fc49c107840 2024-06-09T22:30:17.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc4a20dc700 1 -- 172.21.15.104:0/1915706577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc49c1a74a0 con 0x7fc49c107840 2024-06-09T22:30:17.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.020+0000 7fc498ff9700 1 -- 172.21.15.104:0/1915706577 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49400ecf0 con 0x7fc49c107840 2024-06-09T22:30:17.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.021+0000 7fc4a20dc700 1 -- 172.21.15.104:0/1915706577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc49c1a7990 con 0x7fc49c107840 2024-06-09T22:30:17.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.022+0000 7fc4a20dc700 1 -- 172.21.15.104:0/1915706577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc49c10aa80 con 0x7fc49c107840 2024-06-09T22:30:17.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.023+0000 7fc498ff9700 1 -- 172.21.15.104:0/1915706577 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc494005020 con 0x7fc49c107840 2024-06-09T22:30:17.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.023+0000 7fc498ff9700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc4840a2d90 0x7fc4840a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:17.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.023+0000 7fc49b7fe700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc4840a2d90 0x7fc4840a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:17.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.023+0000 7fc498ff9700 1 -- 172.21.15.104:0/1915706577 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fc4940cf300 con 0x7fc49c107840 2024-06-09T22:30:17.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.024+0000 7fc49b7fe700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc4840a2d90 0x7fc4840a5250 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009fd0 tx=0x7fc48c00b040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:17.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.028+0000 7fc498ff9700 1 -- 172.21.15.104:0/1915706577 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc494014030 con 0x7fc49c107840 2024-06-09T22:30:17.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.268+0000 7fc4a20dc700 1 -- 172.21.15.104:0/1915706577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc49c10a460 con 0x7fc49c107840 2024-06-09T22:30:17.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.268+0000 7fc498ff9700 1 -- 172.21.15.104:0/1915706577 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7fc49409bb50 con 0x7fc49c107840 2024-06-09T22:30:17.271 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:30:17.272 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:30:17.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.271+0000 7fc4a20dc700 1 -- 172.21.15.104:0/1915706577 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc4840a2d90 msgr2=0x7fc4840a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:17.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.271+0000 7fc4a20dc700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc4840a2d90 0x7fc4840a5250 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009fd0 tx=0x7fc48c00b040 comp rx=0 tx=0).stop 2024-06-09T22:30:17.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.271+0000 7fc4a20dc700 1 -- 172.21.15.104:0/1915706577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c107840 msgr2=0x7fc49c075210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:17.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.271+0000 7fc4a20dc700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c107840 0x7fc49c075210 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc49400c970 tx=0x7fc49400cc80 comp rx=0 tx=0).stop 2024-06-09T22:30:17.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.271+0000 7fc4a20dc700 1 -- 172.21.15.104:0/1915706577 shutdown_connections 2024-06-09T22:30:17.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.271+0000 7fc4a20dc700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc4840a2d90 0x7fc4840a5250 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:17.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.271+0000 7fc4a20dc700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c107840 0x7fc49c075210 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:17.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.271+0000 7fc4a20dc700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc49c102740 0x7fc49c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:17.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.271+0000 7fc4a20dc700 1 --2- 172.21.15.104:0/1915706577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc49c1030f0 0x7fc49c074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:17.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.271+0000 7fc4a20dc700 1 -- 172.21.15.104:0/1915706577 >> 172.21.15.104:0/1915706577 conn(0x7fc49c0795f0 msgr2=0x7fc49c108990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:17.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.271+0000 7fc4a20dc700 1 -- 172.21.15.104:0/1915706577 shutdown_connections 2024-06-09T22:30:17.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:17.271+0000 7fc4a20dc700 1 -- 172.21.15.104:0/1915706577 wait complete. 2024-06-09T22:30:17.275 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:30:17.908 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:17 smithi104 ceph-mon[32389]: pgmap v288: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.0 KiB/s wr, 11 op/s 2024-06-09T22:30:17.908 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:17 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1915706577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:30:17.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-09T22:30:17.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:17 smithi153 ceph-mon[36599]: pgmap v288: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.0 KiB/s wr, 11 op/s 2024-06-09T22:30:17.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:17 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1915706577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:30:18.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:17 smithi194 ceph-mon[36856]: pgmap v288: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.0 KiB/s wr, 11 op/s 2024-06-09T22:30:18.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:17 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1915706577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:30:18.174 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:30:19.824 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:19 smithi104 ceph-mon[32389]: pgmap v289: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.0 KiB/s wr, 11 op/s 2024-06-09T22:30:19.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:19 smithi153 ceph-mon[36599]: pgmap v289: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.0 KiB/s wr, 11 op/s 2024-06-09T22:30:20.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:19 smithi194 ceph-mon[36856]: pgmap v289: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.0 KiB/s wr, 11 op/s 2024-06-09T22:30:20.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.073+0000 7f4ab70b0700 1 -- 172.21.15.104:0/2399521656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab01066b0 msgr2=0x7f4ab010e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:20.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.073+0000 7f4ab70b0700 1 --2- 172.21.15.104:0/2399521656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab01066b0 0x7f4ab010e280 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800b0d0 tx=0x7f4aa800b3e0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.073+0000 7f4ab70b0700 1 -- 172.21.15.104:0/2399521656 shutdown_connections 2024-06-09T22:30:20.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.073+0000 7f4ab70b0700 1 --2- 172.21.15.104:0/2399521656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab01066b0 0x7f4ab010e280 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.073+0000 7f4ab70b0700 1 --2- 172.21.15.104:0/2399521656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab0101620 0x7f4ab0101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.073+0000 7f4ab70b0700 1 --2- 172.21.15.104:0/2399521656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab0101fd0 0x7f4ab0105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.073+0000 7f4ab70b0700 1 -- 172.21.15.104:0/2399521656 >> 172.21.15.104:0/2399521656 conn(0x7f4ab0078340 msgr2=0x7f4ab0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:20.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.073+0000 7f4ab70b0700 1 -- 172.21.15.104:0/2399521656 shutdown_connections 2024-06-09T22:30:20.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.073+0000 7f4ab70b0700 1 -- 172.21.15.104:0/2399521656 wait complete. 2024-06-09T22:30:20.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4ab70b0700 1 Processor -- start 2024-06-09T22:30:20.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4ab70b0700 1 -- start start 2024-06-09T22:30:20.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4ab70b0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab0101620 0x7f4ab0199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4ab70b0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab0101fd0 0x7f4ab019a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4ab70b0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab01066b0 0x7f4ab01a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4ab70b0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01116c0 con 0x7f4ab0101fd0 2024-06-09T22:30:20.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4ab70b0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab0111540 con 0x7f4ab0101620 2024-06-09T22:30:20.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4ab70b0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab0111840 con 0x7f4ab01066b0 2024-06-09T22:30:20.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4aaffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab0101fd0 0x7f4ab019a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4aaffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab0101fd0 0x7f4ab019a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47328/0 (socket says 172.21.15.104:47328) 2024-06-09T22:30:20.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4aaffff700 1 -- 172.21.15.104:0/3395579723 learned_addr learned my addr 172.21.15.104:0/3395579723 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:20.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab0101620 0x7f4ab0199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.074+0000 7f4ab564d700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab01066b0 0x7f4ab01a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.075+0000 7f4aaffff700 1 -- 172.21.15.104:0/3395579723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab01066b0 msgr2=0x7f4ab01a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:20.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.075+0000 7f4aaffff700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab01066b0 0x7f4ab01a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.075+0000 7f4aaffff700 1 -- 172.21.15.104:0/3395579723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab0101620 msgr2=0x7f4ab0199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:20.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.075+0000 7f4aaffff700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab0101620 0x7f4ab0199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.075+0000 7f4aaffff700 1 -- 172.21.15.104:0/3395579723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab01a50a0 con 0x7f4ab0101fd0 2024-06-09T22:30:20.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.075+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab0101620 0x7f4ab0199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:30:20.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.075+0000 7f4aaffff700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab0101fd0 0x7f4ab019a020 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00d980 tx=0x7f4a9c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:20.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.075+0000 7f4aadffb700 1 -- 172.21.15.104:0/3395579723 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c017070 con 0x7f4ab0101fd0 2024-06-09T22:30:20.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.075+0000 7f4aadffb700 1 -- 172.21.15.104:0/3395579723 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c00fd70 con 0x7f4ab0101fd0 2024-06-09T22:30:20.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.075+0000 7f4ab70b0700 1 -- 172.21.15.104:0/3395579723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab01a52d0 con 0x7f4ab0101fd0 2024-06-09T22:30:20.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.075+0000 7f4aadffb700 1 -- 172.21.15.104:0/3395579723 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c00e3b0 con 0x7f4ab0101fd0 2024-06-09T22:30:20.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.075+0000 7f4ab70b0700 1 -- 172.21.15.104:0/3395579723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab01a5810 con 0x7f4ab0101fd0 2024-06-09T22:30:20.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.077+0000 7f4aadffb700 1 -- 172.21.15.104:0/3395579723 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f4a9c00f890 con 0x7f4ab0101fd0 2024-06-09T22:30:20.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.078+0000 7f4aadffb700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4aa00a2d90 0x7f4aa00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.078+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4aa00a2d90 0x7f4aa00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.078+0000 7f4aadffb700 1 -- 172.21.15.104:0/3395579723 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f4a9c0cf200 con 0x7f4ab0101fd0 2024-06-09T22:30:20.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.078+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4aa00a2d90 0x7f4aa00a5250 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40104e0 tx=0x7f4aa400b3c0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:20.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.079+0000 7f4ac1933b80 1 Processor -- start 2024-06-09T22:30:20.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.079+0000 7f4ac1933b80 1 -- start start 2024-06-09T22:30:20.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.079+0000 7f4ac1933b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcf1640 0x55714bcf1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.079+0000 7f4ac1933b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bceb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.079+0000 7f4ac1933b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcebf20 0x55714bcec340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.079+0000 7f4ac1933b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55714bc3d810 con 0x55714bcf1f60 2024-06-09T22:30:20.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.079+0000 7f4ac1933b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55714bbe5720 con 0x55714bcebf20 2024-06-09T22:30:20.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.079+0000 7f4ac1933b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55714bc397d0 con 0x55714bcf1640 2024-06-09T22:30:20.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.079+0000 7f4aaffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bceb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.079+0000 7f4aaffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bceb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47330/0 (socket says 172.21.15.104:47330) 2024-06-09T22:30:20.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.079+0000 7f4aaffff700 1 -- 172.21.15.104:0/3560245302 learned_addr learned my addr 172.21.15.104:0/3560245302 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.079+0000 7f4ab564d700 1 --2- 172.21.15.104:0/3560245302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcebf20 0x55714bcec340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.080+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/3560245302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcf1640 0x55714bcf1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.080+0000 7f4aaffff700 1 -- 172.21.15.104:0/3560245302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcf1640 msgr2=0x55714bcf1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.080+0000 7f4aaffff700 1 --2- 172.21.15.104:0/3560245302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcf1640 0x55714bcf1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.080+0000 7f4aaffff700 1 -- 172.21.15.104:0/3560245302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcebf20 msgr2=0x55714bcec340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.080+0000 7f4aaffff700 1 --2- 172.21.15.104:0/3560245302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcebf20 0x55714bcec340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.080+0000 7f4aaffff700 1 -- 172.21.15.104:0/3560245302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55714bbdbf50 con 0x55714bcf1f60 2024-06-09T22:30:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.080+0000 7f4ab564d700 1 --2- 172.21.15.104:0/3560245302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcebf20 0x55714bcec340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:30:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.080+0000 7f4aaffff700 1 --2- 172.21.15.104:0/3560245302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bceb9e0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0977b0 tx=0x7f4a9c097720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22cd9c58c6712214 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.080+0000 7f4a9affd700 1 -- 172.21.15.104:0/3560245302 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c020050 con 0x55714bcf1f60 2024-06-09T22:30:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.080+0000 7f4a9affd700 1 -- 172.21.15.104:0/3560245302 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c0a3fc0 con 0x55714bcf1f60 2024-06-09T22:30:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.080+0000 7f4a9affd700 1 -- 172.21.15.104:0/3560245302 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c0a3b30 con 0x55714bcf1f60 2024-06-09T22:30:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.081+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3560245302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 msgr2=0x55714bceb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.081+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/3560245302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bceb9e0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0977b0 tx=0x7f4a9c097720 comp rx=0 tx=0).stop 2024-06-09T22:30:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.081+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3560245302 shutdown_connections 2024-06-09T22:30:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.081+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/3560245302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcebf20 0x55714bcec340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.081+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/3560245302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcf1640 0x55714bcf1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.081+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/3560245302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bceb9e0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.081+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3560245302 >> 172.21.15.104:0/3560245302 conn(0x55714bc16cc0 msgr2=0x55714bce6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.081+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3560245302 shutdown_connections 2024-06-09T22:30:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.081+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3560245302 wait complete. 2024-06-09T22:30:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.081+0000 7f4ac1933b80 1 -- start start 2024-06-09T22:30:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.081+0000 7f4ac1933b80 1 Processor -- start 2024-06-09T22:30:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ac1933b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcebf20 0x55714bcf7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ac1933b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcf1640 0x55714bcf5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ac1933b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bcf5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ac1933b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55714bcf8970 con 0x55714bcf1f60 2024-06-09T22:30:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ac1933b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55714bbddbf0 con 0x55714bcf1640 2024-06-09T22:30:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ac1933b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55714bcf8af0 con 0x55714bcebf20 2024-06-09T22:30:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ac1933b80 10 client.0 _ll_register_callbacks cb 0x55714bbd7920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ab564d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bcf5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ab564d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bcf5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47340/0 (socket says 172.21.15.104:47340) 2024-06-09T22:30:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ab564d700 1 -- 172.21.15.104:0/420379580 learned_addr learned my addr 172.21.15.104:0/420379580 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4aaffff700 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcf1640 0x55714bcf5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcebf20 0x55714bcf7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ab564d700 1 -- 172.21.15.104:0/420379580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcebf20 msgr2=0x55714bcf7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ab564d700 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcebf20 0x55714bcf7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ab564d700 1 -- 172.21.15.104:0/420379580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcf1640 msgr2=0x55714bcf5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ab564d700 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcf1640 0x55714bcf5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ab564d700 1 -- 172.21.15.104:0/420379580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55714bb8c8d0 con 0x55714bcf1f60 2024-06-09T22:30:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcebf20 0x55714bcf7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:30:20.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4aaffff700 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcf1640 0x55714bcf5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:30:20.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ab564d700 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bcf5590 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8004cc0 tx=0x7f4aa8004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1e0b03e054d82 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:20.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4a8ffff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:20.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4a8ffff700 1 -- 172.21.15.104:0/420379580 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa80076e0 con 0x55714bcf1f60 2024-06-09T22:30:20.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4a8ffff700 1 -- 172.21.15.104:0/420379580 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f4aa8007880 con 0x55714bcf1f60 2024-06-09T22:30:20.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.082+0000 7f4ac1933b80 1 -- 172.21.15.104:0/420379580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a9c0a3fc0 con 0x55714bcf1f60 2024-06-09T22:30:20.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.083+0000 7f4a8ffff700 1 -- 172.21.15.104:0/420379580 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4aa8007a80 con 0x55714bcf1f60 2024-06-09T22:30:20.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.083+0000 7f4ac1933b80 10 client.17868 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:20.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.083+0000 7f4ac1933b80 1 -- 172.21.15.104:0/420379580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55714bd03320 con 0x55714bcf1f60 2024-06-09T22:30:20.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.083+0000 7f4a8ffff700 1 -- 172.21.15.104:0/420379580 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c0a3fc0 con 0x55714bcf1f60 2024-06-09T22:30:20.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.083+0000 7f4a8ffff700 1 -- 172.21.15.104:0/420379580 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4aa8011070 con 0x55714bcf1f60 2024-06-09T22:30:20.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.084+0000 7f4ac1933b80 10 client.17868 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:20.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.084+0000 7f4ac1933b80 10 client.17868 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:30:20.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.084+0000 7f4ac1933b80 20 client.17868 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:20.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.084+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55714bd059b0 0x55714bd07da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.084+0000 7f4ac1933b80 4 client.17868 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:30:20.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.084+0000 7f4ac1933b80 1 -- 172.21.15.104:0/420379580 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55714bc4ad10 con 0x55714bd059b0 2024-06-09T22:30:20.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.084+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55714bd059b0 0x55714bd07da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.084+0000 7f4a8ffff700 10 client.17868 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:30:20.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.084+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55714bd059b0 0x55714bd07da0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5c9b0ec69d63801d server_cookie=d35610e2218873f in_seq=0 out_seq=0 2024-06-09T22:30:20.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4a8ffff700 1 -- 172.21.15.104:0/420379580 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55714bc4ad10 con 0x55714bd059b0 2024-06-09T22:30:20.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4a8ffff700 10 client.17868 handle_command_reply: tid=0 2024-06-09T22:30:20.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4ac1933b80 1 client.17868 shutdown 2024-06-09T22:30:20.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4ac1933b80 1 -- 172.21.15.104:0/420379580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 msgr2=0x55714bcf5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:20.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bcf5590 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8004cc0 tx=0x7f4aa8004da0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4ac1933b80 1 -- 172.21.15.104:0/420379580 shutdown_connections 2024-06-09T22:30:20.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bcf5590 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55714bd059b0 0x55714bd07da0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcebf20 0x55714bcf7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/420379580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcf1640 0x55714bcf5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4a8ffff700 0 client.17868 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:30:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4ac1933b80 1 -- 172.21.15.104:0/420379580 >> 172.21.15.104:0/420379580 conn(0x55714bc16cc0 msgr2=0x55714bcea9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.085+0000 7f4ac1933b80 1 -- 172.21.15.104:0/420379580 shutdown_connections 2024-06-09T22:30:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.086+0000 7f4ac1933b80 1 -- 172.21.15.104:0/420379580 wait complete. 2024-06-09T22:30:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.086+0000 7f4ac1933b80 20 client.17868 trim_cache size 0 max 16384 2024-06-09T22:30:20.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.100+0000 7f4ac1933b80 1 Processor -- start 2024-06-09T22:30:20.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.100+0000 7f4ac1933b80 1 -- start start 2024-06-09T22:30:20.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ac1933b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcebf20 0x55714bd01c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ac1933b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcf1640 0x55714bceb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ac1933b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bce69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ac1933b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01ae650 con 0x55714bcf1f60 2024-06-09T22:30:20.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ac1933b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01b0190 con 0x55714bcebf20 2024-06-09T22:30:20.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ac1933b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01aed20 con 0x55714bcf1640 2024-06-09T22:30:20.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ab564d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bce69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ab564d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bce69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47348/0 (socket says 172.21.15.104:47348) 2024-06-09T22:30:20.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ab564d700 1 -- 172.21.15.104:0/2801359269 learned_addr learned my addr 172.21.15.104:0/2801359269 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:20.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4aaffff700 1 --2- 172.21.15.104:0/2801359269 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcf1640 0x55714bceb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/2801359269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcebf20 0x55714bd01c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ab564d700 1 -- 172.21.15.104:0/2801359269 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcf1640 msgr2=0x55714bceb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:20.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ab564d700 1 --2- 172.21.15.104:0/2801359269 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcf1640 0x55714bceb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ab564d700 1 -- 172.21.15.104:0/2801359269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcebf20 msgr2=0x55714bd01c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:20.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ab564d700 1 --2- 172.21.15.104:0/2801359269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcebf20 0x55714bd01c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ab564d700 1 -- 172.21.15.104:0/2801359269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55714bcf1a20 con 0x55714bcf1f60 2024-06-09T22:30:20.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4ab564d700 1 --2- 172.21.15.104:0/2801359269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bce69f0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800b0d0 tx=0x7f4aa8026550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4dbb812d6b80743 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:20.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4a9a7fc700 1 -- 172.21.15.104:0/2801359269 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8039070 con 0x55714bcf1f60 2024-06-09T22:30:20.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4a9a7fc700 1 -- 172.21.15.104:0/2801359269 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4aa8007c20 con 0x55714bcf1f60 2024-06-09T22:30:20.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.101+0000 7f4a9a7fc700 1 -- 172.21.15.104:0/2801359269 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8026c00 con 0x55714bcf1f60 2024-06-09T22:30:20.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.102+0000 7f4ac1933b80 1 -- 172.21.15.104:0/2801359269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 msgr2=0x55714bce69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:20.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.102+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/2801359269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bce69f0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800b0d0 tx=0x7f4aa8026550 comp rx=0 tx=0).stop 2024-06-09T22:30:20.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.102+0000 7f4ac1933b80 1 -- 172.21.15.104:0/2801359269 shutdown_connections 2024-06-09T22:30:20.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.102+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/2801359269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bce69f0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.102+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/2801359269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcebf20 0x55714bd01c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.102+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/2801359269 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcf1640 0x55714bceb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.102+0000 7f4ac1933b80 1 -- 172.21.15.104:0/2801359269 >> 172.21.15.104:0/2801359269 conn(0x55714bc16cc0 msgr2=0x55714bcef0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:20.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.102+0000 7f4ac1933b80 1 -- 172.21.15.104:0/2801359269 shutdown_connections 2024-06-09T22:30:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.102+0000 7f4ac1933b80 1 -- 172.21.15.104:0/2801359269 wait complete. 2024-06-09T22:30:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.102+0000 7f4ac1933b80 1 -- start start 2024-06-09T22:30:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.102+0000 7f4ac1933b80 1 Processor -- start 2024-06-09T22:30:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ac1933b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcebf20 0x55714bcfe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ac1933b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcf1640 0x55714bd0cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ac1933b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bd0d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ac1933b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01afac0 con 0x55714bcf1f60 2024-06-09T22:30:20.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ac1933b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01af3f0 con 0x55714bcf1640 2024-06-09T22:30:20.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ac1933b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01ac610 con 0x55714bcebf20 2024-06-09T22:30:20.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ac1933b80 10 client.0 _ll_register_callbacks cb 0x55714bbd7920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:20.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ab564d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bd0d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ab564d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bd0d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47354/0 (socket says 172.21.15.104:47354) 2024-06-09T22:30:20.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ab564d700 1 -- 172.21.15.104:0/3179853205 learned_addr learned my addr 172.21.15.104:0/3179853205 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:20.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcebf20 0x55714bcfe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4aaffff700 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcf1640 0x55714bd0cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ab564d700 1 -- 172.21.15.104:0/3179853205 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcebf20 msgr2=0x55714bcfe1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:20.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ab564d700 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcebf20 0x55714bcfe1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ab564d700 1 -- 172.21.15.104:0/3179853205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcf1640 msgr2=0x55714bd0cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:20.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ab564d700 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcf1640 0x55714bd0cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ab564d700 1 -- 172.21.15.104:0/3179853205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55714bd0d820 con 0x55714bcf1f60 2024-06-09T22:30:20.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4aaffff700 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcf1640 0x55714bd0cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:30:20.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4ab564d700 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bd0d260 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800d010 tx=0x7f4aa8007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41370608023a9e4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.103+0000 7f4a8ffff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4a8ffff700 1 -- 172.21.15.104:0/3179853205 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8039030 con 0x55714bcf1f60 2024-06-09T22:30:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4a8ffff700 1 -- 172.21.15.104:0/3179853205 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f4aa800f050 con 0x55714bcf1f60 2024-06-09T22:30:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3179853205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4aa8007c20 con 0x55714bcf1f60 2024-06-09T22:30:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4a8ffff700 1 -- 172.21.15.104:0/3179853205 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4aa8007dd0 con 0x55714bcf1f60 2024-06-09T22:30:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4ac1933b80 10 client.17874 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:20.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3179853205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55714bd74800 con 0x55714bcf1f60 2024-06-09T22:30:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4a8ffff700 1 -- 172.21.15.104:0/3179853205 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4aa8007c20 con 0x55714bcf1f60 2024-06-09T22:30:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4a8ffff700 1 -- 172.21.15.104:0/3179853205 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4aa8011070 con 0x55714bcf1f60 2024-06-09T22:30:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4ac1933b80 10 client.17874 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4ac1933b80 10 client.17874 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:30:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4ac1933b80 20 client.17874 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55714bd74d40 0x55714bd77160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:20.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4ac1933b80 4 client.17874 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:30:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.104+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3179853205 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55714bc45290 con 0x55714bd74d40 2024-06-09T22:30:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55714bd74d40 0x55714bd77160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ab4e4c700 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55714bd74d40 0x55714bd77160 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cb797d11fc2241c server_cookie=6acd80390805f146 in_seq=0 out_seq=0 2024-06-09T22:30:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4a8ffff700 10 client.17874 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:30:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4a8ffff700 1 -- 172.21.15.104:0/3179853205 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55714bc45290 con 0x55714bd74d40 2024-06-09T22:30:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4a8ffff700 10 client.17874 handle_command_reply: tid=0 2024-06-09T22:30:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ac1933b80 1 client.17874 shutdown 2024-06-09T22:30:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3179853205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 msgr2=0x55714bd0d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bd0d260 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800d010 tx=0x7f4aa8007780 comp rx=0 tx=0).stop 2024-06-09T22:30:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3179853205 shutdown_connections 2024-06-09T22:30:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55714bcf1f60 0x55714bd0d260 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55714bd74d40 0x55714bd77160 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55714bcebf20 0x55714bcfe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ac1933b80 1 --2- 172.21.15.104:0/3179853205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55714bcf1640 0x55714bd0cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3179853205 >> 172.21.15.104:0/3179853205 conn(0x55714bc16cc0 msgr2=0x55714bcf2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4a8ffff700 0 client.17874 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:30:20.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3179853205 shutdown_connections 2024-06-09T22:30:20.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.105+0000 7f4ac1933b80 1 -- 172.21.15.104:0/3179853205 wait complete. 2024-06-09T22:30:20.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ac1933b80 20 client.17874 trim_cache size 0 max 16384 2024-06-09T22:30:20.119 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:30:20.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ab70b0700 1 -- 172.21.15.104:0/3395579723 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4aa00a2d90 msgr2=0x7f4aa00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:20.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ab70b0700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4aa00a2d90 0x7f4aa00a5250 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40104e0 tx=0x7f4aa400b3c0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ab70b0700 1 -- 172.21.15.104:0/3395579723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab0101fd0 msgr2=0x7f4ab019a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:20.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ab70b0700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab0101fd0 0x7f4ab019a020 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00d980 tx=0x7f4a9c00dc90 comp rx=0 tx=0).stop 2024-06-09T22:30:20.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ab70b0700 1 -- 172.21.15.104:0/3395579723 shutdown_connections 2024-06-09T22:30:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ab70b0700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f4aa00a2d90 0x7f4aa00a5250 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ab70b0700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ab01066b0 0x7f4ab01a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ab70b0700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ab0101620 0x7f4ab0199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ab70b0700 1 --2- 172.21.15.104:0/3395579723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ab0101fd0 0x7f4ab019a020 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ab70b0700 1 -- 172.21.15.104:0/3395579723 >> 172.21.15.104:0/3395579723 conn(0x7f4ab0078340 msgr2=0x7f4ab00ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ab70b0700 1 -- 172.21.15.104:0/3395579723 shutdown_connections 2024-06-09T22:30:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:20.106+0000 7f4ab70b0700 1 -- 172.21.15.104:0/3395579723 wait complete. 2024-06-09T22:30:20.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:20 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:30:20.939 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-09T22:30:20.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:20 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:30:21.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:20 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:30:21.186 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:30:21.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:21 smithi153 ceph-mon[36599]: pgmap v290: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-06-09T22:30:21.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:21 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:30:21.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:21 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:30:21.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:21 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:30:22.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:21 smithi104 ceph-mon[32389]: pgmap v290: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-06-09T22:30:22.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:21 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:30:22.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:21 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:30:22.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:21 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:30:22.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:21 smithi194 ceph-mon[36856]: pgmap v290: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-06-09T22:30:22.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:21 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:30:22.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:21 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:30:22.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:21 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:30:23.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.018+0000 7fc95a7d2700 1 -- 172.21.15.104:0/443218753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc954101d30 msgr2=0x7fc95410e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:23.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.018+0000 7fc95a7d2700 1 --2- 172.21.15.104:0/443218753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc954101d30 0x7fc95410e210 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00b0a0 tx=0x7fc94c00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.018+0000 7fc95a7d2700 1 -- 172.21.15.104:0/443218753 shutdown_connections 2024-06-09T22:30:23.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.018+0000 7fc95a7d2700 1 --2- 172.21.15.104:0/443218753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc954101d30 0x7fc95410e210 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.018+0000 7fc95a7d2700 1 --2- 172.21.15.104:0/443218753 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9541073d0 0x7fc9541077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.018+0000 7fc95a7d2700 1 --2- 172.21.15.104:0/443218753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc954101390 0x7fc9541017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.018+0000 7fc95a7d2700 1 -- 172.21.15.104:0/443218753 >> 172.21.15.104:0/443218753 conn(0x7fc9540782c0 msgr2=0x7fc9540786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:23.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.018+0000 7fc95a7d2700 1 -- 172.21.15.104:0/443218753 shutdown_connections 2024-06-09T22:30:23.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.018+0000 7fc95a7d2700 1 -- 172.21.15.104:0/443218753 wait complete. 2024-06-09T22:30:23.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.019+0000 7fc95a7d2700 1 Processor -- start 2024-06-09T22:30:23.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.019+0000 7fc95a7d2700 1 -- start start 2024-06-09T22:30:23.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc95a7d2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc954101390 0x7fc95419d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc95a7d2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc954101d30 0x7fc95419dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc95a7d2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9541073d0 0x7fc9541978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc95a7d2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc954111650 con 0x7fc954101d30 2024-06-09T22:30:23.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc95a7d2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541114d0 con 0x7fc954101390 2024-06-09T22:30:23.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc95a7d2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541117d0 con 0x7fc9541073d0 2024-06-09T22:30:23.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc9537fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc954101d30 0x7fc95419dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc9537fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc954101d30 0x7fc95419dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47366/0 (socket says 172.21.15.104:47366) 2024-06-09T22:30:23.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc9537fe700 1 -- 172.21.15.104:0/3572343270 learned_addr learned my addr 172.21.15.104:0/3572343270 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:23.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc958d6f700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9541073d0 0x7fc9541978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc953fff700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc954101390 0x7fc95419d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc9537fe700 1 -- 172.21.15.104:0/3572343270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9541073d0 msgr2=0x7fc9541978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:23.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc9537fe700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9541073d0 0x7fc9541978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc9537fe700 1 -- 172.21.15.104:0/3572343270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc954101390 msgr2=0x7fc95419d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:23.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc9537fe700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc954101390 0x7fc95419d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc9537fe700 1 -- 172.21.15.104:0/3572343270 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc954198050 con 0x7fc954101d30 2024-06-09T22:30:23.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc9537fe700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc954101d30 0x7fc95419dc70 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00c920 tx=0x7fc93c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:23.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc9517fa700 1 -- 172.21.15.104:0/3572343270 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c007b00 con 0x7fc954101d30 2024-06-09T22:30:23.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc95a7d2700 1 -- 172.21.15.104:0/3572343270 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc954198340 con 0x7fc954101d30 2024-06-09T22:30:23.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc9517fa700 1 -- 172.21.15.104:0/3572343270 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc93c004d10 con 0x7fc954101d30 2024-06-09T22:30:23.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc9517fa700 1 -- 172.21.15.104:0/3572343270 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c00f450 con 0x7fc954101d30 2024-06-09T22:30:23.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.020+0000 7fc95a7d2700 1 -- 172.21.15.104:0/3572343270 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9541a4600 con 0x7fc954101d30 2024-06-09T22:30:23.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.023+0000 7fc9517fa700 1 -- 172.21.15.104:0/3572343270 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc93c004750 con 0x7fc954101d30 2024-06-09T22:30:23.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.023+0000 7fc9517fa700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc9400a2e10 0x7fc9400a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.023+0000 7fc953fff700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc9400a2e10 0x7fc9400a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.023+0000 7fc9517fa700 1 -- 172.21.15.104:0/3572343270 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fc93c0cf7e0 con 0x7fc954101d30 2024-06-09T22:30:23.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.024+0000 7fc953fff700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc9400a2e10 0x7fc9400a52d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc944005fd0 tx=0x7fc944005d90 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:23.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.024+0000 7fc965055b80 1 Processor -- start 2024-06-09T22:30:23.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.024+0000 7fc965055b80 1 -- start start 2024-06-09T22:30:23.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.024+0000 7fc965055b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37640 0x558659c37a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.024+0000 7fc965055b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c37f60 0x558659c319e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.024+0000 7fc965055b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c31f20 0x558659c32340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.024+0000 7fc965055b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x558659b83810 con 0x558659c37640 2024-06-09T22:30:23.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.024+0000 7fc965055b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558659b2b720 con 0x558659c37f60 2024-06-09T22:30:23.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.024+0000 7fc965055b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558659b7f7d0 con 0x558659c31f20 2024-06-09T22:30:23.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.024+0000 7fc953fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37640 0x558659c37a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.024+0000 7fc953fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37640 0x558659c37a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47372/0 (socket says 172.21.15.104:47372) 2024-06-09T22:30:23.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.024+0000 7fc953fff700 1 -- 172.21.15.104:0/3210027398 learned_addr learned my addr 172.21.15.104:0/3210027398 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:23.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc9537fe700 1 --2- 172.21.15.104:0/3210027398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c37f60 0x558659c319e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc958d6f700 1 --2- 172.21.15.104:0/3210027398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c31f20 0x558659c32340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc953fff700 1 -- 172.21.15.104:0/3210027398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c31f20 msgr2=0x558659c32340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:23.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc953fff700 1 --2- 172.21.15.104:0/3210027398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c31f20 0x558659c32340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc953fff700 1 -- 172.21.15.104:0/3210027398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c37f60 msgr2=0x558659c319e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:23.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc953fff700 1 --2- 172.21.15.104:0/3210027398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c37f60 0x558659c319e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc953fff700 1 -- 172.21.15.104:0/3210027398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558659b21f50 con 0x558659c37640 2024-06-09T22:30:23.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc9537fe700 1 --2- 172.21.15.104:0/3210027398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c37f60 0x558659c319e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:30:23.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc958d6f700 1 --2- 172.21.15.104:0/3210027398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c31f20 0x558659c32340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:30:23.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc953fff700 1 --2- 172.21.15.104:0/3210027398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37640 0x558659c37a20 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc94400eba0 tx=0x7fc94400af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401f5b1f14b27c55 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:23.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc94a7fc700 1 -- 172.21.15.104:0/3210027398 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944028070 con 0x558659c37640 2024-06-09T22:30:23.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc94a7fc700 1 -- 172.21.15.104:0/3210027398 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc94401ad50 con 0x558659c37640 2024-06-09T22:30:23.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.025+0000 7fc94a7fc700 1 -- 172.21.15.104:0/3210027398 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944018d40 con 0x558659c37640 2024-06-09T22:30:23.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.026+0000 7fc965055b80 1 -- 172.21.15.104:0/3210027398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37640 msgr2=0x558659c37a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:23.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.026+0000 7fc965055b80 1 --2- 172.21.15.104:0/3210027398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37640 0x558659c37a20 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc94400eba0 tx=0x7fc94400af90 comp rx=0 tx=0).stop 2024-06-09T22:30:23.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.026+0000 7fc965055b80 1 -- 172.21.15.104:0/3210027398 shutdown_connections 2024-06-09T22:30:23.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.026+0000 7fc965055b80 1 --2- 172.21.15.104:0/3210027398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37640 0x558659c37a20 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.026+0000 7fc965055b80 1 --2- 172.21.15.104:0/3210027398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c31f20 0x558659c32340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.026+0000 7fc965055b80 1 --2- 172.21.15.104:0/3210027398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c37f60 0x558659c319e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.026+0000 7fc965055b80 1 -- 172.21.15.104:0/3210027398 >> 172.21.15.104:0/3210027398 conn(0x558659b5ccc0 msgr2=0x558659c2c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:23.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.026+0000 7fc965055b80 1 -- 172.21.15.104:0/3210027398 shutdown_connections 2024-06-09T22:30:23.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.026+0000 7fc965055b80 1 -- 172.21.15.104:0/3210027398 wait complete. 2024-06-09T22:30:23.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.026+0000 7fc965055b80 1 -- start start 2024-06-09T22:30:23.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc965055b80 1 Processor -- start 2024-06-09T22:30:23.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc965055b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c31f20 0x558659c3dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc965055b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 0x558659c3af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc965055b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c37f60 0x558659c3b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc965055b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x558659c3e9d0 con 0x558659c31f20 2024-06-09T22:30:23.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc965055b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558659b23bf0 con 0x558659c37f60 2024-06-09T22:30:23.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc965055b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558659c3eb50 con 0x558659c37640 2024-06-09T22:30:23.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc965055b80 10 client.0 _ll_register_callbacks cb 0x558659b1d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:23.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc953fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c31f20 0x558659c3dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc9537fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 0x558659c3af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc953fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c31f20 0x558659c3dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47386/0 (socket says 172.21.15.104:47386) 2024-06-09T22:30:23.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc953fff700 1 -- 172.21.15.104:0/2739827471 learned_addr learned my addr 172.21.15.104:0/2739827471 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:23.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc958d6f700 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c37f60 0x558659c3b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc953fff700 1 -- 172.21.15.104:0/2739827471 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 msgr2=0x558659c3af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:23.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc953fff700 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 0x558659c3af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc953fff700 1 -- 172.21.15.104:0/2739827471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c37f60 msgr2=0x558659c3b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:23.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc953fff700 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c37f60 0x558659c3b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc953fff700 1 -- 172.21.15.104:0/2739827471 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558659ad28d0 con 0x558659c31f20 2024-06-09T22:30:23.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.027+0000 7fc958d6f700 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c37f60 0x558659c3b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:30:23.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc953fff700 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c31f20 0x558659c3dc20 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fc94400a360 tx=0x7fc944019b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5d620f1a2e11df server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:23.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc9337fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:23.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc9337fe700 1 -- 172.21.15.104:0/2739827471 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944028070 con 0x558659c31f20 2024-06-09T22:30:23.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc9337fe700 1 -- 172.21.15.104:0/2739827471 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fc944018340 con 0x558659c31f20 2024-06-09T22:30:23.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc965055b80 1 -- 172.21.15.104:0/2739827471 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc94401ad50 con 0x558659c31f20 2024-06-09T22:30:23.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc9337fe700 1 -- 172.21.15.104:0/2739827471 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc944018540 con 0x558659c31f20 2024-06-09T22:30:23.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc965055b80 10 client.17886 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:23.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc965055b80 1 -- 172.21.15.104:0/2739827471 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558659c4d620 con 0x558659c31f20 2024-06-09T22:30:23.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc9337fe700 1 -- 172.21.15.104:0/2739827471 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc94401ad50 con 0x558659c31f20 2024-06-09T22:30:23.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc9337fe700 1 -- 172.21.15.104:0/2739827471 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc944005200 con 0x558659c31f20 2024-06-09T22:30:23.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc965055b80 10 client.17886 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:23.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc965055b80 10 client.17886 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:30:23.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc965055b80 20 client.17886 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:23.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc965055b80 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x558659c4fc00 0x558659c51ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc965055b80 4 client.17886 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:30:23.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.028+0000 7fc965055b80 1 -- 172.21.15.104:0/2739827471 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558659b90d10 con 0x558659c4fc00 2024-06-09T22:30:23.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.029+0000 7fc9537fe700 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x558659c4fc00 0x558659c51ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.029+0000 7fc9537fe700 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x558659c4fc00 0x558659c51ff0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a13eb701b3c95d85 server_cookie=dec23f814aa914eb in_seq=0 out_seq=0 2024-06-09T22:30:23.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.029+0000 7fc9337fe700 10 client.17886 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:30:23.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc9337fe700 1 -- 172.21.15.104:0/2739827471 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x558659b90d10 con 0x558659c4fc00 2024-06-09T22:30:23.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc9337fe700 10 client.17886 handle_command_reply: tid=0 2024-06-09T22:30:23.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc965055b80 1 client.17886 shutdown 2024-06-09T22:30:23.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc965055b80 1 -- 172.21.15.104:0/2739827471 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c31f20 msgr2=0x558659c3dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:23.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc965055b80 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c31f20 0x558659c3dc20 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fc94400a360 tx=0x7fc944019b80 comp rx=0 tx=0).stop 2024-06-09T22:30:23.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc965055b80 1 -- 172.21.15.104:0/2739827471 shutdown_connections 2024-06-09T22:30:23.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc965055b80 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x558659c4fc00 0x558659c51ff0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc965055b80 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c31f20 0x558659c3dc20 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc965055b80 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c37f60 0x558659c3b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc9337fe700 0 client.17886 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:30:23.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc965055b80 1 --2- 172.21.15.104:0/2739827471 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 0x558659c3af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc965055b80 1 -- 172.21.15.104:0/2739827471 >> 172.21.15.104:0/2739827471 conn(0x558659b5ccc0 msgr2=0x558659c309d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:23.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc965055b80 1 -- 172.21.15.104:0/2739827471 shutdown_connections 2024-06-09T22:30:23.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.030+0000 7fc965055b80 1 -- 172.21.15.104:0/2739827471 wait complete. 2024-06-09T22:30:23.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.031+0000 7fc965055b80 20 client.17886 trim_cache size 0 max 16384 2024-06-09T22:30:23.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.045+0000 7fc965055b80 1 Processor -- start 2024-06-09T22:30:23.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.045+0000 7fc965055b80 1 -- start start 2024-06-09T22:30:23.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc965055b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c31f20 0x558659c319e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc965055b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 0x558659c38bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc965055b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 0x558659c26510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc965055b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541ac4a0 con 0x558659c37f60 2024-06-09T22:30:23.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc965055b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541adfe0 con 0x558659c31f20 2024-06-09T22:30:23.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc965055b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541acb70 con 0x558659c37640 2024-06-09T22:30:23.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc958d6f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 0x558659c26510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc958d6f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 0x558659c26510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47396/0 (socket says 172.21.15.104:47396) 2024-06-09T22:30:23.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc958d6f700 1 -- 172.21.15.104:0/3017823446 learned_addr learned my addr 172.21.15.104:0/3017823446 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc953fff700 1 --2- 172.21.15.104:0/3017823446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c31f20 0x558659c319e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc9537fe700 1 --2- 172.21.15.104:0/3017823446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 0x558659c38bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc958d6f700 1 -- 172.21.15.104:0/3017823446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 msgr2=0x558659c38bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc958d6f700 1 --2- 172.21.15.104:0/3017823446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 0x558659c38bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc958d6f700 1 -- 172.21.15.104:0/3017823446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c31f20 msgr2=0x558659c319e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc958d6f700 1 --2- 172.21.15.104:0/3017823446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c31f20 0x558659c319e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc958d6f700 1 -- 172.21.15.104:0/3017823446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558659c3e420 con 0x558659c37f60 2024-06-09T22:30:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.046+0000 7fc958d6f700 1 --2- 172.21.15.104:0/3017823446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 0x558659c26510 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fc94c015000 tx=0x7fc94c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29b5db121dd6c8f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.047+0000 7fc949ffb700 1 -- 172.21.15.104:0/3017823446 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c009c70 con 0x558659c37f60 2024-06-09T22:30:23.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.047+0000 7fc949ffb700 1 -- 172.21.15.104:0/3017823446 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc94c0074e0 con 0x558659c37f60 2024-06-09T22:30:23.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.047+0000 7fc949ffb700 1 -- 172.21.15.104:0/3017823446 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c0052a0 con 0x558659c37f60 2024-06-09T22:30:23.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.047+0000 7fc965055b80 1 -- 172.21.15.104:0/3017823446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 msgr2=0x558659c26510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.047+0000 7fc965055b80 1 --2- 172.21.15.104:0/3017823446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 0x558659c26510 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fc94c015000 tx=0x7fc94c009260 comp rx=0 tx=0).stop 2024-06-09T22:30:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.047+0000 7fc965055b80 1 -- 172.21.15.104:0/3017823446 shutdown_connections 2024-06-09T22:30:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.047+0000 7fc965055b80 1 --2- 172.21.15.104:0/3017823446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 0x558659c26510 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.047+0000 7fc965055b80 1 --2- 172.21.15.104:0/3017823446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c31f20 0x558659c319e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.047+0000 7fc965055b80 1 --2- 172.21.15.104:0/3017823446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 0x558659c38bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.047+0000 7fc965055b80 1 -- 172.21.15.104:0/3017823446 >> 172.21.15.104:0/3017823446 conn(0x558659b5ccc0 msgr2=0x558659c25e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.047+0000 7fc965055b80 1 -- 172.21.15.104:0/3017823446 shutdown_connections 2024-06-09T22:30:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.047+0000 7fc965055b80 1 -- 172.21.15.104:0/3017823446 wait complete. 2024-06-09T22:30:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc965055b80 1 -- start start 2024-06-09T22:30:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc965055b80 1 Processor -- start 2024-06-09T22:30:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc965055b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c31f20 0x558659cb2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc965055b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 0x558659cb2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc965055b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 0x558659c54700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc965055b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541ad910 con 0x558659c37f60 2024-06-09T22:30:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc965055b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541ad240 con 0x558659c31f20 2024-06-09T22:30:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc965055b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541aa460 con 0x558659c37640 2024-06-09T22:30:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc965055b80 10 client.0 _ll_register_callbacks cb 0x558659b1d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc953fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c31f20 0x558659cb2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc958d6f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 0x558659c54700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc953fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c31f20 0x558659cb2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:53788/0 (socket says 172.21.15.104:53788) 2024-06-09T22:30:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc953fff700 1 -- 172.21.15.104:0/4003663979 learned_addr learned my addr 172.21.15.104:0/4003663979 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.048+0000 7fc9537fe700 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 0x558659cb2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc958d6f700 1 -- 172.21.15.104:0/4003663979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 msgr2=0x558659cb2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc958d6f700 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 0x558659cb2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc958d6f700 1 -- 172.21.15.104:0/4003663979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c31f20 msgr2=0x558659cb2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc958d6f700 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c31f20 0x558659cb2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc958d6f700 1 -- 172.21.15.104:0/4003663979 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558659c54e00 con 0x558659c37f60 2024-06-09T22:30:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc953fff700 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c31f20 0x558659cb2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:30:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc958d6f700 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 0x558659c54700 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00d010 tx=0x7fc94c0078c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc86e47285e76f2b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc9337fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc9337fe700 1 -- 172.21.15.104:0/4003663979 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c009c50 con 0x558659c37f60 2024-06-09T22:30:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc9337fe700 1 -- 172.21.15.104:0/4003663979 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fc94c00f040 con 0x558659c37f60 2024-06-09T22:30:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc965055b80 1 -- 172.21.15.104:0/4003663979 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc94c0074e0 con 0x558659c37f60 2024-06-09T22:30:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc9337fe700 1 -- 172.21.15.104:0/4003663979 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc94c036050 con 0x558659c37f60 2024-06-09T22:30:23.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc965055b80 10 client.17892 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:23.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc965055b80 1 -- 172.21.15.104:0/4003663979 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558659cba150 con 0x558659c37f60 2024-06-09T22:30:23.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc9337fe700 1 -- 172.21.15.104:0/4003663979 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc94c0074e0 con 0x558659c37f60 2024-06-09T22:30:23.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.049+0000 7fc9337fe700 1 -- 172.21.15.104:0/4003663979 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc94c011070 con 0x558659c37f60 2024-06-09T22:30:23.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.050+0000 7fc965055b80 10 client.17892 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:23.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.050+0000 7fc965055b80 10 client.17892 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:30:23.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.050+0000 7fc965055b80 20 client.17892 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:23.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.050+0000 7fc965055b80 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x558659cba6b0 0x558659cbcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.050+0000 7fc965055b80 4 client.17892 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:30:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.050+0000 7fc965055b80 1 -- 172.21.15.104:0/4003663979 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558659b8b290 con 0x558659cba6b0 2024-06-09T22:30:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.050+0000 7fc953fff700 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x558659cba6b0 0x558659cbcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.050+0000 7fc953fff700 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x558659cba6b0 0x558659cbcaa0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e21126171f7681ac server_cookie=56cd0d2b850470b5 in_seq=0 out_seq=0 2024-06-09T22:30:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.050+0000 7fc9337fe700 10 client.17892 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:30:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc9337fe700 1 -- 172.21.15.104:0/4003663979 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558659b8b290 con 0x558659cba6b0 2024-06-09T22:30:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc9337fe700 10 client.17892 handle_command_reply: tid=0 2024-06-09T22:30:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc965055b80 1 client.17892 shutdown 2024-06-09T22:30:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc965055b80 1 -- 172.21.15.104:0/4003663979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 msgr2=0x558659c54700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc965055b80 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 0x558659c54700 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00d010 tx=0x7fc94c0078c0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc965055b80 1 -- 172.21.15.104:0/4003663979 shutdown_connections 2024-06-09T22:30:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc965055b80 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x558659cba6b0 0x558659cbcaa0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc965055b80 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x558659c37f60 0x558659c54700 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc965055b80 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558659c31f20 0x558659cb2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc9337fe700 0 client.17892 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:30:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc965055b80 1 --2- 172.21.15.104:0/4003663979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558659c37640 0x558659cb2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc965055b80 1 -- 172.21.15.104:0/4003663979 >> 172.21.15.104:0/4003663979 conn(0x558659b5ccc0 msgr2=0x558659c26510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:23.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc965055b80 1 -- 172.21.15.104:0/4003663979 shutdown_connections 2024-06-09T22:30:23.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc965055b80 1 -- 172.21.15.104:0/4003663979 wait complete. 2024-06-09T22:30:23.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.051+0000 7fc965055b80 20 client.17892 trim_cache size 0 max 16384 2024-06-09T22:30:23.063 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:30:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.052+0000 7fc95a7d2700 1 -- 172.21.15.104:0/3572343270 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc9400a2e10 msgr2=0x7fc9400a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.052+0000 7fc95a7d2700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc9400a2e10 0x7fc9400a52d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc944005fd0 tx=0x7fc944005d90 comp rx=0 tx=0).stop 2024-06-09T22:30:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.052+0000 7fc95a7d2700 1 -- 172.21.15.104:0/3572343270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc954101d30 msgr2=0x7fc95419dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.052+0000 7fc95a7d2700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc954101d30 0x7fc95419dc70 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00c920 tx=0x7fc93c00cce0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.052+0000 7fc95a7d2700 1 -- 172.21.15.104:0/3572343270 shutdown_connections 2024-06-09T22:30:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.052+0000 7fc95a7d2700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc9400a2e10 0x7fc9400a52d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.052+0000 7fc95a7d2700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9541073d0 0x7fc9541978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.052+0000 7fc95a7d2700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc954101390 0x7fc95419d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.052+0000 7fc95a7d2700 1 --2- 172.21.15.104:0/3572343270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc954101d30 0x7fc95419dc70 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:23.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.052+0000 7fc95a7d2700 1 -- 172.21.15.104:0/3572343270 >> 172.21.15.104:0/3572343270 conn(0x7fc9540782c0 msgr2=0x7fc9540fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:23.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.052+0000 7fc95a7d2700 1 -- 172.21.15.104:0/3572343270 shutdown_connections 2024-06-09T22:30:23.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:23.052+0000 7fc95a7d2700 1 -- 172.21.15.104:0/3572343270 wait complete. 2024-06-09T22:30:23.848 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:23 smithi104 ceph-mon[32389]: pgmap v291: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-06-09T22:30:23.879 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-09T22:30:23.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:23 smithi153 ceph-mon[36599]: pgmap v291: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-06-09T22:30:24.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:23 smithi194 ceph-mon[36856]: pgmap v291: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-06-09T22:30:24.124 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:30:25.689 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:25 smithi104 ceph-mon[32389]: pgmap v292: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.3 KiB/s wr, 14 op/s 2024-06-09T22:30:25.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.936+0000 7f05d69b2700 1 -- 172.21.15.104:0/138246776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0107400 msgr2=0x7f05d01077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:25.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.936+0000 7f05d69b2700 1 --2- 172.21.15.104:0/138246776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0107400 0x7f05d01077e0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f05c0009990 tx=0x7f05c0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.936+0000 7f05d69b2700 1 -- 172.21.15.104:0/138246776 shutdown_connections 2024-06-09T22:30:25.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.936+0000 7f05d69b2700 1 --2- 172.21.15.104:0/138246776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0107400 0x7f05d01077e0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.936+0000 7f05d69b2700 1 --2- 172.21.15.104:0/138246776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05d0101d50 0x7f05d010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.936+0000 7f05d69b2700 1 --2- 172.21.15.104:0/138246776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d0101390 0x7f05d0101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.936+0000 7f05d69b2700 1 -- 172.21.15.104:0/138246776 >> 172.21.15.104:0/138246776 conn(0x7f05d0078340 msgr2=0x7f05d0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.936+0000 7f05d69b2700 1 -- 172.21.15.104:0/138246776 shutdown_connections 2024-06-09T22:30:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.936+0000 7f05d69b2700 1 -- 172.21.15.104:0/138246776 wait complete. 2024-06-09T22:30:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05d69b2700 1 Processor -- start 2024-06-09T22:30:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05d69b2700 1 -- start start 2024-06-09T22:30:25.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05d69b2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05d0101390 0x7f05d0197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05d69b2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d0101d50 0x7f05d0197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05d69b2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0107400 0x7f05d01983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05d69b2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01116f0 con 0x7f05d0107400 2024-06-09T22:30:25.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05d69b2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05d0111570 con 0x7f05d0101390 2024-06-09T22:30:25.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05d69b2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f05d0111870 con 0x7f05d0101d50 2024-06-09T22:30:25.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05d4f4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0107400 0x7f05d01983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05d4f4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0107400 0x7f05d01983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47426/0 (socket says 172.21.15.104:47426) 2024-06-09T22:30:25.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05d4f4f700 1 -- 172.21.15.104:0/3442387521 learned_addr learned my addr 172.21.15.104:0/3442387521 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:25.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05cffff700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05d0101390 0x7f05d0197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.937+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d0101d50 0x7f05d0197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.938+0000 7f05d4f4f700 1 -- 172.21.15.104:0/3442387521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d0101d50 msgr2=0x7f05d0197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:25.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.938+0000 7f05d4f4f700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d0101d50 0x7f05d0197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.938+0000 7f05d4f4f700 1 -- 172.21.15.104:0/3442387521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05d0101390 msgr2=0x7f05d0197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:25.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.938+0000 7f05d4f4f700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05d0101390 0x7f05d0197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.938+0000 7f05d4f4f700 1 -- 172.21.15.104:0/3442387521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d01a7150 con 0x7f05d0107400 2024-06-09T22:30:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.938+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d0101d50 0x7f05d0197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:30:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.938+0000 7f05d4f4f700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0107400 0x7f05d01983c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f05c800ada0 tx=0x7f05c800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.938+0000 7f05cd7fa700 1 -- 172.21.15.104:0/3442387521 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8017020 con 0x7f05d0107400 2024-06-09T22:30:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.938+0000 7f05cd7fa700 1 -- 172.21.15.104:0/3442387521 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05c800a0d0 con 0x7f05d0107400 2024-06-09T22:30:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.938+0000 7f05d69b2700 1 -- 172.21.15.104:0/3442387521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d01a7440 con 0x7f05d0107400 2024-06-09T22:30:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.938+0000 7f05cd7fa700 1 -- 172.21.15.104:0/3442387521 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8005480 con 0x7f05d0107400 2024-06-09T22:30:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.938+0000 7f05d69b2700 1 -- 172.21.15.104:0/3442387521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d01a7980 con 0x7f05d0107400 2024-06-09T22:30:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.940+0000 7f05cd7fa700 1 -- 172.21.15.104:0/3442387521 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f05c8005620 con 0x7f05d0107400 2024-06-09T22:30:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.941+0000 7f05cd7fa700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f05b80a2d90 0x7f05b80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.941+0000 7f05cffff700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f05b80a2d90 0x7f05b80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.941+0000 7f05cd7fa700 1 -- 172.21.15.104:0/3442387521 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f05c80cef30 con 0x7f05d0107400 2024-06-09T22:30:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.941+0000 7f05cffff700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f05b80a2d90 0x7f05b80a5250 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f05c00098e0 tx=0x7f05c0005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.941+0000 7f05e1235b80 1 Processor -- start 2024-06-09T22:30:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.941+0000 7f05e1235b80 1 -- start start 2024-06-09T22:30:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05e1235b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083640 0x5559c5083a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05e1235b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083f60 0x5559c507d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05e1235b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c507e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05e1235b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5559c4fcf810 con 0x5559c5083f60 2024-06-09T22:30:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05e1235b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5559c4f77720 con 0x5559c507df20 2024-06-09T22:30:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05e1235b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5559c4fcb7d0 con 0x5559c5083640 2024-06-09T22:30:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05cf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083f60 0x5559c507d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05cf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083f60 0x5559c507d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47442/0 (socket says 172.21.15.104:47442) 2024-06-09T22:30:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05cf7fe700 1 -- 172.21.15.104:0/3594293288 learned_addr learned my addr 172.21.15.104:0/3594293288 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05cffff700 1 --2- 172.21.15.104:0/3594293288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083640 0x5559c5083a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05d4f4f700 1 --2- 172.21.15.104:0/3594293288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c507e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05cf7fe700 1 -- 172.21.15.104:0/3594293288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083640 msgr2=0x5559c5083a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/3594293288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083640 0x5559c5083a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05cf7fe700 1 -- 172.21.15.104:0/3594293288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 msgr2=0x5559c507e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/3594293288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c507e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05cf7fe700 1 -- 172.21.15.104:0/3594293288 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559c4f6df50 con 0x5559c5083f60 2024-06-09T22:30:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05cffff700 1 --2- 172.21.15.104:0/3594293288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083640 0x5559c5083a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:30:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/3594293288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083f60 0x5559c507d9e0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f05c4007870 tx=0x7f05c4007b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e760d7097514b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05b67fc700 1 -- 172.21.15.104:0/3594293288 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c400d670 con 0x5559c5083f60 2024-06-09T22:30:25.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05b67fc700 1 -- 172.21.15.104:0/3594293288 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05c4018070 con 0x5559c5083f60 2024-06-09T22:30:25.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.942+0000 7f05b67fc700 1 -- 172.21.15.104:0/3594293288 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c4013ca0 con 0x5559c5083f60 2024-06-09T22:30:25.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.943+0000 7f05e1235b80 1 -- 172.21.15.104:0/3594293288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083f60 msgr2=0x5559c507d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:25.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.943+0000 7f05e1235b80 1 --2- 172.21.15.104:0/3594293288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083f60 0x5559c507d9e0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f05c4007870 tx=0x7f05c4007b80 comp rx=0 tx=0).stop 2024-06-09T22:30:25.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.943+0000 7f05e1235b80 1 -- 172.21.15.104:0/3594293288 shutdown_connections 2024-06-09T22:30:25.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.943+0000 7f05e1235b80 1 --2- 172.21.15.104:0/3594293288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c507e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.943+0000 7f05e1235b80 1 --2- 172.21.15.104:0/3594293288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083640 0x5559c5083a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.943+0000 7f05e1235b80 1 --2- 172.21.15.104:0/3594293288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083f60 0x5559c507d9e0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.943+0000 7f05e1235b80 1 -- 172.21.15.104:0/3594293288 >> 172.21.15.104:0/3594293288 conn(0x5559c4fa8cc0 msgr2=0x5559c5078850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:25.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.943+0000 7f05e1235b80 1 -- 172.21.15.104:0/3594293288 shutdown_connections 2024-06-09T22:30:25.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.943+0000 7f05e1235b80 1 -- 172.21.15.104:0/3594293288 wait complete. 2024-06-09T22:30:25.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.943+0000 7f05e1235b80 1 -- start start 2024-06-09T22:30:25.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05e1235b80 1 Processor -- start 2024-06-09T22:30:25.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05e1235b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c507df20 0x5559c5089ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05e1235b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5086fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05e1235b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c5083f60 0x5559c50874e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05e1235b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5559c508a9d0 con 0x5559c5083640 2024-06-09T22:30:25.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05e1235b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5559c4f6fbf0 con 0x5559c5083f60 2024-06-09T22:30:25.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05e1235b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5559c508ab50 con 0x5559c507df20 2024-06-09T22:30:25.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05e1235b80 10 client.0 _ll_register_callbacks cb 0x5559c4f69920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:25.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05cf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5086fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05cf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5086fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47458/0 (socket says 172.21.15.104:47458) 2024-06-09T22:30:25.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05cf7fe700 1 -- 172.21.15.104:0/1085238937 learned_addr learned my addr 172.21.15.104:0/1085238937 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:25.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05d4f4f700 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c5083f60 0x5559c50874e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05cffff700 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c507df20 0x5559c5089ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05cf7fe700 1 -- 172.21.15.104:0/1085238937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c507df20 msgr2=0x5559c5089ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:25.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c507df20 0x5559c5089ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05cf7fe700 1 -- 172.21.15.104:0/1085238937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c5083f60 msgr2=0x5559c50874e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:25.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c5083f60 0x5559c50874e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05cf7fe700 1 -- 172.21.15.104:0/1085238937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559c4f1e8d0 con 0x5559c5083640 2024-06-09T22:30:25.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.944+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5086fa0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f05c4006e50 tx=0x7f05c4013920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c236baae2585e4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:25.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05af7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:25.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05af7fe700 1 -- 172.21.15.104:0/1085238937 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c4013e50 con 0x5559c5083640 2024-06-09T22:30:25.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05af7fe700 1 -- 172.21.15.104:0/1085238937 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f05c400d670 con 0x5559c5083640 2024-06-09T22:30:25.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05e1235b80 1 -- 172.21.15.104:0/1085238937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05c4018070 con 0x5559c5083640 2024-06-09T22:30:25.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05af7fe700 1 -- 172.21.15.104:0/1085238937 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f05c4030050 con 0x5559c5083640 2024-06-09T22:30:25.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05e1235b80 10 client.17904 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:25.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05e1235b80 1 -- 172.21.15.104:0/1085238937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5559c5095220 con 0x5559c5083640 2024-06-09T22:30:25.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05af7fe700 1 -- 172.21.15.104:0/1085238937 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05c4018070 con 0x5559c5083640 2024-06-09T22:30:25.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05af7fe700 1 -- 172.21.15.104:0/1085238937 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f05c401dab0 con 0x5559c5083640 2024-06-09T22:30:25.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05e1235b80 10 client.17904 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:25.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05e1235b80 10 client.17904 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:30:25.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05e1235b80 20 client.17904 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:25.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05e1235b80 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5559c5097800 0x5559c5099bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05e1235b80 4 client.17904 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:30:25.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.945+0000 7f05e1235b80 1 -- 172.21.15.104:0/1085238937 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559c4fdcd10 con 0x5559c5097800 2024-06-09T22:30:25.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.946+0000 7f05cffff700 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5559c5097800 0x5559c5099bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.946+0000 7f05cffff700 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5559c5097800 0x5559c5099bf0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=85f576fb54d9ef8 server_cookie=34de126deba05753 in_seq=0 out_seq=0 2024-06-09T22:30:25.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.946+0000 7f05af7fe700 10 client.17904 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:30:25.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05af7fe700 1 -- 172.21.15.104:0/1085238937 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5559c4fdcd10 con 0x5559c5097800 2024-06-09T22:30:25.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05af7fe700 10 client.17904 handle_command_reply: tid=0 2024-06-09T22:30:25.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05e1235b80 1 client.17904 shutdown 2024-06-09T22:30:25.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05e1235b80 1 -- 172.21.15.104:0/1085238937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 msgr2=0x5559c5086fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:25.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05e1235b80 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5086fa0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f05c4006e50 tx=0x7f05c4013920 comp rx=0 tx=0).stop 2024-06-09T22:30:25.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05e1235b80 1 -- 172.21.15.104:0/1085238937 shutdown_connections 2024-06-09T22:30:25.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05e1235b80 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5559c5097800 0x5559c5099bf0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05e1235b80 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c5083f60 0x5559c50874e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05e1235b80 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c507df20 0x5559c5089ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05af7fe700 0 client.17904 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:30:25.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05e1235b80 1 --2- 172.21.15.104:0/1085238937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5086fa0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05e1235b80 1 -- 172.21.15.104:0/1085238937 >> 172.21.15.104:0/1085238937 conn(0x5559c4fa8cc0 msgr2=0x5559c507c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:25.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05e1235b80 1 -- 172.21.15.104:0/1085238937 shutdown_connections 2024-06-09T22:30:25.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05e1235b80 1 -- 172.21.15.104:0/1085238937 wait complete. 2024-06-09T22:30:25.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.947+0000 7f05e1235b80 20 client.17904 trim_cache size 0 max 16384 2024-06-09T22:30:25.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05e1235b80 1 Processor -- start 2024-06-09T22:30:25.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05e1235b80 1 -- start start 2024-06-09T22:30:25.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05e1235b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c507d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05e1235b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5072510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05e1235b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083f60 0x5559c5086850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:25.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05e1235b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01b0880 con 0x5559c5083640 2024-06-09T22:30:25.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05e1235b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01b23c0 con 0x5559c507df20 2024-06-09T22:30:25.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05e1235b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01b0f50 con 0x5559c5083f60 2024-06-09T22:30:25.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05cf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5072510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05cf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5072510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47462/0 (socket says 172.21.15.104:47462) 2024-06-09T22:30:25.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2398413749 learned_addr learned my addr 172.21.15.104:0/2398413749 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:25.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2398413749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083f60 msgr2=0x5559c5086850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:25.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/2398413749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083f60 0x5559c5086850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05cffff700 1 --2- 172.21.15.104:0/2398413749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c507d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:25.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2398413749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 msgr2=0x5559c507d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:25.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/2398413749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c507d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:25.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2398413749 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559c507a470 con 0x5559c5083640 2024-06-09T22:30:25.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.962+0000 7f05cffff700 1 --2- 172.21.15.104:0/2398413749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c507d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:30:25.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/2398413749 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5072510 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f05c401f000 tx=0x7f05c401b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8752a1dde6b9ba86 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:25.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05b5ffb700 1 -- 172.21.15.104:0/2398413749 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c401b880 con 0x5559c5083640 2024-06-09T22:30:25.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05b5ffb700 1 -- 172.21.15.104:0/2398413749 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05c4013e50 con 0x5559c5083640 2024-06-09T22:30:25.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05b5ffb700 1 -- 172.21.15.104:0/2398413749 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c400ddc0 con 0x5559c5083640 2024-06-09T22:30:25.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05e1235b80 1 -- 172.21.15.104:0/2398413749 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 msgr2=0x5559c5072510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:26.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05e1235b80 1 --2- 172.21.15.104:0/2398413749 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5072510 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f05c401f000 tx=0x7f05c401b560 comp rx=0 tx=0).stop 2024-06-09T22:30:26.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05e1235b80 1 -- 172.21.15.104:0/2398413749 shutdown_connections 2024-06-09T22:30:26.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05e1235b80 1 --2- 172.21.15.104:0/2398413749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083f60 0x5559c5086850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05e1235b80 1 --2- 172.21.15.104:0/2398413749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c507d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05e1235b80 1 --2- 172.21.15.104:0/2398413749 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5072510 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05e1235b80 1 -- 172.21.15.104:0/2398413749 >> 172.21.15.104:0/2398413749 conn(0x5559c4fa8cc0 msgr2=0x5559c5071e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:26.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05e1235b80 1 -- 172.21.15.104:0/2398413749 shutdown_connections 2024-06-09T22:30:26.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.963+0000 7f05e1235b80 1 -- 172.21.15.104:0/2398413749 wait complete. 2024-06-09T22:30:26.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.964+0000 7f05e1235b80 1 -- start start 2024-06-09T22:30:26.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.964+0000 7f05e1235b80 1 Processor -- start 2024-06-09T22:30:26.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.964+0000 7f05e1235b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c508fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:26.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.964+0000 7f05e1235b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5090530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:26.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.964+0000 7f05e1235b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083f60 0x5559c509e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:26.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.964+0000 7f05e1235b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01b1cf0 con 0x5559c5083640 2024-06-09T22:30:26.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.964+0000 7f05e1235b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01b1620 con 0x5559c507df20 2024-06-09T22:30:26.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.964+0000 7f05e1235b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01ae840 con 0x5559c5083f60 2024-06-09T22:30:26.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.964+0000 7f05e1235b80 10 client.0 _ll_register_callbacks cb 0x5559c4f69920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:26.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.964+0000 7f05cf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5090530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:26.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.964+0000 7f05cf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5090530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47478/0 (socket says 172.21.15.104:47478) 2024-06-09T22:30:26.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.964+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2798872531 learned_addr learned my addr 172.21.15.104:0/2798872531 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:26.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05d4f4f700 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083f60 0x5559c509e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:26.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05cffff700 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c508fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:26.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2798872531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083f60 msgr2=0x5559c509e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:26.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083f60 0x5559c509e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2798872531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 msgr2=0x5559c508fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c508fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2798872531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559c509ec50 con 0x5559c5083640 2024-06-09T22:30:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05d4f4f700 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083f60 0x5559c509e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:30:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5090530 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f05c401b3b0 tx=0x7f05c4007550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d46fbb50974fd2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05af7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05af7fe700 1 -- 172.21.15.104:0/2798872531 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c4038070 con 0x5559c5083640 2024-06-09T22:30:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05af7fe700 1 -- 172.21.15.104:0/2798872531 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f05c400dc50 con 0x5559c5083640 2024-06-09T22:30:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05e1235b80 1 -- 172.21.15.104:0/2798872531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05c4013e50 con 0x5559c5083640 2024-06-09T22:30:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05af7fe700 1 -- 172.21.15.104:0/2798872531 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f05c403f080 con 0x5559c5083640 2024-06-09T22:30:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05e1235b80 10 client.17910 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05e1235b80 1 -- 172.21.15.104:0/2798872531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5559c5101ac0 con 0x5559c5083640 2024-06-09T22:30:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05af7fe700 1 -- 172.21.15.104:0/2798872531 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05c4013e50 con 0x5559c5083640 2024-06-09T22:30:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05af7fe700 1 -- 172.21.15.104:0/2798872531 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f05c401d7a0 con 0x5559c5083640 2024-06-09T22:30:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05e1235b80 10 client.17910 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.965+0000 7f05e1235b80 10 client.17910 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:30:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.966+0000 7f05e1235b80 20 client.17910 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.966+0000 7f05e1235b80 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5559c5102100 0x5559c5104520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.966+0000 7f05e1235b80 4 client.17910 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:30:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.966+0000 7f05e1235b80 1 -- 172.21.15.104:0/2798872531 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5559c4fd7290 con 0x5559c5102100 2024-06-09T22:30:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.966+0000 7f05cffff700 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5559c5102100 0x5559c5104520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.966+0000 7f05af7fe700 10 client.17910 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:30:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.966+0000 7f05cffff700 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5559c5102100 0x5559c5104520 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ef3e31ddfb9baba server_cookie=b75c37f1b14bcb3a in_seq=0 out_seq=0 2024-06-09T22:30:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.966+0000 7f05af7fe700 1 -- 172.21.15.104:0/2798872531 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5559c4fd7290 con 0x5559c5102100 2024-06-09T22:30:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.966+0000 7f05af7fe700 10 client.17910 handle_command_reply: tid=0 2024-06-09T22:30:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.966+0000 7f05e1235b80 1 client.17910 shutdown 2024-06-09T22:30:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05e1235b80 1 -- 172.21.15.104:0/2798872531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 msgr2=0x5559c5090530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05e1235b80 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5090530 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f05c401b3b0 tx=0x7f05c4007550 comp rx=0 tx=0).stop 2024-06-09T22:30:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05e1235b80 1 -- 172.21.15.104:0/2798872531 shutdown_connections 2024-06-09T22:30:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05e1235b80 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x5559c5102100 0x5559c5104520 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.006 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:30:26.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05e1235b80 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5559c5083f60 0x5559c509e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05e1235b80 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5559c507df20 0x5559c508fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05af7fe700 0 client.17910 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:30:26.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05e1235b80 1 --2- 172.21.15.104:0/2798872531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5559c5083640 0x5559c5090530 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05e1235b80 1 -- 172.21.15.104:0/2798872531 >> 172.21.15.104:0/2798872531 conn(0x5559c4fa8cc0 msgr2=0x5559c5084a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:26.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05e1235b80 1 -- 172.21.15.104:0/2798872531 shutdown_connections 2024-06-09T22:30:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05e1235b80 1 -- 172.21.15.104:0/2798872531 wait complete. 2024-06-09T22:30:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05e1235b80 20 client.17910 trim_cache size 0 max 16384 2024-06-09T22:30:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05d69b2700 1 -- 172.21.15.104:0/3442387521 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f05b80a2d90 msgr2=0x7f05b80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05d69b2700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f05b80a2d90 0x7f05b80a5250 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f05c00098e0 tx=0x7f05c0005960 comp rx=0 tx=0).stop 2024-06-09T22:30:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05d69b2700 1 -- 172.21.15.104:0/3442387521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0107400 msgr2=0x7f05d01983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05d69b2700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0107400 0x7f05d01983c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f05c800ada0 tx=0x7f05c800c330 comp rx=0 tx=0).stop 2024-06-09T22:30:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05d69b2700 1 -- 172.21.15.104:0/3442387521 shutdown_connections 2024-06-09T22:30:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05d69b2700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f05b80a2d90 0x7f05b80a5250 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05d69b2700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0107400 0x7f05d01983c0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05d69b2700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05d0101390 0x7f05d0197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05d69b2700 1 --2- 172.21.15.104:0/3442387521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d0101d50 0x7f05d0197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:26.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05d69b2700 1 -- 172.21.15.104:0/3442387521 >> 172.21.15.104:0/3442387521 conn(0x7f05d0078340 msgr2=0x7f05d0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:26.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05d69b2700 1 -- 172.21.15.104:0/3442387521 shutdown_connections 2024-06-09T22:30:26.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:25.967+0000 7f05d69b2700 1 -- 172.21.15.104:0/3442387521 wait complete. 2024-06-09T22:30:26.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:25 smithi153 ceph-mon[36599]: pgmap v292: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.3 KiB/s wr, 14 op/s 2024-06-09T22:30:26.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:25 smithi194 ceph-mon[36856]: pgmap v292: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.3 KiB/s wr, 14 op/s 2024-06-09T22:30:26.881 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-09T22:30:27.122 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:30:27.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:27 smithi153 ceph-mon[36599]: pgmap v293: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2024-06-09T22:30:28.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:27 smithi104 ceph-mon[32389]: pgmap v293: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2024-06-09T22:30:28.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:27 smithi194 ceph-mon[36856]: pgmap v293: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2024-06-09T22:30:28.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.971+0000 7fc597f48700 1 -- 172.21.15.104:0/4177397466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc59006b580 msgr2=0x7fc59010f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:28.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.971+0000 7fc597f48700 1 --2- 172.21.15.104:0/4177397466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc59006b580 0x7fc59010f320 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc58800b0a0 tx=0x7fc58800b3b0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.971+0000 7fc597f48700 1 -- 172.21.15.104:0/4177397466 shutdown_connections 2024-06-09T22:30:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.971+0000 7fc597f48700 1 --2- 172.21.15.104:0/4177397466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc59006b580 0x7fc59010f320 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.971+0000 7fc597f48700 1 --2- 172.21.15.104:0/4177397466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc590108500 0x7fc5901088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.971+0000 7fc597f48700 1 --2- 172.21.15.104:0/4177397466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc59006abc0 0x7fc59006b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.971+0000 7fc597f48700 1 -- 172.21.15.104:0/4177397466 >> 172.21.15.104:0/4177397466 conn(0x7fc590078340 msgr2=0x7fc590078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.971+0000 7fc597f48700 1 -- 172.21.15.104:0/4177397466 shutdown_connections 2024-06-09T22:30:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.971+0000 7fc597f48700 1 -- 172.21.15.104:0/4177397466 wait complete. 2024-06-09T22:30:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.971+0000 7fc597f48700 1 Processor -- start 2024-06-09T22:30:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc597f48700 1 -- start start 2024-06-09T22:30:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc597f48700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc59006abc0 0x7fc59019e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc597f48700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59006b580 0x7fc59019ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc597f48700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc590108500 0x7fc590198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc597f48700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901127e0 con 0x7fc59006abc0 2024-06-09T22:30:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc597f48700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc590112660 con 0x7fc59006b580 2024-06-09T22:30:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc597f48700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc590112960 con 0x7fc590108500 2024-06-09T22:30:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc595ce4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc59006abc0 0x7fc59019e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:28.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc595ce4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc59006abc0 0x7fc59019e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36110/0 (socket says 172.21.15.104:36110) 2024-06-09T22:30:28.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc595ce4700 1 -- 172.21.15.104:0/405450896 learned_addr learned my addr 172.21.15.104:0/405450896 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:28.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc5964e5700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc590108500 0x7fc590198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:28.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc5954e3700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59006b580 0x7fc59019ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:28.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc595ce4700 1 -- 172.21.15.104:0/405450896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc590108500 msgr2=0x7fc590198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:28.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc595ce4700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc590108500 0x7fc590198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc595ce4700 1 -- 172.21.15.104:0/405450896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59006b580 msgr2=0x7fc59019ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:28.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc595ce4700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59006b580 0x7fc59019ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc595ce4700 1 -- 172.21.15.104:0/405450896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5901991c0 con 0x7fc59006abc0 2024-06-09T22:30:28.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc5954e3700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59006b580 0x7fc59019ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:30:28.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.972+0000 7fc595ce4700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc59006abc0 0x7fc59019e8a0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fc58000c9b0 tx=0x7fc58000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:28.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.973+0000 7fc586ffd700 1 -- 172.21.15.104:0/405450896 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580007b00 con 0x7fc59006abc0 2024-06-09T22:30:28.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.973+0000 7fc597f48700 1 -- 172.21.15.104:0/405450896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5901994b0 con 0x7fc59006abc0 2024-06-09T22:30:28.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.973+0000 7fc586ffd700 1 -- 172.21.15.104:0/405450896 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc580007ca0 con 0x7fc59006abc0 2024-06-09T22:30:28.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.973+0000 7fc586ffd700 1 -- 172.21.15.104:0/405450896 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580018780 con 0x7fc59006abc0 2024-06-09T22:30:28.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.973+0000 7fc597f48700 1 -- 172.21.15.104:0/405450896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5901a5770 con 0x7fc59006abc0 2024-06-09T22:30:28.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.975+0000 7fc586ffd700 1 -- 172.21.15.104:0/405450896 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc58000f450 con 0x7fc59006abc0 2024-06-09T22:30:28.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.975+0000 7fc586ffd700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc57c0a2d90 0x7fc57c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:28.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.975+0000 7fc5954e3700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc57c0a2d90 0x7fc57c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:28.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.975+0000 7fc586ffd700 1 -- 172.21.15.104:0/405450896 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fc5800cf8f0 con 0x7fc59006abc0 2024-06-09T22:30:28.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.976+0000 7fc5954e3700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc57c0a2d90 0x7fc57c0a5250 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc590078940 tx=0x7fc58c008040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:28.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.976+0000 7fc5a27cbb80 1 Processor -- start 2024-06-09T22:30:28.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.976+0000 7fc5a27cbb80 1 -- start start 2024-06-09T22:30:28.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5a27cbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99640 0x5645b5b99a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:28.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5a27cbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 0x5645b5b939e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:28.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5a27cbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b94340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:28.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5a27cbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5645b5ae5810 con 0x5645b5b93f20 2024-06-09T22:30:28.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5a27cbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5645b5a8d720 con 0x5645b5b99f60 2024-06-09T22:30:28.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5a27cbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5645b5ae17d0 con 0x5645b5b99640 2024-06-09T22:30:28.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5964e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b94340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:28.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5964e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b94340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36120/0 (socket says 172.21.15.104:36120) 2024-06-09T22:30:28.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5964e5700 1 -- 172.21.15.104:0/1286083509 learned_addr learned my addr 172.21.15.104:0/1286083509 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:28.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc595ce4700 1 --2- 172.21.15.104:0/1286083509 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99640 0x5645b5b99a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:28.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5954e3700 1 --2- 172.21.15.104:0/1286083509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 0x5645b5b939e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:28.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5964e5700 1 -- 172.21.15.104:0/1286083509 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99640 msgr2=0x5645b5b99a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:28.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5964e5700 1 --2- 172.21.15.104:0/1286083509 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99640 0x5645b5b99a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5964e5700 1 -- 172.21.15.104:0/1286083509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 msgr2=0x5645b5b939e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:28.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5964e5700 1 --2- 172.21.15.104:0/1286083509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 0x5645b5b939e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5964e5700 1 -- 172.21.15.104:0/1286083509 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645b5a83f50 con 0x5645b5b93f20 2024-06-09T22:30:28.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc595ce4700 1 --2- 172.21.15.104:0/1286083509 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99640 0x5645b5b99a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:30:28.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.977+0000 7fc5964e5700 1 --2- 172.21.15.104:0/1286083509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b94340 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fc58800b070 tx=0x7fc588009150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8f584b99a16cf server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:28.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.978+0000 7fc57bfff700 1 -- 172.21.15.104:0/1286083509 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc588021070 con 0x5645b5b93f20 2024-06-09T22:30:28.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.978+0000 7fc57bfff700 1 -- 172.21.15.104:0/1286083509 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5880074e0 con 0x5645b5b93f20 2024-06-09T22:30:28.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.978+0000 7fc57bfff700 1 -- 172.21.15.104:0/1286083509 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc588005280 con 0x5645b5b93f20 2024-06-09T22:30:28.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.978+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1286083509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 msgr2=0x5645b5b94340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:28.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.978+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/1286083509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b94340 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fc58800b070 tx=0x7fc588009150 comp rx=0 tx=0).stop 2024-06-09T22:30:28.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.978+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1286083509 shutdown_connections 2024-06-09T22:30:28.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.978+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/1286083509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b94340 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.978+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/1286083509 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99640 0x5645b5b99a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.978+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/1286083509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 0x5645b5b939e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.978+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1286083509 >> 172.21.15.104:0/1286083509 conn(0x5645b5abecc0 msgr2=0x5645b5b8e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:28.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.979+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1286083509 shutdown_connections 2024-06-09T22:30:28.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.979+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1286083509 wait complete. 2024-06-09T22:30:28.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.979+0000 7fc5a27cbb80 1 -- start start 2024-06-09T22:30:28.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.979+0000 7fc5a27cbb80 1 Processor -- start 2024-06-09T22:30:28.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.979+0000 7fc5a27cbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b9fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:28.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.979+0000 7fc5a27cbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99640 0x5645b5b9cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:28.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.979+0000 7fc5a27cbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99f60 0x5645b5b9d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:28.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc5a27cbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5645b5ba0970 con 0x5645b5b93f20 2024-06-09T22:30:28.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc5a27cbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5645b5a85bf0 con 0x5645b5b99640 2024-06-09T22:30:28.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc5a27cbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5645b5ba0af0 con 0x5645b5b99f60 2024-06-09T22:30:28.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc5a27cbb80 10 client.0 _ll_register_callbacks cb 0x5645b5a7f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:28.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc595ce4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b9fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:28.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc595ce4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b9fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36130/0 (socket says 172.21.15.104:36130) 2024-06-09T22:30:28.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc595ce4700 1 -- 172.21.15.104:0/3503857284 learned_addr learned my addr 172.21.15.104:0/3503857284 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:28.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc5954e3700 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99640 0x5645b5b9cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:28.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc5964e5700 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99f60 0x5645b5b9d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:28.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc595ce4700 1 -- 172.21.15.104:0/3503857284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99f60 msgr2=0x5645b5b9d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:28.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc595ce4700 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99f60 0x5645b5b9d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc595ce4700 1 -- 172.21.15.104:0/3503857284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99640 msgr2=0x5645b5b9cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:28.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc595ce4700 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99640 0x5645b5b9cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc595ce4700 1 -- 172.21.15.104:0/3503857284 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645b5a348d0 con 0x5645b5b93f20 2024-06-09T22:30:28.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc595ce4700 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b9fc90 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fc5800074a0 tx=0x7fc58009ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c413af13404aae1b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:28.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc5797fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:28.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc5797fa700 1 -- 172.21.15.104:0/3503857284 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800b3020 con 0x5645b5b93f20 2024-06-09T22:30:28.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc5797fa700 1 -- 172.21.15.104:0/3503857284 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fc58000fa30 con 0x5645b5b93f20 2024-06-09T22:30:28.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3503857284 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5880074e0 con 0x5645b5b93f20 2024-06-09T22:30:28.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc5797fa700 1 -- 172.21.15.104:0/3503857284 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc5800b4be0 con 0x5645b5b93f20 2024-06-09T22:30:28.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.980+0000 7fc5a27cbb80 10 client.17922 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:28.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.981+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3503857284 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5645b5bab200 con 0x5645b5b93f20 2024-06-09T22:30:28.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.981+0000 7fc5797fa700 1 -- 172.21.15.104:0/3503857284 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5880074e0 con 0x5645b5b93f20 2024-06-09T22:30:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.981+0000 7fc5797fa700 1 -- 172.21.15.104:0/3503857284 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc5800a41c0 con 0x5645b5b93f20 2024-06-09T22:30:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.981+0000 7fc5a27cbb80 10 client.17922 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.981+0000 7fc5a27cbb80 10 client.17922 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:30:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.981+0000 7fc5a27cbb80 20 client.17922 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.981+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x5645b5bad790 0x5645b5bafbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.981+0000 7fc5a27cbb80 4 client.17922 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:30:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.981+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3503857284 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645b5af2d10 con 0x5645b5bad790 2024-06-09T22:30:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.981+0000 7fc5954e3700 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x5645b5bad790 0x5645b5bafbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.982+0000 7fc5797fa700 10 client.17922 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:30:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.982+0000 7fc5954e3700 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x5645b5bad790 0x5645b5bafbf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2f43027cf3a6699 server_cookie=30950cb6d0980708 in_seq=0 out_seq=0 2024-06-09T22:30:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.982+0000 7fc5797fa700 1 -- 172.21.15.104:0/3503857284 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5645b5af2d10 con 0x5645b5bad790 2024-06-09T22:30:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.982+0000 7fc5797fa700 10 client.17922 handle_command_reply: tid=0 2024-06-09T22:30:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.982+0000 7fc5a27cbb80 1 client.17922 shutdown 2024-06-09T22:30:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.983+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3503857284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 msgr2=0x5645b5b9fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.983+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b9fc90 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fc5800074a0 tx=0x7fc58009ba30 comp rx=0 tx=0).stop 2024-06-09T22:30:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.983+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3503857284 shutdown_connections 2024-06-09T22:30:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.983+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x5645b5bad790 0x5645b5bafbf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.983+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b9fc90 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.983+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99f60 0x5645b5b9d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.983+0000 7fc5797fa700 0 client.17922 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:30:28.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.983+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/3503857284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99640 0x5645b5b9cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:28.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.983+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3503857284 >> 172.21.15.104:0/3503857284 conn(0x5645b5abecc0 msgr2=0x5645b5b929d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:28.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.983+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3503857284 shutdown_connections 2024-06-09T22:30:28.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.983+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3503857284 wait complete. 2024-06-09T22:30:28.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.983+0000 7fc5a27cbb80 20 client.17922 trim_cache size 0 max 16384 2024-06-09T22:30:29.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc5a27cbb80 1 Processor -- start 2024-06-09T22:30:29.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc5a27cbb80 1 -- start start 2024-06-09T22:30:29.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc5a27cbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b939e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:29.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc5a27cbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99640 0x5645b5b9fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:29.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc5a27cbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 0x5645b5b88510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:29.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc5a27cbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901ae450 con 0x5645b5b93f20 2024-06-09T22:30:29.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc5a27cbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901aff90 con 0x5645b5b99f60 2024-06-09T22:30:29.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc5a27cbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901aeb20 con 0x5645b5b99640 2024-06-09T22:30:29.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc595ce4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b939e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:29.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc595ce4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b939e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36132/0 (socket says 172.21.15.104:36132) 2024-06-09T22:30:29.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc595ce4700 1 -- 172.21.15.104:0/3201948374 learned_addr learned my addr 172.21.15.104:0/3201948374 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:29.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc5954e3700 1 --2- 172.21.15.104:0/3201948374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99640 0x5645b5b9fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:29.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc595ce4700 1 -- 172.21.15.104:0/3201948374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99640 msgr2=0x5645b5b9fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:29.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc595ce4700 1 --2- 172.21.15.104:0/3201948374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99640 0x5645b5b9fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc595ce4700 1 -- 172.21.15.104:0/3201948374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 msgr2=0x5645b5b88510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:29.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc595ce4700 1 --2- 172.21.15.104:0/3201948374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 0x5645b5b88510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.998+0000 7fc595ce4700 1 -- 172.21.15.104:0/3201948374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645b5b88a50 con 0x5645b5b93f20 2024-06-09T22:30:29.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc595ce4700 1 --2- 172.21.15.104:0/3201948374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b939e0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fc58000afd0 tx=0x7fc58000a4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7fa0916e0f0d3ad server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:29.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc57b7fe700 1 -- 172.21.15.104:0/3201948374 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800b3070 con 0x5645b5b93f20 2024-06-09T22:30:29.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc57b7fe700 1 -- 172.21.15.104:0/3201948374 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc58009fd90 con 0x5645b5b93f20 2024-06-09T22:30:29.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc57b7fe700 1 -- 172.21.15.104:0/3201948374 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800cf470 con 0x5645b5b93f20 2024-06-09T22:30:29.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3201948374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 msgr2=0x5645b5b939e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:29.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/3201948374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b939e0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fc58000afd0 tx=0x7fc58000a4a0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3201948374 shutdown_connections 2024-06-09T22:30:29.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/3201948374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b93f20 0x5645b5b939e0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/3201948374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 0x5645b5b88510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/3201948374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b99640 0x5645b5b9fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3201948374 >> 172.21.15.104:0/3201948374 conn(0x5645b5abecc0 msgr2=0x5645b5b87e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:29.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3201948374 shutdown_connections 2024-06-09T22:30:29.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:28.999+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/3201948374 wait complete. 2024-06-09T22:30:29.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc5a27cbb80 1 -- start start 2024-06-09T22:30:29.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc5a27cbb80 1 Processor -- start 2024-06-09T22:30:29.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc5a27cbb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b93f20 0x5645b5ba5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:29.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc5a27cbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b99640 0x5645b5ba64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:29.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc5a27cbb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 0x5645b5bb4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:29.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc5a27cbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901af8c0 con 0x5645b5b99640 2024-06-09T22:30:29.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc5a27cbb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901af1f0 con 0x5645b5b99f60 2024-06-09T22:30:29.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc5a27cbb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901ac410 con 0x5645b5b93f20 2024-06-09T22:30:29.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc5a27cbb80 10 client.0 _ll_register_callbacks cb 0x5645b5a7f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:29.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc5954e3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b99640 0x5645b5ba64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:29.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc5954e3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b99640 0x5645b5ba64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36138/0 (socket says 172.21.15.104:36138) 2024-06-09T22:30:29.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc5954e3700 1 -- 172.21.15.104:0/1023220576 learned_addr learned my addr 172.21.15.104:0/1023220576 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:29.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.000+0000 7fc595ce4700 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b93f20 0x5645b5ba5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:29.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5964e5700 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 0x5645b5bb4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:29.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5954e3700 1 -- 172.21.15.104:0/1023220576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b93f20 msgr2=0x5645b5ba5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:29.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5954e3700 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b93f20 0x5645b5ba5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5954e3700 1 -- 172.21.15.104:0/1023220576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 msgr2=0x5645b5bb4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:29.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5954e3700 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 0x5645b5bb4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5954e3700 1 -- 172.21.15.104:0/1023220576 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645b5bb5240 con 0x5645b5b99640 2024-06-09T22:30:29.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc595ce4700 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b93f20 0x5645b5ba5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:30:29.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5954e3700 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b99640 0x5645b5ba64d0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fc58c007540 tx=0x7fc58c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15d42411ddfc09f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:29.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5797fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:29.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5797fa700 1 -- 172.21.15.104:0/1023220576 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c013990 con 0x5645b5b99640 2024-06-09T22:30:29.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5797fa700 1 -- 172.21.15.104:0/1023220576 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fc58c013b30 con 0x5645b5b99640 2024-06-09T22:30:29.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1023220576 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc58009fd90 con 0x5645b5b99640 2024-06-09T22:30:29.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5797fa700 1 -- 172.21.15.104:0/1023220576 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc58c013d70 con 0x5645b5b99640 2024-06-09T22:30:29.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5a27cbb80 10 client.17928 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:29.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1023220576 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5645b5c17fb0 con 0x5645b5b99640 2024-06-09T22:30:29.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5797fa700 1 -- 172.21.15.104:0/1023220576 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc58009fd90 con 0x5645b5b99640 2024-06-09T22:30:29.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.001+0000 7fc5797fa700 1 -- 172.21.15.104:0/1023220576 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc58c021530 con 0x5645b5b99640 2024-06-09T22:30:29.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc5a27cbb80 10 client.17928 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:29.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc5a27cbb80 10 client.17928 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:30:29.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc5a27cbb80 20 client.17928 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:29.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x5645b5c18670 0x5645b5c1aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:29.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc5a27cbb80 4 client.17928 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:30:29.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1023220576 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5645b5aed290 con 0x5645b5c18670 2024-06-09T22:30:29.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc595ce4700 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x5645b5c18670 0x5645b5c1aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:29.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc595ce4700 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x5645b5c18670 0x5645b5c1aa90 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f745faa9c98342eb server_cookie=cf20e82eb6564de4 in_seq=0 out_seq=0 2024-06-09T22:30:29.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc5797fa700 10 client.17928 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:30:29.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc5797fa700 1 -- 172.21.15.104:0/1023220576 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5645b5aed290 con 0x5645b5c18670 2024-06-09T22:30:29.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc5797fa700 10 client.17928 handle_command_reply: tid=0 2024-06-09T22:30:29.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc5a27cbb80 1 client.17928 shutdown 2024-06-09T22:30:29.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1023220576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b99640 msgr2=0x5645b5ba64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:29.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.002+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b99640 0x5645b5ba64d0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fc58c007540 tx=0x7fc58c004750 comp rx=0 tx=0).stop 2024-06-09T22:30:29.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1023220576 shutdown_connections 2024-06-09T22:30:29.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x5645b5c18670 0x5645b5c1aa90 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645b5b99f60 0x5645b5bb4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5645b5b93f20 0x5645b5ba5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc5a27cbb80 1 --2- 172.21.15.104:0/1023220576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5645b5b99640 0x5645b5ba64d0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc5797fa700 0 client.17928 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:30:29.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1023220576 >> 172.21.15.104:0/1023220576 conn(0x5645b5abecc0 msgr2=0x5645b5b9a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:29.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1023220576 shutdown_connections 2024-06-09T22:30:29.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc5a27cbb80 1 -- 172.21.15.104:0/1023220576 wait complete. 2024-06-09T22:30:29.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc5a27cbb80 20 client.17928 trim_cache size 0 max 16384 2024-06-09T22:30:29.017 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:30:29.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc597f48700 1 -- 172.21.15.104:0/405450896 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc57c0a2d90 msgr2=0x7fc57c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:29.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc597f48700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc57c0a2d90 0x7fc57c0a5250 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc590078940 tx=0x7fc58c008040 comp rx=0 tx=0).stop 2024-06-09T22:30:29.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc597f48700 1 -- 172.21.15.104:0/405450896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc59006abc0 msgr2=0x7fc59019e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:29.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc597f48700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc59006abc0 0x7fc59019e8a0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fc58000c9b0 tx=0x7fc58000cd70 comp rx=0 tx=0).stop 2024-06-09T22:30:29.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc597f48700 1 -- 172.21.15.104:0/405450896 shutdown_connections 2024-06-09T22:30:29.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc597f48700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc57c0a2d90 0x7fc57c0a5250 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc597f48700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc59006abc0 0x7fc59019e8a0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc597f48700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc590108500 0x7fc590198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc597f48700 1 --2- 172.21.15.104:0/405450896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59006b580 0x7fc59019ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:29.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc597f48700 1 -- 172.21.15.104:0/405450896 >> 172.21.15.104:0/405450896 conn(0x7fc590078340 msgr2=0x7fc5900fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:29.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc597f48700 1 -- 172.21.15.104:0/405450896 shutdown_connections 2024-06-09T22:30:29.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:29.003+0000 7fc597f48700 1 -- 172.21.15.104:0/405450896 wait complete. 2024-06-09T22:30:29.692 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:29 smithi104 ceph-mon[32389]: pgmap v294: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2024-06-09T22:30:29.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-09T22:30:29.959 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:30:29.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:29 smithi153 ceph-mon[36599]: pgmap v294: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2024-06-09T22:30:30.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:29 smithi194 ceph-mon[36856]: pgmap v294: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2024-06-09T22:30:31.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.840+0000 7f88f433c700 1 -- 172.21.15.104:0/4107875480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88ec107840 msgr2=0x7f88ec10f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:31.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.840+0000 7f88f433c700 1 --2- 172.21.15.104:0/4107875480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88ec107840 0x7f88ec10f3c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f88e400b0d0 tx=0x7f88e400b3e0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 -- 172.21.15.104:0/4107875480 shutdown_connections 2024-06-09T22:30:31.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 --2- 172.21.15.104:0/4107875480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88ec107840 0x7f88ec10f3c0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 --2- 172.21.15.104:0/4107875480 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88ec102740 0x7f88ec102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 --2- 172.21.15.104:0/4107875480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88ec1030f0 0x7f88ec107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 -- 172.21.15.104:0/4107875480 >> 172.21.15.104:0/4107875480 conn(0x7f88ec0795f0 msgr2=0x7f88ec079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:31.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 -- 172.21.15.104:0/4107875480 shutdown_connections 2024-06-09T22:30:31.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 -- 172.21.15.104:0/4107875480 wait complete. 2024-06-09T22:30:31.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 Processor -- start 2024-06-09T22:30:31.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 -- start start 2024-06-09T22:30:31.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88ec102740 0x7f88ec19e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88ec1030f0 0x7f88ec19ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88ec107840 0x7f88ec198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec112780 con 0x7f88ec1030f0 2024-06-09T22:30:31.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec112600 con 0x7f88ec102740 2024-06-09T22:30:31.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.841+0000 7f88f433c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec112900 con 0x7f88ec107840 2024-06-09T22:30:31.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f18d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88ec1030f0 0x7f88ec19ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f18d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88ec1030f0 0x7f88ec19ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36150/0 (socket says 172.21.15.104:36150) 2024-06-09T22:30:31.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f18d7700 1 -- 172.21.15.104:0/576378047 learned_addr learned my addr 172.21.15.104:0/576378047 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:31.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f20d8700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88ec102740 0x7f88ec19e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f28d9700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88ec107840 0x7f88ec198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f18d7700 1 -- 172.21.15.104:0/576378047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88ec107840 msgr2=0x7f88ec198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:31.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f18d7700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88ec107840 0x7f88ec198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f18d7700 1 -- 172.21.15.104:0/576378047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88ec102740 msgr2=0x7f88ec19e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:31.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f18d7700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88ec102740 0x7f88ec19e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f18d7700 1 -- 172.21.15.104:0/576378047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ec1990c0 con 0x7f88ec1030f0 2024-06-09T22:30:31.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f18d7700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88ec1030f0 0x7f88ec19ece0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00d950 tx=0x7f88dc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:31.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88e37fe700 1 -- 172.21.15.104:0/576378047 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc016070 con 0x7f88ec1030f0 2024-06-09T22:30:31.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88e37fe700 1 -- 172.21.15.104:0/576378047 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88dc00fd70 con 0x7f88ec1030f0 2024-06-09T22:30:31.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f433c700 1 -- 172.21.15.104:0/576378047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88ec1993b0 con 0x7f88ec1030f0 2024-06-09T22:30:31.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88e37fe700 1 -- 172.21.15.104:0/576378047 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc00e610 con 0x7f88ec1030f0 2024-06-09T22:30:31.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.842+0000 7f88f433c700 1 -- 172.21.15.104:0/576378047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88ec1a5670 con 0x7f88ec1030f0 2024-06-09T22:30:31.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.845+0000 7f88e37fe700 1 -- 172.21.15.104:0/576378047 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f88dc019070 con 0x7f88ec1030f0 2024-06-09T22:30:31.850 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:31 smithi104 ceph-mon[32389]: pgmap v295: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.7 KiB/s wr, 13 op/s 2024-06-09T22:30:31.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.845+0000 7f88e37fe700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f88d80a2d90 0x7f88d80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.845+0000 7f88f20d8700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f88d80a2d90 0x7f88d80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.845+0000 7f88e37fe700 1 -- 172.21.15.104:0/576378047 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f88dc0cf2b0 con 0x7f88ec1030f0 2024-06-09T22:30:31.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.846+0000 7f88f20d8700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f88d80a2d90 0x7f88d80a5250 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f88ec199aa0 tx=0x7f88e800b3c0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:31.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.846+0000 7f88febbfb80 1 Processor -- start 2024-06-09T22:30:31.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.846+0000 7f88febbfb80 1 -- start start 2024-06-09T22:30:31.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.846+0000 7f88febbfb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856640 0x56251b856a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.846+0000 7f88febbfb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 0x56251b8509e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.846+0000 7f88febbfb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b850f20 0x56251b851340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.846+0000 7f88febbfb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56251b7a2810 con 0x56251b856f60 2024-06-09T22:30:31.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.846+0000 7f88febbfb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56251b74a720 con 0x56251b856640 2024-06-09T22:30:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.846+0000 7f88febbfb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56251b79e7d0 con 0x56251b850f20 2024-06-09T22:30:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88f18d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 0x56251b8509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88f18d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 0x56251b8509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36164/0 (socket says 172.21.15.104:36164) 2024-06-09T22:30:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88f18d7700 1 -- 172.21.15.104:0/3971459466 learned_addr learned my addr 172.21.15.104:0/3971459466 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88f20d8700 1 --2- 172.21.15.104:0/3971459466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856640 0x56251b856a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88f28d9700 1 --2- 172.21.15.104:0/3971459466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b850f20 0x56251b851340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88f18d7700 1 -- 172.21.15.104:0/3971459466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b850f20 msgr2=0x56251b851340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:31.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88f18d7700 1 --2- 172.21.15.104:0/3971459466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b850f20 0x56251b851340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88f18d7700 1 -- 172.21.15.104:0/3971459466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856640 msgr2=0x56251b856a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:31.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88f18d7700 1 --2- 172.21.15.104:0/3971459466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856640 0x56251b856a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88f18d7700 1 -- 172.21.15.104:0/3971459466 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56251b740f50 con 0x56251b856f60 2024-06-09T22:30:31.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88f20d8700 1 --2- 172.21.15.104:0/3971459466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856640 0x56251b856a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:30:31.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88f18d7700 1 --2- 172.21.15.104:0/3971459466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 0x56251b8509e0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0ccc30 tx=0x7f88dc005910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=132be02ffe67b331 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:31.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88e0d70700 1 -- 172.21.15.104:0/3971459466 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc020070 con 0x56251b856f60 2024-06-09T22:30:31.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88e0d70700 1 -- 172.21.15.104:0/3971459466 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88dc09f7b0 con 0x56251b856f60 2024-06-09T22:30:31.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.847+0000 7f88e0d70700 1 -- 172.21.15.104:0/3971459466 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc09b660 con 0x56251b856f60 2024-06-09T22:30:31.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.848+0000 7f88febbfb80 1 -- 172.21.15.104:0/3971459466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 msgr2=0x56251b8509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:31.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.848+0000 7f88febbfb80 1 --2- 172.21.15.104:0/3971459466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 0x56251b8509e0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0ccc30 tx=0x7f88dc005910 comp rx=0 tx=0).stop 2024-06-09T22:30:31.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.848+0000 7f88febbfb80 1 -- 172.21.15.104:0/3971459466 shutdown_connections 2024-06-09T22:30:31.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.848+0000 7f88febbfb80 1 --2- 172.21.15.104:0/3971459466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b850f20 0x56251b851340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.848+0000 7f88febbfb80 1 --2- 172.21.15.104:0/3971459466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856640 0x56251b856a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.848+0000 7f88febbfb80 1 --2- 172.21.15.104:0/3971459466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 0x56251b8509e0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.848+0000 7f88febbfb80 1 -- 172.21.15.104:0/3971459466 >> 172.21.15.104:0/3971459466 conn(0x56251b77bcc0 msgr2=0x56251b84b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:31.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.848+0000 7f88febbfb80 1 -- 172.21.15.104:0/3971459466 shutdown_connections 2024-06-09T22:30:31.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.848+0000 7f88febbfb80 1 -- 172.21.15.104:0/3971459466 wait complete. 2024-06-09T22:30:31.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88febbfb80 1 -- start start 2024-06-09T22:30:31.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88febbfb80 1 Processor -- start 2024-06-09T22:30:31.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88febbfb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b850f20 0x56251b85cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88febbfb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b856640 0x56251b85a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88febbfb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 0x56251b85a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88febbfb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56251b85d910 con 0x56251b856f60 2024-06-09T22:30:31.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88febbfb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56251b742bf0 con 0x56251b850f20 2024-06-09T22:30:31.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88febbfb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56251b85da90 con 0x56251b856640 2024-06-09T22:30:31.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88febbfb80 10 client.0 _ll_register_callbacks cb 0x56251b73c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:31.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88f28d9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 0x56251b85a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88f28d9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 0x56251b85a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36166/0 (socket says 172.21.15.104:36166) 2024-06-09T22:30:31.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88f28d9700 1 -- 172.21.15.104:0/2415121632 learned_addr learned my addr 172.21.15.104:0/2415121632 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:31.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88f20d8700 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b850f20 0x56251b85cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88f18d7700 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b856640 0x56251b85a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88f28d9700 1 -- 172.21.15.104:0/2415121632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b856640 msgr2=0x56251b85a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:31.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88f28d9700 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b856640 0x56251b85a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88f28d9700 1 -- 172.21.15.104:0/2415121632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b850f20 msgr2=0x56251b85cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:31.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88f28d9700 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b850f20 0x56251b85cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.849+0000 7f88f28d9700 1 -- 172.21.15.104:0/2415121632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56251b6f18d0 con 0x56251b856f60 2024-06-09T22:30:31.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.850+0000 7f88f28d9700 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 0x56251b85a5a0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f88e4004c40 tx=0x7f88e4004d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82263e353121aec server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:31.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.850+0000 7f88d5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:31.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.850+0000 7f88d5ffb700 1 -- 172.21.15.104:0/2415121632 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40074e0 con 0x56251b856f60 2024-06-09T22:30:31.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.850+0000 7f88d5ffb700 1 -- 172.21.15.104:0/2415121632 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f88e4007680 con 0x56251b856f60 2024-06-09T22:30:31.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.850+0000 7f88febbfb80 1 -- 172.21.15.104:0/2415121632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88dc09f7b0 con 0x56251b856f60 2024-06-09T22:30:31.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.850+0000 7f88d5ffb700 1 -- 172.21.15.104:0/2415121632 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f88e4007880 con 0x56251b856f60 2024-06-09T22:30:31.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.850+0000 7f88febbfb80 10 client.17940 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:31.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.850+0000 7f88febbfb80 1 -- 172.21.15.104:0/2415121632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56251b8682f0 con 0x56251b856f60 2024-06-09T22:30:31.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.850+0000 7f88d5ffb700 1 -- 172.21.15.104:0/2415121632 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88dc09f7b0 con 0x56251b856f60 2024-06-09T22:30:31.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.850+0000 7f88d5ffb700 1 -- 172.21.15.104:0/2415121632 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f88e4011070 con 0x56251b856f60 2024-06-09T22:30:31.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.851+0000 7f88febbfb80 10 client.17940 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:31.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.851+0000 7f88febbfb80 10 client.17940 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:30:31.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.851+0000 7f88febbfb80 20 client.17940 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:31.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.851+0000 7f88febbfb80 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56251b86a930 0x56251b86cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.851+0000 7f88febbfb80 4 client.17940 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:30:31.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.851+0000 7f88febbfb80 1 -- 172.21.15.104:0/2415121632 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56251b7afd10 con 0x56251b86a930 2024-06-09T22:30:31.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.851+0000 7f88f20d8700 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56251b86a930 0x56251b86cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.851+0000 7f88d5ffb700 10 client.17940 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:31.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.851+0000 7f88f20d8700 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56251b86a930 0x56251b86cd80 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=469fc10ca9f88227 server_cookie=2783cb7c5d39f1ac in_seq=0 out_seq=0 2024-06-09T22:30:31.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88d5ffb700 1 -- 172.21.15.104:0/2415121632 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56251b7afd10 con 0x56251b86a930 2024-06-09T22:30:31.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88d5ffb700 10 client.17940 handle_command_reply: tid=0 2024-06-09T22:30:31.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88febbfb80 1 client.17940 shutdown 2024-06-09T22:30:31.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88febbfb80 1 -- 172.21.15.104:0/2415121632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 msgr2=0x56251b85a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:31.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88febbfb80 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 0x56251b85a5a0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f88e4004c40 tx=0x7f88e4004d20 comp rx=0 tx=0).stop 2024-06-09T22:30:31.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88febbfb80 1 -- 172.21.15.104:0/2415121632 shutdown_connections 2024-06-09T22:30:31.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88febbfb80 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56251b86a930 0x56251b86cd80 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88febbfb80 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856f60 0x56251b85a5a0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88febbfb80 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b850f20 0x56251b85cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88d5ffb700 0 client.17940 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:31.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88febbfb80 1 --2- 172.21.15.104:0/2415121632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b856640 0x56251b85a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88febbfb80 1 -- 172.21.15.104:0/2415121632 >> 172.21.15.104:0/2415121632 conn(0x56251b77bcc0 msgr2=0x56251b84b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:31.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88febbfb80 1 -- 172.21.15.104:0/2415121632 shutdown_connections 2024-06-09T22:30:31.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88febbfb80 1 -- 172.21.15.104:0/2415121632 wait complete. 2024-06-09T22:30:31.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.852+0000 7f88febbfb80 20 client.17940 trim_cache size 0 max 16384 2024-06-09T22:30:31.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.867+0000 7f88febbfb80 1 Processor -- start 2024-06-09T22:30:31.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.867+0000 7f88febbfb80 1 -- start start 2024-06-09T22:30:31.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.867+0000 7f88febbfb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b850f20 0x56251b866be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.867+0000 7f88febbfb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b856640 0x56251b8509e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.867+0000 7f88febbfb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856f60 0x56251b857c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.867+0000 7f88febbfb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1ae580 con 0x56251b850f20 2024-06-09T22:30:31.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.867+0000 7f88febbfb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1b00c0 con 0x56251b856f60 2024-06-09T22:30:31.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.867+0000 7f88febbfb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1aec50 con 0x56251b856640 2024-06-09T22:30:31.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88f20d8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b850f20 0x56251b866be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88f20d8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b850f20 0x56251b866be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36174/0 (socket says 172.21.15.104:36174) 2024-06-09T22:30:31.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88f20d8700 1 -- 172.21.15.104:0/3336061390 learned_addr learned my addr 172.21.15.104:0/3336061390 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:31.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88f28d9700 1 --2- 172.21.15.104:0/3336061390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856f60 0x56251b857c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88f18d7700 1 --2- 172.21.15.104:0/3336061390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b856640 0x56251b8509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88f20d8700 1 -- 172.21.15.104:0/3336061390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b856640 msgr2=0x56251b8509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:31.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88f20d8700 1 --2- 172.21.15.104:0/3336061390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b856640 0x56251b8509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88f20d8700 1 -- 172.21.15.104:0/3336061390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856f60 msgr2=0x56251b857c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:31.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88f20d8700 1 --2- 172.21.15.104:0/3336061390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856f60 0x56251b857c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88f20d8700 1 -- 172.21.15.104:0/3336061390 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56251b8585d0 con 0x56251b850f20 2024-06-09T22:30:31.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88f18d7700 1 --2- 172.21.15.104:0/3336061390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b856640 0x56251b8509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:30:31.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88f20d8700 1 --2- 172.21.15.104:0/3336061390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b850f20 0x56251b866be0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f88e80046e0 tx=0x7f88e8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2302e6c19aec5fa5 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:31.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88d7fff700 1 -- 172.21.15.104:0/3336061390 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8003e30 con 0x56251b850f20 2024-06-09T22:30:31.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88d7fff700 1 -- 172.21.15.104:0/3336061390 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e8004d10 con 0x56251b850f20 2024-06-09T22:30:31.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.868+0000 7f88d7fff700 1 -- 172.21.15.104:0/3336061390 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8005540 con 0x56251b850f20 2024-06-09T22:30:31.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 -- 172.21.15.104:0/3336061390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b850f20 msgr2=0x56251b866be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:31.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 --2- 172.21.15.104:0/3336061390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b850f20 0x56251b866be0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f88e80046e0 tx=0x7f88e8013000 comp rx=0 tx=0).stop 2024-06-09T22:30:31.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 -- 172.21.15.104:0/3336061390 shutdown_connections 2024-06-09T22:30:31.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 --2- 172.21.15.104:0/3336061390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b850f20 0x56251b866be0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 --2- 172.21.15.104:0/3336061390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856f60 0x56251b857c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 --2- 172.21.15.104:0/3336061390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b856640 0x56251b8509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 -- 172.21.15.104:0/3336061390 >> 172.21.15.104:0/3336061390 conn(0x56251b77bcc0 msgr2=0x56251b844e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:31.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 -- 172.21.15.104:0/3336061390 shutdown_connections 2024-06-09T22:30:31.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 -- 172.21.15.104:0/3336061390 wait complete. 2024-06-09T22:30:31.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 -- start start 2024-06-09T22:30:31.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 Processor -- start 2024-06-09T22:30:31.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b850f20 0x56251b862c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856640 0x56251b863150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856f60 0x56251b871640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1af9f0 con 0x56251b856640 2024-06-09T22:30:31.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1af320 con 0x56251b856f60 2024-06-09T22:30:31.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.869+0000 7f88febbfb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1ac540 con 0x56251b850f20 2024-06-09T22:30:31.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88febbfb80 10 client.0 _ll_register_callbacks cb 0x56251b73c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:31.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88f18d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856640 0x56251b863150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88f18d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856640 0x56251b863150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36180/0 (socket says 172.21.15.104:36180) 2024-06-09T22:30:31.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88f18d7700 1 -- 172.21.15.104:0/2451786065 learned_addr learned my addr 172.21.15.104:0/2451786065 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:31.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88f20d8700 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b850f20 0x56251b862c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88f28d9700 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856f60 0x56251b871640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88f18d7700 1 -- 172.21.15.104:0/2451786065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b850f20 msgr2=0x56251b862c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:31.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88f18d7700 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b850f20 0x56251b862c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88f18d7700 1 -- 172.21.15.104:0/2451786065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856f60 msgr2=0x56251b871640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:31.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88f18d7700 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856f60 0x56251b871640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88f18d7700 1 -- 172.21.15.104:0/2451786065 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56251b863990 con 0x56251b856640 2024-06-09T22:30:31.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88f18d7700 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856640 0x56251b863150 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f88dc09b100 tx=0x7f88dc099910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8612ba665a9c1c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:31.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88d5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:31.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88d5ffb700 1 -- 172.21.15.104:0/2451786065 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc020040 con 0x56251b856640 2024-06-09T22:30:31.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88d5ffb700 1 -- 172.21.15.104:0/2451786065 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f88dc0b4260 con 0x56251b856640 2024-06-09T22:30:31.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88febbfb80 1 -- 172.21.15.104:0/2451786065 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88e8004d10 con 0x56251b856640 2024-06-09T22:30:31.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88d5ffb700 1 -- 172.21.15.104:0/2451786065 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f88dc0976a0 con 0x56251b856640 2024-06-09T22:30:31.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88febbfb80 10 client.17946 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:31.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.870+0000 7f88febbfb80 1 -- 172.21.15.104:0/2451786065 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56251b8d9b50 con 0x56251b856640 2024-06-09T22:30:31.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.871+0000 7f88d5ffb700 1 -- 172.21.15.104:0/2451786065 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e8004d10 con 0x56251b856640 2024-06-09T22:30:31.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.871+0000 7f88d5ffb700 1 -- 172.21.15.104:0/2451786065 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f88dc0a3be0 con 0x56251b856640 2024-06-09T22:30:31.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.871+0000 7f88febbfb80 10 client.17946 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:31.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.871+0000 7f88febbfb80 10 client.17946 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:30:31.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.871+0000 7f88febbfb80 20 client.17946 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:31.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.871+0000 7f88febbfb80 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56251b8d9e30 0x56251b8dc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:31.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.871+0000 7f88febbfb80 4 client.17946 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:30:31.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.871+0000 7f88febbfb80 1 -- 172.21.15.104:0/2451786065 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56251b7aa290 con 0x56251b8d9e30 2024-06-09T22:30:31.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.871+0000 7f88f20d8700 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56251b8d9e30 0x56251b8dc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:31.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.871+0000 7f88d5ffb700 10 client.17946 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:31.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.871+0000 7f88f20d8700 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56251b8d9e30 0x56251b8dc220 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ff6b68cdcbd096 server_cookie=c35e0da89a308518 in_seq=0 out_seq=0 2024-06-09T22:30:31.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88d5ffb700 1 -- 172.21.15.104:0/2451786065 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56251b7aa290 con 0x56251b8d9e30 2024-06-09T22:30:31.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88d5ffb700 10 client.17946 handle_command_reply: tid=0 2024-06-09T22:30:31.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88febbfb80 1 client.17946 shutdown 2024-06-09T22:30:31.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88febbfb80 1 -- 172.21.15.104:0/2451786065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856640 msgr2=0x56251b863150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:31.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88febbfb80 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856640 0x56251b863150 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f88dc09b100 tx=0x7f88dc099910 comp rx=0 tx=0).stop 2024-06-09T22:30:31.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88febbfb80 1 -- 172.21.15.104:0/2451786065 shutdown_connections 2024-06-09T22:30:31.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88febbfb80 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x56251b8d9e30 0x56251b8dc220 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88febbfb80 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56251b856f60 0x56251b871640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88febbfb80 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56251b850f20 0x56251b862c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88febbfb80 1 --2- 172.21.15.104:0/2451786065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56251b856640 0x56251b863150 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88d5ffb700 0 client.17946 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:31.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88febbfb80 1 -- 172.21.15.104:0/2451786065 >> 172.21.15.104:0/2451786065 conn(0x56251b77bcc0 msgr2=0x56251b845510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:31.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88febbfb80 1 -- 172.21.15.104:0/2451786065 shutdown_connections 2024-06-09T22:30:31.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88febbfb80 1 -- 172.21.15.104:0/2451786065 wait complete. 2024-06-09T22:30:31.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.872+0000 7f88febbfb80 20 client.17946 trim_cache size 0 max 16384 2024-06-09T22:30:31.887 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:30:31.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.873+0000 7f88f433c700 1 -- 172.21.15.104:0/576378047 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f88d80a2d90 msgr2=0x7f88d80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:31.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.873+0000 7f88f433c700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f88d80a2d90 0x7f88d80a5250 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f88ec199aa0 tx=0x7f88e800b3c0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.873+0000 7f88f433c700 1 -- 172.21.15.104:0/576378047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88ec1030f0 msgr2=0x7f88ec19ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:31.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.873+0000 7f88f433c700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88ec1030f0 0x7f88ec19ece0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00d950 tx=0x7f88dc00dc60 comp rx=0 tx=0).stop 2024-06-09T22:30:31.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.873+0000 7f88f433c700 1 -- 172.21.15.104:0/576378047 shutdown_connections 2024-06-09T22:30:31.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.873+0000 7f88f433c700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f88d80a2d90 0x7f88d80a5250 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.873+0000 7f88f433c700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88ec107840 0x7f88ec198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.873+0000 7f88f433c700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88ec102740 0x7f88ec19e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.873+0000 7f88f433c700 1 --2- 172.21.15.104:0/576378047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88ec1030f0 0x7f88ec19ece0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:31.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.873+0000 7f88f433c700 1 -- 172.21.15.104:0/576378047 >> 172.21.15.104:0/576378047 conn(0x7f88ec0795f0 msgr2=0x7f88ec1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:31.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.873+0000 7f88f433c700 1 -- 172.21.15.104:0/576378047 shutdown_connections 2024-06-09T22:30:31.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:31.873+0000 7f88f433c700 1 -- 172.21.15.104:0/576378047 wait complete. 2024-06-09T22:30:31.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:31 smithi153 ceph-mon[36599]: pgmap v295: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.7 KiB/s wr, 13 op/s 2024-06-09T22:30:32.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:31 smithi194 ceph-mon[36856]: pgmap v295: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.7 KiB/s wr, 13 op/s 2024-06-09T22:30:33.793 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-09T22:30:33.793 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-09T22:30:33.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:33 smithi153 ceph-mon[36599]: pgmap v296: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2024-06-09T22:30:34.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:33 smithi104 ceph-mon[32389]: pgmap v296: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2024-06-09T22:30:34.048 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:30:34.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:33 smithi194 ceph-mon[36856]: pgmap v296: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2024-06-09T22:30:35.825 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:35 smithi104 ceph-mon[32389]: pgmap v297: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.0 KiB/s wr, 13 op/s 2024-06-09T22:30:35.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.930+0000 7feaa4f35700 1 -- 172.21.15.104:0/698209013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa01066b0 msgr2=0x7feaa010e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:35.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.930+0000 7feaa4f35700 1 --2- 172.21.15.104:0/698209013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa01066b0 0x7feaa010e280 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fea980099c0 tx=0x7fea98009cd0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.930+0000 7feaa4f35700 1 -- 172.21.15.104:0/698209013 shutdown_connections 2024-06-09T22:30:35.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.930+0000 7feaa4f35700 1 --2- 172.21.15.104:0/698209013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa01066b0 0x7feaa010e280 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.930+0000 7feaa4f35700 1 --2- 172.21.15.104:0/698209013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feaa0101620 0x7feaa0101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.930+0000 7feaa4f35700 1 --2- 172.21.15.104:0/698209013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feaa0101fd0 0x7feaa0105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.930+0000 7feaa4f35700 1 -- 172.21.15.104:0/698209013 >> 172.21.15.104:0/698209013 conn(0x7feaa0078340 msgr2=0x7feaa0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:35.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.930+0000 7feaa4f35700 1 -- 172.21.15.104:0/698209013 shutdown_connections 2024-06-09T22:30:35.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.930+0000 7feaa4f35700 1 -- 172.21.15.104:0/698209013 wait complete. 2024-06-09T22:30:35.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7feaa4f35700 1 Processor -- start 2024-06-09T22:30:35.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7feaa4f35700 1 -- start start 2024-06-09T22:30:35.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7feaa4f35700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feaa0101620 0x7feaa019d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7feaa4f35700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa0101fd0 0x7feaa019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7feaa4f35700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feaa01066b0 0x7feaa0197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7feaa4f35700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feaa0111790 con 0x7feaa0101fd0 2024-06-09T22:30:35.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7feaa4f35700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feaa0111610 con 0x7feaa0101620 2024-06-09T22:30:35.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7feaa4f35700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feaa0111910 con 0x7feaa01066b0 2024-06-09T22:30:35.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7fea9dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa0101fd0 0x7feaa019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7fea9dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa0101fd0 0x7feaa019ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36194/0 (socket says 172.21.15.104:36194) 2024-06-09T22:30:35.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7fea9dd9b700 1 -- 172.21.15.104:0/1489729190 learned_addr learned my addr 172.21.15.104:0/1489729190 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:35.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7fea9e59c700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feaa0101620 0x7feaa019d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.931+0000 7fea9ed9d700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feaa01066b0 0x7feaa0197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.932+0000 7fea9dd9b700 1 -- 172.21.15.104:0/1489729190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feaa01066b0 msgr2=0x7feaa0197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:35.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.932+0000 7fea9dd9b700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feaa01066b0 0x7feaa0197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.932+0000 7fea9dd9b700 1 -- 172.21.15.104:0/1489729190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feaa0101620 msgr2=0x7feaa019d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:35.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.932+0000 7fea9dd9b700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feaa0101620 0x7feaa019d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.932+0000 7fea9dd9b700 1 -- 172.21.15.104:0/1489729190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa01981d0 con 0x7feaa0101fd0 2024-06-09T22:30:35.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.932+0000 7fea9dd9b700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa0101fd0 0x7feaa019ddf0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fea9400b500 tx=0x7fea9400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:35.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.932+0000 7fea8f7fe700 1 -- 172.21.15.104:0/1489729190 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94015020 con 0x7feaa0101fd0 2024-06-09T22:30:35.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.932+0000 7feaa4f35700 1 -- 172.21.15.104:0/1489729190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaa01984c0 con 0x7feaa0101fd0 2024-06-09T22:30:35.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.932+0000 7fea8f7fe700 1 -- 172.21.15.104:0/1489729190 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea94011d10 con 0x7feaa0101fd0 2024-06-09T22:30:35.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.932+0000 7fea8f7fe700 1 -- 172.21.15.104:0/1489729190 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9400f2e0 con 0x7feaa0101fd0 2024-06-09T22:30:35.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.932+0000 7feaa4f35700 1 -- 172.21.15.104:0/1489729190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaa006abc0 con 0x7feaa0101fd0 2024-06-09T22:30:35.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.934+0000 7fea8f7fe700 1 -- 172.21.15.104:0/1489729190 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fea94018020 con 0x7feaa0101fd0 2024-06-09T22:30:35.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.935+0000 7fea8f7fe700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fea880a2d90 0x7fea880a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.935+0000 7fea9e59c700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fea880a2d90 0x7fea880a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.935+0000 7fea8f7fe700 1 -- 172.21.15.104:0/1489729190 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fea940ceec0 con 0x7feaa0101fd0 2024-06-09T22:30:35.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.935+0000 7fea9e59c700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fea880a2d90 0x7fea880a5250 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feaa0198bb0 tx=0x7fea9000d040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:35.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7feaaf7b8b80 1 Processor -- start 2024-06-09T22:30:35.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7feaaf7b8b80 1 -- start start 2024-06-09T22:30:35.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7feaaf7b8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d51686c0 0x5617d5168aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7feaaf7b8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5168fe0 0x5617d5162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7feaaf7b8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d5162fa0 0x5617d51633c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7feaaf7b8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5617d50b4890 con 0x5617d51686c0 2024-06-09T22:30:35.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7feaaf7b8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5617d505c7a0 con 0x5617d5162fa0 2024-06-09T22:30:35.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7feaaf7b8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5617d50b0850 con 0x5617d5168fe0 2024-06-09T22:30:35.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7fea9e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d51686c0 0x5617d5168aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7fea9e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d51686c0 0x5617d5168aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36196/0 (socket says 172.21.15.104:36196) 2024-06-09T22:30:35.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7fea9e59c700 1 -- 172.21.15.104:0/2205717295 learned_addr learned my addr 172.21.15.104:0/2205717295 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:35.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7fea9dd9b700 1 --2- 172.21.15.104:0/2205717295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5168fe0 0x5617d5162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7fea9ed9d700 1 --2- 172.21.15.104:0/2205717295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d5162fa0 0x5617d51633c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7fea9e59c700 1 -- 172.21.15.104:0/2205717295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5168fe0 msgr2=0x5617d5162a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:35.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7fea9e59c700 1 --2- 172.21.15.104:0/2205717295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5168fe0 0x5617d5162a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7fea9e59c700 1 -- 172.21.15.104:0/2205717295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d5162fa0 msgr2=0x5617d51633c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:35.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7fea9e59c700 1 --2- 172.21.15.104:0/2205717295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d5162fa0 0x5617d51633c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7fea9e59c700 1 -- 172.21.15.104:0/2205717295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617d5052fd0 con 0x5617d51686c0 2024-06-09T22:30:35.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.936+0000 7fea9dd9b700 1 --2- 172.21.15.104:0/2205717295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5168fe0 0x5617d5162a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:30:35.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.937+0000 7fea9e59c700 1 --2- 172.21.15.104:0/2205717295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d51686c0 0x5617d5168aa0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fea90016730 tx=0x7fea90011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9c791d57378737 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:35.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.937+0000 7fea8cff9700 1 -- 172.21.15.104:0/2205717295 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea90015780 con 0x5617d51686c0 2024-06-09T22:30:35.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.937+0000 7fea8cff9700 1 -- 172.21.15.104:0/2205717295 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea90025070 con 0x5617d51686c0 2024-06-09T22:30:35.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.937+0000 7fea8cff9700 1 -- 172.21.15.104:0/2205717295 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea90020b30 con 0x5617d51686c0 2024-06-09T22:30:35.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/2205717295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d51686c0 msgr2=0x5617d5168aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:35.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/2205717295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d51686c0 0x5617d5168aa0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fea90016730 tx=0x7fea90011000 comp rx=0 tx=0).stop 2024-06-09T22:30:35.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/2205717295 shutdown_connections 2024-06-09T22:30:35.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/2205717295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d51686c0 0x5617d5168aa0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/2205717295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d5162fa0 0x5617d51633c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/2205717295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5168fe0 0x5617d5162a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/2205717295 >> 172.21.15.104:0/2205717295 conn(0x5617d508dd40 msgr2=0x5617d515d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:35.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/2205717295 shutdown_connections 2024-06-09T22:30:35.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/2205717295 wait complete. 2024-06-09T22:30:35.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 -- start start 2024-06-09T22:30:35.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 Processor -- start 2024-06-09T22:30:35.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5162fa0 0x5617d516ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d516c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5168fe0 0x5617d516c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5617d516fa50 con 0x5617d5162fa0 2024-06-09T22:30:35.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5617d5054c70 con 0x5617d51686c0 2024-06-09T22:30:35.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5617d516fbd0 con 0x5617d5168fe0 2024-06-09T22:30:35.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7feaaf7b8b80 10 client.0 _ll_register_callbacks cb 0x5617d504e9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:35.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7fea9e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5162fa0 0x5617d516ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7fea9e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5162fa0 0x5617d516ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36202/0 (socket says 172.21.15.104:36202) 2024-06-09T22:30:35.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.938+0000 7fea9e59c700 1 -- 172.21.15.104:0/54873194 learned_addr learned my addr 172.21.15.104:0/54873194 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:35.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea9ed9d700 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5168fe0 0x5617d516c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea9dd9b700 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d516c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea9e59c700 1 -- 172.21.15.104:0/54873194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5168fe0 msgr2=0x5617d516c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:35.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea9e59c700 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5168fe0 0x5617d516c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea9e59c700 1 -- 172.21.15.104:0/54873194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 msgr2=0x5617d516c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:35.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea9e59c700 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d516c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea9e59c700 1 -- 172.21.15.104:0/54873194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617d5003950 con 0x5617d5162fa0 2024-06-09T22:30:35.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea9e59c700 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5162fa0 0x5617d516ed50 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fea90004c20 tx=0x7fea90020f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee755944fa9de3ae server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:35.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea85ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:35.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea85ffb700 1 -- 172.21.15.104:0/54873194 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea90015780 con 0x5617d5162fa0 2024-06-09T22:30:35.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea85ffb700 1 -- 172.21.15.104:0/54873194 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fea90015920 con 0x5617d5162fa0 2024-06-09T22:30:35.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/54873194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea90025070 con 0x5617d5162fa0 2024-06-09T22:30:35.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea85ffb700 1 -- 172.21.15.104:0/54873194 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fea90015b20 con 0x5617d5162fa0 2024-06-09T22:30:35.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7feaaf7b8b80 10 client.17958 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:35.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/54873194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5617d517a350 con 0x5617d5162fa0 2024-06-09T22:30:35.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.939+0000 7fea85ffb700 1 -- 172.21.15.104:0/54873194 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea90025070 con 0x5617d5162fa0 2024-06-09T22:30:35.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.940+0000 7fea85ffb700 1 -- 172.21.15.104:0/54873194 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fea900293f0 con 0x5617d5162fa0 2024-06-09T22:30:35.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.940+0000 7feaaf7b8b80 10 client.17958 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:35.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.940+0000 7feaaf7b8b80 10 client.17958 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:30:35.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.940+0000 7feaaf7b8b80 20 client.17958 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:35.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.940+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5617d517c8d0 0x5617d517ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.940+0000 7feaaf7b8b80 4 client.17958 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:30:35.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.940+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/54873194 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617d50c1d90 con 0x5617d517c8d0 2024-06-09T22:30:35.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.940+0000 7fea9dd9b700 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5617d517c8d0 0x5617d517ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7fea9dd9b700 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5617d517c8d0 0x5617d517ecc0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f25124b19145e56 server_cookie=b68f8c983a737609 in_seq=0 out_seq=0 2024-06-09T22:30:35.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7fea85ffb700 10 client.17958 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:35.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7fea85ffb700 1 -- 172.21.15.104:0/54873194 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5617d50c1d90 con 0x5617d517c8d0 2024-06-09T22:30:35.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7fea85ffb700 10 client.17958 handle_command_reply: tid=0 2024-06-09T22:30:35.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7feaaf7b8b80 1 client.17958 shutdown 2024-06-09T22:30:35.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/54873194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5162fa0 msgr2=0x5617d516ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:35.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5162fa0 0x5617d516ed50 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fea90004c20 tx=0x7fea90020f70 comp rx=0 tx=0).stop 2024-06-09T22:30:35.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/54873194 shutdown_connections 2024-06-09T22:30:35.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5617d517c8d0 0x5617d517ecc0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5162fa0 0x5617d516ed50 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5168fe0 0x5617d516c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/54873194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d516c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/54873194 >> 172.21.15.104:0/54873194 conn(0x5617d508dd40 msgr2=0x5617d515d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:35.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7fea85ffb700 0 client.17958 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:35.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.941+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/54873194 shutdown_connections 2024-06-09T22:30:35.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.942+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/54873194 wait complete. 2024-06-09T22:30:35.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.942+0000 7feaaf7b8b80 20 client.17958 trim_cache size 0 max 16384 2024-06-09T22:30:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7feaaf7b8b80 1 Processor -- start 2024-06-09T22:30:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7feaaf7b8b80 1 -- start start 2024-06-09T22:30:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7feaaf7b8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5162fa0 0x5617d5162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7feaaf7b8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d5169c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7feaaf7b8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 0x5617d516afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7feaaf7b8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01b0a20 con 0x5617d5168fe0 2024-06-09T22:30:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7feaaf7b8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01b2560 con 0x5617d51686c0 2024-06-09T22:30:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7feaaf7b8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01b10f0 con 0x5617d5162fa0 2024-06-09T22:30:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7fea9ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 0x5617d516afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7fea9ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 0x5617d516afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36214/0 (socket says 172.21.15.104:36214) 2024-06-09T22:30:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7fea9ed9d700 1 -- 172.21.15.104:0/1929747346 learned_addr learned my addr 172.21.15.104:0/1929747346 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7fea9e59c700 1 --2- 172.21.15.104:0/1929747346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5162fa0 0x5617d5162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7fea9dd9b700 1 --2- 172.21.15.104:0/1929747346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d5169c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7fea9ed9d700 1 -- 172.21.15.104:0/1929747346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5162fa0 msgr2=0x5617d5162a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7fea9ed9d700 1 --2- 172.21.15.104:0/1929747346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5162fa0 0x5617d5162a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7fea9ed9d700 1 -- 172.21.15.104:0/1929747346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 msgr2=0x5617d5169c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7fea9ed9d700 1 --2- 172.21.15.104:0/1929747346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d5169c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7fea9ed9d700 1 -- 172.21.15.104:0/1929747346 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617d516e990 con 0x5617d5168fe0 2024-06-09T22:30:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.962+0000 7fea9dd9b700 1 --2- 172.21.15.104:0/1929747346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d5169c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:30:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7fea9ed9d700 1 --2- 172.21.15.104:0/1929747346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 0x5617d516afb0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fea98015000 tx=0x7fea9801c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f1a8e1ff4a3868 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7fea87fff700 1 -- 172.21.15.104:0/1929747346 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea98028070 con 0x5617d5168fe0 2024-06-09T22:30:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7fea87fff700 1 -- 172.21.15.104:0/1929747346 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea98003780 con 0x5617d5168fe0 2024-06-09T22:30:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7fea87fff700 1 -- 172.21.15.104:0/1929747346 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9800bc90 con 0x5617d5168fe0 2024-06-09T22:30:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/1929747346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 msgr2=0x5617d516afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/1929747346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 0x5617d516afb0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fea98015000 tx=0x7fea9801c040 comp rx=0 tx=0).stop 2024-06-09T22:30:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/1929747346 shutdown_connections 2024-06-09T22:30:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/1929747346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 0x5617d516afb0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/1929747346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5162fa0 0x5617d5162a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/1929747346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d5169c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/1929747346 >> 172.21.15.104:0/1929747346 conn(0x5617d508dd40 msgr2=0x5617d5156e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/1929747346 shutdown_connections 2024-06-09T22:30:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.963+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/1929747346 wait complete. 2024-06-09T22:30:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7feaaf7b8b80 1 -- start start 2024-06-09T22:30:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7feaaf7b8b80 1 Processor -- start 2024-06-09T22:30:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7feaaf7b8b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5162fa0 0x5617d517fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7feaaf7b8b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d517ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7feaaf7b8b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 0x5617d51e6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7feaaf7b8b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01b1e90 con 0x5617d5168fe0 2024-06-09T22:30:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7feaaf7b8b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01b17c0 con 0x5617d51686c0 2024-06-09T22:30:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7feaaf7b8b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01ae9e0 con 0x5617d5162fa0 2024-06-09T22:30:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7feaaf7b8b80 10 client.0 _ll_register_callbacks cb 0x5617d504e9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7fea9ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 0x5617d51e6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7fea9ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 0x5617d51e6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36228/0 (socket says 172.21.15.104:36228) 2024-06-09T22:30:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7fea9ed9d700 1 -- 172.21.15.104:0/505264340 learned_addr learned my addr 172.21.15.104:0/505264340 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7fea9dd9b700 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d517ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7fea9e59c700 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5162fa0 0x5617d517fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7fea9ed9d700 1 -- 172.21.15.104:0/505264340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5162fa0 msgr2=0x5617d517fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.964+0000 7fea9ed9d700 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5162fa0 0x5617d517fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7fea9ed9d700 1 -- 172.21.15.104:0/505264340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 msgr2=0x5617d517ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7fea9ed9d700 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d517ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7fea9ed9d700 1 -- 172.21.15.104:0/505264340 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617d5180860 con 0x5617d5168fe0 2024-06-09T22:30:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7fea9ed9d700 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 0x5617d51e6210 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fea9800a010 tx=0x7fea98003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410f6e4833e60225 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7fea85ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7fea85ffb700 1 -- 172.21.15.104:0/505264340 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea98030070 con 0x5617d5168fe0 2024-06-09T22:30:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7fea85ffb700 1 -- 172.21.15.104:0/505264340 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fea980050b0 con 0x5617d5168fe0 2024-06-09T22:30:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/505264340 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea98003780 con 0x5617d5168fe0 2024-06-09T22:30:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7fea85ffb700 1 -- 172.21.15.104:0/505264340 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fea9803c050 con 0x5617d5168fe0 2024-06-09T22:30:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7feaaf7b8b80 10 client.17964 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/505264340 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5617d51e7860 con 0x5617d5168fe0 2024-06-09T22:30:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7fea85ffb700 1 -- 172.21.15.104:0/505264340 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea98003780 con 0x5617d5168fe0 2024-06-09T22:30:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7fea85ffb700 1 -- 172.21.15.104:0/505264340 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fea9802b050 con 0x5617d5168fe0 2024-06-09T22:30:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7feaaf7b8b80 10 client.17964 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7feaaf7b8b80 10 client.17964 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:30:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7feaaf7b8b80 20 client.17964 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5617d51e7e00 0x5617d51ea220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7feaaf7b8b80 4 client.17964 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-09T22:30:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.965+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/505264340 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5617d50bc310 con 0x5617d51e7e00 2024-06-09T22:30:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.966+0000 7fea9e59c700 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5617d51e7e00 0x5617d51ea220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.966+0000 7fea85ffb700 10 client.17964 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:35.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.966+0000 7fea9e59c700 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5617d51e7e00 0x5617d51ea220 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ab90f8cec75096a server_cookie=6ff877dd032a77fb in_seq=0 out_seq=0 2024-06-09T22:30:35.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.967+0000 7fea85ffb700 1 -- 172.21.15.104:0/505264340 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5617d50bc310 con 0x5617d51e7e00 2024-06-09T22:30:35.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.967+0000 7fea85ffb700 10 client.17964 handle_command_reply: tid=0 2024-06-09T22:30:35.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.967+0000 7feaaf7b8b80 1 client.17964 shutdown 2024-06-09T22:30:35.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.967+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/505264340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 msgr2=0x5617d51e6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:35.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.967+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 0x5617d51e6210 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fea9800a010 tx=0x7fea98003dc0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/505264340 shutdown_connections 2024-06-09T22:30:35.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5617d51e7e00 0x5617d51ea220 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5617d5168fe0 0x5617d51e6210 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7fea85ffb700 0 client.17964 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:35.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617d5162fa0 0x5617d517fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaaf7b8b80 1 --2- 172.21.15.104:0/505264340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617d51686c0 0x5617d517ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/505264340 >> 172.21.15.104:0/505264340 conn(0x5617d508dd40 msgr2=0x5617d5157590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:35.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/505264340 shutdown_connections 2024-06-09T22:30:35.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaaf7b8b80 1 -- 172.21.15.104:0/505264340 wait complete. 2024-06-09T22:30:35.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaaf7b8b80 20 client.17964 trim_cache size 0 max 16384 2024-06-09T22:30:35.981 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:30:35.981 INFO:teuthology.orchestra.run.smithi104.stdout: "return_code": 0, 2024-06-09T22:30:35.982 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_tag": "9d2a52f3-a1e8-4921-9f3d-ef95d2749657", 2024-06-09T22:30:35.982 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": "asynchronous" 2024-06-09T22:30:35.982 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:30:35.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaa4f35700 1 -- 172.21.15.104:0/1489729190 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fea880a2d90 msgr2=0x7fea880a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:35.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaa4f35700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fea880a2d90 0x7fea880a5250 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feaa0198bb0 tx=0x7fea9000d040 comp rx=0 tx=0).stop 2024-06-09T22:30:35.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaa4f35700 1 -- 172.21.15.104:0/1489729190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa0101fd0 msgr2=0x7feaa019ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:35.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaa4f35700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa0101fd0 0x7feaa019ddf0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fea9400b500 tx=0x7fea9400b810 comp rx=0 tx=0).stop 2024-06-09T22:30:35.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaa4f35700 1 -- 172.21.15.104:0/1489729190 shutdown_connections 2024-06-09T22:30:35.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaa4f35700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fea880a2d90 0x7fea880a5250 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaa4f35700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feaa01066b0 0x7feaa0197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaa4f35700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feaa0101620 0x7feaa019d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaa4f35700 1 --2- 172.21.15.104:0/1489729190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa0101fd0 0x7feaa019ddf0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:35.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaa4f35700 1 -- 172.21.15.104:0/1489729190 >> 172.21.15.104:0/1489729190 conn(0x7feaa0078340 msgr2=0x7feaa00ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:35.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaa4f35700 1 -- 172.21.15.104:0/1489729190 shutdown_connections 2024-06-09T22:30:35.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:35.968+0000 7feaa4f35700 1 -- 172.21.15.104:0/1489729190 wait complete. 2024-06-09T22:30:35.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:35 smithi153 ceph-mon[36599]: pgmap v297: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.0 KiB/s wr, 13 op/s 2024-06-09T22:30:36.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:35 smithi194 ceph-mon[36856]: pgmap v297: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.0 KiB/s wr, 13 op/s 2024-06-09T22:30:36.907 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:36 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:30:36.907 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:36 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:30:36.907 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:36 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:36.907 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:36 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:30:36.908 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:36 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:36.908 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:36 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:36.939 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:30:36.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:36 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:30:36.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:36 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:30:36.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:36 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:36.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:36 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:30:36.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:36 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:36.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:36 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:37.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:36 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:30:37.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:36 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:30:37.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:36 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:37.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:36 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:30:37.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:36 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:37.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:36 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:30:37.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:37 smithi153 ceph-mon[36599]: pgmap v298: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2024-06-09T22:30:37.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:37 smithi153 ceph-mon[36599]: scrub queued for path: / 2024-06-09T22:30:37.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:37 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:37.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:37 smithi153 ceph-mon[36599]: scrub summary: active paths [/] 2024-06-09T22:30:37.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:37 smithi153 ceph-mon[36599]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:38.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:37 smithi104 ceph-mon[32389]: pgmap v298: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2024-06-09T22:30:38.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:37 smithi104 ceph-mon[32389]: scrub queued for path: / 2024-06-09T22:30:38.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:37 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:38.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:37 smithi104 ceph-mon[32389]: scrub summary: active paths [/] 2024-06-09T22:30:38.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:37 smithi104 ceph-mon[32389]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:38.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:37 smithi194 ceph-mon[36856]: pgmap v298: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2024-06-09T22:30:38.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:37 smithi194 ceph-mon[36856]: scrub queued for path: / 2024-06-09T22:30:38.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:37 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:38.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:37 smithi194 ceph-mon[36856]: scrub summary: active paths [/] 2024-06-09T22:30:38.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:37 smithi194 ceph-mon[36856]: scrub summary: idle+waiting paths [/] 2024-06-09T22:30:38.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.870+0000 7f9649a30700 1 -- 172.21.15.104:0/3567676470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96441013c0 msgr2=0x7f9644101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:38.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.870+0000 7f9649a30700 1 --2- 172.21.15.104:0/3567676470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96441013c0 0x7f9644101840 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f962c009990 tx=0x7f962c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.870+0000 7f9649a30700 1 -- 172.21.15.104:0/3567676470 shutdown_connections 2024-06-09T22:30:38.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.870+0000 7f9649a30700 1 --2- 172.21.15.104:0/3567676470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9644101d80 0x7f964410e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.870+0000 7f9649a30700 1 --2- 172.21.15.104:0/3567676470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9644107430 0x7f9644107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.870+0000 7f9649a30700 1 --2- 172.21.15.104:0/3567676470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96441013c0 0x7f9644101840 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.870+0000 7f9649a30700 1 -- 172.21.15.104:0/3567676470 >> 172.21.15.104:0/3567676470 conn(0x7f96440783d0 msgr2=0x7f96440787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:38.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.870+0000 7f9649a30700 1 -- 172.21.15.104:0/3567676470 shutdown_connections 2024-06-09T22:30:38.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.870+0000 7f9649a30700 1 -- 172.21.15.104:0/3567676470 wait complete. 2024-06-09T22:30:38.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.871+0000 7f9649a30700 1 Processor -- start 2024-06-09T22:30:38.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.871+0000 7f9649a30700 1 -- start start 2024-06-09T22:30:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.871+0000 7f9649a30700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96441013c0 0x7f964419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.871+0000 7f9649a30700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9644101d80 0x7f964419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.871+0000 7f9649a30700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9644107430 0x7f9644198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.871+0000 7f9649a30700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9644111720 con 0x7f96441013c0 2024-06-09T22:30:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.871+0000 7f9649a30700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96441115a0 con 0x7f9644101d80 2024-06-09T22:30:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.871+0000 7f9649a30700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f96441118a0 con 0x7f9644107430 2024-06-09T22:30:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.871+0000 7f9642ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96441013c0 0x7f964419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.871+0000 7f9642ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96441013c0 0x7f964419e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37142/0 (socket says 172.21.15.104:37142) 2024-06-09T22:30:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f9642ffd700 1 -- 172.21.15.104:0/652830971 learned_addr learned my addr 172.21.15.104:0/652830971 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f963a7fc700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9644101d80 0x7f964419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f96437fe700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9644107430 0x7f9644198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f9642ffd700 1 -- 172.21.15.104:0/652830971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9644107430 msgr2=0x7f9644198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f9642ffd700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9644107430 0x7f9644198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f9642ffd700 1 -- 172.21.15.104:0/652830971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9644101d80 msgr2=0x7f964419ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f9642ffd700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9644101d80 0x7f964419ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f9642ffd700 1 -- 172.21.15.104:0/652830971 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9644199240 con 0x7f96441013c0 2024-06-09T22:30:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f9642ffd700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96441013c0 0x7f964419e920 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f963400b530 tx=0x7f963400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f9648a2e700 1 -- 172.21.15.104:0/652830971 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9634015020 con 0x7f96441013c0 2024-06-09T22:30:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f9648a2e700 1 -- 172.21.15.104:0/652830971 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9634011d10 con 0x7f96441013c0 2024-06-09T22:30:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f9649a30700 1 -- 172.21.15.104:0/652830971 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9644199530 con 0x7f96441013c0 2024-06-09T22:30:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f9648a2e700 1 -- 172.21.15.104:0/652830971 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963400f2e0 con 0x7f96441013c0 2024-06-09T22:30:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.872+0000 7f9649a30700 1 -- 172.21.15.104:0/652830971 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96441a57f0 con 0x7f96441013c0 2024-06-09T22:30:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.874+0000 7f9648a2e700 1 -- 172.21.15.104:0/652830971 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f9634018020 con 0x7f96441013c0 2024-06-09T22:30:38.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.875+0000 7f9648a2e700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f96300a2d90 0x7f96300a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.875+0000 7f963a7fc700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f96300a2d90 0x7f96300a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.875+0000 7f9648a2e700 1 -- 172.21.15.104:0/652830971 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f96340cefd0 con 0x7f96441013c0 2024-06-09T22:30:38.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f963a7fc700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f96300a2d90 0x7f96300a5250 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f962c00b5c0 tx=0x7f962c005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:38.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f96542b3b80 1 Processor -- start 2024-06-09T22:30:38.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f96542b3b80 1 -- start start 2024-06-09T22:30:38.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f96542b3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 0x55ac25f933b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f96542b3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25f976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f96542b3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f97c30 0x55ac25f98090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f96542b3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55ac25ee0810 con 0x55ac25f92fd0 2024-06-09T22:30:38.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f96542b3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ac25e88720 con 0x55ac25f97c30 2024-06-09T22:30:38.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f96542b3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ac25edc7d0 con 0x55ac25f938f0 2024-06-09T22:30:38.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f9642ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 0x55ac25f933b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f9642ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 0x55ac25f933b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37148/0 (socket says 172.21.15.104:37148) 2024-06-09T22:30:38.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f9642ffd700 1 -- 172.21.15.104:0/4165783562 learned_addr learned my addr 172.21.15.104:0/4165783562 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:38.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f96437fe700 1 --2- 172.21.15.104:0/4165783562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f97c30 0x55ac25f98090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.876+0000 7f963a7fc700 1 --2- 172.21.15.104:0/4165783562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25f976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.877+0000 7f9642ffd700 1 -- 172.21.15.104:0/4165783562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 msgr2=0x55ac25f976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:38.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.877+0000 7f9642ffd700 1 --2- 172.21.15.104:0/4165783562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25f976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.877+0000 7f9642ffd700 1 -- 172.21.15.104:0/4165783562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f97c30 msgr2=0x55ac25f98090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:38.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.877+0000 7f9642ffd700 1 --2- 172.21.15.104:0/4165783562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f97c30 0x55ac25f98090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.877+0000 7f9642ffd700 1 -- 172.21.15.104:0/4165783562 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac25e7ef50 con 0x55ac25f92fd0 2024-06-09T22:30:38.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.877+0000 7f96437fe700 1 --2- 172.21.15.104:0/4165783562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f97c30 0x55ac25f98090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:30:38.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.877+0000 7f9642ffd700 1 --2- 172.21.15.104:0/4165783562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 0x55ac25f933b0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f963409af30 tx=0x7f963409b130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e4ad4286c4c85b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:38.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.877+0000 7f9639ffb700 1 -- 172.21.15.104:0/4165783562 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963409f5e0 con 0x55ac25f92fd0 2024-06-09T22:30:38.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.877+0000 7f9639ffb700 1 -- 172.21.15.104:0/4165783562 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96340a3ea0 con 0x55ac25f92fd0 2024-06-09T22:30:38.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.877+0000 7f9639ffb700 1 -- 172.21.15.104:0/4165783562 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963409f3d0 con 0x55ac25f92fd0 2024-06-09T22:30:38.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.878+0000 7f96542b3b80 1 -- 172.21.15.104:0/4165783562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 msgr2=0x55ac25f933b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:38.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.878+0000 7f96542b3b80 1 --2- 172.21.15.104:0/4165783562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 0x55ac25f933b0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f963409af30 tx=0x7f963409b130 comp rx=0 tx=0).stop 2024-06-09T22:30:38.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.878+0000 7f96542b3b80 1 -- 172.21.15.104:0/4165783562 shutdown_connections 2024-06-09T22:30:38.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.878+0000 7f96542b3b80 1 --2- 172.21.15.104:0/4165783562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 0x55ac25f933b0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.878+0000 7f96542b3b80 1 --2- 172.21.15.104:0/4165783562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f97c30 0x55ac25f98090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.878+0000 7f96542b3b80 1 --2- 172.21.15.104:0/4165783562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25f976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.878+0000 7f96542b3b80 1 -- 172.21.15.104:0/4165783562 >> 172.21.15.104:0/4165783562 conn(0x55ac25eb9cc0 msgr2=0x55ac25f89850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:38.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.878+0000 7f96542b3b80 1 -- 172.21.15.104:0/4165783562 shutdown_connections 2024-06-09T22:30:38.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.878+0000 7f96542b3b80 1 -- 172.21.15.104:0/4165783562 wait complete. 2024-06-09T22:30:38.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.878+0000 7f96542b3b80 1 -- start start 2024-06-09T22:30:38.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.878+0000 7f96542b3b80 1 Processor -- start 2024-06-09T22:30:38.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96542b3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f92fd0 0x55ac25f9f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96542b3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f938f0 0x55ac25f9c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96542b3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 0x55ac25f9cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96542b3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55ac25f9fd70 con 0x55ac25f97c30 2024-06-09T22:30:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96542b3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ac25e80bf0 con 0x55ac25f938f0 2024-06-09T22:30:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96542b3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ac25f9fef0 con 0x55ac25f92fd0 2024-06-09T22:30:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96542b3b80 10 client.0 _ll_register_callbacks cb 0x55ac25e7a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 0x55ac25f9cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 0x55ac25f9cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37152/0 (socket says 172.21.15.104:37152) 2024-06-09T22:30:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96437fe700 1 -- 172.21.15.104:0/2728476599 learned_addr learned my addr 172.21.15.104:0/2728476599 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f9642ffd700 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f92fd0 0x55ac25f9f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f963a7fc700 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f938f0 0x55ac25f9c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96437fe700 1 -- 172.21.15.104:0/2728476599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f92fd0 msgr2=0x55ac25f9f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:38.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96437fe700 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f92fd0 0x55ac25f9f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96437fe700 1 -- 172.21.15.104:0/2728476599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f938f0 msgr2=0x55ac25f9c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:38.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96437fe700 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f938f0 0x55ac25f9c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96437fe700 1 -- 172.21.15.104:0/2728476599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac25e2f8d0 con 0x55ac25f97c30 2024-06-09T22:30:38.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f963a7fc700 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f938f0 0x55ac25f9c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:30:38.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f96437fe700 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 0x55ac25f9cb60 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f963c008c40 tx=0x7f963c008f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b50159de06ea06 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:38.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f9622ffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:38.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f9622ffd700 1 -- 172.21.15.104:0/2728476599 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963c004bb0 con 0x55ac25f97c30 2024-06-09T22:30:38.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.879+0000 7f9622ffd700 1 -- 172.21.15.104:0/2728476599 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f963c004d50 con 0x55ac25f97c30 2024-06-09T22:30:38.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f96542b3b80 1 -- 172.21.15.104:0/2728476599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96340a3ea0 con 0x55ac25f97c30 2024-06-09T22:30:38.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f9622ffd700 1 -- 172.21.15.104:0/2728476599 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f963c022050 con 0x55ac25f97c30 2024-06-09T22:30:38.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f96542b3b80 10 client.17976 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:38.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f96542b3b80 1 -- 172.21.15.104:0/2728476599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ac25faa7d0 con 0x55ac25f97c30 2024-06-09T22:30:38.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f9622ffd700 1 -- 172.21.15.104:0/2728476599 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96340a3ea0 con 0x55ac25f97c30 2024-06-09T22:30:38.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f9622ffd700 1 -- 172.21.15.104:0/2728476599 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f963c015b70 con 0x55ac25f97c30 2024-06-09T22:30:38.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f96542b3b80 10 client.17976 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:38.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f96542b3b80 10 client.17976 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:30:38.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f96542b3b80 20 client.17976 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:38.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f96542b3b80 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ac25face60 0x55ac25faf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f96542b3b80 4 client.17976 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:30:38.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f96542b3b80 1 -- 172.21.15.104:0/2728476599 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac25eedd10 con 0x55ac25face60 2024-06-09T22:30:38.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.880+0000 7f9642ffd700 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ac25face60 0x55ac25faf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.881+0000 7f9642ffd700 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ac25face60 0x55ac25faf250 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b7b5c8a92060a90 server_cookie=ef874cc3796802d8 in_seq=0 out_seq=0 2024-06-09T22:30:38.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.881+0000 7f9622ffd700 10 client.17976 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:38.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f9622ffd700 1 -- 172.21.15.104:0/2728476599 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55ac25eedd10 con 0x55ac25face60 2024-06-09T22:30:38.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f9622ffd700 10 client.17976 handle_command_reply: tid=0 2024-06-09T22:30:38.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f96542b3b80 1 client.17976 shutdown 2024-06-09T22:30:38.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f96542b3b80 1 -- 172.21.15.104:0/2728476599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 msgr2=0x55ac25f9cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:38.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f96542b3b80 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 0x55ac25f9cb60 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f963c008c40 tx=0x7f963c008f50 comp rx=0 tx=0).stop 2024-06-09T22:30:38.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f96542b3b80 1 -- 172.21.15.104:0/2728476599 shutdown_connections 2024-06-09T22:30:38.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f96542b3b80 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ac25face60 0x55ac25faf250 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f96542b3b80 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 0x55ac25f9cb60 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f96542b3b80 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f92fd0 0x55ac25f9f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f96542b3b80 1 --2- 172.21.15.104:0/2728476599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f938f0 0x55ac25f9c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f96542b3b80 1 -- 172.21.15.104:0/2728476599 >> 172.21.15.104:0/2728476599 conn(0x55ac25eb9cc0 msgr2=0x55ac25f8d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:38.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f9622ffd700 0 client.17976 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:38.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f96542b3b80 1 -- 172.21.15.104:0/2728476599 shutdown_connections 2024-06-09T22:30:38.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f96542b3b80 1 -- 172.21.15.104:0/2728476599 wait complete. 2024-06-09T22:30:38.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.882+0000 7f96542b3b80 20 client.17976 trim_cache size 0 max 16384 2024-06-09T22:30:38.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.902+0000 7f96542b3b80 1 Processor -- start 2024-06-09T22:30:38.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.902+0000 7f96542b3b80 1 -- start start 2024-06-09T22:30:38.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.902+0000 7f96542b3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 0x55ac25f9a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.902+0000 7f96542b3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25fa9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.902+0000 7f96542b3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f97c30 0x55ac25f83510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.902+0000 7f96542b3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f96441ac480 con 0x55ac25f92fd0 2024-06-09T22:30:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.902+0000 7f96542b3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96441adfc0 con 0x55ac25f97c30 2024-06-09T22:30:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.902+0000 7f96542b3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f96441acb50 con 0x55ac25f938f0 2024-06-09T22:30:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.902+0000 7f9642ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 0x55ac25f9a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.902+0000 7f9642ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 0x55ac25f9a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37162/0 (socket says 172.21.15.104:37162) 2024-06-09T22:30:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.902+0000 7f9642ffd700 1 -- 172.21.15.104:0/1335175647 learned_addr learned my addr 172.21.15.104:0/1335175647 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.902+0000 7f963a7fc700 1 --2- 172.21.15.104:0/1335175647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25fa9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.903+0000 7f96437fe700 1 --2- 172.21.15.104:0/1335175647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f97c30 0x55ac25f83510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.903+0000 7f9642ffd700 1 -- 172.21.15.104:0/1335175647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 msgr2=0x55ac25fa9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.903+0000 7f9642ffd700 1 --2- 172.21.15.104:0/1335175647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25fa9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.903+0000 7f9642ffd700 1 -- 172.21.15.104:0/1335175647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f97c30 msgr2=0x55ac25f83510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.903+0000 7f9642ffd700 1 --2- 172.21.15.104:0/1335175647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f97c30 0x55ac25f83510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.903+0000 7f9642ffd700 1 -- 172.21.15.104:0/1335175647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac25f9d140 con 0x55ac25f92fd0 2024-06-09T22:30:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.903+0000 7f9642ffd700 1 --2- 172.21.15.104:0/1335175647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 0x55ac25f9a820 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f9634097690 tx=0x7f9634010a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a908ab3a4c49e1f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.903+0000 7f96397fa700 1 -- 172.21.15.104:0/1335175647 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96340a3ea0 con 0x55ac25f92fd0 2024-06-09T22:30:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.903+0000 7f96397fa700 1 -- 172.21.15.104:0/1335175647 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f963409f5e0 con 0x55ac25f92fd0 2024-06-09T22:30:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.903+0000 7f96397fa700 1 -- 172.21.15.104:0/1335175647 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963409f3d0 con 0x55ac25f92fd0 2024-06-09T22:30:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.903+0000 7f96542b3b80 1 -- 172.21.15.104:0/1335175647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 msgr2=0x55ac25f9a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.903+0000 7f96542b3b80 1 --2- 172.21.15.104:0/1335175647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 0x55ac25f9a820 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f9634097690 tx=0x7f9634010a10 comp rx=0 tx=0).stop 2024-06-09T22:30:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 -- 172.21.15.104:0/1335175647 shutdown_connections 2024-06-09T22:30:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 --2- 172.21.15.104:0/1335175647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f92fd0 0x55ac25f9a820 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 --2- 172.21.15.104:0/1335175647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f97c30 0x55ac25f83510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 --2- 172.21.15.104:0/1335175647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25fa9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 -- 172.21.15.104:0/1335175647 >> 172.21.15.104:0/1335175647 conn(0x55ac25eb9cc0 msgr2=0x55ac25f82e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:38.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 -- 172.21.15.104:0/1335175647 shutdown_connections 2024-06-09T22:30:38.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 -- 172.21.15.104:0/1335175647 wait complete. 2024-06-09T22:30:38.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 -- start start 2024-06-09T22:30:38.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 Processor -- start 2024-06-09T22:30:38.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f92fd0 0x55ac25fa56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25fb4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 0x55ac25fb4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f96441ad8f0 con 0x55ac25f97c30 2024-06-09T22:30:38.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96441ad220 con 0x55ac25f92fd0 2024-06-09T22:30:38.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f96441aa440 con 0x55ac25f938f0 2024-06-09T22:30:38.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.904+0000 7f96542b3b80 10 client.0 _ll_register_callbacks cb 0x55ac25e7a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:30:38.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f96437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 0x55ac25fb4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f96437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 0x55ac25fb4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37168/0 (socket says 172.21.15.104:37168) 2024-06-09T22:30:38.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f96437fe700 1 -- 172.21.15.104:0/869226263 learned_addr learned my addr 172.21.15.104:0/869226263 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:30:38.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f9642ffd700 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f92fd0 0x55ac25fa56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f963a7fc700 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25fb4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f96437fe700 1 -- 172.21.15.104:0/869226263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 msgr2=0x55ac25fb4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f96437fe700 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25fb4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f96437fe700 1 -- 172.21.15.104:0/869226263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f92fd0 msgr2=0x55ac25fa56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:30:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f96437fe700 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f92fd0 0x55ac25fa56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f96437fe700 1 -- 172.21.15.104:0/869226263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac25fb4d00 con 0x55ac25f97c30 2024-06-09T22:30:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f963a7fc700 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25fb4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:30:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f96437fe700 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 0x55ac25fb4740 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f963c002230 tx=0x7f963c007b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd82f9d8cddb704 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:30:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f9622ffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:30:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f9622ffd700 1 -- 172.21.15.104:0/869226263 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963c005c60 con 0x55ac25f97c30 2024-06-09T22:30:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f9622ffd700 1 -- 172.21.15.104:0/869226263 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f963c005e00 con 0x55ac25f97c30 2024-06-09T22:30:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f96542b3b80 1 -- 172.21.15.104:0/869226263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f963409f5e0 con 0x55ac25f97c30 2024-06-09T22:30:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f9622ffd700 1 -- 172.21.15.104:0/869226263 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f963c035050 con 0x55ac25f97c30 2024-06-09T22:30:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f96542b3b80 10 client.17982 fetch_fsmap learned FSMap version 32 2024-06-09T22:30:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.905+0000 7f96542b3b80 1 -- 172.21.15.104:0/869226263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ac260178f0 con 0x55ac25f97c30 2024-06-09T22:30:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.906+0000 7f9622ffd700 1 -- 172.21.15.104:0/869226263 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f963409f5e0 con 0x55ac25f97c30 2024-06-09T22:30:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.906+0000 7f9622ffd700 1 -- 172.21.15.104:0/869226263 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f963c0058d0 con 0x55ac25f97c30 2024-06-09T22:30:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.906+0000 7f96542b3b80 10 client.17982 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:30:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.906+0000 7f96542b3b80 10 client.17982 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:30:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.906+0000 7f96542b3b80 20 client.17982 populate_metadata read hostname 'smithi104' 2024-06-09T22:30:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.906+0000 7f96542b3b80 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ac26018080 0x55ac2601a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:30:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.906+0000 7f96542b3b80 4 client.17982 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:30:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.906+0000 7f96542b3b80 1 -- 172.21.15.104:0/869226263 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ac25ee8290 con 0x55ac26018080 2024-06-09T22:30:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.906+0000 7f9642ffd700 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ac26018080 0x55ac2601a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:30:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.906+0000 7f9642ffd700 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ac26018080 0x55ac2601a4a0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=faf92f3e0f3a846 server_cookie=29f352dba3bcf447 in_seq=0 out_seq=0 2024-06-09T22:30:38.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.906+0000 7f9622ffd700 10 client.17982 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:38.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f9622ffd700 1 -- 172.21.15.104:0/869226263 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ac25ee8290 con 0x55ac26018080 2024-06-09T22:30:38.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f9622ffd700 10 client.17982 handle_command_reply: tid=0 2024-06-09T22:30:38.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f96542b3b80 1 client.17982 shutdown 2024-06-09T22:30:38.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f96542b3b80 1 -- 172.21.15.104:0/869226263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 msgr2=0x55ac25fb4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:38.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f96542b3b80 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 0x55ac25fb4740 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f963c002230 tx=0x7f963c007b60 comp rx=0 tx=0).stop 2024-06-09T22:30:38.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f96542b3b80 1 -- 172.21.15.104:0/869226263 shutdown_connections 2024-06-09T22:30:38.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f96542b3b80 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55ac26018080 0x55ac2601a4a0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f96542b3b80 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55ac25f97c30 0x55ac25fb4740 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f96542b3b80 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ac25f92fd0 0x55ac25fa56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f9622ffd700 0 client.17982 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:30:38.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f96542b3b80 1 --2- 172.21.15.104:0/869226263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ac25f938f0 0x55ac25fb4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f96542b3b80 1 -- 172.21.15.104:0/869226263 >> 172.21.15.104:0/869226263 conn(0x55ac25eb9cc0 msgr2=0x55ac25f9de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:38.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f96542b3b80 1 -- 172.21.15.104:0/869226263 shutdown_connections 2024-06-09T22:30:38.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f96542b3b80 1 -- 172.21.15.104:0/869226263 wait complete. 2024-06-09T22:30:38.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.907+0000 7f96542b3b80 20 client.17982 trim_cache size 0 max 16384 2024-06-09T22:30:38.920 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:30:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-09T22:30:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": { 2024-06-09T22:30:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: "9d2a52f3-a1e8-4921-9f3d-ef95d2749657": { 2024-06-09T22:30:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: "path": "/", 2024-06-09T22:30:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: "tag": "9d2a52f3-a1e8-4921-9f3d-ef95d2749657", 2024-06-09T22:30:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: "options": "recursive,force" 2024-06-09T22:30:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:30:38.922 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:30:38.922 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:30:38.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.908+0000 7f9649a30700 1 -- 172.21.15.104:0/652830971 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f96300a2d90 msgr2=0x7f96300a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:38.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.908+0000 7f9649a30700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f96300a2d90 0x7f96300a5250 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f962c00b5c0 tx=0x7f962c005960 comp rx=0 tx=0).stop 2024-06-09T22:30:38.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.908+0000 7f9649a30700 1 -- 172.21.15.104:0/652830971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96441013c0 msgr2=0x7f964419e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:30:38.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.908+0000 7f9649a30700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96441013c0 0x7f964419e920 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f963400b530 tx=0x7f963400b840 comp rx=0 tx=0).stop 2024-06-09T22:30:38.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.908+0000 7f9649a30700 1 -- 172.21.15.104:0/652830971 shutdown_connections 2024-06-09T22:30:38.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.908+0000 7f9649a30700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f96300a2d90 0x7f96300a5250 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.908+0000 7f9649a30700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96441013c0 0x7f964419e920 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.908+0000 7f9649a30700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9644107430 0x7f9644198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.908+0000 7f9649a30700 1 --2- 172.21.15.104:0/652830971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9644101d80 0x7f964419ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:30:38.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.908+0000 7f9649a30700 1 -- 172.21.15.104:0/652830971 >> 172.21.15.104:0/652830971 conn(0x7f96440783d0 msgr2=0x7f96441006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:30:38.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.908+0000 7f9649a30700 1 -- 172.21.15.104:0/652830971 shutdown_connections 2024-06-09T22:30:38.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:30:38.908+0000 7f9649a30700 1 -- 172.21.15.104:0/652830971 wait complete. 2024-06-09T22:30:39.644 INFO:tasks.cephfs.filesystem:scrub status for tag:9d2a52f3-a1e8-4921-9f3d-ef95d2749657 - {'path': '/', 'tag': '9d2a52f3-a1e8-4921-9f3d-ef95d2749657', 'options': 'recursive,force'} 2024-06-09T22:30:39.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:39 smithi153 ceph-mon[36599]: pgmap v299: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2024-06-09T22:30:40.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:39 smithi104 ceph-mon[32389]: pgmap v299: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2024-06-09T22:30:40.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:39 smithi194 ceph-mon[36856]: pgmap v299: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2024-06-09T22:30:41.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:41 smithi153 ceph-mon[36599]: pgmap v300: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.6 KiB/s wr, 20 op/s 2024-06-09T22:30:42.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:41 smithi104 ceph-mon[32389]: pgmap v300: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.6 KiB/s wr, 20 op/s 2024-06-09T22:30:42.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:41 smithi194 ceph-mon[36856]: pgmap v300: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.6 KiB/s wr, 20 op/s 2024-06-09T22:30:43.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:43 smithi153 ceph-mon[36599]: pgmap v301: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-06-09T22:30:44.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:43 smithi104 ceph-mon[32389]: pgmap v301: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-06-09T22:30:44.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:43 smithi194 ceph-mon[36856]: pgmap v301: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-06-09T22:30:45.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:45 smithi153 ceph-mon[36599]: pgmap v302: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 3.6 KiB/s wr, 19 op/s 2024-06-09T22:30:46.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:45 smithi104 ceph-mon[32389]: pgmap v302: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 3.6 KiB/s wr, 19 op/s 2024-06-09T22:30:46.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:45 smithi194 ceph-mon[36856]: pgmap v302: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 3.6 KiB/s wr, 19 op/s 2024-06-09T22:30:47.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:47 smithi153 ceph-mon[36599]: pgmap v303: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.1 KiB/s wr, 15 op/s 2024-06-09T22:30:48.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:47 smithi104 ceph-mon[32389]: pgmap v303: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.1 KiB/s wr, 15 op/s 2024-06-09T22:30:48.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:47 smithi194 ceph-mon[36856]: pgmap v303: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.1 KiB/s wr, 15 op/s 2024-06-09T22:30:49.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:49 smithi153 ceph-mon[36599]: pgmap v304: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.1 KiB/s wr, 15 op/s 2024-06-09T22:30:50.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:49 smithi104 ceph-mon[32389]: pgmap v304: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.1 KiB/s wr, 15 op/s 2024-06-09T22:30:50.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:49 smithi194 ceph-mon[36856]: pgmap v304: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.1 KiB/s wr, 15 op/s 2024-06-09T22:30:50.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:50 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:30:50.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:50 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:30:50.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:50 smithi153 ceph-mon[36599]: scrub summary: idle 2024-06-09T22:30:51.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:50 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:30:51.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:50 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:30:51.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:50 smithi104 ceph-mon[32389]: scrub summary: idle 2024-06-09T22:30:51.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:50 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:30:51.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:50 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:30:51.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:50 smithi194 ceph-mon[36856]: scrub summary: idle 2024-06-09T22:30:51.910 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:51 smithi104 ceph-mon[32389]: pgmap v305: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.1 KiB/s wr, 19 op/s 2024-06-09T22:30:51.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:51 smithi153 ceph-mon[36599]: pgmap v305: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.1 KiB/s wr, 19 op/s 2024-06-09T22:30:52.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:51 smithi194 ceph-mon[36856]: pgmap v305: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.1 KiB/s wr, 19 op/s 2024-06-09T22:30:53.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:53 smithi153 ceph-mon[36599]: pgmap v306: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:30:54.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:53 smithi104 ceph-mon[32389]: pgmap v306: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:30:54.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:53 smithi194 ceph-mon[36856]: pgmap v306: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:30:55.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:55 smithi194 ceph-mon[36856]: pgmap v307: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:30:55.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:55 smithi153 ceph-mon[36599]: pgmap v307: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:30:56.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:55 smithi104 ceph-mon[32389]: pgmap v307: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:30:57.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:57 smithi153 ceph-mon[36599]: pgmap v308: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:30:58.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:57 smithi104 ceph-mon[32389]: pgmap v308: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:30:58.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:57 smithi194 ceph-mon[36856]: pgmap v308: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:30:58.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:30:58 smithi153 ceph-mon[36599]: pgmap v309: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:30:59.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:30:58 smithi104 ceph-mon[32389]: pgmap v309: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:30:59.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:30:58 smithi194 ceph-mon[36856]: pgmap v309: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:01.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:00 smithi153 ceph-mon[36599]: pgmap v310: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:01.259 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:00 smithi104 ceph-mon[32389]: pgmap v310: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:01.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:00 smithi194 ceph-mon[36856]: pgmap v310: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:03.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:02 smithi104 ceph-mon[32389]: pgmap v311: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:03.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:02 smithi153 ceph-mon[36599]: pgmap v311: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:03.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:02 smithi194 ceph-mon[36856]: pgmap v311: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:05.246 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:04 smithi153 ceph-mon[36599]: pgmap v312: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-06-09T22:31:05.259 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:04 smithi104 ceph-mon[32389]: pgmap v312: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-06-09T22:31:05.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:04 smithi194 ceph-mon[36856]: pgmap v312: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-06-09T22:31:07.246 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:06 smithi153 ceph-mon[36599]: pgmap v313: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-06-09T22:31:07.259 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:06 smithi104 ceph-mon[32389]: pgmap v313: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-06-09T22:31:07.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:06 smithi194 ceph-mon[36856]: pgmap v313: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-06-09T22:31:09.246 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:08 smithi153 ceph-mon[36599]: pgmap v314: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-06-09T22:31:09.259 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:08 smithi104 ceph-mon[32389]: pgmap v314: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-06-09T22:31:09.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:08 smithi194 ceph-mon[36856]: pgmap v314: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-06-09T22:31:09.645 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-09T22:31:09.922 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:31:11.188 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:10 smithi104 ceph-mon[32389]: pgmap v315: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-06-09T22:31:11.188 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:10 smithi104 ceph-mon[32389]: osdmap e107: 12 total, 12 up, 12 in 2024-06-09T22:31:11.246 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:10 smithi153 ceph-mon[36599]: pgmap v315: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-06-09T22:31:11.246 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:10 smithi153 ceph-mon[36599]: osdmap e107: 12 total, 12 up, 12 in 2024-06-09T22:31:11.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:10 smithi194 ceph-mon[36856]: pgmap v315: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-06-09T22:31:11.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:10 smithi194 ceph-mon[36856]: osdmap e107: 12 total, 12 up, 12 in 2024-06-09T22:31:11.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.767+0000 7f602e54d700 1 -- 172.21.15.104:0/1551210786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6028102510 msgr2=0x7f6028102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:11.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.767+0000 7f602e54d700 1 --2- 172.21.15.104:0/1551210786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6028102510 0x7f6028102990 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f601c009990 tx=0x7f601c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.767+0000 7f602e54d700 1 -- 172.21.15.104:0/1551210786 shutdown_connections 2024-06-09T22:31:11.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.767+0000 7f602e54d700 1 --2- 172.21.15.104:0/1551210786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6028102ed0 0x7f602810f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.767+0000 7f602e54d700 1 --2- 172.21.15.104:0/1551210786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6028108520 0x7f6028108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.767+0000 7f602e54d700 1 --2- 172.21.15.104:0/1551210786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6028102510 0x7f6028102990 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.767+0000 7f602e54d700 1 -- 172.21.15.104:0/1551210786 >> 172.21.15.104:0/1551210786 conn(0x7f6028079640 msgr2=0x7f6028079a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:11.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.767+0000 7f602e54d700 1 -- 172.21.15.104:0/1551210786 shutdown_connections 2024-06-09T22:31:11.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.767+0000 7f602e54d700 1 -- 172.21.15.104:0/1551210786 wait complete. 2024-06-09T22:31:11.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.768+0000 7f602e54d700 1 Processor -- start 2024-06-09T22:31:11.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.768+0000 7f602e54d700 1 -- start start 2024-06-09T22:31:11.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.768+0000 7f602e54d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6028102510 0x7f6028074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.768+0000 7f602e54d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6028102ed0 0x7f6028074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.768+0000 7f602e54d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6028108520 0x7f6028075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.768+0000 7f602e54d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f60281127f0 con 0x7f6028102510 2024-06-09T22:31:11.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.768+0000 7f602e54d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6028112670 con 0x7f6028108520 2024-06-09T22:31:11.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.768+0000 7f602e54d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6028112970 con 0x7f6028102ed0 2024-06-09T22:31:11.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f6027fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6028102510 0x7f6028074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f6027fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6028102510 0x7f6028074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34304/0 (socket says 172.21.15.104:34304) 2024-06-09T22:31:11.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f6027fff700 1 -- 172.21.15.104:0/1033207580 learned_addr learned my addr 172.21.15.104:0/1033207580 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:11.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f602caea700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6028108520 0x7f6028075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f60277fe700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6028102ed0 0x7f6028074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f6027fff700 1 -- 172.21.15.104:0/1033207580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6028102ed0 msgr2=0x7f6028074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:11.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f6027fff700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6028102ed0 0x7f6028074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f6027fff700 1 -- 172.21.15.104:0/1033207580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6028108520 msgr2=0x7f6028075210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:11.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f6027fff700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6028108520 0x7f6028075210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f6027fff700 1 -- 172.21.15.104:0/1033207580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60281a70b0 con 0x7f6028102510 2024-06-09T22:31:11.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f6027fff700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6028102510 0x7f6028074790 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f601800d980 tx=0x7f601800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:11.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f60257fa700 1 -- 172.21.15.104:0/1033207580 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6018016070 con 0x7f6028102510 2024-06-09T22:31:11.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f602e54d700 1 -- 172.21.15.104:0/1033207580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60281a73a0 con 0x7f6028102510 2024-06-09T22:31:11.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f60257fa700 1 -- 172.21.15.104:0/1033207580 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6018004d10 con 0x7f6028102510 2024-06-09T22:31:11.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f60257fa700 1 -- 172.21.15.104:0/1033207580 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6018005670 con 0x7f6028102510 2024-06-09T22:31:11.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.769+0000 7f602e54d700 1 -- 172.21.15.104:0/1033207580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60281a78e0 con 0x7f6028102510 2024-06-09T22:31:11.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.771+0000 7f60257fa700 1 -- 172.21.15.104:0/1033207580 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f6018004750 con 0x7f6028102510 2024-06-09T22:31:11.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.772+0000 7f60257fa700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f60100a2d90 0x7f60100a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.773+0000 7f60277fe700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f60100a2d90 0x7f60100a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.773+0000 7f60257fa700 1 -- 172.21.15.104:0/1033207580 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7f60180cf620 con 0x7f6028102510 2024-06-09T22:31:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.773+0000 7f60277fe700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f60100a2d90 0x7f60100a5250 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6028079c40 tx=0x7f601c00b540 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.773+0000 7f6038dd0b80 1 Processor -- start 2024-06-09T22:31:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.773+0000 7f6038dd0b80 1 -- start start 2024-06-09T22:31:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6038dd0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 0x5650b0318a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6038dd0b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318f60 0x5650b03129e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6038dd0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b0313340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6038dd0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5650b0264810 con 0x5650b0318640 2024-06-09T22:31:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6038dd0b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5650b020c720 con 0x5650b0318f60 2024-06-09T22:31:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6038dd0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5650b02607d0 con 0x5650b0312f20 2024-06-09T22:31:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6027fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 0x5650b0318a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6027fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 0x5650b0318a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34312/0 (socket says 172.21.15.104:34312) 2024-06-09T22:31:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6027fff700 1 -- 172.21.15.104:0/206086726 learned_addr learned my addr 172.21.15.104:0/206086726 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f60277fe700 1 --2- 172.21.15.104:0/206086726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318f60 0x5650b03129e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f602caea700 1 --2- 172.21.15.104:0/206086726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b0313340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6027fff700 1 -- 172.21.15.104:0/206086726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 msgr2=0x5650b0313340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6027fff700 1 --2- 172.21.15.104:0/206086726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b0313340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6027fff700 1 -- 172.21.15.104:0/206086726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318f60 msgr2=0x5650b03129e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6027fff700 1 --2- 172.21.15.104:0/206086726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318f60 0x5650b03129e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6027fff700 1 -- 172.21.15.104:0/206086726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650b0202f50 con 0x5650b0318640 2024-06-09T22:31:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f602caea700 1 --2- 172.21.15.104:0/206086726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b0313340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:31:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f6027fff700 1 --2- 172.21.15.104:0/206086726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 0x5650b0318a20 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f60180980b0 tx=0x7f601809b110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd782c3c43628e87 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.774+0000 7f600e7fc700 1 -- 172.21.15.104:0/206086726 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60180860c0 con 0x5650b0318640 2024-06-09T22:31:11.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.775+0000 7f600e7fc700 1 -- 172.21.15.104:0/206086726 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f601809b6e0 con 0x5650b0318640 2024-06-09T22:31:11.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.775+0000 7f600e7fc700 1 -- 172.21.15.104:0/206086726 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601809b940 con 0x5650b0318640 2024-06-09T22:31:11.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.775+0000 7f6038dd0b80 1 -- 172.21.15.104:0/206086726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 msgr2=0x5650b0318a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:11.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.775+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/206086726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 0x5650b0318a20 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f60180980b0 tx=0x7f601809b110 comp rx=0 tx=0).stop 2024-06-09T22:31:11.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.775+0000 7f6038dd0b80 1 -- 172.21.15.104:0/206086726 shutdown_connections 2024-06-09T22:31:11.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.775+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/206086726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 0x5650b0318a20 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.775+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/206086726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b0313340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.775+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/206086726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318f60 0x5650b03129e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.775+0000 7f6038dd0b80 1 -- 172.21.15.104:0/206086726 >> 172.21.15.104:0/206086726 conn(0x5650b023dcc0 msgr2=0x5650b030d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:11.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.775+0000 7f6038dd0b80 1 -- 172.21.15.104:0/206086726 shutdown_connections 2024-06-09T22:31:11.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.775+0000 7f6038dd0b80 1 -- 172.21.15.104:0/206086726 wait complete. 2024-06-09T22:31:11.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.776+0000 7f6038dd0b80 1 -- start start 2024-06-09T22:31:11.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.776+0000 7f6038dd0b80 1 Processor -- start 2024-06-09T22:31:11.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.776+0000 7f6038dd0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b031ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.776+0000 7f6038dd0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 0x5650b031c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.776+0000 7f6038dd0b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318f60 0x5650b031c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.776+0000 7f6038dd0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5650b031f9d0 con 0x5650b0318640 2024-06-09T22:31:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.776+0000 7f6038dd0b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5650b0204bf0 con 0x5650b0318f60 2024-06-09T22:31:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.776+0000 7f6038dd0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5650b031fb50 con 0x5650b0312f20 2024-06-09T22:31:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.776+0000 7f6038dd0b80 10 client.0 _ll_register_callbacks cb 0x5650b01fe920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:31:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f60277fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 0x5650b031c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f6027fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b031ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f60277fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 0x5650b031c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34324/0 (socket says 172.21.15.104:34324) 2024-06-09T22:31:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f6027fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b031ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.104:34560/0 (socket says 172.21.15.104:34560) 2024-06-09T22:31:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f60277fe700 1 -- 172.21.15.104:0/1524954502 learned_addr learned my addr 172.21.15.104:0/1524954502 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f602caea700 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318f60 0x5650b031c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f6027fff700 1 -- 172.21.15.104:0/1524954502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318f60 msgr2=0x5650b031c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f6027fff700 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318f60 0x5650b031c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f6027fff700 1 -- 172.21.15.104:0/1524954502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 msgr2=0x5650b031c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f6027fff700 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 0x5650b031c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f6027fff700 1 -- 172.21.15.104:0/1524954502 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650b01b38d0 con 0x5650b0312f20 2024-06-09T22:31:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f60277fe700 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 0x5650b031c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:31:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f6027fff700 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b031ed10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f601809b3e0 tx=0x7f601809fb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63c0d2df01a6c007 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f60077fe700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:31:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f60077fe700 1 -- 172.21.15.104:0/1524954502 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6018004340 con 0x5650b0312f20 2024-06-09T22:31:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f60077fe700 1 -- 172.21.15.104:0/1524954502 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7f6018004a40 con 0x5650b0312f20 2024-06-09T22:31:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.777+0000 7f6038dd0b80 1 -- 172.21.15.104:0/1524954502 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f601809b6e0 con 0x5650b0312f20 2024-06-09T22:31:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.778+0000 7f60077fe700 1 -- 172.21.15.104:0/1524954502 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6018081050 con 0x5650b0312f20 2024-06-09T22:31:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.778+0000 7f6038dd0b80 10 client.27244 fetch_fsmap learned FSMap version 32 2024-06-09T22:31:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.778+0000 7f6038dd0b80 1 -- 172.21.15.104:0/1524954502 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5650b031f9d0 con 0x5650b0312f20 2024-06-09T22:31:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.778+0000 7f60077fe700 1 -- 172.21.15.104:0/1524954502 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f601809b6e0 con 0x5650b0312f20 2024-06-09T22:31:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.778+0000 7f60077fe700 1 -- 172.21.15.104:0/1524954502 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f60180a3ec0 con 0x5650b0312f20 2024-06-09T22:31:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.778+0000 7f6038dd0b80 10 client.27244 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:31:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.778+0000 7f6038dd0b80 10 client.27244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:31:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.778+0000 7f6038dd0b80 20 client.27244 populate_metadata read hostname 'smithi104' 2024-06-09T22:31:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.778+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5650b032c620 0x5650b032ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.778+0000 7f6038dd0b80 4 client.27244 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:31:11.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.778+0000 7f6038dd0b80 1 -- 172.21.15.104:0/1524954502 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650b0271d10 con 0x5650b032c620 2024-06-09T22:31:11.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.778+0000 7f60277fe700 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5650b032c620 0x5650b032ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.779+0000 7f60277fe700 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5650b032c620 0x5650b032ea10 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d7436557b65d514 server_cookie=d63f938df6e5bd4d in_seq=0 out_seq=0 2024-06-09T22:31:11.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.779+0000 7f60077fe700 10 client.27244 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:31:11.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f60077fe700 1 -- 172.21.15.104:0/1524954502 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5650b0271d10 con 0x5650b032c620 2024-06-09T22:31:11.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f60077fe700 10 client.27244 handle_command_reply: tid=0 2024-06-09T22:31:11.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f6038dd0b80 1 client.27244 shutdown 2024-06-09T22:31:11.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f6038dd0b80 1 -- 172.21.15.104:0/1524954502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 msgr2=0x5650b031ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:11.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b031ed10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f601809b3e0 tx=0x7f601809fb60 comp rx=0 tx=0).stop 2024-06-09T22:31:11.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f6038dd0b80 1 -- 172.21.15.104:0/1524954502 shutdown_connections 2024-06-09T22:31:11.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5650b032c620 0x5650b032ea10 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318f60 0x5650b031c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b031ed10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f60077fe700 0 client.27244 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:31:11.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/1524954502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318640 0x5650b031c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f6038dd0b80 1 -- 172.21.15.104:0/1524954502 >> 172.21.15.104:0/1524954502 conn(0x5650b023dcc0 msgr2=0x5650b03119d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:11.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f6038dd0b80 1 -- 172.21.15.104:0/1524954502 shutdown_connections 2024-06-09T22:31:11.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f6038dd0b80 1 -- 172.21.15.104:0/1524954502 wait complete. 2024-06-09T22:31:11.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.780+0000 7f6038dd0b80 20 client.27244 trim_cache size 0 max 16384 2024-06-09T22:31:11.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.800+0000 7f6038dd0b80 1 Processor -- start 2024-06-09T22:31:11.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.800+0000 7f6038dd0b80 1 -- start start 2024-06-09T22:31:11.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.800+0000 7f6038dd0b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0312f20 0x5650b03129e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.800+0000 7f6038dd0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0318640 0x5650b0319bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.800+0000 7f6038dd0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 0x5650b031aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.800+0000 7f6038dd0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f60281b0a00 con 0x5650b0318f60 2024-06-09T22:31:11.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.800+0000 7f6038dd0b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60281b2540 con 0x5650b0312f20 2024-06-09T22:31:11.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.800+0000 7f6038dd0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f60281b10d0 con 0x5650b0318640 2024-06-09T22:31:11.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.800+0000 7f602caea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 0x5650b031aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.800+0000 7f602caea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 0x5650b031aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34326/0 (socket says 172.21.15.104:34326) 2024-06-09T22:31:11.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.800+0000 7f602caea700 1 -- 172.21.15.104:0/655553317 learned_addr learned my addr 172.21.15.104:0/655553317 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:11.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.800+0000 7f6027fff700 1 --2- 172.21.15.104:0/655553317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0312f20 0x5650b03129e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.801+0000 7f60277fe700 1 --2- 172.21.15.104:0/655553317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0318640 0x5650b0319bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.801+0000 7f602caea700 1 -- 172.21.15.104:0/655553317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0318640 msgr2=0x5650b0319bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:11.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.801+0000 7f602caea700 1 --2- 172.21.15.104:0/655553317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0318640 0x5650b0319bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.801+0000 7f602caea700 1 -- 172.21.15.104:0/655553317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0312f20 msgr2=0x5650b03129e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:11.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.801+0000 7f602caea700 1 --2- 172.21.15.104:0/655553317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0312f20 0x5650b03129e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.801+0000 7f602caea700 1 -- 172.21.15.104:0/655553317 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650b031a580 con 0x5650b0318f60 2024-06-09T22:31:11.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.801+0000 7f60277fe700 1 --2- 172.21.15.104:0/655553317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0318640 0x5650b0319bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:31:11.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.801+0000 7f602caea700 1 --2- 172.21.15.104:0/655553317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 0x5650b031aee0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f6020014660 tx=0x7f60200078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ab77986dc70531 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:11.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.801+0000 7f600dffb700 1 -- 172.21.15.104:0/655553317 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602001a070 con 0x5650b0318f60 2024-06-09T22:31:11.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.801+0000 7f600dffb700 1 -- 172.21.15.104:0/655553317 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6020007da0 con 0x5650b0318f60 2024-06-09T22:31:11.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.801+0000 7f600dffb700 1 -- 172.21.15.104:0/655553317 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602001fcf0 con 0x5650b0318f60 2024-06-09T22:31:11.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.802+0000 7f6038dd0b80 1 -- 172.21.15.104:0/655553317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 msgr2=0x5650b031aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:11.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.802+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/655553317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 0x5650b031aee0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f6020014660 tx=0x7f60200078a0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.802+0000 7f6038dd0b80 1 -- 172.21.15.104:0/655553317 shutdown_connections 2024-06-09T22:31:11.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.802+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/655553317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 0x5650b031aee0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.802+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/655553317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0312f20 0x5650b03129e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.802+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/655553317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0318640 0x5650b0319bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.802+0000 7f6038dd0b80 1 -- 172.21.15.104:0/655553317 >> 172.21.15.104:0/655553317 conn(0x5650b023dcc0 msgr2=0x5650b031a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.802+0000 7f6038dd0b80 1 -- 172.21.15.104:0/655553317 shutdown_connections 2024-06-09T22:31:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.802+0000 7f6038dd0b80 1 -- 172.21.15.104:0/655553317 wait complete. 2024-06-09T22:31:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.802+0000 7f6038dd0b80 1 -- start start 2024-06-09T22:31:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.802+0000 7f6038dd0b80 1 Processor -- start 2024-06-09T22:31:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.802+0000 7f6038dd0b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b0320bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f6038dd0b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318640 0x5650b0321110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f6038dd0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 0x5650b0321650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f6038dd0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f60281b1e70 con 0x5650b0318f60 2024-06-09T22:31:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f6038dd0b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60281b17a0 con 0x5650b0318640 2024-06-09T22:31:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f6038dd0b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f60281ae9a0 con 0x5650b0312f20 2024-06-09T22:31:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f6038dd0b80 10 client.0 _ll_register_callbacks cb 0x5650b01fe920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:31:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f602caea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 0x5650b0321650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f602caea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 0x5650b0321650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34338/0 (socket says 172.21.15.104:34338) 2024-06-09T22:31:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f602caea700 1 -- 172.21.15.104:0/3237030646 learned_addr learned my addr 172.21.15.104:0/3237030646 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f6027fff700 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b0320bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f60277fe700 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318640 0x5650b0321110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f602caea700 1 -- 172.21.15.104:0/3237030646 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 msgr2=0x5650b0320bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f602caea700 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b0320bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f602caea700 1 -- 172.21.15.104:0/3237030646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318640 msgr2=0x5650b0321110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f602caea700 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318640 0x5650b0321110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f602caea700 1 -- 172.21.15.104:0/3237030646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650b0396060 con 0x5650b0318f60 2024-06-09T22:31:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f602caea700 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 0x5650b0321650 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f6020009010 tx=0x7f6020020bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eaad6db8956fd2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:11.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f60077fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:31:11.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f60077fe700 1 -- 172.21.15.104:0/3237030646 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602001a040 con 0x5650b0318f60 2024-06-09T22:31:11.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f6038dd0b80 1 -- 172.21.15.104:0/3237030646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6020007da0 con 0x5650b0318f60 2024-06-09T22:31:11.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.803+0000 7f60077fe700 1 -- 172.21.15.104:0/3237030646 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7f6020016630 con 0x5650b0318f60 2024-06-09T22:31:11.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.804+0000 7f60077fe700 1 -- 172.21.15.104:0/3237030646 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6020016830 con 0x5650b0318f60 2024-06-09T22:31:11.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.804+0000 7f6038dd0b80 10 client.18000 fetch_fsmap learned FSMap version 32 2024-06-09T22:31:11.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.804+0000 7f6038dd0b80 1 -- 172.21.15.104:0/3237030646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5650b03972e0 con 0x5650b0318f60 2024-06-09T22:31:11.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.804+0000 7f60077fe700 1 -- 172.21.15.104:0/3237030646 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6020007da0 con 0x5650b0318f60 2024-06-09T22:31:11.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.804+0000 7f60077fe700 1 -- 172.21.15.104:0/3237030646 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6020027070 con 0x5650b0318f60 2024-06-09T22:31:11.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.804+0000 7f6038dd0b80 10 client.18000 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:31:11.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.804+0000 7f6038dd0b80 10 client.18000 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:31:11.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.804+0000 7f6038dd0b80 20 client.18000 populate_metadata read hostname 'smithi104' 2024-06-09T22:31:11.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.804+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5650b03978b0 0x5650b0399cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:11.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.804+0000 7f6038dd0b80 4 client.18000 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-09T22:31:11.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.804+0000 7f6038dd0b80 1 -- 172.21.15.104:0/3237030646 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5650b026c290 con 0x5650b03978b0 2024-06-09T22:31:11.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.804+0000 7f6027fff700 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5650b03978b0 0x5650b0399cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:11.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6027fff700 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5650b03978b0 0x5650b0399cd0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26d586acfda42024 server_cookie=1302fa7f850bfc82 in_seq=0 out_seq=0 2024-06-09T22:31:11.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f60077fe700 10 client.18000 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:31:11.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f60077fe700 1 -- 172.21.15.104:0/3237030646 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5650b026c290 con 0x5650b03978b0 2024-06-09T22:31:11.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f60077fe700 10 client.18000 handle_command_reply: tid=0 2024-06-09T22:31:11.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6038dd0b80 1 client.18000 shutdown 2024-06-09T22:31:11.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6038dd0b80 1 -- 172.21.15.104:0/3237030646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 msgr2=0x5650b0321650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:11.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 0x5650b0321650 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f6020009010 tx=0x7f6020020bb0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6038dd0b80 1 -- 172.21.15.104:0/3237030646 shutdown_connections 2024-06-09T22:31:11.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x5650b03978b0 0x5650b0399cd0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5650b0318f60 0x5650b0321650 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5650b0312f20 0x5650b0320bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6038dd0b80 1 --2- 172.21.15.104:0/3237030646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5650b0318640 0x5650b0321110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6038dd0b80 1 -- 172.21.15.104:0/3237030646 >> 172.21.15.104:0/3237030646 conn(0x5650b023dcc0 msgr2=0x5650b0317a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:11.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f60077fe700 0 client.18000 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:31:11.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6038dd0b80 1 -- 172.21.15.104:0/3237030646 shutdown_connections 2024-06-09T22:31:11.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6038dd0b80 1 -- 172.21.15.104:0/3237030646 wait complete. 2024-06-09T22:31:11.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.805+0000 7f6038dd0b80 20 client.18000 trim_cache size 0 max 16384 2024-06-09T22:31:11.818 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:31:11.818 INFO:teuthology.orchestra.run.smithi104.stdout: "status": "no active scrubs running", 2024-06-09T22:31:11.818 INFO:teuthology.orchestra.run.smithi104.stdout: "scrubs": {} 2024-06-09T22:31:11.818 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:31:11.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.806+0000 7f602e54d700 1 -- 172.21.15.104:0/1033207580 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f60100a2d90 msgr2=0x7f60100a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:11.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.806+0000 7f602e54d700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f60100a2d90 0x7f60100a5250 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6028079c40 tx=0x7f601c00b540 comp rx=0 tx=0).stop 2024-06-09T22:31:11.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.806+0000 7f602e54d700 1 -- 172.21.15.104:0/1033207580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6028102510 msgr2=0x7f6028074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:11.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.806+0000 7f602e54d700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6028102510 0x7f6028074790 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f601800d980 tx=0x7f601800dc90 comp rx=0 tx=0).stop 2024-06-09T22:31:11.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.806+0000 7f602e54d700 1 -- 172.21.15.104:0/1033207580 shutdown_connections 2024-06-09T22:31:11.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.806+0000 7f602e54d700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f60100a2d90 0x7f60100a5250 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.806+0000 7f602e54d700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6028102510 0x7f6028074790 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.806+0000 7f602e54d700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6028108520 0x7f6028075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.806+0000 7f602e54d700 1 --2- 172.21.15.104:0/1033207580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6028102ed0 0x7f6028074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:11.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.806+0000 7f602e54d700 1 -- 172.21.15.104:0/1033207580 >> 172.21.15.104:0/1033207580 conn(0x7f6028079640 msgr2=0x7f6028101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:11.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.806+0000 7f602e54d700 1 -- 172.21.15.104:0/1033207580 shutdown_connections 2024-06-09T22:31:11.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:11.806+0000 7f602e54d700 1 -- 172.21.15.104:0/1033207580 wait complete. 2024-06-09T22:31:12.660 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-09T22:31:12.661 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:31:12.908 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:31:12.945 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:12 smithi104 ceph-mon[32389]: pgmap v317: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-06-09T22:31:13.113 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:12 smithi194 ceph-mon[36856]: pgmap v317: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-06-09T22:31:13.246 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:12 smithi153 ceph-mon[36599]: pgmap v317: 129 pgs: 129 active+clean; 248 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-06-09T22:31:14.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-06-09T22:31:14.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.114+0000 7f3148a53640 1 -- 172.21.15.104:0/3398199384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144101290 msgr2=0x7f314410d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.114+0000 7f3148a53640 1 --2- 172.21.15.104:0/3398199384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144101290 0x7f314410d750 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f313c009cd0 tx=0x7f313c02a8e0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.114+0000 7f3148a53640 1 -- 172.21.15.104:0/3398199384 shutdown_connections 2024-06-09T22:31:14.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.114+0000 7f3148a53640 1 --2- 172.21.15.104:0/3398199384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144101290 0x7f314410d750 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.114+0000 7f3148a53640 1 --2- 172.21.15.104:0/3398199384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31441008d0 0x7f3144100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.114+0000 7f3148a53640 1 --2- 172.21.15.104:0/3398199384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3144106890 0x7f3144106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.114+0000 7f3148a53640 1 -- 172.21.15.104:0/3398199384 >> 172.21.15.104:0/3398199384 conn(0x7f3144077a20 msgr2=0x7f3144077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.114+0000 7f3148a53640 1 -- 172.21.15.104:0/3398199384 shutdown_connections 2024-06-09T22:31:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.114+0000 7f3148a53640 1 -- 172.21.15.104:0/3398199384 wait complete. 2024-06-09T22:31:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3148a53640 1 Processor -- start 2024-06-09T22:31:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3148a53640 1 -- start start 2024-06-09T22:31:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3148a53640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31441008d0 0x7f314410e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3148a53640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144101290 0x7f314410e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3148a53640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3144106890 0x7f314410ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3148a53640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3144111da0 con 0x7f3144101290 2024-06-09T22:31:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3148a53640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3144111c20 con 0x7f31441008d0 2024-06-09T22:31:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3148a53640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3144111f20 con 0x7f3144106890 2024-06-09T22:31:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3141d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144101290 0x7f314410e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3141d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144101290 0x7f314410e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34356/0 (socket says 172.21.15.104:34356) 2024-06-09T22:31:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3141d9b640 1 -- 172.21.15.104:0/3649539632 learned_addr learned my addr 172.21.15.104:0/3649539632 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3142d9d640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3144106890 0x7f314410ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f314259c640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31441008d0 0x7f314410e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3141d9b640 1 -- 172.21.15.104:0/3649539632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3144106890 msgr2=0x7f314410ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3141d9b640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3144106890 0x7f314410ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3141d9b640 1 -- 172.21.15.104:0/3649539632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31441008d0 msgr2=0x7f314410e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3141d9b640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31441008d0 0x7f314410e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3141d9b640 1 -- 172.21.15.104:0/3649539632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31441a6720 con 0x7f3144101290 2024-06-09T22:31:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.115+0000 7f3141d9b640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144101290 0x7f314410e720 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f313400c970 tx=0x7f313400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.116+0000 7f31337fe640 1 -- 172.21.15.104:0/3649539632 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3134007bb0 con 0x7f3144101290 2024-06-09T22:31:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.116+0000 7f31337fe640 1 -- 172.21.15.104:0/3649539632 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3134007d50 con 0x7f3144101290 2024-06-09T22:31:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.116+0000 7f3148a53640 1 -- 172.21.15.104:0/3649539632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31441a6a10 con 0x7f3144101290 2024-06-09T22:31:14.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.116+0000 7f31337fe640 1 -- 172.21.15.104:0/3649539632 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313400f450 con 0x7f3144101290 2024-06-09T22:31:14.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.116+0000 7f3148a53640 1 -- 172.21.15.104:0/3649539632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31441a6f10 con 0x7f3144101290 2024-06-09T22:31:14.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.117+0000 7f3148a53640 1 -- 172.21.15.104:0/3649539632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f314404f760 con 0x7f3144101290 2024-06-09T22:31:14.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.118+0000 7f31337fe640 1 -- 172.21.15.104:0/3649539632 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f3134004fe0 con 0x7f3144101290 2024-06-09T22:31:14.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.119+0000 7f31337fe640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f312c0a2da0 0x7f312c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:14.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.119+0000 7f31337fe640 1 -- 172.21.15.104:0/3649539632 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7f31340cf9f0 con 0x7f3144101290 2024-06-09T22:31:14.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.119+0000 7f314259c640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f312c0a2da0 0x7f312c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:14.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.119+0000 7f314259c640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f312c0a2da0 0x7f312c0a5260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3138005fd0 tx=0x7f3138005850 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:14.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.122+0000 7f31337fe640 1 -- 172.21.15.104:0/3649539632 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3134014030 con 0x7f3144101290 2024-06-09T22:31:14.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.324+0000 7f3148a53640 1 -- 172.21.15.104:0/3649539632 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f3144061de0 con 0x7f312c0a2da0 2024-06-09T22:31:14.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.333+0000 7f31337fe640 1 -- 172.21.15.104:0/3649539632 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f3144061de0 con 0x7f312c0a2da0 2024-06-09T22:31:14.336 INFO:teuthology.orchestra.run.smithi104.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-06-09T00:00:00", "created": "2024-06-09T22:21:11", "first": "2024-06-09T22:22:00", "last": "2024-06-09T22:31:00", "last_pruned": "2024-06-09T22:31:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-06-09T22:31:14.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.335+0000 7f3148a53640 1 -- 172.21.15.104:0/3649539632 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f312c0a2da0 msgr2=0x7f312c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.335+0000 7f3148a53640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f312c0a2da0 0x7f312c0a5260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3138005fd0 tx=0x7f3138005850 comp rx=0 tx=0).stop 2024-06-09T22:31:14.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.335+0000 7f3148a53640 1 -- 172.21.15.104:0/3649539632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144101290 msgr2=0x7f314410e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.335+0000 7f3148a53640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144101290 0x7f314410e720 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f313400c970 tx=0x7f313400ce30 comp rx=0 tx=0).stop 2024-06-09T22:31:14.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.335+0000 7f3148a53640 1 -- 172.21.15.104:0/3649539632 shutdown_connections 2024-06-09T22:31:14.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.335+0000 7f3148a53640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f312c0a2da0 0x7f312c0a5260 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.335+0000 7f3148a53640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3144106890 0x7f314410ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.335+0000 7f3148a53640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144101290 0x7f314410e720 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.335+0000 7f3148a53640 1 --2- 172.21.15.104:0/3649539632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31441008d0 0x7f314410e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.335+0000 7f3148a53640 1 -- 172.21.15.104:0/3649539632 >> 172.21.15.104:0/3649539632 conn(0x7f3144077a20 msgr2=0x7f31441033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:14.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.335+0000 7f3148a53640 1 -- 172.21.15.104:0/3649539632 shutdown_connections 2024-06-09T22:31:14.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.335+0000 7f3148a53640 1 -- 172.21.15.104:0/3649539632 wait complete. 2024-06-09T22:31:14.349 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-06-09T22:31:14.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.450+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/396931220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa0102c30 msgr2=0x7f9aa0103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.450+0000 7f9aa7ff9640 1 --2- 172.21.15.104:0/396931220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa0102c30 0x7f9aa0103010 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009ca0 tx=0x7f9a9002a9e0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.450+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/396931220 shutdown_connections 2024-06-09T22:31:14.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.450+0000 7f9aa7ff9640 1 --2- 172.21.15.104:0/396931220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa010bf70 0x7f9aa010e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.450+0000 7f9aa7ff9640 1 --2- 172.21.15.104:0/396931220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9aa0103550 0x7f9aa010ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.450+0000 7f9aa7ff9640 1 --2- 172.21.15.104:0/396931220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa0102c30 0x7f9aa0103010 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.450+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/396931220 >> 172.21.15.104:0/396931220 conn(0x7f9aa0077a20 msgr2=0x7f9aa0077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:14.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.450+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/396931220 shutdown_connections 2024-06-09T22:31:14.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.450+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/396931220 wait complete. 2024-06-09T22:31:14.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.450+0000 7f9aa7ff9640 1 Processor -- start 2024-06-09T22:31:14.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.450+0000 7f9aa7ff9640 1 -- start start 2024-06-09T22:31:14.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.450+0000 7f9aa7ff9640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa0102c30 0x7f9aa01982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:14.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa7ff9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9aa0103550 0x7f9aa0198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:14.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa7ff9640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa010bf70 0x7f9aa019cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:14.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa7ff9640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa0111d70 con 0x7f9aa0102c30 2024-06-09T22:31:14.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa7ff9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa0111bf0 con 0x7f9aa0103550 2024-06-09T22:31:14.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa7ff9640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa0111ef0 con 0x7f9aa010bf70 2024-06-09T22:31:14.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa5d95640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa0102c30 0x7f9aa01982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:14.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa5d95640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa0102c30 0x7f9aa01982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34374/0 (socket says 172.21.15.104:34374) 2024-06-09T22:31:14.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa5d95640 1 -- 172.21.15.104:0/1763759235 learned_addr learned my addr 172.21.15.104:0/1763759235 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:14.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa5594640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9aa0103550 0x7f9aa0198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:14.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa5d95640 1 -- 172.21.15.104:0/1763759235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa010bf70 msgr2=0x7f9aa019cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa5d95640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa010bf70 0x7f9aa019cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa5d95640 1 -- 172.21.15.104:0/1763759235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9aa0103550 msgr2=0x7f9aa0198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa5d95640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9aa0103550 0x7f9aa0198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa5d95640 1 -- 172.21.15.104:0/1763759235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa019d270 con 0x7f9aa0102c30 2024-06-09T22:31:14.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa5594640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9aa0103550 0x7f9aa0198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:31:14.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa5d95640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa0102c30 0x7f9aa01982d0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f9a90004af0 tx=0x7f9a90003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:14.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9a9effd640 1 -- 172.21.15.104:0/1763759235 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90047070 con 0x7f9aa0102c30 2024-06-09T22:31:14.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/1763759235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa019d500 con 0x7f9aa0102c30 2024-06-09T22:31:14.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9a9effd640 1 -- 172.21.15.104:0/1763759235 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a9002e870 con 0x7f9aa0102c30 2024-06-09T22:31:14.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/1763759235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa01a4df0 con 0x7f9aa0102c30 2024-06-09T22:31:14.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.451+0000 7f9a9effd640 1 -- 172.21.15.104:0/1763759235 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90033050 con 0x7f9aa0102c30 2024-06-09T22:31:14.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.452+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/1763759235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aa004f760 con 0x7f9aa0102c30 2024-06-09T22:31:14.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.453+0000 7f9a9effd640 1 -- 172.21.15.104:0/1763759235 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f9a9004a070 con 0x7f9aa0102c30 2024-06-09T22:31:14.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.454+0000 7f9a9effd640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9a880a2da0 0x7f9a880a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:14.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.454+0000 7f9a9effd640 1 -- 172.21.15.104:0/1763759235 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7f9a900f0780 con 0x7f9aa0102c30 2024-06-09T22:31:14.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.454+0000 7f9aa5594640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9a880a2da0 0x7f9a880a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:14.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.455+0000 7f9aa5594640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9a880a2da0 0x7f9a880a5260 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9aa010ddb0 tx=0x7f9a94005e50 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:14.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.457+0000 7f9a9effd640 1 -- 172.21.15.104:0/1763759235 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9a900bcf80 con 0x7f9aa0102c30 2024-06-09T22:31:14.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.687+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/1763759235 --> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f9aa0061de0 con 0x7f9a880a2da0 2024-06-09T22:31:14.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.695+0000 7f9a9effd640 1 -- 172.21.15.104:0/1763759235 <== mgr.15279 v2:172.21.15.104:6800/493021072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f9aa0061de0 con 0x7f9a880a2da0 2024-06-09T22:31:14.697 INFO:teuthology.orchestra.run.smithi104.stdout:/ 1m 6m3h 2024-06-09T22:31:14.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.697+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/1763759235 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9a880a2da0 msgr2=0x7f9a880a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.697+0000 7f9aa7ff9640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9a880a2da0 0x7f9a880a5260 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9aa010ddb0 tx=0x7f9a94005e50 comp rx=0 tx=0).stop 2024-06-09T22:31:14.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.697+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/1763759235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa0102c30 msgr2=0x7f9aa01982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.697+0000 7f9aa7ff9640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa0102c30 0x7f9aa01982d0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f9a90004af0 tx=0x7f9a90003ce0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.697+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/1763759235 shutdown_connections 2024-06-09T22:31:14.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.697+0000 7f9aa7ff9640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f9a880a2da0 0x7f9a880a5260 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.697+0000 7f9aa7ff9640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa010bf70 0x7f9aa019cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.697+0000 7f9aa7ff9640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9aa0103550 0x7f9aa0198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.697+0000 7f9aa7ff9640 1 --2- 172.21.15.104:0/1763759235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa0102c30 0x7f9aa01982d0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.697+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/1763759235 >> 172.21.15.104:0/1763759235 conn(0x7f9aa0077a20 msgr2=0x7f9aa00fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:14.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.697+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/1763759235 shutdown_connections 2024-06-09T22:31:14.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.697+0000 7f9aa7ff9640 1 -- 172.21.15.104:0/1763759235 wait complete. 2024-06-09T22:31:14.711 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-06-09T22:31:14.722 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-06-09T22:31:14.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.805+0000 7f160ea47700 1 -- 172.21.15.104:0/4137127806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1608101db0 msgr2=0x7f160810e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.805+0000 7f160ea47700 1 --2- 172.21.15.104:0/4137127806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1608101db0 0x7f160810e240 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f160000b0a0 tx=0x7f160000b3b0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.805+0000 7f160ea47700 1 -- 172.21.15.104:0/4137127806 shutdown_connections 2024-06-09T22:31:14.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.805+0000 7f160ea47700 1 --2- 172.21.15.104:0/4137127806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1608101db0 0x7f160810e240 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.805+0000 7f160ea47700 1 --2- 172.21.15.104:0/4137127806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1608107490 0x7f1608107870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.805+0000 7f160ea47700 1 --2- 172.21.15.104:0/4137127806 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16081013f0 0x7f1608101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.805+0000 7f160ea47700 1 -- 172.21.15.104:0/4137127806 >> 172.21.15.104:0/4137127806 conn(0x7f1608077430 msgr2=0x7f1608077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:14.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.805+0000 7f160ea47700 1 -- 172.21.15.104:0/4137127806 shutdown_connections 2024-06-09T22:31:14.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.805+0000 7f160ea47700 1 -- 172.21.15.104:0/4137127806 wait complete. 2024-06-09T22:31:14.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f160ea47700 1 Processor -- start 2024-06-09T22:31:14.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f160ea47700 1 -- start start 2024-06-09T22:31:14.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f160ea47700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16081013f0 0x7f160819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:14.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f160ea47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1608101db0 0x7f160819b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:14.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f160ea47700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1608107490 0x7f160819f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:14.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f160ea47700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f16081149c0 con 0x7f1608101db0 2024-06-09T22:31:14.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f160ea47700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1608114840 con 0x7f1608107490 2024-06-09T22:31:14.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f160ea47700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1608114b40 con 0x7f16081013f0 2024-06-09T22:31:14.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f15fffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1608101db0 0x7f160819b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:14.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f15fffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1608101db0 0x7f160819b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34392/0 (socket says 172.21.15.104:34392) 2024-06-09T22:31:14.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f15fffff700 1 -- 172.21.15.104:0/2702839464 learned_addr learned my addr 172.21.15.104:0/2702839464 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:14.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f160cfe4700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1608107490 0x7f160819f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:14.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f1607fff700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16081013f0 0x7f160819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:14.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f15fffff700 1 -- 172.21.15.104:0/2702839464 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16081013f0 msgr2=0x7f160819ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f15fffff700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16081013f0 0x7f160819ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f15fffff700 1 -- 172.21.15.104:0/2702839464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1608107490 msgr2=0x7f160819f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:14.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f15fffff700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1608107490 0x7f160819f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:14.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.806+0000 7f15fffff700 1 -- 172.21.15.104:0/2702839464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f160819fda0 con 0x7f1608101db0 2024-06-09T22:31:14.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.807+0000 7f15fffff700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1608101db0 0x7f160819b2d0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f15f000c970 tx=0x7f15f000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:14.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.807+0000 7f1605ffb700 1 -- 172.21.15.104:0/2702839464 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f00078d0 con 0x7f1608101db0 2024-06-09T22:31:14.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.807+0000 7f160ea47700 1 -- 172.21.15.104:0/2702839464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16081a0090 con 0x7f1608101db0 2024-06-09T22:31:14.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.807+0000 7f1605ffb700 1 -- 172.21.15.104:0/2702839464 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15f0018480 con 0x7f1608101db0 2024-06-09T22:31:14.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.807+0000 7f1605ffb700 1 -- 172.21.15.104:0/2702839464 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f000f6b0 con 0x7f1608101db0 2024-06-09T22:31:14.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.807+0000 7f160ea47700 1 -- 172.21.15.104:0/2702839464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16081a7910 con 0x7f1608101db0 2024-06-09T22:31:14.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.808+0000 7f160ea47700 1 -- 172.21.15.104:0/2702839464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1608069180 con 0x7f1608101db0 2024-06-09T22:31:14.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.809+0000 7f1605ffb700 1 -- 172.21.15.104:0/2702839464 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f15f0018630 con 0x7f1608101db0 2024-06-09T22:31:14.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.810+0000 7f1605ffb700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f15f40a2d80 0x7f15f40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:14.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.810+0000 7f1607fff700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f15f40a2d80 0x7f15f40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:14.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.810+0000 7f1605ffb700 1 -- 172.21.15.104:0/2702839464 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7f15f00cf990 con 0x7f1608101db0 2024-06-09T22:31:14.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.810+0000 7f1607fff700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f15f40a2d80 0x7f15f40a5240 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009be0 tx=0x7f15f8009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:14.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:14.814+0000 7f1605ffb700 1 -- 172.21.15.104:0/2702839464 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f15f009c220 con 0x7f1608101db0 2024-06-09T22:31:15.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:14 smithi104 ceph-mon[32389]: pgmap v318: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-06-09T22:31:15.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:14 smithi104 ceph-mon[32389]: from='client.18006 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:31:15.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.060+0000 7f160ea47700 1 -- 172.21.15.104:0/2702839464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f160802dca0 con 0x7f1608101db0 2024-06-09T22:31:15.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.061+0000 7f1605ffb700 1 -- 172.21.15.104:0/2702839464 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7f15f009bb20 con 0x7f1608101db0 2024-06-09T22:31:15.064 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:31:15.064 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:31:15.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.063+0000 7f160ea47700 1 -- 172.21.15.104:0/2702839464 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f15f40a2d80 msgr2=0x7f15f40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.063+0000 7f160ea47700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f15f40a2d80 0x7f15f40a5240 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009be0 tx=0x7f15f8009260 comp rx=0 tx=0).stop 2024-06-09T22:31:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.063+0000 7f160ea47700 1 -- 172.21.15.104:0/2702839464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1608101db0 msgr2=0x7f160819b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.063+0000 7f160ea47700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1608101db0 0x7f160819b2d0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f15f000c970 tx=0x7f15f000cc80 comp rx=0 tx=0).stop 2024-06-09T22:31:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.063+0000 7f160ea47700 1 -- 172.21.15.104:0/2702839464 shutdown_connections 2024-06-09T22:31:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.063+0000 7f160ea47700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f15f40a2d80 0x7f15f40a5240 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.063+0000 7f160ea47700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1608107490 0x7f160819f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.063+0000 7f160ea47700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16081013f0 0x7f160819ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.063+0000 7f160ea47700 1 --2- 172.21.15.104:0/2702839464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1608101db0 0x7f160819b2d0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:15.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.063+0000 7f160ea47700 1 -- 172.21.15.104:0/2702839464 >> 172.21.15.104:0/2702839464 conn(0x7f1608077430 msgr2=0x7f16081000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.063+0000 7f160ea47700 1 -- 172.21.15.104:0/2702839464 shutdown_connections 2024-06-09T22:31:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:15.063+0000 7f160ea47700 1 -- 172.21.15.104:0/2702839464 wait complete. 2024-06-09T22:31:15.068 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:31:15.246 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:14 smithi153 ceph-mon[36599]: pgmap v318: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-06-09T22:31:15.246 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:14 smithi153 ceph-mon[36599]: from='client.18006 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:31:15.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:14 smithi194 ceph-mon[36856]: pgmap v318: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-06-09T22:31:15.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:14 smithi194 ceph-mon[36856]: from='client.18006 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:31:15.913 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-06-09T22:31:16.153 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:31:16.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:15 smithi153 ceph-mon[36599]: from='client.18012 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:31:16.246 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:15 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2702839464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:31:16.259 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:15 smithi104 ceph-mon[32389]: from='client.18012 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:31:16.259 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:15 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2702839464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:31:16.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:15 smithi194 ceph-mon[36856]: from='client.18012 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-09T22:31:16.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:15 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2702839464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:31:17.246 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:16 smithi153 ceph-mon[36599]: pgmap v319: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-06-09T22:31:17.259 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:16 smithi104 ceph-mon[32389]: pgmap v319: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-06-09T22:31:17.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:16 smithi194 ceph-mon[36856]: pgmap v319: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-06-09T22:31:18.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.116+0000 7f771362e700 1 -- 172.21.15.104:0/212921276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f770c101390 msgr2=0x7f770c101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:18.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.116+0000 7f771362e700 1 --2- 172.21.15.104:0/212921276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f770c101390 0x7f770c101810 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f7708009990 tx=0x7f7708009ca0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.117+0000 7f771362e700 1 -- 172.21.15.104:0/212921276 shutdown_connections 2024-06-09T22:31:18.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.117+0000 7f771362e700 1 --2- 172.21.15.104:0/212921276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f770c101d50 0x7f770c10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.117+0000 7f771362e700 1 --2- 172.21.15.104:0/212921276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f770c107400 0x7f770c1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.117+0000 7f771362e700 1 --2- 172.21.15.104:0/212921276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f770c101390 0x7f770c101810 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.117+0000 7f771362e700 1 -- 172.21.15.104:0/212921276 >> 172.21.15.104:0/212921276 conn(0x7f770c078340 msgr2=0x7f770c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:18.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.117+0000 7f771362e700 1 -- 172.21.15.104:0/212921276 shutdown_connections 2024-06-09T22:31:18.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.117+0000 7f771362e700 1 -- 172.21.15.104:0/212921276 wait complete. 2024-06-09T22:31:18.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.117+0000 7f771362e700 1 Processor -- start 2024-06-09T22:31:18.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.117+0000 7f771362e700 1 -- start start 2024-06-09T22:31:18.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.117+0000 7f771362e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f770c101390 0x7f770c19d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.117+0000 7f771362e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f770c101d50 0x7f770c19dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.117+0000 7f771362e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f770c107400 0x7f770c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f771362e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f770c111690 con 0x7f770c107400 2024-06-09T22:31:18.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f771362e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f770c111510 con 0x7f770c101d50 2024-06-09T22:31:18.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f771362e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f770c111810 con 0x7f770c101390 2024-06-09T22:31:18.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f7711bcb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f770c107400 0x7f770c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f7711bcb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f770c107400 0x7f770c1979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39616/0 (socket says 172.21.15.104:39616) 2024-06-09T22:31:18.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f7711bcb700 1 -- 172.21.15.104:0/3336566167 learned_addr learned my addr 172.21.15.104:0/3336566167 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:18.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f77113ca700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f770c101390 0x7f770c19d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f7710bc9700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f770c101d50 0x7f770c19dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f7711bcb700 1 -- 172.21.15.104:0/3336566167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f770c101390 msgr2=0x7f770c19d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:18.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f7711bcb700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f770c101390 0x7f770c19d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f7711bcb700 1 -- 172.21.15.104:0/3336566167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f770c101d50 msgr2=0x7f770c19dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:18.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f7711bcb700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f770c101d50 0x7f770c19dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f7711bcb700 1 -- 172.21.15.104:0/3336566167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770c198110 con 0x7f770c107400 2024-06-09T22:31:18.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f7711bcb700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f770c107400 0x7f770c1979b0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f770400ada0 tx=0x7f770400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:18.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f76fe7fc700 1 -- 172.21.15.104:0/3336566167 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704017020 con 0x7f770c107400 2024-06-09T22:31:18.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f76fe7fc700 1 -- 172.21.15.104:0/3336566167 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7704004bb0 con 0x7f770c107400 2024-06-09T22:31:18.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f76fe7fc700 1 -- 172.21.15.104:0/3336566167 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704005520 con 0x7f770c107400 2024-06-09T22:31:18.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f771362e700 1 -- 172.21.15.104:0/3336566167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770c198400 con 0x7f770c107400 2024-06-09T22:31:18.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.118+0000 7f771362e700 1 -- 172.21.15.104:0/3336566167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f770c1a4670 con 0x7f770c107400 2024-06-09T22:31:18.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.121+0000 7f76fe7fc700 1 -- 172.21.15.104:0/3336566167 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f77040056c0 con 0x7f770c107400 2024-06-09T22:31:18.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.122+0000 7f76fe7fc700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f76f80a2e10 0x7f76f80a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.122+0000 7f77113ca700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f76f80a2e10 0x7f76f80a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.122+0000 7f76fe7fc700 1 -- 172.21.15.104:0/3336566167 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f77040cefa0 con 0x7f770c107400 2024-06-09T22:31:18.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.122+0000 7f77113ca700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f76f80a2e10 0x7f76f80a52d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7700005e10 tx=0x7f7700005d40 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:18.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f771deb1b80 1 Processor -- start 2024-06-09T22:31:18.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f771deb1b80 1 -- start start 2024-06-09T22:31:18.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f771deb1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489640 0x55cf84489a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f771deb1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf844839e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f771deb1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf84484340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f771deb1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55cf843d5810 con 0x55cf84489f60 2024-06-09T22:31:18.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f771deb1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cf8437d720 con 0x55cf84483f20 2024-06-09T22:31:18.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f771deb1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cf843d17d0 con 0x55cf84489640 2024-06-09T22:31:18.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f7710bc9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf844839e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f7710bc9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf844839e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39620/0 (socket says 172.21.15.104:39620) 2024-06-09T22:31:18.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f7710bc9700 1 -- 172.21.15.104:0/3267478571 learned_addr learned my addr 172.21.15.104:0/3267478571 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:18.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f77113ca700 1 --2- 172.21.15.104:0/3267478571 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489640 0x55cf84489a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f7711bcb700 1 --2- 172.21.15.104:0/3267478571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf84484340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f7710bc9700 1 -- 172.21.15.104:0/3267478571 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489640 msgr2=0x55cf84489a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:18.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f7710bc9700 1 --2- 172.21.15.104:0/3267478571 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489640 0x55cf84489a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f7710bc9700 1 -- 172.21.15.104:0/3267478571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 msgr2=0x55cf84484340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:18.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f7710bc9700 1 --2- 172.21.15.104:0/3267478571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf84484340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f7710bc9700 1 -- 172.21.15.104:0/3267478571 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf84373f50 con 0x55cf84489f60 2024-06-09T22:31:18.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f7711bcb700 1 --2- 172.21.15.104:0/3267478571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf84484340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:31:18.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.123+0000 7f77113ca700 1 --2- 172.21.15.104:0/3267478571 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489640 0x55cf84489a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:31:18.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.124+0000 7f7710bc9700 1 --2- 172.21.15.104:0/3267478571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf844839e0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f7708006010 tx=0x7f77080096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c6bd4bec51a35d server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:18.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.124+0000 7f76f77fe700 1 -- 172.21.15.104:0/3267478571 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708027070 con 0x55cf84489f60 2024-06-09T22:31:18.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.124+0000 7f76f77fe700 1 -- 172.21.15.104:0/3267478571 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7708005550 con 0x55cf84489f60 2024-06-09T22:31:18.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.124+0000 7f76f77fe700 1 -- 172.21.15.104:0/3267478571 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770800bb00 con 0x55cf84489f60 2024-06-09T22:31:18.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.124+0000 7f771deb1b80 1 -- 172.21.15.104:0/3267478571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 msgr2=0x55cf844839e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:18.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.124+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3267478571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf844839e0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f7708006010 tx=0x7f77080096a0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 -- 172.21.15.104:0/3267478571 shutdown_connections 2024-06-09T22:31:18.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3267478571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf84484340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3267478571 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489640 0x55cf84489a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3267478571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf844839e0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 -- 172.21.15.104:0/3267478571 >> 172.21.15.104:0/3267478571 conn(0x55cf843aecc0 msgr2=0x55cf8447e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:18.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 -- 172.21.15.104:0/3267478571 shutdown_connections 2024-06-09T22:31:18.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 -- 172.21.15.104:0/3267478571 wait complete. 2024-06-09T22:31:18.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 -- start start 2024-06-09T22:31:18.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 Processor -- start 2024-06-09T22:31:18.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf8448f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489640 0x55cf8448f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf84494330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55cf84490970 con 0x55cf84489f60 2024-06-09T22:31:18.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cf84375bf0 con 0x55cf84483f20 2024-06-09T22:31:18.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cf84490af0 con 0x55cf84489640 2024-06-09T22:31:18.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.125+0000 7f771deb1b80 10 client.0 _ll_register_callbacks cb 0x55cf8436f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:31:18.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f7711bcb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf84494330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f7711bcb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf84494330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39628/0 (socket says 172.21.15.104:39628) 2024-06-09T22:31:18.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f7711bcb700 1 -- 172.21.15.104:0/3241836911 learned_addr learned my addr 172.21.15.104:0/3241836911 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:18.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f7710bc9700 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489640 0x55cf8448f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f77113ca700 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf8448f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f7711bcb700 1 -- 172.21.15.104:0/3241836911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489640 msgr2=0x55cf8448f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:18.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f7711bcb700 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489640 0x55cf8448f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f7711bcb700 1 -- 172.21.15.104:0/3241836911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 msgr2=0x55cf8448f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:18.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f7711bcb700 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf8448f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f7711bcb700 1 -- 172.21.15.104:0/3241836911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf843248d0 con 0x55cf84489f60 2024-06-09T22:31:18.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f7711bcb700 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf84494330 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f770400ad70 tx=0x7f770409b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbce1c82022eb8e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:18.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f76f4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:31:18.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f76f4ff9700 1 -- 172.21.15.104:0/3241836911 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77040b4060 con 0x55cf84489f60 2024-06-09T22:31:18.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f76f4ff9700 1 -- 172.21.15.104:0/3241836911 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f770409f170 con 0x55cf84489f60 2024-06-09T22:31:18.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f771deb1b80 1 -- 172.21.15.104:0/3241836911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7708005550 con 0x55cf84489f60 2024-06-09T22:31:18.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f76f4ff9700 1 -- 172.21.15.104:0/3241836911 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f77040a3d00 con 0x55cf84489f60 2024-06-09T22:31:18.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f771deb1b80 10 client.18030 fetch_fsmap learned FSMap version 32 2024-06-09T22:31:18.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f771deb1b80 1 -- 172.21.15.104:0/3241836911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cf8449d4d0 con 0x55cf84489f60 2024-06-09T22:31:18.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.126+0000 7f76f4ff9700 1 -- 172.21.15.104:0/3241836911 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7708005550 con 0x55cf84489f60 2024-06-09T22:31:18.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.127+0000 7f76f4ff9700 1 -- 172.21.15.104:0/3241836911 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7704012070 con 0x55cf84489f60 2024-06-09T22:31:18.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.127+0000 7f771deb1b80 10 client.18030 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:31:18.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.127+0000 7f771deb1b80 10 client.18030 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:31:18.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.127+0000 7f771deb1b80 20 client.18030 populate_metadata read hostname 'smithi104' 2024-06-09T22:31:18.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.127+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55cf8449fb10 0x55cf844a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.127+0000 7f771deb1b80 4 client.18030 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:31:18.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.127+0000 7f771deb1b80 1 -- 172.21.15.104:0/3241836911 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf843e2d10 con 0x55cf8449fb10 2024-06-09T22:31:18.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.127+0000 7f77113ca700 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55cf8449fb10 0x55cf844a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.127+0000 7f77113ca700 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55cf8449fb10 0x55cf844a1f00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b5128312d06f636b server_cookie=65958ea06538cab1 in_seq=0 out_seq=0 2024-06-09T22:31:18.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.127+0000 7f76f4ff9700 10 client.18030 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:31:18.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f76f4ff9700 1 -- 172.21.15.104:0/3241836911 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55cf843e2d10 con 0x55cf8449fb10 2024-06-09T22:31:18.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f76f4ff9700 10 client.18030 handle_command_reply: tid=0 2024-06-09T22:31:18.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f771deb1b80 1 client.18030 shutdown 2024-06-09T22:31:18.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f771deb1b80 1 -- 172.21.15.104:0/3241836911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 msgr2=0x55cf84494330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:18.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf84494330 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f770400ad70 tx=0x7f770409b950 comp rx=0 tx=0).stop 2024-06-09T22:31:18.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f771deb1b80 1 -- 172.21.15.104:0/3241836911 shutdown_connections 2024-06-09T22:31:18.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf84494330 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55cf8449fb10 0x55cf844a1f00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf8448f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3241836911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489640 0x55cf8448f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f76f4ff9700 0 client.18030 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:31:18.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f771deb1b80 1 -- 172.21.15.104:0/3241836911 >> 172.21.15.104:0/3241836911 conn(0x55cf843aecc0 msgr2=0x55cf844829d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:18.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.128+0000 7f771deb1b80 1 -- 172.21.15.104:0/3241836911 shutdown_connections 2024-06-09T22:31:18.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.129+0000 7f771deb1b80 1 -- 172.21.15.104:0/3241836911 wait complete. 2024-06-09T22:31:18.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.129+0000 7f771deb1b80 20 client.18030 trim_cache size 0 max 16384 2024-06-09T22:31:18.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.143+0000 7f771deb1b80 1 Processor -- start 2024-06-09T22:31:18.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.143+0000 7f771deb1b80 1 -- start start 2024-06-09T22:31:18.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.143+0000 7f771deb1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84483f20 0x55cf844839e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.143+0000 7f771deb1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84489640 0x55cf84494320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.143+0000 7f771deb1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf8448c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.143+0000 7f771deb1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1ac2d0 con 0x55cf84489f60 2024-06-09T22:31:18.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.143+0000 7f771deb1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1ade10 con 0x55cf84489640 2024-06-09T22:31:18.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.143+0000 7f771deb1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1ac9a0 con 0x55cf84483f20 2024-06-09T22:31:18.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.143+0000 7f7711bcb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf8448c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f7711bcb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf8448c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39640/0 (socket says 172.21.15.104:39640) 2024-06-09T22:31:18.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f7711bcb700 1 -- 172.21.15.104:0/3473522521 learned_addr learned my addr 172.21.15.104:0/3473522521 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:18.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f7710bc9700 1 --2- 172.21.15.104:0/3473522521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84489640 0x55cf84494320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f77113ca700 1 --2- 172.21.15.104:0/3473522521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84483f20 0x55cf844839e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f7711bcb700 1 -- 172.21.15.104:0/3473522521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84483f20 msgr2=0x55cf844839e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:18.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f7711bcb700 1 --2- 172.21.15.104:0/3473522521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84483f20 0x55cf844839e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f7711bcb700 1 -- 172.21.15.104:0/3473522521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84489640 msgr2=0x55cf84494320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:18.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f7711bcb700 1 --2- 172.21.15.104:0/3473522521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84489640 0x55cf84494320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f7711bcb700 1 -- 172.21.15.104:0/3473522521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf8448fc70 con 0x55cf84489f60 2024-06-09T22:31:18.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f7710bc9700 1 --2- 172.21.15.104:0/3473522521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84489640 0x55cf84494320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:31:18.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f7711bcb700 1 --2- 172.21.15.104:0/3473522521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf8448c8d0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f770400a0a0 tx=0x7f77040a38f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a2f5d342e9979 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:18.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f76f6ffd700 1 -- 172.21.15.104:0/3473522521 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77040a39c0 con 0x55cf84489f60 2024-06-09T22:31:18.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f76f6ffd700 1 -- 172.21.15.104:0/3473522521 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f770409f170 con 0x55cf84489f60 2024-06-09T22:31:18.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.144+0000 7f76f6ffd700 1 -- 172.21.15.104:0/3473522521 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704005bc0 con 0x55cf84489f60 2024-06-09T22:31:18.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.145+0000 7f771deb1b80 1 -- 172.21.15.104:0/3473522521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 msgr2=0x55cf8448c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:18.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.145+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3473522521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf8448c8d0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f770400a0a0 tx=0x7f77040a38f0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.145+0000 7f771deb1b80 1 -- 172.21.15.104:0/3473522521 shutdown_connections 2024-06-09T22:31:18.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.145+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3473522521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489f60 0x55cf8448c8d0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.145+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3473522521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84483f20 0x55cf844839e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.145+0000 7f771deb1b80 1 --2- 172.21.15.104:0/3473522521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84489640 0x55cf84494320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.145+0000 7f771deb1b80 1 -- 172.21.15.104:0/3473522521 >> 172.21.15.104:0/3473522521 conn(0x55cf843aecc0 msgr2=0x55cf8448a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:18.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.145+0000 7f771deb1b80 1 -- 172.21.15.104:0/3473522521 shutdown_connections 2024-06-09T22:31:18.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.145+0000 7f771deb1b80 1 -- 172.21.15.104:0/3473522521 wait complete. 2024-06-09T22:31:18.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.145+0000 7f771deb1b80 1 -- start start 2024-06-09T22:31:18.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.145+0000 7f771deb1b80 1 Processor -- start 2024-06-09T22:31:18.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f771deb1b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf8449bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f771deb1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489640 0x55cf844a72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f771deb1b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489f60 0x55cf844a7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f771deb1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1ad740 con 0x55cf84489640 2024-06-09T22:31:18.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f771deb1b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1ad070 con 0x55cf84483f20 2024-06-09T22:31:18.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f771deb1b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1aa290 con 0x55cf84489f60 2024-06-09T22:31:18.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f771deb1b80 10 client.0 _ll_register_callbacks cb 0x55cf8436f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:31:18.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f7710bc9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489640 0x55cf844a72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f7710bc9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489640 0x55cf844a72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39642/0 (socket says 172.21.15.104:39642) 2024-06-09T22:31:18.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f7710bc9700 1 -- 172.21.15.104:0/2155690982 learned_addr learned my addr 172.21.15.104:0/2155690982 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:18.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f77113ca700 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf8449bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f7711bcb700 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489f60 0x55cf844a7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f7710bc9700 1 -- 172.21.15.104:0/2155690982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489f60 msgr2=0x55cf844a7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:18.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f7710bc9700 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489f60 0x55cf844a7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f7710bc9700 1 -- 172.21.15.104:0/2155690982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 msgr2=0x55cf8449bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:18.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f7710bc9700 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf8449bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f7710bc9700 1 -- 172.21.15.104:0/2155690982 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf844a7dc0 con 0x55cf84489640 2024-06-09T22:31:18.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f7710bc9700 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489640 0x55cf844a72c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f77080098e0 tx=0x7f7708003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d077d4f714a15ab9 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:18.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f76f4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:31:18.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f76f4ff9700 1 -- 172.21.15.104:0/2155690982 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708030020 con 0x55cf84489640 2024-06-09T22:31:18.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f771deb1b80 1 -- 172.21.15.104:0/2155690982 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f770409f170 con 0x55cf84489640 2024-06-09T22:31:18.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.146+0000 7f76f4ff9700 1 -- 172.21.15.104:0/2155690982 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f7708005590 con 0x55cf84489640 2024-06-09T22:31:18.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.147+0000 7f76f4ff9700 1 -- 172.21.15.104:0/2155690982 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f770803b050 con 0x55cf84489640 2024-06-09T22:31:18.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.147+0000 7f771deb1b80 10 client.18036 fetch_fsmap learned FSMap version 32 2024-06-09T22:31:18.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.147+0000 7f771deb1b80 1 -- 172.21.15.104:0/2155690982 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cf8450a400 con 0x55cf84489640 2024-06-09T22:31:18.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.147+0000 7f76f4ff9700 1 -- 172.21.15.104:0/2155690982 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f770409f170 con 0x55cf84489640 2024-06-09T22:31:18.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.147+0000 7f76f4ff9700 1 -- 172.21.15.104:0/2155690982 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7708021030 con 0x55cf84489640 2024-06-09T22:31:18.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.147+0000 7f771deb1b80 10 client.18036 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:31:18.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.147+0000 7f771deb1b80 10 client.18036 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-06-09T22:31:18.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.147+0000 7f771deb1b80 20 client.18036 populate_metadata read hostname 'smithi104' 2024-06-09T22:31:18.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.147+0000 7f771deb1b80 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55cf8450a6d0 0x55cf8450cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:18.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.147+0000 7f771deb1b80 4 client.18036 mds_command: new command op to 24403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:31:18.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.147+0000 7f771deb1b80 1 -- 172.21.15.104:0/2155690982 --> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf843dd290 con 0x55cf8450a6d0 2024-06-09T22:31:18.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.147+0000 7f77113ca700 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55cf8450a6d0 0x55cf8450cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:18.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f76f4ff9700 10 client.18036 ms_handle_connect on v2:172.21.15.194:6832/3284873174 2024-06-09T22:31:18.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f77113ca700 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55cf8450a6d0 0x55cf8450cac0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b0749cbeadf05ef server_cookie=2c2200ae282decdd in_seq=0 out_seq=0 2024-06-09T22:31:18.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f76f4ff9700 1 -- 172.21.15.104:0/2155690982 <== mds.4 v2:172.21.15.194:6832/3284873174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf843dd290 con 0x55cf8450a6d0 2024-06-09T22:31:18.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f76f4ff9700 10 client.18036 handle_command_reply: tid=0 2024-06-09T22:31:18.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f771deb1b80 1 client.18036 shutdown 2024-06-09T22:31:18.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f771deb1b80 1 -- 172.21.15.104:0/2155690982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489640 msgr2=0x55cf844a72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:18.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f771deb1b80 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489640 0x55cf844a72c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f77080098e0 tx=0x7f7708003f70 comp rx=0 tx=0).stop 2024-06-09T22:31:18.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f771deb1b80 1 -- 172.21.15.104:0/2155690982 shutdown_connections 2024-06-09T22:31:18.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f771deb1b80 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.194:6832/3284873174,v1:172.21.15.194:6833/3284873174] conn(0x55cf8450a6d0 0x55cf8450cac0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f771deb1b80 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf84489f60 0x55cf844a7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f771deb1b80 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cf84483f20 0x55cf8449bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f771deb1b80 1 --2- 172.21.15.104:0/2155690982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cf84489640 0x55cf844a72c0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f771deb1b80 1 -- 172.21.15.104:0/2155690982 >> 172.21.15.104:0/2155690982 conn(0x55cf843aecc0 msgr2=0x55cf8448a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:18.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f76f4ff9700 0 client.18036 ms_handle_reset on v2:172.21.15.194:6832/3284873174 2024-06-09T22:31:18.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f771deb1b80 1 -- 172.21.15.104:0/2155690982 shutdown_connections 2024-06-09T22:31:18.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f771deb1b80 1 -- 172.21.15.104:0/2155690982 wait complete. 2024-06-09T22:31:18.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.148+0000 7f771deb1b80 20 client.18036 trim_cache size 0 max 16384 2024-06-09T22:31:18.161 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:31:18.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.149+0000 7f771362e700 1 -- 172.21.15.104:0/3336566167 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f76f80a2e10 msgr2=0x7f76f80a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:18.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.149+0000 7f771362e700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f76f80a2e10 0x7f76f80a52d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7700005e10 tx=0x7f7700005d40 comp rx=0 tx=0).stop 2024-06-09T22:31:18.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.149+0000 7f771362e700 1 -- 172.21.15.104:0/3336566167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f770c107400 msgr2=0x7f770c1979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:18.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.149+0000 7f771362e700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f770c107400 0x7f770c1979b0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f770400ada0 tx=0x7f770400c330 comp rx=0 tx=0).stop 2024-06-09T22:31:18.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.149+0000 7f771362e700 1 -- 172.21.15.104:0/3336566167 shutdown_connections 2024-06-09T22:31:18.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.149+0000 7f771362e700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f76f80a2e10 0x7f76f80a52d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.149+0000 7f771362e700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f770c107400 0x7f770c1979b0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.149+0000 7f771362e700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f770c101390 0x7f770c19d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.149+0000 7f771362e700 1 --2- 172.21.15.104:0/3336566167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f770c101d50 0x7f770c19dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:18.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.149+0000 7f771362e700 1 -- 172.21.15.104:0/3336566167 >> 172.21.15.104:0/3336566167 conn(0x7f770c078340 msgr2=0x7f770c100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:18.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.149+0000 7f771362e700 1 -- 172.21.15.104:0/3336566167 shutdown_connections 2024-06-09T22:31:18.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:18.149+0000 7f771362e700 1 -- 172.21.15.104:0/3336566167 wait complete. 2024-06-09T22:31:18.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:18 smithi153 ceph-mon[36599]: osdmap e108: 12 total, 12 up, 12 in 2024-06-09T22:31:19.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:18 smithi104 ceph-mon[32389]: osdmap e108: 12 total, 12 up, 12 in 2024-06-09T22:31:19.059 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-06-09T22:31:19.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:18 smithi194 ceph-mon[36856]: osdmap e108: 12 total, 12 up, 12 in 2024-06-09T22:31:19.300 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:31:19.699 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:19 smithi104 ceph-mon[32389]: pgmap v321: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-06-09T22:31:19.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:19 smithi153 ceph-mon[36599]: pgmap v321: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-06-09T22:31:20.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:19 smithi194 ceph-mon[36856]: pgmap v321: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-06-09T22:31:21.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.191+0000 7f5340c8f700 1 -- 172.21.15.104:0/4223722425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f533c105090 msgr2=0x7f533c1115a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:21.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.191+0000 7f5340c8f700 1 --2- 172.21.15.104:0/4223722425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f533c105090 0x7f533c1115a0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f533400b0a0 tx=0x7f533400b3b0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.191+0000 7f5340c8f700 1 -- 172.21.15.104:0/4223722425 shutdown_connections 2024-06-09T22:31:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.191+0000 7f5340c8f700 1 --2- 172.21.15.104:0/4223722425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f533c105090 0x7f533c1115a0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.191+0000 7f5340c8f700 1 --2- 172.21.15.104:0/4223722425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f533c10a6d0 0x7f533c10aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.191+0000 7f5340c8f700 1 --2- 172.21.15.104:0/4223722425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f533c1046d0 0x7f533c104b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.191+0000 7f5340c8f700 1 -- 172.21.15.104:0/4223722425 >> 172.21.15.104:0/4223722425 conn(0x7f533c1001c0 msgr2=0x7f533c1025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.191+0000 7f5340c8f700 1 -- 172.21.15.104:0/4223722425 shutdown_connections 2024-06-09T22:31:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.191+0000 7f5340c8f700 1 -- 172.21.15.104:0/4223722425 wait complete. 2024-06-09T22:31:21.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f5340c8f700 1 Processor -- start 2024-06-09T22:31:21.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f5340c8f700 1 -- start start 2024-06-09T22:31:21.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f5340c8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f533c1046d0 0x7f533c19ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f5340c8f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f533c105090 0x7f533c19b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f5340c8f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f533c10a6d0 0x7f533c19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f5340c8f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f533c114a60 con 0x7f533c1046d0 2024-06-09T22:31:21.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f5340c8f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f533c1148e0 con 0x7f533c105090 2024-06-09T22:31:21.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f5340c8f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f533c114be0 con 0x7f533c10a6d0 2024-06-09T22:31:21.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f533a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f533c1046d0 0x7f533c19ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f533a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f533c1046d0 0x7f533c19ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39664/0 (socket says 172.21.15.104:39664) 2024-06-09T22:31:21.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f533a59c700 1 -- 172.21.15.104:0/3652741288 learned_addr learned my addr 172.21.15.104:0/3652741288 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:21.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f5339d9b700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f533c105090 0x7f533c19b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f533ad9d700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f533c10a6d0 0x7f533c19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f533a59c700 1 -- 172.21.15.104:0/3652741288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f533c10a6d0 msgr2=0x7f533c19f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:21.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f533a59c700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f533c10a6d0 0x7f533c19f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f533a59c700 1 -- 172.21.15.104:0/3652741288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f533c105090 msgr2=0x7f533c19b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:21.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f533a59c700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f533c105090 0x7f533c19b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.192+0000 7f533a59c700 1 -- 172.21.15.104:0/3652741288 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533c19fd50 con 0x7f533c1046d0 2024-06-09T22:31:21.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.193+0000 7f533a59c700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f533c1046d0 0x7f533c19ad90 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f532c00c970 tx=0x7f532c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:21.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.193+0000 7f532b7fe700 1 -- 172.21.15.104:0/3652741288 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c007aa0 con 0x7f533c1046d0 2024-06-09T22:31:21.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.193+0000 7f5340c8f700 1 -- 172.21.15.104:0/3652741288 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f533c1a0040 con 0x7f533c1046d0 2024-06-09T22:31:21.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.193+0000 7f532b7fe700 1 -- 172.21.15.104:0/3652741288 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f532c004d10 con 0x7f533c1046d0 2024-06-09T22:31:21.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.193+0000 7f532b7fe700 1 -- 172.21.15.104:0/3652741288 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c005690 con 0x7f533c1046d0 2024-06-09T22:31:21.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.193+0000 7f5340c8f700 1 -- 172.21.15.104:0/3652741288 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f533c1a78c0 con 0x7f533c1046d0 2024-06-09T22:31:21.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.195+0000 7f532b7fe700 1 -- 172.21.15.104:0/3652741288 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f532c004750 con 0x7f533c1046d0 2024-06-09T22:31:21.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.196+0000 7f532b7fe700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f53240a2d90 0x7f53240a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.196+0000 7f5339d9b700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f53240a2d90 0x7f53240a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.196+0000 7f532b7fe700 1 -- 172.21.15.104:0/3652741288 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f532c0cf590 con 0x7f533c1046d0 2024-06-09T22:31:21.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.196+0000 7f5339d9b700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f53240a2d90 0x7f53240a5250 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f533c109a60 tx=0x7f5330005d50 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:21.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.197+0000 7f534b512b80 1 Processor -- start 2024-06-09T22:31:21.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.197+0000 7f534b512b80 1 -- start start 2024-06-09T22:31:21.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.197+0000 7f534b512b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5640 0x5581a83b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f534b512b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5f60 0x5581a83af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f534b512b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 0x5581a83b0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f534b512b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5581a8301810 con 0x5581a83aff20 2024-06-09T22:31:21.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f534b512b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5581a82a9720 con 0x5581a83b5f60 2024-06-09T22:31:21.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f534b512b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5581a82fd7d0 con 0x5581a83b5640 2024-06-09T22:31:21.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f533ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 0x5581a83b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f533ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 0x5581a83b0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39670/0 (socket says 172.21.15.104:39670) 2024-06-09T22:31:21.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f533ad9d700 1 -- 172.21.15.104:0/2216216057 learned_addr learned my addr 172.21.15.104:0/2216216057 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:21.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f533a59c700 1 --2- 172.21.15.104:0/2216216057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5640 0x5581a83b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f5339d9b700 1 --2- 172.21.15.104:0/2216216057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5f60 0x5581a83af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f533ad9d700 1 -- 172.21.15.104:0/2216216057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5640 msgr2=0x5581a83b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:21.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f533ad9d700 1 --2- 172.21.15.104:0/2216216057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5640 0x5581a83b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f533ad9d700 1 -- 172.21.15.104:0/2216216057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5f60 msgr2=0x5581a83af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:21.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f533ad9d700 1 --2- 172.21.15.104:0/2216216057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5f60 0x5581a83af9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f533ad9d700 1 -- 172.21.15.104:0/2216216057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581a829ff50 con 0x5581a83aff20 2024-06-09T22:31:21.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f5339d9b700 1 --2- 172.21.15.104:0/2216216057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5f60 0x5581a83af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:31:21.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f533ad9d700 1 --2- 172.21.15.104:0/2216216057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 0x5581a83b0340 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f533400b070 tx=0x7f5334009150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5faa812a67637e83 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:21.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.198+0000 7f5328ff9700 1 -- 172.21.15.104:0/2216216057 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5334021070 con 0x5581a83aff20 2024-06-09T22:31:21.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.199+0000 7f5328ff9700 1 -- 172.21.15.104:0/2216216057 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53340074e0 con 0x5581a83aff20 2024-06-09T22:31:21.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.199+0000 7f5328ff9700 1 -- 172.21.15.104:0/2216216057 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5334005280 con 0x5581a83aff20 2024-06-09T22:31:21.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.199+0000 7f534b512b80 1 -- 172.21.15.104:0/2216216057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 msgr2=0x5581a83b0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:21.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.199+0000 7f534b512b80 1 --2- 172.21.15.104:0/2216216057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 0x5581a83b0340 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f533400b070 tx=0x7f5334009150 comp rx=0 tx=0).stop 2024-06-09T22:31:21.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.199+0000 7f534b512b80 1 -- 172.21.15.104:0/2216216057 shutdown_connections 2024-06-09T22:31:21.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.199+0000 7f534b512b80 1 --2- 172.21.15.104:0/2216216057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 0x5581a83b0340 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.199+0000 7f534b512b80 1 --2- 172.21.15.104:0/2216216057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5640 0x5581a83b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.199+0000 7f534b512b80 1 --2- 172.21.15.104:0/2216216057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5f60 0x5581a83af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.199+0000 7f534b512b80 1 -- 172.21.15.104:0/2216216057 >> 172.21.15.104:0/2216216057 conn(0x5581a82dacc0 msgr2=0x5581a83aa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.199+0000 7f534b512b80 1 -- 172.21.15.104:0/2216216057 shutdown_connections 2024-06-09T22:31:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.199+0000 7f534b512b80 1 -- 172.21.15.104:0/2216216057 wait complete. 2024-06-09T22:31:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f534b512b80 1 -- start start 2024-06-09T22:31:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f534b512b80 1 Processor -- start 2024-06-09T22:31:21.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f534b512b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83aff20 0x5581a83bbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f534b512b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 0x5581a83b9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f534b512b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5f60 0x5581a83b9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f534b512b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5581a83bc970 con 0x5581a83b5640 2024-06-09T22:31:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f534b512b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5581a82a1bf0 con 0x5581a83aff20 2024-06-09T22:31:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f534b512b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5581a83bcaf0 con 0x5581a83b5f60 2024-06-09T22:31:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f534b512b80 10 client.0 _ll_register_callbacks cb 0x5581a829b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:31:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f533a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83aff20 0x5581a83bbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f533a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83aff20 0x5581a83bbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:51922/0 (socket says 172.21.15.104:51922) 2024-06-09T22:31:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f533a59c700 1 -- 172.21.15.104:0/1103922286 learned_addr learned my addr 172.21.15.104:0/1103922286 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f5339d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 0x5581a83b9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.200+0000 7f533ad9d700 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5f60 0x5581a83b9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f533a59c700 1 -- 172.21.15.104:0/1103922286 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5f60 msgr2=0x5581a83b9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f533a59c700 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5f60 0x5581a83b9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f533a59c700 1 -- 172.21.15.104:0/1103922286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 msgr2=0x5581a83b9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f533a59c700 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 0x5581a83b9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f533a59c700 1 -- 172.21.15.104:0/1103922286 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581a82508d0 con 0x5581a83aff20 2024-06-09T22:31:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f5339d9b700 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 0x5581a83b9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:31:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f533a59c700 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83aff20 0x5581a83bbd10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f532c09b470 tx=0x7f532c09b3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13d4727871fd1653 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f5321ffb700 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-06-09T22:31:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f5321ffb700 1 -- 172.21.15.104:0/1103922286 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c09b770 con 0x5581a83aff20 2024-06-09T22:31:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f5321ffb700 1 -- 172.21.15.104:0/1103922286 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f532c004a40 con 0x5581a83aff20 2024-06-09T22:31:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f534b512b80 1 -- 172.21.15.104:0/1103922286 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53340074e0 con 0x5581a83aff20 2024-06-09T22:31:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f5321ffb700 1 -- 172.21.15.104:0/1103922286 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f532c09fb20 con 0x5581a83aff20 2024-06-09T22:31:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f534b512b80 10 client.27167 fetch_fsmap learned FSMap version 32 2024-06-09T22:31:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.201+0000 7f534b512b80 1 -- 172.21.15.104:0/1103922286 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5581a83c72b0 con 0x5581a83aff20 2024-06-09T22:31:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.202+0000 7f5321ffb700 1 -- 172.21.15.104:0/1103922286 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53340074e0 con 0x5581a83aff20 2024-06-09T22:31:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.202+0000 7f5321ffb700 1 -- 172.21.15.104:0/1103922286 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f532c01f030 con 0x5581a83aff20 2024-06-09T22:31:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.202+0000 7f534b512b80 10 client.27167 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:31:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.202+0000 7f534b512b80 10 client.27167 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:31:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.202+0000 7f534b512b80 20 client.27167 populate_metadata read hostname 'smithi104' 2024-06-09T22:31:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.202+0000 7f534b512b80 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5581a83c9840 0x5581a83cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.202+0000 7f534b512b80 4 client.27167 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:31:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.202+0000 7f534b512b80 1 -- 172.21.15.104:0/1103922286 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581a830ed10 con 0x5581a83c9840 2024-06-09T22:31:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.202+0000 7f5339d9b700 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5581a83c9840 0x5581a83cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f5321ffb700 10 client.27167 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:31:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f5339d9b700 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5581a83c9840 0x5581a83cbc30 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=49848adc93604b3f server_cookie=55c795d0ed5ca000 in_seq=0 out_seq=0 2024-06-09T22:31:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f5321ffb700 1 -- 172.21.15.104:0/1103922286 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5581a830ed10 con 0x5581a83c9840 2024-06-09T22:31:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f5321ffb700 10 client.27167 handle_command_reply: tid=0 2024-06-09T22:31:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f534b512b80 1 client.27167 shutdown 2024-06-09T22:31:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f534b512b80 1 -- 172.21.15.104:0/1103922286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83aff20 msgr2=0x5581a83bbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f534b512b80 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83aff20 0x5581a83bbd10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f532c09b470 tx=0x7f532c09b3a0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f534b512b80 1 -- 172.21.15.104:0/1103922286 shutdown_connections 2024-06-09T22:31:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f534b512b80 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5f60 0x5581a83b9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f534b512b80 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83aff20 0x5581a83bbd10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f534b512b80 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5581a83c9840 0x5581a83cbc30 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f534b512b80 1 --2- 172.21.15.104:0/1103922286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 0x5581a83b9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f5321ffb700 0 client.27167 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:31:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.203+0000 7f534b512b80 1 -- 172.21.15.104:0/1103922286 >> 172.21.15.104:0/1103922286 conn(0x5581a82dacc0 msgr2=0x5581a83ae9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.204+0000 7f534b512b80 1 -- 172.21.15.104:0/1103922286 shutdown_connections 2024-06-09T22:31:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.204+0000 7f534b512b80 1 -- 172.21.15.104:0/1103922286 wait complete. 2024-06-09T22:31:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.204+0000 7f534b512b80 20 client.27167 trim_cache size 0 max 16384 2024-06-09T22:31:21.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.218+0000 7f534b512b80 1 Processor -- start 2024-06-09T22:31:21.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.218+0000 7f534b512b80 1 -- start start 2024-06-09T22:31:21.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f534b512b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 0x5581a83af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f534b512b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5640 0x5581a83b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f534b512b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5f60 0x5581a83d14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f534b512b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f533c1b0670 con 0x5581a83aff20 2024-06-09T22:31:21.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f534b512b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f533c1b21b0 con 0x5581a83b5640 2024-06-09T22:31:21.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f534b512b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f533c1b0d40 con 0x5581a83b5f60 2024-06-09T22:31:21.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f533a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 0x5581a83af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f533a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 0x5581a83af9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39688/0 (socket says 172.21.15.104:39688) 2024-06-09T22:31:21.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f533a59c700 1 -- 172.21.15.104:0/75708241 learned_addr learned my addr 172.21.15.104:0/75708241 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:21.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f5339d9b700 1 --2- 172.21.15.104:0/75708241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5640 0x5581a83b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f533a59c700 1 -- 172.21.15.104:0/75708241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5f60 msgr2=0x5581a83d14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:21.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f533a59c700 1 --2- 172.21.15.104:0/75708241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5f60 0x5581a83d14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f533a59c700 1 -- 172.21.15.104:0/75708241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5640 msgr2=0x5581a83b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:21.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f533a59c700 1 --2- 172.21.15.104:0/75708241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5640 0x5581a83b6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f533a59c700 1 -- 172.21.15.104:0/75708241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581a83ab520 con 0x5581a83aff20 2024-06-09T22:31:21.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f5339d9b700 1 --2- 172.21.15.104:0/75708241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5640 0x5581a83b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:31:21.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f533a59c700 1 --2- 172.21.15.104:0/75708241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 0x5581a83af9e0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f532c0a3e20 tx=0x7f532c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a2016bf18f0616 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:21.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f5323fff700 1 -- 172.21.15.104:0/75708241 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c065070 con 0x5581a83aff20 2024-06-09T22:31:21.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f5323fff700 1 -- 172.21.15.104:0/75708241 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f532c007d60 con 0x5581a83aff20 2024-06-09T22:31:21.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.219+0000 7f5323fff700 1 -- 172.21.15.104:0/75708241 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c09bbd0 con 0x5581a83aff20 2024-06-09T22:31:21.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.220+0000 7f534b512b80 1 -- 172.21.15.104:0/75708241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 msgr2=0x5581a83af9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:21.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.220+0000 7f534b512b80 1 --2- 172.21.15.104:0/75708241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 0x5581a83af9e0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f532c0a3e20 tx=0x7f532c005850 comp rx=0 tx=0).stop 2024-06-09T22:31:21.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.220+0000 7f534b512b80 1 -- 172.21.15.104:0/75708241 shutdown_connections 2024-06-09T22:31:21.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.220+0000 7f534b512b80 1 --2- 172.21.15.104:0/75708241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83aff20 0x5581a83af9e0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.220+0000 7f534b512b80 1 --2- 172.21.15.104:0/75708241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83b5f60 0x5581a83d14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.220+0000 7f534b512b80 1 --2- 172.21.15.104:0/75708241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5640 0x5581a83b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.220+0000 7f534b512b80 1 -- 172.21.15.104:0/75708241 >> 172.21.15.104:0/75708241 conn(0x5581a82dacc0 msgr2=0x5581a83b6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:21.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.220+0000 7f534b512b80 1 -- 172.21.15.104:0/75708241 shutdown_connections 2024-06-09T22:31:21.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.220+0000 7f534b512b80 1 -- 172.21.15.104:0/75708241 wait complete. 2024-06-09T22:31:21.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.220+0000 7f534b512b80 1 -- start start 2024-06-09T22:31:21.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.220+0000 7f534b512b80 1 Processor -- start 2024-06-09T22:31:21.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f534b512b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83aff20 0x5581a83c22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f534b512b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 0x5581a83d0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f534b512b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5f60 0x5581a83d1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f534b512b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f533c1b1ae0 con 0x5581a83b5640 2024-06-09T22:31:21.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f534b512b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f533c1b1410 con 0x5581a83b5f60 2024-06-09T22:31:21.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f534b512b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f533c1ae630 con 0x5581a83aff20 2024-06-09T22:31:21.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f534b512b80 10 client.0 _ll_register_callbacks cb 0x5581a829b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:31:21.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f5339d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 0x5581a83d0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f5339d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 0x5581a83d0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39692/0 (socket says 172.21.15.104:39692) 2024-06-09T22:31:21.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f5339d9b700 1 -- 172.21.15.104:0/692377901 learned_addr learned my addr 172.21.15.104:0/692377901 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:21.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f533a59c700 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83aff20 0x5581a83c22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f533ad9d700 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5f60 0x5581a83d1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f5339d9b700 1 -- 172.21.15.104:0/692377901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83aff20 msgr2=0x5581a83c22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:21.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f5339d9b700 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83aff20 0x5581a83c22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f5339d9b700 1 -- 172.21.15.104:0/692377901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5f60 msgr2=0x5581a83d1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:21.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f5339d9b700 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5f60 0x5581a83d1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f5339d9b700 1 -- 172.21.15.104:0/692377901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581a83d1730 con 0x5581a83b5640 2024-06-09T22:31:21.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f533ad9d700 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5f60 0x5581a83d1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:31:21.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f5339d9b700 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 0x5581a83d0c00 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f5330012040 tx=0x7f53300047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51de50ca6dd68ad server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:21.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f5321ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:31:21.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f5321ffb700 1 -- 172.21.15.104:0/692377901 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533002c070 con 0x5581a83b5640 2024-06-09T22:31:21.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f5321ffb700 1 -- 172.21.15.104:0/692377901 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f5330026520 con 0x5581a83b5640 2024-06-09T22:31:21.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.221+0000 7f534b512b80 1 -- 172.21.15.104:0/692377901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f532c007d60 con 0x5581a83b5640 2024-06-09T22:31:21.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.222+0000 7f5321ffb700 1 -- 172.21.15.104:0/692377901 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5330038050 con 0x5581a83b5640 2024-06-09T22:31:21.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.222+0000 7f534b512b80 10 client.18054 fetch_fsmap learned FSMap version 32 2024-06-09T22:31:21.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.222+0000 7f534b512b80 1 -- 172.21.15.104:0/692377901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5581a8434240 con 0x5581a83b5640 2024-06-09T22:31:21.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.222+0000 7f5321ffb700 1 -- 172.21.15.104:0/692377901 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f532c007d60 con 0x5581a83b5640 2024-06-09T22:31:21.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.222+0000 7f5321ffb700 1 -- 172.21.15.104:0/692377901 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f533000f9b0 con 0x5581a83b5640 2024-06-09T22:31:21.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.222+0000 7f534b512b80 10 client.18054 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:31:21.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.222+0000 7f534b512b80 10 client.18054 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-06-09T22:31:21.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.222+0000 7f534b512b80 20 client.18054 populate_metadata read hostname 'smithi104' 2024-06-09T22:31:21.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.222+0000 7f534b512b80 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5581a8434980 0x5581a8436da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:21.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.222+0000 7f534b512b80 4 client.18054 mds_command: new command op to 24419 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:31:21.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.222+0000 7f534b512b80 1 -- 172.21.15.104:0/692377901 --> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5581a8309290 con 0x5581a8434980 2024-06-09T22:31:21.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.222+0000 7f533a59c700 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5581a8434980 0x5581a8436da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:21.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f5321ffb700 10 client.18054 ms_handle_connect on v2:172.21.15.153:6838/779682823 2024-06-09T22:31:21.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f533a59c700 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5581a8434980 0x5581a8436da0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2f3612b55134fcb1 server_cookie=c97d8b60fca04c35 in_seq=0 out_seq=0 2024-06-09T22:31:21.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f5321ffb700 1 -- 172.21.15.104:0/692377901 <== mds.3 v2:172.21.15.153:6838/779682823 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5581a8309290 con 0x5581a8434980 2024-06-09T22:31:21.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f5321ffb700 10 client.18054 handle_command_reply: tid=0 2024-06-09T22:31:21.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f534b512b80 1 client.18054 shutdown 2024-06-09T22:31:21.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f534b512b80 1 -- 172.21.15.104:0/692377901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 msgr2=0x5581a83d0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:21.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f534b512b80 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 0x5581a83d0c00 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f5330012040 tx=0x7f53300047c0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f534b512b80 1 -- 172.21.15.104:0/692377901 shutdown_connections 2024-06-09T22:31:21.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f534b512b80 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581a83b5f60 0x5581a83d1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f534b512b80 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5581a83aff20 0x5581a83c22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f534b512b80 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.153:6838/779682823,v1:172.21.15.153:6839/779682823] conn(0x5581a8434980 0x5581a8436da0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f534b512b80 1 --2- 172.21.15.104:0/692377901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5581a83b5640 0x5581a83d0c00 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f534b512b80 1 -- 172.21.15.104:0/692377901 >> 172.21.15.104:0/692377901 conn(0x5581a82dacc0 msgr2=0x5581a83b6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:21.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f5321ffb700 0 client.18054 ms_handle_reset on v2:172.21.15.153:6838/779682823 2024-06-09T22:31:21.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f534b512b80 1 -- 172.21.15.104:0/692377901 shutdown_connections 2024-06-09T22:31:21.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f534b512b80 1 -- 172.21.15.104:0/692377901 wait complete. 2024-06-09T22:31:21.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.223+0000 7f534b512b80 20 client.18054 trim_cache size 0 max 16384 2024-06-09T22:31:21.237 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:31:21.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.224+0000 7f5340c8f700 1 -- 172.21.15.104:0/3652741288 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f53240a2d90 msgr2=0x7f53240a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:21.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.224+0000 7f5340c8f700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f53240a2d90 0x7f53240a5250 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f533c109a60 tx=0x7f5330005d50 comp rx=0 tx=0).stop 2024-06-09T22:31:21.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.224+0000 7f5340c8f700 1 -- 172.21.15.104:0/3652741288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f533c1046d0 msgr2=0x7f533c19ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:21.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.224+0000 7f5340c8f700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f533c1046d0 0x7f533c19ad90 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f532c00c970 tx=0x7f532c00cd30 comp rx=0 tx=0).stop 2024-06-09T22:31:21.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.224+0000 7f5340c8f700 1 -- 172.21.15.104:0/3652741288 shutdown_connections 2024-06-09T22:31:21.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.224+0000 7f5340c8f700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f53240a2d90 0x7f53240a5250 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.224+0000 7f5340c8f700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f533c1046d0 0x7f533c19ad90 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.224+0000 7f5340c8f700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f533c10a6d0 0x7f533c19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.224+0000 7f5340c8f700 1 --2- 172.21.15.104:0/3652741288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f533c105090 0x7f533c19b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:21.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.224+0000 7f5340c8f700 1 -- 172.21.15.104:0/3652741288 >> 172.21.15.104:0/3652741288 conn(0x7f533c1001c0 msgr2=0x7f533c100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:21.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.224+0000 7f5340c8f700 1 -- 172.21.15.104:0/3652741288 shutdown_connections 2024-06-09T22:31:21.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:21.224+0000 7f5340c8f700 1 -- 172.21.15.104:0/3652741288 wait complete. 2024-06-09T22:31:21.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:21 smithi153 ceph-mon[36599]: pgmap v322: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 206 B/s wr, 13 op/s 2024-06-09T22:31:21.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:21 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:31:22.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:21 smithi104 ceph-mon[32389]: pgmap v322: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 206 B/s wr, 13 op/s 2024-06-09T22:31:22.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:21 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:31:22.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:21 smithi194 ceph-mon[36856]: pgmap v322: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 206 B/s wr, 13 op/s 2024-06-09T22:31:22.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:21 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:31:22.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-09T22:31:22.334 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:31:22.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:22 smithi153 ceph-mon[36599]: pgmap v323: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-06-09T22:31:22.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:22 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:22.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:22 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:22.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:22 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:22.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:22 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:22.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:22 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:22.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:22 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:23.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:22 smithi104 ceph-mon[32389]: pgmap v323: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-06-09T22:31:23.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:22 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:23.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:22 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:23.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:22 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:23.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:22 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:23.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:22 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:23.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:22 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:23.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:22 smithi194 ceph-mon[36856]: pgmap v323: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-06-09T22:31:23.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:22 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:23.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:22 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:23.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:22 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:23.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:22 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:23.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:22 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:23.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:22 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.190+0000 7f7f46af8700 1 -- 172.21.15.104:0/62284136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f401013c0 msgr2=0x7f7f40101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:24.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.190+0000 7f7f46af8700 1 --2- 172.21.15.104:0/62284136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f401013c0 0x7f7f40101840 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009990 tx=0x7f7f30009ca0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.190+0000 7f7f46af8700 1 -- 172.21.15.104:0/62284136 shutdown_connections 2024-06-09T22:31:24.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.190+0000 7f7f46af8700 1 --2- 172.21.15.104:0/62284136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f40101d80 0x7f7f4010e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.190+0000 7f7f46af8700 1 --2- 172.21.15.104:0/62284136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f40107430 0x7f7f40107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.190+0000 7f7f46af8700 1 --2- 172.21.15.104:0/62284136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f401013c0 0x7f7f40101840 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.190+0000 7f7f46af8700 1 -- 172.21.15.104:0/62284136 >> 172.21.15.104:0/62284136 conn(0x7f7f400783d0 msgr2=0x7f7f400787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:24.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.190+0000 7f7f46af8700 1 -- 172.21.15.104:0/62284136 shutdown_connections 2024-06-09T22:31:24.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.190+0000 7f7f46af8700 1 -- 172.21.15.104:0/62284136 wait complete. 2024-06-09T22:31:24.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.191+0000 7f7f46af8700 1 Processor -- start 2024-06-09T22:31:24.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.191+0000 7f7f46af8700 1 -- start start 2024-06-09T22:31:24.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.191+0000 7f7f46af8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f401013c0 0x7f7f4019d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.191+0000 7f7f46af8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f40101d80 0x7f7f4019dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.191+0000 7f7f46af8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f40107430 0x7f7f40197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.191+0000 7f7f46af8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7f40111720 con 0x7f7f40101d80 2024-06-09T22:31:24.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.191+0000 7f7f46af8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f401115a0 con 0x7f7f401013c0 2024-06-09T22:31:24.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.191+0000 7f7f46af8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f401118a0 con 0x7f7f40107430 2024-06-09T22:31:24.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.191+0000 7f7f37fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f40101d80 0x7f7f4019dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.191+0000 7f7f37fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f40101d80 0x7f7f4019dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39710/0 (socket says 172.21.15.104:39710) 2024-06-09T22:31:24.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.191+0000 7f7f44894700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f401013c0 0x7f7f4019d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.191+0000 7f7f37fff700 1 -- 172.21.15.104:0/4068008229 learned_addr learned my addr 172.21.15.104:0/4068008229 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:24.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f45095700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f40107430 0x7f7f40197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f37fff700 1 -- 172.21.15.104:0/4068008229 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f40107430 msgr2=0x7f7f40197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:24.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f37fff700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f40107430 0x7f7f40197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f37fff700 1 -- 172.21.15.104:0/4068008229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f401013c0 msgr2=0x7f7f4019d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:24.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f37fff700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f401013c0 0x7f7f4019d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f37fff700 1 -- 172.21.15.104:0/4068008229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f401980a0 con 0x7f7f40101d80 2024-06-09T22:31:24.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f44894700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f401013c0 0x7f7f4019d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:31:24.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f37fff700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f40101d80 0x7f7f4019dcc0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f7f3000bff0 tx=0x7f7f30005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:24.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f35ffb700 1 -- 172.21.15.104:0/4068008229 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f30027070 con 0x7f7f40101d80 2024-06-09T22:31:24.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f35ffb700 1 -- 172.21.15.104:0/4068008229 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f30004050 con 0x7f7f40101d80 2024-06-09T22:31:24.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f46af8700 1 -- 172.21.15.104:0/4068008229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f40198330 con 0x7f7f40101d80 2024-06-09T22:31:24.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f35ffb700 1 -- 172.21.15.104:0/4068008229 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f30022d00 con 0x7f7f40101d80 2024-06-09T22:31:24.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.192+0000 7f7f46af8700 1 -- 172.21.15.104:0/4068008229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f4006ab60 con 0x7f7f40101d80 2024-06-09T22:31:24.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.195+0000 7f7f35ffb700 1 -- 172.21.15.104:0/4068008229 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7f3002a070 con 0x7f7f40101d80 2024-06-09T22:31:24.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.196+0000 7f7f35ffb700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f280a2e10 0x7f7f280a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.196+0000 7f7f44894700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f280a2e10 0x7f7f280a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.196+0000 7f7f35ffb700 1 -- 172.21.15.104:0/4068008229 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f7f300d0640 con 0x7f7f40101d80 2024-06-09T22:31:24.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.196+0000 7f7f44894700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f280a2e10 0x7f7f280a52d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7f40198a80 tx=0x7f7f3c009190 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:24.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.196+0000 7f7f5137bb80 1 Processor -- start 2024-06-09T22:31:24.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f5137bb80 1 -- start start 2024-06-09T22:31:24.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f5137bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd8f60 0x559944cd9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f5137bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 0x559944cd3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f5137bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cd3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f5137bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559944bc5f60 con 0x559944cd9880 2024-06-09T22:31:24.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f5137bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559944c24850 con 0x559944cd37a0 2024-06-09T22:31:24.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f5137bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559944b8c460 con 0x559944cd8f60 2024-06-09T22:31:24.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f37fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 0x559944cd3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f37fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 0x559944cd3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39714/0 (socket says 172.21.15.104:39714) 2024-06-09T22:31:24.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f37fff700 1 -- 172.21.15.104:0/1168407540 learned_addr learned my addr 172.21.15.104:0/1168407540 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:24.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f45095700 1 --2- 172.21.15.104:0/1168407540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cd3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f44894700 1 --2- 172.21.15.104:0/1168407540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd8f60 0x559944cd9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f37fff700 1 -- 172.21.15.104:0/1168407540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd8f60 msgr2=0x559944cd9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:24.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f37fff700 1 --2- 172.21.15.104:0/1168407540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd8f60 0x559944cd9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f37fff700 1 -- 172.21.15.104:0/1168407540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 msgr2=0x559944cd3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:24.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f37fff700 1 --2- 172.21.15.104:0/1168407540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cd3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f37fff700 1 -- 172.21.15.104:0/1168407540 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559944b738d0 con 0x559944cd9880 2024-06-09T22:31:24.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f37fff700 1 --2- 172.21.15.104:0/1168407540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 0x559944cd3260 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f7f300991f0 tx=0x7f7f3000b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a29fd75a4d6a0204 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:24.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.197+0000 7f7f26ffd700 1 -- 172.21.15.104:0/1168407540 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3009cf20 con 0x559944cd9880 2024-06-09T22:31:24.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.198+0000 7f7f26ffd700 1 -- 172.21.15.104:0/1168407540 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3009c820 con 0x559944cd9880 2024-06-09T22:31:24.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.198+0000 7f7f26ffd700 1 -- 172.21.15.104:0/1168407540 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f300d0250 con 0x559944cd9880 2024-06-09T22:31:24.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.198+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1168407540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 msgr2=0x559944cd3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:24.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.198+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/1168407540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 0x559944cd3260 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f7f300991f0 tx=0x7f7f3000b540 comp rx=0 tx=0).stop 2024-06-09T22:31:24.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.198+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1168407540 shutdown_connections 2024-06-09T22:31:24.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.198+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/1168407540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cd3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.198+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/1168407540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd8f60 0x559944cd9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.198+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/1168407540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 0x559944cd3260 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.198+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1168407540 >> 172.21.15.104:0/1168407540 conn(0x559944c00080 msgr2=0x559944cce220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:24.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.198+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1168407540 shutdown_connections 2024-06-09T22:31:24.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.198+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1168407540 wait complete. 2024-06-09T22:31:24.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f5137bb80 1 -- start start 2024-06-09T22:31:24.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f5137bb80 1 Processor -- start 2024-06-09T22:31:24.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f5137bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cdf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f5137bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd8f60 0x559944cdc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f5137bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 0x559944cdceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f5137bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x559944bc4fc0 con 0x559944cd9880 2024-06-09T22:31:24.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f5137bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559944c20810 con 0x559944cd37a0 2024-06-09T22:31:24.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f5137bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559944ce0010 con 0x559944cd8f60 2024-06-09T22:31:24.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f5137bb80 10 client.0 _ll_register_callbacks cb 0x559944bba5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:31:24.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f45095700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 0x559944cdceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f44894700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cdf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f45095700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 0x559944cdceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39718/0 (socket says 172.21.15.104:39718) 2024-06-09T22:31:24.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f44894700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cdf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:51984/0 (socket says 172.21.15.104:51984) 2024-06-09T22:31:24.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f45095700 1 -- 172.21.15.104:0/345214343 learned_addr learned my addr 172.21.15.104:0/345214343 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:24.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.199+0000 7f7f37fff700 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd8f60 0x559944cdc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f45095700 1 -- 172.21.15.104:0/345214343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd8f60 msgr2=0x559944cdc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:24.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f45095700 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd8f60 0x559944cdc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f45095700 1 -- 172.21.15.104:0/345214343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 msgr2=0x559944cdf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:24.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f45095700 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cdf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f45095700 1 -- 172.21.15.104:0/345214343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559944bea750 con 0x559944cd9880 2024-06-09T22:31:24.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f44894700 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cdf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:31:24.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f45095700 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 0x559944cdceb0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f7f38012db0 tx=0x7f7f38010520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed3b9f2615d981e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:24.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f1ffff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:31:24.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f1ffff700 1 -- 172.21.15.104:0/345214343 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3801a070 con 0x559944cd9880 2024-06-09T22:31:24.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f1ffff700 1 -- 172.21.15.104:0/345214343 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f7f38010c90 con 0x559944cd9880 2024-06-09T22:31:24.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f5137bb80 1 -- 172.21.15.104:0/345214343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f3009c820 con 0x559944cd9880 2024-06-09T22:31:24.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f1ffff700 1 -- 172.21.15.104:0/345214343 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7f38024050 con 0x559944cd9880 2024-06-09T22:31:24.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f5137bb80 10 client.18066 fetch_fsmap learned FSMap version 32 2024-06-09T22:31:24.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f5137bb80 1 -- 172.21.15.104:0/345214343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559944cea820 con 0x559944cd9880 2024-06-09T22:31:24.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.200+0000 7f7f1ffff700 1 -- 172.21.15.104:0/345214343 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3009c820 con 0x559944cd9880 2024-06-09T22:31:24.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.201+0000 7f7f1ffff700 1 -- 172.21.15.104:0/345214343 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7f380143e0 con 0x559944cd9880 2024-06-09T22:31:24.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.201+0000 7f7f5137bb80 10 client.18066 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:31:24.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.201+0000 7f7f5137bb80 10 client.18066 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:31:24.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.201+0000 7f7f5137bb80 20 client.18066 populate_metadata read hostname 'smithi104' 2024-06-09T22:31:24.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.201+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x559944cece60 0x559944cef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.201+0000 7f7f5137bb80 4 client.18066 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:31:24.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.201+0000 7f7f5137bb80 1 -- 172.21.15.104:0/345214343 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559944c31d50 con 0x559944cece60 2024-06-09T22:31:24.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.201+0000 7f7f44894700 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x559944cece60 0x559944cef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.201+0000 7f7f44894700 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x559944cece60 0x559944cef2b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1586c2c9aea20420 server_cookie=2e2afef0d8e3b752 in_seq=0 out_seq=0 2024-06-09T22:31:24.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.201+0000 7f7f1ffff700 10 client.18066 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:31:24.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f1ffff700 1 -- 172.21.15.104:0/345214343 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x559944c31d50 con 0x559944cece60 2024-06-09T22:31:24.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f1ffff700 10 client.18066 handle_command_reply: tid=0 2024-06-09T22:31:24.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f5137bb80 1 client.18066 shutdown 2024-06-09T22:31:24.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f5137bb80 1 -- 172.21.15.104:0/345214343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 msgr2=0x559944cdceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:24.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 0x559944cdceb0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f7f38012db0 tx=0x7f7f38010520 comp rx=0 tx=0).stop 2024-06-09T22:31:24.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f5137bb80 1 -- 172.21.15.104:0/345214343 shutdown_connections 2024-06-09T22:31:24.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x559944cece60 0x559944cef2b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd9880 0x559944cdceb0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cdf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f1ffff700 0 client.18066 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:31:24.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/345214343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd8f60 0x559944cdc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f5137bb80 1 -- 172.21.15.104:0/345214343 >> 172.21.15.104:0/345214343 conn(0x559944c00080 msgr2=0x559944ccd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:24.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f5137bb80 1 -- 172.21.15.104:0/345214343 shutdown_connections 2024-06-09T22:31:24.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f5137bb80 1 -- 172.21.15.104:0/345214343 wait complete. 2024-06-09T22:31:24.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.202+0000 7f7f5137bb80 20 client.18066 trim_cache size 0 max 16384 2024-06-09T22:31:24.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.217+0000 7f7f5137bb80 1 Processor -- start 2024-06-09T22:31:24.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f5137bb80 1 -- start start 2024-06-09T22:31:24.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f5137bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cd3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f5137bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 0x559944cdf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f5137bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd9880 0x559944ccde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f5137bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7f401b0750 con 0x559944cd8f60 2024-06-09T22:31:24.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f5137bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f401b2290 con 0x559944cd37a0 2024-06-09T22:31:24.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f5137bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f401b0e20 con 0x559944cd9880 2024-06-09T22:31:24.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f37fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 0x559944cdf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f37fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 0x559944cdf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39720/0 (socket says 172.21.15.104:39720) 2024-06-09T22:31:24.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f37fff700 1 -- 172.21.15.104:0/2294401195 learned_addr learned my addr 172.21.15.104:0/2294401195 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:24.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f45095700 1 --2- 172.21.15.104:0/2294401195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd9880 0x559944ccde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f44894700 1 --2- 172.21.15.104:0/2294401195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cd3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f37fff700 1 -- 172.21.15.104:0/2294401195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd9880 msgr2=0x559944ccde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:24.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f37fff700 1 --2- 172.21.15.104:0/2294401195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd9880 0x559944ccde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f37fff700 1 -- 172.21.15.104:0/2294401195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 msgr2=0x559944cd3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:24.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f37fff700 1 --2- 172.21.15.104:0/2294401195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cd3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f37fff700 1 -- 172.21.15.104:0/2294401195 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559944cca810 con 0x559944cd8f60 2024-06-09T22:31:24.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f45095700 1 --2- 172.21.15.104:0/2294401195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd9880 0x559944ccde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:31:24.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f44894700 1 --2- 172.21.15.104:0/2294401195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cd3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:31:24.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f37fff700 1 --2- 172.21.15.104:0/2294401195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 0x559944cdf1b0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f7f30022750 tx=0x7f7f3001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9911fbc7f30c885e server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:24.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f267fc700 1 -- 172.21.15.104:0/2294401195 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3009cf20 con 0x559944cd8f60 2024-06-09T22:31:24.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f267fc700 1 -- 172.21.15.104:0/2294401195 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f300b4070 con 0x559944cd8f60 2024-06-09T22:31:24.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.218+0000 7f7f267fc700 1 -- 172.21.15.104:0/2294401195 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f30098e90 con 0x559944cd8f60 2024-06-09T22:31:24.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.219+0000 7f7f5137bb80 1 -- 172.21.15.104:0/2294401195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 msgr2=0x559944cdf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:24.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.219+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/2294401195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 0x559944cdf1b0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f7f30022750 tx=0x7f7f3001a040 comp rx=0 tx=0).stop 2024-06-09T22:31:24.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.219+0000 7f7f5137bb80 1 -- 172.21.15.104:0/2294401195 shutdown_connections 2024-06-09T22:31:24.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.219+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/2294401195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd9880 0x559944ccde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.219+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/2294401195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944cd3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.219+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/2294401195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 0x559944cdf1b0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.219+0000 7f7f5137bb80 1 -- 172.21.15.104:0/2294401195 >> 172.21.15.104:0/2294401195 conn(0x559944c00080 msgr2=0x559944ce80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:24.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.219+0000 7f7f5137bb80 1 -- 172.21.15.104:0/2294401195 shutdown_connections 2024-06-09T22:31:24.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.219+0000 7f7f5137bb80 1 -- 172.21.15.104:0/2294401195 wait complete. 2024-06-09T22:31:24.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.219+0000 7f7f5137bb80 1 -- start start 2024-06-09T22:31:24.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.219+0000 7f7f5137bb80 1 Processor -- start 2024-06-09T22:31:24.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f5137bb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944ce5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f5137bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 0x559944cf43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f5137bb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd9880 0x559944cf4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f5137bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7f401b1bc0 con 0x559944cd8f60 2024-06-09T22:31:24.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f5137bb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f401b14f0 con 0x559944cd37a0 2024-06-09T22:31:24.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f5137bb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f401ae730 con 0x559944cd9880 2024-06-09T22:31:24.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f5137bb80 10 client.0 _ll_register_callbacks cb 0x559944bba5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:31:24.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f37fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 0x559944cf43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f37fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 0x559944cf43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39734/0 (socket says 172.21.15.104:39734) 2024-06-09T22:31:24.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f37fff700 1 -- 172.21.15.104:0/1463367552 learned_addr learned my addr 172.21.15.104:0/1463367552 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:24.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f45095700 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd9880 0x559944cf4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f44894700 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944ce5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f37fff700 1 -- 172.21.15.104:0/1463367552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd9880 msgr2=0x559944cf4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:24.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f37fff700 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd9880 0x559944cf4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f37fff700 1 -- 172.21.15.104:0/1463367552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 msgr2=0x559944ce5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:24.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f37fff700 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944ce5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f37fff700 1 -- 172.21.15.104:0/1463367552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559944cf5030 con 0x559944cd8f60 2024-06-09T22:31:24.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f37fff700 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 0x559944cf43f0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f7f300a4ed0 tx=0x7f7f30009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a576b3b993765f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:24.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.220+0000 7f7f1ffff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:31:24.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f1ffff700 1 -- 172.21.15.104:0/1463367552 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3007f020 con 0x559944cd8f60 2024-06-09T22:31:24.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f1ffff700 1 -- 172.21.15.104:0/1463367552 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f7f300858d0 con 0x559944cd8f60 2024-06-09T22:31:24.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1463367552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f300b4070 con 0x559944cd8f60 2024-06-09T22:31:24.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f1ffff700 1 -- 172.21.15.104:0/1463367552 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7f300aa050 con 0x559944cd8f60 2024-06-09T22:31:24.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f5137bb80 10 client.18072 fetch_fsmap learned FSMap version 32 2024-06-09T22:31:24.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1463367552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559944d579f0 con 0x559944cd8f60 2024-06-09T22:31:24.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f1ffff700 1 -- 172.21.15.104:0/1463367552 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f300b4070 con 0x559944cd8f60 2024-06-09T22:31:24.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f1ffff700 1 -- 172.21.15.104:0/1463367552 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7f3009ca80 con 0x559944cd8f60 2024-06-09T22:31:24.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f5137bb80 10 client.18072 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:31:24.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f5137bb80 10 client.18072 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-06-09T22:31:24.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f5137bb80 20 client.18072 populate_metadata read hostname 'smithi104' 2024-06-09T22:31:24.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x559944d57c20 0x559944d5a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:24.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f5137bb80 4 client.18072 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:31:24.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1463367552 --> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559944c2c2d0 con 0x559944d57c20 2024-06-09T22:31:24.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.221+0000 7f7f44894700 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x559944d57c20 0x559944d5a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:24.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f1ffff700 10 client.18072 ms_handle_connect on v2:172.21.15.194:6834/2995819396 2024-06-09T22:31:24.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f44894700 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x559944d57c20 0x559944d5a010 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=920a17a56106647 server_cookie=1aff3af9e8f5579c in_seq=0 out_seq=0 2024-06-09T22:31:24.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f1ffff700 1 -- 172.21.15.104:0/1463367552 <== mds.2 v2:172.21.15.194:6834/2995819396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559944c2c2d0 con 0x559944d57c20 2024-06-09T22:31:24.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f1ffff700 10 client.18072 handle_command_reply: tid=0 2024-06-09T22:31:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f5137bb80 1 client.18072 shutdown 2024-06-09T22:31:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1463367552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 msgr2=0x559944cf43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 0x559944cf43f0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f7f300a4ed0 tx=0x7f7f30009ca0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1463367552 shutdown_connections 2024-06-09T22:31:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.194:6834/2995819396,v1:172.21.15.194:6835/2995819396] conn(0x559944d57c20 0x559944d5a010 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559944cd9880 0x559944cf4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559944cd37a0 0x559944ce5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f5137bb80 1 --2- 172.21.15.104:0/1463367552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x559944cd8f60 0x559944cf43f0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1463367552 >> 172.21.15.104:0/1463367552 conn(0x559944c00080 msgr2=0x559944cd2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f1ffff700 0 client.18072 ms_handle_reset on v2:172.21.15.194:6834/2995819396 2024-06-09T22:31:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1463367552 shutdown_connections 2024-06-09T22:31:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f5137bb80 1 -- 172.21.15.104:0/1463367552 wait complete. 2024-06-09T22:31:24.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.222+0000 7f7f5137bb80 20 client.18072 trim_cache size 0 max 16384 2024-06-09T22:31:24.238 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:31:24.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.223+0000 7f7f46af8700 1 -- 172.21.15.104:0/4068008229 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f280a2e10 msgr2=0x7f7f280a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:24.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.223+0000 7f7f46af8700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f280a2e10 0x7f7f280a52d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7f40198a80 tx=0x7f7f3c009190 comp rx=0 tx=0).stop 2024-06-09T22:31:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.223+0000 7f7f46af8700 1 -- 172.21.15.104:0/4068008229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f40101d80 msgr2=0x7f7f4019dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.223+0000 7f7f46af8700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f40101d80 0x7f7f4019dcc0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f7f3000bff0 tx=0x7f7f30005590 comp rx=0 tx=0).stop 2024-06-09T22:31:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.223+0000 7f7f46af8700 1 -- 172.21.15.104:0/4068008229 shutdown_connections 2024-06-09T22:31:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.223+0000 7f7f46af8700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f7f280a2e10 0x7f7f280a52d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.223+0000 7f7f46af8700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f40107430 0x7f7f40197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.223+0000 7f7f46af8700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f401013c0 0x7f7f4019d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.223+0000 7f7f46af8700 1 --2- 172.21.15.104:0/4068008229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f40101d80 0x7f7f4019dcc0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:24.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.223+0000 7f7f46af8700 1 -- 172.21.15.104:0/4068008229 >> 172.21.15.104:0/4068008229 conn(0x7f7f400783d0 msgr2=0x7f7f400ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:24.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.223+0000 7f7f46af8700 1 -- 172.21.15.104:0/4068008229 shutdown_connections 2024-06-09T22:31:24.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:24.223+0000 7f7f46af8700 1 -- 172.21.15.104:0/4068008229 wait complete. 2024-06-09T22:31:24.733 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:24 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.734 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:24 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.734 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:24 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.734 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:24 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.734 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:24 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.734 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:24 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.734 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:24 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.735 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:24 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.735 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:24 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.735 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:24 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.735 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:24 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.735 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:24 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:24 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:24 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:24 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:24 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:24 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:24 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:24 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:24 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:24 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:24 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:24 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:24 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:24 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:24 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:24 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:24 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:24.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:24 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:24 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:24 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:24 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:24 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:24 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:24 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:24.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:24 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:25.731 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:25 smithi104 ceph-mon[32389]: pgmap v324: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-06-09T22:31:25.732 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:25 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-06-09T22:31:25.732 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:25 smithi104 ceph-mon[32389]: Unable to set osd_memory_target on smithi194 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-06-09T22:31:25.732 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:25 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-06-09T22:31:25.732 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:25 smithi104 ceph-mon[32389]: Unable to set osd_memory_target on smithi153 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-06-09T22:31:25.761 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-09T22:31:25.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:25 smithi194 ceph-mon[36856]: pgmap v324: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-06-09T22:31:25.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:25 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-06-09T22:31:25.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:25 smithi194 ceph-mon[36856]: Unable to set osd_memory_target on smithi194 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-06-09T22:31:25.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:25 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-06-09T22:31:25.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:25 smithi194 ceph-mon[36856]: Unable to set osd_memory_target on smithi153 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-06-09T22:31:25.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:25 smithi153 ceph-mon[36599]: pgmap v324: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-06-09T22:31:25.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:25 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-06-09T22:31:25.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:25 smithi153 ceph-mon[36599]: Unable to set osd_memory_target on smithi194 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-06-09T22:31:25.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:25 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-06-09T22:31:25.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:25 smithi153 ceph-mon[36599]: Unable to set osd_memory_target on smithi153 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-06-09T22:31:26.019 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:31:27.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:27 smithi194 ceph-mon[36856]: pgmap v325: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-06-09T22:31:27.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:27 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:27.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:27 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:27.836 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:27 smithi104 ceph-mon[32389]: pgmap v325: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-06-09T22:31:27.837 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:27 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:27.837 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:27 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:27.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.841+0000 7f977cc66700 1 -- 172.21.15.104:0/830798727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778101d50 msgr2=0x7f977810e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:27.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.841+0000 7f977cc66700 1 --2- 172.21.15.104:0/830798727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778101d50 0x7f977810e240 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f977000b0a0 tx=0x7f977000b3b0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.841+0000 7f977cc66700 1 -- 172.21.15.104:0/830798727 shutdown_connections 2024-06-09T22:31:27.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.841+0000 7f977cc66700 1 --2- 172.21.15.104:0/830798727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778101d50 0x7f977810e240 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.841+0000 7f977cc66700 1 --2- 172.21.15.104:0/830798727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9778107400 0x7f97781077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.841+0000 7f977cc66700 1 --2- 172.21.15.104:0/830798727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9778101390 0x7f9778101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.841+0000 7f977cc66700 1 -- 172.21.15.104:0/830798727 >> 172.21.15.104:0/830798727 conn(0x7f9778078340 msgr2=0x7f9778078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.841+0000 7f977cc66700 1 -- 172.21.15.104:0/830798727 shutdown_connections 2024-06-09T22:31:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.841+0000 7f977cc66700 1 -- 172.21.15.104:0/830798727 wait complete. 2024-06-09T22:31:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.842+0000 7f977cc66700 1 Processor -- start 2024-06-09T22:31:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.842+0000 7f977cc66700 1 -- start start 2024-06-09T22:31:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.842+0000 7f977cc66700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778101390 0x7f977819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.842+0000 7f977cc66700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9778101d50 0x7f977819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.842+0000 7f977cc66700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9778107400 0x7f97781989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.842+0000 7f977cc66700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9778111670 con 0x7f9778101390 2024-06-09T22:31:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.842+0000 7f977cc66700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97781114f0 con 0x7f9778107400 2024-06-09T22:31:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.842+0000 7f977cc66700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f97781117f0 con 0x7f9778101d50 2024-06-09T22:31:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f977659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778101390 0x7f977819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f977659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778101390 0x7f977819e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49732/0 (socket says 172.21.15.104:49732) 2024-06-09T22:31:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f977659c700 1 -- 172.21.15.104:0/3183429574 learned_addr learned my addr 172.21.15.104:0/3183429574 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f9775d9b700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9778101d50 0x7f977819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f9776d9d700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9778107400 0x7f97781989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f977659c700 1 -- 172.21.15.104:0/3183429574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9778101d50 msgr2=0x7f977819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f977659c700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9778101d50 0x7f977819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f977659c700 1 -- 172.21.15.104:0/3183429574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9778107400 msgr2=0x7f97781989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:27.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f977659c700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9778107400 0x7f97781989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f977659c700 1 -- 172.21.15.104:0/3183429574 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9778199120 con 0x7f9778101390 2024-06-09T22:31:27.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f977659c700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778101390 0x7f977819e7b0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f976800c9b0 tx=0x7f976800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:27.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f97677fe700 1 -- 172.21.15.104:0/3183429574 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9768007b00 con 0x7f9778101390 2024-06-09T22:31:27.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f97677fe700 1 -- 172.21.15.104:0/3183429574 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f976800ce50 con 0x7f9778101390 2024-06-09T22:31:27.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f97677fe700 1 -- 172.21.15.104:0/3183429574 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9768018660 con 0x7f9778101390 2024-06-09T22:31:27.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.843+0000 7f977cc66700 1 -- 172.21.15.104:0/3183429574 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9778199410 con 0x7f9778101390 2024-06-09T22:31:27.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.844+0000 7f977cc66700 1 -- 172.21.15.104:0/3183429574 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97781a5680 con 0x7f9778101390 2024-06-09T22:31:27.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.846+0000 7f97677fe700 1 -- 172.21.15.104:0/3183429574 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f976800f450 con 0x7f9778101390 2024-06-09T22:31:27.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.846+0000 7f97677fe700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f97600a2e10 0x7f97600a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.847+0000 7f9775d9b700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f97600a2e10 0x7f97600a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.847+0000 7f97677fe700 1 -- 172.21.15.104:0/3183429574 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f97680cf660 con 0x7f9778101390 2024-06-09T22:31:27.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.847+0000 7f9775d9b700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f97600a2e10 0x7f97600a52d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9778078940 tx=0x7f976c00b410 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:27.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.848+0000 7f97874e9b80 1 Processor -- start 2024-06-09T22:31:27.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.849+0000 7f97874e9b80 1 -- start start 2024-06-09T22:31:27.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.849+0000 7f97874e9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c911288e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.849+0000 7f97874e9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c911282df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.849+0000 7f97874e9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 0x55c911283750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.849+0000 7f97874e9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55c9111d0810 con 0x55c911283330 2024-06-09T22:31:27.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.849+0000 7f977659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c911288e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.849+0000 7f97874e9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c911178720 con 0x55c911288a70 2024-06-09T22:31:27.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.849+0000 7f9776d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 0x55c911283750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.849+0000 7f977659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c911288e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.104:57790/0 (socket says 172.21.15.104:57790) 2024-06-09T22:31:27.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.849+0000 7f977659c700 1 -- 172.21.15.104:0/783181382 learned_addr learned my addr 172.21.15.104:0/783181382 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:27.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.849+0000 7f97874e9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c9111cc7d0 con 0x55c911289390 2024-06-09T22:31:27.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.849+0000 7f9775d9b700 1 --2- 172.21.15.104:0/783181382 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c911282df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.850+0000 7f977659c700 1 -- 172.21.15.104:0/783181382 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 msgr2=0x55c911282df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:27.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.850+0000 7f977659c700 1 --2- 172.21.15.104:0/783181382 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c911282df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.850+0000 7f977659c700 1 -- 172.21.15.104:0/783181382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 msgr2=0x55c911283750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:27.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.850+0000 7f977659c700 1 --2- 172.21.15.104:0/783181382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 0x55c911283750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.850+0000 7f977659c700 1 -- 172.21.15.104:0/783181382 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c91116ef50 con 0x55c911288a70 2024-06-09T22:31:27.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.850+0000 7f9776d9d700 1 --2- 172.21.15.104:0/783181382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 0x55c911283750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:31:27.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.850+0000 7f977659c700 1 --2- 172.21.15.104:0/783181382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c911288e50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f976809b740 tx=0x7f976809b6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7711517847b6894 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:27.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.850+0000 7f9764ff9700 1 -- 172.21.15.104:0/783181382 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97680b4070 con 0x55c911288a70 2024-06-09T22:31:27.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.850+0000 7f9764ff9700 1 -- 172.21.15.104:0/783181382 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f976809fb60 con 0x55c911288a70 2024-06-09T22:31:27.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.850+0000 7f9764ff9700 1 -- 172.21.15.104:0/783181382 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976809ba10 con 0x55c911288a70 2024-06-09T22:31:27.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.851+0000 7f97874e9b80 1 -- 172.21.15.104:0/783181382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 msgr2=0x55c911288e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:27.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.851+0000 7f97874e9b80 1 --2- 172.21.15.104:0/783181382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c911288e50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f976809b740 tx=0x7f976809b6b0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.851+0000 7f97874e9b80 1 -- 172.21.15.104:0/783181382 shutdown_connections 2024-06-09T22:31:27.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.851+0000 7f97874e9b80 1 --2- 172.21.15.104:0/783181382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 0x55c911283750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.851+0000 7f97874e9b80 1 --2- 172.21.15.104:0/783181382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c911288e50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.851+0000 7f97874e9b80 1 --2- 172.21.15.104:0/783181382 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c911282df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.851+0000 7f97874e9b80 1 -- 172.21.15.104:0/783181382 >> 172.21.15.104:0/783181382 conn(0x55c9111a9cc0 msgr2=0x55c911279850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:27.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.851+0000 7f97874e9b80 1 -- 172.21.15.104:0/783181382 shutdown_connections 2024-06-09T22:31:27.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.851+0000 7f97874e9b80 1 -- 172.21.15.104:0/783181382 wait complete. 2024-06-09T22:31:27.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.851+0000 7f97874e9b80 1 -- start start 2024-06-09T22:31:27.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f97874e9b80 1 Processor -- start 2024-06-09T22:31:27.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f97874e9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911283330 0x55c91128f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f97874e9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c91128c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f97874e9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911289390 0x55c91128cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f97874e9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55c91128fdd0 con 0x55c911289390 2024-06-09T22:31:27.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f97874e9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c911170bf0 con 0x55c911288a70 2024-06-09T22:31:27.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f97874e9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c91128ff50 con 0x55c911283330 2024-06-09T22:31:27.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f97874e9b80 10 client.0 _ll_register_callbacks cb 0x55c91116a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:31:27.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f9776d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911289390 0x55c91128cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f9775d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c91128c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f977659c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911283330 0x55c91128f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f9776d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911289390 0x55c91128cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49738/0 (socket says 172.21.15.104:49738) 2024-06-09T22:31:27.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f9776d9d700 1 -- 172.21.15.104:0/3331636481 learned_addr learned my addr 172.21.15.104:0/3331636481 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:27.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f977659c700 1 -- 172.21.15.104:0/3331636481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 msgr2=0x55c91128c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:27.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f977659c700 1 --2- 172.21.15.104:0/3331636481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c91128c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f977659c700 1 -- 172.21.15.104:0/3331636481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911289390 msgr2=0x55c91128cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:27.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f977659c700 1 --2- 172.21.15.104:0/3331636481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911289390 0x55c91128cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.852+0000 7f977659c700 1 -- 172.21.15.104:0/3331636481 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c91111f8d0 con 0x55c911283330 2024-06-09T22:31:27.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.853+0000 7f977659c700 1 --2- 172.21.15.104:0/3331636481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911283330 0x55c91128f280 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f97680cca90 tx=0x7f97680a4420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c800d6103dd4265 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:27.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.853+0000 7f975dffb700 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-06-09T22:31:27.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.853+0000 7f975dffb700 1 -- 172.21.15.104:0/3331636481 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97680b4040 con 0x55c911283330 2024-06-09T22:31:27.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.853+0000 7f975dffb700 1 -- 172.21.15.104:0/3331636481 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f97680ba170 con 0x55c911283330 2024-06-09T22:31:27.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.853+0000 7f97874e9b80 1 -- 172.21.15.104:0/3331636481 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f976809fb60 con 0x55c911283330 2024-06-09T22:31:27.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.853+0000 7f975dffb700 1 -- 172.21.15.104:0/3331636481 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9768086040 con 0x55c911283330 2024-06-09T22:31:27.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.853+0000 7f97874e9b80 10 client.27304 fetch_fsmap learned FSMap version 32 2024-06-09T22:31:27.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.853+0000 7f97874e9b80 1 -- 172.21.15.104:0/3331636481 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c91128fdd0 con 0x55c911283330 2024-06-09T22:31:27.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.853+0000 7f975dffb700 1 -- 172.21.15.104:0/3331636481 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f976809fb60 con 0x55c911283330 2024-06-09T22:31:27.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.853+0000 7f975dffb700 1 -- 172.21.15.104:0/3331636481 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f97680a3f90 con 0x55c911283330 2024-06-09T22:31:27.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.854+0000 7f97874e9b80 10 client.27304 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:31:27.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.854+0000 7f97874e9b80 10 client.27304 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:31:27.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.854+0000 7f97874e9b80 20 client.27304 populate_metadata read hostname 'smithi104' 2024-06-09T22:31:27.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.854+0000 7f97874e9b80 1 --2- 172.21.15.104:0/3331636481 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55c91129cb60 0x55c91129ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.854+0000 7f97874e9b80 4 client.27304 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:31:27.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.854+0000 7f97874e9b80 1 -- 172.21.15.104:0/3331636481 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9111ddd10 con 0x55c91129cb60 2024-06-09T22:31:27.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.854+0000 7f9775d9b700 1 --2- 172.21.15.104:0/3331636481 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55c91129cb60 0x55c91129ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.854+0000 7f975dffb700 10 client.27304 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:31:27.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.854+0000 7f9775d9b700 1 --2- 172.21.15.104:0/3331636481 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55c91129cb60 0x55c91129ef50 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=84e0740544426c49 server_cookie=9839d48a187c7594 in_seq=0 out_seq=0 2024-06-09T22:31:27.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f975dffb700 1 -- 172.21.15.104:0/3331636481 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55c9111ddd10 con 0x55c91129cb60 2024-06-09T22:31:27.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f975dffb700 10 client.27304 handle_command_reply: tid=0 2024-06-09T22:31:27.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f97874e9b80 1 client.27304 shutdown 2024-06-09T22:31:27.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f97874e9b80 1 -- 172.21.15.104:0/3331636481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911283330 msgr2=0x55c91128f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:27.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f97874e9b80 1 --2- 172.21.15.104:0/3331636481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911283330 0x55c91128f280 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f97680cca90 tx=0x7f97680a4420 comp rx=0 tx=0).stop 2024-06-09T22:31:27.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f97874e9b80 1 -- 172.21.15.104:0/3331636481 shutdown_connections 2024-06-09T22:31:27.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f97874e9b80 1 --2- 172.21.15.104:0/3331636481 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55c91129cb60 0x55c91129ef50 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f97874e9b80 1 --2- 172.21.15.104:0/3331636481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911289390 0x55c91128cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f975dffb700 0 client.27304 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:31:27.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f97874e9b80 1 --2- 172.21.15.104:0/3331636481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911283330 0x55c91128f280 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f97874e9b80 1 --2- 172.21.15.104:0/3331636481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c91128c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f97874e9b80 1 -- 172.21.15.104:0/3331636481 >> 172.21.15.104:0/3331636481 conn(0x55c9111a9cc0 msgr2=0x55c91127d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:27.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f97874e9b80 1 -- 172.21.15.104:0/3331636481 shutdown_connections 2024-06-09T22:31:27.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.855+0000 7f97874e9b80 1 -- 172.21.15.104:0/3331636481 wait complete. 2024-06-09T22:31:27.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.856+0000 7f97874e9b80 20 client.27304 trim_cache size 0 max 16384 2024-06-09T22:31:27.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.877+0000 7f97874e9b80 1 Processor -- start 2024-06-09T22:31:27.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.877+0000 7f97874e9b80 1 -- start start 2024-06-09T22:31:27.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.877+0000 7f97874e9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 0x55c911286bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.877+0000 7f97874e9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c911273510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.877+0000 7f97874e9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c911299070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.877+0000 7f97874e9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f97781ac450 con 0x55c911283330 2024-06-09T22:31:27.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.877+0000 7f97874e9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97781adf90 con 0x55c911288a70 2024-06-09T22:31:27.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.877+0000 7f97874e9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f97781acb20 con 0x55c911289390 2024-06-09T22:31:27.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.877+0000 7f977659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 0x55c911286bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.877+0000 7f977659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 0x55c911286bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49748/0 (socket says 172.21.15.104:49748) 2024-06-09T22:31:27.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.877+0000 7f977659c700 1 -- 172.21.15.104:0/957067528 learned_addr learned my addr 172.21.15.104:0/957067528 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:27.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f9776d9d700 1 --2- 172.21.15.104:0/957067528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c911299070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f9775d9b700 1 --2- 172.21.15.104:0/957067528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c911273510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f977659c700 1 -- 172.21.15.104:0/957067528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 msgr2=0x55c911299070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:27.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f977659c700 1 --2- 172.21.15.104:0/957067528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c911299070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f977659c700 1 -- 172.21.15.104:0/957067528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 msgr2=0x55c911273510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:27.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f977659c700 1 --2- 172.21.15.104:0/957067528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c911273510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f977659c700 1 -- 172.21.15.104:0/957067528 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c91128aaf0 con 0x55c911283330 2024-06-09T22:31:27.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f9775d9b700 1 --2- 172.21.15.104:0/957067528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c911273510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:31:27.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f9776d9d700 1 --2- 172.21.15.104:0/957067528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c911299070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:31:27.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f977659c700 1 --2- 172.21.15.104:0/957067528 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 0x55c911286bf0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f97680ccfe0 tx=0x7f9768000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954204066d26e206 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:27.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f975ffff700 1 -- 172.21.15.104:0/957067528 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97680b8070 con 0x55c911283330 2024-06-09T22:31:27.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f975ffff700 1 -- 172.21.15.104:0/957067528 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9768097e20 con 0x55c911283330 2024-06-09T22:31:27.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.878+0000 7f975ffff700 1 -- 172.21.15.104:0/957067528 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97680b4040 con 0x55c911283330 2024-06-09T22:31:27.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 -- 172.21.15.104:0/957067528 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 msgr2=0x55c911286bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:27.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 --2- 172.21.15.104:0/957067528 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 0x55c911286bf0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f97680ccfe0 tx=0x7f9768000bc0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 -- 172.21.15.104:0/957067528 shutdown_connections 2024-06-09T22:31:27.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 --2- 172.21.15.104:0/957067528 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911283330 0x55c911286bf0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 --2- 172.21.15.104:0/957067528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c911299070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 --2- 172.21.15.104:0/957067528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911288a70 0x55c911273510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 -- 172.21.15.104:0/957067528 >> 172.21.15.104:0/957067528 conn(0x55c9111a9cc0 msgr2=0x55c911272e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:27.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 -- 172.21.15.104:0/957067528 shutdown_connections 2024-06-09T22:31:27.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 -- 172.21.15.104:0/957067528 wait complete. 2024-06-09T22:31:27.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 -- start start 2024-06-09T22:31:27.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 Processor -- start 2024-06-09T22:31:27.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911283330 0x55c9112953c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911288a70 0x55c911295900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.879+0000 7f97874e9b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c9112a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f97874e9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f97781ad8c0 con 0x55c911288a70 2024-06-09T22:31:27.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f97874e9b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97781ad1f0 con 0x55c911283330 2024-06-09T22:31:27.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f97874e9b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f97781aa430 con 0x55c911289390 2024-06-09T22:31:27.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f9775d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911288a70 0x55c911295900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f9775d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911288a70 0x55c911295900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49760/0 (socket says 172.21.15.104:49760) 2024-06-09T22:31:27.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f97874e9b80 10 client.0 _ll_register_callbacks cb 0x55c91116a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:31:27.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f9775d9b700 1 -- 172.21.15.104:0/2889725599 learned_addr learned my addr 172.21.15.104:0/2889725599 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:27.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f9776d9d700 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c9112a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f977659c700 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911283330 0x55c9112953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f9775d9b700 1 -- 172.21.15.104:0/2889725599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 msgr2=0x55c9112a39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:27.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f9775d9b700 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c9112a39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f9775d9b700 1 -- 172.21.15.104:0/2889725599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911283330 msgr2=0x55c9112953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:27.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f9775d9b700 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911283330 0x55c9112953c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f9775d9b700 1 -- 172.21.15.104:0/2889725599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9112a4010 con 0x55c911288a70 2024-06-09T22:31:27.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f977659c700 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911283330 0x55c9112953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:31:27.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f9775d9b700 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911288a70 0x55c911295900 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f976c005d90 tx=0x7f976c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5751aa1e07e66be server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:27.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f975dffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:31:27.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f975dffb700 1 -- 172.21.15.104:0/2889725599 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c02f070 con 0x55c911288a70 2024-06-09T22:31:27.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f975dffb700 1 -- 172.21.15.104:0/2889725599 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f976c011da0 con 0x55c911288a70 2024-06-09T22:31:27.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.880+0000 7f97874e9b80 1 -- 172.21.15.104:0/2889725599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9768097e20 con 0x55c911288a70 2024-06-09T22:31:27.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.881+0000 7f975dffb700 1 -- 172.21.15.104:0/2889725599 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f976c03b050 con 0x55c911288a70 2024-06-09T22:31:27.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.881+0000 7f97874e9b80 10 client.18090 fetch_fsmap learned FSMap version 32 2024-06-09T22:31:27.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.881+0000 7f97874e9b80 1 -- 172.21.15.104:0/2889725599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c911306ec0 con 0x55c911288a70 2024-06-09T22:31:27.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.881+0000 7f975dffb700 1 -- 172.21.15.104:0/2889725599 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9768097e20 con 0x55c911288a70 2024-06-09T22:31:27.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.881+0000 7f975dffb700 1 -- 172.21.15.104:0/2889725599 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f976c01f730 con 0x55c911288a70 2024-06-09T22:31:27.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.881+0000 7f97874e9b80 10 client.18090 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:31:27.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.881+0000 7f97874e9b80 10 client.18090 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-06-09T22:31:27.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.881+0000 7f97874e9b80 20 client.18090 populate_metadata read hostname 'smithi104' 2024-06-09T22:31:27.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.881+0000 7f97874e9b80 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55c91129cb60 0x55c911309670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:27.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.881+0000 7f97874e9b80 4 client.18090 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:31:27.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.881+0000 7f97874e9b80 1 -- 172.21.15.104:0/2889725599 --> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c9111d8290 con 0x55c91129cb60 2024-06-09T22:31:27.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.881+0000 7f977659c700 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55c91129cb60 0x55c911309670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:27.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f975dffb700 10 client.18090 ms_handle_connect on v2:172.21.15.194:6836/464494707 2024-06-09T22:31:27.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f977659c700 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55c91129cb60 0x55c911309670 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3399d570a2b0fc3f server_cookie=18008d99389ed525 in_seq=0 out_seq=0 2024-06-09T22:31:27.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f975dffb700 1 -- 172.21.15.104:0/2889725599 <== mds.1 v2:172.21.15.194:6836/464494707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c9111d8290 con 0x55c91129cb60 2024-06-09T22:31:27.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f975dffb700 10 client.18090 handle_command_reply: tid=0 2024-06-09T22:31:27.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f97874e9b80 1 client.18090 shutdown 2024-06-09T22:31:27.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f97874e9b80 1 -- 172.21.15.104:0/2889725599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911288a70 msgr2=0x55c911295900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:27.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f97874e9b80 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911288a70 0x55c911295900 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f976c005d90 tx=0x7f976c0047c0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f97874e9b80 1 -- 172.21.15.104:0/2889725599 shutdown_connections 2024-06-09T22:31:27.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f97874e9b80 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.194:6836/464494707,v1:172.21.15.194:6837/464494707] conn(0x55c91129cb60 0x55c911309670 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f975dffb700 0 client.18090 ms_handle_reset on v2:172.21.15.194:6836/464494707 2024-06-09T22:31:27.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f97874e9b80 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c911289390 0x55c9112a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f97874e9b80 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c911283330 0x55c9112953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f97874e9b80 1 --2- 172.21.15.104:0/2889725599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55c911288a70 0x55c911295900 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f97874e9b80 1 -- 172.21.15.104:0/2889725599 >> 172.21.15.104:0/2889725599 conn(0x55c9111a9cc0 msgr2=0x55c91127e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:27.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f97874e9b80 1 -- 172.21.15.104:0/2889725599 shutdown_connections 2024-06-09T22:31:27.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f97874e9b80 1 -- 172.21.15.104:0/2889725599 wait complete. 2024-06-09T22:31:27.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.882+0000 7f97874e9b80 20 client.18090 trim_cache size 0 max 16384 2024-06-09T22:31:27.895 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:31:27.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.883+0000 7f977cc66700 1 -- 172.21.15.104:0/3183429574 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f97600a2e10 msgr2=0x7f97600a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:27.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.883+0000 7f977cc66700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f97600a2e10 0x7f97600a52d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9778078940 tx=0x7f976c00b410 comp rx=0 tx=0).stop 2024-06-09T22:31:27.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.883+0000 7f977cc66700 1 -- 172.21.15.104:0/3183429574 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778101390 msgr2=0x7f977819e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:27.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.883+0000 7f977cc66700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778101390 0x7f977819e7b0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f976800c9b0 tx=0x7f976800ccc0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.883+0000 7f977cc66700 1 -- 172.21.15.104:0/3183429574 shutdown_connections 2024-06-09T22:31:27.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.883+0000 7f977cc66700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f97600a2e10 0x7f97600a52d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.883+0000 7f977cc66700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778101390 0x7f977819e7b0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.883+0000 7f977cc66700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9778107400 0x7f97781989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.883+0000 7f977cc66700 1 --2- 172.21.15.104:0/3183429574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9778101d50 0x7f977819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:27.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.883+0000 7f977cc66700 1 -- 172.21.15.104:0/3183429574 >> 172.21.15.104:0/3183429574 conn(0x7f9778078340 msgr2=0x7f977806a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:27.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.883+0000 7f977cc66700 1 -- 172.21.15.104:0/3183429574 shutdown_connections 2024-06-09T22:31:27.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:27.883+0000 7f977cc66700 1 -- 172.21.15.104:0/3183429574 wait complete. 2024-06-09T22:31:27.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:27 smithi153 ceph-mon[36599]: pgmap v325: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-06-09T22:31:27.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:27 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:27.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:27 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:28.752 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-09T22:31:28.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:28 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:28.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:28 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:28.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:28 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:28.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:28 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:28.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:28 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:31:28.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:28 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:31:28.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:28 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:28.990 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:31:28.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:28 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:28.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:28 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:28.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:28 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:28.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:28 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:28.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:28 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:31:28.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:28 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:31:28.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:28 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:29.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:28 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:29.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:28 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:29.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:28 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:29.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:28 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-09T22:31:29.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:28 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:31:29.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:28 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:31:29.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:28 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:31:29.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:29 smithi104 ceph-mon[32389]: pgmap v326: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 100 B/s wr, 12 op/s 2024-06-09T22:31:29.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:29 smithi104 ceph-mon[32389]: Adjusting osd_memory_target on smithi104 to 176.1M 2024-06-09T22:31:29.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:29 smithi104 ceph-mon[32389]: Unable to set osd_memory_target on smithi104 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-09T22:31:29.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:29 smithi194 ceph-mon[36856]: pgmap v326: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 100 B/s wr, 12 op/s 2024-06-09T22:31:29.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:29 smithi194 ceph-mon[36856]: Adjusting osd_memory_target on smithi104 to 176.1M 2024-06-09T22:31:29.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:29 smithi194 ceph-mon[36856]: Unable to set osd_memory_target on smithi104 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-09T22:31:29.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:29 smithi153 ceph-mon[36599]: pgmap v326: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 100 B/s wr, 12 op/s 2024-06-09T22:31:29.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:29 smithi153 ceph-mon[36599]: Adjusting osd_memory_target on smithi104 to 176.1M 2024-06-09T22:31:29.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:29 smithi153 ceph-mon[36599]: Unable to set osd_memory_target on smithi104 to 184732979: error parsing value: Value '184732979' is below minimum 939524096 2024-06-09T22:31:30.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.726+0000 7fad2ba0c700 1 -- 172.21.15.104:0/3997585807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad24107840 msgr2=0x7fad2410f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:30.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.726+0000 7fad2ba0c700 1 --2- 172.21.15.104:0/3997585807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad24107840 0x7fad2410f3c0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00b0d0 tx=0x7fad1c00b3e0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.726+0000 7fad2ba0c700 1 -- 172.21.15.104:0/3997585807 shutdown_connections 2024-06-09T22:31:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.726+0000 7fad2ba0c700 1 --2- 172.21.15.104:0/3997585807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad24107840 0x7fad2410f3c0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.726+0000 7fad2ba0c700 1 --2- 172.21.15.104:0/3997585807 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad24102740 0x7fad24102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.726+0000 7fad2ba0c700 1 --2- 172.21.15.104:0/3997585807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad241030f0 0x7fad24107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.726+0000 7fad2ba0c700 1 -- 172.21.15.104:0/3997585807 >> 172.21.15.104:0/3997585807 conn(0x7fad240795f0 msgr2=0x7fad24079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.726+0000 7fad2ba0c700 1 -- 172.21.15.104:0/3997585807 shutdown_connections 2024-06-09T22:31:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.726+0000 7fad2ba0c700 1 -- 172.21.15.104:0/3997585807 wait complete. 2024-06-09T22:31:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad2ba0c700 1 Processor -- start 2024-06-09T22:31:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad2ba0c700 1 -- start start 2024-06-09T22:31:30.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad2ba0c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad24102740 0x7fad2419e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad2ba0c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad241030f0 0x7fad2419ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad2ba0c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad24107840 0x7fad24198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad2ba0c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fad24112780 con 0x7fad241030f0 2024-06-09T22:31:30.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad2ba0c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fad24112600 con 0x7fad24107840 2024-06-09T22:31:30.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad2ba0c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fad24112900 con 0x7fad24102740 2024-06-09T22:31:30.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad28fa7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad241030f0 0x7fad2419ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad28fa7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad241030f0 0x7fad2419ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49768/0 (socket says 172.21.15.104:49768) 2024-06-09T22:31:30.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad28fa7700 1 -- 172.21.15.104:0/2409589201 learned_addr learned my addr 172.21.15.104:0/2409589201 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:30.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad297a8700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad24102740 0x7fad2419e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.727+0000 7fad29fa9700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad24107840 0x7fad24198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.728+0000 7fad297a8700 1 -- 172.21.15.104:0/2409589201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad24107840 msgr2=0x7fad24198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:30.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.728+0000 7fad297a8700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad24107840 0x7fad24198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.728+0000 7fad297a8700 1 -- 172.21.15.104:0/2409589201 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad241030f0 msgr2=0x7fad2419ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:30.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.728+0000 7fad297a8700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad241030f0 0x7fad2419ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.728+0000 7fad297a8700 1 -- 172.21.15.104:0/2409589201 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad241990e0 con 0x7fad24102740 2024-06-09T22:31:30.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.728+0000 7fad28fa7700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad241030f0 0x7fad2419ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:31:30.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.728+0000 7fad297a8700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad24102740 0x7fad2419e7c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fad2000d750 tx=0x7fad2000da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:30.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.728+0000 7fad1a7fc700 1 -- 172.21.15.104:0/2409589201 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20016070 con 0x7fad24102740 2024-06-09T22:31:30.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.728+0000 7fad2ba0c700 1 -- 172.21.15.104:0/2409589201 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad241993d0 con 0x7fad24102740 2024-06-09T22:31:30.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.728+0000 7fad1a7fc700 1 -- 172.21.15.104:0/2409589201 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad20005020 con 0x7fad24102740 2024-06-09T22:31:30.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.728+0000 7fad1a7fc700 1 -- 172.21.15.104:0/2409589201 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2000fb30 con 0x7fad24102740 2024-06-09T22:31:30.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.728+0000 7fad2ba0c700 1 -- 172.21.15.104:0/2409589201 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad240746c0 con 0x7fad24102740 2024-06-09T22:31:30.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.730+0000 7fad1a7fc700 1 -- 172.21.15.104:0/2409589201 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fad200051d0 con 0x7fad24102740 2024-06-09T22:31:30.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.731+0000 7fad1a7fc700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fad100a2d90 0x7fad100a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.731+0000 7fad28fa7700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fad100a2d90 0x7fad100a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.731+0000 7fad1a7fc700 1 -- 172.21.15.104:0/2409589201 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7fad200cf170 con 0x7fad24102740 2024-06-09T22:31:30.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.731+0000 7fad28fa7700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fad100a2d90 0x7fad100a5250 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fad24102030 tx=0x7fad1400b500 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:30.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad3628fb80 1 Processor -- start 2024-06-09T22:31:30.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad3628fb80 1 -- start start 2024-06-09T22:31:30.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad3628fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046ff60 0x55d460470340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad3628fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d46046a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad3628fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046a7a0 0x55d46046abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad3628fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d46035cf60 con 0x55d460470880 2024-06-09T22:31:30.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad3628fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d4603bb850 con 0x55d46046ff60 2024-06-09T22:31:30.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad3628fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d460323460 con 0x55d46046a7a0 2024-06-09T22:31:30.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad28fa7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d46046a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad28fa7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d46046a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49774/0 (socket says 172.21.15.104:49774) 2024-06-09T22:31:30.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad28fa7700 1 -- 172.21.15.104:0/991348317 learned_addr learned my addr 172.21.15.104:0/991348317 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:30.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad297a8700 1 --2- 172.21.15.104:0/991348317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046ff60 0x55d460470340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.732+0000 7fad29fa9700 1 --2- 172.21.15.104:0/991348317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046a7a0 0x55d46046abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.733+0000 7fad28fa7700 1 -- 172.21.15.104:0/991348317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046a7a0 msgr2=0x55d46046abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:30.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.733+0000 7fad28fa7700 1 --2- 172.21.15.104:0/991348317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046a7a0 0x55d46046abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.733+0000 7fad28fa7700 1 -- 172.21.15.104:0/991348317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046ff60 msgr2=0x55d460470340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:30.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.733+0000 7fad28fa7700 1 --2- 172.21.15.104:0/991348317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046ff60 0x55d460470340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.733+0000 7fad28fa7700 1 -- 172.21.15.104:0/991348317 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d46030a8d0 con 0x55d460470880 2024-06-09T22:31:30.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.733+0000 7fad297a8700 1 --2- 172.21.15.104:0/991348317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046ff60 0x55d460470340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:31:30.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.733+0000 7fad28fa7700 1 --2- 172.21.15.104:0/991348317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d46046a260 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fad1400bf60 tx=0x7fad14013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aec6d5ac44d8a93 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:30.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.733+0000 7fad0f7fe700 1 -- 172.21.15.104:0/991348317 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1401e620 con 0x55d460470880 2024-06-09T22:31:30.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.733+0000 7fad0f7fe700 1 -- 172.21.15.104:0/991348317 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1401eca0 con 0x55d460470880 2024-06-09T22:31:30.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.733+0000 7fad0f7fe700 1 -- 172.21.15.104:0/991348317 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1401cd90 con 0x55d460470880 2024-06-09T22:31:30.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.734+0000 7fad3628fb80 1 -- 172.21.15.104:0/991348317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 msgr2=0x55d46046a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:30.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.734+0000 7fad3628fb80 1 --2- 172.21.15.104:0/991348317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d46046a260 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fad1400bf60 tx=0x7fad14013000 comp rx=0 tx=0).stop 2024-06-09T22:31:30.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.734+0000 7fad3628fb80 1 -- 172.21.15.104:0/991348317 shutdown_connections 2024-06-09T22:31:30.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.734+0000 7fad3628fb80 1 --2- 172.21.15.104:0/991348317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046a7a0 0x55d46046abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.734+0000 7fad3628fb80 1 --2- 172.21.15.104:0/991348317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046ff60 0x55d460470340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.734+0000 7fad3628fb80 1 --2- 172.21.15.104:0/991348317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d46046a260 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.734+0000 7fad3628fb80 1 -- 172.21.15.104:0/991348317 >> 172.21.15.104:0/991348317 conn(0x55d460397080 msgr2=0x55d460465220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:30.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.734+0000 7fad3628fb80 1 -- 172.21.15.104:0/991348317 shutdown_connections 2024-06-09T22:31:30.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.734+0000 7fad3628fb80 1 -- 172.21.15.104:0/991348317 wait complete. 2024-06-09T22:31:30.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.734+0000 7fad3628fb80 1 -- start start 2024-06-09T22:31:30.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.734+0000 7fad3628fb80 1 Processor -- start 2024-06-09T22:31:30.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad3628fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d46046a7a0 0x55d460476680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad3628fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046ff60 0x55d460473980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad3628fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d460470880 0x55d460473ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad3628fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d46035bfc0 con 0x55d46046a7a0 2024-06-09T22:31:30.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad3628fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d4603b7810 con 0x55d460470880 2024-06-09T22:31:30.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad3628fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d460477260 con 0x55d46046ff60 2024-06-09T22:31:30.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad3628fb80 10 client.0 _ll_register_callbacks cb 0x55d4603515b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:31:30.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad297a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d46046a7a0 0x55d460476680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad297a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d46046a7a0 0x55d460476680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49786/0 (socket says 172.21.15.104:49786) 2024-06-09T22:31:30.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad297a8700 1 -- 172.21.15.104:0/2020045540 learned_addr learned my addr 172.21.15.104:0/2020045540 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:30.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad29fa9700 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d460470880 0x55d460473ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad28fa7700 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046ff60 0x55d460473980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad297a8700 1 -- 172.21.15.104:0/2020045540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046ff60 msgr2=0x55d460473980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:30.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad297a8700 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046ff60 0x55d460473980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad297a8700 1 -- 172.21.15.104:0/2020045540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d460470880 msgr2=0x55d460473ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:30.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad297a8700 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d460470880 0x55d460473ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad297a8700 1 -- 172.21.15.104:0/2020045540 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d460381750 con 0x55d46046a7a0 2024-06-09T22:31:30.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad297a8700 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d46046a7a0 0x55d460476680 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fad20097c90 tx=0x7fad2009f7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3abb7b04ea6369a server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:30.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.735+0000 7fad0cff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:31:30.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad0cff9700 1 -- 172.21.15.104:0/2020045540 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad200b4060 con 0x55d46046a7a0 2024-06-09T22:31:30.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad0cff9700 1 -- 172.21.15.104:0/2020045540 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7fad2009b520 con 0x55d46046a7a0 2024-06-09T22:31:30.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad3628fb80 1 -- 172.21.15.104:0/2020045540 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad1401eca0 con 0x55d46046a7a0 2024-06-09T22:31:30.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad0cff9700 1 -- 172.21.15.104:0/2020045540 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fad2001f040 con 0x55d46046a7a0 2024-06-09T22:31:30.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad3628fb80 10 client.18102 fetch_fsmap learned FSMap version 32 2024-06-09T22:31:30.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad3628fb80 1 -- 172.21.15.104:0/2020045540 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d460481b40 con 0x55d46046a7a0 2024-06-09T22:31:30.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad0cff9700 1 -- 172.21.15.104:0/2020045540 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1401eca0 con 0x55d46046a7a0 2024-06-09T22:31:30.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad0cff9700 1 -- 172.21.15.104:0/2020045540 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fad2009f170 con 0x55d46046a7a0 2024-06-09T22:31:30.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad3628fb80 10 client.18102 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:31:30.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad3628fb80 10 client.18102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:31:30.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad3628fb80 20 client.18102 populate_metadata read hostname 'smithi104' 2024-06-09T22:31:30.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad3628fb80 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d460484180 0x55d460486570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad3628fb80 4 client.18102 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-09T22:31:30.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad3628fb80 1 -- 172.21.15.104:0/2020045540 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4603c8d50 con 0x55d460484180 2024-06-09T22:31:30.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.736+0000 7fad28fa7700 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d460484180 0x55d460486570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.737+0000 7fad0cff9700 10 client.18102 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:31:30.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.737+0000 7fad28fa7700 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d460484180 0x55d460486570 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ca2a0f03f5f5f1c server_cookie=c535ac72f8c60d05 in_seq=0 out_seq=0 2024-06-09T22:31:30.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad0cff9700 1 -- 172.21.15.104:0/2020045540 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d4603c8d50 con 0x55d460484180 2024-06-09T22:31:30.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad0cff9700 10 client.18102 handle_command_reply: tid=0 2024-06-09T22:31:30.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad3628fb80 1 client.18102 shutdown 2024-06-09T22:31:30.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad3628fb80 1 -- 172.21.15.104:0/2020045540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d46046a7a0 msgr2=0x55d460476680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:30.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad3628fb80 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d46046a7a0 0x55d460476680 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fad20097c90 tx=0x7fad2009f7a0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad3628fb80 1 -- 172.21.15.104:0/2020045540 shutdown_connections 2024-06-09T22:31:30.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad3628fb80 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d460484180 0x55d460486570 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad3628fb80 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d46046a7a0 0x55d460476680 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad3628fb80 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d460470880 0x55d460473ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad3628fb80 1 --2- 172.21.15.104:0/2020045540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046ff60 0x55d460473980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad0cff9700 0 client.18102 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:31:30.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad3628fb80 1 -- 172.21.15.104:0/2020045540 >> 172.21.15.104:0/2020045540 conn(0x55d460397080 msgr2=0x55d460464700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:30.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad3628fb80 1 -- 172.21.15.104:0/2020045540 shutdown_connections 2024-06-09T22:31:30.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad3628fb80 1 -- 172.21.15.104:0/2020045540 wait complete. 2024-06-09T22:31:30.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.738+0000 7fad3628fb80 20 client.18102 trim_cache size 0 max 16384 2024-06-09T22:31:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.753+0000 7fad3628fb80 1 Processor -- start 2024-06-09T22:31:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.753+0000 7fad3628fb80 1 -- start start 2024-06-09T22:31:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.753+0000 7fad3628fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046a7a0 0x55d46046a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.753+0000 7fad3628fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046ff60 0x55d46046f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.753+0000 7fad3628fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d4604761c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.753+0000 7fad3628fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b0770 con 0x55d460470880 2024-06-09T22:31:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.753+0000 7fad3628fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b22b0 con 0x55d46046ff60 2024-06-09T22:31:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.753+0000 7fad3628fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b0e40 con 0x55d46046a7a0 2024-06-09T22:31:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad29fa9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d4604761c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad29fa9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d4604761c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49790/0 (socket says 172.21.15.104:49790) 2024-06-09T22:31:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad29fa9700 1 -- 172.21.15.104:0/1168846713 learned_addr learned my addr 172.21.15.104:0/1168846713 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad297a8700 1 --2- 172.21.15.104:0/1168846713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046a7a0 0x55d46046a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad28fa7700 1 --2- 172.21.15.104:0/1168846713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046ff60 0x55d46046f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad29fa9700 1 -- 172.21.15.104:0/1168846713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046a7a0 msgr2=0x55d46046a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:30.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad29fa9700 1 --2- 172.21.15.104:0/1168846713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046a7a0 0x55d46046a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad29fa9700 1 -- 172.21.15.104:0/1168846713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046ff60 msgr2=0x55d46046f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:30.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad29fa9700 1 --2- 172.21.15.104:0/1168846713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046ff60 0x55d46046f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad29fa9700 1 -- 172.21.15.104:0/1168846713 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d460486ba0 con 0x55d460470880 2024-06-09T22:31:30.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad29fa9700 1 --2- 172.21.15.104:0/1168846713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d4604761c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00b010 tx=0x7fad1c006210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7b442ee5881d34 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:30.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad0effd700 1 -- 172.21.15.104:0/1168846713 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c02a070 con 0x55d460470880 2024-06-09T22:31:30.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad0effd700 1 -- 172.21.15.104:0/1168846713 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c009200 con 0x55d460470880 2024-06-09T22:31:30.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.754+0000 7fad0effd700 1 -- 172.21.15.104:0/1168846713 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c007d30 con 0x55d460470880 2024-06-09T22:31:30.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.755+0000 7fad3628fb80 1 -- 172.21.15.104:0/1168846713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 msgr2=0x55d4604761c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:30.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.755+0000 7fad3628fb80 1 --2- 172.21.15.104:0/1168846713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d4604761c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00b010 tx=0x7fad1c006210 comp rx=0 tx=0).stop 2024-06-09T22:31:30.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.755+0000 7fad3628fb80 1 -- 172.21.15.104:0/1168846713 shutdown_connections 2024-06-09T22:31:30.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.755+0000 7fad3628fb80 1 --2- 172.21.15.104:0/1168846713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d4604761c0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.755+0000 7fad3628fb80 1 --2- 172.21.15.104:0/1168846713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046a7a0 0x55d46046a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.755+0000 7fad3628fb80 1 --2- 172.21.15.104:0/1168846713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046ff60 0x55d46046f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.755+0000 7fad3628fb80 1 -- 172.21.15.104:0/1168846713 >> 172.21.15.104:0/1168846713 conn(0x55d460397080 msgr2=0x55d46046ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:30.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.755+0000 7fad3628fb80 1 -- 172.21.15.104:0/1168846713 shutdown_connections 2024-06-09T22:31:30.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.755+0000 7fad3628fb80 1 -- 172.21.15.104:0/1168846713 wait complete. 2024-06-09T22:31:30.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.755+0000 7fad3628fb80 1 -- start start 2024-06-09T22:31:30.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.755+0000 7fad3628fb80 1 Processor -- start 2024-06-09T22:31:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad3628fb80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046a7a0 0x55d46047c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad3628fb80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046ff60 0x55d46048b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad3628fb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d46048b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad3628fb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b1be0 con 0x55d460470880 2024-06-09T22:31:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad3628fb80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b1510 con 0x55d46046a7a0 2024-06-09T22:31:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad3628fb80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fad241ae730 con 0x55d46046ff60 2024-06-09T22:31:30.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad3628fb80 10 client.0 _ll_register_callbacks cb 0x55d4603515b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-09T22:31:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad29fa9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d46048b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad29fa9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d46048b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49806/0 (socket says 172.21.15.104:49806) 2024-06-09T22:31:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad29fa9700 1 -- 172.21.15.104:0/3832320506 learned_addr learned my addr 172.21.15.104:0/3832320506 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad297a8700 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046a7a0 0x55d46047c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad28fa7700 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046ff60 0x55d46048b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad29fa9700 1 -- 172.21.15.104:0/3832320506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046ff60 msgr2=0x55d46048b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad29fa9700 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046ff60 0x55d46048b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad29fa9700 1 -- 172.21.15.104:0/3832320506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046a7a0 msgr2=0x55d46047c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:31:30.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad29fa9700 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046a7a0 0x55d46047c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad29fa9700 1 -- 172.21.15.104:0/3832320506 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d460461810 con 0x55d460470880 2024-06-09T22:31:30.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad28fa7700 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046ff60 0x55d46048b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:31:30.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad29fa9700 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d46048b9c0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00afe0 tx=0x7fad1c003ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=909057d4a6ad80c1 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:30.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad0cff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-06-09T22:31:30.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad0cff9700 1 -- 172.21.15.104:0/3832320506 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c02a030 con 0x55d460470880 2024-06-09T22:31:30.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad0cff9700 1 -- 172.21.15.104:0/3832320506 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7fad1c0079b0 con 0x55d460470880 2024-06-09T22:31:30.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.756+0000 7fad3628fb80 1 -- 172.21.15.104:0/3832320506 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad1c009200 con 0x55d460470880 2024-06-09T22:31:30.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.757+0000 7fad0cff9700 1 -- 172.21.15.104:0/3832320506 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fad1c007bb0 con 0x55d460470880 2024-06-09T22:31:30.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.757+0000 7fad3628fb80 10 client.18108 fetch_fsmap learned FSMap version 32 2024-06-09T22:31:30.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.757+0000 7fad3628fb80 1 -- 172.21.15.104:0/3832320506 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d4604ee8e0 con 0x55d460470880 2024-06-09T22:31:30.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.757+0000 7fad0cff9700 1 -- 172.21.15.104:0/3832320506 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c009200 con 0x55d460470880 2024-06-09T22:31:30.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.757+0000 7fad0cff9700 1 -- 172.21.15.104:0/3832320506 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fad1c024070 con 0x55d460470880 2024-06-09T22:31:30.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.757+0000 7fad3628fb80 10 client.18108 fetch_fsmap finished waiting for FSMap version 32 2024-06-09T22:31:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.757+0000 7fad3628fb80 10 client.18108 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-06-09T22:31:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.757+0000 7fad3628fb80 20 client.18108 populate_metadata read hostname 'smithi104' 2024-06-09T22:31:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.757+0000 7fad3628fb80 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d4604eee30 0x55d4604f1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.757+0000 7fad3628fb80 4 client.18108 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-09T22:31:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.757+0000 7fad3628fb80 1 -- 172.21.15.104:0/3832320506 --> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d4603c32d0 con 0x55d4604eee30 2024-06-09T22:31:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.757+0000 7fad297a8700 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d4604eee30 0x55d4604f1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad0cff9700 10 client.18108 ms_handle_connect on v2:172.21.15.194:6838/2028005544 2024-06-09T22:31:30.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad297a8700 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d4604eee30 0x55d4604f1250 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7f3d0016feeb568 server_cookie=592b101b4fb21c83 in_seq=0 out_seq=0 2024-06-09T22:31:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad0cff9700 1 -- 172.21.15.104:0/3832320506 <== mds.0 v2:172.21.15.194:6838/2028005544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d4603c32d0 con 0x55d4604eee30 2024-06-09T22:31:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad0cff9700 10 client.18108 handle_command_reply: tid=0 2024-06-09T22:31:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad3628fb80 1 client.18108 shutdown 2024-06-09T22:31:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad3628fb80 1 -- 172.21.15.104:0/3832320506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 msgr2=0x55d46048b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad3628fb80 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d46048b9c0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00afe0 tx=0x7fad1c003ed0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad3628fb80 1 -- 172.21.15.104:0/3832320506 shutdown_connections 2024-06-09T22:31:30.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad3628fb80 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.194:6838/2028005544,v1:172.21.15.194:6839/2028005544] conn(0x55d4604eee30 0x55d4604f1250 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad3628fb80 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d460470880 0x55d46048b9c0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad3628fb80 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d46046a7a0 0x55d46047c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad0cff9700 0 client.18108 ms_handle_reset on v2:172.21.15.194:6838/2028005544 2024-06-09T22:31:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad3628fb80 1 --2- 172.21.15.104:0/3832320506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d46046ff60 0x55d46048b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad3628fb80 1 -- 172.21.15.104:0/3832320506 >> 172.21.15.104:0/3832320506 conn(0x55d460397080 msgr2=0x55d460469250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad3628fb80 1 -- 172.21.15.104:0/3832320506 shutdown_connections 2024-06-09T22:31:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad3628fb80 1 -- 172.21.15.104:0/3832320506 wait complete. 2024-06-09T22:31:30.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.758+0000 7fad3628fb80 20 client.18108 trim_cache size 0 max 16384 2024-06-09T22:31:30.772 INFO:teuthology.orchestra.run.smithi104.stdout:[] 2024-06-09T22:31:30.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.759+0000 7fad2ba0c700 1 -- 172.21.15.104:0/2409589201 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fad100a2d90 msgr2=0x7fad100a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:30.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.759+0000 7fad2ba0c700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fad100a2d90 0x7fad100a5250 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fad24102030 tx=0x7fad1400b500 comp rx=0 tx=0).stop 2024-06-09T22:31:30.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.759+0000 7fad2ba0c700 1 -- 172.21.15.104:0/2409589201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad24102740 msgr2=0x7fad2419e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:30.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.759+0000 7fad2ba0c700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad24102740 0x7fad2419e7c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fad2000d750 tx=0x7fad2000da60 comp rx=0 tx=0).stop 2024-06-09T22:31:30.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.759+0000 7fad2ba0c700 1 -- 172.21.15.104:0/2409589201 shutdown_connections 2024-06-09T22:31:30.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.759+0000 7fad2ba0c700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fad100a2d90 0x7fad100a5250 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.759+0000 7fad2ba0c700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad24107840 0x7fad24198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.759+0000 7fad2ba0c700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad24102740 0x7fad2419e7c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.759+0000 7fad2ba0c700 1 --2- 172.21.15.104:0/2409589201 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad241030f0 0x7fad2419ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.759+0000 7fad2ba0c700 1 -- 172.21.15.104:0/2409589201 >> 172.21.15.104:0/2409589201 conn(0x7fad240795f0 msgr2=0x7fad241006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.759+0000 7fad2ba0c700 1 -- 172.21.15.104:0/2409589201 shutdown_connections 2024-06-09T22:31:30.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:30.759+0000 7fad2ba0c700 1 -- 172.21.15.104:0/2409589201 wait complete. 2024-06-09T22:31:31.605 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-06-09T22:31:31.610 INFO:tasks.fwd_scrub:done joining 2024-06-09T22:31:31.611 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-06-09T22:31:31.622 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-09T22:31:31.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.726+0000 7fcb76d4f640 1 -- 172.21.15.104:0/2475935844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb701058e0 msgr2=0x7fcb70105cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:31.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.726+0000 7fcb76d4f640 1 --2- 172.21.15.104:0/2475935844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb701058e0 0x7fcb70105cc0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009930 tx=0x7fcb6402a4d0 comp rx=0 tx=0).stop 2024-06-09T22:31:31.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.726+0000 7fcb76d4f640 1 -- 172.21.15.104:0/2475935844 shutdown_connections 2024-06-09T22:31:31.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.726+0000 7fcb76d4f640 1 --2- 172.21.15.104:0/2475935844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb70101dc0 0x7fcb7010d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:31.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.726+0000 7fcb76d4f640 1 --2- 172.21.15.104:0/2475935844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb70101400 0x7fcb70101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:31.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.726+0000 7fcb76d4f640 1 --2- 172.21.15.104:0/2475935844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb701058e0 0x7fcb70105cc0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:31.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.726+0000 7fcb76d4f640 1 -- 172.21.15.104:0/2475935844 >> 172.21.15.104:0/2475935844 conn(0x7fcb70075b80 msgr2=0x7fcb70075f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:31.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.726+0000 7fcb76d4f640 1 -- 172.21.15.104:0/2475935844 shutdown_connections 2024-06-09T22:31:31.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.726+0000 7fcb76d4f640 1 -- 172.21.15.104:0/2475935844 wait complete. 2024-06-09T22:31:31.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb76d4f640 1 Processor -- start 2024-06-09T22:31:31.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb76d4f640 1 -- start start 2024-06-09T22:31:31.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb76d4f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb70101400 0x7fcb701990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:31.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb76d4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb70101dc0 0x7fcb701995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:31.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb76d4f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb701058e0 0x7fcb701a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:31.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb76d4f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcb70110ca0 con 0x7fcb701058e0 2024-06-09T22:31:31.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb76d4f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcb70110b20 con 0x7fcb70101dc0 2024-06-09T22:31:31.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb76d4f640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcb70110e20 con 0x7fcb70101400 2024-06-09T22:31:31.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb752ec640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb701058e0 0x7fcb701a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:31.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb752ec640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb701058e0 0x7fcb701a4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49832/0 (socket says 172.21.15.104:49832) 2024-06-09T22:31:31.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb752ec640 1 -- 172.21.15.104:0/1826680247 learned_addr learned my addr 172.21.15.104:0/1826680247 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:31.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb6ffff640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb70101dc0 0x7fcb701995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:31.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb74aeb640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb70101400 0x7fcb701990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:31.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb752ec640 1 -- 172.21.15.104:0/1826680247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb70101400 msgr2=0x7fcb701990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:31.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb752ec640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb70101400 0x7fcb701990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:31.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb752ec640 1 -- 172.21.15.104:0/1826680247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb70101dc0 msgr2=0x7fcb701995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:31.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb752ec640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb70101dc0 0x7fcb701995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:31.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb752ec640 1 -- 172.21.15.104:0/1826680247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb701a4660 con 0x7fcb701058e0 2024-06-09T22:31:31.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb6ffff640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb70101dc0 0x7fcb701995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:31:31.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb752ec640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb701058e0 0x7fcb701a4120 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800eaa0 tx=0x7fcb6800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:31.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.727+0000 7fcb6dffb640 1 -- 172.21.15.104:0/1826680247 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6800cdb0 con 0x7fcb701058e0 2024-06-09T22:31:31.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.728+0000 7fcb6dffb640 1 -- 172.21.15.104:0/1826680247 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb68005020 con 0x7fcb701058e0 2024-06-09T22:31:31.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.728+0000 7fcb76d4f640 1 -- 172.21.15.104:0/1826680247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb701a4830 con 0x7fcb701058e0 2024-06-09T22:31:31.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.728+0000 7fcb6dffb640 1 -- 172.21.15.104:0/1826680247 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb680116b0 con 0x7fcb701058e0 2024-06-09T22:31:31.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.728+0000 7fcb76d4f640 1 -- 172.21.15.104:0/1826680247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb701a4d00 con 0x7fcb701058e0 2024-06-09T22:31:31.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.729+0000 7fcb76d4f640 1 -- 172.21.15.104:0/1826680247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb70068c10 con 0x7fcb701058e0 2024-06-09T22:31:31.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.730+0000 7fcb6dffb640 1 -- 172.21.15.104:0/1826680247 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fcb68005560 con 0x7fcb701058e0 2024-06-09T22:31:31.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.730+0000 7fcb6dffb640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fcb600a2d00 0x7fcb600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:31.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.731+0000 7fcb74aeb640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fcb600a2d00 0x7fcb600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:31.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.731+0000 7fcb6dffb640 1 -- 172.21.15.104:0/1826680247 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7fcb680cfae0 con 0x7fcb701058e0 2024-06-09T22:31:31.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.731+0000 7fcb74aeb640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fcb600a2d00 0x7fcb600a51c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcb64006010 tx=0x7fcb640314b0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:31.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.735+0000 7fcb6dffb640 1 -- 172.21.15.104:0/1826680247 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fcb68016030 con 0x7fcb701058e0 2024-06-09T22:31:31.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:31 smithi194 ceph-mon[36856]: pgmap v327: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 15 op/s 2024-06-09T22:31:31.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.911+0000 7fcb76d4f640 1 -- 172.21.15.104:0/1826680247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb70108860 con 0x7fcb701058e0 2024-06-09T22:31:31.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.913+0000 7fcb6dffb640 1 -- 172.21.15.104:0/1826680247 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+25329 (secure 0 0 0) 0x7fcb6809c330 con 0x7fcb701058e0 2024-06-09T22:31:31.916 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:31:31.916 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":108,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:31:17.575356+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:0/2389242747":"2024-06-10T22:20:57.590784+0000","172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6801/1906048760":"2024-06-10T22:20:57.590784+0000","172.21.15.104:6800/1906048760":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/517954187":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/167857460":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/357628765":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:31:31.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.915+0000 7fcb76d4f640 1 -- 172.21.15.104:0/1826680247 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fcb600a2d00 msgr2=0x7fcb600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:31.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.915+0000 7fcb76d4f640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fcb600a2d00 0x7fcb600a51c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcb64006010 tx=0x7fcb640314b0 comp rx=0 tx=0).stop 2024-06-09T22:31:31.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.915+0000 7fcb76d4f640 1 -- 172.21.15.104:0/1826680247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb701058e0 msgr2=0x7fcb701a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:31.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.915+0000 7fcb76d4f640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb701058e0 0x7fcb701a4120 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800eaa0 tx=0x7fcb6800ef60 comp rx=0 tx=0).stop 2024-06-09T22:31:31.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.915+0000 7fcb76d4f640 1 -- 172.21.15.104:0/1826680247 shutdown_connections 2024-06-09T22:31:31.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.915+0000 7fcb76d4f640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fcb600a2d00 0x7fcb600a51c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:31.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.915+0000 7fcb76d4f640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb701058e0 0x7fcb701a4120 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:31.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.915+0000 7fcb76d4f640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb70101dc0 0x7fcb701995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:31.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.915+0000 7fcb76d4f640 1 --2- 172.21.15.104:0/1826680247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb70101400 0x7fcb701990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:31.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.915+0000 7fcb76d4f640 1 -- 172.21.15.104:0/1826680247 >> 172.21.15.104:0/1826680247 conn(0x7fcb70075b80 msgr2=0x7fcb700fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:31.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.915+0000 7fcb76d4f640 1 -- 172.21.15.104:0/1826680247 shutdown_connections 2024-06-09T22:31:31.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:31.915+0000 7fcb76d4f640 1 -- 172.21.15.104:0/1826680247 wait complete. 2024-06-09T22:31:31.928 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:31 smithi104 ceph-mon[32389]: pgmap v327: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 15 op/s 2024-06-09T22:31:31.929 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-09T22:11:49.547377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-09T22:16:38.369246+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-09T22:16:42.128973+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-09T22:16:51.549770+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-09T22:31:31.930 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-09T22:31:31.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:31 smithi153 ceph-mon[36599]: pgmap v327: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 15 op/s 2024-06-09T22:31:32.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.030+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1398706747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c4074590 msgr2=0x7f84c4112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.030+0000 7f84cbe4d640 1 --2- 172.21.15.104:0/1398706747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c4074590 0x7f84c4112510 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009930 tx=0x7f84b402a4d0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.030+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1398706747 shutdown_connections 2024-06-09T22:31:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.030+0000 7f84cbe4d640 1 --2- 172.21.15.104:0/1398706747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84c4112a50 0x7f84c4114e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.030+0000 7f84cbe4d640 1 --2- 172.21.15.104:0/1398706747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c4074590 0x7f84c4112510 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.030+0000 7f84cbe4d640 1 --2- 172.21.15.104:0/1398706747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4073c70 0x7f84c4074050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.030+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1398706747 >> 172.21.15.104:0/1398706747 conn(0x7f84c4078df0 msgr2=0x7f84c4079200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.031+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1398706747 shutdown_connections 2024-06-09T22:31:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.031+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1398706747 wait complete. 2024-06-09T22:31:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.031+0000 7f84cbe4d640 1 Processor -- start 2024-06-09T22:31:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.031+0000 7f84cbe4d640 1 -- start start 2024-06-09T22:31:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.031+0000 7f84cbe4d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84c4073c70 0x7f84c41a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.031+0000 7f84cbe4d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c4074590 0x7f84c41a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.031+0000 7f84cbe4d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4112a50 0x7f84c419e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.031+0000 7f84cbe4d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f84c4118370 con 0x7f84c4074590 2024-06-09T22:31:32.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.031+0000 7f84cbe4d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84c41181f0 con 0x7f84c4112a50 2024-06-09T22:31:32.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.031+0000 7f84cbe4d640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f84c41184f0 con 0x7f84c4073c70 2024-06-09T22:31:32.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.031+0000 7f84c93e8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c4074590 0x7f84c41a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84c93e8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c4074590 0x7f84c41a4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49846/0 (socket says 172.21.15.104:49846) 2024-06-09T22:31:32.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84c93e8640 1 -- 172.21.15.104:0/1432905473 learned_addr learned my addr 172.21.15.104:0/1432905473 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:32.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84c9be9640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84c4073c70 0x7f84c41a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84ca3ea640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4112a50 0x7f84c419e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84c93e8640 1 -- 172.21.15.104:0/1432905473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84c4073c70 msgr2=0x7f84c41a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84c93e8640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84c4073c70 0x7f84c41a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84c93e8640 1 -- 172.21.15.104:0/1432905473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4112a50 msgr2=0x7f84c419e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84c93e8640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4112a50 0x7f84c419e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84c93e8640 1 -- 172.21.15.104:0/1432905473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c419ef80 con 0x7f84c4074590 2024-06-09T22:31:32.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84c93e8640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c4074590 0x7f84c41a4b80 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009230 tx=0x7f84b4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:32.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84baffd640 1 -- 172.21.15.104:0/1432905473 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b403f030 con 0x7f84c4074590 2024-06-09T22:31:32.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1432905473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c419f210 con 0x7f84c4074590 2024-06-09T22:31:32.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84baffd640 1 -- 172.21.15.104:0/1432905473 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84b4003ad0 con 0x7f84c4074590 2024-06-09T22:31:32.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84baffd640 1 -- 172.21.15.104:0/1432905473 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b4035070 con 0x7f84c4074590 2024-06-09T22:31:32.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.032+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1432905473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c41ab490 con 0x7f84c4074590 2024-06-09T22:31:32.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.033+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1432905473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84c40753b0 con 0x7f84c4074590 2024-06-09T22:31:32.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.034+0000 7f84baffd640 1 -- 172.21.15.104:0/1432905473 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f84b4042030 con 0x7f84c4074590 2024-06-09T22:31:32.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.036+0000 7f84baffd640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84b00a2d00 0x7f84b00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.036+0000 7f84c9be9640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84b00a2d00 0x7f84b00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.036+0000 7f84baffd640 1 -- 172.21.15.104:0/1432905473 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f84b40f0260 con 0x7f84c4074590 2024-06-09T22:31:32.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.036+0000 7f84c9be9640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84b00a2d00 0x7f84b00a51c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f84c000bbe0 tx=0x7f84c000b390 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:32.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.039+0000 7f84baffd640 1 -- 172.21.15.104:0/1432905473 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f84b40bc400 con 0x7f84c4074590 2024-06-09T22:31:32.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.217+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1432905473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f84c4104ff0 con 0x7f84c4074590 2024-06-09T22:31:32.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.217+0000 7f84baffd640 1 -- 172.21.15.104:0/1432905473 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) ==== 93+0+10 (secure 0 0 0) 0x7f84b40c0700 con 0x7f84c4074590 2024-06-09T22:31:32.220 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-06-09T22:31:32.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.219+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1432905473 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84b00a2d00 msgr2=0x7f84b00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.219+0000 7f84cbe4d640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84b00a2d00 0x7f84b00a51c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f84c000bbe0 tx=0x7f84c000b390 comp rx=0 tx=0).stop 2024-06-09T22:31:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.219+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1432905473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c4074590 msgr2=0x7f84c41a4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.219+0000 7f84cbe4d640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c4074590 0x7f84c41a4b80 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009230 tx=0x7f84b4004270 comp rx=0 tx=0).stop 2024-06-09T22:31:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.219+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1432905473 shutdown_connections 2024-06-09T22:31:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.219+0000 7f84cbe4d640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f84b00a2d00 0x7f84b00a51c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.219+0000 7f84cbe4d640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c4112a50 0x7f84c419e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.219+0000 7f84cbe4d640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84c4074590 0x7f84c41a4b80 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.219+0000 7f84cbe4d640 1 --2- 172.21.15.104:0/1432905473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84c4073c70 0x7f84c41a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.219+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1432905473 >> 172.21.15.104:0/1432905473 conn(0x7f84c4078df0 msgr2=0x7f84c40ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:32.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.219+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1432905473 shutdown_connections 2024-06-09T22:31:32.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.219+0000 7f84cbe4d640 1 -- 172.21.15.104:0/1432905473 wait complete. 2024-06-09T22:31:32.234 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-09T22:31:32.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.335+0000 7f29c9cbd640 1 -- 172.21.15.104:0/3601484786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c4101240 msgr2=0x7f29c410d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.335+0000 7f29c9cbd640 1 --2- 172.21.15.104:0/3601484786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c4101240 0x7f29c410d750 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009cd0 tx=0x7f29bc02a8e0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.335+0000 7f29c9cbd640 1 -- 172.21.15.104:0/3601484786 shutdown_connections 2024-06-09T22:31:32.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.335+0000 7f29c9cbd640 1 --2- 172.21.15.104:0/3601484786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c4101240 0x7f29c410d750 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.335+0000 7f29c9cbd640 1 --2- 172.21.15.104:0/3601484786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c4100880 0x7f29c4100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.335+0000 7f29c9cbd640 1 --2- 172.21.15.104:0/3601484786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c4106890 0x7f29c4106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.335+0000 7f29c9cbd640 1 -- 172.21.15.104:0/3601484786 >> 172.21.15.104:0/3601484786 conn(0x7f29c4077900 msgr2=0x7f29c4077d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:32.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.335+0000 7f29c9cbd640 1 -- 172.21.15.104:0/3601484786 shutdown_connections 2024-06-09T22:31:32.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.335+0000 7f29c9cbd640 1 -- 172.21.15.104:0/3601484786 wait complete. 2024-06-09T22:31:32.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.335+0000 7f29c9cbd640 1 Processor -- start 2024-06-09T22:31:32.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c9cbd640 1 -- start start 2024-06-09T22:31:32.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c9cbd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c4100880 0x7f29c41990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c9cbd640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c4101240 0x7f29c4199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c9cbd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c4106890 0x7f29c41a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c9cbd640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f29c4110ca0 con 0x7f29c4106890 2024-06-09T22:31:32.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c9cbd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29c4110b20 con 0x7f29c4100880 2024-06-09T22:31:32.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c9cbd640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f29c4110e20 con 0x7f29c4101240 2024-06-09T22:31:32.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c4106890 0x7f29c41a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c4106890 0x7f29c41a4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49866/0 (socket says 172.21.15.104:49866) 2024-06-09T22:31:32.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c4100880 0x7f29c41990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c3fff640 1 -- 172.21.15.104:0/2327054105 learned_addr learned my addr 172.21.15.104:0/2327054105 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:32.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c2ffd640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c4101240 0x7f29c4199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c3fff640 1 -- 172.21.15.104:0/2327054105 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c4101240 msgr2=0x7f29c4199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c3fff640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c4101240 0x7f29c4199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c3fff640 1 -- 172.21.15.104:0/2327054105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c4100880 msgr2=0x7f29c41990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c3fff640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c4100880 0x7f29c41990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c3fff640 1 -- 172.21.15.104:0/2327054105 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c41a46a0 con 0x7f29c4106890 2024-06-09T22:31:32.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c2ffd640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c4101240 0x7f29c4199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:31:32.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.336+0000 7f29c3fff640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c4106890 0x7f29c41a4160 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f29bc005cc0 tx=0x7f29bc003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:32.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.337+0000 7f29c0ff9640 1 -- 172.21.15.104:0/2327054105 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc004040 con 0x7f29c4106890 2024-06-09T22:31:32.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.337+0000 7f29c9cbd640 1 -- 172.21.15.104:0/2327054105 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c41a4870 con 0x7f29c4106890 2024-06-09T22:31:32.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.337+0000 7f29c0ff9640 1 -- 172.21.15.104:0/2327054105 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29bc0046c0 con 0x7f29c4106890 2024-06-09T22:31:32.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.337+0000 7f29c0ff9640 1 -- 172.21.15.104:0/2327054105 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc046480 con 0x7f29c4106890 2024-06-09T22:31:32.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.337+0000 7f29c9cbd640 1 -- 172.21.15.104:0/2327054105 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c41a4cd0 con 0x7f29c4106890 2024-06-09T22:31:32.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.338+0000 7f29c9cbd640 1 -- 172.21.15.104:0/2327054105 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c4068c10 con 0x7f29c4106890 2024-06-09T22:31:32.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.339+0000 7f29c0ff9640 1 -- 172.21.15.104:0/2327054105 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f29bc0305a0 con 0x7f29c4106890 2024-06-09T22:31:32.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.339+0000 7f29c0ff9640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f29ac0a2d00 0x7f29ac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.339+0000 7f29c37fe640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f29ac0a2d00 0x7f29ac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.339+0000 7f29c0ff9640 1 -- 172.21.15.104:0/2327054105 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f29bc02e030 con 0x7f29c4106890 2024-06-09T22:31:32.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.340+0000 7f29c37fe640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f29ac0a2d00 0x7f29ac0a51c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f29b400bb60 tx=0x7f29b400b390 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:32.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.344+0000 7f29c0ff9640 1 -- 172.21.15.104:0/2327054105 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f29bc04b060 con 0x7f29c4106890 2024-06-09T22:31:32.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.518+0000 7f29c9cbd640 1 -- 172.21.15.104:0/2327054105 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f29c41a4fd0 con 0x7f29c4106890 2024-06-09T22:31:32.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.518+0000 7f29c0ff9640 1 -- 172.21.15.104:0/2327054105 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) ==== 104+0+11 (secure 0 0 0) 0x7f29bc0bd8a0 con 0x7f29c4106890 2024-06-09T22:31:32.522 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-06-09T22:31:32.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.520+0000 7f29c9cbd640 1 -- 172.21.15.104:0/2327054105 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f29ac0a2d00 msgr2=0x7f29ac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.520+0000 7f29c9cbd640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f29ac0a2d00 0x7f29ac0a51c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f29b400bb60 tx=0x7f29b400b390 comp rx=0 tx=0).stop 2024-06-09T22:31:32.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.520+0000 7f29c9cbd640 1 -- 172.21.15.104:0/2327054105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c4106890 msgr2=0x7f29c41a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.520+0000 7f29c9cbd640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c4106890 0x7f29c41a4160 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f29bc005cc0 tx=0x7f29bc003970 comp rx=0 tx=0).stop 2024-06-09T22:31:32.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.520+0000 7f29c9cbd640 1 -- 172.21.15.104:0/2327054105 shutdown_connections 2024-06-09T22:31:32.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.520+0000 7f29c9cbd640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f29ac0a2d00 0x7f29ac0a51c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.520+0000 7f29c9cbd640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29c4106890 0x7f29c41a4160 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.520+0000 7f29c9cbd640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29c4101240 0x7f29c4199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.520+0000 7f29c9cbd640 1 --2- 172.21.15.104:0/2327054105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c4100880 0x7f29c41990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.520+0000 7f29c9cbd640 1 -- 172.21.15.104:0/2327054105 >> 172.21.15.104:0/2327054105 conn(0x7f29c4077900 msgr2=0x7f29c4103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:32.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.520+0000 7f29c9cbd640 1 -- 172.21.15.104:0/2327054105 shutdown_connections 2024-06-09T22:31:32.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.520+0000 7f29c9cbd640 1 -- 172.21.15.104:0/2327054105 wait complete. 2024-06-09T22:31:32.534 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-09T22:31:32.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.637+0000 7f2eccda5640 1 -- 172.21.15.104:0/997887828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ec80ffea0 msgr2=0x7f2ec8100320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.637+0000 7f2eccda5640 1 --2- 172.21.15.104:0/997887828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ec80ffea0 0x7f2ec8100320 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009930 tx=0x7f2eb002a420 comp rx=0 tx=0).stop 2024-06-09T22:31:32.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.637+0000 7f2eccda5640 1 -- 172.21.15.104:0/997887828 shutdown_connections 2024-06-09T22:31:32.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.637+0000 7f2eccda5640 1 --2- 172.21.15.104:0/997887828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ec8100860 0x7f2ec810e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.637+0000 7f2eccda5640 1 --2- 172.21.15.104:0/997887828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ec80ffea0 0x7f2ec8100320 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.637+0000 7f2eccda5640 1 --2- 172.21.15.104:0/997887828 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ec8106980 0x7f2ec8106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.637+0000 7f2eccda5640 1 -- 172.21.15.104:0/997887828 >> 172.21.15.104:0/997887828 conn(0x7f2ec8076e30 msgr2=0x7f2ec80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:32.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.637+0000 7f2eccda5640 1 -- 172.21.15.104:0/997887828 shutdown_connections 2024-06-09T22:31:32.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.637+0000 7f2eccda5640 1 -- 172.21.15.104:0/997887828 wait complete. 2024-06-09T22:31:32.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2eccda5640 1 Processor -- start 2024-06-09T22:31:32.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2eccda5640 1 -- start start 2024-06-09T22:31:32.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2eccda5640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ec80ffea0 0x7f2ec8198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2eccda5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ec8100860 0x7f2ec8198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2eccda5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ec8106980 0x7f2ec819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2eccda5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec8111e90 con 0x7f2ec8100860 2024-06-09T22:31:32.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2eccda5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec8111d10 con 0x7f2ec8106980 2024-06-09T22:31:32.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2eccda5640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec8112010 con 0x7f2ec80ffea0 2024-06-09T22:31:32.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2ebffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ec8100860 0x7f2ec8198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2ebffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ec8100860 0x7f2ec8198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49888/0 (socket says 172.21.15.104:49888) 2024-06-09T22:31:32.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2ebffff640 1 -- 172.21.15.104:0/4225330224 learned_addr learned my addr 172.21.15.104:0/4225330224 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:32.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2ec6d9d640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ec8106980 0x7f2ec819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.638+0000 7f2ec659c640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ec80ffea0 0x7f2ec8198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.639+0000 7f2ebffff640 1 -- 172.21.15.104:0/4225330224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ec80ffea0 msgr2=0x7f2ec8198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.639+0000 7f2ebffff640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ec80ffea0 0x7f2ec8198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.639+0000 7f2ebffff640 1 -- 172.21.15.104:0/4225330224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ec8106980 msgr2=0x7f2ec819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.639+0000 7f2ebffff640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ec8106980 0x7f2ec819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.639+0000 7f2ebffff640 1 -- 172.21.15.104:0/4225330224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec819d300 con 0x7f2ec8100860 2024-06-09T22:31:32.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.639+0000 7f2ebffff640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ec8100860 0x7f2ec8198850 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009a50 tx=0x7f2eb0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:32.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.639+0000 7f2ebf7fe640 1 -- 172.21.15.104:0/4225330224 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb003f030 con 0x7f2ec8100860 2024-06-09T22:31:32.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.639+0000 7f2eccda5640 1 -- 172.21.15.104:0/4225330224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec819d590 con 0x7f2ec8100860 2024-06-09T22:31:32.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.639+0000 7f2ebf7fe640 1 -- 172.21.15.104:0/4225330224 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eb0003ad0 con 0x7f2ec8100860 2024-06-09T22:31:32.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.639+0000 7f2ebf7fe640 1 -- 172.21.15.104:0/4225330224 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0035070 con 0x7f2ec8100860 2024-06-09T22:31:32.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.639+0000 7f2eccda5640 1 -- 172.21.15.104:0/4225330224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec81a4e80 con 0x7f2ec8100860 2024-06-09T22:31:32.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.640+0000 7f2eccda5640 1 -- 172.21.15.104:0/4225330224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ec8068c10 con 0x7f2ec8100860 2024-06-09T22:31:32.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.641+0000 7f2ebf7fe640 1 -- 172.21.15.104:0/4225330224 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f2eb0042030 con 0x7f2ec8100860 2024-06-09T22:31:32.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.642+0000 7f2ebf7fe640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2eb40a2d00 0x7f2eb40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.642+0000 7f2ec659c640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2eb40a2d00 0x7f2eb40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.642+0000 7f2ebf7fe640 1 -- 172.21.15.104:0/4225330224 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f2eb00eff30 con 0x7f2ec8100860 2024-06-09T22:31:32.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.642+0000 7f2ec659c640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2eb40a2d00 0x7f2eb40a51c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009990 tx=0x7f2eb8009110 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:32.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.646+0000 7f2ebf7fe640 1 -- 172.21.15.104:0/4225330224 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2eb00bc780 con 0x7f2ec8100860 2024-06-09T22:31:32.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:32 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1826680247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:31:32.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:32 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1432905473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-09T22:31:32.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:32 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2327054105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-09T22:31:32.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.824+0000 7f2eccda5640 1 -- 172.21.15.104:0/4225330224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2ec8109e60 con 0x7f2ec8100860 2024-06-09T22:31:32.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.825+0000 7f2ebf7fe640 1 -- 172.21.15.104:0/4225330224 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) ==== 100+0+11 (secure 0 0 0) 0x7f2eb00bc080 con 0x7f2ec8100860 2024-06-09T22:31:32.828 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-06-09T22:31:32.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.827+0000 7f2eccda5640 1 -- 172.21.15.104:0/4225330224 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2eb40a2d00 msgr2=0x7f2eb40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.827+0000 7f2eccda5640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2eb40a2d00 0x7f2eb40a51c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009990 tx=0x7f2eb8009110 comp rx=0 tx=0).stop 2024-06-09T22:31:32.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.827+0000 7f2eccda5640 1 -- 172.21.15.104:0/4225330224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ec8100860 msgr2=0x7f2ec8198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.827+0000 7f2eccda5640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ec8100860 0x7f2ec8198850 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009a50 tx=0x7f2eb0004270 comp rx=0 tx=0).stop 2024-06-09T22:31:32.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.827+0000 7f2eccda5640 1 -- 172.21.15.104:0/4225330224 shutdown_connections 2024-06-09T22:31:32.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.827+0000 7f2eccda5640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f2eb40a2d00 0x7f2eb40a51c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.827+0000 7f2eccda5640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ec8106980 0x7f2ec819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.827+0000 7f2eccda5640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ec8100860 0x7f2ec8198850 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.827+0000 7f2eccda5640 1 --2- 172.21.15.104:0/4225330224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ec80ffea0 0x7f2ec8198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.827+0000 7f2eccda5640 1 -- 172.21.15.104:0/4225330224 >> 172.21.15.104:0/4225330224 conn(0x7f2ec8076e30 msgr2=0x7f2ec81049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:32.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.827+0000 7f2eccda5640 1 -- 172.21.15.104:0/4225330224 shutdown_connections 2024-06-09T22:31:32.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.827+0000 7f2eccda5640 1 -- 172.21.15.104:0/4225330224 wait complete. 2024-06-09T22:31:32.840 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:32 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1826680247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:31:32.840 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:32 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1432905473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-09T22:31:32.840 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:32 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2327054105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-09T22:31:32.841 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-09T22:31:32.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.941+0000 7f0f32c15640 1 -- 172.21.15.104:0/3044282174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c100870 msgr2=0x7f0f2c100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.941+0000 7f0f32c15640 1 --2- 172.21.15.104:0/3044282174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c100870 0x7f0f2c100cf0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009930 tx=0x7f0f1802a420 comp rx=0 tx=0).stop 2024-06-09T22:31:32.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 -- 172.21.15.104:0/3044282174 shutdown_connections 2024-06-09T22:31:32.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 --2- 172.21.15.104:0/3044282174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f2c101230 0x7f0f2c10d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 --2- 172.21.15.104:0/3044282174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c100870 0x7f0f2c100cf0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 --2- 172.21.15.104:0/3044282174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f2c106880 0x7f0f2c106c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 -- 172.21.15.104:0/3044282174 >> 172.21.15.104:0/3044282174 conn(0x7f0f2c0779a0 msgr2=0x7f0f2c077db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:32.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 -- 172.21.15.104:0/3044282174 shutdown_connections 2024-06-09T22:31:32.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 -- 172.21.15.104:0/3044282174 wait complete. 2024-06-09T22:31:32.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 Processor -- start 2024-06-09T22:31:32.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 -- start start 2024-06-09T22:31:32.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f2c100870 0x7f0f2c1972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c101230 0x7f0f2c197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f2c106880 0x7f0f2c19bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c110d10 con 0x7f0f2c101230 2024-06-09T22:31:32.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c110b90 con 0x7f0f2c100870 2024-06-09T22:31:32.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f32c15640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c110e90 con 0x7f0f2c106880 2024-06-09T22:31:32.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.942+0000 7f0f2bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c101230 0x7f0f2c197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f2bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c101230 0x7f0f2c197830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49908/0 (socket says 172.21.15.104:49908) 2024-06-09T22:31:32.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f2bfff640 1 -- 172.21.15.104:0/474200589 learned_addr learned my addr 172.21.15.104:0/474200589 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:32.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f309b1640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f2c100870 0x7f0f2c1972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f311b2640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f2c106880 0x7f0f2c19bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f2bfff640 1 -- 172.21.15.104:0/474200589 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f2c106880 msgr2=0x7f0f2c19bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f2bfff640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f2c106880 0x7f0f2c19bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f2bfff640 1 -- 172.21.15.104:0/474200589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f2c100870 msgr2=0x7f0f2c1972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:32.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f2bfff640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f2c100870 0x7f0f2c1972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:32.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f2bfff640 1 -- 172.21.15.104:0/474200589 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2c19c2e0 con 0x7f0f2c101230 2024-06-09T22:31:32.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f309b1640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f2c100870 0x7f0f2c1972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:31:32.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f2bfff640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c101230 0x7f0f2c197830 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009a50 tx=0x7f0f18004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:32.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f29ffb640 1 -- 172.21.15.104:0/474200589 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1803f030 con 0x7f0f2c101230 2024-06-09T22:31:32.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f29ffb640 1 -- 172.21.15.104:0/474200589 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f18003ce0 con 0x7f0f2c101230 2024-06-09T22:31:32.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f32c15640 1 -- 172.21.15.104:0/474200589 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f2c19c570 con 0x7f0f2c101230 2024-06-09T22:31:32.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f29ffb640 1 -- 172.21.15.104:0/474200589 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f18035070 con 0x7f0f2c101230 2024-06-09T22:31:32.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.943+0000 7f0f32c15640 1 -- 172.21.15.104:0/474200589 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f2c1a3e60 con 0x7f0f2c101230 2024-06-09T22:31:32.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.945+0000 7f0f32c15640 1 -- 172.21.15.104:0/474200589 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f2c101fb0 con 0x7f0f2c101230 2024-06-09T22:31:32.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.946+0000 7f0f29ffb640 1 -- 172.21.15.104:0/474200589 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f0f18042030 con 0x7f0f2c101230 2024-06-09T22:31:32.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.946+0000 7f0f29ffb640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0f1c0a2d00 0x7f0f1c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:32.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.946+0000 7f0f309b1640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0f1c0a2d00 0x7f0f1c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:32.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.946+0000 7f0f29ffb640 1 -- 172.21.15.104:0/474200589 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f0f180f0280 con 0x7f0f2c101230 2024-06-09T22:31:32.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.947+0000 7f0f309b1640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0f1c0a2d00 0x7f0f1c0a51c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0f20009930 tx=0x7f0f20008040 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:32.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:32.950+0000 7f0f29ffb640 1 -- 172.21.15.104:0/474200589 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0f180bcad0 con 0x7f0f2c101230 2024-06-09T22:31:32.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:32 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1826680247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:31:32.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:32 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1432905473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-09T22:31:32.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:32 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2327054105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-09T22:31:33.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.124+0000 7f0f32c15640 1 -- 172.21.15.104:0/474200589 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0f2c108e30 con 0x7f0f2c101230 2024-06-09T22:31:33.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.125+0000 7f0f29ffb640 1 -- 172.21.15.104:0/474200589 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v108) ==== 103+0+11 (secure 0 0 0) 0x7f0f180bc3d0 con 0x7f0f2c101230 2024-06-09T22:31:33.128 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-06-09T22:31:33.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.127+0000 7f0f32c15640 1 -- 172.21.15.104:0/474200589 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0f1c0a2d00 msgr2=0x7f0f1c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:33.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.127+0000 7f0f32c15640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0f1c0a2d00 0x7f0f1c0a51c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0f20009930 tx=0x7f0f20008040 comp rx=0 tx=0).stop 2024-06-09T22:31:33.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.127+0000 7f0f32c15640 1 -- 172.21.15.104:0/474200589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c101230 msgr2=0x7f0f2c197830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:33.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.127+0000 7f0f32c15640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c101230 0x7f0f2c197830 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009a50 tx=0x7f0f18004270 comp rx=0 tx=0).stop 2024-06-09T22:31:33.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.127+0000 7f0f32c15640 1 -- 172.21.15.104:0/474200589 shutdown_connections 2024-06-09T22:31:33.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.127+0000 7f0f32c15640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f0f1c0a2d00 0x7f0f1c0a51c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:33.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.127+0000 7f0f32c15640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f2c106880 0x7f0f2c19bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:33.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.127+0000 7f0f32c15640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c101230 0x7f0f2c197830 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:33.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.127+0000 7f0f32c15640 1 --2- 172.21.15.104:0/474200589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f2c100870 0x7f0f2c1972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:33.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.127+0000 7f0f32c15640 1 -- 172.21.15.104:0/474200589 >> 172.21.15.104:0/474200589 conn(0x7f0f2c0779a0 msgr2=0x7f0f2c103290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:33.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.127+0000 7f0f32c15640 1 -- 172.21.15.104:0/474200589 shutdown_connections 2024-06-09T22:31:33.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.127+0000 7f0f32c15640 1 -- 172.21.15.104:0/474200589 wait complete. 2024-06-09T22:31:33.141 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-06-09T22:31:33.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.238+0000 7fd66a86b640 1 -- 172.21.15.104:0/4156166913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd664102c30 msgr2=0x7fd664103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:33.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.238+0000 7fd66a86b640 1 --2- 172.21.15.104:0/4156166913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd664102c30 0x7fd664103010 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fd654009900 tx=0x7fd65402a510 comp rx=0 tx=0).stop 2024-06-09T22:31:33.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 -- 172.21.15.104:0/4156166913 shutdown_connections 2024-06-09T22:31:33.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 --2- 172.21.15.104:0/4156166913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66410bf70 0x7fd66410e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:33.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 --2- 172.21.15.104:0/4156166913 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd664103550 0x7fd66410ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:33.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 --2- 172.21.15.104:0/4156166913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd664102c30 0x7fd664103010 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:33.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 -- 172.21.15.104:0/4156166913 >> 172.21.15.104:0/4156166913 conn(0x7fd6640779a0 msgr2=0x7fd664077db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:33.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 -- 172.21.15.104:0/4156166913 shutdown_connections 2024-06-09T22:31:33.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 -- 172.21.15.104:0/4156166913 wait complete. 2024-06-09T22:31:33.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 Processor -- start 2024-06-09T22:31:33.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 -- start start 2024-06-09T22:31:33.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd664102c30 0x7fd664198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:33.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd664103550 0x7fd6641987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:33.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd66410bf70 0x7fd66419cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:33.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd664111d70 con 0x7fd66410bf70 2024-06-09T22:31:33.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd664111bf0 con 0x7fd664102c30 2024-06-09T22:31:33.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.239+0000 7fd66a86b640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd664111ef0 con 0x7fd664103550 2024-06-09T22:31:33.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd668e08640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd66410bf70 0x7fd66419cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:33.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd668e08640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd66410bf70 0x7fd66419cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49920/0 (socket says 172.21.15.104:49920) 2024-06-09T22:31:33.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd668e08640 1 -- 172.21.15.104:0/1575185266 learned_addr learned my addr 172.21.15.104:0/1575185266 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:31:33.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd663fff640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd664102c30 0x7fd664198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:33.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd6637fe640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd664103550 0x7fd6641987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:33.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd668e08640 1 -- 172.21.15.104:0/1575185266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd664103550 msgr2=0x7fd6641987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:33.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd668e08640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd664103550 0x7fd6641987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:33.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd668e08640 1 -- 172.21.15.104:0/1575185266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd664102c30 msgr2=0x7fd664198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:33.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd668e08640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd664102c30 0x7fd664198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:33.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd668e08640 1 -- 172.21.15.104:0/1575185266 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66419d260 con 0x7fd66410bf70 2024-06-09T22:31:33.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd668e08640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd66410bf70 0x7fd66419cb30 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00eaa0 tx=0x7fd65c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:33.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd6617fa640 1 -- 172.21.15.104:0/1575185266 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c00cdb0 con 0x7fd66410bf70 2024-06-09T22:31:33.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd6617fa640 1 -- 172.21.15.104:0/1575185266 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65c005020 con 0x7fd66410bf70 2024-06-09T22:31:33.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd66a86b640 1 -- 172.21.15.104:0/1575185266 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66419d550 con 0x7fd66410bf70 2024-06-09T22:31:33.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd6617fa640 1 -- 172.21.15.104:0/1575185266 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c0116b0 con 0x7fd66410bf70 2024-06-09T22:31:33.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.240+0000 7fd66a86b640 1 -- 172.21.15.104:0/1575185266 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6641a4de0 con 0x7fd66410bf70 2024-06-09T22:31:33.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.241+0000 7fd66a86b640 1 -- 172.21.15.104:0/1575185266 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6640687c0 con 0x7fd66410bf70 2024-06-09T22:31:33.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.242+0000 7fd6617fa640 1 -- 172.21.15.104:0/1575185266 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fd65c005560 con 0x7fd66410bf70 2024-06-09T22:31:33.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.243+0000 7fd6617fa640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd64c0a2d00 0x7fd64c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:31:33.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.243+0000 7fd663fff640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd64c0a2d00 0x7fd64c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:31:33.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.243+0000 7fd6617fa640 1 -- 172.21.15.104:0/1575185266 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7fd65c0cf980 con 0x7fd66410bf70 2024-06-09T22:31:33.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.243+0000 7fd663fff640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd64c0a2d00 0x7fd64c0a51c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd654009230 tx=0x7fd6540058d0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:31:33.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.247+0000 7fd6617fa640 1 -- 172.21.15.104:0/1575185266 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd65c016030 con 0x7fd66410bf70 2024-06-09T22:31:33.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.474+0000 7fd66a86b640 1 -- 172.21.15.104:0/1575185266 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fd664106b60 con 0x7fd66410bf70 2024-06-09T22:31:33.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.483+0000 7fd6617fa640 1 -- 172.21.15.104:0/1575185266 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+740911 (secure 0 0 0) 0x7fd65c09bb80 con 0x7fd66410bf70 2024-06-09T22:31:33.495 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:31:33.495 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:31:33.495 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 25, 2024-06-09T22:31:33.495 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.495 INFO:teuthology.orchestra.run.smithi104.stdout: "active_gid": 15279, 2024-06-09T22:31:33.495 INFO:teuthology.orchestra.run.smithi104.stdout: "active_name": "x", 2024-06-09T22:31:33.495 INFO:teuthology.orchestra.run.smithi104.stdout: "active_addrs": { 2024-06-09T22:31:33.496 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-06-09T22:31:33.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.496 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-06-09T22:31:33.496 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6800", 2024-06-09T22:31:33.496 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 493021072 2024-06-09T22:31:33.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.496 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-06-09T22:31:33.497 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6801", 2024-06-09T22:31:33.497 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 493021072 2024-06-09T22:31:33.497 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.497 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-06-09T22:31:33.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.497 INFO:teuthology.orchestra.run.smithi104.stdout: "active_addr": "172.21.15.104:6801/493021072", 2024-06-09T22:31:33.497 INFO:teuthology.orchestra.run.smithi104.stdout: "active_change": "2024-06-09T22:20:57.591008+0000", 2024-06-09T22:31:33.497 INFO:teuthology.orchestra.run.smithi104.stdout: "active_mgr_features": 4540701547738038271, 2024-06-09T22:31:33.497 INFO:teuthology.orchestra.run.smithi104.stdout: "available": true, 2024-06-09T22:31:33.498 INFO:teuthology.orchestra.run.smithi104.stdout: "standbys": [ 2024-06-09T22:31:33.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.498 INFO:teuthology.orchestra.run.smithi104.stdout: "gid": 24881, 2024-06-09T22:31:33.498 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "y", 2024-06-09T22:31:33.498 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr_features": 4540701547738038271, 2024-06-09T22:31:33.498 INFO:teuthology.orchestra.run.smithi104.stdout: "available_modules": [ 2024-06-09T22:31:33.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.498 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "alerts", 2024-06-09T22:31:33.499 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.499 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.499 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.499 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:33.499 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:33.499 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.499 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.499 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.500 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:33.500 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.500 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.500 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.500 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "How frequently to reexamine health status", 2024-06-09T22:31:33.500 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.500 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.500 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.501 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.501 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.501 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.501 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.501 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.501 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.501 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.501 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.502 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.502 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.502 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.502 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.502 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.502 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.502 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.502 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.502 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.503 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.503 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.503 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.503 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.503 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.503 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.503 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.504 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.504 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.504 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.504 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.504 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.504 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.504 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.504 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.504 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.505 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.505 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.505 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.505 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.505 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.505 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.505 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.506 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.506 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.506 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.506 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.506 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.506 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.506 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.506 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.507 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.507 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.507 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.507 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.507 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.507 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.507 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.507 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.508 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.508 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.508 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.508 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.508 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.508 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.508 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.508 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.508 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.509 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.509 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_destination": { 2024-06-09T22:31:33.509 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_destination", 2024-06-09T22:31:33.509 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.509 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.509 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.509 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.510 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.510 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.510 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.510 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Email address to send alerts to", 2024-06-09T22:31:33.510 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.510 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.510 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.511 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_from_name": { 2024-06-09T22:31:33.511 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_from_name", 2024-06-09T22:31:33.511 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.511 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.511 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.511 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "Ceph", 2024-06-09T22:31:33.511 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.511 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.511 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.512 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Email From: name", 2024-06-09T22:31:33.512 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.512 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.512 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.512 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_host": { 2024-06-09T22:31:33.512 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_host", 2024-06-09T22:31:33.512 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.513 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.513 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.513 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.513 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.513 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.513 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.513 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "SMTP server", 2024-06-09T22:31:33.513 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.514 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.514 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.514 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_password": { 2024-06-09T22:31:33.514 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_password", 2024-06-09T22:31:33.514 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.514 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.514 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.515 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.515 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.515 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.515 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.515 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Password to authenticate with", 2024-06-09T22:31:33.515 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.515 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.515 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.516 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_port": { 2024-06-09T22:31:33.516 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_port", 2024-06-09T22:31:33.516 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.516 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.516 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.516 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "465", 2024-06-09T22:31:33.516 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.516 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.517 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.517 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "SMTP port", 2024-06-09T22:31:33.517 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.517 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.517 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.517 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_sender": { 2024-06-09T22:31:33.517 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_sender", 2024-06-09T22:31:33.518 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.518 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.518 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.518 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.518 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.518 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.518 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.518 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "SMTP envelope sender", 2024-06-09T22:31:33.519 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.519 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.519 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.519 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_ssl": { 2024-06-09T22:31:33.519 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_ssl", 2024-06-09T22:31:33.519 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.520 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.520 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.520 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.520 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.520 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.520 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.520 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Use SSL to connect to SMTP server", 2024-06-09T22:31:33.520 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.520 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.521 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.521 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_user": { 2024-06-09T22:31:33.521 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_user", 2024-06-09T22:31:33.521 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.521 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.521 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.521 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.522 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.522 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.522 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.522 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "User to authenticate as", 2024-06-09T22:31:33.522 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.522 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.522 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.522 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.523 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.523 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "balancer", 2024-06-09T22:31:33.523 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.523 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.523 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.523 INFO:teuthology.orchestra.run.smithi104.stdout: "active": { 2024-06-09T22:31:33.523 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "active", 2024-06-09T22:31:33.524 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.524 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.524 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.524 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.524 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.524 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.524 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.524 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "automatically balance PGs across cluster", 2024-06-09T22:31:33.524 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.525 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.525 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.525 INFO:teuthology.orchestra.run.smithi104.stdout: "begin_time": { 2024-06-09T22:31:33.525 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "begin_time", 2024-06-09T22:31:33.525 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.525 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.525 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.526 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0000", 2024-06-09T22:31:33.526 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.526 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.526 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.526 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "beginning time of day to automatically balance", 2024-06-09T22:31:33.526 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-09T22:31:33.526 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.526 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.527 INFO:teuthology.orchestra.run.smithi104.stdout: "begin_weekday": { 2024-06-09T22:31:33.527 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "begin_weekday", 2024-06-09T22:31:33.527 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:33.527 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.527 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.527 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:33.527 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "0", 2024-06-09T22:31:33.528 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "6", 2024-06-09T22:31:33.528 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.528 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-06-09T22:31:33.528 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-09T22:31:33.528 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.528 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.528 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_compat_max_iterations": { 2024-06-09T22:31:33.529 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crush_compat_max_iterations", 2024-06-09T22:31:33.529 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:33.529 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.529 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.529 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "25", 2024-06-09T22:31:33.529 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1", 2024-06-09T22:31:33.529 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "250", 2024-06-09T22:31:33.529 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.530 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-06-09T22:31:33.530 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.530 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.530 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.530 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_compat_metrics": { 2024-06-09T22:31:33.530 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crush_compat_metrics", 2024-06-09T22:31:33.530 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.530 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.531 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.531 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "pgs,objects,bytes", 2024-06-09T22:31:33.531 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.531 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.531 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.531 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-06-09T22:31:33.531 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-06-09T22:31:33.531 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.532 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.532 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_compat_step": { 2024-06-09T22:31:33.532 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crush_compat_step", 2024-06-09T22:31:33.532 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:33.532 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.532 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.532 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0.5", 2024-06-09T22:31:33.533 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "0.001", 2024-06-09T22:31:33.533 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "0.999", 2024-06-09T22:31:33.533 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.533 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "aggressiveness of optimization", 2024-06-09T22:31:33.533 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-06-09T22:31:33.533 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.533 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.533 INFO:teuthology.orchestra.run.smithi104.stdout: "end_time": { 2024-06-09T22:31:33.534 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "end_time", 2024-06-09T22:31:33.534 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.534 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.534 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.534 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "2359", 2024-06-09T22:31:33.534 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.534 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.534 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.535 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "ending time of day to automatically balance", 2024-06-09T22:31:33.535 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-09T22:31:33.535 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.535 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.535 INFO:teuthology.orchestra.run.smithi104.stdout: "end_weekday": { 2024-06-09T22:31:33.535 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "end_weekday", 2024-06-09T22:31:33.535 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:33.536 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.536 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.536 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:33.536 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "0", 2024-06-09T22:31:33.536 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "6", 2024-06-09T22:31:33.536 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.536 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-06-09T22:31:33.536 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-09T22:31:33.537 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.537 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.537 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.537 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.537 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.537 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.537 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.537 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.538 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.538 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.538 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.538 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.538 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.538 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.538 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.539 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.539 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.539 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.539 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.539 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.539 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.539 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.539 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.542 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.542 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.542 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.542 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.542 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.542 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.542 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.544 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.544 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.544 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.544 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.544 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.544 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.544 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.545 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.545 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.545 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.545 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.545 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.545 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.545 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.545 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.546 INFO:teuthology.orchestra.run.smithi104.stdout: "min_score": { 2024-06-09T22:31:33.546 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "min_score", 2024-06-09T22:31:33.546 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:33.546 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.546 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.546 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:33.546 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.546 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.547 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.547 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-06-09T22:31:33.547 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.547 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.547 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.547 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": { 2024-06-09T22:31:33.547 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mode", 2024-06-09T22:31:33.548 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.548 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.548 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.548 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "upmap", 2024-06-09T22:31:33.548 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.548 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.548 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.549 INFO:teuthology.orchestra.run.smithi104.stdout: "crush-compat", 2024-06-09T22:31:33.549 INFO:teuthology.orchestra.run.smithi104.stdout: "none", 2024-06-09T22:31:33.549 INFO:teuthology.orchestra.run.smithi104.stdout: "read", 2024-06-09T22:31:33.549 INFO:teuthology.orchestra.run.smithi104.stdout: "upmap", 2024-06-09T22:31:33.549 INFO:teuthology.orchestra.run.smithi104.stdout: "upmap-read" 2024-06-09T22:31:33.549 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.549 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Balancer mode", 2024-06-09T22:31:33.549 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.550 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.550 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.550 INFO:teuthology.orchestra.run.smithi104.stdout: "pool_ids": { 2024-06-09T22:31:33.550 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "pool_ids", 2024-06-09T22:31:33.550 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.550 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.551 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.551 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.551 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.551 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.551 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.551 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-06-09T22:31:33.551 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.551 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.552 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.552 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:33.552 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:33.552 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.552 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.552 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.552 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:33.553 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.553 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.553 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.553 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-06-09T22:31:33.553 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.553 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.553 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.554 INFO:teuthology.orchestra.run.smithi104.stdout: "upmap_max_deviation": { 2024-06-09T22:31:33.554 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "upmap_max_deviation", 2024-06-09T22:31:33.554 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.554 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.554 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.554 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5", 2024-06-09T22:31:33.554 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1", 2024-06-09T22:31:33.555 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.555 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.555 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "deviation below which no optimization is attempted", 2024-06-09T22:31:33.555 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-06-09T22:31:33.555 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.555 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.555 INFO:teuthology.orchestra.run.smithi104.stdout: "upmap_max_optimizations": { 2024-06-09T22:31:33.556 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "upmap_max_optimizations", 2024-06-09T22:31:33.556 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:33.556 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.556 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.556 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:33.556 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.556 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.556 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.557 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-06-09T22:31:33.557 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.557 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.557 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.557 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.557 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.558 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cephadm", 2024-06-09T22:31:33.558 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.558 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.558 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.558 INFO:teuthology.orchestra.run.smithi104.stdout: "agent_down_multiplier": { 2024-06-09T22:31:33.558 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "agent_down_multiplier", 2024-06-09T22:31:33.558 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:33.558 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.559 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.559 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "3.0", 2024-06-09T22:31:33.559 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.559 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.559 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.559 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-06-09T22:31:33.559 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.559 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.559 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.560 INFO:teuthology.orchestra.run.smithi104.stdout: "agent_refresh_rate": { 2024-06-09T22:31:33.560 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "agent_refresh_rate", 2024-06-09T22:31:33.560 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.560 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.560 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.560 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "20", 2024-06-09T22:31:33.561 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.561 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.561 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.561 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-06-09T22:31:33.561 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.561 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.561 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.562 INFO:teuthology.orchestra.run.smithi104.stdout: "agent_starting_port": { 2024-06-09T22:31:33.562 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "agent_starting_port", 2024-06-09T22:31:33.562 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.562 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.562 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.562 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "4721", 2024-06-09T22:31:33.562 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.562 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.563 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.563 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-06-09T22:31:33.563 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.563 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.563 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.563 INFO:teuthology.orchestra.run.smithi104.stdout: "allow_ptrace": { 2024-06-09T22:31:33.564 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "allow_ptrace", 2024-06-09T22:31:33.564 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.564 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.564 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.564 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.564 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.564 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.564 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.565 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-06-09T22:31:33.565 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-06-09T22:31:33.565 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.565 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.565 INFO:teuthology.orchestra.run.smithi104.stdout: "autotune_interval": { 2024-06-09T22:31:33.565 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "autotune_interval", 2024-06-09T22:31:33.566 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.566 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.566 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.566 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:33.566 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.566 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.566 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.567 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to autotune daemon memory", 2024-06-09T22:31:33.567 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.567 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.567 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.567 INFO:teuthology.orchestra.run.smithi104.stdout: "autotune_memory_target_ratio": { 2024-06-09T22:31:33.567 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "autotune_memory_target_ratio", 2024-06-09T22:31:33.567 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:33.568 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.568 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.568 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0.7", 2024-06-09T22:31:33.568 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.568 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.568 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.568 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-06-09T22:31:33.568 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.568 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.569 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.569 INFO:teuthology.orchestra.run.smithi104.stdout: "cephadm_log_destination": { 2024-06-09T22:31:33.569 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cephadm_log_destination", 2024-06-09T22:31:33.569 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.569 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.569 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.569 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.569 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.570 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.570 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.570 INFO:teuthology.orchestra.run.smithi104.stdout: "file", 2024-06-09T22:31:33.570 INFO:teuthology.orchestra.run.smithi104.stdout: "file,syslog", 2024-06-09T22:31:33.570 INFO:teuthology.orchestra.run.smithi104.stdout: "syslog" 2024-06-09T22:31:33.570 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.570 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-06-09T22:31:33.570 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.571 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.571 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.571 INFO:teuthology.orchestra.run.smithi104.stdout: "cgroups_split": { 2024-06-09T22:31:33.571 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cgroups_split", 2024-06-09T22:31:33.571 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.571 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.572 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.572 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.572 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.572 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.572 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.572 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-06-09T22:31:33.572 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.572 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.572 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.573 INFO:teuthology.orchestra.run.smithi104.stdout: "config_checks_enabled": { 2024-06-09T22:31:33.573 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "config_checks_enabled", 2024-06-09T22:31:33.573 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.573 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.573 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.573 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.573 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.573 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.574 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.574 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-06-09T22:31:33.574 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.574 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.574 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.574 INFO:teuthology.orchestra.run.smithi104.stdout: "config_dashboard": { 2024-06-09T22:31:33.574 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "config_dashboard", 2024-06-09T22:31:33.575 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.575 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.575 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.575 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.575 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.575 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.575 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.575 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-06-09T22:31:33.576 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.576 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.576 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.576 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_alertmanager": { 2024-06-09T22:31:33.576 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_alertmanager", 2024-06-09T22:31:33.576 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.576 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.576 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.577 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-06-09T22:31:33.577 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.577 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.577 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.577 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Prometheus container image", 2024-06-09T22:31:33.577 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.577 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.577 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.578 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_base": { 2024-06-09T22:31:33.578 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_base", 2024-06-09T22:31:33.578 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.578 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.578 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.578 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/ceph", 2024-06-09T22:31:33.578 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.578 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.579 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.579 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Container image name, without the tag", 2024-06-09T22:31:33.579 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.579 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.579 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.579 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_elasticsearch": { 2024-06-09T22:31:33.579 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_elasticsearch", 2024-06-09T22:31:33.579 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.580 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.580 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.580 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-06-09T22:31:33.580 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.580 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.580 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.580 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "elasticsearch container image", 2024-06-09T22:31:33.580 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.580 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.581 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.581 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_grafana": { 2024-06-09T22:31:33.581 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_grafana", 2024-06-09T22:31:33.581 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.581 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.581 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.581 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-06-09T22:31:33.581 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.582 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.582 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.582 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Prometheus container image", 2024-06-09T22:31:33.582 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.582 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.582 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.582 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_haproxy": { 2024-06-09T22:31:33.583 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_haproxy", 2024-06-09T22:31:33.583 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.583 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.583 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.583 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-06-09T22:31:33.583 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.583 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.583 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.583 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "HAproxy container image", 2024-06-09T22:31:33.584 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.584 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.584 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.584 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_jaeger_agent": { 2024-06-09T22:31:33.584 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_jaeger_agent", 2024-06-09T22:31:33.584 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.584 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.585 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.585 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-06-09T22:31:33.585 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.585 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.585 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.585 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Jaeger agent container image", 2024-06-09T22:31:33.585 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.585 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.585 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.586 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_jaeger_collector": { 2024-06-09T22:31:33.586 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_jaeger_collector", 2024-06-09T22:31:33.586 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.586 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.586 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.586 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-06-09T22:31:33.586 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.587 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.587 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.587 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Jaeger collector container image", 2024-06-09T22:31:33.587 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.587 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.587 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.587 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_jaeger_query": { 2024-06-09T22:31:33.588 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_jaeger_query", 2024-06-09T22:31:33.588 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.588 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.588 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.588 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-06-09T22:31:33.588 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.588 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.588 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.588 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Jaeger query container image", 2024-06-09T22:31:33.589 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.589 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.589 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.589 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_keepalived": { 2024-06-09T22:31:33.589 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_keepalived", 2024-06-09T22:31:33.589 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.589 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.590 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.590 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-06-09T22:31:33.590 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.590 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.590 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.590 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Keepalived container image", 2024-06-09T22:31:33.590 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.590 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.591 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.591 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_loki": { 2024-06-09T22:31:33.591 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_loki", 2024-06-09T22:31:33.591 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.591 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.591 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.592 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-06-09T22:31:33.592 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.592 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.592 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.592 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Loki container image", 2024-06-09T22:31:33.592 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.592 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.592 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.593 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_node_exporter": { 2024-06-09T22:31:33.593 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_node_exporter", 2024-06-09T22:31:33.593 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.593 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.593 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.593 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-06-09T22:31:33.593 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.594 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.594 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.594 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Prometheus container image", 2024-06-09T22:31:33.594 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.594 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.594 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.594 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_nvmeof": { 2024-06-09T22:31:33.595 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_nvmeof", 2024-06-09T22:31:33.595 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.595 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.595 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.595 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-06-09T22:31:33.595 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.595 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.595 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.596 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Nvme-of container image", 2024-06-09T22:31:33.596 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.596 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.596 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.596 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_prometheus": { 2024-06-09T22:31:33.596 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_prometheus", 2024-06-09T22:31:33.596 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.597 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.597 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.597 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-06-09T22:31:33.597 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.597 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.597 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.597 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Prometheus container image", 2024-06-09T22:31:33.597 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.598 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.598 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.598 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_promtail": { 2024-06-09T22:31:33.598 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_promtail", 2024-06-09T22:31:33.598 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.598 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.598 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.599 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-06-09T22:31:33.599 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.599 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.599 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.599 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Promtail container image", 2024-06-09T22:31:33.599 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.599 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.600 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.600 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_samba": { 2024-06-09T22:31:33.600 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_samba", 2024-06-09T22:31:33.600 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.600 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.600 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.600 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-06-09T22:31:33.601 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.601 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.601 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.601 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Samba/SMB container image", 2024-06-09T22:31:33.601 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.601 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.601 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.602 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_snmp_gateway": { 2024-06-09T22:31:33.602 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_snmp_gateway", 2024-06-09T22:31:33.602 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.602 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.602 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.602 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-06-09T22:31:33.602 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.602 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.603 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.603 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "SNMP Gateway container image", 2024-06-09T22:31:33.603 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.603 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.603 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.603 INFO:teuthology.orchestra.run.smithi104.stdout: "container_init": { 2024-06-09T22:31:33.603 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_init", 2024-06-09T22:31:33.603 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.604 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.604 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.604 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.604 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.604 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.604 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.604 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Run podman/docker with `--init`", 2024-06-09T22:31:33.604 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.605 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.605 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.605 INFO:teuthology.orchestra.run.smithi104.stdout: "daemon_cache_timeout": { 2024-06-09T22:31:33.605 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "daemon_cache_timeout", 2024-06-09T22:31:33.605 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.605 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.605 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.605 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:33.606 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.606 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.606 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.606 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-06-09T22:31:33.606 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.606 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.606 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.607 INFO:teuthology.orchestra.run.smithi104.stdout: "default_cephadm_command_timeout": { 2024-06-09T22:31:33.607 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "default_cephadm_command_timeout", 2024-06-09T22:31:33.607 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.607 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.607 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.607 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "900", 2024-06-09T22:31:33.607 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.607 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.608 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.608 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-06-09T22:31:33.608 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.608 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.608 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.608 INFO:teuthology.orchestra.run.smithi104.stdout: "default_registry": { 2024-06-09T22:31:33.608 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "default_registry", 2024-06-09T22:31:33.608 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.609 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.609 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.609 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "docker.io", 2024-06-09T22:31:33.609 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.609 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.609 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.609 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-06-09T22:31:33.609 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.609 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.609 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.610 INFO:teuthology.orchestra.run.smithi104.stdout: "device_cache_timeout": { 2024-06-09T22:31:33.610 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "device_cache_timeout", 2024-06-09T22:31:33.610 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.610 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.610 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.610 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "1800", 2024-06-09T22:31:33.610 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.611 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.611 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.611 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "seconds to cache device inventory", 2024-06-09T22:31:33.611 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.611 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.611 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.611 INFO:teuthology.orchestra.run.smithi104.stdout: "device_enhanced_scan": { 2024-06-09T22:31:33.611 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "device_enhanced_scan", 2024-06-09T22:31:33.612 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.612 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.612 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.612 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.612 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.612 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.612 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.612 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Use libstoragemgmt during device scans", 2024-06-09T22:31:33.613 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.613 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.613 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.613 INFO:teuthology.orchestra.run.smithi104.stdout: "facts_cache_timeout": { 2024-06-09T22:31:33.613 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "facts_cache_timeout", 2024-06-09T22:31:33.613 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.613 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.613 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.614 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:33.614 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.614 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.614 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.614 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "seconds to cache host facts data", 2024-06-09T22:31:33.614 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.614 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.614 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.615 INFO:teuthology.orchestra.run.smithi104.stdout: "grafana_dashboards_path": { 2024-06-09T22:31:33.615 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "grafana_dashboards_path", 2024-06-09T22:31:33.615 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.615 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.615 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.615 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-06-09T22:31:33.615 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.615 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-06-09T22:31:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: "host_check_interval": { 2024-06-09T22:31:33.616 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "host_check_interval", 2024-06-09T22:31:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to perform a host check", 2024-06-09T22:31:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: "hw_monitoring": { 2024-06-09T22:31:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "hw_monitoring", 2024-06-09T22:31:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.618 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.619 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.619 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.619 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.619 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.619 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-06-09T22:31:33.619 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.619 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.619 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.620 INFO:teuthology.orchestra.run.smithi104.stdout: "inventory_list_all": { 2024-06-09T22:31:33.620 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "inventory_list_all", 2024-06-09T22:31:33.620 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.620 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.620 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.620 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.620 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.621 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.621 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.621 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-06-09T22:31:33.621 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.621 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.621 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.621 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.622 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.622 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.622 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.622 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.622 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.622 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.622 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.622 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.623 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.623 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.623 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.623 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.623 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.623 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.623 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.623 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.624 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.624 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.624 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.624 INFO:teuthology.orchestra.run.smithi104.stdout: "log_refresh_metadata": { 2024-06-09T22:31:33.624 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_refresh_metadata", 2024-06-09T22:31:33.624 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.624 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.625 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.625 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.625 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.625 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.625 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.625 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-06-09T22:31:33.625 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.626 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.626 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.626 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.626 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.626 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.626 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.626 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.627 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.627 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.627 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.627 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.627 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-06-09T22:31:33.627 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.627 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.627 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.628 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.628 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.628 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.628 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.628 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.628 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.628 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.628 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.629 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.629 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.629 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.629 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.629 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.629 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.629 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.629 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.629 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.629 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.630 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.630 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.630 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.630 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.630 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.630 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.630 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.631 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.631 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.631 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.631 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.631 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.631 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.631 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.631 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.631 INFO:teuthology.orchestra.run.smithi104.stdout: "manage_etc_ceph_ceph_conf": { 2024-06-09T22:31:33.632 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-06-09T22:31:33.632 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.632 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.632 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.632 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.632 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.632 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.632 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.632 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-06-09T22:31:33.633 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.633 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.633 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.633 INFO:teuthology.orchestra.run.smithi104.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-06-09T22:31:33.633 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-06-09T22:31:33.633 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.633 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.633 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.633 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "*", 2024-06-09T22:31:33.634 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.634 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.634 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.634 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-06-09T22:31:33.634 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.634 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.634 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.635 INFO:teuthology.orchestra.run.smithi104.stdout: "max_count_per_host": { 2024-06-09T22:31:33.635 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_count_per_host", 2024-06-09T22:31:33.635 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.635 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.635 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.635 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:33.635 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.635 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.636 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.636 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "max number of daemons per service per host", 2024-06-09T22:31:33.636 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.636 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.636 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.636 INFO:teuthology.orchestra.run.smithi104.stdout: "max_osd_draining_count": { 2024-06-09T22:31:33.636 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_osd_draining_count", 2024-06-09T22:31:33.636 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.636 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.637 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.637 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:33.637 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.637 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.637 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.637 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-06-09T22:31:33.637 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.637 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.638 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.638 INFO:teuthology.orchestra.run.smithi104.stdout: "migration_current": { 2024-06-09T22:31:33.638 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "migration_current", 2024-06-09T22:31:33.638 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.638 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.638 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.638 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.639 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.639 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.639 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.639 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "internal - do not modify", 2024-06-09T22:31:33.639 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.639 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.639 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.639 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": { 2024-06-09T22:31:33.640 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mode", 2024-06-09T22:31:33.640 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.640 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.640 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.640 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "root", 2024-06-09T22:31:33.640 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.640 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.640 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.640 INFO:teuthology.orchestra.run.smithi104.stdout: "cephadm-package", 2024-06-09T22:31:33.640 INFO:teuthology.orchestra.run.smithi104.stdout: "root" 2024-06-09T22:31:33.641 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.641 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "mode for remote execution of cephadm", 2024-06-09T22:31:33.641 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.641 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.641 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.641 INFO:teuthology.orchestra.run.smithi104.stdout: "oob_default_addr": { 2024-06-09T22:31:33.641 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "oob_default_addr", 2024-06-09T22:31:33.641 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.642 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.642 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.642 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "169.254.1.1", 2024-06-09T22:31:33.642 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.642 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.642 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.642 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Default address for RedFish API (oob management).", 2024-06-09T22:31:33.642 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.642 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.643 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.643 INFO:teuthology.orchestra.run.smithi104.stdout: "prometheus_alerts_path": { 2024-06-09T22:31:33.643 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "prometheus_alerts_path", 2024-06-09T22:31:33.643 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.646 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.646 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.646 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-06-09T22:31:33.646 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.646 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.647 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.647 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-06-09T22:31:33.647 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.647 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.647 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.647 INFO:teuthology.orchestra.run.smithi104.stdout: "registry_insecure": { 2024-06-09T22:31:33.647 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "registry_insecure", 2024-06-09T22:31:33.648 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.648 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.648 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.648 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.648 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.648 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.648 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.648 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-06-09T22:31:33.648 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.649 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.649 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.649 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.649 INFO:teuthology.orchestra.run.smithi104.stdout: "registry_password": { 2024-06-09T22:31:33.649 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "registry_password", 2024-06-09T22:31:33.649 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.649 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.649 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.649 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.650 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.650 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.650 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.650 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-06-09T22:31:33.650 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.650 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.650 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.650 INFO:teuthology.orchestra.run.smithi104.stdout: "registry_url": { 2024-06-09T22:31:33.651 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "registry_url", 2024-06-09T22:31:33.651 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.651 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.651 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.651 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.651 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.651 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.651 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.651 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-06-09T22:31:33.652 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.652 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.652 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.652 INFO:teuthology.orchestra.run.smithi104.stdout: "registry_username": { 2024-06-09T22:31:33.652 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "registry_username", 2024-06-09T22:31:33.652 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.652 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.652 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.653 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.653 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.653 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.653 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.653 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-06-09T22:31:33.653 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.653 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.653 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.654 INFO:teuthology.orchestra.run.smithi104.stdout: "secure_monitoring_stack": { 2024-06-09T22:31:33.654 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "secure_monitoring_stack", 2024-06-09T22:31:33.654 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.654 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.654 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.654 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.654 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.654 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.655 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.655 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-06-09T22:31:33.655 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.655 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.655 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.655 INFO:teuthology.orchestra.run.smithi104.stdout: "service_discovery_port": { 2024-06-09T22:31:33.655 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "service_discovery_port", 2024-06-09T22:31:33.656 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.656 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.656 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.656 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8765", 2024-06-09T22:31:33.656 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.656 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.656 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.656 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "cephadm service discovery port", 2024-06-09T22:31:33.657 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.657 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.657 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.657 INFO:teuthology.orchestra.run.smithi104.stdout: "ssh_config_file": { 2024-06-09T22:31:33.657 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ssh_config_file", 2024-06-09T22:31:33.657 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.657 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.658 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.658 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.658 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.658 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.658 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.658 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-06-09T22:31:33.658 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.658 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.659 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.659 INFO:teuthology.orchestra.run.smithi104.stdout: "use_agent": { 2024-06-09T22:31:33.659 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "use_agent", 2024-06-09T22:31:33.659 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.659 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.659 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.659 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.659 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.660 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.660 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.660 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-06-09T22:31:33.660 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.660 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.660 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.660 INFO:teuthology.orchestra.run.smithi104.stdout: "use_repo_digest": { 2024-06-09T22:31:33.660 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "use_repo_digest", 2024-06-09T22:31:33.661 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.661 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.661 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.661 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.661 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.661 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.661 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.661 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-06-09T22:31:33.662 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.662 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.662 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.662 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_on_failed_host_check": { 2024-06-09T22:31:33.662 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_on_failed_host_check", 2024-06-09T22:31:33.662 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.663 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.663 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.663 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.663 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.663 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.663 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.663 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "raise a health warning if the host check fails", 2024-06-09T22:31:33.663 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.664 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.664 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.664 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_on_stray_daemons": { 2024-06-09T22:31:33.664 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_on_stray_daemons", 2024-06-09T22:31:33.664 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.664 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.664 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.665 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.665 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.665 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.665 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.665 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-06-09T22:31:33.665 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.665 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.665 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.666 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_on_stray_hosts": { 2024-06-09T22:31:33.666 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_on_stray_hosts", 2024-06-09T22:31:33.666 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.666 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.666 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.666 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.666 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.667 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.667 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.667 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-06-09T22:31:33.667 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.667 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.667 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.667 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.667 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.668 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crash", 2024-06-09T22:31:33.668 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.668 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.668 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.669 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.669 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.669 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.669 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.669 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.669 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.669 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.669 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.670 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.670 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.670 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.670 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.670 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.670 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.670 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.670 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.670 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.671 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.671 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.671 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.671 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.671 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.671 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.671 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.671 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.672 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.672 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.672 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.672 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.672 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.672 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.672 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.672 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.673 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.673 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.673 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.673 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.673 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.673 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.673 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.673 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.673 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.674 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.674 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.674 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.674 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.674 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.674 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.674 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.674 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.675 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.675 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.675 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.675 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.675 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.675 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.675 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.676 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.676 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.676 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.676 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.676 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.676 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.676 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.676 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.676 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.677 INFO:teuthology.orchestra.run.smithi104.stdout: "retain_interval": { 2024-06-09T22:31:33.677 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "retain_interval", 2024-06-09T22:31:33.677 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.677 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.677 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.677 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "31536000", 2024-06-09T22:31:33.677 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.677 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.678 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.678 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how long to retain crashes before pruning them", 2024-06-09T22:31:33.678 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.678 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.678 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.678 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_recent_interval": { 2024-06-09T22:31:33.678 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_recent_interval", 2024-06-09T22:31:33.679 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.679 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.679 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.679 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "1209600", 2024-06-09T22:31:33.679 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.679 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.679 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.679 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "time interval in which to warn about recent crashes", 2024-06-09T22:31:33.679 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.679 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.680 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.680 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.680 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.680 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "dashboard", 2024-06-09T22:31:33.680 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.680 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.680 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.681 INFO:teuthology.orchestra.run.smithi104.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-06-09T22:31:33.681 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-06-09T22:31:33.681 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.681 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.681 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.681 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:33.681 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.681 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.681 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.682 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.682 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.682 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.682 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.682 INFO:teuthology.orchestra.run.smithi104.stdout: "ALERTMANAGER_API_HOST": { 2024-06-09T22:31:33.682 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ALERTMANAGER_API_HOST", 2024-06-09T22:31:33.682 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.682 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.683 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.683 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.683 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.683 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.683 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.683 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.683 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.683 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.683 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.684 INFO:teuthology.orchestra.run.smithi104.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-06-09T22:31:33.684 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-06-09T22:31:33.684 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.684 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.684 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.684 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.684 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.684 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.684 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.685 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.685 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.685 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.685 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.685 INFO:teuthology.orchestra.run.smithi104.stdout: "AUDIT_API_ENABLED": { 2024-06-09T22:31:33.685 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "AUDIT_API_ENABLED", 2024-06-09T22:31:33.685 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.685 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.686 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.686 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.686 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.686 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.686 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.686 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.686 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.686 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.687 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.687 INFO:teuthology.orchestra.run.smithi104.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-06-09T22:31:33.687 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-06-09T22:31:33.687 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.687 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.687 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.687 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.687 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.688 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.688 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.688 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.688 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.688 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.688 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.688 INFO:teuthology.orchestra.run.smithi104.stdout: "ENABLE_BROWSABLE_API": { 2024-06-09T22:31:33.689 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ENABLE_BROWSABLE_API", 2024-06-09T22:31:33.689 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.689 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.689 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.689 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.689 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.689 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.689 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.689 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.689 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.690 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.690 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.690 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-06-09T22:31:33.690 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-06-09T22:31:33.690 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.690 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-06-09T22:31:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-06-09T22:31:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-06-09T22:31:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-06-09T22:31:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.694 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.694 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.694 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.694 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.694 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.694 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.694 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.694 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.695 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.695 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-06-09T22:31:33.695 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-06-09T22:31:33.695 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.695 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.695 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.695 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.695 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.696 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.696 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.696 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.696 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.696 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.696 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.696 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_NFS": { 2024-06-09T22:31:33.697 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-06-09T22:31:33.697 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.697 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.697 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.697 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.697 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.697 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.697 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.697 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.698 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.698 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.698 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.698 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_RBD": { 2024-06-09T22:31:33.698 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-06-09T22:31:33.698 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.698 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.699 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.699 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.699 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.699 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.699 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.699 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.699 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.699 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.700 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.700 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_RGW": { 2024-06-09T22:31:33.700 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-06-09T22:31:33.700 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.700 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.700 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.700 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.700 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.701 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.701 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.701 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.701 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.701 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.701 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.702 INFO:teuthology.orchestra.run.smithi104.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-06-09T22:31:33.702 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-06-09T22:31:33.702 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.702 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.702 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.702 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.702 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.702 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.703 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.703 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.703 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.703 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.703 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.703 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_API_PASSWORD": { 2024-06-09T22:31:33.703 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_API_PASSWORD", 2024-06-09T22:31:33.704 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.704 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.704 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.704 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "admin", 2024-06-09T22:31:33.704 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.704 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.704 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.704 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.705 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.705 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.705 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.705 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-06-09T22:31:33.705 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-06-09T22:31:33.705 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.705 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.706 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.706 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.706 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.706 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.706 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.706 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.706 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.706 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.707 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.707 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_API_URL": { 2024-06-09T22:31:33.707 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_API_URL", 2024-06-09T22:31:33.707 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.707 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.707 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.707 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.708 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.708 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.708 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.708 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.708 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.708 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.708 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.709 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_API_USERNAME": { 2024-06-09T22:31:33.709 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_API_USERNAME", 2024-06-09T22:31:33.709 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.709 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.709 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.709 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "admin", 2024-06-09T22:31:33.709 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.709 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.710 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.710 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.710 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.710 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.710 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.710 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-06-09T22:31:33.710 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-06-09T22:31:33.711 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.711 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.711 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.711 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.711 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.711 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.711 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.711 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.711 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.712 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.712 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.712 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-06-09T22:31:33.712 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-06-09T22:31:33.712 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.712 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.712 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.713 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.713 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.713 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.713 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.713 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.713 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.713 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.713 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.714 INFO:teuthology.orchestra.run.smithi104.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-06-09T22:31:33.714 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-06-09T22:31:33.714 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.714 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.714 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.714 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.714 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.714 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.715 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.715 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.715 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.715 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.715 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.715 INFO:teuthology.orchestra.run.smithi104.stdout: "ISSUE_TRACKER_API_KEY": { 2024-06-09T22:31:33.715 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-06-09T22:31:33.716 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.716 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.716 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.716 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.716 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.716 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.716 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.716 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.717 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.717 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.717 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.717 INFO:teuthology.orchestra.run.smithi104.stdout: "PROMETHEUS_API_HOST": { 2024-06-09T22:31:33.717 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PROMETHEUS_API_HOST", 2024-06-09T22:31:33.717 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.717 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.717 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.718 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.718 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.718 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.718 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.718 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.718 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.718 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.719 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.719 INFO:teuthology.orchestra.run.smithi104.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-06-09T22:31:33.719 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-06-09T22:31:33.719 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.719 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.719 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.719 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.720 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.720 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.720 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.720 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.720 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.720 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.720 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.721 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-06-09T22:31:33.721 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-06-09T22:31:33.721 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.721 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.721 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.721 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.721 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.721 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.721 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.722 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.722 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.722 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.722 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.722 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-06-09T22:31:33.722 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-06-09T22:31:33.722 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.723 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.723 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.723 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.723 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.723 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.723 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.723 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.723 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.724 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.724 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.724 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-06-09T22:31:33.724 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-06-09T22:31:33.724 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.724 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.724 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.724 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.725 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.725 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.725 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.725 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.725 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.725 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.725 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.726 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-06-09T22:31:33.726 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-06-09T22:31:33.726 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.726 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.726 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.726 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.726 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.726 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.727 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.727 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.727 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.727 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.727 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.727 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-06-09T22:31:33.727 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-06-09T22:31:33.728 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.728 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.728 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.728 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.728 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.728 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.728 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.728 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.728 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.729 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.729 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.729 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-06-09T22:31:33.729 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-06-09T22:31:33.729 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.729 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.729 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.730 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.730 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.730 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.730 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.730 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.730 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.730 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.730 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.731 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.731 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-06-09T22:31:33.731 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-06-09T22:31:33.731 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.731 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.731 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.731 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.731 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.732 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.732 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.732 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.732 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.732 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.732 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.732 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.732 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_ENABLED": { 2024-06-09T22:31:33.733 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_ENABLED", 2024-06-09T22:31:33.733 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.733 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.733 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.733 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.733 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.733 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.733 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.734 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.734 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.734 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.734 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.734 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-06-09T22:31:33.734 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-06-09T22:31:33.735 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.735 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.735 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.735 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-06-09T22:31:33.735 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.735 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.735 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.735 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.735 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.736 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.736 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.736 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-06-09T22:31:33.736 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-06-09T22:31:33.736 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.736 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.736 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.737 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:33.737 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.737 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.737 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.737 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.737 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.737 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.737 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.738 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-06-09T22:31:33.738 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-06-09T22:31:33.738 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.738 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.738 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.738 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8", 2024-06-09T22:31:33.738 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.739 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.739 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.739 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.739 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.739 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.739 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.739 INFO:teuthology.orchestra.run.smithi104.stdout: "REST_REQUESTS_TIMEOUT": { 2024-06-09T22:31:33.740 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-06-09T22:31:33.740 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.740 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.740 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.740 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "45", 2024-06-09T22:31:33.740 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.740 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.740 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.741 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.741 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.741 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.741 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.741 INFO:teuthology.orchestra.run.smithi104.stdout: "RGW_API_ACCESS_KEY": { 2024-06-09T22:31:33.741 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RGW_API_ACCESS_KEY", 2024-06-09T22:31:33.741 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.741 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.742 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.742 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.742 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.742 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.742 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.742 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.742 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.742 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.743 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.743 INFO:teuthology.orchestra.run.smithi104.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-06-09T22:31:33.743 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-06-09T22:31:33.743 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.743 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.743 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.743 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "admin", 2024-06-09T22:31:33.744 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.744 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.744 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.744 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.744 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.744 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.744 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.744 INFO:teuthology.orchestra.run.smithi104.stdout: "RGW_API_SECRET_KEY": { 2024-06-09T22:31:33.745 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RGW_API_SECRET_KEY", 2024-06-09T22:31:33.745 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.745 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.745 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.745 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.745 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.745 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.745 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.745 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.746 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.746 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.746 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.746 INFO:teuthology.orchestra.run.smithi104.stdout: "RGW_API_SSL_VERIFY": { 2024-06-09T22:31:33.746 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RGW_API_SSL_VERIFY", 2024-06-09T22:31:33.746 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.747 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.747 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.747 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.747 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.747 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.747 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.747 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.747 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.747 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.748 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.748 INFO:teuthology.orchestra.run.smithi104.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-06-09T22:31:33.748 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-06-09T22:31:33.748 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.748 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.748 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.748 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:33.748 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.749 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.749 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.749 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.749 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.749 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.749 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.749 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.750 INFO:teuthology.orchestra.run.smithi104.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-06-09T22:31:33.750 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-06-09T22:31:33.750 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.750 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.750 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.750 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:33.750 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.750 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.751 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.751 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.751 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.751 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.751 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.751 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.751 INFO:teuthology.orchestra.run.smithi104.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-06-09T22:31:33.751 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-06-09T22:31:33.752 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.752 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.752 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.752 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5", 2024-06-09T22:31:33.752 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.752 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.752 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.752 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.752 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.753 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.753 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.753 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.753 INFO:teuthology.orchestra.run.smithi104.stdout: "cross_origin_url": { 2024-06-09T22:31:33.753 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cross_origin_url", 2024-06-09T22:31:33.753 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.753 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.753 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.754 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.754 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.754 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.754 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.754 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.754 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.754 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.754 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.755 INFO:teuthology.orchestra.run.smithi104.stdout: "crt_file": { 2024-06-09T22:31:33.755 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crt_file", 2024-06-09T22:31:33.755 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.755 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.755 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.755 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.755 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.756 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.756 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.756 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.756 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.756 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.756 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.756 INFO:teuthology.orchestra.run.smithi104.stdout: "debug": { 2024-06-09T22:31:33.757 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "debug", 2024-06-09T22:31:33.757 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.757 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.757 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.757 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.757 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.757 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.757 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.758 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Enable/disable debug options", 2024-06-09T22:31:33.758 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.758 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.758 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.758 INFO:teuthology.orchestra.run.smithi104.stdout: "jwt_token_ttl": { 2024-06-09T22:31:33.758 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "jwt_token_ttl", 2024-06-09T22:31:33.758 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "28800", 2024-06-09T22:31:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: "key_file": { 2024-06-09T22:31:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "key_file", 2024-06-09T22:31:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.761 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.761 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.761 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.761 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.761 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.761 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.761 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.761 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.762 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.762 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.762 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.762 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.762 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.762 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.762 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.763 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.763 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.763 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.763 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.763 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.763 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.763 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.763 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.764 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.764 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.764 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.764 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.764 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.764 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.764 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.765 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.765 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.765 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.765 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.765 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.765 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.765 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.765 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.766 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.766 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.766 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.766 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.766 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.766 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.766 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.767 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.767 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.767 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.767 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.767 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.767 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.767 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.767 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.767 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.768 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.768 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.768 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.768 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.768 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.768 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.768 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.768 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.769 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.769 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.769 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.769 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.769 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.769 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.769 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.769 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.769 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.770 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.770 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.770 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.770 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.770 INFO:teuthology.orchestra.run.smithi104.stdout: "motd": { 2024-06-09T22:31:33.770 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "motd", 2024-06-09T22:31:33.770 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.770 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.771 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.771 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.771 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.771 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.771 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.771 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "The message of the day", 2024-06-09T22:31:33.771 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.771 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.771 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.772 INFO:teuthology.orchestra.run.smithi104.stdout: "redirect_resolve_ip_addr": { 2024-06-09T22:31:33.772 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "redirect_resolve_ip_addr", 2024-06-09T22:31:33.772 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.772 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.772 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.772 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.772 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.772 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.773 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.773 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.773 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.773 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.773 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.773 INFO:teuthology.orchestra.run.smithi104.stdout: "server_addr": { 2024-06-09T22:31:33.773 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_addr", 2024-06-09T22:31:33.773 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.773 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.774 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.774 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "::", 2024-06-09T22:31:33.774 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.774 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.774 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.774 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.774 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.774 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.775 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.775 INFO:teuthology.orchestra.run.smithi104.stdout: "server_port": { 2024-06-09T22:31:33.775 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_port", 2024-06-09T22:31:33.775 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.775 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.775 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.775 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8080", 2024-06-09T22:31:33.775 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.776 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.776 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.776 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.776 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.776 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.776 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.776 INFO:teuthology.orchestra.run.smithi104.stdout: "ssl": { 2024-06-09T22:31:33.776 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ssl", 2024-06-09T22:31:33.777 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.777 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.777 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.777 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.777 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.777 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.777 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.777 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.777 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.778 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.778 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.778 INFO:teuthology.orchestra.run.smithi104.stdout: "ssl_server_port": { 2024-06-09T22:31:33.778 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ssl_server_port", 2024-06-09T22:31:33.778 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.778 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.778 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.778 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8443", 2024-06-09T22:31:33.779 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.779 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.779 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.779 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.779 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.779 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.779 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.779 INFO:teuthology.orchestra.run.smithi104.stdout: "standby_behaviour": { 2024-06-09T22:31:33.780 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "standby_behaviour", 2024-06-09T22:31:33.780 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.780 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.780 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.780 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "redirect", 2024-06-09T22:31:33.780 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.780 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.780 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.781 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.781 INFO:teuthology.orchestra.run.smithi104.stdout: "redirect" 2024-06-09T22:31:33.781 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.781 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.781 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.781 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.781 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.781 INFO:teuthology.orchestra.run.smithi104.stdout: "standby_error_status_code": { 2024-06-09T22:31:33.782 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "standby_error_status_code", 2024-06-09T22:31:33.782 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.782 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.782 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.782 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "500", 2024-06-09T22:31:33.782 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "400", 2024-06-09T22:31:33.782 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "599", 2024-06-09T22:31:33.782 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.783 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.783 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.783 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.783 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.783 INFO:teuthology.orchestra.run.smithi104.stdout: "url_prefix": { 2024-06-09T22:31:33.783 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "url_prefix", 2024-06-09T22:31:33.783 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.784 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.784 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.784 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.784 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.784 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.784 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.784 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.784 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.784 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.785 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.785 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.785 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.785 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "devicehealth", 2024-06-09T22:31:33.785 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.785 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.785 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.785 INFO:teuthology.orchestra.run.smithi104.stdout: "enable_monitoring": { 2024-06-09T22:31:33.786 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "enable_monitoring", 2024-06-09T22:31:33.786 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.786 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.786 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.786 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.786 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.786 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.786 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.787 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "monitor device health metrics", 2024-06-09T22:31:33.787 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.787 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.787 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.787 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.787 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.787 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.787 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.788 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.788 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.788 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.788 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.788 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.788 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.788 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.788 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.788 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.789 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.789 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.789 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.789 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.789 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.789 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.789 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.789 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.790 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.790 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.790 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.790 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.790 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.790 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.790 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.790 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.791 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.791 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.791 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.791 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.791 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.791 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.791 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.792 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.792 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.792 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.792 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.792 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.792 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.792 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.792 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.792 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.793 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.793 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.793 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.793 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.793 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.793 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.793 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.793 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.794 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.794 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.794 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.794 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.794 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.794 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.794 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.794 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.794 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.795 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.795 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.795 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.795 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.795 INFO:teuthology.orchestra.run.smithi104.stdout: "mark_out_threshold": { 2024-06-09T22:31:33.795 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mark_out_threshold", 2024-06-09T22:31:33.795 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "2419200", 2024-06-09T22:31:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-06-09T22:31:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: "pool_name": { 2024-06-09T22:31:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "pool_name", 2024-06-09T22:31:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "device_health_metrics", 2024-06-09T22:31:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "name of pool in which to store device health metrics", 2024-06-09T22:31:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: "retention_period": { 2024-06-09T22:31:33.799 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "retention_period", 2024-06-09T22:31:33.799 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.799 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.799 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.799 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "15552000", 2024-06-09T22:31:33.799 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.799 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.800 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.800 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how long to retain device health metrics", 2024-06-09T22:31:33.800 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.800 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.800 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.800 INFO:teuthology.orchestra.run.smithi104.stdout: "scrape_frequency": { 2024-06-09T22:31:33.800 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "scrape_frequency", 2024-06-09T22:31:33.801 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.801 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.801 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.801 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "86400", 2024-06-09T22:31:33.801 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.801 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.801 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.801 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to scrape device health metrics", 2024-06-09T22:31:33.802 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.802 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.802 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.802 INFO:teuthology.orchestra.run.smithi104.stdout: "self_heal": { 2024-06-09T22:31:33.802 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "self_heal", 2024-06-09T22:31:33.802 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.802 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.803 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.803 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.803 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.803 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.803 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.803 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-06-09T22:31:33.803 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.803 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.803 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.804 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:33.804 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:33.804 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.804 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.804 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.804 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:33.804 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.804 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.805 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.805 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to wake up and check device health", 2024-06-09T22:31:33.805 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.805 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.805 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.805 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_threshold": { 2024-06-09T22:31:33.805 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_threshold", 2024-06-09T22:31:33.806 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.806 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.806 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.806 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "7257600", 2024-06-09T22:31:33.806 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.806 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.806 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.806 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-06-09T22:31:33.807 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.807 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.807 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.807 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.807 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.807 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "diskprediction_local", 2024-06-09T22:31:33.808 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.808 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.808 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.808 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.808 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.808 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.808 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.808 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.808 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.809 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.809 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.809 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.809 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.809 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.809 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.809 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.809 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.809 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.810 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.810 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.810 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.810 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.810 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.810 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.810 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.811 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.811 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.811 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.811 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.811 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.811 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.811 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.811 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.812 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.812 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.812 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.812 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.812 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.812 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.812 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.812 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.813 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.813 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.813 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.813 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.813 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.813 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.813 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.813 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.813 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.814 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.814 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.814 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.814 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.814 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.814 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.814 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.814 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.815 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.815 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.815 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.815 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.815 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.815 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.815 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.815 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.816 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.816 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.816 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.816 INFO:teuthology.orchestra.run.smithi104.stdout: "predict_interval": { 2024-06-09T22:31:33.816 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "predict_interval", 2024-06-09T22:31:33.816 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.816 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.816 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.817 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "86400", 2024-06-09T22:31:33.817 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.817 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.817 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.817 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.817 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.817 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.817 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.818 INFO:teuthology.orchestra.run.smithi104.stdout: "predictor_model": { 2024-06-09T22:31:33.818 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "predictor_model", 2024-06-09T22:31:33.818 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.818 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.818 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.818 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "prophetstor", 2024-06-09T22:31:33.818 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.818 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.819 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.819 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.819 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.819 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.819 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.819 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:33.819 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:33.819 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.819 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.820 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.820 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:33.820 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.820 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.820 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.820 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.820 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.820 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.821 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.821 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "influx", 2024-06-09T22:31:33.821 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": false, 2024-06-09T22:31:33.821 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "influxdb python module not found", 2024-06-09T22:31:33.821 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.822 INFO:teuthology.orchestra.run.smithi104.stdout: "batch_size": { 2024-06-09T22:31:33.822 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "batch_size", 2024-06-09T22:31:33.822 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.822 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.822 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.822 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5000", 2024-06-09T22:31:33.822 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.822 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.823 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.823 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-06-09T22:31:33.823 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.823 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.823 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.823 INFO:teuthology.orchestra.run.smithi104.stdout: "database": { 2024-06-09T22:31:33.823 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "database", 2024-06-09T22:31:33.823 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.824 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.824 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.824 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "ceph", 2024-06-09T22:31:33.824 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.824 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.824 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.824 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-06-09T22:31:33.824 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.824 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.825 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.825 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": { 2024-06-09T22:31:33.825 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "hostname", 2024-06-09T22:31:33.825 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.825 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.825 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.825 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.825 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.826 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.826 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.826 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "InfluxDB server hostname", 2024-06-09T22:31:33.826 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.826 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.826 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.826 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:33.827 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:33.827 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.827 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.827 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.827 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "30", 2024-06-09T22:31:33.827 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "5", 2024-06-09T22:31:33.827 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.827 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.827 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-06-09T22:31:33.828 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.828 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.828 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.828 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.828 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.828 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.828 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.828 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.829 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.829 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.829 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.829 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.829 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.829 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.829 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.829 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.830 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.830 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.830 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.830 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.830 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.830 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.830 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.831 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.831 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.831 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.831 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.831 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.831 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.831 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.831 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.832 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.832 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.832 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.832 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.832 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.832 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.832 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.833 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.833 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.833 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.833 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.833 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.833 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.833 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.833 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.833 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.834 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.834 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.834 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.834 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.834 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.834 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.834 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.834 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.835 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.835 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.835 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.835 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.835 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.835 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.835 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.836 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.836 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.836 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.836 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.836 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.836 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.836 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.837 INFO:teuthology.orchestra.run.smithi104.stdout: "password": { 2024-06-09T22:31:33.837 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "password", 2024-06-09T22:31:33.837 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.837 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.837 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.837 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.837 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.837 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.838 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.838 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "password of InfluxDB server user", 2024-06-09T22:31:33.838 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.838 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.838 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.838 INFO:teuthology.orchestra.run.smithi104.stdout: "port": { 2024-06-09T22:31:33.838 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "port", 2024-06-09T22:31:33.839 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.839 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.839 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.839 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8086", 2024-06-09T22:31:33.839 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.839 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.839 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.839 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "InfluxDB server port", 2024-06-09T22:31:33.840 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.840 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.840 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.840 INFO:teuthology.orchestra.run.smithi104.stdout: "ssl": { 2024-06-09T22:31:33.840 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ssl", 2024-06-09T22:31:33.840 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.840 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.841 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.841 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "false", 2024-06-09T22:31:33.841 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.841 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.841 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.841 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-06-09T22:31:33.841 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.841 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.842 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.842 INFO:teuthology.orchestra.run.smithi104.stdout: "threads": { 2024-06-09T22:31:33.842 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "threads", 2024-06-09T22:31:33.842 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.842 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.842 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.843 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5", 2024-06-09T22:31:33.843 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1", 2024-06-09T22:31:33.843 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "32", 2024-06-09T22:31:33.843 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.843 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-06-09T22:31:33.843 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.843 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.843 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.844 INFO:teuthology.orchestra.run.smithi104.stdout: "username": { 2024-06-09T22:31:33.844 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "username", 2024-06-09T22:31:33.844 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.844 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.844 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.844 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.844 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.844 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.845 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.845 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "username of InfluxDB server user", 2024-06-09T22:31:33.845 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.845 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.845 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.845 INFO:teuthology.orchestra.run.smithi104.stdout: "verify_ssl": { 2024-06-09T22:31:33.845 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "verify_ssl", 2024-06-09T22:31:33.846 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.846 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.846 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.846 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "true", 2024-06-09T22:31:33.846 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.846 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.846 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.846 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-06-09T22:31:33.849 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.850 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.850 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.850 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.850 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.850 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "insights", 2024-06-09T22:31:33.851 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.851 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.851 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.851 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.851 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.851 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.851 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.851 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.852 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.852 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.852 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.852 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.852 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.852 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.852 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.852 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.852 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.853 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.853 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.853 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.853 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.853 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.853 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.853 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.853 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.854 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.854 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.854 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.854 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.854 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.854 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.854 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.854 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.854 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.855 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.855 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.855 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.855 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.855 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.855 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.855 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.856 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.856 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.856 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.856 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.856 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.856 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.856 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.856 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.857 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.857 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.857 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.857 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.857 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.857 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.857 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.857 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.858 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.858 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.858 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.858 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.858 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.858 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.858 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.858 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.859 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.859 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.859 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.859 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.859 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.859 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.859 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "iostat", 2024-06-09T22:31:33.860 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.860 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.860 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.860 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.860 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.860 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.860 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.860 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.860 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.861 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.861 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.861 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.861 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.861 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.861 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.861 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.861 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.861 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.862 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.862 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.862 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.862 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.862 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.862 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.862 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.863 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.863 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.863 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.863 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.863 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.863 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.863 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.863 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.864 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.864 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.864 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.864 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.864 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.864 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.864 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.864 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.865 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.865 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.865 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.865 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.865 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.865 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.865 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.865 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.865 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.866 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.866 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.866 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.866 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.866 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.866 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.866 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.867 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.867 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.867 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.867 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.867 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.867 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.867 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.867 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.868 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.868 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.868 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.868 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.868 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.868 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.869 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "k8sevents", 2024-06-09T22:31:33.869 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.869 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.869 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.869 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_event_retention_days": { 2024-06-09T22:31:33.869 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ceph_event_retention_days", 2024-06-09T22:31:33.869 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.869 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.869 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.870 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "7", 2024-06-09T22:31:33.870 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.870 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.870 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.870 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Days to hold ceph event information within local cache", 2024-06-09T22:31:33.870 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.870 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.870 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.871 INFO:teuthology.orchestra.run.smithi104.stdout: "config_check_secs": { 2024-06-09T22:31:33.871 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "config_check_secs", 2024-06-09T22:31:33.871 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.871 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.871 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.871 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:33.871 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "10", 2024-06-09T22:31:33.871 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.872 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.872 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-06-09T22:31:33.872 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.872 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.872 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.872 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.872 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.873 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.873 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.873 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.873 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.873 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.873 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.873 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.873 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.874 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.874 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.874 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.874 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.874 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.874 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.874 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.874 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.875 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.875 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.875 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.875 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.875 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.875 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.875 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.875 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.875 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.876 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.876 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.876 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.876 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.876 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.876 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.876 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.876 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.877 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.877 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.877 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.877 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.877 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.877 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.877 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.877 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.878 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.878 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.878 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.878 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.878 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.878 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.878 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.878 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.879 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.879 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.879 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.879 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.879 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.879 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.879 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.879 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.880 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.880 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.880 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.880 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.880 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.880 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.880 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.880 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.880 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "localpool", 2024-06-09T22:31:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "failure_domain": { 2024-06-09T22:31:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "failure_domain", 2024-06-09T22:31:33.881 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.882 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.882 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.882 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "host", 2024-06-09T22:31:33.882 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.882 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.882 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.882 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "failure domain for any created local pool", 2024-06-09T22:31:33.882 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-06-09T22:31:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.885 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.885 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.885 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.885 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.885 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.885 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.885 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.886 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.886 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.886 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.886 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.886 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.886 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.886 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.886 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.886 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.887 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.887 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.887 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.887 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.887 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.887 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.887 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.888 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.888 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.888 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.888 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.888 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.888 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.888 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.888 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.889 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.889 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.889 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.889 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.889 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.889 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.889 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.889 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.890 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.890 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.890 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.890 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.890 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.890 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.890 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.890 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.891 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.891 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.891 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.891 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.891 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.891 INFO:teuthology.orchestra.run.smithi104.stdout: "min_size": { 2024-06-09T22:31:33.891 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "min_size", 2024-06-09T22:31:33.892 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.892 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.892 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.892 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.892 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.892 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.892 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.893 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "default min_size for any created local pool", 2024-06-09T22:31:33.893 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-06-09T22:31:33.893 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.893 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.893 INFO:teuthology.orchestra.run.smithi104.stdout: "num_rep": { 2024-06-09T22:31:33.893 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "num_rep", 2024-06-09T22:31:33.893 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.894 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.894 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.894 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "3", 2024-06-09T22:31:33.894 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.894 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.894 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.894 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "default replica count for any created local pool", 2024-06-09T22:31:33.895 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.895 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.895 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.895 INFO:teuthology.orchestra.run.smithi104.stdout: "pg_num": { 2024-06-09T22:31:33.895 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "pg_num", 2024-06-09T22:31:33.895 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.895 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.896 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.896 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "128", 2024-06-09T22:31:33.896 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.896 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.896 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.896 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "default pg_num for any created local pool", 2024-06-09T22:31:33.896 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.896 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.897 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.897 INFO:teuthology.orchestra.run.smithi104.stdout: "prefix": { 2024-06-09T22:31:33.897 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "prefix", 2024-06-09T22:31:33.897 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.897 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.897 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.897 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.898 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.898 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.898 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.898 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "name prefix for any created local pool", 2024-06-09T22:31:33.898 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.898 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.898 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.899 INFO:teuthology.orchestra.run.smithi104.stdout: "subtree": { 2024-06-09T22:31:33.899 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "subtree", 2024-06-09T22:31:33.899 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.899 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.899 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.899 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "rack", 2024-06-09T22:31:33.899 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.899 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.899 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.900 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "CRUSH level for which to create a local pool", 2024-06-09T22:31:33.900 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-06-09T22:31:33.900 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.900 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.900 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.900 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.901 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mds_autoscaler", 2024-06-09T22:31:33.901 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.901 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.901 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.901 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.901 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.901 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.901 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.902 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.902 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.902 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.902 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.902 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.902 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.902 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.902 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.903 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.903 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.903 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.903 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.903 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.903 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.903 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.903 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.904 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.904 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.904 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.904 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.904 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.904 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.904 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.904 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.904 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.905 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.905 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.905 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.905 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.905 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.905 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.905 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.905 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.906 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.906 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.906 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.906 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.906 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.906 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.906 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.906 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.907 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.907 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.907 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.907 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.907 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.907 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.907 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.907 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.908 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.908 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.908 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.908 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.908 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.908 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.908 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.908 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.909 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.909 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.909 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.909 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.909 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.909 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.909 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.910 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mirroring", 2024-06-09T22:31:33.910 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.910 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.910 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.910 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.910 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.910 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.911 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.911 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.911 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.911 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.911 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.911 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.911 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.911 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.912 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.912 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.912 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.912 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.912 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.912 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.912 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.912 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.912 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.913 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.913 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.913 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.913 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.913 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.913 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.913 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.914 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.914 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.914 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.914 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.914 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.914 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.915 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.915 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.915 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.915 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.915 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.915 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.915 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.915 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.915 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.916 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.916 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.916 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.916 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.916 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.916 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.916 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.916 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.917 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.917 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.917 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.917 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.917 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.917 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.917 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.918 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.918 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.918 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.918 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.918 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.918 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.919 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.919 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.919 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.919 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.919 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.919 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "nfs", 2024-06-09T22:31:33.920 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.920 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.920 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.920 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.920 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.920 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.920 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.920 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.920 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.921 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.921 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.921 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.921 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.921 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.921 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.921 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.921 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.921 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.922 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.922 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.922 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.922 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.922 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.922 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.922 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.923 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.923 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.923 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.923 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.923 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.923 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.923 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.923 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.924 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.924 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.924 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.924 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.924 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.924 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.924 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.924 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.925 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.925 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.925 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.925 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.925 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.925 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.925 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.925 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.926 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.926 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.926 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.926 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.926 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.926 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.926 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.927 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.927 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.927 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.927 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.927 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.927 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.927 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.927 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.928 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.928 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.928 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.928 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.928 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.928 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.928 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.929 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "orchestrator", 2024-06-09T22:31:33.929 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.929 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.929 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.929 INFO:teuthology.orchestra.run.smithi104.stdout: "fail_fs": { 2024-06-09T22:31:33.929 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "fail_fs", 2024-06-09T22:31:33.929 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.929 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.930 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.930 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.930 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.930 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.930 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.930 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-06-09T22:31:33.930 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.930 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.931 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.931 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.931 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.931 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.931 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.931 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.931 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.932 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.932 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.932 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.932 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.932 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.932 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.932 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.932 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.933 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.933 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.933 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.933 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.933 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.933 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.933 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.933 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.934 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.934 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.934 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.934 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.934 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.934 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.934 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.934 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.935 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.935 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.935 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.935 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.935 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.935 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.935 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.935 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.936 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.936 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.936 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.936 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.936 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.936 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.936 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.936 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.937 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.937 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.937 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.937 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.937 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.937 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.937 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.938 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.938 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.938 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.938 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.938 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.938 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.938 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.938 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.939 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.939 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.939 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.939 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.939 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.939 INFO:teuthology.orchestra.run.smithi104.stdout: "orchestrator": { 2024-06-09T22:31:33.939 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "orchestrator", 2024-06-09T22:31:33.939 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.940 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.940 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.940 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.940 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.940 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.940 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.940 INFO:teuthology.orchestra.run.smithi104.stdout: "cephadm", 2024-06-09T22:31:33.940 INFO:teuthology.orchestra.run.smithi104.stdout: "rook", 2024-06-09T22:31:33.941 INFO:teuthology.orchestra.run.smithi104.stdout: "test_orchestrator" 2024-06-09T22:31:33.941 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.941 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Orchestrator backend", 2024-06-09T22:31:33.941 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.941 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.941 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.941 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.941 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.942 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "osd_perf_query", 2024-06-09T22:31:33.942 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.942 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.942 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.942 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.942 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.943 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.943 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.943 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.943 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.943 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.943 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.943 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.943 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.943 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.944 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.944 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.944 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.944 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.944 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.944 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.944 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.944 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.945 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.945 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.945 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.945 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.945 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.945 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.945 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.945 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.946 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.946 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.946 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.946 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.946 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.946 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.946 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.947 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.947 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.947 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.947 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.947 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.947 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.947 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.947 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.948 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.948 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.948 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.948 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.948 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.948 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.948 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.948 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.949 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.949 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.949 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.949 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.949 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.949 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.949 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.949 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.950 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.950 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.950 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.950 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.950 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.950 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.950 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.950 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.951 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.951 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.951 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "osd_support", 2024-06-09T22:31:33.951 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.951 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.951 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.951 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.952 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.952 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.952 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.952 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.952 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.952 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.952 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.952 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.953 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.953 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.953 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.953 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.953 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.953 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.953 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.953 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.954 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.954 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.954 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.954 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.954 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.954 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.954 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.955 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.955 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.955 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.955 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.955 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.955 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.955 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.955 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.955 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.956 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.956 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.956 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.956 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.956 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.956 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.956 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.956 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.957 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.957 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.957 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.957 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.957 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.957 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.957 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.958 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.958 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.958 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.958 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.958 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.958 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.958 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.959 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.959 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.959 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.959 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.959 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.959 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.959 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.959 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.959 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.960 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.960 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.960 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.960 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.960 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "pg_autoscaler", 2024-06-09T22:31:33.960 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.960 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.961 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.961 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.961 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.961 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.961 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.961 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.961 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.961 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.961 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.962 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.962 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.962 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.962 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.962 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.962 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.962 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.962 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.963 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.963 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.963 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.963 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.963 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.963 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.963 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.964 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.964 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.964 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.964 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.964 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.964 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.964 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.964 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.965 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.965 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.965 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.965 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.965 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.965 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.965 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.966 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.966 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.966 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.966 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.966 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.966 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.966 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.966 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.967 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.970 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.970 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.970 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:33.970 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.970 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.970 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.970 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.970 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.971 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.971 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.971 INFO:teuthology.orchestra.run.smithi104.stdout: "threshold": { 2024-06-09T22:31:33.971 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "threshold", 2024-06-09T22:31:33.971 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:33.971 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.971 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.971 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "3.0", 2024-06-09T22:31:33.972 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1.0", 2024-06-09T22:31:33.972 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.972 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.972 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "scaling threshold", 2024-06-09T22:31:33.972 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-06-09T22:31:33.972 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.972 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.972 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.973 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.973 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "progress", 2024-06-09T22:31:33.973 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.973 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.973 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.973 INFO:teuthology.orchestra.run.smithi104.stdout: "allow_pg_recovery_event": { 2024-06-09T22:31:33.974 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "allow_pg_recovery_event", 2024-06-09T22:31:33.974 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.974 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.974 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.974 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.974 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.974 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.974 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.975 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "allow the module to show pg recovery progress", 2024-06-09T22:31:33.975 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.975 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.975 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.975 INFO:teuthology.orchestra.run.smithi104.stdout: "enabled": { 2024-06-09T22:31:33.975 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "enabled", 2024-06-09T22:31:33.975 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.976 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.976 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.976 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.976 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.976 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.976 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.976 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.976 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.976 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.977 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.977 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.977 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.977 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.977 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.977 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.977 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.978 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.978 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.978 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.978 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.978 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.978 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.978 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.978 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.979 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.979 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.979 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.979 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.979 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.979 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.979 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.980 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.980 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.980 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.980 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.980 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.980 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.980 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.980 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.980 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.981 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.981 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.981 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.981 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.981 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.981 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.981 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.982 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.982 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.982 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.982 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.982 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.982 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.982 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.982 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.983 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.983 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.983 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.983 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.983 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.983 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.983 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.983 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.984 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:33.984 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:33.984 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.984 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.984 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.984 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.984 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.984 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.985 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.985 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.985 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.985 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.985 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.985 INFO:teuthology.orchestra.run.smithi104.stdout: "max_completed_events": { 2024-06-09T22:31:33.985 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_completed_events", 2024-06-09T22:31:33.986 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:33.986 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.986 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.986 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "50", 2024-06-09T22:31:33.986 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.986 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.986 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.986 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "number of past completed events to remember", 2024-06-09T22:31:33.987 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.987 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.987 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.987 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.987 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:33.987 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:33.987 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:33.987 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.987 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.988 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5", 2024-06-09T22:31:33.988 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.988 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.988 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.988 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how long the module is going to sleep", 2024-06-09T22:31:33.988 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.988 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.989 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.989 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.989 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:33.989 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.989 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:33.989 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "prometheus", 2024-06-09T22:31:33.989 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:33.989 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:33.989 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:33.990 INFO:teuthology.orchestra.run.smithi104.stdout: "cache": { 2024-06-09T22:31:33.990 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cache", 2024-06-09T22:31:33.990 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.990 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.990 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:33.990 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.990 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.990 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.991 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.991 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.991 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.991 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.991 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.991 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.991 INFO:teuthology.orchestra.run.smithi104.stdout: "exclude_perf_counters": { 2024-06-09T22:31:33.991 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "exclude_perf_counters", 2024-06-09T22:31:33.991 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.992 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.992 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.992 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:33.992 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.992 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.992 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.992 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-06-09T22:31:33.992 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-06-09T22:31:33.993 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.993 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.993 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:33.993 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:33.993 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.993 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.993 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.993 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:33.994 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.994 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.994 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.994 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.994 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.994 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.994 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.994 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.994 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.995 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.995 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.995 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.995 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.995 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.995 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.995 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:33.995 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:33.996 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:33.996 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.996 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.996 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:33.996 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.996 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.996 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:33.996 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.997 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.997 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.997 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.997 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:33.997 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:33.997 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:33.997 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:33.997 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:33.998 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:33.998 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:33.998 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:33.998 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:33.998 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:33.998 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:33.998 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:33.998 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:33.999 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:33.999 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:33.999 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:33.999 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:33.999 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:33.999 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:33.999 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:33.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:33.999 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.000 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.000 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.000 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.000 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.000 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.000 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.000 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.000 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.000 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.001 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.001 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.001 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.001 INFO:teuthology.orchestra.run.smithi104.stdout: "rbd_stats_pools": { 2024-06-09T22:31:34.001 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rbd_stats_pools", 2024-06-09T22:31:34.001 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.001 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.001 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.002 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.002 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.002 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.002 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.002 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.002 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.002 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.002 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.003 INFO:teuthology.orchestra.run.smithi104.stdout: "rbd_stats_pools_refresh_interval": { 2024-06-09T22:31:34.003 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-06-09T22:31:34.003 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.003 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.003 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.003 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "300", 2024-06-09T22:31:34.003 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.004 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.004 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.004 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.004 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.004 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.004 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.004 INFO:teuthology.orchestra.run.smithi104.stdout: "scrape_interval": { 2024-06-09T22:31:34.005 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "scrape_interval", 2024-06-09T22:31:34.005 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:34.005 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.005 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.005 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "15.0", 2024-06-09T22:31:34.005 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.005 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.005 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.005 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.006 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.006 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.006 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.006 INFO:teuthology.orchestra.run.smithi104.stdout: "server_addr": { 2024-06-09T22:31:34.006 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_addr", 2024-06-09T22:31:34.006 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.006 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.007 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.007 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "::", 2024-06-09T22:31:34.007 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.007 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.007 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.007 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-06-09T22:31:34.007 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.007 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.007 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.008 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.008 INFO:teuthology.orchestra.run.smithi104.stdout: "server_port": { 2024-06-09T22:31:34.008 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_port", 2024-06-09T22:31:34.008 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.008 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.008 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.008 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "9283", 2024-06-09T22:31:34.008 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.009 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.009 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.009 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-06-09T22:31:34.009 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.009 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.009 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.009 INFO:teuthology.orchestra.run.smithi104.stdout: "stale_cache_strategy": { 2024-06-09T22:31:34.009 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "stale_cache_strategy", 2024-06-09T22:31:34.010 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.010 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.010 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.010 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "log", 2024-06-09T22:31:34.010 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.010 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.010 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.011 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.011 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.011 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.011 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.011 INFO:teuthology.orchestra.run.smithi104.stdout: "standby_behaviour": { 2024-06-09T22:31:34.011 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "standby_behaviour", 2024-06-09T22:31:34.011 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.011 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.011 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.012 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "default", 2024-06-09T22:31:34.012 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.012 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.012 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.012 INFO:teuthology.orchestra.run.smithi104.stdout: "default", 2024-06-09T22:31:34.012 INFO:teuthology.orchestra.run.smithi104.stdout: "error" 2024-06-09T22:31:34.012 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.012 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.013 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.013 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.013 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.013 INFO:teuthology.orchestra.run.smithi104.stdout: "standby_error_status_code": { 2024-06-09T22:31:34.013 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "standby_error_status_code", 2024-06-09T22:31:34.013 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.013 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.014 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.014 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "500", 2024-06-09T22:31:34.014 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "400", 2024-06-09T22:31:34.014 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "599", 2024-06-09T22:31:34.014 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.014 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.014 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.014 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.014 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.015 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.015 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.015 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rbd_support", 2024-06-09T22:31:34.015 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.015 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.016 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.016 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.016 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.016 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.016 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.016 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.016 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.016 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.017 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.017 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.017 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.017 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.017 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.017 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.017 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.017 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.017 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.018 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.018 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.018 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.018 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.018 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.018 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.018 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.019 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.019 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.019 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.019 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.019 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.019 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.019 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.019 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.020 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.020 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.020 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.020 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.020 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.020 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.020 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.021 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.021 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.021 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.021 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.021 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.021 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.021 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.021 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.022 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.022 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.022 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.022 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.022 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.022 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.022 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.022 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.023 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.023 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.023 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.023 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.023 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.023 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.023 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.023 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.024 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.024 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.024 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.024 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.024 INFO:teuthology.orchestra.run.smithi104.stdout: "max_concurrent_snap_create": { 2024-06-09T22:31:34.024 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_concurrent_snap_create", 2024-06-09T22:31:34.024 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.025 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.025 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.025 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:34.025 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.025 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.025 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.025 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.025 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.026 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.026 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.026 INFO:teuthology.orchestra.run.smithi104.stdout: "mirror_snapshot_schedule": { 2024-06-09T22:31:34.026 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mirror_snapshot_schedule", 2024-06-09T22:31:34.026 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.026 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.026 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.027 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.027 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.027 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.027 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.027 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.027 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.027 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.027 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.028 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.028 INFO:teuthology.orchestra.run.smithi104.stdout: "trash_purge_schedule": { 2024-06-09T22:31:34.028 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "trash_purge_schedule", 2024-06-09T22:31:34.028 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.028 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.028 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.028 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.028 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.029 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.029 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.029 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.029 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.029 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.029 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.029 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.029 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.030 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "restful", 2024-06-09T22:31:34.030 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.030 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.030 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.030 INFO:teuthology.orchestra.run.smithi104.stdout: "enable_auth": { 2024-06-09T22:31:34.030 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "enable_auth", 2024-06-09T22:31:34.031 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.031 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.031 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.031 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.031 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.031 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.031 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.031 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.032 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.032 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.032 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.032 INFO:teuthology.orchestra.run.smithi104.stdout: "key_file": { 2024-06-09T22:31:34.032 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "key_file", 2024-06-09T22:31:34.032 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.032 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.032 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.033 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.033 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.033 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.033 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.033 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.033 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.034 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.034 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.034 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.034 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.034 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.034 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.035 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.035 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.035 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.035 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.035 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.035 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.035 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.035 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.035 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.036 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.036 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.036 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.036 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.036 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.036 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.036 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.037 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.037 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.037 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.037 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.037 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.037 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.037 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.037 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.038 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.038 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.038 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.038 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.038 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.038 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.038 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.038 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.039 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.039 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.039 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.039 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.039 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.039 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.039 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.039 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.039 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.040 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.040 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.040 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.040 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.040 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.040 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.040 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.040 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.041 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.041 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.041 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.041 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.041 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.041 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.041 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.041 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.042 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.042 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.042 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.042 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.042 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.042 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.042 INFO:teuthology.orchestra.run.smithi104.stdout: "server_addr": { 2024-06-09T22:31:34.043 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_addr", 2024-06-09T22:31:34.043 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.043 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.043 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.043 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.043 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.043 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.043 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.043 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.044 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.044 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.044 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.044 INFO:teuthology.orchestra.run.smithi104.stdout: "server_port": { 2024-06-09T22:31:34.044 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_port", 2024-06-09T22:31:34.044 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.044 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.044 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.045 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.045 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.045 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.045 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.045 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.045 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.045 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.045 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.045 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.046 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.046 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rgw", 2024-06-09T22:31:34.046 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.046 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.046 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.046 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.047 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.047 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.047 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.047 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.047 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.047 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.047 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.047 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.047 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.048 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.048 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.048 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.048 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.048 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.048 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.048 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.048 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.048 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.049 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.049 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.049 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.049 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.049 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.049 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.049 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.049 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.050 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.050 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.050 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.050 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.050 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.050 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.050 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.051 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.051 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.051 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.051 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.051 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.051 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.051 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.051 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.051 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.053 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.053 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.053 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.053 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.053 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.053 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.054 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.054 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.054 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.054 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.054 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.054 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.054 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.054 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.055 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.055 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.055 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.055 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.055 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.055 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.055 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.055 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.055 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.056 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.056 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.056 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.056 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rook", 2024-06-09T22:31:34.056 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.056 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.057 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.057 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.057 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.057 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.057 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.057 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.057 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.057 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.058 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.058 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.058 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.058 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.058 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.058 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.058 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.059 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.059 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.059 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.059 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.059 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.059 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.059 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.060 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.060 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.060 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.060 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.060 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.060 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.060 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.061 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.061 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.061 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.061 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.061 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.061 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.061 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.062 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.062 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.062 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.062 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.062 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.062 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.062 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.062 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.063 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.063 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.063 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.063 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.063 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.063 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.063 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.063 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.064 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.064 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.064 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.064 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.064 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.064 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.064 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.065 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.065 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.065 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.065 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.065 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.065 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.065 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.065 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.066 INFO:teuthology.orchestra.run.smithi104.stdout: "storage_class": { 2024-06-09T22:31:34.066 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "storage_class", 2024-06-09T22:31:34.066 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.066 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.066 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.066 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "local", 2024-06-09T22:31:34.066 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.067 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.067 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.067 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-06-09T22:31:34.067 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.067 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.067 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.067 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.067 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.068 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "selftest", 2024-06-09T22:31:34.068 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.068 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.068 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.068 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.068 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.068 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "roption1": { 2024-06-09T22:31:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "roption1", 2024-06-09T22:31:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "roption2": { 2024-06-09T22:31:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "roption2", 2024-06-09T22:31:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "xyz", 2024-06-09T22:31:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption1": { 2024-06-09T22:31:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption1", 2024-06-09T22:31:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption2": { 2024-06-09T22:31:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption2", 2024-06-09T22:31:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption3": { 2024-06-09T22:31:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption3", 2024-06-09T22:31:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption4": { 2024-06-09T22:31:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption4", 2024-06-09T22:31:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption5": { 2024-06-09T22:31:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption5", 2024-06-09T22:31:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption6": { 2024-06-09T22:31:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption6", 2024-06-09T22:31:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption7": { 2024-06-09T22:31:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption7", 2024-06-09T22:31:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1", 2024-06-09T22:31:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "42", 2024-06-09T22:31:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "testkey": { 2024-06-09T22:31:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "testkey", 2024-06-09T22:31:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "testlkey": { 2024-06-09T22:31:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "testlkey", 2024-06-09T22:31:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "testnewline": { 2024-06-09T22:31:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "testnewline", 2024-06-09T22:31:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "snap_schedule", 2024-06-09T22:31:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "allow_m_granularity": { 2024-06-09T22:31:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "allow_m_granularity", 2024-06-09T22:31:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "allow minute scheduled snapshots", 2024-06-09T22:31:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "dump_on_update": { 2024-06-09T22:31:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "dump_on_update", 2024-06-09T22:31:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "dump database to debug log on update", 2024-06-09T22:31:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "stats", 2024-06-09T22:31:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "status", 2024-06-09T22:31:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "telegraf", 2024-06-09T22:31:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "address": { 2024-06-09T22:31:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "address", 2024-06-09T22:31:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-06-09T22:31:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "15", 2024-06-09T22:31:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "telemetry", 2024-06-09T22:31:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_basic": { 2024-06-09T22:31:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_basic", 2024-06-09T22:31:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share basic cluster information (size, version)", 2024-06-09T22:31:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_crash": { 2024-06-09T22:31:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_crash", 2024-06-09T22:31:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-06-09T22:31:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_device": { 2024-06-09T22:31:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_device", 2024-06-09T22:31:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-06-09T22:31:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_ident": { 2024-06-09T22:31:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_ident", 2024-06-09T22:31:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-06-09T22:31:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_perf": { 2024-06-09T22:31:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_perf", 2024-06-09T22:31:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share various performance metrics of a cluster", 2024-06-09T22:31:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "contact": { 2024-06-09T22:31:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "contact", 2024-06-09T22:31:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "description": { 2024-06-09T22:31:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "description", 2024-06-09T22:31:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "device_url": { 2024-06-09T22:31:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "device_url", 2024-06-09T22:31:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-06-09T22:31:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "enabled": { 2024-06-09T22:31:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "enabled", 2024-06-09T22:31:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "24", 2024-06-09T22:31:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "8", 2024-06-09T22:31:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "last_opt_revision": { 2024-06-09T22:31:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "last_opt_revision", 2024-06-09T22:31:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "1", 2024-06-09T22:31:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "leaderboard": { 2024-06-09T22:31:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "leaderboard", 2024-06-09T22:31:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "leaderboard_description": { 2024-06-09T22:31:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "leaderboard_description", 2024-06-09T22:31:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.168 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.168 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.168 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.168 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.168 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.168 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.168 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.168 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "organization": { 2024-06-09T22:31:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "organization", 2024-06-09T22:31:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "proxy": { 2024-06-09T22:31:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "proxy", 2024-06-09T22:31:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "url": { 2024-06-09T22:31:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "url", 2024-06-09T22:31:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-06-09T22:31:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "test_orchestrator", 2024-06-09T22:31:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "volumes", 2024-06-09T22:31:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.192 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.192 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.192 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.192 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.192 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.192 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.192 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.193 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.193 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.193 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.193 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.193 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.193 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.193 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.193 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.193 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.194 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.194 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.194 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.194 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.194 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.194 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.194 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.195 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.195 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.195 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.195 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.195 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.195 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.195 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.195 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.196 INFO:teuthology.orchestra.run.smithi104.stdout: "max_concurrent_clones": { 2024-06-09T22:31:34.196 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_concurrent_clones", 2024-06-09T22:31:34.196 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.196 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.196 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.196 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "4", 2024-06-09T22:31:34.196 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.197 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.197 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.197 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Number of asynchronous cloner threads", 2024-06-09T22:31:34.197 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.197 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.197 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.198 INFO:teuthology.orchestra.run.smithi104.stdout: "periodic_async_work": { 2024-06-09T22:31:34.198 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "periodic_async_work", 2024-06-09T22:31:34.198 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.198 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.198 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.198 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.198 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.198 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.199 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.199 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Periodically check for async work", 2024-06-09T22:31:34.199 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.199 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.199 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.199 INFO:teuthology.orchestra.run.smithi104.stdout: "snapshot_clone_delay": { 2024-06-09T22:31:34.200 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "snapshot_clone_delay", 2024-06-09T22:31:34.200 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.200 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.200 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.200 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:34.200 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.200 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.200 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.200 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-06-09T22:31:34.201 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.201 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.201 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.201 INFO:teuthology.orchestra.run.smithi104.stdout: "snapshot_clone_no_wait": { 2024-06-09T22:31:34.201 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "snapshot_clone_no_wait", 2024-06-09T22:31:34.201 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.201 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.201 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.202 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.202 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.202 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.202 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.202 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-06-09T22:31:34.202 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.202 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.202 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.203 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.203 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.203 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "zabbix", 2024-06-09T22:31:34.203 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.203 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.203 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.203 INFO:teuthology.orchestra.run.smithi104.stdout: "discovery_interval": { 2024-06-09T22:31:34.204 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "discovery_interval", 2024-06-09T22:31:34.204 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:34.204 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.204 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.204 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "100", 2024-06-09T22:31:34.204 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.204 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.204 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.204 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.205 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.205 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.205 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.205 INFO:teuthology.orchestra.run.smithi104.stdout: "identifier": { 2024-06-09T22:31:34.205 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "identifier", 2024-06-09T22:31:34.205 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.205 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.206 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.206 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.206 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.206 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.206 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.206 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.206 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.206 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.207 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.207 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:34.207 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:34.207 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.207 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.207 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.207 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:34.208 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.208 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.208 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.208 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.208 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.208 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.208 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.209 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.209 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.209 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.209 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.209 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.209 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.209 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.209 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.210 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.210 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.210 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.210 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.210 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.210 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.210 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.210 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.211 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.211 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.211 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.211 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.211 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.211 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.211 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.212 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.212 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.212 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.212 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.212 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.212 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.212 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.212 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.213 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.213 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.213 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.213 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.213 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.213 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.213 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.214 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.214 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.214 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.214 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.214 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.214 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.214 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.214 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.215 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.215 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.215 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.215 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.215 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.215 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.215 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.215 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.216 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.216 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.216 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.216 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.216 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.216 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.216 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.217 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.217 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.217 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.217 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.217 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.217 INFO:teuthology.orchestra.run.smithi104.stdout: "zabbix_host": { 2024-06-09T22:31:34.217 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "zabbix_host", 2024-06-09T22:31:34.218 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.218 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.218 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.218 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.218 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.218 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.218 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.219 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.219 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.219 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.219 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.219 INFO:teuthology.orchestra.run.smithi104.stdout: "zabbix_port": { 2024-06-09T22:31:34.219 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "zabbix_port", 2024-06-09T22:31:34.219 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.220 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.220 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.220 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10051", 2024-06-09T22:31:34.220 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.220 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.220 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.220 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.220 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.221 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.221 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.221 INFO:teuthology.orchestra.run.smithi104.stdout: "zabbix_sender": { 2024-06-09T22:31:34.221 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "zabbix_sender", 2024-06-09T22:31:34.221 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.221 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.221 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.222 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-06-09T22:31:34.222 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.222 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.222 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.222 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.222 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.222 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.223 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.223 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.223 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.223 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.223 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-06-09T22:31:34.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.224 INFO:teuthology.orchestra.run.smithi104.stdout: "gid": 24907, 2024-06-09T22:31:34.224 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "z", 2024-06-09T22:31:34.224 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr_features": 4540701547738038271, 2024-06-09T22:31:34.224 INFO:teuthology.orchestra.run.smithi104.stdout: "available_modules": [ 2024-06-09T22:31:34.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.224 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "alerts", 2024-06-09T22:31:34.224 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.225 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.225 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.225 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:34.225 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:34.225 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.225 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.225 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.226 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:34.226 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.226 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.226 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.226 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "How frequently to reexamine health status", 2024-06-09T22:31:34.226 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.226 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.227 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.227 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.227 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.227 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.227 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.227 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.228 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.228 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.228 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.228 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.228 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.228 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.228 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.230 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.230 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.230 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.230 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.230 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.230 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.230 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.231 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.231 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.231 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.231 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.231 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.231 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.231 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.232 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.232 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.232 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.232 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.232 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.232 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.232 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.232 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.233 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.233 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.233 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.233 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.233 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.233 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.233 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.233 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.234 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.234 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.234 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.234 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.234 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.234 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.235 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.235 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.235 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.235 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.235 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.235 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.235 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.235 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.236 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.236 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.236 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.236 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_destination": { 2024-06-09T22:31:34.236 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_destination", 2024-06-09T22:31:34.236 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.236 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.237 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.237 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.237 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.237 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.237 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.237 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Email address to send alerts to", 2024-06-09T22:31:34.237 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.238 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.238 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.238 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_from_name": { 2024-06-09T22:31:34.238 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_from_name", 2024-06-09T22:31:34.238 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.238 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.238 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.238 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "Ceph", 2024-06-09T22:31:34.239 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.239 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.239 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.239 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Email From: name", 2024-06-09T22:31:34.239 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.239 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.239 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.239 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_host": { 2024-06-09T22:31:34.240 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_host", 2024-06-09T22:31:34.240 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.240 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.240 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.240 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.240 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.240 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.240 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.241 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "SMTP server", 2024-06-09T22:31:34.241 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.241 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.241 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.241 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_password": { 2024-06-09T22:31:34.241 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_password", 2024-06-09T22:31:34.241 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.241 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.242 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.242 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.242 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.242 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.242 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.242 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Password to authenticate with", 2024-06-09T22:31:34.242 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.242 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.243 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.243 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_port": { 2024-06-09T22:31:34.243 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_port", 2024-06-09T22:31:34.243 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.243 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.243 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.243 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "465", 2024-06-09T22:31:34.243 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.244 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.244 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.244 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "SMTP port", 2024-06-09T22:31:34.244 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.244 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.244 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.244 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_sender": { 2024-06-09T22:31:34.244 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_sender", 2024-06-09T22:31:34.245 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.245 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.245 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.245 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.245 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.245 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.245 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.245 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "SMTP envelope sender", 2024-06-09T22:31:34.246 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.246 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.246 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.246 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_ssl": { 2024-06-09T22:31:34.246 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_ssl", 2024-06-09T22:31:34.246 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.247 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.247 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.247 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.247 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.247 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.247 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.247 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Use SSL to connect to SMTP server", 2024-06-09T22:31:34.247 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.248 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.248 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.248 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_user": { 2024-06-09T22:31:34.248 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_user", 2024-06-09T22:31:34.248 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.248 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.248 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.248 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.249 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.249 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.249 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.249 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "User to authenticate as", 2024-06-09T22:31:34.249 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.249 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.249 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.249 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.249 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.250 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "balancer", 2024-06-09T22:31:34.250 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.250 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.250 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.250 INFO:teuthology.orchestra.run.smithi104.stdout: "active": { 2024-06-09T22:31:34.250 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "active", 2024-06-09T22:31:34.250 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.251 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.251 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.251 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.251 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.251 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.251 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.251 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "automatically balance PGs across cluster", 2024-06-09T22:31:34.251 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.252 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.252 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.252 INFO:teuthology.orchestra.run.smithi104.stdout: "begin_time": { 2024-06-09T22:31:34.252 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "begin_time", 2024-06-09T22:31:34.252 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.252 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.252 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.253 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0000", 2024-06-09T22:31:34.253 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.253 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.253 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.253 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "beginning time of day to automatically balance", 2024-06-09T22:31:34.253 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-09T22:31:34.253 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.253 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.254 INFO:teuthology.orchestra.run.smithi104.stdout: "begin_weekday": { 2024-06-09T22:31:34.254 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "begin_weekday", 2024-06-09T22:31:34.254 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:34.254 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.254 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.254 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:34.254 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "0", 2024-06-09T22:31:34.254 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "6", 2024-06-09T22:31:34.255 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.255 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-06-09T22:31:34.255 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-09T22:31:34.255 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.255 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.255 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_compat_max_iterations": { 2024-06-09T22:31:34.255 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crush_compat_max_iterations", 2024-06-09T22:31:34.257 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:34.257 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.257 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.257 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "25", 2024-06-09T22:31:34.258 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1", 2024-06-09T22:31:34.258 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "250", 2024-06-09T22:31:34.258 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.258 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-06-09T22:31:34.258 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.258 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.258 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.258 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_compat_metrics": { 2024-06-09T22:31:34.259 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crush_compat_metrics", 2024-06-09T22:31:34.259 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.259 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.259 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.259 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "pgs,objects,bytes", 2024-06-09T22:31:34.259 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.259 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.259 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.260 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-06-09T22:31:34.260 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-06-09T22:31:34.260 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.260 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.260 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_compat_step": { 2024-06-09T22:31:34.260 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crush_compat_step", 2024-06-09T22:31:34.260 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:34.260 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.261 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.261 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0.5", 2024-06-09T22:31:34.261 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "0.001", 2024-06-09T22:31:34.261 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "0.999", 2024-06-09T22:31:34.261 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.261 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "aggressiveness of optimization", 2024-06-09T22:31:34.261 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-06-09T22:31:34.261 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.262 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.262 INFO:teuthology.orchestra.run.smithi104.stdout: "end_time": { 2024-06-09T22:31:34.262 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "end_time", 2024-06-09T22:31:34.262 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.262 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.262 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.262 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "2359", 2024-06-09T22:31:34.263 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.263 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.263 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.263 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "ending time of day to automatically balance", 2024-06-09T22:31:34.263 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-09T22:31:34.263 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.263 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.264 INFO:teuthology.orchestra.run.smithi104.stdout: "end_weekday": { 2024-06-09T22:31:34.264 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "end_weekday", 2024-06-09T22:31:34.264 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:34.264 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.264 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.264 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:34.264 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "0", 2024-06-09T22:31:34.264 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "6", 2024-06-09T22:31:34.264 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.265 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-06-09T22:31:34.265 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-09T22:31:34.265 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.265 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.265 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.265 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.265 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.266 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.266 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.266 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.266 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.266 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.266 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.266 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.266 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.267 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.267 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.267 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.267 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.267 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.267 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.267 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.268 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.268 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.268 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.268 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.268 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.268 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.268 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.268 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.269 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.269 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.269 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.269 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.269 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.269 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.269 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.270 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.270 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.270 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.270 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.270 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.270 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.270 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.270 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.271 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.271 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.271 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.271 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.271 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.271 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.271 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.271 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.272 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.272 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.272 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.272 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.272 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.272 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.272 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.273 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.273 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.273 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.273 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.273 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.273 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.273 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.273 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.274 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.274 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.274 INFO:teuthology.orchestra.run.smithi104.stdout: "min_score": { 2024-06-09T22:31:34.274 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "min_score", 2024-06-09T22:31:34.274 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:34.274 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.274 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.275 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:34.275 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.275 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.275 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.275 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-06-09T22:31:34.275 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.275 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.275 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.276 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": { 2024-06-09T22:31:34.276 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mode", 2024-06-09T22:31:34.276 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.276 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.276 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.276 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "upmap", 2024-06-09T22:31:34.276 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.277 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.277 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.277 INFO:teuthology.orchestra.run.smithi104.stdout: "crush-compat", 2024-06-09T22:31:34.277 INFO:teuthology.orchestra.run.smithi104.stdout: "none", 2024-06-09T22:31:34.277 INFO:teuthology.orchestra.run.smithi104.stdout: "read", 2024-06-09T22:31:34.277 INFO:teuthology.orchestra.run.smithi104.stdout: "upmap", 2024-06-09T22:31:34.277 INFO:teuthology.orchestra.run.smithi104.stdout: "upmap-read" 2024-06-09T22:31:34.277 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.278 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Balancer mode", 2024-06-09T22:31:34.278 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.278 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.278 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.278 INFO:teuthology.orchestra.run.smithi104.stdout: "pool_ids": { 2024-06-09T22:31:34.278 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "pool_ids", 2024-06-09T22:31:34.278 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.279 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.279 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.279 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.279 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.279 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.279 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.279 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-06-09T22:31:34.279 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.279 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.280 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.280 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:34.280 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:34.280 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.280 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.280 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.280 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:34.281 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.281 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.281 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.281 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-06-09T22:31:34.281 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.281 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.281 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.282 INFO:teuthology.orchestra.run.smithi104.stdout: "upmap_max_deviation": { 2024-06-09T22:31:34.282 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "upmap_max_deviation", 2024-06-09T22:31:34.282 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.282 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.282 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.282 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5", 2024-06-09T22:31:34.282 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1", 2024-06-09T22:31:34.282 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.283 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.283 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "deviation below which no optimization is attempted", 2024-06-09T22:31:34.283 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-06-09T22:31:34.283 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.283 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.283 INFO:teuthology.orchestra.run.smithi104.stdout: "upmap_max_optimizations": { 2024-06-09T22:31:34.283 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "upmap_max_optimizations", 2024-06-09T22:31:34.283 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:34.284 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.284 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.284 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:34.284 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.284 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.284 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.284 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-06-09T22:31:34.284 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.284 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.285 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.285 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.285 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.285 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cephadm", 2024-06-09T22:31:34.285 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.285 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.286 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.286 INFO:teuthology.orchestra.run.smithi104.stdout: "agent_down_multiplier": { 2024-06-09T22:31:34.286 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "agent_down_multiplier", 2024-06-09T22:31:34.286 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:34.286 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.286 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.286 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "3.0", 2024-06-09T22:31:34.286 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.287 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.287 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.287 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-06-09T22:31:34.287 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.287 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.287 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.287 INFO:teuthology.orchestra.run.smithi104.stdout: "agent_refresh_rate": { 2024-06-09T22:31:34.287 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "agent_refresh_rate", 2024-06-09T22:31:34.288 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.288 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.288 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.288 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "20", 2024-06-09T22:31:34.288 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.288 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.288 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.288 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-06-09T22:31:34.288 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.289 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.289 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.289 INFO:teuthology.orchestra.run.smithi104.stdout: "agent_starting_port": { 2024-06-09T22:31:34.289 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "agent_starting_port", 2024-06-09T22:31:34.289 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.289 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.289 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.290 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "4721", 2024-06-09T22:31:34.290 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.290 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.290 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.290 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-06-09T22:31:34.290 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.290 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.291 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.291 INFO:teuthology.orchestra.run.smithi104.stdout: "allow_ptrace": { 2024-06-09T22:31:34.291 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "allow_ptrace", 2024-06-09T22:31:34.291 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.291 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.291 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.291 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.291 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.292 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.292 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.292 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-06-09T22:31:34.292 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-06-09T22:31:34.292 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.292 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.292 INFO:teuthology.orchestra.run.smithi104.stdout: "autotune_interval": { 2024-06-09T22:31:34.293 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "autotune_interval", 2024-06-09T22:31:34.293 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.293 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.293 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.293 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:34.293 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.293 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.293 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.294 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to autotune daemon memory", 2024-06-09T22:31:34.294 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.294 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.294 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.294 INFO:teuthology.orchestra.run.smithi104.stdout: "autotune_memory_target_ratio": { 2024-06-09T22:31:34.294 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "autotune_memory_target_ratio", 2024-06-09T22:31:34.294 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:34.295 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.295 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.295 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0.7", 2024-06-09T22:31:34.295 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.295 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.295 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.295 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-06-09T22:31:34.295 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.296 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.296 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.296 INFO:teuthology.orchestra.run.smithi104.stdout: "cephadm_log_destination": { 2024-06-09T22:31:34.296 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cephadm_log_destination", 2024-06-09T22:31:34.296 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.296 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.296 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.296 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.297 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.297 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.297 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.297 INFO:teuthology.orchestra.run.smithi104.stdout: "file", 2024-06-09T22:31:34.297 INFO:teuthology.orchestra.run.smithi104.stdout: "file,syslog", 2024-06-09T22:31:34.297 INFO:teuthology.orchestra.run.smithi104.stdout: "syslog" 2024-06-09T22:31:34.297 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.297 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-06-09T22:31:34.297 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.298 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.298 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.298 INFO:teuthology.orchestra.run.smithi104.stdout: "cgroups_split": { 2024-06-09T22:31:34.298 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cgroups_split", 2024-06-09T22:31:34.298 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.298 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.298 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.298 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.299 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.299 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.299 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.299 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-06-09T22:31:34.299 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.299 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.299 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.299 INFO:teuthology.orchestra.run.smithi104.stdout: "config_checks_enabled": { 2024-06-09T22:31:34.300 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "config_checks_enabled", 2024-06-09T22:31:34.300 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.300 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.300 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.300 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.300 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.300 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.300 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.300 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-06-09T22:31:34.301 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.301 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.301 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.301 INFO:teuthology.orchestra.run.smithi104.stdout: "config_dashboard": { 2024-06-09T22:31:34.301 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "config_dashboard", 2024-06-09T22:31:34.301 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.301 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.301 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.302 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.302 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.302 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.302 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.302 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-06-09T22:31:34.302 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.302 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.302 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.303 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_alertmanager": { 2024-06-09T22:31:34.303 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_alertmanager", 2024-06-09T22:31:34.303 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.303 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.303 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.303 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-06-09T22:31:34.303 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.303 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.304 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.304 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Prometheus container image", 2024-06-09T22:31:34.304 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.304 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.304 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.304 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_base": { 2024-06-09T22:31:34.304 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_base", 2024-06-09T22:31:34.305 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.305 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.305 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.305 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/ceph", 2024-06-09T22:31:34.305 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.305 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.305 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.305 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Container image name, without the tag", 2024-06-09T22:31:34.305 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.306 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.306 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.306 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_elasticsearch": { 2024-06-09T22:31:34.306 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_elasticsearch", 2024-06-09T22:31:34.306 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.306 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.306 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.307 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-06-09T22:31:34.307 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.307 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.307 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.307 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "elasticsearch container image", 2024-06-09T22:31:34.307 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.307 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.307 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.308 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_grafana": { 2024-06-09T22:31:34.308 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_grafana", 2024-06-09T22:31:34.308 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.308 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.308 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.308 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-06-09T22:31:34.308 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.308 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.308 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.309 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Prometheus container image", 2024-06-09T22:31:34.309 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.309 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.309 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.309 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_haproxy": { 2024-06-09T22:31:34.309 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_haproxy", 2024-06-09T22:31:34.309 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.310 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.310 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.310 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-06-09T22:31:34.310 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.310 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.310 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.310 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "HAproxy container image", 2024-06-09T22:31:34.310 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.311 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.311 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.311 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_jaeger_agent": { 2024-06-09T22:31:34.311 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_jaeger_agent", 2024-06-09T22:31:34.311 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.311 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.311 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.312 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-06-09T22:31:34.312 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.312 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.312 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.312 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Jaeger agent container image", 2024-06-09T22:31:34.312 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.312 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.312 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.313 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_jaeger_collector": { 2024-06-09T22:31:34.313 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_jaeger_collector", 2024-06-09T22:31:34.313 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.313 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.313 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.313 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-06-09T22:31:34.313 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.314 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.314 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.314 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Jaeger collector container image", 2024-06-09T22:31:34.314 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.314 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.314 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.314 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_jaeger_query": { 2024-06-09T22:31:34.315 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_jaeger_query", 2024-06-09T22:31:34.315 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.315 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.315 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.315 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-06-09T22:31:34.315 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.315 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.315 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.315 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Jaeger query container image", 2024-06-09T22:31:34.316 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.316 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.316 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.316 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_keepalived": { 2024-06-09T22:31:34.316 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_keepalived", 2024-06-09T22:31:34.316 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.317 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.317 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.317 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-06-09T22:31:34.317 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.317 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.317 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.317 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Keepalived container image", 2024-06-09T22:31:34.317 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.317 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.318 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.318 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_loki": { 2024-06-09T22:31:34.318 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_loki", 2024-06-09T22:31:34.318 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.318 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.318 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.318 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-06-09T22:31:34.319 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.319 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.319 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.319 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Loki container image", 2024-06-09T22:31:34.319 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.319 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.319 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.319 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_node_exporter": { 2024-06-09T22:31:34.320 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_node_exporter", 2024-06-09T22:31:34.320 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.320 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.320 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.320 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-06-09T22:31:34.320 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.320 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.320 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.320 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Prometheus container image", 2024-06-09T22:31:34.321 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.321 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.321 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.321 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_nvmeof": { 2024-06-09T22:31:34.321 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_nvmeof", 2024-06-09T22:31:34.321 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.321 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.322 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.322 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-06-09T22:31:34.322 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.322 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.322 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.322 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Nvme-of container image", 2024-06-09T22:31:34.322 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.322 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.323 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.323 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_prometheus": { 2024-06-09T22:31:34.323 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_prometheus", 2024-06-09T22:31:34.323 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.323 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.323 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.323 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-06-09T22:31:34.323 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.324 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.324 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.324 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Prometheus container image", 2024-06-09T22:31:34.324 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.324 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.324 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.324 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_promtail": { 2024-06-09T22:31:34.324 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_promtail", 2024-06-09T22:31:34.325 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.325 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.325 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.325 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-06-09T22:31:34.325 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.325 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.325 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.325 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Promtail container image", 2024-06-09T22:31:34.326 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.326 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.326 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.326 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_samba": { 2024-06-09T22:31:34.326 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_samba", 2024-06-09T22:31:34.326 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.326 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.327 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.327 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-06-09T22:31:34.327 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.327 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.327 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.327 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Samba/SMB container image", 2024-06-09T22:31:34.327 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.327 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.327 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.328 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_snmp_gateway": { 2024-06-09T22:31:34.328 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_snmp_gateway", 2024-06-09T22:31:34.328 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.328 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.328 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.328 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-06-09T22:31:34.328 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.329 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.329 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.329 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "SNMP Gateway container image", 2024-06-09T22:31:34.329 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.329 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.329 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.329 INFO:teuthology.orchestra.run.smithi104.stdout: "container_init": { 2024-06-09T22:31:34.329 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_init", 2024-06-09T22:31:34.330 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.330 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.330 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.330 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.330 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.330 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.330 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.331 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Run podman/docker with `--init`", 2024-06-09T22:31:34.331 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.331 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.331 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.331 INFO:teuthology.orchestra.run.smithi104.stdout: "daemon_cache_timeout": { 2024-06-09T22:31:34.331 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "daemon_cache_timeout", 2024-06-09T22:31:34.331 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.331 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.332 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.332 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:34.332 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.332 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.332 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.332 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-06-09T22:31:34.332 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.332 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.333 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.333 INFO:teuthology.orchestra.run.smithi104.stdout: "default_cephadm_command_timeout": { 2024-06-09T22:31:34.333 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "default_cephadm_command_timeout", 2024-06-09T22:31:34.333 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.333 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.333 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.333 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "900", 2024-06-09T22:31:34.334 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.334 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.334 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.334 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-06-09T22:31:34.334 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.334 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.334 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.335 INFO:teuthology.orchestra.run.smithi104.stdout: "default_registry": { 2024-06-09T22:31:34.335 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "default_registry", 2024-06-09T22:31:34.335 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.335 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.335 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.335 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "docker.io", 2024-06-09T22:31:34.335 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.335 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-06-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi104.stdout: "device_cache_timeout": { 2024-06-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "device_cache_timeout", 2024-06-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "1800", 2024-06-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "seconds to cache device inventory", 2024-06-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi104.stdout: "device_enhanced_scan": { 2024-06-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "device_enhanced_scan", 2024-06-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Use libstoragemgmt during device scans", 2024-06-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi104.stdout: "facts_cache_timeout": { 2024-06-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "facts_cache_timeout", 2024-06-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "seconds to cache host facts data", 2024-06-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.342 INFO:teuthology.orchestra.run.smithi104.stdout: "grafana_dashboards_path": { 2024-06-09T22:31:34.342 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "grafana_dashboards_path", 2024-06-09T22:31:34.342 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.342 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.342 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.342 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-06-09T22:31:34.342 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.342 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.343 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.343 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-06-09T22:31:34.343 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.343 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.343 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.343 INFO:teuthology.orchestra.run.smithi104.stdout: "host_check_interval": { 2024-06-09T22:31:34.343 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "host_check_interval", 2024-06-09T22:31:34.344 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.344 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.344 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.344 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:34.344 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.344 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.344 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.344 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to perform a host check", 2024-06-09T22:31:34.345 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.345 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.345 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.345 INFO:teuthology.orchestra.run.smithi104.stdout: "hw_monitoring": { 2024-06-09T22:31:34.345 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "hw_monitoring", 2024-06-09T22:31:34.345 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.345 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.345 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.346 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.346 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.346 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.346 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.346 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-06-09T22:31:34.346 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.346 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.346 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.347 INFO:teuthology.orchestra.run.smithi104.stdout: "inventory_list_all": { 2024-06-09T22:31:34.347 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "inventory_list_all", 2024-06-09T22:31:34.347 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.347 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.347 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.347 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.347 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.347 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.348 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.348 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-06-09T22:31:34.348 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.348 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.348 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.348 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.348 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.349 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.349 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.349 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.349 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.349 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.349 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.349 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.349 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.350 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.350 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.350 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.350 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.350 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.350 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.350 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.350 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.351 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.351 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.351 INFO:teuthology.orchestra.run.smithi104.stdout: "log_refresh_metadata": { 2024-06-09T22:31:34.351 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_refresh_metadata", 2024-06-09T22:31:34.351 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.351 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.351 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.351 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.351 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.352 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.352 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.352 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-06-09T22:31:34.352 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.352 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.352 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.353 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.353 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.353 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.353 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.353 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.353 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.353 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.353 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.354 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.354 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-06-09T22:31:34.354 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.354 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.354 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.354 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.354 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.355 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.355 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.355 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.355 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.355 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.355 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.355 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.355 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.356 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.356 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.356 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.356 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.356 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.356 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.356 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.356 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.356 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.357 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.357 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.357 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.357 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.357 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.357 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.357 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.358 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.358 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.358 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.358 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.358 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.358 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.358 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.359 INFO:teuthology.orchestra.run.smithi104.stdout: "manage_etc_ceph_ceph_conf": { 2024-06-09T22:31:34.359 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-06-09T22:31:34.359 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.359 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.359 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.359 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.359 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.359 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.360 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.360 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-06-09T22:31:34.360 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.360 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.360 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.360 INFO:teuthology.orchestra.run.smithi104.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-06-09T22:31:34.360 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-06-09T22:31:34.361 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.361 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.361 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.361 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "*", 2024-06-09T22:31:34.361 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.361 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.361 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.362 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-06-09T22:31:34.362 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.362 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.362 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.362 INFO:teuthology.orchestra.run.smithi104.stdout: "max_count_per_host": { 2024-06-09T22:31:34.362 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_count_per_host", 2024-06-09T22:31:34.363 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.363 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.363 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.363 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:34.363 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.363 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.363 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.364 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "max number of daemons per service per host", 2024-06-09T22:31:34.364 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.364 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.364 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.364 INFO:teuthology.orchestra.run.smithi104.stdout: "max_osd_draining_count": { 2024-06-09T22:31:34.365 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_osd_draining_count", 2024-06-09T22:31:34.365 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.365 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.365 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.365 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:34.365 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.365 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.365 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.365 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-06-09T22:31:34.366 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.366 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.366 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.366 INFO:teuthology.orchestra.run.smithi104.stdout: "migration_current": { 2024-06-09T22:31:34.366 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "migration_current", 2024-06-09T22:31:34.366 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.366 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.367 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.367 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.367 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.367 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.367 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.367 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "internal - do not modify", 2024-06-09T22:31:34.367 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.367 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.368 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.368 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": { 2024-06-09T22:31:34.368 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mode", 2024-06-09T22:31:34.368 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.368 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.368 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.368 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "root", 2024-06-09T22:31:34.369 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.369 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.369 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.369 INFO:teuthology.orchestra.run.smithi104.stdout: "cephadm-package", 2024-06-09T22:31:34.369 INFO:teuthology.orchestra.run.smithi104.stdout: "root" 2024-06-09T22:31:34.369 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.369 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "mode for remote execution of cephadm", 2024-06-09T22:31:34.369 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.370 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.370 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.370 INFO:teuthology.orchestra.run.smithi104.stdout: "oob_default_addr": { 2024-06-09T22:31:34.370 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "oob_default_addr", 2024-06-09T22:31:34.370 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.370 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.370 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.371 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "169.254.1.1", 2024-06-09T22:31:34.371 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.371 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.371 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.371 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Default address for RedFish API (oob management).", 2024-06-09T22:31:34.371 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.371 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.371 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.372 INFO:teuthology.orchestra.run.smithi104.stdout: "prometheus_alerts_path": { 2024-06-09T22:31:34.372 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "prometheus_alerts_path", 2024-06-09T22:31:34.372 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.372 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.372 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.372 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-06-09T22:31:34.372 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.372 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.372 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.373 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-06-09T22:31:34.373 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.373 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.373 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.373 INFO:teuthology.orchestra.run.smithi104.stdout: "registry_insecure": { 2024-06-09T22:31:34.373 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "registry_insecure", 2024-06-09T22:31:34.373 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.373 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.374 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.374 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.374 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.374 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.374 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.374 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-06-09T22:31:34.374 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.375 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.375 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.375 INFO:teuthology.orchestra.run.smithi104.stdout: "registry_password": { 2024-06-09T22:31:34.375 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "registry_password", 2024-06-09T22:31:34.375 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.375 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.375 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.376 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.376 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.376 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.376 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.376 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-06-09T22:31:34.376 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.376 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.376 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.377 INFO:teuthology.orchestra.run.smithi104.stdout: "registry_url": { 2024-06-09T22:31:34.377 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "registry_url", 2024-06-09T22:31:34.377 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.377 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.377 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.377 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.377 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.378 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.378 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.378 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-06-09T22:31:34.378 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.378 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.378 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.378 INFO:teuthology.orchestra.run.smithi104.stdout: "registry_username": { 2024-06-09T22:31:34.379 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "registry_username", 2024-06-09T22:31:34.379 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.379 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.379 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.379 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.379 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.379 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.379 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.380 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-06-09T22:31:34.380 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.380 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.380 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.380 INFO:teuthology.orchestra.run.smithi104.stdout: "secure_monitoring_stack": { 2024-06-09T22:31:34.380 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "secure_monitoring_stack", 2024-06-09T22:31:34.380 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.380 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.381 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.381 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.381 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.381 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.381 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.381 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-06-09T22:31:34.381 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.381 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.382 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.382 INFO:teuthology.orchestra.run.smithi104.stdout: "service_discovery_port": { 2024-06-09T22:31:34.382 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "service_discovery_port", 2024-06-09T22:31:34.382 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.382 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.382 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.382 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8765", 2024-06-09T22:31:34.383 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.383 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.383 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.383 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "cephadm service discovery port", 2024-06-09T22:31:34.383 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.383 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.383 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.383 INFO:teuthology.orchestra.run.smithi104.stdout: "ssh_config_file": { 2024-06-09T22:31:34.384 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ssh_config_file", 2024-06-09T22:31:34.384 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.384 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.384 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.384 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.384 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.384 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.384 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.385 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-06-09T22:31:34.385 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.385 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.385 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.385 INFO:teuthology.orchestra.run.smithi104.stdout: "use_agent": { 2024-06-09T22:31:34.385 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "use_agent", 2024-06-09T22:31:34.385 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.386 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.386 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.386 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.386 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.386 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.386 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.386 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-06-09T22:31:34.386 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.387 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.387 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.387 INFO:teuthology.orchestra.run.smithi104.stdout: "use_repo_digest": { 2024-06-09T22:31:34.387 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "use_repo_digest", 2024-06-09T22:31:34.387 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.387 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.387 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.387 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.388 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.388 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.388 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.388 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-06-09T22:31:34.388 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.388 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.388 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.389 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_on_failed_host_check": { 2024-06-09T22:31:34.389 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_on_failed_host_check", 2024-06-09T22:31:34.389 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.389 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.389 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.389 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.389 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.389 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.390 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.390 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "raise a health warning if the host check fails", 2024-06-09T22:31:34.390 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.390 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.390 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.390 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_on_stray_daemons": { 2024-06-09T22:31:34.390 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_on_stray_daemons", 2024-06-09T22:31:34.390 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.391 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.391 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.391 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.391 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.391 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.391 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.391 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-06-09T22:31:34.391 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.391 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.392 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.392 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_on_stray_hosts": { 2024-06-09T22:31:34.392 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_on_stray_hosts", 2024-06-09T22:31:34.392 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.392 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.392 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.392 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.393 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.393 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.393 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.393 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-06-09T22:31:34.393 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.393 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.393 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.393 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.393 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.394 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crash", 2024-06-09T22:31:34.394 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.394 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.394 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.394 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.394 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.395 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.395 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.395 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.395 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.395 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.395 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.395 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.395 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.396 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.396 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.396 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.396 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.396 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.396 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.396 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.397 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.397 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.397 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.397 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.397 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.397 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.397 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.397 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.398 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.398 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.398 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.398 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.398 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.398 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.398 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.398 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.399 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.399 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.399 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.399 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.399 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.399 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.399 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.399 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.400 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.400 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.400 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.400 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.400 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.400 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.400 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.400 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.401 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.401 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.401 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.401 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.401 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.401 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.401 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.402 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.402 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.402 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.402 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.402 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.402 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.402 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.402 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.403 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.403 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.403 INFO:teuthology.orchestra.run.smithi104.stdout: "retain_interval": { 2024-06-09T22:31:34.403 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "retain_interval", 2024-06-09T22:31:34.403 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.403 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.403 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.404 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "31536000", 2024-06-09T22:31:34.404 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.404 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.404 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.404 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how long to retain crashes before pruning them", 2024-06-09T22:31:34.404 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.404 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.404 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.405 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_recent_interval": { 2024-06-09T22:31:34.405 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_recent_interval", 2024-06-09T22:31:34.405 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.405 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.405 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.405 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "1209600", 2024-06-09T22:31:34.405 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.405 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.406 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.406 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "time interval in which to warn about recent crashes", 2024-06-09T22:31:34.406 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.406 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.406 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.406 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.406 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.407 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "dashboard", 2024-06-09T22:31:34.407 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.407 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.407 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.407 INFO:teuthology.orchestra.run.smithi104.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-06-09T22:31:34.407 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-06-09T22:31:34.407 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.408 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.408 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.408 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:34.408 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.408 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.408 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.408 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.408 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.409 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.409 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.409 INFO:teuthology.orchestra.run.smithi104.stdout: "ALERTMANAGER_API_HOST": { 2024-06-09T22:31:34.409 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ALERTMANAGER_API_HOST", 2024-06-09T22:31:34.409 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.409 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.409 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.410 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.410 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.410 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.410 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.410 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.410 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.410 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.410 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.411 INFO:teuthology.orchestra.run.smithi104.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-06-09T22:31:34.411 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-06-09T22:31:34.411 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.411 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.411 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.411 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.411 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.412 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.412 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.412 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.412 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.412 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.412 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.412 INFO:teuthology.orchestra.run.smithi104.stdout: "AUDIT_API_ENABLED": { 2024-06-09T22:31:34.412 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "AUDIT_API_ENABLED", 2024-06-09T22:31:34.413 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.413 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.413 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.413 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.413 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.413 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.413 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.413 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.413 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.414 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.414 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.414 INFO:teuthology.orchestra.run.smithi104.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-06-09T22:31:34.414 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-06-09T22:31:34.414 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.414 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.414 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.415 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.415 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.415 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.415 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.415 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.415 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.415 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.415 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.416 INFO:teuthology.orchestra.run.smithi104.stdout: "ENABLE_BROWSABLE_API": { 2024-06-09T22:31:34.416 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ENABLE_BROWSABLE_API", 2024-06-09T22:31:34.416 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.416 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.416 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.416 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.416 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.416 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.417 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.417 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.417 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.417 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.417 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.417 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-06-09T22:31:34.417 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-06-09T22:31:34.418 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.418 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.418 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.418 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.418 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.418 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.418 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.419 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.419 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.419 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.419 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.419 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-06-09T22:31:34.419 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-06-09T22:31:34.419 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.420 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.420 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.420 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.420 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.420 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.420 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.420 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.420 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.421 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.421 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.421 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-06-09T22:31:34.421 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-06-09T22:31:34.421 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.421 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.421 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.421 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.422 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.422 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.422 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.422 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.422 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.422 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.422 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.423 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-06-09T22:31:34.423 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-06-09T22:31:34.423 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.423 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.423 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.423 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.423 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.423 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.424 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.424 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.424 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.424 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.424 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.424 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_NFS": { 2024-06-09T22:31:34.424 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-06-09T22:31:34.424 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.425 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.425 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.425 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.425 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.425 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.425 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.425 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.425 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.425 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.426 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.426 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_RBD": { 2024-06-09T22:31:34.426 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-06-09T22:31:34.426 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.426 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.426 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.427 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.427 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.427 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.427 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.427 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.427 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.427 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.427 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.428 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_RGW": { 2024-06-09T22:31:34.428 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-06-09T22:31:34.428 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.428 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.428 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.428 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.428 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.428 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.428 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.429 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.429 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.429 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.429 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.429 INFO:teuthology.orchestra.run.smithi104.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-06-09T22:31:34.429 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-06-09T22:31:34.429 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.430 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.430 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.430 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.430 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.430 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.430 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.430 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.430 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.431 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.431 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.431 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_API_PASSWORD": { 2024-06-09T22:31:34.431 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_API_PASSWORD", 2024-06-09T22:31:34.431 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.431 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.431 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.432 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "admin", 2024-06-09T22:31:34.432 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.432 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.432 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.432 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.432 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.432 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.432 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.433 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-06-09T22:31:34.433 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-06-09T22:31:34.433 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.433 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.433 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.433 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.433 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.434 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.434 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.434 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.434 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.434 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.434 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.434 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_API_URL": { 2024-06-09T22:31:34.434 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_API_URL", 2024-06-09T22:31:34.435 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.435 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.435 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.435 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.435 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.435 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.435 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.435 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.435 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.436 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.436 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.436 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_API_USERNAME": { 2024-06-09T22:31:34.436 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_API_USERNAME", 2024-06-09T22:31:34.436 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.436 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.437 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.437 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "admin", 2024-06-09T22:31:34.437 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.437 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.437 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.437 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.437 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.437 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.437 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.438 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-06-09T22:31:34.438 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-06-09T22:31:34.438 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.438 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.438 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.438 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.438 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.439 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.439 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.439 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.439 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.439 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.439 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.439 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-06-09T22:31:34.439 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-06-09T22:31:34.440 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.440 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.440 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.440 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.440 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.440 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.440 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.440 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.441 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.441 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.441 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.441 INFO:teuthology.orchestra.run.smithi104.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-06-09T22:31:34.441 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-06-09T22:31:34.441 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.441 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.442 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.442 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.442 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.442 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.442 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.442 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.442 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.442 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.443 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.443 INFO:teuthology.orchestra.run.smithi104.stdout: "ISSUE_TRACKER_API_KEY": { 2024-06-09T22:31:34.443 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-06-09T22:31:34.443 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.443 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.443 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.444 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.444 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.444 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.444 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.444 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.444 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.444 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.444 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.445 INFO:teuthology.orchestra.run.smithi104.stdout: "PROMETHEUS_API_HOST": { 2024-06-09T22:31:34.445 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PROMETHEUS_API_HOST", 2024-06-09T22:31:34.445 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.445 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.445 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.445 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.445 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.445 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.446 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.446 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.446 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.446 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.446 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.446 INFO:teuthology.orchestra.run.smithi104.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-06-09T22:31:34.446 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-06-09T22:31:34.447 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.447 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.447 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.447 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.447 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.447 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.447 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.447 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.448 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.448 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.448 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.448 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-06-09T22:31:34.448 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-06-09T22:31:34.448 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.448 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.449 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.449 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.449 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.449 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.449 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.449 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.449 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.450 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.450 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.450 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-06-09T22:31:34.450 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-06-09T22:31:34.450 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.450 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.450 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.451 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.451 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.451 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.451 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.451 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.451 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.451 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.451 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.452 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-06-09T22:31:34.452 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-06-09T22:31:34.452 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.452 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.452 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.452 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.452 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.453 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.453 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.453 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.453 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.453 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.453 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.453 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-06-09T22:31:34.454 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-06-09T22:31:34.454 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.454 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.454 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.454 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.454 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.454 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.454 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.455 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.455 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.455 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.455 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.455 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-06-09T22:31:34.455 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-06-09T22:31:34.455 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.456 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.456 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.456 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.456 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.456 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.456 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.456 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.456 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.457 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.457 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.457 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-06-09T22:31:34.457 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-06-09T22:31:34.457 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.457 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.457 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.457 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.458 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.458 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.458 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.458 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.458 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.458 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.458 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.459 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-06-09T22:31:34.459 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-06-09T22:31:34.459 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.459 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.459 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.459 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.459 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.459 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.460 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.460 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.460 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.460 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.460 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.460 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_ENABLED": { 2024-06-09T22:31:34.460 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_ENABLED", 2024-06-09T22:31:34.461 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.461 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.461 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.461 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.461 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.461 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.461 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.461 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.461 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.462 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.462 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.462 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-06-09T22:31:34.462 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-06-09T22:31:34.462 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.462 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.462 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.463 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-06-09T22:31:34.463 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.463 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.463 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.463 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.463 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.463 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.463 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.464 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-06-09T22:31:34.464 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-06-09T22:31:34.464 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.464 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.464 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.464 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:34.464 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.465 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.465 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.465 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.465 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.465 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.465 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.465 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-06-09T22:31:34.466 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-06-09T22:31:34.466 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.466 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.466 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.466 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8", 2024-06-09T22:31:34.466 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.466 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.466 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.467 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.467 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.467 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.467 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.467 INFO:teuthology.orchestra.run.smithi104.stdout: "REST_REQUESTS_TIMEOUT": { 2024-06-09T22:31:34.467 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-06-09T22:31:34.467 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.468 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.468 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.468 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "45", 2024-06-09T22:31:34.468 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.468 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.468 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.468 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.468 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.469 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.469 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.469 INFO:teuthology.orchestra.run.smithi104.stdout: "RGW_API_ACCESS_KEY": { 2024-06-09T22:31:34.469 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RGW_API_ACCESS_KEY", 2024-06-09T22:31:34.469 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.469 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.469 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.469 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.470 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.470 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.470 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.470 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.470 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.470 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.470 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.471 INFO:teuthology.orchestra.run.smithi104.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-06-09T22:31:34.471 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-06-09T22:31:34.471 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.471 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.471 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.471 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "admin", 2024-06-09T22:31:34.471 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.471 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.472 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.472 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.472 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.472 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.472 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.472 INFO:teuthology.orchestra.run.smithi104.stdout: "RGW_API_SECRET_KEY": { 2024-06-09T22:31:34.473 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RGW_API_SECRET_KEY", 2024-06-09T22:31:34.473 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.473 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.473 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.473 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.473 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.473 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.473 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.473 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.474 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.474 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.474 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.474 INFO:teuthology.orchestra.run.smithi104.stdout: "RGW_API_SSL_VERIFY": { 2024-06-09T22:31:34.474 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RGW_API_SSL_VERIFY", 2024-06-09T22:31:34.474 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.474 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.475 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.475 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.475 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.475 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.475 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.475 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.475 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.475 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.476 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.476 INFO:teuthology.orchestra.run.smithi104.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-06-09T22:31:34.476 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-06-09T22:31:34.476 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.476 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.476 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.476 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-06-09T22:31:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-06-09T22:31:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-06-09T22:31:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-06-09T22:31:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5", 2024-06-09T22:31:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: "cross_origin_url": { 2024-06-09T22:31:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cross_origin_url", 2024-06-09T22:31:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: "crt_file": { 2024-06-09T22:31:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crt_file", 2024-06-09T22:31:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "debug": { 2024-06-09T22:31:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "debug", 2024-06-09T22:31:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Enable/disable debug options", 2024-06-09T22:31:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "jwt_token_ttl": { 2024-06-09T22:31:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "jwt_token_ttl", 2024-06-09T22:31:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "28800", 2024-06-09T22:31:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: "key_file": { 2024-06-09T22:31:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "key_file", 2024-06-09T22:31:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: "motd": { 2024-06-09T22:31:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "motd", 2024-06-09T22:31:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "The message of the day", 2024-06-09T22:31:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "redirect_resolve_ip_addr": { 2024-06-09T22:31:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "redirect_resolve_ip_addr", 2024-06-09T22:31:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: "server_addr": { 2024-06-09T22:31:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_addr", 2024-06-09T22:31:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "::", 2024-06-09T22:31:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: "server_port": { 2024-06-09T22:31:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_port", 2024-06-09T22:31:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8080", 2024-06-09T22:31:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: "ssl": { 2024-06-09T22:31:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ssl", 2024-06-09T22:31:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: "ssl_server_port": { 2024-06-09T22:31:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ssl_server_port", 2024-06-09T22:31:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8443", 2024-06-09T22:31:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "standby_behaviour": { 2024-06-09T22:31:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "standby_behaviour", 2024-06-09T22:31:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "redirect", 2024-06-09T22:31:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: "redirect" 2024-06-09T22:31:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "standby_error_status_code": { 2024-06-09T22:31:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "standby_error_status_code", 2024-06-09T22:31:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "500", 2024-06-09T22:31:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "400", 2024-06-09T22:31:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "599", 2024-06-09T22:31:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "url_prefix": { 2024-06-09T22:31:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "url_prefix", 2024-06-09T22:31:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "devicehealth", 2024-06-09T22:31:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "enable_monitoring": { 2024-06-09T22:31:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "enable_monitoring", 2024-06-09T22:31:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "monitor device health metrics", 2024-06-09T22:31:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: "mark_out_threshold": { 2024-06-09T22:31:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mark_out_threshold", 2024-06-09T22:31:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "2419200", 2024-06-09T22:31:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-06-09T22:31:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: "pool_name": { 2024-06-09T22:31:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "pool_name", 2024-06-09T22:31:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "device_health_metrics", 2024-06-09T22:31:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "name of pool in which to store device health metrics", 2024-06-09T22:31:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: "retention_period": { 2024-06-09T22:31:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "retention_period", 2024-06-09T22:31:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "15552000", 2024-06-09T22:31:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how long to retain device health metrics", 2024-06-09T22:31:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: "scrape_frequency": { 2024-06-09T22:31:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "scrape_frequency", 2024-06-09T22:31:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "86400", 2024-06-09T22:31:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to scrape device health metrics", 2024-06-09T22:31:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "self_heal": { 2024-06-09T22:31:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "self_heal", 2024-06-09T22:31:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-06-09T22:31:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to wake up and check device health", 2024-06-09T22:31:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_threshold": { 2024-06-09T22:31:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_threshold", 2024-06-09T22:31:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "7257600", 2024-06-09T22:31:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-06-09T22:31:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "diskprediction_local", 2024-06-09T22:31:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "predict_interval": { 2024-06-09T22:31:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "predict_interval", 2024-06-09T22:31:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "86400", 2024-06-09T22:31:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "predictor_model": { 2024-06-09T22:31:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "predictor_model", 2024-06-09T22:31:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "prophetstor", 2024-06-09T22:31:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "influx", 2024-06-09T22:31:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": false, 2024-06-09T22:31:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "influxdb python module not found", 2024-06-09T22:31:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: "batch_size": { 2024-06-09T22:31:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "batch_size", 2024-06-09T22:31:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5000", 2024-06-09T22:31:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-06-09T22:31:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "database": { 2024-06-09T22:31:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "database", 2024-06-09T22:31:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "ceph", 2024-06-09T22:31:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-06-09T22:31:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": { 2024-06-09T22:31:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "hostname", 2024-06-09T22:31:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "InfluxDB server hostname", 2024-06-09T22:31:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "30", 2024-06-09T22:31:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "5", 2024-06-09T22:31:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-06-09T22:31:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: "password": { 2024-06-09T22:31:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "password", 2024-06-09T22:31:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "password of InfluxDB server user", 2024-06-09T22:31:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: "port": { 2024-06-09T22:31:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "port", 2024-06-09T22:31:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8086", 2024-06-09T22:31:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "InfluxDB server port", 2024-06-09T22:31:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "ssl": { 2024-06-09T22:31:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ssl", 2024-06-09T22:31:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "false", 2024-06-09T22:31:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-06-09T22:31:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "threads": { 2024-06-09T22:31:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "threads", 2024-06-09T22:31:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5", 2024-06-09T22:31:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1", 2024-06-09T22:31:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "32", 2024-06-09T22:31:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-06-09T22:31:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: "username": { 2024-06-09T22:31:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "username", 2024-06-09T22:31:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "username of InfluxDB server user", 2024-06-09T22:31:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: "verify_ssl": { 2024-06-09T22:31:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "verify_ssl", 2024-06-09T22:31:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "true", 2024-06-09T22:31:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-06-09T22:31:34.575 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.575 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.575 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.575 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.575 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.575 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "insights", 2024-06-09T22:31:34.576 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.576 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.576 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.576 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.576 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.576 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.576 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.576 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.578 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.578 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.578 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.578 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.578 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.578 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.578 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.581 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.581 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.581 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.581 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.581 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.581 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.581 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.581 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.582 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.582 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.582 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.582 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.582 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.582 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.582 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "iostat", 2024-06-09T22:31:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.586 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.586 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.586 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.586 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.586 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.586 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.586 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.590 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.590 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.590 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.590 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.590 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.590 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.590 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.590 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "k8sevents", 2024-06-09T22:31:34.594 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.594 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.594 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.594 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_event_retention_days": { 2024-06-09T22:31:34.594 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ceph_event_retention_days", 2024-06-09T22:31:34.594 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.594 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.594 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "7", 2024-06-09T22:31:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Days to hold ceph event information within local cache", 2024-06-09T22:31:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.596 INFO:teuthology.orchestra.run.smithi104.stdout: "config_check_secs": { 2024-06-09T22:31:34.596 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "config_check_secs", 2024-06-09T22:31:34.596 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.596 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.596 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.596 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:34.596 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "10", 2024-06-09T22:31:34.596 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-06-09T22:31:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "localpool", 2024-06-09T22:31:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: "failure_domain": { 2024-06-09T22:31:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "failure_domain", 2024-06-09T22:31:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "host", 2024-06-09T22:31:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "failure domain for any created local pool", 2024-06-09T22:31:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-06-09T22:31:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: "min_size": { 2024-06-09T22:31:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "min_size", 2024-06-09T22:31:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "default min_size for any created local pool", 2024-06-09T22:31:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-06-09T22:31:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: "num_rep": { 2024-06-09T22:31:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "num_rep", 2024-06-09T22:31:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "3", 2024-06-09T22:31:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "default replica count for any created local pool", 2024-06-09T22:31:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: "pg_num": { 2024-06-09T22:31:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "pg_num", 2024-06-09T22:31:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "128", 2024-06-09T22:31:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "default pg_num for any created local pool", 2024-06-09T22:31:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: "prefix": { 2024-06-09T22:31:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "prefix", 2024-06-09T22:31:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.623 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.623 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.623 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "name prefix for any created local pool", 2024-06-09T22:31:34.623 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.623 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.623 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.623 INFO:teuthology.orchestra.run.smithi104.stdout: "subtree": { 2024-06-09T22:31:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "subtree", 2024-06-09T22:31:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "rack", 2024-06-09T22:31:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "CRUSH level for which to create a local pool", 2024-06-09T22:31:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-06-09T22:31:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mds_autoscaler", 2024-06-09T22:31:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.635 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mirroring", 2024-06-09T22:31:34.635 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.635 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.635 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.635 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.635 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.636 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.636 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.636 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.636 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.636 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.636 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.636 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.636 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.637 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.637 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.637 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.637 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.637 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.637 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.637 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.637 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.638 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.638 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.638 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.638 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.638 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.638 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.638 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.639 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.639 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.639 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.639 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.639 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.639 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.639 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.639 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.640 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.640 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.640 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.640 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.640 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.640 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.640 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.640 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.640 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.641 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.641 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.641 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.641 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.641 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.641 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.641 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.641 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "nfs", 2024-06-09T22:31:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "orchestrator", 2024-06-09T22:31:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: "fail_fs": { 2024-06-09T22:31:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "fail_fs", 2024-06-09T22:31:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-06-09T22:31:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.664 INFO:teuthology.orchestra.run.smithi104.stdout: "orchestrator": { 2024-06-09T22:31:34.665 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "orchestrator", 2024-06-09T22:31:34.665 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.665 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.665 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "cephadm", 2024-06-09T22:31:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "rook", 2024-06-09T22:31:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "test_orchestrator" 2024-06-09T22:31:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Orchestrator backend", 2024-06-09T22:31:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "osd_perf_query", 2024-06-09T22:31:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "osd_support", 2024-06-09T22:31:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.686 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.686 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.686 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.686 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.686 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.686 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.686 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.686 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.687 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "pg_autoscaler", 2024-06-09T22:31:34.687 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.687 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.687 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.687 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.687 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.688 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.688 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.688 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.688 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.688 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.688 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.688 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.688 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.688 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.689 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.689 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.689 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.689 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.689 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.689 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.689 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.689 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.690 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.690 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.690 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.690 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.690 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.690 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.690 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.690 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.691 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.691 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.691 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.691 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.691 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.691 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.691 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.691 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.692 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.692 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.692 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.692 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.692 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.692 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.692 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.692 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.692 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.693 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.693 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.693 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.693 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.693 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.693 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.693 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.694 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.694 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.694 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.694 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.694 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.694 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.694 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.695 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.695 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.695 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.695 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.695 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.695 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.695 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.695 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.696 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:34.696 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:34.696 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.696 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.696 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.696 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:34.696 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.696 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.697 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.697 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.697 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.697 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.697 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.697 INFO:teuthology.orchestra.run.smithi104.stdout: "threshold": { 2024-06-09T22:31:34.697 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "threshold", 2024-06-09T22:31:34.697 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:34.698 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.698 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.698 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "3.0", 2024-06-09T22:31:34.698 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1.0", 2024-06-09T22:31:34.698 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.698 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.698 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "scaling threshold", 2024-06-09T22:31:34.698 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-06-09T22:31:34.699 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.699 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.699 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.699 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.699 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "progress", 2024-06-09T22:31:34.699 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.700 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.700 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.700 INFO:teuthology.orchestra.run.smithi104.stdout: "allow_pg_recovery_event": { 2024-06-09T22:31:34.700 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "allow_pg_recovery_event", 2024-06-09T22:31:34.700 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.700 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.700 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.700 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.700 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.700 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.701 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.701 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "allow the module to show pg recovery progress", 2024-06-09T22:31:34.701 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.701 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.701 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.701 INFO:teuthology.orchestra.run.smithi104.stdout: "enabled": { 2024-06-09T22:31:34.701 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "enabled", 2024-06-09T22:31:34.701 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.702 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.702 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.702 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.702 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.702 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.702 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.702 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.702 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.703 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.703 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.703 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.703 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.703 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.703 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.703 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.704 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.704 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.704 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.704 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.704 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.704 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.704 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.704 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.705 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.705 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.705 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.705 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.705 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.705 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.705 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.706 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.706 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.706 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.706 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.706 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.706 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.706 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.706 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.707 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.707 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.707 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.707 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.707 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.707 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.707 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.707 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.708 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.708 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.708 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.708 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.708 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.708 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.708 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.708 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.709 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.709 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.709 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.709 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.709 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.709 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.709 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.709 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.710 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.710 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.710 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.710 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.710 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.710 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.710 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.710 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.711 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.711 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.711 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.711 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.711 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.711 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.711 INFO:teuthology.orchestra.run.smithi104.stdout: "max_completed_events": { 2024-06-09T22:31:34.711 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_completed_events", 2024-06-09T22:31:34.712 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.712 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.712 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.712 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "50", 2024-06-09T22:31:34.712 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.712 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.712 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.712 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "number of past completed events to remember", 2024-06-09T22:31:34.713 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.713 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.713 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.713 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:34.713 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:34.713 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.713 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.714 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.714 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5", 2024-06-09T22:31:34.714 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.714 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.714 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.714 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how long the module is going to sleep", 2024-06-09T22:31:34.714 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.714 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.715 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.715 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.715 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.715 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "prometheus", 2024-06-09T22:31:34.715 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.715 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.716 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.716 INFO:teuthology.orchestra.run.smithi104.stdout: "cache": { 2024-06-09T22:31:34.716 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cache", 2024-06-09T22:31:34.716 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.716 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.716 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.716 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.716 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.717 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.717 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.717 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.717 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.717 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.717 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.717 INFO:teuthology.orchestra.run.smithi104.stdout: "exclude_perf_counters": { 2024-06-09T22:31:34.718 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "exclude_perf_counters", 2024-06-09T22:31:34.718 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.718 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.718 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.718 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.718 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.718 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.718 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.719 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-06-09T22:31:34.719 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-06-09T22:31:34.719 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.719 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.719 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.719 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.719 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.719 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.720 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.720 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.720 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.720 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.720 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.720 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.720 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.720 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.721 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.721 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.721 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.721 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.721 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.721 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.721 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.721 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.722 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.722 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.722 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.722 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.722 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.722 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.722 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.722 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.722 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.723 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.723 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.723 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.723 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.723 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.723 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.723 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.723 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.724 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.724 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.724 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.724 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.724 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.724 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.724 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.724 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.725 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.725 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.725 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.725 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.725 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.725 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.725 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.725 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.726 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.726 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.726 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.726 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.726 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.726 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.726 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.726 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.727 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.727 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.727 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.727 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.727 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.727 INFO:teuthology.orchestra.run.smithi104.stdout: "rbd_stats_pools": { 2024-06-09T22:31:34.728 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rbd_stats_pools", 2024-06-09T22:31:34.728 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.728 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.728 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.728 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.728 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.728 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.728 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.729 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.729 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.729 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.729 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.729 INFO:teuthology.orchestra.run.smithi104.stdout: "rbd_stats_pools_refresh_interval": { 2024-06-09T22:31:34.729 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-06-09T22:31:34.729 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.729 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.730 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.730 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "300", 2024-06-09T22:31:34.730 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.730 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.730 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.730 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.730 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.731 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.731 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.731 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.731 INFO:teuthology.orchestra.run.smithi104.stdout: "scrape_interval": { 2024-06-09T22:31:34.731 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "scrape_interval", 2024-06-09T22:31:34.731 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:34.731 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.731 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.732 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "15.0", 2024-06-09T22:31:34.732 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.732 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.732 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.732 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.732 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.732 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.732 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.732 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.733 INFO:teuthology.orchestra.run.smithi104.stdout: "server_addr": { 2024-06-09T22:31:34.733 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_addr", 2024-06-09T22:31:34.733 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.733 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.733 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.733 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "::", 2024-06-09T22:31:34.733 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.733 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.734 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.734 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-06-09T22:31:34.734 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.734 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.734 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.734 INFO:teuthology.orchestra.run.smithi104.stdout: "server_port": { 2024-06-09T22:31:34.734 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_port", 2024-06-09T22:31:34.735 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.735 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.735 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.735 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "9283", 2024-06-09T22:31:34.735 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.735 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.735 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.736 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-06-09T22:31:34.736 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.736 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.736 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.736 INFO:teuthology.orchestra.run.smithi104.stdout: "stale_cache_strategy": { 2024-06-09T22:31:34.736 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "stale_cache_strategy", 2024-06-09T22:31:34.736 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.736 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.737 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.737 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "log", 2024-06-09T22:31:34.737 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.737 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.737 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.737 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.737 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.737 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.738 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.738 INFO:teuthology.orchestra.run.smithi104.stdout: "standby_behaviour": { 2024-06-09T22:31:34.738 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "standby_behaviour", 2024-06-09T22:31:34.738 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.738 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.738 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.738 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "default", 2024-06-09T22:31:34.739 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.739 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.739 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.739 INFO:teuthology.orchestra.run.smithi104.stdout: "default", 2024-06-09T22:31:34.739 INFO:teuthology.orchestra.run.smithi104.stdout: "error" 2024-06-09T22:31:34.739 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.739 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.739 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.740 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.740 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.740 INFO:teuthology.orchestra.run.smithi104.stdout: "standby_error_status_code": { 2024-06-09T22:31:34.740 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "standby_error_status_code", 2024-06-09T22:31:34.740 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.740 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.740 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.740 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "500", 2024-06-09T22:31:34.740 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "400", 2024-06-09T22:31:34.741 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "599", 2024-06-09T22:31:34.741 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.741 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.741 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.741 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.741 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.741 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.741 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.742 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rbd_support", 2024-06-09T22:31:34.742 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.742 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.742 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.742 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.742 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.742 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.743 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.743 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.743 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.743 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.743 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.743 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.743 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.743 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.743 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.744 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.744 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.744 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.744 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.744 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.744 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.744 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.744 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.745 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.745 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.745 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.745 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.745 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.745 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.745 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.746 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.746 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.746 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.746 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.746 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.746 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.746 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.747 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.747 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.747 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.747 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.747 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.747 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.747 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.747 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.747 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.748 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.748 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.748 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.748 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.748 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.748 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.748 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.748 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.748 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.749 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.749 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.749 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.749 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.749 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.749 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.749 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.749 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.749 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.749 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.750 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.750 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.750 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.750 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.750 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.750 INFO:teuthology.orchestra.run.smithi104.stdout: "max_concurrent_snap_create": { 2024-06-09T22:31:34.750 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_concurrent_snap_create", 2024-06-09T22:31:34.751 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.751 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.751 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.751 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:34.751 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.751 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.751 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.751 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.751 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.751 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.752 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.752 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.752 INFO:teuthology.orchestra.run.smithi104.stdout: "mirror_snapshot_schedule": { 2024-06-09T22:31:34.752 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mirror_snapshot_schedule", 2024-06-09T22:31:34.752 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.752 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.752 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.752 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.753 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.753 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.753 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.753 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.753 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.753 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.753 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.753 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.754 INFO:teuthology.orchestra.run.smithi104.stdout: "trash_purge_schedule": { 2024-06-09T22:31:34.754 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "trash_purge_schedule", 2024-06-09T22:31:34.754 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.754 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.754 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.754 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.754 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.754 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.755 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.755 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.755 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.755 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.755 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.755 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.755 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.756 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "restful", 2024-06-09T22:31:34.756 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.756 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.756 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.756 INFO:teuthology.orchestra.run.smithi104.stdout: "enable_auth": { 2024-06-09T22:31:34.756 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "enable_auth", 2024-06-09T22:31:34.756 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.756 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.757 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.757 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.757 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.757 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.757 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.757 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.757 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.757 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.757 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.758 INFO:teuthology.orchestra.run.smithi104.stdout: "key_file": { 2024-06-09T22:31:34.758 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "key_file", 2024-06-09T22:31:34.758 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.758 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.758 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.758 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.759 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.759 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.759 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.759 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.759 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.759 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.759 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.760 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.760 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.760 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.760 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.760 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.760 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.760 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.760 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.761 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.761 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.761 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.761 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.761 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.761 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.761 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.761 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.762 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.762 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.762 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.762 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.762 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.762 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.763 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.763 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.763 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.763 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.763 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.763 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.763 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.763 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.764 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.764 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.764 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.764 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.764 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.764 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.764 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.765 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.765 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.765 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.765 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.765 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.765 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.765 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.765 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.766 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.766 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.766 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.766 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.766 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.766 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.766 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.767 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.767 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.767 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.767 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.767 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.767 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.767 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.768 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.768 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.768 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.768 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.768 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.768 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.768 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.768 INFO:teuthology.orchestra.run.smithi104.stdout: "server_addr": { 2024-06-09T22:31:34.769 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_addr", 2024-06-09T22:31:34.769 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.769 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.769 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.769 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.769 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.769 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.769 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.769 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.770 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.770 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.770 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.770 INFO:teuthology.orchestra.run.smithi104.stdout: "server_port": { 2024-06-09T22:31:34.770 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_port", 2024-06-09T22:31:34.770 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.770 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.771 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.771 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.771 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.771 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.771 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.771 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.771 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.771 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.772 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.772 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.772 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.772 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rgw", 2024-06-09T22:31:34.772 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.772 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.773 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.773 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.773 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.773 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.773 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.773 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.773 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.773 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.773 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.774 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.774 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.774 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.774 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.774 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.774 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.774 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.774 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.775 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.775 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.775 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.775 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.775 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.775 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.775 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.775 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.776 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.776 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.776 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.776 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.776 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.776 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.776 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.776 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.777 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.777 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.777 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.777 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.777 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.777 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.777 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.778 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.778 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.778 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.778 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.778 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.778 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.778 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.778 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.779 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.779 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.779 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.779 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.779 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.779 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.779 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.779 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.779 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.780 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.780 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.780 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.780 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.780 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.780 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.780 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.780 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.780 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.781 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.781 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.781 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.781 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rook", 2024-06-09T22:31:34.781 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.781 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.781 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.782 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.782 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.782 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.782 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.782 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.782 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.782 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.782 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.783 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.783 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.783 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.783 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.783 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.783 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.783 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.783 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.783 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.784 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.784 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.784 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.784 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.784 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.784 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.784 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.784 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.785 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.785 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.785 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.785 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.785 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.785 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.785 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.785 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.786 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.786 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.786 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.786 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.786 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.786 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.786 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.786 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.787 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.787 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.787 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.787 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.787 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.787 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.787 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.788 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.788 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.788 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.788 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.788 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.788 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.788 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.788 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.789 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.789 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.789 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.789 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.789 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.789 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.789 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.789 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.790 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.790 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.790 INFO:teuthology.orchestra.run.smithi104.stdout: "storage_class": { 2024-06-09T22:31:34.790 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "storage_class", 2024-06-09T22:31:34.790 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.790 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.790 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.790 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "local", 2024-06-09T22:31:34.791 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.791 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.791 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.791 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-06-09T22:31:34.791 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.791 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.791 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.791 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.791 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.792 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "selftest", 2024-06-09T22:31:34.792 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.792 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.792 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.792 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.792 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.792 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.793 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.793 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.793 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.793 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.793 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.793 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.793 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.793 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.793 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.794 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.794 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.794 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.794 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.794 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.794 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.794 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.794 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.795 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.795 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.795 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.795 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.795 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.795 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.795 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.796 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.796 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.796 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.796 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.796 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.796 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.796 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.796 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.797 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.797 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.797 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.797 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.797 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.797 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.797 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.797 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.798 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.798 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.798 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.798 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.798 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.798 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.798 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.798 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.798 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.799 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.799 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.799 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.799 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.799 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.799 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.799 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.799 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.800 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.800 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.800 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.800 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.800 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.800 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.800 INFO:teuthology.orchestra.run.smithi104.stdout: "roption1": { 2024-06-09T22:31:34.800 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "roption1", 2024-06-09T22:31:34.801 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.801 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.801 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.801 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.801 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.801 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.801 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.801 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.801 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.801 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.802 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.802 INFO:teuthology.orchestra.run.smithi104.stdout: "roption2": { 2024-06-09T22:31:34.802 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "roption2", 2024-06-09T22:31:34.802 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.802 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.802 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.802 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "xyz", 2024-06-09T22:31:34.803 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.803 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.803 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.803 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.803 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.803 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.803 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.803 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption1": { 2024-06-09T22:31:34.803 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption1", 2024-06-09T22:31:34.804 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.804 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.804 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.804 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.804 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.804 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.804 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.804 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.804 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.805 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.805 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.805 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption2": { 2024-06-09T22:31:34.805 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption2", 2024-06-09T22:31:34.805 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.805 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.805 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.805 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.806 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.806 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.806 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.806 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.806 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.806 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.806 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.807 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption3": { 2024-06-09T22:31:34.807 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption3", 2024-06-09T22:31:34.807 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:34.807 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.807 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.807 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.807 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.807 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.808 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.808 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.808 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.808 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.808 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.808 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption4": { 2024-06-09T22:31:34.808 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption4", 2024-06-09T22:31:34.808 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.809 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.809 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.809 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.809 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.809 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.809 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.809 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.809 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.810 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.810 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.810 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption5": { 2024-06-09T22:31:34.810 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption5", 2024-06-09T22:31:34.810 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.810 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.810 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.811 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.811 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.811 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.811 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.811 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.811 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.811 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.811 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.811 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption6": { 2024-06-09T22:31:34.812 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption6", 2024-06-09T22:31:34.812 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.812 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.812 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.812 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.812 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.812 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.812 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.812 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.813 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.813 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.813 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.813 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption7": { 2024-06-09T22:31:34.813 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption7", 2024-06-09T22:31:34.813 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.813 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.814 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.814 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.814 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1", 2024-06-09T22:31:34.814 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "42", 2024-06-09T22:31:34.814 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.814 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.814 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.814 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.815 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.815 INFO:teuthology.orchestra.run.smithi104.stdout: "testkey": { 2024-06-09T22:31:34.815 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "testkey", 2024-06-09T22:31:34.815 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.815 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.815 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.815 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.816 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.816 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.816 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.816 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.816 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.816 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.816 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.817 INFO:teuthology.orchestra.run.smithi104.stdout: "testlkey": { 2024-06-09T22:31:34.817 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "testlkey", 2024-06-09T22:31:34.817 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.817 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.817 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.817 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.817 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.817 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.817 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.818 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.818 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.818 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.818 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.818 INFO:teuthology.orchestra.run.smithi104.stdout: "testnewline": { 2024-06-09T22:31:34.818 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "testnewline", 2024-06-09T22:31:34.819 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.819 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.819 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.819 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.819 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.819 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.819 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.819 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.819 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.820 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.820 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.820 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.820 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.820 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "snap_schedule", 2024-06-09T22:31:34.820 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.821 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.821 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.821 INFO:teuthology.orchestra.run.smithi104.stdout: "allow_m_granularity": { 2024-06-09T22:31:34.821 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "allow_m_granularity", 2024-06-09T22:31:34.821 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.821 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.821 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.821 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.822 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.822 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.822 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.822 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "allow minute scheduled snapshots", 2024-06-09T22:31:34.822 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.822 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.822 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.823 INFO:teuthology.orchestra.run.smithi104.stdout: "dump_on_update": { 2024-06-09T22:31:34.823 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "dump_on_update", 2024-06-09T22:31:34.823 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.823 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.823 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.823 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.823 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.823 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.824 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.824 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "dump database to debug log on update", 2024-06-09T22:31:34.824 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.824 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.824 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.824 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.824 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.825 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.825 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.825 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.825 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.825 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.825 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.825 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.825 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.826 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.826 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.826 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.826 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.826 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.826 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.826 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.827 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.827 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.827 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.827 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.827 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.827 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.827 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.828 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.828 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.828 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.828 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.828 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.828 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.828 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.828 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.829 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.829 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.829 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.829 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.829 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.829 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.829 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.830 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.830 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.830 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.830 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.830 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.830 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.830 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.830 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.831 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.831 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.831 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.831 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.831 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.831 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.832 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.832 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.832 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.832 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.832 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.832 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.832 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.832 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.833 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.833 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.833 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.833 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.833 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.833 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.833 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.834 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "stats", 2024-06-09T22:31:34.834 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.834 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.834 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.834 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.834 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.835 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.835 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.835 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.835 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.835 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.835 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.835 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.835 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.835 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.836 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.836 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.836 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.836 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.836 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.836 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.836 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.836 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.837 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.837 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.837 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.837 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.837 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.837 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.838 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.838 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.838 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.838 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.838 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.838 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.838 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.838 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.839 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.839 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.839 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.839 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.839 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.839 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.839 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.840 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.840 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.840 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.840 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.840 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.840 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.840 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.840 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.841 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.841 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.841 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.841 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.841 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.841 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.841 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.842 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.842 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.842 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.842 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.842 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.842 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.842 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.842 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.843 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.843 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.843 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.843 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.843 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.843 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "status", 2024-06-09T22:31:34.844 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.844 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.844 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.844 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.844 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.844 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.844 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.844 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.845 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.845 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.845 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.845 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.845 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.845 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.845 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.845 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.845 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.846 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.846 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.846 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.846 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.846 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.846 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.846 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.847 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.847 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.847 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.847 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.847 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.847 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.847 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.847 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.848 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.848 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.848 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.848 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.848 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.848 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.848 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.849 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.849 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.849 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.849 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.849 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.849 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.849 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.849 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.850 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.850 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.850 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.850 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.850 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.850 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.850 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.850 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.851 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.851 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.851 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.851 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.851 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.851 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.851 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.852 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.852 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.852 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.852 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.852 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.852 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.852 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.852 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.853 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.853 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "telegraf", 2024-06-09T22:31:34.853 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.853 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.853 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.853 INFO:teuthology.orchestra.run.smithi104.stdout: "address": { 2024-06-09T22:31:34.853 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "address", 2024-06-09T22:31:34.854 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.854 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.854 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.854 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-06-09T22:31:34.854 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.854 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.854 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.854 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.855 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.855 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.855 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.855 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:34.855 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:34.855 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.855 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.856 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.856 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "15", 2024-06-09T22:31:34.856 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.856 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.856 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.856 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.856 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.856 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.857 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.857 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.857 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.857 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.857 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.857 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.857 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.857 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.858 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.858 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.858 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.858 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.858 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.858 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.858 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.858 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.859 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.859 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.859 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.859 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.859 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.859 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.859 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.860 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.860 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.860 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.860 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.860 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.860 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.860 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.860 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.861 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.861 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.861 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.861 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.861 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.861 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.861 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.862 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.862 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.862 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.862 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.862 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.862 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.862 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.862 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.863 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.863 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.863 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.863 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.863 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.863 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.863 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.863 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.864 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.864 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.864 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.864 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.864 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.864 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.864 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.864 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.865 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.865 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.865 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.865 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.865 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.865 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.865 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.866 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "telemetry", 2024-06-09T22:31:34.866 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.866 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.866 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.866 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_basic": { 2024-06-09T22:31:34.866 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_basic", 2024-06-09T22:31:34.867 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.867 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.867 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.867 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.867 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.867 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.867 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.867 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share basic cluster information (size, version)", 2024-06-09T22:31:34.867 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.868 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.868 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.868 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_crash": { 2024-06-09T22:31:34.868 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_crash", 2024-06-09T22:31:34.868 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.868 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.868 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.869 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.869 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.869 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.869 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.869 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-06-09T22:31:34.869 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.869 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.869 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.870 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_device": { 2024-06-09T22:31:34.870 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_device", 2024-06-09T22:31:34.870 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.870 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.870 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.870 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.870 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.871 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.871 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.871 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-06-09T22:31:34.871 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.871 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.871 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.871 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_ident": { 2024-06-09T22:31:34.871 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_ident", 2024-06-09T22:31:34.872 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.872 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.872 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.872 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.872 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.872 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.872 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.872 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-06-09T22:31:34.872 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.873 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.873 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.873 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_perf": { 2024-06-09T22:31:34.873 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_perf", 2024-06-09T22:31:34.873 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.873 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.873 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.874 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.874 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.874 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.874 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.874 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share various performance metrics of a cluster", 2024-06-09T22:31:34.874 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.874 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.874 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.875 INFO:teuthology.orchestra.run.smithi104.stdout: "contact": { 2024-06-09T22:31:34.875 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "contact", 2024-06-09T22:31:34.875 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.875 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.875 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.875 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.876 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.876 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.876 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.876 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.876 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.876 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.876 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.877 INFO:teuthology.orchestra.run.smithi104.stdout: "description": { 2024-06-09T22:31:34.877 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "description", 2024-06-09T22:31:34.877 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.877 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.877 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.877 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.877 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.877 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.877 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.878 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.878 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.878 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.878 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.878 INFO:teuthology.orchestra.run.smithi104.stdout: "device_url": { 2024-06-09T22:31:34.879 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "device_url", 2024-06-09T22:31:34.879 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.879 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.879 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.879 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-06-09T22:31:34.879 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.879 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.879 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.880 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.880 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.880 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.880 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.880 INFO:teuthology.orchestra.run.smithi104.stdout: "enabled": { 2024-06-09T22:31:34.880 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "enabled", 2024-06-09T22:31:34.880 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.881 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.881 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.881 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.881 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.881 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.881 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.881 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.881 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.882 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.882 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.882 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:34.882 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:34.882 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.882 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.882 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.883 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "24", 2024-06-09T22:31:34.883 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "8", 2024-06-09T22:31:34.883 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.883 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.883 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.883 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.883 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.883 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.884 INFO:teuthology.orchestra.run.smithi104.stdout: "last_opt_revision": { 2024-06-09T22:31:34.884 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "last_opt_revision", 2024-06-09T22:31:34.884 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.884 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.884 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.884 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "1", 2024-06-09T22:31:34.884 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.884 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: "leaderboard": { 2024-06-09T22:31:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "leaderboard", 2024-06-09T22:31:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.885 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.886 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: "leaderboard_description": { 2024-06-09T22:31:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "leaderboard_description", 2024-06-09T22:31:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.887 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.888 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.889 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.890 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.890 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.890 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.890 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.890 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.890 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.890 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.890 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.891 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.891 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.891 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.891 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.891 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.891 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.891 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.892 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.892 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.892 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.892 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.892 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.892 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.892 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.892 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.893 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.893 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.893 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.893 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.893 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.893 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.893 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.894 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.894 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.894 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.894 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.894 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.894 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.894 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.894 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.895 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.895 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.895 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.895 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.895 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.895 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.895 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.896 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.896 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.896 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.896 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.896 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.896 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.896 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.896 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.896 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.897 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.897 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.897 INFO:teuthology.orchestra.run.smithi104.stdout: "organization": { 2024-06-09T22:31:34.897 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "organization", 2024-06-09T22:31:34.897 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.897 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.897 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.898 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.898 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.898 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.898 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.898 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.898 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.898 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.898 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.899 INFO:teuthology.orchestra.run.smithi104.stdout: "proxy": { 2024-06-09T22:31:34.899 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "proxy", 2024-06-09T22:31:34.899 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.899 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.899 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.899 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.899 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.899 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.900 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.900 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.900 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.900 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.900 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.900 INFO:teuthology.orchestra.run.smithi104.stdout: "url": { 2024-06-09T22:31:34.900 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "url", 2024-06-09T22:31:34.900 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.901 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.901 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.901 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-06-09T22:31:34.901 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.901 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.901 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.901 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.901 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.901 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.902 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.902 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.902 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.902 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "test_orchestrator", 2024-06-09T22:31:34.902 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.902 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.902 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.903 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.903 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.903 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.903 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.903 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.903 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.903 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.903 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.904 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.904 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.904 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.904 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.904 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.904 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.904 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.904 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.904 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.905 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.905 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.905 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.905 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.905 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.905 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.906 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.906 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.906 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.906 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.906 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.906 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.906 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.906 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.906 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.907 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.907 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.907 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.907 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.907 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.907 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.907 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.907 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.908 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.908 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.908 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.908 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.908 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.908 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.908 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.908 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.909 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.909 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.909 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.909 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.909 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.909 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.909 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.909 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.910 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.910 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.910 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.910 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.910 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.910 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.910 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.911 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.911 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.911 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.911 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.911 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.911 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "volumes", 2024-06-09T22:31:34.911 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.911 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.912 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.912 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.912 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.912 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.912 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.912 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.912 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.913 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.913 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.913 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.913 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.913 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.913 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.913 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.913 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.914 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.914 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.914 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.914 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.914 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.914 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.914 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.915 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.915 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.915 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.915 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.915 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.915 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.915 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.915 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.916 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.916 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.916 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.916 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.916 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.916 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.916 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.917 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.917 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.917 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.917 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.917 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.917 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.917 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.917 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.918 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.918 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.918 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.918 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.918 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.918 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.918 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.918 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.919 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.919 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.919 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.919 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.919 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.919 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.919 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.919 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.920 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.920 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.920 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.920 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.920 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.920 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.920 INFO:teuthology.orchestra.run.smithi104.stdout: "max_concurrent_clones": { 2024-06-09T22:31:34.920 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_concurrent_clones", 2024-06-09T22:31:34.921 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.921 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.921 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.921 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "4", 2024-06-09T22:31:34.921 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.921 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.921 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.921 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Number of asynchronous cloner threads", 2024-06-09T22:31:34.921 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.922 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.922 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.922 INFO:teuthology.orchestra.run.smithi104.stdout: "periodic_async_work": { 2024-06-09T22:31:34.922 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "periodic_async_work", 2024-06-09T22:31:34.922 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.922 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.922 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.923 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.923 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.923 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.923 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.923 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Periodically check for async work", 2024-06-09T22:31:34.923 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.923 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.923 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.924 INFO:teuthology.orchestra.run.smithi104.stdout: "snapshot_clone_delay": { 2024-06-09T22:31:34.924 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "snapshot_clone_delay", 2024-06-09T22:31:34.924 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.924 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.924 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.924 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:34.924 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.924 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.924 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.925 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-06-09T22:31:34.925 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.925 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.925 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.925 INFO:teuthology.orchestra.run.smithi104.stdout: "snapshot_clone_no_wait": { 2024-06-09T22:31:34.925 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "snapshot_clone_no_wait", 2024-06-09T22:31:34.926 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.926 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.926 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.926 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.926 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.926 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.926 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.927 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-06-09T22:31:34.927 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.927 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.927 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.927 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.927 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.927 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "zabbix", 2024-06-09T22:31:34.928 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.928 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.928 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.928 INFO:teuthology.orchestra.run.smithi104.stdout: "discovery_interval": { 2024-06-09T22:31:34.928 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "discovery_interval", 2024-06-09T22:31:34.928 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:34.928 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.928 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.928 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "100", 2024-06-09T22:31:34.929 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.929 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.929 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.929 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.929 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.929 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.929 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.929 INFO:teuthology.orchestra.run.smithi104.stdout: "identifier": { 2024-06-09T22:31:34.930 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "identifier", 2024-06-09T22:31:34.930 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.930 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.930 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.930 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.930 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.930 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.930 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.931 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.931 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.931 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.931 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.931 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:34.931 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:34.931 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.931 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.932 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.932 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:34.932 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.932 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.932 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.932 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.932 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.932 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.933 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.933 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.933 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.933 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.933 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.933 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.933 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.934 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.934 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.934 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.934 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.934 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.934 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.934 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.934 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.934 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.935 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.935 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.935 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.935 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.935 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.935 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.935 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.935 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.936 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.936 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.936 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.936 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.936 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.936 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.936 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.936 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.937 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.937 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.937 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.937 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.937 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.937 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.937 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.938 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.938 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.938 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.938 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.938 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.938 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.938 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.938 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.939 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.939 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.939 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.939 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.939 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.939 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.939 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.939 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.940 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.940 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.940 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.940 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.940 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.940 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.940 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.940 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.940 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.941 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.941 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.941 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.941 INFO:teuthology.orchestra.run.smithi104.stdout: "zabbix_host": { 2024-06-09T22:31:34.941 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "zabbix_host", 2024-06-09T22:31:34.941 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.941 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.942 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.942 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.942 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.942 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.942 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.942 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.942 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.942 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.943 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.943 INFO:teuthology.orchestra.run.smithi104.stdout: "zabbix_port": { 2024-06-09T22:31:34.943 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "zabbix_port", 2024-06-09T22:31:34.943 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.943 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.943 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.943 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10051", 2024-06-09T22:31:34.943 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.944 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.944 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.944 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.944 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.944 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.944 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.944 INFO:teuthology.orchestra.run.smithi104.stdout: "zabbix_sender": { 2024-06-09T22:31:34.944 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "zabbix_sender", 2024-06-09T22:31:34.945 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.945 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.945 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:34.945 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-06-09T22:31:34.945 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.945 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.945 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.945 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.945 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.946 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.946 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.946 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.946 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.946 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.946 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-06-09T22:31:34.946 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.946 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.947 INFO:teuthology.orchestra.run.smithi104.stdout: "modules": [ 2024-06-09T22:31:34.947 INFO:teuthology.orchestra.run.smithi104.stdout: "cephadm", 2024-06-09T22:31:34.947 INFO:teuthology.orchestra.run.smithi104.stdout: "dashboard", 2024-06-09T22:31:34.947 INFO:teuthology.orchestra.run.smithi104.stdout: "iostat", 2024-06-09T22:31:34.947 INFO:teuthology.orchestra.run.smithi104.stdout: "nfs", 2024-06-09T22:31:34.947 INFO:teuthology.orchestra.run.smithi104.stdout: "restful", 2024-06-09T22:31:34.947 INFO:teuthology.orchestra.run.smithi104.stdout: "snap_schedule" 2024-06-09T22:31:34.947 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.948 INFO:teuthology.orchestra.run.smithi104.stdout: "available_modules": [ 2024-06-09T22:31:34.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.948 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "alerts", 2024-06-09T22:31:34.948 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.948 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.948 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.948 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:34.948 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:34.949 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:34.949 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.949 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.949 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:34.949 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.949 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.949 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.949 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "How frequently to reexamine health status", 2024-06-09T22:31:34.949 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.950 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.950 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.950 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.950 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.950 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.950 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.950 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.951 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.951 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.951 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.951 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.951 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.951 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.951 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.951 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.951 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.952 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.952 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.952 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.952 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.952 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.952 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.952 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.952 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.952 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.953 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.953 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.953 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.953 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.953 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.953 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.953 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.953 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.954 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.954 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.954 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.954 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.954 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.954 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.954 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.955 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.955 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.955 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.955 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.955 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.955 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.955 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.955 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.955 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.955 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.956 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.956 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.956 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.956 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.956 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.956 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.956 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.957 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.957 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.957 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.957 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.957 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.957 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.957 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.957 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.958 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.958 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.958 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.958 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_destination": { 2024-06-09T22:31:34.958 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_destination", 2024-06-09T22:31:34.958 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.958 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.959 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.959 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.959 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.959 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.959 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.959 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Email address to send alerts to", 2024-06-09T22:31:34.959 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.959 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.959 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.960 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_from_name": { 2024-06-09T22:31:34.960 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_from_name", 2024-06-09T22:31:34.960 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.960 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.960 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.960 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "Ceph", 2024-06-09T22:31:34.960 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.960 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.961 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.961 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Email From: name", 2024-06-09T22:31:34.961 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.961 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.961 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.961 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_host": { 2024-06-09T22:31:34.961 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_host", 2024-06-09T22:31:34.962 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.962 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.962 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.962 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.962 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.962 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.962 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.963 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "SMTP server", 2024-06-09T22:31:34.963 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.963 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.963 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.963 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_password": { 2024-06-09T22:31:34.963 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_password", 2024-06-09T22:31:34.963 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.963 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.964 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.964 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.964 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.964 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.964 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.964 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Password to authenticate with", 2024-06-09T22:31:34.964 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.964 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.965 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.965 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_port": { 2024-06-09T22:31:34.965 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_port", 2024-06-09T22:31:34.965 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:34.965 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.965 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.965 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "465", 2024-06-09T22:31:34.965 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.966 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.966 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.966 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "SMTP port", 2024-06-09T22:31:34.966 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.966 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.966 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.967 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_sender": { 2024-06-09T22:31:34.967 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_sender", 2024-06-09T22:31:34.967 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.967 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.967 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.967 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.967 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.967 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.967 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.968 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "SMTP envelope sender", 2024-06-09T22:31:34.968 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.968 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.968 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.968 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_ssl": { 2024-06-09T22:31:34.968 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_ssl", 2024-06-09T22:31:34.968 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.969 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.969 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.969 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.969 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.969 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.969 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.969 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Use SSL to connect to SMTP server", 2024-06-09T22:31:34.969 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.969 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.970 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.970 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.970 INFO:teuthology.orchestra.run.smithi104.stdout: "smtp_user": { 2024-06-09T22:31:34.970 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "smtp_user", 2024-06-09T22:31:34.970 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.970 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.970 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.970 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.971 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.971 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.971 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.971 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "User to authenticate as", 2024-06-09T22:31:34.971 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.971 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.971 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.971 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.972 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:34.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:34.972 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "balancer", 2024-06-09T22:31:34.972 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:34.972 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:34.972 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:34.972 INFO:teuthology.orchestra.run.smithi104.stdout: "active": { 2024-06-09T22:31:34.973 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "active", 2024-06-09T22:31:34.973 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.973 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.973 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.973 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:34.973 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.973 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.973 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.974 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "automatically balance PGs across cluster", 2024-06-09T22:31:34.974 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.974 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.974 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.974 INFO:teuthology.orchestra.run.smithi104.stdout: "begin_time": { 2024-06-09T22:31:34.974 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "begin_time", 2024-06-09T22:31:34.974 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.975 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.975 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.975 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0000", 2024-06-09T22:31:34.975 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.975 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.975 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.975 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "beginning time of day to automatically balance", 2024-06-09T22:31:34.975 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-09T22:31:34.976 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.976 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.976 INFO:teuthology.orchestra.run.smithi104.stdout: "begin_weekday": { 2024-06-09T22:31:34.976 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "begin_weekday", 2024-06-09T22:31:34.976 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:34.976 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.976 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.977 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:34.977 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "0", 2024-06-09T22:31:34.977 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "6", 2024-06-09T22:31:34.977 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.977 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-06-09T22:31:34.977 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-09T22:31:34.977 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.977 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.978 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_compat_max_iterations": { 2024-06-09T22:31:34.978 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crush_compat_max_iterations", 2024-06-09T22:31:34.978 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:34.978 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.978 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.978 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "25", 2024-06-09T22:31:34.978 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1", 2024-06-09T22:31:34.978 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "250", 2024-06-09T22:31:34.979 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.979 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-06-09T22:31:34.979 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.979 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.979 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.979 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_compat_metrics": { 2024-06-09T22:31:34.979 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crush_compat_metrics", 2024-06-09T22:31:34.980 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.980 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.980 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.980 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "pgs,objects,bytes", 2024-06-09T22:31:34.980 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.980 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.980 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.980 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-06-09T22:31:34.981 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-06-09T22:31:34.981 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.981 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.981 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_compat_step": { 2024-06-09T22:31:34.981 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crush_compat_step", 2024-06-09T22:31:34.981 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:34.981 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.982 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.982 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0.5", 2024-06-09T22:31:34.982 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "0.001", 2024-06-09T22:31:34.982 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "0.999", 2024-06-09T22:31:34.982 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.982 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "aggressiveness of optimization", 2024-06-09T22:31:34.982 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-06-09T22:31:34.982 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.983 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.983 INFO:teuthology.orchestra.run.smithi104.stdout: "end_time": { 2024-06-09T22:31:34.983 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "end_time", 2024-06-09T22:31:34.983 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.983 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.983 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.983 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "2359", 2024-06-09T22:31:34.984 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.984 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.984 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.984 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "ending time of day to automatically balance", 2024-06-09T22:31:34.984 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-09T22:31:34.984 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.984 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.984 INFO:teuthology.orchestra.run.smithi104.stdout: "end_weekday": { 2024-06-09T22:31:34.985 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "end_weekday", 2024-06-09T22:31:34.985 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:34.985 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.985 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.985 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:34.985 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "0", 2024-06-09T22:31:34.985 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "6", 2024-06-09T22:31:34.985 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.985 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-06-09T22:31:34.986 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-09T22:31:34.986 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.986 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.986 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.986 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:34.986 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:34.986 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.986 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.987 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.987 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.987 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.987 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.987 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.987 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.987 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.987 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.988 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.988 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.988 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.988 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.988 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.988 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.988 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.989 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.989 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.989 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:34.989 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:34.989 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.989 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.989 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.989 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.989 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.990 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.990 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.990 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.990 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.990 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.990 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.990 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.990 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:34.991 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:34.991 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.991 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.991 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.991 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:34.991 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.991 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.991 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.992 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:34.992 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:34.992 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:34.992 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:34.992 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:34.992 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:34.992 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.992 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.992 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.993 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.993 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.993 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:34.993 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:34.993 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:34.993 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.993 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.994 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:34.994 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.994 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.994 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.994 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:34.994 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.994 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.994 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.994 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.995 INFO:teuthology.orchestra.run.smithi104.stdout: "min_score": { 2024-06-09T22:31:34.995 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "min_score", 2024-06-09T22:31:34.995 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:34.995 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.995 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.995 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:34.995 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.995 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.996 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:34.996 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-06-09T22:31:34.996 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.996 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.996 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.996 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": { 2024-06-09T22:31:34.996 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mode", 2024-06-09T22:31:34.996 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.997 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.997 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.997 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "upmap", 2024-06-09T22:31:34.997 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:34.997 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:34.997 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:34.997 INFO:teuthology.orchestra.run.smithi104.stdout: "crush-compat", 2024-06-09T22:31:34.997 INFO:teuthology.orchestra.run.smithi104.stdout: "none", 2024-06-09T22:31:34.998 INFO:teuthology.orchestra.run.smithi104.stdout: "read", 2024-06-09T22:31:34.998 INFO:teuthology.orchestra.run.smithi104.stdout: "upmap", 2024-06-09T22:31:34.998 INFO:teuthology.orchestra.run.smithi104.stdout: "upmap-read" 2024-06-09T22:31:34.998 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:34.998 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Balancer mode", 2024-06-09T22:31:34.998 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:34.998 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:34.998 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:34.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:34.999 INFO:teuthology.orchestra.run.smithi104.stdout: "pool_ids": { 2024-06-09T22:31:34.999 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "pool_ids", 2024-06-09T22:31:34.999 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:34.999 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:34.999 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:34.999 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:34.999 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.000 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.000 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.000 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-06-09T22:31:35.000 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.000 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.000 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.000 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.000 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:35.000 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:35.001 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.001 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.001 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.001 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:35.001 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.001 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.001 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.001 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-06-09T22:31:35.001 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.002 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.002 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.002 INFO:teuthology.orchestra.run.smithi104.stdout: "upmap_max_deviation": { 2024-06-09T22:31:35.002 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "upmap_max_deviation", 2024-06-09T22:31:35.002 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.002 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.002 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.003 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5", 2024-06-09T22:31:35.003 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1", 2024-06-09T22:31:35.003 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.003 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.003 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "deviation below which no optimization is attempted", 2024-06-09T22:31:35.003 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-06-09T22:31:35.003 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.003 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.004 INFO:teuthology.orchestra.run.smithi104.stdout: "upmap_max_optimizations": { 2024-06-09T22:31:35.004 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "upmap_max_optimizations", 2024-06-09T22:31:35.004 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:35.004 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.004 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.004 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:35.004 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.004 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.005 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.005 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-06-09T22:31:35.005 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.005 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.005 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.005 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.005 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.006 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cephadm", 2024-06-09T22:31:35.006 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.006 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.006 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.006 INFO:teuthology.orchestra.run.smithi104.stdout: "agent_down_multiplier": { 2024-06-09T22:31:35.006 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "agent_down_multiplier", 2024-06-09T22:31:35.006 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:35.006 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.007 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.007 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "3.0", 2024-06-09T22:31:35.007 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.007 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.007 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.007 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-06-09T22:31:35.007 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.008 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.008 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.008 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.008 INFO:teuthology.orchestra.run.smithi104.stdout: "agent_refresh_rate": { 2024-06-09T22:31:35.008 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "agent_refresh_rate", 2024-06-09T22:31:35.008 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.008 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.008 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.008 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "20", 2024-06-09T22:31:35.009 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.009 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.009 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.009 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-06-09T22:31:35.009 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.009 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.009 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.009 INFO:teuthology.orchestra.run.smithi104.stdout: "agent_starting_port": { 2024-06-09T22:31:35.010 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "agent_starting_port", 2024-06-09T22:31:35.010 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.010 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.010 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.010 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "4721", 2024-06-09T22:31:35.010 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.010 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.011 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.011 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-06-09T22:31:35.011 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.011 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.011 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.011 INFO:teuthology.orchestra.run.smithi104.stdout: "allow_ptrace": { 2024-06-09T22:31:35.011 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "allow_ptrace", 2024-06-09T22:31:35.012 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.012 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.012 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.012 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.012 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.012 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.012 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.012 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-06-09T22:31:35.012 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-06-09T22:31:35.013 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.013 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.013 INFO:teuthology.orchestra.run.smithi104.stdout: "autotune_interval": { 2024-06-09T22:31:35.013 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "autotune_interval", 2024-06-09T22:31:35.013 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.013 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.013 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.014 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:35.014 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.014 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.014 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.014 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to autotune daemon memory", 2024-06-09T22:31:35.014 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.014 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.015 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.015 INFO:teuthology.orchestra.run.smithi104.stdout: "autotune_memory_target_ratio": { 2024-06-09T22:31:35.015 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "autotune_memory_target_ratio", 2024-06-09T22:31:35.015 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:35.015 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.015 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.015 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0.7", 2024-06-09T22:31:35.016 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.016 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.016 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.016 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-06-09T22:31:35.016 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.016 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.016 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.016 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.017 INFO:teuthology.orchestra.run.smithi104.stdout: "cephadm_log_destination": { 2024-06-09T22:31:35.017 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cephadm_log_destination", 2024-06-09T22:31:35.017 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.017 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.017 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.017 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.017 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.017 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.017 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.018 INFO:teuthology.orchestra.run.smithi104.stdout: "file", 2024-06-09T22:31:35.018 INFO:teuthology.orchestra.run.smithi104.stdout: "file,syslog", 2024-06-09T22:31:35.018 INFO:teuthology.orchestra.run.smithi104.stdout: "syslog" 2024-06-09T22:31:35.018 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.018 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-06-09T22:31:35.018 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.018 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.019 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.019 INFO:teuthology.orchestra.run.smithi104.stdout: "cgroups_split": { 2024-06-09T22:31:35.019 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cgroups_split", 2024-06-09T22:31:35.019 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.019 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.019 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.019 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.019 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.020 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.020 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.020 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-06-09T22:31:35.020 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.020 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.020 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.020 INFO:teuthology.orchestra.run.smithi104.stdout: "config_checks_enabled": { 2024-06-09T22:31:35.020 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "config_checks_enabled", 2024-06-09T22:31:35.021 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.021 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.021 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.021 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.021 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.021 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.021 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.021 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-06-09T22:31:35.021 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.022 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.022 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.022 INFO:teuthology.orchestra.run.smithi104.stdout: "config_dashboard": { 2024-06-09T22:31:35.022 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "config_dashboard", 2024-06-09T22:31:35.022 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.022 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.023 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.023 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.023 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.023 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.023 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.023 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-06-09T22:31:35.023 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.023 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.024 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.024 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_alertmanager": { 2024-06-09T22:31:35.024 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_alertmanager", 2024-06-09T22:31:35.024 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.024 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.024 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.024 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-06-09T22:31:35.024 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.025 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.025 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.025 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Prometheus container image", 2024-06-09T22:31:35.025 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.025 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.025 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.025 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_base": { 2024-06-09T22:31:35.025 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_base", 2024-06-09T22:31:35.026 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.026 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.026 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.026 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/ceph", 2024-06-09T22:31:35.026 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.026 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.026 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.027 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Container image name, without the tag", 2024-06-09T22:31:35.027 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.027 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.027 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.027 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.027 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_elasticsearch": { 2024-06-09T22:31:35.027 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_elasticsearch", 2024-06-09T22:31:35.027 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.027 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.028 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.028 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-06-09T22:31:35.028 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.028 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.028 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.028 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "elasticsearch container image", 2024-06-09T22:31:35.028 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.028 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.029 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.029 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.029 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_grafana": { 2024-06-09T22:31:35.029 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_grafana", 2024-06-09T22:31:35.029 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.029 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.029 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.029 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-06-09T22:31:35.030 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.030 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.030 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.030 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Prometheus container image", 2024-06-09T22:31:35.030 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.030 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.030 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.031 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_haproxy": { 2024-06-09T22:31:35.031 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_haproxy", 2024-06-09T22:31:35.031 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.031 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.031 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.031 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-06-09T22:31:35.031 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.031 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.032 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.032 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "HAproxy container image", 2024-06-09T22:31:35.032 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.032 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.032 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.032 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_jaeger_agent": { 2024-06-09T22:31:35.032 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_jaeger_agent", 2024-06-09T22:31:35.033 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.033 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.033 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.033 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-06-09T22:31:35.033 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.033 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.033 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.033 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Jaeger agent container image", 2024-06-09T22:31:35.034 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.034 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.034 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.034 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_jaeger_collector": { 2024-06-09T22:31:35.034 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_jaeger_collector", 2024-06-09T22:31:35.034 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.034 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.035 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.035 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-06-09T22:31:35.035 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.035 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.035 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.035 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Jaeger collector container image", 2024-06-09T22:31:35.035 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.035 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.036 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.036 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_jaeger_query": { 2024-06-09T22:31:35.036 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_jaeger_query", 2024-06-09T22:31:35.036 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.036 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.036 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.036 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-06-09T22:31:35.037 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.037 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.037 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.037 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Jaeger query container image", 2024-06-09T22:31:35.037 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.037 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.037 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.038 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_keepalived": { 2024-06-09T22:31:35.038 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_keepalived", 2024-06-09T22:31:35.038 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.038 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.038 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.038 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-06-09T22:31:35.038 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.038 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.039 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.039 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Keepalived container image", 2024-06-09T22:31:35.039 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.039 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.039 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.039 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_loki": { 2024-06-09T22:31:35.039 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_loki", 2024-06-09T22:31:35.040 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.040 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.040 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.040 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-06-09T22:31:35.040 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.040 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.040 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.040 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Loki container image", 2024-06-09T22:31:35.040 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_node_exporter": { 2024-06-09T22:31:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_node_exporter", 2024-06-09T22:31:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-06-09T22:31:35.042 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.042 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.042 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.042 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Prometheus container image", 2024-06-09T22:31:35.042 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.042 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.042 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.043 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_nvmeof": { 2024-06-09T22:31:35.043 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_nvmeof", 2024-06-09T22:31:35.043 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.043 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.043 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.043 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-06-09T22:31:35.043 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.043 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.044 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.044 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Nvme-of container image", 2024-06-09T22:31:35.044 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.044 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.044 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.044 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_prometheus": { 2024-06-09T22:31:35.044 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_prometheus", 2024-06-09T22:31:35.044 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.045 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.045 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.045 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-06-09T22:31:35.045 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.045 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.045 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.045 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Prometheus container image", 2024-06-09T22:31:35.045 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.046 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.046 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.046 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_promtail": { 2024-06-09T22:31:35.046 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_promtail", 2024-06-09T22:31:35.046 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.046 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.046 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.047 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-06-09T22:31:35.047 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.047 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.047 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.047 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Promtail container image", 2024-06-09T22:31:35.047 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.047 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.047 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.048 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.048 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_samba": { 2024-06-09T22:31:35.048 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_samba", 2024-06-09T22:31:35.048 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.048 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.048 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.048 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-06-09T22:31:35.048 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.049 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.049 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.049 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Samba/SMB container image", 2024-06-09T22:31:35.049 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.049 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.049 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.049 INFO:teuthology.orchestra.run.smithi104.stdout: "container_image_snmp_gateway": { 2024-06-09T22:31:35.050 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_image_snmp_gateway", 2024-06-09T22:31:35.050 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.050 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.050 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.050 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-06-09T22:31:35.050 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.050 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.050 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.051 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "SNMP Gateway container image", 2024-06-09T22:31:35.051 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.051 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.051 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.051 INFO:teuthology.orchestra.run.smithi104.stdout: "container_init": { 2024-06-09T22:31:35.051 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "container_init", 2024-06-09T22:31:35.051 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.051 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.052 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.052 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.052 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.052 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.052 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.052 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Run podman/docker with `--init`", 2024-06-09T22:31:35.052 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.052 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.052 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.053 INFO:teuthology.orchestra.run.smithi104.stdout: "daemon_cache_timeout": { 2024-06-09T22:31:35.053 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "daemon_cache_timeout", 2024-06-09T22:31:35.053 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.053 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.053 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.053 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:35.053 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.054 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.054 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.054 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-06-09T22:31:35.054 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.054 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.054 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.054 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.054 INFO:teuthology.orchestra.run.smithi104.stdout: "default_cephadm_command_timeout": { 2024-06-09T22:31:35.055 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "default_cephadm_command_timeout", 2024-06-09T22:31:35.055 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.055 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.055 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.055 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "900", 2024-06-09T22:31:35.055 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.055 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.055 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.056 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-06-09T22:31:35.056 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.056 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.056 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.056 INFO:teuthology.orchestra.run.smithi104.stdout: "default_registry": { 2024-06-09T22:31:35.056 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "default_registry", 2024-06-09T22:31:35.056 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.056 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.057 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.057 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "docker.io", 2024-06-09T22:31:35.057 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.057 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.057 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.057 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-06-09T22:31:35.057 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.057 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: "device_cache_timeout": { 2024-06-09T22:31:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "device_cache_timeout", 2024-06-09T22:31:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "1800", 2024-06-09T22:31:35.059 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.059 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.059 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.059 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "seconds to cache device inventory", 2024-06-09T22:31:35.059 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.059 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.059 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.060 INFO:teuthology.orchestra.run.smithi104.stdout: "device_enhanced_scan": { 2024-06-09T22:31:35.060 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "device_enhanced_scan", 2024-06-09T22:31:35.060 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.060 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.060 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.060 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.060 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.060 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.060 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.061 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Use libstoragemgmt during device scans", 2024-06-09T22:31:35.061 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.061 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.061 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.061 INFO:teuthology.orchestra.run.smithi104.stdout: "facts_cache_timeout": { 2024-06-09T22:31:35.061 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "facts_cache_timeout", 2024-06-09T22:31:35.061 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.062 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.062 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.062 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:35.062 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.062 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.062 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.062 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "seconds to cache host facts data", 2024-06-09T22:31:35.062 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.062 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.063 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.063 INFO:teuthology.orchestra.run.smithi104.stdout: "grafana_dashboards_path": { 2024-06-09T22:31:35.063 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "grafana_dashboards_path", 2024-06-09T22:31:35.063 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.063 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.063 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.063 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-06-09T22:31:35.064 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.064 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.064 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.064 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-06-09T22:31:35.064 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.064 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.064 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.065 INFO:teuthology.orchestra.run.smithi104.stdout: "host_check_interval": { 2024-06-09T22:31:35.065 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "host_check_interval", 2024-06-09T22:31:35.065 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.065 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.065 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.065 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:35.065 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.065 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.066 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.066 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to perform a host check", 2024-06-09T22:31:35.066 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.066 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.066 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.066 INFO:teuthology.orchestra.run.smithi104.stdout: "hw_monitoring": { 2024-06-09T22:31:35.066 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "hw_monitoring", 2024-06-09T22:31:35.067 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.067 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.067 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.067 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.067 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.067 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.067 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.067 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-06-09T22:31:35.067 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.068 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.068 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.068 INFO:teuthology.orchestra.run.smithi104.stdout: "inventory_list_all": { 2024-06-09T22:31:35.068 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "inventory_list_all", 2024-06-09T22:31:35.068 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.068 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.068 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.069 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.069 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.069 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.069 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.069 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-06-09T22:31:35.069 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.069 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.069 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.070 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.070 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.070 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.070 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.070 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.070 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.070 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.071 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.071 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.071 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.071 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.071 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.071 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.071 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.071 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.071 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.072 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.072 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.072 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.072 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.072 INFO:teuthology.orchestra.run.smithi104.stdout: "log_refresh_metadata": { 2024-06-09T22:31:35.072 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_refresh_metadata", 2024-06-09T22:31:35.072 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.072 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.073 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.073 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.073 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.073 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.073 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.073 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-06-09T22:31:35.073 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.073 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.073 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.074 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.074 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.074 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.074 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.074 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.074 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.074 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.075 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.075 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.075 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-06-09T22:31:35.075 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.075 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.075 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.075 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.075 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.076 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.076 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.076 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.076 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.076 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.076 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.076 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.077 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.077 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.077 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.077 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.077 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.077 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.077 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.077 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.078 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.078 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.078 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.078 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.078 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.078 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.078 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.079 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.079 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.079 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.079 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.079 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.079 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.079 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.079 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.080 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.080 INFO:teuthology.orchestra.run.smithi104.stdout: "manage_etc_ceph_ceph_conf": { 2024-06-09T22:31:35.080 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-06-09T22:31:35.080 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.080 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.080 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.080 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.080 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.081 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.081 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.081 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-06-09T22:31:35.081 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.081 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.081 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.081 INFO:teuthology.orchestra.run.smithi104.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-06-09T22:31:35.081 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-06-09T22:31:35.082 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.082 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.082 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.082 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "*", 2024-06-09T22:31:35.082 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.082 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.082 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.082 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-06-09T22:31:35.082 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.083 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.083 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.083 INFO:teuthology.orchestra.run.smithi104.stdout: "max_count_per_host": { 2024-06-09T22:31:35.083 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_count_per_host", 2024-06-09T22:31:35.083 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.083 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.083 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.084 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:35.084 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.084 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.084 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.084 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "max number of daemons per service per host", 2024-06-09T22:31:35.084 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.084 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.084 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.085 INFO:teuthology.orchestra.run.smithi104.stdout: "max_osd_draining_count": { 2024-06-09T22:31:35.085 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_osd_draining_count", 2024-06-09T22:31:35.085 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.085 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.085 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.085 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:35.085 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.085 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.086 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.086 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-06-09T22:31:35.086 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.086 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.086 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.086 INFO:teuthology.orchestra.run.smithi104.stdout: "migration_current": { 2024-06-09T22:31:35.086 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "migration_current", 2024-06-09T22:31:35.087 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.087 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.087 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.087 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.087 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.087 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.087 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.087 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "internal - do not modify", 2024-06-09T22:31:35.088 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.088 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.088 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.088 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": { 2024-06-09T22:31:35.088 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mode", 2024-06-09T22:31:35.088 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.088 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.088 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.089 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "root", 2024-06-09T22:31:35.089 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.089 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.089 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.089 INFO:teuthology.orchestra.run.smithi104.stdout: "cephadm-package", 2024-06-09T22:31:35.089 INFO:teuthology.orchestra.run.smithi104.stdout: "root" 2024-06-09T22:31:35.089 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.089 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "mode for remote execution of cephadm", 2024-06-09T22:31:35.090 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.090 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.090 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.090 INFO:teuthology.orchestra.run.smithi104.stdout: "oob_default_addr": { 2024-06-09T22:31:35.090 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "oob_default_addr", 2024-06-09T22:31:35.090 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.090 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.091 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.091 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "169.254.1.1", 2024-06-09T22:31:35.091 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.091 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.091 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.091 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Default address for RedFish API (oob management).", 2024-06-09T22:31:35.091 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.091 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.091 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.092 INFO:teuthology.orchestra.run.smithi104.stdout: "prometheus_alerts_path": { 2024-06-09T22:31:35.092 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "prometheus_alerts_path", 2024-06-09T22:31:35.092 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.092 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.092 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.092 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-06-09T22:31:35.092 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.092 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.093 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.093 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-06-09T22:31:35.093 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.093 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.093 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.093 INFO:teuthology.orchestra.run.smithi104.stdout: "registry_insecure": { 2024-06-09T22:31:35.093 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "registry_insecure", 2024-06-09T22:31:35.093 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.094 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.094 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.094 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.094 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.094 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.094 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.094 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-06-09T22:31:35.094 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.095 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.095 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.095 INFO:teuthology.orchestra.run.smithi104.stdout: "registry_password": { 2024-06-09T22:31:35.095 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "registry_password", 2024-06-09T22:31:35.095 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.095 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.095 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.095 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.096 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.096 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.096 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.096 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-06-09T22:31:35.096 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.096 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.096 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.097 INFO:teuthology.orchestra.run.smithi104.stdout: "registry_url": { 2024-06-09T22:31:35.097 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "registry_url", 2024-06-09T22:31:35.097 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.097 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.097 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.097 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.097 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.097 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.098 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.098 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-06-09T22:31:35.098 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.098 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.098 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.098 INFO:teuthology.orchestra.run.smithi104.stdout: "registry_username": { 2024-06-09T22:31:35.098 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "registry_username", 2024-06-09T22:31:35.099 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.099 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.099 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.099 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.099 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.099 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.099 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.099 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-06-09T22:31:35.099 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.100 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.100 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.100 INFO:teuthology.orchestra.run.smithi104.stdout: "secure_monitoring_stack": { 2024-06-09T22:31:35.100 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "secure_monitoring_stack", 2024-06-09T22:31:35.100 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.100 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.100 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.100 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.101 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.101 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.101 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.101 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-06-09T22:31:35.101 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.101 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.101 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.102 INFO:teuthology.orchestra.run.smithi104.stdout: "service_discovery_port": { 2024-06-09T22:31:35.102 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "service_discovery_port", 2024-06-09T22:31:35.102 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.102 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.102 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.102 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8765", 2024-06-09T22:31:35.102 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.102 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.103 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.103 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "cephadm service discovery port", 2024-06-09T22:31:35.103 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.103 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.103 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.103 INFO:teuthology.orchestra.run.smithi104.stdout: "ssh_config_file": { 2024-06-09T22:31:35.103 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ssh_config_file", 2024-06-09T22:31:35.103 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.104 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.104 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.104 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.104 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.104 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.104 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.104 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-06-09T22:31:35.104 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.104 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.105 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.105 INFO:teuthology.orchestra.run.smithi104.stdout: "use_agent": { 2024-06-09T22:31:35.105 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "use_agent", 2024-06-09T22:31:35.105 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.105 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.105 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.105 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.105 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.106 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.106 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.106 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-06-09T22:31:35.106 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.106 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.106 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.106 INFO:teuthology.orchestra.run.smithi104.stdout: "use_repo_digest": { 2024-06-09T22:31:35.107 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "use_repo_digest", 2024-06-09T22:31:35.107 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.107 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.107 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.107 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.107 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.107 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.107 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.108 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-06-09T22:31:35.108 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.108 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.108 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.108 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_on_failed_host_check": { 2024-06-09T22:31:35.108 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_on_failed_host_check", 2024-06-09T22:31:35.108 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.109 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.109 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.109 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.109 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.109 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.109 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.109 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "raise a health warning if the host check fails", 2024-06-09T22:31:35.109 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.110 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.110 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.110 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_on_stray_daemons": { 2024-06-09T22:31:35.110 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_on_stray_daemons", 2024-06-09T22:31:35.110 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.110 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.110 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.111 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.111 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.111 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.111 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.111 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-06-09T22:31:35.111 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.111 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.111 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.112 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_on_stray_hosts": { 2024-06-09T22:31:35.112 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_on_stray_hosts", 2024-06-09T22:31:35.112 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.112 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.112 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.112 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.112 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.112 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.112 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.113 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-06-09T22:31:35.113 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.113 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.113 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.113 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.113 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.113 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crash", 2024-06-09T22:31:35.114 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.114 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.114 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.114 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.114 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.114 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.114 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.114 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.115 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.115 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.115 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.115 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.115 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.115 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.115 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.115 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.116 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.116 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.116 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.116 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.116 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.116 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.116 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.116 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.117 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.117 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.117 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.117 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.117 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.117 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.117 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.117 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.118 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.118 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.118 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.118 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.118 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.118 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.118 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.118 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.119 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.119 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.119 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.119 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.119 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.119 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.119 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.119 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.120 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.120 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.120 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.120 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.120 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.120 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.120 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.120 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.121 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.121 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.121 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.121 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.121 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.121 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.121 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.121 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.122 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.122 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.122 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.122 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.122 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.122 INFO:teuthology.orchestra.run.smithi104.stdout: "retain_interval": { 2024-06-09T22:31:35.123 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "retain_interval", 2024-06-09T22:31:35.123 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.123 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.123 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.123 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "31536000", 2024-06-09T22:31:35.123 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.123 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.123 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.124 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how long to retain crashes before pruning them", 2024-06-09T22:31:35.124 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.124 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.124 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.124 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_recent_interval": { 2024-06-09T22:31:35.124 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_recent_interval", 2024-06-09T22:31:35.124 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.125 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.125 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.125 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "1209600", 2024-06-09T22:31:35.125 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.125 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.125 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.125 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "time interval in which to warn about recent crashes", 2024-06-09T22:31:35.125 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.126 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.126 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.126 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.126 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.126 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "dashboard", 2024-06-09T22:31:35.127 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.127 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.127 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.127 INFO:teuthology.orchestra.run.smithi104.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-06-09T22:31:35.127 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-06-09T22:31:35.127 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.127 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.127 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.128 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:35.128 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.128 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.128 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.128 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.128 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.128 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.129 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.129 INFO:teuthology.orchestra.run.smithi104.stdout: "ALERTMANAGER_API_HOST": { 2024-06-09T22:31:35.129 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ALERTMANAGER_API_HOST", 2024-06-09T22:31:35.129 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.129 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.129 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.129 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.130 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.130 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.130 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.130 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.130 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.130 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.131 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.131 INFO:teuthology.orchestra.run.smithi104.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-06-09T22:31:35.131 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-06-09T22:31:35.131 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.131 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.131 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.131 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.132 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.132 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.132 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.132 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.132 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.132 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.132 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.133 INFO:teuthology.orchestra.run.smithi104.stdout: "AUDIT_API_ENABLED": { 2024-06-09T22:31:35.133 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "AUDIT_API_ENABLED", 2024-06-09T22:31:35.133 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.133 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.133 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.133 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.133 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.133 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.134 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.134 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.134 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.134 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.134 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.134 INFO:teuthology.orchestra.run.smithi104.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-06-09T22:31:35.134 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-06-09T22:31:35.135 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.135 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.135 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.135 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.135 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.135 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.135 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.135 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.136 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.136 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.136 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.136 INFO:teuthology.orchestra.run.smithi104.stdout: "ENABLE_BROWSABLE_API": { 2024-06-09T22:31:35.136 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ENABLE_BROWSABLE_API", 2024-06-09T22:31:35.136 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.136 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.137 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.137 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.137 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.137 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.137 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.137 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.137 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.138 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.138 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.138 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-06-09T22:31:35.138 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-06-09T22:31:35.138 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.138 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.138 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.139 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.139 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.139 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.139 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.139 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.139 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.139 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.139 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.140 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-06-09T22:31:35.140 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-06-09T22:31:35.140 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.140 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.140 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.140 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.140 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.141 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.141 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.141 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.141 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.141 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.141 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.141 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-06-09T22:31:35.141 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-06-09T22:31:35.142 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.142 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.142 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.142 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.142 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.142 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.142 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.142 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.143 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.143 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.143 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.143 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-06-09T22:31:35.143 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-06-09T22:31:35.143 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.143 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.144 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.144 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.144 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.144 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.144 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.144 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.144 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.144 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.145 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.145 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_NFS": { 2024-06-09T22:31:35.145 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-06-09T22:31:35.145 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.145 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.145 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.145 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.145 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.146 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.146 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.146 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.146 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.146 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.146 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.146 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_RBD": { 2024-06-09T22:31:35.147 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-06-09T22:31:35.147 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.147 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.147 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.147 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.147 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.147 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.148 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.148 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.148 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.148 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.148 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.148 INFO:teuthology.orchestra.run.smithi104.stdout: "FEATURE_TOGGLE_RGW": { 2024-06-09T22:31:35.148 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-06-09T22:31:35.148 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.149 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.149 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.149 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.149 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.149 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.149 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.149 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.149 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.150 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.150 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.150 INFO:teuthology.orchestra.run.smithi104.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-06-09T22:31:35.150 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-06-09T22:31:35.150 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.150 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.150 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.150 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.151 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.151 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.151 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.151 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.151 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.151 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.151 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.152 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_API_PASSWORD": { 2024-06-09T22:31:35.152 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_API_PASSWORD", 2024-06-09T22:31:35.152 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.152 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.152 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.152 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "admin", 2024-06-09T22:31:35.152 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.152 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.152 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.153 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.153 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.153 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.153 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.153 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-06-09T22:31:35.153 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-06-09T22:31:35.153 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.154 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.154 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.154 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.154 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.154 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.154 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.154 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.154 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.154 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.155 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.155 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_API_URL": { 2024-06-09T22:31:35.155 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_API_URL", 2024-06-09T22:31:35.155 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.155 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.155 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.155 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.156 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.156 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.156 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.156 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.156 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.156 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.156 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.157 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_API_USERNAME": { 2024-06-09T22:31:35.157 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_API_USERNAME", 2024-06-09T22:31:35.157 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.157 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.157 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.157 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "admin", 2024-06-09T22:31:35.157 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.157 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.158 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.158 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.158 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.158 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.158 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.158 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-06-09T22:31:35.158 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-06-09T22:31:35.159 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.159 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.159 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.159 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.159 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.159 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.159 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.159 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.159 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.160 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.160 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.160 INFO:teuthology.orchestra.run.smithi104.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-06-09T22:31:35.160 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-06-09T22:31:35.160 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.160 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.160 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.161 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.161 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.161 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.161 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.161 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.161 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.161 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.162 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.162 INFO:teuthology.orchestra.run.smithi104.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-06-09T22:31:35.162 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-06-09T22:31:35.162 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.162 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.162 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.162 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.163 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.163 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.163 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.163 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.163 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.163 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.163 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.163 INFO:teuthology.orchestra.run.smithi104.stdout: "ISSUE_TRACKER_API_KEY": { 2024-06-09T22:31:35.164 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-06-09T22:31:35.164 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.164 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.164 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.164 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.164 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.164 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.164 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.165 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.165 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.165 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.165 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.165 INFO:teuthology.orchestra.run.smithi104.stdout: "PROMETHEUS_API_HOST": { 2024-06-09T22:31:35.165 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PROMETHEUS_API_HOST", 2024-06-09T22:31:35.165 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.166 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.166 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.166 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.166 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.166 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.166 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.166 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.166 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.167 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.167 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.167 INFO:teuthology.orchestra.run.smithi104.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-06-09T22:31:35.167 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-06-09T22:31:35.167 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.167 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.168 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.168 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.168 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.168 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.168 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.168 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.168 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.168 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.168 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.169 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-06-09T22:31:35.169 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-06-09T22:31:35.169 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.169 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.169 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.169 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.169 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.169 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.170 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.170 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.170 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.170 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.170 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.170 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-06-09T22:31:35.170 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-06-09T22:31:35.171 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.171 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.171 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.171 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.171 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.171 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.171 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.171 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.171 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.172 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.172 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.172 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-06-09T22:31:35.172 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-06-09T22:31:35.172 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.172 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.172 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.172 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.173 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.173 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.173 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.173 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.173 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.173 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.173 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.173 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-06-09T22:31:35.174 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-06-09T22:31:35.174 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.174 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.174 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.174 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.174 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.174 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.174 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.175 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.175 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.175 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.175 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.175 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-06-09T22:31:35.175 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-06-09T22:31:35.175 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.176 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.176 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.176 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.176 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.176 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.176 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.176 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.176 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.177 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.177 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.177 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-06-09T22:31:35.177 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-06-09T22:31:35.177 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.177 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.177 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.178 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.178 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.178 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.178 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.178 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.178 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.178 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.178 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.179 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-06-09T22:31:35.179 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-06-09T22:31:35.179 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.179 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.179 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.179 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.179 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.179 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.180 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.180 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.180 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.180 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.180 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.180 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_ENABLED": { 2024-06-09T22:31:35.180 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_ENABLED", 2024-06-09T22:31:35.180 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.181 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.181 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.181 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.181 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.181 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.181 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.181 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.181 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.182 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.182 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.182 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-06-09T22:31:35.182 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-06-09T22:31:35.182 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.182 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.182 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.183 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-06-09T22:31:35.183 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.183 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.183 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.183 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.183 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.183 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.183 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.184 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-06-09T22:31:35.184 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-06-09T22:31:35.184 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.184 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.184 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.184 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:35.184 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.184 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.185 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.185 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.185 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.185 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.185 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.185 INFO:teuthology.orchestra.run.smithi104.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-06-09T22:31:35.185 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-06-09T22:31:35.185 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.186 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.186 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.186 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8", 2024-06-09T22:31:35.186 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.186 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.186 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.186 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.186 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.187 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.187 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.187 INFO:teuthology.orchestra.run.smithi104.stdout: "REST_REQUESTS_TIMEOUT": { 2024-06-09T22:31:35.187 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-06-09T22:31:35.187 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.187 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.187 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.188 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "45", 2024-06-09T22:31:35.188 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.188 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.188 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.188 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.188 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.188 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.188 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.189 INFO:teuthology.orchestra.run.smithi104.stdout: "RGW_API_ACCESS_KEY": { 2024-06-09T22:31:35.189 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RGW_API_ACCESS_KEY", 2024-06-09T22:31:35.189 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.189 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.189 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.189 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.189 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.190 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.190 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.190 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.190 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.190 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.190 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.190 INFO:teuthology.orchestra.run.smithi104.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-06-09T22:31:35.191 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-06-09T22:31:35.191 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.191 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.191 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.191 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "admin", 2024-06-09T22:31:35.191 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.191 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.191 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.192 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.192 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.192 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.192 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.192 INFO:teuthology.orchestra.run.smithi104.stdout: "RGW_API_SECRET_KEY": { 2024-06-09T22:31:35.192 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RGW_API_SECRET_KEY", 2024-06-09T22:31:35.192 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.193 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.193 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.193 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.193 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.193 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.193 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.193 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.193 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.194 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.194 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.194 INFO:teuthology.orchestra.run.smithi104.stdout: "RGW_API_SSL_VERIFY": { 2024-06-09T22:31:35.194 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "RGW_API_SSL_VERIFY", 2024-06-09T22:31:35.194 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.194 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.194 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.195 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.195 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.195 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.195 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.195 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.195 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.195 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.195 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.196 INFO:teuthology.orchestra.run.smithi104.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-06-09T22:31:35.196 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-06-09T22:31:35.196 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.196 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.196 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.196 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:35.196 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.197 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.197 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.197 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.197 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.197 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.197 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.197 INFO:teuthology.orchestra.run.smithi104.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-06-09T22:31:35.197 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-06-09T22:31:35.198 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.198 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.198 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.198 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:35.198 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.198 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.198 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.199 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.199 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.199 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.199 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.199 INFO:teuthology.orchestra.run.smithi104.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-06-09T22:31:35.199 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-06-09T22:31:35.199 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.199 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.200 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.200 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5", 2024-06-09T22:31:35.200 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.200 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.200 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.200 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.200 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.200 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.201 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.201 INFO:teuthology.orchestra.run.smithi104.stdout: "cross_origin_url": { 2024-06-09T22:31:35.201 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cross_origin_url", 2024-06-09T22:31:35.201 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.201 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.201 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.201 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.202 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.202 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.202 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.202 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.202 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.202 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.202 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.203 INFO:teuthology.orchestra.run.smithi104.stdout: "crt_file": { 2024-06-09T22:31:35.203 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "crt_file", 2024-06-09T22:31:35.203 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.203 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.203 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.203 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.203 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.203 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.203 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.204 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.204 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.204 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.204 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.204 INFO:teuthology.orchestra.run.smithi104.stdout: "debug": { 2024-06-09T22:31:35.204 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "debug", 2024-06-09T22:31:35.204 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.205 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.205 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.205 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.205 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.205 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.205 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.205 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Enable/disable debug options", 2024-06-09T22:31:35.205 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.205 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.206 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.206 INFO:teuthology.orchestra.run.smithi104.stdout: "jwt_token_ttl": { 2024-06-09T22:31:35.206 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "jwt_token_ttl", 2024-06-09T22:31:35.206 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.206 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.206 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.206 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "28800", 2024-06-09T22:31:35.207 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.207 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.207 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.207 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.207 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.207 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.207 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.208 INFO:teuthology.orchestra.run.smithi104.stdout: "key_file": { 2024-06-09T22:31:35.208 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "key_file", 2024-06-09T22:31:35.208 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.208 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.208 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.208 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.208 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.208 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.208 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.209 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.209 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.209 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.209 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.209 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.209 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.209 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.209 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.210 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.210 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.210 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.210 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.210 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.210 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.210 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.210 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.211 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.211 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.211 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.211 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.211 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.211 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.211 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.211 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.212 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.212 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.212 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.212 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.212 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.212 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.212 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.212 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.212 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.213 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.213 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.213 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.213 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.213 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.213 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.213 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.213 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.214 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.214 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.214 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.214 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.214 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.214 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.214 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.214 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.214 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.215 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.215 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.215 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.215 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.215 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.215 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.215 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.216 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.216 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.216 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.216 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.216 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.216 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.216 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.216 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.217 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.217 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.217 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.217 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.217 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.217 INFO:teuthology.orchestra.run.smithi104.stdout: "motd": { 2024-06-09T22:31:35.217 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "motd", 2024-06-09T22:31:35.218 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.218 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.218 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.218 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.218 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.218 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.218 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.219 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "The message of the day", 2024-06-09T22:31:35.219 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.219 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.219 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.219 INFO:teuthology.orchestra.run.smithi104.stdout: "redirect_resolve_ip_addr": { 2024-06-09T22:31:35.219 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "redirect_resolve_ip_addr", 2024-06-09T22:31:35.219 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.219 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.220 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.220 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.220 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.220 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.220 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.220 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.220 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.220 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.221 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.221 INFO:teuthology.orchestra.run.smithi104.stdout: "server_addr": { 2024-06-09T22:31:35.221 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_addr", 2024-06-09T22:31:35.221 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.221 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.221 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.221 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "::", 2024-06-09T22:31:35.222 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.222 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.222 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.222 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.222 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.222 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.222 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.223 INFO:teuthology.orchestra.run.smithi104.stdout: "server_port": { 2024-06-09T22:31:35.223 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_port", 2024-06-09T22:31:35.223 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.223 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.223 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.223 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8080", 2024-06-09T22:31:35.223 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.223 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.224 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.224 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.224 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.224 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.224 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.224 INFO:teuthology.orchestra.run.smithi104.stdout: "ssl": { 2024-06-09T22:31:35.224 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ssl", 2024-06-09T22:31:35.224 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.225 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.225 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.225 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.225 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.225 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.225 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.225 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.225 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.226 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.226 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.226 INFO:teuthology.orchestra.run.smithi104.stdout: "ssl_server_port": { 2024-06-09T22:31:35.226 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ssl_server_port", 2024-06-09T22:31:35.226 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.226 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.226 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.227 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8443", 2024-06-09T22:31:35.227 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.227 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.227 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.227 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.227 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.227 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.227 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.228 INFO:teuthology.orchestra.run.smithi104.stdout: "standby_behaviour": { 2024-06-09T22:31:35.228 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "standby_behaviour", 2024-06-09T22:31:35.228 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.228 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.228 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.228 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "redirect", 2024-06-09T22:31:35.228 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: "redirect" 2024-06-09T22:31:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "standby_error_status_code": { 2024-06-09T22:31:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "standby_error_status_code", 2024-06-09T22:31:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "500", 2024-06-09T22:31:35.231 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "400", 2024-06-09T22:31:35.231 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "599", 2024-06-09T22:31:35.231 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.231 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.231 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.231 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.231 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.231 INFO:teuthology.orchestra.run.smithi104.stdout: "url_prefix": { 2024-06-09T22:31:35.232 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "url_prefix", 2024-06-09T22:31:35.232 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.232 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.232 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.232 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.232 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.232 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.232 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.233 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.233 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.233 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.233 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.233 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.233 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.234 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "devicehealth", 2024-06-09T22:31:35.234 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.234 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.234 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.234 INFO:teuthology.orchestra.run.smithi104.stdout: "enable_monitoring": { 2024-06-09T22:31:35.234 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "enable_monitoring", 2024-06-09T22:31:35.234 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.234 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.235 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.235 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.235 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.235 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.235 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.235 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "monitor device health metrics", 2024-06-09T22:31:35.235 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.235 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.235 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.236 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.236 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.236 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.236 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.236 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.236 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.237 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.237 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.237 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.237 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.237 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.237 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.237 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.237 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.237 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.237 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.238 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.238 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.238 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.238 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.238 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.238 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.238 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.239 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.239 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.239 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.239 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.239 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.239 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.239 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.239 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.239 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.240 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.240 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.240 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.240 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.240 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.240 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.240 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.240 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.241 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.241 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.241 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.241 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.241 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.241 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.241 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.241 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.242 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.242 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.242 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.242 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.242 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.242 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.242 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.243 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.243 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.243 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.243 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.243 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.243 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.243 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.243 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.244 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.244 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.244 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.244 INFO:teuthology.orchestra.run.smithi104.stdout: "mark_out_threshold": { 2024-06-09T22:31:35.244 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mark_out_threshold", 2024-06-09T22:31:35.244 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.244 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.245 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.245 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "2419200", 2024-06-09T22:31:35.245 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.245 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.245 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.245 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-06-09T22:31:35.245 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.245 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.246 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.246 INFO:teuthology.orchestra.run.smithi104.stdout: "pool_name": { 2024-06-09T22:31:35.246 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "pool_name", 2024-06-09T22:31:35.246 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.246 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.246 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.247 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "device_health_metrics", 2024-06-09T22:31:35.247 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.247 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.247 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.247 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "name of pool in which to store device health metrics", 2024-06-09T22:31:35.247 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.247 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.247 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.248 INFO:teuthology.orchestra.run.smithi104.stdout: "retention_period": { 2024-06-09T22:31:35.248 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "retention_period", 2024-06-09T22:31:35.248 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.248 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.248 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.248 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "15552000", 2024-06-09T22:31:35.248 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.249 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.249 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.249 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how long to retain device health metrics", 2024-06-09T22:31:35.249 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.249 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.249 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.250 INFO:teuthology.orchestra.run.smithi104.stdout: "scrape_frequency": { 2024-06-09T22:31:35.250 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "scrape_frequency", 2024-06-09T22:31:35.250 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.250 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.250 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.250 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "86400", 2024-06-09T22:31:35.250 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.250 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.251 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.251 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to scrape device health metrics", 2024-06-09T22:31:35.251 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.251 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.251 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.251 INFO:teuthology.orchestra.run.smithi104.stdout: "self_heal": { 2024-06-09T22:31:35.251 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "self_heal", 2024-06-09T22:31:35.252 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.252 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.252 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.252 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.252 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.252 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.252 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.252 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-06-09T22:31:35.253 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.253 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.253 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.253 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:35.253 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:35.253 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.253 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.254 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.254 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:35.254 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.254 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.254 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.254 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how frequently to wake up and check device health", 2024-06-09T22:31:35.254 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.254 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.255 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.255 INFO:teuthology.orchestra.run.smithi104.stdout: "warn_threshold": { 2024-06-09T22:31:35.255 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "warn_threshold", 2024-06-09T22:31:35.255 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.255 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.255 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.255 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "7257600", 2024-06-09T22:31:35.256 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.256 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.256 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.256 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-06-09T22:31:35.256 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.256 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.256 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.256 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.257 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.257 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "diskprediction_local", 2024-06-09T22:31:35.257 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.257 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.257 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.257 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.258 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.258 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.258 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.258 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.258 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.258 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.258 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.258 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.259 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.259 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.259 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.259 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.259 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.259 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.259 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.259 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.259 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.260 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.260 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.260 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.260 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.260 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.260 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.260 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.260 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.261 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.261 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.261 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.261 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.261 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.261 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.261 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.261 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.262 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.262 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.262 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.262 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.262 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.262 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.262 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.262 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.263 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.263 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.263 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.263 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.263 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.263 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.263 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.263 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.263 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.264 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.264 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.264 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.264 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.264 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.264 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.264 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.264 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.264 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.265 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.265 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.265 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.265 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.265 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.265 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.265 INFO:teuthology.orchestra.run.smithi104.stdout: "predict_interval": { 2024-06-09T22:31:35.265 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "predict_interval", 2024-06-09T22:31:35.266 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.266 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.266 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.266 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "86400", 2024-06-09T22:31:35.266 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.266 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.267 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.267 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.267 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.267 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.267 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.267 INFO:teuthology.orchestra.run.smithi104.stdout: "predictor_model": { 2024-06-09T22:31:35.267 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "predictor_model", 2024-06-09T22:31:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "prophetstor", 2024-06-09T22:31:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "600", 2024-06-09T22:31:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "influx", 2024-06-09T22:31:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": false, 2024-06-09T22:31:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "influxdb python module not found", 2024-06-09T22:31:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "batch_size": { 2024-06-09T22:31:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "batch_size", 2024-06-09T22:31:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5000", 2024-06-09T22:31:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-06-09T22:31:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "database": { 2024-06-09T22:31:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "database", 2024-06-09T22:31:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "ceph", 2024-06-09T22:31:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-06-09T22:31:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": { 2024-06-09T22:31:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "hostname", 2024-06-09T22:31:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "InfluxDB server hostname", 2024-06-09T22:31:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "30", 2024-06-09T22:31:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "5", 2024-06-09T22:31:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-06-09T22:31:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.284 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.284 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.284 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.284 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.284 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.284 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.284 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.284 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.284 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.285 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.285 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.285 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.285 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.285 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.285 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.285 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.286 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.286 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.286 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.286 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.286 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.286 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.286 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.287 INFO:teuthology.orchestra.run.smithi104.stdout: "password": { 2024-06-09T22:31:35.287 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "password", 2024-06-09T22:31:35.287 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.287 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.287 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.287 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.287 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.287 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.288 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.288 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "password of InfluxDB server user", 2024-06-09T22:31:35.288 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.288 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.288 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.288 INFO:teuthology.orchestra.run.smithi104.stdout: "port": { 2024-06-09T22:31:35.288 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "port", 2024-06-09T22:31:35.288 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.289 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.289 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.289 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "8086", 2024-06-09T22:31:35.289 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.289 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.289 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.289 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "InfluxDB server port", 2024-06-09T22:31:35.289 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.289 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.290 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.290 INFO:teuthology.orchestra.run.smithi104.stdout: "ssl": { 2024-06-09T22:31:35.290 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ssl", 2024-06-09T22:31:35.290 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.290 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.290 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.290 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "false", 2024-06-09T22:31:35.291 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.291 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.291 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.291 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-06-09T22:31:35.291 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.291 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.291 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.291 INFO:teuthology.orchestra.run.smithi104.stdout: "threads": { 2024-06-09T22:31:35.292 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "threads", 2024-06-09T22:31:35.292 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.292 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.292 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.292 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5", 2024-06-09T22:31:35.292 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1", 2024-06-09T22:31:35.292 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "32", 2024-06-09T22:31:35.292 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.292 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-06-09T22:31:35.293 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.293 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.293 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.293 INFO:teuthology.orchestra.run.smithi104.stdout: "username": { 2024-06-09T22:31:35.293 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "username", 2024-06-09T22:31:35.293 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.293 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.293 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.294 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.294 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.294 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.294 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.294 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "username of InfluxDB server user", 2024-06-09T22:31:35.294 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.294 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.294 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.295 INFO:teuthology.orchestra.run.smithi104.stdout: "verify_ssl": { 2024-06-09T22:31:35.295 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "verify_ssl", 2024-06-09T22:31:35.295 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.295 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.295 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.295 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "true", 2024-06-09T22:31:35.295 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.296 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.296 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.296 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-06-09T22:31:35.296 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.296 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.296 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.296 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.296 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.297 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "insights", 2024-06-09T22:31:35.297 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.297 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.297 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.297 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.297 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.297 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.298 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.298 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.298 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.298 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.298 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.298 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.298 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.298 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.298 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.299 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.299 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.299 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.299 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.299 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.299 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.299 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.299 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.300 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.300 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.300 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.300 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.300 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.300 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.300 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.300 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.300 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.301 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.301 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.301 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.301 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.301 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.301 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.301 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.301 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.301 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.302 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.302 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.302 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.302 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.302 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.302 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.302 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.302 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.303 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.303 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.303 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.303 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.303 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.303 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.303 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.303 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.303 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.304 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.304 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.304 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.304 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.304 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.304 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.304 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.304 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.304 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.305 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.305 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.305 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.305 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.305 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "iostat", 2024-06-09T22:31:35.305 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.305 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.306 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.306 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.306 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.306 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.306 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.306 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.306 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.306 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.306 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.307 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.307 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.307 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.307 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.307 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.307 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.307 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.307 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.307 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.308 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.308 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.308 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.308 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.308 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.308 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.308 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.308 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.309 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.309 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.309 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.309 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.309 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.309 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.309 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.309 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.310 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.310 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.310 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.310 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.310 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.310 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.310 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.310 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.311 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.311 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.311 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.311 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.311 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.311 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.311 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.311 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.311 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.312 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.312 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.312 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.312 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.312 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.312 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.312 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.312 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.312 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.313 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.313 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.313 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.313 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.313 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.313 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.313 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.313 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.314 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.314 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "k8sevents", 2024-06-09T22:31:35.314 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.314 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.314 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.314 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_event_retention_days": { 2024-06-09T22:31:35.315 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "ceph_event_retention_days", 2024-06-09T22:31:35.315 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.315 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.315 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.315 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "7", 2024-06-09T22:31:35.315 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.315 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.315 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.316 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Days to hold ceph event information within local cache", 2024-06-09T22:31:35.316 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.316 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.316 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.316 INFO:teuthology.orchestra.run.smithi104.stdout: "config_check_secs": { 2024-06-09T22:31:35.316 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "config_check_secs", 2024-06-09T22:31:35.316 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.316 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.317 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.317 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:35.317 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "10", 2024-06-09T22:31:35.317 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.317 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.317 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-06-09T22:31:35.317 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.317 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.318 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.318 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.318 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.318 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.318 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.318 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.318 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.319 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.319 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.319 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.319 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.319 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.319 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.319 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.320 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.320 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.320 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.320 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.320 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.320 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.320 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.320 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.321 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.321 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.321 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.321 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.321 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.321 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.321 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.321 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.322 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.322 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.322 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.322 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.322 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.322 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.322 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.323 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.323 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.323 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.323 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.323 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.323 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.323 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.323 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.323 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.324 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.324 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.324 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.324 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.324 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.324 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.324 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.324 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.325 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.325 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.325 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.325 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.325 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.325 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.325 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.326 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.326 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.326 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.326 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.326 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.326 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.326 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.327 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.327 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "localpool", 2024-06-09T22:31:35.327 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.327 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.327 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.328 INFO:teuthology.orchestra.run.smithi104.stdout: "failure_domain": { 2024-06-09T22:31:35.328 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "failure_domain", 2024-06-09T22:31:35.328 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.328 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.328 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.329 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "host", 2024-06-09T22:31:35.329 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.329 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.329 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.330 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "failure domain for any created local pool", 2024-06-09T22:31:35.330 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-06-09T22:31:35.330 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.330 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.330 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.330 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.331 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.331 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.331 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.331 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.331 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.331 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.331 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.331 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.331 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.332 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.332 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.332 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.332 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.332 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.332 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.332 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.332 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.333 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.333 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.333 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.333 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.333 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.333 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.333 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.334 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.334 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.334 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.334 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.334 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.334 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.334 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.335 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.335 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.335 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.335 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.335 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.335 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.335 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.335 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.335 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.336 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.336 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.336 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.336 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.336 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.336 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.336 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.336 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.337 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.338 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.338 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.338 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.338 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.338 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.338 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.338 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.338 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.339 INFO:teuthology.orchestra.run.smithi104.stdout: "min_size": { 2024-06-09T22:31:35.339 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "min_size", 2024-06-09T22:31:35.339 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.339 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.339 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.339 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.339 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.339 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.340 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.340 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "default min_size for any created local pool", 2024-06-09T22:31:35.340 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-06-09T22:31:35.340 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.340 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.340 INFO:teuthology.orchestra.run.smithi104.stdout: "num_rep": { 2024-06-09T22:31:35.340 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "num_rep", 2024-06-09T22:31:35.340 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.341 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.341 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.341 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "3", 2024-06-09T22:31:35.341 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.341 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.341 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.341 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "default replica count for any created local pool", 2024-06-09T22:31:35.341 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.342 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.342 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.342 INFO:teuthology.orchestra.run.smithi104.stdout: "pg_num": { 2024-06-09T22:31:35.342 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "pg_num", 2024-06-09T22:31:35.342 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.342 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.342 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.343 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "128", 2024-06-09T22:31:35.343 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.343 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.343 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.343 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "default pg_num for any created local pool", 2024-06-09T22:31:35.343 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.343 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.343 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.343 INFO:teuthology.orchestra.run.smithi104.stdout: "prefix": { 2024-06-09T22:31:35.344 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "prefix", 2024-06-09T22:31:35.344 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.344 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.344 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.344 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.344 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.344 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.344 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.344 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "name prefix for any created local pool", 2024-06-09T22:31:35.345 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.345 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.345 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.345 INFO:teuthology.orchestra.run.smithi104.stdout: "subtree": { 2024-06-09T22:31:35.345 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "subtree", 2024-06-09T22:31:35.345 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.345 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.346 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.346 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "rack", 2024-06-09T22:31:35.346 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.346 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.346 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.346 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "CRUSH level for which to create a local pool", 2024-06-09T22:31:35.346 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-06-09T22:31:35.347 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.347 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.347 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.347 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.347 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mds_autoscaler", 2024-06-09T22:31:35.347 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.347 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.348 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.348 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.348 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.348 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.348 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.348 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.348 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.348 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.348 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.349 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.349 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.349 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.349 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.349 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.349 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.349 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.349 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.349 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.350 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.350 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.350 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.350 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.350 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.350 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.351 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.351 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.351 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.351 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.351 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.351 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.351 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.351 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.351 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.352 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.352 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.352 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.352 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.352 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.352 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.352 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.352 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.353 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.353 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.353 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.353 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.353 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.353 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.353 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.354 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.354 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.354 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.354 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.354 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.354 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.354 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.355 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.355 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.355 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.355 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.355 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.355 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.355 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.355 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.355 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.356 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.356 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.356 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.356 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.356 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.356 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mirroring", 2024-06-09T22:31:35.357 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.357 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.357 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.357 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.357 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.357 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.357 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.357 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.358 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.358 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.358 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.358 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.358 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.358 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.358 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.358 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.359 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.359 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.359 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.359 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.359 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.359 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.359 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.359 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.360 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.360 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.360 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.360 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.360 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.360 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.360 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.360 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.360 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.361 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.361 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.361 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.361 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.361 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.361 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.361 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.361 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.362 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.362 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.362 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.362 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.362 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.362 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.362 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.362 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.363 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.363 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.363 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.363 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.363 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.363 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.363 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.363 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.364 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.364 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.364 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.364 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.364 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.364 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.364 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.364 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.364 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.365 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.365 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.365 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.365 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.365 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.365 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "nfs", 2024-06-09T22:31:35.365 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.366 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.366 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.366 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.366 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.366 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.366 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.366 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.366 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.367 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.367 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.367 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.367 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.367 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.367 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.367 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.367 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.368 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.368 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.368 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.368 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.368 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.368 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.368 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.368 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.368 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.369 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.369 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.369 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.369 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.369 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.369 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.369 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.369 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.369 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.370 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.370 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.370 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.370 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.370 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.370 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.370 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.371 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.371 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.371 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.371 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.371 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.371 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.371 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.371 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.372 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.372 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.372 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.372 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.372 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.372 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.372 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.372 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.372 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.373 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.373 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.373 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.373 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.373 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.373 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.373 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.373 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.374 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.374 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.374 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.374 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.374 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "orchestrator", 2024-06-09T22:31:35.375 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.375 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.375 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.375 INFO:teuthology.orchestra.run.smithi104.stdout: "fail_fs": { 2024-06-09T22:31:35.375 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "fail_fs", 2024-06-09T22:31:35.375 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.375 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.375 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.375 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.376 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.376 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.376 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.376 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-06-09T22:31:35.376 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.376 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.376 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.376 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.377 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.377 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.377 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.377 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.377 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.377 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.377 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.377 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.377 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.378 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.378 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.378 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.378 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.378 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.378 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.378 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.378 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.379 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.379 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.379 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.379 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.379 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.379 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.379 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.379 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.379 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.380 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.380 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.380 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.380 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.380 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.380 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.380 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.380 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.381 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.381 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.381 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.381 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.381 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.381 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.381 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.381 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.381 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.382 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.382 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.382 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.382 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.382 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.382 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.382 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.382 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.382 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.383 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.383 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.383 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.383 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.383 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.383 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.383 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.383 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.384 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.384 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.384 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.384 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.384 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.384 INFO:teuthology.orchestra.run.smithi104.stdout: "orchestrator": { 2024-06-09T22:31:35.384 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "orchestrator", 2024-06-09T22:31:35.384 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.385 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.385 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.385 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.385 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.385 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.385 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.385 INFO:teuthology.orchestra.run.smithi104.stdout: "cephadm", 2024-06-09T22:31:35.385 INFO:teuthology.orchestra.run.smithi104.stdout: "rook", 2024-06-09T22:31:35.385 INFO:teuthology.orchestra.run.smithi104.stdout: "test_orchestrator" 2024-06-09T22:31:35.386 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.386 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Orchestrator backend", 2024-06-09T22:31:35.386 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.386 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.386 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.386 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.386 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.387 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "osd_perf_query", 2024-06-09T22:31:35.387 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.387 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.387 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.387 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.387 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.387 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.387 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.388 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.388 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.388 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.388 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.388 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.388 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.388 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.388 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.388 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.389 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.389 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.389 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.389 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.389 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.389 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.389 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.389 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.389 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.390 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.390 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.390 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.390 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.390 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.390 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.390 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.390 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.391 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.391 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.391 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.391 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.391 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.391 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.391 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.391 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.392 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.392 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.392 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.392 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.392 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.392 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.392 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.392 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.392 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.393 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.393 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.393 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.393 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.393 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.393 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.393 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.393 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.393 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.394 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.394 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.394 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.394 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.394 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.394 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.394 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.394 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.395 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.395 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.395 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.395 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.395 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "osd_support", 2024-06-09T22:31:35.395 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.395 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.395 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.396 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.396 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.396 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.396 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.396 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.396 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.396 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.396 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.397 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.397 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.397 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.397 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.397 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.397 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.397 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.397 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.397 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.398 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.398 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.398 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.398 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.398 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.398 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.398 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.399 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.399 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.399 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.399 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.399 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.399 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.399 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.399 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.399 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.400 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.400 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.400 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.400 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.400 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.400 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.400 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.401 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.401 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.401 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.401 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.401 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.401 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.401 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.401 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.402 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.402 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.402 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.402 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.402 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.402 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.402 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.403 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.403 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.403 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.403 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.403 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.403 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.403 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.403 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.404 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.404 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.404 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.404 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.404 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.404 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "pg_autoscaler", 2024-06-09T22:31:35.405 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.405 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.405 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.405 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.405 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.405 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.405 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.405 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.405 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.406 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.406 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.406 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.406 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.406 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.406 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.406 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.407 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.407 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.407 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.407 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.407 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.407 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.407 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.408 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.408 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.408 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.408 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.408 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.408 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.408 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.408 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.409 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.409 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.409 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.409 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.409 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.409 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.409 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.410 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.410 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.410 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.410 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.410 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.410 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.410 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.410 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.411 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.411 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.411 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.411 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.411 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.411 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.411 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.411 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.412 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.412 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.412 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.412 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.412 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.412 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.412 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.413 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.413 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.413 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.413 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.413 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.413 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.413 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.413 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.414 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:35.414 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:35.414 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.414 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.414 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.414 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:35.414 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.414 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.415 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.415 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.415 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.415 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.415 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.415 INFO:teuthology.orchestra.run.smithi104.stdout: "threshold": { 2024-06-09T22:31:35.415 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "threshold", 2024-06-09T22:31:35.416 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:35.416 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.416 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.416 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "3.0", 2024-06-09T22:31:35.416 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1.0", 2024-06-09T22:31:35.416 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.416 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.416 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "scaling threshold", 2024-06-09T22:31:35.416 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-06-09T22:31:35.417 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.417 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.417 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.417 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.417 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "progress", 2024-06-09T22:31:35.418 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.418 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.418 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.418 INFO:teuthology.orchestra.run.smithi104.stdout: "allow_pg_recovery_event": { 2024-06-09T22:31:35.418 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "allow_pg_recovery_event", 2024-06-09T22:31:35.418 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.418 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.418 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.419 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.419 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.419 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.419 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.419 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "allow the module to show pg recovery progress", 2024-06-09T22:31:35.419 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.419 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.419 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.420 INFO:teuthology.orchestra.run.smithi104.stdout: "enabled": { 2024-06-09T22:31:35.420 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "enabled", 2024-06-09T22:31:35.420 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.420 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.420 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.420 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.420 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.420 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.421 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.421 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.421 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.421 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.421 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.421 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.421 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.422 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.422 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.422 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.422 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.422 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.422 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.422 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.422 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.423 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.423 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.423 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.423 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.423 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.423 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.423 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.423 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.423 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.423 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.424 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.424 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.424 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.424 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.424 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.424 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.424 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.425 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.425 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.425 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.425 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.425 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.425 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.425 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.425 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.425 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.426 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.426 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.426 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.426 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.426 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.426 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.426 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.426 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.427 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.427 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.427 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.427 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.427 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.427 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.427 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.428 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.428 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.428 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.428 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.428 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.428 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.428 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.428 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.429 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.429 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.429 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.429 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.429 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.429 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.429 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.429 INFO:teuthology.orchestra.run.smithi104.stdout: "max_completed_events": { 2024-06-09T22:31:35.430 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_completed_events", 2024-06-09T22:31:35.430 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.430 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.430 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.430 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "50", 2024-06-09T22:31:35.430 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.430 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.430 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.431 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "number of past completed events to remember", 2024-06-09T22:31:35.431 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.431 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.431 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.431 INFO:teuthology.orchestra.run.smithi104.stdout: "sleep_interval": { 2024-06-09T22:31:35.431 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "sleep_interval", 2024-06-09T22:31:35.431 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.432 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.432 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.432 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "5", 2024-06-09T22:31:35.432 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.432 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.432 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.432 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "how long the module is going to sleep", 2024-06-09T22:31:35.432 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.432 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.433 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.433 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.433 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.433 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "prometheus", 2024-06-09T22:31:35.433 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.433 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.434 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.434 INFO:teuthology.orchestra.run.smithi104.stdout: "cache": { 2024-06-09T22:31:35.434 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "cache", 2024-06-09T22:31:35.434 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.434 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.434 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.434 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.434 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.435 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.435 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.435 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.435 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.435 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.435 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.436 INFO:teuthology.orchestra.run.smithi104.stdout: "exclude_perf_counters": { 2024-06-09T22:31:35.436 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "exclude_perf_counters", 2024-06-09T22:31:35.436 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.436 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.436 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.436 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.436 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.436 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.436 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.437 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-06-09T22:31:35.437 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-06-09T22:31:35.437 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.437 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.437 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.437 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.437 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.438 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.438 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.438 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.438 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.438 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.438 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.438 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.439 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.439 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.439 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.439 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.439 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.439 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.439 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.439 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.440 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.440 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.440 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.440 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.440 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.440 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.440 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.441 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.441 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.441 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.441 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.441 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.441 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.441 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.441 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.441 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.442 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.442 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.442 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.442 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.442 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.442 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.442 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.442 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.443 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.443 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.443 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.443 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.443 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.443 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.443 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.444 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.444 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.444 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.444 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.444 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.444 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.444 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.444 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.445 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.445 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.445 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.445 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.445 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.445 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.445 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.445 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.445 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.446 INFO:teuthology.orchestra.run.smithi104.stdout: "rbd_stats_pools": { 2024-06-09T22:31:35.446 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rbd_stats_pools", 2024-06-09T22:31:35.446 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.446 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.446 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.446 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.446 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.447 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.447 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.447 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.447 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.447 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.447 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.447 INFO:teuthology.orchestra.run.smithi104.stdout: "rbd_stats_pools_refresh_interval": { 2024-06-09T22:31:35.447 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-06-09T22:31:35.448 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.448 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.448 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.448 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "300", 2024-06-09T22:31:35.448 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.448 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.448 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.448 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.448 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.448 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.449 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.449 INFO:teuthology.orchestra.run.smithi104.stdout: "scrape_interval": { 2024-06-09T22:31:35.449 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "scrape_interval", 2024-06-09T22:31:35.449 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:35.449 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.449 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.449 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "15.0", 2024-06-09T22:31:35.449 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.450 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.450 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.450 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.450 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.450 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.450 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.450 INFO:teuthology.orchestra.run.smithi104.stdout: "server_addr": { 2024-06-09T22:31:35.451 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_addr", 2024-06-09T22:31:35.451 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.451 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.451 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.451 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "::", 2024-06-09T22:31:35.451 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.451 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.451 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.451 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-06-09T22:31:35.452 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.452 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.452 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.452 INFO:teuthology.orchestra.run.smithi104.stdout: "server_port": { 2024-06-09T22:31:35.452 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_port", 2024-06-09T22:31:35.452 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.452 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.452 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.453 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "9283", 2024-06-09T22:31:35.453 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.453 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.453 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.453 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-06-09T22:31:35.453 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.453 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.453 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.453 INFO:teuthology.orchestra.run.smithi104.stdout: "stale_cache_strategy": { 2024-06-09T22:31:35.454 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "stale_cache_strategy", 2024-06-09T22:31:35.454 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.454 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.454 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.454 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "log", 2024-06-09T22:31:35.454 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.454 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.454 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.455 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.455 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.455 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.455 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.455 INFO:teuthology.orchestra.run.smithi104.stdout: "standby_behaviour": { 2024-06-09T22:31:35.455 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "standby_behaviour", 2024-06-09T22:31:35.456 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.456 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.456 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.456 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "default", 2024-06-09T22:31:35.456 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.456 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.456 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.456 INFO:teuthology.orchestra.run.smithi104.stdout: "default", 2024-06-09T22:31:35.456 INFO:teuthology.orchestra.run.smithi104.stdout: "error" 2024-06-09T22:31:35.457 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.457 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.457 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.457 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.457 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.457 INFO:teuthology.orchestra.run.smithi104.stdout: "standby_error_status_code": { 2024-06-09T22:31:35.457 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "standby_error_status_code", 2024-06-09T22:31:35.457 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.458 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.458 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.458 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "500", 2024-06-09T22:31:35.458 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "400", 2024-06-09T22:31:35.458 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "599", 2024-06-09T22:31:35.458 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.458 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.459 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.459 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.459 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.459 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.459 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.459 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rbd_support", 2024-06-09T22:31:35.459 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.460 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.460 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.460 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.460 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.460 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.460 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.460 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.460 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.460 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.461 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.461 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.461 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.461 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.461 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.461 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.461 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.461 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.461 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.462 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.462 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.462 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.462 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.462 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.462 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.462 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.463 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.463 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.463 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.463 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.463 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.463 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.463 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.463 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.464 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.464 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.464 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.464 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.464 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.464 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.464 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.464 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.465 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.465 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.465 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.465 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.465 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.465 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.465 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.465 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.465 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.466 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.466 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.466 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.466 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.466 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.466 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.466 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.467 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.467 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.467 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.467 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.467 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.467 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.467 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.467 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.468 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.468 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.468 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.468 INFO:teuthology.orchestra.run.smithi104.stdout: "max_concurrent_snap_create": { 2024-06-09T22:31:35.468 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_concurrent_snap_create", 2024-06-09T22:31:35.468 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.468 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.469 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.469 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10", 2024-06-09T22:31:35.469 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.469 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.469 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.469 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.469 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.469 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.470 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.470 INFO:teuthology.orchestra.run.smithi104.stdout: "mirror_snapshot_schedule": { 2024-06-09T22:31:35.470 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "mirror_snapshot_schedule", 2024-06-09T22:31:35.470 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.470 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.470 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.470 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.471 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.471 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.471 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.471 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.471 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.471 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.471 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.472 INFO:teuthology.orchestra.run.smithi104.stdout: "trash_purge_schedule": { 2024-06-09T22:31:35.472 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "trash_purge_schedule", 2024-06-09T22:31:35.472 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.472 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.472 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.472 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.472 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.472 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.473 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.473 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.473 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.473 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.473 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.473 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.473 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.474 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "restful", 2024-06-09T22:31:35.474 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.474 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.474 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.474 INFO:teuthology.orchestra.run.smithi104.stdout: "enable_auth": { 2024-06-09T22:31:35.474 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "enable_auth", 2024-06-09T22:31:35.474 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.476 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.476 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.476 INFO:teuthology.orchestra.run.smithi104.stdout: "key_file": { 2024-06-09T22:31:35.476 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "key_file", 2024-06-09T22:31:35.476 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.476 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.476 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.477 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.477 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.477 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.477 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.477 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.477 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.477 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.477 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.478 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.478 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.478 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.478 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.478 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.478 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.478 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.478 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.479 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.479 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.479 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.479 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.479 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.479 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.479 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.479 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.480 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.480 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.480 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.480 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.480 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.480 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.480 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.480 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.486 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.486 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.486 INFO:teuthology.orchestra.run.smithi104.stdout: "server_addr": { 2024-06-09T22:31:35.486 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_addr", 2024-06-09T22:31:35.486 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.486 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.486 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: "server_port": { 2024-06-09T22:31:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "server_port", 2024-06-09T22:31:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rgw", 2024-06-09T22:31:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rook", 2024-06-09T22:31:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: "storage_class": { 2024-06-09T22:31:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "storage_class", 2024-06-09T22:31:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "local", 2024-06-09T22:31:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-06-09T22:31:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "selftest", 2024-06-09T22:31:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.514 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.514 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.514 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.514 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.514 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.514 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.514 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.515 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.515 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.515 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.515 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.515 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.515 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.515 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.515 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.515 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.516 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.516 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.516 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.516 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.516 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.516 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.516 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.516 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.517 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.517 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.517 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.517 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.517 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.517 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.517 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.517 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.518 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.518 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.518 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.518 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.518 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.518 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.518 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.518 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.519 INFO:teuthology.orchestra.run.smithi104.stdout: "roption1": { 2024-06-09T22:31:35.519 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "roption1", 2024-06-09T22:31:35.519 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.519 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.519 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.519 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.519 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.520 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.520 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.520 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.520 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.520 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.520 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.520 INFO:teuthology.orchestra.run.smithi104.stdout: "roption2": { 2024-06-09T22:31:35.520 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "roption2", 2024-06-09T22:31:35.521 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.521 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.521 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.521 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "xyz", 2024-06-09T22:31:35.521 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.521 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.521 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.521 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.522 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.522 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.522 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.522 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption1": { 2024-06-09T22:31:35.522 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption1", 2024-06-09T22:31:35.522 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.522 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.523 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.523 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.523 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.523 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.523 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.523 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.523 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.523 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.524 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.524 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption2": { 2024-06-09T22:31:35.524 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption2", 2024-06-09T22:31:35.524 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.524 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.524 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.524 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.525 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.525 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.525 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.525 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.525 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.525 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.525 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.525 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption3": { 2024-06-09T22:31:35.526 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption3", 2024-06-09T22:31:35.526 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "float", 2024-06-09T22:31:35.526 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.526 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.526 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.526 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.526 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.527 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.527 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.527 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.527 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.527 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.527 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption4": { 2024-06-09T22:31:35.527 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption4", 2024-06-09T22:31:35.528 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.528 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.528 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.528 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.528 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.528 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.528 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.528 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.528 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.529 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.529 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.529 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption5": { 2024-06-09T22:31:35.529 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption5", 2024-06-09T22:31:35.529 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.529 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.529 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.529 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.530 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.530 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.530 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.530 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.530 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.530 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.530 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.531 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption6": { 2024-06-09T22:31:35.531 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption6", 2024-06-09T22:31:35.531 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.531 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.531 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.531 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.531 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.531 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.532 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.532 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.532 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.532 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.532 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.532 INFO:teuthology.orchestra.run.smithi104.stdout: "rwoption7": { 2024-06-09T22:31:35.532 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rwoption7", 2024-06-09T22:31:35.532 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.533 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.533 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.533 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.533 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "1", 2024-06-09T22:31:35.533 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "42", 2024-06-09T22:31:35.533 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.533 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.534 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.534 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.534 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.534 INFO:teuthology.orchestra.run.smithi104.stdout: "testkey": { 2024-06-09T22:31:35.534 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "testkey", 2024-06-09T22:31:35.534 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.534 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.535 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.535 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.535 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.535 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.535 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.535 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.535 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.535 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.536 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.536 INFO:teuthology.orchestra.run.smithi104.stdout: "testlkey": { 2024-06-09T22:31:35.536 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "testlkey", 2024-06-09T22:31:35.536 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.536 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.536 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.536 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.537 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.537 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.537 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.537 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.537 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.537 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.537 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.538 INFO:teuthology.orchestra.run.smithi104.stdout: "testnewline": { 2024-06-09T22:31:35.538 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "testnewline", 2024-06-09T22:31:35.538 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.538 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.538 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.538 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.538 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.538 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.539 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.539 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.539 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.539 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.539 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.539 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.539 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.540 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "snap_schedule", 2024-06-09T22:31:35.540 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.540 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.540 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.540 INFO:teuthology.orchestra.run.smithi104.stdout: "allow_m_granularity": { 2024-06-09T22:31:35.540 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "allow_m_granularity", 2024-06-09T22:31:35.540 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.540 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.541 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.541 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.541 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.541 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.541 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.541 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "allow minute scheduled snapshots", 2024-06-09T22:31:35.541 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.541 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.541 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.542 INFO:teuthology.orchestra.run.smithi104.stdout: "dump_on_update": { 2024-06-09T22:31:35.542 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "dump_on_update", 2024-06-09T22:31:35.542 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.542 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.542 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.542 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.542 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.543 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.543 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.543 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "dump database to debug log on update", 2024-06-09T22:31:35.543 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.543 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.543 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.543 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.544 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.544 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.544 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.544 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.544 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.544 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.544 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.544 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.544 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.545 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.545 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.545 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.545 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.545 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.545 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.545 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.545 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.545 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.546 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.546 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.546 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.546 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.546 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.546 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.547 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.547 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.547 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.547 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.547 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.547 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.547 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.547 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.548 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.548 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.548 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.548 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.548 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.548 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.548 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.548 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.548 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.549 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.549 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.549 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.549 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.549 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.549 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.549 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.549 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.550 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.550 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.550 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.550 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.550 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.550 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.551 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.551 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.551 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.551 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.551 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.551 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.551 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.551 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.552 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.552 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.552 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.552 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.552 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "stats", 2024-06-09T22:31:35.552 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.552 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.553 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.553 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.553 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.553 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.553 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.553 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.553 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.553 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.553 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.554 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.554 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.554 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.554 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.554 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.554 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.554 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.554 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.555 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.555 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.555 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.555 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.555 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.555 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.555 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.555 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.556 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.556 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.556 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.556 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.556 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.556 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.556 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.556 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.557 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.557 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.557 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.557 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.557 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.557 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.557 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.557 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.558 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.558 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.558 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.558 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.558 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.558 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.558 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.558 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.559 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.559 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.559 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.559 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.559 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.559 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.559 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.560 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.560 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.560 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.560 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.560 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.560 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.560 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.560 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.560 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.561 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.561 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.561 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.561 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.561 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "status", 2024-06-09T22:31:35.561 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.562 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.562 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.562 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.562 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.562 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.562 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.562 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.563 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.563 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.563 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.563 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.563 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.563 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.563 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.563 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.563 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.564 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.564 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.564 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.564 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.564 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.564 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.564 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.565 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.565 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.565 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.565 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.565 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.565 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.565 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.565 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.565 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.566 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.566 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.566 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.566 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.566 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.566 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.566 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.567 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.567 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.567 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.567 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.567 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.567 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.567 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.567 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.568 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.568 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.568 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.568 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.568 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.568 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.568 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.568 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.569 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.569 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.569 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.569 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.569 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.569 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.569 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.569 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.570 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.570 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.570 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.570 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.570 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.570 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.570 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.571 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "telegraf", 2024-06-09T22:31:35.571 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.571 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.571 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.571 INFO:teuthology.orchestra.run.smithi104.stdout: "address": { 2024-06-09T22:31:35.571 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "address", 2024-06-09T22:31:35.571 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.572 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.572 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.572 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-06-09T22:31:35.572 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.572 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.572 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.572 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.572 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.572 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.573 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.573 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:35.573 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:35.573 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.573 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.573 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.573 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "15", 2024-06-09T22:31:35.574 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.574 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.574 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.574 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.574 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.574 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.574 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.575 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.575 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.575 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.575 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.575 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.575 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.575 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.575 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.575 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.576 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.576 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.576 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.576 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.576 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.576 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.576 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.576 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.577 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.577 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.577 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.577 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.577 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.577 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.577 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.578 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.578 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.578 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.578 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.578 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.578 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.578 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.578 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.579 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.579 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.579 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.579 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.579 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.579 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.579 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.579 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.580 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.580 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.580 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.580 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.580 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.580 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.580 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.580 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.580 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.581 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.581 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.581 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.581 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.581 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.581 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.581 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.581 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.582 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.582 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.582 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.582 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.582 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.582 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.582 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.582 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.583 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.583 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.583 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.583 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "telemetry", 2024-06-09T22:31:35.583 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.583 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.584 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.584 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_basic": { 2024-06-09T22:31:35.584 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_basic", 2024-06-09T22:31:35.584 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.584 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.584 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.584 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.584 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.584 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.585 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.585 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share basic cluster information (size, version)", 2024-06-09T22:31:35.585 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.585 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.585 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.585 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_crash": { 2024-06-09T22:31:35.585 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_crash", 2024-06-09T22:31:35.586 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.586 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.586 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.586 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.586 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.586 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.586 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.586 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-06-09T22:31:35.587 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.587 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.587 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.587 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_device": { 2024-06-09T22:31:35.587 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_device", 2024-06-09T22:31:35.587 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.587 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.587 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.588 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.588 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.588 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.588 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.588 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-06-09T22:31:35.588 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.588 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.588 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.589 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_ident": { 2024-06-09T22:31:35.589 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_ident", 2024-06-09T22:31:35.589 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.589 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.589 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.589 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.589 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.589 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.589 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.590 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-06-09T22:31:35.590 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.590 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.590 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.590 INFO:teuthology.orchestra.run.smithi104.stdout: "channel_perf": { 2024-06-09T22:31:35.590 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "channel_perf", 2024-06-09T22:31:35.590 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.590 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.591 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.591 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.591 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.591 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.591 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.591 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Share various performance metrics of a cluster", 2024-06-09T22:31:35.591 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.591 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.591 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.592 INFO:teuthology.orchestra.run.smithi104.stdout: "contact": { 2024-06-09T22:31:35.592 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "contact", 2024-06-09T22:31:35.592 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.592 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.592 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.592 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.592 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.592 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.592 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.593 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.593 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.593 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.593 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.593 INFO:teuthology.orchestra.run.smithi104.stdout: "description": { 2024-06-09T22:31:35.593 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "description", 2024-06-09T22:31:35.593 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.593 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.594 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.594 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.594 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.594 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.594 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.594 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.594 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.594 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.594 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.595 INFO:teuthology.orchestra.run.smithi104.stdout: "device_url": { 2024-06-09T22:31:35.595 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "device_url", 2024-06-09T22:31:35.595 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.595 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.595 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.595 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-06-09T22:31:35.595 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.595 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.596 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.596 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.596 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.596 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.596 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.596 INFO:teuthology.orchestra.run.smithi104.stdout: "enabled": { 2024-06-09T22:31:35.596 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "enabled", 2024-06-09T22:31:35.596 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.597 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.597 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.597 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.597 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.597 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.597 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.597 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.597 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.598 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.598 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.598 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:35.598 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:35.598 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.598 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.598 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.599 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "24", 2024-06-09T22:31:35.599 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "8", 2024-06-09T22:31:35.599 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.599 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.599 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.599 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.599 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.599 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.600 INFO:teuthology.orchestra.run.smithi104.stdout: "last_opt_revision": { 2024-06-09T22:31:35.600 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "last_opt_revision", 2024-06-09T22:31:35.600 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.600 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.600 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.600 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "1", 2024-06-09T22:31:35.600 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.600 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.600 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.601 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.601 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.601 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.601 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.601 INFO:teuthology.orchestra.run.smithi104.stdout: "leaderboard": { 2024-06-09T22:31:35.601 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "leaderboard", 2024-06-09T22:31:35.601 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.601 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.602 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.602 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.602 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.602 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.602 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.602 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.602 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.602 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.602 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.603 INFO:teuthology.orchestra.run.smithi104.stdout: "leaderboard_description": { 2024-06-09T22:31:35.603 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "leaderboard_description", 2024-06-09T22:31:35.603 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.603 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.603 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.603 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.603 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.603 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.604 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.604 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.604 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.604 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.604 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.604 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.604 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.604 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.605 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.605 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.605 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.605 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.605 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.605 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.605 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.605 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.606 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.606 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.606 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.606 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.606 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.606 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.606 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.606 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.607 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.607 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.607 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.607 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.607 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.607 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.607 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.608 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.608 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.608 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.608 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.608 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.608 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.608 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.608 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.609 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.609 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.609 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.609 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.609 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.609 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.609 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.609 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.610 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.610 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.610 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.610 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.610 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.610 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.610 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.610 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.611 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.611 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.611 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.611 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.611 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.611 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.611 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.612 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.612 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.612 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.612 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.612 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.612 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.612 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.612 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.613 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.613 INFO:teuthology.orchestra.run.smithi104.stdout: "organization": { 2024-06-09T22:31:35.613 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "organization", 2024-06-09T22:31:35.613 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.613 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.613 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.613 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.614 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.614 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.614 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.614 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.614 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.614 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.614 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.615 INFO:teuthology.orchestra.run.smithi104.stdout: "proxy": { 2024-06-09T22:31:35.615 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "proxy", 2024-06-09T22:31:35.615 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.615 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.615 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.615 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.615 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.615 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.616 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.616 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.616 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.616 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.616 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.616 INFO:teuthology.orchestra.run.smithi104.stdout: "url": { 2024-06-09T22:31:35.616 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "url", 2024-06-09T22:31:35.617 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.617 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.617 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.617 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-06-09T22:31:35.617 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.617 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.617 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.617 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.618 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.618 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.618 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.618 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.618 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.619 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "test_orchestrator", 2024-06-09T22:31:35.619 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.619 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.619 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.619 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.619 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.619 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.619 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.620 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.620 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.620 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.620 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.620 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.620 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.620 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.621 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.621 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.621 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.621 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.621 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.621 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.621 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.621 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.622 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.622 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.622 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.622 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.622 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.622 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.622 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.623 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.623 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.623 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.623 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.623 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.623 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.623 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.624 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.624 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.624 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.624 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.624 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.624 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.624 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.625 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.625 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.625 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.625 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.625 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.625 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.625 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.625 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.625 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.626 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.626 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.626 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.626 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.626 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.626 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.626 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.627 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.627 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.627 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.627 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.627 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.627 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.627 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.628 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.628 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.628 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.628 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.628 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.628 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "volumes", 2024-06-09T22:31:35.628 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.629 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.629 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.629 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.629 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.629 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.629 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.629 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.629 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.630 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.630 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.630 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.630 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.630 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.630 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.630 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.630 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.631 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.631 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.631 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.631 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.631 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.631 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.631 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.632 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.632 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.632 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.632 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.632 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.632 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.632 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.632 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.633 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.633 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.633 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.633 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.633 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.633 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.634 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.634 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.634 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.634 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.634 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.634 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.634 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.634 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.635 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.635 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.635 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.635 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.635 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.635 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.635 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.635 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.636 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.636 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.636 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.636 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.636 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.636 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.636 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: "max_concurrent_clones": { 2024-06-09T22:31:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "max_concurrent_clones", 2024-06-09T22:31:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "4", 2024-06-09T22:31:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Number of asynchronous cloner threads", 2024-06-09T22:31:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: "periodic_async_work": { 2024-06-09T22:31:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "periodic_async_work", 2024-06-09T22:31:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Periodically check for async work", 2024-06-09T22:31:35.641 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.641 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.641 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.641 INFO:teuthology.orchestra.run.smithi104.stdout: "snapshot_clone_delay": { 2024-06-09T22:31:35.641 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "snapshot_clone_delay", 2024-06-09T22:31:35.641 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.641 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.641 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.641 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "0", 2024-06-09T22:31:35.642 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.642 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.642 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.642 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-06-09T22:31:35.642 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.642 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.642 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.643 INFO:teuthology.orchestra.run.smithi104.stdout: "snapshot_clone_no_wait": { 2024-06-09T22:31:35.643 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "snapshot_clone_no_wait", 2024-06-09T22:31:35.643 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.643 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.643 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.643 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "True", 2024-06-09T22:31:35.643 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.643 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.643 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.644 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-06-09T22:31:35.644 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.644 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.644 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.644 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.644 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.644 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "zabbix", 2024-06-09T22:31:35.645 INFO:teuthology.orchestra.run.smithi104.stdout: "can_run": true, 2024-06-09T22:31:35.645 INFO:teuthology.orchestra.run.smithi104.stdout: "error_string": "", 2024-06-09T22:31:35.645 INFO:teuthology.orchestra.run.smithi104.stdout: "module_options": { 2024-06-09T22:31:35.645 INFO:teuthology.orchestra.run.smithi104.stdout: "discovery_interval": { 2024-06-09T22:31:35.645 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "discovery_interval", 2024-06-09T22:31:35.645 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "uint", 2024-06-09T22:31:35.645 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.645 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.646 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "100", 2024-06-09T22:31:35.646 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.646 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.646 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.646 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.646 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.646 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.646 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.647 INFO:teuthology.orchestra.run.smithi104.stdout: "identifier": { 2024-06-09T22:31:35.647 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "identifier", 2024-06-09T22:31:35.647 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.647 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.647 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.647 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.648 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.648 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.648 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.648 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.648 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.648 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.648 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.649 INFO:teuthology.orchestra.run.smithi104.stdout: "interval": { 2024-06-09T22:31:35.649 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "interval", 2024-06-09T22:31:35.649 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "secs", 2024-06-09T22:31:35.649 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.649 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.649 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "60", 2024-06-09T22:31:35.649 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.650 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.650 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.650 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.650 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.650 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.650 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.651 INFO:teuthology.orchestra.run.smithi104.stdout: "log_level": { 2024-06-09T22:31:35.651 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_level", 2024-06-09T22:31:35.651 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.651 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.651 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.651 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.651 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.651 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.652 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.652 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.652 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.652 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.652 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.652 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.652 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.652 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.653 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.653 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.653 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.653 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.653 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster": { 2024-06-09T22:31:35.653 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster", 2024-06-09T22:31:35.654 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.654 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.654 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.654 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.654 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.654 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.654 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.654 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.655 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.655 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.655 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.655 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_cluster_level": { 2024-06-09T22:31:35.655 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_cluster_level", 2024-06-09T22:31:35.655 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.655 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.656 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.656 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "info", 2024-06-09T22:31:35.656 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.656 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.656 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [ 2024-06-09T22:31:35.656 INFO:teuthology.orchestra.run.smithi104.stdout: "", 2024-06-09T22:31:35.656 INFO:teuthology.orchestra.run.smithi104.stdout: "critical", 2024-06-09T22:31:35.656 INFO:teuthology.orchestra.run.smithi104.stdout: "debug", 2024-06-09T22:31:35.657 INFO:teuthology.orchestra.run.smithi104.stdout: "error", 2024-06-09T22:31:35.657 INFO:teuthology.orchestra.run.smithi104.stdout: "info", 2024-06-09T22:31:35.657 INFO:teuthology.orchestra.run.smithi104.stdout: "warning" 2024-06-09T22:31:35.657 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.657 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.657 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.657 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.657 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.658 INFO:teuthology.orchestra.run.smithi104.stdout: "log_to_file": { 2024-06-09T22:31:35.658 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "log_to_file", 2024-06-09T22:31:35.658 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "bool", 2024-06-09T22:31:35.658 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.658 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 1, 2024-06-09T22:31:35.658 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "False", 2024-06-09T22:31:35.659 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.659 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.659 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.659 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.659 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.659 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.659 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.660 INFO:teuthology.orchestra.run.smithi104.stdout: "zabbix_host": { 2024-06-09T22:31:35.660 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "zabbix_host", 2024-06-09T22:31:35.660 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.660 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.660 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.660 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "", 2024-06-09T22:31:35.660 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.660 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.661 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.661 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.661 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.661 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.661 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.661 INFO:teuthology.orchestra.run.smithi104.stdout: "zabbix_port": { 2024-06-09T22:31:35.661 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "zabbix_port", 2024-06-09T22:31:35.662 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "int", 2024-06-09T22:31:35.662 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.662 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.662 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "10051", 2024-06-09T22:31:35.662 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.662 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.662 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.662 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.662 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.663 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.663 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.663 INFO:teuthology.orchestra.run.smithi104.stdout: "zabbix_sender": { 2024-06-09T22:31:35.663 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "zabbix_sender", 2024-06-09T22:31:35.663 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "str", 2024-06-09T22:31:35.663 INFO:teuthology.orchestra.run.smithi104.stdout: "level": "advanced", 2024-06-09T22:31:35.663 INFO:teuthology.orchestra.run.smithi104.stdout: "flags": 0, 2024-06-09T22:31:35.664 INFO:teuthology.orchestra.run.smithi104.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-06-09T22:31:35.664 INFO:teuthology.orchestra.run.smithi104.stdout: "min": "", 2024-06-09T22:31:35.664 INFO:teuthology.orchestra.run.smithi104.stdout: "max": "", 2024-06-09T22:31:35.664 INFO:teuthology.orchestra.run.smithi104.stdout: "enum_allowed": [], 2024-06-09T22:31:35.664 INFO:teuthology.orchestra.run.smithi104.stdout: "desc": "", 2024-06-09T22:31:35.664 INFO:teuthology.orchestra.run.smithi104.stdout: "long_desc": "", 2024-06-09T22:31:35.664 INFO:teuthology.orchestra.run.smithi104.stdout: "tags": [], 2024-06-09T22:31:35.664 INFO:teuthology.orchestra.run.smithi104.stdout: "see_also": [] 2024-06-09T22:31:35.665 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.665 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.665 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.665 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.665 INFO:teuthology.orchestra.run.smithi104.stdout: "services": { 2024-06-09T22:31:35.665 INFO:teuthology.orchestra.run.smithi104.stdout: "dashboard": "https://172.21.15.104:8443/" 2024-06-09T22:31:35.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.665 INFO:teuthology.orchestra.run.smithi104.stdout: "always_on_modules": { 2024-06-09T22:31:35.666 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus": [ 2024-06-09T22:31:35.666 INFO:teuthology.orchestra.run.smithi104.stdout: "balancer", 2024-06-09T22:31:35.666 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-06-09T22:31:35.666 INFO:teuthology.orchestra.run.smithi104.stdout: "devicehealth", 2024-06-09T22:31:35.666 INFO:teuthology.orchestra.run.smithi104.stdout: "orchestrator", 2024-06-09T22:31:35.666 INFO:teuthology.orchestra.run.smithi104.stdout: "pg_autoscaler", 2024-06-09T22:31:35.666 INFO:teuthology.orchestra.run.smithi104.stdout: "progress", 2024-06-09T22:31:35.667 INFO:teuthology.orchestra.run.smithi104.stdout: "rbd_support", 2024-06-09T22:31:35.667 INFO:teuthology.orchestra.run.smithi104.stdout: "status", 2024-06-09T22:31:35.667 INFO:teuthology.orchestra.run.smithi104.stdout: "telemetry", 2024-06-09T22:31:35.667 INFO:teuthology.orchestra.run.smithi104.stdout: "volumes" 2024-06-09T22:31:35.667 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.667 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific": [ 2024-06-09T22:31:35.667 INFO:teuthology.orchestra.run.smithi104.stdout: "balancer", 2024-06-09T22:31:35.667 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-06-09T22:31:35.667 INFO:teuthology.orchestra.run.smithi104.stdout: "devicehealth", 2024-06-09T22:31:35.668 INFO:teuthology.orchestra.run.smithi104.stdout: "orchestrator", 2024-06-09T22:31:35.668 INFO:teuthology.orchestra.run.smithi104.stdout: "pg_autoscaler", 2024-06-09T22:31:35.668 INFO:teuthology.orchestra.run.smithi104.stdout: "progress", 2024-06-09T22:31:35.668 INFO:teuthology.orchestra.run.smithi104.stdout: "rbd_support", 2024-06-09T22:31:35.668 INFO:teuthology.orchestra.run.smithi104.stdout: "status", 2024-06-09T22:31:35.668 INFO:teuthology.orchestra.run.smithi104.stdout: "telemetry", 2024-06-09T22:31:35.668 INFO:teuthology.orchestra.run.smithi104.stdout: "volumes" 2024-06-09T22:31:35.668 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.668 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy": [ 2024-06-09T22:31:35.669 INFO:teuthology.orchestra.run.smithi104.stdout: "balancer", 2024-06-09T22:31:35.669 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-06-09T22:31:35.669 INFO:teuthology.orchestra.run.smithi104.stdout: "devicehealth", 2024-06-09T22:31:35.669 INFO:teuthology.orchestra.run.smithi104.stdout: "orchestrator", 2024-06-09T22:31:35.669 INFO:teuthology.orchestra.run.smithi104.stdout: "pg_autoscaler", 2024-06-09T22:31:35.669 INFO:teuthology.orchestra.run.smithi104.stdout: "progress", 2024-06-09T22:31:35.669 INFO:teuthology.orchestra.run.smithi104.stdout: "rbd_support", 2024-06-09T22:31:35.669 INFO:teuthology.orchestra.run.smithi104.stdout: "status", 2024-06-09T22:31:35.669 INFO:teuthology.orchestra.run.smithi104.stdout: "telemetry", 2024-06-09T22:31:35.670 INFO:teuthology.orchestra.run.smithi104.stdout: "volumes" 2024-06-09T22:31:35.670 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.670 INFO:teuthology.orchestra.run.smithi104.stdout: "reef": [ 2024-06-09T22:31:35.670 INFO:teuthology.orchestra.run.smithi104.stdout: "balancer", 2024-06-09T22:31:35.670 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-06-09T22:31:35.670 INFO:teuthology.orchestra.run.smithi104.stdout: "devicehealth", 2024-06-09T22:31:35.670 INFO:teuthology.orchestra.run.smithi104.stdout: "orchestrator", 2024-06-09T22:31:35.670 INFO:teuthology.orchestra.run.smithi104.stdout: "pg_autoscaler", 2024-06-09T22:31:35.671 INFO:teuthology.orchestra.run.smithi104.stdout: "progress", 2024-06-09T22:31:35.671 INFO:teuthology.orchestra.run.smithi104.stdout: "rbd_support", 2024-06-09T22:31:35.671 INFO:teuthology.orchestra.run.smithi104.stdout: "status", 2024-06-09T22:31:35.671 INFO:teuthology.orchestra.run.smithi104.stdout: "telemetry", 2024-06-09T22:31:35.671 INFO:teuthology.orchestra.run.smithi104.stdout: "volumes" 2024-06-09T22:31:35.671 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-06-09T22:31:35.671 INFO:teuthology.orchestra.run.smithi104.stdout: "squid": [ 2024-06-09T22:31:35.671 INFO:teuthology.orchestra.run.smithi104.stdout: "balancer", 2024-06-09T22:31:35.671 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-06-09T22:31:35.672 INFO:teuthology.orchestra.run.smithi104.stdout: "devicehealth", 2024-06-09T22:31:35.672 INFO:teuthology.orchestra.run.smithi104.stdout: "orchestrator", 2024-06-09T22:31:35.672 INFO:teuthology.orchestra.run.smithi104.stdout: "pg_autoscaler", 2024-06-09T22:31:35.672 INFO:teuthology.orchestra.run.smithi104.stdout: "progress", 2024-06-09T22:31:35.672 INFO:teuthology.orchestra.run.smithi104.stdout: "rbd_support", 2024-06-09T22:31:35.672 INFO:teuthology.orchestra.run.smithi104.stdout: "status", 2024-06-09T22:31:35.672 INFO:teuthology.orchestra.run.smithi104.stdout: "telemetry", 2024-06-09T22:31:35.672 INFO:teuthology.orchestra.run.smithi104.stdout: "volumes" 2024-06-09T22:31:35.673 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-06-09T22:31:35.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.673 INFO:teuthology.orchestra.run.smithi104.stdout: "last_failure_osd_epoch": 100, 2024-06-09T22:31:35.673 INFO:teuthology.orchestra.run.smithi104.stdout: "active_clients": [ 2024-06-09T22:31:35.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.673 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "devicehealth", 2024-06-09T22:31:35.673 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-06-09T22:31:35.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.674 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-06-09T22:31:35.674 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:0", 2024-06-09T22:31:35.674 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1596553372 2024-06-09T22:31:35.674 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.674 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-06-09T22:31:35.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.674 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "libcephsqlite", 2024-06-09T22:31:35.675 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-06-09T22:31:35.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.675 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-06-09T22:31:35.675 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:0", 2024-06-09T22:31:35.675 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3389010484 2024-06-09T22:31:35.675 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.675 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-06-09T22:31:35.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.676 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "rbd_support", 2024-06-09T22:31:35.676 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-06-09T22:31:35.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.676 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-06-09T22:31:35.676 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:0", 2024-06-09T22:31:35.676 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3538688888 2024-06-09T22:31:35.676 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.676 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-06-09T22:31:35.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.677 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "snap_schedule", 2024-06-09T22:31:35.677 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-06-09T22:31:35.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.677 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-06-09T22:31:35.677 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:0", 2024-06-09T22:31:35.677 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 4140041997 2024-06-09T22:31:35.678 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.678 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-06-09T22:31:35.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.678 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "snap_schedule", 2024-06-09T22:31:35.678 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-06-09T22:31:35.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.678 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-06-09T22:31:35.679 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:0", 2024-06-09T22:31:35.679 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3732827340 2024-06-09T22:31:35.679 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.679 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-06-09T22:31:35.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:35.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.679 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "volumes", 2024-06-09T22:31:35.679 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-06-09T22:31:35.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-06-09T22:31:35.680 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-06-09T22:31:35.680 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:0", 2024-06-09T22:31:35.680 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3301736471 2024-06-09T22:31:35.680 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.680 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-06-09T22:31:35.680 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:35.680 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-06-09T22:31:35.680 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:31:35.682 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:33 smithi153 ceph-mon[36599]: pgmap v328: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:35.682 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:33 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/4225330224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-09T22:31:35.682 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:33 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/474200589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-06-09T22:31:35.682 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:33 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1575185266' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-06-09T22:31:35.683 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:33 smithi194 ceph-mon[36856]: pgmap v328: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:35.683 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:33 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/4225330224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-09T22:31:35.683 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:33 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/474200589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-06-09T22:31:35.683 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:33 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1575185266' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-06-09T22:31:35.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.488+0000 7fd66a86b640 1 -- 172.21.15.104:0/1575185266 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd64c0a2d00 msgr2=0x7fd64c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:35.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.488+0000 7fd66a86b640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd64c0a2d00 0x7fd64c0a51c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd654009230 tx=0x7fd6540058d0 comp rx=0 tx=0).stop 2024-06-09T22:31:35.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.488+0000 7fd66a86b640 1 -- 172.21.15.104:0/1575185266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd66410bf70 msgr2=0x7fd66419cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:31:35.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.488+0000 7fd66a86b640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd66410bf70 0x7fd66419cb30 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00eaa0 tx=0x7fd65c00ef60 comp rx=0 tx=0).stop 2024-06-09T22:31:35.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.488+0000 7fd66a86b640 1 -- 172.21.15.104:0/1575185266 shutdown_connections 2024-06-09T22:31:35.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.488+0000 7fd66a86b640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fd64c0a2d00 0x7fd64c0a51c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:35.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.488+0000 7fd66a86b640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd66410bf70 0x7fd66419cb30 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:35.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.488+0000 7fd66a86b640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd664103550 0x7fd6641987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:35.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.488+0000 7fd66a86b640 1 --2- 172.21.15.104:0/1575185266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd664102c30 0x7fd664198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:31:35.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.488+0000 7fd66a86b640 1 -- 172.21.15.104:0/1575185266 >> 172.21.15.104:0/1575185266 conn(0x7fd6640779a0 msgr2=0x7fd6640fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:31:35.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.488+0000 7fd66a86b640 1 -- 172.21.15.104:0/1575185266 shutdown_connections 2024-06-09T22:31:35.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:31:33.488+0000 7fd66a86b640 1 -- 172.21.15.104:0/1575185266 wait complete. 2024-06-09T22:31:35.691 DEBUG:tasks.check_counter:Getting stats from a 2024-06-09T22:31:35.691 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump 2024-06-09T22:31:35.692 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:33 smithi104 ceph-mon[32389]: pgmap v328: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:35.692 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:33 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/4225330224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-09T22:31:35.692 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:33 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/474200589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-06-09T22:31:35.692 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:33 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1575185266' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-06-09T22:31:35.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:35 smithi194 ceph-mon[36856]: pgmap v329: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:35.945 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:31:35.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:35 smithi153 ceph-mon[36599]: pgmap v329: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:36.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:35 smithi104 ceph-mon[32389]: pgmap v329: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:37.657 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:37 smithi104 ceph-mon[32389]: pgmap v330: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:37.785 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:31:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-0": { 2024-06-09T22:31:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 960, 2024-06-09T22:31:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 960, 2024-06-09T22:31:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 4507295, 2024-06-09T22:31:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 304457, 2024-06-09T22:31:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 7, 2024-06-09T22:31:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 3, 2024-06-09T22:31:37.787 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.090715192, 2024-06-09T22:31:37.787 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.029888254, 2024-06-09T22:31:37.787 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.024473595, 2024-06-09T22:31:37.787 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.028000838, 2024-06-09T22:31:37.787 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:37.787 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 959, 2024-06-09T22:31:37.787 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.036006168, 2024-06-09T22:31:37.788 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000037545 2024-06-09T22:31:37.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.788 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:37.788 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.788 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.788 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.789 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-06-09T22:31:37.789 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 448 2024-06-09T22:31:37.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.789 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-1": { 2024-06-09T22:31:37.789 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 1594, 2024-06-09T22:31:37.789 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 1579, 2024-06-09T22:31:37.789 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 22820190, 2024-06-09T22:31:37.790 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 572427, 2024-06-09T22:31:37.790 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 5, 2024-06-09T22:31:37.790 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 4, 2024-06-09T22:31:37.790 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.143495868, 2024-06-09T22:31:37.790 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.039498309, 2024-06-09T22:31:37.790 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.069719989, 2024-06-09T22:31:37.790 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.036263817, 2024-06-09T22:31:37.791 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:37.791 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 1578, 2024-06-09T22:31:37.791 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.044225640, 2024-06-09T22:31:37.791 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000028026 2024-06-09T22:31:37.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.791 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:37.791 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.791 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.792 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.792 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 1511664, 2024-06-09T22:31:37.792 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 152928 2024-06-09T22:31:37.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.792 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-2": { 2024-06-09T22:31:37.792 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 352, 2024-06-09T22:31:37.792 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 536, 2024-06-09T22:31:37.793 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 9933459, 2024-06-09T22:31:37.793 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 583455, 2024-06-09T22:31:37.793 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 7, 2024-06-09T22:31:37.793 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 5, 2024-06-09T22:31:37.793 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.047756769, 2024-06-09T22:31:37.793 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.017517898, 2024-06-09T22:31:37.793 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.016939133, 2024-06-09T22:31:37.793 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.008748630, 2024-06-09T22:31:37.794 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:37.794 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 533, 2024-06-09T22:31:37.794 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.023671054, 2024-06-09T22:31:37.794 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000044410 2024-06-09T22:31:37.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.794 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:37.794 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.794 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.795 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.795 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-09T22:31:37.795 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 472144 2024-06-09T22:31:37.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.795 INFO:teuthology.orchestra.run.smithi104.stdout: "cct": { 2024-06-09T22:31:37.795 INFO:teuthology.orchestra.run.smithi104.stdout: "total_workers": 1, 2024-06-09T22:31:37.795 INFO:teuthology.orchestra.run.smithi104.stdout: "unhealthy_workers": 0 2024-06-09T22:31:37.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.796 INFO:teuthology.orchestra.run.smithi104.stdout: "finisher-MDSRank": { 2024-06-09T22:31:37.796 INFO:teuthology.orchestra.run.smithi104.stdout: "queue_len": 0, 2024-06-09T22:31:37.796 INFO:teuthology.orchestra.run.smithi104.stdout: "complete_latency": { 2024-06-09T22:31:37.796 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 4445, 2024-06-09T22:31:37.796 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.109606056, 2024-06-09T22:31:37.796 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000024658 2024-06-09T22:31:37.796 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "finisher-PurgeQueue": { 2024-06-09T22:31:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "queue_len": 0, 2024-06-09T22:31:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "complete_latency": { 2024-06-09T22:31:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-06-09T22:31:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "request": 0, 2024-06-09T22:31:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "reply": 0, 2024-06-09T22:31:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "reply_latency": { 2024-06-09T22:31:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "slow_reply": 0, 2024-06-09T22:31:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "forward": 0, 2024-06-09T22:31:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_fetch_complete": 0, 2024-06-09T22:31:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_fetch_keys": 0, 2024-06-09T22:31:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_commit": 0, 2024-06-09T22:31:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_split": 0, 2024-06-09T22:31:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_merge": 0, 2024-06-09T22:31:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes": 4738, 2024-06-09T22:31:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_top": 12, 2024-06-09T22:31:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_bottom": 4726, 2024-06-09T22:31:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_pin_tail": 0, 2024-06-09T22:31:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_pinned": 4721, 2024-06-09T22:31:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_expired": 0, 2024-06-09T22:31:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_with_caps": 0, 2024-06-09T22:31:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": 0, 2024-06-09T22:31:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "subtrees": 4, 2024-06-09T22:31:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse": 0, 2024-06-09T22:31:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_hit": 0, 2024-06-09T22:31:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_forward": 0, 2024-06-09T22:31:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_discover": 0, 2024-06-09T22:31:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_dir_fetch": 0, 2024-06-09T22:31:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_remote_ino": 0, 2024-06-09T22:31:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_lock": 0, 2024-06-09T22:31:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "load_cent": 0, 2024-06-09T22:31:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "q": 0, 2024-06-09T22:31:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "exported": 0, 2024-06-09T22:31:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "exported_inodes": 0, 2024-06-09T22:31:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "imported": 0, 2024-06-09T22:31:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "imported_inodes": 0, 2024-06-09T22:31:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_dir_fetch": 0, 2024-06-09T22:31:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_backtrace_fetch": 0, 2024-06-09T22:31:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_peer_discover": 0, 2024-06-09T22:31:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rfiles": 0, 2024-06-09T22:31:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rbytes": 0, 2024-06-09T22:31:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rsnaps": 0, 2024-06-09T22:31:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_backtrace_fetch": 0, 2024-06-09T22:31:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_set_tag": 0, 2024-06-09T22:31:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_backtrace_repaired": 0, 2024-06-09T22:31:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_inotable_repaired": 0, 2024-06-09T22:31:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dir_inodes": 0, 2024-06-09T22:31:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dir_base_inodes": 0, 2024-06-09T22:31:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dirfrag_rstats": 0, 2024-06-09T22:31:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_file_inodes": 0, 2024-06-09T22:31:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_inode_file_caps": 0, 2024-06-09T22:31:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_revoke": 0, 2024-06-09T22:31:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_grant": 0, 2024-06-09T22:31:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_trunc": 0, 2024-06-09T22:31:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-09T22:31:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-09T22:31:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_caps": 0, 2024-06-09T22:31:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_caps_dirty": 0, 2024-06-09T22:31:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_cap_release": 0, 2024-06-09T22:31:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "process_request_cap_release": 0 2024-06-09T22:31:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_cache": { 2024-06-09T22:31:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_update": 0, 2024-06-09T22:31:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_update_receipt": 0, 2024-06-09T22:31:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_try_discover": 0, 2024-06-09T22:31:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_send_discover": 0, 2024-06-09T22:31:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_handle_discover": 0, 2024-06-09T22:31:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays": 0, 2024-06-09T22:31:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays_delayed": 0, 2024-06-09T22:31:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays_enqueuing": 0, 2024-06-09T22:31:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_created": 0, 2024-06-09T22:31:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_enqueued": 0, 2024-06-09T22:31:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_reintegrated": 0, 2024-06-09T22:31:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_migrated": 0, 2024-06-09T22:31:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_processing": 0, 2024-06-09T22:31:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_enqueued": 0, 2024-06-09T22:31:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_prioritized": 0, 2024-06-09T22:31:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "recovery_started": 0, 2024-06-09T22:31:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "recovery_completed": 0, 2024-06-09T22:31:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_quiesce_path": 0, 2024-06-09T22:31:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_quiesce_inode": 0, 2024-06-09T22:31:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_enqueue_scrub": 0, 2024-06-09T22:31:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_exportdir": 0, 2024-06-09T22:31:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_flush": 0, 2024-06-09T22:31:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_fragmentdir": 0, 2024-06-09T22:31:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_fragstats": 0, 2024-06-09T22:31:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_inodestats": 0 2024-06-09T22:31:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_log": { 2024-06-09T22:31:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: "evlrg": 0, 2024-06-09T22:31:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: "evadd": 0, 2024-06-09T22:31:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: "evex": 0, 2024-06-09T22:31:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: "evtrm": 0, 2024-06-09T22:31:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "ev": 22346, 2024-06-09T22:31:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "evexg": 0, 2024-06-09T22:31:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "evexd": 0, 2024-06-09T22:31:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "segadd": 0, 2024-06-09T22:31:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "segex": 0, 2024-06-09T22:31:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "segtrm": 0, 2024-06-09T22:31:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "seg": 26, 2024-06-09T22:31:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "segmjr": 3, 2024-06-09T22:31:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "segexg": 0, 2024-06-09T22:31:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "segexd": 0, 2024-06-09T22:31:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "expos": 4194304, 2024-06-09T22:31:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "wrpos": 39202605, 2024-06-09T22:31:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "rdpos": 39202006, 2024-06-09T22:31:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "jlat": { 2024-06-09T22:31:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "replayed": 22346 2024-06-09T22:31:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_mem": { 2024-06-09T22:31:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "ino": 84, 2024-06-09T22:31:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "ino+": 4695, 2024-06-09T22:31:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "ino-": 4611, 2024-06-09T22:31:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "dir": 23, 2024-06-09T22:31:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "dir+": 23, 2024-06-09T22:31:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "dir-": 0, 2024-06-09T22:31:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "dn": 4738, 2024-06-09T22:31:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "dn+": 4738, 2024-06-09T22:31:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "dn-": 0, 2024-06-09T22:31:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "cap": 0, 2024-06-09T22:31:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "cap+": 0, 2024-06-09T22:31:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "cap-": 0, 2024-06-09T22:31:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "rss": 57440, 2024-06-09T22:31:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "heap": 215420 2024-06-09T22:31:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_server": { 2024-06-09T22:31:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "dispatch_client_request": 0, 2024-06-09T22:31:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "dispatch_server_request": 0, 2024-06-09T22:31:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_request": 0, 2024-06-09T22:31:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_session": 0, 2024-06-09T22:31:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_peer_request": 0, 2024-06-09T22:31:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "req_create_latency": { 2024-06-09T22:31:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getattr_latency": { 2024-06-09T22:31:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getfilelock_latency": { 2024-06-09T22:31:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "req_link_latency": { 2024-06-09T22:31:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookup_latency": { 2024-06-09T22:31:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookuphash_latency": { 2024-06-09T22:31:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupino_latency": { 2024-06-09T22:31:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupname_latency": { 2024-06-09T22:31:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupparent_latency": { 2024-06-09T22:31:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupsnap_latency": { 2024-06-09T22:31:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lssnap_latency": { 2024-06-09T22:31:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mkdir_latency": { 2024-06-09T22:31:37.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:37 smithi194 ceph-mon[36856]: pgmap v330: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:37.823 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.823 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.823 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.823 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mknod_latency": { 2024-06-09T22:31:37.823 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.823 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.823 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.824 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mksnap_latency": { 2024-06-09T22:31:37.824 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.824 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.824 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.824 INFO:teuthology.orchestra.run.smithi104.stdout: "req_open_latency": { 2024-06-09T22:31:37.824 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.824 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.825 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.825 INFO:teuthology.orchestra.run.smithi104.stdout: "req_readdir_latency": { 2024-06-09T22:31:37.825 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.825 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.825 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.825 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rename_latency": { 2024-06-09T22:31:37.826 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.826 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.826 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.826 INFO:teuthology.orchestra.run.smithi104.stdout: "req_renamesnap_latency": { 2024-06-09T22:31:37.826 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.826 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.827 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.827 INFO:teuthology.orchestra.run.smithi104.stdout: "req_snapdiff_latency": { 2024-06-09T22:31:37.827 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.827 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.827 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.827 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmdir_latency": { 2024-06-09T22:31:37.827 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.828 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.828 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.828 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmsnap_latency": { 2024-06-09T22:31:37.828 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.828 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.828 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.829 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmxattr_latency": { 2024-06-09T22:31:37.829 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.829 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.829 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.829 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setattr_latency": { 2024-06-09T22:31:37.829 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.829 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.830 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.830 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setdirlayout_latency": { 2024-06-09T22:31:37.830 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.830 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.830 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.830 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setfilelock_latency": { 2024-06-09T22:31:37.831 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.831 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.831 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.831 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setlayout_latency": { 2024-06-09T22:31:37.831 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.831 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.831 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.832 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setxattr_latency": { 2024-06-09T22:31:37.832 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.832 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.832 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.832 INFO:teuthology.orchestra.run.smithi104.stdout: "req_symlink_latency": { 2024-06-09T22:31:37.832 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.832 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.833 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.833 INFO:teuthology.orchestra.run.smithi104.stdout: "req_unlink_latency": { 2024-06-09T22:31:37.833 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.833 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.833 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.833 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_revoke_eviction": 0, 2024-06-09T22:31:37.834 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_acquisition_throttle": 0, 2024-06-09T22:31:37.834 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getvxattr_latency": { 2024-06-09T22:31:37.834 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.834 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.834 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.834 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:37.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.835 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_sessions": { 2024-06-09T22:31:37.835 INFO:teuthology.orchestra.run.smithi104.stdout: "session_count": 2, 2024-06-09T22:31:37.835 INFO:teuthology.orchestra.run.smithi104.stdout: "session_add": 4, 2024-06-09T22:31:37.835 INFO:teuthology.orchestra.run.smithi104.stdout: "session_remove": 2, 2024-06-09T22:31:37.835 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions_open": 2, 2024-06-09T22:31:37.835 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions_stale": 0, 2024-06-09T22:31:37.835 INFO:teuthology.orchestra.run.smithi104.stdout: "total_load": 0, 2024-06-09T22:31:37.835 INFO:teuthology.orchestra.run.smithi104.stdout: "average_load": 0, 2024-06-09T22:31:37.836 INFO:teuthology.orchestra.run.smithi104.stdout: "avg_session_uptime": 3169, 2024-06-09T22:31:37.836 INFO:teuthology.orchestra.run.smithi104.stdout: "mdthresh_evicted": 0 2024-06-09T22:31:37.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.836 INFO:teuthology.orchestra.run.smithi104.stdout: "mempool": { 2024-06-09T22:31:37.836 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_bytes": 0, 2024-06-09T22:31:37.836 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_items": 0, 2024-06-09T22:31:37.836 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_bytes": 0, 2024-06-09T22:31:37.836 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_items": 0, 2024-06-09T22:31:37.837 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_bytes": 0, 2024-06-09T22:31:37.837 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_items": 0, 2024-06-09T22:31:37.837 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-09T22:31:37.837 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_items": 0, 2024-06-09T22:31:37.837 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-09T22:31:37.837 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_items": 0, 2024-06-09T22:31:37.837 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_bytes": 0, 2024-06-09T22:31:37.837 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_items": 0, 2024-06-09T22:31:37.838 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-09T22:31:37.838 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_items": 0, 2024-06-09T22:31:37.838 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_bytes": 0, 2024-06-09T22:31:37.838 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_items": 0, 2024-06-09T22:31:37.838 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_bytes": 0, 2024-06-09T22:31:37.838 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_items": 0, 2024-06-09T22:31:37.838 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-09T22:31:37.838 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_items": 0, 2024-06-09T22:31:37.839 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-09T22:31:37.839 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_items": 0, 2024-06-09T22:31:37.839 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_bytes": 0, 2024-06-09T22:31:37.839 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_items": 0, 2024-06-09T22:31:37.839 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_bytes": 0, 2024-06-09T22:31:37.839 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_items": 0, 2024-06-09T22:31:37.839 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-09T22:31:37.839 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_items": 0, 2024-06-09T22:31:37.840 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_bytes": 0, 2024-06-09T22:31:37.840 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_items": 0, 2024-06-09T22:31:37.840 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_bytes": 0, 2024-06-09T22:31:37.840 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_items": 0, 2024-06-09T22:31:37.840 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_bytes": 0, 2024-06-09T22:31:37.840 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_items": 0, 2024-06-09T22:31:37.840 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_bytes": 0, 2024-06-09T22:31:37.840 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_items": 0, 2024-06-09T22:31:37.841 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_bytes": 218485, 2024-06-09T22:31:37.841 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_items": 136, 2024-06-09T22:31:37.841 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_bytes": 0, 2024-06-09T22:31:37.841 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_items": 0, 2024-06-09T22:31:37.841 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_bytes": 0, 2024-06-09T22:31:37.841 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_items": 0, 2024-06-09T22:31:37.841 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_bytes": 0, 2024-06-09T22:31:37.841 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_items": 0, 2024-06-09T22:31:37.842 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_bytes": 0, 2024-06-09T22:31:37.842 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_items": 0, 2024-06-09T22:31:37.842 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_bytes": 9512, 2024-06-09T22:31:37.842 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_items": 182, 2024-06-09T22:31:37.842 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_bytes": 0, 2024-06-09T22:31:37.842 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_items": 0, 2024-06-09T22:31:37.843 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_bytes": 0, 2024-06-09T22:31:37.843 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_items": 0, 2024-06-09T22:31:37.843 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_bytes": 3602177, 2024-06-09T22:31:37.843 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_items": 20802, 2024-06-09T22:31:37.843 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_bytes": 0, 2024-06-09T22:31:37.843 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_items": 0, 2024-06-09T22:31:37.843 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_bytes": 0, 2024-06-09T22:31:37.843 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_items": 0 2024-06-09T22:31:37.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.844 INFO:teuthology.orchestra.run.smithi104.stdout: "objecter": { 2024-06-09T22:31:37.844 INFO:teuthology.orchestra.run.smithi104.stdout: "op_active": 0, 2024-06-09T22:31:37.844 INFO:teuthology.orchestra.run.smithi104.stdout: "op_laggy": 0, 2024-06-09T22:31:37.844 INFO:teuthology.orchestra.run.smithi104.stdout: "op_send": 2625, 2024-06-09T22:31:37.844 INFO:teuthology.orchestra.run.smithi104.stdout: "op_send_bytes": 21, 2024-06-09T22:31:37.844 INFO:teuthology.orchestra.run.smithi104.stdout: "op_resend": 3, 2024-06-09T22:31:37.845 INFO:teuthology.orchestra.run.smithi104.stdout: "op_reply": 2625, 2024-06-09T22:31:37.845 INFO:teuthology.orchestra.run.smithi104.stdout: "op_latency": { 2024-06-09T22:31:37.845 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 2625, 2024-06-09T22:31:37.845 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 1.637981724, 2024-06-09T22:31:37.845 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000623993 2024-06-09T22:31:37.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.845 INFO:teuthology.orchestra.run.smithi104.stdout: "op_inflight": 0, 2024-06-09T22:31:37.845 INFO:teuthology.orchestra.run.smithi104.stdout: "oplen_avg": { 2024-06-09T22:31:37.846 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 2625, 2024-06-09T22:31:37.846 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 2626 2024-06-09T22:31:37.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.846 INFO:teuthology.orchestra.run.smithi104.stdout: "op": 2625, 2024-06-09T22:31:37.846 INFO:teuthology.orchestra.run.smithi104.stdout: "op_r": 2625, 2024-06-09T22:31:37.846 INFO:teuthology.orchestra.run.smithi104.stdout: "op_w": 0, 2024-06-09T22:31:37.846 INFO:teuthology.orchestra.run.smithi104.stdout: "op_rmw": 0, 2024-06-09T22:31:37.847 INFO:teuthology.orchestra.run.smithi104.stdout: "op_pg": 0, 2024-06-09T22:31:37.847 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_stat": 1554, 2024-06-09T22:31:37.847 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_create": 0, 2024-06-09T22:31:37.847 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_read": 1069, 2024-06-09T22:31:37.847 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_write": 0, 2024-06-09T22:31:37.847 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_writefull": 0, 2024-06-09T22:31:37.847 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_writesame": 0, 2024-06-09T22:31:37.847 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_append": 0, 2024-06-09T22:31:37.847 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_zero": 0, 2024-06-09T22:31:37.848 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_truncate": 0, 2024-06-09T22:31:37.848 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_delete": 0, 2024-06-09T22:31:37.848 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_mapext": 0, 2024-06-09T22:31:37.848 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_sparse_read": 0, 2024-06-09T22:31:37.848 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_clonerange": 0, 2024-06-09T22:31:37.848 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_getxattr": 1, 2024-06-09T22:31:37.848 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_setxattr": 0, 2024-06-09T22:31:37.848 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_cmpxattr": 0, 2024-06-09T22:31:37.849 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_rmxattr": 0, 2024-06-09T22:31:37.849 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_resetxattrs": 0, 2024-06-09T22:31:37.849 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_call": 0, 2024-06-09T22:31:37.849 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_watch": 0, 2024-06-09T22:31:37.849 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_notify": 0, 2024-06-09T22:31:37.849 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_src_cmpxattr": 0, 2024-06-09T22:31:37.849 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_pgls": 0, 2024-06-09T22:31:37.849 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_pgls_filter": 0, 2024-06-09T22:31:37.850 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_other": 0, 2024-06-09T22:31:37.850 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_active": 0, 2024-06-09T22:31:37.850 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_send": 0, 2024-06-09T22:31:37.850 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_resend": 0, 2024-06-09T22:31:37.850 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_ping": 0, 2024-06-09T22:31:37.850 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_active": 0, 2024-06-09T22:31:37.850 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_send": 0, 2024-06-09T22:31:37.850 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_resend": 0, 2024-06-09T22:31:37.851 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_active": 0, 2024-06-09T22:31:37.851 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_send": 0, 2024-06-09T22:31:37.851 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_resend": 0, 2024-06-09T22:31:37.851 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_active": 0, 2024-06-09T22:31:37.851 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_send": 0, 2024-06-09T22:31:37.851 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_resend": 0, 2024-06-09T22:31:37.851 INFO:teuthology.orchestra.run.smithi104.stdout: "command_active": 0, 2024-06-09T22:31:37.851 INFO:teuthology.orchestra.run.smithi104.stdout: "command_send": 0, 2024-06-09T22:31:37.852 INFO:teuthology.orchestra.run.smithi104.stdout: "command_resend": 0, 2024-06-09T22:31:37.852 INFO:teuthology.orchestra.run.smithi104.stdout: "map_epoch": 108, 2024-06-09T22:31:37.852 INFO:teuthology.orchestra.run.smithi104.stdout: "map_full": 0, 2024-06-09T22:31:37.852 INFO:teuthology.orchestra.run.smithi104.stdout: "map_inc": 11, 2024-06-09T22:31:37.852 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_sessions": 8, 2024-06-09T22:31:37.852 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_session_open": 8, 2024-06-09T22:31:37.852 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_session_close": 0, 2024-06-09T22:31:37.852 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_laggy": 0, 2024-06-09T22:31:37.853 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_wr": 0, 2024-06-09T22:31:37.853 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_rd": 2, 2024-06-09T22:31:37.853 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_del": 0 2024-06-09T22:31:37.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.853 INFO:teuthology.orchestra.run.smithi104.stdout: "oft": { 2024-06-09T22:31:37.853 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_objs": 0, 2024-06-09T22:31:37.853 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_kv_pairs": 0, 2024-06-09T22:31:37.853 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_updates": 0, 2024-06-09T22:31:37.854 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_removes": 0 2024-06-09T22:31:37.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.854 INFO:teuthology.orchestra.run.smithi104.stdout: "osd-slow-ops": { 2024-06-09T22:31:37.854 INFO:teuthology.orchestra.run.smithi104.stdout: "slow_ops_count": 0 2024-06-09T22:31:37.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.854 INFO:teuthology.orchestra.run.smithi104.stdout: "purge_queue": { 2024-06-09T22:31:37.854 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_ops": 0, 2024-06-09T22:31:37.855 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_ops_high_water": 0, 2024-06-09T22:31:37.855 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing": 0, 2024-06-09T22:31:37.855 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_high_water": 0, 2024-06-09T22:31:37.855 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executed_ops": 0, 2024-06-09T22:31:37.855 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executed": 0, 2024-06-09T22:31:37.855 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_item_in_journal": 0 2024-06-09T22:31:37.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.855 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-09T22:31:37.856 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-06-09T22:31:37.856 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-06-09T22:31:37.856 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-06-09T22:31:37.856 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 2902, 2024-06-09T22:31:37.856 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 36982685, 2024-06-09T22:31:37.856 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:37.856 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 2902, 2024-06-09T22:31:37.856 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-06-09T22:31:37.857 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-06-09T22:31:37.857 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 2902, 2024-06-09T22:31:37.857 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 36982685, 2024-06-09T22:31:37.857 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-06-09T22:31:37.857 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.857 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.857 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.857 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:37.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.858 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-objecter_bytes": { 2024-06-09T22:31:37.858 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-06-09T22:31:37.858 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-06-09T22:31:37.858 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-06-09T22:31:37.858 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-06-09T22:31:37.858 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-06-09T22:31:37.858 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:37.859 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:37.859 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 2625, 2024-06-09T22:31:37.859 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 35010810, 2024-06-09T22:31:37.859 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 2625, 2024-06-09T22:31:37.859 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 35010810, 2024-06-09T22:31:37.859 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-06-09T22:31:37.859 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.859 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.859 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.860 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:37.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.860 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-objecter_ops": { 2024-06-09T22:31:37.860 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-06-09T22:31:37.860 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 1024, 2024-06-09T22:31:37.860 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-06-09T22:31:37.860 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-06-09T22:31:37.860 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-06-09T22:31:37.860 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:37.861 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:37.861 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 2625, 2024-06-09T22:31:37.861 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 2625, 2024-06-09T22:31:37.861 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 2625, 2024-06-09T22:31:37.861 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 2625, 2024-06-09T22:31:37.861 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-06-09T22:31:37.861 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.861 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.861 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.862 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:37.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.862 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-write_buf_throttle": { 2024-06-09T22:31:37.862 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-06-09T22:31:37.862 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 3758096384, 2024-06-09T22:31:37.862 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-06-09T22:31:37.862 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-06-09T22:31:37.862 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-06-09T22:31:37.863 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:37.863 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:37.863 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-06-09T22:31:37.863 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-06-09T22:31:37.863 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 0, 2024-06-09T22:31:37.863 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 0, 2024-06-09T22:31:37.863 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-06-09T22:31:37.863 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.863 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.863 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.864 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:37.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:37.864 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-write_buf_throttle-0x5590982f86e0": { 2024-06-09T22:31:37.864 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-06-09T22:31:37.864 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 3758096384, 2024-06-09T22:31:37.864 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-06-09T22:31:37.864 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-06-09T22:31:37.864 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-06-09T22:31:37.864 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:37.865 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:37.865 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-06-09T22:31:37.865 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-06-09T22:31:37.865 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 0, 2024-06-09T22:31:37.865 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 0, 2024-06-09T22:31:37.865 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-06-09T22:31:37.865 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:37.865 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:37.866 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:37.866 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:37.866 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:37.866 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:31:37.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:37 smithi153 ceph-mon[36599]: pgmap v330: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:38.858 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-06-09T22:31:38.858 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-06-09T22:31:38.858 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-06-09T22:31:38.858 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-06-09T22:31:38.859 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-06-09T22:31:38.859 DEBUG:tasks.check_counter:Getting stats from d 2024-06-09T22:31:38.859 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mds.d perf dump 2024-06-09T22:31:39.098 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:31:39.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:39 smithi194 ceph-mon[36856]: pgmap v331: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:39.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:39 smithi153 ceph-mon[36599]: pgmap v331: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:40.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:39 smithi104 ceph-mon[32389]: pgmap v331: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:40.972 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:31:40.972 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-0": { 2024-06-09T22:31:40.972 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 260, 2024-06-09T22:31:40.973 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 250, 2024-06-09T22:31:40.973 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 1454240, 2024-06-09T22:31:40.973 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 154000, 2024-06-09T22:31:40.973 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 4, 2024-06-09T22:31:40.973 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 0, 2024-06-09T22:31:40.973 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.046535030, 2024-06-09T22:31:40.973 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.014083052, 2024-06-09T22:31:40.973 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.036328051, 2024-06-09T22:31:40.973 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.004540872, 2024-06-09T22:31:40.974 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:40.974 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 249, 2024-06-09T22:31:40.974 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.009649935, 2024-06-09T22:31:40.974 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000038754 2024-06-09T22:31:40.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:40.974 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:40.974 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:40.975 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:40.975 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:40.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:40.975 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 1454240, 2024-06-09T22:31:40.975 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 154000 2024-06-09T22:31:40.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:40.975 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-1": { 2024-06-09T22:31:40.975 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 2, 2024-06-09T22:31:40.976 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 191, 2024-06-09T22:31:40.976 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 320, 2024-06-09T22:31:40.976 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 189440, 2024-06-09T22:31:40.976 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 5, 2024-06-09T22:31:40.976 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 3, 2024-06-09T22:31:40.976 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.014733234, 2024-06-09T22:31:40.976 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.010321883, 2024-06-09T22:31:40.976 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.001900681, 2024-06-09T22:31:40.976 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-09T22:31:40.977 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:40.977 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 188, 2024-06-09T22:31:40.977 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.016158157, 2024-06-09T22:31:40.977 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000085947 2024-06-09T22:31:40.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:40.977 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:40.977 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:40.977 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:40.978 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:40.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:40.978 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-09T22:31:40.978 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 189440 2024-06-09T22:31:40.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:40.978 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-2": { 2024-06-09T22:31:40.978 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 4, 2024-06-09T22:31:40.978 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 4, 2024-06-09T22:31:40.979 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 3024, 2024-06-09T22:31:40.979 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 448, 2024-06-09T22:31:40.979 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 2, 2024-06-09T22:31:40.979 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 1, 2024-06-09T22:31:40.979 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.000659641, 2024-06-09T22:31:40.979 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.000040460, 2024-06-09T22:31:40.979 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.000602332, 2024-06-09T22:31:40.979 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-09T22:31:40.979 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:40.980 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 3, 2024-06-09T22:31:40.980 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000713360, 2024-06-09T22:31:40.980 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000237786 2024-06-09T22:31:40.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:40.980 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:40.980 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:40.980 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:40.980 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:40.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:40.981 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-06-09T22:31:40.981 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 448 2024-06-09T22:31:40.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:40.981 INFO:teuthology.orchestra.run.smithi104.stdout: "cct": { 2024-06-09T22:31:40.981 INFO:teuthology.orchestra.run.smithi104.stdout: "total_workers": 0, 2024-06-09T22:31:40.981 INFO:teuthology.orchestra.run.smithi104.stdout: "unhealthy_workers": 0 2024-06-09T22:31:40.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:40.981 INFO:teuthology.orchestra.run.smithi104.stdout: "mempool": { 2024-06-09T22:31:40.981 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_bytes": 0, 2024-06-09T22:31:40.982 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_items": 0, 2024-06-09T22:31:40.982 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_bytes": 0, 2024-06-09T22:31:40.982 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_items": 0, 2024-06-09T22:31:40.982 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_bytes": 0, 2024-06-09T22:31:40.982 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_items": 0, 2024-06-09T22:31:40.982 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-09T22:31:40.982 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_items": 0, 2024-06-09T22:31:40.983 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-09T22:31:40.983 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_items": 0, 2024-06-09T22:31:40.983 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_bytes": 0, 2024-06-09T22:31:40.983 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_items": 0, 2024-06-09T22:31:40.983 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-09T22:31:40.984 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_items": 0, 2024-06-09T22:31:40.984 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_bytes": 0, 2024-06-09T22:31:40.984 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_items": 0, 2024-06-09T22:31:40.984 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_bytes": 0, 2024-06-09T22:31:40.984 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_items": 0, 2024-06-09T22:31:40.984 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-09T22:31:40.984 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_items": 0, 2024-06-09T22:31:40.984 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-09T22:31:40.985 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_items": 0, 2024-06-09T22:31:40.985 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_bytes": 0, 2024-06-09T22:31:40.985 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_items": 0, 2024-06-09T22:31:40.985 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_bytes": 0, 2024-06-09T22:31:40.985 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_items": 0, 2024-06-09T22:31:40.985 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-09T22:31:40.985 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_items": 0, 2024-06-09T22:31:40.985 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_bytes": 0, 2024-06-09T22:31:40.985 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_items": 0, 2024-06-09T22:31:40.986 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_bytes": 0, 2024-06-09T22:31:40.986 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_items": 0, 2024-06-09T22:31:40.986 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_bytes": 0, 2024-06-09T22:31:40.986 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_items": 0, 2024-06-09T22:31:40.986 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_bytes": 0, 2024-06-09T22:31:40.986 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_items": 0, 2024-06-09T22:31:40.986 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_bytes": 148731, 2024-06-09T22:31:40.986 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_items": 31, 2024-06-09T22:31:40.987 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_bytes": 0, 2024-06-09T22:31:40.987 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_items": 0, 2024-06-09T22:31:40.987 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_bytes": 0, 2024-06-09T22:31:40.987 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_items": 0, 2024-06-09T22:31:40.987 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_bytes": 0, 2024-06-09T22:31:40.987 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_items": 0, 2024-06-09T22:31:40.988 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_bytes": 0, 2024-06-09T22:31:40.988 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_items": 0, 2024-06-09T22:31:40.988 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_bytes": 0, 2024-06-09T22:31:40.988 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_items": 0, 2024-06-09T22:31:40.988 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_bytes": 0, 2024-06-09T22:31:40.988 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_items": 0, 2024-06-09T22:31:40.988 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_bytes": 0, 2024-06-09T22:31:40.989 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_items": 0, 2024-06-09T22:31:40.989 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_bytes": 928, 2024-06-09T22:31:40.989 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_items": 2, 2024-06-09T22:31:40.989 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_bytes": 0, 2024-06-09T22:31:40.990 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_items": 0, 2024-06-09T22:31:40.990 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_bytes": 0, 2024-06-09T22:31:40.990 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_items": 0 2024-06-09T22:31:40.990 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:40.990 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-09T22:31:40.990 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-06-09T22:31:40.990 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-06-09T22:31:40.991 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-06-09T22:31:40.991 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 262, 2024-06-09T22:31:40.991 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 1417585, 2024-06-09T22:31:40.991 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:40.992 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 262, 2024-06-09T22:31:40.992 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-06-09T22:31:40.992 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-06-09T22:31:40.993 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 262, 2024-06-09T22:31:40.993 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 1417585, 2024-06-09T22:31:40.993 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-06-09T22:31:40.993 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:40.994 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:40.994 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:40.994 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:40.995 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:40.995 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:31:41.715 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:41 smithi104 ceph-mon[32389]: pgmap v332: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:41.743 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-06-09T22:31:41.744 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-06-09T22:31:41.744 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-06-09T22:31:41.744 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-06-09T22:31:41.744 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-06-09T22:31:41.744 DEBUG:tasks.check_counter:Getting stats from g 2024-06-09T22:31:41.744 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mds.g perf dump 2024-06-09T22:31:41.985 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:31:41.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:41 smithi153 ceph-mon[36599]: pgmap v332: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:42.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:41 smithi194 ceph-mon[36856]: pgmap v332: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:43.820 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:43 smithi153 ceph-mon[36599]: pgmap v333: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-06-09T22:31:43.853 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:43 smithi104 ceph-mon[32389]: pgmap v333: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-06-09T22:31:43.864 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:43 smithi194 ceph-mon[36856]: pgmap v333: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-06-09T22:31:43.883 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:31:43.883 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-0": { 2024-06-09T22:31:43.883 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 833, 2024-06-09T22:31:43.883 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 835, 2024-06-09T22:31:43.883 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 279833, 2024-06-09T22:31:43.884 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 264895, 2024-06-09T22:31:43.884 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 6, 2024-06-09T22:31:43.884 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 2, 2024-06-09T22:31:43.884 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.084704223, 2024-06-09T22:31:43.884 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.037452764, 2024-06-09T22:31:43.884 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.017587201, 2024-06-09T22:31:43.884 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.021251411, 2024-06-09T22:31:43.884 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:43.884 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 833, 2024-06-09T22:31:43.885 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.038602278, 2024-06-09T22:31:43.885 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000046341 2024-06-09T22:31:43.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.885 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:43.885 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.885 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.885 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.886 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-09T22:31:43.886 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 0 2024-06-09T22:31:43.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.886 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-1": { 2024-06-09T22:31:43.886 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 1485, 2024-06-09T22:31:43.886 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 1672, 2024-06-09T22:31:43.886 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 380331, 2024-06-09T22:31:43.886 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 947555, 2024-06-09T22:31:43.887 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 7, 2024-06-09T22:31:43.887 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 5, 2024-06-09T22:31:43.887 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.096715893, 2024-06-09T22:31:43.887 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.032622640, 2024-06-09T22:31:43.887 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.023653718, 2024-06-09T22:31:43.887 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.031918315, 2024-06-09T22:31:43.887 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:43.887 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 1669, 2024-06-09T22:31:43.888 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.051069226, 2024-06-09T22:31:43.888 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000030598 2024-06-09T22:31:43.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.888 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:43.888 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.888 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.888 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.889 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-09T22:31:43.889 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 475904 2024-06-09T22:31:43.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.889 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-2": { 2024-06-09T22:31:43.889 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 285, 2024-06-09T22:31:43.889 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 268, 2024-06-09T22:31:43.889 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 1516852, 2024-06-09T22:31:43.889 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 156261, 2024-06-09T22:31:43.890 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 4, 2024-06-09T22:31:43.890 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 3, 2024-06-09T22:31:43.890 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.048306940, 2024-06-09T22:31:43.890 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.017323486, 2024-06-09T22:31:43.890 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.034002015, 2024-06-09T22:31:43.890 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.004613574, 2024-06-09T22:31:43.890 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:43.890 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 268, 2024-06-09T22:31:43.890 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.013963248, 2024-06-09T22:31:43.891 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000052101 2024-06-09T22:31:43.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.891 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:43.891 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.891 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.891 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.892 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 1515504, 2024-06-09T22:31:43.892 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 155296 2024-06-09T22:31:43.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.892 INFO:teuthology.orchestra.run.smithi104.stdout: "cct": { 2024-06-09T22:31:43.892 INFO:teuthology.orchestra.run.smithi104.stdout: "total_workers": 1, 2024-06-09T22:31:43.892 INFO:teuthology.orchestra.run.smithi104.stdout: "unhealthy_workers": 0 2024-06-09T22:31:43.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.893 INFO:teuthology.orchestra.run.smithi104.stdout: "finisher-MDSRank": { 2024-06-09T22:31:43.893 INFO:teuthology.orchestra.run.smithi104.stdout: "queue_len": 0, 2024-06-09T22:31:43.893 INFO:teuthology.orchestra.run.smithi104.stdout: "complete_latency": { 2024-06-09T22:31:43.893 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 3968, 2024-06-09T22:31:43.893 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.086795947, 2024-06-09T22:31:43.893 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000021873 2024-06-09T22:31:43.893 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:43.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.893 INFO:teuthology.orchestra.run.smithi104.stdout: "finisher-PurgeQueue": { 2024-06-09T22:31:43.894 INFO:teuthology.orchestra.run.smithi104.stdout: "queue_len": 0, 2024-06-09T22:31:43.894 INFO:teuthology.orchestra.run.smithi104.stdout: "complete_latency": { 2024-06-09T22:31:43.894 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.894 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.894 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:43.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.894 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-06-09T22:31:43.895 INFO:teuthology.orchestra.run.smithi104.stdout: "request": 0, 2024-06-09T22:31:43.895 INFO:teuthology.orchestra.run.smithi104.stdout: "reply": 0, 2024-06-09T22:31:43.895 INFO:teuthology.orchestra.run.smithi104.stdout: "reply_latency": { 2024-06-09T22:31:43.895 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.895 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.895 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.895 INFO:teuthology.orchestra.run.smithi104.stdout: "slow_reply": 0, 2024-06-09T22:31:43.896 INFO:teuthology.orchestra.run.smithi104.stdout: "forward": 0, 2024-06-09T22:31:43.896 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_fetch_complete": 0, 2024-06-09T22:31:43.896 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_fetch_keys": 0, 2024-06-09T22:31:43.896 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_commit": 0, 2024-06-09T22:31:43.896 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_split": 0, 2024-06-09T22:31:43.896 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_merge": 0, 2024-06-09T22:31:43.896 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes": 0, 2024-06-09T22:31:43.896 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_top": 0, 2024-06-09T22:31:43.897 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_bottom": 0, 2024-06-09T22:31:43.897 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_pin_tail": 0, 2024-06-09T22:31:43.897 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_pinned": 0, 2024-06-09T22:31:43.897 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_expired": 0, 2024-06-09T22:31:43.897 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_with_caps": 0, 2024-06-09T22:31:43.897 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": 0, 2024-06-09T22:31:43.897 INFO:teuthology.orchestra.run.smithi104.stdout: "subtrees": 1, 2024-06-09T22:31:43.897 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse": 0, 2024-06-09T22:31:43.897 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_hit": 0, 2024-06-09T22:31:43.898 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_forward": 0, 2024-06-09T22:31:43.898 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_discover": 0, 2024-06-09T22:31:43.898 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_dir_fetch": 0, 2024-06-09T22:31:43.898 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_remote_ino": 0, 2024-06-09T22:31:43.898 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_lock": 0, 2024-06-09T22:31:43.898 INFO:teuthology.orchestra.run.smithi104.stdout: "load_cent": 0, 2024-06-09T22:31:43.898 INFO:teuthology.orchestra.run.smithi104.stdout: "q": 0, 2024-06-09T22:31:43.898 INFO:teuthology.orchestra.run.smithi104.stdout: "exported": 0, 2024-06-09T22:31:43.899 INFO:teuthology.orchestra.run.smithi104.stdout: "exported_inodes": 0, 2024-06-09T22:31:43.899 INFO:teuthology.orchestra.run.smithi104.stdout: "imported": 0, 2024-06-09T22:31:43.899 INFO:teuthology.orchestra.run.smithi104.stdout: "imported_inodes": 0, 2024-06-09T22:31:43.899 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_dir_fetch": 0, 2024-06-09T22:31:43.899 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_backtrace_fetch": 0, 2024-06-09T22:31:43.899 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_peer_discover": 0, 2024-06-09T22:31:43.899 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rfiles": 0, 2024-06-09T22:31:43.899 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rbytes": 0, 2024-06-09T22:31:43.900 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rsnaps": 0, 2024-06-09T22:31:43.900 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_backtrace_fetch": 0, 2024-06-09T22:31:43.900 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_set_tag": 0, 2024-06-09T22:31:43.900 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_backtrace_repaired": 0, 2024-06-09T22:31:43.900 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_inotable_repaired": 0, 2024-06-09T22:31:43.900 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dir_inodes": 0, 2024-06-09T22:31:43.900 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dir_base_inodes": 0, 2024-06-09T22:31:43.900 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dirfrag_rstats": 0, 2024-06-09T22:31:43.901 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_file_inodes": 0, 2024-06-09T22:31:43.901 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_inode_file_caps": 0, 2024-06-09T22:31:43.901 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_revoke": 0, 2024-06-09T22:31:43.901 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_grant": 0, 2024-06-09T22:31:43.901 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_trunc": 0, 2024-06-09T22:31:43.901 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-09T22:31:43.901 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-09T22:31:43.901 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_caps": 0, 2024-06-09T22:31:43.902 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_caps_dirty": 0, 2024-06-09T22:31:43.902 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_cap_release": 0, 2024-06-09T22:31:43.902 INFO:teuthology.orchestra.run.smithi104.stdout: "process_request_cap_release": 0 2024-06-09T22:31:43.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.902 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_cache": { 2024-06-09T22:31:43.902 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_update": 0, 2024-06-09T22:31:43.902 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_update_receipt": 0, 2024-06-09T22:31:43.902 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_try_discover": 0, 2024-06-09T22:31:43.903 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_send_discover": 0, 2024-06-09T22:31:43.903 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_handle_discover": 0, 2024-06-09T22:31:43.903 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays": 0, 2024-06-09T22:31:43.903 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays_delayed": 0, 2024-06-09T22:31:43.903 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays_enqueuing": 0, 2024-06-09T22:31:43.903 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_created": 0, 2024-06-09T22:31:43.903 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_enqueued": 0, 2024-06-09T22:31:43.903 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_reintegrated": 0, 2024-06-09T22:31:43.904 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_migrated": 0, 2024-06-09T22:31:43.904 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_processing": 0, 2024-06-09T22:31:43.904 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_enqueued": 0, 2024-06-09T22:31:43.904 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_prioritized": 0, 2024-06-09T22:31:43.904 INFO:teuthology.orchestra.run.smithi104.stdout: "recovery_started": 0, 2024-06-09T22:31:43.904 INFO:teuthology.orchestra.run.smithi104.stdout: "recovery_completed": 0, 2024-06-09T22:31:43.904 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_quiesce_path": 0, 2024-06-09T22:31:43.904 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_quiesce_inode": 0, 2024-06-09T22:31:43.904 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_enqueue_scrub": 0, 2024-06-09T22:31:43.905 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_exportdir": 0, 2024-06-09T22:31:43.905 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_flush": 0, 2024-06-09T22:31:43.905 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_fragmentdir": 0, 2024-06-09T22:31:43.905 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_fragstats": 0, 2024-06-09T22:31:43.905 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_inodestats": 0 2024-06-09T22:31:43.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.905 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_log": { 2024-06-09T22:31:43.905 INFO:teuthology.orchestra.run.smithi104.stdout: "evlrg": 0, 2024-06-09T22:31:43.905 INFO:teuthology.orchestra.run.smithi104.stdout: "evadd": 0, 2024-06-09T22:31:43.906 INFO:teuthology.orchestra.run.smithi104.stdout: "evex": 0, 2024-06-09T22:31:43.906 INFO:teuthology.orchestra.run.smithi104.stdout: "evtrm": 0, 2024-06-09T22:31:43.906 INFO:teuthology.orchestra.run.smithi104.stdout: "ev": 6, 2024-06-09T22:31:43.906 INFO:teuthology.orchestra.run.smithi104.stdout: "evexg": 0, 2024-06-09T22:31:43.906 INFO:teuthology.orchestra.run.smithi104.stdout: "evexd": 0, 2024-06-09T22:31:43.906 INFO:teuthology.orchestra.run.smithi104.stdout: "segadd": 0, 2024-06-09T22:31:43.906 INFO:teuthology.orchestra.run.smithi104.stdout: "segex": 0, 2024-06-09T22:31:43.906 INFO:teuthology.orchestra.run.smithi104.stdout: "segtrm": 0, 2024-06-09T22:31:43.906 INFO:teuthology.orchestra.run.smithi104.stdout: "seg": 2, 2024-06-09T22:31:43.907 INFO:teuthology.orchestra.run.smithi104.stdout: "segmjr": 2, 2024-06-09T22:31:43.907 INFO:teuthology.orchestra.run.smithi104.stdout: "segexg": 0, 2024-06-09T22:31:43.907 INFO:teuthology.orchestra.run.smithi104.stdout: "segexd": 0, 2024-06-09T22:31:43.907 INFO:teuthology.orchestra.run.smithi104.stdout: "expos": 4194304, 2024-06-09T22:31:43.907 INFO:teuthology.orchestra.run.smithi104.stdout: "wrpos": 4196121, 2024-06-09T22:31:43.907 INFO:teuthology.orchestra.run.smithi104.stdout: "rdpos": 4195980, 2024-06-09T22:31:43.907 INFO:teuthology.orchestra.run.smithi104.stdout: "jlat": { 2024-06-09T22:31:43.907 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.908 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.908 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.908 INFO:teuthology.orchestra.run.smithi104.stdout: "replayed": 6 2024-06-09T22:31:43.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.908 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_mem": { 2024-06-09T22:31:43.908 INFO:teuthology.orchestra.run.smithi104.stdout: "ino": 3, 2024-06-09T22:31:43.908 INFO:teuthology.orchestra.run.smithi104.stdout: "ino+": 3, 2024-06-09T22:31:43.909 INFO:teuthology.orchestra.run.smithi104.stdout: "ino-": 0, 2024-06-09T22:31:43.909 INFO:teuthology.orchestra.run.smithi104.stdout: "dir": 1, 2024-06-09T22:31:43.909 INFO:teuthology.orchestra.run.smithi104.stdout: "dir+": 1, 2024-06-09T22:31:43.909 INFO:teuthology.orchestra.run.smithi104.stdout: "dir-": 0, 2024-06-09T22:31:43.909 INFO:teuthology.orchestra.run.smithi104.stdout: "dn": 0, 2024-06-09T22:31:43.909 INFO:teuthology.orchestra.run.smithi104.stdout: "dn+": 0, 2024-06-09T22:31:43.909 INFO:teuthology.orchestra.run.smithi104.stdout: "dn-": 0, 2024-06-09T22:31:43.909 INFO:teuthology.orchestra.run.smithi104.stdout: "cap": 0, 2024-06-09T22:31:43.909 INFO:teuthology.orchestra.run.smithi104.stdout: "cap+": 0, 2024-06-09T22:31:43.910 INFO:teuthology.orchestra.run.smithi104.stdout: "cap-": 0, 2024-06-09T22:31:43.910 INFO:teuthology.orchestra.run.smithi104.stdout: "rss": 50712, 2024-06-09T22:31:43.910 INFO:teuthology.orchestra.run.smithi104.stdout: "heap": 215420 2024-06-09T22:31:43.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.910 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_server": { 2024-06-09T22:31:43.910 INFO:teuthology.orchestra.run.smithi104.stdout: "dispatch_client_request": 0, 2024-06-09T22:31:43.910 INFO:teuthology.orchestra.run.smithi104.stdout: "dispatch_server_request": 0, 2024-06-09T22:31:43.910 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_request": 0, 2024-06-09T22:31:43.911 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_session": 0, 2024-06-09T22:31:43.911 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_peer_request": 0, 2024-06-09T22:31:43.911 INFO:teuthology.orchestra.run.smithi104.stdout: "req_create_latency": { 2024-06-09T22:31:43.911 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.911 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.911 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.911 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getattr_latency": { 2024-06-09T22:31:43.912 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.912 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.912 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.912 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getfilelock_latency": { 2024-06-09T22:31:43.912 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.912 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.912 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.913 INFO:teuthology.orchestra.run.smithi104.stdout: "req_link_latency": { 2024-06-09T22:31:43.913 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.913 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.913 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.913 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookup_latency": { 2024-06-09T22:31:43.913 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.913 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.913 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.914 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookuphash_latency": { 2024-06-09T22:31:43.914 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.914 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.914 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.914 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupino_latency": { 2024-06-09T22:31:43.914 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.915 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.915 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.915 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupname_latency": { 2024-06-09T22:31:43.915 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.915 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.915 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.916 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupparent_latency": { 2024-06-09T22:31:43.916 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.916 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.916 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.916 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupsnap_latency": { 2024-06-09T22:31:43.916 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.916 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.916 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.917 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lssnap_latency": { 2024-06-09T22:31:43.917 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.917 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.917 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.917 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mkdir_latency": { 2024-06-09T22:31:43.917 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.918 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.918 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.919 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mknod_latency": { 2024-06-09T22:31:43.919 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.919 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.919 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.919 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mksnap_latency": { 2024-06-09T22:31:43.919 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.919 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.920 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.920 INFO:teuthology.orchestra.run.smithi104.stdout: "req_open_latency": { 2024-06-09T22:31:43.920 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.920 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.920 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.920 INFO:teuthology.orchestra.run.smithi104.stdout: "req_readdir_latency": { 2024-06-09T22:31:43.921 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.921 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.921 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.921 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rename_latency": { 2024-06-09T22:31:43.921 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.921 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.921 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.922 INFO:teuthology.orchestra.run.smithi104.stdout: "req_renamesnap_latency": { 2024-06-09T22:31:43.922 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.922 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.922 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.922 INFO:teuthology.orchestra.run.smithi104.stdout: "req_snapdiff_latency": { 2024-06-09T22:31:43.922 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.922 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.923 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.923 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmdir_latency": { 2024-06-09T22:31:43.923 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.923 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.923 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.923 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmsnap_latency": { 2024-06-09T22:31:43.923 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.924 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.924 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.924 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmxattr_latency": { 2024-06-09T22:31:43.924 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.924 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.924 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.924 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setattr_latency": { 2024-06-09T22:31:43.924 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.925 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.925 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.925 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setdirlayout_latency": { 2024-06-09T22:31:43.925 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.925 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.925 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.926 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setfilelock_latency": { 2024-06-09T22:31:43.926 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.926 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.926 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.926 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setlayout_latency": { 2024-06-09T22:31:43.926 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.926 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.926 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.927 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setxattr_latency": { 2024-06-09T22:31:43.927 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.927 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.927 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.927 INFO:teuthology.orchestra.run.smithi104.stdout: "req_symlink_latency": { 2024-06-09T22:31:43.927 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.928 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.928 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.928 INFO:teuthology.orchestra.run.smithi104.stdout: "req_unlink_latency": { 2024-06-09T22:31:43.928 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.928 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.928 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.928 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_revoke_eviction": 0, 2024-06-09T22:31:43.929 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_acquisition_throttle": 0, 2024-06-09T22:31:43.929 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getvxattr_latency": { 2024-06-09T22:31:43.929 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.929 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.929 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.929 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:43.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.929 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_sessions": { 2024-06-09T22:31:43.930 INFO:teuthology.orchestra.run.smithi104.stdout: "session_count": 2, 2024-06-09T22:31:43.930 INFO:teuthology.orchestra.run.smithi104.stdout: "session_add": 3, 2024-06-09T22:31:43.930 INFO:teuthology.orchestra.run.smithi104.stdout: "session_remove": 1, 2024-06-09T22:31:43.930 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions_open": 3, 2024-06-09T22:31:43.930 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions_stale": 0, 2024-06-09T22:31:43.930 INFO:teuthology.orchestra.run.smithi104.stdout: "total_load": 0, 2024-06-09T22:31:43.930 INFO:teuthology.orchestra.run.smithi104.stdout: "average_load": 0, 2024-06-09T22:31:43.930 INFO:teuthology.orchestra.run.smithi104.stdout: "avg_session_uptime": 2542, 2024-06-09T22:31:43.931 INFO:teuthology.orchestra.run.smithi104.stdout: "mdthresh_evicted": 0 2024-06-09T22:31:43.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.931 INFO:teuthology.orchestra.run.smithi104.stdout: "mempool": { 2024-06-09T22:31:43.931 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_bytes": 0, 2024-06-09T22:31:43.931 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_items": 0, 2024-06-09T22:31:43.931 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_bytes": 0, 2024-06-09T22:31:43.931 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_items": 0, 2024-06-09T22:31:43.931 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_bytes": 0, 2024-06-09T22:31:43.931 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_items": 0, 2024-06-09T22:31:43.932 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-09T22:31:43.932 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_items": 0, 2024-06-09T22:31:43.932 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-09T22:31:43.932 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_items": 0, 2024-06-09T22:31:43.932 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_bytes": 0, 2024-06-09T22:31:43.932 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_items": 0, 2024-06-09T22:31:43.932 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-09T22:31:43.932 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_items": 0, 2024-06-09T22:31:43.933 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_bytes": 0, 2024-06-09T22:31:43.933 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_items": 0, 2024-06-09T22:31:43.933 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_bytes": 0, 2024-06-09T22:31:43.933 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_items": 0, 2024-06-09T22:31:43.933 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-09T22:31:43.933 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_items": 0, 2024-06-09T22:31:43.933 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-09T22:31:43.933 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_items": 0, 2024-06-09T22:31:43.933 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_bytes": 0, 2024-06-09T22:31:43.934 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_items": 0, 2024-06-09T22:31:43.934 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_bytes": 0, 2024-06-09T22:31:43.934 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_items": 0, 2024-06-09T22:31:43.934 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-09T22:31:43.934 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_items": 0, 2024-06-09T22:31:43.934 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_bytes": 0, 2024-06-09T22:31:43.934 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_items": 0, 2024-06-09T22:31:43.934 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_bytes": 0, 2024-06-09T22:31:43.935 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_items": 0, 2024-06-09T22:31:43.935 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_bytes": 0, 2024-06-09T22:31:43.935 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_items": 0, 2024-06-09T22:31:43.935 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_bytes": 0, 2024-06-09T22:31:43.935 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_items": 0, 2024-06-09T22:31:43.935 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_bytes": 198236, 2024-06-09T22:31:43.935 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_items": 57, 2024-06-09T22:31:43.935 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_bytes": 0, 2024-06-09T22:31:43.935 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_items": 0, 2024-06-09T22:31:43.935 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_bytes": 0, 2024-06-09T22:31:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_items": 0, 2024-06-09T22:31:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_bytes": 0, 2024-06-09T22:31:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_items": 0, 2024-06-09T22:31:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_bytes": 0, 2024-06-09T22:31:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_items": 0, 2024-06-09T22:31:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_bytes": 9512, 2024-06-09T22:31:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_items": 182, 2024-06-09T22:31:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_bytes": 0, 2024-06-09T22:31:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_items": 0, 2024-06-09T22:31:43.937 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_bytes": 0, 2024-06-09T22:31:43.937 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_items": 0, 2024-06-09T22:31:43.937 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_bytes": 8464, 2024-06-09T22:31:43.937 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_items": 13, 2024-06-09T22:31:43.937 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_bytes": 0, 2024-06-09T22:31:43.937 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_items": 0, 2024-06-09T22:31:43.937 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_bytes": 0, 2024-06-09T22:31:43.937 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_items": 0 2024-06-09T22:31:43.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.938 INFO:teuthology.orchestra.run.smithi104.stdout: "objecter": { 2024-06-09T22:31:43.938 INFO:teuthology.orchestra.run.smithi104.stdout: "op_active": 0, 2024-06-09T22:31:43.938 INFO:teuthology.orchestra.run.smithi104.stdout: "op_laggy": 0, 2024-06-09T22:31:43.938 INFO:teuthology.orchestra.run.smithi104.stdout: "op_send": 2319, 2024-06-09T22:31:43.938 INFO:teuthology.orchestra.run.smithi104.stdout: "op_send_bytes": 21, 2024-06-09T22:31:43.938 INFO:teuthology.orchestra.run.smithi104.stdout: "op_resend": 3, 2024-06-09T22:31:43.939 INFO:teuthology.orchestra.run.smithi104.stdout: "op_reply": 2319, 2024-06-09T22:31:43.939 INFO:teuthology.orchestra.run.smithi104.stdout: "op_latency": { 2024-06-09T22:31:43.939 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 2319, 2024-06-09T22:31:43.939 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 1.120987516, 2024-06-09T22:31:43.939 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000483392 2024-06-09T22:31:43.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.939 INFO:teuthology.orchestra.run.smithi104.stdout: "op_inflight": 0, 2024-06-09T22:31:43.939 INFO:teuthology.orchestra.run.smithi104.stdout: "oplen_avg": { 2024-06-09T22:31:43.939 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 2319, 2024-06-09T22:31:43.940 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 2320 2024-06-09T22:31:43.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.940 INFO:teuthology.orchestra.run.smithi104.stdout: "op": 2319, 2024-06-09T22:31:43.940 INFO:teuthology.orchestra.run.smithi104.stdout: "op_r": 2319, 2024-06-09T22:31:43.940 INFO:teuthology.orchestra.run.smithi104.stdout: "op_w": 0, 2024-06-09T22:31:43.940 INFO:teuthology.orchestra.run.smithi104.stdout: "op_rmw": 0, 2024-06-09T22:31:43.940 INFO:teuthology.orchestra.run.smithi104.stdout: "op_pg": 0, 2024-06-09T22:31:43.940 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_stat": 1477, 2024-06-09T22:31:43.941 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_create": 0, 2024-06-09T22:31:43.941 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_read": 840, 2024-06-09T22:31:43.941 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_write": 0, 2024-06-09T22:31:43.941 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_writefull": 0, 2024-06-09T22:31:43.941 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_writesame": 0, 2024-06-09T22:31:43.941 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_append": 0, 2024-06-09T22:31:43.941 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_zero": 0, 2024-06-09T22:31:43.941 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_truncate": 0, 2024-06-09T22:31:43.941 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_delete": 0, 2024-06-09T22:31:43.942 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_mapext": 0, 2024-06-09T22:31:43.942 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_sparse_read": 0, 2024-06-09T22:31:43.942 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_clonerange": 0, 2024-06-09T22:31:43.942 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_getxattr": 1, 2024-06-09T22:31:43.942 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_setxattr": 0, 2024-06-09T22:31:43.942 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_cmpxattr": 0, 2024-06-09T22:31:43.942 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_rmxattr": 0, 2024-06-09T22:31:43.943 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_resetxattrs": 0, 2024-06-09T22:31:43.943 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_call": 0, 2024-06-09T22:31:43.943 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_watch": 0, 2024-06-09T22:31:43.943 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_notify": 0, 2024-06-09T22:31:43.943 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_src_cmpxattr": 0, 2024-06-09T22:31:43.943 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_pgls": 0, 2024-06-09T22:31:43.943 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_pgls_filter": 0, 2024-06-09T22:31:43.943 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_other": 0, 2024-06-09T22:31:43.944 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_active": 0, 2024-06-09T22:31:43.944 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_send": 0, 2024-06-09T22:31:43.944 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_resend": 0, 2024-06-09T22:31:43.944 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_ping": 0, 2024-06-09T22:31:43.944 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_active": 0, 2024-06-09T22:31:43.944 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_send": 0, 2024-06-09T22:31:43.944 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_resend": 0, 2024-06-09T22:31:43.944 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_active": 0, 2024-06-09T22:31:43.945 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_send": 0, 2024-06-09T22:31:43.945 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_resend": 0, 2024-06-09T22:31:43.945 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_active": 0, 2024-06-09T22:31:43.945 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_send": 0, 2024-06-09T22:31:43.945 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_resend": 0, 2024-06-09T22:31:43.945 INFO:teuthology.orchestra.run.smithi104.stdout: "command_active": 0, 2024-06-09T22:31:43.945 INFO:teuthology.orchestra.run.smithi104.stdout: "command_send": 0, 2024-06-09T22:31:43.945 INFO:teuthology.orchestra.run.smithi104.stdout: "command_resend": 0, 2024-06-09T22:31:43.945 INFO:teuthology.orchestra.run.smithi104.stdout: "map_epoch": 108, 2024-06-09T22:31:43.946 INFO:teuthology.orchestra.run.smithi104.stdout: "map_full": 0, 2024-06-09T22:31:43.946 INFO:teuthology.orchestra.run.smithi104.stdout: "map_inc": 11, 2024-06-09T22:31:43.946 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_sessions": 6, 2024-06-09T22:31:43.946 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_session_open": 6, 2024-06-09T22:31:43.946 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_session_close": 0, 2024-06-09T22:31:43.946 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_laggy": 0, 2024-06-09T22:31:43.946 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_wr": 0, 2024-06-09T22:31:43.947 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_rd": 2, 2024-06-09T22:31:43.947 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_del": 0 2024-06-09T22:31:43.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.947 INFO:teuthology.orchestra.run.smithi104.stdout: "oft": { 2024-06-09T22:31:43.947 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_objs": 0, 2024-06-09T22:31:43.947 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_kv_pairs": 0, 2024-06-09T22:31:43.947 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_updates": 0, 2024-06-09T22:31:43.947 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_removes": 0 2024-06-09T22:31:43.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.948 INFO:teuthology.orchestra.run.smithi104.stdout: "osd-slow-ops": { 2024-06-09T22:31:43.948 INFO:teuthology.orchestra.run.smithi104.stdout: "slow_ops_count": 0 2024-06-09T22:31:43.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.948 INFO:teuthology.orchestra.run.smithi104.stdout: "purge_queue": { 2024-06-09T22:31:43.948 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_ops": 0, 2024-06-09T22:31:43.948 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_ops_high_water": 0, 2024-06-09T22:31:43.948 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing": 0, 2024-06-09T22:31:43.949 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_high_water": 0, 2024-06-09T22:31:43.949 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executed_ops": 0, 2024-06-09T22:31:43.949 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executed": 0, 2024-06-09T22:31:43.949 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_item_in_journal": 0 2024-06-09T22:31:43.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.949 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-09T22:31:43.949 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-06-09T22:31:43.949 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-06-09T22:31:43.949 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-06-09T22:31:43.950 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 2599, 2024-06-09T22:31:43.950 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 1925871, 2024-06-09T22:31:43.950 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:43.950 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 2599, 2024-06-09T22:31:43.950 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-06-09T22:31:43.950 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-06-09T22:31:43.950 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 2599, 2024-06-09T22:31:43.951 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 1925871, 2024-06-09T22:31:43.951 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-06-09T22:31:43.951 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.951 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.951 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.951 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:43.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.951 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-objecter_bytes": { 2024-06-09T22:31:43.952 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-06-09T22:31:43.952 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-06-09T22:31:43.952 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-06-09T22:31:43.952 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-06-09T22:31:43.952 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-06-09T22:31:43.952 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:43.952 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:43.952 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 2319, 2024-06-09T22:31:43.953 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 1822, 2024-06-09T22:31:43.953 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 2319, 2024-06-09T22:31:43.953 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 1822, 2024-06-09T22:31:43.953 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-06-09T22:31:43.953 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.953 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.953 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.953 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:43.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.954 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-objecter_ops": { 2024-06-09T22:31:43.954 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-06-09T22:31:43.954 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 1024, 2024-06-09T22:31:43.954 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-06-09T22:31:43.954 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-06-09T22:31:43.954 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-06-09T22:31:43.954 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:43.955 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:43.955 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 2319, 2024-06-09T22:31:43.955 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 2319, 2024-06-09T22:31:43.955 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 2319, 2024-06-09T22:31:43.955 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 2319, 2024-06-09T22:31:43.955 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-06-09T22:31:43.955 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.955 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.955 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.956 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:43.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.956 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-write_buf_throttle": { 2024-06-09T22:31:43.956 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-06-09T22:31:43.956 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 3758096384, 2024-06-09T22:31:43.956 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-06-09T22:31:43.956 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-06-09T22:31:43.956 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-06-09T22:31:43.957 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:43.957 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:43.957 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-06-09T22:31:43.957 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-06-09T22:31:43.957 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 0, 2024-06-09T22:31:43.957 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 0, 2024-06-09T22:31:43.957 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-06-09T22:31:43.957 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.958 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.958 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.958 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:43.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:43.958 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-write_buf_throttle-0x5601104cc6e0": { 2024-06-09T22:31:43.958 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-06-09T22:31:43.958 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 3758096384, 2024-06-09T22:31:43.958 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-06-09T22:31:43.959 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-06-09T22:31:43.959 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-06-09T22:31:43.959 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:43.959 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:43.959 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-06-09T22:31:43.959 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-06-09T22:31:43.959 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 0, 2024-06-09T22:31:43.959 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 0, 2024-06-09T22:31:43.959 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-06-09T22:31:43.960 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:43.960 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:43.960 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:43.960 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:43.960 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:43.960 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:31:44.666 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-06-09T22:31:44.666 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-06-09T22:31:44.666 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-06-09T22:31:44.667 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-06-09T22:31:44.667 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-06-09T22:31:44.667 DEBUG:tasks.check_counter:Getting stats from j 2024-06-09T22:31:44.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mds.j perf dump 2024-06-09T22:31:44.920 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:31:45.905 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:45 smithi104 ceph-mon[32389]: pgmap v334: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:45.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:45 smithi153 ceph-mon[36599]: pgmap v334: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:46.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:45 smithi194 ceph-mon[36856]: pgmap v334: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:46.684 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-06-09T22:31:46.684 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-0": { 2024-06-09T22:31:46.684 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 0, 2024-06-09T22:31:46.684 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 2, 2024-06-09T22:31:46.685 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 0, 2024-06-09T22:31:46.685 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 0, 2024-06-09T22:31:46.685 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 4, 2024-06-09T22:31:46.685 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 0, 2024-06-09T22:31:46.685 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.001109410, 2024-06-09T22:31:46.685 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.000000000, 2024-06-09T22:31:46.685 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.000288043, 2024-06-09T22:31:46.685 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-09T22:31:46.686 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:46.686 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:46.686 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:46.686 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:46.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:46.686 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:46.686 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:46.686 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:46.687 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:46.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:46.687 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-09T22:31:46.687 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 0 2024-06-09T22:31:46.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:46.687 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-1": { 2024-06-09T22:31:46.687 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 263, 2024-06-09T22:31:46.687 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 253, 2024-06-09T22:31:46.688 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 1455056, 2024-06-09T22:31:46.688 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 155920, 2024-06-09T22:31:46.688 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 2, 2024-06-09T22:31:46.688 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 1, 2024-06-09T22:31:46.688 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.047863432, 2024-06-09T22:31:46.688 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.015510665, 2024-06-09T22:31:46.688 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.036231874, 2024-06-09T22:31:46.688 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.004407260, 2024-06-09T22:31:46.689 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:46.689 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 252, 2024-06-09T22:31:46.689 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.010005466, 2024-06-09T22:31:46.689 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000039704 2024-06-09T22:31:46.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:46.689 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:46.689 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:46.689 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:46.690 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:46.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:46.690 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 1455056, 2024-06-09T22:31:46.690 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 155920 2024-06-09T22:31:46.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:46.690 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-2": { 2024-06-09T22:31:46.690 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 6, 2024-06-09T22:31:46.690 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 195, 2024-06-09T22:31:46.691 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 3344, 2024-06-09T22:31:46.691 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 190240, 2024-06-09T22:31:46.691 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 5, 2024-06-09T22:31:46.691 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 3, 2024-06-09T22:31:46.691 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.015789598, 2024-06-09T22:31:46.691 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.010986031, 2024-06-09T22:31:46.691 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.002206152, 2024-06-09T22:31:46.691 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-09T22:31:46.692 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:46.692 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 193, 2024-06-09T22:31:46.692 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.012835124, 2024-06-09T22:31:46.692 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000066503 2024-06-09T22:31:46.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:46.692 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:46.692 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:46.692 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:46.693 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:46.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:46.693 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-06-09T22:31:46.693 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 190240 2024-06-09T22:31:46.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:46.693 INFO:teuthology.orchestra.run.smithi104.stdout: "cct": { 2024-06-09T22:31:46.693 INFO:teuthology.orchestra.run.smithi104.stdout: "total_workers": 0, 2024-06-09T22:31:46.693 INFO:teuthology.orchestra.run.smithi104.stdout: "unhealthy_workers": 0 2024-06-09T22:31:46.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:46.694 INFO:teuthology.orchestra.run.smithi104.stdout: "mempool": { 2024-06-09T22:31:46.694 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_bytes": 0, 2024-06-09T22:31:46.694 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_items": 0, 2024-06-09T22:31:46.694 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_bytes": 0, 2024-06-09T22:31:46.694 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_items": 0, 2024-06-09T22:31:46.694 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_bytes": 0, 2024-06-09T22:31:46.694 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_items": 0, 2024-06-09T22:31:46.695 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-09T22:31:46.695 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_items": 0, 2024-06-09T22:31:46.695 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-09T22:31:46.695 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_items": 0, 2024-06-09T22:31:46.695 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_bytes": 0, 2024-06-09T22:31:46.695 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_items": 0, 2024-06-09T22:31:46.695 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-09T22:31:46.695 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_items": 0, 2024-06-09T22:31:46.695 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_bytes": 0, 2024-06-09T22:31:46.696 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_items": 0, 2024-06-09T22:31:46.696 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_bytes": 0, 2024-06-09T22:31:46.696 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_items": 0, 2024-06-09T22:31:46.696 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-09T22:31:46.696 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_items": 0, 2024-06-09T22:31:46.696 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-09T22:31:46.696 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_items": 0, 2024-06-09T22:31:46.697 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_bytes": 0, 2024-06-09T22:31:46.697 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_items": 0, 2024-06-09T22:31:46.697 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_bytes": 0, 2024-06-09T22:31:46.697 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_items": 0, 2024-06-09T22:31:46.697 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-09T22:31:46.697 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_items": 0, 2024-06-09T22:31:46.697 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_bytes": 0, 2024-06-09T22:31:46.697 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_items": 0, 2024-06-09T22:31:46.698 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_bytes": 0, 2024-06-09T22:31:46.698 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_items": 0, 2024-06-09T22:31:46.698 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_bytes": 0, 2024-06-09T22:31:46.698 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_items": 0, 2024-06-09T22:31:46.698 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_bytes": 0, 2024-06-09T22:31:46.698 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_items": 0, 2024-06-09T22:31:46.698 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_bytes": 148754, 2024-06-09T22:31:46.699 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_items": 31, 2024-06-09T22:31:46.699 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_bytes": 0, 2024-06-09T22:31:46.699 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_items": 0, 2024-06-09T22:31:46.699 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_bytes": 0, 2024-06-09T22:31:46.699 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_items": 0, 2024-06-09T22:31:46.699 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_bytes": 0, 2024-06-09T22:31:46.699 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_items": 0, 2024-06-09T22:31:46.699 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_bytes": 0, 2024-06-09T22:31:46.700 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_items": 0, 2024-06-09T22:31:46.700 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_bytes": 0, 2024-06-09T22:31:46.700 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_items": 0, 2024-06-09T22:31:46.700 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_bytes": 0, 2024-06-09T22:31:46.700 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_items": 0, 2024-06-09T22:31:46.700 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_bytes": 0, 2024-06-09T22:31:46.700 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_items": 0, 2024-06-09T22:31:46.700 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_bytes": 928, 2024-06-09T22:31:46.701 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_items": 2, 2024-06-09T22:31:46.701 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_bytes": 0, 2024-06-09T22:31:46.701 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_items": 0, 2024-06-09T22:31:46.701 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_bytes": 0, 2024-06-09T22:31:46.701 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_items": 0 2024-06-09T22:31:46.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-06-09T22:31:46.701 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-09T22:31:46.701 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-06-09T22:31:46.702 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-06-09T22:31:46.702 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-06-09T22:31:46.702 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 265, 2024-06-09T22:31:46.702 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 1417975, 2024-06-09T22:31:46.702 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:46.702 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 265, 2024-06-09T22:31:46.702 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-06-09T22:31:46.702 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-06-09T22:31:46.703 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 265, 2024-06-09T22:31:46.703 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 1417975, 2024-06-09T22:31:46.703 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-06-09T22:31:46.703 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-06-09T22:31:46.703 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-06-09T22:31:46.703 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-06-09T22:31:46.703 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:46.703 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-06-09T22:31:46.704 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-06-09T22:31:47.414 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-06-09T22:31:47.414 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-06-09T22:31:47.414 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-06-09T22:31:47.414 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-06-09T22:31:47.415 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-06-09T22:31:47.415 DEBUG:tasks.check_counter:Getting stats from b 2024-06-09T22:31:47.415 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mds.b perf dump 2024-06-09T22:31:47.669 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.b/config 2024-06-09T22:31:47.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:47 smithi153 ceph-mon[36599]: pgmap v335: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-06-09T22:31:47.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:47 smithi104 ceph-mon[32389]: pgmap v335: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-06-09T22:31:48.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:47 smithi194 ceph-mon[36856]: pgmap v335: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-06-09T22:31:49.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:49 smithi153 ceph-mon[36599]: pgmap v336: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-06-09T22:31:49.620 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-06-09T22:31:49.620 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-06-09T22:31:49.620 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 72431, 2024-06-09T22:31:49.620 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 58645, 2024-06-09T22:31:49.620 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 29852626, 2024-06-09T22:31:49.620 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 17928415, 2024-06-09T22:31:49.620 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 28, 2024-06-09T22:31:49.621 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 8, 2024-06-09T22:31:49.621 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 3.400102426, 2024-06-09T22:31:49.621 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 1.357397014, 2024-06-09T22:31:49.621 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 2.492938827, 2024-06-09T22:31:49.621 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.136874308, 2024-06-09T22:31:49.621 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:49.621 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 58644, 2024-06-09T22:31:49.621 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.995553956, 2024-06-09T22:31:49.622 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000016976 2024-06-09T22:31:49.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.622 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:49.622 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 122437, 2024-06-09T22:31:49.622 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.015416603, 2024-06-09T22:31:49.622 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000125 2024-06-09T22:31:49.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.623 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-06-09T22:31:49.623 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 448 2024-06-09T22:31:49.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.623 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-06-09T22:31:49.623 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 43749, 2024-06-09T22:31:49.623 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 66598, 2024-06-09T22:31:49.623 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 16574671, 2024-06-09T22:31:49.623 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 67564167, 2024-06-09T22:31:49.624 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 18, 2024-06-09T22:31:49.624 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 7, 2024-06-09T22:31:49.624 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 2.874069043, 2024-06-09T22:31:49.624 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 1.296141958, 2024-06-09T22:31:49.624 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 1.602557850, 2024-06-09T22:31:49.624 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.218126917, 2024-06-09T22:31:49.624 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:49.624 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 66597, 2024-06-09T22:31:49.624 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 1.370464666, 2024-06-09T22:31:49.624 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000020578 2024-06-09T22:31:49.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.625 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:49.625 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 89781, 2024-06-09T22:31:49.625 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.011454667, 2024-06-09T22:31:49.625 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000127 2024-06-09T22:31:49.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.625 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 14901024, 2024-06-09T22:31:49.625 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 46679872 2024-06-09T22:31:49.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.626 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-06-09T22:31:49.626 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 5244, 2024-06-09T22:31:49.626 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 5438, 2024-06-09T22:31:49.626 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1327419, 2024-06-09T22:31:49.626 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 14778569, 2024-06-09T22:31:49.626 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 13, 2024-06-09T22:31:49.626 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 7, 2024-06-09T22:31:49.627 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.364989760, 2024-06-09T22:31:49.627 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.116043917, 2024-06-09T22:31:49.627 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.090888376, 2024-06-09T22:31:49.627 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.130891973, 2024-06-09T22:31:49.627 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:49.627 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 5435, 2024-06-09T22:31:49.627 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.215632260, 2024-06-09T22:31:49.627 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000039674 2024-06-09T22:31:49.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.628 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:49.628 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 143, 2024-06-09T22:31:49.628 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000021879, 2024-06-09T22:31:49.628 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000153 2024-06-09T22:31:49.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.628 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1888, 2024-06-09T22:31:49.628 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 484992 2024-06-09T22:31:49.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.629 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-06-09T22:31:49.629 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-06-09T22:31:49.629 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-06-09T22:31:49.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.629 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-06-09T22:31:49.629 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-06-09T22:31:49.629 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-06-09T22:31:49.629 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 11578, 2024-06-09T22:31:49.629 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 6.821831736, 2024-06-09T22:31:49.630 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000589206 2024-06-09T22:31:49.630 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:49.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.630 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-06-09T22:31:49.630 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-06-09T22:31:49.630 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-06-09T22:31:49.630 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 9055, 2024-06-09T22:31:49.630 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.607795851, 2024-06-09T22:31:49.630 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000067122 2024-06-09T22:31:49.631 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:49.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.631 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-06-09T22:31:49.631 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 29921, 2024-06-09T22:31:49.631 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 28457, 2024-06-09T22:31:49.631 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-06-09T22:31:49.631 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 28457, 2024-06-09T22:31:49.631 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 56.006925168, 2024-06-09T22:31:49.632 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001968124 2024-06-09T22:31:49.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.632 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-06-09T22:31:49.632 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 1464, 2024-06-09T22:31:49.632 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-06-09T22:31:49.632 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-06-09T22:31:49.632 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 11, 2024-06-09T22:31:49.632 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-06-09T22:31:49.632 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-06-09T22:31:49.633 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 134, 2024-06-09T22:31:49.633 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 16, 2024-06-09T22:31:49.633 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 118, 2024-06-09T22:31:49.633 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-06-09T22:31:49.633 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 110, 2024-06-09T22:31:49.633 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-06-09T22:31:49.633 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 2, 2024-06-09T22:31:49.633 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 2, 2024-06-09T22:31:49.634 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 5, 2024-06-09T22:31:49.634 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 62817, 2024-06-09T22:31:49.634 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 25762, 2024-06-09T22:31:49.634 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 1464, 2024-06-09T22:31:49.634 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 5, 2024-06-09T22:31:49.634 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-06-09T22:31:49.634 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-06-09T22:31:49.634 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 522, 2024-06-09T22:31:49.635 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-06-09T22:31:49.635 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-06-09T22:31:49.635 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-06-09T22:31:49.635 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-06-09T22:31:49.635 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 2, 2024-06-09T22:31:49.635 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 51, 2024-06-09T22:31:49.635 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-06-09T22:31:49.635 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 60, 2024-06-09T22:31:49.636 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-06-09T22:31:49.636 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 67, 2024-06-09T22:31:49.636 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 21176629, 2024-06-09T22:31:49.636 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 6, 2024-06-09T22:31:49.636 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 60, 2024-06-09T22:31:49.636 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-06-09T22:31:49.636 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-06-09T22:31:49.636 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-06-09T22:31:49.637 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-06-09T22:31:49.637 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-06-09T22:31:49.637 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-06-09T22:31:49.637 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 60, 2024-06-09T22:31:49.637 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-06-09T22:31:49.637 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 172, 2024-06-09T22:31:49.637 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 15825, 2024-06-09T22:31:49.637 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-06-09T22:31:49.637 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-09T22:31:49.638 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-09T22:31:49.638 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 2921, 2024-06-09T22:31:49.638 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 1376, 2024-06-09T22:31:49.638 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 1014, 2024-06-09T22:31:49.638 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 26499 2024-06-09T22:31:49.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.638 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-06-09T22:31:49.638 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 67, 2024-06-09T22:31:49.639 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 6, 2024-06-09T22:31:49.639 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 4, 2024-06-09T22:31:49.639 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 7, 2024-06-09T22:31:49.639 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 84, 2024-06-09T22:31:49.639 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-06-09T22:31:49.639 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-06-09T22:31:49.639 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-06-09T22:31:49.640 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 4611, 2024-06-09T22:31:49.640 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 4611, 2024-06-09T22:31:49.640 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-06-09T22:31:49.640 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-06-09T22:31:49.640 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-06-09T22:31:49.640 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-06-09T22:31:49.640 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-06-09T22:31:49.640 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-06-09T22:31:49.641 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-06-09T22:31:49.641 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 5, 2024-06-09T22:31:49.641 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 79, 2024-06-09T22:31:49.641 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-06-09T22:31:49.641 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-06-09T22:31:49.641 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-06-09T22:31:49.641 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-06-09T22:31:49.641 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-06-09T22:31:49.641 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-06-09T22:31:49.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.642 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-06-09T22:31:49.642 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-06-09T22:31:49.642 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 22346, 2024-06-09T22:31:49.642 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-06-09T22:31:49.642 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-06-09T22:31:49.642 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 22346, 2024-06-09T22:31:49.642 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-06-09T22:31:49.643 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-06-09T22:31:49.643 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 26, 2024-06-09T22:31:49.643 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-06-09T22:31:49.643 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-06-09T22:31:49.643 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 26, 2024-06-09T22:31:49.643 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 3, 2024-06-09T22:31:49.643 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-06-09T22:31:49.643 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-06-09T22:31:49.644 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-06-09T22:31:49.644 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 39202605, 2024-06-09T22:31:49.644 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 0, 2024-06-09T22:31:49.644 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-06-09T22:31:49.644 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 5616, 2024-06-09T22:31:49.644 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 9.189961122, 2024-06-09T22:31:49.644 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001636389 2024-06-09T22:31:49.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.645 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 0 2024-06-09T22:31:49.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.645 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-06-09T22:31:49.645 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 86, 2024-06-09T22:31:49.645 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 4697, 2024-06-09T22:31:49.645 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 4611, 2024-06-09T22:31:49.645 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 32, 2024-06-09T22:31:49.645 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 34, 2024-06-09T22:31:49.645 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 2, 2024-06-09T22:31:49.646 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 134, 2024-06-09T22:31:49.646 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 7663, 2024-06-09T22:31:49.646 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 7529, 2024-06-09T22:31:49.646 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 2, 2024-06-09T22:31:49.646 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 4706, 2024-06-09T22:31:49.646 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 4704, 2024-06-09T22:31:49.646 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 63024, 2024-06-09T22:31:49.647 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 223612 2024-06-09T22:31:49.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.647 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-06-09T22:31:49.647 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 53116, 2024-06-09T22:31:49.647 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-06-09T22:31:49.647 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 29921, 2024-06-09T22:31:49.647 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 372, 2024-06-09T22:31:49.647 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 9748, 2024-06-09T22:31:49.648 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-06-09T22:31:49.648 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4611, 2024-06-09T22:31:49.648 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 36.378064329, 2024-06-09T22:31:49.648 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.007889408 2024-06-09T22:31:49.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.648 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-06-09T22:31:49.648 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 86, 2024-06-09T22:31:49.648 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.055010914, 2024-06-09T22:31:49.649 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000639661 2024-06-09T22:31:49.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.649 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-06-09T22:31:49.649 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.649 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.649 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.649 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-06-09T22:31:49.649 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.650 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.650 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.650 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-06-09T22:31:49.650 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 7527, 2024-06-09T22:31:49.650 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2.516026600, 2024-06-09T22:31:49.650 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000334266 2024-06-09T22:31:49.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.651 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-06-09T22:31:49.651 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.651 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.651 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.651 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-06-09T22:31:49.651 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.651 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.651 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.652 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-06-09T22:31:49.652 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.652 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.652 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.652 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-06-09T22:31:49.652 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.653 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.653 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.653 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-06-09T22:31:49.653 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.653 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.653 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.653 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-06-09T22:31:49.654 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.654 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.654 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.655 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-06-09T22:31:49.655 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.655 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.655 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.655 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-06-09T22:31:49.655 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.656 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.656 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.656 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-06-09T22:31:49.656 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.656 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.657 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.657 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-06-09T22:31:49.657 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.657 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.657 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.657 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-06-09T22:31:49.657 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 11093, 2024-06-09T22:31:49.658 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 4.124976998, 2024-06-09T22:31:49.658 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000371854 2024-06-09T22:31:49.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.658 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-06-09T22:31:49.658 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 522, 2024-06-09T22:31:49.658 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 1.084812661, 2024-06-09T22:31:49.658 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002078185 2024-06-09T22:31:49.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.659 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-06-09T22:31:49.659 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.659 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.659 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.659 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-06-09T22:31:49.659 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.659 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.659 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.660 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-06-09T22:31:49.660 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.660 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.660 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.660 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-06-09T22:31:49.660 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.660 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.661 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.661 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-06-09T22:31:49.661 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.661 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.661 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.661 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-06-09T22:31:49.661 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 7, 2024-06-09T22:31:49.662 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.024175033, 2024-06-09T22:31:49.662 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.003453576 2024-06-09T22:31:49.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.662 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-06-09T22:31:49.662 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.662 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.662 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.663 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-06-09T22:31:49.663 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.663 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.663 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.663 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-06-09T22:31:49.663 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.663 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.663 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.664 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-06-09T22:31:49.664 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.664 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.664 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.664 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-06-09T22:31:49.664 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.664 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.665 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.665 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-06-09T22:31:49.665 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4611, 2024-06-09T22:31:49.665 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 11.823858633, 2024-06-09T22:31:49.665 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002564272 2024-06-09T22:31:49.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.665 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-06-09T22:31:49.665 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-06-09T22:31:49.666 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-06-09T22:31:49.666 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.666 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.666 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.666 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:49.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.666 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-06-09T22:31:49.666 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 2, 2024-06-09T22:31:49.667 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 4, 2024-06-09T22:31:49.667 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 2, 2024-06-09T22:31:49.667 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 2, 2024-06-09T22:31:49.667 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-06-09T22:31:49.667 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 9799, 2024-06-09T22:31:49.667 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 4899, 2024-06-09T22:31:49.667 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 584, 2024-06-09T22:31:49.667 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-06-09T22:31:49.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.668 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-06-09T22:31:49.668 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-06-09T22:31:49.668 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-06-09T22:31:49.668 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-06-09T22:31:49.668 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-06-09T22:31:49.668 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-06-09T22:31:49.668 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-06-09T22:31:49.669 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-09T22:31:49.669 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-06-09T22:31:49.669 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-09T22:31:49.669 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-06-09T22:31:49.669 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-06-09T22:31:49.670 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-06-09T22:31:49.670 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-09T22:31:49.670 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-06-09T22:31:49.671 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-06-09T22:31:49.671 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-06-09T22:31:49.671 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-06-09T22:31:49.671 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-06-09T22:31:49.671 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-09T22:31:49.672 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-06-09T22:31:49.672 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-09T22:31:49.672 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-06-09T22:31:49.672 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-06-09T22:31:49.673 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-06-09T22:31:49.673 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-06-09T22:31:49.673 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-06-09T22:31:49.673 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-09T22:31:49.674 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-06-09T22:31:49.674 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-06-09T22:31:49.674 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-06-09T22:31:49.675 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-06-09T22:31:49.675 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-06-09T22:31:49.675 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-06-09T22:31:49.675 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-06-09T22:31:49.676 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-06-09T22:31:49.676 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-06-09T22:31:49.676 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 306256, 2024-06-09T22:31:49.676 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 153, 2024-06-09T22:31:49.677 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-06-09T22:31:49.677 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-06-09T22:31:49.677 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-06-09T22:31:49.677 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-06-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-06-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-06-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-06-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-06-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9512, 2024-06-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 182, 2024-06-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-06-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-06-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-06-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-06-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 379130, 2024-06-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 2515, 2024-06-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-06-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-06-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-06-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-06-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-06-09T22:31:49.683 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-06-09T22:31:49.683 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-06-09T22:31:49.683 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 15910, 2024-06-09T22:31:49.683 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 35963631, 2024-06-09T22:31:49.683 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 33, 2024-06-09T22:31:49.684 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 15910, 2024-06-09T22:31:49.684 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-06-09T22:31:49.684 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 15910, 2024-06-09T22:31:49.685 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 39.108295446, 2024-06-09T22:31:49.685 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002458095 2024-06-09T22:31:49.685 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.685 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-06-09T22:31:49.686 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-06-09T22:31:49.686 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 15910, 2024-06-09T22:31:49.686 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 16290 2024-06-09T22:31:49.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.687 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 15910, 2024-06-09T22:31:49.687 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 342, 2024-06-09T22:31:49.687 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 15568, 2024-06-09T22:31:49.687 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-06-09T22:31:49.688 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-06-09T22:31:49.688 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 0, 2024-06-09T22:31:49.688 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 10, 2024-06-09T22:31:49.689 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 282, 2024-06-09T22:31:49.689 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 5898, 2024-06-09T22:31:49.689 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 170, 2024-06-09T22:31:49.689 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-06-09T22:31:49.690 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-06-09T22:31:49.690 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-06-09T22:31:49.690 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-06-09T22:31:49.690 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 9242, 2024-06-09T22:31:49.691 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-06-09T22:31:49.691 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-06-09T22:31:49.691 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-06-09T22:31:49.691 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 60, 2024-06-09T22:31:49.692 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 20, 2024-06-09T22:31:49.692 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-06-09T22:31:49.692 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-06-09T22:31:49.692 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-06-09T22:31:49.693 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-06-09T22:31:49.693 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-06-09T22:31:49.693 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-06-09T22:31:49.693 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-06-09T22:31:49.694 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-06-09T22:31:49.694 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-06-09T22:31:49.694 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-06-09T22:31:49.694 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-06-09T22:31:49.694 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-06-09T22:31:49.695 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-06-09T22:31:49.695 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-06-09T22:31:49.695 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-06-09T22:31:49.695 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-06-09T22:31:49.696 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-06-09T22:31:49.696 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-06-09T22:31:49.696 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-06-09T22:31:49.696 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-06-09T22:31:49.696 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-06-09T22:31:49.697 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-06-09T22:31:49.697 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-06-09T22:31:49.697 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-06-09T22:31:49.697 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-06-09T22:31:49.697 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-06-09T22:31:49.697 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 108, 2024-06-09T22:31:49.697 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-06-09T22:31:49.697 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 11, 2024-06-09T22:31:49.698 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 12, 2024-06-09T22:31:49.698 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 12, 2024-06-09T22:31:49.698 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-06-09T22:31:49.698 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-06-09T22:31:49.698 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 450, 2024-06-09T22:31:49.698 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 0, 2024-06-09T22:31:49.698 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 158 2024-06-09T22:31:49.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.699 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-06-09T22:31:49.699 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 1, 2024-06-09T22:31:49.699 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-06-09T22:31:49.699 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-06-09T22:31:49.699 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-06-09T22:31:49.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.700 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-06-09T22:31:49.700 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 85 2024-06-09T22:31:49.700 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.700 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-06-09T22:31:49.700 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-06-09T22:31:49.700 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 10, 2024-06-09T22:31:49.700 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-06-09T22:31:49.700 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 5, 2024-06-09T22:31:49.701 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 9222, 2024-06-09T22:31:49.701 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 4611, 2024-06-09T22:31:49.701 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-06-09T22:31:49.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.701 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-09T22:31:49.701 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:49.701 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-06-09T22:31:49.701 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:49.702 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 121420, 2024-06-09T22:31:49.702 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 35181444, 2024-06-09T22:31:49.702 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:49.702 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 121420, 2024-06-09T22:31:49.702 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-06-09T22:31:49.703 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-06-09T22:31:49.703 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 121420, 2024-06-09T22:31:49.703 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 35181444, 2024-06-09T22:31:49.703 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:49.703 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.703 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.703 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.703 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:49.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.704 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-06-09T22:31:49.704 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:49.704 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-06-09T22:31:49.704 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:49.704 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:49.704 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:49.704 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:49.704 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:49.705 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 15910, 2024-06-09T22:31:49.705 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 36466390, 2024-06-09T22:31:49.705 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 15910, 2024-06-09T22:31:49.705 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 36466390, 2024-06-09T22:31:49.705 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:49.705 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.705 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.706 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.706 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:49.706 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.706 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-06-09T22:31:49.706 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:49.707 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-06-09T22:31:49.707 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:49.707 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:49.707 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:49.707 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:49.707 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:49.707 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 15910, 2024-06-09T22:31:49.707 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 15910, 2024-06-09T22:31:49.707 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 15910, 2024-06-09T22:31:49.708 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 15910, 2024-06-09T22:31:49.708 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:49.708 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.708 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.708 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.708 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:49.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.708 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-06-09T22:31:49.709 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:49.709 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-06-09T22:31:49.709 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:49.709 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 4611, 2024-06-09T22:31:49.709 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 502759, 2024-06-09T22:31:49.709 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:49.709 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 4611, 2024-06-09T22:31:49.709 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-06-09T22:31:49.710 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-06-09T22:31:49.710 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 282, 2024-06-09T22:31:49.710 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 502759, 2024-06-09T22:31:49.710 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:49.710 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.710 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.710 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.710 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:49.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:49.711 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x5595a5890500": { 2024-06-09T22:31:49.711 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:49.711 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-06-09T22:31:49.711 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:49.711 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 22346, 2024-06-09T22:31:49.711 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 35008301, 2024-06-09T22:31:49.711 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:49.711 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 22346, 2024-06-09T22:31:49.712 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-06-09T22:31:49.712 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-06-09T22:31:49.712 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 5616, 2024-06-09T22:31:49.712 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 35008301, 2024-06-09T22:31:49.712 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:49.712 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:49.712 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:49.713 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:49.713 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:49.713 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:49.713 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-06-09T22:31:50.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:49 smithi104 ceph-mon[32389]: pgmap v336: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-06-09T22:31:50.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:49 smithi194 ceph-mon[36856]: pgmap v336: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-06-09T22:31:50.424 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-06-09T22:31:50.425 INFO:tasks.check_counter:Daemon mds.b mds.imported=2 2024-06-09T22:31:50.425 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-06-09T22:31:50.425 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=0 2024-06-09T22:31:50.425 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-06-09T22:31:50.425 DEBUG:tasks.check_counter:Getting stats from e 2024-06-09T22:31:50.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mds.e perf dump 2024-06-09T22:31:50.667 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.b/config 2024-06-09T22:31:51.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:51 smithi153 ceph-mon[36599]: pgmap v337: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:51.834 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:51 smithi104 ceph-mon[32389]: pgmap v337: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:52.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:51 smithi194 ceph-mon[36856]: pgmap v337: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:52.347 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-06-09T22:31:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-06-09T22:31:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 2438, 2024-06-09T22:31:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 2438, 2024-06-09T22:31:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 693940, 2024-06-09T22:31:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 774467, 2024-06-09T22:31:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-06-09T22:31:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-06-09T22:31:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.167676277, 2024-06-09T22:31:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.055947620, 2024-06-09T22:31:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.040093403, 2024-06-09T22:31:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.058208604, 2024-06-09T22:31:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2437, 2024-06-09T22:31:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.066974172, 2024-06-09T22:31:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000027482 2024-06-09T22:31:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-06-09T22:31:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 448 2024-06-09T22:31:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-06-09T22:31:52.351 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 4, 2024-06-09T22:31:52.351 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 195, 2024-06-09T22:31:52.351 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 876, 2024-06-09T22:31:52.351 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 478922, 2024-06-09T22:31:52.351 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-06-09T22:31:52.351 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 4, 2024-06-09T22:31:52.351 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.012011896, 2024-06-09T22:31:52.351 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.007443397, 2024-06-09T22:31:52.352 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.002388553, 2024-06-09T22:31:52.352 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000107204, 2024-06-09T22:31:52.352 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:52.352 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 192, 2024-06-09T22:31:52.352 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.017038340, 2024-06-09T22:31:52.352 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000088741 2024-06-09T22:31:52.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.352 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:52.352 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.353 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.353 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.353 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-09T22:31:52.353 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 478224 2024-06-09T22:31:52.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.353 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-06-09T22:31:52.353 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 284, 2024-06-09T22:31:52.353 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 269, 2024-06-09T22:31:52.354 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1514108, 2024-06-09T22:31:52.354 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 158366, 2024-06-09T22:31:52.354 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 3, 2024-06-09T22:31:52.354 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-06-09T22:31:52.354 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.049642742, 2024-06-09T22:31:52.354 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.015878878, 2024-06-09T22:31:52.354 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.035550348, 2024-06-09T22:31:52.354 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.006400997, 2024-06-09T22:31:52.355 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:52.355 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 268, 2024-06-09T22:31:52.355 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.013987073, 2024-06-09T22:31:52.355 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000052190 2024-06-09T22:31:52.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.355 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:52.355 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.355 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.355 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.356 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1513840, 2024-06-09T22:31:52.356 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 158048 2024-06-09T22:31:52.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.356 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-06-09T22:31:52.356 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-06-09T22:31:52.356 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-06-09T22:31:52.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.357 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-06-09T22:31:52.357 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-06-09T22:31:52.357 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-06-09T22:31:52.357 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4116, 2024-06-09T22:31:52.357 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.092363928, 2024-06-09T22:31:52.357 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000022440 2024-06-09T22:31:52.357 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:52.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.358 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-06-09T22:31:52.358 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-06-09T22:31:52.358 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-06-09T22:31:52.358 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.358 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.358 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.358 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:52.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.359 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-06-09T22:31:52.359 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 0, 2024-06-09T22:31:52.359 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 0, 2024-06-09T22:31:52.359 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-06-09T22:31:52.359 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.359 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.359 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.360 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-06-09T22:31:52.360 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-06-09T22:31:52.360 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-06-09T22:31:52.360 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-06-09T22:31:52.360 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 0, 2024-06-09T22:31:52.360 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-06-09T22:31:52.360 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-06-09T22:31:52.360 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 0, 2024-06-09T22:31:52.360 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-06-09T22:31:52.361 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 0, 2024-06-09T22:31:52.361 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-06-09T22:31:52.361 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 0, 2024-06-09T22:31:52.361 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-06-09T22:31:52.361 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-06-09T22:31:52.361 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-06-09T22:31:52.361 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 1, 2024-06-09T22:31:52.361 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 0, 2024-06-09T22:31:52.361 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 0, 2024-06-09T22:31:52.362 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-06-09T22:31:52.362 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-06-09T22:31:52.362 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-06-09T22:31:52.362 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-06-09T22:31:52.362 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-06-09T22:31:52.362 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-06-09T22:31:52.362 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-06-09T22:31:52.363 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-06-09T22:31:52.363 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-06-09T22:31:52.363 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-06-09T22:31:52.363 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-06-09T22:31:52.363 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-06-09T22:31:52.363 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-06-09T22:31:52.363 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-06-09T22:31:52.363 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 0, 2024-06-09T22:31:52.364 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 0, 2024-06-09T22:31:52.364 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-06-09T22:31:52.364 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-06-09T22:31:52.364 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-06-09T22:31:52.364 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-06-09T22:31:52.364 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-06-09T22:31:52.364 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-06-09T22:31:52.364 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-06-09T22:31:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-06-09T22:31:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-06-09T22:31:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-06-09T22:31:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-06-09T22:31:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-06-09T22:31:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-06-09T22:31:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-09T22:31:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-09T22:31:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-06-09T22:31:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-06-09T22:31:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-06-09T22:31:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-06-09T22:31:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-06-09T22:31:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-06-09T22:31:52.366 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-06-09T22:31:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-06-09T22:31:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-06-09T22:31:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-06-09T22:31:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-06-09T22:31:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-06-09T22:31:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-06-09T22:31:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-06-09T22:31:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-06-09T22:31:52.367 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-06-09T22:31:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-06-09T22:31:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-06-09T22:31:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-06-09T22:31:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-06-09T22:31:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-06-09T22:31:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-06-09T22:31:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-06-09T22:31:52.368 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-06-09T22:31:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-06-09T22:31:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-06-09T22:31:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-06-09T22:31:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-06-09T22:31:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-06-09T22:31:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-06-09T22:31:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-06-09T22:31:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-06-09T22:31:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 0, 2024-06-09T22:31:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-06-09T22:31:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-06-09T22:31:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 9, 2024-06-09T22:31:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-06-09T22:31:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-06-09T22:31:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 0, 2024-06-09T22:31:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-06-09T22:31:52.370 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-06-09T22:31:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-06-09T22:31:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-06-09T22:31:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-06-09T22:31:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-06-09T22:31:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-06-09T22:31:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4196793, 2024-06-09T22:31:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 4196485, 2024-06-09T22:31:52.371 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-06-09T22:31:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 9 2024-06-09T22:31:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-06-09T22:31:52.372 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 3, 2024-06-09T22:31:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 3, 2024-06-09T22:31:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 0, 2024-06-09T22:31:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 1, 2024-06-09T22:31:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 1, 2024-06-09T22:31:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 0, 2024-06-09T22:31:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 0, 2024-06-09T22:31:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 0, 2024-06-09T22:31:52.373 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 0, 2024-06-09T22:31:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-06-09T22:31:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 0, 2024-06-09T22:31:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 0, 2024-06-09T22:31:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 51480, 2024-06-09T22:31:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 215420 2024-06-09T22:31:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-06-09T22:31:52.374 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 0, 2024-06-09T22:31:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-06-09T22:31:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 0, 2024-06-09T22:31:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 0, 2024-06-09T22:31:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-06-09T22:31:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-06-09T22:31:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.375 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.376 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-06-09T22:31:52.376 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.376 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.376 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.376 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-06-09T22:31:52.376 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.377 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.377 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.377 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-06-09T22:31:52.377 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.377 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.377 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.378 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-06-09T22:31:52.378 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.378 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.378 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.378 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-06-09T22:31:52.378 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.379 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.379 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.379 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-06-09T22:31:52.379 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.379 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.379 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.380 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-06-09T22:31:52.380 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.380 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.380 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.380 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-06-09T22:31:52.380 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.380 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.380 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.381 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-06-09T22:31:52.381 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.381 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.381 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.381 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-06-09T22:31:52.381 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.381 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.382 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.382 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-06-09T22:31:52.382 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.383 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.383 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.383 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-06-09T22:31:52.383 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.383 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.383 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.384 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-06-09T22:31:52.384 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.384 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.384 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.384 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-06-09T22:31:52.384 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.385 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.385 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.385 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-06-09T22:31:52.385 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.385 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.385 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.386 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-06-09T22:31:52.386 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.386 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.386 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.386 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-06-09T22:31:52.386 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.386 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.387 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.387 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-06-09T22:31:52.387 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.387 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.387 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.387 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-06-09T22:31:52.388 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.388 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.388 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.388 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-06-09T22:31:52.388 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.388 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.388 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.389 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-06-09T22:31:52.389 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.389 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.389 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.389 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-06-09T22:31:52.389 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.390 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.390 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.390 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.390 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-06-09T22:31:52.390 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.390 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.390 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.391 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-06-09T22:31:52.391 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.391 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.391 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.391 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-06-09T22:31:52.391 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.391 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.392 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.392 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-06-09T22:31:52.392 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.392 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.392 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.392 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-06-09T22:31:52.392 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.393 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.393 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.393 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-06-09T22:31:52.393 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.393 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.393 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.393 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-06-09T22:31:52.394 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-06-09T22:31:52.394 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-06-09T22:31:52.394 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.394 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.394 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.394 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:52.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.394 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-06-09T22:31:52.395 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 1, 2024-06-09T22:31:52.395 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 4, 2024-06-09T22:31:52.395 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 3, 2024-06-09T22:31:52.395 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 1, 2024-06-09T22:31:52.395 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-06-09T22:31:52.395 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 0, 2024-06-09T22:31:52.395 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 0, 2024-06-09T22:31:52.395 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 2126, 2024-06-09T22:31:52.396 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-06-09T22:31:52.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.396 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-06-09T22:31:52.396 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-06-09T22:31:52.396 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-06-09T22:31:52.396 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-06-09T22:31:52.396 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-06-09T22:31:52.396 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-06-09T22:31:52.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-06-09T22:31:52.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-09T22:31:52.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-06-09T22:31:52.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-09T22:31:52.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-06-09T22:31:52.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-06-09T22:31:52.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-06-09T22:31:52.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-09T22:31:52.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-06-09T22:31:52.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-06-09T22:31:52.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-06-09T22:31:52.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-06-09T22:31:52.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-06-09T22:31:52.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-09T22:31:52.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-06-09T22:31:52.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-09T22:31:52.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-06-09T22:31:52.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-06-09T22:31:52.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-06-09T22:31:52.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-06-09T22:31:52.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-06-09T22:31:52.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-09T22:31:52.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-06-09T22:31:52.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-06-09T22:31:52.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-06-09T22:31:52.400 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-06-09T22:31:52.400 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-06-09T22:31:52.400 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-06-09T22:31:52.400 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-06-09T22:31:52.400 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-06-09T22:31:52.400 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-06-09T22:31:52.400 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 182071, 2024-06-09T22:31:52.400 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 47, 2024-06-09T22:31:52.400 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-06-09T22:31:52.401 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-06-09T22:31:52.401 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-06-09T22:31:52.401 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-06-09T22:31:52.401 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-06-09T22:31:52.401 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-06-09T22:31:52.401 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-06-09T22:31:52.401 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-06-09T22:31:52.401 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9512, 2024-06-09T22:31:52.401 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 182, 2024-06-09T22:31:52.402 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-06-09T22:31:52.402 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-06-09T22:31:52.402 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-06-09T22:31:52.402 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-06-09T22:31:52.402 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 8464, 2024-06-09T22:31:52.402 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 13, 2024-06-09T22:31:52.402 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-06-09T22:31:52.402 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-06-09T22:31:52.403 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-06-09T22:31:52.403 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-06-09T22:31:52.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.403 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-06-09T22:31:52.403 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-06-09T22:31:52.403 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-06-09T22:31:52.403 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 2437, 2024-06-09T22:31:52.403 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 21, 2024-06-09T22:31:52.403 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 3, 2024-06-09T22:31:52.404 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 2437, 2024-06-09T22:31:52.404 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-06-09T22:31:52.404 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2437, 2024-06-09T22:31:52.404 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 1.149979365, 2024-06-09T22:31:52.404 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000471883 2024-06-09T22:31:52.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.404 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-06-09T22:31:52.404 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-06-09T22:31:52.405 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2437, 2024-06-09T22:31:52.405 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2438 2024-06-09T22:31:52.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.405 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 2437, 2024-06-09T22:31:52.405 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 2437, 2024-06-09T22:31:52.405 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 0, 2024-06-09T22:31:52.405 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-06-09T22:31:52.405 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-06-09T22:31:52.406 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 1586, 2024-06-09T22:31:52.406 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 0, 2024-06-09T22:31:52.406 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 849, 2024-06-09T22:31:52.406 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 0, 2024-06-09T22:31:52.406 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 0, 2024-06-09T22:31:52.406 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-06-09T22:31:52.406 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-06-09T22:31:52.406 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-06-09T22:31:52.407 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-06-09T22:31:52.407 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 0, 2024-06-09T22:31:52.407 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-06-09T22:31:52.407 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-06-09T22:31:52.407 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-06-09T22:31:52.407 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 1, 2024-06-09T22:31:52.407 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 0, 2024-06-09T22:31:52.407 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-06-09T22:31:52.407 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-06-09T22:31:52.408 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-06-09T22:31:52.408 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-06-09T22:31:52.408 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-06-09T22:31:52.408 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-06-09T22:31:52.408 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-06-09T22:31:52.408 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-06-09T22:31:52.408 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-06-09T22:31:52.409 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-06-09T22:31:52.409 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-06-09T22:31:52.409 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-06-09T22:31:52.409 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-06-09T22:31:52.409 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-06-09T22:31:52.409 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-06-09T22:31:52.409 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-06-09T22:31:52.409 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-06-09T22:31:52.410 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-06-09T22:31:52.410 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-06-09T22:31:52.410 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-06-09T22:31:52.410 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-06-09T22:31:52.410 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-06-09T22:31:52.410 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-06-09T22:31:52.410 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-06-09T22:31:52.410 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-06-09T22:31:52.411 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-06-09T22:31:52.411 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 108, 2024-06-09T22:31:52.411 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-06-09T22:31:52.411 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 11, 2024-06-09T22:31:52.411 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 4, 2024-06-09T22:31:52.411 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 4, 2024-06-09T22:31:52.411 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-06-09T22:31:52.411 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-06-09T22:31:52.412 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 0, 2024-06-09T22:31:52.412 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 2, 2024-06-09T22:31:52.412 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 0 2024-06-09T22:31:52.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.412 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-06-09T22:31:52.412 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 0, 2024-06-09T22:31:52.412 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-06-09T22:31:52.412 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-06-09T22:31:52.412 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-06-09T22:31:52.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.413 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-06-09T22:31:52.413 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-06-09T22:31:52.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.413 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-06-09T22:31:52.413 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-06-09T22:31:52.413 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-06-09T22:31:52.413 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-06-09T22:31:52.414 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-06-09T22:31:52.414 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-06-09T22:31:52.414 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-06-09T22:31:52.414 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-06-09T22:31:52.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.414 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-09T22:31:52.414 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:52.414 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-06-09T22:31:52.414 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:52.415 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 2722, 2024-06-09T22:31:52.415 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 1946449, 2024-06-09T22:31:52.415 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:52.415 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 2722, 2024-06-09T22:31:52.415 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-06-09T22:31:52.415 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-06-09T22:31:52.415 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2722, 2024-06-09T22:31:52.415 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1946449, 2024-06-09T22:31:52.415 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:52.416 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.416 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.416 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.416 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:52.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.416 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-06-09T22:31:52.416 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:52.416 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-06-09T22:31:52.416 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:52.417 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:52.417 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:52.417 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:52.417 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:52.417 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2437, 2024-06-09T22:31:52.417 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2494, 2024-06-09T22:31:52.417 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2437, 2024-06-09T22:31:52.417 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2494, 2024-06-09T22:31:52.418 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:52.418 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.418 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.418 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.418 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:52.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.418 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-06-09T22:31:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-06-09T22:31:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2437, 2024-06-09T22:31:52.420 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2437, 2024-06-09T22:31:52.420 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2437, 2024-06-09T22:31:52.420 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2437, 2024-06-09T22:31:52.420 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:52.420 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.420 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.420 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.420 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:52.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.421 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-06-09T22:31:52.421 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:52.421 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-06-09T22:31:52.421 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:52.421 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:52.421 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:52.421 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:52.421 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:52.421 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-06-09T22:31:52.422 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-06-09T22:31:52.422 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-06-09T22:31:52.422 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-06-09T22:31:52.422 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:52.422 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.422 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.422 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.422 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:52.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:52.423 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x55af22ebe6e0": { 2024-06-09T22:31:52.423 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:52.423 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-06-09T22:31:52.423 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:52.423 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:52.423 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:52.423 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:52.424 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:52.424 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-06-09T22:31:52.424 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-06-09T22:31:52.424 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-06-09T22:31:52.424 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-06-09T22:31:52.424 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:52.424 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:52.424 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:52.425 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:52.425 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:52.425 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:52.425 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-06-09T22:31:53.173 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-06-09T22:31:53.173 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-06-09T22:31:53.174 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-06-09T22:31:53.174 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-06-09T22:31:53.174 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-06-09T22:31:53.174 DEBUG:tasks.check_counter:Getting stats from h 2024-06-09T22:31:53.174 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mds.h perf dump 2024-06-09T22:31:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.b/config 2024-06-09T22:31:53.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:53 smithi153 ceph-mon[36599]: pgmap v338: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:54.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:53 smithi104 ceph-mon[32389]: pgmap v338: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:54.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:53 smithi194 ceph-mon[36856]: pgmap v338: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:55.304 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-06-09T22:31:55.304 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-06-09T22:31:55.305 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 1198, 2024-06-09T22:31:55.305 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 1200, 2024-06-09T22:31:55.305 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 4570623, 2024-06-09T22:31:55.305 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 380974, 2024-06-09T22:31:55.306 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-06-09T22:31:55.306 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-06-09T22:31:55.306 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.114963258, 2024-06-09T22:31:55.306 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.044724809, 2024-06-09T22:31:55.307 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.027525653, 2024-06-09T22:31:55.307 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.031514482, 2024-06-09T22:31:55.307 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:55.307 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1198, 2024-06-09T22:31:55.307 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.047304658, 2024-06-09T22:31:55.308 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000039486 2024-06-09T22:31:55.308 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.308 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:55.308 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.309 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.309 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.309 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-09T22:31:55.309 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-06-09T22:31:55.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.309 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-06-09T22:31:55.310 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 533, 2024-06-09T22:31:55.310 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 721, 2024-06-09T22:31:55.310 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 133658, 2024-06-09T22:31:55.310 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 647764, 2024-06-09T22:31:55.310 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 7, 2024-06-09T22:31:55.310 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-06-09T22:31:55.310 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.056329423, 2024-06-09T22:31:55.310 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.018274849, 2024-06-09T22:31:55.311 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.013864002, 2024-06-09T22:31:55.313 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.018868503, 2024-06-09T22:31:55.313 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:55.313 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 719, 2024-06-09T22:31:55.313 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.028049092, 2024-06-09T22:31:55.313 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000039011 2024-06-09T22:31:55.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.314 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:55.314 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.314 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.314 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.314 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-06-09T22:31:55.314 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 480112 2024-06-09T22:31:55.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.315 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-06-09T22:31:55.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 1218, 2024-06-09T22:31:55.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 1203, 2024-06-09T22:31:55.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 4970058, 2024-06-09T22:31:55.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 454899, 2024-06-09T22:31:55.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 3, 2024-06-09T22:31:55.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-06-09T22:31:55.316 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.123616742, 2024-06-09T22:31:55.316 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.037865662, 2024-06-09T22:31:55.316 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.056596604, 2024-06-09T22:31:55.316 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.030430299, 2024-06-09T22:31:55.316 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:55.316 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1202, 2024-06-09T22:31:55.316 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.039570026, 2024-06-09T22:31:55.316 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000032920 2024-06-09T22:31:55.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.317 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:55.317 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.317 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.317 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.317 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1513728, 2024-06-09T22:31:55.317 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 157568 2024-06-09T22:31:55.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.318 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-06-09T22:31:55.318 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-06-09T22:31:55.318 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-06-09T22:31:55.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.318 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-06-09T22:31:55.318 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-06-09T22:31:55.318 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-06-09T22:31:55.318 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4351, 2024-06-09T22:31:55.319 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.110951819, 2024-06-09T22:31:55.319 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000025500 2024-06-09T22:31:55.319 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:55.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.319 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-06-09T22:31:55.319 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-06-09T22:31:55.319 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-06-09T22:31:55.320 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.320 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.320 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.320 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:55.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.320 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-06-09T22:31:55.320 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 0, 2024-06-09T22:31:55.320 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 0, 2024-06-09T22:31:55.321 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-06-09T22:31:55.321 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.321 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.321 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.321 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-06-09T22:31:55.321 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-06-09T22:31:55.321 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-06-09T22:31:55.321 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-06-09T22:31:55.322 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 0, 2024-06-09T22:31:55.322 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-06-09T22:31:55.322 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-06-09T22:31:55.322 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 519, 2024-06-09T22:31:55.322 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 163, 2024-06-09T22:31:55.322 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 356, 2024-06-09T22:31:55.322 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-06-09T22:31:55.323 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 12, 2024-06-09T22:31:55.323 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-06-09T22:31:55.323 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-06-09T22:31:55.323 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-06-09T22:31:55.323 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 3, 2024-06-09T22:31:55.323 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 0, 2024-06-09T22:31:55.324 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 0, 2024-06-09T22:31:55.324 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-06-09T22:31:55.324 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-06-09T22:31:55.324 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-06-09T22:31:55.324 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-06-09T22:31:55.325 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-06-09T22:31:55.325 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-06-09T22:31:55.325 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-06-09T22:31:55.325 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-06-09T22:31:55.325 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-06-09T22:31:55.325 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-06-09T22:31:55.325 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-06-09T22:31:55.325 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-06-09T22:31:55.325 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-06-09T22:31:55.326 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-06-09T22:31:55.326 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 67, 2024-06-09T22:31:55.326 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 21176629, 2024-06-09T22:31:55.326 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 6, 2024-06-09T22:31:55.326 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-06-09T22:31:55.326 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-06-09T22:31:55.326 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-06-09T22:31:55.326 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-06-09T22:31:55.327 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-06-09T22:31:55.327 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-06-09T22:31:55.327 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-06-09T22:31:55.327 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-06-09T22:31:55.327 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-06-09T22:31:55.327 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-06-09T22:31:55.327 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-06-09T22:31:55.327 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-06-09T22:31:55.328 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-09T22:31:55.328 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-09T22:31:55.328 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-06-09T22:31:55.328 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-06-09T22:31:55.328 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-06-09T22:31:55.328 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-06-09T22:31:55.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.328 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-06-09T22:31:55.329 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-06-09T22:31:55.329 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-06-09T22:31:55.329 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-06-09T22:31:55.329 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-06-09T22:31:55.329 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-06-09T22:31:55.329 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-06-09T22:31:55.329 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-06-09T22:31:55.330 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-06-09T22:31:55.330 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-06-09T22:31:55.330 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-06-09T22:31:55.330 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-06-09T22:31:55.330 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-06-09T22:31:55.330 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-06-09T22:31:55.330 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-06-09T22:31:55.331 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-06-09T22:31:55.331 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-06-09T22:31:55.331 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-06-09T22:31:55.331 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-06-09T22:31:55.331 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-06-09T22:31:55.331 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-06-09T22:31:55.332 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-06-09T22:31:55.332 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-06-09T22:31:55.332 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-06-09T22:31:55.332 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-06-09T22:31:55.332 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-06-09T22:31:55.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.332 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-06-09T22:31:55.333 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-06-09T22:31:55.333 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 0, 2024-06-09T22:31:55.333 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-06-09T22:31:55.333 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-06-09T22:31:55.333 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 2504, 2024-06-09T22:31:55.333 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-06-09T22:31:55.333 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-06-09T22:31:55.333 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 0, 2024-06-09T22:31:55.334 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-06-09T22:31:55.334 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-06-09T22:31:55.334 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-06-09T22:31:55.334 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 38, 2024-06-09T22:31:55.334 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-06-09T22:31:55.334 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-06-09T22:31:55.334 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 11260279, 2024-06-09T22:31:55.334 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 11606297, 2024-06-09T22:31:55.335 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 11606161, 2024-06-09T22:31:55.335 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-06-09T22:31:55.335 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.335 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.335 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.335 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 2504 2024-06-09T22:31:55.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.336 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-06-09T22:31:55.336 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 20, 2024-06-09T22:31:55.336 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 589, 2024-06-09T22:31:55.336 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 569, 2024-06-09T22:31:55.336 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 15, 2024-06-09T22:31:55.336 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 26, 2024-06-09T22:31:55.336 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 11, 2024-06-09T22:31:55.336 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 519, 2024-06-09T22:31:55.337 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 666, 2024-06-09T22:31:55.337 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 147, 2024-06-09T22:31:55.337 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-06-09T22:31:55.337 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 0, 2024-06-09T22:31:55.337 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 0, 2024-06-09T22:31:55.338 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 53824, 2024-06-09T22:31:55.338 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 215420 2024-06-09T22:31:55.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.338 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-06-09T22:31:55.338 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 0, 2024-06-09T22:31:55.338 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-06-09T22:31:55.338 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 0, 2024-06-09T22:31:55.338 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 0, 2024-06-09T22:31:55.339 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-06-09T22:31:55.339 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-06-09T22:31:55.339 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.339 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.339 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.339 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-06-09T22:31:55.339 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.340 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.340 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.340 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-06-09T22:31:55.340 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.340 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.340 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.341 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-06-09T22:31:55.341 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.341 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.341 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.342 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-06-09T22:31:55.342 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.342 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.342 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.343 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-06-09T22:31:55.343 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.344 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.344 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.344 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-06-09T22:31:55.345 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.345 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.345 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.346 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-06-09T22:31:55.346 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.346 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.347 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.347 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-06-09T22:31:55.347 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.347 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.348 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.348 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-06-09T22:31:55.348 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.349 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.349 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.349 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-06-09T22:31:55.350 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.350 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.350 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.351 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-06-09T22:31:55.352 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.353 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.353 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.353 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-06-09T22:31:55.353 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.353 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.353 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.354 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-06-09T22:31:55.354 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.354 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.354 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.354 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-06-09T22:31:55.354 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.354 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.355 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.355 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-06-09T22:31:55.355 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.355 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.356 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.356 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-06-09T22:31:55.356 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.357 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.357 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.357 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-06-09T22:31:55.357 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.357 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.357 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.357 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-06-09T22:31:55.358 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.358 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.358 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.358 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-06-09T22:31:55.358 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.359 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.359 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.359 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-06-09T22:31:55.359 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.359 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.359 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.360 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-06-09T22:31:55.360 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.360 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.360 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.360 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-06-09T22:31:55.361 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.361 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.361 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.361 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-06-09T22:31:55.361 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.361 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.361 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.362 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-06-09T22:31:55.362 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.362 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.362 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.362 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-06-09T22:31:55.362 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-06-09T22:31:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-06-09T22:31:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-06-09T22:31:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-06-09T22:31:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-06-09T22:31:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-06-09T22:31:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-06-09T22:31:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 2, 2024-06-09T22:31:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 7, 2024-06-09T22:31:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 5, 2024-06-09T22:31:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 5, 2024-06-09T22:31:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-06-09T22:31:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 0, 2024-06-09T22:31:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 0, 2024-06-09T22:31:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 4345, 2024-06-09T22:31:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-06-09T22:31:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-06-09T22:31:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-06-09T22:31:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-06-09T22:31:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-06-09T22:31:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-06-09T22:31:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-06-09T22:31:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-06-09T22:31:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-09T22:31:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-06-09T22:31:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-09T22:31:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-06-09T22:31:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-06-09T22:31:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-06-09T22:31:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-09T22:31:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-06-09T22:31:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-06-09T22:31:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-06-09T22:31:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-06-09T22:31:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-06-09T22:31:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-09T22:31:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-06-09T22:31:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-09T22:31:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-06-09T22:31:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-06-09T22:31:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-06-09T22:31:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-06-09T22:31:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-06-09T22:31:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-09T22:31:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-06-09T22:31:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-06-09T22:31:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-06-09T22:31:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-06-09T22:31:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-06-09T22:31:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-06-09T22:31:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-06-09T22:31:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-06-09T22:31:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-06-09T22:31:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 191129, 2024-06-09T22:31:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 78, 2024-06-09T22:31:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-06-09T22:31:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-06-09T22:31:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-06-09T22:31:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-06-09T22:31:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-06-09T22:31:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-06-09T22:31:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-06-09T22:31:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-06-09T22:31:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9512, 2024-06-09T22:31:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 182, 2024-06-09T22:31:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-06-09T22:31:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-06-09T22:31:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-06-09T22:31:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-06-09T22:31:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 478634, 2024-06-09T22:31:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 2844, 2024-06-09T22:31:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-06-09T22:31:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-06-09T22:31:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-06-09T22:31:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-06-09T22:31:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-06-09T22:31:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-06-09T22:31:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-06-09T22:31:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 2660, 2024-06-09T22:31:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 26, 2024-06-09T22:31:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 4, 2024-06-09T22:31:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 2660, 2024-06-09T22:31:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-06-09T22:31:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2660, 2024-06-09T22:31:55.376 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 1.258977397, 2024-06-09T22:31:55.376 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000473299 2024-06-09T22:31:55.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.376 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-06-09T22:31:55.376 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-06-09T22:31:55.376 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2660, 2024-06-09T22:31:55.376 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2661 2024-06-09T22:31:55.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.377 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 2660, 2024-06-09T22:31:55.377 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 2660, 2024-06-09T22:31:55.377 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 0, 2024-06-09T22:31:55.377 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-06-09T22:31:55.377 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-06-09T22:31:55.377 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 1686, 2024-06-09T22:31:55.377 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 0, 2024-06-09T22:31:55.377 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 971, 2024-06-09T22:31:55.378 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 0, 2024-06-09T22:31:55.378 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 0, 2024-06-09T22:31:55.378 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-06-09T22:31:55.378 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-06-09T22:31:55.378 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-06-09T22:31:55.378 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-06-09T22:31:55.378 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 0, 2024-06-09T22:31:55.379 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-06-09T22:31:55.379 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-06-09T22:31:55.379 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-06-09T22:31:55.379 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 2, 2024-06-09T22:31:55.379 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 0, 2024-06-09T22:31:55.379 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-06-09T22:31:55.379 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-06-09T22:31:55.380 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-06-09T22:31:55.380 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-06-09T22:31:55.380 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-06-09T22:31:55.380 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-06-09T22:31:55.380 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-06-09T22:31:55.380 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-06-09T22:31:55.380 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-06-09T22:31:55.380 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-06-09T22:31:55.381 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-06-09T22:31:55.381 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-06-09T22:31:55.381 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-06-09T22:31:55.381 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-06-09T22:31:55.381 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-06-09T22:31:55.381 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-06-09T22:31:55.381 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-06-09T22:31:55.381 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-06-09T22:31:55.382 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-06-09T22:31:55.382 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-06-09T22:31:55.382 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-06-09T22:31:55.382 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-06-09T22:31:55.382 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-06-09T22:31:55.382 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-06-09T22:31:55.382 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-06-09T22:31:55.382 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-06-09T22:31:55.383 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 108, 2024-06-09T22:31:55.383 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-06-09T22:31:55.383 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 11, 2024-06-09T22:31:55.383 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 5, 2024-06-09T22:31:55.383 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 5, 2024-06-09T22:31:55.383 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-06-09T22:31:55.383 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-06-09T22:31:55.383 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 0, 2024-06-09T22:31:55.384 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 2, 2024-06-09T22:31:55.384 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 0 2024-06-09T22:31:55.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.384 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-06-09T22:31:55.384 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 0, 2024-06-09T22:31:55.384 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-06-09T22:31:55.385 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-06-09T22:31:55.385 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-06-09T22:31:55.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.385 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-06-09T22:31:55.385 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-06-09T22:31:55.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.386 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-06-09T22:31:55.386 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-06-09T22:31:55.386 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-06-09T22:31:55.386 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-06-09T22:31:55.386 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-06-09T22:31:55.386 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-06-09T22:31:55.387 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-06-09T22:31:55.387 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-06-09T22:31:55.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.387 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-09T22:31:55.387 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:55.387 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-06-09T22:31:55.387 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:55.387 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 2945, 2024-06-09T22:31:55.388 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 9391800, 2024-06-09T22:31:55.388 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:55.388 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 2945, 2024-06-09T22:31:55.388 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-06-09T22:31:55.388 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-06-09T22:31:55.388 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2945, 2024-06-09T22:31:55.388 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 9391800, 2024-06-09T22:31:55.389 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:55.389 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.389 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.389 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.389 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:55.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.390 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-06-09T22:31:55.390 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:55.390 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-06-09T22:31:55.390 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:55.390 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:55.390 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:55.390 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:55.390 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:55.391 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2660, 2024-06-09T22:31:55.391 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 7412003, 2024-06-09T22:31:55.391 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2660, 2024-06-09T22:31:55.391 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 7412003, 2024-06-09T22:31:55.391 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:55.391 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.391 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.391 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.392 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:55.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.392 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-06-09T22:31:55.392 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:55.392 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-06-09T22:31:55.392 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:55.392 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:55.392 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:55.392 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:55.393 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:55.393 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2660, 2024-06-09T22:31:55.393 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2660, 2024-06-09T22:31:55.393 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2660, 2024-06-09T22:31:55.393 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2660, 2024-06-09T22:31:55.393 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:55.393 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.393 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.394 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.394 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:55.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.394 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-06-09T22:31:55.394 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:55.394 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-06-09T22:31:55.394 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:55.394 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:55.395 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:55.395 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:55.395 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:55.395 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-06-09T22:31:55.396 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-06-09T22:31:55.396 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-06-09T22:31:55.396 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-06-09T22:31:55.396 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:55.396 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.396 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.396 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.397 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:55.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:55.397 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x5619c6cec6e0": { 2024-06-09T22:31:55.397 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:55.397 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-06-09T22:31:55.398 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:55.398 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:55.398 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:55.398 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:55.399 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:55.399 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-06-09T22:31:55.399 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-06-09T22:31:55.399 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-06-09T22:31:55.400 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-06-09T22:31:55.400 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:55.400 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:55.401 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:55.401 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:55.401 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:55.401 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:55.401 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-06-09T22:31:55.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:55 smithi153 ceph-mon[36599]: pgmap v339: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:56.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:55 smithi104 ceph-mon[32389]: pgmap v339: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:56.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:55 smithi194 ceph-mon[36856]: pgmap v339: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:31:56.063 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-06-09T22:31:56.063 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-06-09T22:31:56.064 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-06-09T22:31:56.064 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-06-09T22:31:56.064 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-06-09T22:31:56.064 DEBUG:tasks.check_counter:Getting stats from k 2024-06-09T22:31:56.064 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mds.k perf dump 2024-06-09T22:31:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.b/config 2024-06-09T22:31:57.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:57 smithi153 ceph-mon[36599]: pgmap v340: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:58.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:57 smithi104 ceph-mon[32389]: pgmap v340: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:58.039 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-06-09T22:31:58.039 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-06-09T22:31:58.039 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 546, 2024-06-09T22:31:58.040 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 531, 2024-06-09T22:31:58.040 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 47719032, 2024-06-09T22:31:58.040 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 243298, 2024-06-09T22:31:58.040 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-06-09T22:31:58.041 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-06-09T22:31:58.041 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.094443138, 2024-06-09T22:31:58.041 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.021994014, 2024-06-09T22:31:58.041 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.065304223, 2024-06-09T22:31:58.042 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.013363768, 2024-06-09T22:31:58.042 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:58.042 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 530, 2024-06-09T22:31:58.042 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.020719174, 2024-06-09T22:31:58.043 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000039092 2024-06-09T22:31:58.043 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.043 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:58.043 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.044 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.044 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.044 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1515200, 2024-06-09T22:31:58.045 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 161248 2024-06-09T22:31:58.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.045 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-06-09T22:31:58.045 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 1008, 2024-06-09T22:31:58.046 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 1200, 2024-06-09T22:31:58.046 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 92531534, 2024-06-09T22:31:58.046 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 799457, 2024-06-09T22:31:58.047 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 8, 2024-06-09T22:31:58.047 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 6, 2024-06-09T22:31:58.047 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.146216235, 2024-06-09T22:31:58.047 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.031280818, 2024-06-09T22:31:58.048 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.075132416, 2024-06-09T22:31:58.048 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.029772689, 2024-06-09T22:31:58.048 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:58.048 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1198, 2024-06-09T22:31:58.049 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.044445555, 2024-06-09T22:31:58.049 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000037099 2024-06-09T22:31:58.049 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.049 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:58.050 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.050 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.050 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.051 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-06-09T22:31:58.051 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 480816 2024-06-09T22:31:58.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.051 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-06-09T22:31:58.051 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 1647, 2024-06-09T22:31:58.052 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 1649, 2024-06-09T22:31:58.052 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 49360428, 2024-06-09T22:31:58.052 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 523804, 2024-06-09T22:31:58.052 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-06-09T22:31:58.053 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-06-09T22:31:58.053 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.181870599, 2024-06-09T22:31:58.053 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.053036776, 2024-06-09T22:31:58.054 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.064758135, 2024-06-09T22:31:58.054 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.049870676, 2024-06-09T22:31:58.054 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:31:58.054 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1647, 2024-06-09T22:31:58.055 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.065288177, 2024-06-09T22:31:58.055 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000039640 2024-06-09T22:31:58.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.055 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:31:58.056 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.056 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.056 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.057 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-09T22:31:58.057 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-06-09T22:31:58.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.057 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-06-09T22:31:58.058 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-06-09T22:31:58.058 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-06-09T22:31:58.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.059 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-06-09T22:31:58.059 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-06-09T22:31:58.059 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-06-09T22:31:58.060 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4748, 2024-06-09T22:31:58.060 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.126287118, 2024-06-09T22:31:58.060 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000026597 2024-06-09T22:31:58.060 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:58.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.061 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-06-09T22:31:58.061 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-06-09T22:31:58.061 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-06-09T22:31:58.062 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.062 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.062 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.063 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:58.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.063 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-06-09T22:31:58.063 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 0, 2024-06-09T22:31:58.064 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 0, 2024-06-09T22:31:58.064 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-06-09T22:31:58.064 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.064 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.065 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.065 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-06-09T22:31:58.065 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-06-09T22:31:58.065 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-06-09T22:31:58.066 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-06-09T22:31:58.066 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 0, 2024-06-09T22:31:58.066 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-06-09T22:31:58.066 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-06-09T22:31:58.066 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 6708, 2024-06-09T22:31:58.067 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-06-09T22:31:58.067 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 6708, 2024-06-09T22:31:58.067 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-06-09T22:31:58.067 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 6708, 2024-06-09T22:31:58.067 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-06-09T22:31:58.068 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-06-09T22:31:58.068 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-06-09T22:31:58.068 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 5, 2024-06-09T22:31:58.068 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 0, 2024-06-09T22:31:58.068 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 0, 2024-06-09T22:31:58.069 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-06-09T22:31:58.069 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-06-09T22:31:58.069 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-06-09T22:31:58.069 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-06-09T22:31:58.069 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-06-09T22:31:58.069 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-06-09T22:31:58.070 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-06-09T22:31:58.070 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-06-09T22:31:58.070 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-06-09T22:31:58.070 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-06-09T22:31:58.070 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-06-09T22:31:58.070 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-06-09T22:31:58.070 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-06-09T22:31:58.071 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-06-09T22:31:58.071 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 0, 2024-06-09T22:31:58.071 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 0, 2024-06-09T22:31:58.071 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-06-09T22:31:58.071 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-06-09T22:31:58.071 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-06-09T22:31:58.072 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-06-09T22:31:58.072 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-06-09T22:31:58.072 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-06-09T22:31:58.072 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-06-09T22:31:58.072 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-06-09T22:31:58.072 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-06-09T22:31:58.072 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-06-09T22:31:58.073 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-06-09T22:31:58.073 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-06-09T22:31:58.073 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-06-09T22:31:58.073 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-09T22:31:58.073 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-09T22:31:58.073 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-06-09T22:31:58.074 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-06-09T22:31:58.074 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-06-09T22:31:58.074 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-06-09T22:31:58.074 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.074 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-06-09T22:31:58.074 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-06-09T22:31:58.074 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-06-09T22:31:58.075 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-06-09T22:31:58.075 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-06-09T22:31:58.075 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-06-09T22:31:58.075 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-06-09T22:31:58.075 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-06-09T22:31:58.075 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-06-09T22:31:58.076 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-06-09T22:31:58.076 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-06-09T22:31:58.076 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-06-09T22:31:58.076 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-06-09T22:31:58.076 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-06-09T22:31:58.076 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-06-09T22:31:58.076 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-06-09T22:31:58.076 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-06-09T22:31:58.077 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-06-09T22:31:58.077 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-06-09T22:31:58.077 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-06-09T22:31:58.077 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-06-09T22:31:58.077 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-06-09T22:31:58.077 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-06-09T22:31:58.077 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-06-09T22:31:58.078 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-06-09T22:31:58.078 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-06-09T22:31:58.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.078 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-06-09T22:31:58.078 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-06-09T22:31:58.078 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 0, 2024-06-09T22:31:58.078 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-06-09T22:31:58.079 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-06-09T22:31:58.079 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 39589, 2024-06-09T22:31:58.079 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-06-09T22:31:58.079 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-06-09T22:31:58.079 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 0, 2024-06-09T22:31:58.079 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-06-09T22:31:58.080 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-06-09T22:31:58.080 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 60, 2024-06-09T22:31:58.080 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 5, 2024-06-09T22:31:58.080 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-06-09T22:31:58.081 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-06-09T22:31:58.081 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-06-09T22:31:58.081 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 191475077, 2024-06-09T22:31:58.081 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 191474503, 2024-06-09T22:31:58.081 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-06-09T22:31:58.081 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.081 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.082 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.082 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 39589 2024-06-09T22:31:58.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.083 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-06-09T22:31:58.083 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 474, 2024-06-09T22:31:58.083 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 6667, 2024-06-09T22:31:58.083 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 6193, 2024-06-09T22:31:58.083 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 37, 2024-06-09T22:31:58.083 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 38, 2024-06-09T22:31:58.083 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 1, 2024-06-09T22:31:58.083 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 6708, 2024-06-09T22:31:58.084 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 6708, 2024-06-09T22:31:58.084 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 0, 2024-06-09T22:31:58.084 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-06-09T22:31:58.084 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 0, 2024-06-09T22:31:58.084 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 0, 2024-06-09T22:31:58.084 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 59920, 2024-06-09T22:31:58.085 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 215420 2024-06-09T22:31:58.085 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.085 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-06-09T22:31:58.085 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 0, 2024-06-09T22:31:58.086 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-06-09T22:31:58.086 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 0, 2024-06-09T22:31:58.086 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 0, 2024-06-09T22:31:58.086 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-06-09T22:31:58.086 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-06-09T22:31:58.086 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.087 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.087 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.087 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.087 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-06-09T22:31:58.087 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.088 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.088 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.089 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-06-09T22:31:58.089 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.089 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.089 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.090 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-06-09T22:31:58.090 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.091 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.091 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.092 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-06-09T22:31:58.092 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.092 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.092 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.092 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-06-09T22:31:58.093 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.093 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.093 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.093 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-06-09T22:31:58.093 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.093 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.094 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.094 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-06-09T22:31:58.094 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.094 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.094 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.095 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-06-09T22:31:58.095 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.095 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.095 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.095 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-06-09T22:31:58.095 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.095 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.095 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.096 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-06-09T22:31:58.096 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.096 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.096 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.096 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-06-09T22:31:58.097 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:57 smithi194 ceph-mon[36856]: pgmap v340: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:58.097 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.097 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.098 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.098 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-06-09T22:31:58.098 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.098 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.098 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.099 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-06-09T22:31:58.099 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.099 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.099 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.099 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-06-09T22:31:58.099 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.099 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.100 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.100 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-06-09T22:31:58.100 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.100 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.100 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.100 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-06-09T22:31:58.101 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.101 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.101 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.101 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-06-09T22:31:58.101 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.101 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.101 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.102 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-06-09T22:31:58.102 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.102 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.102 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.103 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-06-09T22:31:58.103 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.103 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.103 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.103 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-06-09T22:31:58.103 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.103 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.103 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.104 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-06-09T22:31:58.104 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.104 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.104 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.104 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-06-09T22:31:58.105 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.105 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.105 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.105 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-06-09T22:31:58.105 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.106 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.106 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.107 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-06-09T22:31:58.107 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.107 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.107 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.108 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-06-09T22:31:58.108 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.108 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.108 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.109 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-06-09T22:31:58.109 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.109 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.109 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.110 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-06-09T22:31:58.110 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.110 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.110 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.111 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-06-09T22:31:58.111 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.111 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.111 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.111 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-06-09T22:31:58.111 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-06-09T22:31:58.112 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-06-09T22:31:58.112 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.112 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.112 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.112 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:58.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.112 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-06-09T22:31:58.112 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 1, 2024-06-09T22:31:58.112 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 3, 2024-06-09T22:31:58.113 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 2, 2024-06-09T22:31:58.113 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 2, 2024-06-09T22:31:58.113 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-06-09T22:31:58.113 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 0, 2024-06-09T22:31:58.113 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 0, 2024-06-09T22:31:58.113 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 3611, 2024-06-09T22:31:58.113 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-06-09T22:31:58.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.114 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-06-09T22:31:58.114 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-06-09T22:31:58.114 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-06-09T22:31:58.114 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-06-09T22:31:58.114 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-06-09T22:31:58.114 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-06-09T22:31:58.114 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-06-09T22:31:58.115 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-09T22:31:58.115 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-06-09T22:31:58.115 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-09T22:31:58.115 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-06-09T22:31:58.115 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-06-09T22:31:58.115 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-06-09T22:31:58.115 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-09T22:31:58.115 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-06-09T22:31:58.116 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-06-09T22:31:58.116 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-06-09T22:31:58.116 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-06-09T22:31:58.116 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-06-09T22:31:58.116 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-09T22:31:58.116 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-06-09T22:31:58.116 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-09T22:31:58.116 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-06-09T22:31:58.117 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-06-09T22:31:58.117 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-06-09T22:31:58.117 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-06-09T22:31:58.117 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-06-09T22:31:58.117 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-09T22:31:58.117 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-06-09T22:31:58.117 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-06-09T22:31:58.117 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-06-09T22:31:58.118 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-06-09T22:31:58.118 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-06-09T22:31:58.118 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-06-09T22:31:58.118 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-06-09T22:31:58.118 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-06-09T22:31:58.118 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-06-09T22:31:58.118 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 245041, 2024-06-09T22:31:58.118 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 620, 2024-06-09T22:31:58.119 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-06-09T22:31:58.119 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-06-09T22:31:58.119 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-06-09T22:31:58.119 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-06-09T22:31:58.119 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-06-09T22:31:58.119 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-06-09T22:31:58.120 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-06-09T22:31:58.120 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-06-09T22:31:58.120 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9512, 2024-06-09T22:31:58.120 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 182, 2024-06-09T22:31:58.120 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-06-09T22:31:58.120 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-06-09T22:31:58.120 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-06-09T22:31:58.121 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-06-09T22:31:58.121 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 6002535, 2024-06-09T22:31:58.121 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 39643, 2024-06-09T22:31:58.121 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-06-09T22:31:58.121 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-06-09T22:31:58.121 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-06-09T22:31:58.121 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-06-09T22:31:58.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.122 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-06-09T22:31:58.122 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-06-09T22:31:58.122 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-06-09T22:31:58.122 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 2907, 2024-06-09T22:31:58.122 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 21, 2024-06-09T22:31:58.122 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 3, 2024-06-09T22:31:58.123 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 2907, 2024-06-09T22:31:58.123 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-06-09T22:31:58.123 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2907, 2024-06-09T22:31:58.123 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2.008963883, 2024-06-09T22:31:58.123 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000691078 2024-06-09T22:31:58.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.123 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-06-09T22:31:58.123 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-06-09T22:31:58.124 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2907, 2024-06-09T22:31:58.124 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2908 2024-06-09T22:31:58.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.124 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 2907, 2024-06-09T22:31:58.124 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 2907, 2024-06-09T22:31:58.124 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 0, 2024-06-09T22:31:58.124 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-06-09T22:31:58.125 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-06-09T22:31:58.125 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 1757, 2024-06-09T22:31:58.125 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 0, 2024-06-09T22:31:58.125 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 1148, 2024-06-09T22:31:58.125 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 0, 2024-06-09T22:31:58.125 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 0, 2024-06-09T22:31:58.126 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-06-09T22:31:58.126 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-06-09T22:31:58.126 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-06-09T22:31:58.126 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-06-09T22:31:58.126 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 0, 2024-06-09T22:31:58.126 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-06-09T22:31:58.127 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-06-09T22:31:58.127 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-06-09T22:31:58.127 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 1, 2024-06-09T22:31:58.127 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 0, 2024-06-09T22:31:58.127 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-06-09T22:31:58.127 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-06-09T22:31:58.127 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-06-09T22:31:58.128 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-06-09T22:31:58.128 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-06-09T22:31:58.128 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-06-09T22:31:58.128 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-06-09T22:31:58.128 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-06-09T22:31:58.128 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-06-09T22:31:58.128 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-06-09T22:31:58.128 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-06-09T22:31:58.129 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-06-09T22:31:58.129 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-06-09T22:31:58.129 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-06-09T22:31:58.129 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-06-09T22:31:58.129 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-06-09T22:31:58.129 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-06-09T22:31:58.129 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-06-09T22:31:58.129 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-06-09T22:31:58.130 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-06-09T22:31:58.130 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-06-09T22:31:58.130 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-06-09T22:31:58.130 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-06-09T22:31:58.130 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-06-09T22:31:58.131 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-06-09T22:31:58.131 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-06-09T22:31:58.131 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 108, 2024-06-09T22:31:58.131 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-06-09T22:31:58.131 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 11, 2024-06-09T22:31:58.131 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 9, 2024-06-09T22:31:58.131 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 9, 2024-06-09T22:31:58.131 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-06-09T22:31:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-06-09T22:31:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 0, 2024-06-09T22:31:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 2, 2024-06-09T22:31:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 0 2024-06-09T22:31:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-06-09T22:31:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 0, 2024-06-09T22:31:58.133 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-06-09T22:31:58.133 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-06-09T22:31:58.133 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-06-09T22:31:58.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.133 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-06-09T22:31:58.133 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-06-09T22:31:58.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.133 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-06-09T22:31:58.134 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-06-09T22:31:58.134 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-06-09T22:31:58.134 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-06-09T22:31:58.134 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-06-09T22:31:58.134 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-06-09T22:31:58.134 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-06-09T22:31:58.134 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-06-09T22:31:58.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.135 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-09T22:31:58.135 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:58.135 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-06-09T22:31:58.135 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:58.135 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 3197, 2024-06-09T22:31:58.136 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 189305509, 2024-06-09T22:31:58.136 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:58.136 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 3197, 2024-06-09T22:31:58.136 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-06-09T22:31:58.136 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-06-09T22:31:58.136 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 3197, 2024-06-09T22:31:58.136 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 189305509, 2024-06-09T22:31:58.136 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:58.136 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.137 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.137 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.137 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:58.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.137 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-06-09T22:31:58.137 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:58.137 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-06-09T22:31:58.137 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:58.138 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:58.138 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:58.138 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:58.138 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:58.139 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2907, 2024-06-09T22:31:58.139 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 187281558, 2024-06-09T22:31:58.139 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2907, 2024-06-09T22:31:58.139 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 187281558, 2024-06-09T22:31:58.139 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:58.139 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.139 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.140 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.140 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:58.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.140 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-06-09T22:31:58.140 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:58.140 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-06-09T22:31:58.140 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:58.141 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:58.141 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:58.141 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:58.141 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:58.141 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2907, 2024-06-09T22:31:58.141 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2907, 2024-06-09T22:31:58.141 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2907, 2024-06-09T22:31:58.141 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2907, 2024-06-09T22:31:58.142 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:58.142 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.142 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.142 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.142 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:58.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.142 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-06-09T22:31:58.143 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:58.143 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-06-09T22:31:58.143 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:58.143 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:58.143 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:58.143 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:58.143 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:58.144 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-06-09T22:31:58.144 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-06-09T22:31:58.144 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-06-09T22:31:58.144 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-06-09T22:31:58.144 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:58.144 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.144 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.145 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.145 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:58.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-06-09T22:31:58.145 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x560e740c06e0": { 2024-06-09T22:31:58.145 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-06-09T22:31:58.145 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-06-09T22:31:58.145 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-06-09T22:31:58.145 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-06-09T22:31:58.146 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-06-09T22:31:58.146 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-06-09T22:31:58.146 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-06-09T22:31:58.146 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-06-09T22:31:58.146 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-06-09T22:31:58.146 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-06-09T22:31:58.146 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-06-09T22:31:58.147 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-06-09T22:31:58.147 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-06-09T22:31:58.147 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-06-09T22:31:58.147 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-06-09T22:31:58.147 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:58.147 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-06-09T22:31:58.147 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-06-09T22:31:58.826 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-06-09T22:31:58.827 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-06-09T22:31:58.827 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-06-09T22:31:58.827 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-06-09T22:31:58.827 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-06-09T22:31:58.827 DEBUG:tasks.check_counter:Getting stats from c 2024-06-09T22:31:58.828 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mds.c perf dump 2024-06-09T22:31:59.078 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:31:59.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:31:59 smithi194 ceph-mon[36856]: pgmap v341: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:31:59.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:31:59 smithi153 ceph-mon[36599]: pgmap v341: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:32:00.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:31:59 smithi104 ceph-mon[32389]: pgmap v341: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:32:00.931 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-06-09T22:32:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-0": { 2024-06-09T22:32:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 2703, 2024-06-09T22:32:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 2725, 2024-06-09T22:32:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 2287559, 2024-06-09T22:32:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 689851, 2024-06-09T22:32:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 9, 2024-06-09T22:32:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 4, 2024-06-09T22:32:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.243035261, 2024-06-09T22:32:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.097304911, 2024-06-09T22:32:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.165719844, 2024-06-09T22:32:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.011359361, 2024-06-09T22:32:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:32:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2724, 2024-06-09T22:32:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.077873040, 2024-06-09T22:32:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000028587 2024-06-09T22:32:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:32:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 4737, 2024-06-09T22:32:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000899064, 2024-06-09T22:32:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000189 2024-06-09T22:32:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 2082752, 2024-06-09T22:32:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 413744 2024-06-09T22:32:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-1": { 2024-06-09T22:32:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 48282, 2024-06-09T22:32:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 24207, 2024-06-09T22:32:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 22928587, 2024-06-09T22:32:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 4373755, 2024-06-09T22:32:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 20, 2024-06-09T22:32:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 5, 2024-06-09T22:32:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 2.378230628, 2024-06-09T22:32:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 1.051546719, 2024-06-09T22:32:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 1.819422339, 2024-06-09T22:32:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.000345631, 2024-06-09T22:32:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:32:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 24205, 2024-06-09T22:32:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.546051036, 2024-06-09T22:32:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000022559 2024-06-09T22:32:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:32:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 72373, 2024-06-09T22:32:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.012018257, 2024-06-09T22:32:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000166 2024-06-09T22:32:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 5024, 2024-06-09T22:32:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 160608 2024-06-09T22:32:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-2": { 2024-06-09T22:32:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 41, 2024-06-09T22:32:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 176, 2024-06-09T22:32:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 8289, 2024-06-09T22:32:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 583943, 2024-06-09T22:32:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 28, 2024-06-09T22:32:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 5, 2024-06-09T22:32:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.016295977, 2024-06-09T22:32:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.005967326, 2024-06-09T22:32:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.008165147, 2024-06-09T22:32:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.000474981, 2024-06-09T22:32:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:32:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 174, 2024-06-09T22:32:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.026921188, 2024-06-09T22:32:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000154719 2024-06-09T22:32:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:32:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 27, 2024-06-09T22:32:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000002943, 2024-06-09T22:32:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000109 2024-06-09T22:32:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-06-09T22:32:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 326704 2024-06-09T22:32:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "cct": { 2024-06-09T22:32:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "total_workers": 1, 2024-06-09T22:32:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "unhealthy_workers": 0 2024-06-09T22:32:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-MDSRank": { 2024-06-09T22:32:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-06-09T22:32:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-06-09T22:32:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 60, 2024-06-09T22:32:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.001907358, 2024-06-09T22:32:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000031789 2024-06-09T22:32:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-PurgeQueue": { 2024-06-09T22:32:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-06-09T22:32:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-06-09T22:32:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2, 2024-06-09T22:32:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000013061, 2024-06-09T22:32:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000006530 2024-06-09T22:32:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "mds": { 2024-06-09T22:32:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "request": 1684, 2024-06-09T22:32:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "reply": 5, 2024-06-09T22:32:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "reply_latency": { 2024-06-09T22:32:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 5, 2024-06-09T22:32:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.001298888, 2024-06-09T22:32:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000259777 2024-06-09T22:32:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_reply": 0, 2024-06-09T22:32:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "forward": 1679, 2024-06-09T22:32:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_complete": 0, 2024-06-09T22:32:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_keys": 0, 2024-06-09T22:32:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_commit": 11, 2024-06-09T22:32:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_split": 0, 2024-06-09T22:32:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_merge": 0, 2024-06-09T22:32:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes": 22, 2024-06-09T22:32:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_top": 2, 2024-06-09T22:32:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_bottom": 20, 2024-06-09T22:32:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pin_tail": 0, 2024-06-09T22:32:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pinned": 19, 2024-06-09T22:32:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_expired": 1, 2024-06-09T22:32:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_with_caps": 0, 2024-06-09T22:32:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "caps": 0, 2024-06-09T22:32:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "subtrees": 3, 2024-06-09T22:32:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse": 1736, 2024-06-09T22:32:00.946 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_hit": 43, 2024-06-09T22:32:00.946 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_forward": 1679, 2024-06-09T22:32:00.946 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_discover": 12, 2024-06-09T22:32:00.946 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_dir_fetch": 0, 2024-06-09T22:32:00.946 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_remote_ino": 0, 2024-06-09T22:32:00.946 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_lock": 0, 2024-06-09T22:32:00.946 INFO:teuthology.orchestra.run.smithi194.stdout: "load_cent": 0, 2024-06-09T22:32:00.946 INFO:teuthology.orchestra.run.smithi194.stdout: "q": 0, 2024-06-09T22:32:00.947 INFO:teuthology.orchestra.run.smithi194.stdout: "exported": 0, 2024-06-09T22:32:00.947 INFO:teuthology.orchestra.run.smithi194.stdout: "exported_inodes": 0, 2024-06-09T22:32:00.947 INFO:teuthology.orchestra.run.smithi194.stdout: "imported": 0, 2024-06-09T22:32:00.947 INFO:teuthology.orchestra.run.smithi194.stdout: "imported_inodes": 0, 2024-06-09T22:32:00.947 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_dir_fetch": 0, 2024-06-09T22:32:00.947 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_backtrace_fetch": 0, 2024-06-09T22:32:00.947 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_peer_discover": 0, 2024-06-09T22:32:00.947 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rfiles": 67, 2024-06-09T22:32:00.948 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rbytes": 21176629, 2024-06-09T22:32:00.948 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rsnaps": 6, 2024-06-09T22:32:00.948 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_fetch": 0, 2024-06-09T22:32:00.948 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_set_tag": 0, 2024-06-09T22:32:00.948 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_repaired": 0, 2024-06-09T22:32:00.948 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_inotable_repaired": 0, 2024-06-09T22:32:00.948 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_inodes": 0, 2024-06-09T22:32:00.948 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_base_inodes": 0, 2024-06-09T22:32:00.949 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dirfrag_rstats": 0, 2024-06-09T22:32:00.949 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_file_inodes": 0, 2024-06-09T22:32:00.949 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_inode_file_caps": 0, 2024-06-09T22:32:00.949 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_revoke": 0, 2024-06-09T22:32:00.949 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_grant": 0, 2024-06-09T22:32:00.949 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_trunc": 0, 2024-06-09T22:32:00.949 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-09T22:32:00.949 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-09T22:32:00.950 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps": 0, 2024-06-09T22:32:00.950 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps_dirty": 0, 2024-06-09T22:32:00.950 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_cap_release": 0, 2024-06-09T22:32:00.950 INFO:teuthology.orchestra.run.smithi194.stdout: "process_request_cap_release": 0 2024-06-09T22:32:00.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.950 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_cache": { 2024-06-09T22:32:00.950 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update": 0, 2024-06-09T22:32:00.950 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update_receipt": 55, 2024-06-09T22:32:00.951 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_try_discover": 30, 2024-06-09T22:32:00.951 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_send_discover": 67, 2024-06-09T22:32:00.951 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_handle_discover": 0, 2024-06-09T22:32:00.951 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays": 0, 2024-06-09T22:32:00.951 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_delayed": 0, 2024-06-09T22:32:00.951 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_enqueuing": 0, 2024-06-09T22:32:00.951 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_created": 0, 2024-06-09T22:32:00.951 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_enqueued": 0, 2024-06-09T22:32:00.952 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_reintegrated": 0, 2024-06-09T22:32:00.952 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_migrated": 0, 2024-06-09T22:32:00.952 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_processing": 0, 2024-06-09T22:32:00.952 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_enqueued": 0, 2024-06-09T22:32:00.952 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_prioritized": 0, 2024-06-09T22:32:00.952 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_started": 0, 2024-06-09T22:32:00.952 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_completed": 0, 2024-06-09T22:32:00.952 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_path": 5, 2024-06-09T22:32:00.952 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_inode": 52, 2024-06-09T22:32:00.953 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_enqueue_scrub": 0, 2024-06-09T22:32:00.953 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_exportdir": 0, 2024-06-09T22:32:00.953 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_flush": 0, 2024-06-09T22:32:00.953 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragmentdir": 0, 2024-06-09T22:32:00.953 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragstats": 0, 2024-06-09T22:32:00.953 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_inodestats": 0 2024-06-09T22:32:00.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.953 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_log": { 2024-06-09T22:32:00.954 INFO:teuthology.orchestra.run.smithi194.stdout: "evlrg": 0, 2024-06-09T22:32:00.954 INFO:teuthology.orchestra.run.smithi194.stdout: "evadd": 9, 2024-06-09T22:32:00.954 INFO:teuthology.orchestra.run.smithi194.stdout: "evex": 0, 2024-06-09T22:32:00.954 INFO:teuthology.orchestra.run.smithi194.stdout: "evtrm": 0, 2024-06-09T22:32:00.954 INFO:teuthology.orchestra.run.smithi194.stdout: "ev": 9, 2024-06-09T22:32:00.954 INFO:teuthology.orchestra.run.smithi194.stdout: "evexg": 0, 2024-06-09T22:32:00.954 INFO:teuthology.orchestra.run.smithi194.stdout: "evexd": 0, 2024-06-09T22:32:00.954 INFO:teuthology.orchestra.run.smithi194.stdout: "segadd": 2, 2024-06-09T22:32:00.955 INFO:teuthology.orchestra.run.smithi194.stdout: "segex": 0, 2024-06-09T22:32:00.955 INFO:teuthology.orchestra.run.smithi194.stdout: "segtrm": 0, 2024-06-09T22:32:00.955 INFO:teuthology.orchestra.run.smithi194.stdout: "seg": 2, 2024-06-09T22:32:00.955 INFO:teuthology.orchestra.run.smithi194.stdout: "segmjr": 2, 2024-06-09T22:32:00.955 INFO:teuthology.orchestra.run.smithi194.stdout: "segexg": 0, 2024-06-09T22:32:00.955 INFO:teuthology.orchestra.run.smithi194.stdout: "segexd": 0, 2024-06-09T22:32:00.955 INFO:teuthology.orchestra.run.smithi194.stdout: "expos": 4194304, 2024-06-09T22:32:00.955 INFO:teuthology.orchestra.run.smithi194.stdout: "wrpos": 4196793, 2024-06-09T22:32:00.956 INFO:teuthology.orchestra.run.smithi194.stdout: "rdpos": 0, 2024-06-09T22:32:00.956 INFO:teuthology.orchestra.run.smithi194.stdout: "jlat": { 2024-06-09T22:32:00.956 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 8, 2024-06-09T22:32:00.956 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.013424660, 2024-06-09T22:32:00.956 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001678082 2024-06-09T22:32:00.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.956 INFO:teuthology.orchestra.run.smithi194.stdout: "replayed": 0 2024-06-09T22:32:00.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.956 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_mem": { 2024-06-09T22:32:00.957 INFO:teuthology.orchestra.run.smithi194.stdout: "ino": 25, 2024-06-09T22:32:00.957 INFO:teuthology.orchestra.run.smithi194.stdout: "ino+": 25, 2024-06-09T22:32:00.957 INFO:teuthology.orchestra.run.smithi194.stdout: "ino-": 0, 2024-06-09T22:32:00.957 INFO:teuthology.orchestra.run.smithi194.stdout: "dir": 30, 2024-06-09T22:32:00.957 INFO:teuthology.orchestra.run.smithi194.stdout: "dir+": 60, 2024-06-09T22:32:00.957 INFO:teuthology.orchestra.run.smithi194.stdout: "dir-": 30, 2024-06-09T22:32:00.957 INFO:teuthology.orchestra.run.smithi194.stdout: "dn": 22, 2024-06-09T22:32:00.957 INFO:teuthology.orchestra.run.smithi194.stdout: "dn+": 23, 2024-06-09T22:32:00.957 INFO:teuthology.orchestra.run.smithi194.stdout: "dn-": 1, 2024-06-09T22:32:00.958 INFO:teuthology.orchestra.run.smithi194.stdout: "cap": 0, 2024-06-09T22:32:00.958 INFO:teuthology.orchestra.run.smithi194.stdout: "cap+": 4, 2024-06-09T22:32:00.958 INFO:teuthology.orchestra.run.smithi194.stdout: "cap-": 4, 2024-06-09T22:32:00.958 INFO:teuthology.orchestra.run.smithi194.stdout: "rss": 57980, 2024-06-09T22:32:00.958 INFO:teuthology.orchestra.run.smithi194.stdout: "heap": 223612 2024-06-09T22:32:00.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.958 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_server": { 2024-06-09T22:32:00.959 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_client_request": 1691, 2024-06-09T22:32:00.959 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_server_request": 0, 2024-06-09T22:32:00.959 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_request": 1684, 2024-06-09T22:32:00.959 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_session": 20, 2024-06-09T22:32:00.959 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_peer_request": 0, 2024-06-09T22:32:00.959 INFO:teuthology.orchestra.run.smithi194.stdout: "req_create_latency": { 2024-06-09T22:32:00.959 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.959 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.959 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.960 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.960 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getattr_latency": { 2024-06-09T22:32:00.960 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2, 2024-06-09T22:32:00.960 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000397632, 2024-06-09T22:32:00.960 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000198816 2024-06-09T22:32:00.960 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.960 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getfilelock_latency": { 2024-06-09T22:32:00.960 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.961 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.961 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.961 INFO:teuthology.orchestra.run.smithi194.stdout: "req_link_latency": { 2024-06-09T22:32:00.961 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.961 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.961 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.962 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookup_latency": { 2024-06-09T22:32:00.962 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 3, 2024-06-09T22:32:00.962 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000901256, 2024-06-09T22:32:00.962 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000300418 2024-06-09T22:32:00.962 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.962 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookuphash_latency": { 2024-06-09T22:32:00.962 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.962 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.963 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.963 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupino_latency": { 2024-06-09T22:32:00.963 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.963 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.963 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.963 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupname_latency": { 2024-06-09T22:32:00.964 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.964 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.964 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.964 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupparent_latency": { 2024-06-09T22:32:00.964 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.964 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.964 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.965 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.965 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupsnap_latency": { 2024-06-09T22:32:00.965 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.965 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.965 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.965 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.965 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lssnap_latency": { 2024-06-09T22:32:00.965 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.965 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.966 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.966 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.966 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mkdir_latency": { 2024-06-09T22:32:00.967 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.967 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.967 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.967 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.967 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mknod_latency": { 2024-06-09T22:32:00.967 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.967 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.967 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.968 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.968 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mksnap_latency": { 2024-06-09T22:32:00.968 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.968 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.968 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.968 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.968 INFO:teuthology.orchestra.run.smithi194.stdout: "req_open_latency": { 2024-06-09T22:32:00.968 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.968 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.969 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.969 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.969 INFO:teuthology.orchestra.run.smithi194.stdout: "req_readdir_latency": { 2024-06-09T22:32:00.969 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.969 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.969 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.969 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.969 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rename_latency": { 2024-06-09T22:32:00.970 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.970 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.970 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.970 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.970 INFO:teuthology.orchestra.run.smithi194.stdout: "req_renamesnap_latency": { 2024-06-09T22:32:00.970 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.970 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.971 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.971 INFO:teuthology.orchestra.run.smithi194.stdout: "req_snapdiff_latency": { 2024-06-09T22:32:00.971 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.971 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.971 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.971 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmdir_latency": { 2024-06-09T22:32:00.971 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.972 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.972 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.972 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmsnap_latency": { 2024-06-09T22:32:00.972 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.972 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.972 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.972 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmxattr_latency": { 2024-06-09T22:32:00.973 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.973 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.973 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.973 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.973 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setattr_latency": { 2024-06-09T22:32:00.973 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.973 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.973 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.973 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.973 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setdirlayout_latency": { 2024-06-09T22:32:00.974 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.974 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.974 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.974 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.974 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setfilelock_latency": { 2024-06-09T22:32:00.974 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.974 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.974 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.975 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.975 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setlayout_latency": { 2024-06-09T22:32:00.975 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.975 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.975 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.975 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.975 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setxattr_latency": { 2024-06-09T22:32:00.976 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.976 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.976 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.976 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.976 INFO:teuthology.orchestra.run.smithi194.stdout: "req_symlink_latency": { 2024-06-09T22:32:00.976 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.976 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.976 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.977 INFO:teuthology.orchestra.run.smithi194.stdout: "req_unlink_latency": { 2024-06-09T22:32:00.977 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.977 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.977 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.977 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_revoke_eviction": 0, 2024-06-09T22:32:00.977 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_acquisition_throttle": 0, 2024-06-09T22:32:00.978 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getvxattr_latency": { 2024-06-09T22:32:00.978 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:00.978 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:00.978 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:00.978 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:00.978 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.978 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_sessions": { 2024-06-09T22:32:00.978 INFO:teuthology.orchestra.run.smithi194.stdout: "session_count": 1, 2024-06-09T22:32:00.979 INFO:teuthology.orchestra.run.smithi194.stdout: "session_add": 4, 2024-06-09T22:32:00.979 INFO:teuthology.orchestra.run.smithi194.stdout: "session_remove": 3, 2024-06-09T22:32:00.979 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_open": 1, 2024-06-09T22:32:00.979 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_stale": 0, 2024-06-09T22:32:00.979 INFO:teuthology.orchestra.run.smithi194.stdout: "total_load": 4, 2024-06-09T22:32:00.979 INFO:teuthology.orchestra.run.smithi194.stdout: "average_load": 2, 2024-06-09T22:32:00.979 INFO:teuthology.orchestra.run.smithi194.stdout: "avg_session_uptime": 538, 2024-06-09T22:32:00.979 INFO:teuthology.orchestra.run.smithi194.stdout: "mdthresh_evicted": 0 2024-06-09T22:32:00.980 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.980 INFO:teuthology.orchestra.run.smithi194.stdout: "mempool": { 2024-06-09T22:32:00.980 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_bytes": 0, 2024-06-09T22:32:00.980 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_items": 0, 2024-06-09T22:32:00.980 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_bytes": 0, 2024-06-09T22:32:00.980 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_items": 0, 2024-06-09T22:32:00.980 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_bytes": 0, 2024-06-09T22:32:00.980 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_items": 0, 2024-06-09T22:32:00.981 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-09T22:32:00.981 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_items": 0, 2024-06-09T22:32:00.981 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-09T22:32:00.981 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_items": 0, 2024-06-09T22:32:00.981 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_bytes": 0, 2024-06-09T22:32:00.981 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_items": 0, 2024-06-09T22:32:00.981 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-09T22:32:00.981 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_items": 0, 2024-06-09T22:32:00.981 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_bytes": 0, 2024-06-09T22:32:00.982 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_items": 0, 2024-06-09T22:32:00.982 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_bytes": 0, 2024-06-09T22:32:00.982 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_items": 0, 2024-06-09T22:32:00.982 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-09T22:32:00.982 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_items": 0, 2024-06-09T22:32:00.982 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-09T22:32:00.982 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_items": 0, 2024-06-09T22:32:00.982 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_bytes": 0, 2024-06-09T22:32:00.983 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_items": 0, 2024-06-09T22:32:00.983 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_bytes": 0, 2024-06-09T22:32:00.983 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_items": 0, 2024-06-09T22:32:00.983 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-09T22:32:00.983 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_items": 0, 2024-06-09T22:32:00.983 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_bytes": 0, 2024-06-09T22:32:00.983 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_items": 0, 2024-06-09T22:32:00.983 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_bytes": 0, 2024-06-09T22:32:00.983 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_items": 0, 2024-06-09T22:32:00.984 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_bytes": 0, 2024-06-09T22:32:00.984 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_items": 0, 2024-06-09T22:32:00.984 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, 2024-06-09T22:32:00.984 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, 2024-06-09T22:32:00.984 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 263656, 2024-06-09T22:32:00.984 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_items": 102, 2024-06-09T22:32:00.984 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_bytes": 0, 2024-06-09T22:32:00.984 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_items": 0, 2024-06-09T22:32:00.984 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_bytes": 0, 2024-06-09T22:32:00.985 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_items": 0, 2024-06-09T22:32:00.985 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_bytes": 0, 2024-06-09T22:32:00.985 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_items": 0, 2024-06-09T22:32:00.985 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_bytes": 0, 2024-06-09T22:32:00.985 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_items": 0, 2024-06-09T22:32:00.985 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_bytes": 9512, 2024-06-09T22:32:00.985 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_items": 182, 2024-06-09T22:32:00.985 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_bytes": 0, 2024-06-09T22:32:00.985 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_items": 0, 2024-06-09T22:32:00.986 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_bytes": 0, 2024-06-09T22:32:00.986 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_items": 0, 2024-06-09T22:32:00.986 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_bytes": 119346, 2024-06-09T22:32:00.986 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_items": 664, 2024-06-09T22:32:00.986 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_bytes": 0, 2024-06-09T22:32:00.986 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_items": 0, 2024-06-09T22:32:00.986 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_bytes": 0, 2024-06-09T22:32:00.986 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_items": 0 2024-06-09T22:32:00.987 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.987 INFO:teuthology.orchestra.run.smithi194.stdout: "objecter": { 2024-06-09T22:32:00.987 INFO:teuthology.orchestra.run.smithi194.stdout: "op_active": 0, 2024-06-09T22:32:00.987 INFO:teuthology.orchestra.run.smithi194.stdout: "op_laggy": 0, 2024-06-09T22:32:00.987 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send": 48, 2024-06-09T22:32:00.987 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send_bytes": 13116, 2024-06-09T22:32:00.987 INFO:teuthology.orchestra.run.smithi194.stdout: "op_resend": 33, 2024-06-09T22:32:00.987 INFO:teuthology.orchestra.run.smithi194.stdout: "op_reply": 48, 2024-06-09T22:32:00.988 INFO:teuthology.orchestra.run.smithi194.stdout: "op_latency": { 2024-06-09T22:32:00.988 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 48, 2024-06-09T22:32:00.988 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.248995022, 2024-06-09T22:32:00.988 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.005187396 2024-06-09T22:32:00.988 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.988 INFO:teuthology.orchestra.run.smithi194.stdout: "op_inflight": 0, 2024-06-09T22:32:00.988 INFO:teuthology.orchestra.run.smithi194.stdout: "oplen_avg": { 2024-06-09T22:32:00.988 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 48, 2024-06-09T22:32:00.989 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 72 2024-06-09T22:32:00.989 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.989 INFO:teuthology.orchestra.run.smithi194.stdout: "op": 48, 2024-06-09T22:32:00.989 INFO:teuthology.orchestra.run.smithi194.stdout: "op_r": 0, 2024-06-09T22:32:00.989 INFO:teuthology.orchestra.run.smithi194.stdout: "op_w": 48, 2024-06-09T22:32:00.989 INFO:teuthology.orchestra.run.smithi194.stdout: "op_rmw": 0, 2024-06-09T22:32:00.989 INFO:teuthology.orchestra.run.smithi194.stdout: "op_pg": 0, 2024-06-09T22:32:00.989 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_stat": 0, 2024-06-09T22:32:00.989 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_create": 10, 2024-06-09T22:32:00.990 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_read": 0, 2024-06-09T22:32:00.990 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_write": 8, 2024-06-09T22:32:00.990 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writefull": 9, 2024-06-09T22:32:00.990 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writesame": 0, 2024-06-09T22:32:00.990 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_append": 0, 2024-06-09T22:32:00.990 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_zero": 0, 2024-06-09T22:32:00.990 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_truncate": 0, 2024-06-09T22:32:00.990 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_delete": 6, 2024-06-09T22:32:00.991 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_mapext": 0, 2024-06-09T22:32:00.991 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_sparse_read": 0, 2024-06-09T22:32:00.991 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_clonerange": 0, 2024-06-09T22:32:00.991 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_getxattr": 0, 2024-06-09T22:32:00.991 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_setxattr": 20, 2024-06-09T22:32:00.991 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_cmpxattr": 0, 2024-06-09T22:32:00.991 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_rmxattr": 0, 2024-06-09T22:32:00.991 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_resetxattrs": 0, 2024-06-09T22:32:00.992 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_call": 0, 2024-06-09T22:32:00.992 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_watch": 0, 2024-06-09T22:32:00.992 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_notify": 0, 2024-06-09T22:32:00.992 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_src_cmpxattr": 0, 2024-06-09T22:32:00.992 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls": 0, 2024-06-09T22:32:00.992 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls_filter": 0, 2024-06-09T22:32:00.992 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_other": 0, 2024-06-09T22:32:00.992 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_active": 0, 2024-06-09T22:32:00.993 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_send": 0, 2024-06-09T22:32:00.993 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_resend": 0, 2024-06-09T22:32:00.993 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_ping": 0, 2024-06-09T22:32:00.993 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_active": 0, 2024-06-09T22:32:00.993 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_send": 0, 2024-06-09T22:32:00.993 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_resend": 0, 2024-06-09T22:32:00.993 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_active": 0, 2024-06-09T22:32:00.993 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_send": 0, 2024-06-09T22:32:00.993 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_resend": 0, 2024-06-09T22:32:00.994 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_active": 0, 2024-06-09T22:32:00.994 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_send": 0, 2024-06-09T22:32:00.994 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_resend": 0, 2024-06-09T22:32:00.994 INFO:teuthology.orchestra.run.smithi194.stdout: "command_active": 0, 2024-06-09T22:32:00.994 INFO:teuthology.orchestra.run.smithi194.stdout: "command_send": 0, 2024-06-09T22:32:00.994 INFO:teuthology.orchestra.run.smithi194.stdout: "command_resend": 0, 2024-06-09T22:32:00.994 INFO:teuthology.orchestra.run.smithi194.stdout: "map_epoch": 108, 2024-06-09T22:32:00.994 INFO:teuthology.orchestra.run.smithi194.stdout: "map_full": 0, 2024-06-09T22:32:00.995 INFO:teuthology.orchestra.run.smithi194.stdout: "map_inc": 11, 2024-06-09T22:32:00.995 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_sessions": 10, 2024-06-09T22:32:00.995 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_open": 10, 2024-06-09T22:32:00.995 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_close": 0, 2024-06-09T22:32:00.995 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_laggy": 0, 2024-06-09T22:32:00.995 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_wr": 17, 2024-06-09T22:32:00.995 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_rd": 0, 2024-06-09T22:32:00.995 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_del": 2 2024-06-09T22:32:00.996 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.996 INFO:teuthology.orchestra.run.smithi194.stdout: "oft": { 2024-06-09T22:32:00.996 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_objs": 1, 2024-06-09T22:32:00.996 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_kv_pairs": 0, 2024-06-09T22:32:00.996 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_updates": 0, 2024-06-09T22:32:00.996 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_removes": 0 2024-06-09T22:32:00.996 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.996 INFO:teuthology.orchestra.run.smithi194.stdout: "osd-slow-ops": { 2024-06-09T22:32:00.996 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_ops_count": 57 2024-06-09T22:32:00.997 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.997 INFO:teuthology.orchestra.run.smithi194.stdout: "purge_queue": { 2024-06-09T22:32:00.997 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops": 0, 2024-06-09T22:32:00.997 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops_high_water": 0, 2024-06-09T22:32:00.997 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing": 0, 2024-06-09T22:32:00.997 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_high_water": 0, 2024-06-09T22:32:00.997 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed_ops": 0, 2024-06-09T22:32:00.997 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed": 0, 2024-06-09T22:32:00.998 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_item_in_journal": 0 2024-06-09T22:32:00.998 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:00.998 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-09T22:32:00.998 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:00.998 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-06-09T22:32:00.998 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:00.998 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 51022, 2024-06-09T22:32:00.998 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 20543178, 2024-06-09T22:32:00.999 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:00.999 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 51022, 2024-06-09T22:32:00.999 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-06-09T22:32:00.999 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-06-09T22:32:00.999 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 51022, 2024-06-09T22:32:00.999 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 20543178, 2024-06-09T22:32:00.999 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:00.999 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:01.000 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:01.000 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:01.000 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:01.000 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:01.000 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_bytes": { 2024-06-09T22:32:01.000 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:01.000 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-06-09T22:32:01.000 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:01.001 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-06-09T22:32:01.001 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-06-09T22:32:01.001 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:01.001 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-06-09T22:32:01.001 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 48, 2024-06-09T22:32:01.001 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 13116, 2024-06-09T22:32:01.001 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 48, 2024-06-09T22:32:01.001 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 13116, 2024-06-09T22:32:01.002 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:01.002 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:01.002 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:01.002 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:01.002 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:01.002 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:01.002 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_ops": { 2024-06-09T22:32:01.002 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:01.002 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 1024, 2024-06-09T22:32:01.003 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:01.003 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-06-09T22:32:01.003 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-06-09T22:32:01.003 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:01.003 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-06-09T22:32:01.003 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 48, 2024-06-09T22:32:01.003 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 48, 2024-06-09T22:32:01.003 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 48, 2024-06-09T22:32:01.004 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 48, 2024-06-09T22:32:01.004 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:01.004 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:01.004 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:01.004 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:01.004 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:01.004 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:01.004 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle": { 2024-06-09T22:32:01.005 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:01.005 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-06-09T22:32:01.005 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:01.005 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-06-09T22:32:01.005 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-06-09T22:32:01.005 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:01.005 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-06-09T22:32:01.005 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-06-09T22:32:01.006 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-06-09T22:32:01.006 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 0, 2024-06-09T22:32:01.006 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 0, 2024-06-09T22:32:01.006 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:01.006 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:01.006 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:01.006 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:01.006 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:01.007 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:01.007 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle-0x55ca24f0e500": { 2024-06-09T22:32:01.007 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:01.007 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-06-09T22:32:01.007 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:01.007 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 9, 2024-06-09T22:32:01.007 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 2489, 2024-06-09T22:32:01.007 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:01.008 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 9, 2024-06-09T22:32:01.008 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-06-09T22:32:01.008 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-06-09T22:32:01.008 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 8, 2024-06-09T22:32:01.008 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 2489, 2024-06-09T22:32:01.008 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:01.008 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:01.008 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:01.008 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:01.009 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:01.009 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:01.009 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-06-09T22:32:01.763 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-06-09T22:32:01.763 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-06-09T22:32:01.763 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-06-09T22:32:01.764 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-06-09T22:32:01.764 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-06-09T22:32:01.764 DEBUG:tasks.check_counter:Getting stats from f 2024-06-09T22:32:01.764 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mds.f perf dump 2024-06-09T22:32:01.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:01 smithi194 ceph-mon[36856]: pgmap v342: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:32:01.873 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:01 smithi104 ceph-mon[32389]: pgmap v342: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:32:01.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:01 smithi153 ceph-mon[36599]: pgmap v342: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-06-09T22:32:01.998 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:32:03.706 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:03 smithi194 ceph-mon[36856]: pgmap v343: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:32:03.834 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-06-09T22:32:03.835 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-0": { 2024-06-09T22:32:03.835 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 74231, 2024-06-09T22:32:03.835 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 138292, 2024-06-09T22:32:03.836 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 14332422, 2024-06-09T22:32:03.836 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 115901742, 2024-06-09T22:32:03.836 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 14, 2024-06-09T22:32:03.836 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 5, 2024-06-09T22:32:03.837 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 6.115767619, 2024-06-09T22:32:03.837 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 2.774745418, 2024-06-09T22:32:03.837 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 3.502197220, 2024-06-09T22:32:03.837 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.268470287, 2024-06-09T22:32:03.837 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:32:03.837 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 138290, 2024-06-09T22:32:03.837 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 2.895716486, 2024-06-09T22:32:03.838 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000020939 2024-06-09T22:32:03.838 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.838 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:32:03.838 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 192562, 2024-06-09T22:32:03.838 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.043783652, 2024-06-09T22:32:03.838 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000227 2024-06-09T22:32:03.838 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-09T22:32:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 0 2024-06-09T22:32:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-1": { 2024-06-09T22:32:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 125156, 2024-06-09T22:32:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 177527, 2024-06-09T22:32:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 41073367, 2024-06-09T22:32:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 187166481, 2024-06-09T22:32:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 14, 2024-06-09T22:32:03.840 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 6, 2024-06-09T22:32:03.840 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 8.795424638, 2024-06-09T22:32:03.840 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 4.207491735, 2024-06-09T22:32:03.840 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 5.399634912, 2024-06-09T22:32:03.840 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.312220256, 2024-06-09T22:32:03.840 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:32:03.840 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 177527, 2024-06-09T22:32:03.840 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 3.627732829, 2024-06-09T22:32:03.840 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000020434 2024-06-09T22:32:03.841 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.841 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:32:03.841 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 267177, 2024-06-09T22:32:03.841 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.044539400, 2024-06-09T22:32:03.841 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000166 2024-06-09T22:32:03.841 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.841 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 34459840, 2024-06-09T22:32:03.841 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 118426784 2024-06-09T22:32:03.842 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.842 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-2": { 2024-06-09T22:32:03.842 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 66027, 2024-06-09T22:32:03.842 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 80021, 2024-06-09T22:32:03.842 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 14408181, 2024-06-09T22:32:03.842 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 120630626, 2024-06-09T22:32:03.842 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 31, 2024-06-09T22:32:03.842 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 10, 2024-06-09T22:32:03.843 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 4.213618851, 2024-06-09T22:32:03.843 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 1.668078702, 2024-06-09T22:32:03.843 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 2.609962245, 2024-06-09T22:32:03.843 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.322359338, 2024-06-09T22:32:03.843 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:32:03.843 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 80018, 2024-06-09T22:32:03.843 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 1.586501094, 2024-06-09T22:32:03.843 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000019826 2024-06-09T22:32:03.843 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.844 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:32:03.844 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 121060, 2024-06-09T22:32:03.844 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.025648378, 2024-06-09T22:32:03.844 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000211 2024-06-09T22:32:03.844 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.844 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 2496, 2024-06-09T22:32:03.844 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 487152 2024-06-09T22:32:03.844 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.845 INFO:teuthology.orchestra.run.smithi194.stdout: "cct": { 2024-06-09T22:32:03.845 INFO:teuthology.orchestra.run.smithi194.stdout: "total_workers": 1, 2024-06-09T22:32:03.845 INFO:teuthology.orchestra.run.smithi194.stdout: "unhealthy_workers": 0 2024-06-09T22:32:03.845 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.845 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-MDSRank": { 2024-06-09T22:32:03.845 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-06-09T22:32:03.845 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-06-09T22:32:03.845 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 35264, 2024-06-09T22:32:03.846 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 24.227157377, 2024-06-09T22:32:03.846 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000687022 2024-06-09T22:32:03.846 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:03.846 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.846 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-PurgeQueue": { 2024-06-09T22:32:03.846 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-06-09T22:32:03.846 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-06-09T22:32:03.846 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 11942, 2024-06-09T22:32:03.847 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 1.123327123, 2024-06-09T22:32:03.847 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000094065 2024-06-09T22:32:03.847 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:03.847 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.847 INFO:teuthology.orchestra.run.smithi194.stdout: "mds": { 2024-06-09T22:32:03.847 INFO:teuthology.orchestra.run.smithi194.stdout: "request": 64076, 2024-06-09T22:32:03.847 INFO:teuthology.orchestra.run.smithi194.stdout: "reply": 60933, 2024-06-09T22:32:03.848 INFO:teuthology.orchestra.run.smithi194.stdout: "reply_latency": { 2024-06-09T22:32:03.848 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 60933, 2024-06-09T22:32:03.848 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 167.639302549, 2024-06-09T22:32:03.848 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.002751207 2024-06-09T22:32:03.848 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.848 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_reply": 0, 2024-06-09T22:32:03.848 INFO:teuthology.orchestra.run.smithi194.stdout: "forward": 3143, 2024-06-09T22:32:03.848 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_complete": 0, 2024-06-09T22:32:03.849 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_keys": 0, 2024-06-09T22:32:03.849 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_commit": 19, 2024-06-09T22:32:03.849 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_split": 1, 2024-06-09T22:32:03.849 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_merge": 0, 2024-06-09T22:32:03.849 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes": 521, 2024-06-09T22:32:03.849 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_top": 65, 2024-06-09T22:32:03.849 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_bottom": 456, 2024-06-09T22:32:03.849 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pin_tail": 0, 2024-06-09T22:32:03.850 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pinned": 429, 2024-06-09T22:32:03.850 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_expired": 1, 2024-06-09T22:32:03.850 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_with_caps": 16, 2024-06-09T22:32:03.850 INFO:teuthology.orchestra.run.smithi194.stdout: "caps": 16, 2024-06-09T22:32:03.850 INFO:teuthology.orchestra.run.smithi194.stdout: "subtrees": 5, 2024-06-09T22:32:03.850 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse": 97718, 2024-06-09T22:32:03.850 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_hit": 65592, 2024-06-09T22:32:03.851 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_forward": 3140, 2024-06-09T22:32:03.851 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_discover": 12, 2024-06-09T22:32:03.851 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_dir_fetch": 0, 2024-06-09T22:32:03.851 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_remote_ino": 0, 2024-06-09T22:32:03.851 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_lock": 833, 2024-06-09T22:32:03.851 INFO:teuthology.orchestra.run.smithi194.stdout: "load_cent": 0, 2024-06-09T22:32:03.851 INFO:teuthology.orchestra.run.smithi194.stdout: "q": 0, 2024-06-09T22:32:03.851 INFO:teuthology.orchestra.run.smithi194.stdout: "exported": 0, 2024-06-09T22:32:03.851 INFO:teuthology.orchestra.run.smithi194.stdout: "exported_inodes": 0, 2024-06-09T22:32:03.852 INFO:teuthology.orchestra.run.smithi194.stdout: "imported": 7, 2024-06-09T22:32:03.852 INFO:teuthology.orchestra.run.smithi194.stdout: "imported_inodes": 113, 2024-06-09T22:32:03.852 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_dir_fetch": 0, 2024-06-09T22:32:03.852 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_backtrace_fetch": 568, 2024-06-09T22:32:03.852 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_peer_discover": 0, 2024-06-09T22:32:03.852 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rfiles": 67, 2024-06-09T22:32:03.852 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rbytes": 21176629, 2024-06-09T22:32:03.852 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rsnaps": 6, 2024-06-09T22:32:03.853 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_fetch": 568, 2024-06-09T22:32:03.853 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_set_tag": 0, 2024-06-09T22:32:03.853 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_repaired": 0, 2024-06-09T22:32:03.853 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_inotable_repaired": 0, 2024-06-09T22:32:03.853 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_inodes": 88, 2024-06-09T22:32:03.853 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_base_inodes": 0, 2024-06-09T22:32:03.853 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dirfrag_rstats": 88, 2024-06-09T22:32:03.853 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_file_inodes": 480, 2024-06-09T22:32:03.854 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_inode_file_caps": 7, 2024-06-09T22:32:03.854 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_revoke": 8079, 2024-06-09T22:32:03.854 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_grant": 25028, 2024-06-09T22:32:03.854 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_trunc": 0, 2024-06-09T22:32:03.854 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-09T22:32:03.854 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-09T22:32:03.854 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps": 21959, 2024-06-09T22:32:03.854 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps_dirty": 7430, 2024-06-09T22:32:03.855 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_cap_release": 3413, 2024-06-09T22:32:03.855 INFO:teuthology.orchestra.run.smithi194.stdout: "process_request_cap_release": 40997 2024-06-09T22:32:03.855 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.855 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_cache": { 2024-06-09T22:32:03.855 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update": 6, 2024-06-09T22:32:03.855 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update_receipt": 0, 2024-06-09T22:32:03.855 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_try_discover": 0, 2024-06-09T22:32:03.855 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_send_discover": 15, 2024-06-09T22:32:03.856 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_handle_discover": 143, 2024-06-09T22:32:03.856 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays": 0, 2024-06-09T22:32:03.856 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_delayed": 0, 2024-06-09T22:32:03.856 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_enqueuing": 0, 2024-06-09T22:32:03.857 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_created": 6193, 2024-06-09T22:32:03.857 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_enqueued": 6193, 2024-06-09T22:32:03.857 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_reintegrated": 0, 2024-06-09T22:32:03.857 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_migrated": 0, 2024-06-09T22:32:03.858 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_processing": 0, 2024-06-09T22:32:03.858 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_enqueued": 0, 2024-06-09T22:32:03.858 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_prioritized": 0, 2024-06-09T22:32:03.858 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_started": 0, 2024-06-09T22:32:03.859 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_completed": 0, 2024-06-09T22:32:03.859 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_path": 5, 2024-06-09T22:32:03.859 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_inode": 299, 2024-06-09T22:32:03.859 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_enqueue_scrub": 0, 2024-06-09T22:32:03.859 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_exportdir": 0, 2024-06-09T22:32:03.859 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_flush": 0, 2024-06-09T22:32:03.860 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragmentdir": 1, 2024-06-09T22:32:03.860 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragstats": 0, 2024-06-09T22:32:03.860 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_inodestats": 0 2024-06-09T22:32:03.860 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.861 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_log": { 2024-06-09T22:32:03.861 INFO:teuthology.orchestra.run.smithi194.stdout: "evlrg": 0, 2024-06-09T22:32:03.861 INFO:teuthology.orchestra.run.smithi194.stdout: "evadd": 39589, 2024-06-09T22:32:03.861 INFO:teuthology.orchestra.run.smithi194.stdout: "evex": 0, 2024-06-09T22:32:03.861 INFO:teuthology.orchestra.run.smithi194.stdout: "evtrm": 0, 2024-06-09T22:32:03.861 INFO:teuthology.orchestra.run.smithi194.stdout: "ev": 39589, 2024-06-09T22:32:03.861 INFO:teuthology.orchestra.run.smithi194.stdout: "evexg": 0, 2024-06-09T22:32:03.862 INFO:teuthology.orchestra.run.smithi194.stdout: "evexd": 0, 2024-06-09T22:32:03.862 INFO:teuthology.orchestra.run.smithi194.stdout: "segadd": 60, 2024-06-09T22:32:03.862 INFO:teuthology.orchestra.run.smithi194.stdout: "segex": 0, 2024-06-09T22:32:03.862 INFO:teuthology.orchestra.run.smithi194.stdout: "segtrm": 0, 2024-06-09T22:32:03.862 INFO:teuthology.orchestra.run.smithi194.stdout: "seg": 60, 2024-06-09T22:32:03.862 INFO:teuthology.orchestra.run.smithi194.stdout: "segmjr": 5, 2024-06-09T22:32:03.863 INFO:teuthology.orchestra.run.smithi194.stdout: "segexg": 0, 2024-06-09T22:32:03.863 INFO:teuthology.orchestra.run.smithi194.stdout: "segexd": 0, 2024-06-09T22:32:03.863 INFO:teuthology.orchestra.run.smithi194.stdout: "expos": 4194304, 2024-06-09T22:32:03.863 INFO:teuthology.orchestra.run.smithi194.stdout: "wrpos": 191475077, 2024-06-09T22:32:03.863 INFO:teuthology.orchestra.run.smithi194.stdout: "rdpos": 0, 2024-06-09T22:32:03.863 INFO:teuthology.orchestra.run.smithi194.stdout: "jlat": { 2024-06-09T22:32:03.863 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 17540, 2024-06-09T22:32:03.863 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 29.677541445, 2024-06-09T22:32:03.864 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001691992 2024-06-09T22:32:03.864 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.864 INFO:teuthology.orchestra.run.smithi194.stdout: "replayed": 0 2024-06-09T22:32:03.864 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.864 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_mem": { 2024-06-09T22:32:03.864 INFO:teuthology.orchestra.run.smithi194.stdout: "ino": 474, 2024-06-09T22:32:03.864 INFO:teuthology.orchestra.run.smithi194.stdout: "ino+": 6667, 2024-06-09T22:32:03.864 INFO:teuthology.orchestra.run.smithi194.stdout: "ino-": 6193, 2024-06-09T22:32:03.865 INFO:teuthology.orchestra.run.smithi194.stdout: "dir": 37, 2024-06-09T22:32:03.865 INFO:teuthology.orchestra.run.smithi194.stdout: "dir+": 39, 2024-06-09T22:32:03.865 INFO:teuthology.orchestra.run.smithi194.stdout: "dir-": 2, 2024-06-09T22:32:03.865 INFO:teuthology.orchestra.run.smithi194.stdout: "dn": 521, 2024-06-09T22:32:03.865 INFO:teuthology.orchestra.run.smithi194.stdout: "dn+": 12846, 2024-06-09T22:32:03.865 INFO:teuthology.orchestra.run.smithi194.stdout: "dn-": 12325, 2024-06-09T22:32:03.865 INFO:teuthology.orchestra.run.smithi194.stdout: "cap": 16, 2024-06-09T22:32:03.865 INFO:teuthology.orchestra.run.smithi194.stdout: "cap+": 7415, 2024-06-09T22:32:03.865 INFO:teuthology.orchestra.run.smithi194.stdout: "cap-": 7399, 2024-06-09T22:32:03.866 INFO:teuthology.orchestra.run.smithi194.stdout: "rss": 69520, 2024-06-09T22:32:03.866 INFO:teuthology.orchestra.run.smithi194.stdout: "heap": 223612 2024-06-09T22:32:03.866 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.866 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_server": { 2024-06-09T22:32:03.866 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_client_request": 102409, 2024-06-09T22:32:03.866 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_server_request": 9749, 2024-06-09T22:32:03.866 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_request": 64076, 2024-06-09T22:32:03.867 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_session": 1166, 2024-06-09T22:32:03.867 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_peer_request": 19589, 2024-06-09T22:32:03.867 INFO:teuthology.orchestra.run.smithi194.stdout: "req_create_latency": { 2024-06-09T22:32:03.867 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 6198, 2024-06-09T22:32:03.867 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 78.105930560, 2024-06-09T22:32:03.867 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.012601795 2024-06-09T22:32:03.867 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.867 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getattr_latency": { 2024-06-09T22:32:03.867 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 6395, 2024-06-09T22:32:03.868 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 31.598151189, 2024-06-09T22:32:03.868 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.004941071 2024-06-09T22:32:03.868 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.868 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getfilelock_latency": { 2024-06-09T22:32:03.868 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.868 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.868 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.868 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.869 INFO:teuthology.orchestra.run.smithi194.stdout: "req_link_latency": { 2024-06-09T22:32:03.869 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.869 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.869 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.869 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.869 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookup_latency": { 2024-06-09T22:32:03.869 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 15981, 2024-06-09T22:32:03.869 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 6.821486882, 2024-06-09T22:32:03.869 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000426849 2024-06-09T22:32:03.870 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.870 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookuphash_latency": { 2024-06-09T22:32:03.870 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.870 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.870 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.870 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.870 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupino_latency": { 2024-06-09T22:32:03.871 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.871 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.871 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.871 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.871 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupname_latency": { 2024-06-09T22:32:03.871 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.872 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.872 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.872 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.872 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupparent_latency": { 2024-06-09T22:32:03.872 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.872 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.872 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.873 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.873 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupsnap_latency": { 2024-06-09T22:32:03.873 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.873 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.873 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.874 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.874 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lssnap_latency": { 2024-06-09T22:32:03.874 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.875 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.875 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.875 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.875 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mkdir_latency": { 2024-06-09T22:32:03.875 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.875 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.875 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.876 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.876 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mknod_latency": { 2024-06-09T22:32:03.876 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.876 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.876 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.876 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.876 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mksnap_latency": { 2024-06-09T22:32:03.876 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.877 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.877 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.877 INFO:teuthology.orchestra.run.smithi194.stdout: "req_open_latency": { 2024-06-09T22:32:03.877 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.877 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.877 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.878 INFO:teuthology.orchestra.run.smithi194.stdout: "req_readdir_latency": { 2024-06-09T22:32:03.878 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 24066, 2024-06-09T22:32:03.878 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 36.751751568, 2024-06-09T22:32:03.878 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001527123 2024-06-09T22:32:03.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.878 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rename_latency": { 2024-06-09T22:32:03.878 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1748, 2024-06-09T22:32:03.879 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 3.021631968, 2024-06-09T22:32:03.879 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001728622 2024-06-09T22:32:03.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.879 INFO:teuthology.orchestra.run.smithi194.stdout: "req_renamesnap_latency": { 2024-06-09T22:32:03.879 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.879 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.879 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.879 INFO:teuthology.orchestra.run.smithi194.stdout: "req_snapdiff_latency": { 2024-06-09T22:32:03.880 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.880 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.880 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.880 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.880 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmdir_latency": { 2024-06-09T22:32:03.880 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.880 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.880 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.880 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.881 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmsnap_latency": { 2024-06-09T22:32:03.881 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.881 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.881 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.881 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.881 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmxattr_latency": { 2024-06-09T22:32:03.881 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.881 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.882 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.882 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.882 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setattr_latency": { 2024-06-09T22:32:03.882 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 4, 2024-06-09T22:32:03.882 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.057319290, 2024-06-09T22:32:03.882 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.014329822 2024-06-09T22:32:03.882 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.882 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setdirlayout_latency": { 2024-06-09T22:32:03.883 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.883 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.883 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.883 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.883 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setfilelock_latency": { 2024-06-09T22:32:03.883 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 348, 2024-06-09T22:32:03.883 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.206060385, 2024-06-09T22:32:03.883 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000592127 2024-06-09T22:32:03.883 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.884 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setlayout_latency": { 2024-06-09T22:32:03.884 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.884 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.884 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.884 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setxattr_latency": { 2024-06-09T22:32:03.884 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.884 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.885 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.885 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.885 INFO:teuthology.orchestra.run.smithi194.stdout: "req_symlink_latency": { 2024-06-09T22:32:03.885 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.885 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.885 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.885 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.885 INFO:teuthology.orchestra.run.smithi194.stdout: "req_unlink_latency": { 2024-06-09T22:32:03.885 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 6193, 2024-06-09T22:32:03.886 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 11.076970707, 2024-06-09T22:32:03.886 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001788627 2024-06-09T22:32:03.886 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.886 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_revoke_eviction": 0, 2024-06-09T22:32:03.886 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_acquisition_throttle": 0, 2024-06-09T22:32:03.886 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getvxattr_latency": { 2024-06-09T22:32:03.886 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.886 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.887 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.887 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:03.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.887 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_sessions": { 2024-06-09T22:32:03.887 INFO:teuthology.orchestra.run.smithi194.stdout: "session_count": 1, 2024-06-09T22:32:03.887 INFO:teuthology.orchestra.run.smithi194.stdout: "session_add": 3, 2024-06-09T22:32:03.887 INFO:teuthology.orchestra.run.smithi194.stdout: "session_remove": 2, 2024-06-09T22:32:03.887 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_open": 1, 2024-06-09T22:32:03.888 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_stale": 0, 2024-06-09T22:32:03.888 INFO:teuthology.orchestra.run.smithi194.stdout: "total_load": 22652, 2024-06-09T22:32:03.888 INFO:teuthology.orchestra.run.smithi194.stdout: "average_load": 22652, 2024-06-09T22:32:03.888 INFO:teuthology.orchestra.run.smithi194.stdout: "avg_session_uptime": 666, 2024-06-09T22:32:03.888 INFO:teuthology.orchestra.run.smithi194.stdout: "mdthresh_evicted": 0 2024-06-09T22:32:03.888 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.888 INFO:teuthology.orchestra.run.smithi194.stdout: "mempool": { 2024-06-09T22:32:03.888 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_bytes": 0, 2024-06-09T22:32:03.888 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_items": 0, 2024-06-09T22:32:03.889 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_bytes": 0, 2024-06-09T22:32:03.889 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_items": 0, 2024-06-09T22:32:03.889 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_bytes": 0, 2024-06-09T22:32:03.889 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_items": 0, 2024-06-09T22:32:03.889 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-09T22:32:03.889 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_items": 0, 2024-06-09T22:32:03.889 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-09T22:32:03.890 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_items": 0, 2024-06-09T22:32:03.890 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_bytes": 0, 2024-06-09T22:32:03.890 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_items": 0, 2024-06-09T22:32:03.890 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-09T22:32:03.890 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_items": 0, 2024-06-09T22:32:03.890 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_bytes": 0, 2024-06-09T22:32:03.890 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_items": 0, 2024-06-09T22:32:03.891 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_bytes": 0, 2024-06-09T22:32:03.891 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_items": 0, 2024-06-09T22:32:03.891 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-09T22:32:03.891 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_items": 0, 2024-06-09T22:32:03.891 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-09T22:32:03.891 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_items": 0, 2024-06-09T22:32:03.891 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_bytes": 0, 2024-06-09T22:32:03.891 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_items": 0, 2024-06-09T22:32:03.891 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_bytes": 0, 2024-06-09T22:32:03.892 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_items": 0, 2024-06-09T22:32:03.892 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-09T22:32:03.892 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_items": 0, 2024-06-09T22:32:03.892 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_bytes": 0, 2024-06-09T22:32:03.892 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_items": 0, 2024-06-09T22:32:03.892 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_bytes": 0, 2024-06-09T22:32:03.892 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_items": 0, 2024-06-09T22:32:03.892 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_bytes": 0, 2024-06-09T22:32:03.892 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_items": 0, 2024-06-09T22:32:03.893 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, 2024-06-09T22:32:03.893 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, 2024-06-09T22:32:03.893 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 326644, 2024-06-09T22:32:03.893 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_items": 273, 2024-06-09T22:32:03.893 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_bytes": 88, 2024-06-09T22:32:03.893 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_items": 1, 2024-06-09T22:32:03.893 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_bytes": 0, 2024-06-09T22:32:03.893 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_items": 0, 2024-06-09T22:32:03.893 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_bytes": 0, 2024-06-09T22:32:03.894 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_items": 0, 2024-06-09T22:32:03.894 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_bytes": 0, 2024-06-09T22:32:03.894 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_items": 0, 2024-06-09T22:32:03.894 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_bytes": 9512, 2024-06-09T22:32:03.894 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_items": 182, 2024-06-09T22:32:03.894 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_bytes": 0, 2024-06-09T22:32:03.894 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_items": 0, 2024-06-09T22:32:03.894 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_bytes": 0, 2024-06-09T22:32:03.895 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_items": 0, 2024-06-09T22:32:03.895 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_bytes": 1620230, 2024-06-09T22:32:03.897 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_items": 12344, 2024-06-09T22:32:03.897 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_bytes": 0, 2024-06-09T22:32:03.897 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_items": 0, 2024-06-09T22:32:03.897 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_bytes": 0, 2024-06-09T22:32:03.898 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_items": 0 2024-06-09T22:32:03.898 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.898 INFO:teuthology.orchestra.run.smithi194.stdout: "objecter": { 2024-06-09T22:32:03.898 INFO:teuthology.orchestra.run.smithi194.stdout: "op_active": 0, 2024-06-09T22:32:03.898 INFO:teuthology.orchestra.run.smithi194.stdout: "op_laggy": 0, 2024-06-09T22:32:03.898 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send": 31818, 2024-06-09T22:32:03.898 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send_bytes": 188748249, 2024-06-09T22:32:03.898 INFO:teuthology.orchestra.run.smithi194.stdout: "op_resend": 33, 2024-06-09T22:32:03.899 INFO:teuthology.orchestra.run.smithi194.stdout: "op_reply": 31818, 2024-06-09T22:32:03.899 INFO:teuthology.orchestra.run.smithi194.stdout: "op_latency": { 2024-06-09T22:32:03.899 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 31818, 2024-06-09T22:32:03.899 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 69.750601856, 2024-06-09T22:32:03.899 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.002192174 2024-06-09T22:32:03.899 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.899 INFO:teuthology.orchestra.run.smithi194.stdout: "op_inflight": 0, 2024-06-09T22:32:03.899 INFO:teuthology.orchestra.run.smithi194.stdout: "oplen_avg": { 2024-06-09T22:32:03.900 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 31818, 2024-06-09T22:32:03.900 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 32303 2024-06-09T22:32:03.900 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.900 INFO:teuthology.orchestra.run.smithi194.stdout: "op": 31818, 2024-06-09T22:32:03.900 INFO:teuthology.orchestra.run.smithi194.stdout: "op_r": 946, 2024-06-09T22:32:03.900 INFO:teuthology.orchestra.run.smithi194.stdout: "op_w": 30872, 2024-06-09T22:32:03.900 INFO:teuthology.orchestra.run.smithi194.stdout: "op_rmw": 0, 2024-06-09T22:32:03.900 INFO:teuthology.orchestra.run.smithi194.stdout: "op_pg": 0, 2024-06-09T22:32:03.901 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_stat": 0, 2024-06-09T22:32:03.901 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_create": 10, 2024-06-09T22:32:03.901 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_read": 378, 2024-06-09T22:32:03.901 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_write": 17918, 2024-06-09T22:32:03.901 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writefull": 220, 2024-06-09T22:32:03.901 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writesame": 0, 2024-06-09T22:32:03.901 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_append": 0, 2024-06-09T22:32:03.901 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_zero": 0, 2024-06-09T22:32:03.902 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_truncate": 1, 2024-06-09T22:32:03.902 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_delete": 12442, 2024-06-09T22:32:03.902 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_mapext": 0, 2024-06-09T22:32:03.902 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_sparse_read": 0, 2024-06-09T22:32:03.902 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_clonerange": 0, 2024-06-09T22:32:03.902 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_getxattr": 568, 2024-06-09T22:32:03.902 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_setxattr": 20, 2024-06-09T22:32:03.902 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_cmpxattr": 0, 2024-06-09T22:32:03.903 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_rmxattr": 0, 2024-06-09T22:32:03.903 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_resetxattrs": 0, 2024-06-09T22:32:03.903 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_call": 0, 2024-06-09T22:32:03.903 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_watch": 0, 2024-06-09T22:32:03.903 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_notify": 0, 2024-06-09T22:32:03.903 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_src_cmpxattr": 0, 2024-06-09T22:32:03.903 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls": 0, 2024-06-09T22:32:03.904 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls_filter": 0, 2024-06-09T22:32:03.904 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_other": 0, 2024-06-09T22:32:03.904 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_active": 0, 2024-06-09T22:32:03.904 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_send": 0, 2024-06-09T22:32:03.904 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_resend": 0, 2024-06-09T22:32:03.904 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_ping": 0, 2024-06-09T22:32:03.904 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_active": 0, 2024-06-09T22:32:03.905 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_send": 0, 2024-06-09T22:32:03.905 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_resend": 0, 2024-06-09T22:32:03.905 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_active": 0, 2024-06-09T22:32:03.905 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_send": 0, 2024-06-09T22:32:03.905 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_resend": 0, 2024-06-09T22:32:03.905 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_active": 0, 2024-06-09T22:32:03.905 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_send": 0, 2024-06-09T22:32:03.905 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_resend": 0, 2024-06-09T22:32:03.906 INFO:teuthology.orchestra.run.smithi194.stdout: "command_active": 0, 2024-06-09T22:32:03.906 INFO:teuthology.orchestra.run.smithi194.stdout: "command_send": 0, 2024-06-09T22:32:03.906 INFO:teuthology.orchestra.run.smithi194.stdout: "command_resend": 0, 2024-06-09T22:32:03.906 INFO:teuthology.orchestra.run.smithi194.stdout: "map_epoch": 108, 2024-06-09T22:32:03.906 INFO:teuthology.orchestra.run.smithi194.stdout: "map_full": 0, 2024-06-09T22:32:03.906 INFO:teuthology.orchestra.run.smithi194.stdout: "map_inc": 11, 2024-06-09T22:32:03.907 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_sessions": 12, 2024-06-09T22:32:03.907 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_open": 12, 2024-06-09T22:32:03.907 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_close": 0, 2024-06-09T22:32:03.907 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_laggy": 0, 2024-06-09T22:32:03.907 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_wr": 511, 2024-06-09T22:32:03.907 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_rd": 0, 2024-06-09T22:32:03.907 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_del": 235 2024-06-09T22:32:03.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.908 INFO:teuthology.orchestra.run.smithi194.stdout: "oft": { 2024-06-09T22:32:03.908 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_objs": 1, 2024-06-09T22:32:03.908 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_kv_pairs": 0, 2024-06-09T22:32:03.908 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_updates": 0, 2024-06-09T22:32:03.908 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_removes": 0 2024-06-09T22:32:03.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.908 INFO:teuthology.orchestra.run.smithi194.stdout: "osd-slow-ops": { 2024-06-09T22:32:03.908 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_ops_count": 307 2024-06-09T22:32:03.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.909 INFO:teuthology.orchestra.run.smithi194.stdout: "purge_queue": { 2024-06-09T22:32:03.909 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops": 0, 2024-06-09T22:32:03.909 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops_high_water": 10, 2024-06-09T22:32:03.909 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing": 0, 2024-06-09T22:32:03.909 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_high_water": 5, 2024-06-09T22:32:03.909 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed_ops": 12386, 2024-06-09T22:32:03.909 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed": 6193, 2024-06-09T22:32:03.909 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_item_in_journal": 0 2024-06-09T22:32:03.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.910 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-09T22:32:03.910 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:03.910 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-06-09T22:32:03.910 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:03.910 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 265410, 2024-06-09T22:32:03.910 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 41792046, 2024-06-09T22:32:03.910 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:03.911 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 265410, 2024-06-09T22:32:03.911 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-06-09T22:32:03.911 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-06-09T22:32:03.911 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 265410, 2024-06-09T22:32:03.911 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 41792046, 2024-06-09T22:32:03.911 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:03.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.911 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.912 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:03.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.912 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_bytes": { 2024-06-09T22:32:03.912 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:03.912 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-06-09T22:32:03.912 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:03.912 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-06-09T22:32:03.912 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-06-09T22:32:03.913 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:03.913 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-06-09T22:32:03.913 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 31818, 2024-06-09T22:32:03.913 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 189423926, 2024-06-09T22:32:03.913 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 31818, 2024-06-09T22:32:03.913 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 189423926, 2024-06-09T22:32:03.913 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:03.913 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.914 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.914 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.914 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:03.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.914 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_ops": { 2024-06-09T22:32:03.914 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:03.914 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 1024, 2024-06-09T22:32:03.914 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:03.915 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-06-09T22:32:03.915 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-06-09T22:32:03.915 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:03.915 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-06-09T22:32:03.915 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 31818, 2024-06-09T22:32:03.915 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 31818, 2024-06-09T22:32:03.915 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 31818, 2024-06-09T22:32:03.915 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 31818, 2024-06-09T22:32:03.916 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:03.916 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.916 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.916 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.916 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:03.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.916 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle": { 2024-06-09T22:32:03.916 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:03.917 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-06-09T22:32:03.917 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:03.917 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 6193, 2024-06-09T22:32:03.917 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 675677, 2024-06-09T22:32:03.917 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:03.917 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 6193, 2024-06-09T22:32:03.917 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-06-09T22:32:03.917 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-06-09T22:32:03.917 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 378, 2024-06-09T22:32:03.918 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 675677, 2024-06-09T22:32:03.918 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:03.918 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.918 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.918 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.918 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:03.918 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:03.918 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle-0x565460aa8500": { 2024-06-09T22:32:03.919 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:03.919 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-06-09T22:32:03.919 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:03.919 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 39589, 2024-06-09T22:32:03.919 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 187280773, 2024-06-09T22:32:03.920 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:03.920 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 39589, 2024-06-09T22:32:03.920 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-06-09T22:32:03.920 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-06-09T22:32:03.920 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 17540, 2024-06-09T22:32:03.920 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 187280773, 2024-06-09T22:32:03.920 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:03.920 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:03.920 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:03.921 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:03.921 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:03.921 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:03.921 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-06-09T22:32:03.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:03 smithi153 ceph-mon[36599]: pgmap v343: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:32:04.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:03 smithi104 ceph-mon[32389]: pgmap v343: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-06-09T22:32:04.669 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-06-09T22:32:04.669 INFO:tasks.check_counter:Daemon mds.f mds.imported=7 2024-06-09T22:32:04.669 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-06-09T22:32:04.670 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-06-09T22:32:04.670 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-06-09T22:32:04.670 DEBUG:tasks.check_counter:Getting stats from i 2024-06-09T22:32:04.670 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mds.i perf dump 2024-06-09T22:32:04.912 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:32:05.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:05 smithi153 ceph-mon[36599]: pgmap v344: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-06-09T22:32:06.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:05 smithi104 ceph-mon[32389]: pgmap v344: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-06-09T22:32:06.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:05 smithi194 ceph-mon[36856]: pgmap v344: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-06-09T22:32:06.610 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-06-09T22:32:06.611 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-0": { 2024-06-09T22:32:06.611 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 3064, 2024-06-09T22:32:06.611 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 3003, 2024-06-09T22:32:06.611 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 933726, 2024-06-09T22:32:06.611 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 10897402, 2024-06-09T22:32:06.611 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 186, 2024-06-09T22:32:06.612 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 5, 2024-06-09T22:32:06.612 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.330624787, 2024-06-09T22:32:06.612 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.104249447, 2024-06-09T22:32:06.612 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.129121912, 2024-06-09T22:32:06.612 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.069780489, 2024-06-09T22:32:06.612 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:32:06.612 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 3004, 2024-06-09T22:32:06.612 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.211466712, 2024-06-09T22:32:06.613 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000070395 2024-06-09T22:32:06.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.613 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:32:06.613 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1815, 2024-06-09T22:32:06.613 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000415830, 2024-06-09T22:32:06.613 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000229 2024-06-09T22:32:06.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.613 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 102640, 2024-06-09T22:32:06.614 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 362288 2024-06-09T22:32:06.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.614 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-1": { 2024-06-09T22:32:06.614 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 92749, 2024-06-09T22:32:06.614 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 56782, 2024-06-09T22:32:06.614 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 36344361, 2024-06-09T22:32:06.614 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 16168443, 2024-06-09T22:32:06.614 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 11, 2024-06-09T22:32:06.615 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 6, 2024-06-09T22:32:06.615 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 4.550744179, 2024-06-09T22:32:06.615 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 2.007252488, 2024-06-09T22:32:06.615 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 3.329723207, 2024-06-09T22:32:06.615 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.091629028, 2024-06-09T22:32:06.615 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:32:06.615 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 56783, 2024-06-09T22:32:06.615 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 1.376215060, 2024-06-09T22:32:06.616 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000024236 2024-06-09T22:32:06.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.616 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:32:06.616 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 144440, 2024-06-09T22:32:06.616 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.022470312, 2024-06-09T22:32:06.616 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000155 2024-06-09T22:32:06.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.616 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 4008320, 2024-06-09T22:32:06.617 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 8130464 2024-06-09T22:32:06.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.617 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-2": { 2024-06-09T22:32:06.617 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 3558, 2024-06-09T22:32:06.617 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 2834, 2024-06-09T22:32:06.617 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 2917573, 2024-06-09T22:32:06.617 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 1723593, 2024-06-09T22:32:06.617 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 26, 2024-06-09T22:32:06.617 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 9, 2024-06-09T22:32:06.618 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.361333868, 2024-06-09T22:32:06.618 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.108963218, 2024-06-09T22:32:06.618 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.151272113, 2024-06-09T22:32:06.618 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.103370067, 2024-06-09T22:32:06.618 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:32:06.618 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2827, 2024-06-09T22:32:06.618 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.528844613, 2024-06-09T22:32:06.618 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000187069 2024-06-09T22:32:06.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.619 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:32:06.619 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 3838, 2024-06-09T22:32:06.619 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000838827, 2024-06-09T22:32:06.619 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000218 2024-06-09T22:32:06.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.619 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 1923552, 2024-06-09T22:32:06.619 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 280336 2024-06-09T22:32:06.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.620 INFO:teuthology.orchestra.run.smithi194.stdout: "cct": { 2024-06-09T22:32:06.620 INFO:teuthology.orchestra.run.smithi194.stdout: "total_workers": 1, 2024-06-09T22:32:06.620 INFO:teuthology.orchestra.run.smithi194.stdout: "unhealthy_workers": 0 2024-06-09T22:32:06.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.620 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-MDSRank": { 2024-06-09T22:32:06.620 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-06-09T22:32:06.620 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-06-09T22:32:06.621 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2750, 2024-06-09T22:32:06.621 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 1.658208659, 2024-06-09T22:32:06.621 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000602984 2024-06-09T22:32:06.621 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:06.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.621 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-PurgeQueue": { 2024-06-09T22:32:06.621 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-06-09T22:32:06.621 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-06-09T22:32:06.622 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 893, 2024-06-09T22:32:06.622 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.112136297, 2024-06-09T22:32:06.622 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000125572 2024-06-09T22:32:06.622 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:06.622 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.622 INFO:teuthology.orchestra.run.smithi194.stdout: "mds": { 2024-06-09T22:32:06.622 INFO:teuthology.orchestra.run.smithi194.stdout: "request": 7968, 2024-06-09T22:32:06.622 INFO:teuthology.orchestra.run.smithi194.stdout: "reply": 3056, 2024-06-09T22:32:06.623 INFO:teuthology.orchestra.run.smithi194.stdout: "reply_latency": { 2024-06-09T22:32:06.623 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 3056, 2024-06-09T22:32:06.623 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 114.659120455, 2024-06-09T22:32:06.623 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.037519345 2024-06-09T22:32:06.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.623 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_reply": 0, 2024-06-09T22:32:06.623 INFO:teuthology.orchestra.run.smithi194.stdout: "forward": 4912, 2024-06-09T22:32:06.623 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_complete": 5, 2024-06-09T22:32:06.623 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_keys": 2, 2024-06-09T22:32:06.624 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_commit": 261, 2024-06-09T22:32:06.624 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_split": 0, 2024-06-09T22:32:06.624 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_merge": 0, 2024-06-09T22:32:06.624 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes": 43, 2024-06-09T22:32:06.624 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_top": 15, 2024-06-09T22:32:06.624 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_bottom": 18, 2024-06-09T22:32:06.624 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pin_tail": 10, 2024-06-09T22:32:06.624 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pinned": 21, 2024-06-09T22:32:06.624 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_expired": 302, 2024-06-09T22:32:06.625 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_with_caps": 9, 2024-06-09T22:32:06.625 INFO:teuthology.orchestra.run.smithi194.stdout: "caps": 9, 2024-06-09T22:32:06.625 INFO:teuthology.orchestra.run.smithi194.stdout: "subtrees": 4, 2024-06-09T22:32:06.625 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse": 9307, 2024-06-09T22:32:06.625 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_hit": 2783, 2024-06-09T22:32:06.625 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_forward": 4912, 2024-06-09T22:32:06.625 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_discover": 48, 2024-06-09T22:32:06.625 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_dir_fetch": 0, 2024-06-09T22:32:06.625 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_remote_ino": 0, 2024-06-09T22:32:06.626 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_lock": 68, 2024-06-09T22:32:06.626 INFO:teuthology.orchestra.run.smithi194.stdout: "load_cent": 0, 2024-06-09T22:32:06.626 INFO:teuthology.orchestra.run.smithi194.stdout: "q": 0, 2024-06-09T22:32:06.626 INFO:teuthology.orchestra.run.smithi194.stdout: "exported": 9, 2024-06-09T22:32:06.626 INFO:teuthology.orchestra.run.smithi194.stdout: "exported_inodes": 164, 2024-06-09T22:32:06.626 INFO:teuthology.orchestra.run.smithi194.stdout: "imported": 0, 2024-06-09T22:32:06.626 INFO:teuthology.orchestra.run.smithi194.stdout: "imported_inodes": 0, 2024-06-09T22:32:06.626 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_dir_fetch": 0, 2024-06-09T22:32:06.627 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_backtrace_fetch": 217, 2024-06-09T22:32:06.627 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_peer_discover": 0, 2024-06-09T22:32:06.627 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rfiles": 67, 2024-06-09T22:32:06.627 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rbytes": 21176629, 2024-06-09T22:32:06.627 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rsnaps": 6, 2024-06-09T22:32:06.627 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_fetch": 217, 2024-06-09T22:32:06.627 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_set_tag": 0, 2024-06-09T22:32:06.627 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_repaired": 0, 2024-06-09T22:32:06.627 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_inotable_repaired": 0, 2024-06-09T22:32:06.628 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_inodes": 116, 2024-06-09T22:32:06.628 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_base_inodes": 11, 2024-06-09T22:32:06.628 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dirfrag_rstats": 105, 2024-06-09T22:32:06.628 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_file_inodes": 101, 2024-06-09T22:32:06.628 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_inode_file_caps": 2, 2024-06-09T22:32:06.628 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_revoke": 1851, 2024-06-09T22:32:06.628 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_grant": 3850, 2024-06-09T22:32:06.628 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_trunc": 0, 2024-06-09T22:32:06.629 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-09T22:32:06.629 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-09T22:32:06.629 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps": 2462, 2024-06-09T22:32:06.629 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps_dirty": 404, 2024-06-09T22:32:06.629 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_cap_release": 193, 2024-06-09T22:32:06.629 INFO:teuthology.orchestra.run.smithi194.stdout: "process_request_cap_release": 1802 2024-06-09T22:32:06.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.629 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_cache": { 2024-06-09T22:32:06.629 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update": 0, 2024-06-09T22:32:06.630 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update_receipt": 44, 2024-06-09T22:32:06.630 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_try_discover": 22, 2024-06-09T22:32:06.630 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_send_discover": 95, 2024-06-09T22:32:06.630 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_handle_discover": 24, 2024-06-09T22:32:06.630 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays": 0, 2024-06-09T22:32:06.630 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_delayed": 0, 2024-06-09T22:32:06.630 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_enqueuing": 0, 2024-06-09T22:32:06.630 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_created": 500, 2024-06-09T22:32:06.631 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_enqueued": 500, 2024-06-09T22:32:06.631 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_reintegrated": 0, 2024-06-09T22:32:06.631 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_migrated": 0, 2024-06-09T22:32:06.631 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_processing": 0, 2024-06-09T22:32:06.631 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_enqueued": 0, 2024-06-09T22:32:06.631 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_prioritized": 0, 2024-06-09T22:32:06.631 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_started": 0, 2024-06-09T22:32:06.631 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_completed": 0, 2024-06-09T22:32:06.631 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_path": 5, 2024-06-09T22:32:06.632 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_inode": 114, 2024-06-09T22:32:06.632 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_enqueue_scrub": 11, 2024-06-09T22:32:06.632 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_exportdir": 10, 2024-06-09T22:32:06.632 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_flush": 0, 2024-06-09T22:32:06.632 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragmentdir": 0, 2024-06-09T22:32:06.632 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragstats": 0, 2024-06-09T22:32:06.632 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_inodestats": 0 2024-06-09T22:32:06.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.632 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_log": { 2024-06-09T22:32:06.633 INFO:teuthology.orchestra.run.smithi194.stdout: "evlrg": 0, 2024-06-09T22:32:06.633 INFO:teuthology.orchestra.run.smithi194.stdout: "evadd": 2512, 2024-06-09T22:32:06.633 INFO:teuthology.orchestra.run.smithi194.stdout: "evex": 2476, 2024-06-09T22:32:06.633 INFO:teuthology.orchestra.run.smithi194.stdout: "evtrm": 2476, 2024-06-09T22:32:06.633 INFO:teuthology.orchestra.run.smithi194.stdout: "ev": 36, 2024-06-09T22:32:06.633 INFO:teuthology.orchestra.run.smithi194.stdout: "evexg": 0, 2024-06-09T22:32:06.633 INFO:teuthology.orchestra.run.smithi194.stdout: "evexd": 11, 2024-06-09T22:32:06.633 INFO:teuthology.orchestra.run.smithi194.stdout: "segadd": 40, 2024-06-09T22:32:06.633 INFO:teuthology.orchestra.run.smithi194.stdout: "segex": 39, 2024-06-09T22:32:06.634 INFO:teuthology.orchestra.run.smithi194.stdout: "segtrm": 39, 2024-06-09T22:32:06.634 INFO:teuthology.orchestra.run.smithi194.stdout: "seg": 1, 2024-06-09T22:32:06.634 INFO:teuthology.orchestra.run.smithi194.stdout: "segmjr": 1, 2024-06-09T22:32:06.634 INFO:teuthology.orchestra.run.smithi194.stdout: "segexg": 0, 2024-06-09T22:32:06.634 INFO:teuthology.orchestra.run.smithi194.stdout: "segexd": 1, 2024-06-09T22:32:06.634 INFO:teuthology.orchestra.run.smithi194.stdout: "expos": 11260279, 2024-06-09T22:32:06.634 INFO:teuthology.orchestra.run.smithi194.stdout: "wrpos": 11614293, 2024-06-09T22:32:06.634 INFO:teuthology.orchestra.run.smithi194.stdout: "rdpos": 0, 2024-06-09T22:32:06.635 INFO:teuthology.orchestra.run.smithi194.stdout: "jlat": { 2024-06-09T22:32:06.635 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1029, 2024-06-09T22:32:06.635 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 1.999494370, 2024-06-09T22:32:06.635 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001943143 2024-06-09T22:32:06.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.635 INFO:teuthology.orchestra.run.smithi194.stdout: "replayed": 0 2024-06-09T22:32:06.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.635 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_mem": { 2024-06-09T22:32:06.635 INFO:teuthology.orchestra.run.smithi194.stdout: "ino": 47, 2024-06-09T22:32:06.636 INFO:teuthology.orchestra.run.smithi194.stdout: "ino+": 712, 2024-06-09T22:32:06.636 INFO:teuthology.orchestra.run.smithi194.stdout: "ino-": 665, 2024-06-09T22:32:06.636 INFO:teuthology.orchestra.run.smithi194.stdout: "dir": 47, 2024-06-09T22:32:06.636 INFO:teuthology.orchestra.run.smithi194.stdout: "dir+": 77, 2024-06-09T22:32:06.636 INFO:teuthology.orchestra.run.smithi194.stdout: "dir-": 30, 2024-06-09T22:32:06.636 INFO:teuthology.orchestra.run.smithi194.stdout: "dn": 43, 2024-06-09T22:32:06.636 INFO:teuthology.orchestra.run.smithi194.stdout: "dn+": 1416, 2024-06-09T22:32:06.636 INFO:teuthology.orchestra.run.smithi194.stdout: "dn-": 1373, 2024-06-09T22:32:06.636 INFO:teuthology.orchestra.run.smithi194.stdout: "cap": 9, 2024-06-09T22:32:06.637 INFO:teuthology.orchestra.run.smithi194.stdout: "cap+": 593, 2024-06-09T22:32:06.637 INFO:teuthology.orchestra.run.smithi194.stdout: "cap-": 584, 2024-06-09T22:32:06.637 INFO:teuthology.orchestra.run.smithi194.stdout: "rss": 67720, 2024-06-09T22:32:06.637 INFO:teuthology.orchestra.run.smithi194.stdout: "heap": 231804 2024-06-09T22:32:06.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.637 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_server": { 2024-06-09T22:32:06.637 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_client_request": 8688, 2024-06-09T22:32:06.637 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_server_request": 91, 2024-06-09T22:32:06.637 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_request": 7968, 2024-06-09T22:32:06.638 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_session": 120, 2024-06-09T22:32:06.638 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_peer_request": 93, 2024-06-09T22:32:06.638 INFO:teuthology.orchestra.run.smithi194.stdout: "req_create_latency": { 2024-06-09T22:32:06.638 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 562, 2024-06-09T22:32:06.638 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.579237296, 2024-06-09T22:32:06.638 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001030671 2024-06-09T22:32:06.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.638 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getattr_latency": { 2024-06-09T22:32:06.639 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 120, 2024-06-09T22:32:06.639 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.036874513, 2024-06-09T22:32:06.639 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000307287 2024-06-09T22:32:06.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.639 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getfilelock_latency": { 2024-06-09T22:32:06.639 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.639 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.639 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.640 INFO:teuthology.orchestra.run.smithi194.stdout: "req_link_latency": { 2024-06-09T22:32:06.640 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.640 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.640 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.640 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookup_latency": { 2024-06-09T22:32:06.640 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 931, 2024-06-09T22:32:06.640 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 91.392926889, 2024-06-09T22:32:06.640 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.098166409 2024-06-09T22:32:06.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.641 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookuphash_latency": { 2024-06-09T22:32:06.641 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.641 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.641 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.641 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupino_latency": { 2024-06-09T22:32:06.641 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1, 2024-06-09T22:32:06.642 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000124884, 2024-06-09T22:32:06.642 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000124884 2024-06-09T22:32:06.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.642 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupname_latency": { 2024-06-09T22:32:06.642 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.642 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.642 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.642 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupparent_latency": { 2024-06-09T22:32:06.643 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.643 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.643 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.643 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupsnap_latency": { 2024-06-09T22:32:06.643 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.643 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.643 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.644 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lssnap_latency": { 2024-06-09T22:32:06.644 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 11, 2024-06-09T22:32:06.645 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.002182108, 2024-06-09T22:32:06.645 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000198373 2024-06-09T22:32:06.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.645 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mkdir_latency": { 2024-06-09T22:32:06.645 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 20, 2024-06-09T22:32:06.645 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 19.887650029, 2024-06-09T22:32:06.645 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.994382501 2024-06-09T22:32:06.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.645 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mknod_latency": { 2024-06-09T22:32:06.646 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.646 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.646 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.646 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mksnap_latency": { 2024-06-09T22:32:06.646 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 11, 2024-06-09T22:32:06.646 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.078400726, 2024-06-09T22:32:06.646 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.007127338 2024-06-09T22:32:06.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.647 INFO:teuthology.orchestra.run.smithi194.stdout: "req_open_latency": { 2024-06-09T22:32:06.647 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1, 2024-06-09T22:32:06.647 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000251888, 2024-06-09T22:32:06.647 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000251888 2024-06-09T22:32:06.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.647 INFO:teuthology.orchestra.run.smithi194.stdout: "req_readdir_latency": { 2024-06-09T22:32:06.647 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 759, 2024-06-09T22:32:06.647 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 1.820108311, 2024-06-09T22:32:06.648 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.002398034 2024-06-09T22:32:06.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.648 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rename_latency": { 2024-06-09T22:32:06.648 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 108, 2024-06-09T22:32:06.648 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.145571089, 2024-06-09T22:32:06.648 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001347880 2024-06-09T22:32:06.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.648 INFO:teuthology.orchestra.run.smithi194.stdout: "req_renamesnap_latency": { 2024-06-09T22:32:06.649 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.649 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.649 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.649 INFO:teuthology.orchestra.run.smithi194.stdout: "req_snapdiff_latency": { 2024-06-09T22:32:06.649 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.649 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.649 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.650 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmdir_latency": { 2024-06-09T22:32:06.650 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.650 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.650 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.650 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmsnap_latency": { 2024-06-09T22:32:06.650 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 5, 2024-06-09T22:32:06.650 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.026350515, 2024-06-09T22:32:06.651 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.005270103 2024-06-09T22:32:06.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.651 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmxattr_latency": { 2024-06-09T22:32:06.651 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.651 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.651 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.651 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setattr_latency": { 2024-06-09T22:32:06.652 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2, 2024-06-09T22:32:06.652 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.001487742, 2024-06-09T22:32:06.652 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000743871 2024-06-09T22:32:06.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.652 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setdirlayout_latency": { 2024-06-09T22:32:06.652 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.652 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.652 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.653 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setfilelock_latency": { 2024-06-09T22:32:06.653 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 16, 2024-06-09T22:32:06.653 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.004635175, 2024-06-09T22:32:06.653 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000289698 2024-06-09T22:32:06.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.653 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setlayout_latency": { 2024-06-09T22:32:06.653 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.653 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.654 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.654 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setxattr_latency": { 2024-06-09T22:32:06.654 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 9, 2024-06-09T22:32:06.654 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.023802463, 2024-06-09T22:32:06.654 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.002644718 2024-06-09T22:32:06.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.655 INFO:teuthology.orchestra.run.smithi194.stdout: "req_symlink_latency": { 2024-06-09T22:32:06.655 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.655 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.655 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.655 INFO:teuthology.orchestra.run.smithi194.stdout: "req_unlink_latency": { 2024-06-09T22:32:06.655 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 500, 2024-06-09T22:32:06.655 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.659516827, 2024-06-09T22:32:06.656 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001319033 2024-06-09T22:32:06.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.656 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_revoke_eviction": 0, 2024-06-09T22:32:06.656 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_acquisition_throttle": 0, 2024-06-09T22:32:06.656 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getvxattr_latency": { 2024-06-09T22:32:06.656 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.656 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.656 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.657 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:06.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.657 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_sessions": { 2024-06-09T22:32:06.657 INFO:teuthology.orchestra.run.smithi194.stdout: "session_count": 2, 2024-06-09T22:32:06.657 INFO:teuthology.orchestra.run.smithi194.stdout: "session_add": 7, 2024-06-09T22:32:06.657 INFO:teuthology.orchestra.run.smithi194.stdout: "session_remove": 5, 2024-06-09T22:32:06.657 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_open": 2, 2024-06-09T22:32:06.657 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_stale": 0, 2024-06-09T22:32:06.657 INFO:teuthology.orchestra.run.smithi194.stdout: "total_load": 33, 2024-06-09T22:32:06.658 INFO:teuthology.orchestra.run.smithi194.stdout: "average_load": 16, 2024-06-09T22:32:06.658 INFO:teuthology.orchestra.run.smithi194.stdout: "avg_session_uptime": 636, 2024-06-09T22:32:06.658 INFO:teuthology.orchestra.run.smithi194.stdout: "mdthresh_evicted": 0 2024-06-09T22:32:06.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.658 INFO:teuthology.orchestra.run.smithi194.stdout: "mempool": { 2024-06-09T22:32:06.658 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_bytes": 0, 2024-06-09T22:32:06.658 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_items": 0, 2024-06-09T22:32:06.658 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_bytes": 0, 2024-06-09T22:32:06.659 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_items": 0, 2024-06-09T22:32:06.659 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_bytes": 0, 2024-06-09T22:32:06.659 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_items": 0, 2024-06-09T22:32:06.659 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-09T22:32:06.659 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_items": 0, 2024-06-09T22:32:06.659 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-09T22:32:06.659 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_items": 0, 2024-06-09T22:32:06.659 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_bytes": 0, 2024-06-09T22:32:06.660 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_items": 0, 2024-06-09T22:32:06.660 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-09T22:32:06.660 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_items": 0, 2024-06-09T22:32:06.660 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_bytes": 0, 2024-06-09T22:32:06.660 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_items": 0, 2024-06-09T22:32:06.660 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_bytes": 0, 2024-06-09T22:32:06.660 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_items": 0, 2024-06-09T22:32:06.660 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-09T22:32:06.660 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_items": 0, 2024-06-09T22:32:06.661 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-09T22:32:06.661 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_items": 0, 2024-06-09T22:32:06.661 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_bytes": 0, 2024-06-09T22:32:06.661 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_items": 0, 2024-06-09T22:32:06.661 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_bytes": 0, 2024-06-09T22:32:06.661 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_items": 0, 2024-06-09T22:32:06.661 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-09T22:32:06.661 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_items": 0, 2024-06-09T22:32:06.662 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_bytes": 0, 2024-06-09T22:32:06.662 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_items": 0, 2024-06-09T22:32:06.662 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_bytes": 0, 2024-06-09T22:32:06.662 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_items": 0, 2024-06-09T22:32:06.662 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_bytes": 0, 2024-06-09T22:32:06.662 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_items": 0, 2024-06-09T22:32:06.662 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, 2024-06-09T22:32:06.662 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, 2024-06-09T22:32:06.663 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 320269, 2024-06-09T22:32:06.663 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_items": 127, 2024-06-09T22:32:06.663 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_bytes": 0, 2024-06-09T22:32:06.663 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_items": 0, 2024-06-09T22:32:06.663 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_bytes": 0, 2024-06-09T22:32:06.663 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_items": 0, 2024-06-09T22:32:06.663 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_bytes": 0, 2024-06-09T22:32:06.663 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_items": 0, 2024-06-09T22:32:06.664 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_bytes": 0, 2024-06-09T22:32:06.664 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_items": 0, 2024-06-09T22:32:06.664 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_bytes": 9512, 2024-06-09T22:32:06.664 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_items": 182, 2024-06-09T22:32:06.664 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_bytes": 0, 2024-06-09T22:32:06.664 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_items": 0, 2024-06-09T22:32:06.664 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_bytes": 0, 2024-06-09T22:32:06.664 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_items": 0, 2024-06-09T22:32:06.665 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_bytes": 269221, 2024-06-09T22:32:06.665 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_items": 2056, 2024-06-09T22:32:06.665 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_bytes": 0, 2024-06-09T22:32:06.665 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_items": 0, 2024-06-09T22:32:06.665 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_bytes": 0, 2024-06-09T22:32:06.665 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_items": 0 2024-06-09T22:32:06.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.665 INFO:teuthology.orchestra.run.smithi194.stdout: "objecter": { 2024-06-09T22:32:06.666 INFO:teuthology.orchestra.run.smithi194.stdout: "op_active": 0, 2024-06-09T22:32:06.666 INFO:teuthology.orchestra.run.smithi194.stdout: "op_laggy": 0, 2024-06-09T22:32:06.666 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send": 4059, 2024-06-09T22:32:06.666 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send_bytes": 8673243, 2024-06-09T22:32:06.666 INFO:teuthology.orchestra.run.smithi194.stdout: "op_resend": 31, 2024-06-09T22:32:06.666 INFO:teuthology.orchestra.run.smithi194.stdout: "op_reply": 4059, 2024-06-09T22:32:06.666 INFO:teuthology.orchestra.run.smithi194.stdout: "op_latency": { 2024-06-09T22:32:06.666 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 4059, 2024-06-09T22:32:06.667 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 77.999440265, 2024-06-09T22:32:06.667 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.019216417 2024-06-09T22:32:06.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.667 INFO:teuthology.orchestra.run.smithi194.stdout: "op_inflight": 0, 2024-06-09T22:32:06.667 INFO:teuthology.orchestra.run.smithi194.stdout: "oplen_avg": { 2024-06-09T22:32:06.667 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 4059, 2024-06-09T22:32:06.667 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 5753 2024-06-09T22:32:06.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.668 INFO:teuthology.orchestra.run.smithi194.stdout: "op": 4059, 2024-06-09T22:32:06.668 INFO:teuthology.orchestra.run.smithi194.stdout: "op_r": 258, 2024-06-09T22:32:06.668 INFO:teuthology.orchestra.run.smithi194.stdout: "op_w": 3801, 2024-06-09T22:32:06.668 INFO:teuthology.orchestra.run.smithi194.stdout: "op_rmw": 0, 2024-06-09T22:32:06.668 INFO:teuthology.orchestra.run.smithi194.stdout: "op_pg": 0, 2024-06-09T22:32:06.668 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_stat": 229, 2024-06-09T22:32:06.668 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_create": 763, 2024-06-09T22:32:06.668 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_read": 23, 2024-06-09T22:32:06.669 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_write": 1041, 2024-06-09T22:32:06.669 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writefull": 144, 2024-06-09T22:32:06.669 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writesame": 0, 2024-06-09T22:32:06.669 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_append": 0, 2024-06-09T22:32:06.669 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_zero": 0, 2024-06-09T22:32:06.669 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_truncate": 0, 2024-06-09T22:32:06.669 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_delete": 1514, 2024-06-09T22:32:06.669 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_mapext": 0, 2024-06-09T22:32:06.669 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_sparse_read": 0, 2024-06-09T22:32:06.670 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_clonerange": 0, 2024-06-09T22:32:06.670 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_getxattr": 235, 2024-06-09T22:32:06.670 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_setxattr": 1163, 2024-06-09T22:32:06.670 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_cmpxattr": 0, 2024-06-09T22:32:06.670 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_rmxattr": 0, 2024-06-09T22:32:06.670 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_resetxattrs": 0, 2024-06-09T22:32:06.670 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_call": 0, 2024-06-09T22:32:06.670 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_watch": 0, 2024-06-09T22:32:06.671 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_notify": 0, 2024-06-09T22:32:06.671 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_src_cmpxattr": 0, 2024-06-09T22:32:06.671 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls": 0, 2024-06-09T22:32:06.671 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls_filter": 0, 2024-06-09T22:32:06.671 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_other": 0, 2024-06-09T22:32:06.671 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_active": 0, 2024-06-09T22:32:06.671 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_send": 0, 2024-06-09T22:32:06.671 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_resend": 0, 2024-06-09T22:32:06.672 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_ping": 0, 2024-06-09T22:32:06.672 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_active": 0, 2024-06-09T22:32:06.672 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_send": 0, 2024-06-09T22:32:06.672 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_resend": 0, 2024-06-09T22:32:06.672 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_active": 0, 2024-06-09T22:32:06.672 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_send": 0, 2024-06-09T22:32:06.672 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_resend": 0, 2024-06-09T22:32:06.672 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_active": 0, 2024-06-09T22:32:06.673 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_send": 0, 2024-06-09T22:32:06.673 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_resend": 0, 2024-06-09T22:32:06.673 INFO:teuthology.orchestra.run.smithi194.stdout: "command_active": 0, 2024-06-09T22:32:06.673 INFO:teuthology.orchestra.run.smithi194.stdout: "command_send": 0, 2024-06-09T22:32:06.673 INFO:teuthology.orchestra.run.smithi194.stdout: "command_resend": 0, 2024-06-09T22:32:06.673 INFO:teuthology.orchestra.run.smithi194.stdout: "map_epoch": 108, 2024-06-09T22:32:06.673 INFO:teuthology.orchestra.run.smithi194.stdout: "map_full": 0, 2024-06-09T22:32:06.673 INFO:teuthology.orchestra.run.smithi194.stdout: "map_inc": 20, 2024-06-09T22:32:06.674 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_sessions": 12, 2024-06-09T22:32:06.674 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_open": 12, 2024-06-09T22:32:06.674 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_close": 0, 2024-06-09T22:32:06.674 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_laggy": 0, 2024-06-09T22:32:06.674 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_wr": 603, 2024-06-09T22:32:06.674 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_rd": 14, 2024-06-09T22:32:06.674 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_del": 24 2024-06-09T22:32:06.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.675 INFO:teuthology.orchestra.run.smithi194.stdout: "oft": { 2024-06-09T22:32:06.675 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_objs": 1, 2024-06-09T22:32:06.675 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_kv_pairs": 0, 2024-06-09T22:32:06.675 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_updates": 0, 2024-06-09T22:32:06.675 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_removes": 0 2024-06-09T22:32:06.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.675 INFO:teuthology.orchestra.run.smithi194.stdout: "osd-slow-ops": { 2024-06-09T22:32:06.675 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_ops_count": 114 2024-06-09T22:32:06.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.676 INFO:teuthology.orchestra.run.smithi194.stdout: "purge_queue": { 2024-06-09T22:32:06.676 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops": 0, 2024-06-09T22:32:06.676 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops_high_water": 10, 2024-06-09T22:32:06.676 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing": 0, 2024-06-09T22:32:06.676 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_high_water": 5, 2024-06-09T22:32:06.676 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed_ops": 1000, 2024-06-09T22:32:06.676 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed": 500, 2024-06-09T22:32:06.677 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_item_in_journal": 0 2024-06-09T22:32:06.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.677 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-09T22:32:06.677 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:06.677 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-06-09T22:32:06.677 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:06.677 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 99367, 2024-06-09T22:32:06.677 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 30685660, 2024-06-09T22:32:06.677 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:06.678 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 99367, 2024-06-09T22:32:06.678 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-06-09T22:32:06.678 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-06-09T22:32:06.678 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 99367, 2024-06-09T22:32:06.678 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 30685660, 2024-06-09T22:32:06.678 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:06.678 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.678 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.679 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.679 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:06.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.679 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_bytes": { 2024-06-09T22:32:06.679 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:06.679 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-06-09T22:32:06.679 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:06.679 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-06-09T22:32:06.680 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-06-09T22:32:06.680 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:06.680 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-06-09T22:32:06.680 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 4059, 2024-06-09T22:32:06.680 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 8727655, 2024-06-09T22:32:06.680 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 4059, 2024-06-09T22:32:06.680 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 8727655, 2024-06-09T22:32:06.680 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:06.681 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.681 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.681 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.681 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:06.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.681 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_ops": { 2024-06-09T22:32:06.681 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:06.681 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 1024, 2024-06-09T22:32:06.682 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:06.682 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-06-09T22:32:06.682 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-06-09T22:32:06.682 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:06.682 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-06-09T22:32:06.682 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 4059, 2024-06-09T22:32:06.682 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 4059, 2024-06-09T22:32:06.682 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 4059, 2024-06-09T22:32:06.682 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 4059, 2024-06-09T22:32:06.683 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:06.683 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.683 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.683 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.683 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:06.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.683 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle": { 2024-06-09T22:32:06.683 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:06.684 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-06-09T22:32:06.684 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:06.684 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 500, 2024-06-09T22:32:06.684 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 54500, 2024-06-09T22:32:06.684 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:06.684 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 500, 2024-06-09T22:32:06.684 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-06-09T22:32:06.684 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-06-09T22:32:06.684 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 12, 2024-06-09T22:32:06.685 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 54500, 2024-06-09T22:32:06.685 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:06.685 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.685 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.685 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.685 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:06.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:06.685 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle-0x55d255bba500": { 2024-06-09T22:32:06.686 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 60, 2024-06-09T22:32:06.686 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-06-09T22:32:06.686 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:06.686 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 2512, 2024-06-09T22:32:06.686 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 7419989, 2024-06-09T22:32:06.686 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:06.686 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 2512, 2024-06-09T22:32:06.686 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-06-09T22:32:06.687 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-06-09T22:32:06.687 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 1029, 2024-06-09T22:32:06.687 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 7419929, 2024-06-09T22:32:06.687 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:06.687 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:06.687 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:06.687 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:06.687 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:06.688 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:06.688 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-06-09T22:32:07.409 INFO:tasks.check_counter:Daemon mds.i mds.exported=9 2024-06-09T22:32:07.409 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-06-09T22:32:07.409 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-06-09T22:32:07.409 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=11 2024-06-09T22:32:07.409 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=5 2024-06-09T22:32:07.409 DEBUG:tasks.check_counter:Getting stats from l 2024-06-09T22:32:07.410 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph daemon mds.l perf dump 2024-06-09T22:32:07.646 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.c/config 2024-06-09T22:32:07.767 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:07 smithi194 ceph-mon[36856]: pgmap v345: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-06-09T22:32:07.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:07 smithi153 ceph-mon[36599]: pgmap v345: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-06-09T22:32:08.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:07 smithi104 ceph-mon[32389]: pgmap v345: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-06-09T22:32:09.386 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-06-09T22:32:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-0": { 2024-06-09T22:32:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 48244, 2024-06-09T22:32:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 24142, 2024-06-09T22:32:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 22898943, 2024-06-09T22:32:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 4210271, 2024-06-09T22:32:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 19, 2024-06-09T22:32:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 5, 2024-06-09T22:32:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 2.433815958, 2024-06-09T22:32:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 1.075192147, 2024-06-09T22:32:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 1.855124108, 2024-06-09T22:32:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.000423155, 2024-06-09T22:32:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:32:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 24138, 2024-06-09T22:32:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.465747538, 2024-06-09T22:32:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000019295 2024-06-09T22:32:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:32:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 72312, 2024-06-09T22:32:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.012002307, 2024-06-09T22:32:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000165 2024-06-09T22:32:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 4976, 2024-06-09T22:32:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 13712 2024-06-09T22:32:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-1": { 2024-06-09T22:32:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 1054, 2024-06-09T22:32:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 1068, 2024-06-09T22:32:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 1766960, 2024-06-09T22:32:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 604698, 2024-06-09T22:32:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 11, 2024-06-09T22:32:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 5, 2024-06-09T22:32:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.133352652, 2024-06-09T22:32:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.054068016, 2024-06-09T22:32:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.080461819, 2024-06-09T22:32:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.010599168, 2024-06-09T22:32:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:32:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1070, 2024-06-09T22:32:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.053185393, 2024-06-09T22:32:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000049705 2024-06-09T22:32:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:32:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1439, 2024-06-09T22:32:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000351300, 2024-06-09T22:32:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000244 2024-06-09T22:32:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 1531776, 2024-06-09T22:32:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 169056 2024-06-09T22:32:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-2": { 2024-06-09T22:32:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 188, 2024-06-09T22:32:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 418, 2024-06-09T22:32:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 55081, 2024-06-09T22:32:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 719381, 2024-06-09T22:32:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 25, 2024-06-09T22:32:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 8, 2024-06-09T22:32:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.038048220, 2024-06-09T22:32:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.015643506, 2024-06-09T22:32:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.021653619, 2024-06-09T22:32:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.000386981, 2024-06-09T22:32:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-06-09T22:32:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 415, 2024-06-09T22:32:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.029848018, 2024-06-09T22:32:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000071922 2024-06-09T22:32:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-06-09T22:32:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 350, 2024-06-09T22:32:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000072831, 2024-06-09T22:32:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000208 2024-06-09T22:32:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 48688, 2024-06-09T22:32:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 517040 2024-06-09T22:32:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "cct": { 2024-06-09T22:32:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "total_workers": 1, 2024-06-09T22:32:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "unhealthy_workers": 0 2024-06-09T22:32:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-MDSRank": { 2024-06-09T22:32:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-06-09T22:32:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-06-09T22:32:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 46, 2024-06-09T22:32:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.002167485, 2024-06-09T22:32:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000047119 2024-06-09T22:32:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-PurgeQueue": { 2024-06-09T22:32:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-06-09T22:32:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-06-09T22:32:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2, 2024-06-09T22:32:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000007818, 2024-06-09T22:32:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000003909 2024-06-09T22:32:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "mds": { 2024-06-09T22:32:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "request": 151, 2024-06-09T22:32:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "reply": 4, 2024-06-09T22:32:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "reply_latency": { 2024-06-09T22:32:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 4, 2024-06-09T22:32:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000861473, 2024-06-09T22:32:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000215368 2024-06-09T22:32:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_reply": 0, 2024-06-09T22:32:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: "forward": 147, 2024-06-09T22:32:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_complete": 0, 2024-06-09T22:32:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_keys": 0, 2024-06-09T22:32:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_commit": 11, 2024-06-09T22:32:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_split": 0, 2024-06-09T22:32:09.400 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_merge": 0, 2024-06-09T22:32:09.400 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes": 22, 2024-06-09T22:32:09.400 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_top": 2, 2024-06-09T22:32:09.400 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_bottom": 20, 2024-06-09T22:32:09.400 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pin_tail": 0, 2024-06-09T22:32:09.400 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pinned": 19, 2024-06-09T22:32:09.400 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_expired": 0, 2024-06-09T22:32:09.400 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_with_caps": 1, 2024-06-09T22:32:09.400 INFO:teuthology.orchestra.run.smithi194.stdout: "caps": 2, 2024-06-09T22:32:09.401 INFO:teuthology.orchestra.run.smithi194.stdout: "subtrees": 3, 2024-06-09T22:32:09.401 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse": 196, 2024-06-09T22:32:09.401 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_hit": 40, 2024-06-09T22:32:09.401 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_forward": 143, 2024-06-09T22:32:09.401 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_discover": 12, 2024-06-09T22:32:09.401 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_dir_fetch": 0, 2024-06-09T22:32:09.401 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_remote_ino": 0, 2024-06-09T22:32:09.401 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_lock": 0, 2024-06-09T22:32:09.402 INFO:teuthology.orchestra.run.smithi194.stdout: "load_cent": 0, 2024-06-09T22:32:09.402 INFO:teuthology.orchestra.run.smithi194.stdout: "q": 0, 2024-06-09T22:32:09.402 INFO:teuthology.orchestra.run.smithi194.stdout: "exported": 0, 2024-06-09T22:32:09.402 INFO:teuthology.orchestra.run.smithi194.stdout: "exported_inodes": 0, 2024-06-09T22:32:09.402 INFO:teuthology.orchestra.run.smithi194.stdout: "imported": 0, 2024-06-09T22:32:09.402 INFO:teuthology.orchestra.run.smithi194.stdout: "imported_inodes": 0, 2024-06-09T22:32:09.402 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_dir_fetch": 0, 2024-06-09T22:32:09.402 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_backtrace_fetch": 0, 2024-06-09T22:32:09.403 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_peer_discover": 1, 2024-06-09T22:32:09.403 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rfiles": 67, 2024-06-09T22:32:09.403 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rbytes": 21176629, 2024-06-09T22:32:09.403 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rsnaps": 6, 2024-06-09T22:32:09.403 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_fetch": 0, 2024-06-09T22:32:09.403 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_set_tag": 0, 2024-06-09T22:32:09.403 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_repaired": 0, 2024-06-09T22:32:09.403 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_inotable_repaired": 0, 2024-06-09T22:32:09.403 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_inodes": 0, 2024-06-09T22:32:09.403 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_base_inodes": 0, 2024-06-09T22:32:09.404 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dirfrag_rstats": 0, 2024-06-09T22:32:09.404 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_file_inodes": 0, 2024-06-09T22:32:09.404 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_inode_file_caps": 0, 2024-06-09T22:32:09.404 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_revoke": 2, 2024-06-09T22:32:09.404 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_grant": 0, 2024-06-09T22:32:09.404 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_trunc": 0, 2024-06-09T22:32:09.404 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-09T22:32:09.404 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-09T22:32:09.404 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps": 2, 2024-06-09T22:32:09.405 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps_dirty": 0, 2024-06-09T22:32:09.405 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_cap_release": 0, 2024-06-09T22:32:09.405 INFO:teuthology.orchestra.run.smithi194.stdout: "process_request_cap_release": 0 2024-06-09T22:32:09.405 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.405 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_cache": { 2024-06-09T22:32:09.405 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update": 0, 2024-06-09T22:32:09.405 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update_receipt": 53, 2024-06-09T22:32:09.405 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_try_discover": 29, 2024-06-09T22:32:09.406 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_send_discover": 67, 2024-06-09T22:32:09.406 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_handle_discover": 0, 2024-06-09T22:32:09.406 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays": 0, 2024-06-09T22:32:09.406 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_delayed": 0, 2024-06-09T22:32:09.406 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_enqueuing": 0, 2024-06-09T22:32:09.406 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_created": 0, 2024-06-09T22:32:09.406 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_enqueued": 0, 2024-06-09T22:32:09.407 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_reintegrated": 0, 2024-06-09T22:32:09.407 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_migrated": 0, 2024-06-09T22:32:09.407 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_processing": 0, 2024-06-09T22:32:09.407 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_enqueued": 0, 2024-06-09T22:32:09.407 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_prioritized": 0, 2024-06-09T22:32:09.407 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_started": 0, 2024-06-09T22:32:09.407 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_completed": 0, 2024-06-09T22:32:09.407 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_path": 5, 2024-06-09T22:32:09.408 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_inode": 51, 2024-06-09T22:32:09.408 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_enqueue_scrub": 0, 2024-06-09T22:32:09.408 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_exportdir": 0, 2024-06-09T22:32:09.408 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_flush": 0, 2024-06-09T22:32:09.408 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragmentdir": 0, 2024-06-09T22:32:09.408 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragstats": 0, 2024-06-09T22:32:09.408 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_inodestats": 0 2024-06-09T22:32:09.408 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.408 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_log": { 2024-06-09T22:32:09.409 INFO:teuthology.orchestra.run.smithi194.stdout: "evlrg": 0, 2024-06-09T22:32:09.409 INFO:teuthology.orchestra.run.smithi194.stdout: "evadd": 6, 2024-06-09T22:32:09.409 INFO:teuthology.orchestra.run.smithi194.stdout: "evex": 0, 2024-06-09T22:32:09.409 INFO:teuthology.orchestra.run.smithi194.stdout: "evtrm": 0, 2024-06-09T22:32:09.409 INFO:teuthology.orchestra.run.smithi194.stdout: "ev": 6, 2024-06-09T22:32:09.409 INFO:teuthology.orchestra.run.smithi194.stdout: "evexg": 0, 2024-06-09T22:32:09.409 INFO:teuthology.orchestra.run.smithi194.stdout: "evexd": 0, 2024-06-09T22:32:09.409 INFO:teuthology.orchestra.run.smithi194.stdout: "segadd": 2, 2024-06-09T22:32:09.409 INFO:teuthology.orchestra.run.smithi194.stdout: "segex": 0, 2024-06-09T22:32:09.410 INFO:teuthology.orchestra.run.smithi194.stdout: "segtrm": 0, 2024-06-09T22:32:09.410 INFO:teuthology.orchestra.run.smithi194.stdout: "seg": 2, 2024-06-09T22:32:09.410 INFO:teuthology.orchestra.run.smithi194.stdout: "segmjr": 2, 2024-06-09T22:32:09.410 INFO:teuthology.orchestra.run.smithi194.stdout: "segexg": 0, 2024-06-09T22:32:09.410 INFO:teuthology.orchestra.run.smithi194.stdout: "segexd": 0, 2024-06-09T22:32:09.410 INFO:teuthology.orchestra.run.smithi194.stdout: "expos": 4194304, 2024-06-09T22:32:09.410 INFO:teuthology.orchestra.run.smithi194.stdout: "wrpos": 4196121, 2024-06-09T22:32:09.410 INFO:teuthology.orchestra.run.smithi194.stdout: "rdpos": 0, 2024-06-09T22:32:09.411 INFO:teuthology.orchestra.run.smithi194.stdout: "jlat": { 2024-06-09T22:32:09.411 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 5, 2024-06-09T22:32:09.411 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.009296107, 2024-06-09T22:32:09.411 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001859221 2024-06-09T22:32:09.411 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.411 INFO:teuthology.orchestra.run.smithi194.stdout: "replayed": 0 2024-06-09T22:32:09.411 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.412 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_mem": { 2024-06-09T22:32:09.412 INFO:teuthology.orchestra.run.smithi194.stdout: "ino": 25, 2024-06-09T22:32:09.412 INFO:teuthology.orchestra.run.smithi194.stdout: "ino+": 25, 2024-06-09T22:32:09.412 INFO:teuthology.orchestra.run.smithi194.stdout: "ino-": 0, 2024-06-09T22:32:09.412 INFO:teuthology.orchestra.run.smithi194.stdout: "dir": 30, 2024-06-09T22:32:09.412 INFO:teuthology.orchestra.run.smithi194.stdout: "dir+": 61, 2024-06-09T22:32:09.412 INFO:teuthology.orchestra.run.smithi194.stdout: "dir-": 31, 2024-06-09T22:32:09.412 INFO:teuthology.orchestra.run.smithi194.stdout: "dn": 22, 2024-06-09T22:32:09.413 INFO:teuthology.orchestra.run.smithi194.stdout: "dn+": 22, 2024-06-09T22:32:09.413 INFO:teuthology.orchestra.run.smithi194.stdout: "dn-": 0, 2024-06-09T22:32:09.413 INFO:teuthology.orchestra.run.smithi194.stdout: "cap": 2, 2024-06-09T22:32:09.413 INFO:teuthology.orchestra.run.smithi194.stdout: "cap+": 2, 2024-06-09T22:32:09.413 INFO:teuthology.orchestra.run.smithi194.stdout: "cap-": 0, 2024-06-09T22:32:09.413 INFO:teuthology.orchestra.run.smithi194.stdout: "rss": 53620, 2024-06-09T22:32:09.413 INFO:teuthology.orchestra.run.smithi194.stdout: "heap": 223612 2024-06-09T22:32:09.414 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.414 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_server": { 2024-06-09T22:32:09.414 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_client_request": 158, 2024-06-09T22:32:09.414 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_server_request": 0, 2024-06-09T22:32:09.414 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_request": 151, 2024-06-09T22:32:09.414 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_session": 22, 2024-06-09T22:32:09.414 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_peer_request": 0, 2024-06-09T22:32:09.414 INFO:teuthology.orchestra.run.smithi194.stdout: "req_create_latency": { 2024-06-09T22:32:09.415 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.415 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.415 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.415 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.415 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getattr_latency": { 2024-06-09T22:32:09.415 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 3, 2024-06-09T22:32:09.415 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000757208, 2024-06-09T22:32:09.415 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000252402 2024-06-09T22:32:09.416 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.416 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getfilelock_latency": { 2024-06-09T22:32:09.416 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.416 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.416 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.416 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.416 INFO:teuthology.orchestra.run.smithi194.stdout: "req_link_latency": { 2024-06-09T22:32:09.417 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.417 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.417 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.417 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.417 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookup_latency": { 2024-06-09T22:32:09.417 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.417 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.417 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.418 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.418 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookuphash_latency": { 2024-06-09T22:32:09.418 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.418 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.418 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.418 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.418 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupino_latency": { 2024-06-09T22:32:09.418 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1, 2024-06-09T22:32:09.419 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000104265, 2024-06-09T22:32:09.419 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000104265 2024-06-09T22:32:09.419 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.419 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupname_latency": { 2024-06-09T22:32:09.419 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.419 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.420 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.420 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.420 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupparent_latency": { 2024-06-09T22:32:09.420 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.420 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.420 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.420 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.420 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupsnap_latency": { 2024-06-09T22:32:09.420 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.421 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.421 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.421 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.421 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lssnap_latency": { 2024-06-09T22:32:09.421 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.421 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.421 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.421 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.422 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mkdir_latency": { 2024-06-09T22:32:09.422 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.422 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.422 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.423 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.423 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mknod_latency": { 2024-06-09T22:32:09.423 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.423 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.423 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.423 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.423 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mksnap_latency": { 2024-06-09T22:32:09.423 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.423 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.424 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.424 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.424 INFO:teuthology.orchestra.run.smithi194.stdout: "req_open_latency": { 2024-06-09T22:32:09.424 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.424 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.424 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.424 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.424 INFO:teuthology.orchestra.run.smithi194.stdout: "req_readdir_latency": { 2024-06-09T22:32:09.424 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.425 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.425 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.425 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.425 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rename_latency": { 2024-06-09T22:32:09.425 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.425 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.425 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.425 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.425 INFO:teuthology.orchestra.run.smithi194.stdout: "req_renamesnap_latency": { 2024-06-09T22:32:09.426 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.426 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.426 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.426 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.426 INFO:teuthology.orchestra.run.smithi194.stdout: "req_snapdiff_latency": { 2024-06-09T22:32:09.426 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.426 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.426 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.426 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.429 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmdir_latency": { 2024-06-09T22:32:09.429 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.429 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.429 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.429 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.430 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmsnap_latency": { 2024-06-09T22:32:09.430 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.430 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.430 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.430 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.430 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmxattr_latency": { 2024-06-09T22:32:09.430 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.430 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.431 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.431 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.431 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setattr_latency": { 2024-06-09T22:32:09.431 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.431 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.431 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.431 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.431 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setdirlayout_latency": { 2024-06-09T22:32:09.432 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.432 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.432 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.432 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.432 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setfilelock_latency": { 2024-06-09T22:32:09.432 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.432 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.432 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.433 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.433 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setlayout_latency": { 2024-06-09T22:32:09.433 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.433 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.433 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.433 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.433 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setxattr_latency": { 2024-06-09T22:32:09.433 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.433 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.434 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.434 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.434 INFO:teuthology.orchestra.run.smithi194.stdout: "req_symlink_latency": { 2024-06-09T22:32:09.434 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.434 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.434 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.434 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.434 INFO:teuthology.orchestra.run.smithi194.stdout: "req_unlink_latency": { 2024-06-09T22:32:09.435 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.435 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.435 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.435 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.435 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_revoke_eviction": 0, 2024-06-09T22:32:09.435 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_acquisition_throttle": 0, 2024-06-09T22:32:09.435 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getvxattr_latency": { 2024-06-09T22:32:09.435 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.435 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.436 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.436 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:09.436 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.436 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_sessions": { 2024-06-09T22:32:09.436 INFO:teuthology.orchestra.run.smithi194.stdout: "session_count": 2, 2024-06-09T22:32:09.436 INFO:teuthology.orchestra.run.smithi194.stdout: "session_add": 3, 2024-06-09T22:32:09.436 INFO:teuthology.orchestra.run.smithi194.stdout: "session_remove": 1, 2024-06-09T22:32:09.436 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_open": 2, 2024-06-09T22:32:09.436 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_stale": 0, 2024-06-09T22:32:09.437 INFO:teuthology.orchestra.run.smithi194.stdout: "total_load": 3, 2024-06-09T22:32:09.437 INFO:teuthology.orchestra.run.smithi194.stdout: "average_load": 1, 2024-06-09T22:32:09.437 INFO:teuthology.orchestra.run.smithi194.stdout: "avg_session_uptime": 619, 2024-06-09T22:32:09.437 INFO:teuthology.orchestra.run.smithi194.stdout: "mdthresh_evicted": 0 2024-06-09T22:32:09.437 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.437 INFO:teuthology.orchestra.run.smithi194.stdout: "mempool": { 2024-06-09T22:32:09.437 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_bytes": 0, 2024-06-09T22:32:09.437 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_items": 0, 2024-06-09T22:32:09.438 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_bytes": 0, 2024-06-09T22:32:09.438 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_items": 0, 2024-06-09T22:32:09.438 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_bytes": 0, 2024-06-09T22:32:09.438 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_items": 0, 2024-06-09T22:32:09.438 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-09T22:32:09.438 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_items": 0, 2024-06-09T22:32:09.438 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-09T22:32:09.438 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_items": 0, 2024-06-09T22:32:09.438 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_bytes": 0, 2024-06-09T22:32:09.439 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_items": 0, 2024-06-09T22:32:09.439 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-09T22:32:09.439 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_items": 0, 2024-06-09T22:32:09.439 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_bytes": 0, 2024-06-09T22:32:09.439 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_items": 0, 2024-06-09T22:32:09.439 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_bytes": 0, 2024-06-09T22:32:09.439 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_items": 0, 2024-06-09T22:32:09.439 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-09T22:32:09.440 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_items": 0, 2024-06-09T22:32:09.440 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-09T22:32:09.440 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_items": 0, 2024-06-09T22:32:09.440 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_bytes": 0, 2024-06-09T22:32:09.440 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_items": 0, 2024-06-09T22:32:09.440 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_bytes": 0, 2024-06-09T22:32:09.440 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_items": 0, 2024-06-09T22:32:09.440 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-09T22:32:09.440 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_items": 0, 2024-06-09T22:32:09.441 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_bytes": 0, 2024-06-09T22:32:09.441 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_items": 0, 2024-06-09T22:32:09.441 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_bytes": 0, 2024-06-09T22:32:09.441 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_items": 0, 2024-06-09T22:32:09.441 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_bytes": 0, 2024-06-09T22:32:09.441 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_items": 0, 2024-06-09T22:32:09.441 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, 2024-06-09T22:32:09.441 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, 2024-06-09T22:32:09.441 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 248168, 2024-06-09T22:32:09.442 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_items": 103, 2024-06-09T22:32:09.442 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_bytes": 0, 2024-06-09T22:32:09.442 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_items": 0, 2024-06-09T22:32:09.442 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_bytes": 0, 2024-06-09T22:32:09.442 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_items": 0, 2024-06-09T22:32:09.442 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_bytes": 0, 2024-06-09T22:32:09.442 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_items": 0, 2024-06-09T22:32:09.442 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_bytes": 0, 2024-06-09T22:32:09.442 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_items": 0, 2024-06-09T22:32:09.443 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_bytes": 9512, 2024-06-09T22:32:09.443 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_items": 182, 2024-06-09T22:32:09.443 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_bytes": 0, 2024-06-09T22:32:09.443 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_items": 0, 2024-06-09T22:32:09.443 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_bytes": 0, 2024-06-09T22:32:09.443 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_items": 0, 2024-06-09T22:32:09.443 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_bytes": 120850, 2024-06-09T22:32:09.443 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_items": 661, 2024-06-09T22:32:09.444 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_bytes": 0, 2024-06-09T22:32:09.444 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_items": 0, 2024-06-09T22:32:09.444 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_bytes": 0, 2024-06-09T22:32:09.444 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_items": 0 2024-06-09T22:32:09.444 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.444 INFO:teuthology.orchestra.run.smithi194.stdout: "objecter": { 2024-06-09T22:32:09.444 INFO:teuthology.orchestra.run.smithi194.stdout: "op_active": 0, 2024-06-09T22:32:09.444 INFO:teuthology.orchestra.run.smithi194.stdout: "op_laggy": 0, 2024-06-09T22:32:09.444 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send": 43, 2024-06-09T22:32:09.445 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send_bytes": 12299, 2024-06-09T22:32:09.445 INFO:teuthology.orchestra.run.smithi194.stdout: "op_resend": 33, 2024-06-09T22:32:09.445 INFO:teuthology.orchestra.run.smithi194.stdout: "op_reply": 43, 2024-06-09T22:32:09.445 INFO:teuthology.orchestra.run.smithi194.stdout: "op_latency": { 2024-06-09T22:32:09.445 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 43, 2024-06-09T22:32:09.445 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.223995520, 2024-06-09T22:32:09.445 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.005209198 2024-06-09T22:32:09.445 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.445 INFO:teuthology.orchestra.run.smithi194.stdout: "op_inflight": 0, 2024-06-09T22:32:09.446 INFO:teuthology.orchestra.run.smithi194.stdout: "oplen_avg": { 2024-06-09T22:32:09.446 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 43, 2024-06-09T22:32:09.446 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 65 2024-06-09T22:32:09.446 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.446 INFO:teuthology.orchestra.run.smithi194.stdout: "op": 43, 2024-06-09T22:32:09.446 INFO:teuthology.orchestra.run.smithi194.stdout: "op_r": 0, 2024-06-09T22:32:09.446 INFO:teuthology.orchestra.run.smithi194.stdout: "op_w": 43, 2024-06-09T22:32:09.446 INFO:teuthology.orchestra.run.smithi194.stdout: "op_rmw": 0, 2024-06-09T22:32:09.447 INFO:teuthology.orchestra.run.smithi194.stdout: "op_pg": 0, 2024-06-09T22:32:09.447 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_stat": 0, 2024-06-09T22:32:09.447 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_create": 10, 2024-06-09T22:32:09.447 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_read": 0, 2024-06-09T22:32:09.447 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_write": 5, 2024-06-09T22:32:09.447 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writefull": 9, 2024-06-09T22:32:09.447 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writesame": 0, 2024-06-09T22:32:09.447 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_append": 0, 2024-06-09T22:32:09.448 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_zero": 0, 2024-06-09T22:32:09.448 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_truncate": 0, 2024-06-09T22:32:09.448 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_delete": 6, 2024-06-09T22:32:09.448 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_mapext": 0, 2024-06-09T22:32:09.448 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_sparse_read": 0, 2024-06-09T22:32:09.448 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_clonerange": 0, 2024-06-09T22:32:09.448 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_getxattr": 0, 2024-06-09T22:32:09.448 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_setxattr": 20, 2024-06-09T22:32:09.449 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_cmpxattr": 0, 2024-06-09T22:32:09.449 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_rmxattr": 0, 2024-06-09T22:32:09.449 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_resetxattrs": 0, 2024-06-09T22:32:09.449 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_call": 0, 2024-06-09T22:32:09.449 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_watch": 0, 2024-06-09T22:32:09.449 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_notify": 0, 2024-06-09T22:32:09.449 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_src_cmpxattr": 0, 2024-06-09T22:32:09.449 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls": 0, 2024-06-09T22:32:09.449 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls_filter": 0, 2024-06-09T22:32:09.450 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_other": 0, 2024-06-09T22:32:09.450 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_active": 0, 2024-06-09T22:32:09.450 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_send": 0, 2024-06-09T22:32:09.450 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_resend": 0, 2024-06-09T22:32:09.450 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_ping": 0, 2024-06-09T22:32:09.450 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_active": 0, 2024-06-09T22:32:09.450 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_send": 0, 2024-06-09T22:32:09.451 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_resend": 0, 2024-06-09T22:32:09.451 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_active": 0, 2024-06-09T22:32:09.451 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_send": 0, 2024-06-09T22:32:09.451 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_resend": 0, 2024-06-09T22:32:09.451 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_active": 0, 2024-06-09T22:32:09.451 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_send": 0, 2024-06-09T22:32:09.451 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_resend": 0, 2024-06-09T22:32:09.451 INFO:teuthology.orchestra.run.smithi194.stdout: "command_active": 0, 2024-06-09T22:32:09.451 INFO:teuthology.orchestra.run.smithi194.stdout: "command_send": 0, 2024-06-09T22:32:09.452 INFO:teuthology.orchestra.run.smithi194.stdout: "command_resend": 0, 2024-06-09T22:32:09.452 INFO:teuthology.orchestra.run.smithi194.stdout: "map_epoch": 108, 2024-06-09T22:32:09.452 INFO:teuthology.orchestra.run.smithi194.stdout: "map_full": 0, 2024-06-09T22:32:09.452 INFO:teuthology.orchestra.run.smithi194.stdout: "map_inc": 11, 2024-06-09T22:32:09.452 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_sessions": 9, 2024-06-09T22:32:09.452 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_open": 9, 2024-06-09T22:32:09.452 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_close": 0, 2024-06-09T22:32:09.452 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_laggy": 0, 2024-06-09T22:32:09.452 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_wr": 14, 2024-06-09T22:32:09.453 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_rd": 0, 2024-06-09T22:32:09.453 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_del": 1 2024-06-09T22:32:09.453 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.453 INFO:teuthology.orchestra.run.smithi194.stdout: "oft": { 2024-06-09T22:32:09.453 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_objs": 1, 2024-06-09T22:32:09.453 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_kv_pairs": 0, 2024-06-09T22:32:09.453 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_updates": 0, 2024-06-09T22:32:09.453 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_removes": 0 2024-06-09T22:32:09.453 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.454 INFO:teuthology.orchestra.run.smithi194.stdout: "osd-slow-ops": { 2024-06-09T22:32:09.454 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_ops_count": 56 2024-06-09T22:32:09.454 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.454 INFO:teuthology.orchestra.run.smithi194.stdout: "purge_queue": { 2024-06-09T22:32:09.454 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops": 0, 2024-06-09T22:32:09.454 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops_high_water": 0, 2024-06-09T22:32:09.454 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing": 0, 2024-06-09T22:32:09.454 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_high_water": 0, 2024-06-09T22:32:09.454 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed_ops": 0, 2024-06-09T22:32:09.455 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed": 0, 2024-06-09T22:32:09.455 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_item_in_journal": 0 2024-06-09T22:32:09.455 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.455 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-09T22:32:09.455 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:09.455 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-06-09T22:32:09.455 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:09.455 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 49482, 2024-06-09T22:32:09.456 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 20240681, 2024-06-09T22:32:09.456 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:09.456 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 49482, 2024-06-09T22:32:09.456 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-06-09T22:32:09.456 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-06-09T22:32:09.456 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 49482, 2024-06-09T22:32:09.456 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 20240681, 2024-06-09T22:32:09.456 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:09.456 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.457 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.457 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.457 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:09.457 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.457 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_bytes": { 2024-06-09T22:32:09.457 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:09.457 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-06-09T22:32:09.457 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:09.457 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-06-09T22:32:09.458 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-06-09T22:32:09.458 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:09.458 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-06-09T22:32:09.458 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 43, 2024-06-09T22:32:09.458 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 12299, 2024-06-09T22:32:09.458 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 43, 2024-06-09T22:32:09.458 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 12299, 2024-06-09T22:32:09.459 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:09.459 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.459 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.459 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.459 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:09.459 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.459 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_ops": { 2024-06-09T22:32:09.459 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:09.460 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 1024, 2024-06-09T22:32:09.460 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:09.460 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-06-09T22:32:09.460 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-06-09T22:32:09.460 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:09.460 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-06-09T22:32:09.460 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 43, 2024-06-09T22:32:09.460 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 43, 2024-06-09T22:32:09.460 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 43, 2024-06-09T22:32:09.461 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 43, 2024-06-09T22:32:09.461 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:09.461 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.461 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.461 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.461 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:09.461 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.462 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle": { 2024-06-09T22:32:09.462 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:09.462 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-06-09T22:32:09.462 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:09.462 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-06-09T22:32:09.462 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-06-09T22:32:09.462 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:09.462 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-06-09T22:32:09.463 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-06-09T22:32:09.463 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-06-09T22:32:09.463 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 0, 2024-06-09T22:32:09.463 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 0, 2024-06-09T22:32:09.463 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:09.463 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.463 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.463 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.464 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:09.464 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-06-09T22:32:09.464 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle-0x5623cf476500": { 2024-06-09T22:32:09.464 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-06-09T22:32:09.464 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-06-09T22:32:09.464 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-06-09T22:32:09.464 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 6, 2024-06-09T22:32:09.464 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 1817, 2024-06-09T22:32:09.465 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-06-09T22:32:09.465 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 6, 2024-06-09T22:32:09.465 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-06-09T22:32:09.465 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-06-09T22:32:09.465 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 5, 2024-06-09T22:32:09.465 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 1817, 2024-06-09T22:32:09.465 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-06-09T22:32:09.465 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-06-09T22:32:09.466 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-06-09T22:32:09.466 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-06-09T22:32:09.466 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:09.466 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-06-09T22:32:09.466 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-06-09T22:32:09.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:09 smithi194 ceph-mon[36856]: pgmap v346: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-06-09T22:32:09.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:09 smithi153 ceph-mon[36599]: pgmap v346: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-06-09T22:32:10.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:09 smithi104 ceph-mon[32389]: pgmap v346: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-06-09T22:32:10.085 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-06-09T22:32:10.085 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-06-09T22:32:10.085 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=6 2024-06-09T22:32:10.086 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-06-09T22:32:10.086 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-06-09T22:32:10.086 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-06-09T22:32:10.099 INFO:tasks.kclient:Unmounting kernel clients... 2024-06-09T22:32:10.099 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:32:10.100 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-09T22:32:10.122 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-06-09T22:32:10.122 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-09T22:32:10.182 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-06-09T22:32:10.472 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:32:11.899 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:11 smithi104 ceph-mon[32389]: pgmap v347: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-06-09T22:32:11.900 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:11 smithi104 ceph-mon[32389]: osdmap e109: 12 total, 12 up, 12 in 2024-06-09T22:32:11.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:11 smithi153 ceph-mon[36599]: pgmap v347: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-06-09T22:32:11.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:11 smithi153 ceph-mon[36599]: osdmap e109: 12 total, 12 up, 12 in 2024-06-09T22:32:12.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:11 smithi194 ceph-mon[36856]: pgmap v347: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-06-09T22:32:12.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:11 smithi194 ceph-mon[36856]: osdmap e109: 12 total, 12 up, 12 in 2024-06-09T22:32:12.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.270+0000 7f99db11d700 1 -- 172.21.15.104:0/2880594350 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d4101d30 msgr2=0x7f99d410e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:12.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.270+0000 7f99db11d700 1 --2- 172.21.15.104:0/2880594350 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d4101d30 0x7f99d410e210 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00b0a0 tx=0x7f99cc00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:32:12.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.270+0000 7f99db11d700 1 -- 172.21.15.104:0/2880594350 shutdown_connections 2024-06-09T22:32:12.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.270+0000 7f99db11d700 1 --2- 172.21.15.104:0/2880594350 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d4101d30 0x7f99d410e210 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:12.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.270+0000 7f99db11d700 1 --2- 172.21.15.104:0/2880594350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99d41073d0 0x7f99d41077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:12.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.270+0000 7f99db11d700 1 --2- 172.21.15.104:0/2880594350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d4101390 0x7f99d41017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:12.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.270+0000 7f99db11d700 1 -- 172.21.15.104:0/2880594350 >> 172.21.15.104:0/2880594350 conn(0x7f99d40782c0 msgr2=0x7f99d40786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:12.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.270+0000 7f99db11d700 1 -- 172.21.15.104:0/2880594350 shutdown_connections 2024-06-09T22:32:12.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.270+0000 7f99db11d700 1 -- 172.21.15.104:0/2880594350 wait complete. 2024-06-09T22:32:12.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99db11d700 1 Processor -- start 2024-06-09T22:32:12.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99db11d700 1 -- start start 2024-06-09T22:32:12.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99db11d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d4101390 0x7f99d419d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:12.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99db11d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99d4101d30 0x7f99d419dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:12.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99db11d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d41073d0 0x7f99d4197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99db11d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f99d4111690 con 0x7f99d4101390 2024-06-09T22:32:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99db11d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99d4111510 con 0x7f99d41073d0 2024-06-09T22:32:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99db11d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f99d4111810 con 0x7f99d4101d30 2024-06-09T22:32:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99d8eb9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d4101390 0x7f99d419d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99d8eb9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d4101390 0x7f99d419d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45198/0 (socket says 172.21.15.104:45198) 2024-06-09T22:32:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99d8eb9700 1 -- 172.21.15.104:0/3555599550 learned_addr learned my addr 172.21.15.104:0/3555599550 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:32:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99cbfff700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99d4101d30 0x7f99d419dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99d96ba700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d41073d0 0x7f99d4197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99d8eb9700 1 -- 172.21.15.104:0/3555599550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99d4101d30 msgr2=0x7f99d419dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.271+0000 7f99d8eb9700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99d4101d30 0x7f99d419dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.272+0000 7f99d8eb9700 1 -- 172.21.15.104:0/3555599550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d41073d0 msgr2=0x7f99d4197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.272+0000 7f99d8eb9700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d41073d0 0x7f99d4197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.272+0000 7f99d8eb9700 1 -- 172.21.15.104:0/3555599550 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d4198070 con 0x7f99d4101390 2024-06-09T22:32:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.272+0000 7f99cbfff700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99d4101d30 0x7f99d419dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:32:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.272+0000 7f99d96ba700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d41073d0 0x7f99d4197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:32:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.272+0000 7f99d8eb9700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d4101390 0x7f99d419d750 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f99c400b530 tx=0x7f99c400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.272+0000 7f99c9ffb700 1 -- 172.21.15.104:0/3555599550 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c4015020 con 0x7f99d4101390 2024-06-09T22:32:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.272+0000 7f99c9ffb700 1 -- 172.21.15.104:0/3555599550 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99c4011d10 con 0x7f99d4101390 2024-06-09T22:32:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.272+0000 7f99db11d700 1 -- 172.21.15.104:0/3555599550 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99d4198360 con 0x7f99d4101390 2024-06-09T22:32:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.272+0000 7f99c9ffb700 1 -- 172.21.15.104:0/3555599550 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c4010e50 con 0x7f99d4101390 2024-06-09T22:32:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.272+0000 7f99db11d700 1 -- 172.21.15.104:0/3555599550 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99d406a170 con 0x7f99d4101390 2024-06-09T22:32:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.273+0000 7f99db11d700 1 -- 172.21.15.104:0/3555599550 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99d4069180 con 0x7f99d4101390 2024-06-09T22:32:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.274+0000 7f99c9ffb700 1 -- 172.21.15.104:0/3555599550 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f99c4018020 con 0x7f99d4101390 2024-06-09T22:32:12.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.275+0000 7f99c9ffb700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f99bc0a2f30 0x7f99bc0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:12.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.275+0000 7f99cbfff700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f99bc0a2f30 0x7f99bc0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:12.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.275+0000 7f99c9ffb700 1 -- 172.21.15.104:0/3555599550 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9584+0+0 (secure 0 0 0) 0x7f99c40cf040 con 0x7f99d4101390 2024-06-09T22:32:12.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.275+0000 7f99cbfff700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f99bc0a2f30 0x7f99bc0a53f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f99d0005fd0 tx=0x7f99d0005e30 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:12.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.280+0000 7f99c9ffb700 1 -- 172.21.15.104:0/3555599550 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f99c401f020 con 0x7f99d4101390 2024-06-09T22:32:12.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.489+0000 7f99db11d700 1 -- 172.21.15.104:0/3555599550 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f99d4198e40 con 0x7f99d4101390 2024-06-09T22:32:12.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.490+0000 7f99c9ffb700 1 -- 172.21.15.104:0/3555599550 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v109) ==== 81+0+955 (secure 0 0 0) 0x7f99c409b950 con 0x7f99d4101390 2024-06-09T22:32:12.493 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/2389242747 2024-06-10T22:20:57.590784+0000 2024-06-09T22:32:12.494 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6801/85752866 2024-06-10T22:09:25.221731+0000 2024-06-09T22:32:12.494 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/3559678620 2024-06-10T22:08:58.969000+0000 2024-06-09T22:32:12.494 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/3864904957 2024-06-10T22:09:25.221731+0000 2024-06-09T22:32:12.494 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6800/85752866 2024-06-10T22:09:25.221731+0000 2024-06-09T22:32:12.494 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6801/1879478526 2024-06-10T22:08:58.969000+0000 2024-06-09T22:32:12.494 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6801/1906048760 2024-06-10T22:20:57.590784+0000 2024-06-09T22:32:12.494 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6800/1906048760 2024-06-10T22:20:57.590784+0000 2024-06-09T22:32:12.494 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/517954187 2024-06-10T22:20:57.590784+0000 2024-06-09T22:32:12.495 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/167857460 2024-06-10T22:20:57.590784+0000 2024-06-09T22:32:12.495 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/2471278860 2024-06-10T22:08:58.969000+0000 2024-06-09T22:32:12.495 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/357628765 2024-06-10T22:20:57.590784+0000 2024-06-09T22:32:12.495 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/4119286692 2024-06-10T22:09:25.221731+0000 2024-06-09T22:32:12.495 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/2266338289 2024-06-10T22:08:58.969000+0000 2024-06-09T22:32:12.495 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:6800/1879478526 2024-06-10T22:08:58.969000+0000 2024-06-09T22:32:12.495 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104:0/3072042273 2024-06-10T22:09:25.221731+0000 2024-06-09T22:32:12.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.492+0000 7f99db11d700 1 -- 172.21.15.104:0/3555599550 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f99bc0a2f30 msgr2=0x7f99bc0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:12.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.492+0000 7f99db11d700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f99bc0a2f30 0x7f99bc0a53f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f99d0005fd0 tx=0x7f99d0005e30 comp rx=0 tx=0).stop 2024-06-09T22:32:12.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.492+0000 7f99db11d700 1 -- 172.21.15.104:0/3555599550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d4101390 msgr2=0x7f99d419d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:12.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.492+0000 7f99db11d700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d4101390 0x7f99d419d750 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f99c400b530 tx=0x7f99c400b8f0 comp rx=0 tx=0).stop 2024-06-09T22:32:12.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.493+0000 7f99db11d700 1 -- 172.21.15.104:0/3555599550 shutdown_connections 2024-06-09T22:32:12.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.493+0000 7f99db11d700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f99bc0a2f30 0x7f99bc0a53f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:12.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.493+0000 7f99db11d700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d4101390 0x7f99d419d750 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:12.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.493+0000 7f99db11d700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d41073d0 0x7f99d4197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:12.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.493+0000 7f99db11d700 1 --2- 172.21.15.104:0/3555599550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99d4101d30 0x7f99d419dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:12.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.493+0000 7f99db11d700 1 -- 172.21.15.104:0/3555599550 >> 172.21.15.104:0/3555599550 conn(0x7f99d40782c0 msgr2=0x7f99d40ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:12.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.493+0000 7f99db11d700 1 -- 172.21.15.104:0/3555599550 shutdown_connections 2024-06-09T22:32:12.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:12.493+0000 7f99db11d700 1 -- 172.21.15.104:0/3555599550 wait complete. 2024-06-09T22:32:12.498 INFO:teuthology.orchestra.run.smithi104.stderr:listed 16 entries 2024-06-09T22:32:12.759 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:12 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/3555599550' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-09T22:32:12.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:12 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/3555599550' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-09T22:32:13.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:12 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/3555599550' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-09T22:32:13.203 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-09T22:32:13.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:32:13.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-09T22:32:13.347 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2024-06-09T22:32:13.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:32:13.348 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-09T22:32:13.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:32:13.367 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2024-06-09T22:32:13.423 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-09T22:32:13.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:32:13.423 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-09T22:32:13.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-09T22:32:13.489 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link delete ceph-brx 2024-06-09T22:32:13.576 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-09T22:32:13.589 INFO:tasks.cephadm:Teardown begin 2024-06-09T22:32:13.589 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-09T22:32:13.590 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:32:13.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:13 smithi104 ceph-mon[32389]: pgmap v349: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-06-09T22:32:13.834 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:32:13.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:13 smithi153 ceph-mon[36599]: pgmap v349: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-06-09T22:32:13.963 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:13 smithi194 ceph-mon[36856]: pgmap v349: 129 pgs: 129 active+clean; 247 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-06-09T22:32:15.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.654+0000 7f95bd0e9700 1 -- 172.21.15.104:0/358511003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95b80746c0 msgr2=0x7f95b8074aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:15.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.654+0000 7f95bd0e9700 1 --2- 172.21.15.104:0/358511003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95b80746c0 0x7f95b8074aa0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009990 tx=0x7f95a8009ca0 comp rx=0 tx=0).stop 2024-06-09T22:32:15.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.654+0000 7f95bd0e9700 1 -- 172.21.15.104:0/358511003 shutdown_connections 2024-06-09T22:32:15.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.654+0000 7f95bd0e9700 1 --2- 172.21.15.104:0/358511003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95b80746c0 0x7f95b8074aa0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:15.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.654+0000 7f95bd0e9700 1 --2- 172.21.15.104:0/358511003 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95b810eef0 0x7f95b81112e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:15.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.654+0000 7f95bd0e9700 1 --2- 172.21.15.104:0/358511003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b8074fe0 0x7f95b810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:15.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.654+0000 7f95bd0e9700 1 -- 172.21.15.104:0/358511003 >> 172.21.15.104:0/358511003 conn(0x7f95b8079520 msgr2=0x7f95b8079930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:15.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.654+0000 7f95bd0e9700 1 -- 172.21.15.104:0/358511003 shutdown_connections 2024-06-09T22:32:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.654+0000 7f95bd0e9700 1 -- 172.21.15.104:0/358511003 wait complete. 2024-06-09T22:32:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.655+0000 7f95bd0e9700 1 Processor -- start 2024-06-09T22:32:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.655+0000 7f95bd0e9700 1 -- start start 2024-06-09T22:32:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.655+0000 7f95bd0e9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95b80746c0 0x7f95b819ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.655+0000 7f95bd0e9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b8074fe0 0x7f95b819b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.655+0000 7f95bd0e9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95b810eef0 0x7f95b819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.655+0000 7f95bd0e9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f95b8114990 con 0x7f95b80746c0 2024-06-09T22:32:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.655+0000 7f95bd0e9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95b8114810 con 0x7f95b8074fe0 2024-06-09T22:32:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.655+0000 7f95bd0e9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f95b8114b10 con 0x7f95b810eef0 2024-06-09T22:32:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95b80746c0 0x7f95b819ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95b80746c0 0x7f95b819ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:45220/0 (socket says 172.21.15.104:45220) 2024-06-09T22:32:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b6d9d700 1 -- 172.21.15.104:0/2526888614 learned_addr learned my addr 172.21.15.104:0/2526888614 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:32:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b659c700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b8074fe0 0x7f95b819b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b759e700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95b810eef0 0x7f95b819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b6d9d700 1 -- 172.21.15.104:0/2526888614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95b810eef0 msgr2=0x7f95b819f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b6d9d700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95b810eef0 0x7f95b819f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:15.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b6d9d700 1 -- 172.21.15.104:0/2526888614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b8074fe0 msgr2=0x7f95b819b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:15.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b6d9d700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b8074fe0 0x7f95b819b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:15.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b6d9d700 1 -- 172.21.15.104:0/2526888614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b819fe00 con 0x7f95b80746c0 2024-06-09T22:32:15.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b659c700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b8074fe0 0x7f95b819b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:32:15.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b759e700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95b810eef0 0x7f95b819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:32:15.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95b6d9d700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95b80746c0 0x7f95b819ae40 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f95a8000c00 tx=0x7f95a800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:15.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f959ffff700 1 -- 172.21.15.104:0/2526888614 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a8027070 con 0x7f95b80746c0 2024-06-09T22:32:15.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f959ffff700 1 -- 172.21.15.104:0/2526888614 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95a8003d70 con 0x7f95b80746c0 2024-06-09T22:32:15.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95bd0e9700 1 -- 172.21.15.104:0/2526888614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b81a0090 con 0x7f95b80746c0 2024-06-09T22:32:15.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f959ffff700 1 -- 172.21.15.104:0/2526888614 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a800f460 con 0x7f95b80746c0 2024-06-09T22:32:15.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.656+0000 7f95bd0e9700 1 -- 172.21.15.104:0/2526888614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b81a7970 con 0x7f95b80746c0 2024-06-09T22:32:15.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.658+0000 7f95bd0e9700 1 -- 172.21.15.104:0/2526888614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95b8069180 con 0x7f95b80746c0 2024-06-09T22:32:15.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.659+0000 7f959ffff700 1 -- 172.21.15.104:0/2526888614 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f95a802a070 con 0x7f95b80746c0 2024-06-09T22:32:15.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.659+0000 7f959ffff700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f95a00a2d90 0x7f95a00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:15.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.659+0000 7f95b659c700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f95a00a2d90 0x7f95a00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:15.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.659+0000 7f959ffff700 1 -- 172.21.15.104:0/2526888614 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9584+0+0 (secure 0 0 0) 0x7f95a80d0600 con 0x7f95b80746c0 2024-06-09T22:32:15.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.660+0000 7f95b659c700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f95a00a2d90 0x7f95a00a5250 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f95b8110ab0 tx=0x7f95ac009260 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:15.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.664+0000 7f959ffff700 1 -- 172.21.15.104:0/2526888614 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f95a8013070 con 0x7f95b80746c0 2024-06-09T22:32:15.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:15 smithi104 ceph-mon[32389]: pgmap v350: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-06-09T22:32:15.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.911+0000 7f95bd0e9700 1 -- 172.21.15.104:0/2526888614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95b81a7b40 con 0x7f95b80746c0 2024-06-09T22:32:15.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.912+0000 7f959ffff700 1 -- 172.21.15.104:0/2526888614 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7f95a809ce90 con 0x7f95b80746c0 2024-06-09T22:32:15.915 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:32:15.915 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:32:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.915+0000 7f95bd0e9700 1 -- 172.21.15.104:0/2526888614 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f95a00a2d90 msgr2=0x7f95a00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.915+0000 7f95bd0e9700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f95a00a2d90 0x7f95a00a5250 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f95b8110ab0 tx=0x7f95ac009260 comp rx=0 tx=0).stop 2024-06-09T22:32:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.915+0000 7f95bd0e9700 1 -- 172.21.15.104:0/2526888614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95b80746c0 msgr2=0x7f95b819ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.915+0000 7f95bd0e9700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95b80746c0 0x7f95b819ae40 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f95a8000c00 tx=0x7f95a800bab0 comp rx=0 tx=0).stop 2024-06-09T22:32:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.915+0000 7f95bd0e9700 1 -- 172.21.15.104:0/2526888614 shutdown_connections 2024-06-09T22:32:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.915+0000 7f95bd0e9700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f95a00a2d90 0x7f95a00a5250 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.915+0000 7f95bd0e9700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95b80746c0 0x7f95b819ae40 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.915+0000 7f95bd0e9700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f95b810eef0 0x7f95b819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.915+0000 7f95bd0e9700 1 --2- 172.21.15.104:0/2526888614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b8074fe0 0x7f95b819b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.915+0000 7f95bd0e9700 1 -- 172.21.15.104:0/2526888614 >> 172.21.15.104:0/2526888614 conn(0x7f95b8079520 msgr2=0x7f95b80ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.915+0000 7f95bd0e9700 1 -- 172.21.15.104:0/2526888614 shutdown_connections 2024-06-09T22:32:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:15.915+0000 7f95bd0e9700 1 -- 172.21.15.104:0/2526888614 wait complete. 2024-06-09T22:32:15.919 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:32:15.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:15 smithi153 ceph-mon[36599]: pgmap v350: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-06-09T22:32:16.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:15 smithi194 ceph-mon[36856]: pgmap v350: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-06-09T22:32:16.764 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-09T22:32:16.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:16 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2526888614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:32:17.005 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:32:17.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:16 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2526888614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:32:17.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:16 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2526888614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:32:17.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:17 smithi104 ceph-mon[32389]: pgmap v351: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-06-09T22:32:17.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:17 smithi104 ceph-mon[32389]: osdmap e110: 12 total, 12 up, 12 in 2024-06-09T22:32:17.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:17 smithi153 ceph-mon[36599]: pgmap v351: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-06-09T22:32:17.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:17 smithi153 ceph-mon[36599]: osdmap e110: 12 total, 12 up, 12 in 2024-06-09T22:32:18.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:17 smithi194 ceph-mon[36856]: pgmap v351: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-06-09T22:32:18.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:17 smithi194 ceph-mon[36856]: osdmap e110: 12 total, 12 up, 12 in 2024-06-09T22:32:18.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.669+0000 7f6db64d3700 1 -- 172.21.15.104:0/2550013289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db0107400 msgr2=0x7f6db01077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:18.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.669+0000 7f6db64d3700 1 --2- 172.21.15.104:0/2550013289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db0107400 0x7f6db01077e0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009990 tx=0x7f6da0009ca0 comp rx=0 tx=0).stop 2024-06-09T22:32:18.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.669+0000 7f6db64d3700 1 -- 172.21.15.104:0/2550013289 shutdown_connections 2024-06-09T22:32:18.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.669+0000 7f6db64d3700 1 --2- 172.21.15.104:0/2550013289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db0107400 0x7f6db01077e0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:18.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.669+0000 7f6db64d3700 1 --2- 172.21.15.104:0/2550013289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6db0101d50 0x7f6db010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:18.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.669+0000 7f6db64d3700 1 --2- 172.21.15.104:0/2550013289 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6db0101390 0x7f6db0101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:18.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.669+0000 7f6db64d3700 1 -- 172.21.15.104:0/2550013289 >> 172.21.15.104:0/2550013289 conn(0x7f6db0078340 msgr2=0x7f6db0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:18.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.669+0000 7f6db64d3700 1 -- 172.21.15.104:0/2550013289 shutdown_connections 2024-06-09T22:32:18.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.670+0000 7f6db64d3700 1 -- 172.21.15.104:0/2550013289 wait complete. 2024-06-09T22:32:18.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.670+0000 7f6db64d3700 1 Processor -- start 2024-06-09T22:32:18.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.670+0000 7f6db64d3700 1 -- start start 2024-06-09T22:32:18.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db64d3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6db0101390 0x7f6db010eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:18.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db64d3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6db0101d50 0x7f6db010f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:18.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db64d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db0107400 0x7f6db010f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:18.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db64d3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6db0112870 con 0x7f6db0107400 2024-06-09T22:32:18.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db64d3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6db01126f0 con 0x7f6db0101390 2024-06-09T22:32:18.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db64d3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6db01129f0 con 0x7f6db0101d50 2024-06-09T22:32:18.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db4a70700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db0107400 0x7f6db010f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:18.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db4a70700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db0107400 0x7f6db010f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36554/0 (socket says 172.21.15.104:36554) 2024-06-09T22:32:18.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db4a70700 1 -- 172.21.15.104:0/2749655595 learned_addr learned my addr 172.21.15.104:0/2749655595 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:32:18.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6daffff700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6db0101390 0x7f6db010eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:18.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6daf7fe700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6db0101d50 0x7f6db010f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:18.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db4a70700 1 -- 172.21.15.104:0/2749655595 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6db0101d50 msgr2=0x7f6db010f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:18.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db4a70700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6db0101d50 0x7f6db010f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:18.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db4a70700 1 -- 172.21.15.104:0/2749655595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6db0101390 msgr2=0x7f6db010eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:18.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db4a70700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6db0101390 0x7f6db010eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:18.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.671+0000 7f6db4a70700 1 -- 172.21.15.104:0/2749655595 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db01a7290 con 0x7f6db0107400 2024-06-09T22:32:18.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.672+0000 7f6db4a70700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db0107400 0x7f6db010f5d0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f6da800b970 tx=0x7f6da800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:18.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.672+0000 7f6dad7fa700 1 -- 172.21.15.104:0/2749655595 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8019070 con 0x7f6db0107400 2024-06-09T22:32:18.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.672+0000 7f6db64d3700 1 -- 172.21.15.104:0/2749655595 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6db01a7580 con 0x7f6db0107400 2024-06-09T22:32:18.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.672+0000 7f6dad7fa700 1 -- 172.21.15.104:0/2749655595 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da800ce10 con 0x7f6db0107400 2024-06-09T22:32:18.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.672+0000 7f6dad7fa700 1 -- 172.21.15.104:0/2749655595 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8007370 con 0x7f6db0107400 2024-06-09T22:32:18.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.672+0000 7f6db64d3700 1 -- 172.21.15.104:0/2749655595 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6db01a7ac0 con 0x7f6db0107400 2024-06-09T22:32:18.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.673+0000 7f6db64d3700 1 -- 172.21.15.104:0/2749655595 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6db0069180 con 0x7f6db0107400 2024-06-09T22:32:18.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.674+0000 7f6dad7fa700 1 -- 172.21.15.104:0/2749655595 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f6da801c070 con 0x7f6db0107400 2024-06-09T22:32:18.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.675+0000 7f6dad7fa700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6d980a2d90 0x7f6d980a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:18.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.675+0000 7f6daffff700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6d980a2d90 0x7f6d980a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:18.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.675+0000 7f6dad7fa700 1 -- 172.21.15.104:0/2749655595 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9496+0+0 (secure 0 0 0) 0x7f6da80cef10 con 0x7f6db0107400 2024-06-09T22:32:18.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.676+0000 7f6daffff700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6d980a2d90 0x7f6d980a5250 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6da000b5c0 tx=0x7f6da0005960 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:18.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.679+0000 7f6dad7fa700 1 -- 172.21.15.104:0/2749655595 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6da809b870 con 0x7f6db0107400 2024-06-09T22:32:18.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.922+0000 7f6db64d3700 1 -- 172.21.15.104:0/2749655595 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6db002dca0 con 0x7f6db0107400 2024-06-09T22:32:18.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.923+0000 7f6dad7fa700 1 -- 172.21.15.104:0/2749655595 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11953 (secure 0 0 0) 0x7f6da809b170 con 0x7f6db0107400 2024-06-09T22:32:18.927 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:32:18.927 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3517082076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3517082076},{"type":"v1","addr":"172.21.15.104:6835","nonce":3517082076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/3060956100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":3060956100},{"type":"v1","addr":"172.21.15.104:6837","nonce":3060956100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-09T22:16:45.150445+0000","modified":"2024-06-09T22:25:34.695814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24460,"mds_1":24448,"mds_2":24442,"mds_3":24419,"mds_4":24403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14571":{"gid":14571,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/647544853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":647544853},{"type":"v1","addr":"172.21.15.104:6839","nonce":647544853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14583":{"gid":14583,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6841/3830332242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6840","nonce":3830332242},{"type":"v1","addr":"172.21.15.104:6841","nonce":3830332242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24362":{"gid":24362,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6833/3968887191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3968887191},{"type":"v1","addr":"172.21.15.153:6833","nonce":3968887191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24383":{"gid":24383,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/3075465176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3075465176},{"type":"v1","addr":"172.21.15.153:6835","nonce":3075465176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/3294454895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3294454895},{"type":"v1","addr":"172.21.15.153:6837","nonce":3294454895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":31,"addr":"172.21.15.194:6833/3284873174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":3284873174},{"type":"v1","addr":"172.21.15.194:6833","nonce":3284873174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":21,"addr":"172.21.15.153:6839/779682823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":779682823},{"type":"v1","addr":"172.21.15.153:6839","nonce":779682823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":26,"addr":"172.21.15.194:6835/2995819396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":2995819396},{"type":"v1","addr":"172.21.15.194:6835","nonce":2995819396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":25,"addr":"172.21.15.194:6837/464494707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":464494707},{"type":"v1","addr":"172.21.15.194:6837","nonce":464494707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.194:6839/2028005544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2028005544},{"type":"v1","addr":"172.21.15.194:6839","nonce":2028005544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24460,"qdb_cluster":[24403,24419,24442,24448,24460]},"id":1}]} 2024-06-09T22:32:18.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.925+0000 7f6db64d3700 1 -- 172.21.15.104:0/2749655595 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6d980a2d90 msgr2=0x7f6d980a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:18.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.925+0000 7f6db64d3700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6d980a2d90 0x7f6d980a5250 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6da000b5c0 tx=0x7f6da0005960 comp rx=0 tx=0).stop 2024-06-09T22:32:18.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.925+0000 7f6db64d3700 1 -- 172.21.15.104:0/2749655595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db0107400 msgr2=0x7f6db010f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:18.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.925+0000 7f6db64d3700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db0107400 0x7f6db010f5d0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f6da800b970 tx=0x7f6da800bc80 comp rx=0 tx=0).stop 2024-06-09T22:32:18.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.926+0000 7f6db64d3700 1 -- 172.21.15.104:0/2749655595 shutdown_connections 2024-06-09T22:32:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.926+0000 7f6db64d3700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f6d980a2d90 0x7f6d980a5250 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.926+0000 7f6db64d3700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db0107400 0x7f6db010f5d0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.926+0000 7f6db64d3700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6db0101390 0x7f6db010eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.926+0000 7f6db64d3700 1 --2- 172.21.15.104:0/2749655595 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6db0101d50 0x7f6db010f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.926+0000 7f6db64d3700 1 -- 172.21.15.104:0/2749655595 >> 172.21.15.104:0/2749655595 conn(0x7f6db0078340 msgr2=0x7f6db01001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.926+0000 7f6db64d3700 1 -- 172.21.15.104:0/2749655595 shutdown_connections 2024-06-09T22:32:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:18.926+0000 7f6db64d3700 1 -- 172.21.15.104:0/2749655595 wait complete. 2024-06-09T22:32:18.931 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 32 2024-06-09T22:32:19.823 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-09T22:32:19.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:19 smithi153 ceph-mon[36599]: pgmap v353: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-06-09T22:32:19.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:19 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2749655595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:32:20.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:19 smithi104 ceph-mon[32389]: pgmap v353: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-06-09T22:32:20.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:19 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2749655595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:32:20.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:19 smithi194 ceph-mon[36856]: pgmap v353: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-06-09T22:32:20.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:19 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2749655595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-09T22:32:20.063 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:32:21.792 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:21 smithi104 ceph-mon[32389]: pgmap v354: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-06-09T22:32:21.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.891+0000 7f1b691af700 1 -- 172.21.15.104:0/2371143721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b64101d50 msgr2=0x7f1b6410e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:21.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.891+0000 7f1b691af700 1 --2- 172.21.15.104:0/2371143721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b64101d50 0x7f1b6410e240 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00b0a0 tx=0x7f1b5c00b3b0 comp rx=0 tx=0).stop 2024-06-09T22:32:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.891+0000 7f1b691af700 1 -- 172.21.15.104:0/2371143721 shutdown_connections 2024-06-09T22:32:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.891+0000 7f1b691af700 1 --2- 172.21.15.104:0/2371143721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b64101d50 0x7f1b6410e240 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.891+0000 7f1b691af700 1 --2- 172.21.15.104:0/2371143721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b64107400 0x7f1b641077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.891+0000 7f1b691af700 1 --2- 172.21.15.104:0/2371143721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b64101390 0x7f1b64101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.891+0000 7f1b691af700 1 -- 172.21.15.104:0/2371143721 >> 172.21.15.104:0/2371143721 conn(0x7f1b64078340 msgr2=0x7f1b64078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.891+0000 7f1b691af700 1 -- 172.21.15.104:0/2371143721 shutdown_connections 2024-06-09T22:32:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.891+0000 7f1b691af700 1 -- 172.21.15.104:0/2371143721 wait complete. 2024-06-09T22:32:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b691af700 1 Processor -- start 2024-06-09T22:32:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b691af700 1 -- start start 2024-06-09T22:32:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b691af700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b64101390 0x7f1b6419e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b691af700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b64101d50 0x7f1b6419ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b691af700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b64107400 0x7f1b641989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b691af700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1b64111690 con 0x7f1b64107400 2024-06-09T22:32:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b691af700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1b64111510 con 0x7f1b64101d50 2024-06-09T22:32:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b691af700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1b64111810 con 0x7f1b64101390 2024-06-09T22:32:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b6359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b64107400 0x7f1b641989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:21.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b6359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b64107400 0x7f1b641989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36576/0 (socket says 172.21.15.104:36576) 2024-06-09T22:32:21.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b6359e700 1 -- 172.21.15.104:0/2461187261 learned_addr learned my addr 172.21.15.104:0/2461187261 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:32:21.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b62d9d700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b64101390 0x7f1b6419e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:21.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.892+0000 7f1b6259c700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b64101d50 0x7f1b6419ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:21.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.893+0000 7f1b6359e700 1 -- 172.21.15.104:0/2461187261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b64101390 msgr2=0x7f1b6419e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:21.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.893+0000 7f1b6359e700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b64101390 0x7f1b6419e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:21.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.893+0000 7f1b6359e700 1 -- 172.21.15.104:0/2461187261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b64101d50 msgr2=0x7f1b6419ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:21.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.893+0000 7f1b6359e700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b64101d50 0x7f1b6419ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:21.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.893+0000 7f1b6359e700 1 -- 172.21.15.104:0/2461187261 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b64199140 con 0x7f1b64107400 2024-06-09T22:32:21.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.893+0000 7f1b6359e700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b64107400 0x7f1b641989e0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c0062a0 tx=0x7f1b5c004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:21.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.893+0000 7f1b4bfff700 1 -- 172.21.15.104:0/2461187261 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c021070 con 0x7f1b64107400 2024-06-09T22:32:21.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.893+0000 7f1b691af700 1 -- 172.21.15.104:0/2461187261 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b641993d0 con 0x7f1b64107400 2024-06-09T22:32:21.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.893+0000 7f1b4bfff700 1 -- 172.21.15.104:0/2461187261 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b5c007c00 con 0x7f1b64107400 2024-06-09T22:32:21.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.893+0000 7f1b4bfff700 1 -- 172.21.15.104:0/2461187261 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c01d450 con 0x7f1b64107400 2024-06-09T22:32:21.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.893+0000 7f1b691af700 1 -- 172.21.15.104:0/2461187261 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b641a56a0 con 0x7f1b64107400 2024-06-09T22:32:21.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.894+0000 7f1b691af700 1 -- 172.21.15.104:0/2461187261 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b64069180 con 0x7f1b64107400 2024-06-09T22:32:21.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.895+0000 7f1b4bfff700 1 -- 172.21.15.104:0/2461187261 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f1b5c004000 con 0x7f1b64107400 2024-06-09T22:32:21.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.896+0000 7f1b4bfff700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1b4c0a2d90 0x7f1b4c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:21.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.896+0000 7f1b62d9d700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1b4c0a2d90 0x7f1b4c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:21.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.896+0000 7f1b4bfff700 1 -- 172.21.15.104:0/2461187261 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9496+0+0 (secure 0 0 0) 0x7f1b5c00f020 con 0x7f1b64107400 2024-06-09T22:32:21.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.896+0000 7f1b62d9d700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1b4c0a2d90 0x7f1b4c0a5250 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1b64107c80 tx=0x7f1b5800b3c0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:21.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:21.899+0000 7f1b4bfff700 1 -- 172.21.15.104:0/2461187261 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1b5c02a020 con 0x7f1b64107400 2024-06-09T22:32:21.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:21 smithi153 ceph-mon[36599]: pgmap v354: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-06-09T22:32:22.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:21 smithi194 ceph-mon[36856]: pgmap v354: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-06-09T22:32:22.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.096+0000 7f1b691af700 1 -- 172.21.15.104:0/2461187261 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b641a5870 con 0x7f1b64107400 2024-06-09T22:32:22.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.098+0000 7f1b4bfff700 1 -- 172.21.15.104:0/2461187261 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+25329 (secure 0 0 0) 0x7f1b5c09d630 con 0x7f1b64107400 2024-06-09T22:32:22.101 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-06-09T22:32:22.102 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":110,"fsid":"a948c0ca-26ac-11ef-bc9b-c7b262605968","created":"2024-06-09T22:08:31.471440+0000","modified":"2024-06-09T22:32:17.588095+0000","last_up_change":"2024-06-09T22:15:38.458547+0000","last_in_change":"2024-06-09T22:15:18.023674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:11:49.547377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:16:38.369246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:16:42.128973+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:16:51.549770+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5d95a0f1-5dfc-4481-bb1f-ff3f9627b6e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6803","nonce":2202624505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6805","nonce":2202624505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6809","nonce":2202624505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2202624505},{"type":"v1","addr":"172.21.15.104:6807","nonce":2202624505}]},"public_addr":"172.21.15.104:6803/2202624505","cluster_addr":"172.21.15.104:6805/2202624505","heartbeat_back_addr":"172.21.15.104:6809/2202624505","heartbeat_front_addr":"172.21.15.104:6807/2202624505","state":["exists","up"]},{"osd":1,"uuid":"aa718c73-5826-43d9-9b5b-c99ddec256da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6801","nonce":628783097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6803","nonce":628783097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6807","nonce":628783097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":628783097},{"type":"v1","addr":"172.21.15.153:6805","nonce":628783097}]},"public_addr":"172.21.15.153:6801/628783097","cluster_addr":"172.21.15.153:6803/628783097","heartbeat_back_addr":"172.21.15.153:6807/628783097","heartbeat_front_addr":"172.21.15.153:6805/628783097","state":["exists","up"]},{"osd":2,"uuid":"35204639-9a6e-485d-a3b7-2d2330406b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6801","nonce":120935084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6803","nonce":120935084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6807","nonce":120935084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":120935084},{"type":"v1","addr":"172.21.15.194:6805","nonce":120935084}]},"public_addr":"172.21.15.194:6801/120935084","cluster_addr":"172.21.15.194:6803/120935084","heartbeat_back_addr":"172.21.15.194:6807/120935084","heartbeat_front_addr":"172.21.15.194:6805/120935084","state":["exists","up"]},{"osd":3,"uuid":"3acd0acc-fcf7-4122-a939-d961bfc03c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6811","nonce":3003003340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6813","nonce":3003003340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6817","nonce":3003003340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3003003340},{"type":"v1","addr":"172.21.15.104:6815","nonce":3003003340}]},"public_addr":"172.21.15.104:6811/3003003340","cluster_addr":"172.21.15.104:6813/3003003340","heartbeat_back_addr":"172.21.15.104:6817/3003003340","heartbeat_front_addr":"172.21.15.104:6815/3003003340","state":["exists","up"]},{"osd":4,"uuid":"e0d2ceee-535e-48d8-90e7-5c5edb3eebe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6809","nonce":1237651981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6811","nonce":1237651981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6815","nonce":1237651981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1237651981},{"type":"v1","addr":"172.21.15.153:6813","nonce":1237651981}]},"public_addr":"172.21.15.153:6809/1237651981","cluster_addr":"172.21.15.153:6811/1237651981","heartbeat_back_addr":"172.21.15.153:6815/1237651981","heartbeat_front_addr":"172.21.15.153:6813/1237651981","state":["exists","up"]},{"osd":5,"uuid":"49a8983d-15aa-47ef-83b9-72ea75fe4505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6809","nonce":3138023494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6811","nonce":3138023494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6815","nonce":3138023494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3138023494},{"type":"v1","addr":"172.21.15.194:6813","nonce":3138023494}]},"public_addr":"172.21.15.194:6809/3138023494","cluster_addr":"172.21.15.194:6811/3138023494","heartbeat_back_addr":"172.21.15.194:6815/3138023494","heartbeat_front_addr":"172.21.15.194:6813/3138023494","state":["exists","up"]},{"osd":6,"uuid":"08563350-3ded-4ff6-a214-12d9242f984a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6819","nonce":4097161419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6821","nonce":4097161419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6825","nonce":4097161419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4097161419},{"type":"v1","addr":"172.21.15.104:6823","nonce":4097161419}]},"public_addr":"172.21.15.104:6819/4097161419","cluster_addr":"172.21.15.104:6821/4097161419","heartbeat_back_addr":"172.21.15.104:6825/4097161419","heartbeat_front_addr":"172.21.15.104:6823/4097161419","state":["exists","up"]},{"osd":7,"uuid":"b23baa2c-c27d-43a3-9820-8626c26d3f5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6817","nonce":1289060024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6819","nonce":1289060024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6823","nonce":1289060024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1289060024},{"type":"v1","addr":"172.21.15.153:6821","nonce":1289060024}]},"public_addr":"172.21.15.153:6817/1289060024","cluster_addr":"172.21.15.153:6819/1289060024","heartbeat_back_addr":"172.21.15.153:6823/1289060024","heartbeat_front_addr":"172.21.15.153:6821/1289060024","state":["exists","up"]},{"osd":8,"uuid":"8beddca3-bc69-4ef9-955b-f0296e0f28bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6817","nonce":2851208300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6819","nonce":2851208300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6823","nonce":2851208300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2851208300},{"type":"v1","addr":"172.21.15.194:6821","nonce":2851208300}]},"public_addr":"172.21.15.194:6817/2851208300","cluster_addr":"172.21.15.194:6819/2851208300","heartbeat_back_addr":"172.21.15.194:6823/2851208300","heartbeat_front_addr":"172.21.15.194:6821/2851208300","state":["exists","up"]},{"osd":9,"uuid":"d82b0c48-b957-4209-afc9-020ded569c2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6827","nonce":3652005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6829","nonce":3652005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6833","nonce":3652005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3652005298},{"type":"v1","addr":"172.21.15.104:6831","nonce":3652005298}]},"public_addr":"172.21.15.104:6827/3652005298","cluster_addr":"172.21.15.104:6829/3652005298","heartbeat_back_addr":"172.21.15.104:6833/3652005298","heartbeat_front_addr":"172.21.15.104:6831/3652005298","state":["exists","up"]},{"osd":10,"uuid":"98aa6a7b-baac-4816-bf0c-fa0b01931b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6825","nonce":1184232655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6827","nonce":1184232655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6831","nonce":1184232655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1184232655},{"type":"v1","addr":"172.21.15.153:6829","nonce":1184232655}]},"public_addr":"172.21.15.153:6825/1184232655","cluster_addr":"172.21.15.153:6827/1184232655","heartbeat_back_addr":"172.21.15.153:6831/1184232655","heartbeat_front_addr":"172.21.15.153:6829/1184232655","state":["exists","up"]},{"osd":11,"uuid":"f107b3c6-8206-4cd0-bc41-5cc8eb8f179d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6825","nonce":2145259515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6827","nonce":2145259515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6831","nonce":2145259515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2145259515},{"type":"v1","addr":"172.21.15.194:6829","nonce":2145259515}]},"public_addr":"172.21.15.194:6825/2145259515","cluster_addr":"172.21.15.194:6827/2145259515","heartbeat_back_addr":"172.21.15.194:6831/2145259515","heartbeat_front_addr":"172.21.15.194:6829/2145259515","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:10:59.462566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:22.994421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:11:46.965696+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:11.982300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:12:36.725765+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:01.655054+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:27.539850+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:13:53.044159+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:17.907395+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:14:43.874362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:10.074986+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:15:36.595028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:0/2389242747":"2024-06-10T22:20:57.590784+0000","172.21.15.104:6801/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/3559678620":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3864904957":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6800/85752866":"2024-06-10T22:09:25.221731+0000","172.21.15.104:6801/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6801/1906048760":"2024-06-10T22:20:57.590784+0000","172.21.15.104:6800/1906048760":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/517954187":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/167857460":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/2471278860":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/357628765":"2024-06-10T22:20:57.590784+0000","172.21.15.104:0/4119286692":"2024-06-10T22:09:25.221731+0000","172.21.15.104:0/2266338289":"2024-06-10T22:08:58.969000+0000","172.21.15.104:6800/1879478526":"2024-06-10T22:08:58.969000+0000","172.21.15.104:0/3072042273":"2024-06-10T22:09:25.221731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:32:22.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.100+0000 7f1b691af700 1 -- 172.21.15.104:0/2461187261 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1b4c0a2d90 msgr2=0x7f1b4c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:22.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.100+0000 7f1b691af700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1b4c0a2d90 0x7f1b4c0a5250 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1b64107c80 tx=0x7f1b5800b3c0 comp rx=0 tx=0).stop 2024-06-09T22:32:22.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.100+0000 7f1b691af700 1 -- 172.21.15.104:0/2461187261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b64107400 msgr2=0x7f1b641989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:22.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.100+0000 7f1b691af700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b64107400 0x7f1b641989e0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c0062a0 tx=0x7f1b5c004d70 comp rx=0 tx=0).stop 2024-06-09T22:32:22.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.100+0000 7f1b691af700 1 -- 172.21.15.104:0/2461187261 shutdown_connections 2024-06-09T22:32:22.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.100+0000 7f1b691af700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f1b4c0a2d90 0x7f1b4c0a5250 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:22.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.100+0000 7f1b691af700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b64107400 0x7f1b641989e0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:22.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.100+0000 7f1b691af700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b64101390 0x7f1b6419e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:22.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.100+0000 7f1b691af700 1 --2- 172.21.15.104:0/2461187261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b64101d50 0x7f1b6419ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:22.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.100+0000 7f1b691af700 1 -- 172.21.15.104:0/2461187261 >> 172.21.15.104:0/2461187261 conn(0x7f1b64078340 msgr2=0x7f1b6406a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:22.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.100+0000 7f1b691af700 1 -- 172.21.15.104:0/2461187261 shutdown_connections 2024-06-09T22:32:22.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:22.100+0000 7f1b691af700 1 -- 172.21.15.104:0/2461187261 wait complete. 2024-06-09T22:32:22.888 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:22 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2461187261' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:32:22.918 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs fail cephfs 2024-06-09T22:32:22.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:22 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2461187261' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:32:23.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:22 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2461187261' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-09T22:32:23.157 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:32:23.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:23 smithi153 ceph-mon[36599]: pgmap v355: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-06-09T22:32:24.009 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:23 smithi104 ceph-mon[32389]: pgmap v355: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-06-09T22:32:24.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:23 smithi194 ceph-mon[36856]: pgmap v355: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-06-09T22:32:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.010+0000 7fa6a391e700 1 -- 172.21.15.104:0/429539173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa69c101390 msgr2=0x7fa69c101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.010+0000 7fa6a391e700 1 --2- 172.21.15.104:0/429539173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa69c101390 0x7fa69c101810 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009990 tx=0x7fa68c009ca0 comp rx=0 tx=0).stop 2024-06-09T22:32:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.010+0000 7fa6a391e700 1 -- 172.21.15.104:0/429539173 shutdown_connections 2024-06-09T22:32:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.010+0000 7fa6a391e700 1 --2- 172.21.15.104:0/429539173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa69c101d50 0x7fa69c10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.010+0000 7fa6a391e700 1 --2- 172.21.15.104:0/429539173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa69c107400 0x7fa69c1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.010+0000 7fa6a391e700 1 --2- 172.21.15.104:0/429539173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa69c101390 0x7fa69c101810 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.010+0000 7fa6a391e700 1 -- 172.21.15.104:0/429539173 >> 172.21.15.104:0/429539173 conn(0x7fa69c078340 msgr2=0x7fa69c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.010+0000 7fa6a391e700 1 -- 172.21.15.104:0/429539173 shutdown_connections 2024-06-09T22:32:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.010+0000 7fa6a391e700 1 -- 172.21.15.104:0/429539173 wait complete. 2024-06-09T22:32:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a391e700 1 Processor -- start 2024-06-09T22:32:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a391e700 1 -- start start 2024-06-09T22:32:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a391e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa69c101390 0x7fa69c19e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a391e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa69c101d50 0x7fa69c19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a391e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa69c107400 0x7fa69c198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a391e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa69c06ae20 con 0x7fa69c101d50 2024-06-09T22:32:25.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a391e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa69c06aca0 con 0x7fa69c101390 2024-06-09T22:32:25.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a391e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa69c06afa0 con 0x7fa69c107400 2024-06-09T22:32:25.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a0eb9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa69c101d50 0x7fa69c19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:25.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a0eb9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa69c101d50 0x7fa69c19edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36598/0 (socket says 172.21.15.104:36598) 2024-06-09T22:32:25.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a0eb9700 1 -- 172.21.15.104:0/879620516 learned_addr learned my addr 172.21.15.104:0/879620516 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:32:25.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a16ba700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa69c101390 0x7fa69c19e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:25.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a0eb9700 1 -- 172.21.15.104:0/879620516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa69c107400 msgr2=0x7fa69c198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:25.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.011+0000 7fa6a1ebb700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa69c107400 0x7fa69c198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.012+0000 7fa6a0eb9700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa69c107400 0x7fa69c198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.012+0000 7fa6a0eb9700 1 -- 172.21.15.104:0/879620516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa69c101390 msgr2=0x7fa69c19e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.012+0000 7fa6a0eb9700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa69c101390 0x7fa69c19e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.012+0000 7fa6a0eb9700 1 -- 172.21.15.104:0/879620516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa69c1991d0 con 0x7fa69c101d50 2024-06-09T22:32:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.012+0000 7fa6a16ba700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa69c101390 0x7fa69c19e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:32:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.012+0000 7fa6a0eb9700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa69c101d50 0x7fa69c19edf0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fa68c000c00 tx=0x7fa68c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.012+0000 7fa6927fc700 1 -- 172.21.15.104:0/879620516 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c027070 con 0x7fa69c101d50 2024-06-09T22:32:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.012+0000 7fa6927fc700 1 -- 172.21.15.104:0/879620516 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa68c004050 con 0x7fa69c101d50 2024-06-09T22:32:25.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.012+0000 7fa6a391e700 1 -- 172.21.15.104:0/879620516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa69c199460 con 0x7fa69c101d50 2024-06-09T22:32:25.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.012+0000 7fa6927fc700 1 -- 172.21.15.104:0/879620516 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c00f460 con 0x7fa69c101d50 2024-06-09T22:32:25.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.012+0000 7fa6a391e700 1 -- 172.21.15.104:0/879620516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa69c1a5780 con 0x7fa69c101d50 2024-06-09T22:32:25.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.013+0000 7fa6a391e700 1 -- 172.21.15.104:0/879620516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa69c102ab0 con 0x7fa69c101d50 2024-06-09T22:32:25.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.014+0000 7fa6927fc700 1 -- 172.21.15.104:0/879620516 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa68c02a070 con 0x7fa69c101d50 2024-06-09T22:32:25.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.015+0000 7fa6927fc700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa6880a2d90 0x7fa6880a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:25.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.015+0000 7fa6a16ba700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa6880a2d90 0x7fa6880a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:25.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.015+0000 7fa6927fc700 1 -- 172.21.15.104:0/879620516 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9496+0+0 (secure 0 0 0) 0x7fa68c0d0bd0 con 0x7fa69c101d50 2024-06-09T22:32:25.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.015+0000 7fa6a16ba700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa6880a2d90 0x7fa6880a5250 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa698005950 tx=0x7fa69800a5b0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:25.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.018+0000 7fa6927fc700 1 -- 172.21.15.104:0/879620516 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa68c031070 con 0x7fa69c101d50 2024-06-09T22:32:25.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.256+0000 7fa6a391e700 1 -- 172.21.15.104:0/879620516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fa69c02dca0 con 0x7fa69c101d50 2024-06-09T22:32:25.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.258+0000 7fa6927fc700 1 -- 172.21.15.104:0/879620516 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) ==== 161+0+0 (secure 0 0 0) 0x7fa68c013070 con 0x7fa69c101d50 2024-06-09T22:32:25.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.262+0000 7fa6a391e700 1 -- 172.21.15.104:0/879620516 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa6880a2d90 msgr2=0x7fa6880a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:25.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.262+0000 7fa6a391e700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa6880a2d90 0x7fa6880a5250 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa698005950 tx=0x7fa69800a5b0 comp rx=0 tx=0).stop 2024-06-09T22:32:25.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.262+0000 7fa6a391e700 1 -- 172.21.15.104:0/879620516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa69c101d50 msgr2=0x7fa69c19edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.262+0000 7fa6a391e700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa69c101d50 0x7fa69c19edf0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fa68c000c00 tx=0x7fa68c0047b0 comp rx=0 tx=0).stop 2024-06-09T22:32:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.262+0000 7fa6a391e700 1 -- 172.21.15.104:0/879620516 shutdown_connections 2024-06-09T22:32:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.262+0000 7fa6a391e700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fa6880a2d90 0x7fa6880a5250 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.262+0000 7fa6a391e700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa69c107400 0x7fa69c198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.262+0000 7fa6a391e700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa69c101390 0x7fa69c19e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.262+0000 7fa6a391e700 1 --2- 172.21.15.104:0/879620516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa69c101d50 0x7fa69c19edf0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.262+0000 7fa6a391e700 1 -- 172.21.15.104:0/879620516 >> 172.21.15.104:0/879620516 conn(0x7fa69c078340 msgr2=0x7fa69c0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.263+0000 7fa6a391e700 1 -- 172.21.15.104:0/879620516 shutdown_connections 2024-06-09T22:32:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:25.263+0000 7fa6a391e700 1 -- 172.21.15.104:0/879620516 wait complete. 2024-06-09T22:32:25.268 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-09T22:32:25.278 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a[32384]: 2024-06-09T22:32:25.257+0000 7f3ada095700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-09T22:32:25.563 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f[67095]: ignoring --setuser ceph since I am not root 2024-06-09T22:32:25.563 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f[67095]: ignoring --setgroup ceph since I am not root 2024-06-09T22:32:25.563 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f[67095]: starting mds.f at 2024-06-09T22:32:25.563 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i[67699]: ignoring --setuser ceph since I am not root 2024-06-09T22:32:25.564 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i[67699]: ignoring --setgroup ceph since I am not root 2024-06-09T22:32:25.564 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-mds[67703]: -- 172.21.15.194:0/477754278 <== mon.0 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f0c0ed7a40 con 0x55f0c0eaa800 2024-06-09T22:32:25.564 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i[67699]: starting mds.i at 2024-06-09T22:32:25.564 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-l[66781]: ignoring --setuser ceph since I am not root 2024-06-09T22:32:25.564 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-l[66781]: ignoring --setgroup ceph since I am not root 2024-06-09T22:32:25.565 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-mds[66785]: -- 172.21.15.194:0/2128929008 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560af0c39a40 con 0x560af0c0cc00 2024-06-09T22:32:25.565 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-l[66781]: starting mds.l at 2024-06-09T22:32:25.565 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-c[67396]: ignoring --setuser ceph since I am not root 2024-06-09T22:32:25.565 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-c[67396]: ignoring --setgroup ceph since I am not root 2024-06-09T22:32:25.565 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-mds[67400]: -- 172.21.15.194:0/3539133943 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560d2a1344e0 con 0x560d2a10b000 2024-06-09T22:32:25.565 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-mds[67400]: -- 172.21.15.194:0/3539133943 <== mon.0 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560d2a137a40 con 0x560d2a10b000 2024-06-09T22:32:25.565 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-c[67396]: starting mds.c at 2024-06-09T22:32:25.585 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-e[66112]: ignoring --setuser ceph since I am not root 2024-06-09T22:32:25.585 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-e[66112]: ignoring --setgroup ceph since I am not root 2024-06-09T22:32:25.585 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-mds[66116]: -- 172.21.15.153:0/1591948092 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56404e3cba40 con 0x56404e39ec00 2024-06-09T22:32:25.585 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-e[66112]: starting mds.e at 2024-06-09T22:32:25.585 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-h[66412]: ignoring --setuser ceph since I am not root 2024-06-09T22:32:25.585 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-h[66412]: ignoring --setgroup ceph since I am not root 2024-06-09T22:32:25.586 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-mds[66416]: -- 172.21.15.153:0/3928708862 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557b272a7a40 con 0x557b2727a800 2024-06-09T22:32:25.586 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-h[66412]: starting mds.h at 2024-06-09T22:32:25.586 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b[66716]: ignoring --setuser ceph since I am not root 2024-06-09T22:32:25.586 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b[66716]: ignoring --setgroup ceph since I am not root 2024-06-09T22:32:25.586 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-mds[66720]: -- 172.21.15.153:0/1055669550 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561bec947a40 con 0x561bec91b000 2024-06-09T22:32:25.586 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b[66716]: starting mds.b at 2024-06-09T22:32:25.587 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-k[65798]: ignoring --setuser ceph since I am not root 2024-06-09T22:32:25.587 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-k[65798]: ignoring --setgroup ceph since I am not root 2024-06-09T22:32:25.587 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-mds[65802]: -- 172.21.15.153:0/3965788079 <== mon.0 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55581c2b9a40 con 0x55581c28c800 2024-06-09T22:32:25.587 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-k[65798]: starting mds.k at 2024-06-09T22:32:25.642 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-g[71521]: ignoring --setuser ceph since I am not root 2024-06-09T22:32:25.642 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-g[71521]: ignoring --setgroup ceph since I am not root 2024-06-09T22:32:25.642 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-mds[71525]: -- 172.21.15.104:0/1112700700 <== mon.0 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e81864da40 con 0x55e818621000 2024-06-09T22:32:25.642 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-g[71521]: starting mds.g at 2024-06-09T22:32:25.642 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-a[71945]: ignoring --setuser ceph since I am not root 2024-06-09T22:32:25.643 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-a[71945]: ignoring --setgroup ceph since I am not root 2024-06-09T22:32:25.643 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-mds[71971]: -- 172.21.15.104:0/1714295791 <== mon.0 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a5f882fa40 con 0x55a5f8802c00 2024-06-09T22:32:25.643 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-a[71945]: starting mds.a at 2024-06-09T22:32:25.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-mon[36599]: pgmap v356: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-06-09T22:32:25.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/879620516' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-06-09T22:32:25.996 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-mon[36599]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-06-09T22:32:25.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-mon[36599]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-09T22:32:25.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/879620516' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-06-09T22:32:25.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-mon[36599]: osdmap e111: 12 total, 12 up, 12 in 2024-06-09T22:32:25.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:25 smithi153 ceph-mon[36599]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-06-09T22:32:26.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-mon[32389]: pgmap v356: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-06-09T22:32:26.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/879620516' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-06-09T22:32:26.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-mon[32389]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-06-09T22:32:26.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-mon[32389]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-09T22:32:26.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/879620516' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-06-09T22:32:26.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-mon[32389]: osdmap e111: 12 total, 12 up, 12 in 2024-06-09T22:32:26.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:25 smithi104 ceph-mon[32389]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-06-09T22:32:26.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-mon[36856]: pgmap v356: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-06-09T22:32:26.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/879620516' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-06-09T22:32:26.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-mon[36856]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-06-09T22:32:26.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-mon[36856]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-09T22:32:26.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/879620516' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-06-09T22:32:26.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-mon[36856]: osdmap e111: 12 total, 12 up, 12 in 2024-06-09T22:32:26.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:25 smithi194 ceph-mon[36856]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-06-09T22:32:26.122 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-06-09T22:32:26.365 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:32:27.510 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: pgmap v358: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-06-09T22:32:27.510 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.104:6838/3475660337,v1:172.21.15.104:6839/3475660337] up:boot 2024-06-09T22:32:27.511 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.104:6840/4069205836,v1:172.21.15.104:6841/4069205836] up:boot 2024-06-09T22:32:27.511 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.153:6832/268770264,v1:172.21.15.153:6834/268770264] up:boot 2024-06-09T22:32:27.511 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.153:6833/2036588455,v1:172.21.15.153:6835/2036588455] up:boot 2024-06-09T22:32:27.511 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.194:6832/1100968959,v1:172.21.15.194:6835/1100968959] up:boot 2024-06-09T22:32:27.511 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.194:6833/1341347368,v1:172.21.15.194:6836/1341347368] up:boot 2024-06-09T22:32:27.511 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.194:6834/1569714373,v1:172.21.15.194:6837/1569714373] up:boot 2024-06-09T22:32:27.511 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.194:6838/4119659985,v1:172.21.15.194:6839/4119659985] up:boot 2024-06-09T22:32:27.511 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.153:6836/3664922294,v1:172.21.15.153:6838/3664922294] up:boot 2024-06-09T22:32:27.512 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: mds.? [v2:172.21.15.153:6837/359908808,v1:172.21.15.153:6839/359908808] up:boot 2024-06-09T22:32:27.512 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-06-09T22:32:27.512 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-09T22:32:27.512 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-09T22:32:27.512 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-09T22:32:27.512 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-09T22:32:27.512 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-09T22:32:27.512 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-09T22:32:27.513 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-09T22:32:27.513 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-09T22:32:27.513 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-09T22:32:27.513 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:27 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-09T22:32:27.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: pgmap v358: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-06-09T22:32:27.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.104:6838/3475660337,v1:172.21.15.104:6839/3475660337] up:boot 2024-06-09T22:32:27.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.104:6840/4069205836,v1:172.21.15.104:6841/4069205836] up:boot 2024-06-09T22:32:27.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.153:6832/268770264,v1:172.21.15.153:6834/268770264] up:boot 2024-06-09T22:32:27.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.153:6833/2036588455,v1:172.21.15.153:6835/2036588455] up:boot 2024-06-09T22:32:27.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.194:6832/1100968959,v1:172.21.15.194:6835/1100968959] up:boot 2024-06-09T22:32:27.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.194:6833/1341347368,v1:172.21.15.194:6836/1341347368] up:boot 2024-06-09T22:32:27.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.194:6834/1569714373,v1:172.21.15.194:6837/1569714373] up:boot 2024-06-09T22:32:27.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.194:6838/4119659985,v1:172.21.15.194:6839/4119659985] up:boot 2024-06-09T22:32:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.153:6836/3664922294,v1:172.21.15.153:6838/3664922294] up:boot 2024-06-09T22:32:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: mds.? [v2:172.21.15.153:6837/359908808,v1:172.21.15.153:6839/359908808] up:boot 2024-06-09T22:32:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-06-09T22:32:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-09T22:32:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-09T22:32:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-09T22:32:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-09T22:32:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-09T22:32:27.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-09T22:32:27.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-09T22:32:27.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-09T22:32:27.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-09T22:32:27.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:27 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-09T22:32:27.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: pgmap v358: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-06-09T22:32:27.747 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.104:6838/3475660337,v1:172.21.15.104:6839/3475660337] up:boot 2024-06-09T22:32:27.747 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.104:6840/4069205836,v1:172.21.15.104:6841/4069205836] up:boot 2024-06-09T22:32:27.747 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.153:6832/268770264,v1:172.21.15.153:6834/268770264] up:boot 2024-06-09T22:32:27.747 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.153:6833/2036588455,v1:172.21.15.153:6835/2036588455] up:boot 2024-06-09T22:32:27.747 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.194:6832/1100968959,v1:172.21.15.194:6835/1100968959] up:boot 2024-06-09T22:32:27.747 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.194:6833/1341347368,v1:172.21.15.194:6836/1341347368] up:boot 2024-06-09T22:32:27.747 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.194:6834/1569714373,v1:172.21.15.194:6837/1569714373] up:boot 2024-06-09T22:32:27.747 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.194:6838/4119659985,v1:172.21.15.194:6839/4119659985] up:boot 2024-06-09T22:32:27.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.153:6836/3664922294,v1:172.21.15.153:6838/3664922294] up:boot 2024-06-09T22:32:27.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: mds.? [v2:172.21.15.153:6837/359908808,v1:172.21.15.153:6839/359908808] up:boot 2024-06-09T22:32:27.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-06-09T22:32:27.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-09T22:32:27.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-09T22:32:27.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-09T22:32:27.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-09T22:32:27.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-09T22:32:27.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-09T22:32:27.749 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-09T22:32:27.749 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-09T22:32:27.749 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-09T22:32:27.749 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:27 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-09T22:32:28.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.215+0000 7f13ac75e700 1 -- 172.21.15.104:0/1884773908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13a4107840 msgr2=0x7f13a410f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:28.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.215+0000 7f13ac75e700 1 --2- 172.21.15.104:0/1884773908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13a4107840 0x7f13a410f3c0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f139c00b0d0 tx=0x7f139c00b3e0 comp rx=0 tx=0).stop 2024-06-09T22:32:28.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.215+0000 7f13ac75e700 1 -- 172.21.15.104:0/1884773908 shutdown_connections 2024-06-09T22:32:28.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.215+0000 7f13ac75e700 1 --2- 172.21.15.104:0/1884773908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13a4107840 0x7f13a410f3c0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:28.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.215+0000 7f13ac75e700 1 --2- 172.21.15.104:0/1884773908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13a4102740 0x7f13a4102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:28.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.215+0000 7f13ac75e700 1 --2- 172.21.15.104:0/1884773908 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13a41030f0 0x7f13a4107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:28.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.215+0000 7f13ac75e700 1 -- 172.21.15.104:0/1884773908 >> 172.21.15.104:0/1884773908 conn(0x7f13a40795f0 msgr2=0x7f13a4079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:28.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.215+0000 7f13ac75e700 1 -- 172.21.15.104:0/1884773908 shutdown_connections 2024-06-09T22:32:28.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.215+0000 7f13ac75e700 1 -- 172.21.15.104:0/1884773908 wait complete. 2024-06-09T22:32:28.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.215+0000 7f13ac75e700 1 Processor -- start 2024-06-09T22:32:28.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13ac75e700 1 -- start start 2024-06-09T22:32:28.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13ac75e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13a4102740 0x7f13a419e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:28.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13ac75e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13a41030f0 0x7f13a419ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:28.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13ac75e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13a4107840 0x7f13a4198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:28.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13ac75e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f13a4112760 con 0x7f13a4102740 2024-06-09T22:32:28.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13ac75e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f13a41125e0 con 0x7f13a4107840 2024-06-09T22:32:28.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13ac75e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f13a41128e0 con 0x7f13a41030f0 2024-06-09T22:32:28.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13aa4fa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13a4102740 0x7f13a419e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:28.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13aa4fa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13a4102740 0x7f13a419e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55100/0 (socket says 172.21.15.104:55100) 2024-06-09T22:32:28.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13aa4fa700 1 -- 172.21.15.104:0/1471138143 learned_addr learned my addr 172.21.15.104:0/1471138143 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:32:28.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13aacfb700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13a4107840 0x7f13a4198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:28.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13a9cf9700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13a41030f0 0x7f13a419ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:28.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13aa4fa700 1 -- 172.21.15.104:0/1471138143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13a41030f0 msgr2=0x7f13a419ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:28.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13aa4fa700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13a41030f0 0x7f13a419ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:28.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13aa4fa700 1 -- 172.21.15.104:0/1471138143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13a4107840 msgr2=0x7f13a4198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:28.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13aa4fa700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13a4107840 0x7f13a4198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:28.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13aa4fa700 1 -- 172.21.15.104:0/1471138143 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a41990c0 con 0x7f13a4102740 2024-06-09T22:32:28.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13aacfb700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13a4107840 0x7f13a4198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-09T22:32:28.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13a9cf9700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13a41030f0 0x7f13a419ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:32:28.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.216+0000 7f13aa4fa700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13a4102740 0x7f13a419e7a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f13a000d950 tx=0x7f13a000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:28.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.217+0000 7f139b7fe700 1 -- 172.21.15.104:0/1471138143 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0016070 con 0x7f13a4102740 2024-06-09T22:32:28.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.217+0000 7f139b7fe700 1 -- 172.21.15.104:0/1471138143 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a0009b40 con 0x7f13a4102740 2024-06-09T22:32:28.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.217+0000 7f13ac75e700 1 -- 172.21.15.104:0/1471138143 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a41993b0 con 0x7f13a4102740 2024-06-09T22:32:28.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.217+0000 7f139b7fe700 1 -- 172.21.15.104:0/1471138143 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a000f620 con 0x7f13a4102740 2024-06-09T22:32:28.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.217+0000 7f13ac75e700 1 -- 172.21.15.104:0/1471138143 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a41a5670 con 0x7f13a4102740 2024-06-09T22:32:28.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.218+0000 7f13ac75e700 1 -- 172.21.15.104:0/1471138143 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13a4069180 con 0x7f13a4102740 2024-06-09T22:32:28.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.219+0000 7f139b7fe700 1 -- 172.21.15.104:0/1471138143 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f13a0019070 con 0x7f13a4102740 2024-06-09T22:32:28.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.219+0000 7f139b7fe700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13900a2d90 0x7f13900a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:28.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.220+0000 7f13a9cf9700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13900a2d90 0x7f13900a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:28.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.220+0000 7f139b7fe700 1 -- 172.21.15.104:0/1471138143 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9838+0+0 (secure 0 0 0) 0x7f13a00cf9b0 con 0x7f13a4102740 2024-06-09T22:32:28.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.220+0000 7f13a9cf9700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13900a2d90 0x7f13900a5250 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1394005fd0 tx=0x7f1394005ee0 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:28.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.224+0000 7f139b7fe700 1 -- 172.21.15.104:0/1471138143 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f13a009c0b0 con 0x7f13a4102740 2024-06-09T22:32:28.483 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:28 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:32:28.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:28.479+0000 7f13ac75e700 1 -- 172.21.15.104:0/1471138143 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f13a402dca0 con 0x7f13a4102740 2024-06-09T22:32:28.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:28 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:32:28.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:28 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-09T22:32:29.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.274+0000 7f139b7fe700 1 -- 172.21.15.104:0/1471138143 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) ==== 104+0+0 (secure 0 0 0) 0x7f13a009b9b0 con 0x7f13a4102740 2024-06-09T22:32:29.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.276+0000 7f13ac75e700 1 -- 172.21.15.104:0/1471138143 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13900a2d90 msgr2=0x7f13900a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:29.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.276+0000 7f13ac75e700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13900a2d90 0x7f13900a5250 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1394005fd0 tx=0x7f1394005ee0 comp rx=0 tx=0).stop 2024-06-09T22:32:29.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.276+0000 7f13ac75e700 1 -- 172.21.15.104:0/1471138143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13a4102740 msgr2=0x7f13a419e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:29.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.276+0000 7f13ac75e700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13a4102740 0x7f13a419e7a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f13a000d950 tx=0x7f13a000dd10 comp rx=0 tx=0).stop 2024-06-09T22:32:29.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.276+0000 7f13ac75e700 1 -- 172.21.15.104:0/1471138143 shutdown_connections 2024-06-09T22:32:29.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.276+0000 7f13ac75e700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7f13900a2d90 0x7f13900a5250 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:29.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.276+0000 7f13ac75e700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13a4102740 0x7f13a419e7a0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:29.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.276+0000 7f13ac75e700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13a4107840 0x7f13a4198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:29.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.276+0000 7f13ac75e700 1 --2- 172.21.15.104:0/1471138143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13a41030f0 0x7f13a419ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:29.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.277+0000 7f13ac75e700 1 -- 172.21.15.104:0/1471138143 >> 172.21.15.104:0/1471138143 conn(0x7f13a40795f0 msgr2=0x7f13a4100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:29.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.277+0000 7f13ac75e700 1 -- 172.21.15.104:0/1471138143 shutdown_connections 2024-06-09T22:32:29.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:29.277+0000 7f13ac75e700 1 -- 172.21.15.104:0/1471138143 wait complete. 2024-06-09T22:32:29.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:29 smithi194 ceph-mon[36856]: pgmap v359: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-06-09T22:32:29.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:29 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1471138143' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-06-09T22:32:29.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:29 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:32:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:29 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:32:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:29 smithi194 ceph-mon[36856]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:32:29.648 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:29 smithi104 ceph-mon[32389]: pgmap v359: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-06-09T22:32:29.649 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:29 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1471138143' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-06-09T22:32:29.649 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:29 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:32:29.649 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:29 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:32:29.649 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:29 smithi104 ceph-mon[32389]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:32:29.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:29 smithi153 ceph-mon[36599]: pgmap v359: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-06-09T22:32:29.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:29 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1471138143' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-06-09T22:32:29.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:29 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-09T22:32:29.747 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:29 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-09T22:32:29.747 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:29 smithi153 ceph-mon[36599]: from='mgr.15279 172.21.15.104:0/2320493861' entity='mgr.x' 2024-06-09T22:32:30.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-09T22:32:30.320 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:32:30.442 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:30 smithi104 ceph-mon[32389]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-06-09T22:32:30.442 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:30 smithi104 ceph-mon[32389]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-06-09T22:32:30.442 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:30 smithi104 ceph-mon[32389]: Cluster is now healthy 2024-06-09T22:32:30.442 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:30 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1471138143' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-06-09T22:32:30.442 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:30 smithi104 ceph-mon[32389]: fsmap 12 up:standby 2024-06-09T22:32:30.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:30 smithi194 ceph-mon[36856]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-06-09T22:32:30.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:30 smithi194 ceph-mon[36856]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-06-09T22:32:30.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:30 smithi194 ceph-mon[36856]: Cluster is now healthy 2024-06-09T22:32:30.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:30 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1471138143' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-06-09T22:32:30.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:30 smithi194 ceph-mon[36856]: fsmap 12 up:standby 2024-06-09T22:32:30.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:30 smithi153 ceph-mon[36599]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-06-09T22:32:30.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:30 smithi153 ceph-mon[36599]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-06-09T22:32:30.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:30 smithi153 ceph-mon[36599]: Cluster is now healthy 2024-06-09T22:32:30.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:30 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1471138143' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-06-09T22:32:30.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:30 smithi153 ceph-mon[36599]: fsmap 12 up:standby 2024-06-09T22:32:31.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:31 smithi194 ceph-mon[36856]: pgmap v360: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-06-09T22:32:31.583 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:31 smithi104 ceph-mon[32389]: pgmap v360: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-06-09T22:32:31.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:31 smithi153 ceph-mon[36599]: pgmap v360: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-06-09T22:32:32.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.161+0000 7fc031610700 1 -- 172.21.15.104:0/1532805863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc02c101440 msgr2=0x7fc02c10f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:32.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.161+0000 7fc031610700 1 --2- 172.21.15.104:0/1532805863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc02c101440 0x7fc02c10f330 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fc02400b0a0 tx=0x7fc02400b3b0 comp rx=0 tx=0).stop 2024-06-09T22:32:32.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.161+0000 7fc031610700 1 -- 172.21.15.104:0/1532805863 shutdown_connections 2024-06-09T22:32:32.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.161+0000 7fc031610700 1 --2- 172.21.15.104:0/1532805863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc02c101440 0x7fc02c10f330 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:32.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.161+0000 7fc031610700 1 --2- 172.21.15.104:0/1532805863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc02c108510 0x7fc02c1088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:32.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.161+0000 7fc031610700 1 --2- 172.21.15.104:0/1532805863 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100a80 0x7fc02c100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:32.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.161+0000 7fc031610700 1 -- 172.21.15.104:0/1532805863 >> 172.21.15.104:0/1532805863 conn(0x7fc02c077a00 msgr2=0x7fc02c0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:32.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.161+0000 7fc031610700 1 -- 172.21.15.104:0/1532805863 shutdown_connections 2024-06-09T22:32:32.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.161+0000 7fc031610700 1 -- 172.21.15.104:0/1532805863 wait complete. 2024-06-09T22:32:32.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.162+0000 7fc031610700 1 Processor -- start 2024-06-09T22:32:32.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.162+0000 7fc031610700 1 -- start start 2024-06-09T22:32:32.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.162+0000 7fc031610700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100a80 0x7fc02c19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:32.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.162+0000 7fc031610700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc02c101440 0x7fc02c19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:32.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.162+0000 7fc031610700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc02c108510 0x7fc02c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:32.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.162+0000 7fc031610700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc02c112770 con 0x7fc02c101440 2024-06-09T22:32:32.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.162+0000 7fc031610700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc02c1125f0 con 0x7fc02c108510 2024-06-09T22:32:32.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.162+0000 7fc031610700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc02c1128f0 con 0x7fc02c100a80 2024-06-09T22:32:32.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.162+0000 7fc02a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc02c101440 0x7fc02c19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:32.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.162+0000 7fc02a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc02c101440 0x7fc02c19ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55108/0 (socket says 172.21.15.104:55108) 2024-06-09T22:32:32.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.162+0000 7fc02affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100a80 0x7fc02c19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:32.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.162+0000 7fc02a7fc700 1 -- 172.21.15.104:0/325734138 learned_addr learned my addr 172.21.15.104:0/325734138 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:32:32.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.163+0000 7fc02b7fe700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc02c108510 0x7fc02c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:32.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.163+0000 7fc02a7fc700 1 -- 172.21.15.104:0/325734138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100a80 msgr2=0x7fc02c19e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:32.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.163+0000 7fc02a7fc700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100a80 0x7fc02c19e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:32.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.163+0000 7fc02a7fc700 1 -- 172.21.15.104:0/325734138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc02c108510 msgr2=0x7fc02c198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:32.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.163+0000 7fc02a7fc700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc02c108510 0x7fc02c198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:32.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.163+0000 7fc02a7fc700 1 -- 172.21.15.104:0/325734138 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02c198fd0 con 0x7fc02c101440 2024-06-09T22:32:32.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.163+0000 7fc02a7fc700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc02c101440 0x7fc02c19ebf0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc02000c9b0 tx=0x7fc02000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:32.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.163+0000 7fc013fff700 1 -- 172.21.15.104:0/325734138 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc020007b00 con 0x7fc02c101440 2024-06-09T22:32:32.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.163+0000 7fc013fff700 1 -- 172.21.15.104:0/325734138 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc02000ce50 con 0x7fc02c101440 2024-06-09T22:32:32.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.163+0000 7fc031610700 1 -- 172.21.15.104:0/325734138 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc02c1992c0 con 0x7fc02c101440 2024-06-09T22:32:32.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.163+0000 7fc013fff700 1 -- 172.21.15.104:0/325734138 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc020018660 con 0x7fc02c101440 2024-06-09T22:32:32.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.163+0000 7fc031610700 1 -- 172.21.15.104:0/325734138 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc02c1a5580 con 0x7fc02c101440 2024-06-09T22:32:32.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.164+0000 7fc031610700 1 -- 172.21.15.104:0/325734138 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc02c199da0 con 0x7fc02c101440 2024-06-09T22:32:32.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.166+0000 7fc013fff700 1 -- 172.21.15.104:0/325734138 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc02000f450 con 0x7fc02c101440 2024-06-09T22:32:32.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.167+0000 7fc013fff700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0140a2d90 0x7fc0140a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:32.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.167+0000 7fc02affd700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0140a2d90 0x7fc0140a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:32.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.167+0000 7fc013fff700 1 -- 172.21.15.104:0/325734138 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9838+0+0 (secure 0 0 0) 0x7fc0200cfb20 con 0x7fc02c101440 2024-06-09T22:32:32.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.167+0000 7fc02affd700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0140a2d90 0x7fc0140a5250 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc02c1999b0 tx=0x7fc01c00a380 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:32.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.170+0000 7fc013fff700 1 -- 172.21.15.104:0/325734138 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc0200cfe20 con 0x7fc02c101440 2024-06-09T22:32:32.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:32.369+0000 7fc031610700 1 -- 172.21.15.104:0/325734138 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc02c102200 con 0x7fc02c101440 2024-06-09T22:32:33.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.288+0000 7fc013fff700 1 -- 172.21.15.104:0/325734138 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v112) ==== 181+0+0 (secure 0 0 0) 0x7fc020014030 con 0x7fc02c101440 2024-06-09T22:32:33.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.290+0000 7fc031610700 1 -- 172.21.15.104:0/325734138 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0140a2d90 msgr2=0x7fc0140a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:33.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.290+0000 7fc031610700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0140a2d90 0x7fc0140a5250 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc02c1999b0 tx=0x7fc01c00a380 comp rx=0 tx=0).stop 2024-06-09T22:32:33.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.290+0000 7fc031610700 1 -- 172.21.15.104:0/325734138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc02c101440 msgr2=0x7fc02c19ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:33.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.290+0000 7fc031610700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc02c101440 0x7fc02c19ebf0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc02000c9b0 tx=0x7fc02000ccc0 comp rx=0 tx=0).stop 2024-06-09T22:32:33.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.290+0000 7fc031610700 1 -- 172.21.15.104:0/325734138 shutdown_connections 2024-06-09T22:32:33.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.291+0000 7fc031610700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0140a2d90 0x7fc0140a5250 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:33.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.291+0000 7fc031610700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc02c108510 0x7fc02c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:33.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.291+0000 7fc031610700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc02c100a80 0x7fc02c19e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:33.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.291+0000 7fc031610700 1 --2- 172.21.15.104:0/325734138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc02c101440 0x7fc02c19ebf0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:33.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.291+0000 7fc031610700 1 -- 172.21.15.104:0/325734138 >> 172.21.15.104:0/325734138 conn(0x7fc02c077a00 msgr2=0x7fc02c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:33.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.291+0000 7fc031610700 1 -- 172.21.15.104:0/325734138 shutdown_connections 2024-06-09T22:32:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:33.291+0000 7fc031610700 1 -- 172.21.15.104:0/325734138 wait complete. 2024-06-09T22:32:33.296 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2024-06-09T22:32:33.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:33 smithi194 ceph-mon[36856]: pgmap v361: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-06-09T22:32:33.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:33 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/325734138' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-09T22:32:33.678 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:33 smithi104 ceph-mon[32389]: pgmap v361: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-06-09T22:32:33.679 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:33 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/325734138' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-09T22:32:33.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:33 smithi153 ceph-mon[36599]: pgmap v361: 129 pgs: 129 active+clean; 246 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-06-09T22:32:33.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:33 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/325734138' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-09T22:32:34.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-09T22:32:34.361 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:32:34.483 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:34 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/325734138' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-06-09T22:32:34.483 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:34 smithi104 ceph-mon[32389]: osdmap e112: 12 total, 12 up, 12 in 2024-06-09T22:32:34.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:34 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/325734138' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-06-09T22:32:34.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:34 smithi194 ceph-mon[36856]: osdmap e112: 12 total, 12 up, 12 in 2024-06-09T22:32:34.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:34 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/325734138' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-06-09T22:32:34.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:34 smithi153 ceph-mon[36599]: osdmap e112: 12 total, 12 up, 12 in 2024-06-09T22:32:35.510 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:35 smithi104 ceph-mon[32389]: pgmap v363: 97 pgs: 97 active+clean; 29 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:32:35.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:35 smithi194 ceph-mon[36856]: pgmap v363: 97 pgs: 97 active+clean; 29 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:32:35.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:35 smithi153 ceph-mon[36599]: pgmap v363: 97 pgs: 97 active+clean; 29 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:32:36.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 -- 172.21.15.104:0/2570328975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4101390 msgr2=0x7fc0e4101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:36.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 --2- 172.21.15.104:0/2570328975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4101390 0x7fc0e4101810 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009990 tx=0x7fc0cc009ca0 comp rx=0 tx=0).stop 2024-06-09T22:32:36.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 -- 172.21.15.104:0/2570328975 shutdown_connections 2024-06-09T22:32:36.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 --2- 172.21.15.104:0/2570328975 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0e4101d50 0x7fc0e410e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 --2- 172.21.15.104:0/2570328975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0e4107400 0x7fc0e41077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 --2- 172.21.15.104:0/2570328975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4101390 0x7fc0e4101810 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 -- 172.21.15.104:0/2570328975 >> 172.21.15.104:0/2570328975 conn(0x7fc0e4078340 msgr2=0x7fc0e4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 -- 172.21.15.104:0/2570328975 shutdown_connections 2024-06-09T22:32:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 -- 172.21.15.104:0/2570328975 wait complete. 2024-06-09T22:32:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 Processor -- start 2024-06-09T22:32:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 -- start start 2024-06-09T22:32:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0e4101390 0x7fc0e4103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.170+0000 7fc0e9b44700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0e4101d50 0x7fc0e41035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e9b44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4107400 0x7fc0e4103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e9b44700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e41116f0 con 0x7fc0e4107400 2024-06-09T22:32:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e9b44700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e4111570 con 0x7fc0e4101390 2024-06-09T22:32:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e9b44700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e4111870 con 0x7fc0e4101d50 2024-06-09T22:32:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4107400 0x7fc0e4103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4107400 0x7fc0e4103af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52178/0 (socket says 172.21.15.104:52178) 2024-06-09T22:32:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e3fff700 1 -- 172.21.15.104:0/1553983593 learned_addr learned my addr 172.21.15.104:0/1553983593 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:32:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0dbfff700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0e4101d50 0x7fc0e41035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e37fe700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0e4101390 0x7fc0e4103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e3fff700 1 -- 172.21.15.104:0/1553983593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0e4101d50 msgr2=0x7fc0e41035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e3fff700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0e4101d50 0x7fc0e41035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e3fff700 1 -- 172.21.15.104:0/1553983593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0e4101390 msgr2=0x7fc0e4103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e3fff700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0e4101390 0x7fc0e4103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e3fff700 1 -- 172.21.15.104:0/1553983593 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e41a7120 con 0x7fc0e4107400 2024-06-09T22:32:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e37fe700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0e4101390 0x7fc0e4103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-09T22:32:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e3fff700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4107400 0x7fc0e4103af0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00b970 tx=0x7fc0dc00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e17fa700 1 -- 172.21.15.104:0/1553983593 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc019070 con 0x7fc0e4107400 2024-06-09T22:32:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e9b44700 1 -- 172.21.15.104:0/1553983593 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e41a7410 con 0x7fc0e4107400 2024-06-09T22:32:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e17fa700 1 -- 172.21.15.104:0/1553983593 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0dc00ce10 con 0x7fc0e4107400 2024-06-09T22:32:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e17fa700 1 -- 172.21.15.104:0/1553983593 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc007370 con 0x7fc0e4107400 2024-06-09T22:32:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.171+0000 7fc0e9b44700 1 -- 172.21.15.104:0/1553983593 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e41a78d0 con 0x7fc0e4107400 2024-06-09T22:32:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.172+0000 7fc0e9b44700 1 -- 172.21.15.104:0/1553983593 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e4069180 con 0x7fc0e4107400 2024-06-09T22:32:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.174+0000 7fc0e17fa700 1 -- 172.21.15.104:0/1553983593 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc0dc01c070 con 0x7fc0e4107400 2024-06-09T22:32:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.174+0000 7fc0e17fa700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0d00a2d90 0x7fc0d00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.175+0000 7fc0e37fe700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0d00a2d90 0x7fc0d00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.175+0000 7fc0e17fa700 1 -- 172.21.15.104:0/1553983593 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9313+0+0 (secure 0 0 0) 0x7fc0dc0cea30 con 0x7fc0e4107400 2024-06-09T22:32:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.175+0000 7fc0e37fe700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0d00a2d90 0x7fc0d00a5250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4005fd0 tx=0x7fc0d4005e20 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:36.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.179+0000 7fc0e17fa700 1 -- 172.21.15.104:0/1553983593 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc0dc09b440 con 0x7fc0e4107400 2024-06-09T22:32:36.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:36.377+0000 7fc0e9b44700 1 -- 172.21.15.104:0/1553983593 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc0e41a7b60 con 0x7fc0e4107400 2024-06-09T22:32:37.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.302+0000 7fc0e17fa700 1 -- 172.21.15.104:0/1553983593 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v113) ==== 169+0+0 (secure 0 0 0) 0x7fc0dc09ad40 con 0x7fc0e4107400 2024-06-09T22:32:37.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.307+0000 7fc0e9b44700 1 -- 172.21.15.104:0/1553983593 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0d00a2d90 msgr2=0x7fc0d00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:37.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.307+0000 7fc0e9b44700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0d00a2d90 0x7fc0d00a5250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4005fd0 tx=0x7fc0d4005e20 comp rx=0 tx=0).stop 2024-06-09T22:32:37.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.307+0000 7fc0e9b44700 1 -- 172.21.15.104:0/1553983593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4107400 msgr2=0x7fc0e4103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:37.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.307+0000 7fc0e9b44700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4107400 0x7fc0e4103af0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00b970 tx=0x7fc0dc00bc80 comp rx=0 tx=0).stop 2024-06-09T22:32:37.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.307+0000 7fc0e9b44700 1 -- 172.21.15.104:0/1553983593 shutdown_connections 2024-06-09T22:32:37.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.307+0000 7fc0e9b44700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fc0d00a2d90 0x7fc0d00a5250 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:37.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.307+0000 7fc0e9b44700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4107400 0x7fc0e4103af0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:37.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.307+0000 7fc0e9b44700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0e4101390 0x7fc0e4103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:37.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.307+0000 7fc0e9b44700 1 --2- 172.21.15.104:0/1553983593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0e4101d50 0x7fc0e41035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:37.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.307+0000 7fc0e9b44700 1 -- 172.21.15.104:0/1553983593 >> 172.21.15.104:0/1553983593 conn(0x7fc0e4078340 msgr2=0x7fc0e41005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:37.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.307+0000 7fc0e9b44700 1 -- 172.21.15.104:0/1553983593 shutdown_connections 2024-06-09T22:32:37.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:37.308+0000 7fc0e9b44700 1 -- 172.21.15.104:0/1553983593 wait complete. 2024-06-09T22:32:37.312 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2024-06-09T22:32:37.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:37 smithi194 ceph-mon[36856]: pgmap v364: 97 pgs: 97 active+clean; 29 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:32:37.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:37 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1553983593' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-09T22:32:37.685 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:37 smithi104 ceph-mon[32389]: pgmap v364: 97 pgs: 97 active+clean; 29 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:32:37.686 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:37 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1553983593' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-09T22:32:37.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:37 smithi153 ceph-mon[36599]: pgmap v364: 97 pgs: 97 active+clean; 29 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:32:37.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:37 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1553983593' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-09T22:32:38.128 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-06-09T22:32:38.373 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/mon.a/config 2024-06-09T22:32:38.494 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:38 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/1553983593' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-06-09T22:32:38.495 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:38 smithi104 ceph-mon[32389]: osdmap e113: 12 total, 12 up, 12 in 2024-06-09T22:32:38.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:38 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/1553983593' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-06-09T22:32:38.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:38 smithi194 ceph-mon[36856]: osdmap e113: 12 total, 12 up, 12 in 2024-06-09T22:32:38.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:38 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/1553983593' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-06-09T22:32:38.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:38 smithi153 ceph-mon[36599]: osdmap e113: 12 total, 12 up, 12 in 2024-06-09T22:32:39.510 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:39 smithi104 ceph-mon[32389]: pgmap v366: 33 pgs: 33 active+clean; 29 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:32:39.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:39 smithi153 ceph-mon[36599]: pgmap v366: 33 pgs: 33 active+clean; 29 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:32:39.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:39 smithi194 ceph-mon[36856]: pgmap v366: 33 pgs: 33 active+clean; 29 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:32:40.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.278+0000 7fbb92b63700 1 -- 172.21.15.104:0/1880274745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb8c1024b0 msgr2=0x7fbb8c10e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:40.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.278+0000 7fbb92b63700 1 --2- 172.21.15.104:0/1880274745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb8c1024b0 0x7fbb8c10e260 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fbb8400b0a0 tx=0x7fbb8400b3b0 comp rx=0 tx=0).stop 2024-06-09T22:32:40.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.278+0000 7fbb92b63700 1 -- 172.21.15.104:0/1880274745 shutdown_connections 2024-06-09T22:32:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.278+0000 7fbb92b63700 1 --2- 172.21.15.104:0/1880274745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb8c1024b0 0x7fbb8c10e260 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.278+0000 7fbb92b63700 1 --2- 172.21.15.104:0/1880274745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb8c1063c0 0x7fbb8c1067a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.278+0000 7fbb92b63700 1 --2- 172.21.15.104:0/1880274745 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb8c101af0 0x7fbb8c101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.278+0000 7fbb92b63700 1 -- 172.21.15.104:0/1880274745 >> 172.21.15.104:0/1880274745 conn(0x7fbb8c077990 msgr2=0x7fbb8c0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.278+0000 7fbb92b63700 1 -- 172.21.15.104:0/1880274745 shutdown_connections 2024-06-09T22:32:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.278+0000 7fbb92b63700 1 -- 172.21.15.104:0/1880274745 wait complete. 2024-06-09T22:32:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.278+0000 7fbb92b63700 1 Processor -- start 2024-06-09T22:32:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.278+0000 7fbb92b63700 1 -- start start 2024-06-09T22:32:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb92b63700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb8c101af0 0x7fbb8c19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb92b63700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb8c1024b0 0x7fbb8c19ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb92b63700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb8c1063c0 0x7fbb8c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb92b63700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c111690 con 0x7fbb8c1063c0 2024-06-09T22:32:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb92b63700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c111510 con 0x7fbb8c1024b0 2024-06-09T22:32:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb92b63700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c111810 con 0x7fbb8c101af0 2024-06-09T22:32:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb91100700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb8c1063c0 0x7fbb8c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb91100700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb8c1063c0 0x7fbb8c1988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52190/0 (socket says 172.21.15.104:52190) 2024-06-09T22:32:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb91100700 1 -- 172.21.15.104:0/2575370620 learned_addr learned my addr 172.21.15.104:0/2575370620 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-06-09T22:32:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb908ff700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb8c101af0 0x7fbb8c19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb91100700 1 -- 172.21.15.104:0/2575370620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb8c101af0 msgr2=0x7fbb8c19e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb8bfff700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb8c1024b0 0x7fbb8c19ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb91100700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb8c101af0 0x7fbb8c19e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb91100700 1 -- 172.21.15.104:0/2575370620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb8c1024b0 msgr2=0x7fbb8c19ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb91100700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb8c1024b0 0x7fbb8c19ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb91100700 1 -- 172.21.15.104:0/2575370620 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8c199040 con 0x7fbb8c1063c0 2024-06-09T22:32:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb91100700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb8c1063c0 0x7fbb8c1988e0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fbb840062a0 tx=0x7fbb84004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb89ffb700 1 -- 172.21.15.104:0/2575370620 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb84021070 con 0x7fbb8c1063c0 2024-06-09T22:32:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb89ffb700 1 -- 172.21.15.104:0/2575370620 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb84007c80 con 0x7fbb8c1063c0 2024-06-09T22:32:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb92b63700 1 -- 172.21.15.104:0/2575370620 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb8c1992d0 con 0x7fbb8c1063c0 2024-06-09T22:32:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.279+0000 7fbb89ffb700 1 -- 172.21.15.104:0/2575370620 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8401d450 con 0x7fbb8c1063c0 2024-06-09T22:32:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.280+0000 7fbb92b63700 1 -- 172.21.15.104:0/2575370620 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb8c1a55f0 con 0x7fbb8c1063c0 2024-06-09T22:32:40.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.281+0000 7fbb92b63700 1 -- 172.21.15.104:0/2575370620 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb8c069180 con 0x7fbb8c1063c0 2024-06-09T22:32:40.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.282+0000 7fbb89ffb700 1 -- 172.21.15.104:0/2575370620 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fbb84004020 con 0x7fbb8c1063c0 2024-06-09T22:32:40.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.282+0000 7fbb89ffb700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbb7c0a2d90 0x7fbb7c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:32:40.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.282+0000 7fbb908ff700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbb7c0a2d90 0x7fbb7c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:32:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.282+0000 7fbb89ffb700 1 -- 172.21.15.104:0/2575370620 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8754+0+0 (secure 0 0 0) 0x7fbb8400f020 con 0x7fbb8c1063c0 2024-06-09T22:32:40.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.283+0000 7fbb908ff700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbb7c0a2d90 0x7fbb7c0a5250 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbb80005fd0 tx=0x7fbb80005f60 comp rx=0 tx=0).ready entity=mgr.15279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:32:40.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.287+0000 7fbb89ffb700 1 -- 172.21.15.104:0/2575370620 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbb8402a020 con 0x7fbb8c1063c0 2024-06-09T22:32:40.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:40.484+0000 7fbb92b63700 1 -- 172.21.15.104:0/2575370620 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbb8c1a57c0 con 0x7fbb8c1063c0 2024-06-09T22:32:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.318+0000 7fbb89ffb700 1 -- 172.21.15.104:0/2575370620 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v114) ==== 178+0+0 (secure 0 0 0) 0x7fbb8409d6d0 con 0x7fbb8c1063c0 2024-06-09T22:32:41.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.323+0000 7fbb92b63700 1 -- 172.21.15.104:0/2575370620 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbb7c0a2d90 msgr2=0x7fbb7c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:41.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.323+0000 7fbb92b63700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbb7c0a2d90 0x7fbb7c0a5250 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbb80005fd0 tx=0x7fbb80005f60 comp rx=0 tx=0).stop 2024-06-09T22:32:41.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.323+0000 7fbb92b63700 1 -- 172.21.15.104:0/2575370620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb8c1063c0 msgr2=0x7fbb8c1988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:32:41.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.323+0000 7fbb92b63700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb8c1063c0 0x7fbb8c1988e0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fbb840062a0 tx=0x7fbb84004e20 comp rx=0 tx=0).stop 2024-06-09T22:32:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.324+0000 7fbb92b63700 1 -- 172.21.15.104:0/2575370620 shutdown_connections 2024-06-09T22:32:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.324+0000 7fbb92b63700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.104:6800/493021072,v1:172.21.15.104:6801/493021072] conn(0x7fbb7c0a2d90 0x7fbb7c0a5250 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.324+0000 7fbb92b63700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb8c1063c0 0x7fbb8c1988e0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.324+0000 7fbb92b63700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb8c101af0 0x7fbb8c19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.324+0000 7fbb92b63700 1 --2- 172.21.15.104:0/2575370620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb8c1024b0 0x7fbb8c19ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:32:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.324+0000 7fbb92b63700 1 -- 172.21.15.104:0/2575370620 >> 172.21.15.104:0/2575370620 conn(0x7fbb8c077990 msgr2=0x7fbb8c100ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:32:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.324+0000 7fbb92b63700 1 -- 172.21.15.104:0/2575370620 shutdown_connections 2024-06-09T22:32:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-06-09T22:32:41.324+0000 7fbb92b63700 1 -- 172.21.15.104:0/2575370620 wait complete. 2024-06-09T22:32:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data_ec' removed 2024-06-09T22:32:41.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:41 smithi153 ceph-mon[36599]: pgmap v367: 33 pgs: 33 active+clean; 29 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:32:41.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:41 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2575370620' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-09T22:32:41.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:41 smithi104 ceph-mon[32389]: pgmap v367: 33 pgs: 33 active+clean; 29 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:32:41.760 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:41 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2575370620' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-09T22:32:41.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:41 smithi194 ceph-mon[36856]: pgmap v367: 33 pgs: 33 active+clean; 29 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-09T22:32:41.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:41 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2575370620' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-09T22:32:42.272 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-09T22:32:42.302 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-09T22:32:42.335 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-09T22:32:42.365 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-09T22:32:42.365 DEBUG:teuthology.orchestra.run.smithi104:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-09T22:32:42.381 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-09T22:32:42.399 DEBUG:teuthology.orchestra.run.smithi194:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-09T22:32:42.422 INFO:tasks.cephadm:Stopping all daemons... 2024-06-09T22:32:42.422 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-06-09T22:32:42.422 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.a 2024-06-09T22:32:42.510 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:42 smithi104 ceph-mon[32389]: from='client.? 172.21.15.104:0/2575370620' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-06-09T22:32:42.510 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:42 smithi104 ceph-mon[32389]: osdmap e114: 12 total, 12 up, 12 in 2024-06-09T22:32:42.510 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:42 smithi104 systemd[1]: Stopping Ceph mon.a for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:32:42.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:42 smithi153 ceph-mon[36599]: from='client.? 172.21.15.104:0/2575370620' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-06-09T22:32:42.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:42 smithi153 ceph-mon[36599]: osdmap e114: 12 total, 12 up, 12 in 2024-06-09T22:32:42.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:42 smithi194 ceph-mon[36856]: from='client.? 172.21.15.104:0/2575370620' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-06-09T22:32:42.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:42 smithi194 ceph-mon[36856]: osdmap e114: 12 total, 12 up, 12 in 2024-06-09T22:32:43.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:42 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a[32384]: 2024-06-09T22:32:42.714+0000 7f3adf8a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:32:43.010 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:42 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a[32384]: 2024-06-09T22:32:42.714+0000 7f3adf8a0700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-06-09T22:32:43.011 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:42 smithi104 podman[122385]: 2024-06-09 22:32:42.847951527 +0000 UTC m=+0.244133270 container died 2cb3d4ded8f69f6e28cefd01ca38469df63e8442bf5ea5c405eed635a0b1e429 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0) 2024-06-09T22:32:43.510 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:43 smithi104 podman[122385]: 2024-06-09 22:32:43.3407926 +0000 UTC m=+0.736974335 container cleanup 2cb3d4ded8f69f6e28cefd01ca38469df63e8442bf5ea5c405eed635a0b1e429 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, io.buildah.version=1.33.5) 2024-06-09T22:32:43.510 INFO:journalctl@ceph.mon.a.smithi104.stdout:Jun 09 22:32:43 smithi104 bash[122385]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-a 2024-06-09T22:32:43.745 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.a.service' 2024-06-09T22:32:43.791 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:32:43.791 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-06-09T22:32:43.791 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-06-09T22:32:43.791 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.b 2024-06-09T22:32:44.100 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:43 smithi153 systemd[1]: Stopping Ceph mon.b for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:32:44.496 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:44 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-b[36595]: 2024-06-09T22:32:44.096+0000 7ff4c3e32700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:32:44.496 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:44 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-b[36595]: 2024-06-09T22:32:44.096+0000 7ff4c3e32700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-06-09T22:32:44.496 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:44 smithi153 podman[73362]: 2024-06-09 22:32:44.290172898 +0000 UTC m=+0.304856719 container died 7de2111927b650159a58879354316b7c456b55da167ba0e194667720cd2bf765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-b, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, ceph=True, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS) 2024-06-09T22:32:44.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:44 smithi153 podman[73362]: 2024-06-09 22:32:44.697100988 +0000 UTC m=+0.711784808 container cleanup 7de2111927b650159a58879354316b7c456b55da167ba0e194667720cd2bf765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-09T22:32:44.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:44 smithi153 bash[73362]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-b 2024-06-09T22:32:45.097 INFO:journalctl@ceph.mon.b.smithi153.stdout:Jun 09 22:32:44 smithi153 podman[73376]: 2024-06-09 22:32:44.940301053 +0000 UTC m=+0.647459338 container remove 7de2111927b650159a58879354316b7c456b55da167ba0e194667720cd2bf765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2) 2024-06-09T22:32:45.112 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.b.service' 2024-06-09T22:32:45.159 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:32:45.159 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-06-09T22:32:45.159 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-06-09T22:32:45.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.c 2024-06-09T22:32:45.516 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:45 smithi194 systemd[1]: Stopping Ceph mon.c for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:32:46.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:45 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-c[36844]: 2024-06-09T22:32:45.797+0000 7ff388f39700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:32:46.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:45 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-c[36844]: 2024-06-09T22:32:45.797+0000 7ff388f39700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-06-09T22:32:46.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:45 smithi194 podman[73962]: 2024-06-09 22:32:45.900013531 +0000 UTC m=+0.549291410 container died 1e622cac1cc78614d23a20f71f7b142bbccb0a63fc633bbe57e46f2d5415c573 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-c, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:32:46.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:46 smithi194 podman[73962]: 2024-06-09 22:32:46.28166095 +0000 UTC m=+0.930938838 container cleanup 1e622cac1cc78614d23a20f71f7b142bbccb0a63fc633bbe57e46f2d5415c573 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:32:46.373 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:46 smithi194 bash[73962]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-c 2024-06-09T22:32:46.727 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jun 09 22:32:46 smithi194 podman[73975]: 2024-06-09 22:32:46.566993022 +0000 UTC m=+0.665215418 container remove 1e622cac1cc78614d23a20f71f7b142bbccb0a63fc633bbe57e46f2d5415c573 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:32:46.740 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mon.c.service' 2024-06-09T22:32:46.790 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:32:46.790 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-06-09T22:32:46.790 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-06-09T22:32:46.791 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.x 2024-06-09T22:32:47.123 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:32:46 smithi104 systemd[1]: Stopping Ceph mgr.x for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:32:47.450 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:32:47 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x[32617]: 2024-06-09T22:32:47.119+0000 7f2de4bd0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:32:47.450 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:32:47 smithi104 podman[122507]: 2024-06-09 22:32:47.151110053 +0000 UTC m=+0.165283048 container died b6c0d3ce81dbb43d65a11537b077ccb874d44886be4274548f615e6faaf21166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-06-09T22:32:47.760 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:32:47 smithi104 podman[122507]: 2024-06-09 22:32:47.604752185 +0000 UTC m=+0.618925171 container cleanup b6c0d3ce81dbb43d65a11537b077ccb874d44886be4274548f615e6faaf21166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528) 2024-06-09T22:32:47.760 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Jun 09 22:32:47 smithi104 bash[122507]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-x 2024-06-09T22:32:48.020 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.x.service' 2024-06-09T22:32:48.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:32:48.056 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-06-09T22:32:48.056 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-06-09T22:32:48.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.y 2024-06-09T22:32:48.391 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:32:48 smithi153 systemd[1]: Stopping Ceph mgr.y for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:32:48.705 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:32:48 smithi153 podman[73498]: 2024-06-09 22:32:48.38902321 +0000 UTC m=+0.142738191 container died 0827a1ba0af3fc1257e8739c8c2ce0d821daeb55a4e097f708754f23cbe478f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux ) 2024-06-09T22:32:48.996 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:32:48 smithi153 podman[73498]: 2024-06-09 22:32:48.794212881 +0000 UTC m=+0.547927859 container cleanup 0827a1ba0af3fc1257e8739c8c2ce0d821daeb55a4e097f708754f23cbe478f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:32:48.996 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Jun 09 22:32:48 smithi153 bash[73498]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-y 2024-06-09T22:32:49.176 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.y.service' 2024-06-09T22:32:49.211 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:32:49.211 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-06-09T22:32:49.211 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-06-09T22:32:49.211 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.z 2024-06-09T22:32:49.562 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:32:49 smithi194 systemd[1]: Stopping Ceph mgr.z for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:32:49.890 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:32:49 smithi194 podman[74094]: 2024-06-09 22:32:49.583966901 +0000 UTC m=+0.169963202 container died 0b98115dc341f2deb77d99544c2f0de42feac370f4740c3fb39bfeffacfd119e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-06-09T22:32:50.232 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:32:49 smithi194 podman[74094]: 2024-06-09 22:32:49.961884587 +0000 UTC m=+0.547880878 container cleanup 0b98115dc341f2deb77d99544c2f0de42feac370f4740c3fb39bfeffacfd119e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:32:50.233 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jun 09 22:32:49 smithi194 bash[74094]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mgr-z 2024-06-09T22:32:50.393 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mgr.z.service' 2024-06-09T22:32:50.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:32:50.427 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-06-09T22:32:50.427 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-09T22:32:50.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.0 2024-06-09T22:32:50.760 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:50 smithi104 systemd[1]: Stopping Ceph osd.0 for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:32:51.260 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0[44576]: 2024-06-09T22:32:50.783+0000 7f29be855700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:32:51.260 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0[44576]: 2024-06-09T22:32:50.783+0000 7f29be855700 -1 osd.0 114 *** Got signal Terminated *** 2024-06-09T22:32:51.260 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0[44576]: 2024-06-09T22:32:50.783+0000 7f29be855700 -1 osd.0 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-09T22:32:56.655 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:56 smithi104 podman[122627]: 2024-06-09 22:32:56.313369586 +0000 UTC m=+5.686326272 container died 00b495df162d12d259f4ec0a586ab0665ff24715e2811941bfa63b190aafae90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, io.buildah.version=1.33.5) 2024-06-09T22:32:56.980 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:56 smithi104 podman[122627]: 2024-06-09 22:32:56.743115653 +0000 UTC m=+6.116072340 container cleanup 00b495df162d12d259f4ec0a586ab0665ff24715e2811941bfa63b190aafae90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:32:56.981 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:56 smithi104 bash[122627]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0 2024-06-09T22:32:57.260 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:56 smithi104 podman[122655]: 2024-06-09 22:32:56.978246616 +0000 UTC m=+0.667655417 container remove 00b495df162d12d259f4ec0a586ab0665ff24715e2811941bfa63b190aafae90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:32:57.573 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:57 smithi104 podman[122743]: 2024-06-09 22:32:57.326497791 +0000 UTC m=+0.016970397 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:32:57.573 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:57 smithi104 podman[122743]: 2024-06-09 22:32:57.445777072 +0000 UTC m=+0.136249659 container create 15332774ec5ed33863b18b4890d1631f6435c4c52a85f8a60f6d4d00850da2fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240528) 2024-06-09T22:32:57.934 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:57 smithi104 podman[122743]: 2024-06-09 22:32:57.70667533 +0000 UTC m=+0.397147919 container init 15332774ec5ed33863b18b4890d1631f6435c4c52a85f8a60f6d4d00850da2fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-06-09T22:32:57.934 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:57 smithi104 podman[122743]: 2024-06-09 22:32:57.70993667 +0000 UTC m=+0.400409265 container start 15332774ec5ed33863b18b4890d1631f6435c4c52a85f8a60f6d4d00850da2fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528) 2024-06-09T22:32:57.934 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:57 smithi104 podman[122743]: 2024-06-09 22:32:57.931760399 +0000 UTC m=+0.622232987 container attach 15332774ec5ed33863b18b4890d1631f6435c4c52a85f8a60f6d4d00850da2fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-09T22:32:58.260 INFO:journalctl@ceph.osd.0.smithi104.stdout:Jun 09 22:32:58 smithi104 podman[122743]: 2024-06-09 22:32:58.012783409 +0000 UTC m=+0.703255997 container died 15332774ec5ed33863b18b4890d1631f6435c4c52a85f8a60f6d4d00850da2fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:32:58.652 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.0.service' 2024-06-09T22:32:58.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:32:58.692 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-09T22:32:58.693 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-09T22:32:58.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.1 2024-06-09T22:32:58.996 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:32:58 smithi153 systemd[1]: Stopping Ceph osd.1 for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:32:59.497 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:32:59 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1[41180]: 2024-06-09T22:32:59.006+0000 7fd798a85700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:32:59.497 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:32:59 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1[41180]: 2024-06-09T22:32:59.006+0000 7fd798a85700 -1 osd.1 114 *** Got signal Terminated *** 2024-06-09T22:32:59.497 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:32:59 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1[41180]: 2024-06-09T22:32:59.006+0000 7fd798a85700 -1 osd.1 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-09T22:33:04.746 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:33:04 smithi153 podman[73622]: 2024-06-09 22:33:04.484288769 +0000 UTC m=+5.593931076 container died 3f6c64c59e84ef293af8f69c98c982b9b98eff8e694170f2e4ce6f1a80b3165d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-06-09T22:33:05.144 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:33:04 smithi153 podman[73622]: 2024-06-09 22:33:04.88208628 +0000 UTC m=+5.991728589 container cleanup 3f6c64c59e84ef293af8f69c98c982b9b98eff8e694170f2e4ce6f1a80b3165d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1, org.label-schema.build-date=20240528, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc) 2024-06-09T22:33:05.144 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:33:04 smithi153 bash[73622]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1 2024-06-09T22:33:05.462 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:33:05 smithi153 podman[73638]: 2024-06-09 22:33:05.141851629 +0000 UTC m=+0.656651247 container remove 3f6c64c59e84ef293af8f69c98c982b9b98eff8e694170f2e4ce6f1a80b3165d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD) 2024-06-09T22:33:05.732 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:33:05 smithi153 podman[73729]: 2024-06-09 22:33:05.362448303 +0000 UTC m=+0.017337652 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:33:05.732 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:33:05 smithi153 podman[73729]: 2024-06-09 22:33:05.492914152 +0000 UTC m=+0.147803488 container create 9c27f9045ce20160cb530d550e06ac319bb27784f7580f4341c66573dcea0d12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-06-09T22:33:05.732 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:33:05 smithi153 podman[73729]: 2024-06-09 22:33:05.728962024 +0000 UTC m=+0.383851352 container init 9c27f9045ce20160cb530d550e06ac319bb27784f7580f4341c66573dcea0d12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:33:05.997 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:33:05 smithi153 podman[73729]: 2024-06-09 22:33:05.732335457 +0000 UTC m=+0.387224796 container start 9c27f9045ce20160cb530d550e06ac319bb27784f7580f4341c66573dcea0d12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-09T22:33:05.997 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:33:05 smithi153 podman[73729]: 2024-06-09 22:33:05.828973397 +0000 UTC m=+0.483862736 container attach 9c27f9045ce20160cb530d550e06ac319bb27784f7580f4341c66573dcea0d12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1-deactivate, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-06-09T22:33:06.299 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 09 22:33:06 smithi153 podman[73729]: 2024-06-09 22:33:06.017104849 +0000 UTC m=+0.671994188 container died 9c27f9045ce20160cb530d550e06ac319bb27784f7580f4341c66573dcea0d12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:33:06.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.1.service' 2024-06-09T22:33:06.669 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:33:06.669 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-09T22:33:06.669 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-09T22:33:06.670 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.2 2024-06-09T22:33:07.052 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:06 smithi194 systemd[1]: Stopping Ceph osd.2 for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:33:07.312 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:07 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2[41983]: 2024-06-09T22:33:07.049+0000 7fbdc0440700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:33:07.312 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:07 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2[41983]: 2024-06-09T22:33:07.049+0000 7fbdc0440700 -1 osd.2 114 *** Got signal Terminated *** 2024-06-09T22:33:07.313 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:07 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2[41983]: 2024-06-09T22:33:07.049+0000 7fbdc0440700 -1 osd.2 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-09T22:33:12.753 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:12 smithi194 podman[74222]: 2024-06-09 22:33:12.447563612 +0000 UTC m=+5.588611924 container died 3aacd8b0174f9575aae255fd52681c2dd7308b5152894c1f3d20abdf4cf8b065 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:33:13.062 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:12 smithi194 podman[74222]: 2024-06-09 22:33:12.849249945 +0000 UTC m=+5.990298248 container cleanup 3aacd8b0174f9575aae255fd52681c2dd7308b5152894c1f3d20abdf4cf8b065 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:33:13.063 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:12 smithi194 bash[74222]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2 2024-06-09T22:33:13.472 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:13 smithi194 podman[74239]: 2024-06-09 22:33:13.117834698 +0000 UTC m=+0.671956769 container remove 3aacd8b0174f9575aae255fd52681c2dd7308b5152894c1f3d20abdf4cf8b065 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:33:13.799 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:13 smithi194 podman[74328]: 2024-06-09 22:33:13.372201858 +0000 UTC m=+0.017766049 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:33:13.799 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:13 smithi194 podman[74328]: 2024-06-09 22:33:13.485504933 +0000 UTC m=+0.131069117 container create 21656d8a490eb78ce39b5d1a8382c5e5726f367b854b68440e5507f4667dac36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240528) 2024-06-09T22:33:13.799 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:13 smithi194 podman[74328]: 2024-06-09 22:33:13.713350688 +0000 UTC m=+0.358914873 container init 21656d8a490eb78ce39b5d1a8382c5e5726f367b854b68440e5507f4667dac36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:33:13.799 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:13 smithi194 podman[74328]: 2024-06-09 22:33:13.716578153 +0000 UTC m=+0.362142348 container start 21656d8a490eb78ce39b5d1a8382c5e5726f367b854b68440e5507f4667dac36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc) 2024-06-09T22:33:13.799 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:13 smithi194 podman[74328]: 2024-06-09 22:33:13.796614903 +0000 UTC m=+0.442179095 container attach 21656d8a490eb78ce39b5d1a8382c5e5726f367b854b68440e5507f4667dac36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-06-09T22:33:14.062 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jun 09 22:33:14 smithi194 podman[74328]: 2024-06-09 22:33:14.006106402 +0000 UTC m=+0.651670594 container died 21656d8a490eb78ce39b5d1a8382c5e5726f367b854b68440e5507f4667dac36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True) 2024-06-09T22:33:14.649 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.2.service' 2024-06-09T22:33:14.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:33:14.688 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-09T22:33:14.688 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-09T22:33:14.688 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.3 2024-06-09T22:33:15.010 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:14 smithi104 systemd[1]: Stopping Ceph osd.3 for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:33:15.510 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:15 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3[51484]: 2024-06-09T22:33:15.011+0000 7f1f631f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:33:15.511 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:15 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3[51484]: 2024-06-09T22:33:15.011+0000 7f1f631f5700 -1 osd.3 114 *** Got signal Terminated *** 2024-06-09T22:33:15.511 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:15 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3[51484]: 2024-06-09T22:33:15.011+0000 7f1f631f5700 -1 osd.3 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-09T22:33:17.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:16 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:16.590+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:18.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:17 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:17.621+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:18.510 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:18 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:18.066+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:18.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:18 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:18.192+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:19.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:18 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:18.627+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:19.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:19 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:19.102+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:19.261 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:18 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3[51484]: 2024-06-09T22:33:18.938+0000 7f1f5f7e9700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.128249+0000 front 2024-06-09T22:32:54.128277+0000 (oldest deadline 2024-06-09T22:33:18.828358+0000) 2024-06-09T22:33:19.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:19 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:19.045+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:19.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:19.241+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:20.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:19.652+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:20.260 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:19 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3[51484]: 2024-06-09T22:33:19.941+0000 7f1f5f7e9700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.128249+0000 front 2024-06-09T22:32:54.128277+0000 (oldest deadline 2024-06-09T22:33:18.828358+0000) 2024-06-09T22:33:20.261 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:20 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:20.073+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:20.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:20 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:20.094+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:20.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:20 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:20.086+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:20.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:20 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:20.194+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:20.823 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:20 smithi104 podman[122847]: 2024-06-09 22:33:20.51481976 +0000 UTC m=+5.635039697 container died 04dcef22b25662368deb849771d31f0b0f7aad9397f3e0cd051966e101474003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-06-09T22:33:21.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:20 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:20.668+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:21.119 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:21 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:21.069+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:21.119 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:21 smithi104 podman[122847]: 2024-06-09 22:33:21.01129259 +0000 UTC m=+6.131512532 container cleanup 04dcef22b25662368deb849771d31f0b0f7aad9397f3e0cd051966e101474003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-09T22:33:21.120 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:21 smithi104 bash[122847]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3 2024-06-09T22:33:21.383 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:21 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:21.116+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:21.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:21 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:21.105+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:21.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:21 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:21.172+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:21.760 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:21 smithi104 podman[122862]: 2024-06-09 22:33:21.380395554 +0000 UTC m=+0.864690714 container remove 04dcef22b25662368deb849771d31f0b0f7aad9397f3e0cd051966e101474003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-06-09T22:33:22.036 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:21 smithi104 podman[122950]: 2024-06-09 22:33:21.677180004 +0000 UTC m=+0.017050281 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:33:22.036 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:21 smithi104 podman[122950]: 2024-06-09 22:33:21.798002588 +0000 UTC m=+0.137872856 container create f7b8dc29f95a3ce17436700f05b523e06e3adcaca173c2cd47439ed618572c2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3-deactivate, RELEASE=squid-3d89acc, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-09T22:33:22.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:21 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:21.689+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:22.362 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:22 smithi104 podman[122950]: 2024-06-09 22:33:22.058675598 +0000 UTC m=+0.398545876 container init f7b8dc29f95a3ce17436700f05b523e06e3adcaca173c2cd47439ed618572c2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-06-09T22:33:22.362 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:22 smithi104 podman[122950]: 2024-06-09 22:33:22.06199339 +0000 UTC m=+0.401863669 container start f7b8dc29f95a3ce17436700f05b523e06e3adcaca173c2cd47439ed618572c2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc) 2024-06-09T22:33:22.362 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:22 smithi104 podman[122950]: 2024-06-09 22:33:22.151101975 +0000 UTC m=+0.490972245 container attach f7b8dc29f95a3ce17436700f05b523e06e3adcaca173c2cd47439ed618572c2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-09T22:33:22.363 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:22 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:22.033+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:22.363 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:22 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:22.128+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:22.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:22 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:22.116+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:22.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:22 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:22.134+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:22.634 INFO:journalctl@ceph.osd.3.smithi104.stdout:Jun 09 22:33:22 smithi104 podman[122950]: 2024-06-09 22:33:22.359530039 +0000 UTC m=+0.699400339 container died f7b8dc29f95a3ce17436700f05b523e06e3adcaca173c2cd47439ed618572c2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-3-deactivate, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc) 2024-06-09T22:33:23.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:22 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:22.729+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:23.081 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.3.service' 2024-06-09T22:33:23.119 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:33:23.120 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-09T22:33:23.120 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-09T22:33:23.120 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.4 2024-06-09T22:33:23.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:23 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:23.116+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:23.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:22 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:22.990+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:23.480 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:23 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:23.104+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:23.481 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:23 smithi153 systemd[1]: Stopping Ceph osd.4 for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:33:23.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:23 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:23.134+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:23.746 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:23 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4[48121]: 2024-06-09T22:33:23.476+0000 7f94a4ca6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:33:23.747 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:23 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4[48121]: 2024-06-09T22:33:23.476+0000 7f94a4ca6700 -1 osd.4 114 *** Got signal Terminated *** 2024-06-09T22:33:23.747 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:23 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4[48121]: 2024-06-09T22:33:23.476+0000 7f94a4ca6700 -1 osd.4 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-09T22:33:24.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:23 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:23.741+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:24.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:24 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:24.139+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:24.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:24 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:24.114+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:24.510 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:24 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:24.017+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:24.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:24 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:24.154+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:25.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:24 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:24.777+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:25.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:25 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:25.119+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:25.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:25 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:25.109+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:25.510 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:25 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:25.045+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:25.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:25.177+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:26.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:25 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:25.743+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:26.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:26 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:26.097+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:26.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:26 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:26.097+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:26.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:26 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:26.062+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:26.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:26 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:26.062+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:26.510 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:26 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:26.011+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:26.511 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:26 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:26.011+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:26.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:26 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:26.208+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:27.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:26 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:26.701+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:27.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:26 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:26.701+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.556189+0000 front 2024-06-09T22:33:00.556218+0000 (oldest deadline 2024-06-09T22:33:25.856217+0000) 2024-06-09T22:33:27.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:27 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:27.084+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:27.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:27 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:27.084+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:27.261 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:26 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:26.984+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:27.261 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:26 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:26.984+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:27.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:27 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:27.119+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:27.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:27 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:27.119+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:27.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:27 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:27.190+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:27.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:27 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:27.133+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:28.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:27 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:27.691+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:28.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:27 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:27.691+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.556189+0000 front 2024-06-09T22:33:00.556218+0000 (oldest deadline 2024-06-09T22:33:25.856217+0000) 2024-06-09T22:33:28.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:28 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:28.037+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:28.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:28 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:28.037+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:28.261 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:27 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:27.990+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:28.261 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:27 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:27.990+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:28.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:28 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:28.128+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:28.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:28 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:28.128+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:28.562 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:28 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:28.167+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:28.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:28 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:28.175+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:29.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:28 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:28.711+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:29.063 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:28 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:28.711+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.556189+0000 front 2024-06-09T22:33:00.556218+0000 (oldest deadline 2024-06-09T22:33:25.856217+0000) 2024-06-09T22:33:29.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:29 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:29.032+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:29.247 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:28 smithi153 podman[73858]: 2024-06-09 22:33:28.990637306 +0000 UTC m=+5.670713492 container died 26a600117843a8b2e0d61bed2f3f4443209e09ffc3f29bb1fd7ec31054646f08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-06-09T22:33:29.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:29 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:29.152+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:29.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:29 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:29.152+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:29.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:29 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:29.073+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:29.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:29 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:29.073+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:29.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:28 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:28.968+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:29.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:28 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:28.968+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:29.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:29 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:29.131+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:29.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:29 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:29.194+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:29.623 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:29 smithi153 podman[73858]: 2024-06-09 22:33:29.352739357 +0000 UTC m=+6.032815534 container cleanup 26a600117843a8b2e0d61bed2f3f4443209e09ffc3f29bb1fd7ec31054646f08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-09T22:33:29.623 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:29 smithi153 bash[73858]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4 2024-06-09T22:33:29.973 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:29 smithi153 podman[73875]: 2024-06-09 22:33:29.620638838 +0000 UTC m=+0.624912278 container remove 26a600117843a8b2e0d61bed2f3f4443209e09ffc3f29bb1fd7ec31054646f08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-09T22:33:30.063 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:29 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:29.724+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:30.063 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:29 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:29.724+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.556189+0000 front 2024-06-09T22:33:00.556218+0000 (oldest deadline 2024-06-09T22:33:25.856217+0000) 2024-06-09T22:33:30.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:29 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:29.993+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:30.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:30 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:30.108+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:30.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:30 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:30.108+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:30.248 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:29 smithi153 podman[73965]: 2024-06-09 22:33:29.874041695 +0000 UTC m=+0.017885717 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:33:30.248 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:30 smithi153 podman[73965]: 2024-06-09 22:33:30.032207931 +0000 UTC m=+0.176051941 container create 05716d29db3adace27329b289f9f56be823cb4d495f37a2698d752e03fcefb58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, io.buildah.version=1.33.5) 2024-06-09T22:33:30.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:29 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:29.961+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:30.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:29 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:29.961+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:30.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:30 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:30.120+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:30.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:30 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:30.120+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:30.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:30 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:30.105+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:30.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:30 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:30.222+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:30.597 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:30 smithi153 podman[73965]: 2024-06-09 22:33:30.294056734 +0000 UTC m=+0.437900752 container init 05716d29db3adace27329b289f9f56be823cb4d495f37a2698d752e03fcefb58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-06-09T22:33:30.597 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:30 smithi153 podman[73965]: 2024-06-09 22:33:30.297532537 +0000 UTC m=+0.441376558 container start 05716d29db3adace27329b289f9f56be823cb4d495f37a2698d752e03fcefb58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc) 2024-06-09T22:33:30.598 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:30 smithi153 podman[73965]: 2024-06-09 22:33:30.390818435 +0000 UTC m=+0.534662452 container attach 05716d29db3adace27329b289f9f56be823cb4d495f37a2698d752e03fcefb58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-09T22:33:30.879 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:30 smithi153 podman[73965]: 2024-06-09 22:33:30.595021538 +0000 UTC m=+0.738865563 container died 05716d29db3adace27329b289f9f56be823cb4d495f37a2698d752e03fcefb58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD) 2024-06-09T22:33:31.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:30 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:30.692+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:31.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:30 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:30.692+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.556189+0000 front 2024-06-09T22:33:00.556218+0000 (oldest deadline 2024-06-09T22:33:25.856217+0000) 2024-06-09T22:33:31.137 INFO:journalctl@ceph.osd.4.smithi153.stdout:Jun 09 22:33:31 smithi153 podman[73992]: 2024-06-09 22:33:31.11955392 +0000 UTC m=+0.519021508 container remove 05716d29db3adace27329b289f9f56be823cb4d495f37a2698d752e03fcefb58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-06-09T22:33:31.137 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:30 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:30.956+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:31.138 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:31 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:31.112+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:31.138 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:31 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:31.112+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:31.142 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.4.service' 2024-06-09T22:33:31.179 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:33:31.180 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-09T22:33:31.180 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-09T22:33:31.180 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.5 2024-06-09T22:33:31.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:30 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:30.927+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:31.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:30 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:30.927+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:31.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:31 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:31.147+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:31.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:31 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:31.147+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:31.506 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:31 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:31.198+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:31.507 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:31 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:31.122+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:31.507 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:31 smithi194 systemd[1]: Stopping Ceph osd.5 for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:33:31.812 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:31 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:31.502+0000 7f4a9db06700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:33:31.812 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:31 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:31.502+0000 7f4a9db06700 -1 osd.5 114 *** Got signal Terminated *** 2024-06-09T22:33:31.813 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:31 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:31.502+0000 7f4a9db06700 -1 osd.5 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-09T22:33:31.813 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:31 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:31.662+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:31.813 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:31 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:31.662+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.556189+0000 front 2024-06-09T22:33:00.556218+0000 (oldest deadline 2024-06-09T22:33:25.856217+0000) 2024-06-09T22:33:32.246 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:31 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:31.963+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:32.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:32 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:32.096+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:32.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:32 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:32.096+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:32.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:31 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:31.958+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:32.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:31 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:31.958+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:32.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:32 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:32.123+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:32.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:32 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:32.123+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:32.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:32 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:32.111+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:32.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:32 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:32.199+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:33.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:32 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:32.620+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:33.063 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:32 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:32.620+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.556189+0000 front 2024-06-09T22:33:00.556218+0000 (oldest deadline 2024-06-09T22:33:25.856217+0000) 2024-06-09T22:33:33.246 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:32 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:32.970+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:33.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:33 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:33.129+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:33.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:33 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:33.129+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:33.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:32 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:32.974+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:33.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:32 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:32.974+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:33.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:33 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:33.137+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:33.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:33 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:33.137+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:33.562 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:33 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:33.156+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:33.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:33 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:33.121+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:34.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:33 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:33.647+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:34.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:33 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:33.647+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.556189+0000 front 2024-06-09T22:33:00.556218+0000 (oldest deadline 2024-06-09T22:33:25.856217+0000) 2024-06-09T22:33:34.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:33 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:33.950+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:34.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:33 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:33.950+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:34.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:34 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:34.138+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:34.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:34 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:34.138+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:34.496 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:33 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:33.993+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:34.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:34 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:34.099+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:34.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:34 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:34.099+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:34.562 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:34 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:34.190+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:34.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:34 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:34.168+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:34.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:34 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:34.168+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:35.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:34 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:34.692+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:35.063 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:34 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:34.692+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.556189+0000 front 2024-06-09T22:33:00.556218+0000 (oldest deadline 2024-06-09T22:33:25.856217+0000) 2024-06-09T22:33:35.246 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:34 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:34.956+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:35.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:35 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:35.084+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:35.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:35 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:35.084+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:35.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:34 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:34.967+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:35.261 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:34 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:34.967+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:35.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:35 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:35.127+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:35.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:35 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:35.127+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:35.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:35 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:35.127+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:35.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:35 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:35.192+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:35.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:35 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:35.192+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:35.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:35 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:35.148+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:35.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:35 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:35.148+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:36.062 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:35 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:35.688+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.354907+0000 front 2024-06-09T22:32:55.354946+0000 (oldest deadline 2024-06-09T22:33:16.455027+0000) 2024-06-09T22:33:36.063 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:35 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5[48458]: 2024-06-09T22:33:35.688+0000 7f4a960f2700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.556189+0000 front 2024-06-09T22:33:00.556218+0000 (oldest deadline 2024-06-09T22:33:25.856217+0000) 2024-06-09T22:33:36.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:35 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:35.967+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:36.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:35 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:35.967+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:36.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:36 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:36.101+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:36.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:36 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:36.101+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:36.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:36 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:36.101+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:36.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:35 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:35.970+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:36.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:35 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:35.970+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:36.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:36 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:36.098+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:36.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:36 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:36.098+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:36.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:36 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:36.098+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:36.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:36 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:36.116+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:36.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:36 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:36.116+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:36.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:36 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:36.188+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:36.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:36 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:36.188+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:37.236 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:36 smithi194 podman[74448]: 2024-06-09 22:33:36.904754358 +0000 UTC m=+5.536317337 container died 44790d2272978ec3255eecabcfd820468507ad73e5cdbd3d31401684c053a547 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-3d89acc, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-06-09T22:33:37.237 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:37 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:37.115+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:37.237 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:37 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:37.115+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:37.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:36 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:36.960+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:37.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:36 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:36.960+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:37.260 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:36 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:36.960+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.337171+0000 front 2024-06-09T22:33:11.337139+0000 (oldest deadline 2024-06-09T22:33:36.036968+0000) 2024-06-09T22:33:37.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:37 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:37.115+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:37.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:37 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:37.115+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:37.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:37 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:37.115+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:37.496 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:37 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:37.002+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:37.497 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:37 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:37.002+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:37.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:37 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:37.109+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:37.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:37 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:37.109+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:37.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:37 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:37.109+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:37.562 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:37 smithi194 podman[74448]: 2024-06-09 22:33:37.326905625 +0000 UTC m=+5.958468593 container cleanup 44790d2272978ec3255eecabcfd820468507ad73e5cdbd3d31401684c053a547 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:33:37.563 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:37 smithi194 bash[74448]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5 2024-06-09T22:33:37.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:37 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:37.233+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:37.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:37 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:37.233+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:38.003 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:37 smithi194 podman[74464]: 2024-06-09 22:33:37.578271123 +0000 UTC m=+0.670571877 container remove 44790d2272978ec3255eecabcfd820468507ad73e5cdbd3d31401684c053a547 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-09T22:33:38.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:37 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:37.968+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:38.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:37 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:37.968+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:38.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:38 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:38.084+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:38.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:38 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:38.084+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:38.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:38 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:38.084+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:38.259 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:38 smithi194 podman[74551]: 2024-06-09 22:33:37.903061498 +0000 UTC m=+0.017250286 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:33:38.259 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:38 smithi194 podman[74551]: 2024-06-09 22:33:38.012390925 +0000 UTC m=+0.126579728 container create 7a0c8ed49abc8edbf3d6ce09fe2148dd0b492d3728cce66625ec628981de805d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5-deactivate, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-06-09T22:33:38.259 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:38 smithi194 podman[74551]: 2024-06-09 22:33:38.256532709 +0000 UTC m=+0.370721488 container init 7a0c8ed49abc8edbf3d6ce09fe2148dd0b492d3728cce66625ec628981de805d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:33:38.260 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:38 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:38.145+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:38.260 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:38 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:38.145+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:38.260 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:38 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:38.238+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:38.261 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:38 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:38.238+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:38.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:38 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:38.146+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:38.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:38 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:38.146+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:38.261 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:38 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:38.146+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:38.262 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:37 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:37.921+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:38.262 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:37 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:37.921+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:38.262 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:37 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:37.921+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.337171+0000 front 2024-06-09T22:33:11.337139+0000 (oldest deadline 2024-06-09T22:33:36.036968+0000) 2024-06-09T22:33:38.551 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:38 smithi194 podman[74551]: 2024-06-09 22:33:38.259834702 +0000 UTC m=+0.374023486 container start 7a0c8ed49abc8edbf3d6ce09fe2148dd0b492d3728cce66625ec628981de805d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-06-09T22:33:38.552 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:38 smithi194 podman[74551]: 2024-06-09 22:33:38.364855503 +0000 UTC m=+0.479044289 container attach 7a0c8ed49abc8edbf3d6ce09fe2148dd0b492d3728cce66625ec628981de805d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-06-09T22:33:38.813 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jun 09 22:33:38 smithi194 podman[74551]: 2024-06-09 22:33:38.549035782 +0000 UTC m=+0.663224563 container died 7a0c8ed49abc8edbf3d6ce09fe2148dd0b492d3728cce66625ec628981de805d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-06-09T22:33:39.162 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:38 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:38.874+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:39.162 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:38 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:38.874+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:39.162 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:38 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:38.874+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.337171+0000 front 2024-06-09T22:33:11.337139+0000 (oldest deadline 2024-06-09T22:33:36.036968+0000) 2024-06-09T22:33:39.237 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.5.service' 2024-06-09T22:33:39.246 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:38 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:38.982+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:39.246 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:38 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:38.982+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:39.250 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:39 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:39.133+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:39.250 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:39 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:39.133+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:39.250 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:39 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:39.133+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:39.275 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:33:39.275 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-09T22:33:39.275 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-06-09T22:33:39.275 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.6 2024-06-09T22:33:39.510 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:39 smithi104 systemd[1]: Stopping Ceph osd.6 for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:33:39.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:39 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:39.158+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:39.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:39 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:39.158+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:39.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:39 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:39.158+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:39.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:39 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:39.201+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:39.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:39 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:39.201+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:39.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:39 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:39.159+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:39.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:39 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:39.159+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:39.873 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:39 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:39.590+0000 7f81089d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:33:39.874 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:39 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:39.590+0000 7f81089d6700 -1 osd.6 114 *** Got signal Terminated *** 2024-06-09T22:33:39.874 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:39 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:39.590+0000 7f81089d6700 -1 osd.6 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-09T22:33:39.874 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:39 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:39.870+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:39.874 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:39 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:39.870+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:39.874 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:39 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:39.870+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.337171+0000 front 2024-06-09T22:33:11.337139+0000 (oldest deadline 2024-06-09T22:33:36.036968+0000) 2024-06-09T22:33:40.246 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:40 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:40.119+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:40.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:40 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:40.119+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:40.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:40 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:40.119+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:40.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:39 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:39.954+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:40.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:39 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:39.954+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:40.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:40 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:40.189+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:40.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:40 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:40.189+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:40.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:40 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:40.189+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:40.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:40 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:40.243+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:40.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:40 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:40.243+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:40.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:40 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:40.127+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:40.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:40 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:40.127+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:41.220 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:40 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:40.900+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:41.220 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:40 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:40.900+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:41.220 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:40 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:40.900+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.337171+0000 front 2024-06-09T22:33:11.337139+0000 (oldest deadline 2024-06-09T22:33:36.036968+0000) 2024-06-09T22:33:41.246 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:40 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:40.929+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:41.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:40 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:40.929+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:41.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:41 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:41.091+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:41.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:41 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:41.091+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:41.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:41 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:41.091+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:41.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:41 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:41.217+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:41.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:41 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:41.217+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:41.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:41 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:41.217+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:41.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:41 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:41.257+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:41.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:41 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:41.257+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:41.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:41 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:41.128+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:41.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:41 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:41.128+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:41.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:41 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:41.128+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:42.206 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:41 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:41.882+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:42.206 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:41 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:41.882+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:42.206 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:41 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:41.882+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.337171+0000 front 2024-06-09T22:33:11.337139+0000 (oldest deadline 2024-06-09T22:33:36.036968+0000) 2024-06-09T22:33:42.246 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:41 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:41.973+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:42.246 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:41 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:41.973+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:42.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:42.140+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:42.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:42.140+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:42.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:42.140+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:42.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:42 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:42.203+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:42.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:42 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:42.203+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:42.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:42 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:42.203+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:42.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:42 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:42.305+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:42.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:42 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:42.305+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:42.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:42 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:42.305+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:42.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:42 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:42.143+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:42.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:42 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:42.143+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:42.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:42 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:42.143+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:43.205 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:42 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:42.867+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:43.206 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:42 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:42.867+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:43.206 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:42 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:42.867+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.337171+0000 front 2024-06-09T22:33:11.337139+0000 (oldest deadline 2024-06-09T22:33:36.036968+0000) 2024-06-09T22:33:43.246 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:42.971+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:43.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:42.971+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:43.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:42.971+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.652082+0000 front 2024-06-09T22:33:18.652034+0000 (oldest deadline 2024-06-09T22:33:42.751854+0000) 2024-06-09T22:33:43.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:43 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:43.153+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:43.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:43 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:43.153+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:43.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:43 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:43.153+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:43.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:43 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:43.153+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:43.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:43 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:43.203+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:43.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:43 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:43.203+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:43.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:43 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:43.203+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:43.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:43 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:43.203+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:43.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:43 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:43.279+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:43.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:43 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:43.279+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:43.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:43 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:43.279+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:43.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:43 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:43.122+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:43.564 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:43 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:43.122+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:43.564 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:43 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:43.122+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:44.237 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:43 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:43.912+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:53.233733+0000 front 2024-06-09T22:32:53.233758+0000 (oldest deadline 2024-06-09T22:33:17.933874+0000) 2024-06-09T22:33:44.238 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:43 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:43.912+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.634778+0000 front 2024-06-09T22:32:59.634806+0000 (oldest deadline 2024-06-09T22:33:25.534973+0000) 2024-06-09T22:33:44.238 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:43 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6[58010]: 2024-06-09T22:33:43.912+0000 7f8100fc2700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.337171+0000 front 2024-06-09T22:33:11.337139+0000 (oldest deadline 2024-06-09T22:33:36.036968+0000) 2024-06-09T22:33:44.496 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:44 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:44.008+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:44.497 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:44 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:44.008+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:44.497 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:44 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:44.008+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.652082+0000 front 2024-06-09T22:33:18.652034+0000 (oldest deadline 2024-06-09T22:33:42.751854+0000) 2024-06-09T22:33:44.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:44 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:44.144+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:44.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:44 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:44.144+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:44.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:44 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:44.144+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:44.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:44 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:44.144+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:44.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:44 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:44.235+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:44.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:44 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:44.235+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:44.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:44 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:44.235+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:44.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:44 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:44.235+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:44.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:44 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:44.265+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:44.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:44 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:44.265+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:44.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:44 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:44.265+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:44.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:44 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:44.086+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:44.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:44 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:44.086+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:44.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:44 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:44.086+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:45.303 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:45 smithi104 podman[123065]: 2024-06-09 22:33:45.027578506 +0000 UTC m=+5.556678639 container died da309d07233963d59e1f063e8769434fb587b84251e10c623014aeb88d620283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_CLEAN=True, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-09T22:33:45.303 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:45 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:45.197+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:45.303 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:45 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:45.197+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:45.303 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:45 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:45.197+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:45.304 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:45 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:45.197+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:45.464 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:45 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:45.115+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:45.464 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:45 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:45.115+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:45.464 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:45 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:45.115+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:45.465 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:45 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:45.221+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:45.465 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:45 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:45.221+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:45.465 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:45 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:45.221+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:45.496 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:45 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:45.020+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:45.496 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:45 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:45.020+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:45.497 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:45 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:45.020+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.652082+0000 front 2024-06-09T22:33:18.652034+0000 (oldest deadline 2024-06-09T22:33:42.751854+0000) 2024-06-09T22:33:45.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:45 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:45.099+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:45.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:45 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:45.099+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:45.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:45 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:45.099+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:45.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:45 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:45.099+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:45.760 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:45 smithi104 podman[123065]: 2024-06-09 22:33:45.499291824 +0000 UTC m=+6.028391942 container cleanup da309d07233963d59e1f063e8769434fb587b84251e10c623014aeb88d620283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, ceph=True) 2024-06-09T22:33:45.761 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:45 smithi104 bash[123065]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6 2024-06-09T22:33:46.129 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:45 smithi104 podman[123080]: 2024-06-09 22:33:45.77603043 +0000 UTC m=+0.745313748 container remove da309d07233963d59e1f063e8769434fb587b84251e10c623014aeb88d620283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-06-09T22:33:46.399 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:46 smithi104 podman[123169]: 2024-06-09 22:33:46.126914705 +0000 UTC m=+0.113627722 container create 9407fc79ca2f139a49a7471d796ee77b07ee9591593e716e25cd6aaf47e125e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-06-09T22:33:46.399 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:46 smithi104 podman[123169]: 2024-06-09 22:33:46.030544908 +0000 UTC m=+0.017257927 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:33:46.400 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:46 smithi104 podman[123169]: 2024-06-09 22:33:46.396554361 +0000 UTC m=+0.383267383 container init 9407fc79ca2f139a49a7471d796ee77b07ee9591593e716e25cd6aaf47e125e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6-deactivate, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-06-09T22:33:46.400 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:46 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:46.168+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:46.400 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:46 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:46.168+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:46.400 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:46 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:46.168+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:46.400 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:46 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:46.168+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:46.496 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:46 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:46.022+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:46.497 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:46 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:46.022+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:46.497 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:46 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:46.022+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.652082+0000 front 2024-06-09T22:33:18.652034+0000 (oldest deadline 2024-06-09T22:33:42.751854+0000) 2024-06-09T22:33:46.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:46 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:46.114+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:46.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:46 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:46.114+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:46.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:46 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:46.114+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:46.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:46 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:46.114+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:46.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:46 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:46.224+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:46.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:46 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:46.224+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:46.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:46 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:46.224+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:46.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:46 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:46.079+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:46.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:46 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:46.079+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:46.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:46 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:46.079+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:46.696 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:46 smithi104 podman[123169]: 2024-06-09 22:33:46.401163728 +0000 UTC m=+0.387876742 container start 9407fc79ca2f139a49a7471d796ee77b07ee9591593e716e25cd6aaf47e125e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-06-09T22:33:46.696 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:46 smithi104 podman[123169]: 2024-06-09 22:33:46.496544399 +0000 UTC m=+0.483257438 container attach 9407fc79ca2f139a49a7471d796ee77b07ee9591593e716e25cd6aaf47e125e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-09T22:33:47.010 INFO:journalctl@ceph.osd.6.smithi104.stdout:Jun 09 22:33:46 smithi104 podman[123169]: 2024-06-09 22:33:46.693932122 +0000 UTC m=+0.680645137 container died 9407fc79ca2f139a49a7471d796ee77b07ee9591593e716e25cd6aaf47e125e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-09T22:33:47.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:46 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:46.980+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:47.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:46 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:46.980+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:47.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:46 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:46.980+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.652082+0000 front 2024-06-09T22:33:18.652034+0000 (oldest deadline 2024-06-09T22:33:42.751854+0000) 2024-06-09T22:33:47.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:47 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:47.071+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:47.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:47 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:47.071+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:47.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:47 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:47.071+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:47.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:47 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:47.071+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:47.311 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:47 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:47.217+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:47.311 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:47 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:47.217+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:47.311 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:47 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:47.217+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:47.312 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:47 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:47.217+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:47.336 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.6.service' 2024-06-09T22:33:47.384 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:33:47.384 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-06-09T22:33:47.385 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-06-09T22:33:47.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.7 2024-06-09T22:33:47.562 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:47 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:47.082+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:47.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:47 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:47.082+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:47.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:47 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:47.082+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:47.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:47 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:47.200+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:47.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:47 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:47.200+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:47.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:47 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:47.200+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:47.676 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:47 smithi153 systemd[1]: Stopping Ceph osd.7 for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:33:47.978 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:47 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:47.713+0000 7fced6811700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:33:47.978 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:47 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:47.713+0000 7fced6811700 -1 osd.7 114 *** Got signal Terminated *** 2024-06-09T22:33:47.978 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:47 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:47.713+0000 7fced6811700 -1 osd.7 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-09T22:33:47.978 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:47 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:47.974+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:47.978 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:47 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:47.974+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:47.978 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:47 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:47.974+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.652082+0000 front 2024-06-09T22:33:18.652034+0000 (oldest deadline 2024-06-09T22:33:42.751854+0000) 2024-06-09T22:33:48.246 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:48 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:48.078+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:48.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:48 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:48.078+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:48.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:48 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:48.078+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:48.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:48 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:48.078+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:48.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:48 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:48.241+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:48.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:48 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:48.241+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:48.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:48 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:48.241+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:48.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:48 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:48.241+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:48.562 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:48 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:48.093+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:48.562 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:48 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:48.093+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:48.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:48 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:48.093+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:48.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:48 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:48.237+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:48.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:48 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:48.237+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:48.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:48 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:48.237+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:49.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:48 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:48.957+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:49.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:48 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:48.957+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:49.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:48 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:48.957+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.652082+0000 front 2024-06-09T22:33:18.652034+0000 (oldest deadline 2024-06-09T22:33:42.751854+0000) 2024-06-09T22:33:49.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:49 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:49.106+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:49.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:49 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:49.106+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:49.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:49 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:49.106+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:49.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:49 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:49.106+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:49.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:49 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:49.102+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:49.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:49 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:49.102+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:49.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:49 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:49.103+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:49.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:49 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:49.201+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:49.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:49 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:49.201+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:49.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:49 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:49.201+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:49.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:49 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:49.287+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:49.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:49 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:49.287+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:49.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:49 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:49.287+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:49.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:49 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:49.287+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:50.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:49 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:49.919+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:50.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:49 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:49.919+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:50.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:49 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:49.919+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.652082+0000 front 2024-06-09T22:33:18.652034+0000 (oldest deadline 2024-06-09T22:33:42.751854+0000) 2024-06-09T22:33:50.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:49 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:49.919+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.452669+0000 front 2024-06-09T22:33:27.452698+0000 (oldest deadline 2024-06-09T22:33:49.752846+0000) 2024-06-09T22:33:50.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:50.133+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:50.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:50.133+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:50.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:50.133+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:50.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:50.133+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:50.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:50.119+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:50.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:50.119+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:50.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:50.119+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:50.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:50.243+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:50.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:50.243+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:50.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:50.243+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:50.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:50 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:50.243+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:33:50.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:50.264+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:50.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:50.264+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:50.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:50.264+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:50.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:50.264+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:50.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:50 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:50.264+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:33:51.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:50.925+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:51.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:50.925+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:51.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:50.925+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.652082+0000 front 2024-06-09T22:33:18.652034+0000 (oldest deadline 2024-06-09T22:33:42.751854+0000) 2024-06-09T22:33:51.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:50.925+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.452669+0000 front 2024-06-09T22:33:27.452698+0000 (oldest deadline 2024-06-09T22:33:49.752846+0000) 2024-06-09T22:33:51.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:51 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:51.122+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:51.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:51 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:51.122+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:51.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:51 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:51.122+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:51.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:51 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:51.122+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:51.562 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:51 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:51.140+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:51.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:51 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:51.140+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:51.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:51 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:51.140+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:51.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:51 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:51.241+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:51.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:51 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:51.241+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:51.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:51 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:51.241+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:51.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:51 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:51.241+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:33:51.662 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:51 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:51.262+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:51.662 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:51 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:51.262+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:51.662 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:51 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:51.262+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:51.662 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:51 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:51.262+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:51.663 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:51 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:51.262+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:33:52.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:51 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:51.949+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:03.548767+0000 front 2024-06-09T22:33:03.548823+0000 (oldest deadline 2024-06-09T22:33:28.848955+0000) 2024-06-09T22:33:52.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:51 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:51.949+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.650638+0000 front 2024-06-09T22:33:11.650758+0000 (oldest deadline 2024-06-09T22:33:35.750788+0000) 2024-06-09T22:33:52.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:51 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:51.949+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.652082+0000 front 2024-06-09T22:33:18.652034+0000 (oldest deadline 2024-06-09T22:33:42.751854+0000) 2024-06-09T22:33:52.247 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:51 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7[54626]: 2024-06-09T22:33:51.949+0000 7fcecedfd700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.452669+0000 front 2024-06-09T22:33:27.452698+0000 (oldest deadline 2024-06-09T22:33:49.752846+0000) 2024-06-09T22:33:52.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:52 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:52.133+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:52.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:52 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:52.133+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:52.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:52 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:52.133+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:52.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:52 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:52.133+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:52.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:52 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:52.123+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:52.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:52 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:52.123+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:52.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:52 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:52.123+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:52.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:52 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:52.274+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:52.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:52 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:52.274+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:52.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:52 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:52.274+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:52.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:52 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:52.274+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:33:52.762 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:52.286+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:52.762 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:52.286+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:52.762 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:52.286+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:52.762 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:52.286+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:52.762 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:52 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:52.286+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:33:53.450 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:53 smithi153 podman[74071]: 2024-06-09 22:33:53.200287975 +0000 UTC m=+5.613203368 container died 85308217697b8ef5db440f9fe78e86181e7080de2109f7063401ae0e79c756f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:33:53.451 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:53 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:53.119+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:53.451 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:53 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:53.119+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:53.451 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:53 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:53.119+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:53.451 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:53 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:53.119+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:53.451 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:53 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:53.119+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:33:53.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:53 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:53.173+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:53.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:53 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:53.173+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:53.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:53 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:53.173+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:53.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:53 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:53.273+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:53.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:53 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:53.273+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:53.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:53 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:53.273+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:53.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:53 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:53.273+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:33:53.747 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:53 smithi153 podman[74071]: 2024-06-09 22:33:53.529588836 +0000 UTC m=+5.942504221 container cleanup 85308217697b8ef5db440f9fe78e86181e7080de2109f7063401ae0e79c756f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-06-09T22:33:53.747 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:53 smithi153 bash[74071]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7 2024-06-09T22:33:53.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:53 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:53.284+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:53.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:53 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:53.284+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:53.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:53 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:53.284+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:53.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:53 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:53.284+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:53.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:53 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:53.284+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:33:54.102 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:53 smithi153 podman[74097]: 2024-06-09 22:33:53.755888424 +0000 UTC m=+0.557883920 container remove 85308217697b8ef5db440f9fe78e86181e7080de2109f7063401ae0e79c756f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528) 2024-06-09T22:33:54.454 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:54 smithi153 podman[74188]: 2024-06-09 22:33:54.061899599 +0000 UTC m=+0.016843294 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:33:54.454 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:54 smithi153 podman[74188]: 2024-06-09 22:33:54.190593469 +0000 UTC m=+0.145537196 container create 1db613f18f37551bf2ac50f42ba4bdd2055eadce727b4b451b185940021fc444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2) 2024-06-09T22:33:54.454 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:54 smithi153 podman[74188]: 2024-06-09 22:33:54.451163587 +0000 UTC m=+0.406107266 container init 1db613f18f37551bf2ac50f42ba4bdd2055eadce727b4b451b185940021fc444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:33:54.455 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:54 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:54.099+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:54.455 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:54 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:54.099+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:54.455 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:54 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:54.099+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:54.455 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:54 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:54.099+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:54.455 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:54 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:54.099+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:33:54.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:54 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:54.201+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:54.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:54 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:54.201+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:54.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:54 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:54.201+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:54.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:54 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:54.201+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:22.624121+0000 front 2024-06-09T22:33:22.624108+0000 (oldest deadline 2024-06-09T22:33:53.224598+0000) 2024-06-09T22:33:54.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:54 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:54.303+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:54.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:54 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:54.303+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:54.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:54 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:54.303+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:54.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:54 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:54.303+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:33:54.743 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:54 smithi153 podman[74188]: 2024-06-09 22:33:54.454550242 +0000 UTC m=+0.409493933 container start 1db613f18f37551bf2ac50f42ba4bdd2055eadce727b4b451b185940021fc444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-06-09T22:33:54.743 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:54 smithi153 podman[74188]: 2024-06-09 22:33:54.576516891 +0000 UTC m=+0.531460582 container attach 1db613f18f37551bf2ac50f42ba4bdd2055eadce727b4b451b185940021fc444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528) 2024-06-09T22:33:54.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:54.296+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:54.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:54.296+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:54.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:54.296+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:54.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:54.296+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:54.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:54 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:54.296+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:33:54.996 INFO:journalctl@ceph.osd.7.smithi153.stdout:Jun 09 22:33:54 smithi153 podman[74188]: 2024-06-09 22:33:54.740674692 +0000 UTC m=+0.695618379 container died 1db613f18f37551bf2ac50f42ba4bdd2055eadce727b4b451b185940021fc444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-06-09T22:33:55.399 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:55 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:55.097+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:55.400 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:55 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:55.097+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:55.400 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:55 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:55.097+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:55.400 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:55 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:55.097+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:55.400 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:55 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:55.097+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:33:55.419 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.7.service' 2024-06-09T22:33:55.461 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:33:55.462 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-06-09T22:33:55.462 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-06-09T22:33:55.462 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.8 2024-06-09T22:33:55.489 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:55.159+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:55.489 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:55.159+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:55.490 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:55.159+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:55.490 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:55.159+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:22.624121+0000 front 2024-06-09T22:33:22.624108+0000 (oldest deadline 2024-06-09T22:33:53.224598+0000) 2024-06-09T22:33:55.490 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:55.291+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:55.490 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:55.291+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:55.490 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:55.291+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:55.490 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:55.291+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:33:55.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:55 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:55.252+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:55.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:55 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:55.252+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:55.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:55 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:55.252+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:55.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:55 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:55.252+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:55.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:55 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:55.252+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:33:55.774 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:55 smithi194 systemd[1]: Stopping Ceph osd.8 for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:33:56.062 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:55.770+0000 7fa9df8ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:33:56.063 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:55.770+0000 7fa9df8ec700 -1 osd.8 114 *** Got signal Terminated *** 2024-06-09T22:33:56.063 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:55 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:55.770+0000 7fa9df8ec700 -1 osd.8 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-09T22:33:56.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:56 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:56.076+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:56.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:56 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:56.076+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:56.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:56 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:56.076+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:56.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:56 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:56.076+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:56.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:56 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:56.076+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:33:56.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:56.243+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:56.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:56.243+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:56.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:56.243+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:56.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:56.243+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:56.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:56 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:56.243+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:33:56.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:56 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:56.124+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:56.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:56 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:56.124+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:56.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:56 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:56.124+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:56.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:56 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:56.124+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:22.624121+0000 front 2024-06-09T22:33:22.624108+0000 (oldest deadline 2024-06-09T22:33:53.224598+0000) 2024-06-09T22:33:56.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:56 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:56.277+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:56.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:56 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:56.277+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:56.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:56 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:56.277+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:56.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:56 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:56.277+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:33:57.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:57 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:57.105+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:57.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:57 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:57.105+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:57.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:57 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:57.105+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:57.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:57 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:57.105+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:57.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:57 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:57.105+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:33:57.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:57.204+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:57.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:57.204+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:57.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:57.204+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:57.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:57.204+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:57.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:57 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:57.204+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:33:57.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:57 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:57.282+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:57.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:57 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:57.282+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:57.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:57 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:57.282+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:57.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:57 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:57.282+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:33:57.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:57 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:57.127+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:57.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:57 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:57.127+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:57.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:57 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:57.127+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:57.564 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:57 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:57.127+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:22.624121+0000 front 2024-06-09T22:33:22.624108+0000 (oldest deadline 2024-06-09T22:33:53.224598+0000) 2024-06-09T22:33:58.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:58 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:58.131+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:58.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:58 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:58.131+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:58.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:58 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:58.131+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:58.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:58 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:58.131+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:58.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:58 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:58.131+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:33:58.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:58.222+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:58.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:58.222+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:58.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:58.222+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:58.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:58.222+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:58.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:58 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:58.222+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:33:58.562 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:58 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:58.086+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:58.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:58 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:58.086+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:58.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:58 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:58.086+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:58.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:58 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:58.086+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:22.624121+0000 front 2024-06-09T22:33:22.624108+0000 (oldest deadline 2024-06-09T22:33:53.224598+0000) 2024-06-09T22:33:58.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:58 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:58.270+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:58.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:58 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:58.270+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:58.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:58 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:58.270+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:58.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:58 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:58.270+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:33:59.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:59 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:59.127+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:33:59.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:59 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:59.127+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:33:59.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:59 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:59.127+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:33:59.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:59 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:59.127+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:33:59.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:59 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:59.127+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:33:59.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:33:59 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:33:59.127+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:33:59.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:59 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:59.217+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:33:59.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:59 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:59.217+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:33:59.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:59 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:59.217+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:33:59.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:59 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:59.217+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:33:59.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:33:59 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:33:59.217+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:33:59.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:59 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:59.122+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:33:59.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:59 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:59.123+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:33:59.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:59 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:59.123+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:33:59.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:59 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:59.123+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:22.624121+0000 front 2024-06-09T22:33:22.624108+0000 (oldest deadline 2024-06-09T22:33:53.224598+0000) 2024-06-09T22:33:59.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:33:59 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:33:59.123+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:36.025969+0000 front 2024-06-09T22:33:36.026015+0000 (oldest deadline 2024-06-09T22:33:58.925996+0000) 2024-06-09T22:33:59.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:59 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:59.293+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:33:59.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:59 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:59.293+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:33:59.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:59 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:59.293+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:33:59.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:59 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:59.293+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:33:59.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:33:59 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:33:59.293+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:00.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:00 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:00.101+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:00.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:00 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:00.101+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:00.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:00 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:00.101+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:00.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:00 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:00.101+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:00.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:00 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:00.101+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:00.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:00 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:00.101+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:00.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:00.252+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:34:00.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:00.252+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:34:00.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:00.252+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:34:00.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:00.252+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:34:00.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:00.252+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:34:00.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:00 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:00.252+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:36.363231+0000 front 2024-06-09T22:33:36.363212+0000 (oldest deadline 2024-06-09T22:33:59.263163+0000) 2024-06-09T22:34:00.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:00 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:34:00.118+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:00.520393+0000 front 2024-06-09T22:33:00.520308+0000 (oldest deadline 2024-06-09T22:33:26.420381+0000) 2024-06-09T22:34:00.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:00 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:34:00.118+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.721458+0000 front 2024-06-09T22:33:11.721485+0000 (oldest deadline 2024-06-09T22:33:34.621608+0000) 2024-06-09T22:34:00.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:00 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:34:00.118+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.823194+0000 front 2024-06-09T22:33:19.823157+0000 (oldest deadline 2024-06-09T22:33:40.923066+0000) 2024-06-09T22:34:00.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:00 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:34:00.118+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:22.624121+0000 front 2024-06-09T22:33:22.624108+0000 (oldest deadline 2024-06-09T22:33:53.224598+0000) 2024-06-09T22:34:00.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:00 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8[55435]: 2024-06-09T22:34:00.118+0000 7fa9d7ed8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:36.025969+0000 front 2024-06-09T22:33:36.026015+0000 (oldest deadline 2024-06-09T22:33:58.925996+0000) 2024-06-09T22:34:00.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:00 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:00.247+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:00.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:00 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:00.247+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:00.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:00 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:00.247+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:00.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:00 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:00.247+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:00.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:00 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:00.247+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:01.467 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:01 smithi194 podman[74669]: 2024-06-09 22:34:01.199089242 +0000 UTC m=+5.544637171 container died 42996772a59ebab7c4530eca75fe3b45ad8bc8f182b6e4e57a2bc1f779e85b75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, GIT_CLEAN=True, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:34:01.467 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:01 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:01.250+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:01.467 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:01 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:01.250+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:01.467 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:01 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:01.250+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:01.468 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:01 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:01.250+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:01.468 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:01 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:01.250+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:01.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:01 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:01.126+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:01.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:01 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:01.126+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:01.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:01 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:01.126+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:01.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:01 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:01.126+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:01.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:01 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:01.126+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:01.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:01 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:01.126+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:01.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:01 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:01.227+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:34:01.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:01 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:01.227+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:34:01.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:01 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:01.227+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:34:01.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:01 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:01.227+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:34:01.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:01 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:01.227+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:34:01.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:01 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:01.227+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:36.363231+0000 front 2024-06-09T22:33:36.363212+0000 (oldest deadline 2024-06-09T22:33:59.263163+0000) 2024-06-09T22:34:01.813 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:01 smithi194 podman[74669]: 2024-06-09 22:34:01.570267366 +0000 UTC m=+5.915815284 container cleanup 42996772a59ebab7c4530eca75fe3b45ad8bc8f182b6e4e57a2bc1f779e85b75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS) 2024-06-09T22:34:01.813 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:01 smithi194 bash[74669]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8 2024-06-09T22:34:02.189 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:01 smithi194 podman[74684]: 2024-06-09 22:34:01.854797182 +0000 UTC m=+0.652830562 container remove 42996772a59ebab7c4530eca75fe3b45ad8bc8f182b6e4e57a2bc1f779e85b75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240528) 2024-06-09T22:34:02.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:02 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:02.149+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:02.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:02 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:02.149+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:02.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:02 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:02.149+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:02.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:02 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:02.149+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:02.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:02 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:02.149+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:02.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:02 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:02.149+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:02.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:02 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:02.188+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:34:02.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:02 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:02.188+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:34:02.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:02 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:02.188+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:34:02.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:02 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:02.188+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:34:02.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:02 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:02.188+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:34:02.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:02 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:02.188+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:36.363231+0000 front 2024-06-09T22:33:36.363212+0000 (oldest deadline 2024-06-09T22:33:59.263163+0000) 2024-06-09T22:34:02.521 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:02 smithi194 podman[74777]: 2024-06-09 22:34:02.089312628 +0000 UTC m=+0.017248119 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:34:02.521 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:02 smithi194 podman[74777]: 2024-06-09 22:34:02.19097875 +0000 UTC m=+0.118914255 container create 0f2261fcc6cb588820bf07ec01f1a3aa492f51413d5cc3683780c9ab88b30274 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:34:02.521 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:02 smithi194 podman[74777]: 2024-06-09 22:34:02.410101105 +0000 UTC m=+0.338036587 container init 0f2261fcc6cb588820bf07ec01f1a3aa492f51413d5cc3683780c9ab88b30274 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2) 2024-06-09T22:34:02.521 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:02 smithi194 podman[74777]: 2024-06-09 22:34:02.413123852 +0000 UTC m=+0.341059341 container start 0f2261fcc6cb588820bf07ec01f1a3aa492f51413d5cc3683780c9ab88b30274 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux ) 2024-06-09T22:34:02.521 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:02 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:02.296+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:02.522 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:02 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:02.296+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:02.522 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:02 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:02.296+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:02.522 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:02 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:02.296+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:02.522 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:02 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:02.296+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:02.813 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:02 smithi194 podman[74777]: 2024-06-09 22:34:02.518190107 +0000 UTC m=+0.446125603 container attach 0f2261fcc6cb588820bf07ec01f1a3aa492f51413d5cc3683780c9ab88b30274 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-09T22:34:02.813 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jun 09 22:34:02 smithi194 podman[74777]: 2024-06-09 22:34:02.705334681 +0000 UTC m=+0.633270168 container died 0f2261fcc6cb588820bf07ec01f1a3aa492f51413d5cc3683780c9ab88b30274 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, ceph=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-06-09T22:34:03.346 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.8.service' 2024-06-09T22:34:03.424 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:34:03.424 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-06-09T22:34:03.425 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-06-09T22:34:03.425 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.9 2024-06-09T22:34:03.451 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:03.185+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:34:03.452 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:03.185+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:34:03.452 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:03.185+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:34:03.452 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:03.185+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:34:03.452 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:03.185+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:34:03.452 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:03.185+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:36.363231+0000 front 2024-06-09T22:33:36.363212+0000 (oldest deadline 2024-06-09T22:33:59.263163+0000) 2024-06-09T22:34:03.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:03 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:03.160+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:03.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:03 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:03.160+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:03.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:03 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:03.160+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:03.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:03 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:03.160+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:03.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:03 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:03.160+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:03.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:03 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:03.160+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:03.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:03 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:03.296+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:03.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:03 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:03.296+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:03.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:03 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:03.296+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:03.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:03 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:03.296+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:03.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:03 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:03.296+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:03.730 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:03 smithi104 systemd[1]: Stopping Ceph osd.9 for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:34:04.010 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:03.726+0000 7f1978d22700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:34:04.011 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:03.726+0000 7f1978d22700 -1 osd.9 114 *** Got signal Terminated *** 2024-06-09T22:34:04.011 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:03 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:03.726+0000 7f1978d22700 -1 osd.9 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-09T22:34:04.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:04 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:04.122+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:04.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:04 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:04.122+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:04.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:04 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:04.122+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:04.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:04 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:04.122+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:04.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:04 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:04.122+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:04.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:04 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:04.122+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:04.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:04 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:04.212+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:34:04.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:04 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:04.212+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:34:04.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:04 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:04.212+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:34:04.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:04 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:04.212+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:34:04.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:04 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:04.212+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:34:04.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:04 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:04.212+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:36.363231+0000 front 2024-06-09T22:33:36.363212+0000 (oldest deadline 2024-06-09T22:33:59.263163+0000) 2024-06-09T22:34:04.521 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:04 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:04.248+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:04.521 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:04 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:04.248+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:04.521 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:04 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:04.248+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:04.521 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:04 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:04.248+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:04.521 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:04 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:04.248+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:05.469 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:05 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:05.095+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:05.470 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:05 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:05.095+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:05.470 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:05 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:05.095+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:05.470 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:05 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:05.095+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:05.470 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:05 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:05.095+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:05.470 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:05 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:05.095+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:05.510 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:05 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:05.256+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:34:05.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:05 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:05.256+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:34:05.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:05 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:05.256+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:34:05.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:05 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:05.256+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:34:05.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:05 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:05.256+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:34:05.511 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:05 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:05.256+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:36.363231+0000 front 2024-06-09T22:33:36.363212+0000 (oldest deadline 2024-06-09T22:33:59.263163+0000) 2024-06-09T22:34:05.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:05 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:05.199+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:05.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:05 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:05.199+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:05.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:05 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:05.199+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:05.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:05 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:05.199+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:05.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:05 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:05.199+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:06.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:06 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:06.111+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:06.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:06 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:06.111+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:06.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:06 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:06.111+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:06.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:06 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:06.111+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:06.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:06 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:06.111+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:06.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:06 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:06.111+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:06.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:06 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:06.208+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:06.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:06 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:06.208+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:06.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:06 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:06.208+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:06.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:06 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:06.208+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:06.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:06 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:06.208+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:06.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:06 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:06.278+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:34:06.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:06 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:06.278+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:34:06.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:06 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:06.278+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:34:06.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:06 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:06.278+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:34:06.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:06 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:06.278+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:34:06.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:06 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:06.278+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:36.363231+0000 front 2024-06-09T22:33:36.363212+0000 (oldest deadline 2024-06-09T22:33:59.263163+0000) 2024-06-09T22:34:07.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:07 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:07.071+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:07.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:07 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:07.071+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:07.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:07 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:07.071+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:07.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:07 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:07.071+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:07.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:07 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:07.071+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:07.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:07 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:07.071+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:07.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:07 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:07.172+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:07.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:07 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:07.172+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:07.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:07 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:07.172+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:07.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:07 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:07.172+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:07.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:07 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:07.172+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:07.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:07 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:07.274+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:34:07.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:07 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:07.274+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:34:07.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:07 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:07.274+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:34:07.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:07 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:07.274+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:34:07.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:07 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:07.274+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:34:07.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:07 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:07.274+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:36.363231+0000 front 2024-06-09T22:33:36.363212+0000 (oldest deadline 2024-06-09T22:33:59.263163+0000) 2024-06-09T22:34:08.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:08 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:08.116+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:08.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:08 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:08.116+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:08.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:08 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:08.116+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:08.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:08 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:08.116+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:08.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:08 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:08.116+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:08.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:08 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:08.116+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:08.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:08 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:08.165+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:08.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:08 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:08.165+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:08.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:08 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:08.165+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:08.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:08 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:08.165+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:08.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:08 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:08.165+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:08.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:08 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:08.165+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:08.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:08 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:08.312+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.154818+0000 front 2024-06-09T22:32:55.154847+0000 (oldest deadline 2024-06-09T22:33:19.254977+0000) 2024-06-09T22:34:08.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:08 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:08.312+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:32:59.255398+0000 front 2024-06-09T22:32:59.255418+0000 (oldest deadline 2024-06-09T22:33:25.155519+0000) 2024-06-09T22:34:08.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:08 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:08.312+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:11.458022+0000 front 2024-06-09T22:33:11.458100+0000 (oldest deadline 2024-06-09T22:33:34.958107+0000) 2024-06-09T22:34:08.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:08 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:08.312+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:17.159697+0000 front 2024-06-09T22:33:17.159732+0000 (oldest deadline 2024-06-09T22:33:42.459792+0000) 2024-06-09T22:34:08.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:08 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:08.312+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:27.061570+0000 front 2024-06-09T22:33:27.061529+0000 (oldest deadline 2024-06-09T22:33:49.961409+0000) 2024-06-09T22:34:08.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:08 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9[64562]: 2024-06-09T22:34:08.312+0000 7f1975316700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:36.363231+0000 front 2024-06-09T22:33:36.363212+0000 (oldest deadline 2024-06-09T22:33:59.263163+0000) 2024-06-09T22:34:09.496 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:09 smithi104 podman[123286]: 2024-06-09 22:34:09.210351949 +0000 UTC m=+5.585938312 container died 336722146f563db52735f06e901a7a7a31564b47766c14d5966a59258add9509 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9, ceph=True, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-09T22:34:09.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:09 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:09.067+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:09.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:09 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:09.067+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:09.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:09 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:09.067+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:09.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:09 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:09.067+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:09.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:09 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:09.067+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:09.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:09 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:09.067+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:09.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:09 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:09.067+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:44.353376+0000 front 2024-06-09T22:33:44.353307+0000 (oldest deadline 2024-06-09T22:34:08.453117+0000) 2024-06-09T22:34:09.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:09 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:09.137+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:09.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:09 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:09.137+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:09.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:09 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:09.137+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:09.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:09 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:09.137+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:09.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:09 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:09.137+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:09.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:09 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:09.137+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:09.760 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:09 smithi104 podman[123286]: 2024-06-09 22:34:09.642734647 +0000 UTC m=+6.018321031 container cleanup 336722146f563db52735f06e901a7a7a31564b47766c14d5966a59258add9509 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:34:09.761 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:09 smithi104 bash[123286]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9 2024-06-09T22:34:10.260 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:09 smithi104 podman[123304]: 2024-06-09 22:34:09.810884256 +0000 UTC m=+0.602665723 container remove 336722146f563db52735f06e901a7a7a31564b47766c14d5966a59258add9509 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:34:10.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:10 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:10.061+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:10.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:10 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:10.061+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:10.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:10 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:10.061+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:10.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:10 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:10.061+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:10.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:10 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:10.061+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:10.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:10 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:10.061+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:10.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:10 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:10.061+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:44.353376+0000 front 2024-06-09T22:33:44.353307+0000 (oldest deadline 2024-06-09T22:34:08.453117+0000) 2024-06-09T22:34:10.550 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:10 smithi104 podman[123392]: 2024-06-09 22:34:10.162205925 +0000 UTC m=+0.016865301 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:34:10.550 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:10 smithi104 podman[123392]: 2024-06-09 22:34:10.269965224 +0000 UTC m=+0.124624585 container create d0ab1deeafac07c362ec8a24697f4e175a51c487dc29f502bf944b607abbdb88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-06-09T22:34:10.551 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:10 smithi104 podman[123392]: 2024-06-09 22:34:10.464122716 +0000 UTC m=+0.318782086 container init d0ab1deeafac07c362ec8a24697f4e175a51c487dc29f502bf944b607abbdb88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-3d89acc) 2024-06-09T22:34:10.551 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:10 smithi104 podman[123392]: 2024-06-09 22:34:10.467328296 +0000 UTC m=+0.321987666 container start d0ab1deeafac07c362ec8a24697f4e175a51c487dc29f502bf944b607abbdb88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-06-09T22:34:10.551 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:10 smithi104 podman[123392]: 2024-06-09 22:34:10.547346825 +0000 UTC m=+0.402006205 container attach d0ab1deeafac07c362ec8a24697f4e175a51c487dc29f502bf944b607abbdb88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-06-09T22:34:10.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:10 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:10.088+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:10.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:10 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:10.088+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:10.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:10 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:10.088+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:10.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:10 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:10.088+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:10.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:10 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:10.088+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:10.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:10 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:10.088+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:11.010 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:10 smithi104 podman[123392]: 2024-06-09 22:34:10.752819368 +0000 UTC m=+0.607478738 container died d0ab1deeafac07c362ec8a24697f4e175a51c487dc29f502bf944b607abbdb88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-06-09T22:34:11.294 INFO:journalctl@ceph.osd.9.smithi104.stdout:Jun 09 22:34:11 smithi104 podman[123419]: 2024-06-09 22:34:11.275904053 +0000 UTC m=+0.517103250 container remove d0ab1deeafac07c362ec8a24697f4e175a51c487dc29f502bf944b607abbdb88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-06-09T22:34:11.299 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.9.service' 2024-06-09T22:34:11.378 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:34:11.378 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-06-09T22:34:11.379 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-06-09T22:34:11.379 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.10 2024-06-09T22:34:11.405 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:11 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:11.028+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:11.405 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:11 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:11.028+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:11.405 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:11 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:11.028+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:11.405 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:11 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:11.028+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:11.405 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:11 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:11.028+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:11.405 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:11 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:11.028+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:11.406 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:11 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:11.028+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:44.353376+0000 front 2024-06-09T22:33:44.353307+0000 (oldest deadline 2024-06-09T22:34:08.453117+0000) 2024-06-09T22:34:11.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:11 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:11.114+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:11.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:11 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:11.114+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:11.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:11 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:11.114+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:11.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:11 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:11.114+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:11.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:11 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:11.114+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:11.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:11 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:11.114+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:11.693 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:11 smithi153 systemd[1]: Stopping Ceph osd.10 for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:34:11.996 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:11 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:11.689+0000 7ffa8d952700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:34:11.997 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:11 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:11.689+0000 7ffa8d952700 -1 osd.10 114 *** Got signal Terminated *** 2024-06-09T22:34:11.997 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:11 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:11.689+0000 7ffa8d952700 -1 osd.10 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-09T22:34:12.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:12 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:12.047+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:12.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:12 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:12.047+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:12.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:12 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:12.047+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:12.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:12 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:12.047+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:12.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:12 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:12.047+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:12.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:12 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:12.047+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:12.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:12 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:12.047+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:44.353376+0000 front 2024-06-09T22:33:44.353307+0000 (oldest deadline 2024-06-09T22:34:08.453117+0000) 2024-06-09T22:34:12.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:12 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:12.073+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:12.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:12 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:12.073+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:12.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:12 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:12.073+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:12.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:12 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:12.073+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:12.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:12 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:12.073+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:12.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:12 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:12.073+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:13.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:13 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:13.056+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:13.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:13 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:13.056+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:13.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:13 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:13.056+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:13.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:13 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:13.056+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:13.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:13 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:13.056+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:13.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:13 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:13.056+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:13.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:13 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:13.056+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:44.353376+0000 front 2024-06-09T22:33:44.353307+0000 (oldest deadline 2024-06-09T22:34:08.453117+0000) 2024-06-09T22:34:13.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:13 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:13.079+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:13.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:13 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:13.079+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:13.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:13 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:13.079+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:13.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:13 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:13.079+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:13.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:13 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:13.079+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:13.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:13 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:13.079+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:14.313 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:14 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:14.066+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:14.313 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:14 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:14.066+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:14.313 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:14 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:14.066+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:14.313 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:14 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:14.066+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:14.313 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:14 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:14.066+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:14.313 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:14 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:14.066+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:14.314 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:14 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:14.066+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-06-09T22:33:49.244611+0000 front 2024-06-09T22:33:49.244715+0000 (oldest deadline 2024-06-09T22:34:13.344605+0000) 2024-06-09T22:34:14.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.007+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:14.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.007+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:14.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.007+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:14.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.007+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:14.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.007+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:14.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.007+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:14.498 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.007+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:44.353376+0000 front 2024-06-09T22:33:44.353307+0000 (oldest deadline 2024-06-09T22:34:08.453117+0000) 2024-06-09T22:34:15.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:15.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:15.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:15.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:15.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:15.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:15.248 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:14 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:14.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:44.353376+0000 front 2024-06-09T22:33:44.353307+0000 (oldest deadline 2024-06-09T22:34:08.453117+0000) 2024-06-09T22:34:15.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:15 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:15.112+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:15.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:15 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:15.112+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:15.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:15 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:15.112+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:15.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:15 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:15.112+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:15.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:15 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:15.112+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:15.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:15 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:15.112+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:15.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:15 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:15.112+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-06-09T22:33:49.244611+0000 front 2024-06-09T22:33:49.244715+0000 (oldest deadline 2024-06-09T22:34:13.344605+0000) 2024-06-09T22:34:16.246 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:15 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:15.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:55.545728+0000 front 2024-06-09T22:32:55.545513+0000 (oldest deadline 2024-06-09T22:33:19.045625+0000) 2024-06-09T22:34:16.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:15 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:15.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-06-09T22:33:01.346444+0000 front 2024-06-09T22:33:01.346466+0000 (oldest deadline 2024-06-09T22:33:25.347094+0000) 2024-06-09T22:34:16.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:15 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:15.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.648165+0000 front 2024-06-09T22:33:10.648101+0000 (oldest deadline 2024-06-09T22:33:35.348158+0000) 2024-06-09T22:34:16.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:15 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:15.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:18.749788+0000 front 2024-06-09T22:33:18.749698+0000 (oldest deadline 2024-06-09T22:33:42.249664+0000) 2024-06-09T22:34:16.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:15 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:15.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:26.350455+0000 front 2024-06-09T22:33:26.350507+0000 (oldest deadline 2024-06-09T22:33:52.150952+0000) 2024-06-09T22:34:16.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:15 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:15.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:32.651947+0000 front 2024-06-09T22:33:32.651866+0000 (oldest deadline 2024-06-09T22:33:58.551776+0000) 2024-06-09T22:34:16.247 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:15 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10[61198]: 2024-06-09T22:34:15.977+0000 7ffa85f3e700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:44.353376+0000 front 2024-06-09T22:33:44.353307+0000 (oldest deadline 2024-06-09T22:34:08.453117+0000) 2024-06-09T22:34:16.332 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:16 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:16.133+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:16.332 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:16 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:16.133+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:16.332 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:16 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:16.133+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:16.332 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:16 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:16.133+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:16.332 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:16 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:16.133+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:16.332 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:16 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:16.133+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:16.332 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:16 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:16.133+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-06-09T22:33:49.244611+0000 front 2024-06-09T22:33:49.244715+0000 (oldest deadline 2024-06-09T22:34:13.344605+0000) 2024-06-09T22:34:17.444 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:17 smithi153 podman[74299]: 2024-06-09 22:34:17.183275436 +0000 UTC m=+5.600352019 container died 12f75f292eea3be8249fd1e849b2b5af1d96be376a6f4c41e12730539ca51079 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:34:17.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:17 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:17.138+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:17.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:17 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:17.138+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:17.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:17 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:17.138+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:17.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:17 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:17.138+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:17.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:17 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:17.138+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:17.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:17 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:17.138+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:17.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:17 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:17.138+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-06-09T22:33:49.244611+0000 front 2024-06-09T22:33:49.244715+0000 (oldest deadline 2024-06-09T22:34:13.344605+0000) 2024-06-09T22:34:17.746 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:17 smithi153 podman[74299]: 2024-06-09 22:34:17.539939286 +0000 UTC m=+5.957015863 container cleanup 12f75f292eea3be8249fd1e849b2b5af1d96be376a6f4c41e12730539ca51079 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-06-09T22:34:17.747 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:17 smithi153 bash[74299]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10 2024-06-09T22:34:18.214 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:17 smithi153 podman[74315]: 2024-06-09 22:34:17.84142534 +0000 UTC m=+0.654852647 container remove 12f75f292eea3be8249fd1e849b2b5af1d96be376a6f4c41e12730539ca51079 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10, ceph=True, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-09T22:34:18.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:18 smithi153 podman[74405]: 2024-06-09 22:34:18.114231084 +0000 UTC m=+0.017621946 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:34:18.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:18 smithi153 podman[74405]: 2024-06-09 22:34:18.21712703 +0000 UTC m=+0.120517932 container create d04d4680c98e5d545d1aa86408feca627106d30975e6a9c2454704ff835d421d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-06-09T22:34:18.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:18 smithi153 podman[74405]: 2024-06-09 22:34:18.427744634 +0000 UTC m=+0.331135484 container init d04d4680c98e5d545d1aa86408feca627106d30975e6a9c2454704ff835d421d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-09T22:34:18.497 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:18 smithi153 podman[74405]: 2024-06-09 22:34:18.430748643 +0000 UTC m=+0.334139500 container start d04d4680c98e5d545d1aa86408feca627106d30975e6a9c2454704ff835d421d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc) 2024-06-09T22:34:18.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:18 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:18.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:18.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:18 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:18.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:18.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:18 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:18.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:18.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:18 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:18.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:18.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:18 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:18.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:18.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:18 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:18.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:18.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:18 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:18.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-06-09T22:33:49.244611+0000 front 2024-06-09T22:33:49.244715+0000 (oldest deadline 2024-06-09T22:34:13.344605+0000) 2024-06-09T22:34:18.942 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:18 smithi153 podman[74405]: 2024-06-09 22:34:18.502541452 +0000 UTC m=+0.405932312 container attach d04d4680c98e5d545d1aa86408feca627106d30975e6a9c2454704ff835d421d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:34:18.942 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:18 smithi153 podman[74405]: 2024-06-09 22:34:18.718259155 +0000 UTC m=+0.621650017 container died d04d4680c98e5d545d1aa86408feca627106d30975e6a9c2454704ff835d421d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10-deactivate, ceph=True, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528) 2024-06-09T22:34:19.194 INFO:journalctl@ceph.osd.10.smithi153.stdout:Jun 09 22:34:19 smithi153 podman[74431]: 2024-06-09 22:34:19.180777933 +0000 UTC m=+0.453621355 container remove d04d4680c98e5d545d1aa86408feca627106d30975e6a9c2454704ff835d421d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-06-09T22:34:19.200 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.10.service' 2024-06-09T22:34:19.277 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:34:19.277 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-06-09T22:34:19.277 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-06-09T22:34:19.278 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.11 2024-06-09T22:34:19.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:19.191+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:19.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:19.191+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:19.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:19.191+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:19.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:19.191+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:19.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:19.191+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:19.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:19.191+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:19.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:19.191+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-06-09T22:33:49.244611+0000 front 2024-06-09T22:33:49.244715+0000 (oldest deadline 2024-06-09T22:34:13.344605+0000) 2024-06-09T22:34:19.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:19 smithi194 systemd[1]: Stopping Ceph osd.11 for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:34:20.063 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:19.598+0000 7f1a91689700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:34:20.063 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:19.598+0000 7f1a91689700 -1 osd.11 114 *** Got signal Terminated *** 2024-06-09T22:34:20.063 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:19 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:19.598+0000 7f1a91689700 -1 osd.11 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-09T22:34:20.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:20 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:20.163+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:20.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:20 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:20.163+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:20.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:20 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:20.163+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:20.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:20 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:20.163+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:20.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:20 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:20.163+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:20.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:20 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:20.163+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:20.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:20 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:20.163+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-06-09T22:33:49.244611+0000 front 2024-06-09T22:33:49.244715+0000 (oldest deadline 2024-06-09T22:34:13.344605+0000) 2024-06-09T22:34:21.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:21 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:21.130+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:21.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:21 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:21.130+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:21.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:21 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:21.130+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:21.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:21 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:21.130+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:21.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:21 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:21.131+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:21.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:21 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:21.131+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:21.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:21 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:21.131+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-06-09T22:33:49.244611+0000 front 2024-06-09T22:33:49.244715+0000 (oldest deadline 2024-06-09T22:34:13.344605+0000) 2024-06-09T22:34:22.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:22 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:22.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:22.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:22 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:22.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:22.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:22 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:22.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:22.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:22 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:22.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:22.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:22 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:22.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:22.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:22 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:22.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:22.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:22 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:22.142+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-06-09T22:33:49.244611+0000 front 2024-06-09T22:33:49.244715+0000 (oldest deadline 2024-06-09T22:34:13.344605+0000) 2024-06-09T22:34:23.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:23 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:23.146+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:23.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:23 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:23.146+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:23.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:23 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:23.146+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:23.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:23 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:23.146+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:23.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:23 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:23.146+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:23.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:23 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:23.146+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:23.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:23 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:23.146+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-06-09T22:33:49.244611+0000 front 2024-06-09T22:33:49.244715+0000 (oldest deadline 2024-06-09T22:34:13.344605+0000) 2024-06-09T22:34:24.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:24 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:24.120+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-06-09T22:32:54.532297+0000 front 2024-06-09T22:32:54.532440+0000 (oldest deadline 2024-06-09T22:33:17.432228+0000) 2024-06-09T22:34:24.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:24 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:24.120+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-06-09T22:33:10.334449+0000 front 2024-06-09T22:33:10.334402+0000 (oldest deadline 2024-06-09T22:33:33.834506+0000) 2024-06-09T22:34:24.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:24 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:24.120+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6814 osd.3 since back 2024-06-09T22:33:19.437270+0000 front 2024-06-09T22:33:19.437423+0000 (oldest deadline 2024-06-09T22:33:41.737259+0000) 2024-06-09T22:34:24.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:24 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:24.120+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-06-09T22:33:28.440716+0000 front 2024-06-09T22:33:28.440749+0000 (oldest deadline 2024-06-09T22:33:49.540653+0000) 2024-06-09T22:34:24.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:24 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:24.120+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-06-09T22:33:34.742261+0000 front 2024-06-09T22:33:34.742394+0000 (oldest deadline 2024-06-09T22:33:58.842399+0000) 2024-06-09T22:34:24.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:24 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:24.120+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.104:6822 osd.6 since back 2024-06-09T22:33:42.343320+0000 front 2024-06-09T22:33:42.343195+0000 (oldest deadline 2024-06-09T22:34:07.643170+0000) 2024-06-09T22:34:24.564 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:24 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-06-09T22:34:24.120+0000 7f1a8dc7d700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-06-09T22:33:49.244611+0000 front 2024-06-09T22:33:49.244715+0000 (oldest deadline 2024-06-09T22:34:13.344605+0000) 2024-06-09T22:34:25.284 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:24 smithi194 podman[74907]: 2024-06-09 22:34:24.990340831 +0000 UTC m=+5.516733233 container died 4e38c921712f7d2243ed488959c9680b7573dc9d6af3cc8b5f1ac5f9b684c239 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-09T22:34:25.284 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:25 smithi194 podman[74907]: 2024-06-09 22:34:25.281416777 +0000 UTC m=+5.807809171 container cleanup 4e38c921712f7d2243ed488959c9680b7573dc9d6af3cc8b5f1ac5f9b684c239 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-06-09T22:34:25.562 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:25 smithi194 bash[74907]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11 2024-06-09T22:34:25.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:25 smithi194 podman[74923]: 2024-06-09 22:34:25.524169305 +0000 UTC m=+0.531458576 container remove 4e38c921712f7d2243ed488959c9680b7573dc9d6af3cc8b5f1ac5f9b684c239 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:34:26.163 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:25 smithi194 podman[75012]: 2024-06-09 22:34:25.75016931 +0000 UTC m=+0.017139867 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-09T22:34:26.163 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:25 smithi194 podman[75012]: 2024-06-09 22:34:25.849770764 +0000 UTC m=+0.116741297 container create 8afcbb52ea85f70b1e4ec1885c656bfbad4d4f84a76827cfd2fbc7ab19f595f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS) 2024-06-09T22:34:26.164 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:26 smithi194 podman[75012]: 2024-06-09 22:34:26.076949191 +0000 UTC m=+0.343919721 container init 8afcbb52ea85f70b1e4ec1885c656bfbad4d4f84a76827cfd2fbc7ab19f595f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-06-09T22:34:26.164 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:26 smithi194 podman[75012]: 2024-06-09 22:34:26.080290562 +0000 UTC m=+0.347261103 container start 8afcbb52ea85f70b1e4ec1885c656bfbad4d4f84a76827cfd2fbc7ab19f595f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11-deactivate, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-06-09T22:34:26.164 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:26 smithi194 podman[75012]: 2024-06-09 22:34:26.160141037 +0000 UTC m=+0.427111579 container attach 8afcbb52ea85f70b1e4ec1885c656bfbad4d4f84a76827cfd2fbc7ab19f595f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-09T22:34:26.563 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jun 09 22:34:26 smithi194 podman[75012]: 2024-06-09 22:34:26.370024191 +0000 UTC m=+0.636994732 container died 8afcbb52ea85f70b1e4ec1885c656bfbad4d4f84a76827cfd2fbc7ab19f595f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-09T22:34:26.939 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@osd.11.service' 2024-06-09T22:34:26.975 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:34:26.975 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-06-09T22:34:26.975 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-06-09T22:34:26.975 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.a 2024-06-09T22:34:27.260 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:34:27 smithi104 systemd[1]: Stopping Ceph mds.a for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:34:27.760 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:34:27 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-a[71945]: 2024-06-09T22:34:27.310+0000 7efece7a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:34:27.760 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:34:27 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-a[71945]: 2024-06-09T22:34:27.310+0000 7efece7a0700 -1 mds.a *** got signal Terminated *** 2024-06-09T22:34:29.604 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:34:29 smithi104 podman[123499]: 2024-06-09 22:34:29.327294059 +0000 UTC m=+2.155545975 container died cdde291a6221532b97b0d9cb4b8206f8307140dcb3b80ac4f3dbcb4ea60af0d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-a, RELEASE=squid-3d89acc, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:34:30.010 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:34:29 smithi104 podman[123499]: 2024-06-09 22:34:29.784825484 +0000 UTC m=+2.613077391 container cleanup cdde291a6221532b97b0d9cb4b8206f8307140dcb3b80ac4f3dbcb4ea60af0d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240528) 2024-06-09T22:34:30.010 INFO:journalctl@ceph.mds.a.smithi104.stdout:Jun 09 22:34:29 smithi104 bash[123499]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-a 2024-06-09T22:34:30.228 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.a.service' 2024-06-09T22:34:30.264 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:34:30.265 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-06-09T22:34:30.265 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-06-09T22:34:30.265 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.d 2024-06-09T22:34:30.760 INFO:journalctl@ceph.mds.d.smithi104.stdout:Jun 09 22:34:30 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-d[70720]: 2024-06-09T22:34:30.572+0000 7ff3c04c1700 -1 mds.d *** got signal Terminated *** 2024-06-09T22:34:33.910 INFO:journalctl@ceph.mds.d.smithi104.stdout:Jun 09 22:34:33 smithi104 podman[123629]: 2024-06-09 22:34:33.64582086 +0000 UTC m=+3.166837556 container died 5cc820ab0504190e0ce47bf7595040b7dcbef0e8b490870d5f9da9c0c8025172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-09T22:34:34.260 INFO:journalctl@ceph.mds.d.smithi104.stdout:Jun 09 22:34:34 smithi104 podman[123629]: 2024-06-09 22:34:34.080406645 +0000 UTC m=+3.601423343 container cleanup 5cc820ab0504190e0ce47bf7595040b7dcbef0e8b490870d5f9da9c0c8025172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-3d89acc, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:34:34.261 INFO:journalctl@ceph.mds.d.smithi104.stdout:Jun 09 22:34:34 smithi104 bash[123629]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-d 2024-06-09T22:34:34.645 INFO:journalctl@ceph.mds.d.smithi104.stdout:Jun 09 22:34:34 smithi104 podman[123640]: 2024-06-09 22:34:34.34905443 +0000 UTC m=+0.701322858 container remove 5cc820ab0504190e0ce47bf7595040b7dcbef0e8b490870d5f9da9c0c8025172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-09T22:34:34.663 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.d.service' 2024-06-09T22:34:34.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:34:34.696 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-06-09T22:34:34.696 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-06-09T22:34:34.696 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.g 2024-06-09T22:34:35.005 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:34:34 smithi104 systemd[1]: Stopping Ceph mds.g for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:34:35.260 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:34:35 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-g[71521]: 2024-06-09T22:34:35.001+0000 7f31250db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:34:35.260 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:34:35 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-g[71521]: 2024-06-09T22:34:35.001+0000 7f31250db700 -1 mds.g *** got signal Terminated *** 2024-06-09T22:34:37.597 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:34:37 smithi104 podman[123750]: 2024-06-09 22:34:37.325849435 +0000 UTC m=+2.418626249 container died 14860765eeef8b3083e6c5843a2e9d29a20604db94bb1d246d4a6afba8264110 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-06-09T22:34:38.010 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:34:37 smithi104 podman[123750]: 2024-06-09 22:34:37.77669601 +0000 UTC m=+2.869472829 container cleanup 14860765eeef8b3083e6c5843a2e9d29a20604db94bb1d246d4a6afba8264110 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-06-09T22:34:38.011 INFO:journalctl@ceph.mds.g.smithi104.stdout:Jun 09 22:34:37 smithi104 bash[123750]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-g 2024-06-09T22:34:38.161 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.g.service' 2024-06-09T22:34:38.194 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:34:38.194 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-06-09T22:34:38.194 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-06-09T22:34:38.195 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.j 2024-06-09T22:34:38.356 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:34:38 smithi104 systemd[1]: Stopping Ceph mds.j for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:34:38.760 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:34:38 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-j[70278]: 2024-06-09T22:34:38.513+0000 7f0a901f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:34:38.761 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:34:38 smithi104 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-j[70278]: 2024-06-09T22:34:38.513+0000 7f0a901f1700 -1 mds.j *** got signal Terminated *** 2024-06-09T22:34:41.991 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:34:41 smithi104 podman[123871]: 2024-06-09 22:34:41.639902584 +0000 UTC m=+3.217446042 container died b1baca679f74bd6cf4f8d00dcb32fa91f079e70107ef1a1603f1fbd34510d72d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-06-09T22:34:41.991 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:34:41 smithi104 podman[123871]: 2024-06-09 22:34:41.988288798 +0000 UTC m=+3.565832256 container cleanup b1baca679f74bd6cf4f8d00dcb32fa91f079e70107ef1a1603f1fbd34510d72d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-j, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:34:42.260 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:34:41 smithi104 bash[123871]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-j 2024-06-09T22:34:42.261 INFO:journalctl@ceph.mds.j.smithi104.stdout:Jun 09 22:34:42 smithi104 podman[123884]: 2024-06-09 22:34:42.173097556 +0000 UTC m=+0.526980467 container remove b1baca679f74bd6cf4f8d00dcb32fa91f079e70107ef1a1603f1fbd34510d72d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:34:42.399 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.j.service' 2024-06-09T22:34:42.431 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:34:42.431 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-06-09T22:34:42.431 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-06-09T22:34:42.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.b 2024-06-09T22:34:42.734 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:34:42 smithi153 systemd[1]: Stopping Ceph mds.b for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:34:42.997 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:34:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b[66716]: 2024-06-09T22:34:42.731+0000 7f528c6ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:34:42.997 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:34:42 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b[66716]: 2024-06-09T22:34:42.731+0000 7f528c6ee700 -1 mds.b *** got signal Terminated *** 2024-06-09T22:34:45.600 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:34:45 smithi153 podman[74521]: 2024-06-09 22:34:45.329005756 +0000 UTC m=+2.701902020 container died 0c85ecc8056baf0e4c3aefd1c003fcaf4e0138bf6ea97a63d9b96c40c492c84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-09T22:34:45.601 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:34:45 smithi153 podman[74521]: 2024-06-09 22:34:45.597635438 +0000 UTC m=+2.970531703 container cleanup 0c85ecc8056baf0e4c3aefd1c003fcaf4e0138bf6ea97a63d9b96c40c492c84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-09T22:34:45.978 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:34:45 smithi153 bash[74521]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b 2024-06-09T22:34:45.978 INFO:journalctl@ceph.mds.b.smithi153.stdout:Jun 09 22:34:45 smithi153 podman[74532]: 2024-06-09 22:34:45.840223396 +0000 UTC m=+0.508613383 container remove 0c85ecc8056baf0e4c3aefd1c003fcaf4e0138bf6ea97a63d9b96c40c492c84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2) 2024-06-09T22:34:45.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.b.service' 2024-06-09T22:34:46.028 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:34:46.028 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-06-09T22:34:46.028 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-06-09T22:34:46.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.e 2024-06-09T22:34:46.246 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:34:46 smithi153 systemd[1]: Stopping Ceph mds.e for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:34:46.746 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:34:46 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-e[66112]: 2024-06-09T22:34:46.384+0000 7f3d389c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:34:46.747 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:34:46 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-e[66112]: 2024-06-09T22:34:46.384+0000 7f3d389c1700 -1 mds.e *** got signal Terminated *** 2024-06-09T22:34:49.630 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:34:49 smithi153 podman[74644]: 2024-06-09 22:34:49.335144389 +0000 UTC m=+3.083415186 container died c5ae1e6e050f7721c32b7bf9d21a321a58add6a76c57edcfd4ca232e1e551cf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-e, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc) 2024-06-09T22:34:49.997 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:34:49 smithi153 podman[74644]: 2024-06-09 22:34:49.734717671 +0000 UTC m=+3.482988463 container cleanup c5ae1e6e050f7721c32b7bf9d21a321a58add6a76c57edcfd4ca232e1e551cf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-e, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-06-09T22:34:49.998 INFO:journalctl@ceph.mds.e.smithi153.stdout:Jun 09 22:34:49 smithi153 bash[74644]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-e 2024-06-09T22:34:50.163 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.e.service' 2024-06-09T22:34:50.196 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:34:50.196 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-06-09T22:34:50.197 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-06-09T22:34:50.197 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.h 2024-06-09T22:34:50.525 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:34:50 smithi153 systemd[1]: Stopping Ceph mds.h for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:34:50.829 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:34:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-h[66412]: 2024-06-09T22:34:50.521+0000 7f40c342b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:34:50.829 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:34:50 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-h[66412]: 2024-06-09T22:34:50.521+0000 7f40c342b700 -1 mds.h *** got signal Terminated *** 2024-06-09T22:34:53.618 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:34:53 smithi153 podman[74775]: 2024-06-09 22:34:53.327158224 +0000 UTC m=+2.893696804 container died 6a8f04b33aece99c5ed5fbbc063d3bf6aad6a8fd24d82f2ab4678dbcaf29f7cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-h, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux ) 2024-06-09T22:34:53.934 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:34:53 smithi153 podman[74775]: 2024-06-09 22:34:53.705202079 +0000 UTC m=+3.271740659 container cleanup 6a8f04b33aece99c5ed5fbbc063d3bf6aad6a8fd24d82f2ab4678dbcaf29f7cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc) 2024-06-09T22:34:53.934 INFO:journalctl@ceph.mds.h.smithi153.stdout:Jun 09 22:34:53 smithi153 bash[74775]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-h 2024-06-09T22:34:54.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.h.service' 2024-06-09T22:34:54.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:34:54.112 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-06-09T22:34:54.112 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-06-09T22:34:54.112 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.k 2024-06-09T22:34:54.471 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:34:54 smithi153 systemd[1]: Stopping Ceph mds.k for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:34:54.747 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:34:54 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-k[65798]: 2024-06-09T22:34:54.467+0000 7fc372610700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:34:54.747 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:34:54 smithi153 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-k[65798]: 2024-06-09T22:34:54.467+0000 7fc372610700 -1 mds.k *** got signal Terminated *** 2024-06-09T22:34:57.638 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:34:57 smithi153 podman[74897]: 2024-06-09 22:34:57.337402642 +0000 UTC m=+3.013834021 container died d8e017896f09ecbc0d605ec8dc16da9e3cd702c32cb38c94940624861c03229c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-09T22:34:57.962 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:34:57 smithi153 podman[74897]: 2024-06-09 22:34:57.736712184 +0000 UTC m=+3.413143558 container cleanup d8e017896f09ecbc0d605ec8dc16da9e3cd702c32cb38c94940624861c03229c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-k, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-06-09T22:34:57.963 INFO:journalctl@ceph.mds.k.smithi153.stdout:Jun 09 22:34:57 smithi153 bash[74897]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-k 2024-06-09T22:34:58.083 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.k.service' 2024-06-09T22:34:58.115 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:34:58.115 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-06-09T22:34:58.115 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-06-09T22:34:58.115 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.c 2024-06-09T22:34:58.407 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:34:58 smithi194 systemd[1]: Stopping Ceph mds.c for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:34:58.813 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:34:58 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-c[67396]: 2024-06-09T22:34:58.403+0000 7fb5d2af9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:34:58.813 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:34:58 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-c[67396]: 2024-06-09T22:34:58.403+0000 7fb5d2af9700 -1 mds.c *** got signal Terminated *** 2024-06-09T22:35:01.599 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:35:01 smithi194 podman[75119]: 2024-06-09 22:35:01.323498554 +0000 UTC m=+3.018788001 container died d0aec5967e0adea7dfbfa62dce793e7f18969e9cd0ea56283db69825f6bf7286 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-c, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:35:01.949 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:35:01 smithi194 podman[75119]: 2024-06-09 22:35:01.695819013 +0000 UTC m=+3.391108454 container cleanup d0aec5967e0adea7dfbfa62dce793e7f18969e9cd0ea56283db69825f6bf7286 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-06-09T22:35:01.950 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jun 09 22:35:01 smithi194 bash[75119]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-c 2024-06-09T22:35:02.086 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.c.service' 2024-06-09T22:35:02.120 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:35:02.121 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-06-09T22:35:02.121 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-06-09T22:35:02.121 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.f 2024-06-09T22:35:02.469 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:35:02 smithi194 systemd[1]: Stopping Ceph mds.f for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:35:02.813 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:35:02 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f[67095]: 2024-06-09T22:35:02.465+0000 7fcf73cc7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:35:02.813 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:35:02 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f[67095]: 2024-06-09T22:35:02.465+0000 7fcf73cc7700 -1 mds.f *** got signal Terminated *** 2024-06-09T22:35:05.627 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:35:05 smithi194 podman[75252]: 2024-06-09 22:35:05.334002861 +0000 UTC m=+2.990212916 container died ea3e90d5e7ea8e91584e570c18ea74d117513d9cf0e55fb07c54584a347c9f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-09T22:35:06.063 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:35:05 smithi194 podman[75252]: 2024-06-09 22:35:05.707355142 +0000 UTC m=+3.363565197 container cleanup ea3e90d5e7ea8e91584e570c18ea74d117513d9cf0e55fb07c54584a347c9f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528) 2024-06-09T22:35:06.063 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:35:05 smithi194 bash[75252]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f 2024-06-09T22:35:06.063 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jun 09 22:35:05 smithi194 podman[75264]: 2024-06-09 22:35:05.866824485 +0000 UTC m=+0.528800077 container remove ea3e90d5e7ea8e91584e570c18ea74d117513d9cf0e55fb07c54584a347c9f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-f, ceph=True, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-09T22:35:06.085 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.f.service' 2024-06-09T22:35:06.117 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:35:06.118 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-06-09T22:35:06.118 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-06-09T22:35:06.118 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.i 2024-06-09T22:35:06.439 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:35:06 smithi194 systemd[1]: Stopping Ceph mds.i for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:35:06.813 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:35:06 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i[67699]: 2024-06-09T22:35:06.435+0000 7f567b647700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:35:06.813 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:35:06 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i[67699]: 2024-06-09T22:35:06.435+0000 7f567b647700 -1 mds.i *** got signal Terminated *** 2024-06-09T22:35:09.606 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:35:09 smithi194 podman[75373]: 2024-06-09 22:35:09.327460157 +0000 UTC m=+2.993490046 container died 8ffd8f55d181714ee06007c3e3ae3abe9e8b2d682a78e34267a7356e8fdcbf37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i, ceph=True, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528) 2024-06-09T22:35:09.900 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:35:09 smithi194 podman[75373]: 2024-06-09 22:35:09.685651458 +0000 UTC m=+3.351681338 container cleanup 8ffd8f55d181714ee06007c3e3ae3abe9e8b2d682a78e34267a7356e8fdcbf37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5) 2024-06-09T22:35:09.901 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jun 09 22:35:09 smithi194 bash[75373]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-i 2024-06-09T22:35:10.122 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.i.service' 2024-06-09T22:35:10.153 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:35:10.154 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-06-09T22:35:10.154 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-06-09T22:35:10.154 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.l 2024-06-09T22:35:10.473 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:35:10 smithi194 systemd[1]: Stopping Ceph mds.l for a948c0ca-26ac-11ef-bc9b-c7b262605968... 2024-06-09T22:35:10.813 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:35:10 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-l[66781]: 2024-06-09T22:35:10.472+0000 7f1936846700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-09T22:35:10.813 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:35:10 smithi194 ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-l[66781]: 2024-06-09T22:35:10.472+0000 7f1936846700 -1 mds.l *** got signal Terminated *** 2024-06-09T22:35:13.602 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:35:13 smithi194 podman[75495]: 2024-06-09 22:35:13.32968629 +0000 UTC m=+2.965324516 container died 9373f7a032cae5b48d733c7fb6419c1af3e10da484169c4a206f9e61a07945ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-l, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc) 2024-06-09T22:35:13.859 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:35:13 smithi194 podman[75495]: 2024-06-09 22:35:13.66395517 +0000 UTC m=+3.299593389 container cleanup 9373f7a032cae5b48d733c7fb6419c1af3e10da484169c4a206f9e61a07945ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-l, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-09T22:35:13.860 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jun 09 22:35:13 smithi194 bash[75495]: ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968-mds-l 2024-06-09T22:35:14.087 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a948c0ca-26ac-11ef-bc9b-c7b262605968@mds.l.service' 2024-06-09T22:35:14.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-09T22:35:14.118 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-06-09T22:35:14.118 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-09T22:35:14.310 INFO:teuthology.orchestra.run.smithi104.stdout:Deleting cluster with fsid: a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:35:16.711 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-09T22:35:16.904 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:35:19.210 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-09T22:35:19.400 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:35:21.681 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-09T22:35:21.715 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-09T22:35:21.741 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-09T22:35:21.766 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-09T22:35:21.768 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213/remote/smithi104/crash 2024-06-09T22:35:21.768 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/crash -- . 2024-06-09T22:35:21.813 INFO:teuthology.orchestra.run.smithi104.stderr:tar: /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-09T22:35:21.814 INFO:teuthology.orchestra.run.smithi104.stderr:tar: Error is not recoverable: exiting now 2024-06-09T22:35:21.815 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213/remote/smithi153/crash 2024-06-09T22:35:21.816 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/crash -- . 2024-06-09T22:35:21.856 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-09T22:35:21.857 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-06-09T22:35:21.858 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213/remote/smithi194/crash 2024-06-09T22:35:21.859 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/crash -- . 2024-06-09T22:35:21.904 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /var/lib/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-09T22:35:21.904 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now 2024-06-09T22:35:21.905 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-09T22:35:21.905 DEBUG:teuthology.orchestra.run.smithi104:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-06-09T22:35:21.943 INFO:tasks.cephadm:Compressing logs... 2024-06-09T22:35:21.943 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-09T22:35:21.986 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-09T22:35:21.988 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-09T22:35:22.010 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.log 2024-06-09T22:35:22.010 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-06-09T22:35:22.010 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-09T22:35:22.021 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.log 2024-06-09T22:35:22.022 INFO:teuthology.orchestra.run.smithi194.stderr:: No such file or directory 2024-06-09T22:35:22.022 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-06-09T22:35:22.022 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.f.log 2024-06-09T22:35:22.022 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-09T22:35:22.022 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-06-09T22:35:22.022 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.c.log 2024-06-09T22:35:22.022 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.audit.log: /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mon.c.log 2024-06-09T22:35:22.022 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mgr.z.log 2024-06-09T22:35:22.023 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mon.c.log: /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mgr.z.log: 87.2% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-09T22:35:22.023 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.i.log 2024-06-09T22:35:22.023 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-09T22:35:22.023 INFO:teuthology.orchestra.run.smithi194.stderr: 90.5%/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.i.log: -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-09T22:35:22.023 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.7.log 2024-06-09T22:35:22.023 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.log: /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.b.log 2024-06-09T22:35:22.023 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-06-09T22:35:22.024 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-06-09T22:35:22.024 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.b.log: /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mgr.y.log 2024-06-09T22:35:22.024 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.h.log 2024-06-09T22:35:22.024 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mgr.y.log: 87.2% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.log.gz 90.4% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-09T22:35:22.024 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-06-09T22:35:22.024 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.10.log 2024-06-09T22:35:22.024 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-09T22:35:22.029 INFO:teuthology.orchestra.run.smithi194.stderr: 92.1% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mgr.z.log.gz 2024-06-09T22:35:22.029 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.11.log 2024-06-09T22:35:22.030 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.8.log 2024-06-09T22:35:22.030 INFO:teuthology.orchestra.run.smithi194.stderr: 84.1% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-09T22:35:22.031 INFO:teuthology.orchestra.run.smithi104.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.log 2024-06-09T22:35:22.031 INFO:teuthology.orchestra.run.smithi104.stderr:: No such file or directory 2024-06-09T22:35:22.031 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-09T22:35:22.031 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.g.log 2024-06-09T22:35:22.031 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.log: /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.j.log 2024-06-09T22:35:22.031 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-06-09T22:35:22.031 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-06-09T22:35:22.032 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mgr.x.log 2024-06-09T22:35:22.032 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mon.a.log 2024-06-09T22:35:22.032 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.6.log 2024-06-09T22:35:22.032 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mon.a.log: 90.3% 87.2% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-09T22:35:22.032 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-09T22:35:22.032 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-09T22:35:22.032 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.a.log 2024-06-09T22:35:22.032 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.cephadm.log: 84.3% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-09T22:35:22.033 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-09T22:35:22.033 INFO:teuthology.orchestra.run.smithi104.stderr: 94.6% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.j.log.gz 2024-06-09T22:35:22.033 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.10.log: 92.1% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mgr.y.log.gz 2024-06-09T22:35:22.033 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.e.log 2024-06-09T22:35:22.033 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.cephadm.log: 84.1% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-09T22:35:22.034 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-09T22:35:22.034 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.d.log 2024-06-09T22:35:22.037 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-09T22:35:22.037 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mon.b.log 2024-06-09T22:35:22.044 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.k.log 2024-06-09T22:35:22.045 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.9.log 2024-06-09T22:35:22.047 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.d.log: 91.0% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-09T22:35:22.047 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-09T22:35:22.049 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.l.log 2024-06-09T22:35:22.051 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-09T22:35:22.057 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-09T22:35:22.059 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.9.log: 94.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-06-09T22:35:22.059 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.d.log.gz 2024-06-09T22:35:22.061 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.k.log: 91.1%/var/log/ceph/cephadm.log: -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-09T22:35:22.072 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-09T22:35:22.075 INFO:teuthology.orchestra.run.smithi153.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-09T22:35:22.076 INFO:teuthology.orchestra.run.smithi194.stderr: 91.0% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-09T22:35:22.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.admin.log: 92.6% 94.5% 94.5% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.g.log.gz 2024-06-09T22:35:22.083 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-09T22:35:22.090 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-06-09T22:35:22.156 INFO:teuthology.orchestra.run.smithi153.stderr: 94.6% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.e.log.gz 2024-06-09T22:35:22.309 INFO:teuthology.orchestra.run.smithi104.stderr: 90.1% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mgr.x.log.gz 2024-06-09T22:35:22.404 INFO:teuthology.orchestra.run.smithi153.stderr: 94.7% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.h.log.gz 2024-06-09T22:35:22.552 INFO:teuthology.orchestra.run.smithi153.stderr: 91.7% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mon.b.log.gz 2024-06-09T22:35:22.819 INFO:teuthology.orchestra.run.smithi194.stderr: 92.3% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mon.c.log.gz 2024-06-09T22:35:23.316 INFO:teuthology.orchestra.run.smithi104.stderr: 95.6% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.a.log.gz 2024-06-09T22:35:23.389 INFO:teuthology.orchestra.run.smithi104.stderr: 90.4% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mon.a.log.gz 2024-06-09T22:35:23.686 INFO:teuthology.orchestra.run.smithi194.stderr: 96.4% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.c.log.gz 2024-06-09T22:35:23.765 INFO:teuthology.orchestra.run.smithi194.stderr: 96.7% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.l.log.gz 2024-06-09T22:35:24.049 INFO:teuthology.orchestra.run.smithi153.stderr: 95.7% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.k.log.gz 2024-06-09T22:35:27.817 INFO:teuthology.orchestra.run.smithi153.stderr: 92.0% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.10.log.gz 2024-06-09T22:35:28.256 INFO:teuthology.orchestra.run.smithi104.stderr: 92.2% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-06-09T22:35:28.924 INFO:teuthology.orchestra.run.smithi194.stderr: 94.9% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.i.log.gz 2024-06-09T22:35:29.133 INFO:teuthology.orchestra.run.smithi194.stderr: 92.0% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.11.log.gz 2024-06-09T22:35:29.235 INFO:teuthology.orchestra.run.smithi104.stderr: 91.6% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.6.log.gz 2024-06-09T22:35:29.340 INFO:teuthology.orchestra.run.smithi104.stderr: 91.6% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-06-09T22:35:29.373 INFO:teuthology.orchestra.run.smithi153.stderr: 92.1% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-06-09T22:35:29.942 INFO:teuthology.orchestra.run.smithi153.stderr: 91.7% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-06-09T22:35:30.080 INFO:teuthology.orchestra.run.smithi104.stderr: 91.3% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.9.log.gz 2024-06-09T22:35:30.083 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-06-09T22:35:30.083 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m8.083s 2024-06-09T22:35:30.083 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m30.817s 2024-06-09T22:35:30.083 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m1.044s 2024-06-09T22:35:30.322 INFO:teuthology.orchestra.run.smithi194.stderr: 91.4% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.8.log.gz 2024-06-09T22:35:30.949 INFO:teuthology.orchestra.run.smithi153.stderr: 91.8% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.7.log.gz 2024-06-09T22:35:32.045 INFO:teuthology.orchestra.run.smithi194.stderr: 91.5% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-06-09T22:35:34.617 INFO:teuthology.orchestra.run.smithi194.stderr: 91.4% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-06-09T22:35:42.503 INFO:teuthology.orchestra.run.smithi153.stderr: 94.0% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.b.log.gz 2024-06-09T22:35:42.505 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-06-09T22:35:42.505 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m20.505s 2024-06-09T22:35:42.505 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m51.393s 2024-06-09T22:35:42.505 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m1.649s 2024-06-09T22:36:11.170 INFO:teuthology.orchestra.run.smithi194.stderr: 93.6% -- replaced with /var/log/ceph/a948c0ca-26ac-11ef-bc9b-c7b262605968/ceph-mds.f.log.gz 2024-06-09T22:36:11.171 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-06-09T22:36:11.172 INFO:teuthology.orchestra.run.smithi194.stderr:real 0m49.169s 2024-06-09T22:36:11.172 INFO:teuthology.orchestra.run.smithi194.stderr:user 1m34.429s 2024-06-09T22:36:11.172 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m2.953s 2024-06-09T22:36:11.172 INFO:tasks.cephadm:Archiving logs... 2024-06-09T22:36:11.173 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213/remote/smithi104/log 2024-06-09T22:36:11.174 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-09T22:36:15.836 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213/remote/smithi153/log 2024-06-09T22:36:15.837 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-09T22:36:22.683 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213/remote/smithi194/log 2024-06-09T22:36:22.684 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-09T22:36:35.559 INFO:tasks.cephadm:Removing cluster... 2024-06-09T22:36:35.560 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 --force 2024-06-09T22:36:35.750 INFO:teuthology.orchestra.run.smithi104.stdout:Deleting cluster with fsid: a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:36:36.071 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 --force 2024-06-09T22:36:36.266 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:36:36.653 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a948c0ca-26ac-11ef-bc9b-c7b262605968 --force 2024-06-09T22:36:36.843 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: a948c0ca-26ac-11ef-bc9b-c7b262605968 2024-06-09T22:36:37.393 INFO:tasks.cephadm:Removing cephadm ... 2024-06-09T22:36:37.393 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-09T22:36:37.410 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-09T22:36:37.427 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-09T22:36:37.442 INFO:tasks.cephadm:Teardown complete 2024-06-09T22:36:37.443 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-09T22:36:37.454 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-09T22:36:37.454 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-09T22:36:37.457 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-09T22:36:37.470 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-09T22:36:37.581 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-09T22:36:37.592 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-09T22:36:37.592 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-09T22:36:37.594 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-09T22:36:37.597 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-09T22:36:37.606 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-06-09T22:36:37.609 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-06-09T22:36:37.610 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-06-09T22:36:37.611 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-09T22:36:37.612 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-06-09T22:36:37.612 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 +477us[ +491us] +/- 58ms 2024-06-09T22:36:37.612 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 43 -738us[ -738us] +/- 65ms 2024-06-09T22:36:37.612 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +272us[ +272us] +/- 74ms 2024-06-09T22:36:37.612 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:36:37.613 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-09T22:36:37.613 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-06-09T22:36:37.613 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv01.front.sepia.ceph.com 3 6 377 46 +470us[ +481us] +/- 58ms 2024-06-09T22:36:37.614 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 47 -740us[ -729us] +/- 65ms 2024-06-09T22:36:37.614 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +267us[ +278us] +/- 74ms 2024-06-09T22:36:37.614 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:36:37.615 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-09T22:36:37.615 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-06-09T22:36:37.616 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 +445us[ +458us] +/- 58ms 2024-06-09T22:36:37.616 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 46 -755us[ -742us] +/- 65ms 2024-06-09T22:36:37.616 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +256us[ +269us] +/- 74ms 2024-06-09T22:36:37.616 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:36:37.617 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-09T22:36:37.627 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-09T22:36:37.628 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-09T22:36:37.637 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-09T22:36:37.736 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-09T22:36:37.806 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-09T22:36:37.873 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-09T22:36:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:36:37.902 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-09T22:36:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:36:37.931 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-09T22:36:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:36:37.957 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-09T22:36:37.968 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-09T22:36:37.979 INFO:teuthology.task.internal:Duration was 2150.142024 seconds 2024-06-09T22:36:37.980 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-09T22:36:37.991 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-09T22:36:37.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-09T22:36:37.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-09T22:36:37.996 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-09T22:36:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-09T22:36:38.033 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-09T22:36:38.033 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-09T22:36:38.545 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-09T22:36:38.546 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-06-09T22:36:38.546 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-09T22:36:38.567 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-06-09T22:36:38.567 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-09T22:36:38.608 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-06-09T22:36:38.608 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-09T22:36:38.647 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-09T22:36:38.648 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-09T22:36:38.650 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-09T22:36:38.653 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-09T22:36:38.715 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-09T22:36:38.715 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-09T22:36:38.717 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-09T22:36:38.721 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-09T22:36:39.925 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-09T22:36:39.938 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-09T22:36:39.938 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-09T22:36:39.966 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-09T22:36:39.995 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-09T22:36:40.023 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-09T22:36:40.035 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-09T22:36:40.037 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-09T22:36:40.040 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-09T22:36:40.057 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-06-09T22:36:40.063 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-06-09T22:36:40.089 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-06-09T22:36:40.342 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-09T22:36:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:36:40.358 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-09T22:36:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:36:40.399 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-09T22:36:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:36:40.417 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-09T22:36:40.429 INFO:teuthology.task.internal:Transferring archived files... 2024-06-09T22:36:40.429 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213/remote/smithi104 2024-06-09T22:36:40.430 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-09T22:36:40.492 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213/remote/smithi153 2024-06-09T22:36:40.492 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-09T22:36:40.531 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747213/remote/smithi194 2024-06-09T22:36:40.532 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-09T22:36:40.570 INFO:teuthology.task.internal:Removing archive directory... 2024-06-09T22:36:40.570 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-09T22:36:40.573 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-09T22:36:40.575 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-09T22:36:40.622 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-09T22:36:40.636 INFO:teuthology.task.internal:Not uploading archives. 2024-06-09T22:36:40.636 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-09T22:36:40.648 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-09T22:36:40.648 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-09T22:36:40.650 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-09T22:36:40.653 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-09T22:36:40.663 INFO:teuthology.orchestra.run.smithi104.stdout: 265314 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 9 22:36 /home/ubuntu/cephtest 2024-06-09T22:36:40.667 INFO:teuthology.orchestra.run.smithi153.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 9 22:36 /home/ubuntu/cephtest 2024-06-09T22:36:40.680 INFO:teuthology.orchestra.run.smithi194.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 9 22:36 /home/ubuntu/cephtest 2024-06-09T22:36:40.681 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-09T22:36:40.693 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-09T22:36:40.722 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 2150.142023563385 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi104 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7b7a5d375cf4e4afab96b9d4d229b699ae8304bb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=97aa6dc78f564db98a050a8253e0ba2a status: fail success: false 2024-06-09T22:36:40.722 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-09T22:36:40.815 INFO:teuthology.run:FAIL